Category: Uncategorized

Introduction to Jenkins

Overview of CI/CD

Definition and Importance:

• Continuous Integration (CI) and Continuous Delivery (CD) are foundational practices in modern software development that aim to improve software delivery speed and quality.
• CI is the practice of automating the integration of code changes from multiple contributors into a single software project. It involves automated testing to detect integration errors as quickly as possible.
• CD extends CI by automating the delivery of applications to selected infrastructure environments. It ensures that the software can be reliably released at any time.

Continuous Integration vs. Continuous Delivery vs. Continuous Deployment:

• Continuous Integration: Developers frequently merge their code changes into a central repository, after which automated builds and tests are run.
• Continuous Delivery: This is an extension of CI, where the software release process is automated. This ensures that the software can be released to production at any time with the push of a button.
• Continuous Deployment: A step beyond Continuous Delivery. Every change that passes all stages of the production pipeline is released to customers. There’s no human intervention, and only a failed test will prevent a new change to be deployed to production.

Introduction to Jenkins

History and Evolution:

• Jenkins was originally developed as the Hudson project in 2004 by Kohsuke Kawaguchi, a Sun Microsystems employee.
• It was renamed Jenkins in 2011 after a dispute with Oracle, which had acquired Sun Microsystems.
• Jenkins has evolved to become one of the most popular automation servers, with a strong community and a vast plugin ecosystem.

Jenkins in the DevOps Culture:

• Jenkins plays a pivotal role in DevOps by providing a robust platform for automating the various stages of the DevOps pipeline.
• It bridges the gap between software development and IT operations, enabling faster and more efficient delivery of software.

Key Features and Benefits:

• Extensibility: Jenkins can be extended via its vast plugin ecosystem, making it adaptable to almost any tool or technology in the CI/CD pipeline.
• Flexibility: It supports various SCM tools like Git, SVN, and Mercurial and can integrate with numerous testing and deployment technologies.
• Ease of Use: Jenkins is relatively easy to set up and configure, and it offers a user-friendly web interface for managing the CI/CD process.
• Distributed Nature: Jenkins can distribute work across multiple machines for faster builds, tests, and deployments.
• Rich Community: Being open-source, Jenkins has a large and active community, providing a wealth of plugins and shared knowledge.

Examples:

• A typical Jenkins CI pipeline includes pulling code from a Git repository, building the code using a tool like Maven or Gradle, running tests, and then packaging the application for deployment.
• In a CD setup, Jenkins could further automate the deployment of the built application to a staging server, run additional tests, and prepare it for production deployment.

Setting Up Jenkins

Installation and Configuration

System Requirements:

• Java: Jenkins requires Java (JRE or JDK) to run. The recommended version is Java 11, but it also supports Java 8.
• Memory: Minimum of 256 MB of heap space and 1 GB of RAM.
• Disk Space: At least 10 GB of disk space for Jenkins and additional space for builds and jobs.
• Web Browser: A modern web browser for accessing the Jenkins web interface.

Installing Jenkins on Various Platforms:

1. Windows:

• Download the Jenkins Windows installer from the Jenkins website.
• Run the installer and follow the on-screen instructions.
• Jenkins will be installed as a Windows service.

1. Linux:

• Jenkins can be installed on Linux using package managers like apt (for Ubuntu/Debian) or yum (for Red Hat/CentOS).
• Example for Ubuntu:
  bash wget -q -O - https://pkg.jenkins.io/debian/jenkins.io.key | sudo apt-key add - sudo sh -c 'echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list' sudo apt-get update sudo apt-get install jenkins
• Jenkins will start as a daemon on Linux.

1. macOS:

• The easiest way is to use Homebrew:
  bash brew install jenkins-lts
• Start Jenkins using:
  bash brew services start jenkins-lts

Initial Setup and Configuration:

• After installation, open your browser and go to http://localhost:8080.
• The first time you access Jenkins, it will ask for an initial admin password, which can be found in a file specified in the console output.
• After entering the password, you’ll be prompted to install suggested plugins or select specific plugins.
• Create an admin user and configure the Jenkins instance.

Jenkins Dashboard

Navigating the Interface:

• The Jenkins dashboard is the central point for managing Jenkins.
• It displays a summary of Jenkins jobs, including the status of recent builds.
• The left-hand side menu provides options for managing Jenkins, including creating new jobs, managing users, and system configuration.

Basic Configuration Options:

• Manage Jenkins: This section allows you to configure system settings, manage plugins, and set up global security.
• Creating Jobs: From the dashboard, you can create new Jenkins jobs by selecting “New Item.”
• System Configuration: Here, you can configure system-level settings like JDK installations, Maven configurations, and environment variables.
• Security Configuration: In “Configure Global Security,” you can set up authentication methods, authorize users, and configure security realms.

Examples:

• Creating a Freestyle Job:
• Go to the Jenkins dashboard.
• Click on “New Item.”
• Enter a name for the job, select “Freestyle project,” and click OK.
• Configure the job by specifying source code management, build triggers, and build steps.
• Save the job and run it to see the results.
• Setting Up a Maven Project:
• From the dashboard, create a new item and select “Maven project.”
• Provide the details of your Maven project, including repository URL and build goals.
• Jenkins will build the Maven project based on the provided POM file and goals.

Jenkins Jobs and Builds

Creating Jobs

Job Types in Jenkins:

1. Freestyle Project: The most flexible and easy-to-use type. Suitable for most use cases.
2. Maven Project: Optimized for projects built with Apache Maven. It uses information from the POM file.
3. Pipeline: For complex pipelines (as code), typically using a Jenkinsfile. Allows for implementing sophisticated CI/CD workflows.
4. Multibranch Pipeline: Automatically creates a pipeline for each branch in your source control.
5. External Job: Monitor executions run outside of Jenkins.

Configuring Source Code Management (Git, SVN):

• Jenkins can integrate with various SCM tools like Git, Subversion (SVN), Mercurial, etc.
• Git Example:
  • In the job configuration, select “Git” in the Source Code Management section.
  • Enter the Repository URL (e.g., https://github.com/user/repo.git).
  • Add credentials if the repository is private.
  • Optionally specify branches to build.
• SVN Example:
  • Select “Subversion” in the Source Code Management section.
  • Enter the Repository URL (e.g., http://svn.example.com/project).
  • Configure credentials and additional options as needed.

Build Triggers and Scheduling:

• Trigger Types:
  • Poll SCM: Checks the SCM for changes at specified intervals.
  • Build after other projects are built: Triggers a build after the completion of a specified project.
  • Build periodically: Schedule at specific intervals (e.g., H/15 * * * * for every 15 minutes).
  • GitHub hook trigger for GITScm polling: Triggers a build when a change is pushed to GitHub (requires webhook configuration in GitHub).
  • Example: To build every night at 2 AM, use 0 2 * * * in “Build periodically.”

Build Process

Understanding Build Steps:

• Build steps are actions to execute during the build process.
• Common steps include executing shell scripts or batch commands, invoking build tools like Maven or Gradle, running tests, etc.
• Example: A simple shell script step could be echo "Building project" for a Linux-based system or a batch command like echo Building project on Windows.

Build Environment Configuration:

• In the job configuration, you can set various environment options like:
  • Delete workspace before build starts: To ensure a clean environment for each build.
  • Use secret text(s) or file(s): For handling credentials.
  • Set environment variables: To define or override environment variables for the build.

Post-build Actions:

• Actions to perform after a build is completed.
• Common actions include:
  • Archiving artifacts: Save build outputs for later use.
  • Publishing JUnit test results: Process and display test results.
  • Sending email notifications: Notify team members of build results.
  • Deploying to a server: Automatically deploy successful builds.
  • Example: To archive all jar files produced in a build, use **/*.jar in “Archive the artifacts.”

Jenkins Pipeline

Pipeline as Code

Concept:

• Pipeline as Code refers to defining the deployment pipeline through code, rather than manual job creation in Jenkins.
• This is typically done using a Jenkinsfile, which is a text file that contains the definition of a Jenkins Pipeline and is checked into source control.

Advantages:

• Version Control: Pipelines can be versioned and reviewed like any other code.
• Reusability: Pipelines can be shared across different projects.
• Consistency: Ensures consistency in the build process across environments.

Example of a Jenkinsfile:

pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                echo 'Building..'
                // Add build steps here
            }
        }
        stage('Test') {
            steps {
                echo 'Testing..'
                // Add test steps here
            }
        }
        stage('Deploy') {
            steps {
                echo 'Deploying..'
                // Add deployment steps here
            }
        }
    }
}

Creating and Managing Pipelines

Creating a Pipeline:

1. Using a Jenkinsfile:

• Create a Jenkinsfile in your SCM repository.
• In Jenkins, create a new item and select “Pipeline.”
• In the Pipeline section, specify the SCM and the path to the Jenkinsfile.

1. Directly in Jenkins:

• Create a new Pipeline item in Jenkins.
• Directly write or paste the pipeline script in the Pipeline section.

Managing Pipelines:

• Pipelines are managed in Jenkins just like any other job.
• They can be triggered manually, by SCM commits, or on a schedule.
• Jenkins provides visualization of pipeline stages and progress.

Scripted vs. Declarative Pipelines

Scripted Pipeline:

• Definition: Uses a more traditional Groovy syntax. Offers more flexibility and control.
• Syntax: Written in Groovy-based DSL.
• Control Structures: Allows complex logic, loops, and conditionals.
• Example:

  node {
      stage('Build') {
          echo 'Building..'
          // Build steps
      }
      stage('Test') {
          echo 'Testing..'
          // Test steps
      }
      stage('Deploy') {
          echo 'Deploying..'
          // Deploy steps
      }
  }

Declarative Pipeline:

• Definition: Introduced for a simpler and more opinionated syntax for authoring Jenkins Pipeline.
• Syntax: More straightforward and easier to read.
• Structure: Has a predefined structure and sections.
• Example: (Same as the example provided in the Pipeline as Code section).

Key Differences:

• Flexibility: Scripted pipelines offer more flexibility and control but are more complex.
• Ease of Use: Declarative pipelines are easier to write and understand, especially for beginners.
• Syntax: Scripted pipelines use a Groovy-based DSL, while Declarative pipelines have a more structured and pre-defined format.

Managing Plugins in Jenkins

Finding and Installing Plugins

Finding Plugins:

• Jenkins Plugin Manager: The primary method to find plugins is through the Jenkins Plugin Manager in the Jenkins web interface.
• Jenkins Plugin Site: The Jenkins Plugin Site is also a valuable resource for exploring available plugins, where you can search and read documentation.

Installing Plugins:

1. Via Jenkins Web Interface:

• Navigate to Manage Jenkins > Manage Plugins.
• Switch to the Available tab to browse or search for plugins.
• Select the desired plugin(s) and click Install without restart or Download now and install after restart.
• Jenkins will download and install the plugin(s).

1. Manual Installation:

• If a plugin is not available in the Plugin Manager, it can be manually downloaded from the Jenkins Plugin Site and uploaded.
• Navigate to Manage Jenkins > Manage Plugins > Advanced tab.
• Under Upload Plugin, choose the .hpi file and click Upload.

Example:

• Installing the Git Plugin:
• Go to Manage Plugins.
• In the Available tab, search for “Git plugin.”
• Select it and click Install without restart.
• Jenkins will install the plugin and may require a restart.

Plugin Configuration and Management

Configuring Plugins:

• After installation, many plugins require configuration.
• Configuration can typically be done through Manage Jenkins > Configure System or a specific section in the Jenkins dashboard.
• For example, the Git plugin requires setting up Git installations and global configurations.

Managing Existing Plugins:

• Updating Plugins:
• Regularly update plugins for new features and security fixes.
• Go to Manage Plugins > Updates tab to see available updates.
• Disabling/Enabling Plugins:
• Plugins can be disabled without uninstalling them.
• Navigate to Manage Plugins > Installed tab, and use the Enable/Disable button as needed.
• Uninstalling Plugins:
• If a plugin is no longer needed, it can be uninstalled.
• In the Installed tab, select the plugin and click Uninstall.

Example:

• Configuring the Mailer Plugin:
• After installing the Mailer plugin, go to Manage Jenkins > Configure System.
• Scroll to the E-mail Notification section.
• Enter your SMTP server details and email address.
• Save the configuration.

Distributed Builds in Jenkins

Master-Slave Architecture

Concept:

• Jenkins uses a Master-Slave architecture to manage distributed builds.
• The Master is the main Jenkins server, responsible for scheduling builds, dispatching jobs to nodes (slaves), and monitoring them.
• Slaves (or Nodes) are servers where the actual job execution takes place.

Advantages:

• Scalability: Distributes workload across multiple machines, improving build times.
• Flexibility: Different jobs can be run in different environments.
• Resource Optimization: Utilizes various hardware and software configurations as needed.

Reference:

• Distributed builds – Jenkins

Configuring and Managing Nodes

Setting Up a Slave Node:

1. Adding a Node:

• In Jenkins, navigate to Manage Jenkins > Manage Nodes and Clouds.
• Click on New Node, enter a name, select Permanent Agent, and click OK.
• Configure the node details (remote root directory, labels, usage, launch method, etc.).

1. Launch Methods:

• SSH: Connects to the slave via SSH. Requires Java on the slave machine.
• JNLP (Java Web Start): The slave connects to the master using a JNLP agent.
• Windows agents: Can be connected using Windows-specific methods like DCOM.

Managing Nodes:

• Monitoring: The master provides a monitoring view for all nodes, showing their status and workload.
• Configuring Executors: Executors are individual build slots on a node. The number of executors can be configured based on the node’s capacity.
• Maintaining Nodes: Nodes can be temporarily taken offline for maintenance or permanently removed.

Example:

• Configuring a Linux Node via SSH:
• Add a new node as described above.
• In the Launch method, select Launch agents via SSH.
• Enter the host IP, credentials, and other SSH settings.
• Save and Jenkins will try to establish a connection to the node.

Reference:

• Adding agents – Jenkins

Jenkins Security

Access Control

User Authentication and Authorization:

• Objective: Ensure that only authorized users can access Jenkins and perform specific tasks.
• Process:
• Jenkins supports various authentication methods like LDAP, Active Directory, and internal Jenkins user database.
• Authorization strategies define what authenticated users are allowed to do. Common strategies include Matrix-based security and Project-based Matrix Authorization.
• Example:
• Configuring LDAP authentication:
  • Navigate to Manage Jenkins > Configure Global Security.
  • Select LDAP in the Security Realm section and enter LDAP server details.

Role-Based Access Control (RBAC)

Concept:

• RBAC in Jenkins allows fine-grained access control based on roles assigned to users or groups.
• Roles can be defined globally or per project, with specific permissions.

Implementation:

• Install the Role-based Authorization Strategy plugin.
• Define roles in Manage and Assign Roles under Manage Jenkins.
• Assign roles to users or groups with specific permissions.

Reference:

• Role-based Authorization Strategy – Jenkins

Securing Jenkins

Best Practices for Jenkins Security:

• Regular Updates: Keep Jenkins and its plugins updated to the latest versions.
• Secure Configuration: Follow the principle of least privilege. Limit permissions and access to what is necessary.
• Use HTTPS: Configure Jenkins to use HTTPS for secure communication.
• Audit Logs: Enable and monitor audit logs to track changes and actions in Jenkins.
• Firewall Configuration: Restrict access to Jenkins servers using firewalls.

Managing Credentials:

• Objective: Securely store and manage credentials used in Jenkins jobs.
• Process:
• Use the Credentials Plugin to store credentials securely in Jenkins.
• Credentials can be scoped globally or to specific Jenkins items.
• Supports various credential types like username/password, SSH keys, and secret text.
• Example:
• Adding SSH credentials:
  • Navigate to Credentials > System > Global credentials > Add Credentials.
  • Select SSH Username with private key and enter the required details.
• Reference:
• Credentials Plugin – Jenkins

1. Introduction to Maven

• What Maven is and its role in software development.
• Brief history and comparison with tools like Ant and Gradle.

2. Maven Basics

• Installation and basic setup.
• Key concepts: Project Object Model (POM), lifecycles, dependencies, and repositories.

3. Project Configuration

• Understanding and setting up the POM file.
• Managing project dependencies.

4. Maven Build Lifecycle

• Overview of Maven’s standard build phases.
• Customizing build processes.

5. Repositories in Maven

• Types: local, central, and remote.
• Managing and configuring repositories.

6. Multi-Module Projects

• Structuring and managing larger projects with multiple modules.

7. Dependency Management

• Handling dependency conflicts and complex scenarios.

8. Maven Plugins

• Using and creating plugins for custom functionality.

9. Integration and Optimization

• Integrating Maven with IDEs and CI/CD tools.
• Tips for optimizing Maven builds.

Introduction to Maven

What is Maven?

• Definition: Apache Maven is a powerful project management and comprehension tool used primarily for Java projects. It is based on the concept of a project object model (POM) and can manage a project’s build, reporting, and documentation from a central piece of information.
• Role in Software Development:
  • Build Automation: Automates the process of building software, including compiling source code, packaging binary code, and running tests.
  • Dependency Management: Manages libraries and other dependencies a project needs, automatically downloading and integrating them from a central repository.
  • Standardization: Provides a uniform build system, so developers only need to learn Maven to work on different Maven projects.

Brief History

• Origins: Maven was created by Jason van Zyl in 2002 as part of the Apache Turbine project. It was a response to the need for a more standardized and flexible project building tool.
• Evolution: Over the years, Maven has evolved, with the release of Maven 2 in 2005 introducing significant changes in its build process and dependency management. Maven 3, released in 2010, brought further improvements in performance and configuration.

Comparison with Ant and Gradle

• Maven vs. Ant:
  • Ant: An older build tool, primarily focused on building Java applications. It uses XML for configuration and is more procedural, requiring explicit instructions for each build step.
  • Maven: Focuses on convention over configuration, providing a standardized build process with less need for detailed scripting. It’s more about describing the desired end state rather than the steps to get there.
  • Example: In Maven, compiling a Java project is a matter of defining the project structure according to Maven’s standards. In Ant, each step (like source code compilation, testing, packaging) must be explicitly defined in the build script.
• Maven vs. Gradle:
  • Gradle: A newer tool that combines the strengths of both Maven and Ant. It uses a domain-specific language based on Groovy, offering more powerful scripting capabilities than Maven.
  • Maven: Known for its simplicity and ease of use, especially in projects that fit well into its conventional structure. However, it can be less flexible than Gradle in handling non-standard project layouts.
  • Example: Dependency management in Gradle can be more customizable and can handle scenarios that Maven might struggle with, such as dynamic versioning.

Maven Basics

Installation and Basic Setup

• Installation:
  • Prerequisites: Java Development Kit (JDK) must be installed.
  • Steps: Download Maven from the Apache website and extract it to your chosen directory. Add the bin directory of the extracted Maven to the PATH environment variable.
  • Verification: Run mvn -v in the command line to verify the installation.

Key Concepts

1. Project Object Model (POM):

• Definition: POM is an XML file (pom.xml) in a Maven project that contains information about the project and configuration details used by Maven to build the project.
• Components: Includes project dependencies, plugins, goals, build profiles, and project metadata like version, description, and developers.

1. Lifecycles:

• Explanation: Maven is based on a lifecycle to handle project building and management. The primary lifecycles are default (handling project deployment), clean (cleaning the project), and site (creating the project’s site documentation).
• Phases: Examples include compile, test, package, and install.

1. Dependencies and Repositories:

• Dependencies: Libraries or modules that a project needs to function.
• Repositories: Places where dependencies are stored. Maven can retrieve dependencies from local (on your machine), central (default Maven repository), or remote (custom or third-party) repositories.

Project Configuration

1. Setting Up the POM File:

• Basic Structure:
  xml <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.example</groupId> <artifactId>my-app</artifactId> <version>1.0-SNAPSHOT</version> </project>
• Explanation: groupId identifies your project uniquely across all projects, artifactId is the name of the jar without version, and version is the version of the artifact.

1. Managing Project Dependencies:

• Adding a Dependency: Dependencies are added in the <dependencies> section of the pom.xml.
• Example:
  xml <dependencies> <dependency> <groupId>org.apache.commons</groupId> <artifactId>commons-lang3</artifactId> <version>3.10</version> </dependency> </dependencies>
• Explanation: This example adds Apache Commons Lang, which provides extra functionality for classes in java.lang.

Maven Build Lifecycle

Overview of Maven’s Standard Build Phases

Maven’s build lifecycle is a sequence of phases that define the order in which goals are executed. Here are the key phases:

1. validate: Checks if all necessary information is available.
2. compile: Compiles the source code of the project.
3. test: Tests the compiled source code using a suitable unit testing framework.
4. package: Packages the compiled code in its distributable format, such as a JAR.
5. verify: Runs any checks to validate the package is valid and meets quality criteria.
6. install: Installs the package into the local repository, for use as a dependency in other projects locally.
7. deploy: Copies the final package to the remote repository for sharing with other developers and projects.

Customizing Build Processes

• Custom Phases and Goals: You can customize the build process by adding or configuring goals in your pom.xml.
• Example: Binding a custom plugin goal to a lifecycle phase.

  <build>
      <plugins>
          <plugin>
              <groupId>org.apache.maven.plugins</groupId>
              <artifactId>maven-antrun-plugin</artifactId>
              <version>1.8</version>
              <executions>
                  <execution>
                      <phase>compile</phase>
                      <goals>
                          <goal>run</goal>
                      </goals>
                      <configuration>
                          <!-- Custom configuration here -->
                      </configuration>
                  </execution>
              </executions>
          </plugin>
      </plugins>
  </build>

Repositories in Maven

Types of Repositories

1. Local Repository: A local machine’s cache of the artifacts downloaded from central or remote repositories. It can also contain projects built locally.
2. Central Repository: The default repository provided by Maven. It contains a large number of commonly used libraries.
3. Remote Repository: Any other repository accessed over a network, which can be a private or third-party repository.

Managing and Configuring Repositories

• Configuring a Repository in pom.xml:
  • Example: Adding a remote repository.
    xml <repositories> <repository> <id>my-remote-repo</id> <url>http://repo.mycompany.com/maven2</url> </repository> </repositories>
• Using a Mirror:
  • Purpose: Mirrors can be used to redirect requests to a central repository to another location.
  • Example: Configuring a mirror in settings.xml.
    xml <mirrors> <mirror> <id>mirrorId</id> <mirrorOf>central</mirrorOf> <name>Human Readable Name for this Mirror.</name> <url>http://my.repository.com/repo/path</url> </mirror> </mirrors>

Multi-Module Projects

Structuring and Managing Larger Projects with Multiple Modules

• Overview: In Maven, a multi-module project is a structure that allows you to manage several modules (or sub-projects) in a single project. Each module is a separate project, but they are all built together.
• Example:
  • Parent POM (pom.xml):
    xml <groupId>com.example</groupId> <artifactId>multi-module-project</artifactId> <version>1.0</version> <packaging>pom</packaging> <modules> <module>module1</module> <module>module2</module> </modules>
  • Module POM (module1/pom.xml):
    xml <parent> <groupId>com.example</groupId> <artifactId>multi-module-project</artifactId> <version>1.0</version> </parent> <artifactId>module1</artifactId>

Dependency Management

Handling Dependency Conflicts and Complex Scenarios

• Dependency Conflicts: Occur when different modules or libraries require different versions of the same dependency.
• Example: Using <dependencyManagement> in the parent POM to manage versions.

  <dependencyManagement>
      <dependencies>
          <dependency>
              <groupId>org.apache.commons</groupId>
              <artifactId>commons-lang3</artifactId>
              <version>3.10</version>
          </dependency>
      </dependencies>
  </dependencyManagement>

Maven Plugins

Using and Creating Plugins for Custom Functionality

• Using Plugins: Plugins extend Maven’s capabilities and can be used for tasks like code generation, testing, and packaging.
• Creating Plugins: Involves writing a Maven plugin in Java and configuring it in your POM.
• Example: Adding a plugin to a POM.

  <build>
      <plugins>
          <plugin>
              <groupId>org.apache.maven.plugins</groupId>
              <artifactId>maven-compiler-plugin</artifactId>
              <version>3.8.1</version>
              <configuration>
                  <source>1.8</source>
                  <target>1.8</target>
              </configuration>
          </plugin>
      </plugins>
  </build>

Integration and Optimization

Integrating Maven with IDEs and CI/CD Tools

• IDE Integration: Most modern IDEs like Eclipse or IntelliJ IDEA have built-in support for Maven. They can automatically detect pom.xml and manage dependencies.
• CI/CD Integration: Maven integrates well with CI/CD tools like Jenkins, allowing automated builds and deployments.

Tips for Optimizing Maven Builds

• Dependency Management: Keep your dependencies up to date and remove unused ones.
• Maven Profiles: Use profiles for different build environments.
• Incremental Builds: Leverage Maven’s incremental build features to avoid rebuilding unchanged modules.
• Parallel Builds: Use Maven’s parallel build option (-T option) to speed up the build process.

1. Introduction to Linux

• History and Philosophy
• Linux Distributions
• Open Source Licensing

2. Getting Started with Linux

• Installing Linux
• Basic Linux Commands
• Navigating the File System
• File and Directory Operations

3. System Administration

• User and Group Management
• File Permissions and Ownership
• System Monitoring
• Installing and Updating Software

4. Text Editors and File Processing

• Using Editors (Vi, Nano, Emacs)
• Text Processing (grep, sed, awk)
• Shell Scripting

5. Command Line Proficiency

• Advanced Bash Scripting
• Advanced File Operations
• Process Management
• Networking Commands

6. System Configuration and Management

• System Services and Daemons
• System Logs and Journaling
• Task Scheduling (cron, at)

7. Networking and Security

• Network Configuration
• Firewall Management (iptables, ufw)
• SSH for Remote Access
• Security Best Practices

8. Kernel and System Optimization

• Understanding the Linux Kernel
• Kernel Modules and Parameters
• Performance Tuning

9. Storage and File Systems

• Disk Partitioning
• File System Types and Management
• Logical Volume Management (LVM)
• Network File Systems (NFS, Samba)

10. Advanced Networking
    • Network Troubleshooting
    • VPNs and Routing
    • Network File Systems and Storage Solutions
11. Server Management and Automation
    • Web Server Setup (Apache, Nginx)
    • Database Management (MySQL, PostgreSQL)
    • Automation Tools (Ansible, Puppet)
12. Real-Time Commands and Tools
    • Monitoring and Diagnostics: top, htop, netstat, ss, dmesg, iotop
    • System Information: uname, lscpu, lsblk, df, du
    • Networking: ifconfig, ip, traceroute, ping, nmap
    • Security and Auditing: journalctl, auditd, fail2ban

1. Introduction to Linux

History and Philosophy

Origins: Linux was created by Linus Torvalds in 1991. It was developed as a free and open-source alternative to the UNIX operating system.

Philosophy: The core philosophy of Linux is centered around freedom and collaboration. It’s built on the principles of open-source software, where the source code is freely available for anyone to view, modify, and distribute.

Growth and Community: Over the years, Linux has grown significantly, supported by a large community of developers and users. It’s known for its stability, security, and flexibility.

Linux Distributions

• Definition: A Linux distribution (often called a distro) is an operating system made from a software collection based on the Linux kernel and often a package management system.
• Popular Distributions:
• Debian: Known for its stability and the basis for many other distributions like Ubuntu.
• Ubuntu: Popular in both desktop and server environments, known for its user-friendliness.
• Fedora: Features cutting-edge technology and innovations; a community version of Red Hat Enterprise Linux.
• Red Hat Enterprise Linux (RHEL): Widely used in enterprise environments, known for its robustness and support.
• CentOS: A free version of RHEL, known for its enterprise-oriented features.
• Arch Linux: Known for its simplicity and customization.
• Choosing a Distribution: The choice depends on the user’s needs – stability, support, cutting-edge features, or simplicity.

Open Source Licensing

• Definition: Open source licensing allows software to be freely used, modified, and shared.

• Types of Licenses:
  • GNU General Public License (GPL): Used by the Linux kernel, it requires that modified versions also be open source.
  • Apache License: Allows modification and distribution of the software for any purpose, without the requirement for modified versions to be open source.
  • MIT License: A permissive license with minimal restrictions on reuse.

• Impact on Development: Open source licensing has led to collaborative development, where a global community contributes to software projects.

• Benefits: Promotes innovation, ensures security (through transparency), and fosters a community-driven approach to software development.

2. Getting Started with Linux

Installing Linux

• Choosing a Distribution: Select a Linux distribution based on your needs. Popular choices for beginners include Ubuntu, Fedora, and Linux Mint.
• Installation Process:
  • Download ISO: Obtain the ISO file from the distribution’s website.
  • Create Bootable USB: Use tools like Rufus or Etcher to create a bootable USB drive.
  • Boot from USB: Restart your computer and boot from the USB drive.
  • Installation Wizard: Follow the on-screen instructions to complete the installation. This typically includes setting language, time zone, keyboard layout, disk partitioning, and user account creation.

Basic Linux Commands

• pwd: Print the current working directory.
• ls: List files and directories in the current directory.
• cd: Change the current directory.
• touch: Create a new empty file.
• cp: Copy files and directories.
• mv: Move or rename files and directories.
• rm: Remove files and directories.
• cat: Concatenate and display file content.
• echo: Display a line of text/string that is passed as an argument.
• man: Display the user manual of any command.

Navigating the File System

• File System Structure:
  • /: Root directory.
  • /home: Home directories for users.
  • /etc: Configuration files.
  • /var: Variable files like logs.
  • /usr: User binaries and software.

• Navigation Commands:
  • cd: Change directory (e.g., cd /home/user).
  • cd ..: Move up one directory.
  • cd ~ or cd: Go to the home directory.

File and Directory Operations

• Creating Files and Directories:
  • touch filename: Create a new file.
  • mkdir directoryname: Create a new directory.

• Copying and Moving:
  • cp source destination: Copy files or directories.
  • mv source destination: Move/rename files or directories.

• Deleting:
  • rm filename: Delete a file.
  • rm -r directoryname: Recursively delete a directory and its contents.

• Viewing and Editing Files:
  • cat filename: View the content of a file.
  • nano filename or vi filename: Edit a file using Nano or Vi editor.

• File Permissions:
  • chmod: Change file mode bits.
  • chown: Change file owner and group.
  • ls -l: List files with permissions, ownership, and size.

3. System Administration in Linux

User and Group Management

• Users and Groups: In Linux, users are the accounts that can access the system, while groups are collections of users.

• Managing Users:
  • useradd: Create a new user (e.g., useradd username).
  • usermod: Modify a user account (e.g., usermod -aG groupname username to add a user to a group).
  • userdel: Delete a user account (e.g., userdel username).

• Managing Groups:
  • groupadd: Create a new group (e.g., groupadd groupname).
  • groupdel: Delete a group (e.g., groupdel groupname).
  • groups: List all groups a user belongs to.

File Permissions and Ownership

• Understanding Permissions: Linux file permissions determine who can read, write, or execute a file.

• Permission Types:
  • Read (r): View the contents of the file.
  • Write (w): Modify the file.
  • Execute (x): Run the file as a program.

• Changing Permissions:
  • chmod: Change file permissions (e.g., chmod 755 filename).

• Ownership: Files and directories are owned by users and groups.

• Changing Ownership:
  • chown: Change the owner of a file (e.g., chown username filename).
  • chgrp: Change the group of a file (e.g., chgrp groupname filename).

System Monitoring

• Monitoring Tools:
  • top/htop: View real-time system processes and resource usage.
  • df: Report file system disk space usage.
  • du: Estimate file space usage.
  • free: Display amount of free and used memory in the system.
  • iostat: Monitor system input/output device loading.

• System Logs:
  • Located in /var/log/, system logs provide a history of system activities and errors.
  • journalctl: Used to query and display messages from the journal (systemd).

Installing and Updating Software

• Package Management:
  • Debian/Ubuntu: Use apt or apt-get (e.g., apt update, apt upgrade, apt install packagename).
  • Red Hat/CentOS: Use yum or dnf (e.g., yum update, yum install packagename).

• Software Installation:
  • Install software from repositories or download and install packages manually.

• Updating System:
  • Regularly update the system to ensure security and stability.
  • Use update to refresh package index and upgrade to install available updates.

4. Text Editors and File Processing in Linux

Using Editors

Vi/Vim:

• Description: Vi (or Vim, which is Vi improved) is a powerful text editor with a modal interface, widely used in Unix and Linux systems.
• Basic Commands:
  • i: Enter insert mode.
  • :w: Save the file.
  • :q: Quit (add ! to force quit without saving).
  • :wq: Save and quit.

• Example: To edit a file named example.txt, use vi example.txt.

Nano:

• Description: Nano is a simple, user-friendly text editor for Unix and Linux systems.
• Usage: Commands are displayed at the bottom of the screen.
• Example: To edit a file, use nano example.txt.

Emacs:

• Description: Emacs is a highly customizable text editor with a wide range of features.
• Basic Commands:
  • Ctrl-x Ctrl-f: Open a file.
  • Ctrl-x Ctrl-s: Save a file.
  • Ctrl-x Ctrl-c: Exit Emacs.

• Example: To start Emacs, simply type emacs in the terminal.

Text Processing

• grep:
  • Usage: Search for patterns in files.
  • Example: grep 'search_term' filename – Finds ‘search_term’ in ‘filename’.

• sed:
  • Usage: Stream editor for filtering and transforming text.
  • Example: sed 's/original/replacement/' filename – Replaces the first instance of ‘original’ with ‘replacement’ in each line of ‘filename’.
• awk:
  • Usage: Programming language designed for text processing.
  • Example: awk '{print $1}' filename – Prints the first field of each line in ‘filename’.

Shell Scripting

Basics:

• Shell scripting allows for automating tasks in Unix/Linux.
• Scripts are written in plain text and can be executed.

Creating a Script:

• Start with the shebang line: #!/bin/bash.
• Write commands as you would in the shell.
• Make the script executable: chmod +x scriptname.sh.

• Example Script (backup.sh):

  #!/bin/bash
  tar -czf /backup/my_backup.tar.gz /home/user/documents

• This script creates a compressed tarball of the ‘documents’ directory.

5. Command Line Proficiency in Linux

Advanced Bash Scripting

• Concept: Bash scripting allows for automating tasks in a Linux environment using the Bash shell.
• Features:
  • Variables: Storing and using data.
  • Control Structures: if-else, for, while, case statements for decision making and looping.
  • Functions: Reusable code blocks.
  • Script Parameters: Passing arguments to scripts.
• Example Script:

  #!/bin/bash
  echo "Starting backup process..."
  tar -czf /backup/$(date +%Y%m%d)_backup.tar.gz /home/user/documents
  echo "Backup completed."

Advanced File Operations

• File Globbing: Using wildcard patterns to match file names (e.g., *.txt).

• Find Command:
  • Usage: Search for files in a directory hierarchy.
  • Example: find /home/user -name "*.txt" – Finds all .txt files in /home/user.

Sort, Cut, and Join:

• Sort: Sort lines of text files.
• Cut: Remove sections from each line of files.
• Join: Join lines of two files on a common field.
• Example: sort file.txt | cut -d':' -f2 | uniq – Sorts file.txt, cuts out the second field, and filters unique lines.

Process Management

• Viewing Processes: ps, top, htop for real-time process monitoring.
• Killing Processes:
  • kill: Send a signal to a process (e.g., kill -9 PID).
  • pkill: Kill processes by name (e.g., pkill nginx).
• Background Processes:
  • &: Run a command in the background (e.g., command &).
  • jobs: List background jobs.
  • fg: Bring a job to the foreground.

Networking Commands

• ifconfig/ip: Display or configure network interfaces.
• ping: Check connectivity to a host.
• netstat: Display network connections, routing tables, interface statistics.
• ssh: Securely connect to a remote machine.
• scp: Securely copy files between hosts.
• wget/curl: Download files from the internet.
• Example: ssh user@192.168.1.10 – Connects to a remote machine with IP 192.168.1.10.

6. System Configuration and Management in Linux

System Services and Daemons

Overview: Services and daemons are background processes that start during boot or after logging into a system.

• Managing Services:
• Systemd: The most common init system and service manager in modern Linux distributions.

• Commands:
  • systemctl start service_name: Start a service.
  • systemctl stop service_name: Stop a service.
  • systemctl restart service_name: Restart a service.
  • systemctl enable service_name: Enable a service to start on boot.
  • systemctl disable service_name: Disable a service from starting on boot.
• Example: systemctl start nginx – Starts the Nginx service.

System Logs and Journaling

System Logs:

• Location: Typically found in /var/log/.
• Common Logs:
  • /var/log/syslog or /var/log/messages: General system logs.
  • /var/log/auth.log: Authentication logs.

Journaling with Systemd:

• journalctl: A utility to query and display messages from the systemd journal.
• Example: journalctl -u nginx – Shows logs for the Nginx service.

Task Scheduling

• cron:
  • Usage: Schedule scripts or commands to run at specific times and dates.
  • Crontab File: Lists scheduled tasks (crontab -e to edit).
  • Syntax: minute hour day month day_of_week command.
  • Example: 0 5 * * * /path/to/script.sh – Runs script.sh daily at 5:00 AM.

• at:
  • Usage: Execute commands or scripts at a specific future time.
  • Command: at followed by the time for execution.
  • Example: echo "/path/to/script.sh" | at now + 5 minutes – Schedules script.sh to run 5 minutes from now.

Conclusion

Understanding system configuration and management is crucial for maintaining a Linux system. Managing system services and daemons ensures that essential processes run correctly. System logs and journaling provide valuable insights into system operations and help in troubleshooting. Task scheduling with cron and at is essential for automating routine tasks, contributing to efficient system management.

7. Networking and Security in Linux

Network Configuration

• Basics: Involves setting up IP addresses, subnet masks, and routing information.
• Tools:
  • ifconfig/ip: For viewing and configuring network interfaces.
  • nmcli (NetworkManager): A command-line tool for controlling NetworkManager.
• Example:
  • Setting a Static IP:
    bash nmcli con mod enp0s3 ipv4.addresses 192.168.1.100/24 nmcli con mod enp0s3 ipv4.gateway 192.168.1.1 nmcli con up enp0s3

Firewall Management

• iptables:
  • Description: A powerful tool for configuring the Linux kernel firewall.
  • Example: Allow HTTP traffic:
    bash iptables -A INPUT -p tcp --dport 80 -j ACCEPT
• ufw (Uncomplicated Firewall):
  • Description: A user-friendly interface for iptables.
  • Example:
    • Enable UFW: ufw enable
    • Allow SSH: ufw allow 22

SSH for Remote Access

• SSH (Secure Shell):
  • Usage: Securely access remote systems over an unsecured network.
  • Setting Up SSH:
    • Server: Install openssh-server.
    • Client: Connect using ssh user@host.
  • Example: Connect to a server:
    bash ssh user@192.168.1.10

Security Best Practices

• Regular Updates: Keep the system and all software up to date.
• User Account Management:
  • Use strong, unique passwords.
  • Implement two-factor authentication where possible.
• Firewall Configuration: Ensure only necessary ports are open.
• SSH Security:
  • Disable root login (PermitRootLogin no in sshd_config).
  • Use key-based authentication.
  • Change the default SSH port (e.g., to 2222).
• System Monitoring:
  • Regularly review system and application logs.
  • Use intrusion detection systems like fail2ban.
• Data Encryption: Use tools like gpg for encrypting files and openssl for secure communication.

Conclusion

Effective networking and security in Linux are crucial for maintaining system integrity and protecting data. This involves configuring network settings, managing firewall rules, using SSH for secure remote access, and following best practices for system security. Regular updates, strong user account management, and vigilant monitoring are key to maintaining a secure Linux environment.

8.Kernel and System Optimization in Linux

Understanding the Linux Kernel

• Overview: The kernel is the core part of Linux, responsible for managing the system’s resources and the communication between hardware and software.
• Components:
  • Process Management: Handles scheduling and execution of processes.
  • Memory Management: Manages system memory allocation.
  • Device Drivers: Interface for hardware devices.
  • System Calls: Interface between user applications and the kernel.
• Exploration:
  • uname -a: Displays kernel information.
  • lsmod: Lists loaded kernel modules.

Kernel Modules and Parameters

• Kernel Modules:
  • Description: Modules are pieces of code that can be loaded and unloaded into the kernel upon demand.
  • Management:
    • insmod: Insert a module into the kernel.
    • rmmod: Remove a module from the kernel.
    • modprobe: Add or remove modules from the kernel.
  • Example: Load a module:
    bash sudo modprobe vboxdrv
• Kernel Parameters:
  • Usage: Parameters can be used to customize the behavior of the kernel.
  • Setting Parameters:
    • Temporary: Modify at boot time in the bootloader menu.
    • Permanent: Edit /etc/sysctl.conf or files in /etc/sysctl.d/.
  • Example: Increase maximum number of open files:
    bash echo 'fs.file-max = 100000' | sudo tee -a /etc/sysctl.conf sudo sysctl -p

Performance Tuning

• Concept: Adjusting various system settings to optimize performance.
• Areas:
  • CPU Scheduling: Adjusting process priorities with nice and renice.
  • Memory Management: Tuning swap usage and memory overcommit.
  • I/O Scheduling: Adjusting I/O priorities and choosing the right I/O scheduler.
  • Network Tuning: Adjusting network parameters for better throughput.
• Tools:
  • top/htop: Monitor system performance.
  • iotop: Monitor I/O usage.
  • ifconfig/ip: Network configuration.
  • Example: Optimize network buffer sizes:

  echo 'net.core.rmem_max = 16777216' | sudo tee -a /etc/sysctl.conf
  echo 'net.core.wmem_max = 16777216' | sudo tee -a /etc/sysctl.conf
  sudo sysctl -p

Conclusion

Understanding and optimizing the Linux kernel is key to enhancing system performance. This involves managing kernel modules and parameters, as well as tuning various aspects of system behavior like CPU scheduling, memory management, I/O operations, and network settings. Regular monitoring and adjustments based on the system’s workload can lead to significant improvements in performance.

Suggested Approach for Learning

• Start with the Basics: If you’re new to Kubernetes, begin with the introduction and core concepts.
• Hands-on Practice: Parallel to learning, set up a minikube or a small cluster to experiment with or you can use Play With Kubernetes.
• Explore Advanced Topics: Once comfortable with basics, move to advanced topics like CRDs, Helm, and cluster administration.
• Apply to Use Cases: Understand how Kubernetes fits into different architectures and operational models.
• Stay Updated: Kubernetes is evolving; keep up with the latest trends and community updates.

Creating complete notes with examples and images for each topic in Kubernetes is a substantial endeavor, due to the depth and breadth of the subject. However, I can provide an overview of each section with key points and examples where applicable. For detailed notes with images, you might need to refer to comprehensive Kubernetes guides or official documentation. Let’s start with the first few topics.

1. Introduction to Kubernetes

What is Kubernetes:

• Kubernetes (K8s) is an open-source container orchestration platform designed to automate the deployment, scaling, and operation of application containers.
• It groups containers into logical units for easy management and discovery.

History and Evolution:

• Developed by Google, Kubernetes was released as an open-source project in 2014.
• It builds on 15+ years of experience running production workloads at Google.

Basic Concepts and Terminology:

• Cluster: A set of node machines for running containerized applications.
• Node: A worker machine in Kubernetes.
• Pod: The smallest deployable units that can be created, scheduled, and managed.

Kubernetes vs. Traditional Deployment:

• Traditional deployments had challenges like scalability, availability, and resource utilization.
• Kubernetes provides solutions with container orchestration.

Kubernetes vs. Other Container Orchestration Tools:

• Compared to tools like Docker Swarm and Apache Mesos, Kubernetes is more feature-rich, widely adopted, and has a strong community.
• are then applied to a Kubernetes cluster using the kubectl apply -f <filename>.yaml command.

2. Architecture

Certainly! Let’s delve into the components of both Master and Worker Nodes in a Kubernetes cluster, providing explanations and examples where applicable.

Master Node Components

1. API Server

• Explanation: The API Server acts as the front-end for Kubernetes. It validates and configures data for the API objects such as pods, services, replication controllers, etc. It provides the API for Kubernetes, allowing different tools and libraries to interact with the cluster.
• Example: When you run kubectl commands, these are processed by the API Server. It takes the command, processes it, and updates the etcd store with the state of the Kubernetes objects.

2. etcd

• Explanation: etcd is a distributed key-value store used by Kubernetes to store all data used to manage the cluster. It ensures data consistency and is the ultimate source of truth for your cluster.
• Example: When you create a new Pod, its configuration is stored in etcd. If the Pod crashes, the Kubernetes system can check etcd to know its intended state.

3. Scheduler

• Explanation: The Scheduler watches for newly created Pods with no assigned node and selects a node for them to run on based on resource availability, constraints, affinity specifications, data locality, and other factors.
• Example: When you submit a new Pod with specific CPU and memory requirements, the Scheduler decides which node the Pod runs on, based on resource availability.

4. Controllers

• Explanation: Controllers are control loops that watch the state of your cluster, then make or request changes where needed. Each controller tries to move the current cluster state closer to the desired state.
• Example: The ReplicaSet Controller ensures that the number of replicas defined for a service matches the number currently deployed in the cluster.

Worker Node Components

1. Kubelet

• Explanation: The Kubelet is an agent that runs on each node in the cluster. It ensures that containers are running in a Pod.
• Example: The Kubelet takes a set of PodSpecs (provided by the API Server) and ensures that the containers described in those PodSpecs are running and healthy.

2. Kube-Proxy

• Explanation: Kube-Proxy is a network proxy that runs on each node in your cluster, implementing part of the Kubernetes Service concept. It maintains network rules on nodes which allow network communication to your Pods from network sessions inside or outside of your cluster.
• Example: Kube-Proxy can route traffic coming to a node’s IP address to the appropriate Pods running on that node.

3. Container Runtime

Explanation: The container runtime is the software responsible for running containers. Kubernetes supports several runtimes: Docker, containerd, CRI-O, and any implementation of the Kubernetes CRI (Container Runtime Interface).

Example: If you’re using Docker as your container runtime, when a Pod is scheduled on a node, the Kubelet talks to Docker to start the container(s) as per the Pod specification.

Here’s a basic diagram that displays the components of the Control (Master) Node and Worker Node in a Kubernetes environment:

This diagram provides a straightforward representation of the key components in both the Control Node and Worker Node, along with their basic interactions and functions within a Kubernetes cluster.

These components work together to create a robust, scalable, and efficient Kubernetes environment. The Master Node components make global decisions about the cluster (like scheduling), whereas the Worker Node components execute these decisions and run the application containers.

3. Core Concepts

Pods

• Explanation: A Pod is the smallest deployable unit in Kubernetes and can contain one or more containers. Containers in a Pod share the same network namespace, meaning they can communicate with each other using localhost. They also share storage volumes.
• Example YAML:

  apiVersion: v1
  kind: Pod
  metadata:
    name: example-pod
  spec:
    containers:
    - name: nginx-container
      image: nginx

Nodes and Clusters

• Explanation: A Node is a physical or virtual machine that runs Kubernetes workloads. A Cluster is a set of Nodes managed by a master node. Clusters are the foundation of Kubernetes, enabling high availability and scalability.
• Example YAML: Nodes and Clusters are typically set up and managed outside of Kubernetes YAML files, often through cloud providers or Kubernetes installation tools like kubeadm.

Services

• Explanation: A Service in Kubernetes defines a logical set of Pods and a policy by which to access them. Services enable network access to a set of Pods, often providing load balancing.

Let’s provide examples for each type of Kubernetes service (ClusterIP, NodePort, LoadBalancer, and ExternalName) using Nginx as the application:

1. ClusterIP Service Example

A ClusterIP service is the default Kubernetes service that exposes the service on a cluster-internal IP. This makes the service only reachable within the cluster.

Here’s an example YAML configuration for a ClusterIP service for Nginx:

apiVersion: v1
kind: Service
metadata:
  name: nginx-clusterip
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: ClusterIP

2. NodePort Service Example

NodePort exposes the service on each Node’s IP at a static port. This service is accessible from outside the cluster using <NodeIP>:<NodePort>.

Example for NodePort service with Nginx:

apiVersion: v1
kind: Service
metadata:
  name: nginx-nodeport
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
      nodePort: 30007
  type: NodePort

3. LoadBalancer Service Example

LoadBalancer exposes the service externally using a cloud provider’s load balancer. This is a common way to expose services to the internet.

Example for a LoadBalancer service with Nginx:

apiVersion: v1
kind: Service
metadata:
  name: nginx-loadbalancer
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer

4. ExternalName Service Example

ExternalName service is a special type of service that has no selectors and does not define any ports or endpoints. Instead, it allows the service to return a CNAME record for an external name.

Example for an ExternalName service pointing to an external Nginx server:

apiVersion: v1
kind: Service
metadata:
  name: nginx-externalname
spec:
  type: ExternalName
  externalName: my-nginx.example.com

In this example, my-nginx.example.com would be the domain where your external Nginx server is located.

Deployments

• Explanation: A Deployment provides declarative updates for Pods and ReplicaSets. It allows you to describe the desired state in a definition file, and the Deployment Controller changes the actual state to the desired state at a controlled rate.
• Example YAML:

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: example-deployment
  spec:
    replicas: 3
    selector:
      matchLabels:
        app: example
    template:
      metadata:
        labels:
          app: example
      spec:
        containers:
        - name: nginx
          image: nginx:1.14.2

StatefulSets

• Explanation: StatefulSets are used for applications that require stable, unique network identifiers, stable persistent storage, and ordered, graceful deployment and scaling.
• Example YAML:

  apiVersion: apps/v1
  kind: StatefulSet
  metadata:
    name: example-statefulset
  spec:
    serviceName: "nginx"
    replicas: 3
    selector:
      matchLabels:
        app: example
    template:
      metadata:
        labels:
          app: example
      spec:
        containers:
        - name: nginx
          image: nginx

DaemonSets

• Explanation: A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected.
• Example YAML:

  apiVersion: apps/v1
  kind: DaemonSet
  metadata:
    name: example-daemonset
  spec:
    selector:
      matchLabels:
        app: example
    template:
      metadata:
        labels:
          app: example
      spec:
        containers:
        - name: busybox
          image: busybox
          args:
          - /bin/sh
          - -c
          - 'while true; do sleep 1000; done'

Namespaces

• Explanation: Namespaces in Kubernetes are intended for use in environments with many users spread across multiple teams or projects. Namespaces provide a scope for names and can be used to divide cluster resources between multiple users.
• Example YAML:

  apiVersion: v1
  kind: Namespace
  metadata:
    name: example-namespace

These examples illustrate how you can define various Kubernetes resources using YAML files. These files are then applied to a Kubernetes cluster using the kubectl apply -f <filename>.yaml command.

4. Configuration and Management

ConfigMaps and Secrets in Kubernetes

ConfigMaps and Secrets are Kubernetes objects used to store non-confidential and confidential data, respectively. They are key tools for managing configuration data and sensitive information in Kubernetes.

ConfigMaps

Explanation: ConfigMaps allow you to decouple configuration artifacts from image content, keeping containerized applications portable. They store non-confidential data in key-value pairs and can be used by pods.

Creating a ConfigMap:

You can create a ConfigMap from literal values, files, or directories.

Example using literal values:
bash kubectl create configmap my-config --from-literal=key1=value1 --from-literal=key2=value2

Example using a file (config-file.yaml): kubectl create configmap my-config --from-file=path/to/config-file.yaml

Accessing ConfigMaps in Pods:

ConfigMaps can be used in a Pod by referencing them in the Pod’s environment variables, command-line arguments, or as configuration files in a volume.

Example of using a ConfigMap in a Pod definition:

apiVersion: v1
kind: Pod
metadata:
name: mypod
spec:
containers:
– name: mycontainer
image: nginx
envFrom:
– configMapRef:
name: my-config

Secrets

Explanation: Secrets are used to store and manage sensitive information, such as passwords, OAuth tokens, and SSH keys. They are similar to ConfigMaps but are specifically intended to hold confidential data.

Creating a Secret:

Secrets can be created from literal values or from files.

Example using literal values:
bash kubectl create secret generic my-secret --from-literal=password=my-password

Example using a file: kubectl create secret generic my-secret --from-file=path/to/secret/file

Accessing Secrets in Pods:

Secrets can be mounted as data volumes or be exposed as environment variables to be used by a container in a Pod.

Example of using a Secret in a Pod definition:

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mycontainer
    image: nginx
    envFrom:
    - configMapRef:
        name: my-config

Using ConfigMaps and Secrets in Deployments

• ConfigMaps and Secrets can also be used in Deployments. The process is similar to using them in Pods.
• Example of a Deployment using a ConfigMap and a Secret:

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: my-deployment
  spec:
    replicas: 2
    selector:
      matchLabels:
        app: myapp
    template:
      metadata:
        labels:
          app: myapp
      spec:
        containers:
        - name: nginx
          image: nginx
          env:
            - name: CONFIG_VALUE
              valueFrom:
                configMapKeyRef:
                  name: my-config
                  key: key1
            - name: SECRET_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: my-secret
                  key: password

In this example, the Deployment creates Pods that use both a ConfigMap and a Secret. The ConfigMap provides a non-confidential configuration value, while the Secret provides a confidential password, both of which are used as environment variables in the containers.

Resource Quotas and Limits

• Explanation: Resource quotas are used to limit the overall resource consumption in a namespace, ensuring fair usage of resources. Resource limits, on the other hand, are applied to individual pods or containers to restrict their resource usage.
• Example: A Resource Quota limiting the total memory and CPU that can be used in a namespace.

  apiVersion: v1
  kind: ResourceQuota
  metadata:
    name: example-quota
  spec:
    hard:
      cpu: "10"
      memory: 10Gi

Labels and Selectors

• Explanation: Labels are key/value pairs attached to objects (like pods) used for identifying and organizing them. Selectors are used to select a set of objects based on their labels.
• Example: A Deployment using a selector to identify the pods it manages.

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: example-deployment
  spec:
    replicas: 2
    selector:
      matchLabels:
        app: myapp
    template:
      metadata:
        labels:
          app: myapp
      spec:
        containers:
        - name: nginx
          image: nginx

Ingress Controllers and Resources

Certainly! Let’s delve into Ingress and Ingress Controllers in Kubernetes, with a focus on deploying in an AWS environment.

Ingress in Kubernetes

• Explanation: In Kubernetes, an Ingress is an API object that manages external access to the services in a cluster, typically HTTP/HTTPS. Ingress can provide load balancing, SSL termination, and name-based virtual hosting. It’s a way to route external traffic to your internal Kubernetes services.

Ingress Controller

• Explanation: The Ingress resource alone is not enough; you also need an Ingress Controller, which is the component responsible for fulfilling the Ingress, usually with a load balancer. The Ingress Controller reads the Ingress Resource information and processes the data accordingly.

AWS Context

When deploying on AWS, you typically use the AWS Load Balancer Controller (formerly known as the ALB Ingress Controller). This controller allows you to leverage AWS Elastic Load Balancing features like Application Load Balancer for distributing external HTTP(S) traffic to Kubernetes services.

Setup and Configuration

1. Install the AWS Load Balancer Controller:

• Ensure your Kubernetes cluster is running in AWS.
• Install the AWS Load Balancer Controller in your cluster. This can be done using Helm or by applying YAML files directly. Using Helm:

   helm repo add eks https://aws.github.io/eks-charts
   helm install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system

2. Create an IAM Policy:

• The AWS Load Balancer Controller needs permissions to interact with AWS resources. Create an IAM policy that grants the necessary permissions.

2. Associate the IAM Role with your Kubernetes Service Account:

• Use the AWS IAM roles for Kubernetes Service Accounts (IRSA) feature to assign the IAM role to the AWS Load Balancer Controller service account in your cluster.

2. Define an Ingress Resource:

• Create an Ingress resource that specifies how you want to route traffic to your Kubernetes services. Example Ingress YAML:

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: example-ingress
     annotations:
       kubernetes.io/ingress.class: "alb"
       alb.ingress.kubernetes.io/scheme: internet-facing
   spec:
     rules:
     - host: myapp.example.com
       http:
         paths:
         - path: /
           pathType: Prefix
           backend:
             service:
               name: my-service
               port:
                 number: 80

5. Deploy the Ingress Resource:

• Apply the Ingress resource to your cluster using kubectl apply -f ingress.yaml.

5. DNS Configuration:

• Once the Ingress is created, it will be assigned a URL by the AWS Load Balancer. Update your DNS records to point your domain to this URL.

Considerations

• Security: Ensure you configure security groups and access control lists correctly to restrict access where necessary.
• SSL/TLS: For HTTPS, you’ll need to configure SSL/TLS certificates, which can be managed by AWS Certificate Manager.
• Monitoring and Logging: Utilize AWS CloudWatch for monitoring and logging the performance and health of your Ingress.

By following these steps, you can set up an Ingress in a Kubernetes cluster running on AWS, leveraging AWS’s native load balancing capabilities to efficiently route external traffic to your internal services.

Persistent Volumes and Claims

Certainly! Let’s delve into Persistent Volumes (PVs), Persistent Volume Claims (PVCs), and Storage Classes in Kubernetes, including how they are defined and used in deployments.

Persistent Volumes (PVs)

• Explanation: A Persistent Volume (PV) is a cluster-level resource that represents a piece of storage capacity in the cluster. It is provisioned by an administrator or dynamically provisioned using Storage Classes.
• Creating a PV: Here’s an example of a PV definition with a specified Storage Class:

  apiVersion: v1
  kind: PersistentVolume
  metadata:
    name: example-pv
  spec:
    capacity:
      storage: 10Gi
    volumeMode: Filesystem
    accessModes:
      - ReadWriteOnce
    persistentVolumeReclaimPolicy: Retain
    storageClassName: slow
    nfs:
      path: /path/to/nfs/share
      server: nfs-server.example.com

Persistent Volume Claims (PVCs)

• Explanation: A Persistent Volume Claim (PVC) is a request for storage by a user. It specifies the size and access modes (like ReadWriteOnce, ReadOnlyMany).
• Creating a PVC: Here’s an example of a PVC definition that requests a volume from the slow Storage Class:

  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: example-pvc
  spec:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 5Gi
    storageClassName: slow

Storage Classes

• Explanation: Storage Classes allow you to define different classes of storage (like slow and fast). This abstraction enables dynamic provisioning of PVs.
• Creating a Storage Class: Here’s an example of a Storage Class definition:

  apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    name: slow
  provisioner: kubernetes.io/aws-ebs
  parameters:
    type: gp2
    zone: us-west-2a
  reclaimPolicy: Retain
  allowVolumeExpansion: true

Using PVCs in Deployments

• Explanation: PVCs can be mounted as volumes in pods. This is useful in deployments to provide persistent storage for your applications.
• Example: Here’s an example of a Deployment using a PVC:

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: my-deployment
  spec:
    replicas: 2
    selector:
      matchLabels:
        app: myapp
    template:
      metadata:
        labels:
          app: myapp
      spec:
        containers:
        - name: mycontainer
          image: nginx
          volumeMounts:
          - mountPath: "/var/www/html"
            name: my-volume
        volumes:
        - name: my-volume
          persistentVolumeClaim:
            claimName: example-pvc

In this Deployment, the example-pvc PVC is mounted into the containers as a volume at /var/www/html. The data in this directory will persist across pod restarts and rescheduling, thanks to the underlying persistent storage provided by the PV.

Key Points

• Match Storage Classes: Ensure the storageClassName in your PVC matches the one defined in your PV or Storage Class for dynamic provisioning.
• Access Modes: The access modes in the PVC should be compatible with those supported by the PV.
• Size Considerations: The requested storage size in the PVC should not exceed the capacity of the PV.

By integrating PVs, PVCs, and Storage Classes, Kubernetes offers a flexible and powerful way to handle persistent storage needs, making it suitable for stateful applications that require stable and persistent data storage.

5. Advanced Topics

Custom Resource Definitions (CRDs)

• Explanation: CRDs allow you to extend Kubernetes with custom resources. You can create new resource types with properties you define, allowing your Kubernetes cluster to manage a broader range of configurations.
• Example: Defining a CRD for a custom resource named MyResource.

  apiVersion: apiextensions.k8s.io/v1
  kind: CustomResourceDefinition
  metadata:
    name: myresources.example.com
  spec:
    group: example.com
    versions:
      - name: v1
        served: true
        storage: true
    scope: Namespaced
    names:
      plural: myresources
      singular: myresource
      kind: MyResource

Helm: Kubernetes Package Management

• Explanation: Helm is a package manager for Kubernetes, allowing you to define, install, and upgrade even the most complex Kubernetes applications.
• Example: Installing a package (chart) using Helm.

  helm install my-release stable/my-chart

This command installs a chart from the stable repository with the release name my-release.

Networking in Kubernetes

• Explanation: Kubernetes networking addresses four main concerns: container-to-container communication, pod-to-pod communication, pod-to-service communication, and external-to-service communication.
• Example: Defining a Network Policy to control traffic flow at the IP address or port level.

  apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata:
    name: example-network-policy
  spec:
    podSelector:
      matchLabels:
        role: db
    policyTypes:
    - Ingress
    ingress:
    - from:
      - podSelector:
          matchLabels:
            role: frontend
      ports:
      - protocol: TCP
        port: 6379

Security in Kubernetes

• Explanation: Kubernetes security encompasses securing the cluster components themselves, securing applications running on Kubernetes, and securing the data within those applications.
• Example: Creating a Role-Based Access Control (RBAC) Role and RoleBinding.

  # Role definition
  apiVersion: rbac.authorization.k8s.io/v1
  kind: Role
  metadata:
    namespace: default
    name: pod-reader
  rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "watch", "list"]

  # RoleBinding definition
  apiVersion: rbac.authorization.k8s.io/v1
  kind: RoleBinding
  metadata:
    name: read-pods
    namespace: default
  subjects:
  - kind: User
    name: "jane"
    apiGroup: rbac.authorization.k8s.io
  roleRef:
    kind: Role
    name: pod-reader
    apiGroup: rbac.authorization.k8s.io

Autoscaling: HPA and CA

• Explanation: Horizontal Pod Autoscaler (HPA) automatically scales the number of pods in a replication controller, deployment, or replica set based on observed CPU utilization. Cluster Autoscaler (CA) automatically adjusts the size of a Kubernetes Cluster so that all pods have a place to run and there are no unneeded nodes.
• Example: Defining an HPA.

  apiVersion: autoscaling/v1
  kind: HorizontalPodAutoscaler
  metadata:
    name: example-hpa
  spec:
    scaleTargetRef:
      apiVersion: apps/v1
      kind: Deployment
      name: my-deployment
    minReplicas: 1
    maxReplicas: 10
    targetCPUUtilizationPercentage: 80

Observability: Logging and Monitoring

• Explanation: Observability in Kubernetes involves monitoring the health of your applications and Kubernetes clusters, as well as logging and tracing to understand the behavior of your applications.
• Example: While specific examples of logging and monitoring configurations depend on the tools used (like Prometheus for monitoring and Fluentd for logging), you can set up a basic logging mechanism using a sidecar container.

  apiVersion: v1
  kind: Pod
  metadata:
    name: counter
  spec:
    containers:
    - name: count
      image: busybox
      args: [/bin/sh, -c, 'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done']
    - name: log-viewer
      image: busybox
      args: [/bin/sh, -c, 'tail -f /var/log/count.log']
      volumeMounts:
      - name: log
        mountPath: /var/log
    volumes:
    - name: log
      emptyDir: {}

These notes and examples provide a comprehensive overview of advanced Kubernetes topics, from extending Kubernetes capabilities with CRDs to ensuring robust observability with logging and monitoring solutions.

6. Cluster Administration and Maintenance

Managing a Kubernetes cluster involves various tasks including setting up the cluster, performing upgrades and rollbacks, ensuring backup and disaster recovery, and maintaining nodes. Let’s delve into each of these aspects.

1. Setting Up a Kubernetes Cluster

• Explanation: Setting up a Kubernetes cluster involves configuring a group of machines to run containerized applications managed by Kubernetes. This can be done on-premises, in the cloud, or in a hybrid environment.
• Tools and Services: Tools like kubeadm, Minikube, Kubespray, and cloud services like Amazon EKS, Google GKE, and Microsoft AKS can be used for cluster setup.
• Example: Using kubeadm to set up a basic cluster:

  # On the master node
  kubeadm init --pod-network-cidr=192.168.0.0/16

  # Setting up kubeconfig
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

  # On each worker node
  kubeadm join [your unique string from the kubeadm init output]

2. Cluster Upgrades and Rollbacks

• Explanation: Upgrading a Kubernetes cluster involves updating the software of all components (API server, controller manager, scheduler, kubelet) to a new version. Rollbacks are performed if the upgrade encounters issues.
• Process: Upgrades should be planned and tested in a non-production environment first. Rollbacks require a backup of the etcd database and cluster state.
• Example: Upgrading a cluster using kubeadm:

  # Drain the node
  kubectl drain <node-name> --ignore-daemonsets

  # Upgrade the kubeadm tool
  apt-get update && apt-get upgrade -y kubeadm

  # Upgrade the cluster
  kubeadm upgrade apply <new-version>

3. Backup and Disaster Recovery

• Explanation: Regular backups of the Kubernetes cluster state and data are crucial for disaster recovery. This includes backing up the etcd database, Kubernetes resource configurations, and persistent data.
• Tools: Tools like Velero can be used for backup and recovery.
• Example: Setting up Velero for backups:

  velero install --provider aws --bucket my-backup-bucket --secret-file ./credentials-velero
  velero schedule create daily-backup --schedule="@daily"

4. Node Maintenance and Management

• Explanation: Node maintenance involves managing the lifecycle of nodes, monitoring node health, and ensuring nodes are properly provisioned and configured.
• Tasks: This includes adding/removing nodes, updating node software, monitoring node health, and troubleshooting node issues.
• Example: Safely draining a node for maintenance:

  kubectl drain <node-name> --ignore-daemonsets --delete-local-data
  # Perform maintenance tasks
  kubectl uncordon <node-name>

Key Points

• Automation and Tools: Utilize automation tools and Kubernetes features to streamline cluster management tasks.
• Monitoring and Alerts: Implement comprehensive monitoring and alerting to quickly identify and respond to issues.
• Documentation and Best Practices: Follow Kubernetes best practices and document your cluster architecture and maintenance procedures.

7. Use Cases and Patterns

Microservices Architecture on Kubernetes

Explanation

Kubernetes is well-suited for a microservices architecture due to its ability to manage and scale a large number of small, independent services efficiently.

Key Features

• Pods and Services: Each microservice can be deployed as a set of Pods, managed by a Service.
• Service Discovery: Kubernetes Services provide a stable endpoint for discovering and communicating with a set of Pods.

Example

Deploying a simple microservice:

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app: my-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 9376
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app
        image: my-app-image

CI/CD Pipelines with Kubernetes

Explanation

Continuous Integration and Continuous Deployment (CI/CD) pipelines in Kubernetes automate the process of building, testing, and deploying applications.

Key Features

• Automated Deployment: Tools like Jenkins, GitLab CI, and ArgoCD can be integrated with Kubernetes.
• Rolling Updates: Kubernetes supports rolling updates for zero-downtime deployments.

Example

Using a Jenkins pipeline to deploy to Kubernetes:

pipeline {
    agent any
    stages {
        stage('Deploy') {
            steps {
                script {
                    kubernetesDeploy(
                        configs: 'k8s/deployment.yaml',
                        kubeconfigId: 'KUBE_CONFIG'
                    )
                }
            }
        }
    }
}

High Availability and Load Balancing

Explanation

Kubernetes enhances high availability and load balancing of applications through various mechanisms.

Key Features

• ReplicaSets: Ensure a specified number of pod replicas are running at all times.
• Load Balancing Services: Distribute traffic among multiple pods.

Example

Creating a LoadBalancer service:

apiVersion: v1
kind: Service
metadata:
  name: my-loadbalancer
spec:
  selector:
    app: my-app
  ports:
    - protocol: TCP
      port: 80
  type: LoadBalancer

Multi-Tenancy and Resource Isolation

Explanation

Kubernetes supports multi-tenancy, allowing multiple users or teams to share a cluster while maintaining isolation.

Key Features

• Namespaces: Logical separation of cluster resources.
• Resource Quotas: Limit resource usage per namespace.
• Network Policies: Control traffic flow at the IP address or port level.

Example

Creating a namespace with resource quotas:

apiVersion: v1
kind: Namespace
metadata:
  name: team-a
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-a-quota
  namespace: team-a
spec:
  hard:
    pods: "10"
    limits.cpu: "4"
    limits.memory: 2Gi

Conclusion

Kubernetes provides a robust platform for microservices architecture, CI/CD pipelines, high availability, and multi-tenancy. By leveraging Kubernetes features, you can build scalable, resilient, and efficient applications.

8. Troubleshooting Common Issues in Kubernetes

Troubleshooting in Kubernetes involves identifying and resolving issues that arise in your cluster. Common issues range from pod failures, networking issues, to resource constraints. Here’s a guide to help you navigate these challenges.

1. Pod Failures

• Symptoms: Pods are in CrashLoopBackOff, Error, or ImagePullBackOff state.
• Troubleshooting Steps:
• Check pod logs: kubectl logs <pod-name>
• Describe the pod to see events and status: kubectl describe pod <pod-name>
• Check if the container image is correct and accessible.
• Ensure resource limits are not too low (CPU, memory).

2. Networking Issues

• Symptoms: Services are not reachable, inter-pod communication fails.
• Troubleshooting Steps:
• Verify network policies and ensure they are not overly restrictive.
• Check service and pod selectors for mismatches.
• Ensure the DNS service within the cluster is functioning correctly.
• Test network connectivity between nodes.

3. Persistent Volume Claims (PVCs) Issues

• Symptoms: PVCs are stuck in Pending state.
• Troubleshooting Steps:
• Check if the Persistent Volumes (PVs) meet the requirements of the PVCs.
• Ensure the storage class and access modes are correctly configured.
• Verify dynamic provisioning configurations if applicable.

4. Resource Constraints and Quotas

• Symptoms: Pods are not being scheduled due to insufficient resources.
• Troubleshooting Steps:
• Check resource quotas: kubectl describe quota
• Review node resource utilization: kubectl describe node <node-name>
• Consider scaling up the cluster or optimizing application resource usage.

5. Cluster Component Failures

• Symptoms: API server is unresponsive, etcd issues, scheduler or controller manager problems.
• Troubleshooting Steps:
• Check the status of master components.
• Review logs of Kubernetes system components.
• Ensure etcd cluster is healthy.

6. Security and Access Issues

• Symptoms: Unauthorized access errors, RBAC issues.
• Troubleshooting Steps:
• Review RBAC configurations: roles, rolebindings, clusterroles, clusterrolebindings.
• Check service account tokens and permissions.
• Verify API server access logs for unauthorized access attempts.

7. Application Performance Issues

• Symptoms: Slow application response, timeouts.
• Troubleshooting Steps:
• Monitor and analyze pod metrics for CPU and memory usage.
• Use tools like Prometheus and Grafana for in-depth monitoring.
• Check for network latency or bandwidth issues.

8. Upgrade Related Issues

• Symptoms: Problems after upgrading the cluster or applications.
• Troubleshooting Steps:
• Review change logs and upgrade notes for breaking changes.
• Roll back upgrades if necessary and feasible.
• Test upgrades in a staging environment before applying them to production.

Tools for Troubleshooting

• kubectl: Primary CLI tool for interacting with Kubernetes.
• Prometheus and Grafana: For monitoring and visualizing metrics.
• Elastic Stack (ELK): For log aggregation and analysis.
• Lens or K9s: Kubernetes IDEs for easier cluster management and troubleshooting.

Conclusion

Effective troubleshooting in Kubernetes requires a solid understanding of its components and architecture. Regular monitoring, log analysis, and staying informed about the cluster’s state are key to quickly identifying and resolving issues.

Assigning Pods to Nodes in Kubernetes is crucial for managing application workloads effectively. Here are the main mechanisms to assign pods to specific nodes in Kubernetes, with clear examples:

1. Node Affinity

Node Affinity is a flexible method to control pod placement based on specific node labels. You can set hard (required) or soft (preferred) constraints using Node Affinity rules.

Hard Constraint Example (requiredDuringSchedulingIgnoredDuringExecution):

This enforces strict placement on nodes with specific labels.

Example: Only place pods on nodes labeled as “high-performance.”

apiVersion: v1
kind: Pod
metadata:
  name: high-performance-app
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: node-type
                operator: In
                values:
                  - high-performance
  containers:
    - name: app
      image: my-app-image

Soft Constraint Example (preferredDuringSchedulingIgnoredDuringExecution):

This makes placement preferential but not mandatory.

Example: Prefer placing pods on nodes in the “us-east-1a” zone.

apiVersion: v1
kind: Pod
metadata:
  name: regional-app
spec:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 1
          preference:
            matchExpressions:
              - key: topology.kubernetes.io/zone
                operator: In
                values:
                  - us-east-1a
  containers:
    - name: app
      image: my-app-image

2. Node Selector

Node Selector is a simpler, label-based method for assigning pods to specific nodes. It’s a straightforward approach where pods are scheduled only on nodes that match specified labels.

Example: Schedule the pod on nodes with the label env=production.

apiVersion: v1
kind: Pod
metadata:
  name: production-app
spec:
  nodeSelector:
    env: production
  containers:
    - name: app
      image: my-app-image

3. Taints and Tolerations

Taints and tolerations are used to repel certain pods from specific nodes, unless those pods have a matching toleration. This mechanism helps reserve nodes for specific purposes, like dedicated nodes for sensitive applications.

Node Tainting Example

To taint a node, use:

kubectl taint nodes <node-name> key=value:NoSchedule

Pod Toleration Example

To allow a pod to be scheduled on a tainted node, add a toleration:

apiVersion: v1
kind: Pod
metadata:
  name: tolerable-app
spec:
  tolerations:
    - key: "key"
      operator: "Equal"
      value: "value"
      effect: "NoSchedule"
  containers:
    - name: app
      image: my-app-image

4. Pod Affinity and Anti-Affinity

Pod Affinity and Anti-Affinity manage pod placement based on other pods. Pod Affinity places pods close to each other, while Anti-Affinity ensures pods are scheduled away from each other.

Pod Affinity Example

Place a pod on the same node as other pods with label app: frontend.

apiVersion: v1
kind: Pod
metadata:
  name: frontend-helper
spec:
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: app
                operator: In
                values:
                  - frontend
          topologyKey: "kubernetes.io/hostname"
  containers:
    - name: app
      image: my-helper-app

Pod Anti-Affinity Example

Ensure each pod replica is placed on a different node.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-deployment
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: my-app
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - my-app
              topologyKey: "kubernetes.io/hostname"
      containers:
        - name: app
          image: my-app-image

5. Static Pods

Static Pods are managed directly by the kubelet on each node, not by the API server, so they are automatically placed on the node where they are configured.

Static Pod Example

To create a static pod, place a pod configuration file in the directory specified by the kubelet (usually /etc/kubernetes/manifests).

# /etc/kubernetes/manifests/static-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: static-app
spec:
  containers:
    - name: app
      image: my-static-app

Summary Table

This summary should help guide the assignment of pods to nodes based on various requirements and use cases. Let me know if there’s a particular mechanism you’d like more details on!

 jenkins

For continuous integration and for continuous deployment we are using jenkins

CI:ContinuousIntegration

It is the combination of continuous build + continuous test

Whenever Developer commits the code using source code management like GIT, then the CI Pipeline gets the change of the code runs automatically build and unit test

 Due to integrating the new code with old code, we can easily get to know the code is a success (or) failure

 It finds the errors more quickly

  Delivery the products to client more frequently Developers don’t need to do manual tasks

 It reduces the developer time 20% to 30%

CI Server

Here only Build, test & Deploy all these activities are performed in a single CI Server Overall, CI Server = Build + Test + Deploy

CD: ContinuousDelivery/Development

Continuous Delivery

CD is making it available for deployment. Anytime a new build artifact is available, the artifact is automatically placed in the desired environment and deployed

 Here, Deploy to production is manual here

Continuous Deployment

 CD is when you commit your code then its gets automatically tested, build and deploy on the production server.

 It does not require approval

  99% of customers don’t follow this Here, Deploy to production is automatic

CI/CD Pipeline

It looks like a Software Development Life Cycle (SDLC). Here we are having 6 phases

Version Control

Here developers need to write code for web applications. So it needs to be committed using version control system like GIT (or) SVN

Build

 Let’s consider your code is written in java, it needs to be compiled before execution. In this build step code gets compiled

 For build purpose we’re using maven

Unit Test

  If the build step is completed, then move to testing phase in this step unit step will be done. Here we can use sonarqube/mvn test

  Here, application/program components are perfectly worked/not we will check in this testing Overall, It is code level testing

Deploy

 If the test step is completed, then move to deploy phase

  In this step, you can deploy your code in dev, testing environment Here, you can see your application output

 Overall, we are deploying our application in Pre-Prod server. So, Internally we can access

Auto Test

  Once your code is working fine in testing servers, then we need to do Automation testing So, overall it is Application level testing

 Using Selenium (or) Junit testing

Deploy to Production

If everything is fine then you can directly deploy your code in production server

Becauseofthispipeline,bugswillbereportedfastandgetrectifiedsoentiredevelopmentisfast Here,OverallSDLCwillbeautomaticusingJenkins

Note:

If we have error in code then it will give feedback and it will be corrected, if we have error in build then it will give feedback and it will be corrected, pipeline will work like this until it reaches deploy

what is jenkins

 It is an open source project written in Java by kohsuke kawaguchi

 The leading open source automation server, Jenkins provides hundreds of plugins to support building, deploying and automating any project.

 It is platform independent

  It is community-supported, free to use It is used for CI/CD

 If we want to use continuous integration first choice is jenkins

  It consists of plugins. Through plugins we can do whatever we want. Overall without plugins we can’t run anything in jenkins

  It is used to detect the faults in the software development It automates the code whenever developer commits

  It was originally developed by SUN Microsystem in 2004 as HUDSON HUDSON was an enterprise addition we need to pay for it

  The project was renamed jenkins when oracle brought the microsystems Main thing is It supports master & slave concepts

 It can run on any major platform without complexity issues

 Whenever developers write code we integrate all the code of all developers at any point int time and we build, test and deliver/deploy it to the client. This is called CI/CD

  We can create the pipelines by our own We have speed release cycles

 Jenkins default port number is 8080

Jenkins Installation

1. Launch an linux server in AWS and add security group rule [Custom TCP and 8080]
2. Install java – amazon-linux-extras install java-openjdk11 -y
3. Getting keys and repo i.e.. copy those commands from “jenkins.io” in browser and paste in terminal open browser → jenkins.io → download → Download Jenkins 2.401.3 LTS for under →Redhat

 sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo sudo rpm –import https://pkg.jenkins.io/redhat-stable/jenkins.io-2023.key

 Copy above 2 links and enter in terminal

1. Install Jenkins – yum install jenkins -y
2. systemctl status jenkins – It is in inactive/dead state
3. systemctl start/restart jenkins – Start the jenkins

Now, open the jenkins in browser – publicIP:8080 JENKINS Default Path : /var/lib/jenkins

  Enter the password go to the particular path i.e. cd path Click on install suggested plugins

Now, Start using jenkins

Alternative way to install jenkins:

 Everytime we have to setup jenkins manually means it will takes time instead of that we can use shell scripting i.e.

  vim jenkins.sh  > add all the manual commands here > :wq Now, we execute the file

  First we need to check whether the file has executable permissions/not, if it’s not #chmod +x jenkins.sh

 Run the file

 ./ jenkins.sh (or) sh jenkins.sh

Create a new Job/task

Job: To perform some set of tasks we use a job in jenkins In Jenkins jobs are of two types

   Freestyle (old)    Pipeline (new)

Now, we are creating the jobs in freestyle

1. Click on create a job (or) new item
2. Enter task name
3. click on freestyle project (or) pipeline [Depends on your requirement] These are the basic steps to Create a Job

Get the Git Repo

Follow above 3 steps then after

1. Copy the github repo url and paste in under SCM. It is showing error
2. So, now in your AWS terminal → Install GIT → yum install git -y
3. Whenever we are using private repo, then we have to create credentials. But right now, we are using public repo. So, none credentials

1. If we want to get the data from particular branch means you can mention the branch name in branch section.But default it takes master

1. Click on save and Build now and build success

If you want to see output in jenkins. Click on console output i.e., (click green tick mark)

If you want to see the repo in our linux terminal

Go to this path →  cd /var/lib/jenkins/workspace/task_name ⟶ now you can see the files from git repo

   If we edit the data in github, then again we have to do build, otherwise that change didn’t reflect in linux server

   Once run the build, open the file in server whether the data is present/not

   So, if we’re doing like this means this is completely under manual work. But, we are DevOps engineers we need automatically

How are you triggering your jenkins Jobs ?

Jenkins job can be triggered either manually (or) automatically

1. Github Webhook
2. Build Periodically
3. Poll SCM

WebHooks

Whenever developer commits the code that change will be automatically applied in server. For this, we use WebHooks

How to add webhooks from gitHub

Open repository → settings → webhooks → Add webhook →

  payload URL : jenkinsURL:8080/github-webhook/ Content-type : Application/json

 Click on Add webhook

So, we are created webhooks from github

Now, we have to activate in jenkins dashboard, here Go to Job → Configure → select below option → save

Schedule the Jobs

Build Periodically

Select one job → configure → build periodically Here, it is working on “CRON SYNTAX”

  Here we have 5 starts i.e.  * * * * * 1st  star represents minutes

  2nd star represents hours [24 hours format] 3rd star represents date

 4th star represents month

  5th star represents day of the week i.e.., Sunday – 0

  Monday – 1

 Tuesday  – 2

 Wednesday – 3

 Thursday – 4

 Friday – 5

 Saturday – 6

 Eg: Aug 28, 11:30 am, sunday Build has to be done → 30 11 28 08 0 → copy this in build periodically

  If we give “ * * * * * “ 5 stars means → At every minute build will happen If i want every 5 minutes build means  → */5 * * * *

Click on Save and Build

Note: Here changes ‘happen/not’ automatically build will happen in “schedule the jobs” For Reference, Go to browser → Crontab-guru

Poll SCM

Select one job → configure → select Poll SCM

  It only works whenever the changes happened in “GIT” tool (or) github We have to mention between the time like 9am-6pm in a day

  same it’s also working on cron syntax Eg: * 9-17 * * *

Difference between WebHooks, Build periodically, Poll SCM (FAQ) Webhooks:

  Whenever developer commits the code, on that time only build will happen. It is 24×7 no time limit

 It is also working based on GIT Tool (or) github

Poll SCM:

  Same as webhooks, But here we have time limit Only for GIT

Build Periodically:

  Automatically build, whether the changes happen/not (24×7) It is used for all devops tools not only for git

  It will support on every work Every Time as per our schedule

   Discard old builds

  Here, we remove the builds, i.e., Here we can see how many builds we have to see (or) max of builds to keep (or) how many days to keep builds. we can do this thing here

  But, when we are in jenkins, it is little bit confusion to see all the builds So, here max. 3 days we can store the builds.

 In our dashboard we can see latest 25 builds

  More than 25 means automatically old builds get deleted So, overall here we can store, delete builds.

 These type of activities are done here

In server, If you want to see build history ?

Go to jenkins path (cd /var/lib/jenkins) → jobs → select the job → builds

If we want to see log info i.e., we can see console o/p info

Go inside builds → 1 → log Here, In server we don’t have any problem

Parameter Types:

1. String → Any combination of characters & numbers
2. Choice → A pre-defined set of strings from which a user can pick a value
3. Credentials → A pre-defined jenkins credentials
4. File → The full path to a file on the file system
5. Multi-line string → Same as string, but allows newline characters
6. password → Similar to the credentials type, but allows us to pass a plain text parameter specific to the job (or) pipeline
7. Run → An absolute URL to a single run of another job

   This project is parameterized

Here, we are having so many parameters, In real life we will use this

1. 1.Boolean parameter

Boolean means used in true (or) false conditions

Here, Set by Default enable means true, Otherwise false

1. 2.Choice parameter

 This parameter is used when we have multiple options to generate a build but need to use only on specific one

 If we have multiple options i.e., either branches (or) files (or) folders etc.., anything we have multiple means we use this parameter

Suppose, If you want to execute a linux command through jenkins means

Job → Configure → build steps → Execute shell → Enter the command → save & build

After build, Check in server → go to workspace → job → we got file data So, above step is we are creating a file through jenkins

Now, $filename it is variable name, we have to mentioned in choice parameterized inside the name

Save and build we got options select the file and build, we will see that output in server

So, overall it provides the choices. Based on requirements we will build So, every time we no need to do configure and change the settings

   File parameter

  This parameter is used when we want to build our local files Local/computer files we can build here

 file location → starts with user

 select a file and see the info and copy the path eg: users/sandeep/downloads

Build with → browse a file → open and build

 So, here at a time we can build a single file

   String parameters (for single line)

  This parameter is used when we need to pass an parameter as input by default String it is a group/sequence of characters

  If we want to give input in the middle of the build we will use this first, write the command in execute shell

Then write the data in string parameter

Save & build

   Multi-line string parameters(for multiple lines)

 Multi-line string parameters are text parameters in pipeline syntax. They are described on the jenkins pipeline syntax page

 This will work as same as string parameter but the difference is instead of one single line string we can use multiple strings at a time as a parameters

How to access the private repo in git

1. Copy the github repo url and paste in under SCM. It is showing error
2. So, now in your AWS terminal → Install GIT → yum install git -y
3. Now, we are using private repo then we have to create credentials.
4. So, for credentials Go to github, open profile settings → developer settings → personal access tokens → Tokens(classic) → Generate new token (general use) → give any name

Same do like above image and create token. So this is your password

 Now, In jenkins go to credentials → add credentials → select username and password → username (github username) → password (paste token) → Description(github-credentials) → save

So, whenever if you want to get private repo from github in jenkins follow above steps

Linked Jobs

This is used when a job is linked with another job

Upstream & Downstream

An upstream job is a configured project that triggers a project as part of it execution.

A downstream job is a configured project that is triggered as part of a execution of pipeline

So, here I want to run the jobs automatically i.e., here we need to run the 1st job, So automatically job-2, job-3 has also build. Once the 1st build is done

 Here for Job-1, Job-2 is downstream

   For Job-2 upstream is Job-1 and downstream is job-3 & Job-4 For Job-3 upstream is Job-1 & Job-2 and downstream is Job-4 For Job-4 both Job-1 & Job-2 & Job-3 are upstream

So, here upstream and downstream jobs help you to configure the sequence of execution for different operations. Hence, you can arrange/orchestrate the flow of execution

 First, create a job-1 and save

 Create another job-2 and here perform below image steps like this and save. So do same for remaining job-3 and job-4

So, we can select based on our requirements

  Then build Job-1, So automatically other jobs builds also started after successfully job-1 builded. because we linked the jobs using upstream and downstream concept

 If you open any task/job, It will show like below

In this dashboard we can see the changes, this is step by step pipeline process

Create the pipeline in freestyle

If I want to see my pipeline in step by step process like above. So, we have to create a pipeline for these

 In dashboard we have builds and click on (+) symbol

 But we need plugin for that pipeline view

  So, Go to manage jenkins → plugins → available plugins we need to add plugin – (build pipeline) and click install without restart

once you got success, go to dashboard and click the (+ New view) symbol

  Perform above steps and click on create and select initial job – Job1, So here once job-1 is build successfully so remaining jobs will be automatically builded

 Don’t touch/do anything and click OK

 So, we can see the visualized pipeline below like this

Here, when you click on ‘RUN’ Trigger a Pipeline you got the view,

  Here, trigger means it is in queue/progress for build

  whenever, you refresh the page, trigger will change from old job to new job history : If you want to see history, select above pipeline history option

  Configure : This is option in pipeline, If you want to configure the job instead of Job-1, click this Add Step : Suppose, you want to add a new job after Job-4

  So, first create a job, with option build after other projects, and give Job-4 So, we have that in pipeline and when you click on run

 But, If your new job wants to come in first (or) middle of the pipeline you have to do it in manually Note : If parameters is on inside a job means we can’t see the pipeline view

Master & Slave Architecture

Here, the communication between these servers, we will use master & slave communication Here, Master is Jenkins server and Slave is other servers

   Jenkins uses a Master-Slave architecture to manage distributed builds.

 In this architecture, master & slave nodes communicate through TCP/IP protocol

  Using Slaves, the jobs can be distributed and load on master reduces and jenkins can run more concurrent jobs and can perform more

  It allows set up various different environments such as java, .Net, terraform, etc.., It supports various types of slaves

     Linux slaves Windows slaves Docker slaves Kubernetes slaves ECS (AWS) slaves

 If Slaves are not there means by default master only do the work

Setup for Master & Slave

1. Launch 3 instances at a time with key-pair, because for server to server communication we are using key-pair
   1. Here name the 3 instances like master, slave-1, slave-2 for better understanding
   2. In master server do jenkins setup
   3. In slave servers you have to install one dependency i.e., java.
   4. Here, in master server whatever the java version you installed right, same you have to install the same version in slave server.
2. Open Jenkins-master server and do setup
   1. Here Go to manage jenkins → click on set up agent

(or)

Go to manage jenkins → nodes & clouds → click on new node → Give node name any → click on permanent agent and create

1. Number of executors –

  Default we have 2 executors. Maximum we can take 5 executors

If we take more executors then build will perform speed and parallely we can do some other builds.

For that purpose we are taking this nodes

c. Remote root directory –

 we have to give slave server path. Here, jenkins related information stored here

So, on that remote path jenkins folder created. we can see build details, workspace, etc..,

1. Labels –

  When creating a slave node, Jenkins allows us to tag a slave node with a label Labels represent a way of naming one or more slaves

  Here we can give environment (or) slave names i.e., dev server – take dev

 production server means take prod (or) take linux, docker

1. Usage –

 Usage describing, how we are using that labels .!

  Whenever label is matches to the server then only build will perform i.e., select “only build jobs with label expressions matching this node”

1. Launch method –

  It describes how we are launching master & slave server Here, we are launching this agents via SSH connection

1. Host –

 Here, we have to give slave server public IP address

1. Credentials –

   Here, we are using our key-pair pem file in SSH connection

Here, In the key you have to add the slave key-pair pem data

click on add and select this credentials

g. Host Key Verification Strategy –

 Here, when you are communicating from one server to another server, on that time if you don’t want verification means

 we can select “Non verifying verification strategy” option

h. Availability –

 We need our Agent always must be running i.e., keep this agent online as much as possible

Perform above steps and Click on save

Here, If everything is success means we will get like below image

Note:Sometimes in Build Executor status under, It will shows one error. That is dependency issue. For that one you have to install the same java version in slave server, which we installed in master server

 Now, Go to Jenkins dashboard, create a job

select above option and give label name

   create one file in execute shell under build steps. perform save & build

So, whatever the jobs data we’re having, we can see in slave server. not in master.

Because you’re using master & slave concept that means slave is working behalf of master.

Note:If you don’t give the above Label option inside a job means, it will runs inside a master This is all about Master & Slave Architecture in Jenkins

User Management in Jenkins

For security configuration purpose we’re using user management

1. Security is all about authentication and authorization.
2. By default, jenkins requires username and password to access
3. By default, all new users will have full access
4. Jenkins stored details about users in local file system
   1. In the real word we use third party identity management systems such as active directory, LDAP etc..,
5. here, we are having 2 types
   1. Role-based strategy
   2. Project based Matrix Authorization strategy
   3. Matrix-based security (Advanced concept)

1. Role-based strategy

In our dashboard, we have 3 main roles

1. Developer – Here we can give read permissions i.e., he can see the build
2. Tester – Read, cancel, testing permissions we can give
3. DevOps – Here we can give full permissions

Steps :

Default we’re having one user. Go to dashboard → people → we can see users

1. Add Users : Go to manage jenkins → users → create user

Here, we can’t directly mention the roles. For that we need plugin

Go to manage plugins → Add plugins →  Role-based Authorization Strategy → Install

1. Configure the plugin

 Go to manage jenkins → Security → Authentication → select role-based strategy → save

 Once you configured the plugin, automatically you will get a new feature in manage jenkins i.e.., manage & assign roles

1. Adding roles

 Now, go inside Manage & Assign roles → Manage roles → Add roles

  Give the permissions for developer, tester and check the boxes based on their roles and save eg: Developer can only see the view, DevOps engineer can do anything like that

1. Assign the roles

 In the above path we’re having assign roles

 Go inside → Add User → give user name → save

  If you give wrong user name, it will take but we can see the user name is striked Do Above process, save

1. Login

  After done above 4 steps, click on log out and login as another user Go to dashboard, Here we can see the changes

 Like this you can login as multiple user and do perform the operations

1. Project-based matrix authorization strategy

Here, we can give job-level permissions. that means specific users can access only specific jobs

1. First install the plugin – Role-based authorization
2. Go to manage jenkins → add user → save
3. Go to security → Authorization → project-based matrix authorization strategy → add user → give either read/view any permissions → save

1. Go to dashboard → select a job → configure → click enable project based security → add user → give permissions → save

Now, that job is only access for that particular user FYI, open dashboard and see the jobs

1. Logout and login as another user

Now that user can see only that particular job in his dashboard. User can’t see any jobs This is the way you can restrict the users inside a job

  Jenkins- i eline

 Jenkins pipeline is a combination of plugins that supports integration and implementation of continuous delivery pipelines

  A pipeline is a group of events interlinked with each other in a sequence Here, using Groovy syntax we’re writing a pipeline

We have 3 types of pipelines

1. Freestyle pipeline
2. scripted pipeline
3. Declarative pipeline

Difference between freestyle and pipeline

 In pipeline, we are writing the script for deployment. It is updated

  In freestyle we are having manual options we can go through that. It is little bit old In real time we use 2 pipelines based on our requirement

Jenkins file – it is nothing but it contains the scripted (or) declarative code

Scripted pipeline syntax:

Eg: node {

stage (”stage 1″) { echo “hi”

}

}

Declarative pipeline syntax:

pipeline { agent any stages {

stage(”code”) {

steps {

echo “hi”

}

}

}

}

Here, In our pipeline we’re using declarative syntax

Declarative pipeline :

 Here, pipeline is a block

 In this block we have agents

  Through agent we will decide in which server we have to run our tasks/job So, here we created a label, through label we will define

  Inside the stages we have multiples stages Eg: Code, build, test, deploy

 Inside every stages we have one step

 Inside the steps we can write our code/commands

Launch jenkins server and open dashboard

1. Create a job → select pipeline → OK

1. Select pipeline → here we have to write the groovy syntax

1. Write the script

   Single stage pipeline

Here, automatically indentations will take i.e, a tab space (or) 4 spaces

 Once you write your script → build

  GUI will be different. Here we can see step by step process

If you want to see the output, click on the build view click on logs

 Multi stage pipeline

Click on build, you will see the o/p like given below

Variables :

variables are used to store the values (or) data. Here, we are having 2 types of variables

1. Global variable

2. Local variable

Global variable

  Here, we’re declaring the environment variable after the agent. And we have to use $variable in stages to call the variables

Click on build and click on logs to see the output

Multiple Global variables

Click on build and click on logs to see the output

Local variable

  Local variable override the Global variable We’re declaring local variable inside the stages

Click on build and here we have 2 stages. First is global and second is local variable. Now, you can easily find out the difference between local and global

So, when we’re using local variable means, some specific/particular stage we need another value. On that case we’re using local

This is all about local and global variables

 Parameters pipeline

Instead of manually selecting parameters, we can write the code in pipeline

  For the first time build, Automatically selecting the parameters based on our code. for the 1st build → code will executed

  After 1st build, Go to configure and check the parameters selected (or) not and do save For the second time build, click on build with parameters, we can see the output

 for the 2nd build → parameters executed

  Here, Overall we have to build 2 times to see our output We have to take parameters block after the agent

Whenever we’re using parameters we don’t need to use Environment block

String parameter pipeline

This is our code, click on save and build. Here, our code will get executed

 After the first time build, Automatically selecting the parameters based on our code.

 Click on build with parameters, you will get above image and now click on build

This is all about string parameters

Boolean parameter pipeline

This is our code, click on save and build. Here, our code will get executed

 After the first time build, Automatically selecting the parameters based on our code.

 Click on build with parameters, you will get above image and now click on build

 In the above code We written defaultValue is true. So, db checkbox is enabled. if we write false it is disabled

Choice parameter pipeline

This is our code, click on save and build. Here, our code will get executed

 After the first time build, Automatically selecting the parameters based on our code.

Here, we select the file names based on our requirements

  Click on build with parameters, you will get above image and now click on build

After build click on logs we can see the output

  Input function pipeline

It takes the input from the user, based on the input it will performs the operations

  Here, we are taking the input from the user If User said OK means build will happen

 If User said NO means build will fail

So, here we are having one condition. That condition we can called input function

 Here continuous integration performed. i/e.., build +test

 But when it comes to deploy stage. It has to be asked the input from the user

Real-time Scenario :

 Whenever you’re doing deployment, this input function we have to give to approval manager. So, manager check everything. If everything is correct he will click OK i.e., he will approve the deployment

 Here, how we’re giving the permissions means we’re using role based strategy and for all managers we have to give full build perform permissions.

Click on save and build you will get below image

 Here, if we click on yes means build will success, Click on abort means build aborted/stopped Once you click on yes, you will get below image

This is all about input function

 Post Build Actions/Functions pipeline

A Jenkins Post-build action is a task executed after the build has been completed

 When you perform build, you won’t care about the build whether it is success(or) fail. Automatically, you want to build the particular stage

 on that case we’re using post build actions Here, we are having post conditions in jenkins

1. Always
2. Success
3. Failure

Success:

When the above stage build gets success means, then the post block will executed

click on save and build

Failure:

When the above stage build gets failed means, then the post block will executed

click on save and build

Always:

When the above stage build either success (or) Failure. This post block don’t care it will always executed

Click on save and build

This is all about Post-Build Actions

Setup for Master & Slave

1. Launch 3 instances at a time with key-pair, because for server to server communication we are using key-pair
   1. Here name the 3 instances like master, slave-1, slave-2 for better understanding
   2. In master server do jenkins setup
   3. In slave servers you have to install one dependency i.e., java.
   4. Here, in master server whatever the java version you installed right, same you have to install the same version in slave server.
2. Open Jenkins-master server and do setup
   1. Here Go to manage jenkins → click on set up agent

(or)

Go to manage jenkins → nodes & clouds → click on new node → Give node name any → click on permanent agent and create

1. Number of executors –

  Default we have 2 executors. Maximum we can take 5 executors

If we take more executors then build will perform speed and parallely we can do some other builds.

For that purpose we are taking this nodes

1. Remote root directory –

 we have to give slave server path. Here, jenkins related information stored here

So, on that remote path jenkins folder created. we can see build details, workspace, etc..,

1. Labels –

  When creating a slave node, Jenkins allows us to tag a slave node with a label Labels represent a way of naming one or more slaves

  Here we can give environment (or) slave names i.e., dev server – take dev

 production server means take prod (or) take linux, docker

e. Usage –

 Usage describing, how we are using that labels .!

  Whenever label is matches to the server then only build will perform i.e., select “only build jobs with label expressions matching this node”

1. Launch method –

  It describes how we are launching master & slave server Here, we are launching this agents via SSH connection

1. Host –

 Here, we have to give slave server public IP address

1. Credentials –

  Here, we are using our key-pair pem file in SSH connection

Here, In the key you have to add the slave key-pair pem data

 click on add and select this credentials

g. Host Key Verification Strategy –

 Here, when you are communicating from one server to another server, on that time if you don’t want verification means

 we can select “Non verifying verification strategy” option

h. Availability –

 We need our Agent always must be running i.e., keep this agent online as much as possible

Perform above steps and Click on save

Here, If everything is success means we will get like below image

Note:Sometimes in Build Executor status under, It will shows one error. That is dependency issue. For that one you have to install the same java version in slave server, which we installed in master server

Now, create the pipeline for master-slave

Save and Build and see the output. Now, go to the jenkins path & check This is all about Jenkins

1. What is Terraform?
   Terraform is an infrastructure as code (IaC) tool for building, changing, and versioning infrastructure efficiently.
2. What are the advantages of using Terraform or IaC in general?

• Automation of infrastructure provisioning
• Consistency and standardization
• Scalability
• Collaboration facilitation
• Error reduction
• Version control integration

3. What are some of Terraform features?

• Infrastructure as Code
• Execution Plans
• Resource Graph
• Change Automation
• Modularity with Modules
• State Management

4. What language does Terraform use?
   Terraform uses its own language called HCL (HashiCorp Configuration Language).
5. What’s a typical Terraform workflow?

• Write Terraform definitions: .tf files written in HCL that described the desired infrastructure state (and run terraform init at the very beginning)
• Review: With command such as terraform plan you can get a glance at what Terraform will perform with the written definitions
• Apply definitions: With the command terraform apply Terraform will apply the given definitions, by adding, modifying or removing the resources

6. What are some use cases for using Terraform?

• Multi-cloud deployments
• Self-service clusters
• Development environment setup
• Resource scaling
• Infrastructure audit and compliance

7. What’s the difference between Terraform and technologies such as Ansible, Puppet, Chef, etc.?
   Terraform is primarily focused on infrastructure provisioning while Ansible, Puppet, Chef, etc., are configuration management tools focused on software and configuration on existing servers. Terraform can be used to provision the servers that configuration management tools then configure. Terraform is immutable, whereas the others can be mutable.

8.Explain the following block of Terraform code

resource "aws_instance" "some-instance" {
  ami           = "ami-201720221991yay"
  instance_type = "t2.micro"
}

This Terraform code defines an AWS EC2 instance resource named "some-instance" with a specified AMI ID "ami-201720221991yay" and instance type "t2.micro".

9. What do you do next after writing the following in main.tf file?

resource "aws_instance" "some-instance" {
  ami           = "ami-201720221991yay"
  instance_type = "t2.micro"
}

Run terraform init. This will scan the code in the directory to figure out which providers are used (in this case AWS provider) and will download them.

10. You’ve executed terraform init and now you would like to move forward to creating the resources but you have concerns and would like to make be 100% sure on what you are going to execute. What should you be doing?

Execute terraform plan. That will provide a detailed information on what Terraform will do once you apply the changes.

11. You’ve downloaded the providers, seen the what Terraform will do (with terraform plan) and you are ready to actually apply the changes. What should you do next?

Run terraform apply. That will apply the changes described in your .tf files.

12. Explain the meaning of the following strings that seen at the beginning of each line When you run terraform apply

• ‘+’
• ‘-‘
• ‘-/+’

• ‘+’ – The resource or attribute is going to be added
• ‘-‘ – the resource or attribute is going to be removed
• ‘-/+’ – the resource or attribute is going to be replaced

13. How to cleanup Terraform resources? Why the user should be careful doing so?
terraform destroy will cleanup all the resources tracked by Terraform.

A user should be careful with this command because there is no way to revert it. Sure, you can always run again “apply” but that can take time, generates completely new resources, etc.

Dependencies

14. Sometimes you need to reference some resources in the same or separate .tf file. Why and how it’s done?

Why: Resources are referenced to establish dependencies and relations, such as attaching a security group to an EC2 instance to control its traffic.

How it’s done: Use the resource type and name to reference attributes of another resource.

Example:

resource "aws_security_group" "example_sg" {
  # ... security group configuration ...
}

resource "aws_instance" "example" {
  ami                    = "some-ami"
  instance_type          = "t2.micro"
  vpc_security_group_ids = [aws_security_group.example_sg.id]  # Reference to the security group's ID
}

In this example, the security group example_sg is defined first, and its ID is referenced in the aws_instance to associate the two resources.

15. Does it matter in which order Terraform creates resources?
Yes, when there is a dependency between different Terraform resources, you want the resources to be created in the right order and this is exactly what Terraform does.

To make it ever more clear, if you have a resource X that references the ID of resource Y, it doesn’t makes sense to create first resource X because it won’t have any ID to get from a resource that wasn’t created yet.

16. Is there a way to print/see the dependencies between the different resources?
Yes, with terraform graph

Providers

17. Explain what is a “provider”

Terraform relies on plugins called “providers” to interact with cloud providers, SaaS providers, and other APIs…Each provider adds a set of resource types and/or data sources that Terraform can manage. Every resource type is implemented by a provider; without providers, Terraform can’t manage any kind of infrastructure.

18. Where can you find publicly available providers?

In the Terraform Registry

19. What are the names of the providers in this case?

terraform {
    required_providers {
      aws = {
        source  = "hashicorp/aws"
      }
      azurerm = {
        source  = "hashicorp/azurerm"
        version = "~> 3.0.2"
      }
    }
  }

azurerm and aws

20. You write a provider block like the following one and run terraform init

provider "aws" {
  region = "us-west-1"
}

21. Write a configuration of a Terraform provider (any type you would like)
AWS is one of the most popular providers in Terraform. Here is an example of how to configure it to use one specific region and specifying a specific version of the provider

terraform {
required_providers {
aws = {
source = “hashicorp/aws”
version = “~> 3.0”
}
}
}

# Configure the AWS Provider

provider “aws” {
region = “us-west-2”
}

22. Where Terraform installs providers from by default?

By default Terraform providers are installed from Terraform Registry

23. What is the Terraform Registry?

The Terraform Registry provides a centralized location for official and community-managed providers and modules.

24. Where providers are downloaded to? (when for example you run terraform init)

.terraform directory.

23. Describe in high level what happens behind the scenes when you run terraform init on on the following Terraform configuration

terraform {
required_providers {
aws = {
source = “hashicorp/aws”
version = “~> 3.0”
}
}
}