I want to share my experience in implementing the Jenkins pipeline. In this tutorial, we will implement the Jenkins CI/CD Pipeline for Python applications. We will be building a pipeline as code, aka Declarative pipeline.
What is the declarative pipeline? The pipeline is the new feature of Jenkins, where we can write the Jenkins job as a sequence of steps. It is written in groovy. Since, it is a script we can keep it along with our code and can be easily replicated to other projects. We can also invoke Jenkins plugins in pipeline scripts.
Directives: Jenkins pipeline provides a list of directives that will be used in defining various operations.
- Agent: specifies in which environment the job will be run.
- Stages: a collection of stage or a working bundle.
- Stage: a set of steps to perform a certain operation.
- Steps: one or more steps that will be executed under each stage. Typically, it is a command.
- Post: it will run on job completion. we can trigger different options based on build status, if successful, unstable, or failure.
Step 1: choosing a server and setting working directories. By default, Jenkins chooses the home directory of the user to run jobs. We can change this if required.
agent {
node {
label "my-local-suer'
customWorkspace "/projects/debitcardcollection"
}
}Agent: specifies which environment we need to run the job. It can be server/host, docker, or Kubernetes.
- To run in any available server or host, we can specify agent as "any."
- To run a specific server, we need to specify it in "label."
- To add extra parameters like a custom working directory, we need to use node. "customWorkspace" is where our code will be checked out, and steps will be performed.
Checking out code: To checkout the code, we used the GIT plugin. We need to specify the following things. - branch: what branches we need to checkout - credentialsId: if we use the HTTP method to checkout, we need to specify the credentials to checkout. If it SSH, it's not required. - url: the repo URL.
stage("Checkout Code") {
steps {
script {
git branch: "master",
credentialsId: 'my-credentials',
url: 'https://user@github.org/myproject/sample-repo.git'
}
}
}The code will be checked out in the current working directory.
Installing the python packages: to install the python packages, we can use the shell command.
stage('Installing packages') {
steps {
script {
sh 'pip -r requirements.txt'
}
}
}Running Static Code Analysis: We choose Pylint to check the static code analysis and Warnings Next Generation Plugin to analyze the Pylint report. The plugin has features to mark build as unstable or fail based on the scores.
stage('Static Code Checking') {
steps {
script {
sh 'find . -name \\*.py | xargs pylint --load-plugins=pylint_django -f parseable | tee pylint.log'
recordIssues(
tool: pyLint(pattern: 'pylint.log'),
failTotalHigh: 10,
)
}
}
}- We used to shell command to run the Pylint for all python files in the project folder.
- recordIssues method of warnings next-generation plugin, and we have specified what kind of log and parameters. The build fails if the High category issues count is more than or equal to 10. there are multiple options available. Please visit https://www.jenkins.io/doc/pipeline/steps/warnings-ng/ for more information on configurations.
Running unit test cases: We can use any test tools like a pytest or nose to run the unit test. To publish the report on Jenkins we choose "CoberturaPublisher" plugin. There is no straight way to use the plugin in the pipeline. We need to invoke its class.
stage('Running Unit tests') {
steps {
script {
sh 'python3.7 manage.py test --keepdb --with-xunit --xunit-file=pyunit.xml --cover-xml --cover-xml-file=cov.xml tests/*.py || true'
step([$class: 'CoberturaPublisher',
coberturaReportFile: "cov.xml",
])
junit "pyunit.xml"
}
}- We need to specify the report file, and it must be "XML" file.
- We also have options to fail the build if tests fail or when the coverage is less. for detailed information available options, please visit. https://www.jenkins.io/doc/pipeline/steps/cobertura/
- To publish the test results we use Junit plugin
The completed pipeline script
pipeline {
agent {
node {
label 'my_local_server'
customWorkspace '/projects/'
}
}
stages {
stage('Checkout project') {
steps {
script {
git branch: "master",
credentialsId: 'my-credentials',
url: 'https://user@github.org/myproject/sample-repo.git'
}
}
}
stage('Installing packages') {
steps {
script {
sh 'pip install -r requirements.txt'
}
}
}
stage('Static Code Checking') {
steps {
script {
sh 'find . -name \\*.py | xargs pylint -f parseable | tee pylint.log'
recordIssues(
tool: pyLint(pattern: 'pylint.log'),
unstableTotalHigh: 100,
)
}
}
}
stage('Running Unit tests') {
steps {
script {
sh 'pytest --with-xunit --xunit-file=pyunit.xml --cover-xml --cover-xml-file=cov.xml tests/*.py || true'
step([$class: 'CoberturaPublisher',
coberturaReportFile: "cov.xml",
onlyStable: false,
failNoReports: true,
failUnhealthy: false,
failUnstable: false,
autoUpdateHealth: true,
autoUpdateStability: true,
zoomCoverageChart: true,
maxNumberOfBuilds: 10,
lineCoverageTargets: '80, 80, 80',
conditionalCoverageTargets: '80, 80, 80',
classCoverageTargets: '80, 80, 80',
fileCoverageTargets: '80, 80, 80',
])
junit "pyunit.xml"
}
}
}
}
}gist link: https://gist.github.com/srkama/0600ec839ca675f1a461d7739cb4e404