This is part of a series of blog posts I’m writing about hosting this blog from my homelab. You can find part 1 here: Hosting a blog from your Homelab

jenkins-pipeline-final

My current workflow for creating a new blog post is this:

  • Run a Hugo command to create a new markdown file.
  • Write my content in the file.
  • Run the hugo command to generate the build files.
  • Run a command to create a new docker image based on httpd which will serve the build files.
  • Bring down the old docker container hosting this blog using another docker command.
  • Finally, run another command to bring up the new container.

This works fine, but there’s definitely room for optimization. I decided that this is a great opportunity to learn about CI/CD pipelines. I had a vague idea about how CI/CD pipelines work, and this is the initial steps that I laid down:

  • Install Jenkins on my homelab.

  • Create a new Jenkins pipeline for my blog. Jenkins should watch for commits to the Gitea repo where I push new content to my blog.

  • I’ll have two branches on the repo: master and develop. develop will be where I push any new content that I write to first. Once I’m happy with the content and I’m ready to show it to the world, I’ll merge the changes from develop into master.

  • Jenkins will run different actions based on which branch I’m pushing to:

    If I’m pushing to develop:

    • Run codespell to check for spelling errors in any new content that I write.
    • Run the hugo command to generate the build files. Also pass in the --buildDrafts flag when running this command so that draft blog posts are also included in the final build.
    • Create a new docker image based on the Dockerfile in the repo with the tag personal-blog:develop.
    • Spin up a new docker container based on that image and map port 80 in the container to port 12345 on the host so that I can see a development version of my blog.

    If I’m pushing to master:

    • It’s almost the same flow as above, except, don’t pass in the --buildDrafts flag when running the hugo command.
    • Tag the docker image with personal-blog:master
    • Expose port 12346 to the host from the container.

With this pipeline, my new workflow would be much more smoother. I just write my content, commit and push to a repo in my self hosted Gitea instance, and I can see the result almost instantly.

Installing Jenkins

Installing Jenkins is pretty straight-forward. You have the usual option of installing it on bare metal and running it as a systemd service, or you can install it in Docker (there are many more options, but I believe these are the most relevant options for homelabbers. See here for all installation options). I decided to install it on Docker. The process was, again, pretty straight-forward. You create a new docker-compose.yml file and write these lines:

services:
  jenkins:
    image: john-jenkins:latest
    user: "jenkins:984"
    environment:
      - TZ=Asia/Kolkata
    ports:
      - 8075:8080
    volumes:
      - ./jenkins_home/:/var/jenkins_home
      - /var/run/docker.sock:/var/run/docker.sock
      - /usr/bin/docker:/usr/bin/docker
    restart: always

There are a few things I’ve modified here:

Note that I’m not using the official Jenkins image. This is because I needed a few CLI tools for my pipeline that are not included in the jenkins image. So I wrote a Dockerfile based on the jenkins image and added a few lines to install any package that I need.

FROM jenkins/jenkins:lts
USER root
RUN apt update
RUN apt install wget
RUN apt -y install codespell
USER jenkins

I’ve also added the line

user: "jenkins:984"

This basically runs the docker container as the user jenkins (you can create a new user in debian using the adduser command) and 984 group. This is because the pipeline that I need to setup requires creating a Docker container. There are a few different methods for setting up Docker in Jenkins in Docker, and this is one of them. You mount the docker binary and the docker unix socket in the container like so:

volumes:
  - /var/run/docker.sock:/var/run/docker.sock
  - /usr/bin/docker:/usr/bin/docker

Then you need to open the /etc/group file in your host OS and find the group number that corresponds to the docker group. Also remember to add the jenkins user to that group.

I’ve also added the line:

environment:
  - TZ=Asia/Kolkata

This just sets the timezone for Jenkins. This is necessary to make it easy to read logs in the Jenkins UI.

Create a new Docker image based on the Dockerfile that we created by running the command:

docker build -t john-jenkins:latest .

Now run:

docker compose up

in the directory that you have the docker-compose.yml file and your custom version of Jenkins should be up and running.

Configuring Jenkins for Gitea

Jenkins has a plugin for Gitea readily available to install. To do that, go to Manage Jenkins -> Plugins -> Available plugins and search for and install the ‘Gitea Plugin’.

The Gitea plugin for Jenkins has pretty good documentation on how to configure Gitea to work with Jenkins. The official link is here. Basically, you tell Jenkins about where your Gitea server is, create a new user in Gitea for Jenkins (you might need to add that user to your organization if you’re planning on using Gitea Organizations) and finally create a personal access token in Gitea for the Jenkins user to use in Jenkins.

Since I’m planning on creating a multi-branch pipeline (I have the master branch and the develop branch), I decided to go with the Organization Folder item in Jenkins. This means I just need to create an Organization in Gitea and Jenkins will automatically pickup any new repositories that I create under that Organization.

One thing to note when configuring the Organization Folder in Jenkins for Gitea is to make sure that you fill in the ‘Owner’ field under ‘Repository Sources’ with the name of your organization. It’s not a required field, but Jenkins won’t be able to scan your Organization without filling in this field correctly.

jenkins-gitea-organization-owner

Once that’s setup, you should be able to see your repository under your organization in Jenkins.

jenkins-repo

As one final step, also make sure that the option to scan multibranch pipeline triggers periodically in the item configuration is enabled. For some reason (probably due to something I’m missing), Jenkins doesn’t seem to be triggering a new job on push even though I’ve setup webhooks on Gitea. For now though, periodic 1 minute scans seem to work fine.

jenkins-periodic-trigger

Note: Jenkins is designed to run in a distributed environment. Meaning, there will be one machine running Jenkins and multiple client/slave machines running as Jenkins agents. These agents will run the actual pipeline jobs and report back to the main Jenkins machine. But Jenkins also allows us to run agents in the master machine too. This is the method I’ve adopted for now cause I won’t really be running a lot of concurrent jobs.

Creating the Jenkinsfile

The next step is to create the actual Jenkinsfile where we will define what we need the pipeline to do. For this we just need to create a new file named Jenkinsfile in our repository. The documentation for the syntax for the Jenkinsfile is available here. The contents of my Jenkinsfile for this blog look like this right now:

pipeline {
    agent any

    environment {
        HUGO_VERSION = '0.129.0'
        DEV_IP_ADDRESS = '192.168.1.100'
        DEV_PORT = '12345'
        PROD_PORT = '12346'
        HUGO_THEME_URL = 'https://github.com/adityatelange/hugo-PaperMod'
        HUGO_THEME_NAME = 'PaperMod'
    }

    stages {
        stage('Check for codespell errors') {
            steps {
                script {
                    sh '''
                        cd personal-website/content
                        codespell
                    '''
                }
            }
            post {
                failure {
                    echo 'codespell errors. FAILURE!'
                }
            }
        }
        stage('Download Hugo') {
            steps {
                cleanWs()
                checkout scm
                script {
                    def hugoUrl = "https://github.com/gohugoio/hugo/releases/download/v${env.HUGO_VERSION}/hugo_${env.HUGO_VERSION}_Linux-64bit.tar.gz"
                    sh """
                        wget -O hugo ${hugoUrl}
                        tar -xf hugo
                    """
                }
            }
        }
        stage('Setup Hugo theme') {
            steps {
                script {
                    sh """
                        cd personal-website
                        git clone ${env.HUGO_THEME_URL} themes/${env.HUGO_THEME_NAME} --depth=1
                    """
                }
            }
        }
        stage('Deploy to develop') {
            when {
                branch 'develop'
            }
            steps {
                script {
                    sh """
                    cd personal-website
                    ../hugo --baseURL  http://${env.DEV_IP_ADDRESS}:${DEV_PORT} --buildDrafts
                    cd ..
                    docker build -t personal-blog-develop:latest .
                    docker stop personal-blog-develop || true
                    docker rm personal-blog-develop || true
                    docker run -d --name personal-blog-develop -p ${env.DEV_PORT}:80 personal-blog-develop:latest
                """
                }
            }
        }
        stage('Deploy to prod') {
            when {
                branch 'master'
            }
            steps {
                script {
                    sh """
                    cd personal-website
                    ../hugo
                    cd ..
                    docker build -t personal-blog-master:latest .
                    docker stop personal-blog-master || true
                    docker rm personal-blog-master || true
                    docker run -d --name personal-blog-master -p ${env.PROD_PORT}:80 personal-blog-master:latest
                """
                }
            }
        }
    }
}

I won’t be going into the details of the syntax for a Jenkinsfile since that’s another entire blog post on its own. But I’ll just explain each stage in this pipeline.

The agent any line just means that Jenkins can use any agent to run this pipeline.

The environment block basically defines a bunch of variables that we use in the Jenkinsfile. I could have skipped this block entirely and hard-coded all the version numbers, port numbers and URLs directly, but storing them as environment variables means I just need to change them at one place. Storing them as variables is also best practise for readability and debugging.

The Check for codespell errors stage uses the Codespell CLI tool to check for spelling errors in the contents folder of my blog. My thought process for adding this stage was that since a blog will primarily consist of text content, checking for spell errors would be a relevant stage to include in a pipeline. If codespell finds any spelling errors in any of my blog post markdown files, the job will abort with an error.

The Download Hugo stage just downloads the Hugo binary from their Github Releases page and extracts the tar file. I make use of the HUGO_VERSION environment variable that I defined earlier in the environment block. If I need to upgrade to a newer version of Hugo, I just need to update the environment variable.

The Setup Hugo theme stage clones the Hugo theme called PaperMod into the /themes directory.

Next, the job will run one of two different stages depending on which branch I’ve pushed a commit to.

If the branch is develop (denoted by the when line), the job will execute the Deploy to develop stage. This stage basically runs the hugo command to generate the final HTML, CSS and JS files for the blog while also passing in the --buildDrafts parameter. This is required because when I’m writing a new blog post, the draft = true line will be present in the markdown files. This tells Hugo that the post is not ready for publishing. This can act as a sort of failsafe against pushing content still in progress to the public internet. I also pass in a --baseURL parameter to the command. This is because the develop version of the blog will be deployed to a private IP address in my local network and not the akjohn.dev domain. Next, this stage will run the docker build command to create a new Docker image based on the Dockerfile that I’ve created in the repository. Finally, we run the docker run command to spin up a new Docker container running this image, while mapping a port that I’ve defined in the environment section from the container.

If the branch is master, the steps we run are almost exactly the same, except for a few minor changes. We don’t pass in any parameters to the hugo command. And we also map a different port, one I’ve already mapped Nginx Proxy Manager to, from the container.

New workflow with Jenkins

With this new setup, all I need to create a new blog post in my blog is this:

  • Checkout to the develop branch in my repository.
  • Run the hugo new content command to create a new markdown file in the content directory.
  • Write my content in the markdown file.
  • Commit and push the changes.
  • Now I can see how the blog will look like by going to port 12345 on my home lab.
  • If I’m happy with the changes, I just merge the changes in develop to master and push again. My new blog post is now public on the internet.

This is significantly much less effort than the process that I used before!

I’ve also installed the Blue Ocean plugin in Jenkins. This gives me a pretty UI to monitor my pipeline in real-time.

jenkins-blue-ocean

Future Plans

I’ve still got a few more ideas on how to improve this pipeline.

  • Integrate some sort of AI/tool to proof read the blog post content. Right now I’m just checking for spell errors, but if I could somehow check for grammatical errors or logical errors in the text, it would be much more useful. I’ve found this repo called Language Tool which seems to do what I need.
  • Upload the production Docker image to a remote repository (like DockerHub) so that I can setup redundant servers to host my blog in case my home lab goes down.
  • Setup notifications on pipeline success or failure.

Overall, this was a fun little weekend project. Since I now have the skeleton of how to create a Jenkins pipeline, I’ll probably be creating pipelines for all my personal projects.