My Selfhosting Workflow

Hi there! I have here another update on selfhosting. I hosting all services with docker and using docker-compose.yml file to deploy services. I wrote about it more in Notes on Selfhosted Services. Ok, now i have one repository for configuration and each application has it own repository where I have source codes and where i building image. All my sorce codes are stored on GitLab and all jobs runs on Gitlab ci after pushing to repository.

How looks my workflow?

#

• Pull changes from original repository
• Merge it to my production branch on my server
• CI:build job will build image and push it to private registry
• CI:deploy job will connect to docker server over ssh, pull latest changes in my configuration registry (workbench), pulls latest image from registry and update service.

What im missing for now?

#

Backups. All services are backed up automatically every day to offsite location (BackBlaze B2). But when i want to upgrade service i have to do it manually (for now).

Let’s see closer to my workflow.

Pulling changes from original repository

#

In my private repository i created meow/production branch from which is image built. When is release new version of application manually pulling changes from repository and creating new branch meow/v.X.Y.Z which i will merge with GitLab later after review. So it looks like follows:

git clone my-private-repo.git
cd my-private-repo
git remote add downstream official-repo.git
git fetch downstream
git checkout -b meow/vX.Y.Z vX.Y.Z
git push origin meow/vX.Y.Z

git checkout -b meow/vX.Y.Z vX.Y.Z this is how you checking out new branch from tag. Next i updating theme for example or making some other changes (like add version suffix so i know i using image from my server) and creating merge request. Next is on GitLab CI.

Building image

#

Im using pretty standard script, you can see it below

docker-build:
  # Use the official docker image.
  image: docker:latest
  stage: build
  services:
    - docker:dind
  before_script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
  # Default branch leaves tag empty (= latest tag)
  # All other branches are tagged with the escaped branch name (commit ref slug)
  script:
    - |
      if [[ "$CI_COMMIT_BRANCH" == "$CI_DEFAULT_BRANCH" ]]; then
        tag=""
        echo "Running on default branch '$CI_DEFAULT_BRANCH': tag = 'latest'"
      else
        tag=":$CI_COMMIT_REF_SLUG"
        echo "Running on branch '$CI_COMMIT_BRANCH': tag = $tag"
      fi      
    - docker build --pull -t "$CI_REGISTRY_IMAGE${tag}" .
    - docker push "$CI_REGISTRY_IMAGE${tag}"
  # Run this job in a branch where a Dockerfile exists
  rules:
    - if: $CI_COMMIT_BRANCH
      exists:
        - Dockerfile
  tags:
    - hetzner

This script will build image from Dockerfile and pushing image to private repository. Next step is deploying to production server.

Deploying to server

#

This job runs only on default branch. (production branch)

There was needed some configuration. Each eserver has it’s own ssh key with read rights for workbench repository. Next i had to create user on which i running jobs. This user is not in sudo group (so it is cannot run privileged tasks on server), it has disabled password login - only option is connect with ssh key and at last it is in docker group so it can run docker-compose or docker commands.

Ok, after successful build gitlab fire deployment job. Each information for it are stored in Environmental variables. In before_script i configure ssh agent and importing private key for connect to server. (this is standard). Next in script i have to login to my private repository

ssh $SSH_USER@$SSH_HOST -p $SSH_PORT -o LogLevel=ERROR "docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY"

Login informations are passed from CI by default. When this is done I Checking if i have on server my workbench repository and based on this I clonning it or pulling chcnges.

- |
      ssh $SSH_USER@$SSH_HOST -p $SSH_PORT -o LogLevel=ERROR << 'EOF'
        if [ -d "$HOME/workbench" ]; then 
          cd $HOME/workbench
          git pull origin main
        else 
          git clone ssh://git@git.<my-server-url-redacted>/workbench.git $HOME/workbench
        fi
      EOF      

After this i just run taks which is needed for deployment (pulliing docker, backup database - in progress, not done yet, and more). As example here is my homer dashboard deployment script:

deploy-to-homeserver:
  stage: deploy
  variables:
    GIT_STRATEGY: none
  image: glcr.themeow.cloud/maymeow/toolkit:latest
  before_script:
    - mkdir -p ~/.ssh
    - eval $(ssh-agent -s)
    - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
    - ssh-add <(echo "$SSH_PRIVATE_KEY" | tr -d '\r')
  script:
    - ssh $SSH_USER@$SSH_HOST -p $SSH_PORT -o LogLevel=ERROR "docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY"
    - |
      ssh $SSH_USER@$SSH_HOST -p $SSH_PORT -o LogLevel=ERROR << 'EOF'
        if [ -d "$HOME/workbench" ]; then 
          cd $HOME/workbench
          git pull origin main
        else 
          git clone ssh://git@git.<my-server-url-redacted>/workbench.git $HOME/workbench
        fi
      EOF      
    - ssh $SSH_USER@$SSH_HOST -p $SSH_PORT -o LogLevel=ERROR 'cd $HOME/workbench && docker-compose -f applications/homer/docker-compose.yml pull'
    - ssh $SSH_USER@$SSH_HOST -p $SSH_PORT -o LogLevel=ERROR 'cd $HOME/workbench && docker-compose -f applications/homer/docker-compose.yml down && docker-compose -f applications/homer/docker-compose.yml up -d'
  tags:
    - hetzner
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'

So thats all. Thank you for reading.