Create a specific Gitlab Runner with Kubernetes and GKE

Hi ! Do you know Gitlab, Kubernetes and Google Container Engine (GKE) ? I suppose. In this paper, we are going to combine those three technologies to create and use a specific Gitlab Runner. Enjoy !

1. Gitlab Runner ?

A Gitlab runner is a kind of build machine. Each time you trigger a pipeline on Gitlab with Gitlab-ci, your actions describe into .gitlab-ci.yaml will be processed onto a Gitlab runner.
When you start a project, by default, those runners are shared (for free!) with other Gitlab users. You do not have to configure anything, just write your ci file and push on your repo.
It is correct when you work on a personal project and you do not need to have a 100% uptime. But if you want to have your own runner, you have to create one and plug it to your project repository.

2. Configure your cluster

We are going to see one method for runner creation. It is the basic method, but if you want to go faster you can use helm a Kubernetes “package manager”, with a few commands you can create your pods. In fact, Helm does what we are going to see together. But first, we need to create our cluster !

1
2
3
$ gcloud container clusters create demo-gitlab-runner --zone us-east1-c --num-nodes 3
NAME ZONE MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS
demo-gitlab-runner us-east1-c 1.6.4 35.185.104.110 n1-standard-1 1.6.4 3 RUNNING

We are going to create a custom namespace gitlab-runner-ns.

1
2
3
4
5
6
7
8
9
$ kubectl create namespace gitlab-runner-ns
namespace "gitlab-runner-ns" created

$ kubectl get namespaces
NAME STATUS AGE
default Active 6m
gitlab-runner-ns Active 8s
kube-public Active 6m
kube-system Active 6m

Now, we set this new namespace as default, we can avoid to write --namespace gitlab-runner-ns after each command !

1
$ kubectl config set-context $(kubectl config current-context) --namespace=gitlab-runner-ns

Ok, let’$ go !

Homemade way!

For deploy our runner, we need a couple of things: a Docker gitlab-runner image, a deployment file, a config map file and a secret. In fact, runner does not need a bunch of thing, just a config.toml into our config map, an url and a token.

We can start with config-map.yaml (doc)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
apiVersion: v1
kind: ConfigMap
metadata:
name: gitlab-runner-cm
namespace: gitlab-runner-ns
data:
config.toml: |
concurrent = 4
check_interval = 30

entrypoint: |
#!/bin/bash

set -xe

cp /scripts/config.toml /etc/gitlab-runner/

# Register the runner
/entrypoint register --non-interactive \
--url $GITLAB_URL \
--executor kubernetes

# Start the runner
/entrypoint run --user=gitlab-runner \
--working-directory=/home/gitlab-runner

Now you can create your config map and check his content with describe.

1
2
3
4
5
6
7
8
$ kubectl create -f config-map.yaml
configmap "gitlab-runner-cm" created

$ kubectl get configmaps
NAME DATA AGE
gitlab-runner-cm 1 7s

$ kubectl describe configmap gitlab-runner-cm

Now, we need a token but we do not want to see this token in clear in our deployment file. So we need to create a secret volume with those two data. Do not forget to hash your token with a base64 !

1
$ echo -n "my_token" | base64
1
2
3
4
5
6
7
8
apiVersion: v1
kind: Secret
metadata:
name: gitlab-runner-secret
namespace: gitlab-runner-ns
type: Opaque
data:
runner-registration-token: my base 64 string

And create your secret with

1
$ kubectl create --validate -f secret.yaml

Now, we can pass to the “main” file, the deployment.yaml file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: gitlab-runner
namespace: gitlab-runner-ns
spec:
replicas: 1
selector:
matchLabels:
name: gitlab-runner
template:
metadata:
labels:
name: gitlab-runner
spec:
containers:
- name: gitlab-runner
image: gitlab/gitlab-runner:alpine-v9.3.0
command: ["/bin/bash", "/scripts/entrypoint"]
env:
- name: GITLAB_URL
value: "https://gitlab.com/"
- name: REGISTRATION_TOKEN
valueFrom:
secretKeyRef:
name: gitlab-runner-secret
key: runner-registration-token
imagePullPolicy: Always
volumeMounts:
- name: config
mountPath: /scripts
- name: cacerts
mountPath: /etc/gitlab-runner/certs
readOnly: true
restartPolicy: Always
volumes:
- name: config
configMap:
name: gitlab-runner-cm
- name: cacerts
hostPath:
path: /var/mozilla

As you can see, we pass two data as env variable : one from secret volume and one like a simple env variable.

The only specificity is the two mounted volumes: one for our configmap (config.toml) and the other for our SSL certificates.

Now , we create our deployment:

1
$ kubectl create --validate -f deployment.yaml

I suppose you should have this kind of error:

1
2
3
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
gitlab-runner-4147958637-1qtwc 0/1 rpc error: code = 2 desc = failed to start container "9089667a3f922a78a1e7f174b0203c1af536c0e65911d20ea9995e9af0d3ca5e": Error response from daemon: mkdir /usr/share/ca-certificates/mozilla: read-only file system 10 1m

Because Google security mount your “/“ as a file system (file system on Google Cloud). So you do what you want to do but for this demo, I am going to simply move my certs into /var folder.

Now, you are suppose to see your pod running:

1
2
3
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
gitlab-runner-1934066984-58wh9 1/1 Running 0 2m

Go on your CI/CD settings on Gitlab, I suppose you should see your specific runner as active with same id as GKE id. You can now start your CI/CD pipelines as usual ! If you build some docker images during your CI/CD, you have to add a parameter to your deployment.yaml and mount a volume pointing to Docker daemon. If not, you will have this error while building an image.

1
2
3
4
$ docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
Warning: failed to get default registry endpoint from daemon (Cannot connect to the Docker daemon at tcp://localhost:2375. Is the docker daemon running?). Using system default: https://index.docker.io/v1/
Cannot connect to the Docker daemon at tcp://localhost:2375. Is the docker daemon running?
ERROR: Job failed: error executing remote command: command terminated with non-zero exit code: Error executing in Docker Container: 1

This is the two parameters:

1
2
3
4
5
6
7
8
9
10
...
env:
- name: KUBERNETES_PRIVILEGED
value: "true"
...
volumes:
- name: var-run-docker-sock
hostPath:
path: /var/run/docker.sock
...

And into your .gitlab-ci.yaml, you have to add this variable

1
2
3
4
...
variables:
DOCKER_HOST: tcp://localhost:2375
...

You are now ready to manage your own specific runners with Kubernetes and Gitlab !