I have a couple of static websites I need to host in Kubernetes - straightforward enough, just with the minor extra challenge that the cluster has a security policy preventing pods from running as root.

First step is to create an image of my website running in Nginx that runs as a non-root user. There are a few posts online showing how to do this, in summary we need to:

  • Grant some additional permissions to the nginx user which already exists in the container.
  • Change to listen on a port > 1024, instead of port 80.
  • Remove the user directive from the default nginx configuration file. I don’t know if this is strictly necessary, but if you don’t remove it then nginx warns you that this directive is ignored.

Lets tackle the config files first - to see what the default config file looks like get a shell using docker run -it nginx bash, and then cat /etc/nginx/nginx.conf.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

Its the very top line user nginx; that that needs to be removed. While I’m here, I also need to update the port that nginx listens on. By default nginx includes the configuration from files in /etc/nginx/conf.d/ (bottom line), but in this case I think its cleaner to keep all the nginx config in a single file. Using ls /etc/nginx/conf.d/ I can see that there is only a single file default.conf being included, so I can inline this file, update the port number and clean up the commented out config to leave me with:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;

    keepalive_timeout  65;

    server {
        listen       8080;
        server_name  localhost;

        location / {
            root   /usr/share/nginx/html;
            index  index.html index.htm;
        }

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/html;
        }
    }
}

I also need to run id nginx to get the uid of the “nginx” user (101) - when specifying which user the image is going to run as it needs to be specified as a uid insted of the name so that Kubernetes can validate it against the pod security policy.

Next up, the Dockefile, the bulk of which is granting the “nginx” user some additional permissions:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

FROM nginx

# First, copy in the nginx configuration file (above)
COPY nginx.conf /etc/nginx/nginx.conf

# Magic commands to grant the required permissions to the nginx user
RUN chown -R nginx:nginx /usr/share/nginx/html && \
    chmod -R 755 /usr/share/nginx/html && \
    chown -R nginx:nginx /var/cache/nginx && \
    chown -R nginx:nginx /var/log/nginx && \
    chown -R nginx:nginx /etc/nginx/conf.d

# nginx also needs access to the pid file, which needs to be created first
RUN touch /var/run/nginx.pid && \
    chown -R nginx:nginx /var/run/nginx.pid

# Run as nginx user
USER 101

# Copy the static website into the default location
COPY html /usr/share/nginx/html

Finally, to run this in Kubernetes I need to build and push this image to a container registry and create a deployment (to run the container), and an ingress resource (to access it externally):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: site
spec:
  replicas: 1
  selector:
    matchLabels:
      name: site
  template:
    metadata:
      labels:
        name: site
    spec:
      containers:
      - name: site
        image: justinpealing/site:1
        ports:
        - name: http
          containerPort: 8080

---
apiVersion: v1
kind: Service
metadata:
  name: site
  labels:
    name: site
spec:
  selector:
    name: site
  ports:
  - protocol: TCP
    port: 80
    targetPort: http

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: site
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: site
          servicePort: 80