benw2/wezm.net
1
0
Fork 0
forked from wezm/wezm.net
Code Pull requests Activity
wezm.net/content/technical/2019/02/alpine-linux-docker-infrastructure.md

17 KiB
Raw Blame History

For more than a decade I have run one or more servers to host a number of personal websites and web applications. Recently I decided it was time to rebuild the servers to address some issues and make improvements. The last time I did this was in 2016 when I switched the servers from Ubuntu to FreeBSD. The outgoing servers were managed with Ansible. After being a Docker skeptic for a long time I have finally come around to it recently and decided to rebuild on Docker. This post aims to describe some of the choices made, and why I made them.

Before we start I'd like to take a moment to acknowledge a few things. This infrastructure is mine and is built to my values in a way that works for me. You might make different choices and that's ok. You may think some parts are overly complex or that I jumped through too many hoops just to avoid some dependency. That's fine, you're free to build systems however you like. This post is my story and is not intended to tell you what to do, but to share how and why I built my systems the way I did.

Before the rebuild this is what my infrastructure looked like:

You'll note 3 servers, across 2 countries, and 2 hosting providers. Also the Rust Melbourne server was not managed by Ansible like the other two were.

I had a number of goals in mind with the rebuild:

  • Move everything to Australia (where I live)
  • Consolidate onto one server
  • https enable all websites

I set up my original infrastructure in the US because it was cheaper at the time and most traffic to the websites I host comes from the US. The Wizards Mattermost instance was added later. It's for a group of friends that are all in Australia. Being in the US made it quite slow at times, especially when sharing and viewing images.

Another drawback to administering servers in the US from AU was that it makes the Ansible cycle time of "make a change, run it, fix it, repeat", excruciatingly slow. It had been on my to do list for a long time to move Wizards to Australia but I kept putting it off because I didn't want to deal with Ansible.

While having a single server that does everything wouldn't be the recommended architecture for business systems, for personal hosting where the small chance of downtime isn't going to result in loss of income the simplicity won out, at least for now.

This is what I ended up building. Each box is a Docker container running on the host machine:

Graph of services

I haven't always been in favour of Docker but I think enough time has passed to show that it's probably here to stay. There are some really nice benefits to Docker managed services too. Such as, building locally and then shipping the image to production, and isolation from the host system (in the sense you can just nuke the container and rebuild it if needed).

Picking a Host OS

Moving to Docker unfortunately ruled out FreeBSD as the host system. There is a very old Docker port for FreeBSD but my previous attempts at using it showed that it was not in a good enough state to use for hosting. That meant I needed to find a suitable Linux distro to act as the Docker host.

Coming from FreeBSD I'm a fan of the stable base + up-to-date packages model. For me this ruled out Debian (stable) based systems, which I find often have out-of-date or missing packages -- especially in the latter stages of the release cycle. I did some research to see if there were any distros that used a BSD style model. Most I found were either abandoned or one person operations.

I then recalled that as part of his Sourcehut work, Drew DeVault was migrating things to Alpine Linux. I had played with Alpine in the past (before it became famous in the Docker world), and I consider Drew's use some evidence in its favour.

Alpine describes itself as follows:

Alpine Linux is an independent, non-commercial, general purpose Linux distribution designed for power users who appreciate security, simplicity and resource efficiency.

Now that's a value statement I can get behind! Other things I like about Alpine Linux:

  • It's small, only including the bare essentials:
    • It avoids bloat by using musl-libc (which is MIT licensed) and busybox userland.
    • It has a 37Mb installation ISO intended for virtualised server installations.
  • It was likely to be (and ended up being) the base of my Docker images.
  • It enables a number of security features by default.
  • Releases are made every ~6 months and are supported for 2 years.

Each release also has binary packages available in a stable channel that receives bug fixes and security updates for the lifetime of the release as well as a rolling edge channel that's always up-to-date.

Note that Alpine Linux doesn't use systemd, it uses OpenRC. This didn't factor into my decision at all. systemd has worked well for me on my Arch Linux systems. It may not be perfect but it does do a lot of things well. Benno Rice did a great talk at linux.conf.au 2019, titled, The Tragedy of systemd, that makes for interesting viewing on this topic.

Building Images

So with the host OS selected I set about building Docker images for each of the services I needed to run. There are a lot of pre-built Docker images for software like nginx, and PostgreSQL available on Docker Hub. Often they also have an alpine variant that builds the image from an Alpine base image. I decided early on that these weren't really for me:

  • A lot of them build the package from source instead of just installing the Alpine package.
  • The Docker build was more complicated that I needed as it was trying to be a generic image that anyone could pull and use.
  • I wasn't a huge fan of pulling random Docker images from the Internet, even if they were official images.

In the end I only need to trust one image from Docker Hub: The 5Mb Alpine image. All of my images are built on top of this one image.

An aspect of Docker that I don't really like is that inside the container you are root by default. When building my images I made a point of making the entrypoint processes run as a non-privileged user or configure the service drop down to a regular user after starting.

Most services were fairly easy to Dockerise. For example here is my nginx Dockerfile:

FROM alpine:3.9

RUN apk update && apk add --no-cache nginx

COPY nginx.conf /etc/nginx/nginx.conf

RUN mkdir -p /usr/share/www/ /run/nginx/ && \
  rm /etc/nginx/conf.d/default.conf

EXPOSE 80

STOPSIGNAL SIGTERM

ENTRYPOINT ["/usr/sbin/nginx", "-g", "daemon off;"]

I did not strive to make the images especially generic. They just need to work for me. However I did make a point not to bake any credentials into the images and instead used environment variables for things like that.

Let's Encrypt

I've been avoiding Let's Encrypt up until now. Partly because the short expiry of the certificates seems easy to mishandle. Partly because of certbot, the recommended client. By default certbot is interactive, prompting for answers when you run it the first time, it wants to be installed alongside the webserver so it can manipulate the configuration, it's over 30,000 lines of Python (excluding tests, and dependencies), the documentation suggests running magical certbot-auto scripts to install it... Too big and too magical for my liking.

Despite my reservations I wanted to enable https on all my sites and I wanted to avoid paying for certificates. This meant I had to make Let's Encrypt work for me. I did some research and finally settled on acme.sh. It's written in POSIX shell and uses curl and openssl to do its bidding.

To avoid the need for acme.sh to manipulate the webserver config I opted to use the DNS validation method (certbot can do this too). This requires a DNS provider that has an API so the client can dynamically manipulate the records. I looked through the large list of supported providers and settled on LuaDNS.

LuaDNS has a nice git based workflow where you define the DNS zones with small Lua scripts and the records are published when you push to the repo. They also have the requisite API for acme.sh. You can see my DNS repo at: https://github.com/wezm/dns

Getting the acme.sh + hitch combo to play nice proved to be bit of a challenge. acme.sh needs to periodically renew certificates from Let's Encrypt, these then need to be formatted for hitch and hitch told about them. In the end I built the hitch image off my acme.sh image. This goes against the Docker ethos of one service per container but acme.sh doesn't run a daemon, it's periodically invoked by cron so this seemed reasonable.

Docker and cron is also a challenge. I ended up solving that with a simple solution: use the host cron to docker exec acme.sh in the hitch container. Perhaps not "pure" Docker but a lot simpler than some of the options I saw.

Hosting

I've been a happy DigitalOcean customer for 5 years but they don't have a data centre in Australia. Vultr, which have a similar offering -- low cost, high performance servers and a well-designed admin interface -- do have a Sydney data centre. Other obvious options include AWS and GCP. I wanted to avoid these where possible as their server offerings are more expensive, and their platforms have a tendency to lock you in with platform specific features. Also in the case of Google, they are a massive surveillance capitalist that I don't trust at all. So Vultr were my host of choice for the new server.

Having said that, the thing with building your own images is that you need to make them available to the Docker host somehow. For this I used an Amazon Elastic Container Registry. It's much cheaper than Docker Hub for private images and is just a standard container registry so I'm not locked in.

Orchestration

Once all the services were Dockerised, there needed to be a way to run the containers, and make them aware of each other. A popular option for this is Kubernetes and for a larger, multi-server deployment it might be the right choice. For my single server operation I opted for Docker Compose, which is, "a tool for defining and running multi-container Docker applications". With Compose you specify all the services in a YAML file and it takes care of running them all together.

My Docker Compose file looks like this:

version: '3'
services:
  hitch:
    image: 791569612186.dkr.ecr.ap-southeast-2.amazonaws.com/hitch
    command: ["--config", "/etc/hitch/hitch.conf", "-b", "[varnish]:6086"]
    volumes:
      - ./hitch/hitch.conf:/etc/hitch/hitch.conf:ro
      - ./private/hitch/dhparams.pem:/etc/hitch/dhparams.pem:ro
      - certs:/etc/hitch/cert.d:rw
      - acme:/etc/acme.sh:rw
    ports:
      - "443:443"
    env_file:
      - private/hitch/development.env
    depends_on:
      - varnish
    restart: unless-stopped
  varnish:
    image: 791569612186.dkr.ecr.ap-southeast-2.amazonaws.com/varnish
    command: ["-F", "-a", ":80", "-a", ":6086,PROXY", "-p", "feature=+http2", "-f", "/etc/varnish/default.vcl", "-s", "malloc,256M"]
    volumes:
      - ./varnish/default.vcl:/etc/varnish/default.vcl:ro
    ports:
      - "80:80"
    depends_on:
      - nginx
      - pkb
      - binary_trance
      - wizards
      - rust_melbourne
    restart: unless-stopped
  nginx:
    image: 791569612186.dkr.ecr.ap-southeast-2.amazonaws.com/nginx
    volumes:
      - ./nginx/conf.d:/etc/nginx/conf.d:ro
      - ./volumes/www:/usr/share/www:ro
    restart: unless-stopped
  pkb:
    image: 791569612186.dkr.ecr.ap-southeast-2.amazonaws.com/pkb
    volumes:
      - pages:/home/pkb/pages:ro
    env_file:
      - private/pkb/development.env
    depends_on:
      - syncthing
    restart: unless-stopped
  binary_trance:
    image: 791569612186.dkr.ecr.ap-southeast-2.amazonaws.com/binary_trance
    env_file:
      - private/binary_trance/development.env
    depends_on:
      - db
    restart: unless-stopped
  wizards:
    image: 791569612186.dkr.ecr.ap-southeast-2.amazonaws.com/mattermost
    volumes:
      - ./private/wizards/config:/mattermost/config:rw
      - ./volumes/wizards/data:/mattermost/data:rw
      - ./volumes/wizards/logs:/mattermost/logs:rw
      - ./volumes/wizards/plugins:/mattermost/plugins:rw
      - ./volumes/wizards/client-plugins:/mattermost/client/plugins:rw
      - /etc/localtime:/etc/localtime:ro
    depends_on:
      - db
    restart: unless-stopped
  rust_melbourne:
    image: 791569612186.dkr.ecr.ap-southeast-2.amazonaws.com/mattermost
    volumes:
      - ./private/rust_melbourne/config:/mattermost/config:rw
      - ./volumes/rust_melbourne/data:/mattermost/data:rw
      - ./volumes/rust_melbourne/logs:/mattermost/logs:rw
      - ./volumes/rust_melbourne/plugins:/mattermost/plugins:rw
      - ./volumes/rust_melbourne/client-plugins:/mattermost/client/plugins:rw
      - /etc/localtime:/etc/localtime:ro
    depends_on:
      - db
    restart: unless-stopped
  db:
    image: 791569612186.dkr.ecr.ap-southeast-2.amazonaws.com/postgresql
    volumes:
      - postgresql:/var/lib/postgresql/data
    ports:
      - "127.0.0.1:5432:5432"
    env_file:
      - private/postgresql/development.env
    restart: unless-stopped
  syncthing:
    image: 791569612186.dkr.ecr.ap-southeast-2.amazonaws.com/syncthing
    volumes:
      - syncthing:/var/lib/syncthing:rw
      - pages:/var/lib/syncthing/Sync:rw
    ports:
      - "127.0.0.1:8384:8384"
      - "22000:22000"
      - "21027:21027/udp"
    restart: unless-stopped
volumes:
  postgresql:
  certs:
  acme:
  pages:
  syncthing:

Bringing all the services up is one command:

docker-compose -f docker-compose.yml -f production.yml up -d

The best bit is I can develop and test it all in isolation locally. Then when it's working, push to ECR and then run docker-compose on the server to bring in the changes. This is a huge improvement over my previous Ansible workflow and should make adding or removing new services in the future fairly painless.

Closing Thoughts

The new server has been running issue free so far. All sites are now redirecting to their https variants with Strict-Transport-Security headers set and get an A grade on the SSL Labs test. The Wizards Mattermost is much faster now that it's in Australia too.

There is one drawback to this move though: my sites are now slower for a lot of visitors. https adds some initial negotiation overhead and if you're reading this from outside Australia there's probably a bunch more latency than before.

I did some testing with WebPageTest to get a feel for the impact of this. My sites are already quite compact. Firefox tells me this page and all resources is 171KB / 54KB transferred. So there's not a lot of slimming to be done there. One thing I did notice was the TLS negotiation was happening for each of the parallel connections the browser opened to load the site.

Some research suggested HTTP/2 might help as it multiplexes requests on a single connection and only performs the TLS negotiation once. So I decided to live on the edge a little and enable Varnish's experimental HTTP/2 support. Retrieving the site over HTTP/2 did in fact reduce the TLS negotiations to one.

Thanks for reading, I hope the bits didn't take to long to get from Australia to wherever you are. Happy computing!