diff --git a/v2/content/posts/2022/alpine-linux-docker-infrastructure-three-years/index.md b/v2/content/posts/2022/alpine-linux-docker-infrastructure-three-years/index.md new file mode 100644 index 0000000..d54ab24 --- /dev/null +++ b/v2/content/posts/2022/alpine-linux-docker-infrastructure-three-years/index.md @@ -0,0 +1,106 @@ ++++ +title = "Alpine Linux and Docker Infrastructure Three Years Later" +date = 2022-02-28T09:52:10+10:00 + +[extra] +#updated = 2022-01-27T21:07:32+10:00 ++++ + +Three years ago I published, +[Rebuilding My Personal Infrastructure With Alpine Linux and Docker][original-post], +in which I described how I was hosting various applications using an [Alpine +Linux][alpine] host and [Docker] on a virtual machine at [Vultr]. I thought it +would be good to write a follow-up on how this worked out. + + + +The server I set up in 2019 is still running, although it has been through a +few Alpine upgrades in that time. Since there are very few packages installed +on the host, upgrades are painless. When I originally configured the +server Docker Compose was not in the stable Alpine package repos. It was added +in the mid-2019 Alpine 3.10 release. I was glad to be able to remove `pip` and +not have to manually manage compose updates after migrating to the package from +that point on. + +In 2019 the services I was hosting looked like this: + +![Before](https://www.wezm.net/images/2019/services.svg) + +In 2022 the situation is now: + +![After](services.svg) + +As you can see the server is now running more containers. It now hosts a number +of new services including a [Pleroma] instance and several web-applications I +built in Rust ([leaf], [quotes], [dewpoint]). I also retired the [Binary +Trance] Rails instance and migrated from [acme.sh] to [lego]. I ran a [Nitter] +instance for a while but removed it after it bots made it consume a huge amount +of bandwidth. See my [Burning 2.5Tb of Bandwidth Hosting a Nitter +Instance](@/posts/2021/nitter-bandwidth/index.md) post for more details on +that. + +The [Read Rust] application is implemented in [Crystal]. This one posed a bit +of a challenge. It is the only container I run that is not derived from a +minimal Alpine base image that I build myself. Instead, it uses `debian-slim`. I +need to use a specific version of the Crystal compiler to build my application, +which means I can't use the version in the Alpine package repos. Additionally +the Linux binaries that the project published for this version are not +dynamically linked but the bundled `libgc.a` assumes a `glibc` based system: + + $ crystal build src/asdf.cr + /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /crystal-0.34.0-1/bin/../lib/crystal/lib/libgc.a(pthread_support.o): in function `GC_thr_init': + pthread_support.c:(.text+0x1137): undefined reference to `gnu_get_libc_version' + ⋮ + +For this reason I opted to use a Debian base image for this container and that's +worked fine. + +Another notable change is the move from [acme.sh] to [lego] for managing TLS +certificates. In my original post I noted the following regarding how the +certificates were renewed: + +> Docker and cron is also a challenge. I ended up solving that with a simple +> solution: use the host cron to `docker exec` acme.sh in the `hitch` container. +> Perhaps not “pure” Docker but a lot simpler than some of the options I saw. + +It turned out that this never worked. I could see that cron was running the +script but the certs would not get renewed. For a while I ran the +script manually to renew them, which did work. Eventually I got sick of this, and +thinking `acme.sh` was to blame I searched for an alternative. + +I settled on [lego], an ACME client implemented in Go. I [discovered and +suggested a fix for a bug][lego-bug] and [contributed LuaDNS +support][lego-luadns] in May 2020 and then migrated over to using it instead of +`acme.sh`. It was in the final stages of test this that I discovered the cron +bug was in my script all along. Adding `-T` to inhibit `docker-compose exec` +from allocating a TTY fixed the issue. It's likely this would have fixed the +issue for `acme.sh` as well. Ultimately lego felt like the better option as the +code in `acme.sh` for constructing API requests and [parsing their +results][acme-parse] seemed quite fragile + +[Jokes aside][it-works-on-my-machine], I still find the Docker workflow of +iterating on an image locally then shipping it when it's working to be quite +pleasant. In summary this server has served me well over the last three years +and I have no immediate plans to rebuild again. It's been reliable and mostly +hassle free. Hosting on [Vultr] has also been reliable and stable over that +whole time with only the odd interruption for network maintenance or network +issues. + +[original-post]: https://www.wezm.net/technical/2019/02/alpine-linux-docker-infrastructure/ +[alpine]: https://alpinelinux.org/ +[Docker]: https://www.docker.com/ +[Vultr]: https://www.vultr.com/?ref=7903263 +[lego-luadns]: https://github.com/go-acme/lego/pull/1135 +[lego-bug]: https://github.com/go-acme/lego/issues/1150 +[acme-parse]: https://github.com/acmesh-official/acme.sh/blob/2a2d556551e9266a3924da205c1ede55d89a689d/dnsapi/dns_lua.sh#L117-L126 +[Pleroma]: https://pleroma.social/ +[lego]: https://go-acme.github.io/lego/ +[acme.sh]: https://github.com/acmesh-official/acme.sh +[Read Rust]: https://github.com/wezm/read-rust/ +[Crystal]: https://crystal-lang.org/ +[Binary Trance]: https://binarytrance.com/ +[leaf]: https://github.com/wezm/leaf +[quotes]: https://github.com/wezm/Quotes +[dewpoint]: https://github.com/wezm/dewpoint.7bit.org +[Nitter]: https://github.com/zedeus/nitter +[it-works-on-my-machine]: https://www.reddit.com/r/ProgrammerHumor/comments/cw58z7/it_works_on_my_machine/ diff --git a/v2/content/posts/2022/alpine-linux-docker-infrastructure-three-years/services.dot b/v2/content/posts/2022/alpine-linux-docker-infrastructure-three-years/services.dot new file mode 100644 index 0000000..dfaba0e --- /dev/null +++ b/v2/content/posts/2022/alpine-linux-docker-infrastructure-three-years/services.dot @@ -0,0 +1,24 @@ +digraph G { + node[shape=box]; + edge[headport=n]; + + subgraph cluster { + hitch + lego + } + hitch -> varnish; + varnish -> nginx; + varnish -> pkb; + varnish -> wizards; + varnish -> rust_melbourne; + varnish -> pleroma; + varnish -> read_rust; + varnish -> read_rust_leaf; + varnish -> quotes; + varnish -> dewpoint; + pkb -> syncthing; + read_rust -> postgresql:ne; + wizards -> postgresql; + rust_melbourne -> postgresql:ne; + pleroma -> postgresql:ne; +} diff --git a/v2/content/posts/2022/alpine-linux-docker-infrastructure-three-years/services.svg b/v2/content/posts/2022/alpine-linux-docker-infrastructure-three-years/services.svg new file mode 100644 index 0000000..17f7f99 --- /dev/null +++ b/v2/content/posts/2022/alpine-linux-docker-infrastructure-three-years/services.svg @@ -0,0 +1,191 @@ + + + + + + +G + + +cluster + + + + +hitch + +hitch + + + +varnish + +varnish + + + +hitch->varnish:n + + + + + +lego + +lego + + + +nginx + +nginx + + + +varnish->nginx:n + + + + + +pkb + +pkb + + + +varnish->pkb:n + + + + + +wizards + +wizards + + + +varnish->wizards:n + + + + + +rust_melbourne + +rust_melbourne + + + +varnish->rust_melbourne:n + + + + + +pleroma + +pleroma + + + +varnish->pleroma:n + + + + + +read_rust + +read_rust + + + +varnish->read_rust:n + + + + + +read_rust_leaf + +read_rust_leaf + + + +varnish->read_rust_leaf:n + + + + + +quotes + +quotes + + + +varnish->quotes:n + + + + + +dewpoint + +dewpoint + + + +varnish->dewpoint:n + + + + + +syncthing + +syncthing + + + +pkb->syncthing:n + + + + + +postgresql + +postgresql + + + +wizards->postgresql:n + + + + + +rust_melbourne->postgresql:ne + + + + + +pleroma->postgresql:ne + + + + + +read_rust->postgresql:ne + + + + +