diff --git a/v2/content/posts/2024/tiny-cdn/gcore-dns-config.png b/v2/content/posts/2024/tiny-cdn/gcore-dns-config.png
new file mode 100644
index 0000000..6e6124c
Binary files /dev/null and b/v2/content/posts/2024/tiny-cdn/gcore-dns-config.png differ
diff --git a/v2/content/posts/2024/tiny-cdn/index.md b/v2/content/posts/2024/tiny-cdn/index.md
new file mode 100644
index 0000000..d089e5f
--- /dev/null
+++ b/v2/content/posts/2024/tiny-cdn/index.md
@@ -0,0 +1,1125 @@
++++
+title = "Building a Tiny CDN With pyinfra and Chimera Linux"
+date = 2024-12-08T10:18:01+10:00
+
+# [extra]
+# updated = 2024-07-26T10:34:50+10:00
++++
+
+In my quest to make my [linkedlist.org][linkedlist]—my link blog—faster, I set
+up multiple deployments around the world. I used pyinfra to automate the
+process and Chimera Linux as the host operating system. Join me on this
+adventure in over-engineering to see how I dropped the average response time
+across nine global locations from 807ms to 189ms without spending a fortune.
+
+
+
+
+
+A digram of what we're building.[^1]
+
+
+
+
+
+
+I previously wrote about my Linked List project in [Building and Launching My
+New Link Blog, linkedlist.org (Twice)](@/posts/2024/linked-list.md). While I
+should have been focussing on writing content for the site, I instead continued
+optimising what is currently a very low traffic website. Since the last post, I
+added caching so that the application only renders a page once. After that, the
+cached render result is reused for subsequent responses. These cached responses
+are typically generated in about a third of a millisecond (~323µs on average),
+which I was pretty happy with.
+
+The problem was, for my convenience the server that hosts Linked List was
+located in Australia, where I live. Unfortunately, most other people do not
+live in Australia, and we're a long way from everywhere. This meant that
+visitors would often encounter a lot of latency, just due to the
+distances covered. Is this _really_ a problem for a lightweight website with
+low traffic? Not really, but it bothered me, so I set about looking into
+options to improve the situation.
+
+{{ figure(image="posts/2024/tiny-cdn/response-times-before.png",
+ link="posts/2024/tiny-cdn/response-times-before.png",
+ resize_width=1600,
+ alt="Screenshot of updown.io monitoring. It has response times for nine locations around the world. The average total time is 807ms.",
+ caption="Average response timing from my monitoring at updown.io before the changes.") }}
+
+I briefly explored options like Cloudflare and Fastly. These would have been
+a cheap and sensible choice, but I didn't really feel like giving them (especially
+Cloudflare) even more of the Internet's traffic, no matter how miniscule. Plus,
+this is a personal project I do for fun, and just sticking a hosted cache in
+front of it is no fun.
+
+My stats in [GoatCounter] showed the top 10 visitor locations were:
+
+1. Australia
+1. United States
+1. Germany
+1. United Kingdom
+1. Netherlands
+1. France
+1. Ireland
+1. Japan
+1. Canada
+1. Poland
+
+This suggested I'd get a decent improvement in latency for the most number of
+visitors by adding a presence in Europe and the US. It's probably worth noting
+at this point that due to the nature of Linked List there is no central
+database, so it's a relatively simple to improve performance by deploying an
+instance in each location to be sped up.
+
+
+Linked List is implemented in Rust and has very meagre system requirements. I
+did some research into bargain-basement VPS's on [LowEndBox]. I found
+[RackNerd](affiliate link) were offering [KVM] based VMs with 1Gb of
+RAM[^2], custom ISO support, and US and European data centres for about US$12
+per **year**—probably still more than Cloudflare or Fastly but still cheap.
+
+I created servers in the New York US datacentre, as well as France. These
+servers were so cheap that I wanted to make sure the network and underlying
+hardware was not over provisioned before committing to them. They were running
+on older hardware[^3], but after some basic testing seemed fine.
+
+### Provisioning & Configuration
+
+To provision the servers I used some some [pyinfra] code I'd written previously
+to automate the installation of [Chimera Linux] in virtual machines. I picked
+Chimera Linux because I like it and it's easy to do minimal installs with just
+the things I want, and nothing more. There were some other benefits
+described later on too.
+
+The basic steps for provisioning are:
+
+1. Boot from the ISO and login as `root`
+2. Bootstrap ssh access to the live environment with [xdotool]:
+ * `xdotool windowfocus --sync $(xdotool selectwindow) type 'dinitctl start sshd && mkdir ~/.ssh && echo "ssh-rsa wmoore-key" > ~/.ssh/authorized_keys && ip addr && fdisk -l'`
+ * This works with local VMs in [virt-manager] as well as most web-based VNC
+ consoles provided by VPC hosts, including RackNerd.
+ * It uses `xdotool` to type into the selected window. The typed text enables
+ sshd and adds my public key to the SSH authorized_keys, then prints some
+ info about the network and disks that will be needed later.
+3. Run the pyinfra installation deployment against the server:
+ * `pyinfra -vvv --user root IP install-bios.py`
+
+After that, the server is rebooted and boots off its disk. I wrote subsequent
+pyinfra code to install and configure the server for hosting Linked List. This
+includes things like installing packages, creating users, nginx configuration,
+and ensuring services are started.
+
+To deploy the Linked List application I defined a [cports] template to
+build Linked List as a system (apk) package:
+
+```python
+pkgname = "linkedlist"
+# pkgver = "2.0.0_git20241108"
+# until this is fixed: https://github.com/pyinfra-dev/pyinfra/pull/1233
+pkgver = "2.0.12"
+pkgrel = 1
+_gitrev = "5e1aed8"
+_token = self.get_data("forge_token")
+build_style = "cargo"
+make_build_args = ["--bin", "linkedlistd"]
+make_install_args = [*make_build_args]
+hostmakedepends = ["cargo-auditable", "pkgconf"]
+makedepends = [
+ "oniguruma-devel",
+ "rust-std",
+]
+pkgdesc = "Linked List web application"
+maintainer = "Wesley Moore "
+license = "custom:none"
+url = "https://forge.wezm.net/wezm/linkedlist"
+source = f"https://forge.wezm.net/api/v1/repos/wezm/linkedlist/archive/{_gitrev}.tar.gz?access_token={_token}>linkedlist-{pkgver}.tar.gz"
+sha256 = "907978d15960b46f96fb1e5afaf3ff8dff888a00711dbe9c887e106314d85d70"
+
+def post_install(self):
+ self.install_sysusers(self.files_path / "sysusers.conf")
+ self.install_tmpfiles(self.files_path / "tmpfiles.conf")
+ self.install_service(self.files_path / "linkedlist")
+```
+
+As shown in the `post_install` hook, I was also able to make use of cports
+support for `systemd-tmpfiles` and `systemd-sysusers`[^4] to create the
+`linkedlist` user and data directory that will hold the content of the site.
+There's also a [Dinit] service definition to run and manage the server:
+
+```dinit
+# linkedlist service
+
+type = process
+command = /usr/bin/linkedlistd
+run-as = _linkedlist
+env-file = linkedlist.env
+logfile = /var/log/linkedlist.log
+depends-on = local.target
+smooth-recovery = true
+```
+
+The `env-file` allows the configuration to be changed without needing to rebuild the package.
+
+The servers are configured with an additional `apk` repository that points at
+my locally built packages. A pyinfa deployment takes care of building the
+packages and syncing the repo to the servers.
+
+Using Chimera Linux and building the binary through `cports` has some really
+neat benefits:
+
+1. The cports build tool, `cbuild`, does not require a Chimera Linux host.
+ I am able to re-build the package on my desktop that is still running Arch Linux.
+ All that's required is a recent version of `apk-tools`, which is in the [AUR].
+2. [My laptop](@/posts/2024/yoga-7x-snapdragon-developer-review/index.md) is
+ aarch64 based, but the servers are not. Cross-compiling is trivial with
+ `cbuild` though, just add `-a x86_64` to the `./cbuild -a x86_64 pkg
+ user/linkedlist` command, and it doesn't matter what host architecture I'm
+ on. This allows me to build and deploy an updated package from the
+ aarch64 WSL2 Chimera install on my laptop as well as my x86\_64 desktop.
+
+Cross-compiling Rust binaries is easy when just Rust is involved, but it gets
+harder when there are system library dependencies. For example, [oniguruma], which
+Linked List uses transitively. Using `cports` makes managing this easy.
+
+### TLS Certificates
+
+At this point I had the application running on the servers. Next I needed to
+handle certificates for `https`. I usually use [Lego] to manage certificates
+from Let's Encrypt. This posed a challenge though, as each server needed to get
+a copy of the certs.
+
+I explored various options here. It's common problem with a bunch of hosted and
+self-hosted solutions. All seemed too complicated for my need of syncing two
+files between three servers. I also wasn't super keen on one of the internet
+facing app servers having access to the others in order to push updated
+certificates to them.
+
+In the end I revived a [fanless Qotom Mini
+PC](https://qotom.net/product/29.html) I had at home. I again used my pyinfra
+Chimera install scripts to set it up, followed by more pyinfra code to
+configure it. This machine is responsible for managing the certs with Lego. It
+pushes out updated files when they're renewed via a renew-hook script. The
+script is managed by pyinfra and templated so that it syncs to each server in
+my pyinfra inventory within the `linkedlist_servers` group:
+
+```j2
+#!/bin/sh -x
+
+# push updated files to each server
+# restart nginx on each server
+
+if [ "$LEGO_CERT_DOMAIN" = "linkedlist.org" ]; then
+ {%- for server in linkedlist_servers %}
+ scp "$LEGO_CERT_PATH" "$LEGO_CERT_KEY_PATH" {{ server }}:/etc/ssl/lego/certificates/
+ ssh {{ server }} doas dinitctl restart nginx
+ {%- endfor %}
+fi
+```
+
+The compromise here was that I had to allow the cert server `ssh` access to
+each of the app servers. I created a dedicated user for this purpose. A `doas`
+rule allows this user to restart `nginx` to pick up the updated certificates:
+
+```doas
+permit nopass lego as root cmd dinitctl args restart nginx
+```
+
+### GeoDNS
+
+The final piece of the puzzle was how to determine which server to send a
+visitor to in order to minimise their latency. One option would be to use an
+edge compute service like [Deno Deploy], and proxy the request to the desired
+host. However, that adds another request, with its own latency—a bit over 20ms
+in my testing. I wanted to avoid the extra hop, so I looked into GeoDNS where
+the source IP of DNS requests is geo-located in order to resolve to an server
+IP that should minimise latency. Geo-location from IP address is imperfect but
+good enough for this project.
+
+I settled on [Gcore's Managed DNS service][gcore] as it had the necessary Geo
+features, a generous free tier, and a reasonable paid tier if that was ever
+reached. The UI is perhaps not super intuitive but I eventually got it set up
+as desired:
+
+{{ figure(image="posts/2024/tiny-cdn/gcore-dns-config.png",
+ link="posts/2024/tiny-cdn/gcore-dns-config.png",
+ width="709",
+ alt="Screenshot of the Gcore DNS configuration. There's three records. The first is assigned to North America, the second Europe, Africa, and South America, the last one is the default fallback record.",
+ caption="Gcore DNS configuration for linkedlist.org.") }}
+
+* North America is served by the US server
+* Europe, Africa, and South America are served by the French server
+* Everything else is served by the Australian server
+
+The default is the AU server because it's an existing server I already had that
+has more RAM and CPU than the others. Some of these mappings were informed by
+results from [PingBear]. I was also able to utilise the health checking in
+Gcore to avoid resolving DNS requests to servers that are down for some reason.
+
+### Taking It Live
+
+With the DNS sorted, I finally switched the NS records for `linkedlist.org` over
+to Gcore and started seeing each of the servers handle traffic 🎉.
+
+One final detail: how is new content deployed? The content is independent of
+the application, which monitors the content directory for changes, reloading as
+necessary. Therefore deploying new content is done with a simple `rsync`. All
+three servers are synced in parallel, with a bit of help from `xargs`:
+
+```sh
+printf "au\nny\nfr\n" | xargs -P 0 -I {host} \
+ rsync -azhP --delete content/ {host}.linkedlist.org:/var/lib/linkedlist/content/
+```
+
+Deployment is actually initiated via a Makefile, so I don't have to remember
+all that: `make deploy`.
+
+So how did I go, was this whole exercise in over-engineering worth it? Yep, it
+looks like it was. This is the end result in [updown.io], which shows a decent
+improvement over the stats at the beginning of the post:
+
+{{ figure(image="posts/2024/tiny-cdn/response-times-after.png",
+ link="posts/2024/tiny-cdn/response-times-after.png",
+ resize_width=1600,
+ alt="Screenshot of updown.io monitoring. It has response times for nine locations around the world. The average total time is 189ms.",
+ caption="Average response timing from my monitoring after the changes") }}
+
+Most of the monitored locations are showing lower average response times. In
+particular, the times in Europe are much improved. It also scores 100 for
+performance in a Lighthouse test in Chromium:
+
+{{ figure(image="posts/2024/tiny-cdn/linked-list-lighthouse.png",
+ link="posts/2024/tiny-cdn/linked-list-lighthouse.png",
+ width="741",
+ alt="Screenshot of a Lighthouse report for linkedlist.org in Chromium. It shows 100 for performance, 82 for accessibility, 100 for best practices, adn 92 for SEO.",
+ caption="linkedlist.org Lighthouse report.") }}
+
+While these results are good, I still only have a presence in three places
+around the world. Notably absent are servers in Asia and Africa. Should I start
+to see regular visitors from these or other countries the pyinfa config should
+make it straightforward to add servers as needed.
+
+For now though, I need to get back to writing on Linked List. If you haven't
+checked it out already please do. It's also easy to [follow via RSS, Mastodon,
+and Bluesky][follow].
+
+### References
+
+This series of three blog posts by Stefano Marinelli served as a good reference for
+what I was trying to achieve. Stefano uses Varnish to add a layer of caching, which
+I didn't do since the application manages caching itself.
+
+* [Building a Self-Hosted CDN for BSD Cafe Media](https://it-notes.dragas.net/2024/08/26/building-a-self-hosted-cdn-for-bsd-cafe-media/)
+* [Make Your Own CDN With OpenBSD Base and Just 2 Packages](https://it-notes.dragas.net/2024/08/29/make-your-own-cdn-openbsd/)
+* [Make Your Own CDN With NetBSD](https://it-notes.dragas.net/2024/09/03/make-your-own-cdn-netbsd/)
+
+[^1]: The icons in this diagram are from the [Haiku project][Haiku] used under
+ the terms of [their BSD license][haiku-licence]. Haiku is a really cool
+ operating system, you should check it out.
+[^2]: This give plenty of headroom as each server is currently only using about 240MiB of memory.
+[^3]: `/proc/cpuinfo` reports: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
+[^4]: I covered `systemd-sysusers` in [a previous post](@/posts/2023/systemd-sysusers-and-chimera-linux.md)
+
+[AUR]: https://aur.archlinux.org/
+[Chimera Linux]: https://chimera-linux.org/
+[cports]: https://github.com/chimera-linux/cports
+[Deno Deploy]: https://deno.com/deploy
+[dinit]: https://davmac.org/projects/dinit/
+[follow]: https://linkedlist.org/follow
+[gcore]: https://gcore.com/dns
+[GoatCounter]: https://www.goatcounter.com/
+[haiku-licence]: https://github.com/haiku/haiku/blob/8ada0c0e7c645e90d9e8d2addb10e7ffbd2bdf56/License.md
+[Haiku]: https://www.haiku-os.org/
+[KVM]: https://en.wikipedia.org/wiki/Kernel-based_Virtual_Machine
+[Lego]: https://go-acme.github.io/lego/
+[linkedlist]: https://linkedlist.org/
+[LowEndBox]: https://lowendbox.com/
+[oniguruma]: https://github.com/kkos/oniguruma
+[PingBear]: https://pingbear.com/
+[pyinfra]: https://pyinfra.com/
+[RackNerd]: https://my.racknerd.com/aff.php?aff=12841
+[updown.io]: https://updown.io/
+[virt-manager]: https://virt-manager.org/
+[xdotool]: https://github.com/jordansissel/xdotool
diff --git a/v2/content/posts/2024/tiny-cdn/linked-list-lighthouse.png b/v2/content/posts/2024/tiny-cdn/linked-list-lighthouse.png
new file mode 100644
index 0000000..f79d43c
Binary files /dev/null and b/v2/content/posts/2024/tiny-cdn/linked-list-lighthouse.png differ
diff --git a/v2/content/posts/2024/tiny-cdn/network-diagram.svg b/v2/content/posts/2024/tiny-cdn/network-diagram.svg
new file mode 100644
index 0000000..6f8fd6c
--- /dev/null
+++ b/v2/content/posts/2024/tiny-cdn/network-diagram.svg
@@ -0,0 +1,781 @@
+
+
+
+
diff --git a/v2/content/posts/2024/tiny-cdn/response-times-after.png b/v2/content/posts/2024/tiny-cdn/response-times-after.png
new file mode 100644
index 0000000..2eb52e7
Binary files /dev/null and b/v2/content/posts/2024/tiny-cdn/response-times-after.png differ
diff --git a/v2/content/posts/2024/tiny-cdn/response-times-before.png b/v2/content/posts/2024/tiny-cdn/response-times-before.png
new file mode 100644
index 0000000..5ec6e6e
Binary files /dev/null and b/v2/content/posts/2024/tiny-cdn/response-times-before.png differ