wezm/wezm.net
1
0
Fork 1
mirror of https://github.com/wezm/wezm.net.git synced 2024-09-19 21:18:27 +00:00
Code Issues Projects Releases Packages Wiki Activity

Add alpine-linux-docker-infrastructure

Browse source
This commit is contained in:
Wesley Moore 2019-02-28 18:09:53 +11:00
parent 5d02e6572d
commit cd37a544b8
No known key found for this signature in database
GPG key ID: BF67766C0BC2D0EE
4 changed files with 573 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

408
content/technical/2019/02/alpine-linux-docker-infrastructure.md Normal file
View file

@ -0,0 +1,408 @@
For more than a decade I have run one or more servers to host a number of
personal websites and web applications. Recently I decided it was time to
rebuild the servers to address some issues and make improvements. The last time
I did this was in 2016 when I switched the servers from [Ubuntu] to [FreeBSD].
The outgoing servers were managed with [Ansible]. After being a Docker skeptic
for a long time I have finally come around to it recently and decided to
rebuild on [Docker]. This post aims to describe some of the choices made, and
why I made them.
_Before we start I'd like to take a moment to acknowledge a few things. This
infrastructure is mine and is built to my values in a way that works for me.
You might make different choices and that's ok. You may think some parts are
overly complex or that I jumped through too many hoops just to avoid some
dependency. That's fine, you're free to build systems however you like.
This post is my story and is not intended to tell you what to do, but to
share how and why I built my systems the way I did._
Before the rebuild this is what my infrastructure looked like:
- FreeBSD 11 server in [DigitalOcean] hosting:
- [PostgreSQL] 9
- [nginx]
- [Varnish]
- 2 [Rails] apps
- Static sites
- Debian 9 server in DigitalOcean hosting:
- Wizards [Mattermost] instance
- FreeBSD 12 server in [Vultr] hosting:
- [rust.melbourne] Mattermost instance
- PostgreSQL 11
You'll note 3 servers, across 2 countries, and 2 hosting providers. Also the
Rust Melbourne server was not managed by Ansible like the other two were.
I had a number of goals in mind with the rebuild:
- Move everything to Australia (where I live)
- Consolidate onto one server
- https enable all websites
I set up my original infrastructure in the US because it was cheaper at the
time and most traffic to the websites I host comes from the US. The Wizards
Mattermost instance was added later. It's for a group of friends that are all
in Australia. Being in the US made it quite slow at times, especially when
sharing and viewing images.
Another drawback to administering servers in the US from AU was that it makes
the Ansible cycle time of "make a change, run it, fix it, repeat", excruciatingly
slow. It had been on my to do list for a long time to move Wizards to Australia
but I kept putting it off because I didn't want to deal with Ansible.
While having a single server that does everything wouldn't be the
recommended architecture for business systems, for personal hosting where the
small chance of downtime isn't going to result in loss of income the simplicity
won out, at least for now.
This is what I ended up building. Each box is a Docker container running on the
host machine:
![Graph of services](/images/2019/services.svg)
- `pkb` is <https://linkedlist.org>
- `binary_trance` is <https://binarytrance.com>
- `wizards` and `rust_melbourne` are [Mattermost] instances
- The rest are software of the same name
I haven't always been in favour of Docker but I think enough time has passed to
show that it's probably here to stay. There are some really nice benefits to
Docker managed services too. Such as, building locally and then shipping the image
to production, and isolation from the host system (in the sense you can just
nuke the container and rebuild it if needed).
## Picking a Host OS
Moving to Docker unfortunately ruled out FreeBSD as the host system. There is a
[very old Docker port for FreeBSD][docker-freebsd] but my previous attempts at using it showed
that it was not in a good enough state to use for hosting. That meant I
needed to find a suitable Linux distro to act as the Docker host.
Coming from FreeBSD I'm a fan of the stable base + up-to-date packages model.
For me this ruled out Debian (stable) based systems, which I find often
have out-of-date or missing packages -- especially in the latter stages of
the release cycle. I did some research to see if there were any distros that
used a BSD style model. Most I found were either abandoned or one person
operations.
I then recalled that as part of his [Sourcehut] work, [Drew DeVault was
migrating][sr.ht-announce] things to [Alpine Linux]. I had played with Alpine
in the past (before it became famous in the Docker world), and I consider Drew's
use some evidence in its favour.
Alpine describes itself as follows:
> Alpine Linux is an independent, non-commercial, general purpose Linux
> distribution designed for power users who appreciate security, simplicity and
> resource efficiency.
Now that's a value statement I can get behind! Other things I like about Alpine
Linux:
- It's small, only including the bare essentials:
- It avoids bloat by using [musl-libc] (which is MIT licensed) and
[busybox userland][busybox].
- It has a 37Mb installation ISO intended for virtualised server
installations.
- It was likely to be (and ended up being) the base of my Docker images.
- It enables a number of security features by default.
- Releases are made every ~6 months and are supported for 2 years.
Each release also has binary packages available in a stable channel that
receives bug fixes and security updates for the lifetime of the release as well
as a rolling edge channel that's always up-to-date.
Note that Alpine Linux doesn't use [systemd], it uses [OpenRC]. This didn't
factor into my decision at all. `systemd` has worked well for me on my Arch
Linux systems. It may not be perfect but it does do a lot of things well. Benno
Rice did a great talk at linux.conf.au 2019, titled, [The Tragedy of
systemd][systemd-tragedy], that makes for interesting viewing on this topic.
## Building Images
So with the host OS selected I set about building Docker images for each of the
services I needed to run. There are a lot of pre-built Docker images for
software like nginx, and PostgreSQL available on [Docker Hub]. Often they also
have an `alpine` variant that builds the image from an Alpine base image. I
decided early on that these weren't really for me:
- A lot of them build the package from source instead of just installing the
Alpine package.
- The Docker build was more complicated that I needed as it was trying to be
a generic image that anyone could pull and use.
- I wasn't a huge fan of pulling random Docker images from the Internet, even
if they were official images.
In the end I only need to trust one image from [Docker Hub]: The 5Mb [Alpine
image][alpine-docker-image]. All of my images are built on top of this one
image.
An aspect of Docker that I don't really like is that inside the container you
are root by default. When building my images I made a point of making the
entrypoint processes run as a non-privileged user or configure the service drop
down to a regular user after starting.
Most services were fairly easy to Dockerise. For example here is my nginx
`Dockerfile`:
```language-docker
FROM alpine:3.9
RUN apk update && apk add --no-cache nginx
COPY nginx.conf /etc/nginx/nginx.conf
RUN mkdir -p /usr/share/www/ /run/nginx/ && \
rm /etc/nginx/conf.d/default.conf
EXPOSE 80
STOPSIGNAL SIGTERM
ENTRYPOINT ["/usr/sbin/nginx", "-g", "daemon off;"]
```
I did not strive to make the images especially generic. They just need to work
for me. However I did make a point not to bake any credentials into the images
and instead used environment variables for things like that.
## Let's Encrypt
I've been avoiding [Let's Encrypt] up until now. Partly because the short expiry of the
certificates seems easy to mishandle. Partly because of [certbot], the recommended
client. By default `certbot` is interactive, prompting for answers when you run
it the first time, it wants to be installed alongside the webserver so it can
manipulate the configuration, it's over 30,000 lines of Python (excluding
tests, and dependencies), the documentation suggests running magical
`certbot-auto` scripts to install it... Too big and too magical for my liking.
Despite my reservations I wanted to enable https on all my sites and I wanted
to avoid paying for certificates. This meant I had to make Let's Encrypt work
for me. I did some research and finally settled on [acme.sh]. It's
written in POSIX shell and uses `curl` and `openssl` to do its bidding.
To avoid the need for `acme.sh` to manipulate the webserver config I opted to
use the DNS validation method (`certbot` can do this too). This requires a DNS
provider that has an API so the client can dynamically manipulate the records.
I looked through the large list of supported providers and settled on [LuaDNS].
LuaDNS has a nice git based workflow where you define the DNS zones with small
Lua scripts and the records are published when you push to the repo. They also
have the requisite API for `acme.sh`. You can see my DNS repo at:
<https://github.com/wezm/dns>
Getting the [acme.sh] + [hitch] combo to play nice proved to be bit of a
challenge. `acme.sh` needs to periodically renew certificates from Let's
Encrypt, these then need to be formatted for `hitch` and `hitch` told about
them. In the end I built the `hitch` image off my `acme.sh` image. This goes
against the Docker ethos of one service per container but `acme.sh` doesn't run
a daemon, it's periodically invoked by cron so this seemed reasonable.
Docker and cron is also a challenge. I ended up solving that with a
simple solution: use the host cron to `docker exec` `acme.sh` in the `hitch`
container. Perhaps not "pure" Docker but a lot simpler than some of the options
I saw.
## Hosting
I've been a happy [DigitalOcean] customer for 5 years but they don't have a
data centre in Australia. [Vultr], which have a similar offering -- low cost,
high performance servers and a well-designed admin interface -- do have a
Sydney data centre. Other obvious options include AWS and GCP. I wanted to
avoid these where possible as their server offerings are more expensive, and
their platforms have a tendency to lock you in with platform specific features.
Also in the case of Google, they are a massive [surveillance capitalist] that I
don't trust at all. So Vultr were my host of choice for the new server.
Having said that, the thing with building your own images is that you need to
make them available to the Docker host somehow. For this I used an [Amazon
Elastic Container Registry][ECR]. It's much cheaper than Docker Hub for private
images and is just a standard container registry so I'm not locked in.
## Orchestration
Once all the services were Dockerised, there needed to be a way to run the
containers, and make them aware of each other. A popular option for this is
[Kubernetes] and for a larger, multi-server deployment it might be the right
choice. For my single server operation I opted for [Docker Compose], which is,
"a tool for defining and running multi-container Docker applications". With
Compose you specify all the services in a YAML file and it takes care of
running them all together.
My Docker Compose file looks like this:
```language-yaml
version: '3'
services:
hitch:
image: 791569612186.dkr.ecr.ap-southeast-2.amazonaws.com/hitch
command: ["--config", "/etc/hitch/hitch.conf", "-b", "[varnish]:6086"]
volumes:
- ./hitch/hitch.conf:/etc/hitch/hitch.conf:ro
- ./private/hitch/dhparams.pem:/etc/hitch/dhparams.pem:ro
- certs:/etc/hitch/cert.d:rw
- acme:/etc/acme.sh:rw
ports:
- "443:443"
env_file:
- private/hitch/development.env
depends_on:
- varnish
restart: unless-stopped
varnish:
image: 791569612186.dkr.ecr.ap-southeast-2.amazonaws.com/varnish
command: ["-F", "-a", ":80", "-a", ":6086,PROXY", "-p", "feature=+http2", "-f", "/etc/varnish/default.vcl", "-s", "malloc,256M"]
volumes:
- ./varnish/default.vcl:/etc/varnish/default.vcl:ro
ports:
- "80:80"
depends_on:
- nginx
- pkb
- binary_trance
- wizards
- rust_melbourne
restart: unless-stopped
nginx:
image: 791569612186.dkr.ecr.ap-southeast-2.amazonaws.com/nginx
volumes:
- ./nginx/conf.d:/etc/nginx/conf.d:ro
- ./volumes/www:/usr/share/www:ro
restart: unless-stopped
pkb:
image: 791569612186.dkr.ecr.ap-southeast-2.amazonaws.com/pkb
volumes:
- pages:/home/pkb/pages:ro
env_file:
- private/pkb/development.env
depends_on:
- syncthing
restart: unless-stopped
binary_trance:
image: 791569612186.dkr.ecr.ap-southeast-2.amazonaws.com/binary_trance
env_file:
- private/binary_trance/development.env
depends_on:
- db
restart: unless-stopped
wizards:
image: 791569612186.dkr.ecr.ap-southeast-2.amazonaws.com/mattermost
volumes:
- ./private/wizards/config:/mattermost/config:rw
- ./volumes/wizards/data:/mattermost/data:rw
- ./volumes/wizards/logs:/mattermost/logs:rw
- ./volumes/wizards/plugins:/mattermost/plugins:rw
- ./volumes/wizards/client-plugins:/mattermost/client/plugins:rw
- /etc/localtime:/etc/localtime:ro
depends_on:
- db
restart: unless-stopped
rust_melbourne:
image: 791569612186.dkr.ecr.ap-southeast-2.amazonaws.com/mattermost
volumes:
- ./private/rust_melbourne/config:/mattermost/config:rw
- ./volumes/rust_melbourne/data:/mattermost/data:rw
- ./volumes/rust_melbourne/logs:/mattermost/logs:rw
- ./volumes/rust_melbourne/plugins:/mattermost/plugins:rw
- ./volumes/rust_melbourne/client-plugins:/mattermost/client/plugins:rw
- /etc/localtime:/etc/localtime:ro
depends_on:
- db
restart: unless-stopped
db:
image: 791569612186.dkr.ecr.ap-southeast-2.amazonaws.com/postgresql
volumes:
- postgresql:/var/lib/postgresql/data
ports:
- "127.0.0.1:5432:5432"
env_file:
- private/postgresql/development.env
restart: unless-stopped
syncthing:
image: 791569612186.dkr.ecr.ap-southeast-2.amazonaws.com/syncthing
volumes:
- syncthing:/var/lib/syncthing:rw
- pages:/var/lib/syncthing/Sync:rw
ports:
- "127.0.0.1:8384:8384"
- "22000:22000"
- "21027:21027/udp"
restart: unless-stopped
volumes:
postgresql:
certs:
acme:
pages:
syncthing:
```
Bringing all the services up is one command:
docker-compose -f docker-compose.yml -f production.yml up -d
The best bit is I can develop and test it all in isolation locally. Then when
it's working, push to ECR and then run `docker-compose` on the server to bring
in the changes. This is a huge improvement over my previous Ansible workflow
and should make adding or removing new services in the future fairly painless.
## Closing Thoughts
The new server has been running issue free so far. All sites are now
redirecting to their https variants with `Strict-Transport-Security` headers
set and get an A grade on the [SSL Labs test]. The Wizards Mattermost is _much_
faster now that it's in Australia too.
There is one drawback to this move though: my sites are now slower for a lot of
visitors. https adds some initial negotiation overhead and if you're reading
this from outside Australia there's probably a bunch more latency than
before.
I did some testing with [WebPageTest] to get a feel for the impact of this.
My sites are already quite compact. Firefox tells me this page and all
resources is 171KB / 54KB transferred. So there's not a lot of slimming
to be done there. One thing I did notice was the TLS negotiation was happening
for each of the parallel connections the browser opened to load the site.
Some research suggested HTTP/2 might help as it multiplexes requests on a
single connection and only performs the TLS negotiation once. So I decided to
live on the edge a little and enable [Varnish's experimental HTTP/2
support][varnish-http2]. Retrieving the site over HTTP/2 did in fact reduce the
TLS negotiations to one.
Thanks for reading, I hope the bits didn't take to long to get from Australia
to wherever you are. Happy computing!
[acme.sh]: https://github.com/Neilpang/acme.sh
[Alpine Linux]: https://alpinelinux.org/
[alpine-docker-image]: https://hub.docker.com/_/alpine
[Ansible]: https://www.ansible.com/
[busybox]: https://www.busybox.net/
[certbot]: https://certbot.eff.org/
[DigitalOcean]: https://m.do.co/c/0eb3d3d839ea
[Docker Compose]: https://docs.docker.com/compose/overview/
[Docker Hub]: https://hub.docker.com/
[docker-freebsd]: https://www.freshports.org/sysutils/docker-freebsd/
[Docker]: https://www.docker.com/
[ECR]: https://aws.amazon.com/ecr/
[FreeBSD]: https://www.freebsd.org/
[hitch]: https://hitch-tls.org/
[Kubernetes]: https://kubernetes.io/
[Let's Encrypt]: https://letsencrypt.org/
[LuaDNS]: https://luadns.com/
[Mattermost]: https://mattermost.com/
[musl-libc]: http://www.musl-libc.org/
[OpenRC]: https://wiki.gentoo.org/wiki/Project:OpenRC
[rust.melbourne]: https://rust.melbourne/
[Sourcehut]: https://sourcehut.org/
[sr.ht-announce]: https://lists.sr.ht/~sircmpwn/sr.ht-announce/%3C20190117003837.GA6037%40homura.localdomain%3E
[SSL Labs test]: https://www.ssllabs.com/ssltest/index.html
[surveillance capitalist]: https://en.wikipedia.org/wiki/Surveillance_capitalism
[systemd-tragedy]: https://youtu.be/o_AIw9bGogo
[systemd]: https://freedesktop.org/wiki/Software/systemd/
[Ubuntu]: https://www.ubuntu.com/
[Vultr]: https://www.vultr.com/?ref=7903263
[WebPageTest]: https://www.webpagetest.org/
[Varnish]: https://varnish-cache.org/
[varnish-http2]: https://info.varnish-software.com/blog/varnish-cache-5-http2-support
[nginx]: http://nginx.org/
[PostgreSQL]: https://www.postgresql.org/
[Rails]: https://rubyonrails.org/

9
content/technical/2019/02/alpine-linux-docker-infrastructure.yaml Normal file
View file

@ -0,0 +1,9 @@
---
title: Rebuilding My Personal Infrastructure With Alpine Linux and Docker
extra: My story of replacing three servers, across two countries, running two operating systems with a single server running Alpine Linux and Docker.
kind: article
section: technical
created_at: 2019-02-28 09:45:00.000000000 +11:00
keywords:
- linux
short_url:

19
output/images/2019/services.dot Normal file
View file

@ -0,0 +1,19 @@
digraph G {
node[shape=box];
edge[headport=n];
subgraph cluster {
hitch
"acme.sh"
}
hitch -> varnish;
varnish -> nginx;
varnish -> pkb;
varnish -> binary_trance;
varnish -> wizards;
varnish -> rust_melbourne;
pkb -> syncthing;
binary_trance -> postgresql:nw;
wizards -> postgresql;
rust_melbourne -> postgresql:ne;
}

137
output/images/2019/services.svg Normal file
View file

@ -0,0 +1,137 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 2.40.1 (20161225.0304)
-->
<!-- Title: G Pages: 1 -->
<svg width="452pt" height="276pt"
viewBox="0.00 0.00 451.50 276.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 272)">
<title>G</title>
<polygon fill="#ffffff" stroke="transparent" points="-4,4 -4,-272 447.5,-272 447.5,4 -4,4"/>
<g id="clust1" class="cluster">
<title>cluster</title>
<polygon fill="none" stroke="#000000" points="77,-208 77,-260 227,-260 227,-208 77,-208"/>
</g>
<!-- hitch -->
<g id="node1" class="node">
<title>hitch</title>
<polygon fill="none" stroke="#000000" points="219,-252 165,-252 165,-216 219,-216 219,-252"/>
<text text-anchor="middle" x="192" y="-230.3" font-family="Times,serif" font-size="14.00" fill="#000000">hitch</text>
</g>
<!-- varnish -->
<g id="node3" class="node">
<title>varnish</title>
<polygon fill="none" stroke="#000000" points="221.5,-180 162.5,-180 162.5,-144 221.5,-144 221.5,-180"/>
<text text-anchor="middle" x="192" y="-158.3" font-family="Times,serif" font-size="14.00" fill="#000000">varnish</text>
</g>
<!-- hitch&#45;&gt;varnish -->
<g id="edge1" class="edge">
<title>hitch&#45;&gt;varnish:n</title>
<path fill="none" stroke="#000000" d="M192,-215.5056C192,-208.0111 192,-199.0501 192,-190.2162"/>
<polygon fill="#000000" stroke="#000000" points="195.5001,-190 192,-180 188.5001,-190 195.5001,-190"/>
</g>
<!-- acme.sh -->
<g id="node2" class="node">
<title>acme.sh</title>
<polygon fill="none" stroke="#000000" points="147,-252 85,-252 85,-216 147,-216 147,-252"/>
<text text-anchor="middle" x="116" y="-230.3" font-family="Times,serif" font-size="14.00" fill="#000000">acme.sh</text>
</g>
<!-- nginx -->
<g id="node4" class="node">
<title>nginx</title>
<polygon fill="none" stroke="#000000" points="54,-108 0,-108 0,-72 54,-72 54,-108"/>
<text text-anchor="middle" x="27" y="-86.3" font-family="Times,serif" font-size="14.00" fill="#000000">nginx</text>
</g>
<!-- varnish&#45;&gt;nginx -->
<g id="edge2" class="edge">
<title>varnish&#45;&gt;nginx:n</title>
<path fill="none" stroke="#000000" d="M162.1197,-158.4356C120.3291,-152.7095 47.6218,-139.8 30.6258,-117.6751"/>
<polygon fill="#000000" stroke="#000000" points="33.7867,-116.1358 27,-108 27.2319,-118.5923 33.7867,-116.1358"/>
</g>
<!-- pkb -->
<g id="node5" class="node">
<title>pkb</title>
<polygon fill="none" stroke="#000000" points="126,-108 72,-108 72,-72 126,-72 126,-108"/>
<text text-anchor="middle" x="99" y="-86.3" font-family="Times,serif" font-size="14.00" fill="#000000">pkb</text>
</g>
<!-- varnish&#45;&gt;pkb -->
<g id="edge3" class="edge">
<title>varnish&#45;&gt;pkb:n</title>
<path fill="none" stroke="#000000" d="M162.3185,-157.4598C138.3972,-152.145 107.613,-140.9697 100.5083,-118.1231"/>
<polygon fill="#000000" stroke="#000000" points="103.9355,-117.375 99,-108 97.0119,-118.4066 103.9355,-117.375"/>
</g>
<!-- binary_trance -->
<g id="node6" class="node">
<title>binary_trance</title>
<polygon fill="none" stroke="#000000" points="239.5,-108 144.5,-108 144.5,-72 239.5,-72 239.5,-108"/>
<text text-anchor="middle" x="192" y="-86.3" font-family="Times,serif" font-size="14.00" fill="#000000">binary_trance</text>
</g>
<!-- varnish&#45;&gt;binary_trance -->
<g id="edge4" class="edge">
<title>varnish&#45;&gt;binary_trance:n</title>
<path fill="none" stroke="#000000" d="M192,-143.5056C192,-136.0111 192,-127.0501 192,-118.2162"/>
<polygon fill="#000000" stroke="#000000" points="195.5001,-118 192,-108 188.5001,-118 195.5001,-118"/>
</g>
<!-- wizards -->
<g id="node7" class="node">
<title>wizards</title>
<polygon fill="none" stroke="#000000" points="318,-108 258,-108 258,-72 318,-72 318,-108"/>
<text text-anchor="middle" x="288" y="-86.3" font-family="Times,serif" font-size="14.00" fill="#000000">wizards</text>
</g>
<!-- varnish&#45;&gt;wizards -->
<g id="edge5" class="edge">
<title>varnish&#45;&gt;wizards:n</title>
<path fill="none" stroke="#000000" d="M221.5865,-157.9823C246.4881,-152.9582 279.1827,-141.9194 286.5035,-118.153"/>
<polygon fill="#000000" stroke="#000000" points="290.0043,-118.4035 288,-108 283.0792,-117.3827 290.0043,-118.4035"/>
</g>
<!-- rust_melbourne -->
<g id="node8" class="node">
<title>rust_melbourne</title>
<polygon fill="none" stroke="#000000" points="443.5,-108 336.5,-108 336.5,-72 443.5,-72 443.5,-108"/>
<text text-anchor="middle" x="390" y="-86.3" font-family="Times,serif" font-size="14.00" fill="#000000">rust_melbourne</text>
</g>
<!-- varnish&#45;&gt;rust_melbourne -->
<g id="edge6" class="edge">
<title>varnish&#45;&gt;rust_melbourne:n</title>
<path fill="none" stroke="#000000" d="M221.6426,-160.0798C271.2681,-156.1302 367.6192,-145.0651 386.6786,-117.8585"/>
<polygon fill="#000000" stroke="#000000" points="390.124,-118.5941 390,-108 383.4904,-116.3591 390.124,-118.5941"/>
</g>
<!-- syncthing -->
<g id="node9" class="node">
<title>syncthing</title>
<polygon fill="none" stroke="#000000" points="135.5,-36 62.5,-36 62.5,0 135.5,0 135.5,-36"/>
<text text-anchor="middle" x="99" y="-14.3" font-family="Times,serif" font-size="14.00" fill="#000000">syncthing</text>
</g>
<!-- pkb&#45;&gt;syncthing -->
<g id="edge7" class="edge">
<title>pkb&#45;&gt;syncthing:n</title>
<path fill="none" stroke="#000000" d="M99,-71.5056C99,-64.0111 99,-55.0501 99,-46.2162"/>
<polygon fill="#000000" stroke="#000000" points="102.5001,-46 99,-36 95.5001,-46 102.5001,-46"/>
</g>
<!-- postgresql -->
<g id="node10" class="node">
<title>postgresql</title>
<polygon fill="none" stroke="#000000" points="325.5,-36 250.5,-36 250.5,0 325.5,0 325.5,-36"/>
<text text-anchor="middle" x="288" y="-14.3" font-family="Times,serif" font-size="14.00" fill="#000000">postgresql</text>
</g>
<!-- binary_trance&#45;&gt;postgresql -->
<g id="edge8" class="edge">
<title>binary_trance&#45;&gt;postgresql:nw</title>
<path fill="none" stroke="#000000" d="M212.6205,-71.9381C221.9822,-63.5759 233.3005,-53.2446 243.8515,-43.0286"/>
<polygon fill="#000000" stroke="#000000" points="246.3232,-45.5068 251,-36 241.4155,-40.5153 246.3232,-45.5068"/>
</g>
<!-- wizards&#45;&gt;postgresql -->
<g id="edge9" class="edge">
<title>wizards&#45;&gt;postgresql:n</title>
<path fill="none" stroke="#000000" d="M288,-71.5056C288,-64.0111 288,-55.0501 288,-46.2162"/>
<polygon fill="#000000" stroke="#000000" points="291.5001,-46 288,-36 284.5001,-46 291.5001,-46"/>
</g>
<!-- rust_melbourne&#45;&gt;postgresql -->
<g id="edge10" class="edge">
<title>rust_melbourne&#45;&gt;postgresql:ne</title>
<path fill="none" stroke="#000000" d="M366.2372,-71.9949C355.9011,-63.8213 343.6302,-53.6595 332.4922,-43.2365"/>
<polygon fill="#000000" stroke="#000000" points="334.6243,-40.4298 325,-36 329.7612,-45.4648 334.6243,-40.4298"/>
</g>
</g>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 6.8 KiB