forked from wezm/wezm.net
222 lines
9.4 KiB
Markdown
222 lines
9.4 KiB
Markdown
|
+++
|
||
|
|
title = "Debugging a Docker Core Dump"
|
||
|
|
date = 2023-02-25T10:39:49+10:00
|
||
|
|
|
||
|
|
#[extra]
|
||
|
|
#updated = 2023-01-11T21:11:28+10:00
|
||
|
|
+++
|
||
|
|
|
||
|
|
On my main machine I use an excellent cross-platform tool called [Docuum] that
|
||
|
|
automatically cleans up unused docker images. This allows me to use Docker
|
||
|
|
without the need to periodically wonder why I'm out of disk space, run `docker
|
||
|
|
system prune` and recover half my disk.
|
||
|
|
|
||
|
|
I installed Docuum via [the AUR package][aurpkg] (although tweaked to build the
|
||
|
|
latest Docuum release) and ran it via the bundled systemd service definition.
|
||
|
|
This worked great for a while but some time back it started failing. Every time
|
||
|
|
Docuum would try to check for things to clean up I'd see the following in the
|
||
|
|
system journal:
|
||
|
|
|
||
|
|
<!-- more -->
|
||
|
|
|
||
|
|
```
|
||
|
|
Feb 25 10:03:12 ryzen docuum[77751]: [2023-02-25 10:03:12 +10:00 INFO] Performing an initial vacuum on startup…
|
||
|
|
Feb 25 10:03:12 ryzen kernel: audit: type=1326 audit(1677283392.831:2): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=77763 comm="docker" exe="/usr/bin/docke>
|
||
|
|
Feb 25 10:03:12 ryzen systemd[1]: Created slice Slice /system/systemd-coredump.
|
||
|
|
Feb 25 10:03:12 ryzen systemd[1]: Started Process Core Dump (PID 77768/UID 0).
|
||
|
|
Feb 25 10:03:13 ryzen systemd-coredump[77769]: [🡕] Process 77763 (docker) of user 0 dumped core.
|
||
|
|
|
||
|
|
Stack trace of thread 77763:
|
||
|
|
#0 0x00005568dbcb5c4e n/a (docker + 0x243c4e)
|
||
|
|
#1 0x00005568dbd35a3b n/a (docker + 0x2c3a3b)
|
||
|
|
#2 0x00005568dbd3482f n/a (docker + 0x2c282f)
|
||
|
|
#3 0x00005568dbd6c2ee n/a (docker + 0x2fa2ee)
|
||
|
|
#4 0x00005568dbcfafa8 n/a (docker + 0x288fa8)
|
||
|
|
#5 0x00005568dbcfaef1 n/a (docker + 0x288ef1)
|
||
|
|
#6 0x00005568dbced953 n/a (docker + 0x27b953)
|
||
|
|
#7 0x00005568dbd1eb41 n/a (docker + 0x2acb41)
|
||
|
|
ELF object binary architecture: AMD x86-64
|
||
|
|
Feb 25 10:03:13 ryzen docuum[77751]: [2023-02-25 10:03:13 +10:00 ERROR] Unable to list images.
|
||
|
|
Feb 25 10:03:13 ryzen docuum[77751]: [2023-02-25 10:03:13 +10:00 INFO] Retrying in 5 seconds…
|
||
|
|
Feb 25 10:03:13 ryzen systemd[1]: systemd-coredump@0-77768-0.service: Deactivated successfully.
|
||
|
|
```
|
||
|
|
|
||
|
|
This would repeat every 5 seconds. I ignored this for a while but finally decided
|
||
|
|
to investigate it today. To find the failing command I ran
|
||
|
|
`coredumpctl list` then identified one of the docker crashes and ran
|
||
|
|
`coredumpctl info` with its PID:
|
||
|
|
|
||
|
|
```
|
||
|
|
$ coredumpctl info 78255
|
||
|
|
PID: 78255 (docker)
|
||
|
|
UID: 0 (root)
|
||
|
|
GID: 0 (root)
|
||
|
|
Signal: 31 (SYS)
|
||
|
|
Timestamp: Sat 2023-02-25 10:03:23 AEST (44min ago)
|
||
|
|
Command Line: docker image ls --all --no-trunc --format $'{{.ID}}\\t{{.Repository}}\\t{{.Tag}}\\t{{.CreatedAt}}'
|
||
|
|
Executable: /usr/bin/docker
|
||
|
|
Control Group: /system.slice/docuum.service
|
||
|
|
Unit: docuum.service
|
||
|
|
Slice: system.slice
|
||
|
|
Boot ID: 0ac9f0dd246548949c3a90a0e7494665
|
||
|
|
Machine ID: affcb0b7a7d1464385d65464d9be450e
|
||
|
|
Hostname: ryzen
|
||
|
|
Storage: /var/lib/systemd/coredump/core.docker.0.0ac9f0dd246548949c3a90a0e7494665.78255.1677283403000000.zst (inaccessible)
|
||
|
|
Message: Process 78255 (docker) of user 0 dumped core.
|
||
|
|
```
|
||
|
|
|
||
|
|
Strangely I could run the command myself (`Command Line` line) just fine. I
|
||
|
|
figured I needed to see where in docker it was crashing. I learned how to
|
||
|
|
access systemd coredumps with gdb and ran: `sudo coredumpctl gdb 78255`. `sudo`
|
||
|
|
is needed because the core dump belongs to root due the crashing `docker`
|
||
|
|
process belonging to root. This didn't yield much extra info as debug symbols
|
||
|
|
were not present for the binary. It did identify why it crashed though:
|
||
|
|
|
||
|
|
> Program terminated with signal SIGSYS, Bad system call.
|
||
|
|
|
||
|
|
Knowing that docker was implemented in Go and that they make system calls
|
||
|
|
manually on Linux I wondered if this was some sort of Go bug—although given
|
||
|
|
the popularity of Docker this did seem unlikely.
|
||
|
|
|
||
|
|
To get more info I needed debug symbols. Arch Linux makes these available via
|
||
|
|
[debuginfod](https://wiki.archlinux.org/title/Debuginfod) and gdb can
|
||
|
|
automatically download debuginfo files if `DEBUGINFOD_URLS` is set. I reran
|
||
|
|
`gdb`, telling `sudo` to pass the `DEBUGINFOD_URLS` environment variable
|
||
|
|
through (I had already set `DEBUGINFOD_URLS=https://debuginfod.archlinux.org/`
|
||
|
|
in my Zsh config some time ago):
|
||
|
|
|
||
|
|
```
|
||
|
|
sudo --preserve-env=DEBUGINFOD_URLS coredumpctl gdb 78255
|
||
|
|
```
|
||
|
|
|
||
|
|
Now there was a proper backtrace:
|
||
|
|
|
||
|
|
```
|
||
|
|
#0 runtime/internal/syscall.Syscall6 () at runtime/internal/syscall/asm_linux_amd64.s:36
|
||
|
|
#1 0x00005588e7b93c33 in syscall.RawSyscall6 (num=160, a1=7, a2=94046491589710, a3=7, a4=824634289408, a5=0, a6=0, r1=<optimized out>, r2=<optimized out>,
|
||
|
|
errno=<optimized out>) at runtime/internal/syscall/syscall_linux.go:38
|
||
|
|
#2 0x00005588e7c13a3b in syscall.RawSyscall (trap=160, a1=7, a2=94046491589710, a3=7, r1=<optimized out>, r2=<optimized out>, err=<optimized out>)
|
||
|
|
at syscall/syscall_linux.go:62
|
||
|
|
#3 0x00005588e7c1282f in syscall.Setrlimit (resource=<optimized out>, rlim=<optimized out>, err=...) at syscall/zsyscall_linux_amd64.go:1326
|
||
|
|
#4 0x00005588e7c4a2ee in os.init.1 () at os/rlimit.go:30
|
||
|
|
#5 0x00005588e7bd8fa8 in runtime.doInit (t=0x5588e91a7be0 <os.[inittask]>) at runtime/proc.go:6506
|
||
|
|
#6 0x00005588e7bd8ef1 in runtime.doInit (t=0x5588e91a9900 <main.[inittask]>) at runtime/proc.go:6483
|
||
|
|
#7 0x00005588e7bcb953 in runtime.main () at runtime/proc.go:233
|
||
|
|
#8 0x00005588e7bfcb41 in runtime.goexit () at runtime/asm_amd64.s:1598
|
||
|
|
#9 0x0000000000000000 in ?? ()
|
||
|
|
```
|
||
|
|
|
||
|
|
So the issue seems to be a call to `setrlimit`. Looking at [the code][gocode]
|
||
|
|
and searching the Go issue tracker didn't turn up anyone else having this
|
||
|
|
issue, which pointed at an issue on my system.
|
||
|
|
|
||
|
|
I'm honestly not sure what led me to the next step but I decided to take a look
|
||
|
|
at the Docuum service definition. I was surprised to see that it was more
|
||
|
|
complicated than most definitions I'm used to seeing:
|
||
|
|
|
||
|
|
```ini
|
||
|
|
[Unit]
|
||
|
|
Description=LRU eviction of Docker images
|
||
|
|
Documentation=https://github.com/stepchowfun/docuum
|
||
|
|
DefaultDependencies=false
|
||
|
|
After=docker.service docker.socket
|
||
|
|
Requires=docker.service docker.socket
|
||
|
|
|
||
|
|
[Service]
|
||
|
|
Type=simple
|
||
|
|
Environment=DOCUUM_THRESHOLD=10GB
|
||
|
|
EnvironmentFile=-/etc/default/docuum
|
||
|
|
ExecStart=/usr/bin/docuum --threshold $DOCUUM_THRESHOLD
|
||
|
|
ProtectSystem=full
|
||
|
|
PrivateTmp=true
|
||
|
|
PrivateDevices=true
|
||
|
|
PrivateNetwork=true
|
||
|
|
CapabilityBoundingSet=
|
||
|
|
KeyringMode=private
|
||
|
|
RestrictNamespaces=~cgroup ipc net mnt pid user uts
|
||
|
|
RestrictAddressFamilies=AF_UNIX
|
||
|
|
ReadWritePaths=/var/run/docker.sock
|
||
|
|
DeviceAllow=
|
||
|
|
IPAddressDeny=any
|
||
|
|
NoNewPrivileges=true
|
||
|
|
PrivateTmp=true
|
||
|
|
PrivateDevices=true
|
||
|
|
PrivateMounts=true
|
||
|
|
PrivateUsers=true
|
||
|
|
ProtectControlGroups=true
|
||
|
|
ProtectSystem=strict
|
||
|
|
ProtectHome=tmpfs
|
||
|
|
ProtectKernelModules=true
|
||
|
|
ProtectKernelTunables=true
|
||
|
|
RestrictSUIDSGID=true
|
||
|
|
SystemCallArchitectures=native
|
||
|
|
SystemCallFilter=@system-service
|
||
|
|
SystemCallFilter=~@privileged @resources
|
||
|
|
RestrictRealtime=true
|
||
|
|
LockPersonality=true
|
||
|
|
MemoryDenyWriteExecute=true
|
||
|
|
UMask=0077
|
||
|
|
ProtectHostname=true
|
||
|
|
|
||
|
|
[Install]
|
||
|
|
WantedBy=multi-user.target
|
||
|
|
```
|
||
|
|
|
||
|
|
Suspiciously it seemed to be doing some sandboxing and filtering system calls
|
||
|
|
(`SystemCallFilter`). A bit more research pointed me to `systemd-analyze
|
||
|
|
syscall-filter`, which lists which system calls belong to the predefined system
|
||
|
|
call sets (`@privileged`, `@resources`, etc.).
|
||
|
|
|
||
|
|
`setrlimit` was listed under `@resources`:
|
||
|
|
|
||
|
|
```
|
||
|
|
@resources
|
||
|
|
# Alter resource settings
|
||
|
|
ioprio_set
|
||
|
|
mbind
|
||
|
|
migrate_pages
|
||
|
|
move_pages
|
||
|
|
nice
|
||
|
|
sched_setaffinity
|
||
|
|
sched_setattr
|
||
|
|
sched_setparam
|
||
|
|
sched_setscheduler
|
||
|
|
set_mempolicy
|
||
|
|
setpriority
|
||
|
|
setrlimit
|
||
|
|
```
|
||
|
|
|
||
|
|
The [systemd docs for `SystemCallFilter`][SystemCallFilter] also mentioned:
|
||
|
|
|
||
|
|
> If the first character of the list is "~", the effect is inverted: only the
|
||
|
|
> listed system calls will result in immediate process termination
|
||
|
|
> (deny-listing)
|
||
|
|
|
||
|
|
So we finally had our culprit: the service definition was denying system calls in
|
||
|
|
the `@resources` set and at some point `docker` had started making `setrlimit` calls,
|
||
|
|
which were resulting in termination.
|
||
|
|
|
||
|
|
The fix was simple enough: I removed `@resources` from the deny list, rebuilt
|
||
|
|
the package, and then re-enabled the `docuum` service (I'd previously disabled
|
||
|
|
it do to the constant crashes). I was pleased to see it start successfully and
|
||
|
|
begin vacuuming up a few months of Docker image detritus.
|
||
|
|
|
||
|
|
### Conclusion
|
||
|
|
|
||
|
|
This small debugging session taught me a number of things: I learned how to
|
||
|
|
find and list core dumps managed by systemd, how to open them in GDB with
|
||
|
|
symbols present, and that systemd has powerful, fine-grained system call
|
||
|
|
sandboxing.
|
||
|
|
|
||
|
|
I was ultimately able to resolve the issue and get Docuum working again.
|
||
|
|
I have published my patched version of the AUR package to my personal AUR
|
||
|
|
repo in case it's useful to anyone else:
|
||
|
|
|
||
|
|
<https://github.com/wezm/aur/tree/master/docuum>.
|
||
|
|
|
||
|
|
[aurpkg]: https://aur.archlinux.org/packages/docuum
|
||
|
|
[Docuum]: https://github.com/stepchowfun/docuum
|
||
|
|
[gocode]: https://github.com/golang/go/blob/203e59ad41bd288e1d92b6f617c2f55e70d3c8e3/src/syscall/zsyscall_linux_amd64.go#L1335
|
||
|
|
[service]: https://aur.archlinux.org/cgit/aur.git/tree/docuum.service?h=docuum&id=650d2c24fe9df712e8a98dde37f3ee47d3af4e47
|
||
|
|
[SystemCallFilter]: https://www.freedesktop.org/software/systemd/man/systemd.exec.html#System%20Call%20Filtering
|