A 40ms Go GC pause attributable to swap · Fernando Simões

I’m scripting this so that you don’t slap your brow like I virtually did after I determined to run swap in manufacturing to soak up reminiscence spikes.

I had a cgroup with two processes: one is a Go course of that calls io.ReadAll after which proto.Unmarshal, making a blob after which a graph struct (which is marked as scan by Go’s allocator). The different course of is an HTTP server that principally stays quiet.

Whenever the collector runs, it reads these scan spans, pointer by pointer, and decides what to do with them. So I believed: okay, beneath reminiscence strain, the kernel goes to evict pages to the swap gadget, however because the eviction is per cgroup, and never per course of, each processes’ pages are going to be evicted – so there may be solely a small probability that this may flip into a tragic dance of swap-in and swap-out between the kernel and the rubbish collector.

I used to be fallacious. While experimenting with this, I discovered an issue that might’ve harm me: Go’s rubbish collector reads its metadata (outdoors the heap, in a area that isn’t freed) in a stop-the-world pause, and that metadata may be in swap.

I did a mock run on a Hetzner field utilizing kernel 6.8 with MGLRU enabled^{1 You can discover every part about these experiments: plots, the mock allocator, bpf scripts, python scripts, and so forth., right here: https://github.com/frnsimoes/go-gc-swap-cost. The median pause was round 51 us. With the metadata on the NVMe, the worst pause was 40ms.}

To test the place these 40ms went, I wrote a small bpf script that counts web page faults whereas the international stage is stopped. This was the worst one: 39902 us, faults throughout it 228, 39013 us in faults. 39 of these 40ms had been spent in 228 web page faults. Those faults occurred contained in the GC’s bookkeeping:

// addr2line output
0x42e5c8  runtime.(*spanSet).reset         /usr/native/go/src/inside/runtime/atomic/varieties.go:194
0x4219de  runtime.finishsweep_m            /usr/native/go/src/runtime/mcentral.go:71
0x4629cf  runtime.gcStart.func2            /usr/native/go/src/runtime/mgc.go:724
0x46dd8a  runtime.systemstack              /usr/native/go/src/runtime/asm_amd64.s:518
0x4169dc  runtime.gcStart                  /usr/native/go/src/runtime/mgc.go:722

0x4276a4  runtime.nextMarkBitArenaEpoch    /usr/native/go/src/runtime/mheap.go:2481
0x421a65  runtime.finishsweep_m            /usr/native/go/src/runtime/mgcsweep.go:268
0x4629cf  runtime.gcStart.func2            /usr/native/go/src/runtime/mgc.go:724
0x46dd8a  runtime.systemstack              /usr/native/go/src/runtime/asm_amd64.s:518
0x4169dc  runtime.gcStart                  /usr/native/go/src/runtime/mgc.go:722

That’s a possible failure mode. Go’s GC has to cease the international stage at two factors: when it performs a sweep termination, and when it performs a mark termination. We had 312 of these pauses in half-hour.

So right here is why this occurs: the runtime allocates these pages. They are usually not freed, however reused. Those pages are learn in GC cycles. Because the kernel evicts pages by age, it sends the least just lately accessed pages to swap. The GC runs, stops the international stage, tries to learn these pages, however now we now have a significant web page fault. The kernel must read PTEs, after which call do_swap_page, discover a new frame, charge it to the cgroup, read the pages, submit a bio, wait for the disk, and put them back in memory – simply to maintain it brief.

Those 40ms appear innocent at first. But we’re speaking a few stop-the-world pause. Those 40ms imply every part has stopped – in Go’s terminology, every P has stopped, so, for instance, if a goroutine was ready for I/O, throughout that pause the I/O would possibly return and there could be nobody to deal with it. 40ms is 800 instances the median pause. It occurs two or 3 times per reminiscence spike throughout the check. It is rather a lot.

And then I observed one other factor: constructing one 511 KiB message, which often takes 3-5 ms, jumped to 105 ms on the NVMe and 903 ms on Hetzner’s community quantity. Per message, this prices greater than the metadata pause. But solely the goroutine doing the allocation pays that worth, so a minimum of it’s localized, and never international just like the metadata one.

I haven’t confirmed the place that point goes, besides I needed to say it right here, as a result of that’s one other price you would need to pay. In any case, I agree with Chris Down, swap isn’t evil. But, yeah, it didn’t behave properly with rubbish assortment, and, in manufacturing, I’m accumulating rather a lot.

Update, September 14.

Someone requested me if Go 1.26’s Green Tea garbage collector modified the best way the GC reads metadata. I measured it, and the influence is negligible.