Goroutine Leak Profiles – The Go Programming Language
Go’s concurrency features are highly effective and simple to make use of, however
that very same ease can generally lead even seasoned builders to make
errors.
Fortunately, the Go ecosystem comes outfitted with helpful instruments for
debugging, e.g., the race detector,
however even present instruments might miss some concurrency bugs,
similar to the subject of this text, the goroutine leak.
Goroutines synchronize or alternate data
by way of shared concurrency primitives, e.g., channels, locks, and wait teams.
While speaking, goroutines usually block on these primitives,
as in, wait till some situation is met;
ubiquitous examples embody ready to accumulate a held mutex,
or obtain a message over a channel.
Goroutines also can block on working system operations, like studying from a community socket or a file.
We might think about a goroutine leaked whether it is blocked,
however the situations wanted to unblock it could actually by no means be met.
Over time, an accumulation of leaked goroutines degrades
efficiency by extreme reminiscence utilization (by the leaked
goroutines themselves or the reminiscence they reference), in addition to
CPU utilization from the rubbish collector, particularly
if GOMEMLIMIT is in use.
Goroutine leaks will be notoriously troublesome to detect.
In unit testing, probably the most important breakthroughs embody
the open-source library goleak,
which might instrument particular person assessments to sign any
un-terminated goroutines after the check wraps up as suspicious.
Similarly, Go 1.25 launched the synctest package to
the usual library; it could actually considerably enhance
the standard of unit assessments in concurrent code by giving
Go builders extra management over the ordering of concurrent occasions
so as to reliably check hard-to-reproduce eventualities.
Unfortunately, neither method can examine for goroutine leaks
in manufacturing programs, particularly at bigger scales,
which can behave in methods unaccounted for by assessments.
Goroutine profiles are a rudimentary method to examine for operations
that block too many goroutines, or analyze development developments.
However, goroutine profiles can’t distinguish between
goroutines that are leaked, and people that are briefly blocked
in excessive numbers by design, e.g., as brought on by elevated
site visitors in a microservice.
Likewise, leaks that are low in quantity might slip by undetected for a few years.
Go 1.27 introduces the goroutine leak profiler,
a versatile and light-weight mechanism for locating
goroutine leaks in operating Go applications, together with manufacturing programs.
Unlike earlier approaches, which require human evaluation,
this mechanism is exact and generates little-to-no false positives.
The trade-off is that it’s restricted to a subset of goroutine leaks:
goroutines completely blocked on channels or primitives
within the sync package.
Luckily for us, this already covers a really massive subset of goroutine leaks,
as we’ll see in our examples.
In the next sections, we showcase the way to use the characteristic, adopted by
some further examples of detectable leaks, and an outline of the
underlying implementation and trade-offs.
Example: concurrent staff
Consider a perform that processes work objects concurrently:
kind end result struct {
res workResult
err error
}
func processWorkItems(ws []workItem) ([]workResult, error) {
// Process work objects in parallel, aggregating leads to ch.
ch := make(chan end result)
for _, w := vary ws {
go func() {
res, err := processWorkItem(w)
ch <- end result{res, err}
}()
}
// Collect the outcomes from ch, or return an error if one is discovered.
var outcomes []workResult
for vary len(ws) {
r := <-ch
if r.err != nil {
// This early return might trigger goroutine leaks.
return nil, r.err
}
outcomes = append(outcomes, r.res)
}
return outcomes, nil
}
Because ch is an unbuffered channel, every employee goroutine blocks when sending
its end result till the primary goroutine receives from the channel.
If processWorkItems returns early attributable to an error, the receiving loop terminates,
and all remaining sender goroutines block without end.
This instance is emblematic of a typical mistake found in actual Go applications,
together with Uber manufacturing providers.
Let’s see how we are able to discover these leaks through the use of the
new goroutine leak profiler.
Debugging with the goroutine leak profiler
The profile is obtainable by the
runtime/pprof package, because the
goroutineleak profile kind, or by putting in the profile handlers outlined
by the net/http/pprof package.
If you have already got internet/http/pprof arrange in your service,
you then don’t must do anything! The profile shall be
mechanically made accessible for assortment on the /debug/pprof/goroutineleak
endpoint on no matter host and port the handlers are put in.
Let’s put our concurrency bug in context and arrange the internet/http/pprof bundle.
This approach, you may strive it your self!
bundle essential
import (
"errors"
"log"
"internet/http"
_ "internet/http/pprof"
"time"
)
kind workItem int
kind workResult int
func processWorkItem(w workItem) (workResult, error) {
time.Sleep(10 * time.Millisecond)
if w == 5 {
return 0, errors.New("simulated error")
}
return workResult(w * 2), nil
}
kind end result struct {
res workResult
err error
}
func processWorkItems(ws []workItem) ([]workResult, error) {
ch := make(chan end result)
for _, w := vary ws {
go func() {
res, err := processWorkItem(w)
ch <- end result{res, err}
}()
}
var outcomes []workResult
for vary len(ws) {
r := <-ch
if r.err != nil {
return nil, r.err
}
outcomes = append(outcomes, r.res)
}
return outcomes, nil
}
func essential() {
// Start pprof server
go func() {
log.Println(http.Pay attentionAndServe("localhost:6060", nil))
}()
// Repeatedly set off the leak
for {
objects := []workItem{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
_, err := processWorkItems(objects)
if err != nil {
log.Printf("Error processing objects: %v", err)
}
time.Sleep(time.Second)
}
}
Build this system above, then run it:
$ go construct -o leaky
$ ./leaky
Collecting the profile
It gained’t take lengthy for this system to start out accumulating
leaks, which you’ll be able to then view through the use of the online UI
at http://localhost:6060/debug/pprof.
Alternatively, you may gather the goroutine
leak profile utilizing curl, after which look at it with go device pprof:
$ curl http://localhost:6060/debug/pprof/goroutineleak > leak.prof
$ go device pprof leak.prof
Type: goroutineleak
Time: 2026-03-01 13:19:49 UTC
Entering interactive mode (kind "assist" for instructions, "o" for choices)
(pprof) checklist processWorkItems
Total: 116
ROUTINE ======================== essential.processWorkItems.func1 in .../essential.go
0 116 (flat, cum) 100% of Total
. . 31: go func() {
. . 32: res, err := processWorkItem(w)
. 116 33: ch <- end result{res, err}
. . 34: }()
The profile reveals the goroutines leaked at
ch <- end result{res, err} (line 33), pinpointing the wrongdoer operation.
Notably, the longer this system is operating, the bigger the variety of leaked
goroutines.
Addressing the leak
This leak will be merely fastened by giving ch a buffer:
ch := make(chan end result, len(ws))
This permits all of the work merchandise goroutines to ship a message with out blocking
within the occasion of a untimely return of processWorkItems.
We checklist extra real-world examples in this section.
Implementation
This part is for these how leak detection
works below the hood of the goroutine leak profiler.
For particulars strictly pertaining to efficiency overhead and limitations,
skip forward to this section.
Core idea
Let’s begin with an preliminary commentary: if a goroutine
is blocked over some concurrency primitive that no different goroutine has entry to
(on this case, by way of a reference in reminiscence), then it’s clearly leaked.
This is already a powerful lead, we are able to generalize it additional right into a definition
for when a goroutine is not leaked, a property we time period as liveness.
We formally outline liveness, an inductive property
as follows:
A goroutine is reside if:
- it isn’t blocked by a concurrency primitive, or
- not less than one concurrency primitive that blocks it’s referenced
by one other reside goroutine.
In the trivial case, goroutines which aren’t blocked are clearly
not leaked.
In the inductive case, the underlying assumption is that
any goroutine which isn’t leaked might finally use
concurrency primitives it references to unblock any
different goroutines blocked by these primitives.
To discover all reside goroutines, we begin from the clearly reside
unblocked goroutines and hint any references
they maintain, i.e., by their native variables, to seek out
the concurrency primitives they’ve entry to.
We then incrementally embody any goroutines blocked over these
primitives as reside, and repeat the method till no
further reside goroutines are found.
Fortunately for us, the Go runtime already computes reminiscence reachability
by the garbage collector (GC),
so the subsequent step is to adapt the GC to swimsuit our functions.
You can rapidly evaluate the 2 GCs with the next diagrams:
A whole overhaul of the GC will not be mandatory.
The Go runtime makes use of a concurrent tri-color mark-and-sweep rubbish collector,
(now with the Green Tea variant!),
so its MO already neatly aligns with our objectives.
Only just a few key adjustments are wanted:
- In the preliminary phases, the common GC marks all goroutines (and world variables)
as reachable, such that they might by no means be thought-about rubbish,
i.e., they’re mark roots.
We change it to as a substitute solely embody unblocked goroutines,
since these are assured to be reside. - This is adopted by the marking section, the place the GC traces objects referenced
(transitively) by the mark roots, and marks them as usable reminiscence.
Even although we don’t modify this section instantly, the adjustments in step 1.
implicitly be sure that the GC solely marks reminiscence referenced by reside goroutines. - The marking section is finalized by inspecting all of the blocked
goroutines not included as mark roots in step 1.
If a goroutine is blocked by not less than one concurrency
primitive that has been marked in step 2., it’s added as a mark root,
and the GC resumes the marking section from step 2.
This coincides with the inductive step within the definition
of liveness. - Once all reside goroutines have been found, any goroutine
which has not been added as a mark root has its standing set to leaked. - The marking section then resumes one final time with all of the leaked goroutines
added as mark roots, permitting the GC to mark all of the reminiscence it will have
marked throughout a daily run.
Once the GC cycle is full, the goroutine leak profiler picks up
like in a daily goroutine profile, and filters for strictly
leaked goroutines.
Limitations
The examples above reveal the usefulness of goroutine leak profiles.
Nevertheless, the rubbish collector has some limitations which will lead
it to overlook leaks:
-
Memory overreach: if a concurrency primitive is
constantly reachable by world variables or runnable goroutines,
then goroutines blocking on it are by no means reported as leaked, even when
that concurrency primitive isn’t used sooner or later.This will be alleviated by higher regimenting entry to
concurrency primitive references, and extra clearly
delineating their lifecycle. -
Non-standard blocking:
For the sake of correctness, goroutine leak detection is strictly restricted
to Go first-class concurrency primitives, which incorporates:
channel ship and obtain operations (together with overnilchannels),
blockingchoosestatements, i.e., with nodefaultcase, as much as, and together with
choosestatements with no instances, and members of the
syncbundle, particularlyMutex,
RWMutex,WaitGroupandCond.Goroutines blocked for another cause, e.g.,
file and community IO, or direct system calls
are by no means thought-about as leaked.
This likewise applies for customized, user-defined concurrency,
e.g., spin locks, until they depend on the primitives outlined above
for his or her underlying implementation. -
Non-determinism: leaks will be detected solely after
they’ve occurred, however can’t be in any other case predicted,
so reproducing and diagnosing leaks in flaky applications
continues to be a problem.
For the most effective outcomes, we encourage mixing approaches, through the use of
goroutine leak profiles at numerous layers, as much as, and together with manufacturing,
in addition to complete check suites instrumented withgoleakandsynctest.
Performance influence
Goroutine leak detection is rigorously designed to attenuate
efficiency influence, however there are, nonetheless, some prices.
While reminiscence overhead is negligible, solely restricted to small additions
required for bookkeeping, goroutine leak detection will be slower
than the common GC.
This is finest illustrated by a pathological case we
name the “daisy-chain”:
In this leak-free instance, runnable goroutine G₀ has a
reference to primitive P₁ which blocks G₁, and so forth.
This implies that proving liveness for some Pᵢ₊₁,
requires proving liveness for Pᵢ, which introduces
two prices:
- The GC marking section is successfully serialized relative to the
order by which goroutines will be scanned, as all of the reminiscence reachable
from some Pᵢ should be marked earlier than Pᵢ₊₁ will be added as a root. - The inspection at the moment checks all blocked goroutines
on the finish of every marking spherical, for a worst-case of O(n²) steps for one
GC cycle, the place n is the full variety of goroutines.
While the second level can finally be optimized for,
the primary level is an intrinsic limitation of leak detection
that can not be circumvented.
Regardless, we remind the reader that, until configured in any other case
by way of runtime flags, the GC nonetheless operates concurrently with consumer code.
Furthermore, if a goroutine leak will be noticed sooner or later in time, then it
may also be noticed at any future level throughout the identical execution.
Periodic profiling infrastructures can due to this fact tune profiling frequency,
e.g., each 4 hours, to attenuate overhead at just about no value in
leak detection capabilities.
Acknowledgements
Goroutine leak detection is the results of a analysis collaboration between
Aarhus University, Washington University in St. Louis, and Uber, as introduced in
“Dynamic Partial Deadlock Detection and Recovery via Garbage Collection”
(Saioc et al., ASPLOS 2025).
The transition from educational prototype to precise Go characteristic was made doable
with the steerage of Michael Knyszek and Michael Pratt on the Go crew at Google, and
PJ Malloy (@thepudds).
Additional examples
The following are coding patterns that result in leaks, as
noticed in industrial-scale codebases and open supply initiatives,
in ascending order of complexity.
You can rapidly check drive the goroutine leak detector on them in
the Go playground, in addition to
experiment with your individual leaks.
Example: Double ship
Some of the only leaks happen when extra messages
are despatched over a channel than anticipated.
Below, a goroutine is predicted to ship one message to the primary goroutine
over an unbuffered channel.
However, the return assertion is lacking after the ship operation
within the error case.
For each error, the sender will, due to this fact, try to ship two messages,
which causes a leak.
func DoubleSend() {
ch := make(chan any)
go func(err error) {
if err != nil {
// In case of an error, ship nil.
ch <- nil
// Return assertion is lacking.
}
// Otherwise, proceed with regular behaviour.
// This ship continues to be executed, which causes a leak within the error case.
ch <- struct{}{}
}(fmt.Errorf("error"))
// Receive just one message.
<-ch
}
While the profile doesn’t explicitly spotlight the lacking return
because the trigger, it not less than directs you to the defective perform, by
highlighting the leaking ship operation.
(pprof) checklist DoubleSend
Total: 1
ROUTINE ======================== essential.DoubleSend.func1 in .../essential.go
0 1 (flat, cum) 100% of Total
. . 118: go func(err error) {
. . 119: if err != nil {
. . 121: ch <- nil
. . 123: }
. 1 126: ch <- struct{}{}
. . 127: }(fmt.Errorf("error"))
. . 129: <-ch
This leak will be addressed just by including a return assertion after the
ship operation within the error case.
Example: Early return
The inverse scenario is simply as frequent, the place the receiver
omits communication on some management movement paths,
in what’s successfully a simplified model of the introductory instance.
// Incoming error simulates an error produced internally.
func EarlyReturn(err error) {
ch := make(chan any)
// Create a employee goroutine.
go func() {
// Send one thing to the channel.
// Leaks if the dad or mum goroutine terminates early.
ch <- struct{}{}
}()
if err != nil {
// The dad or mum goroutine quits too early in case of an error.
// Sender leaks.
return
}
// Receive is barely executed if there isn't any error.
<-ch
}
The goroutine leak is uncovered by the profile:
ROUTINE ======================== essential.EarlyReturn.func1 in .../essential.go
0 1 (flat, cum) 100% of Total
. . 140: go func() {
. 1 143: ch <- struct{}{}
. . 144: }()
. . 145:
. . 146: if err != nil {
The leak will be addressed by giving ch a buffer of measurement 1.
Example: Timeout
A variation of the Early return sample above entails contexts
and non-deterministic selection (choose statements):
func Timeout(ctx context.Context) {
// An unbuffered channel is used to coordinate
// a employee and dad or mum thread
ch := make(chan any)
// Create employee goroutine
go func() {
// Perform some work then sign to the dad or mum thread.
ch <- struct{}{}
}()
// Wait for message from employee or context
// to be cancelled or timed out.
choose {
case <-ch: // Receive message from employee
case <-ctx.Done():
// Sender leaks as a result of there isn't any
// future rendezvous over the channel.
}
}
If the context is cancelled earlier than the sender synchronizes with the dad or mum,
the sender will leak:
(pprof) checklist Timeout
Total: 10
ROUTINE ======================== essential.Timeout.func1.1 in .../essential.go
0 10 (flat, cum) 100% of Total
. . 198: go func() {
. 10 201: ch <- struct{}{}
. . 202: }()
As within the earlier instance, the repair is to offer the channel
buffer of measurement 1.
Example: Range over channel with out closing
Iterating over channels through the use of vary
permits you to repeatedly obtain values from a channel in a loop.
Once the channel is closed and all values which were enqueued
within the channel’s buffer have been acquired, the loop exits.
Importantly, if the channel isn’t closed, a vary loop will block
the executing goroutine without end.
Omitting the shut operation is a typical mistake, as beneath:
// Incoming checklist of things and the variety of staff.
func noCloseRange(checklist []any, staff int) {
// Create a channel that distributes work objects.
ch := make(chan any)
// Create the employee goroutines.
for i := 0; i < staff; i++ {
go func() {
// Each employee pulls objects from the channel
// after which processes it.
for merchandise := vary ch {
// Process every merchandise
_ = merchandise
}
}()
}
// Queue objects to the employees through the use of the channel.
for _, merchandise := vary checklist {
// The dad or mum leaks by sending an merchandise if staff == 0
// or if all the employees panic, however the panic is recovered.
ch <- merchandise
}
// Otherwise, the channel isn't closed, so staff
// leak as soon as there are not any extra objects left to course of.
}
...
go noCloseRange([]any{1, 2, 3}, 3) // Leaks all 3 staff
A goroutine leak profile for such a program would come with the next:
Type: goroutineleak
(pprof) checklist noCloseRange.func1
Total: 4
ROUTINE ======================== essential.noCloseRange.func1 in .../essential.go
0 3 (flat, cum) 75.00% of Total
. . 82: go func() {
. 3 84: for merchandise := vary ch {
. . 86: _ = merchandise
. . 87: }
. . 88: }()
We see the three staff blocked on the vary ch operation, which
offers an ample trace as to the reason for the leak. The leak will be
addressed by merely closing the channel as soon as all messages have been despatched:
for _, merchandise := vary checklist {
ch <- merchandise
}
// All objects have been despatched. It is now secure to shut.
shut(ch)
Bonus! Eagle-eyed readers might have noticed one other potential
leak on this instance, if the variety of staff is mistakenly set to zero,
which can lead the dad or mum sender to leak:
go noCloseRange([]any{1, 2, 3}, 0) // Sender leaks with 0 staff
This can be captured by the profile:
(pprof) checklist noCloseRange$
Total: 4
ROUTINE ======================== essential.noCloseRange in .../essential.go
0 1 (flat, cum) 25.00% of Total
. . 76:func noCloseRange(checklist []any, staff int) {
...
. . 92: for _, merchandise := vary checklist {
. 1 95: ch <- merchandise
. . 96: }
While staff > 0 will be assumed to carry in practical manufacturing programs,
goroutine leak profiles can nonetheless be used to implicitly monitor for off-chance
violations with out conservative staff <= 0 checks.
Example: Method contract violations
The patterns seen thus far have been comparatively constrained of their lexical scope.
However, as performance is unfold out throughout features, strategies and packages, and
implementations are obfuscated by interfaces, the problem of manually detecting
leaks drastically will increase.
Such a case is exemplified on this part, with the customized employee kind that embeds two channel
fields, ch and executed and creates a looping goroutine with its Start methodology that
reads from each channels with a choose assertion.
Said goroutine can solely be terminated by receiving a message by the executed channel,
which is closed by the Stop methodology.
The Start methodology will be invoked any variety of occasions, however whether it is invoked
not less than as soon as, Stop ought to finally be referred to as.
As a end result, Start and Stop type an implicit contract that dictates the order
by which the strategies needs to be invoked.
Breaking that contract can result in undesirable habits,
on this case, goroutine leaks:
func MethodContractViolation() {
objects := make([]any, 10)
// Create a brand new employee
w := NewWorker()
// Start employee
w.Start()
// Operate on employee
for _, merchandise := vary objects {
w.AddToQueue(merchandise)
}
// Exits with out calling ’Stop’.
}
kind employee struct {
ch chan any
executed chan any
}
kind Worker interface {
Start()
Stop()
AddToQueue(merchandise any)
}
func NewWorker() Worker {
return &employee{
ch: make(chan any),
executed: make(chan any),
}
}
// Start spawns a background goroutine that extracts objects pushed to the queue.
func (w *employee) Start() {
go func() {
for {
choose {
case <-w.ch: // Normal workflow
case <-w.executed:
return // Shut down
}
}
}()
}
func (w *employee) Stop() {
// Allows goroutine created by Start to terminate
shut(w.executed)
}
func (w *employee) AddToQueue(merchandise any) {
w.ch <- merchandise
}
This subject is additional exacerbated in observe, the place such customized sorts are solely
exported as interfaces, on this case, by the non-descript
Worker kind.
Clients might not even concentrate on the underlying implementation and,
consequently, violate the implicit contract with out realizing.
Fortunately, soliciting a goroutine leak profile can reveal the defect:
(pprof) checklist Start
Total: 1
ROUTINE ======================== essential.(*employee).Start.func1 in .../essential.go
0 1 (flat, cum) 100% of Total
. . 266: go func() {
. . 267: for {
. 1 268: choose {
. . 269: case <-w.ch:
. . 270: case <-w.executed:
. . 271: return
Naturally, the repair entails following the path to the Start name
and including an invocation of Stop.
Example (Cockroach): Missing unlock
The following example
is taken from CockroachDB.
It entails buying and releasing a lock in a loop,
however forgetting to unlock it
earlier than executing a break assertion:
kind Gossip struct {
mu sync.Mutex
closed bool
}
func (g *Gossip) bootstrap() {
for {
g.mu.Lock()
if g.closed {
// Missing g.mu.Unlock
break
}
g.mu.Unlock()
}
}
func Cockroach584() {
g := &Gossip{
closed: true,
}
// ...
g.bootstrap()
g.bootstrap() // Causes a leak
}
In such a case, the goroutine will leak when failing to accumulate the lock.
(pprof) checklist Gossip
Total: 1
ROUTINE ======================== essential.(*Gossip).bootstrap in .../essential.go
0 1 (flat, cum) 100% of Total
. . 165:func (g *Gossip) bootstrap() {
. . 166: for {
. 1 167: g.mu.Lock()
. . 168: if g.closed {
. . 170: break
. . 171: }
. . 172: g.mu.Unlock()
Adding a name to Unlock earlier than the break addresses the difficulty.
Example (etcd): Unexpected channel operation orderings
This example,
present in etcd,
reveals how an surprising ordering between channel
operations can result in a goroutine leak:
kind node struct {
standing chan chan struct{}
cease chan struct{}
executed chan struct{}
}
func (n *node) Status() struct{} {
c := make(chan struct{})
n.standing <- c
return <-c
}
func (n *node) run() {
for {
choose {
case c := <-n.standing:
c <- struct{}{}
case <-n.cease:
shut(n.executed)
return
}
}
}
func (n *node) Stop() {
choose {
case n.cease <- struct{}{}:
case <-n.executed:
return
}
<-n.executed
}
func Etcd6857() {
n := &node{
standing: make(chan chan struct{}),
cease: make(chan struct{}),
executed: make(chan struct{}),
}
go n.run()
go n.Status()
go n.Stop()
}
The run methodology fires a loop which expects to
repeatedly obtain messages over the standing channel
(despatched by invoking the Status methodology).
At the identical time, it could actually additionally obtain one message over the
cease channel (despatched by way of the Stop methodology),
at which level it closes the executed channel and exits.
The Stop methodology itself then waits to obtain message
over executed, which is unblocked as soon as executed is closed.
A leak might happen if the run, Status, and Stop strategies
run concurrently.
The Stop and run goroutines can synchronize
and exit with out receiving the message issued
by Status, inflicting it to dam without end.
(pprof) checklist Status
Total: 8
ROUTINE ======================== essential.(*node).Status in .../essential.go
0 8 (flat, cum) 100% of Total
. . 16:func (n *node) Status() struct{} {
. . 17: c := make(chan struct{})
. 8 18: n.standing <- c
. . 19: return <-c
. . 20:}
Wrapping the ship to standing in a choose assertion
the place the opposite case department tries to obtain a message
over executed permits the goroutine operating to Status
to gracefully exit if it misplaced the race with a Stop
name.
Example (Kubernetes): Mutual blocking between channels and mutexes
This example
happens in Kubernetes,
because of mixing channels and locks:
kind Connection struct {
closeChan chan bool
}
kind idleAwareFramer struct {
resetChan chan bool
writeLock sync.Mutex
conn *Connection
}
func (i *idleAwareFramer) monitor() {
var resetChan = i.resetChan
for vary i.conn.closeChan {
i.writeLock.Lock()
shut(resetChan)
i.resetChan = nil
i.writeLock.Unlock()
break
}
}
func (i *idleAwareFramer) WriteFrame() {
i.writeLock.Lock()
defer i.writeLock.Unlock()
if i.resetChan == nil {
return
}
i.resetChan <- true
}
func NewIdleAwareFramer() *idleAwareFramer {
return &idleAwareFramer{
resetChan: make(chan bool),
conn: &Connection{
closeChan: make(chan bool),
},
}
}
func Kubernetes6632() {
i := NewIdleAwareFramer()
go func() {
i.conn.closeChan <- true
}()
go i.monitor()
go i.WriteFrame()
}
The goroutine operating WriteFrame might purchase the
idle-aware framer lock, adopted by sending a message over the
resetChan channel, whereas the monitor goroutine
waits to obtain a message over the closeChan channel.
Once a message has been dispatched, the monitor goroutine
will try to accumulate the identical lock.
However, since there isn’t any site visitors over resetChan, the ship operation
blocks without end, stopping the monitor goroutine from releasing
the lock.
This, in flip, causes each goroutines to leak.
(pprof) checklist AwareFramer
Total: 200
ROUTINE ======================== essential.(*idleAwareFramer).WriteFrame in .../essential.go
0 100 (flat, cum) 50.00% of Total
. . 32:func (i *idleAwareFramer) WriteFrame() {
. . 33: i.writeLock.Lock()
. . 34: defer i.writeLock.Unlock()
. . 35: if i.resetChan == nil {
. . 36: return
. . 37: }
. 100 38: i.resetChan <- true
. . 39:}
ROUTINE ======================== essential.(*idleAwareFramer).monitor in .../essential.go
0 100 (flat, cum) 50.00% of Total
. . 21:func (i *idleAwareFramer) monitor() {
. . 22: var resetChan = i.resetChan
. . 23: for vary i.conn.closeChan {
. 100 24: i.writeLock.Lock()
. . 25: shut(resetChan)
The repair is to arrange a separate goroutine after a message is acquired
over closeChan within the monitor goroutine that drains the resetChan
earlier than making an attempt to accumulate the lock.
Example (Moby): Misusing sync.WaitGroup
The following example in
Moby showcases how wait teams might
trigger leaks:
kind Manager struct {
plugins []int
}
func (pm *Manager) init() {
var group sync.WaitGroup
group.Add(len(pm.plugins))
for _, p := vary pm.plugins {
go func(p int) {
defer group.Done()
}(p)
group.Wait() // Block right here
}
}
func Moby25384() {
pm := &Manager{
plugins: []int{1, 2},
}
go pm.init()
}
The group wait group increments its counter
relying on the variety of plugins held by the
plugin supervisor pm, then iterates over every plugin
and spawns a goroutine.
Each goroutine decrements the counter as soon as it finishes
its activity with the Done methodology.
However, group erroneously invokes Wait inside
the loop physique, as a substitute of after it!
This will trigger any goroutine operating the init methodology
when the supervisor has multiple plugin to leak.
(pprof) checklist init
Total: 1
ROUTINE ======================== essential.(*Manager).init in .../essential.go
0 1 (flat, cum) 100% of Total
. . 17: group.Add(len(pm.plugins))
. . 18: for _, p := vary pm.plugins {
. . 19: go func(p int) {
. . 20: defer group.Done()
. . 21: }(p)
. 1 22: group.Wait() // Block right here
. . 23: }
This will be simply addressed by transferring the Wait exterior
the loop.
Example (Moby): Mutual blocking between channels and mutexes
Another example
in Moby
showcases a blended channel-lock leak:
kind (
State struct {
Health *Health
}
Container struct {
sync.Mutex
State *State
}
Store struct {
ctr *Container
}
Daemon struct {
containers Store
}
Health struct {
cease chan struct{}
}
)
func (d *Daemon) StateModified() {
c := d.containers.ctr
c.Lock()
d.replaceHealthMonitorElseBranch(c)
defer c.Unlock()
}
func (d *Daemon) replaceHealthMonitorElseBranch(c *Container) {
c.State.Health.CloseMonitorChannel()
}
func (s *Health) CloseMonitorChannel() {
if s.cease != nil {
s.cease <- struct{}{}
}
}
func monitor(c *Container, cease chan struct{}) {
for {
choose {
case <-stop:
return
default:
handleProbeResult(c)
}
}
}
func handleProbeResult(c *Container) {
c.Lock()
defer c.Unlock()
// Additional work...
}
func NewDaemonAndContainer() (*Daemon, *Container) {
c := &Container{
State: &State{&Health{
cease: make(chan struct{}),
}},
}
d := &Daemon{Store{c}}
return d, c
}
func Moby28462() {
d, c := NewDaemonAndContainer()
go monitor(c, c.State.Health.cease)
go d.StateModified()
}
The goroutine invoking StateModified might purchase the lock
of the container saved by the daemon, then invoke
the replaceHealthMonitorElseBranch methodology on
the daemon, which makes an attempt to ship a message over
the cease channel of the container.
However, the goroutine operating monitor
might fail to obtain a message over cease, if the message
will not be already in-flight, and as a substitute unblock by selecting
the default case of the choose assertion.
This will lead it to attempt to purchase the identical container
lock that’s already held by the StateModified
goroutine, main each goroutines to leak.
(pprof) checklist .CloseMonitorChannel
Total: 2
ROUTINE ======================== essential.(*Health).CloseMonitorChannel in .../essential.go
0 1 (flat, cum) 50.00% of Total
. . 66:func (s *Health) CloseMonitorChannel() {
. . 67: if s.cease != nil {
. 1 68: s.cease <- struct{}{}
. . 69: }
. . 70:}
(pprof) checklist essential.handleProbeResult
Total: 2
ROUTINE ======================== essential.handleProbeResult in .../essential.go
0 1 (flat, cum) 50.00% of Total
. . 83:func handleProbeResult(c *Container) {
. 1 84: c.Lock()
. . 85: // Additional work...
. . 86: defer c.Unlock()
. . 87:}
The repair is to shut the cease channel as a substitute
of sending a message over it.
Since closing a channel will not be a blocking operation,
the StateModified goroutine is then in a position to launch
the lock.
In flip, this unblocks the monitor goroutine,
which can now terminate by selecting unblocked
<-stop case department within the choose assertion
on the subsequent loop iteration.
