As this year comes to an end, a new golang release approaches which promises to fix the existing issue of the reference loop values.

But first, let’s start with understanding what the ‘issue’ is by showing the difference between passing by value and by reference. The following code will print what you probably expect: the values 0 to 4 in some random order.

Now, we’ve all probably made the error at some point of referencing a variable loop outside its loop scope. If we update the above code to pass by reference, the result will now be 5 5 5 5 5, probably not what we were expecting.

The issue we're encountering is related to the closure capturing the loop variable i by reference. By the time the goroutines are executed in the main function, they all refer to the final value of i (which is 5) because the loop has completed.

To fix this, you need to create a new variable inside the loop for each iteration, allowing each goroutine to capture a distinct variable. You can achieve this by introducing a new variable in the loop scope:

In the above, each goroutine captures a different variable val, and the sync.WaitGroup ensures that the main function waits for all goroutines to finish before exiting. The output should now be the values 0 to 4 as expected.

This problem was so common that Go 1.21 contains a new experimental feature which changes the semantics so that each iteration would use a new variable. You can turn it on by exporting the flag export GOEXPERIMENT=loopvar. This behaviour will be enabled by default in Go 1.22. Let’s say we have the following example:

If we enable the flag, this time it will produce the following output:

With modifications like these, it's evident that Go is becoming increasingly welcoming to beginners. Tackling issues such as loop variable capture will significantly accelerate the onboarding process for new developers, enhancing the overall experience for gophers at any skill level.

Resources:

• Golang Blog Announcement

• Proposal of Per-Iteration Loop Variable

• Loopvar Experiment Proposal

Last week we’ve delved into the basics of golang’s concurrency, how goroutines are scheduled and ways of synchronisation. This week we’ll take a closer look into some advanced patterns, how they work and when it’s best to use them.

Package Singleflight

• provides a duplicate function call suppression mechanism.

• executes and returns the results of the given function, making sure that only one execution is in-flight for a given key at a time. If a duplicate comes in, the duplicate caller waits for the original to complete and receives the same results.

• especially useful in scenarios where multiple goroutines may request the same data simultaneously; makes sure that a specific piece of work is done only once, even if multiple goroutines request it simultaneously; suited for optimising your code by removing redundant processes that generate the same result.

Let’s say we have the following example:

The fetchData function simulates fetching data. The singleflight.Group is used to manage concurrent requests for the same key. Each goroutine calls sf.Do with a unique key. The function passed to sf.Do is executed only once for a given key, and subsequent goroutines with the same key will wait for the initial call to complete. The sync.WaitGroup is used to wait for all goroutines to finish.

Use Cases:

• Cache Loading - to avoid redundant fetching or computation, making sure the operation is only done once.

• Resource Intensive Operations - for fetching data or performing an operation that is expensive and should be done only once even if requested by multiple goroutines simultaneously.

• Rate Limiting - for limiting the rate at which a specific function is executed to avoid exceeding certain limits.

Sync’s Once Function

• provides a mechanism to ensure that a particular function is executed only once, regardless of the number of goroutines calling it.

• useful for lazy initialisation, one-time setup, or any scenario where a specific action needs to be performed exactly once in a concurrent program.

Use Cases:

• Lazy Initialisation - sync.Once can be used for lazy initialisation of resources that are expensive to create or set up. The initialisation function will be executed only when the resource is first accessed.

• Global Initialisation - When you have global variables or settings that need to be initialised before the application starts, sync.Once can ensure that the initialisation function is executed only once.

• Singleton Pattern - sync.Once can be used to implement a simple form of the singleton pattern, ensuring that an instance of an object is created only once.

• Resource Cleanup - sync.Once can be used to perform resource cleanup or teardown operations exactly once, ensuring that cleanup is not duplicated.

Singleflight and sync.Once might look like they both achieve the same purpose but that’s not the case.

• singleflight :

  • You want to limit the concurrency of a specific function to avoid redundant work.

  • You are dealing with scenarios where multiple goroutines may request the same data or perform similar work concurrently.

• sync.Once:

  • You want to ensure that a particular function is executed exactly once, regardless of the number of goroutines calling it.

  • You are dealing with scenarios like lazy initialization, global setup, or singleton pattern implementation.

Errgroup

• similar to sync.WaitGroup but where the tasks return errors that are propagated back to the waiter.

• useful when you have multiple operations that you want to wait for, but you also want to determine if they all completed successfully.

• errgroup is designed to manage a group of goroutines. You can add goroutines to the group, and the group will keep track of their execution. If any goroutine in the group returns an error, it cancels the entire group, causing all other goroutines to be signaled to stop. This helps to prevent goroutines from continuing to run when an error has occurred. The Wait method of errgroup.Group can be used to block until all goroutines in the group have completed their execution. The Wait method returns an error if any of the goroutines in the group returned an error. This allows you to handle errors centrally.

Use cases:

• Fan-Out Concurrency - When you need to perform multiple independent tasks concurrently (fan-out), and you want to wait for all of them to complete or return on the first error.

• Error Handling - When you want to handle errors collectively instead of checking errors from each goroutine individually.

• Context Cancellation - errgroup integrates well with the Go context package, allowing you to use context cancellation to stop the entire group if needed.

Bounded concurrency via buffered channels

• pattern where you use channels with a specified capacity (buffered channels) to control the level of concurrency.

• can be used to limit the number of goroutines actively working on a task, preventing unbounded resource consumption.

• useful in situations where you want to limit the number of simultaneous network requests, file I/O operations, or database queries to avoid overwhelming external resources.

Use cases:

• Rate Limiting - When you want to limit the rate at which certain operations are performed, such as API requests, file reads/writes, or database queries.

• Resource Constraints - When you need to prevent unbounded resource consumption by limiting the number of concurrent operations.

Let’s say we want to control the rate of our API requests by allowing only two requests to be processed concurrently. In the following example, the processRequest function simulates the processing of each request and we limit the concurrency to 2 using a buffered channel (ch) with a capacity of 2.

Weighted bounded concurrency

• in some cases, not all tasks are equally expensive. So instead of reasoning about the number of tasks we want to run concurrently, we come up with a "cost" for every task and acquire and release that cost from a semaphore.

• mechanism where tasks are assigned different weights, and the system ensures that the sum of weights of concurrently executing tasks does not exceed a certain limit.

• allows for fine-grained control over the level of concurrency based on the computational or resource intensity of each task.

Use cases:

• Resource Management - When tasks have varying resource requirements, and you want to ensure that resource-intensive tasks do not overwhelm the system.

• Fine-Grained Control - When you need fine-grained control over the level of concurrency based on the specific characteristics of each task.

• Adaptive Scaling - When the system needs to adaptively adjust the concurrency limit based on factors such as system load, resource availability, or external conditions.

In the above example, tasks are assigned weights based on their IDs (for illustration purposes). The worker goroutines read tasks from the tasks channel and process them, taking into account their weights. The concurrencyLimit ensures that the sum of weights of concurrently executing tasks does not exceed the specified limit.

Those concurrency mechanisms provide a way to allow you to build highly efficient and responsive systems. Whether handling complex distributed scenarios or managing parallel execution of tasks, Golang's concurrency features contribute to the development of robust and scalable applications in a concurrent world.

This week’s newsletter aims to approach the basics of Go concurrency by going through a couple of concepts:

• the difference between concurrency and parallelism

• how different concepts are done in the JVM (Java) vs Go

• how goroutines are scheduled

• ways of declaring goroutines

• ways of synchronising goroutines by going through the different sync package primitives

Concurrency vs Parallelism

“Concurrency is about dealing with lots of things at once. Parallelism is about doing lots of things at once.” — Rob Pike

Concurrency is a property of the code; parallelism is a property of the running program.

Concurrency is a semantic property of a program or system. Concurrency is when multiple tasks are in progress for overlapping periods of time. Concurrency is a conceptual property of a program or a system, it’s more about how the program or system has been designed. Long story short, concurrency happens when you have context switching between sequential tasks.

Using the same example as Kirill Bobrov uses in Grokking Concurrency, imagine that one cook is chopping salad while occasionally stirring the soup on the stove. He has to stop chopping, check the stove top, and then start chopping again, and repeat this process until everything is done.

As you can see, we only have one processing resource here, the chef, and his concurrency is mostly related to logistics; without concurrency, the chef has to wait until the soup on the stove is ready to chop the salad.

Parallelism is an implementation property. It resides on the hardware layer.

Parallelism is about multiple tasks or subtasks of the same task that literally run at the same time on a hardware with multiple computing resources like multi-core processor.

Back in the kitchen, now we have two chefs, one who can do stirring and one who can chop the salad. We’ve divided the work by having another processing resource, another chef.

Concurrency can be parallelised but concurrency does not imply parallelism. 
e.g. In a single-core CPU, you can have concurrency but not parallelism.

=> We don’t write parallel code, only concurrent code that we hope might be ran in parallel.

What are GoRoutines?

GoRoutines are light-weight execution threads integrated into Go’s runtime that run independently, along with the initialised functions.

Concurrency in Go vs Java

Scheduling

Go’s mechanism for hosting goroutines is an implementation of what’s called an M:N scheduler: which states that M number of goroutines can be distributed over N number of OS threads.

When a Go program starts => it is given a logical processor P for every virtual core => Every P is assigned an OS thread M => Every Go program is also given an initial G which is the path of execution for a Go program. OS threads are context-switched on and off a core, goroutines are context-switched on and off a M.

There are two run queues in the Go scheduler.

• Global Run Queue (GRQ)

• Local Run Queue (LRQ)

Each P is given given a LRQ that manages the goroutines assigned to be executed within the context of P. These goroutines take turn being context-switched on and off the M assigned to that P. GRQ is for goroutines that have not been assigned to a P yet.

When a goroutine is performing an asynchronous system call, P can swap the G off M and put in a different G for execution. However, when a goroutine is performing a synchronous system call, the OS thread is effectively blocked. Go scheduler will create a new thread to continue servicing the existing goroutines in the LRQ.

Go follows a model of concurrency called the fork-join model:

• fork — at any point in the program, a child branch of execution can be split off and run concurrently with its parent

• join — at some point in the future, the concurrent branches of execution will join back together

Ways of declaring goroutines

Synchronizing goroutines — via sync package

To make sure your goroutines execute before the main goroutine we need join points. These can be created via:

The WaitGroup primitive

used for waiting for a set of concurrent operations to complete when you either don’t care about the result of the concurrent operation, or you have other means of collecting their results.

Closures = a function value that references variables from outside its body.

With closures, we’d have to pass a copy of the variable into the closure so by the time a goroutine is run, it will be operating on the data from its iteration of the loop.

The Mutex primitive

provides a concurrent-safe way to express exclusive access to these shared resources.

How to: sync.Mutex interface with Lock() and Unlock() methods

Shares memory by creating a convention developers must follow to synchronise access to the memory.

The RWMutex primitive

same as Mutex but it provides a read/write lock. We can have a multiple number of readers holding a reader lock as long as nobody is holding a writer lock.

How to: sync.RWMutex interface with RLock() and RUnlock() methods

RWMutex can only be held by n readers at a time, or by a single writer

A way of choosing which primitive to use is emphasised in the following diagram:

Other basic primtives

The Cond Primitive

a rendezvous point for goroutines waiting for or announcing the occurrence of an event (=signal between 2 or more goroutines, has no info other than it happened).

How to: sync.NewCond(&sync.Mutex{}) with 2 methods:

1. Signal - notifies goroutines (runtime picks the one that has been waiting the longest) blocked on a Wait call that the condition has been triggered

2. Brodcast - sends signal to all waiting goroutines

Once

ensures that only one call to Do ever calls the function passed in

counts the no of times Do is called, not how many unique functions passed into Do are called

Pool

= concurrent-safe implementation of the object pool pattern.

How to:

1. Get interface - checks wether the are any available instances within the pool to return to the caller, and if not, call its New member variable to create one.

2. Put interface - to put the instance they were working with back in the pool

Uses cases:

• memory optimisations as instantiated objects are garbage collected.

• warming up a cache of pre-allocated objects for operations that must run as quickly as possible. (by trying to guard the host machine’s memory front-loading the time it takes to get a reference to another object)

Channels

Can be used to synchronise access of the memory and to communicate information between goroutines.

They support unidirectional flow of data:

• channel that can only read

• channel that can only send

Sending to/Receiving from a channel

Channels are blocking.

Any goroutine that attempts to write to a channel that is full will wait until the channel has been emptied.

Any goroutine that attempts to read from a channel that is empty will wait until at least one item is placed on it.

Closing a channel

Ranging over a channel

Closing a channel signals to multiple goroutines

Buffered Channels

= channels that are given a capacity when they’re instantiated.

Buffered channels are in-memory FIFO queue for concurrent processes to communicate over.

In the next newsletter, we’ll dive deeper into the result of channel operation given a channel’s state, how to design your code into channel owners and consumers and some other go concurrency patterns.

Go’s Memory Management

Golang’s memory is divided into two areas, one for the stack and one for the heap. The stack is usually used for data with a known, fixed size that is determined at compile time, such as function variables and function call frames. It’s automatically allocated and deallocated as functions are called and returned. Each goroutine has its own stack frame.

On the other hand, the heap is usually used for global variables, local variables with a large memory footprint, local variables that cannot be reclaimed immediately after a function call. Other examples might include objects created with new or make and in the case of reference types, such as slices and maps, which store pointers to the underlying data, which in turn resides on the heap. Objects stored on the heap are shared between all the goroutines of a process. The heap is managed by the Go’s garbage collector.

Why is the stack faster than the heap?

Allocating variables on the stack is generally faster than if we do it on the heap. This performance difference is primarily due to the characteristics of how CPUs and memory access work. The stack is a region of memory that is often small and is closely tied to the execution of a specific thread or goroutine. As a result, the stack is more likely to reside in the CPU cache. Accessing data in the cache is significantly faster than fetching data from the main memory (heap). But let’s do some testing to prove it:

• Allocating small objects to stack vs heap

We can see it’s around ~22.5x faster:

• Allocating large objects to stack vs heap

Now it’s ~1,268x faster:

• Allocating huge objects to stack vs heap

Now we can see its ~648,532x faster:

How does the compiler decide whether to put a variable on the stack or on the heap?

The size threshold when it comes to deciding whether a variable is allocated on the heap rather than the stack is determined by the compiler using some heuristic rules. Usually, small arrays (e.g., a few kilobytes) are more likely to be allocated on the stack, while larger arrays will most certainly be allocated on the heap. The exact threshold cannot be known and depends on various factors, including the specific compiler and its settings. The decision might also be subject to change based on improvements or adjustments to the compiler itself. Developers typically don't have direct control over it.

Why do we care what stays on the heap?

Memory allocated on the heap is reclaimed through the process of garbage collection, which can sometimes be an expensive process, causing latency.

The garbage collector periodically scans the program's heap, looking for objects that are no longer reachable. Once the garbage collector determines that an object is no longer reachable, it will release the memory associated with that object and make it available for reuse. It is a concurrent operation, which means it runs concurrently with the application's execution. However it has stop-the-world (STW) phases where all the application’s goroutines are paused. The duration of these pauses is influenced by factors such as the size of the heap and the activity of the program.

When a collection starts, the collector runs through three phases of work:

• Mark Setup - STW

  • At the start of a collection process, the initial step involves activating the Write Barrier. The Write Barrier serves the purpose of ensuring data integrity on the heap throughout the collection, as both the collector and application goroutines operate concurrently.

  • To enable the Write Barrier, it is necessary to halt every currently running goroutine. Typically, this interruption is swiftly executed, averaging between 10 to 30 microseconds, provided that the application goroutines are functioning as expected.

• Marking - Concurrent

  • Once the Write Barrier is turned on, the Marking phase is started.

  • The collector uses CPU resources to scan and identify unreferenced objects, trace the object graph, and perform other necessary tasks.

  • The Marking phase involves identifying and marking values in heap memory that are still in use. This process starts by examining the stacks of all active goroutines to locate root pointers leading to heap memory. Subsequently, the collector traverses the graph of heap memory starting from these root pointers.

    • starts with a set of "root" objects, such as global variables, and then "traces" the pointers from these objects to other objects, marking all of the objects that are reachable. Any objects that are not marked as reachable are considered garbage and are eligible for collection.

• Mark Termination - STW

  • Once the Marking work is done, the next phase is Mark Termination, when the Write Barrier is turned off, various clean up tasks are performed, and the next collection goal is calculated.

Go’s Escape Analysis

Escape analysis determines whether the variables can reside on the stack or should be escaped to the heap during the compile time. It tries to keep variables on the stack as much as possible. If a variable is allocated on the heap, the variable is escaped (from the stack).

In performance-critical domains where you want to maximise the efficiency of your application, having a thorough grasp of Go’s escape analysis can prove highly advantageous.

The way a variable is used - not declared - determines whether it lives on the stack or the heap.

Sharing up (returning pointers) typically escapes to the heap.

Go encourages the use of value types, which are allocated on the stack, as they don't require garbage collection.

Variables escape to the heap when their lifetime extends beyond the function or scope in which they are defined. This can happen when the variable is returned from a function or stored in a data structure that outlives the function. Garbage collection is responsible for reclaiming memory occupied by variables that have escaped to the heap.

Here's a simple example to demonstrate variable escape and garbage collection:

If we run go build -gcflags=-m main.go we can see which variables will escape.

# command-line-arguments
./main.go:23:22: moved to heap: user
./main.go:18:21: ... argument does not escape
./main.go:18:63: func literal does not escape

1. The createObject function creates a UserData object and returns a pointer to it.

2. In the main function, a loop calls createObject repeatedly, and the returned pointers are assigned to the user variable.

3. The user variable escapes to the heap because it is returned from the createObject function.

We can investigate the effects the above code has on the heap. If we run the above code, the defer profile.Start(profile.TraceProfile, profile.ProfilePath(".")).Stop() will generate a file called trace.out which we can investigate by running
go tool trace trace.out

You can notice the heap starts increasing with time and at a certain point, the GC runs and cleans up the heap, before it starts growing again.

If we modify the above example so that the createObject function directly returns the UserData object instead of a pointer to it, we can see the variable no longer escapes.

This time if we run go build -gcflags=-m main.go and go tool trace trace.out we get:

# command-line-arguments
./main.go:10:6: can inline createObject
./main.go:16:63: inlining call to profile.ProfilePath
./main.go:20:22: inlining call to createObject
./main.go:16:21: ... argument does not escape
./main.go:16:63: func literal does not escape

Now, the createObject function returns the UserData object directly, and there's no need to use a pointer. This ensures that the UserData object is allocated on the stack within the loop, preventing it from escaping to the heap.

Other examples of variables escaping

Generally, a variable escapes if:

• its address has been captured by ​the address-of operand (&).

• huge objects

  • For explicit declarations (var or :=)

    • The variables escape if their sizes are over 10MB

  • For implicit declarations (new or make)

    • ​​The variables escape if their sizes are over 64KB 

• a slice variable escapes if its size is variable and not constant

• return values if:

  • they are pointers

  • they are of map or slice types

• closure functions - a variable escapes if the source variable is captured by a closure function

Before we go, if you ever found it difficult to read the cli results of the go build -gcflags=-m, here is a command line tool that translates the escape analysis results into a human-readable format to output which variables will be stored on the stack and which on the heap. Code available on github.

Conclusion

While the garbage collector introduces some overhead, it also provides memory safety and automatic memory management, reducing the risk of memory-related bugs. If you have specific concerns about stack vs. heap allocation for performance reasons, it's recommended to rely on profiling and benchmarks to identify potential bottlenecks in your code. The Go runtime and compiler are designed to handle memory allocation efficiently, and the decision of stack vs. heap allocation is part of their optimisation strategies. Premature optimisation can lead to various issues. Only optimise services when they have performance or cost issues. In most cases, you don’t need to worry about where variables are allocated, as the garbage collection knows what it’s doing.