As reflection is an essential part of the learning process, this article is all about acknowledging that even with the right intentions and much dedication, you don’t always get it right. This is about accepting this reality and reminding myself that growth is a learning curve and despite setbacks we need to continue to take ownership.

What problem did we try to solve?

Maintaining accountability within data-driven systems along with keeping track of who did what and when within isn't just a security requirement—sometimes it's a crucial business necessity. Whether if it’s just for meeting compliance requirements, troubleshooting and understanding system behaviour, investigating production incidents or providing insights for post-mortem analysis, a robust audit logging solution serves as your system's source of truth.

But building such a solution comes with a lot of challenges in terms of scalability, performance and most of all, data integrity. In this article, we’ll explore what are the possible approaches to implement such a system that captures meaningful events, stores them reliably, and makes them easily accessible when you need them most. We'll cover the essential components that make up a production-grade audit logging solution and emphasise some lessons learnt from implementing a “tamper-proof solution”.

What is an audit log?

We want to know all database interactions that happened during a period of time, along with detailed information about each event. Those logs should be queryable and be able to answer questions like "who changed this value" or "which process caused this item to be removed." Audit logs record the occurrence of an event, the time at which it occurred, the responsible user or service, and the impacted entity.

A good audit logging system should not only record event data but also consider various usability criteria for the teams inspecting the logs (i.e the end users) and the future maintainers (e.g. engineers) of the system. The main requirements we’re going to work with:

• audit logs should be reliable - any database change should have a corresponding audit log entry.

• have no or minimal setup - should be easy to set up across the ecosystem and ideally require no or minimal code changes for new services.

• audit logs should be immutable - users and services should only have the ability to read data the logs and not update them.

• audit logs should be easily queryable (by service, action, date, user etc) - users must be able to efficiently search through the logs by applying any one or more filters.

• audit logs should be parsable - logs should also be easy to parse and work with.

Since the ecosystem we work with is only using DynamoDB as a storage option, we’ll design the audit log solution to work especially with this kind of database.

Solutions Considered

1. via the Transactional Outbox Pattern

One solution to implement the audit logs would be to emit an event with every database change. This event would contain all the necessary information to track the change (what got changed, who changed it etc). The consumer of those events would process the data and store it in a way that would be easily accessible and queryable by all devs.

In our current ecosystem, we use an internal library go-kit (similar to the open source one go-kit) to help us with the scaffolding and management of our golang microservices. This library also provides a standardised way for doing all db interactions. This approach would mean just modifying this library to emit an event when doing a CRUD operation. The advantage would be services don’t need to be aware of this logic and no logic would need to be updated.

The problem with this solution is the general dual-write issue where you want to make 2 write operations in one atomic transaction. If we want to update 2 different systems, persist data in the database and send an event to notify other systems, because these two systems aren't linked, we have no way to update both in a transactional fashion. When a microservice sends an event notification after a database update, these two operations should run atomically to ensure data consistency and reliability.

The way to solve this is usually through the Outbox Pattern. If we have a database that supports transactional updates, then we can use it to overcome the dual-write problem. In our case, we would need another table, say an “Outbox Table“, and on a DB CRUD operation we would 1. do the CRUD operation and 2. save the change we did in the outbox table (both as part of a transaction which DynamoDB supports). The records in the outbox table will then be read by some workers which will emit an event with the changed data. The events will be read by an “Auditor Service“ that will process them and finally store them in a queryable way.

This approach comes with a fair bit of complexities. We would have at least one other table (we can choose to use one single outbox table for all services or create a seperate outbox table for each service). We would introduce some latency due to the additional database write for every transaction (to outbox). There is also the consideration of how we’re going to manage the outbox table growth/cleanup. We would need to think how to coordinate the allocation of work between the worker services.

2. via the CDC Approach

A second solution would be relying on a Change Data Capture (CDC) approach. CDC is the process of identifying and capturing changes made to data in a database and then delivering those changes in real-time to a downstream process or system). The basic idea is that we will store our data to a storage that supports CDC. Then the CDC infrastructure will notify external systems of the changes. Fortunately for us, DynamoDB provides such an option, called DynamoDB Streams.

DynamoDB Streams captures a time-ordered sequence of item-level modifications in any DynamoDB table and stores this information in a log for up to 24 hours (can be extended to 7 days). Applications can access this log and view the data items as they appeared before and after they were modified, in near-real time.

At first glance this approach sounds like a good solution for our use case. No other resources are required to be setup. DynamoDB provides a CDC capability out of the box. Scalability and reliability are provided natively by AWS. The code what we would need to add would be minimal as we just need a service which reads the data from the dynamodb streams. So we went ahead with this.

What’s in a DynamoDB Stream?

Any dynamodb table with dynamodb streams enabled can have one dynamodb stream active. At any point in time, we can only have one active stream. A stream can have multiple shards. A shard can contain zero or more records. A record is the change that we want to be captured.

Depending on how the stream was set up when it was enabled, the shard record can contain different types of data (source):

• KEYS_ONLY - Only the key attributes of the modified item are written to the stream.

• NEW_IMAGE - The entire item, as it appears after it was modified, is written to the stream.

• OLD_IMAGE - The entire item, as it appeared before it was modified, is written to the stream.

• NEW_AND_OLD_IMAGES - Both the new and the old item images of the item are written to the stream.

We’re interested in the last type, NEW_AND_OLD_IMAGES, and so in our case, we’re expecting our record to look similar to the following:

Costs

DynamoDB Streams does not cost anything to enable. DynamoDB charges on the number of read requests made towards the table’s DynamoDB Stream. Each read request in terms of the stream is in the form of a GetRecords API call that can return up to 1000 records or 1 MB worth of records in the response, whichever is reached first. None of the other DynamoDB Stream APIs (source) For DynamoDB Streams the cost is $0.25 per million reads.

Approaches to Reading from Streams

DynamoDB streams can be processed in multiple ways. The most common approaches (source) are:

• trigger based via AWS Lambdas

• polling based with a standalone application that uses the Kinesis Client Library with the DynamoDB Streams Kinesis Adapter

• polling based with a standalone application connecting directly to the Dynamo Stream APIs through the AWS SDK

The AWS Lambdas Approach

Amazon DynamoDB comes integrated with AWS Lambda so that you can create triggers—pieces of code that automatically respond to events in DynamoDB Streams. Lambdas will poll the DynamoDB stream and, when they detect a new record, invoke your Lambda function and pass in one or more events.

Under the hood though, the AWS Lambda service will poll the stream for new records four times per second. (source) The polling interval cannot be changed, you can only change things like the batch size or the parallelization factor. (source)

Going with this approach would have a couple of advantages. The polling logic for data streams is automatically managed by Lambda functions, removing the need for manual implementation. Lambda functions provide built-in autoscaling capabilities, handling varying workloads efficiently without manual intervention. When using Lambda functions, you avoid the costs associated with making GetRecords calls to the streams, as this is handled by the AWS infrastructure.

On the other hand, there are significant additional costs involved, with an estimated expense of approximately $194 per month when handling 100 tables. The setup process requires extra effort, introducing changes to the deployment topology and adding complexity when establishing new data pipelines. Observability is limited as CloudWatch is the only available tool for monitoring and understanding what occurs within Lambda functions. Lambda functions must be designed to execute quickly, with minimal processing time, making them unsuitable for complex or long-running operations within the function logic.

Using the DynamoDB Streams Kinesis Adapter

The Amazon Kinesis Adapter is Amazon's recommended solution for consuming DynamoDB streams. However, a significant limitation is that this adapter is exclusively available in Java, making it unsuitable for non-Java applications. The Kinesis Client Library (KCL) offers a higher level of abstraction above the basic Kinesis Data Streams API, making the development process more straightforward and efficient. It provides robust batch processing capabilities, enabling efficient handling of multiple records simultaneously. The library includes automatic checkpointing functionality, maintaining accurate records of which data has been processed without requiring manual tracking.

The most significant limitation is that the DynamoDB Streams Kinesis Adapter is only implemented in Java and available through the awslabs/dynamodb-streams-kinesis-adapter repository, making it inaccessible for applications written in other programming languages.

Directly through the Dynamo Streams APIs using the AWS SDK

The third option is to have a service that constantly polls the streams for new records using the available aws sdk for go. This has a couple of implications, we need to keep track of an index to know which records have been processed so far and due to the replicated nature of our systems, we need an idempotent way to achieve this.

And some things worth considering:

• Streams have rate limits - the DescribeStream endpoint can be called at a maximum rate of 10 times per second; the GetRecords endpoint can be be called at 5 TPS per stream shard or 2MB/s, whichever comes first (source). Therefore, we cannot read more than 10 streams per sec.

• You cannot have more than one active stream per table.

• DynamoDB streams guarantee no duplicates and ordering of records.

Implemented Solution

Most of the time reinventing the wheel is the worst solution you can choose. But what do you do when the current company policies and/or ecosystem makes it hard to integrate these libraries or this new set up in particular? Do you use the existing wheel but build a completely special car to work with it, or build a new wheel that fits into your existing car? All of the sudden, crafting this new wheel sounds like the easiest option. So here we are, reinventing the wheel because…it looked easy enough.

Polling Processing Logic

Before going into the high-level overview on how we’re using the aws go sdk to continuously poll the Dynamo Streams APIs, a couple of key concepts on how DynamoDB Streams work (which took us quite a lot of searching, digging and calls with AWS Support):

• We can have 2 types of shards: OPEN and CLOSED.

• There is a one to one mapping between the dynamodb table partition and an open shard.

• A shard is OPEN if EndingSequenceNumber == nil and CLOSED otherwise.

• A shard can have (a lot of) successor shards (but every shard can only 1 child)

  • for CLOSED shards, you can iterate through all its successors until you get to the end denoted by NextShardIterator == nil.

  • for OPEN shards, NextShardIterator is never nil.

• Every shard (parent or child) can contain records and the SequenceNo is a pointer to that shard record.

We will store some shard/record information to be able to keep track of where we left off:

• a streams table - containing the table_name (which is the dynamodb table name we’re going to audit log), is_enabled (whether the dynamodb streams are currently enabled for it) and stream_arn (which is dynamodb stream identifier for the stream)

• a streams_shards table - containing the stream_arn, shard_id (the identifier of the stream shard), sequence_no (the sequence number of the given shard which we will use to know when in the shard we left off processing, similar to how an index works) and status (which for us can be open meaning the shard is still open on dynamodb side, closed meaning the shard is closed on dynamodb shard but we still have records left to process and closedAndProcessed meaning the shard is closed on dynamodb side and we have processed all records)

The way our processing logic will work is that given a stream_arn:

• we get a list of all the parent shards with their dynamodb statuses.

• if it's dynamodb CLOSED:

  • we check if we have already processed all records in it (where our status is closedAndProcessed) and if so skip

  • otherwise (where our status=closed) loop through all the successor shards until NextShardIterator == nil, process all the records in them and update shard status to closedAndProcessed.

• if OPEN:

  • loop through the shards using NextShardIterator.

Our main issue is what do we do with open shards, since NextShardIterator is never nil? When do we know when to stop? At first, we made the assumption that if we get to a couple of consecutive shards with 0 length records, we can assume the shard is caught up. Unfortunately, this is not the case (confirmed with AWS Support team). Even if we encounter multiple consecutive shards with 0 records, it's generally recommended to continue iterating through subsequent shards, as we can have data in later shards. Now you would say, that’s not a big deal, just continuously loop and if you see new records, save those. To point out, you might not need extra logic here for making sure you’re not processing duplicate records, remember we’re always storing the map of shard_id with sequence_no. So when you’re processing a shard that’s already been seen, we’re going to start from the given sequence_no.

The problem with this approach is that we don’t have just a couple of dozens of successor shards for open shards, we proved to have a couple of thousands. A couple of thousands with zero records in them. This is for a table with quite a reasonable (not very high) throughput. Overall, this overcomplicates our approach quite a bit as the time it takes to complete doing a full read looking for new changes increased significantly.

How is dynamodb streams spinning successor shards so chaotically? Why do we have so many with zero records in them? Why aren’t records written sequentially, why do we have large “gaps” (zero length record shards) in between shards with data? Would love to get an answer to those questions. We ended up not getting one so far.

Lessons Learnt

Even though every project starts with an MVP to prove the feasibility of the solution, as it started in this case, it might be worth to do a test on a production scale environment. If we would have faked production workload on local/staging to test the MVP, maybe we would have noticed some of the issues and started asking questions at an earlier stage.

This time, we’ll have a look at how Bluesky is setup and go through the novel approaches they use to solve certain challenges. As of December 2025, Bluesky has crossed the 20 million users mark while being maintained by no more than 15 engineers. Martin Kleppman, author of the “Designing Data Intensive Applications”, currently a technical advisor, published a paper1 outlining how Bluesky is functioning as a federated and decentralised social media platform. Gergely Orosz's has two good articles covering the company’s early days, culture and evolutionary architecture if you would like to check them out.2 In the meantime, let’s start with the basics.

Centralised, decentralised and federated

Decentralised networks have no central authority or server, and all participants (nodes) are equal and have the same capabilities. Data and control are distributed across all nodes. If one node fails, the network continues to function. One example of a fully decentralised network is Bitcoin - no single entity controls it, and all participants maintain a copy of the ledger.

Federated networks have multiple independent servers that can communicate with each other, where each server has its own administrator and user base. Users on different servers can interact with each other. Each server follows its own rules while adhering to a common protocol. For example, email providers (gmail, yahoo, and other providers) run their own servers but can communicate with each other via standard protocols.

A system can be both federated and decentralised. This is often called a federated decentralised network. The federation part provides standard protocols for communication, interoperability between different servers/instances and local administration and rules. The decentralisation part ensures no central authority controls the entire network, anyone can set up their own server/instance and there is no single point of failure.

In which category should X/Mastodon be part of?

X maintains complete centralised control over its platform. They have exclusive authority over your personal data, your identity, and content management (which includes both moderation policies and feed generation). While X has taken steps toward transparency by partially open-sourcing their algorithm3, users remain largely passive observers - able to understand parts of how the system works but with minimal ability to influence or modify its operation.

Mastodon is federated because different servers can communicate with each other using the ActivityPub protocol. It's partially decentralised because no one owns or controls the entire network and anyone can set up their own Mastodon server. If one server goes down, the rest of the network continues functioning. The key difference from purely decentralised systems (like Bitcoin) is that these networks maintain some organisational structure through federation while still preventing central control. They strike a balance between complete decentralisation and practical usability. One disadvantage though, is that in the same way there are huge email providers, there are big Mastodon servers. Your user identity becomes tied to a server and it might get challenging (if not impossible) to move to another server.

Bluesky architecture high-level overview

Bluesky uses a protocol called the AT Protocol (Authenticated Transfer Protocol)4 aka atproto, which works similar to email in some ways. Just like how email allows you to use Gmail but still message someone with a Yahoo account, Bluesky lets you choose different providers (called Personal Data Servers or PDS) to host your account, communicate with users on different servers, and take your account and data with you if you want to switch providers.

The key aspects through which Bluesky encourages decentralisation is that anyone can run their own server and connect to the network, users have data portability (can move their account between providers) and the protocol is open, meaning anyone can build compatible apps or services.

Let’s go through the main components that make the Bluesky’s architecture.

We have the PDS (Personal Data Server) which is the central part of the architecture. It hosts user repositories and data, manages user identity, orchestrates requests to other services, handles data sync and federation, can be self-hosted or provided (like Bluesky PDS). User Data Repository holds the User Identity (DID), which is the decentralised identifier for authentication. It handles the cryptographic authentication, signing and verification and holds the records structure like posts, follows, likes and profile data. In order to consume data, aggregate it to your liking and distribute it in real time, you have the Network Relays which index and collect content from PDSs and then use Firehose to distribute it so that you can process updates in real time. The firehose data is consumed by the Feed Generator (Discover) to create custom content feeds and by the Labeling Service (Ozone) to provide content moderation, tagging and content classification. The data from those two Opinionated Services is consumed by the App Views, which acts as the user-facing platform.

Interoperation and federability

The protocol uses a system called "Lexicons", these are like agreed-upon vocabularies that define how different types of data (posts, likes, follows, etc.) should be structured. This standardization allows different servers and clients to understand each other, similar to how different email clients all understand what an email header should look like.

Each server supports specific features by implementing "lexicons" - there are core ones (starting with com.atproto.*) that handle basic stuff like syncing user data, and social-focused ones (starting with app.bsky.*) that enable social media features.

Unlike the regular web, which shares HTML pages that determine both content and appearance, the AT Protocol only shares the underlying data in a structured format. This means apps can display the same information in their own unique ways, without needing to download and run someone else's display code (HTML/JS/CSS). It's like getting pure information that each app can then present however it wants. 5

The user data repository

A noteworthy feature is repositories ("repos"). Each user has their own personal repository that contains all their data - posts, follows, likes, etc. These repos are designed to be efficiently synced between servers, maintain a verifiable history of changes and allow users to take their entire social presence with them if they switch providers.

The AT Protocol (ATP) has some interesting technical characteristics that enable decentralisation. The core of ATP is built around something called "Self-Authenticating Data Structures". Every post, profile, and interaction is signed cryptographically, which means data can be verified as authentic regardless of which server it comes from, users truly own their identity through cryptographic keys and content can't be tampered with without detection.

An interesting technical choice is that ATP uses DIDs (Decentralised Identifiers) as the foundation for user identity, which are persistent identifiers that remain valid even if you change providers.

Every user has two identifiers:

• A human-readable handle (like username.bsky.social or custom domain)

• A permanent DID (Decentralised Identifier, like did:plc:xyz123...)

User Handle (e.g., alice.bsky.social)
       ↓
Resolves to DID (did:plc:xyz123...)
       ↓
DID looks up in PLC server
       ↓
Returns DID Document containing:
- Public key
- PDS server location
- Handle verification

Users can change their handle without losing their identity because the DID remains constant. The DID document points back to your handle, creating a verification loop. Any changes to the DID document must be signed with your private key and create a verifiable chain of updates. All updates require cryptographic signatures. Changes create a verifiable chain (like a "mini-blockchain"). Public data is self-verifiable. Third parties can detect manipulation.

Some limitations with the current approach are that it relies on centralised DNS system, there’s currently a single PLC server managed by Bluesky and most users' private keys are currently managed by Bluesky.

How decentralised and federated is Bluesky actually?

Bluesky doesn't use a traditional message-passing system like other federated networks (ActivityPub, XMPP, email). Instead, it uses a "shared heap" architecture where data exists in one global pool. There's no direct message-passing between hosts, which is a common federation pattern. Most users are currently hosted on Bluesky's PDS (Personal Data Server) instances. The chat/DMs service is completely centralised and most services rely on Bluesky's relay as a firehose.

The platform is not yet fully decentralised as the power is not evenly diffused throughout the system, most users run the Bluesky app, most developers work with Bluesky-defined application schemas (Lexicons), most self-hosted users run Bluesky's PDS software. Running a full-network relay requires significant resources (16TB of fast NVMe disk). Most DID PLC accounts don't have independently controlled PLC rotation keys configured.

One thing I would not debate is that much of the protocol and the network is reliant on Bluesky in material terms today. Almost everybody in the atproto network is hosted on a Bluesky PDS instance. Most self-hosted folks run the Bluesky PDS software. Most services use a Bluesky relay as a firehose. Most users run the Bluesky app. Most developers are working with the Bluesky-defined application schemas (Lexicons). - Bryan Newbold, Protocol Engineer at Bluesky 6

While Bluesky's underlying protocol (atproto) is designed to support decentralisation, in practice most of the network currently runs through Bluesky the company. Almost everyone uses Bluesky's servers, app, and relay services - making it functionally similar to Twitter/X today. The key difference is in the architecture.

It allows for "credible exit", users can leave without losing their data or connections. It Supports independent Lexicons (like the whtwnd.com blogging platform). Every major infrastructure component can be substituted without friction. Users can self-host their own instances. Lets users own their identity and data. The protocol is designed to prevent any single party from moderating the entire network.

So, unlike Twitter, Bluesky's infrastructure is built to allow users and developers to run their own independent services, migrate their data, and participate in the network without relying on Bluesky's systems. This capability exists even though relatively few are using it right now.

The early cron days

The original cron was designed by Ken Thompson in Unix V7 (around 1979) and was later rewritten by Dennis Ritchie. It used to work in a simple way. It would read the crontab files from /usr/lib/crontab, used a simple in memory table for job scheduling and had one main process that woke up every minute and determine if any commands must run at the current date and time, and if so, run them as the superuser, root.

This method presented a couple of challenges. It could easily overload the system if too many jobs ran simultaneously. It consumed resources every minute whether there was a job to be run or not. Performance degraded with more entries. It wasn’t suitable for large systems with many users. Memory usage grew linearly with number of jobs. There was no way to limit the number of jobs per user.

These limitations led to the development of more sophisticated cron implementations for multi-user capability.

The Franta-Maly Event List Approach

The Unix System V came with an upgraded version of cron based on the 1977 paper by W.R. Franta and Kurt Maly, An Efficient Data Structure for the Simulation Event Set. [1] In this paper, they presented a new event scheduling algorithm that improves on previously published algorithms. Robert Brown, a Purdue graduate student, noticed similarities between cron and discrete event simulators, leading him to implement the Franta-Maly event list manager. In 1979, Keith Williamson, a new graduate student, developed Brown's prototype into a production-ready multi-user cron service at Purdue.

The cron algorithm begins by searching for .crontab files in all users' home directories at startup. For each discovered crontab file, it calculates the next future execution time for each command and adds these commands to the Franta-Maly event list, along with their execution times and five-field time specifiers. The main loop then continuously examines the first task in the queue, calculating its future execution time. The system sleeps until that time, then wakes to verify the time and execute the queued task in the background using the original user's privileges. After execution, it recalculates the next runtime for that command and reinserts it into the event list accordingly.

The daemon implements two monitoring mechanisms for crontab file changes: it processes SIGHUP signals to trigger immediate rescans of modified crontab files, and it maintains scheduled wake-up checks at 30-minute intervals (on the hour and half-hour) to detect any crontab modifications.

The Franta-Maly event list (Two-Level algorithm) has a time complexity of O(√n) in the worst case scenario, driven by its two-tier data structure design and event insertion mechanism.

The Two-Level (TL) Algorithm Data Structure

The Two-Level (TL) algorithm proposed by Franta and Maly utilises a sophisticated data structure to efficiently manage event notices in discrete event simulations. The primary goal of this structure is to reduce the number of comparisons required to find the correct position to insert a new event notice, thus optimising the simulation's performance. The algorithm utilises a data structure inspired by indexed lists and balanced tree concepts (3-2 trees), to minimise the computational cost of scheduling and processing events.

The data structure consists of 3 layers:

• the index list layer - The TL algorithm uses an index list to divide the range of event times into a number of equal-sized intervals. Each element in the index list points to a "dummy key" which represents the boundary of an interval.

• the secondary key layer - The efficiency of the TL algorithm lies in the introduction of secondary keys within each interval. These keys, associated with the right boundary dummy key, point to sublists of event notices within the interval. This two-level indexing scheme enables more efficient searching and insertion operations.

• the event notices layer - Each interval is further divided into sublists, each headed by a secondary key. The sublists are designed to be balanced, meaning they don't grow excessively large. The maximum size of each sublist is controlled by nlim parameter.

The Insert Operation - Worst Case Complexity: O(√n)

Let’s say we want to insert a new event with t=1.3. The Franta-Maly insertion process follows a systematic sequence through the data structure's layers. First, we calculate the target interval by applying the formula i = [(t - lowerbound)/dt] to determine the precise interval containing the event time, which leads us to the corresponding dummy key entry in the index list.

Next, we traverse the sequence of secondary keys associated with the identified dummy key until locating the appropriate insertion point. For an event with time t=1.3, this involves scanning until we find it belongs between secondary keys 1.2 and 1.5.

The actual insertion occurs by placing the event notice in its correct time-ordered position within the identified sublist. In our example, the new event with t=1.3 is inserted between existing events with times 1.2 and 1.4, maintaining the temporal ordering of the sublist.

Finally, we execute a balance check to maintain the data structure's efficiency. The system verifies if the insertion has caused the sublist to exceed its maximum size (nlim). When the size limit is exceeded, the structure triggers one of two rebalancing operations: either transferring the last notice to an adjacent list or creating a new secondary key. If the sublist remains within size constraints, the insertion completes without requiring rebalancing.

The worst case complexity occurs when all n notices are associated with the same dummy key and the new event being inserted has a time smaller than all other notices in the structure. The complexity is calculated as: cw(n) = n/nlim + nlim, where nlim is optimally set to √n to minimize the worst case complexity.

The Search Operation - Worst Case Complexity: O(√n)

Let’s say we want to search for an event with t=1.3.

The search process in the Franta-Maly structure executes in three sequential phases. The first phase calculates the target interval using the formula i = [(t - lowerbound)/dt], which maps directly to the corresponding dummy key in the index list. This initial lookup operates in constant time O(1).

The second phase traverses the secondary key sequence within the identified interval until locating the appropriate sublist. For a target time t=1.3, the algorithm scans between secondary keys 1.2 and 1.5, ultimately selecting SK2's sublist. This traversal has a worst-case complexity of O(√n).

The final phase executes a linear search within the selected sublist to find the exact match or appropriate position. Since the maximum sublist size is constrained by nlim (√n), this search also operates with worst-case complexity O(√n). The combination of these three phases yields an overall worst-case complexity of O(√n), though practical performance often approaches constant time.

Calendar Queues (CQ)

A calendar queue is a priority queue data structure specifically designed for efficient handling of time-based events. It was introduced by Randy Brown in 1988 [2] and is particularly useful in discrete event simulation systems.

The queue is organized like a calendar, divided into "buckets" (like days or time slots). Each bucket represents a fixed-width time interval. Events within each bucket are sorted by their timestamp. The buckets are arranged in a circular array (wrapping around like a year calendar)

The data structure consists of 4 components:

• Buckets: An array representing days in a year, each element pointing to a sorted linked list of events scheduled for that day.

• Event Nodes: Contain information about the event, including its time (priority) and any associated data. These are organised into sorted linked lists within each bucket.

• Circular Calendar: Events can be scheduled up to a year in advance. Once the last day of the year is reached, the calendar wraps around to the first day, allowing for continuous event scheduling without data structure manipulation2.

• Event Dates: Each event node stores its scheduled date to handle events scheduled beyond a year. Events with future dates are skipped during dequeue operations until the calendar reaches their scheduled year

The above image shows how Brown’s Calendar Queue looks like, where the year starts at 12.0 and end at 16.0 and the bucket width is 0.5 time units.

How are events inserted?

Events are assigned to buckets using this formula:

bucket_number = floor(priority/bucket_width) % number_of_buckets

Let’s say we want to insert a new element 14.6 and we have the following initial state:

We have the current year that starts at 12.0 and ends at 16.0 (range of 4.0 units), with a bucket width = 0.5 time units and 8 buckets. We calculate the

bucket_number = floor(14.6/0.5)%8 = 29%8 = 5

The element belongs to bucket 5 and will be inserted into sorted position into the bucket’s linked list.

This insertion process maintains O(1) average complexity since bucket calculation is O(1), each bucket contains a small, roughly constant number of events, and searching within a bucket's linked list is therefore bounded by a constant.

How is bucket width calculated?

According to Brown's paper, the bucket width is calculated in a clever way to maintain efficiency:

• bucket width is adjusted each time the queue is copied onto a new calendar (when resizing)

• events are sampled to find the average separation:

  • For small queues (≤ 5 events): sample all events

  • For larger queues: sample 5 + size/10 events (up to max of 25 samples)

  • Dequeue these events, record their priorities, then re-insert them

• there’s a two-step average calculation:

  • First calculate initial average separation between sampled events

  • Then recalculate average using only separations smaller than twice the initial average (this handles bimodal distributions by ignoring large gaps)

• and finally, the bucket width = 3.0 * final average

The reasoning behind this method is that we aim to have 2-3 events per bucket on average. Multiplier of 3.0 was found experimentally to give best performance. Ignoring large gaps helps handle skewed/bimodal distributions. Using a sample rather than all events makes the calculation efficient.

When does resizing occur?

According to Brown's paper, resizing occurs based on queue size thresholds, more specifically:

• Growing (Doubling):

  • Resize occurs when queue size > 2 * number_of_buckets

  • The new calendar gets twice as many buckets

  • Example: If current size is 50 buckets, resize at 101 events

• Shrinking (Halving):

  • Resize occurs when queue size < number_of_buckets/2

  • The new calendar gets half as many buckets

  • Example: If current size is 50 buckets, resize at 24 events

During the resizing process, a new calendar array is created, a new bucket width is calculated using the sampling method, all events are copied to their new bucket positions and finally the switch to the new calendar is done.

Brown analyzed the impact of resizing, the average event is copied less than 2 times during its lifetime. In the worst case, the queue size oscillates between just above 2^n and just below 2^(n-1), and even in worst case, only about 3 events are copied per hold operation.

The key innovation of Brown's Calendar Queue was simplifying the two-level approach by using fixed-size buckets instead of dynamic sublists, eliminating the overflow problem through circular wrapping and providing automatic adaptation through bucket resizing. This made the Calendar Queue both simpler to implement and more efficient in practice compared to the TL algorithm, especially for larger queue sizes and varying event distributions.

FELT - An optimisation to CQ

One of the disadvantages of Brown’s CQs is that resize operation would involve creating a new CQ structure and then moving each item from the old CQ to the new CQ before discarding the old CQ. Hence, such resizes can be costly if the size of the queue is very large. And this is what Hui’s and Thng’s FELT (Far Future Event Leaf Tree) [3] structure is trying to solve.

FELT uses two different structures together to manage events:

1. A regular calendar queue for near-future events (Primary Tier)

2. A special tree structure for far-future events (FELT) (Secondary Tier)

Primary Tier (Calendar Queue)

Works like a normal calendar queue with a fixed maximum size (typically 4096 events)

The FELT primary Calendar Queue works like a normal calendar queue with dual thresholds. The UPPBOUND threshold, typically set at 4096 events, represents the maximum capacity of the primary queue. When exceeded, the system initiates a transfer operation, moving the oldest events into the secondary tree structure. This transfer specifically moves UPPBOUND minus LOWBOUND events, maintaining optimal queue size for performance.

Conversely, the LOWBOUND threshold, typically set at 256 events, establishes the minimum required event count in the primary queue. When the queue size drops below LOWBOUND, the system initiates a replenishment operation. During replenishment, the system identifies the leaf node containing events with the earliest timestamps in the secondary tier, transfers its entire contents to the primary queue, and removes the depleted leaf node. This process ensures the primary queue maintains sufficient events for efficient operation.

Secondary Tier (Tree Structure)

FELT acts as a semi sorted binary tree, where the left branch always holds events with higher priority (earlier timestamps) than the right branch. Only the leaf nodes store events. These events are kept in an unsorted linked list called NodeList. Parent nodes act as branching points. They do not store events themselves but maintain information about the minimum timestamp (MinTimeStamp) of all events in the branches below them. This helps efficiently route new events to the correct leaf node during insertion.

FELT is size-based, meaning it manages events based on the number of events stored in the primary CQ. This contrasts with time-based approaches like the Future Event Tree (FET) used in the Dynamic Lazy Calendar Queue (DLCQ), which can lead to inefficiencies when the CQ resizes.

                   [Parent Node(PN): MinTimeStamp(t) = X]
                         /                 \
    [Parent Node: MinTimeStamp = Y]    [Parent Node: MinTimeStamp = Z]
             /   \                                 /  \
 [Leaf Node(LN): MinTimeStamp = A] ... ... [Leaf Node: MinTimeStamp = B]
               |                                    |
[Unsorted Linked List of Events(ULLE)]          [(ULLE)]

How are events inserted?

FELT processes each new event through a two-stage decision process for optimal placement. Initially, the system examines the event's timestamp to determine whether it belongs in the near-future or far-future category. This classification happens by comparing the event's timestamp against a threshold calculated as the current simulation time plus one calendar period.

For near-future events, the system directs them into the primary calendar queue using standard calendar queue insertion operations. However, if this insertion causes the primary queue to exceed its size threshold, the system automatically transfers the oldest events from the primary queue into the secondary tree structure to maintain optimal queue performance.

Far-future events take a different path, entering the secondary tree structure where they navigate through internal nodes based on timestamp comparisons. The event travels down the tree, with each internal node guiding it left or right until it reaches an appropriate leaf node. Once at the leaf node, the event is simply appended to an unsorted list maintained at that node:

1. Start at the Top Node: The insertion process begins at the topmost node of the FELT structure (TopNode).

2. Check if Current Node is a Leaf Node:

   1. If the current node is a leaf node: The event is inserted into the unsorted linked list (NodeList) at that leaf node.

   2. If the current node is a parent node: Proceed to step 3.

3. Compare Event Timestamp with Right Child's MinTimeStamp:

   1. If the event's timestamp is greater than or equal to the MinTimeStamp of the right child node, AND the right child node is not NULL: Move to the right child node and repeat step 2.

   2. Otherwise: Proceed to step 4.

4. Check Left Child:

   1. If the left child node is not NULL: Move to the left child node and repeat step 2.

   2. Otherwise: Insert the event into the NodeList of the current node (which must be a leaf node).

5. Check for Node Splitting: After the event is inserted, the algorithm checks if the leaf node has exceeded its maximum capacity (MaxSize). When a leaf node has reached its maximum capacity (MaxSize) of events that it can store, which is set a priori, it will spawn two new leaf nodes and split NodeList into 2 and transfer the events into the left and right leaf nodes.

   1. If the node needs splitting: The splitNode function is called to create two new child leaf nodes and redistribute the events based on their timestamps.

                  [Parent Node(PN): MinTimeStamp(t) = 10]
                    /                             \
               [PN:t = 5]                     [PN:t = 15]
                /   \                             /   \
       [LN:t = 2]   [LN:t = 7]          [LN:t = 12]   [LN:t = 17]
          |              |                  |           |
     [(ULLE)]         [(ULLE)]          [(ULLE)]      [(ULLE)]

New Event (Timestamp = 8)

1. Start at TopNode (MinTimeStamp = 10).

2. 8 < 15 (Right Child's MinTimeStamp), so move to Right Child (MinTimeStamp = 12).

3. Left Child is NULL, so insert event into current node.

Result:
                  [Parent Node(PN): MinTimeStamp(t) = 8]
                    /                             \
               [PN:t = 5]                     [PN:t = 8]
                /   \                             /   \
       [LN:t = 2]   [LN:t = 7]          [LN:t = 8]   [LN:t = 17]
          |              |                  |           |
     [(ULLE)]         [(ULLE)]          [(ULLE)]      [(ULLE)]
                                            |   
                                 [New Event (Timestamp = 8)]

When it comes to complexity, the tree traversal is O(log n) in best case when tree is balanced and O(n) when it’s a completely skewed tree.

While the theoretical worst case is O(n), the actual performance in practice tends toward O(1) because: the size-based splitting keeps the tree reasonably balanced. The large maxSize (3840) means splits are rare, no sorting is required within leaf nodes and the split cost is bounded by a constant.

The authors show empirically that FELT maintains near O(1) performance even for large queue sizes and skewed event distributions, performing better than other approaches like DLCQ.

The Snoopy CQ

The Snoopy Calendar Queue, proposed by Tan, Kah Leong, and Li-Jin Thng. "SNOOPy Calendar Queue." [4], builds upon the concepts of the conventional Calendar Queue (CQ) and the Dynamic Calendar Queue (DCQ), addressing their limitations, particularly in handling skewed event distributions. SNOOPy CQ extends the calendar queue with statistical optimization mechanisms. Instead of using sampling for bucket width calculation like traditional CQ, it tracks operational costs and uses these statistics to optimize queue parameters. The name "SNOOPy" stands for Statistically eNhanced with Optimum Operating Parameter Calendar Queue. The core of SNOOPy CQ lies in its ability to dynamically adjust its operating parameter – the bucket width – based on the queue's performance statistics. This adaptive mechanism ensures consistent and efficient performance, achieving near-O(1) complexity even in challenging scenarios.

                       +-----------------------+
                       |      SNOOPy CQ        |
                       +-----------------------+
                                 |  ^
                                 |  |
                                 v  |
                       +-----------------------+
                       |  Enqueue/Dequeue      |
                       |  Operations           |
                       +-----------------------+
                                  |  ^
                                  |  |
                                  v  |
                       +-----------------------+
                       | Performance Statistics|
                       |     (CE, CD)          |
                       +-----------------------+
                                  |  ^
                                  |  |
                                  v  |
      +---------------+-----------------------+----------------+
      |               |                       |                |
      v               v                       v                v
  +-------------+-------------------+--------------+--------------+
  |Trigger Check|  Bucket Width     |Calendar Resize|   Cost       |
  | (CE, CD,    |  Optimization     |  (BW, NB)     |  Calculation |
  | Thresholds) |(Minimize CE + CD) |               |  (CE, CD)    |
  +-------------+-------------------+---------------+--------------+

1. Enqueue/Dequeue Operations: These are the fundamental operations performed on the SNOOPy CQ. Events are enqueued (added to the queue) based on their timestamps, and the event with the earliest timestamp is dequeued (removed from the queue) for processing.

2. Performance Statistics Collection: As events are enqueued and dequeued, SNOOPy CQ meticulously tracks two crucial performance metrics:

   1. Average Enqueue Cost (CE): This represents the average number of events traversed before an insertion can be made in a bucket's linked list.

   2. Average Dequeue Cost (CD): This signifies the average number of buckets that need to be examined before finding the event with the earliest timestamp.

3. Trigger Check: SNOOPy CQ continuously monitors the collected performance statistics (CE and CD). It employs multiple trigger mechanisms:

   1. Inherited from CQ and DCQ: These triggers are based on thresholds for CE and CD (e.g., if either exceeds a value of 2 or 3), as in the CQ and DCQ structures.

   2. Unique to SNOOPy CQ: Additional triggers are based on the relative difference between CE and CD over multiple slots (time intervals corresponding to a certain number of enqueue or dequeue operations). If the difference exceeds a predefined factor, it signals a need for adjustment.

4. Bucket Width Optimization: When any of the trigger conditions are met, indicating a suboptimal bucket width, SNOOPy CQ initiates its bucket width optimization process. The goal is to find a new bucket width (BW) that minimizes the total cost (CE + CD), keeping the number of buckets (NB) constant. SNOOPy CQ employs a novel approach to determine the optimum BW based on the relationship between CE and CD, as detailed in section 3.1 of "download.pdf".

5. Calendar Resize: Once the optimal bucket width is calculated, the SNOOPy CQ undergoes a resize operation. A new calendar queue is created with the calculated BW and the same NB as before. Events from the old calendar are then transferred to their appropriate buckets in the new calendar.

6. Cost Calculation: After each slot (a series of enqueue or dequeue operations), the average enqueue cost (CE) and average dequeue cost (CD) are recalculated based on the accumulated statistics from that slot. These updated costs are then used in subsequent trigger checks and optimization processes.

Conclusion

Starting from the simple Unix V7 cron implementation, each subsequent event scheduling structure addressed specific limitations of its predecessors.

The Franta-Maly event list introduced sophisticated two-level indexing, achieving O(√n) complexity through its hierarchical structure. Brown's Calendar Queue (CQ) then simplified this approach with fixed-size buckets and circular wrapping, providing O(1) average performance for uniform distributions.

FELT built upon CQ's foundation by introducing a dual-tier architecture that efficiently manages near and far-future events, effectively addressing the resize overhead issues. Its size-based approach, rather than time-based, provides consistent performance across varying event distributions.

SNOOPy CQ represents the current state-of-the-art, employing statistical optimisation to dynamically adjust bucket widths based on operational costs. This adaptive approach ensures robust O(1) performance even under skewed distributions.

Data Structures Comparison

Franta-Maly Structure (1977)

Primary Components

• Employs a two-level structure based on indexed lists and the balanced terminal node structure of 3-2 trees.

• Divides the range of event times into intervals, bounded by "dummy keys (notices)" pointed to by an index list.

• Each interval further divided into sublists, marked by secondary keys associated with the right boundary dummy key.

Key Operations Performance

Insertion (O(√n) worst case):

• Binary search index array to find appropriate terminal node

• Linear search within terminal node for insertion position

• Split terminal node if size exceeds 2√n

• Update index pointers if split occurs

Deletion (O(√n) worst case):

• Search index array for target terminal node

• Remove event from terminal node

• Merge underfull terminal nodes if necessary

• Update index pointers after merge

Terminal Node Management:

• Split Threshold: 2√n events

• Merge Threshold: ½√n events

• Maintains roughly balanced terminal node sizes

Performance Bottlenecks

Index Array Search:

• Binary search cost increases with structure size

• More frequent searches in skewed distributions

Terminal Node Operations:

• Linear search within nodes

• Split/merge overhead with uneven distributions

Maintenance Overhead:

• Index updates after node modifications

• Balance maintenance across terminal nodes

Advantages, Limitations and Selection Criteria

Advantages:

1. Bounded Operations:

   1. Guaranteed O(√n) worst case

   2. Predictable performance characteristics

2. Memory Locality:

   1. Events clustered in terminal nodes

   2. Efficient cache utilisation for sequential access

3. Flexible Structure:

   1. Adapts to varying event distributions

   2. Self-balancing capabilities

Consider Using When:

1. Predictable Performance is critical:

   1. Operations must have guaranteed bounds

   2. Worst-case performance matters more than average case

2. Memory Access Patterns are important:

   1. System benefits from clustered event storage

   2. Cache efficiency is a priority

Limitations:

1. Complex Implementation:

   1. Significant code complexity for node management

   2. Error-prone split/merge operations

2. Performance Issues:

   1. High overhead for small event sets

   2. Degraded performance with skewed distributions

3. Maintenance Overhead:

   1. Frequent index updates

   2. Complex rebalancing operations

Avoid Using When:

1. Simple Implementation is preferred:

   1. Calendar queue or modern alternatives offer simpler solutions

   2. Development time is constrained

2. Optimal Average Performance is required:

   1. Calendar queue provides better average-case performance

   2. Event distribution is relatively uniform

Calendar Queue (Brown, 1988)

Primary Components

Based on the analogy of a desk calendar, with each "day" represented by a sorted linked list. An array of pointers provides access to each day's linked list.

Bucket Array:

• Fixed-size array of bucket heads

• Each bucket contains sorted linked list of events

• Array size maintained as power of 2 for efficient modulo operations

• Circular structure with wrap-around for year boundaries

Year Mechanism:

• Year = NB * BW (Number of buckets * Bucket width)

• Tracks current year for wrap-around handling

• Enables infinite future time handling with finite structure

Key Operations Performance

Enqueue (O(n) worst case, O(1) average case):

• Calculate virtual bucket using timestamp

• Map to actual bucket using modulo

• Insert in sorted order within bucket

• O(n) complexity when all events are in one bucket

Dequeue (O(n) worst case, O(1) average case):

• Start from lastBucket

• Remove earliest event from current bucket

• Advance to next bucket if empty

• Handle year wrap-around when needed

• O(n) when searching empty buckets

Resize (O(n)):

• Triggered when queue size crosses thresholds:

• Upsize: size > 2 * nbuckets

• Downsize: size < nbuckets/2

• Creates new bucket array with doubled/halved size

• Recalculates bucket width by sampling

• Redistributes all events

Performance Bottlenecks

Bucket Width Impact:

• Too small: Events spread across too many buckets

• Too large: Too many events per bucket

• Optimal: 2-3 events per bucket near dequeue point

Distribution Effects:

• Uniform: Achieves O(1) performance

• Skewed: Performance degrades

• Bimodal: May require frequent resizing

Advantages, Limitations and Selection Criteria

Advantages:

1. Uniform Distributions - for network simulation, regular event patterns, predictable timings etc

2. Medium-sized Queues - for 1,000 to 10,000 events, stable event counts, regular dequeue patterns

Limitations:

1. Skewed Distributions - lead to performance degradation, frequent resizing, bucket imbalance

2. Very Large Queues - lead to resize overhead, memory locality issues, width calculation challenges

FELT (Far Future Event Leaf Tree, 2002)

Primary Components

Uses a Primary Tier (Calendar Queue) and a Secondary Tier (Tree Structure).

Primary Tier (Calendar Queue):

• Works like a normal calendar queue with a fixed maximum size (typically 4096 events)

• Has both an upper limit (UPPBOUND) and lower limit (LOWBOUND) for number of events

• When too full, moves events to the secondary tier

• When too empty, gets events back from the secondary tier

Secondary Tier (Tree Structure):

• Uses a binary tree where only leaf nodes store events

• Leaf nodes contain unsorted lists of events

• Parent nodes just help route to the right leaf node

• Each node keeps track of the smallest timestamp in its subtree

Key Operations Performance

When Adding a New Event:

1. Decision Point: System first checks if the event is near-future or far-future

2. Near-Future Events:

   1. Go directly into the calendar queue

   2. If this makes the queue too full, oldest events are moved to the tree

3. Far-Future Events:

   1. Go into the tree structure

   2. Get routed down the tree to the appropriate leaf node

   3. Added to that leaf node's unsorted list

Near-future events (Primary Calendar Queue): O(1) average case, as it uses standard calendar queue insertion
Far-future events (Secondary Tree): O(log n) to traverse the tree to the correct leaf node, then O(1) to append to unsorted list
Overall: O(1) for near events, O(log n) for far events

When Retrieving Events:

1. Always take events from the calendar queue first

2. When calendar queue gets too empty:

   1. Find the leaf node with earliest events in the tree

   2. Move those events to the calendar queue

   3. Continue processing from calendar queue

3. Primary Calendar Queue: O(1) average case

   1. When primary queue needs replenishment:

      1. Finding minimum leaf node: O(1) using tree's cached minimum timestamps

      2. Transferring batch of events: O(k) where k is transfer batch size

Overall: Amortized O(1) since transfer costs are spread across many operations

Performance Bottlenecks

Transfer Operations Impact:
When the primary queue hits its UPPBOUND or LOWBOUND thresholds, the transfer operations between tiers can cause significant overhead. These transfers require moving large batches of events between structures and can temporarily pause normal operations. While the cost is amortized, it can cause noticeable performance spikes.

Leaf Node Management:
When leaf nodes in the secondary tier get too full and need splitting, the operation requires scanning all events in the node to find the average timestamp, then redistributing them. This can be expensive for large leaf nodes and may happen frequently with certain event patterns.

Threshold Tuning:
Choosing optimal values for UPPBOUND and LOWBOUND is critical but challenging. Poor choices can lead to either too frequent transfers or inefficient primary queue operation, significantly impacting performance.

These bottlenecks become particularly apparent in scenarios with:
- Highly skewed temporal distributions
- Memory-constrained environments
- Large numbers of far-future events

Advantages, Limitations and Selection Criteria

Optimal Use Cases

1. Large Event Sets (10,000 events)

   1. Wide timestamp distribution

   2. Frequent far-future events

2. Variable Distributions:

   1. Skewed timestamp patterns

   2. Mixed temporal localities

   3. Dynamic workload patterns

Limitations

1. Implementation Complexity:

   1. Complex maintenance logic

   2. Debugging challenges

   3. Higher development overhead

2. Resource Requirements:

   1. Additional memory overhead

   2. Complex pointer management

   3. Transfer operation costs

SNOOPy Calendar Queue (2000)

Primary Components

The structure maintains traditional calendar queue elements plus statistical tracking:

Calendar Base Structure:

• Array of buckets containing sorted event lists

• Each bucket represents a time interval

• Year size = number of buckets × bucket width

Statistical Enhancement Layer:

• CE (Enqueue Cost): Tracks average number of events traversed during insertion

• CD (Dequeue Cost): Tracks average number of empty buckets traversed during dequeuing

• Moving average window over multiple operations

• Performance metrics for bucket width optimisation

Key Operations Performance

Enqueue Process (O(1) average case):

• Calculates target bucket using timestamp

• Inserts event in sorted order within bucket

• Updates enqueue cost statistics based on traversal length

• Triggers resize if statistical thresholds exceeded

Dequeue Process (O(1) average case):

• Retrieves earliest event from current bucket

• Advances through empty buckets if necessary

• Updates dequeue cost statistics

• May trigger optimisation based on performance metrics

Resize Operation (O(n) when restructuring):

• Triggered by statistical cost metrics rather than just size

• Uses performance history to calculate optimal bucket width

• Redistributes events across new structure

• Resets statistical counters after resize

Performance Bottlenecks

Statistical Overhead:

• Continuous cost tracking adds per-operation overhead

• Moving average calculations require additional computation

• Statistical threshold checks on every operation

Resize Decisions:

• More complex than traditional CQ resize triggers

• May require more frequent resizes initially while optimising

• Statistical calculation overhead during resize operations

Dynamic Adaptation:

• Initial period of suboptimal performance while gathering statistics

• May oscillate between different configurations before settling

• Additional memory references for statistical tracking

Advantages, Limitations and Selection Criteria

Advantages:

• Robust Performance:

  • Better handling of skewed distributions than traditional CQ

  • Self-tuning capability reduces manual parameter tuning

  • More consistent performance across varying workloads

• Adaptive Optimisation:

  • Automatically adjusts to changing event patterns

  • Reduces impact of poor initial parameter choices

  • Maintains efficiency under varying conditions

• Statistical Insight:

  • Provides performance metrics for monitoring

  • Enables informed optimization decisions

  • Can predict and prevent performance degradation

Best Used When:

• Event distribution is unknown or varies

• Long-running simulations justify optimization overhead

• Performance consistency is critical

• Manual tuning is impractical

Limitations:

• Implementation Complexity:

  • More complex than basic calendar queue

  • Additional overhead for statistical tracking

  • More difficult to debug and maintain

Avoid When:

• Simple, known event distributions

• Short-running simulations

• Memory or CPU resources are very constrained

• Implementation simplicity is priority

Resources

[1] Franta, W. R., and Kurt Maly. "An Efficient Data Structure for the Simulation Event Set." Communications of the ACM 20, no. 8 (August 1977): 596-602.

[2] Brown, Randy. "Calendar Queues: A Fast O(1) Priority Queue Implementation for the Simulation Event Set Problem." Communications of the ACM 31, no. 10 (October 1988): 1220-1227.

[3] Hui, Timothy Chee-Kin, and Ian Li-Jin Thng. "FELT: A Far Future Event List Structure Optimized for Calendar Queues." SIMULATION: Transactions of The Society for Modeling and Simulation International 78, no. 6 (June 2002): 343-361.

[4] Tan, Kah Leong, and Li-Jin Thng. "SNOOPy Calendar Queue." In Proceedings of the 2000 Winter Simulation Conference, edited by J. A. Joines, R. R. Barton, K. Kang, and P. A. Fishwick, 487-495. IEEE, 2000.

As Uber evolved from a simple ride-hailing app into a super-app offering food delivery, grocery shopping, and even car rentals, each new service built its own payment system. The result? A labyrinth of 70 different payment endpoints, each speaking its own language.

This fragmentation created real-world headaches. A user could pay for an immediate ride with Apple Pay but couldn't use it for a scheduled pickup. Popular payment methods like PIX (Brazil) and UPI (India) worked inconsistently across services. For any new payment feature or compliance requirement that needed to be integrated, engineers would have to spend countless more hours maintaining this new duplicate code.

The cost wasn't just technical. When the European Union mandated Strong Customer Authentication for transactions, requiring features like fingerprinting and biometric verification, it became clear: Uber needed to revolutionize its payment infrastructure or risk losing hundreds of millions in revenue.

Unified Checkout - One Gateway to Rule Them All

What Uber’s engineering team created is a payment universal translator, which they called Unified Checkout. The way it works is that instead of each service (rides, eats, rentals) handling payments differently, everything flows through one complex system.

This payments orchestration layer while it helps to process millions of transactions and to maintain consistency across all services, it keeps individual services blissfully unaware of the underlying complexity.

The architecture revolves around a chain of microservices that handle everything from payment profile preparation to risk evaluation. When a user starts a transaction, the platform exchanges transactional data with third-party processors, generates authentication tokens, and coordinates with risk systems to determine the optimal payment strategy. The system is able to adapt its approach based on the risk algorithms, whether if it’s for an authorization hold or an upfront charge.

One notable capability of the Unified Checkout is that it provides two distinct integration paths.

• The modular approach allows services to embed pre-built components into their existing checkout flows, maintaining their unique user experience while leveraging the unified payment infrastructure.

• For newer or smaller services, the hosted solution provides a complete out-of-the-box payment experience, accelerating time-to-market without compromising on functionality.

Another key advantage is the "Checkout Actions" framework - a stateless system that orchestrates complex payment flows, from two-factor authentication to identity challenges, this framework handles every possible payment scenario while maintaining a consistent user experience.

By treating each Uber service as an external company, the platform maintains strict boundaries between payment logic and business operations. This architectural decision forced the team to build a solution that could handle any payment scenario while remaining service-agnostic, and hence building a good foundation for future payment innovations.

The Business Impact

The Unified Checkout delivered a remarkable 3% increase in checkout conversion rate and a 4.5% improvement in session recovery, which meant saving hundreds of millions in additional annual revenue. These results were shown in an experiment on UberEats, where the team compared user behaviour between the old and new systems, leading to rapid adoption across all services.

This transformation dramatically simplified the payment landscape at Uber. Adding another payment method like Apple Pay doesn’t require a full implementation anymore, but only some configuration changes. Moving away from 70 endpoints to one single unified system streamlined operations and changed on a fundamental level how Uber does payments. Duplicate development work vanished. The launch of new payment methods accelerated dramatically. A consistent user experience across services built stronger trust with customers. Payment method decisions shifted from technical limitations to pure strategy, allowing Uber to optimise its offerings based on market needs rather than implementation constraints.

Maintenance costs plummeted as teams no longer needed to maintain multiple implementations. Compliance became more straightforward, error rates dropped significantly, and customer support issues decreased. And probably most importantly, new services could now launch faster, equipped with a complete payment infrastructure from day one.

Lessons Learnt

It comes a time in a company’s scaling journey where maintaining multiple systems becomes more expensive than building a unified platform. For Uber, this moment came when they faced 70 different payment endpoints across their services, each requiring separate maintenance and updates.

Uber's experience demonstrates how external pressures, like EU regulations, can serve as catalysts for positive change. What began as a compliance requirement evolved into a platform that generated hundreds of millions in additional revenue.

Looking toward the future, the key is creating modular systems that can evolve with changing requirements, while maintaining a clear separation between business logic and payment processing. This separation allows for easier updates and maintenance as regulatory requirements change or new payment methods emerge.

The most valuable lesson from Uber's journey is recognizing that technical debt isn't always a burden – it can be a signal that your system is ready for platformization. When multiple teams struggle with similar problems, it might be time to step back and consider a unified approach.

The Uber article is available here:
Unified Checkout: Streamlining Uber’s Payment Ecosystem

Similar stories you might enjoy 📚

• How Stripe’s document databases supported 99.999% uptime with zero-downtime data migrations

• How PayPal Scaled Kafka to 1.3 Trillion Messages a Day

• How Monzo vertically scaled ordered consumption using Kafka

The dual-write problem is a critical issue that arises when updating multiple systems simultaneously. This challenge significantly impacts data consistency, system reliability, and overall performance in modern software architectures.

At its core, the dual-write problem stems from the difficulty of ensuring that all write operations across different systems either succeed or fail together. Traditional database transactions provide atomicity within a single database, but this guarantee doesn't extend to multiple systems. This limitation creates a risk of partial updates, which can lead to data inconsistencies and compromise the integrity of the overall system.

Failure scenarios further complicate the dual-write problem. System crashes, network failures, and partial outages can occur at any point during the update process, potentially leaving systems out of sync. For instance, if a failure occurs after writing to one system but before updating another, the result is inconsistent data across the environment.

To illustrate the dual-write problem, consider a common scenario in microservice architectures: updating a database and sending a notification message. Ideally, these operations should be atomic to ensure data consistency and reliability. However, the lack of a unified transaction mechanism across different systems makes achieving this atomicity challenging.

• If the database update is successful but the event notification fails, the downstream service will not be aware of the change, and the system can enter an inconsistent state.

• If the database update fails but the event notification is sent, data could get corrupted, which might affect the reliability of the system.

As organisations continue to adopt microservices and distributed architectures, addressing the dual-write problem becomes increasingly crucial.

The Transactional Outbox Pattern

One way to solve this is through the Transactional Outbox pattern. If we have a database that supports transactional updates, then we can use it to overcome the dual-write problem.

Let’s take the example of a food delivery company system. When a customer places an order, several things need to happen: 1. The order details need to be saved in the database 2. A notification needs to be sent to the restaurant 3. The customer should receive an order confirmation 4. The order should be made available for driver assignment and so on.

The Order Service receives the order request. Then it starts a database transaction. It inserts the order details into the Orders table and in the same transaction, it also inserts messages into the Outbox table. These messages might include: a) OrderCreated event b) NotifyRestaurant message c) SendCustomerConfirmation message. The database transaction is committed, ensuring both the order and the outbox messages are saved atomically.

The Publisher Service runs continuously or at frequent intervals. It queries the Outbox table for unpublished messages. When it finds messages, it reads them and publishes them to the Message Broker. After successful publication, it marks the messages as processed or deletes them from the Outbox table.

The Message Broker receives the published messages from the Publisher Service and routes these messages to the appropriate queues or topics.

Then we have all the Downstream Services. A Restaurant Service might subscribe to the "OrderCreated" event. When received, it notifies the restaurant about the new order. A Notification Service might pick up the "SendCustomerConfirmation" message and sends an order confirmation to the customer. A Driver Assignment Service might listen for "OrderCreated" events to begin the process of assigning a driver.

When to use:

• You need to ensure consistency between your database and a message broker.

• You want to implement reliable messaging in a microservices architecture.

• You need to guarantee that a message is sent if and only if a database transaction commits.

Pros:

• Ensures consistency between database and messaging system.

• Works well with existing relational databases.

• Simpler to implement than full event sourcing.

• Doesn't require changing your entire architecture.

Cons:

• Doesn't provide a full audit trail like event sourcing.

• Requires polling or additional mechanisms to publish messages.

• Can add complexity to your database schema.

Change Data Capture

Store the state record to a data store that supports CDC. Then the CDC infrastructure will notify external systems of the changes. One such example would be DynamoDB, using its DynamoDB Streams.

Coming back to the example earlier, in a CDC approach, the Order Service receives the order request, it processes the order and inserts the order details into the Orders table in the database. The CDC tool continuously monitors the database's transaction log. When it detects a change in the Orders table (in this case, a new order insertion), it captures this change. The CDC tool transforms the captured change into a structured event message and publishes the captured change event to a Message Broker/Streams. The event contains details about the new order, such as order ID, customer ID, restaurant ID, and other relevant information. And same as above, multiple Downstream Services can then subscribe to the Order change events from the Message Broker.

When to use:

• You want to track changes in your database for replication or integration purposes.

• You need to sync data between different systems in near real-time.

• You want to implement event-driven architectures without changing your existing systems.

• You need to create data lakes or data warehouses from operational databases.

Pros:

• Non-intrusive - can be implemented without changing existing applications.

• Captures all changes, including those made outside your application.

• Can be used with legacy systems.

• Supports real-time data integration.

Cons:

• May not capture the full business context of changes.

• Can be complex to set up and manage.

• May impact database performance. CDC can add some overhead to the database, as it needs to read the transaction logs.

• Might require specific database features or additional tools.

• When the Orders table schema changes, downstream consumers need to be prepared to handle both old and new event formats.

Event Sourcing

Event sourcing is a pattern for storing data or changes to data as events in an append-only log. This represents a business decision to accept the request and start processing it.

The idea of event sourcing is that the event log is the source of truth, and all other databases are projections of the event log. Any write must first be made to the event log. After this write succeeds, one or more event handlers consume this new event and writes it to the other databases. (Cloud Native Patterns by Cornelia Davis (Manning Publications, 2019))

In event sourcing, every change to an entity's state is captured as a discrete, fine-grained event. These events are not merely notifications, each event represents a specific, atomic alteration to the entity's state.

As these state-changing events occur, they are published to interested parties and simultaneously persisted in a chronological log. This log becomes the source of truth for all the changes that have occurred to the entity over time. It's akin to a detailed historical ledger, capturing every nuance of the entity's evolution.

The power of event sourcing lies in its ability to reconstruct an entity state. Subscribers to this event log can process the sequence of events to derive the current state of an entity at any given point in time. By "replaying" the events in order, these subscribers can build a complete and accurate picture of the entity's current condition.

Take the design of an order processing system for a food delivery platform as an example. Event sourcing simplifies this process. All these events (orderCreated, orderAcceptedByRestaurant, orderPreparing etc.) are stored in an append-only log. Each event contains details like timestamp, entity_id, event_type and relevant data. The current state of an order can be reconstructed by replaying all events for that order from the beginning.

When to use:

• You need a complete audit trail of all changes.

• You want to reconstruct past states of your system.

• Your domain model is complex with many state transitions.

• You need to support temporal queries (e.g., "What was the state at time X?").

• You want to enable easy implementation of new features based on historical data.

Pros:

• Provides a full history of changes.

• Enables easy debugging and auditing.

• Allows for event replay and system recovery.

• Supports complex domain modelling.

Cons:

• Can be complex to implement and maintain.

• May require more storage.

• Querying current state can be slower (mitigated by snapshots).

Listen to Yourself

The Listen to Yourself pattern is a variation of event-driven architecture that aims to solve the problem of data consistency between a service's database and its published events. The service publishes events to a message broker as part of its operations. The service then subscribes to its own events from the message broker. Upon receiving its own event, the service updates its database.

The Order Service receives the order request. It prepares the order data and instead of writing directly to the database, it publishes an "OrderCreated" event to the Message Broker. This event contains all necessary order details (order ID, customer ID, restaurant ID, items, total, etc.). The Message Broker receives the "OrderCreated" event from the Order Service and makes the event available to all subscribed services, including the Order Service itself. The Order Service subscribes to its own "OrderCreated" events. When it receives the event it just published: a) It writes the order details to the Orders table in the database. b) It updates its internal state if necessary. Multiple Downstream Services also subscribe to the "OrderCreated" events.

How it's different:

• Unlike the Transactional Outbox pattern, it doesn't require a separate outbox table in the database.

• It's simpler than full Event Sourcing but still provides a level of eventual consistency.

• It differs from CDC in that the service actively publishes events, rather than relying on database log parsing.

Pros:

• Simplifies the architecture by using the message broker as the source of truth.

• Ensures eventual consistency between the published events and the database state.

• Can be easier to implement than some other patterns.

Cons:

• There's a potential for temporary inconsistency between the event being published and the database being updated.

• It may introduce additional latency due to the round-trip through the message broker.

• It can be more difficult to debug, as the flow of data is less straightforward.

Conclusion

We had a look at the different solutions that can be used to solve the dual-write problem. Which approach is used varies based on a given use case, specific requirements or any constraints of the system being designed or modified. To wrap things up, below is a comparison that might help you in making your decision:

Data Consistency:

• Event Sourcing: High consistency, events are the source of truth

• Transactional Outbox: Strong consistency between database and messaging

• Listen to Yourself: Eventual consistency, with potential for brief inconsistencies

• CDC: Eventual consistency, with potential for small delays

Complexity of Implementation:

• Event Sourcing: High - requires significant architectural changes

• Transactional Outbox: Moderate - requires additional table and message relay process

• Listen to Yourself: Low to Moderate - requires event publishing and self-subscription

• CDC: Moderate - requires CDC tool setup and configuration

Performance Impact:

• Event Sourcing: Can be high for read operations, depending on implementation

• Transactional Outbox: Low impact on write operations, additional load for message relay

• Listen to Yourself: Moderate - additional network round-trip for each operation

• CDC: Low to Moderate - some overhead on database for log reading

Scalability:

• Event Sourcing: Highly scalable, especially for write operations

• Transactional Outbox: Scalable, with separate scaling for application and message relay

• Listen to Yourself: Scalable, but may require careful management of self-subscriptions

• CDC: Highly scalable, especially for high-volume changes

Auditability:

• Event Sourcing: Excellent - full history of all changes

• Transactional Outbox: Good - outbox can serve as an audit log

• Listen to Yourself: Good - events can serve as an audit log

• CDC: Excellent - captures all database changes

Ability to Reconstruct Past States:

• Event Sourcing: Excellent - core feature of the pattern

• Transactional Outbox: Limited - not a primary feature

• Listen to Yourself: Possible but not straightforward

• CDC: Possible if all changes are retained, but not a primary feature

Impact on Existing Systems:

• Event Sourcing: High - requires fundamental architectural changes

• Transactional Outbox: Moderate - requires changes to data access layer

• Listen to Yourself: Moderate - requires changes to service logic

• CDC: Low - can often be implemented without changing existing applications

Handling of Schema Changes:

• Event Sourcing: Challenging - requires careful event versioning

• Transactional Outbox: Moderate - affects outbox table and consumers

• Listen to Yourself: Moderate - affects event structure and consumers

• CDC: Challenging - changes in source database schema need to be carefully managed

Real-time Data Integration:

• Event Sourcing: Excellent - events can be processed in real-time

• Transactional Outbox: Good - near real-time, depending on relay frequency

• Listen to Yourself: Excellent - events are processed immediately

• CDC: Very Good - near real-time, with minimal delay

Resilience to System Failures:

• Event Sourcing: High - can rebuild state from event log

• Transactional Outbox: Good - messages persist in outbox until processed

• Listen to Yourself: Moderate - depends on message broker reliability

• CDC: Good - can resume from last processed transaction log position

Latency:

• Event Sourcing: Can be high for complex queries on current state

• Transactional Outbox: Low for writes, potential delay in message processing

• Listen to Yourself: Moderate - additional network round-trip

• CDC: Low to Moderate - depends on CDC tool and configuration

In 2020, with the constant increase in their user base, the team decided to move from a Django monolith to a microservice architecture. This allowed for better scalability options, shorter waits for tests completion, faster deployment times and increased developer velocity. They wrote a great blog post on how they managed the transition. But this change also brought a lot of complexity with it.

The new architecture introduced other types of issues which we’re going to talk about in this article. We’re going to have a look at some of the common pitfalls and anti-patterns that appear in a microservice architecture, how Doordash solved them at a local level and how they’re attempting to mitigate them at a global level.

Common pitfalls with microservice architectures

1. Cascading Failure - A cascading failure happens when the failure of one service leads to the failure of other dependent services. This can cause a chain reaction, potentially bringing down the entire system.

DoorDash had an outage of this kind that they talked about in this blog post. In their case, the chain of failure started from a seemingly innocuous database maintenance, which increased the database latency. The latency then bubbled up to the upstream services, causing errors from timeouts and resource exhaustion. The increased error rates triggered a misconfigured circuit breaker, which stopped traffic between a lot of unrelated services, resulting in an outage with a wide blast radius.

Why it happens:

• Tight Coupling: Services are too dependent on each other, leading to a domino effect.

• Lack of Isolation: Failures in one service are not contained and propagate to others.

• Resource Exhaustion: Failure in a critical service can lead to resource exhaustion (e.g., CPU, memory) in dependent services.

2. Retry Storm - A retry storm occurs when a service failure triggers multiple retries from dependent services, which can overwhelm the failing service even more. Retries can worsen the issue when the downstream services are unavailable or slow, leading to work amplification, as each failed request is retried multiple times, which can cause an already degraded service to deteriorate further.

Why it happens:

• Uncontrolled Retries: Services automatically retry failed requests without considering the state of the failing service.

• Lack of Backoff: Retries happen too frequently, without appropriate delay, exacerbating the problem.

3. Death Spiral - A death spiral happens when the system starts to fail under load, and the attempts to handle the failure (like retries or additional resource allocation) further degrade the system’s performance, leading to a vicious cycle of deteriorating performance.

We’ve seen earlier how issues can spread vertically through different services dependant on one another. But they can also spread horizontally, inside the service cluster, from one node to another.

Why this happens:

• Resource Contention: Excessive retries and fallback operations consume more resources, reducing availability for normal operations.

• Unbalanced Load: Efforts to handle failures, like adding more instances, might lead to other parts of the system becoming overloaded.

4. Metastable Failure - Metastable failure refers to a state where the system appears to be stable under low load but becomes unstable and fails when the load increases beyond a certain threshold. Metastable failures occur in open systems with an uncontrolled source of load where a trigger causes the system to enter a bad state that persists even when the trigger is removed. When any of several potential triggers cause the system to enter a metastable state, a feedback loop sustains the failure, keeping the system in this state until a significant (usually manual) corrective action is applied.

For example, an initial trigger such as a surge in users, might cause one of the backend services to load shed and start responding to certain requests with 429 (rate limit). The callers will then retry their calls, but these retries, combined with requests from new users, overwhelm the backend service even more, leading to further load shedding. This results in a positive feedback loop where calls are continuously retried (along with new calls), get rate limited, and then retried again, perpetuating the cycle.

The above is called a Thundering Herd problem and is one example of a Metastable failure.

Why this happens:

• Insufficient Capacity: The system is not designed to handle peak loads, leading to instability.

• Hidden Bottlenecks: Bottlenecks that are not apparent under normal conditions become critical under high load.

• Latency Sensitivity: Small increases in latency can have a disproportionate effect on the system's stability.

• Request Retries: Retrying failed requests is widely used to mask transient issues. However, it also results in work amplification, which can lead to additional failures

Solutions at a local level

There are a couple of known techniques which are used to solve the above, some of which are being used by DoorDash as well:

• Exponential Backoff - by gradually increasing the delay between retries to reduce the load

• Retry Limits - by setting a maximum limit on the number of retries to prevent endless retry loops.

• Circuit Breakers - by activating and reducing the number of calls a dependent service makes to another one. Some examples of how circuit breakers are used by other companies:

  • Netflix's Hystrix library: Netflix developed Hystrix as a library to manage failures within their distributed system. When Hystrix identifies that a remote service is down or unresponsive, it activates the circuit breaker, stopping further requests from being sent to the problematic service. It can also offer alternative responses or retry the request after a designated period.

  • AWS ECS: ECS employs a circuit breaker pattern to automatically isolate failing services, thus preventing cascading failures within applications. This pattern also reduces latency and resource consumption during recovery from a service failure. It effectively ensures high availability and reliability for containerized applications.

  • SoundCloud: SoundCloud also uses circuit breaker to handle failures in their distributed architecture.

  

• Bulkheads - by isolating critical resources (like thread pools, database connections, or service instances) ensuring that failures or high resource usage in one microservice do not impact others sharing the same resources.

• Timeouts & Fallbacks - by defining the maximum acceptable duration for a microservice to respond to a request and preventing indefinite waiting and and resource consumption when a service is slow or unresponsive.

• Load Shedding - by prioritising and limiting incoming requests to prevent overload.

• Graceful Degradation - by minimising the amount of work that needs to be done, degrading non-essential functionalities or reduce service levels during times of high load, resource scarcity, or service unavailability. This can be achieved by determining essential functionalities that must remain operational even under degraded conditions, simplifying non-critical features to prioritise core functionalities and implementing fallback strategies to maintain basic service levels when primary functionalities are unavailable.

• Capacity Planning - by analysing historical data, user patterns, and trends to forecast future demand for each microservice and determining the scalability requirements based on expected growth, seasonal fluctuations, and special events.

Shortcomings of the local countermeasures

The current localised mechanisms face similar limitations:

• They rely on metrics specific to the individual service to determine its health. However, many types of failures involve interactions across multiple components, requiring a comprehensive system-wide perspective to effectively address overload conditions.

• They use general metrics and heuristics to assess the system health, which may lack precision. For instance, high latency alone may not indicate service overload; it could stem from slow downstream services.

• Their corrective actions are constrained. Operating within the local service, these mechanisms can only take local actions which may not be optimal for restoring the system health, as the root cause of the issue might lie elsewhere.

Solutions at a global level

One limitation of load shedding, circuit breakers, and graceful degradation is their narrow perspective within the system. These tools assess factors like their own resource usage, immediate dependencies, and incoming request volume. However, they lack the ability to adopt a global view of the entire system and decide based on that.

Aperture, an open-source system for reliability management, goes beyond local solutions by implementing a centralised load control. It provides a unified system for managing loads across multiple services during outages. It has 3 main components:

• Observe: Aperture gathers reliability metrics from each node and consolidates them using Prometheus.

• Analyse: A standalone Aperture controller continuously monitors these metrics and detects deviations from Service Level Objectives (SLOs).

• Actuate: Upon detecting anomalies, the Aperture controller triggers policies tailored to observed patterns and applies actions on each node, such as load shedding or distributed rate limiting.

Aperture utilizes YAML-based policies that guides its actions during system disruptions. When an alert is triggered, Aperture automatically executes actions based on these configured policies. Some of the actions it offers include distributed rate limiting and concurrency control (also known as load shedding). By maintaining a centralized oversight and control of the entire system, Aperture enables various strategies to mitigate outages. For instance, it can be configured to implement a policy that throttles traffic to an upstream service when a downstream service is overwhelmed, preventing excessive requests from reaching the problematic area and thereby enhancing system responsiveness and cost-efficiency.

DoorDash deployed Aperture within one of their core services and conducted artificial request simulations to evaluate its performance. They discovered that Aperture effectively operated as a robust and user-friendly global rate limiter and load shedding solution, providing a concurrency limiting algorithm which minimises the impact of unexpected load or latency.

The Aperture blog also provides good use cases of how their solution can be used to solve production problems:

• Protecting PostgreSQL with Adaptive Rate Limiting

• Balancing Cost and Efficiency in Mistral with Concurrency Scheduling

• Squeezing Water from Stone: Managing OpenAI Rate Limits with Request Prioritization

Conclusion

We had a look into the different pitfalls that arise in distributed systems, what are the triggers that might cause them, what are some of the localised solutions to prevent them from happening and what can be done at a more global level.

In the next articles, we’re going to take a deeper look into how Aperture works, how we can configure it and how to define a set of policies for a given use case.

So here I am, starting a new series of articles diving into Coding Theory. From Hamming distance, parity checks, linear codes, Reed-Solomon codes to BCH and maybe beyond? We’ll see how long my curiosity lasts.

This article will talk about:

• what is noise, error detection and error correction

• definitions - hamming distance, ball, volume, code dimension, rate, weight and hamming bound

And at the end of it, you should be able to:

• find and use the Hamming distance for a code

• understand the trade-off between rate and distance

The Goal of Coding Theory

One of the main goals of coding theory is developing methods to detect/correct errors caused by the transmission of information through noisy channels.

message m of length k → (encoding) codeword c  → (channel) received word c’ →  (decoding) message m’ of length n

Let’s say we have a message m of length k which we are going to encode (somehow) and get a word c of length n. Here n > k as we are adding redundancy data to our original message. We transmit the encoded word c through some communication channel but some of the bits get corrupted and so we receive a corrupted codeword c~ of length n.

We are concerned about 4 things:

1. Handling the communication noise.

2. Recovering the initial message.

3. Minimising overhead, meaning k/n should be large.

4. Being able to do the above through an encoding method that is efficient.

Binary bit-error correcting codes

To start off, there are a couple of assumptions we’ll make. All messages that we want to send are binary strings and we will only be transmitting binary strings across the channel and the only errors that occur are bit-errors, which consist of replacing 0 by 1 or 1 by 0.

Let’s start by providing the formal definition of what a codeword c is.

Example 1 An example would be the following where C is a matrix of vectors of length 4 over an alphabet {0,1}. C is a code which has 8 codewords, each of length 4 and all digits are either 0 or 1. Such codes are called binary codes.

Example 2

Suppose we have the following code (also known as the Hamming Code)

Then C is a binary code of blocklength 7. There is a nice way to visualise the bove encoding using 3 circles. Our message bits will be at the intersection of any 2 circles and all the other 3 parity bits ((x2,x3,x4)%2, etc) correspond to the remaining empty cells. You can also notice that each circle sums up to 0mod2.

The Hamming Code is capable of detecting up to two simultaneous bit errors and correcting single-bit errors. Let’s say we get the following corrupted code word c~={0111010}. Are we able to figure out c? One way to check which bit got flipped is to use the same circle representation above and use the bits that we got:

We know that the sum of the bits in each circle should sum up to 0mod%2 but the yellow and blue one don’t, which means the bit at their intersection (aka the 3rd bit) has been corrupted. So now we know the corrected codeword is c={0101010}

Let’s continue with some more formal definitions. The Hamming Distance between two strings is the number of positions in which they differ.

For example Δ(001, 100) = 2.

The Relative Hamming Distance and the Minimum Distance are defined as:

The distance of a code is the minimum Hamming distance between two distinct codewords in the code.

For example, the distance of {001, 010, 100, 111} is 2.

Another definition of the volume of a Hamming Ball can be defined as:

Example Given the following code C = {001,101,100,111} where first 2 bits are information bits, the Hamming Ball can be calculated as follows:

• Calculate the Hamming Distance: For each codeword, calculate the Hamming distance to all other codewords within the specified radius. Let's say we want to calculate the Hamming ball of radius 1.

  • For codeword 001: Hamming distance of 1 to 010 and 100.

  • For codeword 010: Hamming distance of 1 to 001 and 111.

  • For codeword 100: Hamming distance of 1 to 001 and 111.

  • For codeword 111: Hamming distance of 1 to 010 and 100.

• Count the Codewords within the Radius: Count the number of codewords that fall within the specified radius from each codeword.

  • For codeword 001: 2 codewords (010, 100) within radius 1.

  • For codeword 010: 2 codewords (001, 111) within radius 1.

  • For codeword 100: 2 codewords (001, 111) within radius 1.

  • For codeword 111: 2 codewords (010, 100) within radius 1.

• Calculate the Volume: The volume of the Hamming ball is the sum of the counts of codewords within the radius from each codeword.

  • Volume = 2 (from 001) + 2 (from 010) + 2 (from 100) + 2 (from 111) = 8.

Detecting and Correcting Errors

The error-correcting capability of a code refers to its ability to detect and correct errors that occur during transmission. A code with a larger minimum distance can correct more errors.

Detecting errors involves finding out if an error has occurred.

We say a code can detect d errors if the following holds:
Whenever a codeword is transmitted across a channel capable of producing d bit-errors, it is possible to deduce whether an error occurred using only the received word.

If the distance of a code is d, then the code can detect exactly d−1 errors.

Proof If a channel can produce at most d−1 bit-errors, it cannot turn one codeword into another by flipping bits, because the minimum Hamming distance between any two codewords is at least d. Thus, if the received word c’ is not in the code, we can deduce that some errors occurred. The code is not guaranteed to detect d errors, because one codeword can be transformed into another by d bit-errors.

We say a code can correct d errors if the following holds:
Whenever a codeword is transmitted across a channel capable of producing d bit-errors, one can deduce which codeword was sent using only the received word.

If the distance of a code is d, then the code can correct exactly ⌊(d − 1)/2⌋ errors.

Proof Suppose C is a code of distance d, and a codeword c is sent across a channel capable of producing at most ⌊(d − 1)/2⌋ bit-errors. Consider we have received the word c’ that is the result of transmitting the codeword c and introducing some errors. We claim that c is the only codeword within distance ⌊(d − 1)/2⌋ of c’. If there were another codeword, say c’’, then it would mean Δ(c,c’’) ≤ 2⌊(d − 1)/2⌋ ≤ d-1 (which is a contradiction based on the properties of non-overlapping neighborhoods)

Example The following code C = {001,101,100,111} has distance 2 which means it can detect 1 error, but it can correct 0 errors.

A code with minimum Hamming distance d between its codewords, can:

• detect at most d-1 errors

• correct at most ⌊(d − 1)/2⌋ errors

How much information can we send: the Rate

The rate of a code is defined as the ratio of the number of information bits to the total number of bits in a codeword, which is a value between [0,1]. The closer to 1 the better.

Given the following code C={000,101,011,110} where let’s say we use an encoding where the 2 bits are the information bits. This means our rate is:
R = number of information bits/length of the codeword = 2/3

In coding theory, higher-rate codes are desirable because they allow more information to be transmitted per unit of time or bandwidth. However, increasing the rate often comes at the cost of reduced error-correction capability, as higher-rate codes typically have shorter minimum distances. Therefore, there is often a trade-off between rate and error-correction performance in code design.

Rate and Distance Trade-off: the Hamming Bound

How does rate and distance influence each other?

• Higher Rate, Lower Distance: Increasing the rate of a code often comes at the expense of reducing its distance. This is because increasing the rate typically involves reducing redundancy in the codewords, which means fewer bits are available for error detection and correction. As a result, the distance of the code decreases, making it less capable of correcting errors.

• Lower Rate, Higher Distance: Conversely, decreasing the rate of a code allows for more redundancy to be added to the codewords, which increases the distance of the code. This provides better error-correcting capability but at the cost of transmitting fewer information bits per codeword.

Increasing the rate improves efficiency but may reduce error-correcting capability, while increasing the distance enhances error-correcting capability but may reduce efficiency.

So what is the best trade-off between rate and distance? The Hamming Bound establishes some limitations of how good this trade off can be.

Let’s try and visualise this. Let’s say we have our alphabet ∑ with codewords c in code C, where C has distance d. All of the codewords neighborhoods are non-overlapping since each ball has radius ⌊(d − 1)/2⌋ and the distance between any 2 codewords is d.

From this we can get an idea of what the Hamming Bound is all about. We have those |C| disjoint balls of radius ⌊(d − 1)/2⌋but all of those balls need to fit in ∑^n, so we can’t have too many balls. Therefore, we can’t have too many codewords and the rate of the code can’t be too big. All those disjoint Hamming Balls need to live in ∑^n

Using the definition for the volume of the Hamming Ball, we can define the following:

The Hamming Bound gives us a bound in terms of the distance and rate and is important in coding theory as it places an upper limit on the number of distinct codewords that can be encoded using a block of symbols of particular length, to provide a specified error-correcting capability.

Conclusion

As this article is getting quite long and there are still a lot of things to talk about we’ll stop here. Now that we have most of the definitions and theorems defined, next article will cover more practical examples on how to calculate the Hamming Bound for different codes. We’ll also discuss what are generator and parity matrixes and how to decode corrupted codewords.

Resources:

• Huge appreciation to Mary Wootters’ for her free Algebraic Coding Theory YT Lessons

System Design

1. SYSTEM DESIGN COMPONENTS

   1. CACHING

      1. Mastering Cache Management: Eviction, Invalidation, and Optimal Data Access Strategies

      2. Least Frequently Used (LFU) Cache - Design and Implementation

2. PATTERNS OF DISTRIBUTED TRANSACTIONS

   1. Understanding the Dual-Write Challenge in Multi-System Environments

3. MICROSERVICES

   1. Refactoring legacy code: The Strangler Fig Migration Pattern

4. REAL WORLD SCALING STORIES

   1. Common Microservice Failures and How DoorDash Mitigates Them

   2. Payments Systems at Scale: Lessons from Uber's Unified Platform

5. UNDER THE HOOD - HOW EXISTING SYSTEMS WORK

   1. From Pixels to Information: A Comprehensive Guide to QR Codes

      1. Coding Theory: Hamming Distance, Code, and Bound

   2. Advanced Data Structures for Discrete Event Simulation

   3. Decentralised Social Media Explained: Inside Bluesky's Federated Architecture

6. OPTIMISATION TECHNIQUES

   1. Optimising Performance: The Cool Magic of Bloom Filters

7. MY LEARNINGS

   1. Building an Audit Logging Solution: Production Scars and Evolutionary Architectures

Engineering Interviews

1. INTERVIEWS’ HOW-TO’S

   1. Cracking the Product Interview for Engineers

   2. Cracking the System Design Interview

   3. Red Flags to Look for in Software Engineering Interviews

2. DATA STRUCTURES & ALGORITHMS

   1. LFU Cache (leetcode 460)

   2. Bloom Filter Data Structure

Golang in Depth

1. MEMORY MANAGEMENT

   1. Breaking Free: Navigating Golang's Memory Management

2. CONCURRENCY

   1. Cracking the Go Concurrency Basics - Part 1

   2. Beyond Goroutines: Exploring Advanced Concurrency Patterns in Go - Part 2

3. GOTCHAS

   1. Unbreaking the Loop: Go 1.22 fixes for loops

Coding Theory

1. Coding Theory: Hamming Distance, Code, and Bound

Career Growth/Leadership

1. BUILDING TEAMS

   1. The Bear: How a show about a restaurant provides the best leadership takeaways

2. MENTORSHIP

   1. Mentorship That Matters: Building Effective Knowledge-Sharing Systems

Book Reviews

1. Book Review: "The Worlds I See: Curiosity, Exploration, and Discovery at the Dawn of AI", by Dr. Fei-Fei Li

AI/ML

1. Crafting Conversations: Techniques and Tips for Effective Prompting

Non-Tech

1. The 'Lazy-to-Crazy' Runner's Guide to Marathon Glory

How and why QR codes came into being

Masahiro Hara, a Japanese engineer who worked at Denso Wave then, invented the QR Code (or Quick Response codes) in 1994 to meet the need for a more efficient and versatile method of tracking automotive parts during manufacturing. Hara and his team developed QR codes as an improvement over traditional barcodes.

QR Codes vs Barcodes

Traditional barcodes could only store a limited amount of information horizontally, while QR codes can store much more data both horizontally and vertically. A barcode can store 1,556 bytes, whereas a QR Code can store up to 7,089 numeric characters, 4,296 alphanumeric characters, 2,953 bytes, and 1,817 kanji characters.

Components

And now let’s get into the fun part, what do all those black and white squares represent? QR Codes consist of black and white elements called modules, where black represents 1 and white 0. These modules can be combined into meaningful binary messages.

Before diving into how we encode/decode the information, let’s first analyse what the main components of the QR code are:

Function Patterns

1. Quiet Zone - helps the scanner to distinguish between the QR code and its surroundings, making it easier to detect the QR Code.

2. Position Pattern - located in 3 of the QR code’s corners, it allows the scanner to quickly recognise the code orientation. This means the code can be scanned at any angle.

3. Separator - used to keep space between the position pattern and the other information, helping the scanner to identify the position pattern.

   

4. Alignment Pattern - used for correcting distortion by straightening the code on a curved surface. On higher versions, QR codes tend to have more alignment patterns. Ver40 which is the largest QR code consists of 46 alignment patterns. Ver1 doesn’t require any alignment patterns.

5. Timing Pattern - helps the scanner to determine the width of a single module.

Encoding Region

1. Version Information - there are 40 versions of QR codes, from 1 up to 40; this region specifies which version is being used.

2. Format Information - contains information such as error correction level, mask pattern etc.

3. Data and error correction keys - the remaining available code is used for storing the data along with the error correction data. The correction data is done by the Reed-Solomon codes which allows the QR code to still be scanned if some information is corrupted/not visible etc.

Putting everything together:

How does versioning work?

There are 40 versions of the conventional QR code. Each version has a different number of data modules, commencing with Version 1 (21 × 21 modules) up to Version 40 (177 × 177 modules).

With a new higher version, another 4 additional modules per side are added.

How would we know which version we need to use?

QRCode.com provides a comprehensive list of versions given the type of input data you want to use (numeric, alphanumeric, binary, kanji) and the correction level.

Let’s say I want to create a QR Code for the text “The Crafty Dev“ which means we’re only interested in a 14 letter alphanumeric input. And let’s suppose we’re going to stick the QR code in various places outside, so we need a pretty high error correction capability (in case it rains or it gets stained etc). We need to find a figure in the QRCode versioning table, that is 14 or over and with an H correction level, so a good version for us to use would be version 2.

The Error Correction Level

The QR code error correction is based on the Reed-Solomon algorithm which works in a similar way to the parity error correction - it adds extra bits to the data so that errors can be corrected.

Fun Fact

The Voyager spacecraft uses the Reed-Solomon error correction to ensure that data transmitted from the outer reaches of the solar system can be reconstructed even if it encounters interference, which is highly probable. Given that signals from Voyager 1 can take up to 20 hours to reach Earth, employing an ARQ (Automatic Repeat reQuest) mechanism to request retransmission in case of errors would be highly impractical. (source)

The way Reed-Solomon does encoding deserves a completely different article so I’ll link it here once it’s published, so stay tuned.

Unmasking

Now comes the crazy part. Let’s say we have a QR code and we want to manually decode it. Where do we start? The first thing we’ll need to look at is the Format Information section.

The first 2 modules (colored red in the picture above) will give you the error correction level. And the next 3 (colored green) will give you the mask pattern.

This means we have a 00 error correction level, which means high, 30% of data bytes can be restored; and a 111 masking pattern.

The QR codes are usually masked, so we’ll have to take off the mask first. We put the coordinates of the current bit into the formula, and if the result is 0, we use the opposite bit of that. After removing the mask on the original QR code, we’ll get the following unmasked QR code:

Decoding

Now, finally getting into the decoding part. The way a QR code is read is from bottom right to top left following the path shown below. The first 4 bits give us the encoding and the next 8 the length of the message.

This means, for our QR code, we would have a 0100 encoding, which is a byte encoding and a 00001110 message length which is 14. Byte encoding tells us we’re expecting blocks of 8 bits and that we're expecting 14 of them.

We’re going to continue splitting our QR code into blocks of 8 bits and following the pattern of interleaved blocks below, we should have something similar to the right picture below:

We can see we have indeed 14 blocks. The last block containing the 4 bits of white modules is the End of message (Terminator) representing the end of our message. Now that we have our blocks splitted we can start decoding each character. We do this by summing up all the turned on (black) bits and converting the decimal number obtained into its specific character using an ASCII Table.

This will give us the following:

64+16+4 = 84 representing T
64+32+8 = 104 representing h
64+32+4+1 = 101 representing e
32 representing " "
64+2+1 = 67 representing C
64+32+14+4 = 114 representing r
64+32+1 = 97 representing a
64+32+4+2 = 102 representing f
64+32+16+4 = 116 representing t
64+32+16+8+1 = 121 representing y
32 representing " "
64+4 = 68 representing D
64+32+4+1 = 101 representing e
64+32+16+4+2 = representing v

We finally decoded our QR code which says “The Crafty Dev“ 🙌 🙌 🙌 .

The rest of the QR code message is part of the error correction data which we’ll have a look at in the next article, where we’ll talk about how QR Codes do error correction while we’ll also dive into how the Reed-Solomon algorithm works.

Conclusion

In this article we had a general look at QR Codes, their main components, what unmasking is and how it works, and finally how to decode the QR Code message.

Stay tuned, as in the upcoming articles we'll dive deeper into:

1. On Error Correction: From Hamming to Reed-Solomon

2. QR Code Models

See you soon 👋

The basic principle of LFU caching is to remove the least frequently used items first when the cache reaches its capacity limit. This means that the cache keeps track of the frequency with which each item is accessed, and when it needs to make space for a new item, it removes the item that has been accessed the least frequently.

LFU Cache Design

To implement a Least Frequently Used (LFU) cache solution we need to design a data structure that would support the following operations:

• LFUCache(int capacity) - initialises the object with the capacity of the data structure.

• int get(int key) - gets the value of the key if the key exists in the cache. Otherwise, returns -1.

• void put(int key, int value) - updates the value of the key if present, or inserts the key if not already present. When the cache reaches its capacity, it should invalidate and remove the least frequently used key before inserting a new item. For this problem, when there is a tie (i.e., two or more keys with the same frequency), the least recently used key would be invalidated.

The frequency for a key is incremented whenever a get or a put operation is called on the element.

Let’s run through an example to better understand how this would work:

1. Let’s say we have a cache with a capacity of 2 LFU(2) and we call put(1,1)
→ Our LFU will become {1,1,1} as we add the key 1 with value 1 which currently has a frequency of 1

2. Then we call put(2,2) → Our LFU will become {2,2,1},{1,1,1}. We added a new element with key 2 and value 2 which currently has a frequency of 1. Since elements with key 1 and 2 have the same frequency of 1, we add the most recently used element (the one with key 2) to the front of the list.

3. Then we call get(1). Our LFU will return 1 and the new LFU will be {1,1,2}, {2,2,1}. This is because we increase the frequency of element with key 2 to 2 (due to the read operation) and we move it to the front of the list because it has higher frequency than the other element.

4. Then we call put(3, 3). Our LFU will become {1,1,2}, {3,3,1}. A new element comes in but our cache is at full capacity so we need to remove an element. We remove from the end of the list (as this is where the element with the least frequency and least recency is stored), so {2,2,1} gets removed. We then add the new element with a frequency of 1, {3,3,1}.

5. We call get(2). We return -1 as the element no longer exists in the cache.

6. We call get(3). We return 3 and the LFU becomes {3,3,2}, {1,1,2}. We increase the frequency of 3 and move it before key 1 as even though they have the same frequency, key 3 has been accessed more recently.

7. We call put(4,4). The LFU becomes {3,3,2}, {4,4,1}. New element comes in, cache is full, we remove the last element with key 1 and add the new element with key 4.

8. We call get(1). We return -1 as element is no longer present in the cache.

9. We call get(3). We return 3 and our LFU becomes {3,3,3}, {4,4,1}. The read operation increases the frequency of element at key 3 to 3.

10. We call get(4). We return 4 and our LFU becomes {3,3,3}, {4,4,2}.

Possible Implementations

• Using a brute force approach, we could have an array with the size of the cache. Each element in the array would store the item along with its frequency and last access time. When we get an element, we’d have to iterate through the array to find our element and update the item with the incremented frequency value. This means getting an element would be on average O(n). To put an item when the cache is full, we’d have to look for an item to evict. So we’ll iterate through the whole array and look for the item with the least frequency and oldest access time. This would take O(n).

• Using a min-heap data structure would be an improvement - each node will hold the cached element along with its frequency. This makes it easy to remove an element when the cache is full as the least recently and frequently used element will be a the top. The problem comes when inserting or reading from the cache, as any operation will mean the min-heap will need to be rebalanced which will take O(logn).

Solution in constant time

To implement an LFU that uses constant time for pushing and getting items, we’re going to need 2 data structures: a key map and a frequency map.

• Key Map - stores the key of the item and the item node containing the key, value and frequency.

• Frequency Map - stores the frequency as key and a doubly linked list of all the cached items having that frequency as value. The items in the doubly linked list are sorted by recency such that most recently accessed items will be at the head of the list and the least recently used at the tail. Due to the nature of the doubly linked list, removing, adding and updating a cached item will take O(1).

Given those two data structures, the cache operations would have the following logic:

• get operation for an item X:

  • check that X is in our key map. If it doesn’t exist return -1. Otherwise get the frequency from the value node.

  • for the frequency we’ve obtained above, get the doubly linked list from the frequency map. Delete the node from the doubly linked list. Create a new node with the incremented frequency and added it to the frequency map at the head of the doubly linked list.

    
    Following the above instructions, the implementation for the get operation should look similar to this:

• put operation for an item X:

  • if the item already exists in the cache, it returns the node and updates the frequency (hence, we call getNode which does all of these)

  • if the item doesn’t exist in the cache:

    • if the cache is full - we remove the least recently and frequency used item, which should be the tail of the doubly linked list value in the frequency map with the minimum frequency.

    • then we add the item at the head of the doubly linked list with has the frequency of 1

    
    Considering the above, the implementation for the put operation would look like:

Putting everything together, the code should look like this.

Conclusion

In this article we had a look on how we would go about implementing an LFU.

To learn more about caching, have a look at the following articles:

• Mastering Cache Management: Eviction, Invalidation, and Optimal Data Access Strategies

Or stay tuned as in the upcoming articles, we'll dive deeper into:

• Types of Caches: Exploring different caching mechanisms like in-memory, distributed, and CDN caching.

• Deployment Strategies: Understanding how to integrate caching into your architecture.

• Metrics: How to monitor cache performance for optimal results.

• Implementation Solutions: Practical examples with popular caching tools and libraries.

When to use it :

• for frequently accessed data - e.g. website assets, session data, popular content

• for expensive operations - database queries (if the same complex query is frequently executed), API calls (e.g. weather data, currency conversions), compute intensive calculations

• for times when speed is paramount - high-traffic websites (reduce strain on the backend by serving frequent requests), search suggestion (autocomplete features often use a cache to display results instantly as the user types)

When accessing data from a cache, there's a best-case and a fallback scenario. A cache hit means the data was found in the cache, leading to a speedy response. A cache miss means the data wasn't present, so the system will need to fetch it from the slower main storage.

What to consider:

• Amdahl’s Law - “the overall performance improvement gained by optimizing a single part of a system is limited by the fraction of time that the improved part is actually used”

  • Diminishing Returns: Cache hit rates matter immensely. The initial gains from adding a cache are often dramatic. But, according to Amdahl's Law, as you reach 90, 95, 98% hit rates, the effort to squeeze more performance decreases in effectiveness.

• Pareto Distribution - “also known as the “80-20 rule” stating that 80% of outcomes are due to 20% of causes“

  • Cache Misses Are Not Evenly Distributed: In many systems, some data is far more popular than others. Picture the top 10% of products on an e-commerce site getting 60% of the traffic. This skewed access pattern aligns with the Pareto principle. To get the caching wins, you need to first analyse your data to see if the access patterns truly exhibit Pareto-like characteristics.

Cache Benefits

• Faster Access and Reduced Latency - serving data from in-memory stores vs. databases or external APIs offers massive latency gains.

• Bandwidth Optimisation - when external API calls or media assets are cached, you can potentially decrease bandwidth usage.

• Reduced Database Costs - databases are often a significant cost factor. Offloading queries to a cache can directly lower your database-related expenses.

• Improved Throughput - when a cache satisfies a request, your main database or origin servers are untouched. This allows your system to handle more concurrent users without overwhelming core resources.

• Handle Spikes - sudden surges that would cripple an uncached system can often be served smoothly when a cache absorbs the brunt of those requests.

Cache Eviction Policies

A caching eviction policy determines how a cache handles the data eviction when the cache is full. When an item is requested from the cache, it may result in a miss, which means data needs to be fetched from the database and the cache needs to be updated. But before doing that, an item needs to be removed. The following section describes several strategies on how to choose which item to pick.

• Queue-Based Policies

  • FIFO (First-In-First-Out)

    • Process: The first in/oldest item gets removed to make space for the new item, which is then added to the end.

    • Pros:

      • Simplicity: Very easy to implement and understand.

      • Fairness (Of Sorts): Prevents any item from 'hogging' the cache indefinitely if there's a steady stream of new data.

    • Cons:

      • Inefficient for Many Workloads: Real-world data access patterns often aren't purely FIFO order. An item used constantly might get evicted simply because it was added a while ago, leading to repetitive refetches of the same thing.

      • Ignores Popularity: A popular item accessed right after eviction will need to be fetched again, even though there may be other rarely-used data in the cache.

  

  • LIFO (Last-In-First-Out)

    • Process: The last in/newest item gets removed to make space for the new item, which is then added to the end.

    • Pros: Same as FIFO

    • Cons: Same as FIFO

• Recency-Based Policies

  • LRU (Least Recently Used)

    • Process: The basic principle of LRU caching is to remove the least recently used items first when the cache reaches its capacity limit. This means that the cache keeps track of the order in which items are accessed, and when it needs to make space for a new item, it removes the least recently accessed item.

    • Pros:

      • Effective use of cache space: LRU caching maximizes the utilization of cache space by keeping frequently accessed items in memory.

      • Good performance: LRU caching performs well in scenarios where there is a high degree of temporal locality, such as web servers and databases.

      • Simple implementation: The basic LRU caching algorithm is relatively simple to implement and understand.

    • Cons:

      • Overhead: LRU caching requires additional data structures to keep track of access history, which can introduce overhead in terms of memory and computational resources.

      • Not suitable for all access patterns: LRU caching may not perform optimally in scenarios with non-temporal access patterns or where access frequencies vary widely.

    • Possible implementation:

      • An LRU cache is often implemented by pairing a doubly linked list with a hash map. The linked list is set up with the most-recently used item at the head of the list and the least-recently used item at the tail.

      • When we get a request, we first look up the item in our hash map:

        • if it’s there, it’s a cache hit:

          • we use the hash map to quickly find the corresponding linked list node.

          • we then move the item's linked list node to the head of the linked list, since it's now the most recently used (so it shouldn't get evicted any time soon).

        • if it’s not there, it’s a cache miss:

          • if the cache is full - we evit the least recently used item which should be at the end of the queue.

          • we create a new linked list node and add it at the head of the queue and update the hashmap where the value points to the node we’ve just inserted.

  

  • MRU (Most Recently Used)

    • Process: evicts the most recently accessed data item from the cache. When a new item needs to be added to the cache and it's already at full capacity, the cache algorithm looks at the usage history of the items and evicts the most recently used one. This makes room for the new item to be added. MRU can be useful in scenarios where a small subset of items is accessed frequently.

    • Pros: Same as LRU

    • Cons: Same as LRU

• Frequency-Based Policies

  • LFU (Least Frequently Used)

    • Process: The basic principle of LFU caching is to remove the least frequently used items first when the cache reaches its capacity limit. This means that the cache keeps track of the frequency with which each item is accessed, and when it needs to make space for a new item, it removes the item that has been accessed the least frequently.

    • Pros:

      • Effective use of cache space: LFU caching maximizes the utilization of cache space by keeping frequently accessed items in memory.

      • Good performance: LFU caching performs well in scenarios where there are consistent access patterns and certain items are accessed more frequently than others.

      • Adaptive: LFU caching dynamically adjusts to changes in access patterns, ensuring that frequently accessed items remain in the cache.

    • Cons:

      • Complexity: LFU caching algorithms can be more complex to implement and maintain compared to simpler caching strategies like LRU (Least Recently Used).

      • Overhead: LFU caching requires additional data structures to track the access frequency of items, which can introduce overhead in terms of memory and computational resources.

      • Sensitivity to outliers: LFU caching may not perform optimally in scenarios with highly variable access patterns or sudden spikes in access frequency. Outliers in access frequency can skew the effectiveness of the cache.

    • Possible Implementation:

      • We are going to need 2 data structures, a cache map HashMap<item, Node> which holds our item along with a Node object
        
        Node(int val, int key) {

         this.val = val;

         this.key = key;

         this.frequency = 1;

        }
        
        and a frequency map HashMap<count, DoublyLinkedList<Node object>> which holds the count or frequency value a long with a list of all the items that have that given frequency.

      • When the cache is full, the next element that needs to be removed is the tail element of the doubly linked list with the least frequency.

    

  • LFRU (Least Frequently Recently Used)

    • Process: based on a hybrid algorithm that combines aspects of both LFU (Least Frequently Used) and LRU (Least Recently Used) caching strategies. LFRU caching maintains both frequency and recency information for each item in the cache. It tracks how frequently an item is accessed (similar to LFU) and how recently it was accessed (similar to LRU). When a new item needs to be added to the cache and it's already at full capacity, the cache algorithm uses a combination of frequency and recency information to decide which item to evict.

      • If an item has a low frequency of access but was accessed recently, it may be considered valuable and kept in the cache.

      • If an item has a high frequency of access but hasn't been accessed recently, it may be considered less valuable and evicted from the cache.

    • Pros:

      • Balance: LFRU caching strikes a balance between considering both frequency and recency of access, leading to more nuanced cache management decisions.

      • Adaptability: LFRU caching adapts well to varying access patterns and workload changes, ensuring efficient use of cache space.

      • Performance: LFRU caching can provide good performance by retaining frequently accessed items while also accommodating recent access patterns.

    • Cons:

      • Complexity: LFRU caching algorithms can be complex to implement and may require additional computational overhead compared to simpler caching strategies.

      • Resource Consumption: Maintaining both frequency and recency information for each item in the cache may consume more memory and computational resources.

      • Tuning: Tuning the parameters of an LFRU caching algorithm to optimize performance may require careful experimentation and analysis.

Cache Invalidation

Cache invalidation is the process of removing or marking as invalid data in a cache system. Implementing effective cache invalidation strategies is vital for preserving data integrity and preventing the delivery of stale or obsolete data.

• Invalidating on Write

  • This cache invalidation happens upon modifications to the underlying data source. Whenever a modification operation, like an update or deletion, takes place, the cache is notified or marked to invalidate the associated cached data. Subsequent attempts to access the invalidated data result in cache misses, prompting the retrieval of the latest information from the data source. While this method reduces the risk of serving outdated data, it may introduce a slight delay due to cache misses during the invalidation process.

• Invalidating on Read

  • Cache invalidation occurs during data access or retrieval. When a read request is received, the cache verifies whether the cached data is still valid or has become outdated. If the data has expired or is marked as invalid, the cache retrieves the latest information from the data source and updates its contents before fulfilling the request. While this method ensures the delivery of up-to-date data, it introduces overhead to each read operation as the cache must confirm the freshness of the data before responding.

• Write-Through Invalidation

  • In systems where data is written directly to the cache and the source of truth simultaneously, cache entries are invalidated whenever data is written or updated in the source of truth.

• Time Based Invalidation

  • Cached data is invalidated after a certain period of time has elapsed. For example, a web page cache may be configured to invalidate its content every hour to ensure that users see the most recent version of the page.

• Time-to-Live (TTL)

  • Similar to time based invalidation, TTL associates a time duration with each cached item. When an item is stored in the cache, it is marked with a TTL value indicating how long the item is considered valid.

Caching Strategies

• Read Strategies

  • Cache Aside - also known as lazy loading or lazy population, is a caching strategy where the application is responsible for managing the cache directly. When data is requested, the application first checks the cache. If the data is found, it is returned from the cache. If the data is not in the cache, the application retrieves it from the data source, stores it in the cache, and then returns it to the caller.

    • Pros:

      • Flexibility: Cache-aside allows applications to cache data selectively, based on their specific requirements. This flexibility enables developers to optimize caching for performance and efficiency.

      • Control: Applications have full control over cache management, including when to populate, update, or evict cached data. This level of control allows for fine-tuning cache behavior to suit the application's needs.

      • Reduced Load: By caching frequently accessed data, cache-aside can reduce the load on the primary data source, leading to improved overall system performance and scalability.

    • Cons:

      • Stale Data: Without proactive cache population or expiration mechanisms, cache-aside may lead to serving stale or outdated data to users, particularly if data changes frequently in the primary data source.

      • Consistency Challenges: Ensuring consistency between the cache and the primary data source can be complex, particularly in scenarios involving distributed caching or multiple cache instances. Inconsistent data can lead to data integrity issues and user experience problems.

  

  • Read Through - caching strategy where the application reads data from the cache, and if the requested data is not found in the cache (cache miss), the cache automatically fetches the data from the primary data source (such as a database) and populates the cache with the retrieved data before returning it to the application.

    • Pros:

      • Automatic Population: Cache read-through automatically populates the cache with requested data, reducing the need for explicit cache management.

      • Simplicity: Implementing cache read-through is relatively straightforward, as the caching logic is encapsulated within the cache itself, simplifying application code.

    • Cons:

      • Complexity in Eviction: Managing cache eviction policies to make room for new data can be challenging, especially in scenarios with limited cache capacity or high data volatility.

      • Resource Consumption: Cache read-through can consume additional system resources, such as memory and network bandwidth, particularly during cache population operations.

  

  • Refresh Ahead - also known as cache pre-loading or cache warm-up, is a caching strategy where the cache proactively fetches and stores data from the primary data source before it is requested by the application. This anticipatory approach aims to populate the cache with frequently accessed or critical data in advance, thereby reducing latency and improving overall system performance.

    • Pros:

      • Reduced Latency: By pre-loading frequently accessed data into the cache, cache refresh-ahead reduces the latency associated with fetching data from the primary source on demand.

      • Improved Performance: Proactively populating the cache with data that is likely to be accessed soon improves overall system performance by ensuring that frequently accessed data is readily available in memory.

      • Consistency: By periodically refreshing pre-loaded data, cache refresh-ahead helps maintain data consistency between the cache and the primary data source, ensuring that cached data remains up-to-date.

    • Cons:

      • Resource Overhead: Proactive data fetching and cache pre-loading consume additional system resources, such as memory, CPU, and network bandwidth, especially for large or frequently updated data sets.

      • Over-Population: Inefficient cache pre-loading strategies may lead to over-population of the cache with unnecessary or infrequently accessed data, resulting in wasted resources and reduced cache effectiveness.

• Write Strategies

  • Write Through - caching strategy where data is written to both the cache and the underlying data store simultaneously whenever a write operation occurs.

    • Pros:

      • Data Consistency: By writing data to both the cache and the underlying data store simultaneously, cache write-through ensures that the cache remains synchronized with the primary data source, minimizing the risk of data inconsistency.

      • Simplicity: Cache write-through simplifies cache management by eliminating the need for separate cache invalidation or synchronization mechanisms, as data is always written to both the cache and the data store.

    • Cons:

      • Write Latency: Write operations in cache write-through may experience increased latency due to the additional step of writing data to both the cache and the data store, which can impact application performance, especially for write-heavy workloads.

      • Potential Bottleneck: In highly concurrent or distributed environments, cache write-through may introduce a potential bottleneck if multiple write operations contend for access to the cache and the data store simultaneously, leading to contention and performance degradation.

  

  • Write Around - caching strategy where write operations bypass the cache and are written directly to the underlying data store. Data is cached when it is read from the data store, allowing the cache to store frequently accessed data while avoiding unnecessary caching of infrequently accessed or transient data.

    • Pros:

      • Reduced Cache Pollution: By bypassing the cache for write operations, cache write-around avoids caching infrequently accessed or transient data, reducing cache pollution

      • Scalability: Cache write-around is well-suited for scenarios with write-heavy workloads or large datasets, as it minimizes the impact on cache performance and storage capacity by avoiding unnecessary caching of write operations.

    • Cons:

      • Read Latency: The first read operation for data that has not been cached incurs additional latency, as the data must be fetched from the underlying data store instead of the cache, potentially impacting application performance.

      • Stale Data: If data is frequently updated or modified in the underlying data store, cache write-around may result in cached data becoming stale, as the cache does not proactively invalidate or update cached data based on write operations.

  

  • Write Back - caching strategy where write operations are initially written to the cache and later asynchronously propagated to the underlying data store.

    • Pros:

      • Improved Write Performance: Cache write-back improves write performance by acknowledging write operations as soon as they are written to the cache, without waiting for them to be persisted to the data store. This reduces latency and improves throughput, especially for write-heavy workloads.

      • Optimized Resource Utilization: Cache write-back optimizes resource utilization by buffering write operations in the cache, allowing the data store to handle write operations more efficiently and effectively, especially during peak load periods.

    • Cons:

      • Increased Complexity: Implementing cache write-back requires additional mechanisms for asynchronous data propagation, periodic flushing, and handling of failure scenarios, increasing system complexity and maintenance overhead.

      • Potential Data Loss: In the event of cache failure or system crash, cached data that has not been flushed to the data store may be lost, leading to potential data loss or corruption. Proper backup and recovery mechanisms are required to mitigate this risk.

Each caching strategy has its own advantages and considerations, and the selection of an appropriate strategy depends on the specific requirements and characteristics of the system. 

Summary

In this article, we've explored the fundamentals of caching, covering the core definition of caching, scenarios where it offers significant advantages, the importance of eviction policies (like LRU or MRU) to manage cache size, and strategies for invalidation to ensure your cache aligns with the primary data source. We also touched on the flow of data between the cache and your database for both read and write operations.

Stay Tuned! In upcoming articles, we'll dive deeper into:

• Types of Caches: Exploring different caching mechanisms like in-memory, distributed, and CDN caching.

• Deployment Strategies: Understanding how to integrate caching into your architecture.

• Metrics: How to monitor cache performance for optimal results.

• Implementation Solutions: Practical examples with popular caching tools and libraries.

In the whirlwind of software development, success too often depends not just on what you know, but on who you know. Picture this scene:

A promising new hire, Maya, joins your team with solid fundamentals but limited experience in your specific tech stack. She tackles an assigned task, diligently studying documentation and online resources. Days turn into a frustrating week as Maya gets stuck on an integration issue that seasoned team members know is notoriously finicky. Meanwhile, across the office, another developer, Alex, breezes through a similar task, muttering, “Oh, the usual quirks with that library...” under his breath. Maya doesn't know to connect with Alex, and Alex hasn't been prompted to share the workaround. What unfolds?

• Wasted Effort: Maya's progress stalled, while precious time is spent reinventing the wheel.

• Missed Growth: A learning opportunity evaporates. This could have been a prime moment for Maya to master a tricky corner of the codebase and solidify a positive working relationship.

• Teamwide Impact: Frustration festers. Maya feels unsupported; Alex becomes the go-to person, soon overburdened with questions which could have been prevented by a better knowledge sharing accros teams.

Real-Life Story: The Cost of “I'll Figure It Out”

Early in my career, I was hesitant to “bother” senior engineers, assuming struggling through was a rite of passage. This led to weeks of spinning my wheels and missed chances to understand design rationale. While I eventually brute-forced the solutions, more was lost than just my time. This mentality fostered an aversion to asking for help, harming my progress long-term.

Beyond Missed Opportunities

As seen so many times, these issues can easily scale throughout your organisation:

• Knowledge Silos: Critical insights locked with individuals or teams impede agility and can jeopardize a project if someone leaves.

• Reinventing the Wheel: Lack of awareness of existing in-house solutions drains company resources and morale.

• Uneven Workload: A few “gurus” get swamped with basic questions, while the potential of less outspoken team members remains untapped.

The problem isn't about a lack of talent or documentation—it's about failing to create the channels through which knowledge, wisdom, and experience can flow seamlessly.

Why Traditional Approaches Fall Short

While onboarding documents and internal wikis are valuable tools, they often fall short of providing the depth and personalisation critical to true mentorship. Here's why:

• Onboarding documentation or wikis represent a snapshot of knowledge at a particular point in time. Technology, teams, and best practices evolve rapidly, rendering them quickly outdated. Mentorship, in contrast, is a dynamic, two-way process that flexes with evolving needs. A mentor can pinpoint what needs updating and address the “why” behind processes, not just the “how.”

• Boilerplate documentation lacks the nuanced, individualized context vital for deep understanding. Imagine a new engineer just out of school joining your heavily regulated industry. Onboarding docs tell them the company forms to fill out, but a mentor helps them grasp the underlying reasons for the strict compliance rules, fostering both adherence and innovation within the constraints.

• Wikis present facts but lack the interpretive layer a mentor provides. A new hire wrestling with code architecture benefits infinitely more from a mentor patiently explaining design choices and tradeoffs than a page describing abstract patterns.

My early days at a fast-growing startup were marked by frantic scrambling through the company wiki. There was always a page covering some technical process, but rarely addressing the edge cases I inevitably ran into. The breakthrough came when paired with an experienced developer. Turns out, half the wiki was outdated, filled with workarounds for since-fixed bugs! My mentor knew what to skip, the undocumented “tribal knowledge”, and crucially, was invested in helping me bridge those information gaps.

Beyond Technical Knowledge

It's crucial to stress that wikis and docs fall shortest in mentorship's “softer” aspects such as:

• Confidence Building: The simple act of a senior engineer dedicating time boosts a mentee's sense that they belong.

• Company Culture Navigation: How and when to escalate issues, who is truly the expert on 'X' technology, informal communication norms – a good mentor accelerates this decoding process.

• Career Pathing: Wikis won't provide honest insights into growth opportunities or challenges within the organization.

While documentation will always play a crucial role, the power of human connection in knowledge transfer and professional development shouldn’t be underestimated.

Key Elements of Effective Systems

Transforming good intentions into a true force for growth requires structure. What truly makes a mentorship program thrive? Let's dive into the key elements that separate well-meant initiatives from transformative programs:

1. ✅ Finding the Right Match

   Skillset is just a start for finding a good match but what sometimes gets overlooked are the other aspects such as: ways of working, personality, common aspirations etc.

   • Communication Match: An introverted mentee may struggle with an overly extroverted mentor, regardless of technical skills. Similar communication styles streamline interaction and reduce misunderstandings. For example, both being naturally direct benefits efficiency, while an introvert/extrovert pairing may require adapting to each other's preferences.

   • Personality Fit: Grumpy pessimist + eager optimist = potential frustration. Friction over clashing personality styles can dampen enthusiasm on both sides. A mentor who seems overly negative or dismissive can discourage the mentee, impacting their progress.

   • Career Path Honesty: Is the mentor in a position to truly help with the mentee's aspirations?

2. ✅ Setting SMART Goals and Keeping a Live Document

   SMART stands for Specific, Measurable, Achievable, Relevant, Time-bound. Don't just say “improve coding”, “get better at X“ - break that down into smaller focused and actionable goals. Goals shouldn't be set and forgotten. Regular check-ins allow mentors/mentees to celebrate progress and refine goals as the relationship evolves.

3. ✅ Mentoring on Values

   1. The Hidden Culture Code: Every company has unspoken ways decisions get made, what truly signals being a team player, etc. This can be harder to navigate than the tech stack itself!

   2. Ethical Compass: This is especially significant in fast-paced, 'move fast, break things' environments. A mentor showing when to push back, uphold quality standards despite deadlines – that models values in action.

4. ✅ Network Creation: Opening Doors and Breaking Invisible Barriers

   1. Advocate, Don’t Gatekeep: Simply knowing names gets a foot in the door, but true mentorship paves the way. (“May I intro you to Maya? Her work on X would be relevant to your team...”)

   2. Beyond Internal Networking: Mentors with broad industry connections become allies. Recommending the mentee for conference talks, open-source collaborations – this amplifies growth.

   3. “Paying it Forward” Loop: This isn't about indebted favors. Rather, the mentee, now armed with a wider network, is positioned to eventually open doors for others.

5. ✅ Keeping it Structured, but Flexible

   Guidance and autonomy should balance.

   • The Power of Suggestion: Offer templates for goal setting, sample agendas for early meetings, etc. This eases pairs into the process without a rigid script.

   • Mentorship Menu: Could have varying levels of engagement. Is there a 'quick question' channel AND an option for longer term pairing?

   • Adaptability Test: Mentor is NOT a crutch. If a mentee fails to prepare for meetings, repeatedly disregards agreed-upon goals, that's feedback for the program coordinators.

6. ✅ The Feedback Loop: Mentors Aren't Mind Readers

   • Safe Space: Emphasize to mentees they WON'T hurt their mentor's feelings by being honest about what's not working. This protects from passive resentment.

   • Org Level Feedback: How did this mentorship improve (or not) the mentee's onboarding? Their contributions?

7. ✅ Celebrating Success & Recognising Contributions

   • Success Is Variable: Not all mentees become rockstars (that's okay!). Did they get unstuck on a blocker? First significant code contribution? Celebrate it to encourage the relationship.

   • Make it Visible: Internal newsletter, 'thank you' board, etc. Showcases that the company is serious about this, not just paying lip service to mentorship.

8. ✅ Fostering a Psychological Safe Environment

   1. Combatting Imposter Syndrome: Especially prevalent among new hires, the fear of appearing incompetent stifles questions. A mentor explicitly stating 'No question is stupid' is a start, but must be consistent in their response.

   2. Error as Opportunity: This takes more than words. Can a mentor show how they diagnose their OWN mistakes? This teaches more than perfect demos ever could.

   3. The Long Game: Psychological safety creates long-term loyalty. That sense of being 'seen' and supported can be the deciding factor on whether a mentee sticks with the company during hard times.

Common Pitfalls

If you've been involved in mentorship, chances are you've encountered a few stumbling blocks. In this section, we'll uncover the most prevalent pitfalls:

❌ Pitfall 1: The 'Hero' Mentor

Swooping in to solve problems instead of teaching problem-solving strategies. The mentee ends up reliant, the mentor burnt out. Mentoring is about giving a toolbox, not just building things for others. Empowering mentees is empowering for the long term.

❌ Pitfall 2: Assuming Interest

A 'passion project' mentorship match – brilliant engineer paired with someone just seeking stable paycheck. Mutual frustration ensues. Upfront discussion about goals and what excites the mentee prevents wasted time and potential resentment.

❌ Pitfall 3: Rigid 'By-The-Book' Mentorship

The program dictates all the focus areas, meeting frequency, etc., with no room for adaptation based on the pair's unique needs. Structure is good, rigidity is stifling. Allowing mentor/mentee input on format fosters both ownership and better matching in future iterations.

❌ Pitfall 4: Neglecting Mentor Development

We assume technical experts already possess all the skills needed to be effective mentors. An underdeveloped mentor may flounder when providing constructive feedback, navigating difficult conversations about the mentee's performance, etc. This directly impacts the mentee's growth and can even breed disillusionment with the concept of mentorship altogether. Beyond the tech side, mentors need to give feedback, motivate, and sometimes manage underperformance. This is skill, not inherent instinct.

Key Takeaways

Let's recap the essential lessons covered:

1. Be on the Lookout for Static Resources

   1. These risk leaving mentees feeling overwhelmed, misinformed, and frustrated.

   2. “Information overload” hinders progress more than lack of access to facts. Mentors serve as filters and navigators.

   3. Wikis can actually impede adaptation— they perpetuate legacy knowledge unless meticulously maintained.

2. Have Structured Mentorship Programs

   • Pairing: Match new hires or less-experienced team members with seasoned developers for focused knowledge transfer and ongoing support. This program can be formal (designated time each week) or more flexible as needed.

   • Goal Setting: Help mentor/mentee pairs define what success looks like for both parties, creating focus and increasing progress tracking.

   • Company-Wide Mentorship Culture: Leaders must model this behavior by mentoring within their teams and recognizing those who dedicate time to develop others. This prevents mentorship from being viewed as a mere 'extra' duty.

3. Put in place Knowledge-Sharing Incentives and Tools

   • Collaboration Hubs: Dedicated Slack channels, or other platforms designed for asking questions, offering tips, and recognizing team members for helping.

   • Celebration of “Eureka” Moments: Encourage developers to share their hard-won solutions publicly, whether a brief internal knowledge article or a 'lunch and learn' presentation. Normalises both asking and sharing.

   • Reward Structures (If Feasible): Public shout-outs for those actively spreading knowledge creates positive reinforcement.

4. Make Sure your Team’s Internal Documentation is always a “Living Document”: Set clear expectations for who keeps internal resources up-to-date. A version control system (like Git) allows seeing when modifications were made, aiding in trust of content.

Once you’ve been offered the opportunity to interview, you’d think all you need to do is perform well, right? In an ideal scenario interviews should go both ways, you’re trying to convince them to hire you but they should also do their best to convince you to come work for them. So this is your chance to test the grounds and get a feel of the culture and what working there would be like.

But what should you look for in your next company? 5 companies and +70 interviews later, here’s a list of red flags and points to consider I now go through to help set my expectations:

The Interview Process

• You only get a technical interview

As an engineer, yes, you need to know how to do your job but that’s not the only thing you need in order to succeed. If a company is interested in having only the technical interview, it might suggest that they do not value all the other non-technical but essential skills: communication, collaboration, product knowledge etc. Which means that they’ve probably hired people whose skills in those areas have not been evaluated, so you might end up working with people who are all hard skills and no soft skills. In the end there’s more to a good engineer than his technical knowledge.

• You do not get any technical interviews or the interviews lack any in-depth tech questions

They’re either hiring weak engineers or that is an entry level position.

• Non-technical interviewer asking technical questions

If a people person starts asking you technical questions, they are probably reading off from a script and you probably need to say exactly what is written on it. It’s like playing engineering trivia.

• Did you meet any of your co-workers during the interview process? If not, ask to before making a decision.

Company Culture

• Non-existent vision, purpose, or reasonable goals

What is the point of working on a new feature if it doesn’t integrate with the wider picture and it doesn’t contribute to the product’s growth? That means you’ll probably be working on anything the customer asks and in this case your team plays more of a customer support role.

• No of senior people in the team/Lack of diversity

You want to be part of an engineering squad that has a mix of junior/mid-level, and senior engineers. It gives you the opportunity to be surrounded by people you can learn from and to also strengthen your knowledge by mentoring others.

• Staring at interviewers who look like stressed out zombies

Are engineers excited for the projects they’re working on or do they look stressed out and overworked?

• Growth opportunities

“Is a learning mentality encouraged?” “Do they have 20% time?” (time — usually a Friday off every now and then — when engineers are encouraged to learn/experiment with new things, prepare for exams etc) “Do they have demos/brown-bags sessions?” “Is the sharing of knowledge encouraged?”

“Are there any personal development plans put into place?” “And is there a clear strategy/action points for you to progress/level up in your career?”

• Your interviewer uses a lot of buzz words.

“We’re using AI, social networks, cloud computing, big data, IoT. Machine learning this, machine learning that.” Buzz-words combined with the lack of explaining the tech behind it and how the company has implemented any of that, big red flag.

• The work-life balance — Did they mention they look for an engineer with a “strong work ethic”?

“We want you to work insane hours and weekends but we don’t want to compensate you accordingly. We want you to work those insane hours and weekends because you want to.”

We’re all in this industry because we love what to do. Someone with a spark in their eyes will usually go the extra mile because they’re passionate about what they do and they care about the impact their work has. But everyone needs time to breath, to spend time with family, recharge, pursue other passions etc. Long hours are sometimes necessary but shouldn’t be the norm.

• Do some “stalking” aka research the people you’re going to work with.

Are there any engineers who have been working at the company for some time now? If not, there might be a good reason they’ve left.

Day-to-day Work

• Do they deal with a lot of tech debt?

“What state is the code base in?” and “How much documentation is there?”. Devs will generally give honest answers to this kind of questions. “How long is the onboarding expected to take? How long do you expect it will take for a new hire to become productive in your code base?” This will give you a feel of how complex their systems are. Also, if they have lots of technical debt, and they think you will be instantly productive, they have ridiculously high expectations.

• Support/On-Call Rota

“How often will you have to be on call?” “Will it cover your working hours or will you have to do 24h?” “How many tier 1 systems does your team support?” Tier 1 meaning having to deal with systems that impact your customer the most if they’re down. (e.g. payment systems, authentication etc) “How busy will on call be?” “How many issues per week does the team usually have to deal with?” “Does the team put emphasis on writing proper, in-depth run-books and postmortems?” “How are incidents dealt with? Is the root caused being investigated/ is there a list of actions the team is planning to work on?”

• Do you need to own your feature from start — end? From gathering requirements, design/architecture proposal, infrastructure provisioning, implementation, testing, release, maintenance?

This might depend from company to company and it’s not really a red flag if you are or not involved in every step of the SDLC but it might be a good question to ask to set your expectations of what you will be asked to do.

• How are things prioritised and how much of a voice can you have?

“Are incoming tasks discussed among the team before being assigned?”

Who gets to add tasks to the team’s backlog? Are customer requests always prioritised before anything else (bug fixes, dev experience improvements)?

Ask how and who decides on estimates and deadlines. If the developers do not participate at all in how tasks are prioritised and how long the completion should take, then it is likely the leadership makes promises to customers that are next to impossible to keep and pressure you to meet these deadlines while you’ll probably have to deal with a really slow and painful dev environment/deploy/release and who knows what else is in a bad state because no one had the time to fix it.

• How does the dev experience look like?

“Do you have admin access on your work computer?” “Do applications need to go through an approval process to be granted the permission to be downloaded?” “Will you need to rise a request to download an approved application?”

I worked in a company where my request to have Notepad++ was declined on the premise I don’t need it for the job. However regulated the industry you’re working in needs to be, this is a sign of how much lack of trust the company has in its employees. Any security, regulatory or compliance checks should be run on the code you put into production as part of your build/deploy pipelines. As an engineer, how can I innovate/improve/grow if I can’t even try a postman replacement on the basis that I can already do my job with postman.

“How does the deployment and release work?” “Do they have a QA/review process?” “Can you release any time or do you need to wait for the end of the sprint?” “Will your team’s releases depend on other teams’?”

Having to wait for the end of the spring to release is not great.

“What’s the split percentage between improving the dev experience/working on legacy (e.g. automating any current manual work, smooth local running and testing of systems, easy straightforward deploys etc) and implementing new features?”

It should be a good balance between the two. If the only work the team is doing is working on new features it might be that the leadership team is a customer pleaser or you’ll be working in a brilliant team that doesn’t produce any tech debt. If the management is not able to say no to the business, it will have a major impact on your workload and how tasks are being prioritised.

• Other things to double check — glassdor reviews

That’s a good place to find the reasons former employees left the company.

In the end, you need to trust your gut

It’s unlikely that you’ll manage to keep track of all these red flags. The best thing you can do is trust your gut and try to find out if their working style will be a good fit for you.

Useful resources

Here’s a list of questions I use when interviewing (what I ask changes depending of the company I’m interviewing with, who I’m talking to and what stage of the interviewing process I’m in):

General/Processes

• What would you change around here if you could? Or is there anything you’d like to improve?

• What do you enjoy the most about working here?

• What does a typical day look like for you?

• What is your policy on working from home/remotely?

Culture

• How would you describe the culture at your company?

  • is there anything that you are particularly happy/excited about when it comes to the company culture? anything that stands out to you?

• Are there any particular qualities that you are looking for in new joiners?

  • what are some of the areas in the org that you are looking to bolster?

    • what about in the next 1-2 years?

Career Growth

• Are there any personal development plans put into place? Is there a clear strategy/action points for you to progress/level up in your career?

Planning and Roadmaps

• What does the planning and goal setting process look like at your company?

  • Do you use use OKRs? Are they quarterly?

  • How do you keep the whole organisation aligned?

  • How do you ensure you’re building the right thing for the customers?

    • How do you get feedback from the customers?

• Do you have a roadmap for the next 1-2 years?

• What is a recent piece of feedback that you received from the customer?

• What are some of the key performance metrics or KPIs that you measure and monitor?

• What are some of the recent milestones at your company?

  • was there anything that could have been done better/faster? (in hindsight)

  • was there anything that you found particularly challenging while working on this?

  • how were the new features/products received by customers?

• What are your highest priorities right now?

• What are some areas of growth that your organisation is planning to focus on in the next few year?

  • Any particular areas of expertise you are looking to bring/grow?

    • examples: data analytics, growth engineering, platform, etc.

• How do you prioritise internal/technical roadmap items?

  • Do you have a separate internal roadmap?

  • How do you choose where to allocate bandwidth?

Technical/Engineering

• What are the engineering challenges that the company/team is facing?

• What is the most fulfilling/exciting/technically complex project that you've worked on here so far?

• How do you evaluate new technologies? Who makes the final decisions?

• What is your stack? What is the rationale for/story behind this specific stack?

• Do you tend to roll your own solutions or rely on third party tools? What's the rationale behind it?

• How often have you moved teams? What made you join the team you're on right now? If you wanted to move teams, what would need to happen?

• What would I work on if I joined this team and who would I work most closely with?

• How long is the onboarding expected to take? How long do you expect it will take for a new hire to become productive in your code base?

• Support/On-Call

  • How often will you have to be on call? Will it cover your working hours or will you have to do 24h? How many tier 1 systems does your team support?

  • How busy will on call be? How many issues per week does the team usually have to deal with?

  • Does the team put emphasis on writing proper, in-depth run-books and postmortems? How are incidents dealt with? Is the root caused being investigated/is there a list of actions the team is planning to work on?

• Release

  • How does the deployment and release work?

  • Do you have a QA/review process?

  • Can you release any time or do you need to wait for the end of the sprint?

  • Will your team’s releases depend on other teams’?

• Dealing with legacy

  • How does the engineering team balance the effort between feature requests and engineering maintenance/legacy work?

  • What’s the split percentage between improving the dev experience/working on legacy (e.g. automating any current manual work, smooth local running and testing of systems, easy straightforward deploys etc) and implementing new features?

Microservices have been getting more and more popular in recent years, and several companies are migrating their monolithic applications to microservices. But just a reminder that switching to microservices might not always be the good option. Martin Fowler wrote a nice article regarding the monolith-first approach. Here are a couple of useful statements:

• Almost all the successful microservice stories have started with a monolith that grew too big and was broken up.

• Almost all the cases I’ve heard of a system that was built as a microservice system from scratch, it has ended up in serious trouble.

A migration technique that has been frequently used when doing system rewrites is called the Strangler Fig Application. This migration method is based on incrementally replacing existing functionalities of the old legacy system with new applications and services in a phased approach until the new application system eventually replaces all of the old legacy system’s features.

The name of this type of tree, the Strangler Fig, comes from their pattern of growth around the host tree, which often results in the host’s death. The roots grow down to the forest floor where they take root and begin to take nutrients from the soil. Gradually the roots wrap around the host tree, widen, and slowly form a lattice-work that surrounds the host’s trunk. The fig’s crown grows foliage which soon overshadows the tree. Eventually, the host tree dies leaving the fig with a hollow trunk.

Martin Fowler chose this as the name of the pattern as a metaphor of describing a way of doing rewrite of an important system. Our new system is initially supported by and wrapping the existing system. The old and the new can co coexist while the new system can grow and potentially replace the entire old system.

“An alternative route is to gradually create a new system around the edges of the old, letting it grow slowly over several years until the old system is strangled.”

~ Martin Fowler

Advantages

• It allows for incremental migration to a new system.

• It allows to pause and even stop the migration while still taking advantage of the new system built so far.

• Each step is reversible, reducing the risk of each incremental step.

“The most important reason to consider a strangler application over a cut-over rewrite is reduced risk. A strangler can give value steadily and the frequent releases allow you to monitor its progress more carefully. Many people still don’t consider a strangler since they think it will cost more — I’m not convinced about that. Since you can use shorter release cycles with a strangler you can avoid a lot of the unnecessary features that cut over rewrites often generate.”
~ Martin Fowler

When not to use it

• The pattern will not be suitable for small systems where the complexity and size are low.

• The pattern cannot be used in systems where requests to the back-end system cannot be intercepted and routed.

Issues

• Considerable effort is required for the legacy and new system integrations.

• Considerable effort is required for the parallel run of the same functionality.

• Big commitment. Lack of will and resources to finish the strangling job might lead you into a bigger mess where your system now has two ways of doing everything with an awkward interface between the two.

How it works

The strangler fig pattern allows you to move functionality over to your new service without having to touch or make changes to your existing system.

The pattern involves 4 steps:

• Identify -> identify parts of the existing system that you wish to migrate. (use Domain Driven Design to identify the various bounded contexts)

• Transform -> implement this functionality in your new microservice.

• Co-exist -> leave the existing module in the legacy application as-is. Incrementally reroute calls from the monolith over to the new microservice.

• Eliminate -> once the traffic is completely redirected to the microservice, eliminate the legacy module.

First, add a proxy (facade), which sits between the legacy application and the user. This can be a “bare bones” routing app or an API Gateway. Initially, this proxy doesn’t do anything and all the application traffic is routed to the legacy application. Once a new component is built, you can link it to the proxy and you can allow the proxy to serve traffic to that page. You can also test your new functionality in parallel against the existing monolithic code. Keep adding more services and link them to the proxy. Repeat until all required functionality is handled by the new stack. Both the monolith and the new built component need to be functional for a period of time. Sometimes the transitional phase can last for an extended duration. When the new component has been incrementally developed and tested, the legacy monolithic application can be removed.

Conclusion

Moving from your existing legacy monolithic application into a microserviced architecture is a nice end goal to have but the journey is challenging and needs to be well planned. In this article, we have discussed about the Strangler Fig migration pattern which can help you in your journey. Keep in mind that strangling a monolith is not a quick process and may require some time.

Resources

• Martin Fowler Strangler Application

• Michiel Rook The Strangler pattern in practice

• Azure Cloud Patterns Strangler pattern

System design interviews are in general 45–60 minutes long where you are given a broad topic like “Design Twitter”. You’ll be expected to generate a high-level design, showing the different system components that will be required, how they’re connected, and any trade-offs in the approach you’ve taken. Time permitting, the interviewers will ask you to go deeper into how some of your system components would function and talk about how you would handle several scenarios and edge cases.

For a junior, mid-level engineer, this article aims to give you a fair good start at the System Design Interview and it’s not an exhausting list since system design is a vast topic. But from here, you can dig deeper with other resources.

Before starting with the theory basics, let’s briefly talk about how you should approach the interview.

1. Understand the goal and requirements.
   Never go head-first into the design, make sure you understand the problem and ask clarifying questions — What is the goal of the system? Who are the users of the system? What do they need it for? How are they going to use it? Clarify your assumptions.

2. Establish the scope
   Go through all the features of the system that you can think of and write them down.

3. Establish the scale
   What is the scale of the system? How many customers does the system need to serve per day? How many transactions? What are the performance requirements (e.g. latency, throughput, availability)? What is the expected read-to-write ratio? What’s the average expected response time?

4. Design high-level. The first 20 min of the interview you should have a draft of your high level design. Make sure to discuss the trade offs and why you chose those types of technologies. What type of database would you use and why? What caching solutions are out there? Which would you choose and why?

5. Drill-down. You should drill-down into more details, starting with the components that you are most familiar with. This is to ensure that you have time to go in-depth into the topics you have the most knowledge in.

6. Wrap-up. Before the end of your interview, you should spend some time to summarize your solution and highlight important bottlenecks or improvement opportunities.

   ---

Now let’s touch onto the most encountered components you’ll probably have to talk about.

Distributed Systems

= group of computers working together as to appear as a single computer to the end-user. They have a shared state, operate concurrently and can fail independently without affecting the whole system’s uptime.

Key Characteristics:

• Stability or Serviceability or Manageability = simplicity and speed with which a system can be repaired or maintained.

• Reliability = probability a system will fail in a given period. A distributed system is considered reliable if it keeps delivering its services even when one or several of its software or hardware components fail.

• Scalability = capability of a system, process, or a network to grow and manage increased demand.

• Efficiency/Performance = response time (or latency = delay to obtain the first item) + throughput (or bandwidth = the number of items delivered in a given time unit)

• Availability = simple measure of the percentage of time that a system, service, or a machine remains operational under normal conditions.

• Fault tolerance and Catastrophe-preparedness

Load Balancer

= component that acts as a reverse proxy and distributes network or application traffic across a number of servers to improve responsiveness and availability of applications, websites or databases.

• Benefits: faster, uninterrupted service, less downtime and higher throughput, predictive analytics that predict traffic bottlenecks (smart LB), fewer failed or stressed components.

Algorithms (used to balance the load):

• Least Connection Method (directs traffic to the server with the fewest active connections)

• Least Response Time Method (directs traffic to the server with the fewest active connections)

• Least Bandwidth Method (selects the server that is currently serving the least amount of traffic)

• Round Robin Method (cycles through a list of servers and sends each new request to the next server)

• Weighted Round Robin Method (servers with higher weights (= an integer value that indicates the processing capacity) receive new and more connections before those with less weights) — to handle servers with different characteristics (e.g. processing power, availability, load)

• IP Hash (a hash of the IP address of the client is calculated to redirect the request to a server)

Caching

= temporary storage area that stores the result of expensive responses or frequently accessed data in memory so that subsequent requests are served more quickly.

When to use it — when data is read frequently but modified infrequently

Considerations:

• Set an expiration policy that is not too short (as it reloads data too fast from the database) and not too long (since data can become stale).

• How to keep the cache and database consistent (see the cache invalidation section below)

• The cache can easily become a single point of failure, you might need to use multiple caches

• Choose an eviction policy (see the cache eviction policies section below)

• Think how you handle concurrency (aka the reader-writer problem) — When multiple clients are trying to update the cache at the same time, there can be conflicts. A solution would be to use commit logs. To update the cache, we can store all the mutations into logs rather than update immediately. And then some background processes will execute all the logs asynchronously. This strategy is commonly adopted in database design.

Reading from cache:

Cache Invalidation = happens when data is modified in the database and becomes inconsistent with the cache.

Updating the cache:

• Write-aside cache — data is written into the cache and the corresponding database at the same time (also known as lazy loading) — cons: higher latency

• Write-through cache — application adds or updates an entry in the cache. Cache synchronously writes entry to the data store. Cache returns the data to the application.

• Write-around cache — data is written directly to permanent storage, bypassing the cache — cons: read request for recently written data will create a “cache miss”.

• Write-back cache — data is written to cache alone and completion is immediately confirmed to the client and then data is asynchronously updated to the data store — pros: low latency and high throughput; cons: risk of data loss in case of a crash

Cache eviction policies

• First In First Out (FIFO) — evicts the first block accessed first

• Last In First Out (LIFO) — evicts the block accessed most recently

• Least Recently Used (LRU) — evicts the least recently used items first. When the client requests resource A, it happens as follow:
  If A exists in the cache, we just return immediately.
  If not and the cache has extra storage slots, we fetch resource A and return to the client. In addition, insert A into the cache.
  If the cache is full, we kick out the resource that is least recently used and replace it with resource A.

• Most Recently Used (MRU) — evicts the most recently used items first

• Least Frequently Used (LFU) — counts how often an item is needed. Those that are used least often are discarded first.

• Random Replacement (RR) — randomly evicts a candidate item

Concurrency — reader-writer problem — When multiple clients are trying to update the cache at the same time, there can be conflicts. => use commit logs. To update the cache, we can store all the mutations into logs rather than update immediately. And then some background processes will execute all the logs asynchronously. This strategy is commonly adopted in database design.

Content Distribution Network (CDN)

= system of distributed servers (network) that deliver webpages and other web content to a user based on the geographic locations of the user, the origin of the webpage and a content delivery server.

System flow: a request will first ask the CDN for a piece of static media; the CDN will serve that content if it has it locally available. If it isn’t available, the CDN will query the back-end servers for the file, cache it locally, and serve it to the requesting user.

Message Queues

= durable component, stored in memory, that supports asynchronous communication. Helps reduce request times for expensive operations that would otherwise be performed in-line. Receives, holds, and delivers messages.

If an operation is too slow to perform inline, you can use a message queue with the following workflow:

• An application publishes a job to the queue, then notifies the user of job status.

• A worker picks up the job from the queue, processes it, then signals the job is complete.

e.g. RabbitMQ, Amazon SQS

Proxies

= piece of software or hardware that acts as an intermediary for requests from clients seeking resources from other servers.

• used to filter requests, log requests, or sometimes transform requests. Its cache can serve a lot of requests (If multiple clients access a particular resource, the proxy server can cache it and serve it to all the clients without going to the remote server.)

Types:

• Open Proxy — accessible by any Internet user. Allows users within a network group to store and forward Internet services such as DNS or web pages to reduce and control the bandwidth used by the group.

Anonymous Proxy — reveals іts іdentіty аs а server but does not disclose the іnіtіаl IP address

Trаnspаrent Proxy — іdentіfіes іtself, аnd wіth the support of HTTP heаders, the fіrst IP аddress cаn be vіewed.

• Reverse Proxy — retrieves resources on behalf of a client from one or more servers

Databases

SQL vs NoSQL

NoSQL Types

Sharding or Data Partitioning

= process of splitting up a DB/table across multiple machines to improve the manageability, performance, availability, and load balancing of an application

Partitioning Methods:

• Horizontal Partitioning — put different rows into different tables — cons: unbalanced servers (if the value whose range is used for sharding isn’t chosen carefully)

• Vertical Partitioning divide our data to store tables related to a specific feature in their own server (instagram use case — user profile information stored on one DB server, friend lists on another) — cons: if our application experiences additional growth, then it may be necessary to further partition a feature specific DB across various servers

• Directory Based Partitioning — create a lookup service which knows your current partitioning scheme and abstracts it away from the DB access code.

Partitioning Criteria:

• Key or Hash-based partitioning — apply a hash function to some key attributes of the entity we are storing — cons: adding new servers means changing the hash function which would require redistribution of data and downtime for the service. (use Consistent Hashing)

• List partitioning — each partition is assigned a list of values, so whenever we want to insert a new record, we will see which partition contains our key and then store it there

• Round-robin partitioning — With ’n’ partitions, the ‘i’ tuple is assigned to partition (i mod n).

• Composite partitioning — combine any of the above partitioning schemes to devise a new scheme

Complexities:

Joins and Denormalization — not feasible to perform joins that span database shards (denormalize the database so that queries that previously required joins can be performed from a single table)

Referential Integrity — enforcing foreign keys in a sharded database can be extremely difficult. (not supported by most RDBMS — has to be enforced in application code.)

Celebrity problem — how would you deal with hotspots? (one way would be to add a shard for each celebrity)

Rebalancing/Resharding — data distribution is not uniform and there is a lot of load on a shard => have to create more DB shards or have to rebalance existing shards.

Strategies:

• Fixed number of partitions — Create many more partitions than there are nodes, and assign several partitions to each node

• Dynamic partitioning — when a partition grows to exceed a configured size, it is split into 2 partitions so that approximately half of the data ends up on each side of the split (no of partitions adapt to the total data volume)

• Partitioning proportionally to nodes — have a fixed no of partitions per node. The size of each partition grows proportionally to the dataset size while the no of nodes remains unchanged

Examples of Real World Scenarios:

• Read about Tinder’s Geosharded Recommendations Part 1: Sharding Approach — Based on Google’s S2 which uses the Hilbert curve (=a space-filling curve that preserves spatial locality: two points that are close on the Hilbert curve are close in physical space. Each smallest Hilbert curve clone is a cell, and 4 adjacent cells form a bigger cell.) Create the geoshards by enumerating all the possible container sizes, and calculate the standard deviation of each sharding configuration, the one with smallest standard deviation will be the most balanced geo sharding configuration we are looking for.

• Read about Uber’s Unwinding Uber’s Most Efficient Service — using a QuadTree — Take a flat projection of your search space and divide it into quarters that we’ll call cells. You then divide each of those cells into quarters recursively until you hit a defined maximum depth which will be the leaves of the tree.

• Read about Instragram’s Sharding & IDs at Instagram

Replication

= keeping a copy of the same data on multiple machines that are connected via a network.

Reasons:

• Latency — Keep data geographically close to your users (reduce access latency)

• High availability — Allow the system to continue working even if some parts of it have failed

• Scalability — Scale out the number of machines that can serve read queries

Leader-based replication:

• One of the replicas is designated as the leader. When clients want to write to DB, they send their requests to the leader, which writes the data to its local storage

• The other replicas (followers) receive the data from the leader in a process called replication log or change stream.

• When a client wants to read from the DB, it can query both leader/any of the followers.

Indexes

= data structure that improves the speed of data retrieval operations on a database table at the cost of additional writes and storage space to maintain the index data structure.

Can be created using one or more columns of a database table.

ACID

= set of properties of database transactions intended to guarantee validity even in the event of errors, power failures, etc.

In the context of databases, a sequence of database operations that satisfies the ACID properties (and these can be perceived as a single logical operation on the data) is called a transaction. For example, a transfer of funds from one bank account to another, even involving multiple changes such as debiting one account and crediting another, is a single transaction.

Atomicity — means that you guarantee that either all of the transaction succeeds or none of it does. You don’t get part of it succeeding and part of it not. If one part of the transaction fails, the whole transaction fails. With atomicity, it’s either “all or nothing”.

Consistency — this ensures that you guarantee that all data will be consistent. All data will be valid according to all defined rules, including any constraints, cascades, and triggers that have been applied on the database.

Isolation — guarantees that all transactions will occur in isolation. No transaction will be affected by any other transaction. So a transaction cannot read data from any other transaction that has not yet completed.

Durability — means that, once a transaction is committed, it will remain in the system — even if there’s a system crash immediately following the transaction. Any changes from the transaction must be stored permanently. If the system tells the user that the transaction has succeeded, the transaction must have, in fact, succeeded.

CAP

impossible for a distributed software system to simultaneously provide more than two out of three of the following guarantees (CAP): Consistency, Availability, and Partition tolerance

• Consistency: All nodes see the same data at the same time. Consistency is achieved by updating several nodes before allowing further reads.

• Availability: Every request gets a response on success/failure. Availability is achieved by replicating the data across different servers.

• Partition tolerance: The system continues to work despite message loss or partial failure. Means that the cluster continues to function even if there is a “partition” (communication break) between two nodes (both nodes are up, but can’t communicate).

All the combinations available are:

• CA — data is consistent between all nodes — as long as all nodes are online — and you can read/write from any node and be sure that the data is the same, but if you ever develop a partition between nodes, the data will be out of sync (and won’t re-sync once the partition is resolved).

• CP — data is consistent between all nodes, and maintains partition tolerance (preventing data desync) by becoming unavailable when a node goes down.

• AP — nodes remain online even if they can’t communicate with each other and will resync data once the partition is resolved, but you aren’t guaranteed that all nodes will have the same data (either during or after the partition)

  ---

Conclusion

I hope this article can be the first stepping stone on your journey to ace your system design interview. Good luck and reach out for any questions or feedback 😃

As this year comes to an end, a new golang release approaches which promises to fix the existing issue of the reference loop values.

But first, let’s start with understanding what the ‘issue’ is by showing the difference between passing by value and by reference. The following code will print what you probably expect: the values 0 to 4 in some random order.

Now, we’ve all probably made the error at some point of referencing a variable loop outside its loop scope. If we update the above code to pass by reference, the result will now be 5 5 5 5 5, probably not what we were expecting.

The issue we're encountering is related to the closure capturing the loop variable i by reference. By the time the goroutines are executed in the main function, they all refer to the final value of i (which is 5) because the loop has completed.

To fix this, you need to create a new variable inside the loop for each iteration, allowing each goroutine to capture a distinct variable. You can achieve this by introducing a new variable in the loop scope:

In the above, each goroutine captures a different variable val, and the sync.WaitGroup ensures that the main function waits for all goroutines to finish before exiting. The output should now be the values 0 to 4 as expected.

This problem was so common that Go 1.21 contains a new experimental feature which changes the semantics so that each iteration would use a new variable. You can turn it on by exporting the flag export GOEXPERIMENT=loopvar. This behaviour will be enabled by default in Go 1.22. Let’s say we have the following example:

If we enable the flag, this time it will produce the following output:

With modifications like these, it's evident that Go is becoming increasingly welcoming to beginners. Tackling issues such as loop variable capture will significantly accelerate the onboarding process for new developers, enhancing the overall experience for gophers at any skill level.

Resources:

• Golang Blog Announcement

• Proposal of Per-Iteration Loop Variable

• Loopvar Experiment Proposal

Some time ago I finished watching the TV series The Bear, and some parts of it keep resurfacing in my brain. And yes, you’ve guessed it. In order to declutter my mind, I’ll obviously need to write about it.

As I watched the show, my mind couldn't help but dwell on the various aspects of it that I come across in my daily work. “Hey, I know a guy that behaves like that and I have the same issues working with him“, or “Hey, I’ve seen my manager do the same thing to resolve a similar conflict“. What I found really cool about the show is that each main character seemed to be represented as an accentuated archetype of the different people you might encounter in your work environment: the controller, the perfectionist, the slacker, the over-achiever, the challenger, and somehow (after a really long struggle) the team manages to make it all work. We’ll get into how in a second. I also found bits of myself in almost every character and made me realise the issues I might be dealing with myself.

So what I want from this article is to present a reflection of the different dysfunctions of a team, different conflict resolutions, and what leadership lessons we can get from the show.

But first things first, what is the show about? The Bear unfolds the story of Carmen (Carmy) a young chef who, following his brother's death, returns to his hometown of Chicago. He inherits not only his family restaurant but also his brother's co-manager and a team of disorderly cooks. With experience in some of the world's finest restaurants, he is confident in transforming his family establishment into a culinary landmark. Easy, right? But what Carmy’s journey will unravel is an endless road of blockers: tight deadlines, challenging budget, difficult stakeholders, endless milestones, the pain of legacy work (in this case his brother’s huge unpaid debts), a ton of dependencies, unknown risks, burn out, lack of alignment, no shared vision, others’ pursue of self-interest over the well being of the team, personal commitments and so on. These are all real issues we’ve been dealing with which the show exposes in a really well-crafted manner.

The Archetypes

I couldn’t help but analyse how well the characters fitted in the Enneagram’s classic archetypes roles.

Firstly, we have Tina, the Challenger, who is resentful of change and of new-comers. She likes to be independent and refuses any type of help, even if it’s needed. When Sydney is appointed chef de cuisine, she constantly undermines Sydney's authority and disregards her advice. Her respect is earned through reason and competency, and not through age or status.

Sydney is The Achiever, the innovator, passionate about cooking, and very hard-working. She is always open and pushes for change. And sometimes pushing for too much at once, especially when the team is already dealing with a lot, leads to conflicts, as we’ll soon see.

Carmy is The Enthusiast, the observant, focused, and experienced person but with high anxiety, a problem of expressing himself, and the deep frustration that comes when he fails at that. Carmy loses his temper quickly. He doesn’t know how to communicate with his team. Routine and control calm him down.

Ritchie is The Giver (you doubt it but listen to me), the stubborn, lazy guy who takes it out on every one out of fear and anger at his own life. But under the angry hood, he is the loving and caring guy who cares strongly for his family even if he doesn't show it in words.

Marcus is The Peacemaker and The Perfectionist, an emphatic and thoughtful person but his dedication to his passion as a baker sometimes stands in the way of seeing the bigger picture as he fails to put the team’s wellbeing first before his own. This can be seen when he decides to work on his special doughnut rather than keep up with Carmy’s orders and his work at the restaurant during a busy and stressful time.

Getting to know your teammates’ ways of working and needs is crucial to making the team work well together: develop a common language, identify strengths and weaknesses, and build an authentic culture that works for you and them.

Lessons

I can’t share and analyse all the things I really appreciated about this show in the hope that I can still keep this article fairly short. So this is just a compressed list of lessons that came out from the different situations the team dealt with.

1. Grow your team; hire for passion and invest in skills

   Later in the show Tina and Ebra are sent to culinary school. Marcus is sent to Copenhagen to learn from a well-known baker and Natalie (Carmy’s sister) is brought in as a project manager because she might be a good fit. Even Richie is sent to a restaurant called Ever because Carmy “believes in him“. This completely changes the team’s dynamics as this constant investment in their skills makes them feel seen and appreciated. So they improve and contribute, and most importantly learn to care and show up.

2. Don’t stop the learning
   Another aspect I appreciated about the show was presenting how the team got unstuck when they were blocked (mentally). How they found inspiration and new creative ways to move forward. They would go outside and look at architecture, they would taste different foods, try out new restaurants, and travel to new places.
   Now this sounds expensive, and the thing is growth is not only hard but it can also come with a price. This is where Carmy stepped in with his network. As a leader, you might not be aware of the different realities people come from. Not everyone has your network or resources.
   
   During the show we can see Syd reading “Leading with the Heart: Coach K’s Successful Strategies for Basketball, Business, and Life” which emphasises the importance of leading with passion, empathy, and integrity. I haven’t read it but this approach to leadership will be emphasised in a lot of scenes during the show.
   

3. Trust your team

   One of my ‘but why’s’ moments was when Carmy sent Richie to a high-end, three-Michelin-star Chicago restaurant called Ever, to learn about how fine dining establishments are run. At this point, I was wondering how Richie fit in all of that. He’s the last person who would understand the importance of order and discipline. But what I wasn’t getting at that time is that Richie was battling some pretty big demons around his own ability and self-worth, like all of us out here, with our ever-growing imposter syndrome voice reminding us every day that we don’t belong and that we’ll never be good enough. And this is where Carmy steps in, and tells the crew over at Ever that he believes in Richie and his ability to work with people. Yes, Ritchie was supported by multiple people to be able to develop and thrive but without the trust from the others, maybe he wouldn’t have been able to step in and help Sydney when the tickets started to back up on opening night. Maybe he wouldn’t have been able to step up and become the “Expo” on that same night.
   

4. Cultivate your shared rituals
   Do what works for you to solve the issues you’re dealing with.
   
   Another aspect that made me “I need this in my relationship with others. This is good advice.” was Carmy and Sydney’s hand gesture. One of the things Carmy learned when he worked for another chef was that when tensions were high and there was no time to smooth things out, they would do this heart hand gesture to tell each other that they were sorry and that they could readdress the issue later. Sometimes too many emotions or having more high-priority issues on your mind don’t let you have the necessary bandwidth to deal with yet another issue. Having a shared ritual in your team for communicating the “I’m sorry, we’re both well-intentioned, but this is not a good time so let’s come back to this later.” can save your situation from overflowing into an out-of-control mess. Later on, Carmy and Sydney’s gesture ritual turns into a lack of communication and avoidance problem but we won’t talk about it now. What’s important is that while developing your rituals, you also need to adapt them or get rid of them accordingly when they don’t make sense for your team anymore. So, don’t continue doing something just because it used to work in the past.
   
   Other communication rituals help them maintain safety, optimise their work and show each other respect, like “Behind!” - I'm walking behind you, so please don't back up or turn quickly because one of us probably has a piping hot pot of something., “Corner!” - I'm coming around the corner, so be aware., “Hands”: Someone, please come grab this food and deliver it to the customer while it's hot. “Thank you, Chef.”
   

5. Show respect

   Talking about respect, I really enjoyed the “Yes, Chef“ acknowledgment. Even if they didn’t agree with one another, they would say “Yes, Chef“ as a type of “No matter what you're saying or asking, I hear you” and I find it a nice and thoughtful way of showing respect.
   

6. Take Ownership

   Another important lesson is shown during Ritchie’s time at Ever. During a staff meeting, a member of the team asks who was responsible for a smudge on a plate during a past service. The problem was not the smudge but that no one owned it. No one stepped up. Haven’t we been in similar meetings at work? Realising that something as small as that detail shouldn't matter, but it does. It matters because it’s proof of the team’s culture - a lack of ownership where no one will try to improve a process they’re not responsible for.

   
   “We're not children. It's okay to make mistakes. We can smudge things, but we need to own up to them with immediacy, integrity, and honesty.” (scene)
   

7. Be customer-obsessed

   Yet another interesting subject presented during Richie’s time at Ever is how much emphasis an established luxury restaurant puts on attention to detail. Some customers were chatting at their table about how disappointed they were to leave Chicago without trying the deep-dish pizza. The servers listened to the guests and carried the message to the kitchen, where they tracked down a real deep-dish pizza, which the chef remixed and sent it out to be served. Why is this important? The end product is not only the food but also the customer experience. Isn’t it the same in the tech space? The attention to detail, going above and beyond to understand your users and make their journey more enjoyable is what sets apart a successful company from the others.

8. Experiment, build, and break into small iterations

   When Syd and Carmy were working together on setting up the menu, they would start small. Experiment. Fail. Try Again. Fail some more. Try something else. When something works, go from there. Iterate.
   

9. Discover and embrace your leadership style

   Not everyone has the same talents or skills, so we all have different ways of conquering a problem. As so beautifully presented in the show, Richie has a way with people, Carmy brings with him his expertise, and Syd is the relentless innovator. While they all struggled for power at the beginning, they learned when it was time to take the lead and when it was time to take a step back and let the others do their thing.

"The Bear" serves as a captivating reflection of the intricacies that make a team work together, their diverse personalities, and ways of working. What adds further to the show's brilliance is its portrayal of how the team manages to make it all work. It’s a journey full of frustration, high cortisol levels, and sometimes desperation but when everybody works towards the same goal, nice things start to happen.

Last week we’ve delved into the basics of golang’s concurrency, how goroutines are scheduled and ways of synchronisation. This week we’ll take a closer look into some advanced patterns, how they work and when it’s best to use them.

Package Singleflight

• provides a duplicate function call suppression mechanism.

• executes and returns the results of the given function, making sure that only one execution is in-flight for a given key at a time. If a duplicate comes in, the duplicate caller waits for the original to complete and receives the same results.

• especially useful in scenarios where multiple goroutines may request the same data simultaneously; makes sure that a specific piece of work is done only once, even if multiple goroutines request it simultaneously; suited for optimising your code by removing redundant processes that generate the same result.

Let’s say we have the following example:

The fetchData function simulates fetching data. The singleflight.Group is used to manage concurrent requests for the same key. Each goroutine calls sf.Do with a unique key. The function passed to sf.Do is executed only once for a given key, and subsequent goroutines with the same key will wait for the initial call to complete. The sync.WaitGroup is used to wait for all goroutines to finish.

Use Cases:

• Cache Loading - to avoid redundant fetching or computation, making sure the operation is only done once.

• Resource Intensive Operations - for fetching data or performing an operation that is expensive and should be done only once even if requested by multiple goroutines simultaneously.

• Rate Limiting - for limiting the rate at which a specific function is executed to avoid exceeding certain limits.

Sync’s Once Function

• provides a mechanism to ensure that a particular function is executed only once, regardless of the number of goroutines calling it.

• useful for lazy initialisation, one-time setup, or any scenario where a specific action needs to be performed exactly once in a concurrent program.

Use Cases:

• Lazy Initialisation - sync.Once can be used for lazy initialisation of resources that are expensive to create or set up. The initialisation function will be executed only when the resource is first accessed.

• Global Initialisation - When you have global variables or settings that need to be initialised before the application starts, sync.Once can ensure that the initialisation function is executed only once.

• Singleton Pattern - sync.Once can be used to implement a simple form of the singleton pattern, ensuring that an instance of an object is created only once.

• Resource Cleanup - sync.Once can be used to perform resource cleanup or teardown operations exactly once, ensuring that cleanup is not duplicated.

Singleflight and sync.Once might look like they both achieve the same purpose but that’s not the case.

• singleflight :

  • You want to limit the concurrency of a specific function to avoid redundant work.

  • You are dealing with scenarios where multiple goroutines may request the same data or perform similar work concurrently.

• sync.Once:

  • You want to ensure that a particular function is executed exactly once, regardless of the number of goroutines calling it.

  • You are dealing with scenarios like lazy initialization, global setup, or singleton pattern implementation.

Errgroup

• similar to sync.WaitGroup but where the tasks return errors that are propagated back to the waiter.

• useful when you have multiple operations that you want to wait for, but you also want to determine if they all completed successfully.

• errgroup is designed to manage a group of goroutines. You can add goroutines to the group, and the group will keep track of their execution. If any goroutine in the group returns an error, it cancels the entire group, causing all other goroutines to be signaled to stop. This helps to prevent goroutines from continuing to run when an error has occurred. The Wait method of errgroup.Group can be used to block until all goroutines in the group have completed their execution. The Wait method returns an error if any of the goroutines in the group returned an error. This allows you to handle errors centrally.

Use cases:

• Fan-Out Concurrency - When you need to perform multiple independent tasks concurrently (fan-out), and you want to wait for all of them to complete or return on the first error.

• Error Handling - When you want to handle errors collectively instead of checking errors from each goroutine individually.

• Context Cancellation - errgroup integrates well with the Go context package, allowing you to use context cancellation to stop the entire group if needed.

Bounded concurrency via buffered channels

• pattern where you use channels with a specified capacity (buffered channels) to control the level of concurrency.

• can be used to limit the number of goroutines actively working on a task, preventing unbounded resource consumption.

• useful in situations where you want to limit the number of simultaneous network requests, file I/O operations, or database queries to avoid overwhelming external resources.

Use cases:

• Rate Limiting - When you want to limit the rate at which certain operations are performed, such as API requests, file reads/writes, or database queries.

• Resource Constraints - When you need to prevent unbounded resource consumption by limiting the number of concurrent operations.

Let’s say we want to control the rate of our API requests by allowing only two requests to be processed concurrently. In the following example, the processRequest function simulates the processing of each request and we limit the concurrency to 2 using a buffered channel (ch) with a capacity of 2.

Weighted bounded concurrency

• in some cases, not all tasks are equally expensive. So instead of reasoning about the number of tasks we want to run concurrently, we come up with a "cost" for every task and acquire and release that cost from a semaphore.

• mechanism where tasks are assigned different weights, and the system ensures that the sum of weights of concurrently executing tasks does not exceed a certain limit.

• allows for fine-grained control over the level of concurrency based on the computational or resource intensity of each task.

Use cases:

• Resource Management - When tasks have varying resource requirements, and you want to ensure that resource-intensive tasks do not overwhelm the system.

• Fine-Grained Control - When you need fine-grained control over the level of concurrency based on the specific characteristics of each task.

• Adaptive Scaling - When the system needs to adaptively adjust the concurrency limit based on factors such as system load, resource availability, or external conditions.

In the above example, tasks are assigned weights based on their IDs (for illustration purposes). The worker goroutines read tasks from the tasks channel and process them, taking into account their weights. The concurrencyLimit ensures that the sum of weights of concurrently executing tasks does not exceed the specified limit.

Those concurrency mechanisms provide a way to allow you to build highly efficient and responsive systems. Whether handling complex distributed scenarios or managing parallel execution of tasks, Golang's concurrency features contribute to the development of robust and scalable applications in a concurrent world.

I have been following Dr Fei Fei Li’s research since my master's degree days and when I saw the news that she has just published her memoir, couldn’t be more excited to find out more about her days as a researcher.

Dr. Li is a renowned computer scientist who played a crucial role in shaping the evolution of contemporary artificial intelligence. Her new memoir presents her journey from her childhood years in China, navigating the hurdles of relocating to the United States during high school, to her eventual ascent as a prominent researcher and professor at Stanford, at the forefront of technological innovation.

In “The Worlds I See”, she presents her experiences and challenges as an immigrant, a woman in STEM, a teacher and a scientist, emphasising her profound passion for the work that has solidified her standing as a well-known thought leader in the field of AI.

Born in Chengdu, China, she moved to Parsippany, New Jersey in the early 1990s where she met Mr Sabella, her math teacher and her first American friend. She majored in physics and then went on to study at Princeton on a full scholarship, followed by a PhD at Caltech. Dr. Li completed her dissertation, titled "Visual Recognition: Computational Models and Human Psychophysics", under the primary supervision of Pietro Perona and secondary supervision of Christof Koch, where she put together ImageNet, a fourteen million-image dataset that enabled rapid advances in computer vision in the 2010s.

The birth of ImageNet

During an era when the predominant focus of AI research was on models and algorithms, Dr Li aimed to broaden and enhance the dataset available for training AI algorithms. She admits this was not an easy challenge as she recounts her struggles and efforts in encouraging other academics’ use of the dataset in the early years:

“We’d dedicated years of our lives to a data set that was orders of magnitude beyond anything that has ever existed, orchestrated an international competition to explore its capabilities, and, for all that, accomplished little more than simply reifying the status quo. If ImageNet was a bet, it was time to start wondering if we’d simply lost.“

ImageNet contains more than 20,000 categories where a typical category consists of several hundred images. Images have been annotated to indicate what objects are pictured and in at least one million of the images, bounding boxes are also provided.

As her husband Silvio mentioned “That’s another thing about hypotheses - sometimes they take time to win everyone over.“, and he was right. During one of the annual ImageNet Large Scale Visual Recognition Challenge (ILSVRC) where programs compete to correctly classify and detect objects and scenes, one of the contestants’ algorithms had the surprising results of an 85% accuracy. They had used a neural network, a concept talked about only decades earlier but considered “a dusty artifact, encased in glass and protected by velvet ropes.“ More precisely, the algorithm was based on a CNN (convolutional neural network), where each of its layers integrates further details into higher and higher levels of awareness until a real-world object comes fully into view.

A CNN behaves like a retina, “its outermost layer applies thousands of receptive fields to the pixels of a photograph, each turned to a unique, tiny pattern and activating when it encounters it. […] At this level of awareness, these filters could be reacting to anything […] Rather than arbitrarily deciding in advance which feature to look for, the authors allowed each of its hundreds of thousands of neurons to learn their own sensitivities from the training data, without manual intervention. […] Signals from those thousands of receptive fields travel deep into the network, merging and clustering into larger, clearer hints. Each new layer, operating at a more sophisticated level of perception than the last, responds when sensing something familiar - something it had been trained to recognize.“

The team that published the entry was a three-man team, made of Alex Krizhevsky (hence the name of the algorithm, AlexNet), Ilya Sutskever, and Geoffrey E. Hinton (the same person that has developed the backpropagation technique in the mid-80s).

The intersection of computer vision with other domains

A second-year grad student under Dr Li’s supervision, Andrej Karpathy’s thesis focused on the intersection of natural language processing with computer vision, investigating deep learning models suited for this task (Connecting images and natural language, 2016). The exploration started with the idea of having an algorithm that could describe an entire scene and not just being able to label whichever object appears in the background.

This posed a couple of challenges. The fundamental unit of a spoken language is a word and unlike a pixel, words convey distinct meaning, in isolation or as part of group of words. (e.g. “rock bottom“, “rock fragments“, “rock music“ etc)

Karpathy’s idea to solve the challenge would be to pair a CNN with an RNN. Another neural network, the RNN (recurrent neural network), born a couple of decades ago as well would start to gain attention. The model is a bi-directional artificial neural network, meaning that it allows the output from some nodes to affect subsequent input to the same nodes, and hence being able to infer basic properties of text.

“One to encode visual information and pair it with words, and the other to generate language. We’ll train our model on pairs of images and human-written description. […] the RNN generates each new word in the description conditionally, based on the words already in the sentence.“

And the birth of deep learning

A sense of eagerness gripped the field as the demand for more intensified: more deeper neural networks, more layers, more compute power to speed up the training process, more data, more of … everything. It wasn't merely the era of machine learning, but as with more and more layered networks being experimented with across labs, it was the era of deep learning.

In 2015, a retrospective was published on the ILSVRC competition, including the results conducted by Andrej estimating the human error rate when labelling a thousand images at around 5.1%. In 2014, GoogLeNet, Google’s neural network classifier, achieved an all-time-low error rate of just 6.67%, coming close to surpassing humans at the top.

Later that year, Deep Residual Network (ResNet) led by Kaiming He, a network with 152 layers presented a unique innovation: some of the layers could be bypassed during the training phase, allowing different images to direct their influence toward smaller subregions in the network.

Google later acquired DeepMind with the mission of working on AGI (artificial general intelligence), a form of AI that rather than just performing tasks like classification or translation, it can replicate all human cognitive capabilities.

In 2018, at the Google Brain conference, one of the topics was “neural architecture research“, a domain looking into the optimisation of the neural network’s architecture. Models are defined by a set of parameters which govern the trade-off between speed vs accuracy, memory vs efficiency etc, and finding a way to balance this trade-off is a complex task. One method to solve this would be to automate it, have a machine learning model design other machine learning models. But from a computational side, this would be a really expensive task.

More innovations followed, such as LLMs, transformers, “multimodal“ networks, which allowed models to provide solutions to more complex tasks.

Future challenges concerning AI policies

Another project Dr Li’s lab was working on was in the healthcare domain and its goal was to have an automated technique for ensuring that caregivers wash their hands consistently and thoroughly throughout hospital spaces. The aim was to have some type of technology that would blend into the background discreetly, keeping a silent watch and speaking up only when it sensed danger (what Dr Li called ambient intelligence). But the unexpected that followed was not only the complexity of the sheer technical task but more the implications of it from a human perspective. The technology was intended for patient safety, but it raised concerns when it came to performance reviews. This was Dr Li’s first encounter with an aspect of AI that would soon haunt the public’s mind: its capacity for surveillance.

Another concern towards AI would soon be raised by the emerging threat known as “adversarial attacks“, in which input is designed for the sole purpose of confusing an ML algorithm to counterintuitive or destructive ends. For instance, an input photo can be modified with subtle fluctuations in the colours of individual pixels in such a way that would trigger a cascade of failures with the NN. This would have enormous consequences in the self-driving field where for example, a stop sign could be misclassified, putting at risk our own safety.

In 2016, ProPublica investigations documented the widespread use of biased AI in helping lenders process loan applications and even in assisting judges in making decisions about parole. Similar reports discovered other questionable uses of technology in domains like pre-screening job applicants before being called for interviews.

“As scary as each of these issues was in isolation, they pointed toward a future that would be characterised by oversight, more inequality, and, in the wrong hands, possibly even a kind of looming, digital authoritarianism.“

Providing concrete examples of the initial processes and challenges in AI up to the present, Dr. Li contemplates the principles that should underpin AI in both academic and private sector contexts. Lopsided data sets, susceptibility to bias, adversarial influence, fairness, and privacy are all concerns that need to be addressed. This is another challenge Dr Li tries to solve as a Co-Director of the Stanford Institute for Human-Centered Artificial Intelligence.

A memoir of one woman’s curiosity and courage

In one of her talks, she mentions her friend, John Etchemendy, reading her first manuscript saying it’s no good and convincing her she needs to write a memoir which she was reluctant at first.

“Many AI scientists and technologists can write this book, but there are so many young women, young people, immigrants, people of all walks of life out there, who need to identify with the voice that is not a typical AI voice, and you have a journey that you need to share with these people.“

The challenges of being an immigrant and woman in STEM paired with the demands of having a career, a marriage and a motherhood are some of the hardships Dr. Li describes so heartedly. As an immigrant and woman in engineering, I’m really happy she listened to her friend’s advice as her book has become a cosy journey of reminiscing of my own struggles and the spark and passion that made me move forward.

Her book’s final chapter contains probably the most fitting message for all the immigrants scientists out there, “No matter who you are, or where you came from - you belong here“ so that we can build the future together.

The Worlds I See is an inspiring story of one woman’s curiosity and courage following her passion to become a scientist. Today, Dr. Li is a computer science professor and director of the Stanford Human-Centered AI which she founded while she also plays a role as a consultant to the Senate and House sub-committees and the Executive branch on matters concerning AI security and policies.

This week’s newsletter aims to approach the basics of Go concurrency by going through a couple of concepts:

• the difference between concurrency and parallelism

• how different concepts are done in the JVM (Java) vs Go

• how goroutines are scheduled

• ways of declaring goroutines

• ways of synchronising goroutines by going through the different sync package primitives

Concurrency vs Parallelism

“Concurrency is about dealing with lots of things at once. Parallelism is about doing lots of things at once.” — Rob Pike

Concurrency is a property of the code; parallelism is a property of the running program.

Concurrency is a semantic property of a program or system. Concurrency is when multiple tasks are in progress for overlapping periods of time. Concurrency is a conceptual property of a program or a system, it’s more about how the program or system has been designed. Long story short, concurrency happens when you have context switching between sequential tasks.

Using the same example as Kirill Bobrov uses in Grokking Concurrency, imagine that one cook is chopping salad while occasionally stirring the soup on the stove. He has to stop chopping, check the stove top, and then start chopping again, and repeat this process until everything is done.

As you can see, we only have one processing resource here, the chef, and his concurrency is mostly related to logistics; without concurrency, the chef has to wait until the soup on the stove is ready to chop the salad.

Parallelism is an implementation property. It resides on the hardware layer.

Parallelism is about multiple tasks or subtasks of the same task that literally run at the same time on a hardware with multiple computing resources like multi-core processor.

Back in the kitchen, now we have two chefs, one who can do stirring and one who can chop the salad. We’ve divided the work by having another processing resource, another chef.

Concurrency can be parallelised but concurrency does not imply parallelism. 
e.g. In a single-core CPU, you can have concurrency but not parallelism.

=> We don’t write parallel code, only concurrent code that we hope might be ran in parallel.

What are GoRoutines?

GoRoutines are light-weight execution threads integrated into Go’s runtime that run independently, along with the initialised functions.

Concurrency in Go vs Java

Scheduling

Go’s mechanism for hosting goroutines is an implementation of what’s called an M:N scheduler: which states that M number of goroutines can be distributed over N number of OS threads.

When a Go program starts => it is given a logical processor P for every virtual core => Every P is assigned an OS thread M => Every Go program is also given an initial G which is the path of execution for a Go program. OS threads are context-switched on and off a core, goroutines are context-switched on and off a M.

There are two run queues in the Go scheduler.

• Global Run Queue (GRQ)

• Local Run Queue (LRQ)

Each P is given given a LRQ that manages the goroutines assigned to be executed within the context of P. These goroutines take turn being context-switched on and off the M assigned to that P. GRQ is for goroutines that have not been assigned to a P yet.

When a goroutine is performing an asynchronous system call, P can swap the G off M and put in a different G for execution. However, when a goroutine is performing a synchronous system call, the OS thread is effectively blocked. Go scheduler will create a new thread to continue servicing the existing goroutines in the LRQ.

Go follows a model of concurrency called the fork-join model:

• fork — at any point in the program, a child branch of execution can be split off and run concurrently with its parent

• join — at some point in the future, the concurrent branches of execution will join back together

Ways of declaring goroutines

Synchronizing goroutines — via sync package

To make sure your goroutines execute before the main goroutine we need join points. These can be created via:

The WaitGroup primitive

used for waiting for a set of concurrent operations to complete when you either don’t care about the result of the concurrent operation, or you have other means of collecting their results.

Closures = a function value that references variables from outside its body.

With closures, we’d have to pass a copy of the variable into the closure so by the time a goroutine is run, it will be operating on the data from its iteration of the loop.

The Mutex primitive

provides a concurrent-safe way to express exclusive access to these shared resources.

How to: sync.Mutex interface with Lock() and Unlock() methods

Shares memory by creating a convention developers must follow to synchronise access to the memory.

The RWMutex primitive

same as Mutex but it provides a read/write lock. We can have a multiple number of readers holding a reader lock as long as nobody is holding a writer lock.

How to: sync.RWMutex interface with RLock() and RUnlock() methods

RWMutex can only be held by n readers at a time, or by a single writer

A way of choosing which primitive to use is emphasised in the following diagram:

Other basic primtives

The Cond Primitive

a rendezvous point for goroutines waiting for or announcing the occurrence of an event (=signal between 2 or more goroutines, has no info other than it happened).

How to: sync.NewCond(&sync.Mutex{}) with 2 methods:

1. Signal - notifies goroutines (runtime picks the one that has been waiting the longest) blocked on a Wait call that the condition has been triggered

2. Brodcast - sends signal to all waiting goroutines

Once

ensures that only one call to Do ever calls the function passed in

counts the no of times Do is called, not how many unique functions passed into Do are called

Pool

= concurrent-safe implementation of the object pool pattern.

How to:

1. Get interface - checks wether the are any available instances within the pool to return to the caller, and if not, call its New member variable to create one.

2. Put interface - to put the instance they were working with back in the pool

Uses cases:

• memory optimisations as instantiated objects are garbage collected.

• warming up a cache of pre-allocated objects for operations that must run as quickly as possible. (by trying to guard the host machine’s memory front-loading the time it takes to get a reference to another object)

Channels

Can be used to synchronise access of the memory and to communicate information between goroutines.

They support unidirectional flow of data:

• channel that can only read

• channel that can only send

Sending to/Receiving from a channel

Channels are blocking.

Any goroutine that attempts to write to a channel that is full will wait until the channel has been emptied.

Any goroutine that attempts to read from a channel that is empty will wait until at least one item is placed on it.

Closing a channel

Ranging over a channel

Closing a channel signals to multiple goroutines

Buffered Channels

= channels that are given a capacity when they’re instantiated.

Buffered channels are in-memory FIFO queue for concurrent processes to communicate over.

In the next newsletter, we’ll dive deeper into the result of channel operation given a channel’s state, how to design your code into channel owners and consumers and some other go concurrency patterns.