As reflection is an essential part of the learning process, this article is all about acknowledging that even with the right intentions and much dedication, you don’t always get it right. This is about accepting this reality and reminding myself that growth is a learning curve and despite setbacks we need to continue to take ownership.

What problem did we try to solve?

Maintaining accountability within data-driven systems along with keeping track of who did what and when within isn't just a security requirement—sometimes it's a crucial business necessity. Whether if it’s just for meeting compliance requirements, troubleshooting and understanding system behaviour, investigating production incidents or providing insights for post-mortem analysis, a robust audit logging solution serves as your system's source of truth.

But building such a solution comes with a lot of challenges in terms of scalability, performance and most of all, data integrity. In this article, we’ll explore what are the possible approaches to implement such a system that captures meaningful events, stores them reliably, and makes them easily accessible when you need them most. We'll cover the essential components that make up a production-grade audit logging solution and emphasise some lessons learnt from implementing a “tamper-proof solution”.

What is an audit log?

We want to know all database interactions that happened during a period of time, along with detailed information about each event. Those logs should be queryable and be able to answer questions like "who changed this value" or "which process caused this item to be removed." Audit logs record the occurrence of an event, the time at which it occurred, the responsible user or service, and the impacted entity.

A good audit logging system should not only record event data but also consider various usability criteria for the teams inspecting the logs (i.e the end users) and the future maintainers (e.g. engineers) of the system. The main requirements we’re going to work with:

• audit logs should be reliable - any database change should have a corresponding audit log entry.

• have no or minimal setup - should be easy to set up across the ecosystem and ideally require no or minimal code changes for new services.

• audit logs should be immutable - users and services should only have the ability to read data the logs and not update them.

• audit logs should be easily queryable (by service, action, date, user etc) - users must be able to efficiently search through the logs by applying any one or more filters.

• audit logs should be parsable - logs should also be easy to parse and work with.

Since the ecosystem we work with is only using DynamoDB as a storage option, we’ll design the audit log solution to work especially with this kind of database.

Solutions Considered

1. via the Transactional Outbox Pattern

One solution to implement the audit logs would be to emit an event with every database change. This event would contain all the necessary information to track the change (what got changed, who changed it etc). The consumer of those events would process the data and store it in a way that would be easily accessible and queryable by all devs.

In our current ecosystem, we use an internal library go-kit (similar to the open source one go-kit) to help us with the scaffolding and management of our golang microservices. This library also provides a standardised way for doing all db interactions. This approach would mean just modifying this library to emit an event when doing a CRUD operation. The advantage would be services don’t need to be aware of this logic and no logic would need to be updated.

The problem with this solution is the general dual-write issue where you want to make 2 write operations in one atomic transaction. If we want to update 2 different systems, persist data in the database and send an event to notify other systems, because these two systems aren't linked, we have no way to update both in a transactional fashion. When a microservice sends an event notification after a database update, these two operations should run atomically to ensure data consistency and reliability.

The way to solve this is usually through the Outbox Pattern. If we have a database that supports transactional updates, then we can use it to overcome the dual-write problem. In our case, we would need another table, say an “Outbox Table“, and on a DB CRUD operation we would 1. do the CRUD operation and 2. save the change we did in the outbox table (both as part of a transaction which DynamoDB supports). The records in the outbox table will then be read by some workers which will emit an event with the changed data. The events will be read by an “Auditor Service“ that will process them and finally store them in a queryable way.

This approach comes with a fair bit of complexities. We would have at least one other table (we can choose to use one single outbox table for all services or create a seperate outbox table for each service). We would introduce some latency due to the additional database write for every transaction (to outbox). There is also the consideration of how we’re going to manage the outbox table growth/cleanup. We would need to think how to coordinate the allocation of work between the worker services.

2. via the CDC Approach

A second solution would be relying on a Change Data Capture (CDC) approach. CDC is the process of identifying and capturing changes made to data in a database and then delivering those changes in real-time to a downstream process or system). The basic idea is that we will store our data to a storage that supports CDC. Then the CDC infrastructure will notify external systems of the changes. Fortunately for us, DynamoDB provides such an option, called DynamoDB Streams.

DynamoDB Streams captures a time-ordered sequence of item-level modifications in any DynamoDB table and stores this information in a log for up to 24 hours (can be extended to 7 days). Applications can access this log and view the data items as they appeared before and after they were modified, in near-real time.

At first glance this approach sounds like a good solution for our use case. No other resources are required to be setup. DynamoDB provides a CDC capability out of the box. Scalability and reliability are provided natively by AWS. The code what we would need to add would be minimal as we just need a service which reads the data from the dynamodb streams. So we went ahead with this.

What’s in a DynamoDB Stream?

Any dynamodb table with dynamodb streams enabled can have one dynamodb stream active. At any point in time, we can only have one active stream. A stream can have multiple shards. A shard can contain zero or more records. A record is the change that we want to be captured.

Depending on how the stream was set up when it was enabled, the shard record can contain different types of data (source):

• KEYS_ONLY - Only the key attributes of the modified item are written to the stream.

• NEW_IMAGE - The entire item, as it appears after it was modified, is written to the stream.

• OLD_IMAGE - The entire item, as it appeared before it was modified, is written to the stream.

• NEW_AND_OLD_IMAGES - Both the new and the old item images of the item are written to the stream.

We’re interested in the last type, NEW_AND_OLD_IMAGES, and so in our case, we’re expecting our record to look similar to the following:

Costs

DynamoDB Streams does not cost anything to enable. DynamoDB charges on the number of read requests made towards the table’s DynamoDB Stream. Each read request in terms of the stream is in the form of a GetRecords API call that can return up to 1000 records or 1 MB worth of records in the response, whichever is reached first. None of the other DynamoDB Stream APIs (source) For DynamoDB Streams the cost is $0.25 per million reads.

Approaches to Reading from Streams

DynamoDB streams can be processed in multiple ways. The most common approaches (source) are:

• trigger based via AWS Lambdas

• polling based with a standalone application that uses the Kinesis Client Library with the DynamoDB Streams Kinesis Adapter

• polling based with a standalone application connecting directly to the Dynamo Stream APIs through the AWS SDK

The AWS Lambdas Approach

Amazon DynamoDB comes integrated with AWS Lambda so that you can create triggers—pieces of code that automatically respond to events in DynamoDB Streams. Lambdas will poll the DynamoDB stream and, when they detect a new record, invoke your Lambda function and pass in one or more events.

Under the hood though, the AWS Lambda service will poll the stream for new records four times per second. (source) The polling interval cannot be changed, you can only change things like the batch size or the parallelization factor. (source)

Going with this approach would have a couple of advantages. The polling logic for data streams is automatically managed by Lambda functions, removing the need for manual implementation. Lambda functions provide built-in autoscaling capabilities, handling varying workloads efficiently without manual intervention. When using Lambda functions, you avoid the costs associated with making GetRecords calls to the streams, as this is handled by the AWS infrastructure.

On the other hand, there are significant additional costs involved, with an estimated expense of approximately $194 per month when handling 100 tables. The setup process requires extra effort, introducing changes to the deployment topology and adding complexity when establishing new data pipelines. Observability is limited as CloudWatch is the only available tool for monitoring and understanding what occurs within Lambda functions. Lambda functions must be designed to execute quickly, with minimal processing time, making them unsuitable for complex or long-running operations within the function logic.

Using the DynamoDB Streams Kinesis Adapter

The Amazon Kinesis Adapter is Amazon's recommended solution for consuming DynamoDB streams. However, a significant limitation is that this adapter is exclusively available in Java, making it unsuitable for non-Java applications. The Kinesis Client Library (KCL) offers a higher level of abstraction above the basic Kinesis Data Streams API, making the development process more straightforward and efficient. It provides robust batch processing capabilities, enabling efficient handling of multiple records simultaneously. The library includes automatic checkpointing functionality, maintaining accurate records of which data has been processed without requiring manual tracking.

The most significant limitation is that the DynamoDB Streams Kinesis Adapter is only implemented in Java and available through the awslabs/dynamodb-streams-kinesis-adapter repository, making it inaccessible for applications written in other programming languages.

Directly through the Dynamo Streams APIs using the AWS SDK

The third option is to have a service that constantly polls the streams for new records using the available aws sdk for go. This has a couple of implications, we need to keep track of an index to know which records have been processed so far and due to the replicated nature of our systems, we need an idempotent way to achieve this.

And some things worth considering:

• Streams have rate limits - the DescribeStream endpoint can be called at a maximum rate of 10 times per second; the GetRecords endpoint can be be called at 5 TPS per stream shard or 2MB/s, whichever comes first (source). Therefore, we cannot read more than 10 streams per sec.

• You cannot have more than one active stream per table.

• DynamoDB streams guarantee no duplicates and ordering of records.

Implemented Solution

Most of the time reinventing the wheel is the worst solution you can choose. But what do you do when the current company policies and/or ecosystem makes it hard to integrate these libraries or this new set up in particular? Do you use the existing wheel but build a completely special car to work with it, or build a new wheel that fits into your existing car? All of the sudden, crafting this new wheel sounds like the easiest option. So here we are, reinventing the wheel because…it looked easy enough.

Polling Processing Logic

Before going into the high-level overview on how we’re using the aws go sdk to continuously poll the Dynamo Streams APIs, a couple of key concepts on how DynamoDB Streams work (which took us quite a lot of searching, digging and calls with AWS Support):

• We can have 2 types of shards: OPEN and CLOSED.

• There is a one to one mapping between the dynamodb table partition and an open shard.

• A shard is OPEN if EndingSequenceNumber == nil and CLOSED otherwise.

• A shard can have (a lot of) successor shards (but every shard can only 1 child)

  • for CLOSED shards, you can iterate through all its successors until you get to the end denoted by NextShardIterator == nil.

  • for OPEN shards, NextShardIterator is never nil.

• Every shard (parent or child) can contain records and the SequenceNo is a pointer to that shard record.

We will store some shard/record information to be able to keep track of where we left off:

• a streams table - containing the table_name (which is the dynamodb table name we’re going to audit log), is_enabled (whether the dynamodb streams are currently enabled for it) and stream_arn (which is dynamodb stream identifier for the stream)

• a streams_shards table - containing the stream_arn, shard_id (the identifier of the stream shard), sequence_no (the sequence number of the given shard which we will use to know when in the shard we left off processing, similar to how an index works) and status (which for us can be open meaning the shard is still open on dynamodb side, closed meaning the shard is closed on dynamodb shard but we still have records left to process and closedAndProcessed meaning the shard is closed on dynamodb side and we have processed all records)

The way our processing logic will work is that given a stream_arn:

• we get a list of all the parent shards with their dynamodb statuses.

• if it's dynamodb CLOSED:

  • we check if we have already processed all records in it (where our status is closedAndProcessed) and if so skip

  • otherwise (where our status=closed) loop through all the successor shards until NextShardIterator == nil, process all the records in them and update shard status to closedAndProcessed.

• if OPEN:

  • loop through the shards using NextShardIterator.

Our main issue is what do we do with open shards, since NextShardIterator is never nil? When do we know when to stop? At first, we made the assumption that if we get to a couple of consecutive shards with 0 length records, we can assume the shard is caught up. Unfortunately, this is not the case (confirmed with AWS Support team). Even if we encounter multiple consecutive shards with 0 records, it's generally recommended to continue iterating through subsequent shards, as we can have data in later shards. Now you would say, that’s not a big deal, just continuously loop and if you see new records, save those. To point out, you might not need extra logic here for making sure you’re not processing duplicate records, remember we’re always storing the map of shard_id with sequence_no. So when you’re processing a shard that’s already been seen, we’re going to start from the given sequence_no.

The problem with this approach is that we don’t have just a couple of dozens of successor shards for open shards, we proved to have a couple of thousands. A couple of thousands with zero records in them. This is for a table with quite a reasonable (not very high) throughput. Overall, this overcomplicates our approach quite a bit as the time it takes to complete doing a full read looking for new changes increased significantly.

How is dynamodb streams spinning successor shards so chaotically? Why do we have so many with zero records in them? Why aren’t records written sequentially, why do we have large “gaps” (zero length record shards) in between shards with data? Would love to get an answer to those questions. We ended up not getting one so far.

Lessons Learnt

Even though every project starts with an MVP to prove the feasibility of the solution, as it started in this case, it might be worth to do a test on a production scale environment. If we would have faked production workload on local/staging to test the MVP, maybe we would have noticed some of the issues and started asking questions at an earlier stage.

This time, we’ll have a look at how Bluesky is setup and go through the novel approaches they use to solve certain challenges. As of December 2025, Bluesky has crossed the 20 million users mark while being maintained by no more than 15 engineers. Martin Kleppman, author of the “Designing Data Intensive Applications”, currently a technical advisor, published a paper1 outlining how Bluesky is functioning as a federated and decentralised social media platform. Gergely Orosz's has two good articles covering the company’s early days, culture and evolutionary architecture if you would like to check them out.2 In the meantime, let’s start with the basics.

Centralised, decentralised and federated

Decentralised networks have no central authority or server, and all participants (nodes) are equal and have the same capabilities. Data and control are distributed across all nodes. If one node fails, the network continues to function. One example of a fully decentralised network is Bitcoin - no single entity controls it, and all participants maintain a copy of the ledger.

Federated networks have multiple independent servers that can communicate with each other, where each server has its own administrator and user base. Users on different servers can interact with each other. Each server follows its own rules while adhering to a common protocol. For example, email providers (gmail, yahoo, and other providers) run their own servers but can communicate with each other via standard protocols.

A system can be both federated and decentralised. This is often called a federated decentralised network. The federation part provides standard protocols for communication, interoperability between different servers/instances and local administration and rules. The decentralisation part ensures no central authority controls the entire network, anyone can set up their own server/instance and there is no single point of failure.

In which category should X/Mastodon be part of?

X maintains complete centralised control over its platform. They have exclusive authority over your personal data, your identity, and content management (which includes both moderation policies and feed generation). While X has taken steps toward transparency by partially open-sourcing their algorithm3, users remain largely passive observers - able to understand parts of how the system works but with minimal ability to influence or modify its operation.

Mastodon is federated because different servers can communicate with each other using the ActivityPub protocol. It's partially decentralised because no one owns or controls the entire network and anyone can set up their own Mastodon server. If one server goes down, the rest of the network continues functioning. The key difference from purely decentralised systems (like Bitcoin) is that these networks maintain some organisational structure through federation while still preventing central control. They strike a balance between complete decentralisation and practical usability. One disadvantage though, is that in the same way there are huge email providers, there are big Mastodon servers. Your user identity becomes tied to a server and it might get challenging (if not impossible) to move to another server.

Bluesky architecture high-level overview

Bluesky uses a protocol called the AT Protocol (Authenticated Transfer Protocol)4 aka atproto, which works similar to email in some ways. Just like how email allows you to use Gmail but still message someone with a Yahoo account, Bluesky lets you choose different providers (called Personal Data Servers or PDS) to host your account, communicate with users on different servers, and take your account and data with you if you want to switch providers.

The key aspects through which Bluesky encourages decentralisation is that anyone can run their own server and connect to the network, users have data portability (can move their account between providers) and the protocol is open, meaning anyone can build compatible apps or services.

Let’s go through the main components that make the Bluesky’s architecture.

We have the PDS (Personal Data Server) which is the central part of the architecture. It hosts user repositories and data, manages user identity, orchestrates requests to other services, handles data sync and federation, can be self-hosted or provided (like Bluesky PDS). User Data Repository holds the User Identity (DID), which is the decentralised identifier for authentication. It handles the cryptographic authentication, signing and verification and holds the records structure like posts, follows, likes and profile data. In order to consume data, aggregate it to your liking and distribute it in real time, you have the Network Relays which index and collect content from PDSs and then use Firehose to distribute it so that you can process updates in real time. The firehose data is consumed by the Feed Generator (Discover) to create custom content feeds and by the Labeling Service (Ozone) to provide content moderation, tagging and content classification. The data from those two Opinionated Services is consumed by the App Views, which acts as the user-facing platform.

Interoperation and federability

The protocol uses a system called "Lexicons", these are like agreed-upon vocabularies that define how different types of data (posts, likes, follows, etc.) should be structured. This standardization allows different servers and clients to understand each other, similar to how different email clients all understand what an email header should look like.

Each server supports specific features by implementing "lexicons" - there are core ones (starting with com.atproto.*) that handle basic stuff like syncing user data, and social-focused ones (starting with app.bsky.*) that enable social media features.

Unlike the regular web, which shares HTML pages that determine both content and appearance, the AT Protocol only shares the underlying data in a structured format. This means apps can display the same information in their own unique ways, without needing to download and run someone else's display code (HTML/JS/CSS). It's like getting pure information that each app can then present however it wants. 5

The user data repository

A noteworthy feature is repositories ("repos"). Each user has their own personal repository that contains all their data - posts, follows, likes, etc. These repos are designed to be efficiently synced between servers, maintain a verifiable history of changes and allow users to take their entire social presence with them if they switch providers.

The AT Protocol (ATP) has some interesting technical characteristics that enable decentralisation. The core of ATP is built around something called "Self-Authenticating Data Structures". Every post, profile, and interaction is signed cryptographically, which means data can be verified as authentic regardless of which server it comes from, users truly own their identity through cryptographic keys and content can't be tampered with without detection.

An interesting technical choice is that ATP uses DIDs (Decentralised Identifiers) as the foundation for user identity, which are persistent identifiers that remain valid even if you change providers.

Every user has two identifiers:

• A human-readable handle (like username.bsky.social or custom domain)

• A permanent DID (Decentralised Identifier, like did:plc:xyz123...)

User Handle (e.g., alice.bsky.social)
       ↓
Resolves to DID (did:plc:xyz123...)
       ↓
DID looks up in PLC server
       ↓
Returns DID Document containing:
- Public key
- PDS server location
- Handle verification

Users can change their handle without losing their identity because the DID remains constant. The DID document points back to your handle, creating a verification loop. Any changes to the DID document must be signed with your private key and create a verifiable chain of updates. All updates require cryptographic signatures. Changes create a verifiable chain (like a "mini-blockchain"). Public data is self-verifiable. Third parties can detect manipulation.

Some limitations with the current approach are that it relies on centralised DNS system, there’s currently a single PLC server managed by Bluesky and most users' private keys are currently managed by Bluesky.

How decentralised and federated is Bluesky actually?

Bluesky doesn't use a traditional message-passing system like other federated networks (ActivityPub, XMPP, email). Instead, it uses a "shared heap" architecture where data exists in one global pool. There's no direct message-passing between hosts, which is a common federation pattern. Most users are currently hosted on Bluesky's PDS (Personal Data Server) instances. The chat/DMs service is completely centralised and most services rely on Bluesky's relay as a firehose.

The platform is not yet fully decentralised as the power is not evenly diffused throughout the system, most users run the Bluesky app, most developers work with Bluesky-defined application schemas (Lexicons), most self-hosted users run Bluesky's PDS software. Running a full-network relay requires significant resources (16TB of fast NVMe disk). Most DID PLC accounts don't have independently controlled PLC rotation keys configured.

One thing I would not debate is that much of the protocol and the network is reliant on Bluesky in material terms today. Almost everybody in the atproto network is hosted on a Bluesky PDS instance. Most self-hosted folks run the Bluesky PDS software. Most services use a Bluesky relay as a firehose. Most users run the Bluesky app. Most developers are working with the Bluesky-defined application schemas (Lexicons). - Bryan Newbold, Protocol Engineer at Bluesky 6

While Bluesky's underlying protocol (atproto) is designed to support decentralisation, in practice most of the network currently runs through Bluesky the company. Almost everyone uses Bluesky's servers, app, and relay services - making it functionally similar to Twitter/X today. The key difference is in the architecture.

It allows for "credible exit", users can leave without losing their data or connections. It Supports independent Lexicons (like the whtwnd.com blogging platform). Every major infrastructure component can be substituted without friction. Users can self-host their own instances. Lets users own their identity and data. The protocol is designed to prevent any single party from moderating the entire network.

So, unlike Twitter, Bluesky's infrastructure is built to allow users and developers to run their own independent services, migrate their data, and participate in the network without relying on Bluesky's systems. This capability exists even though relatively few are using it right now.

The early cron days

The original cron was designed by Ken Thompson in Unix V7 (around 1979) and was later rewritten by Dennis Ritchie. It used to work in a simple way. It would read the crontab files from /usr/lib/crontab, used a simple in memory table for job scheduling and had one main process that woke up every minute and determine if any commands must run at the current date and time, and if so, run them as the superuser, root.

This method presented a couple of challenges. It could easily overload the system if too many jobs ran simultaneously. It consumed resources every minute whether there was a job to be run or not. Performance degraded with more entries. It wasn’t suitable for large systems with many users. Memory usage grew linearly with number of jobs. There was no way to limit the number of jobs per user.

These limitations led to the development of more sophisticated cron implementations for multi-user capability.

The Franta-Maly Event List Approach

The Unix System V came with an upgraded version of cron based on the 1977 paper by W.R. Franta and Kurt Maly, An Efficient Data Structure for the Simulation Event Set. [1] In this paper, they presented a new event scheduling algorithm that improves on previously published algorithms. Robert Brown, a Purdue graduate student, noticed similarities between cron and discrete event simulators, leading him to implement the Franta-Maly event list manager. In 1979, Keith Williamson, a new graduate student, developed Brown's prototype into a production-ready multi-user cron service at Purdue.

The cron algorithm begins by searching for .crontab files in all users' home directories at startup. For each discovered crontab file, it calculates the next future execution time for each command and adds these commands to the Franta-Maly event list, along with their execution times and five-field time specifiers. The main loop then continuously examines the first task in the queue, calculating its future execution time. The system sleeps until that time, then wakes to verify the time and execute the queued task in the background using the original user's privileges. After execution, it recalculates the next runtime for that command and reinserts it into the event list accordingly.

The daemon implements two monitoring mechanisms for crontab file changes: it processes SIGHUP signals to trigger immediate rescans of modified crontab files, and it maintains scheduled wake-up checks at 30-minute intervals (on the hour and half-hour) to detect any crontab modifications.

The Franta-Maly event list (Two-Level algorithm) has a time complexity of O(√n) in the worst case scenario, driven by its two-tier data structure design and event insertion mechanism.

The Two-Level (TL) Algorithm Data Structure

The Two-Level (TL) algorithm proposed by Franta and Maly utilises a sophisticated data structure to efficiently manage event notices in discrete event simulations. The primary goal of this structure is to reduce the number of comparisons required to find the correct position to insert a new event notice, thus optimising the simulation's performance. The algorithm utilises a data structure inspired by indexed lists and balanced tree concepts (3-2 trees), to minimise the computational cost of scheduling and processing events.

The data structure consists of 3 layers:

• the index list layer - The TL algorithm uses an index list to divide the range of event times into a number of equal-sized intervals. Each element in the index list points to a "dummy key" which represents the boundary of an interval.

• the secondary key layer - The efficiency of the TL algorithm lies in the introduction of secondary keys within each interval. These keys, associated with the right boundary dummy key, point to sublists of event notices within the interval. This two-level indexing scheme enables more efficient searching and insertion operations.

• the event notices layer - Each interval is further divided into sublists, each headed by a secondary key. The sublists are designed to be balanced, meaning they don't grow excessively large. The maximum size of each sublist is controlled by nlim parameter.

The Insert Operation - Worst Case Complexity: O(√n)

Let’s say we want to insert a new event with t=1.3. The Franta-Maly insertion process follows a systematic sequence through the data structure's layers. First, we calculate the target interval by applying the formula i = [(t - lowerbound)/dt] to determine the precise interval containing the event time, which leads us to the corresponding dummy key entry in the index list.

Next, we traverse the sequence of secondary keys associated with the identified dummy key until locating the appropriate insertion point. For an event with time t=1.3, this involves scanning until we find it belongs between secondary keys 1.2 and 1.5.

The actual insertion occurs by placing the event notice in its correct time-ordered position within the identified sublist. In our example, the new event with t=1.3 is inserted between existing events with times 1.2 and 1.4, maintaining the temporal ordering of the sublist.

Finally, we execute a balance check to maintain the data structure's efficiency. The system verifies if the insertion has caused the sublist to exceed its maximum size (nlim). When the size limit is exceeded, the structure triggers one of two rebalancing operations: either transferring the last notice to an adjacent list or creating a new secondary key. If the sublist remains within size constraints, the insertion completes without requiring rebalancing.

The worst case complexity occurs when all n notices are associated with the same dummy key and the new event being inserted has a time smaller than all other notices in the structure. The complexity is calculated as: cw(n) = n/nlim + nlim, where nlim is optimally set to √n to minimize the worst case complexity.

The Search Operation - Worst Case Complexity: O(√n)

Let’s say we want to search for an event with t=1.3.

The search process in the Franta-Maly structure executes in three sequential phases. The first phase calculates the target interval using the formula i = [(t - lowerbound)/dt], which maps directly to the corresponding dummy key in the index list. This initial lookup operates in constant time O(1).

The second phase traverses the secondary key sequence within the identified interval until locating the appropriate sublist. For a target time t=1.3, the algorithm scans between secondary keys 1.2 and 1.5, ultimately selecting SK2's sublist. This traversal has a worst-case complexity of O(√n).

The final phase executes a linear search within the selected sublist to find the exact match or appropriate position. Since the maximum sublist size is constrained by nlim (√n), this search also operates with worst-case complexity O(√n). The combination of these three phases yields an overall worst-case complexity of O(√n), though practical performance often approaches constant time.

Calendar Queues (CQ)

A calendar queue is a priority queue data structure specifically designed for efficient handling of time-based events. It was introduced by Randy Brown in 1988 [2] and is particularly useful in discrete event simulation systems.

The queue is organized like a calendar, divided into "buckets" (like days or time slots). Each bucket represents a fixed-width time interval. Events within each bucket are sorted by their timestamp. The buckets are arranged in a circular array (wrapping around like a year calendar)

The data structure consists of 4 components:

• Buckets: An array representing days in a year, each element pointing to a sorted linked list of events scheduled for that day.

• Event Nodes: Contain information about the event, including its time (priority) and any associated data. These are organised into sorted linked lists within each bucket.

• Circular Calendar: Events can be scheduled up to a year in advance. Once the last day of the year is reached, the calendar wraps around to the first day, allowing for continuous event scheduling without data structure manipulation2.

• Event Dates: Each event node stores its scheduled date to handle events scheduled beyond a year. Events with future dates are skipped during dequeue operations until the calendar reaches their scheduled year

The above image shows how Brown’s Calendar Queue looks like, where the year starts at 12.0 and end at 16.0 and the bucket width is 0.5 time units.

How are events inserted?

Events are assigned to buckets using this formula:

bucket_number = floor(priority/bucket_width) % number_of_buckets

Let’s say we want to insert a new element 14.6 and we have the following initial state:

We have the current year that starts at 12.0 and ends at 16.0 (range of 4.0 units), with a bucket width = 0.5 time units and 8 buckets. We calculate the

bucket_number = floor(14.6/0.5)%8 = 29%8 = 5

The element belongs to bucket 5 and will be inserted into sorted position into the bucket’s linked list.

This insertion process maintains O(1) average complexity since bucket calculation is O(1), each bucket contains a small, roughly constant number of events, and searching within a bucket's linked list is therefore bounded by a constant.

How is bucket width calculated?

According to Brown's paper, the bucket width is calculated in a clever way to maintain efficiency:

• bucket width is adjusted each time the queue is copied onto a new calendar (when resizing)

• events are sampled to find the average separation:

  • For small queues (≤ 5 events): sample all events

  • For larger queues: sample 5 + size/10 events (up to max of 25 samples)

  • Dequeue these events, record their priorities, then re-insert them

• there’s a two-step average calculation:

  • First calculate initial average separation between sampled events

  • Then recalculate average using only separations smaller than twice the initial average (this handles bimodal distributions by ignoring large gaps)

• and finally, the bucket width = 3.0 * final average

The reasoning behind this method is that we aim to have 2-3 events per bucket on average. Multiplier of 3.0 was found experimentally to give best performance. Ignoring large gaps helps handle skewed/bimodal distributions. Using a sample rather than all events makes the calculation efficient.

When does resizing occur?

According to Brown's paper, resizing occurs based on queue size thresholds, more specifically:

• Growing (Doubling):

  • Resize occurs when queue size > 2 * number_of_buckets

  • The new calendar gets twice as many buckets

  • Example: If current size is 50 buckets, resize at 101 events

• Shrinking (Halving):

  • Resize occurs when queue size < number_of_buckets/2

  • The new calendar gets half as many buckets

  • Example: If current size is 50 buckets, resize at 24 events

During the resizing process, a new calendar array is created, a new bucket width is calculated using the sampling method, all events are copied to their new bucket positions and finally the switch to the new calendar is done.

Brown analyzed the impact of resizing, the average event is copied less than 2 times during its lifetime. In the worst case, the queue size oscillates between just above 2^n and just below 2^(n-1), and even in worst case, only about 3 events are copied per hold operation.

The key innovation of Brown's Calendar Queue was simplifying the two-level approach by using fixed-size buckets instead of dynamic sublists, eliminating the overflow problem through circular wrapping and providing automatic adaptation through bucket resizing. This made the Calendar Queue both simpler to implement and more efficient in practice compared to the TL algorithm, especially for larger queue sizes and varying event distributions.

FELT - An optimisation to CQ

One of the disadvantages of Brown’s CQs is that resize operation would involve creating a new CQ structure and then moving each item from the old CQ to the new CQ before discarding the old CQ. Hence, such resizes can be costly if the size of the queue is very large. And this is what Hui’s and Thng’s FELT (Far Future Event Leaf Tree) [3] structure is trying to solve.

FELT uses two different structures together to manage events:

1. A regular calendar queue for near-future events (Primary Tier)

2. A special tree structure for far-future events (FELT) (Secondary Tier)

Primary Tier (Calendar Queue)

Works like a normal calendar queue with a fixed maximum size (typically 4096 events)

The FELT primary Calendar Queue works like a normal calendar queue with dual thresholds. The UPPBOUND threshold, typically set at 4096 events, represents the maximum capacity of the primary queue. When exceeded, the system initiates a transfer operation, moving the oldest events into the secondary tree structure. This transfer specifically moves UPPBOUND minus LOWBOUND events, maintaining optimal queue size for performance.

Conversely, the LOWBOUND threshold, typically set at 256 events, establishes the minimum required event count in the primary queue. When the queue size drops below LOWBOUND, the system initiates a replenishment operation. During replenishment, the system identifies the leaf node containing events with the earliest timestamps in the secondary tier, transfers its entire contents to the primary queue, and removes the depleted leaf node. This process ensures the primary queue maintains sufficient events for efficient operation.

Secondary Tier (Tree Structure)

FELT acts as a semi sorted binary tree, where the left branch always holds events with higher priority (earlier timestamps) than the right branch. Only the leaf nodes store events. These events are kept in an unsorted linked list called NodeList. Parent nodes act as branching points. They do not store events themselves but maintain information about the minimum timestamp (MinTimeStamp) of all events in the branches below them. This helps efficiently route new events to the correct leaf node during insertion.

FELT is size-based, meaning it manages events based on the number of events stored in the primary CQ. This contrasts with time-based approaches like the Future Event Tree (FET) used in the Dynamic Lazy Calendar Queue (DLCQ), which can lead to inefficiencies when the CQ resizes.

                   [Parent Node(PN): MinTimeStamp(t) = X]
                         /                 \
    [Parent Node: MinTimeStamp = Y]    [Parent Node: MinTimeStamp = Z]
             /   \                                 /  \
 [Leaf Node(LN): MinTimeStamp = A] ... ... [Leaf Node: MinTimeStamp = B]
               |                                    |
[Unsorted Linked List of Events(ULLE)]          [(ULLE)]

How are events inserted?

FELT processes each new event through a two-stage decision process for optimal placement. Initially, the system examines the event's timestamp to determine whether it belongs in the near-future or far-future category. This classification happens by comparing the event's timestamp against a threshold calculated as the current simulation time plus one calendar period.

For near-future events, the system directs them into the primary calendar queue using standard calendar queue insertion operations. However, if this insertion causes the primary queue to exceed its size threshold, the system automatically transfers the oldest events from the primary queue into the secondary tree structure to maintain optimal queue performance.

Far-future events take a different path, entering the secondary tree structure where they navigate through internal nodes based on timestamp comparisons. The event travels down the tree, with each internal node guiding it left or right until it reaches an appropriate leaf node. Once at the leaf node, the event is simply appended to an unsorted list maintained at that node:

1. Start at the Top Node: The insertion process begins at the topmost node of the FELT structure (TopNode).

2. Check if Current Node is a Leaf Node:

   1. If the current node is a leaf node: The event is inserted into the unsorted linked list (NodeList) at that leaf node.

   2. If the current node is a parent node: Proceed to step 3.

3. Compare Event Timestamp with Right Child's MinTimeStamp:

   1. If the event's timestamp is greater than or equal to the MinTimeStamp of the right child node, AND the right child node is not NULL: Move to the right child node and repeat step 2.

   2. Otherwise: Proceed to step 4.

4. Check Left Child:

   1. If the left child node is not NULL: Move to the left child node and repeat step 2.

   2. Otherwise: Insert the event into the NodeList of the current node (which must be a leaf node).

5. Check for Node Splitting: After the event is inserted, the algorithm checks if the leaf node has exceeded its maximum capacity (MaxSize). When a leaf node has reached its maximum capacity (MaxSize) of events that it can store, which is set a priori, it will spawn two new leaf nodes and split NodeList into 2 and transfer the events into the left and right leaf nodes.

   1. If the node needs splitting: The splitNode function is called to create two new child leaf nodes and redistribute the events based on their timestamps.

                  [Parent Node(PN): MinTimeStamp(t) = 10]
                    /                             \
               [PN:t = 5]                     [PN:t = 15]
                /   \                             /   \
       [LN:t = 2]   [LN:t = 7]          [LN:t = 12]   [LN:t = 17]
          |              |                  |           |
     [(ULLE)]         [(ULLE)]          [(ULLE)]      [(ULLE)]

New Event (Timestamp = 8)

1. Start at TopNode (MinTimeStamp = 10).

2. 8 < 15 (Right Child's MinTimeStamp), so move to Right Child (MinTimeStamp = 12).

3. Left Child is NULL, so insert event into current node.

Result:
                  [Parent Node(PN): MinTimeStamp(t) = 8]
                    /                             \
               [PN:t = 5]                     [PN:t = 8]
                /   \                             /   \
       [LN:t = 2]   [LN:t = 7]          [LN:t = 8]   [LN:t = 17]
          |              |                  |           |
     [(ULLE)]         [(ULLE)]          [(ULLE)]      [(ULLE)]
                                            |   
                                 [New Event (Timestamp = 8)]

When it comes to complexity, the tree traversal is O(log n) in best case when tree is balanced and O(n) when it’s a completely skewed tree.

While the theoretical worst case is O(n), the actual performance in practice tends toward O(1) because: the size-based splitting keeps the tree reasonably balanced. The large maxSize (3840) means splits are rare, no sorting is required within leaf nodes and the split cost is bounded by a constant.

The authors show empirically that FELT maintains near O(1) performance even for large queue sizes and skewed event distributions, performing better than other approaches like DLCQ.

The Snoopy CQ

The Snoopy Calendar Queue, proposed by Tan, Kah Leong, and Li-Jin Thng. "SNOOPy Calendar Queue." [4], builds upon the concepts of the conventional Calendar Queue (CQ) and the Dynamic Calendar Queue (DCQ), addressing their limitations, particularly in handling skewed event distributions. SNOOPy CQ extends the calendar queue with statistical optimization mechanisms. Instead of using sampling for bucket width calculation like traditional CQ, it tracks operational costs and uses these statistics to optimize queue parameters. The name "SNOOPy" stands for Statistically eNhanced with Optimum Operating Parameter Calendar Queue. The core of SNOOPy CQ lies in its ability to dynamically adjust its operating parameter – the bucket width – based on the queue's performance statistics. This adaptive mechanism ensures consistent and efficient performance, achieving near-O(1) complexity even in challenging scenarios.

                       +-----------------------+
                       |      SNOOPy CQ        |
                       +-----------------------+
                                 |  ^
                                 |  |
                                 v  |
                       +-----------------------+
                       |  Enqueue/Dequeue      |
                       |  Operations           |
                       +-----------------------+
                                  |  ^
                                  |  |
                                  v  |
                       +-----------------------+
                       | Performance Statistics|
                       |     (CE, CD)          |
                       +-----------------------+
                                  |  ^
                                  |  |
                                  v  |
      +---------------+-----------------------+----------------+
      |               |                       |                |
      v               v                       v                v
  +-------------+-------------------+--------------+--------------+
  |Trigger Check|  Bucket Width     |Calendar Resize|   Cost       |
  | (CE, CD,    |  Optimization     |  (BW, NB)     |  Calculation |
  | Thresholds) |(Minimize CE + CD) |               |  (CE, CD)    |
  +-------------+-------------------+---------------+--------------+

1. Enqueue/Dequeue Operations: These are the fundamental operations performed on the SNOOPy CQ. Events are enqueued (added to the queue) based on their timestamps, and the event with the earliest timestamp is dequeued (removed from the queue) for processing.

2. Performance Statistics Collection: As events are enqueued and dequeued, SNOOPy CQ meticulously tracks two crucial performance metrics:

   1. Average Enqueue Cost (CE): This represents the average number of events traversed before an insertion can be made in a bucket's linked list.

   2. Average Dequeue Cost (CD): This signifies the average number of buckets that need to be examined before finding the event with the earliest timestamp.

3. Trigger Check: SNOOPy CQ continuously monitors the collected performance statistics (CE and CD). It employs multiple trigger mechanisms:

   1. Inherited from CQ and DCQ: These triggers are based on thresholds for CE and CD (e.g., if either exceeds a value of 2 or 3), as in the CQ and DCQ structures.

   2. Unique to SNOOPy CQ: Additional triggers are based on the relative difference between CE and CD over multiple slots (time intervals corresponding to a certain number of enqueue or dequeue operations). If the difference exceeds a predefined factor, it signals a need for adjustment.

4. Bucket Width Optimization: When any of the trigger conditions are met, indicating a suboptimal bucket width, SNOOPy CQ initiates its bucket width optimization process. The goal is to find a new bucket width (BW) that minimizes the total cost (CE + CD), keeping the number of buckets (NB) constant. SNOOPy CQ employs a novel approach to determine the optimum BW based on the relationship between CE and CD, as detailed in section 3.1 of "download.pdf".

5. Calendar Resize: Once the optimal bucket width is calculated, the SNOOPy CQ undergoes a resize operation. A new calendar queue is created with the calculated BW and the same NB as before. Events from the old calendar are then transferred to their appropriate buckets in the new calendar.

6. Cost Calculation: After each slot (a series of enqueue or dequeue operations), the average enqueue cost (CE) and average dequeue cost (CD) are recalculated based on the accumulated statistics from that slot. These updated costs are then used in subsequent trigger checks and optimization processes.

Conclusion

Starting from the simple Unix V7 cron implementation, each subsequent event scheduling structure addressed specific limitations of its predecessors.

The Franta-Maly event list introduced sophisticated two-level indexing, achieving O(√n) complexity through its hierarchical structure. Brown's Calendar Queue (CQ) then simplified this approach with fixed-size buckets and circular wrapping, providing O(1) average performance for uniform distributions.

FELT built upon CQ's foundation by introducing a dual-tier architecture that efficiently manages near and far-future events, effectively addressing the resize overhead issues. Its size-based approach, rather than time-based, provides consistent performance across varying event distributions.

SNOOPy CQ represents the current state-of-the-art, employing statistical optimisation to dynamically adjust bucket widths based on operational costs. This adaptive approach ensures robust O(1) performance even under skewed distributions.

Data Structures Comparison

Franta-Maly Structure (1977)

Primary Components

• Employs a two-level structure based on indexed lists and the balanced terminal node structure of 3-2 trees.

• Divides the range of event times into intervals, bounded by "dummy keys (notices)" pointed to by an index list.

• Each interval further divided into sublists, marked by secondary keys associated with the right boundary dummy key.

Key Operations Performance

Insertion (O(√n) worst case):

• Binary search index array to find appropriate terminal node

• Linear search within terminal node for insertion position

• Split terminal node if size exceeds 2√n

• Update index pointers if split occurs

Deletion (O(√n) worst case):

• Search index array for target terminal node

• Remove event from terminal node

• Merge underfull terminal nodes if necessary

• Update index pointers after merge

Terminal Node Management:

• Split Threshold: 2√n events

• Merge Threshold: ½√n events

• Maintains roughly balanced terminal node sizes

Performance Bottlenecks

Index Array Search:

• Binary search cost increases with structure size

• More frequent searches in skewed distributions

Terminal Node Operations:

• Linear search within nodes

• Split/merge overhead with uneven distributions

Maintenance Overhead:

• Index updates after node modifications

• Balance maintenance across terminal nodes

Advantages, Limitations and Selection Criteria

Advantages:

1. Bounded Operations:

   1. Guaranteed O(√n) worst case

   2. Predictable performance characteristics

2. Memory Locality:

   1. Events clustered in terminal nodes

   2. Efficient cache utilisation for sequential access

3. Flexible Structure:

   1. Adapts to varying event distributions

   2. Self-balancing capabilities

Consider Using When:

1. Predictable Performance is critical:

   1. Operations must have guaranteed bounds

   2. Worst-case performance matters more than average case

2. Memory Access Patterns are important:

   1. System benefits from clustered event storage

   2. Cache efficiency is a priority

Limitations:

1. Complex Implementation:

   1. Significant code complexity for node management

   2. Error-prone split/merge operations

2. Performance Issues:

   1. High overhead for small event sets

   2. Degraded performance with skewed distributions

3. Maintenance Overhead:

   1. Frequent index updates

   2. Complex rebalancing operations

Avoid Using When:

1. Simple Implementation is preferred:

   1. Calendar queue or modern alternatives offer simpler solutions

   2. Development time is constrained

2. Optimal Average Performance is required:

   1. Calendar queue provides better average-case performance

   2. Event distribution is relatively uniform

Calendar Queue (Brown, 1988)

Primary Components

Based on the analogy of a desk calendar, with each "day" represented by a sorted linked list. An array of pointers provides access to each day's linked list.

Bucket Array:

• Fixed-size array of bucket heads

• Each bucket contains sorted linked list of events

• Array size maintained as power of 2 for efficient modulo operations

• Circular structure with wrap-around for year boundaries

Year Mechanism:

• Year = NB * BW (Number of buckets * Bucket width)

• Tracks current year for wrap-around handling

• Enables infinite future time handling with finite structure

Key Operations Performance

Enqueue (O(n) worst case, O(1) average case):

• Calculate virtual bucket using timestamp

• Map to actual bucket using modulo

• Insert in sorted order within bucket

• O(n) complexity when all events are in one bucket

Dequeue (O(n) worst case, O(1) average case):

• Start from lastBucket

• Remove earliest event from current bucket

• Advance to next bucket if empty

• Handle year wrap-around when needed

• O(n) when searching empty buckets

Resize (O(n)):

• Triggered when queue size crosses thresholds:

• Upsize: size > 2 * nbuckets

• Downsize: size < nbuckets/2

• Creates new bucket array with doubled/halved size

• Recalculates bucket width by sampling

• Redistributes all events

Performance Bottlenecks

Bucket Width Impact:

• Too small: Events spread across too many buckets

• Too large: Too many events per bucket

• Optimal: 2-3 events per bucket near dequeue point

Distribution Effects:

• Uniform: Achieves O(1) performance

• Skewed: Performance degrades

• Bimodal: May require frequent resizing

Advantages, Limitations and Selection Criteria

Advantages:

1. Uniform Distributions - for network simulation, regular event patterns, predictable timings etc

2. Medium-sized Queues - for 1,000 to 10,000 events, stable event counts, regular dequeue patterns

Limitations:

1. Skewed Distributions - lead to performance degradation, frequent resizing, bucket imbalance

2. Very Large Queues - lead to resize overhead, memory locality issues, width calculation challenges

FELT (Far Future Event Leaf Tree, 2002)

Primary Components

Uses a Primary Tier (Calendar Queue) and a Secondary Tier (Tree Structure).

Primary Tier (Calendar Queue):

• Works like a normal calendar queue with a fixed maximum size (typically 4096 events)

• Has both an upper limit (UPPBOUND) and lower limit (LOWBOUND) for number of events

• When too full, moves events to the secondary tier

• When too empty, gets events back from the secondary tier

Secondary Tier (Tree Structure):

• Uses a binary tree where only leaf nodes store events

• Leaf nodes contain unsorted lists of events

• Parent nodes just help route to the right leaf node

• Each node keeps track of the smallest timestamp in its subtree

Key Operations Performance

When Adding a New Event:

1. Decision Point: System first checks if the event is near-future or far-future

2. Near-Future Events:

   1. Go directly into the calendar queue

   2. If this makes the queue too full, oldest events are moved to the tree

3. Far-Future Events:

   1. Go into the tree structure

   2. Get routed down the tree to the appropriate leaf node

   3. Added to that leaf node's unsorted list

Near-future events (Primary Calendar Queue): O(1) average case, as it uses standard calendar queue insertion
Far-future events (Secondary Tree): O(log n) to traverse the tree to the correct leaf node, then O(1) to append to unsorted list
Overall: O(1) for near events, O(log n) for far events

When Retrieving Events:

1. Always take events from the calendar queue first

2. When calendar queue gets too empty:

   1. Find the leaf node with earliest events in the tree

   2. Move those events to the calendar queue

   3. Continue processing from calendar queue

3. Primary Calendar Queue: O(1) average case

   1. When primary queue needs replenishment:

      1. Finding minimum leaf node: O(1) using tree's cached minimum timestamps

      2. Transferring batch of events: O(k) where k is transfer batch size

Overall: Amortized O(1) since transfer costs are spread across many operations

Performance Bottlenecks

Transfer Operations Impact:
When the primary queue hits its UPPBOUND or LOWBOUND thresholds, the transfer operations between tiers can cause significant overhead. These transfers require moving large batches of events between structures and can temporarily pause normal operations. While the cost is amortized, it can cause noticeable performance spikes.

Leaf Node Management:
When leaf nodes in the secondary tier get too full and need splitting, the operation requires scanning all events in the node to find the average timestamp, then redistributing them. This can be expensive for large leaf nodes and may happen frequently with certain event patterns.

Threshold Tuning:
Choosing optimal values for UPPBOUND and LOWBOUND is critical but challenging. Poor choices can lead to either too frequent transfers or inefficient primary queue operation, significantly impacting performance.

These bottlenecks become particularly apparent in scenarios with:
- Highly skewed temporal distributions
- Memory-constrained environments
- Large numbers of far-future events

Advantages, Limitations and Selection Criteria

Optimal Use Cases

1. Large Event Sets (10,000 events)

   1. Wide timestamp distribution

   2. Frequent far-future events

2. Variable Distributions:

   1. Skewed timestamp patterns

   2. Mixed temporal localities

   3. Dynamic workload patterns

Limitations

1. Implementation Complexity:

   1. Complex maintenance logic

   2. Debugging challenges

   3. Higher development overhead

2. Resource Requirements:

   1. Additional memory overhead

   2. Complex pointer management

   3. Transfer operation costs

SNOOPy Calendar Queue (2000)

Primary Components

The structure maintains traditional calendar queue elements plus statistical tracking:

Calendar Base Structure:

• Array of buckets containing sorted event lists

• Each bucket represents a time interval

• Year size = number of buckets × bucket width

Statistical Enhancement Layer:

• CE (Enqueue Cost): Tracks average number of events traversed during insertion

• CD (Dequeue Cost): Tracks average number of empty buckets traversed during dequeuing

• Moving average window over multiple operations

• Performance metrics for bucket width optimisation

Key Operations Performance

Enqueue Process (O(1) average case):

• Calculates target bucket using timestamp

• Inserts event in sorted order within bucket

• Updates enqueue cost statistics based on traversal length

• Triggers resize if statistical thresholds exceeded

Dequeue Process (O(1) average case):

• Retrieves earliest event from current bucket

• Advances through empty buckets if necessary

• Updates dequeue cost statistics

• May trigger optimisation based on performance metrics

Resize Operation (O(n) when restructuring):

• Triggered by statistical cost metrics rather than just size

• Uses performance history to calculate optimal bucket width

• Redistributes events across new structure

• Resets statistical counters after resize

Performance Bottlenecks

Statistical Overhead:

• Continuous cost tracking adds per-operation overhead

• Moving average calculations require additional computation

• Statistical threshold checks on every operation

Resize Decisions:

• More complex than traditional CQ resize triggers

• May require more frequent resizes initially while optimising

• Statistical calculation overhead during resize operations

Dynamic Adaptation:

• Initial period of suboptimal performance while gathering statistics

• May oscillate between different configurations before settling

• Additional memory references for statistical tracking

Advantages, Limitations and Selection Criteria

Advantages:

• Robust Performance:

  • Better handling of skewed distributions than traditional CQ

  • Self-tuning capability reduces manual parameter tuning

  • More consistent performance across varying workloads

• Adaptive Optimisation:

  • Automatically adjusts to changing event patterns

  • Reduces impact of poor initial parameter choices

  • Maintains efficiency under varying conditions

• Statistical Insight:

  • Provides performance metrics for monitoring

  • Enables informed optimization decisions

  • Can predict and prevent performance degradation

Best Used When:

• Event distribution is unknown or varies

• Long-running simulations justify optimization overhead

• Performance consistency is critical

• Manual tuning is impractical

Limitations:

• Implementation Complexity:

  • More complex than basic calendar queue

  • Additional overhead for statistical tracking

  • More difficult to debug and maintain

Avoid When:

• Simple, known event distributions

• Short-running simulations

• Memory or CPU resources are very constrained

• Implementation simplicity is priority

Resources

[1] Franta, W. R., and Kurt Maly. "An Efficient Data Structure for the Simulation Event Set." Communications of the ACM 20, no. 8 (August 1977): 596-602.

[2] Brown, Randy. "Calendar Queues: A Fast O(1) Priority Queue Implementation for the Simulation Event Set Problem." Communications of the ACM 31, no. 10 (October 1988): 1220-1227.

[3] Hui, Timothy Chee-Kin, and Ian Li-Jin Thng. "FELT: A Far Future Event List Structure Optimized for Calendar Queues." SIMULATION: Transactions of The Society for Modeling and Simulation International 78, no. 6 (June 2002): 343-361.

[4] Tan, Kah Leong, and Li-Jin Thng. "SNOOPy Calendar Queue." In Proceedings of the 2000 Winter Simulation Conference, edited by J. A. Joines, R. R. Barton, K. Kang, and P. A. Fishwick, 487-495. IEEE, 2000.

As Uber evolved from a simple ride-hailing app into a super-app offering food delivery, grocery shopping, and even car rentals, each new service built its own payment system. The result? A labyrinth of 70 different payment endpoints, each speaking its own language.

This fragmentation created real-world headaches. A user could pay for an immediate ride with Apple Pay but couldn't use it for a scheduled pickup. Popular payment methods like PIX (Brazil) and UPI (India) worked inconsistently across services. For any new payment feature or compliance requirement that needed to be integrated, engineers would have to spend countless more hours maintaining this new duplicate code.

The cost wasn't just technical. When the European Union mandated Strong Customer Authentication for transactions, requiring features like fingerprinting and biometric verification, it became clear: Uber needed to revolutionize its payment infrastructure or risk losing hundreds of millions in revenue.

Unified Checkout - One Gateway to Rule Them All

What Uber’s engineering team created is a payment universal translator, which they called Unified Checkout. The way it works is that instead of each service (rides, eats, rentals) handling payments differently, everything flows through one complex system.

This payments orchestration layer while it helps to process millions of transactions and to maintain consistency across all services, it keeps individual services blissfully unaware of the underlying complexity.

The architecture revolves around a chain of microservices that handle everything from payment profile preparation to risk evaluation. When a user starts a transaction, the platform exchanges transactional data with third-party processors, generates authentication tokens, and coordinates with risk systems to determine the optimal payment strategy. The system is able to adapt its approach based on the risk algorithms, whether if it’s for an authorization hold or an upfront charge.

One notable capability of the Unified Checkout is that it provides two distinct integration paths.

• The modular approach allows services to embed pre-built components into their existing checkout flows, maintaining their unique user experience while leveraging the unified payment infrastructure.

• For newer or smaller services, the hosted solution provides a complete out-of-the-box payment experience, accelerating time-to-market without compromising on functionality.

Another key advantage is the "Checkout Actions" framework - a stateless system that orchestrates complex payment flows, from two-factor authentication to identity challenges, this framework handles every possible payment scenario while maintaining a consistent user experience.

By treating each Uber service as an external company, the platform maintains strict boundaries between payment logic and business operations. This architectural decision forced the team to build a solution that could handle any payment scenario while remaining service-agnostic, and hence building a good foundation for future payment innovations.

The Business Impact

The Unified Checkout delivered a remarkable 3% increase in checkout conversion rate and a 4.5% improvement in session recovery, which meant saving hundreds of millions in additional annual revenue. These results were shown in an experiment on UberEats, where the team compared user behaviour between the old and new systems, leading to rapid adoption across all services.

This transformation dramatically simplified the payment landscape at Uber. Adding another payment method like Apple Pay doesn’t require a full implementation anymore, but only some configuration changes. Moving away from 70 endpoints to one single unified system streamlined operations and changed on a fundamental level how Uber does payments. Duplicate development work vanished. The launch of new payment methods accelerated dramatically. A consistent user experience across services built stronger trust with customers. Payment method decisions shifted from technical limitations to pure strategy, allowing Uber to optimise its offerings based on market needs rather than implementation constraints.

Maintenance costs plummeted as teams no longer needed to maintain multiple implementations. Compliance became more straightforward, error rates dropped significantly, and customer support issues decreased. And probably most importantly, new services could now launch faster, equipped with a complete payment infrastructure from day one.

Lessons Learnt

It comes a time in a company’s scaling journey where maintaining multiple systems becomes more expensive than building a unified platform. For Uber, this moment came when they faced 70 different payment endpoints across their services, each requiring separate maintenance and updates.

Uber's experience demonstrates how external pressures, like EU regulations, can serve as catalysts for positive change. What began as a compliance requirement evolved into a platform that generated hundreds of millions in additional revenue.

Looking toward the future, the key is creating modular systems that can evolve with changing requirements, while maintaining a clear separation between business logic and payment processing. This separation allows for easier updates and maintenance as regulatory requirements change or new payment methods emerge.

The most valuable lesson from Uber's journey is recognizing that technical debt isn't always a burden – it can be a signal that your system is ready for platformization. When multiple teams struggle with similar problems, it might be time to step back and consider a unified approach.

The Uber article is available here:
Unified Checkout: Streamlining Uber’s Payment Ecosystem

Similar stories you might enjoy 📚

• How Stripe’s document databases supported 99.999% uptime with zero-downtime data migrations

• How PayPal Scaled Kafka to 1.3 Trillion Messages a Day

• How Monzo vertically scaled ordered consumption using Kafka

The dual-write problem is a critical issue that arises when updating multiple systems simultaneously. This challenge significantly impacts data consistency, system reliability, and overall performance in modern software architectures.

At its core, the dual-write problem stems from the difficulty of ensuring that all write operations across different systems either succeed or fail together. Traditional database transactions provide atomicity within a single database, but this guarantee doesn't extend to multiple systems. This limitation creates a risk of partial updates, which can lead to data inconsistencies and compromise the integrity of the overall system.

Failure scenarios further complicate the dual-write problem. System crashes, network failures, and partial outages can occur at any point during the update process, potentially leaving systems out of sync. For instance, if a failure occurs after writing to one system but before updating another, the result is inconsistent data across the environment.

To illustrate the dual-write problem, consider a common scenario in microservice architectures: updating a database and sending a notification message. Ideally, these operations should be atomic to ensure data consistency and reliability. However, the lack of a unified transaction mechanism across different systems makes achieving this atomicity challenging.

• If the database update is successful but the event notification fails, the downstream service will not be aware of the change, and the system can enter an inconsistent state.

• If the database update fails but the event notification is sent, data could get corrupted, which might affect the reliability of the system.

As organisations continue to adopt microservices and distributed architectures, addressing the dual-write problem becomes increasingly crucial.

The Transactional Outbox Pattern

One way to solve this is through the Transactional Outbox pattern. If we have a database that supports transactional updates, then we can use it to overcome the dual-write problem.

Let’s take the example of a food delivery company system. When a customer places an order, several things need to happen: 1. The order details need to be saved in the database 2. A notification needs to be sent to the restaurant 3. The customer should receive an order confirmation 4. The order should be made available for driver assignment and so on.

The Order Service receives the order request. Then it starts a database transaction. It inserts the order details into the Orders table and in the same transaction, it also inserts messages into the Outbox table. These messages might include: a) OrderCreated event b) NotifyRestaurant message c) SendCustomerConfirmation message. The database transaction is committed, ensuring both the order and the outbox messages are saved atomically.

The Publisher Service runs continuously or at frequent intervals. It queries the Outbox table for unpublished messages. When it finds messages, it reads them and publishes them to the Message Broker. After successful publication, it marks the messages as processed or deletes them from the Outbox table.

The Message Broker receives the published messages from the Publisher Service and routes these messages to the appropriate queues or topics.

Then we have all the Downstream Services. A Restaurant Service might subscribe to the "OrderCreated" event. When received, it notifies the restaurant about the new order. A Notification Service might pick up the "SendCustomerConfirmation" message and sends an order confirmation to the customer. A Driver Assignment Service might listen for "OrderCreated" events to begin the process of assigning a driver.

When to use:

• You need to ensure consistency between your database and a message broker.

• You want to implement reliable messaging in a microservices architecture.

• You need to guarantee that a message is sent if and only if a database transaction commits.

Pros:

• Ensures consistency between database and messaging system.

• Works well with existing relational databases.

• Simpler to implement than full event sourcing.

• Doesn't require changing your entire architecture.

Cons:

• Doesn't provide a full audit trail like event sourcing.

• Requires polling or additional mechanisms to publish messages.

• Can add complexity to your database schema.

Change Data Capture

Store the state record to a data store that supports CDC. Then the CDC infrastructure will notify external systems of the changes. One such example would be DynamoDB, using its DynamoDB Streams.

Coming back to the example earlier, in a CDC approach, the Order Service receives the order request, it processes the order and inserts the order details into the Orders table in the database. The CDC tool continuously monitors the database's transaction log. When it detects a change in the Orders table (in this case, a new order insertion), it captures this change. The CDC tool transforms the captured change into a structured event message and publishes the captured change event to a Message Broker/Streams. The event contains details about the new order, such as order ID, customer ID, restaurant ID, and other relevant information. And same as above, multiple Downstream Services can then subscribe to the Order change events from the Message Broker.

When to use:

• You want to track changes in your database for replication or integration purposes.

• You need to sync data between different systems in near real-time.

• You want to implement event-driven architectures without changing your existing systems.

• You need to create data lakes or data warehouses from operational databases.

Pros:

• Non-intrusive - can be implemented without changing existing applications.

• Captures all changes, including those made outside your application.

• Can be used with legacy systems.

• Supports real-time data integration.

Cons:

• May not capture the full business context of changes.

• Can be complex to set up and manage.

• May impact database performance. CDC can add some overhead to the database, as it needs to read the transaction logs.

• Might require specific database features or additional tools.

• When the Orders table schema changes, downstream consumers need to be prepared to handle both old and new event formats.

Event Sourcing

Event sourcing is a pattern for storing data or changes to data as events in an append-only log. This represents a business decision to accept the request and start processing it.

The idea of event sourcing is that the event log is the source of truth, and all other databases are projections of the event log. Any write must first be made to the event log. After this write succeeds, one or more event handlers consume this new event and writes it to the other databases. (Cloud Native Patterns by Cornelia Davis (Manning Publications, 2019))

In event sourcing, every change to an entity's state is captured as a discrete, fine-grained event. These events are not merely notifications, each event represents a specific, atomic alteration to the entity's state.

As these state-changing events occur, they are published to interested parties and simultaneously persisted in a chronological log. This log becomes the source of truth for all the changes that have occurred to the entity over time. It's akin to a detailed historical ledger, capturing every nuance of the entity's evolution.

The power of event sourcing lies in its ability to reconstruct an entity state. Subscribers to this event log can process the sequence of events to derive the current state of an entity at any given point in time. By "replaying" the events in order, these subscribers can build a complete and accurate picture of the entity's current condition.

Take the design of an order processing system for a food delivery platform as an example. Event sourcing simplifies this process. All these events (orderCreated, orderAcceptedByRestaurant, orderPreparing etc.) are stored in an append-only log. Each event contains details like timestamp, entity_id, event_type and relevant data. The current state of an order can be reconstructed by replaying all events for that order from the beginning.

When to use:

• You need a complete audit trail of all changes.

• You want to reconstruct past states of your system.

• Your domain model is complex with many state transitions.

• You need to support temporal queries (e.g., "What was the state at time X?").

• You want to enable easy implementation of new features based on historical data.

Pros:

• Provides a full history of changes.

• Enables easy debugging and auditing.

• Allows for event replay and system recovery.

• Supports complex domain modelling.

Cons:

• Can be complex to implement and maintain.

• May require more storage.

• Querying current state can be slower (mitigated by snapshots).

Listen to Yourself

The Listen to Yourself pattern is a variation of event-driven architecture that aims to solve the problem of data consistency between a service's database and its published events. The service publishes events to a message broker as part of its operations. The service then subscribes to its own events from the message broker. Upon receiving its own event, the service updates its database.

The Order Service receives the order request. It prepares the order data and instead of writing directly to the database, it publishes an "OrderCreated" event to the Message Broker. This event contains all necessary order details (order ID, customer ID, restaurant ID, items, total, etc.). The Message Broker receives the "OrderCreated" event from the Order Service and makes the event available to all subscribed services, including the Order Service itself. The Order Service subscribes to its own "OrderCreated" events. When it receives the event it just published: a) It writes the order details to the Orders table in the database. b) It updates its internal state if necessary. Multiple Downstream Services also subscribe to the "OrderCreated" events.

How it's different:

• Unlike the Transactional Outbox pattern, it doesn't require a separate outbox table in the database.

• It's simpler than full Event Sourcing but still provides a level of eventual consistency.

• It differs from CDC in that the service actively publishes events, rather than relying on database log parsing.

Pros:

• Simplifies the architecture by using the message broker as the source of truth.

• Ensures eventual consistency between the published events and the database state.

• Can be easier to implement than some other patterns.

Cons:

• There's a potential for temporary inconsistency between the event being published and the database being updated.

• It may introduce additional latency due to the round-trip through the message broker.

• It can be more difficult to debug, as the flow of data is less straightforward.

Conclusion

We had a look at the different solutions that can be used to solve the dual-write problem. Which approach is used varies based on a given use case, specific requirements or any constraints of the system being designed or modified. To wrap things up, below is a comparison that might help you in making your decision:

Data Consistency:

• Event Sourcing: High consistency, events are the source of truth

• Transactional Outbox: Strong consistency between database and messaging

• Listen to Yourself: Eventual consistency, with potential for brief inconsistencies

• CDC: Eventual consistency, with potential for small delays

Complexity of Implementation:

• Event Sourcing: High - requires significant architectural changes

• Transactional Outbox: Moderate - requires additional table and message relay process

• Listen to Yourself: Low to Moderate - requires event publishing and self-subscription

• CDC: Moderate - requires CDC tool setup and configuration

Performance Impact:

• Event Sourcing: Can be high for read operations, depending on implementation

• Transactional Outbox: Low impact on write operations, additional load for message relay

• Listen to Yourself: Moderate - additional network round-trip for each operation

• CDC: Low to Moderate - some overhead on database for log reading

Scalability:

• Event Sourcing: Highly scalable, especially for write operations

• Transactional Outbox: Scalable, with separate scaling for application and message relay

• Listen to Yourself: Scalable, but may require careful management of self-subscriptions

• CDC: Highly scalable, especially for high-volume changes

Auditability:

• Event Sourcing: Excellent - full history of all changes

• Transactional Outbox: Good - outbox can serve as an audit log

• Listen to Yourself: Good - events can serve as an audit log

• CDC: Excellent - captures all database changes

Ability to Reconstruct Past States:

• Event Sourcing: Excellent - core feature of the pattern

• Transactional Outbox: Limited - not a primary feature

• Listen to Yourself: Possible but not straightforward

• CDC: Possible if all changes are retained, but not a primary feature

Impact on Existing Systems:

• Event Sourcing: High - requires fundamental architectural changes

• Transactional Outbox: Moderate - requires changes to data access layer

• Listen to Yourself: Moderate - requires changes to service logic

• CDC: Low - can often be implemented without changing existing applications

Handling of Schema Changes:

• Event Sourcing: Challenging - requires careful event versioning

• Transactional Outbox: Moderate - affects outbox table and consumers

• Listen to Yourself: Moderate - affects event structure and consumers

• CDC: Challenging - changes in source database schema need to be carefully managed

Real-time Data Integration:

• Event Sourcing: Excellent - events can be processed in real-time

• Transactional Outbox: Good - near real-time, depending on relay frequency

• Listen to Yourself: Excellent - events are processed immediately

• CDC: Very Good - near real-time, with minimal delay

Resilience to System Failures:

• Event Sourcing: High - can rebuild state from event log

• Transactional Outbox: Good - messages persist in outbox until processed

• Listen to Yourself: Moderate - depends on message broker reliability

• CDC: Good - can resume from last processed transaction log position

Latency:

• Event Sourcing: Can be high for complex queries on current state

• Transactional Outbox: Low for writes, potential delay in message processing

• Listen to Yourself: Moderate - additional network round-trip

• CDC: Low to Moderate - depends on CDC tool and configuration

In 2020, with the constant increase in their user base, the team decided to move from a Django monolith to a microservice architecture. This allowed for better scalability options, shorter waits for tests completion, faster deployment times and increased developer velocity. They wrote a great blog post on how they managed the transition. But this change also brought a lot of complexity with it.

The new architecture introduced other types of issues which we’re going to talk about in this article. We’re going to have a look at some of the common pitfalls and anti-patterns that appear in a microservice architecture, how Doordash solved them at a local level and how they’re attempting to mitigate them at a global level.

Common pitfalls with microservice architectures

1. Cascading Failure - A cascading failure happens when the failure of one service leads to the failure of other dependent services. This can cause a chain reaction, potentially bringing down the entire system.

DoorDash had an outage of this kind that they talked about in this blog post. In their case, the chain of failure started from a seemingly innocuous database maintenance, which increased the database latency. The latency then bubbled up to the upstream services, causing errors from timeouts and resource exhaustion. The increased error rates triggered a misconfigured circuit breaker, which stopped traffic between a lot of unrelated services, resulting in an outage with a wide blast radius.

Why it happens:

• Tight Coupling: Services are too dependent on each other, leading to a domino effect.

• Lack of Isolation: Failures in one service are not contained and propagate to others.

• Resource Exhaustion: Failure in a critical service can lead to resource exhaustion (e.g., CPU, memory) in dependent services.

2. Retry Storm - A retry storm occurs when a service failure triggers multiple retries from dependent services, which can overwhelm the failing service even more. Retries can worsen the issue when the downstream services are unavailable or slow, leading to work amplification, as each failed request is retried multiple times, which can cause an already degraded service to deteriorate further.

Why it happens:

• Uncontrolled Retries: Services automatically retry failed requests without considering the state of the failing service.

• Lack of Backoff: Retries happen too frequently, without appropriate delay, exacerbating the problem.

3. Death Spiral - A death spiral happens when the system starts to fail under load, and the attempts to handle the failure (like retries or additional resource allocation) further degrade the system’s performance, leading to a vicious cycle of deteriorating performance.

We’ve seen earlier how issues can spread vertically through different services dependant on one another. But they can also spread horizontally, inside the service cluster, from one node to another.

Why this happens:

• Resource Contention: Excessive retries and fallback operations consume more resources, reducing availability for normal operations.

• Unbalanced Load: Efforts to handle failures, like adding more instances, might lead to other parts of the system becoming overloaded.

4. Metastable Failure - Metastable failure refers to a state where the system appears to be stable under low load but becomes unstable and fails when the load increases beyond a certain threshold. Metastable failures occur in open systems with an uncontrolled source of load where a trigger causes the system to enter a bad state that persists even when the trigger is removed. When any of several potential triggers cause the system to enter a metastable state, a feedback loop sustains the failure, keeping the system in this state until a significant (usually manual) corrective action is applied.

For example, an initial trigger such as a surge in users, might cause one of the backend services to load shed and start responding to certain requests with 429 (rate limit). The callers will then retry their calls, but these retries, combined with requests from new users, overwhelm the backend service even more, leading to further load shedding. This results in a positive feedback loop where calls are continuously retried (along with new calls), get rate limited, and then retried again, perpetuating the cycle.

The above is called a Thundering Herd problem and is one example of a Metastable failure.

Why this happens:

• Insufficient Capacity: The system is not designed to handle peak loads, leading to instability.

• Hidden Bottlenecks: Bottlenecks that are not apparent under normal conditions become critical under high load.

• Latency Sensitivity: Small increases in latency can have a disproportionate effect on the system's stability.

• Request Retries: Retrying failed requests is widely used to mask transient issues. However, it also results in work amplification, which can lead to additional failures

Solutions at a local level

There are a couple of known techniques which are used to solve the above, some of which are being used by DoorDash as well:

• Exponential Backoff - by gradually increasing the delay between retries to reduce the load

• Retry Limits - by setting a maximum limit on the number of retries to prevent endless retry loops.

• Circuit Breakers - by activating and reducing the number of calls a dependent service makes to another one. Some examples of how circuit breakers are used by other companies:

  • Netflix's Hystrix library: Netflix developed Hystrix as a library to manage failures within their distributed system. When Hystrix identifies that a remote service is down or unresponsive, it activates the circuit breaker, stopping further requests from being sent to the problematic service. It can also offer alternative responses or retry the request after a designated period.

  • AWS ECS: ECS employs a circuit breaker pattern to automatically isolate failing services, thus preventing cascading failures within applications. This pattern also reduces latency and resource consumption during recovery from a service failure. It effectively ensures high availability and reliability for containerized applications.

  • SoundCloud: SoundCloud also uses circuit breaker to handle failures in their distributed architecture.

  

• Bulkheads - by isolating critical resources (like thread pools, database connections, or service instances) ensuring that failures or high resource usage in one microservice do not impact others sharing the same resources.

• Timeouts & Fallbacks - by defining the maximum acceptable duration for a microservice to respond to a request and preventing indefinite waiting and and resource consumption when a service is slow or unresponsive.

• Load Shedding - by prioritising and limiting incoming requests to prevent overload.

• Graceful Degradation - by minimising the amount of work that needs to be done, degrading non-essential functionalities or reduce service levels during times of high load, resource scarcity, or service unavailability. This can be achieved by determining essential functionalities that must remain operational even under degraded conditions, simplifying non-critical features to prioritise core functionalities and implementing fallback strategies to maintain basic service levels when primary functionalities are unavailable.

• Capacity Planning - by analysing historical data, user patterns, and trends to forecast future demand for each microservice and determining the scalability requirements based on expected growth, seasonal fluctuations, and special events.

Shortcomings of the local countermeasures

The current localised mechanisms face similar limitations:

• They rely on metrics specific to the individual service to determine its health. However, many types of failures involve interactions across multiple components, requiring a comprehensive system-wide perspective to effectively address overload conditions.

• They use general metrics and heuristics to assess the system health, which may lack precision. For instance, high latency alone may not indicate service overload; it could stem from slow downstream services.

• Their corrective actions are constrained. Operating within the local service, these mechanisms can only take local actions which may not be optimal for restoring the system health, as the root cause of the issue might lie elsewhere.

Solutions at a global level

One limitation of load shedding, circuit breakers, and graceful degradation is their narrow perspective within the system. These tools assess factors like their own resource usage, immediate dependencies, and incoming request volume. However, they lack the ability to adopt a global view of the entire system and decide based on that.

Aperture, an open-source system for reliability management, goes beyond local solutions by implementing a centralised load control. It provides a unified system for managing loads across multiple services during outages. It has 3 main components:

• Observe: Aperture gathers reliability metrics from each node and consolidates them using Prometheus.

• Analyse: A standalone Aperture controller continuously monitors these metrics and detects deviations from Service Level Objectives (SLOs).

• Actuate: Upon detecting anomalies, the Aperture controller triggers policies tailored to observed patterns and applies actions on each node, such as load shedding or distributed rate limiting.

Aperture utilizes YAML-based policies that guides its actions during system disruptions. When an alert is triggered, Aperture automatically executes actions based on these configured policies. Some of the actions it offers include distributed rate limiting and concurrency control (also known as load shedding). By maintaining a centralized oversight and control of the entire system, Aperture enables various strategies to mitigate outages. For instance, it can be configured to implement a policy that throttles traffic to an upstream service when a downstream service is overwhelmed, preventing excessive requests from reaching the problematic area and thereby enhancing system responsiveness and cost-efficiency.

DoorDash deployed Aperture within one of their core services and conducted artificial request simulations to evaluate its performance. They discovered that Aperture effectively operated as a robust and user-friendly global rate limiter and load shedding solution, providing a concurrency limiting algorithm which minimises the impact of unexpected load or latency.

The Aperture blog also provides good use cases of how their solution can be used to solve production problems:

• Protecting PostgreSQL with Adaptive Rate Limiting

• Balancing Cost and Efficiency in Mistral with Concurrency Scheduling

• Squeezing Water from Stone: Managing OpenAI Rate Limits with Request Prioritization

Conclusion

We had a look into the different pitfalls that arise in distributed systems, what are the triggers that might cause them, what are some of the localised solutions to prevent them from happening and what can be done at a more global level.

In the next articles, we’re going to take a deeper look into how Aperture works, how we can configure it and how to define a set of policies for a given use case.

How and why QR codes came into being

Masahiro Hara, a Japanese engineer who worked at Denso Wave then, invented the QR Code (or Quick Response codes) in 1994 to meet the need for a more efficient and versatile method of tracking automotive parts during manufacturing. Hara and his team developed QR codes as an improvement over traditional barcodes.

QR Codes vs Barcodes

Traditional barcodes could only store a limited amount of information horizontally, while QR codes can store much more data both horizontally and vertically. A barcode can store 1,556 bytes, whereas a QR Code can store up to 7,089 numeric characters, 4,296 alphanumeric characters, 2,953 bytes, and 1,817 kanji characters.

Components

And now let’s get into the fun part, what do all those black and white squares represent? QR Codes consist of black and white elements called modules, where black represents 1 and white 0. These modules can be combined into meaningful binary messages.

Before diving into how we encode/decode the information, let’s first analyse what the main components of the QR code are:

Function Patterns

1. Quiet Zone - helps the scanner to distinguish between the QR code and its surroundings, making it easier to detect the QR Code.

2. Position Pattern - located in 3 of the QR code’s corners, it allows the scanner to quickly recognise the code orientation. This means the code can be scanned at any angle.

3. Separator - used to keep space between the position pattern and the other information, helping the scanner to identify the position pattern.

   

4. Alignment Pattern - used for correcting distortion by straightening the code on a curved surface. On higher versions, QR codes tend to have more alignment patterns. Ver40 which is the largest QR code consists of 46 alignment patterns. Ver1 doesn’t require any alignment patterns.

5. Timing Pattern - helps the scanner to determine the width of a single module.

Encoding Region

1. Version Information - there are 40 versions of QR codes, from 1 up to 40; this region specifies which version is being used.

2. Format Information - contains information such as error correction level, mask pattern etc.

3. Data and error correction keys - the remaining available code is used for storing the data along with the error correction data. The correction data is done by the Reed-Solomon codes which allows the QR code to still be scanned if some information is corrupted/not visible etc.

Putting everything together:

How does versioning work?

There are 40 versions of the conventional QR code. Each version has a different number of data modules, commencing with Version 1 (21 × 21 modules) up to Version 40 (177 × 177 modules).

With a new higher version, another 4 additional modules per side are added.

How would we know which version we need to use?

QRCode.com provides a comprehensive list of versions given the type of input data you want to use (numeric, alphanumeric, binary, kanji) and the correction level.

Let’s say I want to create a QR Code for the text “The Crafty Dev“ which means we’re only interested in a 14 letter alphanumeric input. And let’s suppose we’re going to stick the QR code in various places outside, so we need a pretty high error correction capability (in case it rains or it gets stained etc). We need to find a figure in the QRCode versioning table, that is 14 or over and with an H correction level, so a good version for us to use would be version 2.

The Error Correction Level

The QR code error correction is based on the Reed-Solomon algorithm which works in a similar way to the parity error correction - it adds extra bits to the data so that errors can be corrected.

Fun Fact

The Voyager spacecraft uses the Reed-Solomon error correction to ensure that data transmitted from the outer reaches of the solar system can be reconstructed even if it encounters interference, which is highly probable. Given that signals from Voyager 1 can take up to 20 hours to reach Earth, employing an ARQ (Automatic Repeat reQuest) mechanism to request retransmission in case of errors would be highly impractical. (source)

The way Reed-Solomon does encoding deserves a completely different article so I’ll link it here once it’s published, so stay tuned.

Unmasking

Now comes the crazy part. Let’s say we have a QR code and we want to manually decode it. Where do we start? The first thing we’ll need to look at is the Format Information section.

The first 2 modules (colored red in the picture above) will give you the error correction level. And the next 3 (colored green) will give you the mask pattern.

This means we have a 00 error correction level, which means high, 30% of data bytes can be restored; and a 111 masking pattern.

The QR codes are usually masked, so we’ll have to take off the mask first. We put the coordinates of the current bit into the formula, and if the result is 0, we use the opposite bit of that. After removing the mask on the original QR code, we’ll get the following unmasked QR code:

Decoding

Now, finally getting into the decoding part. The way a QR code is read is from bottom right to top left following the path shown below. The first 4 bits give us the encoding and the next 8 the length of the message.

This means, for our QR code, we would have a 0100 encoding, which is a byte encoding and a 00001110 message length which is 14. Byte encoding tells us we’re expecting blocks of 8 bits and that we're expecting 14 of them.

We’re going to continue splitting our QR code into blocks of 8 bits and following the pattern of interleaved blocks below, we should have something similar to the right picture below:

We can see we have indeed 14 blocks. The last block containing the 4 bits of white modules is the End of message (Terminator) representing the end of our message. Now that we have our blocks splitted we can start decoding each character. We do this by summing up all the turned on (black) bits and converting the decimal number obtained into its specific character using an ASCII Table.

This will give us the following:

64+16+4 = 84 representing T
64+32+8 = 104 representing h
64+32+4+1 = 101 representing e
32 representing " "
64+2+1 = 67 representing C
64+32+14+4 = 114 representing r
64+32+1 = 97 representing a
64+32+4+2 = 102 representing f
64+32+16+4 = 116 representing t
64+32+16+8+1 = 121 representing y
32 representing " "
64+4 = 68 representing D
64+32+4+1 = 101 representing e
64+32+16+4+2 = representing v

We finally decoded our QR code which says “The Crafty Dev“ 🙌 🙌 🙌 .

The rest of the QR code message is part of the error correction data which we’ll have a look at in the next article, where we’ll talk about how QR Codes do error correction while we’ll also dive into how the Reed-Solomon algorithm works.

Conclusion

In this article we had a general look at QR Codes, their main components, what unmasking is and how it works, and finally how to decode the QR Code message.

Stay tuned, as in the upcoming articles we'll dive deeper into:

1. On Error Correction: From Hamming to Reed-Solomon

2. QR Code Models

See you soon 👋

The basic principle of LFU caching is to remove the least frequently used items first when the cache reaches its capacity limit. This means that the cache keeps track of the frequency with which each item is accessed, and when it needs to make space for a new item, it removes the item that has been accessed the least frequently.

LFU Cache Design

To implement a Least Frequently Used (LFU) cache solution we need to design a data structure that would support the following operations:

• LFUCache(int capacity) - initialises the object with the capacity of the data structure.

• int get(int key) - gets the value of the key if the key exists in the cache. Otherwise, returns -1.

• void put(int key, int value) - updates the value of the key if present, or inserts the key if not already present. When the cache reaches its capacity, it should invalidate and remove the least frequently used key before inserting a new item. For this problem, when there is a tie (i.e., two or more keys with the same frequency), the least recently used key would be invalidated.

The frequency for a key is incremented whenever a get or a put operation is called on the element.

Let’s run through an example to better understand how this would work:

1. Let’s say we have a cache with a capacity of 2 LFU(2) and we call put(1,1)
→ Our LFU will become {1,1,1} as we add the key 1 with value 1 which currently has a frequency of 1

2. Then we call put(2,2) → Our LFU will become {2,2,1},{1,1,1}. We added a new element with key 2 and value 2 which currently has a frequency of 1. Since elements with key 1 and 2 have the same frequency of 1, we add the most recently used element (the one with key 2) to the front of the list.

3. Then we call get(1). Our LFU will return 1 and the new LFU will be {1,1,2}, {2,2,1}. This is because we increase the frequency of element with key 2 to 2 (due to the read operation) and we move it to the front of the list because it has higher frequency than the other element.

4. Then we call put(3, 3). Our LFU will become {1,1,2}, {3,3,1}. A new element comes in but our cache is at full capacity so we need to remove an element. We remove from the end of the list (as this is where the element with the least frequency and least recency is stored), so {2,2,1} gets removed. We then add the new element with a frequency of 1, {3,3,1}.

5. We call get(2). We return -1 as the element no longer exists in the cache.

6. We call get(3). We return 3 and the LFU becomes {3,3,2}, {1,1,2}. We increase the frequency of 3 and move it before key 1 as even though they have the same frequency, key 3 has been accessed more recently.

7. We call put(4,4). The LFU becomes {3,3,2}, {4,4,1}. New element comes in, cache is full, we remove the last element with key 1 and add the new element with key 4.

8. We call get(1). We return -1 as element is no longer present in the cache.

9. We call get(3). We return 3 and our LFU becomes {3,3,3}, {4,4,1}. The read operation increases the frequency of element at key 3 to 3.

10. We call get(4). We return 4 and our LFU becomes {3,3,3}, {4,4,2}.

Possible Implementations

• Using a brute force approach, we could have an array with the size of the cache. Each element in the array would store the item along with its frequency and last access time. When we get an element, we’d have to iterate through the array to find our element and update the item with the incremented frequency value. This means getting an element would be on average O(n). To put an item when the cache is full, we’d have to look for an item to evict. So we’ll iterate through the whole array and look for the item with the least frequency and oldest access time. This would take O(n).

• Using a min-heap data structure would be an improvement - each node will hold the cached element along with its frequency. This makes it easy to remove an element when the cache is full as the least recently and frequently used element will be a the top. The problem comes when inserting or reading from the cache, as any operation will mean the min-heap will need to be rebalanced which will take O(logn).

Solution in constant time

To implement an LFU that uses constant time for pushing and getting items, we’re going to need 2 data structures: a key map and a frequency map.

• Key Map - stores the key of the item and the item node containing the key, value and frequency.

• Frequency Map - stores the frequency as key and a doubly linked list of all the cached items having that frequency as value. The items in the doubly linked list are sorted by recency such that most recently accessed items will be at the head of the list and the least recently used at the tail. Due to the nature of the doubly linked list, removing, adding and updating a cached item will take O(1).

Given those two data structures, the cache operations would have the following logic:

• get operation for an item X:

  • check that X is in our key map. If it doesn’t exist return -1. Otherwise get the frequency from the value node.

  • for the frequency we’ve obtained above, get the doubly linked list from the frequency map. Delete the node from the doubly linked list. Create a new node with the incremented frequency and added it to the frequency map at the head of the doubly linked list.

    
    Following the above instructions, the implementation for the get operation should look similar to this:

• put operation for an item X:

  • if the item already exists in the cache, it returns the node and updates the frequency (hence, we call getNode which does all of these)

  • if the item doesn’t exist in the cache:

    • if the cache is full - we remove the least recently and frequency used item, which should be the tail of the doubly linked list value in the frequency map with the minimum frequency.

    • then we add the item at the head of the doubly linked list with has the frequency of 1

    
    Considering the above, the implementation for the put operation would look like:

Putting everything together, the code should look like this.

Conclusion

In this article we had a look on how we would go about implementing an LFU.

To learn more about caching, have a look at the following articles:

• Mastering Cache Management: Eviction, Invalidation, and Optimal Data Access Strategies

Or stay tuned as in the upcoming articles, we'll dive deeper into:

• Types of Caches: Exploring different caching mechanisms like in-memory, distributed, and CDN caching.

• Deployment Strategies: Understanding how to integrate caching into your architecture.

• Metrics: How to monitor cache performance for optimal results.

• Implementation Solutions: Practical examples with popular caching tools and libraries.

When to use it :

• for frequently accessed data - e.g. website assets, session data, popular content

• for expensive operations - database queries (if the same complex query is frequently executed), API calls (e.g. weather data, currency conversions), compute intensive calculations

• for times when speed is paramount - high-traffic websites (reduce strain on the backend by serving frequent requests), search suggestion (autocomplete features often use a cache to display results instantly as the user types)

When accessing data from a cache, there's a best-case and a fallback scenario. A cache hit means the data was found in the cache, leading to a speedy response. A cache miss means the data wasn't present, so the system will need to fetch it from the slower main storage.

What to consider:

• Amdahl’s Law - “the overall performance improvement gained by optimizing a single part of a system is limited by the fraction of time that the improved part is actually used”

  • Diminishing Returns: Cache hit rates matter immensely. The initial gains from adding a cache are often dramatic. But, according to Amdahl's Law, as you reach 90, 95, 98% hit rates, the effort to squeeze more performance decreases in effectiveness.

• Pareto Distribution - “also known as the “80-20 rule” stating that 80% of outcomes are due to 20% of causes“

  • Cache Misses Are Not Evenly Distributed: In many systems, some data is far more popular than others. Picture the top 10% of products on an e-commerce site getting 60% of the traffic. This skewed access pattern aligns with the Pareto principle. To get the caching wins, you need to first analyse your data to see if the access patterns truly exhibit Pareto-like characteristics.

Cache Benefits

• Faster Access and Reduced Latency - serving data from in-memory stores vs. databases or external APIs offers massive latency gains.

• Bandwidth Optimisation - when external API calls or media assets are cached, you can potentially decrease bandwidth usage.

• Reduced Database Costs - databases are often a significant cost factor. Offloading queries to a cache can directly lower your database-related expenses.

• Improved Throughput - when a cache satisfies a request, your main database or origin servers are untouched. This allows your system to handle more concurrent users without overwhelming core resources.

• Handle Spikes - sudden surges that would cripple an uncached system can often be served smoothly when a cache absorbs the brunt of those requests.

Cache Eviction Policies

A caching eviction policy determines how a cache handles the data eviction when the cache is full. When an item is requested from the cache, it may result in a miss, which means data needs to be fetched from the database and the cache needs to be updated. But before doing that, an item needs to be removed. The following section describes several strategies on how to choose which item to pick.

• Queue-Based Policies

  • FIFO (First-In-First-Out)

    • Process: The first in/oldest item gets removed to make space for the new item, which is then added to the end.

    • Pros:

      • Simplicity: Very easy to implement and understand.

      • Fairness (Of Sorts): Prevents any item from 'hogging' the cache indefinitely if there's a steady stream of new data.

    • Cons:

      • Inefficient for Many Workloads: Real-world data access patterns often aren't purely FIFO order. An item used constantly might get evicted simply because it was added a while ago, leading to repetitive refetches of the same thing.

      • Ignores Popularity: A popular item accessed right after eviction will need to be fetched again, even though there may be other rarely-used data in the cache.

  

  • LIFO (Last-In-First-Out)

    • Process: The last in/newest item gets removed to make space for the new item, which is then added to the end.

    • Pros: Same as FIFO

    • Cons: Same as FIFO

• Recency-Based Policies

  • LRU (Least Recently Used)

    • Process: The basic principle of LRU caching is to remove the least recently used items first when the cache reaches its capacity limit. This means that the cache keeps track of the order in which items are accessed, and when it needs to make space for a new item, it removes the least recently accessed item.

    • Pros:

      • Effective use of cache space: LRU caching maximizes the utilization of cache space by keeping frequently accessed items in memory.

      • Good performance: LRU caching performs well in scenarios where there is a high degree of temporal locality, such as web servers and databases.

      • Simple implementation: The basic LRU caching algorithm is relatively simple to implement and understand.

    • Cons:

      • Overhead: LRU caching requires additional data structures to keep track of access history, which can introduce overhead in terms of memory and computational resources.

      • Not suitable for all access patterns: LRU caching may not perform optimally in scenarios with non-temporal access patterns or where access frequencies vary widely.

    • Possible implementation:

      • An LRU cache is often implemented by pairing a doubly linked list with a hash map. The linked list is set up with the most-recently used item at the head of the list and the least-recently used item at the tail.

      • When we get a request, we first look up the item in our hash map:

        • if it’s there, it’s a cache hit:

          • we use the hash map to quickly find the corresponding linked list node.

          • we then move the item's linked list node to the head of the linked list, since it's now the most recently used (so it shouldn't get evicted any time soon).

        • if it’s not there, it’s a cache miss:

          • if the cache is full - we evit the least recently used item which should be at the end of the queue.

          • we create a new linked list node and add it at the head of the queue and update the hashmap where the value points to the node we’ve just inserted.

  

  • MRU (Most Recently Used)

    • Process: evicts the most recently accessed data item from the cache. When a new item needs to be added to the cache and it's already at full capacity, the cache algorithm looks at the usage history of the items and evicts the most recently used one. This makes room for the new item to be added. MRU can be useful in scenarios where a small subset of items is accessed frequently.

    • Pros: Same as LRU

    • Cons: Same as LRU

• Frequency-Based Policies

  • LFU (Least Frequently Used)

    • Process: The basic principle of LFU caching is to remove the least frequently used items first when the cache reaches its capacity limit. This means that the cache keeps track of the frequency with which each item is accessed, and when it needs to make space for a new item, it removes the item that has been accessed the least frequently.

    • Pros:

      • Effective use of cache space: LFU caching maximizes the utilization of cache space by keeping frequently accessed items in memory.

      • Good performance: LFU caching performs well in scenarios where there are consistent access patterns and certain items are accessed more frequently than others.

      • Adaptive: LFU caching dynamically adjusts to changes in access patterns, ensuring that frequently accessed items remain in the cache.

    • Cons:

      • Complexity: LFU caching algorithms can be more complex to implement and maintain compared to simpler caching strategies like LRU (Least Recently Used).

      • Overhead: LFU caching requires additional data structures to track the access frequency of items, which can introduce overhead in terms of memory and computational resources.

      • Sensitivity to outliers: LFU caching may not perform optimally in scenarios with highly variable access patterns or sudden spikes in access frequency. Outliers in access frequency can skew the effectiveness of the cache.

    • Possible Implementation:

      • We are going to need 2 data structures, a cache map HashMap<item, Node> which holds our item along with a Node object
        
        Node(int val, int key) {

         this.val = val;

         this.key = key;

         this.frequency = 1;

        }
        
        and a frequency map HashMap<count, DoublyLinkedList<Node object>> which holds the count or frequency value a long with a list of all the items that have that given frequency.

      • When the cache is full, the next element that needs to be removed is the tail element of the doubly linked list with the least frequency.

    

  • LFRU (Least Frequently Recently Used)

    • Process: based on a hybrid algorithm that combines aspects of both LFU (Least Frequently Used) and LRU (Least Recently Used) caching strategies. LFRU caching maintains both frequency and recency information for each item in the cache. It tracks how frequently an item is accessed (similar to LFU) and how recently it was accessed (similar to LRU). When a new item needs to be added to the cache and it's already at full capacity, the cache algorithm uses a combination of frequency and recency information to decide which item to evict.

      • If an item has a low frequency of access but was accessed recently, it may be considered valuable and kept in the cache.

      • If an item has a high frequency of access but hasn't been accessed recently, it may be considered less valuable and evicted from the cache.

    • Pros:

      • Balance: LFRU caching strikes a balance between considering both frequency and recency of access, leading to more nuanced cache management decisions.

      • Adaptability: LFRU caching adapts well to varying access patterns and workload changes, ensuring efficient use of cache space.

      • Performance: LFRU caching can provide good performance by retaining frequently accessed items while also accommodating recent access patterns.

    • Cons:

      • Complexity: LFRU caching algorithms can be complex to implement and may require additional computational overhead compared to simpler caching strategies.

      • Resource Consumption: Maintaining both frequency and recency information for each item in the cache may consume more memory and computational resources.

      • Tuning: Tuning the parameters of an LFRU caching algorithm to optimize performance may require careful experimentation and analysis.

Cache Invalidation

Cache invalidation is the process of removing or marking as invalid data in a cache system. Implementing effective cache invalidation strategies is vital for preserving data integrity and preventing the delivery of stale or obsolete data.

• Invalidating on Write

  • This cache invalidation happens upon modifications to the underlying data source. Whenever a modification operation, like an update or deletion, takes place, the cache is notified or marked to invalidate the associated cached data. Subsequent attempts to access the invalidated data result in cache misses, prompting the retrieval of the latest information from the data source. While this method reduces the risk of serving outdated data, it may introduce a slight delay due to cache misses during the invalidation process.

• Invalidating on Read

  • Cache invalidation occurs during data access or retrieval. When a read request is received, the cache verifies whether the cached data is still valid or has become outdated. If the data has expired or is marked as invalid, the cache retrieves the latest information from the data source and updates its contents before fulfilling the request. While this method ensures the delivery of up-to-date data, it introduces overhead to each read operation as the cache must confirm the freshness of the data before responding.

• Write-Through Invalidation

  • In systems where data is written directly to the cache and the source of truth simultaneously, cache entries are invalidated whenever data is written or updated in the source of truth.

• Time Based Invalidation

  • Cached data is invalidated after a certain period of time has elapsed. For example, a web page cache may be configured to invalidate its content every hour to ensure that users see the most recent version of the page.

• Time-to-Live (TTL)

  • Similar to time based invalidation, TTL associates a time duration with each cached item. When an item is stored in the cache, it is marked with a TTL value indicating how long the item is considered valid.

Caching Strategies

• Read Strategies

  • Cache Aside - also known as lazy loading or lazy population, is a caching strategy where the application is responsible for managing the cache directly. When data is requested, the application first checks the cache. If the data is found, it is returned from the cache. If the data is not in the cache, the application retrieves it from the data source, stores it in the cache, and then returns it to the caller.

    • Pros:

      • Flexibility: Cache-aside allows applications to cache data selectively, based on their specific requirements. This flexibility enables developers to optimize caching for performance and efficiency.

      • Control: Applications have full control over cache management, including when to populate, update, or evict cached data. This level of control allows for fine-tuning cache behavior to suit the application's needs.

      • Reduced Load: By caching frequently accessed data, cache-aside can reduce the load on the primary data source, leading to improved overall system performance and scalability.

    • Cons:

      • Stale Data: Without proactive cache population or expiration mechanisms, cache-aside may lead to serving stale or outdated data to users, particularly if data changes frequently in the primary data source.

      • Consistency Challenges: Ensuring consistency between the cache and the primary data source can be complex, particularly in scenarios involving distributed caching or multiple cache instances. Inconsistent data can lead to data integrity issues and user experience problems.

  

  • Read Through - caching strategy where the application reads data from the cache, and if the requested data is not found in the cache (cache miss), the cache automatically fetches the data from the primary data source (such as a database) and populates the cache with the retrieved data before returning it to the application.

    • Pros:

      • Automatic Population: Cache read-through automatically populates the cache with requested data, reducing the need for explicit cache management.

      • Simplicity: Implementing cache read-through is relatively straightforward, as the caching logic is encapsulated within the cache itself, simplifying application code.

    • Cons:

      • Complexity in Eviction: Managing cache eviction policies to make room for new data can be challenging, especially in scenarios with limited cache capacity or high data volatility.

      • Resource Consumption: Cache read-through can consume additional system resources, such as memory and network bandwidth, particularly during cache population operations.

  

  • Refresh Ahead - also known as cache pre-loading or cache warm-up, is a caching strategy where the cache proactively fetches and stores data from the primary data source before it is requested by the application. This anticipatory approach aims to populate the cache with frequently accessed or critical data in advance, thereby reducing latency and improving overall system performance.

    • Pros:

      • Reduced Latency: By pre-loading frequently accessed data into the cache, cache refresh-ahead reduces the latency associated with fetching data from the primary source on demand.

      • Improved Performance: Proactively populating the cache with data that is likely to be accessed soon improves overall system performance by ensuring that frequently accessed data is readily available in memory.

      • Consistency: By periodically refreshing pre-loaded data, cache refresh-ahead helps maintain data consistency between the cache and the primary data source, ensuring that cached data remains up-to-date.

    • Cons:

      • Resource Overhead: Proactive data fetching and cache pre-loading consume additional system resources, such as memory, CPU, and network bandwidth, especially for large or frequently updated data sets.

      • Over-Population: Inefficient cache pre-loading strategies may lead to over-population of the cache with unnecessary or infrequently accessed data, resulting in wasted resources and reduced cache effectiveness.

• Write Strategies

  • Write Through - caching strategy where data is written to both the cache and the underlying data store simultaneously whenever a write operation occurs.

    • Pros:

      • Data Consistency: By writing data to both the cache and the underlying data store simultaneously, cache write-through ensures that the cache remains synchronized with the primary data source, minimizing the risk of data inconsistency.

      • Simplicity: Cache write-through simplifies cache management by eliminating the need for separate cache invalidation or synchronization mechanisms, as data is always written to both the cache and the data store.

    • Cons:

      • Write Latency: Write operations in cache write-through may experience increased latency due to the additional step of writing data to both the cache and the data store, which can impact application performance, especially for write-heavy workloads.

      • Potential Bottleneck: In highly concurrent or distributed environments, cache write-through may introduce a potential bottleneck if multiple write operations contend for access to the cache and the data store simultaneously, leading to contention and performance degradation.

  

  • Write Around - caching strategy where write operations bypass the cache and are written directly to the underlying data store. Data is cached when it is read from the data store, allowing the cache to store frequently accessed data while avoiding unnecessary caching of infrequently accessed or transient data.

    • Pros:

      • Reduced Cache Pollution: By bypassing the cache for write operations, cache write-around avoids caching infrequently accessed or transient data, reducing cache pollution

      • Scalability: Cache write-around is well-suited for scenarios with write-heavy workloads or large datasets, as it minimizes the impact on cache performance and storage capacity by avoiding unnecessary caching of write operations.

    • Cons:

      • Read Latency: The first read operation for data that has not been cached incurs additional latency, as the data must be fetched from the underlying data store instead of the cache, potentially impacting application performance.

      • Stale Data: If data is frequently updated or modified in the underlying data store, cache write-around may result in cached data becoming stale, as the cache does not proactively invalidate or update cached data based on write operations.

  

  • Write Back - caching strategy where write operations are initially written to the cache and later asynchronously propagated to the underlying data store.

    • Pros:

      • Improved Write Performance: Cache write-back improves write performance by acknowledging write operations as soon as they are written to the cache, without waiting for them to be persisted to the data store. This reduces latency and improves throughput, especially for write-heavy workloads.

      • Optimized Resource Utilization: Cache write-back optimizes resource utilization by buffering write operations in the cache, allowing the data store to handle write operations more efficiently and effectively, especially during peak load periods.

    • Cons:

      • Increased Complexity: Implementing cache write-back requires additional mechanisms for asynchronous data propagation, periodic flushing, and handling of failure scenarios, increasing system complexity and maintenance overhead.

      • Potential Data Loss: In the event of cache failure or system crash, cached data that has not been flushed to the data store may be lost, leading to potential data loss or corruption. Proper backup and recovery mechanisms are required to mitigate this risk.

Each caching strategy has its own advantages and considerations, and the selection of an appropriate strategy depends on the specific requirements and characteristics of the system. 

Summary

In this article, we've explored the fundamentals of caching, covering the core definition of caching, scenarios where it offers significant advantages, the importance of eviction policies (like LRU or MRU) to manage cache size, and strategies for invalidation to ensure your cache aligns with the primary data source. We also touched on the flow of data between the cache and your database for both read and write operations.

Stay Tuned! In upcoming articles, we'll dive deeper into:

• Types of Caches: Exploring different caching mechanisms like in-memory, distributed, and CDN caching.

• Deployment Strategies: Understanding how to integrate caching into your architecture.

• Metrics: How to monitor cache performance for optimal results.

• Implementation Solutions: Practical examples with popular caching tools and libraries.

Microservices have been getting more and more popular in recent years, and several companies are migrating their monolithic applications to microservices. But just a reminder that switching to microservices might not always be the good option. Martin Fowler wrote a nice article regarding the monolith-first approach. Here are a couple of useful statements:

• Almost all the successful microservice stories have started with a monolith that grew too big and was broken up.

• Almost all the cases I’ve heard of a system that was built as a microservice system from scratch, it has ended up in serious trouble.

A migration technique that has been frequently used when doing system rewrites is called the Strangler Fig Application. This migration method is based on incrementally replacing existing functionalities of the old legacy system with new applications and services in a phased approach until the new application system eventually replaces all of the old legacy system’s features.

The name of this type of tree, the Strangler Fig, comes from their pattern of growth around the host tree, which often results in the host’s death. The roots grow down to the forest floor where they take root and begin to take nutrients from the soil. Gradually the roots wrap around the host tree, widen, and slowly form a lattice-work that surrounds the host’s trunk. The fig’s crown grows foliage which soon overshadows the tree. Eventually, the host tree dies leaving the fig with a hollow trunk.

Martin Fowler chose this as the name of the pattern as a metaphor of describing a way of doing rewrite of an important system. Our new system is initially supported by and wrapping the existing system. The old and the new can co coexist while the new system can grow and potentially replace the entire old system.

“An alternative route is to gradually create a new system around the edges of the old, letting it grow slowly over several years until the old system is strangled.”

~ Martin Fowler

Advantages

• It allows for incremental migration to a new system.

• It allows to pause and even stop the migration while still taking advantage of the new system built so far.

• Each step is reversible, reducing the risk of each incremental step.

“The most important reason to consider a strangler application over a cut-over rewrite is reduced risk. A strangler can give value steadily and the frequent releases allow you to monitor its progress more carefully. Many people still don’t consider a strangler since they think it will cost more — I’m not convinced about that. Since you can use shorter release cycles with a strangler you can avoid a lot of the unnecessary features that cut over rewrites often generate.”
~ Martin Fowler

When not to use it

• The pattern will not be suitable for small systems where the complexity and size are low.

• The pattern cannot be used in systems where requests to the back-end system cannot be intercepted and routed.

Issues

• Considerable effort is required for the legacy and new system integrations.

• Considerable effort is required for the parallel run of the same functionality.

• Big commitment. Lack of will and resources to finish the strangling job might lead you into a bigger mess where your system now has two ways of doing everything with an awkward interface between the two.

How it works

The strangler fig pattern allows you to move functionality over to your new service without having to touch or make changes to your existing system.

The pattern involves 4 steps:

• Identify -> identify parts of the existing system that you wish to migrate. (use Domain Driven Design to identify the various bounded contexts)

• Transform -> implement this functionality in your new microservice.

• Co-exist -> leave the existing module in the legacy application as-is. Incrementally reroute calls from the monolith over to the new microservice.

• Eliminate -> once the traffic is completely redirected to the microservice, eliminate the legacy module.

First, add a proxy (facade), which sits between the legacy application and the user. This can be a “bare bones” routing app or an API Gateway. Initially, this proxy doesn’t do anything and all the application traffic is routed to the legacy application. Once a new component is built, you can link it to the proxy and you can allow the proxy to serve traffic to that page. You can also test your new functionality in parallel against the existing monolithic code. Keep adding more services and link them to the proxy. Repeat until all required functionality is handled by the new stack. Both the monolith and the new built component need to be functional for a period of time. Sometimes the transitional phase can last for an extended duration. When the new component has been incrementally developed and tested, the legacy monolithic application can be removed.

Conclusion

Moving from your existing legacy monolithic application into a microserviced architecture is a nice end goal to have but the journey is challenging and needs to be well planned. In this article, we have discussed about the Strangler Fig migration pattern which can help you in your journey. Keep in mind that strangling a monolith is not a quick process and may require some time.

Resources

• Martin Fowler Strangler Application

• Michiel Rook The Strangler pattern in practice

• Azure Cloud Patterns Strangler pattern

System design interviews are in general 45–60 minutes long where you are given a broad topic like “Design Twitter”. You’ll be expected to generate a high-level design, showing the different system components that will be required, how they’re connected, and any trade-offs in the approach you’ve taken. Time permitting, the interviewers will ask you to go deeper into how some of your system components would function and talk about how you would handle several scenarios and edge cases.

For a junior, mid-level engineer, this article aims to give you a fair good start at the System Design Interview and it’s not an exhausting list since system design is a vast topic. But from here, you can dig deeper with other resources.

Before starting with the theory basics, let’s briefly talk about how you should approach the interview.

1. Understand the goal and requirements.
   Never go head-first into the design, make sure you understand the problem and ask clarifying questions — What is the goal of the system? Who are the users of the system? What do they need it for? How are they going to use it? Clarify your assumptions.

2. Establish the scope
   Go through all the features of the system that you can think of and write them down.

3. Establish the scale
   What is the scale of the system? How many customers does the system need to serve per day? How many transactions? What are the performance requirements (e.g. latency, throughput, availability)? What is the expected read-to-write ratio? What’s the average expected response time?

4. Design high-level. The first 20 min of the interview you should have a draft of your high level design. Make sure to discuss the trade offs and why you chose those types of technologies. What type of database would you use and why? What caching solutions are out there? Which would you choose and why?

5. Drill-down. You should drill-down into more details, starting with the components that you are most familiar with. This is to ensure that you have time to go in-depth into the topics you have the most knowledge in.

6. Wrap-up. Before the end of your interview, you should spend some time to summarize your solution and highlight important bottlenecks or improvement opportunities.

   ---

Now let’s touch onto the most encountered components you’ll probably have to talk about.

Distributed Systems

= group of computers working together as to appear as a single computer to the end-user. They have a shared state, operate concurrently and can fail independently without affecting the whole system’s uptime.

Key Characteristics:

• Stability or Serviceability or Manageability = simplicity and speed with which a system can be repaired or maintained.

• Reliability = probability a system will fail in a given period. A distributed system is considered reliable if it keeps delivering its services even when one or several of its software or hardware components fail.

• Scalability = capability of a system, process, or a network to grow and manage increased demand.

• Efficiency/Performance = response time (or latency = delay to obtain the first item) + throughput (or bandwidth = the number of items delivered in a given time unit)

• Availability = simple measure of the percentage of time that a system, service, or a machine remains operational under normal conditions.

• Fault tolerance and Catastrophe-preparedness

Load Balancer

= component that acts as a reverse proxy and distributes network or application traffic across a number of servers to improve responsiveness and availability of applications, websites or databases.

• Benefits: faster, uninterrupted service, less downtime and higher throughput, predictive analytics that predict traffic bottlenecks (smart LB), fewer failed or stressed components.

Algorithms (used to balance the load):

• Least Connection Method (directs traffic to the server with the fewest active connections)

• Least Response Time Method (directs traffic to the server with the fewest active connections)

• Least Bandwidth Method (selects the server that is currently serving the least amount of traffic)

• Round Robin Method (cycles through a list of servers and sends each new request to the next server)

• Weighted Round Robin Method (servers with higher weights (= an integer value that indicates the processing capacity) receive new and more connections before those with less weights) — to handle servers with different characteristics (e.g. processing power, availability, load)

• IP Hash (a hash of the IP address of the client is calculated to redirect the request to a server)

Caching

= temporary storage area that stores the result of expensive responses or frequently accessed data in memory so that subsequent requests are served more quickly.

When to use it — when data is read frequently but modified infrequently

Considerations:

• Set an expiration policy that is not too short (as it reloads data too fast from the database) and not too long (since data can become stale).

• How to keep the cache and database consistent (see the cache invalidation section below)

• The cache can easily become a single point of failure, you might need to use multiple caches

• Choose an eviction policy (see the cache eviction policies section below)

• Think how you handle concurrency (aka the reader-writer problem) — When multiple clients are trying to update the cache at the same time, there can be conflicts. A solution would be to use commit logs. To update the cache, we can store all the mutations into logs rather than update immediately. And then some background processes will execute all the logs asynchronously. This strategy is commonly adopted in database design.

Reading from cache:

Cache Invalidation = happens when data is modified in the database and becomes inconsistent with the cache.

Updating the cache:

• Write-aside cache — data is written into the cache and the corresponding database at the same time (also known as lazy loading) — cons: higher latency

• Write-through cache — application adds or updates an entry in the cache. Cache synchronously writes entry to the data store. Cache returns the data to the application.

• Write-around cache — data is written directly to permanent storage, bypassing the cache — cons: read request for recently written data will create a “cache miss”.

• Write-back cache — data is written to cache alone and completion is immediately confirmed to the client and then data is asynchronously updated to the data store — pros: low latency and high throughput; cons: risk of data loss in case of a crash

Cache eviction policies

• First In First Out (FIFO) — evicts the first block accessed first

• Last In First Out (LIFO) — evicts the block accessed most recently

• Least Recently Used (LRU) — evicts the least recently used items first. When the client requests resource A, it happens as follow:
  If A exists in the cache, we just return immediately.
  If not and the cache has extra storage slots, we fetch resource A and return to the client. In addition, insert A into the cache.
  If the cache is full, we kick out the resource that is least recently used and replace it with resource A.

• Most Recently Used (MRU) — evicts the most recently used items first

• Least Frequently Used (LFU) — counts how often an item is needed. Those that are used least often are discarded first.

• Random Replacement (RR) — randomly evicts a candidate item

Concurrency — reader-writer problem — When multiple clients are trying to update the cache at the same time, there can be conflicts. => use commit logs. To update the cache, we can store all the mutations into logs rather than update immediately. And then some background processes will execute all the logs asynchronously. This strategy is commonly adopted in database design.

Content Distribution Network (CDN)

= system of distributed servers (network) that deliver webpages and other web content to a user based on the geographic locations of the user, the origin of the webpage and a content delivery server.

System flow: a request will first ask the CDN for a piece of static media; the CDN will serve that content if it has it locally available. If it isn’t available, the CDN will query the back-end servers for the file, cache it locally, and serve it to the requesting user.

Message Queues

= durable component, stored in memory, that supports asynchronous communication. Helps reduce request times for expensive operations that would otherwise be performed in-line. Receives, holds, and delivers messages.

If an operation is too slow to perform inline, you can use a message queue with the following workflow:

• An application publishes a job to the queue, then notifies the user of job status.

• A worker picks up the job from the queue, processes it, then signals the job is complete.

e.g. RabbitMQ, Amazon SQS

Proxies

= piece of software or hardware that acts as an intermediary for requests from clients seeking resources from other servers.

• used to filter requests, log requests, or sometimes transform requests. Its cache can serve a lot of requests (If multiple clients access a particular resource, the proxy server can cache it and serve it to all the clients without going to the remote server.)

Types:

• Open Proxy — accessible by any Internet user. Allows users within a network group to store and forward Internet services such as DNS or web pages to reduce and control the bandwidth used by the group.

Anonymous Proxy — reveals іts іdentіty аs а server but does not disclose the іnіtіаl IP address

Trаnspаrent Proxy — іdentіfіes іtself, аnd wіth the support of HTTP heаders, the fіrst IP аddress cаn be vіewed.

• Reverse Proxy — retrieves resources on behalf of a client from one or more servers

Databases

SQL vs NoSQL

NoSQL Types

Sharding or Data Partitioning

= process of splitting up a DB/table across multiple machines to improve the manageability, performance, availability, and load balancing of an application

Partitioning Methods:

• Horizontal Partitioning — put different rows into different tables — cons: unbalanced servers (if the value whose range is used for sharding isn’t chosen carefully)

• Vertical Partitioning divide our data to store tables related to a specific feature in their own server (instagram use case — user profile information stored on one DB server, friend lists on another) — cons: if our application experiences additional growth, then it may be necessary to further partition a feature specific DB across various servers

• Directory Based Partitioning — create a lookup service which knows your current partitioning scheme and abstracts it away from the DB access code.

Partitioning Criteria:

• Key or Hash-based partitioning — apply a hash function to some key attributes of the entity we are storing — cons: adding new servers means changing the hash function which would require redistribution of data and downtime for the service. (use Consistent Hashing)

• List partitioning — each partition is assigned a list of values, so whenever we want to insert a new record, we will see which partition contains our key and then store it there

• Round-robin partitioning — With ’n’ partitions, the ‘i’ tuple is assigned to partition (i mod n).

• Composite partitioning — combine any of the above partitioning schemes to devise a new scheme

Complexities:

Joins and Denormalization — not feasible to perform joins that span database shards (denormalize the database so that queries that previously required joins can be performed from a single table)

Referential Integrity — enforcing foreign keys in a sharded database can be extremely difficult. (not supported by most RDBMS — has to be enforced in application code.)

Celebrity problem — how would you deal with hotspots? (one way would be to add a shard for each celebrity)

Rebalancing/Resharding — data distribution is not uniform and there is a lot of load on a shard => have to create more DB shards or have to rebalance existing shards.

Strategies:

• Fixed number of partitions — Create many more partitions than there are nodes, and assign several partitions to each node

• Dynamic partitioning — when a partition grows to exceed a configured size, it is split into 2 partitions so that approximately half of the data ends up on each side of the split (no of partitions adapt to the total data volume)

• Partitioning proportionally to nodes — have a fixed no of partitions per node. The size of each partition grows proportionally to the dataset size while the no of nodes remains unchanged

Examples of Real World Scenarios:

• Read about Tinder’s Geosharded Recommendations Part 1: Sharding Approach — Based on Google’s S2 which uses the Hilbert curve (=a space-filling curve that preserves spatial locality: two points that are close on the Hilbert curve are close in physical space. Each smallest Hilbert curve clone is a cell, and 4 adjacent cells form a bigger cell.) Create the geoshards by enumerating all the possible container sizes, and calculate the standard deviation of each sharding configuration, the one with smallest standard deviation will be the most balanced geo sharding configuration we are looking for.

• Read about Uber’s Unwinding Uber’s Most Efficient Service — using a QuadTree — Take a flat projection of your search space and divide it into quarters that we’ll call cells. You then divide each of those cells into quarters recursively until you hit a defined maximum depth which will be the leaves of the tree.

• Read about Instragram’s Sharding & IDs at Instagram

Replication

= keeping a copy of the same data on multiple machines that are connected via a network.

Reasons:

• Latency — Keep data geographically close to your users (reduce access latency)

• High availability — Allow the system to continue working even if some parts of it have failed

• Scalability — Scale out the number of machines that can serve read queries

Leader-based replication:

• One of the replicas is designated as the leader. When clients want to write to DB, they send their requests to the leader, which writes the data to its local storage

• The other replicas (followers) receive the data from the leader in a process called replication log or change stream.

• When a client wants to read from the DB, it can query both leader/any of the followers.

Indexes

= data structure that improves the speed of data retrieval operations on a database table at the cost of additional writes and storage space to maintain the index data structure.

Can be created using one or more columns of a database table.

ACID

= set of properties of database transactions intended to guarantee validity even in the event of errors, power failures, etc.

In the context of databases, a sequence of database operations that satisfies the ACID properties (and these can be perceived as a single logical operation on the data) is called a transaction. For example, a transfer of funds from one bank account to another, even involving multiple changes such as debiting one account and crediting another, is a single transaction.

Atomicity — means that you guarantee that either all of the transaction succeeds or none of it does. You don’t get part of it succeeding and part of it not. If one part of the transaction fails, the whole transaction fails. With atomicity, it’s either “all or nothing”.

Consistency — this ensures that you guarantee that all data will be consistent. All data will be valid according to all defined rules, including any constraints, cascades, and triggers that have been applied on the database.

Isolation — guarantees that all transactions will occur in isolation. No transaction will be affected by any other transaction. So a transaction cannot read data from any other transaction that has not yet completed.

Durability — means that, once a transaction is committed, it will remain in the system — even if there’s a system crash immediately following the transaction. Any changes from the transaction must be stored permanently. If the system tells the user that the transaction has succeeded, the transaction must have, in fact, succeeded.

CAP

impossible for a distributed software system to simultaneously provide more than two out of three of the following guarantees (CAP): Consistency, Availability, and Partition tolerance

• Consistency: All nodes see the same data at the same time. Consistency is achieved by updating several nodes before allowing further reads.

• Availability: Every request gets a response on success/failure. Availability is achieved by replicating the data across different servers.

• Partition tolerance: The system continues to work despite message loss or partial failure. Means that the cluster continues to function even if there is a “partition” (communication break) between two nodes (both nodes are up, but can’t communicate).

All the combinations available are:

• CA — data is consistent between all nodes — as long as all nodes are online — and you can read/write from any node and be sure that the data is the same, but if you ever develop a partition between nodes, the data will be out of sync (and won’t re-sync once the partition is resolved).

• CP — data is consistent between all nodes, and maintains partition tolerance (preventing data desync) by becoming unavailable when a node goes down.

• AP — nodes remain online even if they can’t communicate with each other and will resync data once the partition is resolved, but you aren’t guaranteed that all nodes will have the same data (either during or after the partition)

  ---

Conclusion

I hope this article can be the first stepping stone on your journey to ace your system design interview. Good luck and reach out for any questions or feedback 😃

Motivation

When you have a large set of structured data (identified by record IDs) stored in a set of data files, what is the most efficient way to know which file might contain our required data? We need to keep track of large entities with the smallest memory print possible.

Reading each file would be slow as we have to read a lot of data from the disk. We could build an index on each data file and store it in a separate index file. Each index file will be sorted on the record ID. If we want to search an ID in this index, the best we can do is a Binary Search. Would there be an even faster approach?

Solution

A Bloom filter is a probabilistic data structure, named after Burton Howard Bloom, who invented them in the 1970s. It is based on hashing designed to tell you, rapidly and memory-efficiently, whether an element is present in a set. Elements are not added to the set but their hash is.

The price paid for this efficiency is that a Bloom filter is a probabilistic data structure: it tells us whether the element is either definitely not in the set or that it may be in the set, thus false positives are possible.

How does it work?

• A Bloom filter is made of two elements: an array of m elements and a set of k hash functions. An empty Bloom filter is a bit-array of m bits, all set to 0 initially; and each hash function outputs an index between 0 and m-1. There is no 1-1 correspondence between the elements of the array and the keys we add to the Bloom filter. We use k bits to store each entry for the Bloom filter.

• k is a constant that we pick when we create the data structure, so each and every entry we add is stored using the same amount of memory, k bits.

• To add an element, feed it to the hash functions to get k bit positions, and set the bits at these positions to 1.

• To test if an element is in the set, feed it to the hash functions to get k bit positions:
  1. If any of the bits at these positions is 0, the element is definitely not in the set.
  2. If all are 1, then the element may be in the set.

Choosing the Hash Functions

The hash functions used in a Bloom filter should be independent and uniformly distributed. Some examples of independent enough include murmur, xxHash, the fnv series of hashes, and HashMix. The more hash functions you have, the slower your bloom filter, and the quicker it fills up. If you have too few, however, you may suffer too many false positives.

The false-positive ratio formula

The false positive rate of your filter can vary on the length of the filter. A larger filter will have less false positives, and a smaller one more. Considering that

n = number of elements you expect to insert, 
m = number of bits in the bloom filter and 
k = number of hash functions we use to map a key to k different positions in the array

The rate of the false positives will be approximately

Let’s see why. After a single bit has been stored in the bloom filter with a capacity of m bits, the probability that a specific bit is set to 1 is 1/m. Then the probability that the same bit is set to 0 after all the k bits used to store an element have been flipped is (1–1/m)^k.

If we consider the events of flipping any specific bit to 1 as independent events, then after inserting n elements, for each individual bit the probability that the bit is still 0 is: 

To have a false positive, all the k bits corresponding to an element V must have been set to 1 independently, and the probability that all of those k bits are 1 is given by:

Choosing the size of the bloom filter

To choose the size of a bloom filter, we:

1. Choose an estimate value for n

2. Choose a value for m

3. Calculate the optimal value of k

4. Calculate the error rate for the chosen values of n, m, and k. If it’s unacceptable, return to step 2 and change m; otherwise we’re done.

Thomas Hurst has a nice online page that will help you choose an optimal size for your filter, or explore how the different parameters interact.

Time and Space Complexity

Let’s say we’re using a bloom filter with m bits and k hash functions, then insertion and search will both take O(k) time. We just need to run the input through all of the hash functions and then check the output bits.

If we make a bloom filter where k=1 then we just have a hash table that ignores collisions. The lower the false positive rate low, the larger the m. So, the space of the actual data structure is simply O(m).

Possible Implementation

Let’s first define an interface for our implementation. Firstly, we’re going to need a method to add a key and one to check if the key exists. PrintStats() will print the number of total items we have added to the filter and the total number of bits sets. SetHashFunction() allows us to set a custom hash function and Reset() to clear the bloom filter.

type Bloom interface {
 Add(key []byte) Bloom
 Check(key []byte) (bool, float64)
 PrintStats()
 SetHashFunction(hash.Hash)
 Reset()
}

func K(e float64) uint {
 return uint(math.Ceil(math.Log2(1 / e)))
}

func M(n uint, p, e float64) uint {
 // m =~ n / ((log(p)*log(1-p))/abs(log e))
 return uint(math.Ceil(float64(n) / ((math.Log(p) * math.Log(1-p)) / math.Abs(math.Log(e)))))
}

And now let’s dive into the implementation. We’ll define a BloomFunction with the following attributes:

h = hash function
m = number of bits in the bloom filter, will be divided into k partitions, or slices
k = number of hash values we use to set and test bits
s = the size of the partition, or slice; s = m / k
p = the fill ratio of the filter partitions. It’s mainly used to calculate m at the start. p is not checked when new items are added. So if the fill ratio goes above p, the likelihood of false positives (error rate) will increase. By default, we use the fill ratio of p = 0.5
e = the desired error rate of the bloom filter. The lower the e, the higher the k. By default, we use the error rate of e = 0.1% = 0.001.
n = number of elements the filter is predicted to hold
b = b is the set of bit array holding the bloom filters. There will be k b’s. We’re going to use the golang’s bitset package which is a mapping between non-negative integers and boolean values.
c = number of items we have added to the filter
bs = the list of bits

type BloomFunction struct {
 // h is the hash function
 h hash.Hash

 // the total number of bits for the bloom filter.
 m uint

 // the number of hash values used to set and test bits.
 k uint

 // the size of the partition, or slice.
 s uint

 // the fill ratio of the filter partitions. 
 p float64

 // the desired error rate of the bloom filter.
 e float64

 // number of elements the filter is predicted to hold
 n uint

 // b is the set of bit array holding the bloom filters.
 b *bitset.BitSet

 // number of items we have added to the filter
 c uint

 // the list of bits
 bs []uint
}

var _ Bloom = (*BloomFunction)(nil)

func NewBloomFilter(n uint) Bloom {
 var (
  p float64 = 0.5
  e float64 = 0.001
  k uint    = K(e)
  m uint    = M(n, p, e)
 )

 return &BloomFunction{
  h:  fnv.New64(),
  n:  n,
  p:  p,
  e:  e,
  k:  k,
  m:  m,
  b:  bitset.New(m),
  bs: make([]uint, k),
 }
}

When adding an item we simply pass it through the hash functions and store the results. To check if an item is in the set we iterate through the bitset and check whether every bit is set or not. If any of the bits are not, then it’s safe to assume the word doesn’t exist. If all bits are set then the item possibly exists with the probability of the formula we proved earlier.

func (f *BloomFunction) SetHashFunction (h hash.Hash) {
 f.h = h
}

func (f *BloomFunction) Reset() {
 f.k = K(f.e)
 f.m = M(f.n, f.p, f.e)
 f.b = bitset.New(f.m)
 f.bs = make([]uint, f.k)

 if f.h == nil {
  f.h = fnv.New64()
 } else {
  f.h.Reset()
 }
}

func (f *BloomFunction) Add(item []byte) Bloom {
 f.bits(item)
 for _, v := range f.bs[:f.k] {
  f.b.Set(v)
 }
 f.c++
 return f
}

func (f *BloomFunction) Check(item []byte) (bool, float64) {
 f.bits(item)
 for _, v := range f.bs[:f.k] {
  if !f.b.Test(v) {
   return false, 0
  }
 }
 fPosProb := math.Pow(1.0-math.Pow(1.0-1.0/float64(f.m), float64(f.k*f.n)), float64(f.k))

 return true, fPosProb
}

func (f *BloomFunction) Count() uint {
 return f.c
}

func (f *BloomFunction) PrintStats() {
 fmt.Printf("m = %d, n = %d, k = %d, s = %d, p = %f, e = %f\n", f.m, f.n, f.k, f.s, f.p, f.e)
 fmt.Println("Total items:", f.c)
 c := f.b.Count()
 fmt.Printf("Total bits set: %d (%.1f%%)\n", c, float32(c)/float32(f.m)*100)
}

func (f *BloomFunction) bits(item []byte) {
 f.h.Reset()
 f.h.Write(item)
 s := f.h.Sum(nil)
 a := binary.BigEndian.Uint32(s[4:8])
 b := binary.BigEndian.Uint32(s[0:4])

 for i := range f.bs[:f.k] {
  f.bs[i] = (uint(a) + uint(b)*uint(i)) % f.m
 }
}

Now let’s add 2 words to the filter “hello” and “world”. We can see that the “hello” word exists with a false positive probability of 0.001. 

func main() {
 bf := NewBloomFilter(10)
 bf.Add([]byte("hello"))
 bf.Add([]byte("world"))
 check, prob := bf.Check([]byte("hello"))
 if check {
  fmt.Printf("Word exists with a false positive probability of %v\n", prob)
 } else {
  fmt.Printf("Word does not exists.")
 }
}
bf.PrintStats()

// Prints 

/*

Word exists with a false positive probability of 0.0010134325478480376

m = 144, n = 10, k = 10, s = 0, p = 0.500000, e = 0.001000
Total items: 2
Total bits set: 20 (13.9%)

*/

Code available on github.

Applications

In BigTable (and Cassandra), any read operation has to read from all SSTables that make up a Tablet. If these SSTables are not in memory, the read operation may end up doing many disk accesses. To reduce the number of disk accesses, BigTable uses Bloom filters. (source)

Bloom filters are created for SSTables (particularly for the locality groups). They help reduce the number of disk accesses by predicting if an SSTable may contain data corresponding to a particular row or column pair. For certain applications, a small amount of Tablet server memory used for storing Bloom filters drastically reduces the number of disk-seeks, thereby improving read performance.

Medium uses Bloom filters in its Recommendation module to avoid showing the posts that the user has already seen.

Resources:

• Visualisation by Thomas Hurst showing how parameters influence each other

• Bloom Filters by Example