Skip to main content

Comprehensive Reference List

System Design References

Real-world Systems

Company Engineering Blogs

General References

All References


A JSON Meta Application Protocol (JMAP) Subprotocol for WebSocket. Visit


Ad traffic quality. Visit


Aeron. Visit


All of the World's Stock Exchanges by Size. Visit


Booking(.com) iOS app. Visit


Books, an immutable double-entry accounting database service. Visit


Cassandra - A Decentralized Structured Storage System. Visit


Cassandra Limitations. Visit


CCPA (California Consumer Privacy Act). Visit


Clickthrough rate (CTR): Definition. Visit


Comparing Random and Sequential Access in Disk and Memory. Visit


Conversion rate: Definition. Visit


CS168: The Modern Algorithmic Toolbox Lecture #1: Introduction and Consistent Hashing. Visit


Datadog Homepage. Visit


DB-Engines Ranking of Search Engines. Visit


DB-Engines Ranking of time-series DBMS. Visit


Developing with kdb+ and the q language. Visit


Domain-based Message Authentication, Reporting & Conformance. Visit


Dynamo: Amazon's Highly Available Key- value Store. Visit


Epoch. Visit


Fix trading. Visit


Google Directions API. Visit


Google Drive. Visit


Google Maps 101: How AI helps predict traffic and determine routes. Visit


Google Maps Homepage. Visit


Google Maps Platform. Visit


Grafana Demo. Visit


Graphite Homepage. Visit


HdrHistogram: A High Dynamic Range Histogram. Visit


Hilbert mapping. Visit


HKEX daily trading volume. Visit


How Discord Scaled Elixir to 5,000,000 Concurrent Users. Visit


How Many Emails Are Sent Per Day in 2021?. Visit


How to Build a Fast Limit Order Book. Visit


InfluxDB Homepage. Visit


Kafka No Longer Requires ZooKeeper. Visit


Latency Numbers Every Programmer Should Know. Visit


LMAX exchange was famous for its open-source Disruptor. Visit


Maglev: A Fast and Reliable Software Network Load Balancer. Visit


Message filtering in RocketMQ. Visit


mmap. Visit


Order book. Visit


QQ Email System Optimization (in Chinese). Visit


RFC 1939 - Post Office Protocol - Version 3. Visit


Scheduled messages and delayed messages in Apache RocketMQ. Visit


Service Discovery with Zookeeper. Visit


Splunk Homepage. Visit


Stamen Design. Visit


Yelp Homepage. Visit


YugabyteDB Homepage. Visit


ACM. Comparison of disk and memory performance. Visit


Airbnb Blog. Scaling Airbnb's Payment Platform. Visit


Akamai. Akamai homepage. Visit


All Things Distributed. Dynamo: Amazon's Highly Available Key-value Store. Visit


Allan Heydon and Marc Najork. Mercator: A scalable, extensible web crawler World Wide Web, 2 (4) (1999), pp. 219-229. Visit


Amazon Web Services. Amazon CloudFront Dynamic Content Delivery. Visit


Amazon Web Services. Amazon CloudWatch. Visit


Amazon Web Services. Amazon Compute Service Level Agreement. Visit


Amazon Web Services. Amazon DynamoDB. Visit


Amazon Web Services. Amazon EC2 High Memory Instances. Visit


Amazon Web Services. Amazon S3. Visit


Amazon Web Services. Amazon S3 Glacier. Visit


Amazon Web Services. Amazon S3 Service Level Agreement. Visit


Amazon Web Services. Amazon S3 Strong Consistency. Visit


Amazon Web Services. Amazon Timestream. Visit


Amazon Web Services. AWS CLI ls command. Visit


Amazon Web Services. AWS Coinbase Case Study. Visit


Amazon Web Services. Build a real-time gaming leaderboard with Amazon ElastiCache for Redis. Visit


Amazon Web Services. Building Applications with Serverless Architectures. Visit


Amazon Web Services. Choosing your node size. Visit


Amazon Web Services. Cloudfront Pricing. Visit


Amazon Web Services. Configure Sticky Sessions for Your Classic Load Balancer. Visit


Amazon Web Services. Lambda. Visit


Amazon Web Services. Netflix on AWS. Visit


Amazon Web Services. Regions and Zones. Visit


Amazon Web Services. Throttle API requests for better throughput. Visit


Amazon Web Services. Using Global Secondary Indexes in DynamoDB. Visit


Amazon Web Services. Warming up dedicated IP addresses. Visit


Amit. End-to-end Exactly-once Aggregation Over Ad Streams. Visit


Andrew Tridgell and Paul Mackerras. The rsync algorithm. Visit


Android Docs. Get the last known location. Visit


Antin Harasymiv. Prototyping a Smoother Map. Visit


Apache. Apache Druid Homepage. Visit


Apache. Apache Kafka allow consumers fetch from closest replica. Visit


Apache. Flink on Apache Yarn. Visit


Apache. Flink performance tuning. Visit


Apache. Kafka mirroring. Visit


Apache. LanguageManual ORC. Visit


Apache Flink. An Overview of End-to-End Exactly-Once Processing in Apache Flink. Visit


Apache Flink. Apache Flink Homepage. Visit


Apache Hadoop. Apache hadoop. Visit


Apache Hadoop. Apache Hadoop Homepage. Visit


Apache James. Apache James Homepage. Visit


Apache Kafka. Apache kafka. Visit


Apache Kafka. Apache Kafka Homepage. Visit


Apache Kafka. Kafka 2.0 Documentation. Visit


Apache Kafka. Kafka protocol guide. Visit


Apache Kafka. Push vs. pull. Visit


Apache Spark. Spark streaming. Visit


Apache Storm. Apache storm. Visit


Apache ZooKeeper. Apache ZooKeeper. Visit


Apache Zookeeper. Apache Zookeeper Homepage. Visit


Apache Zookeeper. Apache Zookeeper Homepage. Visit


AppDynamics. What Are The Benefits of Microservices Architecture?. Visit


Atlassian. Pagination in the REST API. Visit


Backblaze. Data Durability Calculation. Visit


Burton H. Bloom. Space/Time Trade-offs in Hash Coding with Allowable Errors. 1970. Visit


Burton H. Bloom. Space/Time Trade-offs in Hash Coding with Allowable Errors. Visit


Cassandra. Cassandra. Visit


Cassandra. Cassandra architecture. Visit


Ceph Docs. Ceph's Rados Gateway. Visit


Chargebee Blog. How Payment Gateways Can Detect and Prevent Online Fraud. Visit


Chris Richardson. Monolithic Architecture. Visit


Chris Richardson. Saga. Visit


Christopher Olston and Marc Najork. Web Crawling. . Visit


ClickHouse. ClickHouse Homepage. Visit


Cloudflare. What is DDoS attack. Visit


Cloudflare Blog. How we built rate limiting capable of scaling to millions of domains. Visit


Cloudflare Learning. What is edge computing. Visit


cloudkarafka. ISR in Apache Kafka. Visit


CME Group. CME Co-Location and Data Center Services. Visit


CME Group. Supported Matching Algorithms. Visit


Cockroach Labs. Cockroachdb Homepage. Visit


Code Ahoy. Caching Strategies and How to Choose the Right One. Visit


Colin Scott. Latency Numbers Every Programmer Should Know. Visit


Columbia CS. Hashed and hierarchical timing wheels. Visit


Confluent. Chain Services with Exactly-Once Guarantees. Visit


Confluent. Hands-free Kafka Replication. Visit


Confluent. Kafka Connect Deep Dive - Error Handling and Dead Letter Queues. Visit


Cornell CS. SAGAS, HectorGarcia-Molina. Visit


Dacast. Here's What You Need to Know About Streaming Protocols. Visit


Data Dog. Key metrics for RabbitMQ monitoring. Visit


databricks. Lambda architecture. Visit


databricks. Parquet. Visit


DataKwery. Big Data. Visit


DataStax. How data is distributed across a cluster (using virtual nodes). Visit


DataStax. TIMEUUID. Visit


DataStax Docs. UUID and timeuuid types. Visit


DataTracker. IMAP LIST Extension for Special-Use Mailboxes. Visit


Debezium. Debizium Homepage. Visit


Debezium. Redis sink. Visit


DeepMind. ETAs with GNNs. Visit


Digimarc. Digimarc Homepage. Visit


Discord. How Discord Stores Billions of Messages. Visit


Donald J. Patterson. Web Crawling: Introduction to Information Retrieval. Visit


Donne Martin. System design primer. Visit


Dropbox. Dropbox security white paper. Visit


DynamoDBGuide. Leaderboard & Write Sharding. Visit


Earth Lab. Global map in a Geographic Coordinate Reference System. Visit


Elastic. Elastic stack. Visit


Elastic. Geo Shape Type. Visit


Elastic. Geohash grid aggregation. Visit


Elixir. Elixir Homepage. Visit


Eric Evans. Domain-Driven Design: Tackling Complexity in the Heart of Software. 2003. Visit


Erlang. A brief introduction to BEAM. Visit


Erlang. Erlang Homepage. Visit


Erlang. OTP. Visit


etcd. etcd Homepage. Visit


etcd. etcd Homepage. Visit


etcd. etcd Homepage. Visit


Facebook. Erlang at Facebook. Visit


Facebook. How News Feed Works. Visit


Facebook. RocksDB. Visit


Facebook. The Life of a Typeahead Query. Visit


Facebook. The Underlying Technology of Messages. Visit


Fay Change et al.. Bigtable: A Distributed Storage System for Structured Data. Visit


FinancesOnline. Number of Active Gmail Users. Visit


Flickr. Ticket Servers: Distributed Unique Primary Keys on the Cheap. Visit


Frank Ren. Geosharded Recommendations Part 1: Sharding Approach. Visit


GeekBang. Geekbang. Visit


Geeks with Blogs. Friend of Friend recommendations Neo4j and SQL Sever. Visit


Giedrius Blog. Push vs Pull in Monitoring Systems. Visit


Google Cloud. Compute Engine Service Level Agreement (SLA). Visit


Google Cloud. Google Cloud Functions. Visit


Google Cloud. Rate-limiting strategies and techniques. Visit


Google Cloud. Schema design for time-series data | Cloud Bigtable Documentation. Visit


Google Docs. Google docs usage limits. Visit


Google Docs. Google places API. Visit


Google Docs. Upload file data. Visit


Google Research. Dapper, a Large-Scale Distributed Systems Tracing Infrastructure. Visit


Google Safety. Email safety. Visit


Google Search Central. Google Dynamic Rendering: . Visit


gRPC. grpc Homepage. Visit


gRPC. gRPC Introduction. Visit


Hazelcast. Kappa architecture. Visit


High Scalability Blog. Google Pro Tip: Use Back-Of-The-Envelope-Calculations To Choose The Best Design. Visit


High Scalability Blog. What The Heck Are You Actually Using NoSQL For. Visit


Hsin-Tsang Lee et al.. IRLbot: Scaling to 6 Billion Pages and Beyond. 2008. Visit


HubSpot. 2019 YouTube Demographics. Visit


IBM. IBM microservices. Visit


IBM. Optimistic and pessimistic record locking. Visit


IBM. Security in Push Notifications. Visit


IBM. What is avro. Visit


Igvita. SSTable. Visit


Ilya Grigorik. SStable. Visit


InfluxDB Docs. InfluxDB storage engine. Visit


InfoQ. From nothing: the evolution of WeChat background system (Article in Chinese). Visit


Investopedia. Dark pool. Visit


Jamie Zawinski. Message threading. Visit


Jim Fisher. Redis Pub/Sub under the hood. Visit


Josh Constine. Facebook Launches 'Nearby Friends'. Visit


Juan Manuel Villegas. How we created a real-time Leaderboard for a million Users. Visit


Jun Wang et al.. Display Advertising with Real-Time Bidding (RTB) and Behavioural Targeting. Visit


Kevin Modzelewski. How We've Scaled Dropbox. Visit


Kousik Nath. Geohashing. Visit


Larry Page and Sergey Brin. The PageRank Citation Ranking: Bringing Order to the Web. 1998. Visit


Lyft. Lyft rate limiting. Visit


Mailchimp. Mailchimp. Visit


Martin Fowler. Blue green deployment. Visit


Martin Fowler. CQRS. Visit


Martin Fowler. Event Sourcing. Visit


Martin Kleppmann. 'Replication' in Designing Data-Intensive Applications. Visit


Martin Kleppmann. Designing Data-Intensive Applications. 2017. Visit


Martin Kleppmann. Designing Data-Intensive Applications. 2017. Visit


Martin Kleppmann. Designing Data-Intensive Applications. 2017. Visit


Martin Pool et al.. Librsync. Visit


memcached. memcached. Visit


Michael O. Rabin. Fingerprinting by Random Polynomials. 1981. Visit


Microsoft. Bing Maps Tile System. Visit


Microsoft. Delegate access with a shared access signature. Visit


Microsoft. Understand star schema and the importance for Power BI. Visit


Microsoft Azure. Azure Functions. Visit


Microsoft Azure. SLA summary for Azure services. Visit


MongoDB. MongoDB. Visit


Movable Type Scripts. Geohash. Visit


MySQL Docs. NDB Cluster Replication: Multi-Master and Circular Replication. Visit


Neil Fraser. Differential Synchronization. Visit


Neil Fraser. Differential Synchronization YouTube talk. Visit


Netflix Tech Blog. Active-Active for Multi-Regional Resiliency. Visit


Netflix Tech Blog. Content Popularity for Open Connect. Visit


Nexmo. Nexmo SMS. Visit


Nick Craver. What it takes to run Stack Overflow. Visit


NYSE. NYSE matched volume. Visit


Omnicore. YouTube by the numbers. Visit


Onesmus Mbaabu. Understanding MapReduce in Hadoop. Visit


OpenStreetMap. OpenStreetMap Homepage. Visit


OpenStreetMap. OpenStreetMap Homepage. Visit


OpenStreetMap. OpenStreetMap Homepage. Visit


OpenTSDB. OpenTSDB - A Distributed, Scalable Monitoring System - Homepage. Visit


Oracle. Change data capture. Visit


Oracle Docs. Java stream. Visit


Oracle Docs. OLAP functions. Visit


Oracle Docs. Transactional guarantees. Visit


Patrick O'Leary. Cartesian tiers. Visit


Paul Dix. Why We're Building Flux, a New Data Scripting and Query Language. Visit


Paul Tarjan. Scaling your API with rate limiters. Visit


PayPal. Idempotency. Visit


Peter Hayes. Better Rate Limiting With Redis Sorted Sets. Visit




Prefixy Team. How We Built Prefixy: A Scalable Prefix Search Service for Powering Autocomplete. Visit


Princeton CS. SVE: Distributed Video Processing at Facebook Scale. Visit


ProgrammerSought. Refactoring practice: build full-text search of QQ mailbox based on Tencent Cloud Elasticsearch. Visit


Programster Blog. Rate Limit Requests with Iptables. Visit


Prometheus. Data model. Visit


Prometheus. Prometheus Introduction. Visit


Prometheus. Pushgateway. Visit


Prometheus Blog. Pull doesn't scale - or does it?. Visit


Publications Office of the European Union. EU Web Archive. Visit


RabbitMQ. Queue Length Limit. Visit


Raft. Designing for Understandability: the Raft Consensus Algorithm. Visit


Raft. Raft Homepage. Visit


Raft. Raft Homepage. Visit


Raft. Raft Homepage. Visit


Raft. Raft Homepage. Visit


Rajesh Nishtala et al.. Scaling Memcache at Facebook. Visit


Redis. Geospatial Indexing: The 10 Million QPS Redis Architecture Powering Lyft. Visit


Redis. How fast is Redis?. Visit


Redis. Info command. Visit


Redis. Redis. Visit


Redis. Redis GEOHASH. Visit


Redis. Redis Pub/Sub. Visit


Redis. Redis Sorted Set source code. Visit


Redis. Redis website. Visit


RedisLabs. Leaderboards. Visit


RocksDB. RocksDB Homepage. Visit


ronginx Blog. Kafka high watermark. Visit


S2 Geometry. Region cover. Visit


S2 Geometry Homepage. S2. Visit


Sai Sandeep Mopuri. System Design — Rate limiter and Data modelling. Visit


Sandeep Verma. Building real-time Leaderboard with Redis. Visit


Search Engine Land. 29% Of Sites Face Duplicate Content Issues. Visit


Sendgrid. Sendgrid. Visit


sflow Blog. Push vs. Pull. Visit


Shadi A. Noghabi et al.. Ambry: LinkedIn's Scalable Geo-Distributed Object Store. Visit


Shopify. Shopify REST Admin API rate limits. Visit


Slack Engineering. Flannel: An Application-Level Edge Cache to Make Slack Scale. Visit


SQLite. SQLite Homepage. Visit


SQLite. SQLite Homepage. Visit


Sriram Ramabhadran. Prefix Hash Tree An Indexing Data Structure over Distributed Hash Tables. Visit


Stack Overflow. How many leaves has a quadtree. Visit


Stack Overflow. Why redis cluster only have 16384 slots. Visit


Stamen Design. Map tiles by Stamen Design. Visit


Statista. Global spam volume. Visit


Steven Skiena. Algorithm Design Manual. 2020. Visit


Stripe. Customize your success page. Visit


Stripe. Stripe API Reference. Visit


Stripe. Stripe idempotent requests. Visit


Stripe. Stripe rate limiters. Visit


Stripe. Webhooks. Visit


Tanguy Urvoy et al.. Tracking Web Spam with Hidden Style Similarity. 2006. Visit


Team Treehouse. Should you go Beyond Relational Databases?. Visit


Tech Hub. Monitoring Architecture. Visit


The Verge. Messenger and WhatsApp process 60 billion messages a day. Visit


Tipalti. Tipalti Homepage. Visit


Todd Fredrich. A RESTful Tutorial. Visit


Tom White. Consistent Hashing. Visit


Toptal. Consistent hashing. Visit


Toptal. Consistent hashing. Visit


TPC. TPC-E Top Price/Performance Results: . Visit


Trisha Gee. Cache line padding. Visit


Twilio. Twilio SMS. Visit


Twitter Blog. Announcing Snowflake. Visit


Twitter Blog. Announcing Snowflake. Visit


Twitter Blog. Distributed Systems Tracing with Zipkin. Visit


Twitter Blog. MetricsDB: TimeSeries Database for storing metrics at Twitter. Visit


Twitter Docs. Twitter rate limits. Visit


Tyler Treat. You Cannot Have Exactly-Once Delivery. Visit


Uber Blog. Advanced Technologies for Detecting and Preventing Fraud at Uber. Visit


Uber Blog. Re-Architecting Cash and Digital Wallet Payments for India with Uber Engineering. Visit


Uber Blog. Real-Time Exactly-Once Ad Event Processing with Apache Flink, Kafka, and Pinot. Visit


Uber Engineering. Payments Integration at Uber: A Case Study. Visit


Unicode. Unicode frequently asked questions. Visit


Upyun. Weibo video processing architecture (in Chinese). Visit


US Library of Congress. Search Web Archives. Visit


Valhalla. Routing Tiles. Visit


Verizon. 2018 Data Breach Investigations Report. Visit


VLDB. Gorilla: A Fast, Scalable, In-Memory Time Series Database. Visit


WhatsApp. End-to-end encryption. Visit


Wikipedia. 3D Secure. Visit


Wikipedia. ACID. Visit


Wikipedia. ActiveSync. Visit


Wikipedia. Address geocoding. Visit


Wikipedia. Adjacency list. Visit


Wikipedia. Advanced Message Queuing Protocol. Visit


Wikipedia. AML/CFT. Visit


Wikipedia. Apache ZooKeeper. Visit


Wikipedia. B+ tree. Visit


Wikipedia. Base32. Visit


Wikipedia. Binary large object. Visit


Wikipedia. Bloom filter. Visit


Wikipedia. Bloom filter. Visit


Wikipedia. CAP theorem. Visit


Wikipedia. Card scheme. Visit


Wikipedia. Chaos engineering. Visit


Wikipedia. Checksum. Visit


Wikipedia. Compensating transaction. Visit


Wikipedia. Consistent hashing. Visit


Wikipedia. Context switch. Visit


Wikipedia. Cyclic redundancy check. Visit


Wikipedia. Cyclic redundancy check. Visit


Wikipedia. DAG model. Visit


Wikipedia. Denial of service attack. Visit


Wikipedia. DomainKeys Identified Mail. Visit


Wikipedia. Double-entry bookkeeping. Visit


Wikipedia. Email attachment. Visit


Wikipedia. Erasure Coding. Visit


Wikipedia. Exponential backoff. Visit


Wikipedia. Exponential backoff. Visit


Wikipedia. Fibre channel. Visit


Wikipedia. Gall–Peters projection. Visit


Wikipedia. GDPR (Data Protection Regulation). Visit


Wikipedia. General Data Protection Regulation. Visit


Wikipedia. Geo-fence. Visit


Wikipedia. Hilbert curve. Visit


Wikipedia. Hmac. Visit


Wikipedia. HotSpot (virtual machine). Visit


Wikipedia. HTTP keep-alive. Visit


Wikipedia. Hypertext Transfer Protocol. Visit


Wikipedia. Idempotence. Visit


Wikipedia. IEX attracts investors by 'playing fair', also is the 'Flash Boys Exchange'. Visit


Wikipedia. inode. Visit


Wikipedia. Inverted index. Visit


Wikipedia. IOPS. Visit


Wikipedia. iSCSI. Visit


Wikipedia. ISO 4217. Visit


Wikipedia. ISO 4217 CURRENCY CODES. Visit


Wikipedia. Lawful interception. Visit


Wikipedia. Log-structured merge-tree. Visit


Wikipedia. Long tail. Visit


Wikipedia. Market impact. Visit


Wikipedia. Md5. Visit


Wikipedia. Mercator projection. Visit


Wikipedia. Merkle tree. Visit


Wikipedia. Microservices. Visit


Wikipedia. MIME. Visit


Wikipedia. mmap. Visit


Wikipedia. MongoDB wikipedia. Visit


Wikipedia. Multi-master replication. Visit


Wikipedia. NACK-Oriented Reliable Multicast. Visit


Wikipedia. Network File System. Visit


Wikipedia. Network time protocol. Visit


Wikipedia. Nonce. Visit


Wikipedia. Optimistic concurrency control. Visit


Wikipedia. Order book. Visit


Wikipedia. OSI model. Visit


Wikipedia. Outlook. Visit


Wikipedia. Paxos. Visit


Wikipedia. Paxos. Visit


Wikipedia. Payment Card Industry Data Security Standard. Visit


Wikipedia. Payment system. Visit


Wikipedia. Peirce quincuncial projection. Visit


Wikipedia. Quadtree. Visit


Wikipedia. R-tree. Visit


Wikipedia. Rack. Visit


Wikipedia. Reed-Solomon error correction. Visit


Wikipedia. Reliable User Datagram Protocol. Visit


Wikipedia. Replication (Computing). Visit


Wikipedia. Reverse proxy. Visit


Wikipedia. Sender Policy Framework. Visit


Wikipedia. Serial Attached SCSI. Visit


Wikipedia. Serializability. Visit


Wikipedia. Server Message Block. Visit


Wikipedia. Sha1. Visit


Wikipedia. Single point of failure. Visit


Wikipedia. Threading. Visit


Wikipedia. Two-phase commit protocol. Visit


Wikipedia. Universally unique identifier. Visit


Wikipedia. Winkel tripel projection. Visit


Wikipedia. X/Open XA. Visit


Wikipedia. YAML. Visit


Xu Cheng et al.. Understanding the characteristics of internet short video sharing: A youtube-based measurement study. Visit


Yelp. Yelp business endpoints. Visit


Yext Engineering Blog. Improved Location Caching with Quadtrees. Visit


YouTube. Erasure Coding Demystified. Visit


YouTube. Microsoft Exchange Conference 2014 Search in Exchange. Visit


YouTube. Reliable Processing in a Streaming Payment System. Visit


YouTube. YouTube scalability talk by early YouTube employee. Visit