System Design Advanced · Week 3
Kafka internals: partitions, offsets, consumer groups
00:00
System Design Advanced • Week 3
01 / 13

Kafka internals: partitions, offsets, consumer groups

Delivery guarantees define your consumer design, not the broker.

StackShade deck
COURSE CONTENTS

Kafka internals: partitions, offsets, consumer groups

01Consistent Hashing & Data PartitioningWeek 1 · 5 topics0/5
Sharding & the modulo-N trap (visual deep dive)12 minConsistent hashing ring simulator15 minVirtual nodes and balanced load12 minDynamo-style partitioning in the wild (Cassandra, Memcached)14 minCompute key movement when a 5th node joins (by hand)20 min
02Distributed Caching at ScaleWeek 2 · 5 topics0/5
Cache-aside vs write-behind (with the stampede bug)12 minReproduce + fix a cache stampede in the simulator12 minHot keys, key skew and celebrity mitigation12 minMemcached vs Redis architectures; cache clusters14 minDesign the cache layer for a viral-tweet scenario30 min
03Event-Driven Architecture & Message QueuesWeek 3 · 5 topics0/5
Delivery guarantees & idempotent consumers14 minKafka internals: partitions, offsets, consumer groups16 minOrdering, retries, dead-letter queues12 minInteractive: partition assignment across a consumer group12 minDesign an event pipeline for order processing40 min
04Consensus, Replication & Leader ElectionWeek 4 · 5 topics0/5
Distributed consensus & the replication log12 minRaft walkthrough: terms, votes, log matching16 minSplit-brain and why quorum saves you12 minPaxos vs Raft vs gossip — picking a coordination style12 minSimulate: 5-node cluster survives 2 failures?25 min
05Distributed TransactionsWeek 5 · 4 topics0/4
2PC and why it blocks (and where it's still used)14 minSagas: choreography vs orchestration14 minOutbox pattern: events that never lie about the DB12 minDesign payment flow with saga compensations35 min
06Rate Limiting, Gates & Real-TimeWeek 6 · 5 topics0/5
Token bucket, leaky bucket, sliding window14 minInteractive: rate limiter algorithms visualizer12 minAPI gateways: auth, quotas, routing in one hop12 minReal-time at scale: WebSocket fan-out, presence, rooms14 minDesign rate limiting for a public API (per-user + global)30 min
07Case Studies I: Social & MessagingWeek 7 · 4 topics0/4
Design Twitter: feeds, fan-out, celebrities22 minDesign WhatsApp: chats, delivery ticks, groups22 minDesign Instagram: media pipeline + feed ranking hooks18 minSolo: design a chat system with read receipts45 min
08Case Studies II: Infra-Scale SystemsWeek 8 · 5 topics0/5
Design Uber: geohashes, matching, surge24 minDesign YouTube: upload, transcode, CDN delivery22 minDesign Google Drive: sync, chunks, conflicts20 minDesign a web crawler + typeahead (bonus)18 minFull mock: 45-min design of your choice + rubric self-review60 min