LinkedIn System Design: Feed, Graph and Kafka

LinkedIn System Design: More Than a Feed

LinkedIn has over 1.2 billion members and earned about $19.8 billion in Microsoft's fiscal 2026. Underneath the job posts and "I'm happy to share" updates are some of the most influential pieces of data infrastructure ever built, starting with Apache Kafka, which was created at LinkedIn.

LinkedIn is a great system design study because it combines a massive social graph ("you're a 2nd-degree connection"), a ranked feed, and real-time data pipelines that feed everything else.

🎯 Quick Answer (30-Second Read)

The graph: members and connections form a huge graph; "2nd and 3rd degree" queries need special graph services and caching.
The feed: candidates from your network and followed pages are ranked by ML for relevance, not just time.
Kafka: LinkedIn built Kafka to move activity data between systems in real time, then open-sourced it in 2011.
Data stores: LinkedIn built its own, such as Espresso (document store) and Venice (derived data for serving ML features).
Search: people, jobs and posts, powered by LinkedIn's in-house search platform.

High-Level Architecture

flowchart TB M["📱 Member apps"] --> API["API layer (Rest.li)"] API --> FEED["Feed service
candidates + ranking"] API --> GRAPH["Graph service
connections, degrees"] API --> PROFILE["Profile service"] API --> SEARCH["Search
people, jobs, posts"] PROFILE --> ESP[("Espresso
member data")] GRAPH --> GDB[("Graph store
+ caches")] FEED --> VEN[("Venice
ML features")] API -->|"activity events"| K[["Kafka"]] K --> STREAM["Stream processing
(Samza)"] K --> HADOOP["Offline / batch
(training ML models)"] STREAM --> VEN HADOOP --> VEN

1. The Social Graph and Degrees

"How are you connected to this person?" is a graph question:

Degree	Meaning	Rough scale for an active member
1st	Direct connections	Hundreds
2nd	Connections of connections	Tens of thousands or more
3rd	One step further	Millions

Computing 2nd- and 3rd-degree connections on every page view would be too slow. LinkedIn's engineers have described a distributed graph service that keeps the connection graph in memory, partitioned across many machines, with caching for expensive network-distance queries.

2. The Feed

flowchart LR C1["Posts from
connections"] --> CAND["Candidate
generation"] C2["Followed companies,
creators, hashtags"] --> CAND C3["Recommended posts"] --> CAND CAND --> RANK["ML ranking
predicted value: reactions,
comments, dwell time"] RANK --> RULES["Diversity &
quality rules"] RULES --> F["📱 Your feed"] F -.->|"what you do"| K[["Kafka events"]] K -.-> RANK

Like other modern feeds, LinkedIn gathers candidate posts, scores them with machine-learning models predicting whether you'll find them valuable (reactions, comments, time spent), then applies rules for variety and quality. Every interaction flows back as an event to improve the models.

3. Kafka: Born at LinkedIn

LinkedIn had many systems that needed the same data: who viewed a profile, who clicked a job, what got shared. Point-to-point pipelines between every pair of systems became unmanageable. Jay Kreps, Neha Narkhede and Jun Rao built Kafka: a durable, replicated log that any system can write events to and any system can read from.

Key takeaway: instead of N×N connections between systems, everything publishes to and reads from one shared log. That idea, described in Jay Kreps' essay "The Log," now underpins data infrastructure at thousands of companies.

Kafka was open-sourced in 2011, became an Apache project, and its creators later founded Confluent. (See Message Queue Explained and Write-Ahead Log Explained.)

4. Purpose-Built Data Stores

System	What LinkedIn uses it for
Espresso	Distributed document store for primary member data
Venice	Serving derived data (like ML features and recommendations) computed by batch and streaming jobs
Voldemort (earlier)	Key-value store, inspired by Amazon's Dynamo; Venice later replaced it for derived data
Pinot	Real-time analytics (e.g. "who viewed your profile" stats), later an Apache project

LinkedIn open-sourced many of these, including Kafka, Samza, Pinot and Venice.

5. Search

Searching for people, jobs and posts needs results personalised by your network ("people you might know at Google"). LinkedIn built its own search platform, Galene, combining keyword matching with ranking based on your connections and activity.

Interview Checklist

Graph storage partitioned in memory, with cached degree queries
Feed: candidate generation + ML ranking + rules
Event pipeline (Kafka) feeding both streaming and batch systems
Derived data stores for serving ML features quickly
Personalised search across people, jobs and content
Privacy controls on profile visibility and network data

Frequently Asked Questions

Was Kafka created at LinkedIn?

Yes. Apache Kafka was built at LinkedIn by Jay Kreps, Neha Narkhede and Jun Rao to move high volumes of activity data between systems in real time. LinkedIn open-sourced it in 2011, and it became an Apache Software Foundation project.

How does LinkedIn calculate 2nd-degree connections?

LinkedIn's engineers have described a distributed graph service that keeps the member connection graph in memory across many machines, with caching for network-distance queries. That makes looking up 2nd- and 3rd-degree connections fast enough for every page view.

How does the LinkedIn feed decide what to show?

It collects candidate posts from your connections, the companies and people you follow, and recommendations, then ranks them with machine-learning models that predict how valuable each post is to you, with rules to keep the feed varied and high quality.

Conclusion

LinkedIn's system design combines an in-memory social graph, an ML-ranked feed, personalised search and, most famously, Kafka as the central nervous system moving activity data everywhere. Many of the tools it built for itself, from Kafka to Pinot, are now standard parts of other companies' data stacks.

Sources

Last verified: 2 November 2026.

I write about system design and how real products are built. Follow on LinkedIn or X, or invite me to speak.