🎓 A1Learner
🔍
0 of 42 lessons completed

The Celebrity Problem

The celebrity problem is the strain that one account with millions of followers puts on fan-out on write. This lesson deep-dives that problem and the hybrid fix that answers it.

One post, millions of writes

The arithmetic uses the follower count and the write rate from the points. The printout shows the writes and the time they take.

followers = 50000000
writes_per_second = 100000

writes = followers
seconds = writes // writes_per_second

print("feed writes for one post:", writes)
print("seconds to finish:", seconds)
print("minutes, about:", round(seconds / 60))

A threshold splits accounts

Accounts at or under the threshold push their posts. Accounts over the threshold are pulled at read time instead.

accounts = {"ana": 120, "ben": 350, "celebrity": 25000}
threshold = 10000

for name in sorted(accounts):
    if accounts[name] > threshold:
        plan = "pull"
    else:
        plan = "push"
    print(name, accounts[name], "followers ->", plan)

Write counts under the hybrid

A normal post and a celebrity post arrive in the same run. The printout shows the feed writes each one makes under the hybrid.

threshold = 10000
posts = [("ana", 200), ("celebrity", 50000000)]

total = 0
for author, followers in posts:
    writes = followers if followers <= threshold else 0
    total += writes
    print(author, "post feed writes:", writes)
print("total feed writes:", total)

Example

threshold = 10000
posts = [("ana", 200), ("ben", 150), ("celebrity", 50000000)]

full = 0
hybrid = 0
for author, followers in posts:
    writes = followers if followers <= threshold else 0
    full += followers
    hybrid += writes
    print(author, "hybrid writes:", writes)

print("full push writes:", full)
print("hybrid writes:", hybrid)
print("writes saved:", full - hybrid)

Key points

  • Celebrity: an account with far more followers than a normal account.
  • Celebrity problem: under fan-out on write, one celebrity post must be copied into every follower's feed.
  • Scale here: the celebrity in this design has 50,000,000 followers.
  • One post: that one post makes 50,000,000 feed writes under fan-out on write.
  • Write time: at 100,000 writes per second, those writes take 500 seconds, about 8 minutes.
  • Threshold: this design treats accounts with more than 10,000 followers as huge.
  • Hybrid: ordinary accounts use fan-out on write, and huge accounts are left for fan-out on read.

In this example, the hybrid pushes the two ordinary posts and pulls the celebrity post at read time. Almost all feed writes in the run disappear, because the celebrity post makes none.

Note

the hybrid keeps normal posts cheap to read, and huge posts cost no copies at all.

Try it Yourself

Edit the code, then press Run.

Running is simulated

Exercise

What is the hybrid answer to the celebrity problem?