🎓 A1Learner
🔍
0 of 47 lessons completed

Why Partition Data

Partitioning is splitting one large dataset across several machines so no single machine must store all of it. Sharding is partitioning where each part, called a shard, lives on its own machine.

One dataset against one budget

This demo adds up the full dataset and compares it with the budget of one machine. The totals show the dataset outgrowing a single machine.

rows = 12000
row_kb = 1
total_mb = rows * row_kb // 1000
budget_mb = 5
print("Rows:", rows)
print("Total size:", total_mb, "MB")
print("Budget for one machine:", budget_mb, "MB")
print("Fits on one machine:", total_mb <= budget_mb)

The same rows across shards

This demo splits the generated rows across 3 shards. Each shard holds 4,000 rows and uses 4 MB.

rows = list(range(1, 12001))
shards = [rows[0:4000], rows[4000:8000], rows[8000:12000]]
number = 1
for shard in shards:
    size_mb = len(shard) * 1 // 1000
    print("Shard", number, "holds", len(shard), "rows using", size_mb, "MB")
    number = number + 1
print("Budget per machine: 5 MB")

Each shard holds its own range

This demo prints the first and last row of each shard. The ranges sit side by side and cover the full dataset.

rows = list(range(1, 12001))
shards = [rows[0:4000], rows[4000:8000], rows[8000:12000]]
number = 1
for shard in shards:
    print("Shard", number, "range:", shard[0], "to", shard[-1])
    number = number + 1

Example

rows = list(range(1, 12001))
total_mb = len(rows) * 1 // 1000
print("Total rows:", len(rows))
print("Total size:", total_mb, "MB")
shards = [rows[0:4000], rows[4000:8000], rows[8000:12000]]
number = 1
stored = 0
for shard in shards:
    size_mb = len(shard) * 1 // 1000
    stored = stored + len(shard)
    print("Shard", number, ":", len(shard), "rows,", size_mb, "MB, under the 5 MB budget:", size_mb <= 5)
    number = number + 1
print("Rows stored across all shards:", stored)

Key points

  • Partitioning: one dataset is split into parts, and the parts live on different machines.
  • Shard: one part of a split dataset, together with the machine that holds it.
  • Row size: each row in this lesson takes 1 KB of disk.
  • Disk budget: one machine holds at most 5 MB, so one machine holds at most 5,000 rows.
  • Dataset size: the full dataset holds 12,000 rows and takes 12 MB of disk.

In this example, the full dataset outgrows the budget of one machine, so the rows are split across shards. Each shard stays under the budget, and the shard counts add back to the full dataset.

Note

Splitting the data also splits the work, because each shard serves only its own part.

Try it Yourself

Edit the code, then press Run.

Running is simulated

Exercise

A dataset grows past the disk budget of one machine. What does partitioning do about it?