🎓 A1Learner
🔍
0 of 48 lessons completed

Julia Project: Data Pipeline

A data pipeline is a set of steps that turns stored values into a short summary. This lesson builds one from a small sales file.

Write the file, then read it back

The file is written first in the same block. Reading it back shows its header and its size.

using DelimitedFiles
write("sales.csv", "category,price\napple,4\nbanana,3\n")
data, header = readdlm("sales.csv", ',', header=true)
println(join(header, ","))
println(size(data))

Count the rows in each group

Rows are grouped by their category. Each group count is printed in sorted key order.

using DelimitedFiles
write("sales.csv", "category,price\napple,4\napple,6\nbanana,3\n")
data, header = readdlm("sales.csv", ',', header=true)
function group_counts(data)
  counts = Dict{String,Int}()
  for i in 1:size(data, 1)
    cat = String(data[i, 1])
    counts[cat] = get(counts, cat, 0) + 1
  end
  return counts
end
counts = group_counts(data)
for cat in sort(collect(keys(counts)))
  println(cat, ": ", counts[cat])
end

Compute the mean of each group

Each group total is divided by its row count. The means are printed in sorted key order.

using DelimitedFiles
write("sales.csv", "category,price\napple,4\napple,6\nbanana,3\nbanana,5\n")
data, header = readdlm("sales.csv", ',', header=true)
function group_means(data)
  sums = Dict{String,Float64}()
  counts = Dict{String,Int}()
  for i in 1:size(data, 1)
    cat = String(data[i, 1])
    price = data[i, 2]
    sums[cat] = get(sums, cat, 0.0) + price
    counts[cat] = get(counts, cat, 0) + 1
  end
  means = Dict{String,Float64}()
  for cat in keys(sums)
    means[cat] = sums[cat] / counts[cat]
  end
  return means
end
means = group_means(data)
for cat in sort(collect(keys(means)))
  println(cat, ": ", means[cat])
end

Example

using DelimitedFiles
write("sales.csv", "category,price\napple,4\napple,6\nbanana,3\nbanana,5\n")
data, header = readdlm("sales.csv", ',', header=true)
function summarize(data)
  sums = Dict{String,Float64}()
  counts = Dict{String,Int}()
  for i in 1:size(data, 1)
    cat = String(data[i, 1])
    price = data[i, 2]
    sums[cat] = get(sums, cat, 0.0) + price
    counts[cat] = get(counts, cat, 0) + 1
  end
  means = Dict{String,Float64}()
  for cat in keys(sums)
    means[cat] = sums[cat] / counts[cat]
  end
  return counts, means
end
counts, means = summarize(data)
for cat in sort(collect(keys(means)))
  println(cat, ": ", counts[cat], " rows, mean ", means[cat])
end

Key points

  • Pipeline: a pipeline is a set of steps that turns stored values into a short summary.
  • Reading: readdlm reads a file you wrote first and returns its rows as a matrix.
  • Header: the header row names each part of a row, and readdlm can return it with the data.
  • Grouping: grouping puts rows with the same category together so each group can be summarized on its own.
  • Mean: a mean is the total of a group divided by the number of rows in that group.
  • Order: sorted keys give the same group order on every run.

In this example, the sales file is written first, then readdlm reads it with its header. The summarize function groups rows by category, counts each group, and divides each total by its count. The loop prints each group in sorted key order with its row count and its mean.

Note

a block that reads a file must write it first, so a second run gives the same result.

Try it Yourself

Edit the code, then press Run.

Running is simulated · JuliaHub

Exercise

A sales file has apple prices 4 and 6 and banana prices 3 and 5. What is the mean apple price?