post://designing-data-pipelines-that-stay-cheap
Designing data pipelines that stay cheap
Efficient pipeline design is mostly a set of decisions about bytes scanned, work repeated, and failures survived. Here is how I reason through them, with the arithmetic shown.