Designing data pipelines that stay cheap
Efficient pipeline design is mostly a set of decisions about bytes scanned, work repeated, and failures survived. Here is how I reason through them, with the arithmetic shown.
7 articles in this category
Efficient pipeline design is mostly a set of decisions about bytes scanned, work repeated, and failures survived. Here is how I reason through them, with the arithmetic shown.
The requirement was 100K rps. The work was arithmetic, a stateless tier, a cache that absorbs most of it, and a long list of things that break before the CPU does.
Paillier homomorphic encryption, an honest-but-curious proxy, and a simulator that measures what the network costs when the server never sees plaintext.
JustAPI is a Python web framework where the web server, the router, the TLS termination, the request validation, the JSON serialization, and the database access all run in Rust. Your Python code handles the application logic. That is the entire premise, and it is the one sentence that explains every design decision in the project.
Most web players ship a 30 MB software decoder and hope the phone survives it. VidoLib decodes through WebCodecs instead, and ships every parser as its own plugin.
Boringrats served a million users on donations, with a fifty-dollar monthly budget and a bill that landed near thirty-five. Here is the architecture, the numbers from the busiest day, and what it cost outside the invoice.