Designing data pipelines that stay cheap
Efficient pipeline design is mostly a set of decisions about bytes scanned, work repeated, and failures survived. Here is how I reason through them, with the arithmetic shown.
25 published posts across 14 top-level categories.
Efficient pipeline design is mostly a set of decisions about bytes scanned, work repeated, and failures survived. Here is how I reason through them, with the arithmetic shown.
The requirement was 100K rps. The work was arithmetic, a stateless tier, a cache that absorbs most of it, and a long list of things that break before the CPU does.
I built Flare to run mail for my own domain without an IMAP server. Astro on a Worker, Turso for state, Resend or Maileroo for delivery, webhooks for inbound.
Point it at a folder and every create, edit, rename and delete lands in a local database. Undo, replay, fork and diff, without git in the way.
Hand-curated source data, generated gold tables, and a published dataset. How I keep a directory of TOML honest with code generation and CI.
Playwright for the pages that need JavaScript, cron for the schedule, and a dedupe key that keeps DAAD, Erasmus and Chevening from colliding.
Detect, track, check liveness, embed, match. One worker per camera, a SQLite source of truth, and a vector index that heals itself when the model changes.
Paillier homomorphic encryption, an honest-but-curious proxy, and a simulator that measures what the network costs when the server never sees plaintext.
Removing a Linux app often leaves configs and caches on disk. Veet scans 17 sources at once, stages every residual path, and removes them in one confirmed step.
An OpenAI-compatible endpoint, guardrails, a model registry, drift detection, and Prometheus metrics, packaged as a single Docker container.