11 articles in this category
Efficient pipeline design is mostly a set of decisions about bytes scanned, work repeated, and failures survived. Here is how I reason through them, with the arithmetic shown.
Point it at a folder and every create, edit, rename and delete lands in a local database. Undo, replay, fork and diff, without git in the way.
Hand-curated source data, generated gold tables, and a published dataset. How I keep a directory of TOML honest with code generation and CI.
Playwright for the pages that need JavaScript, cron for the schedule, and a dedupe key that keeps DAAD, Erasmus and Chevening from colliding.
Detect, track, check liveness, embed, match. One worker per camera, a SQLite source of truth, and a vector index that heals itself when the model changes.
An OpenAI-compatible endpoint, guardrails, a model registry, drift detection, and Prometheus metrics, packaged as a single Docker container.
Four public sources, a versioned lake on Hugging Face, dbt transforms, an IsolationForest, and a dashboard at the edge. The compute bill is zero.
Kafka and Parquet are append-only by design. Article 17 is not. This is a medallion lakehouse built around a six-layer deletion cascade.
Kafka on a small VPS, DuckDB and dbt in CI, and a read-only Turso copy for the browser. How I built a live European airspace pipeline without a cloud budget.
Boringrats served a million users on donations, with a fifty-dollar monthly budget and a bill that landed near thirty-five. Here is the architecture, the numbers from the busiest day, and what it cost outside the invoice.
An agent that runs long enough will forget what it decided. I implemented the GCC framework, which treats agent memory like Git, in seven languages against one on-disk format.
