Most data infrastructure is built on one assumption: writes are append-only. Kafka appends events into partitions, and Iceberg, Delta and Hudi write immutable Parquet files. Under the GDPR that assumption runs into a legal obligation. Article 17 says a person can ask you to erase their data, and Article 12 puts a clock on the response.
EuroStream is my attempt to build a lakehouse where erasure is a first-class operation instead of a support ticket. Repo: swadhinbiswas/eurostream · Lake: Hugging Face · Architecture RFC
Where append-only breaks
I mapped the problem into five failures that show up in production.
A raw event log retains the original payload, and rewriting a partition or mutating consumer offsets to remove one person's data tends to corrupt downstream state. Materialized aggregates keep ghost records: delete a customer upstream and gold.customer_360 still reports their lifetime spend and marketing flags. Stateful stream processors hold rolling window state in memory, so an erased customer can sit in a deque for hours and trigger false fraud alerts. Serverless and ephemeral containers wipe in-memory suppression caches on restart, which produces split-brain governance. And microservices keep adding PII columns nobody classified.
The deletion cascade
The core of the project is a synchronized six-step transaction:
Suppression -> Bronze mask -> Silver hard DELETE -> Gold hard DELETE
-> Alert state purge -> Lake re-export
When Article 17 fires, raw Bronze columns are masked in place to <anonymized> so the ledger keeps its row ordering. Silver records are hard deleted, since PII there is already pseudonymized with a salted SHA-256 hash H(s, x). Every affected Gold aggregate partition is recomputed, and the public Parquet lake is replaced atomically so a de-identified copy does not keep a ghost. A global suppression registry intercepts replayed or delayed events before they reach a consumer.
The streaming side does the same thing in memory. Before a payment is scored for velocity, Z-score or geo-mismatch, FraudScorer checks erasure.is_suppressed(cust_id). On deletion the customer's window deque and alert history are evicted immediately, so an erased identity cannot set off an alert later.
Two engines, one watermark
EuroStream pairs a local embedded DuckDB for microsecond analytics with a distributed Turso libSQL replica for durability. Every write, merge, watermark advance and erasure mutation is dual-written over HTTP v2. On a container restart the suppression sets and warehouse state come back automatically, which is what keeps a serverless deployment from drifting.
The event bus has two implementations behind one interface: SqliteBus locally, in WAL mode with BEGIN IMMEDIATE for concurrency, and KafkaBus in production over Aiven with SASL_SSL and SCRAM-SHA-256. The speed path and the batch path share no state, which is what makes the erasure transaction safe to reason about.
Governance in CI
PII detection includes IBAN checksum verification (ISO 13616, ISO 7064 mod 97), and a contract baseline gate blocks any pull request that introduces an unclassified column or a breaking schema change. The medallion layers carry the policy directly: Bronze keeps PII internally and is never exported, Silver is pseudonymized, and Gold marketing analytics are gated on bool_and(marketing_consent) so a withdrawn consent removes a person from the aggregates.
The project is pure Python, typed strictly with mypy and linted with ruff, with 59 tests across the streaming and batch paths.
What I took from it
Erasure is the hardest test of a data architecture, because it forces you to know where every copy of a record lives. If you cannot name each place a person's data sits, you cannot delete it. Building the cascade meant documenting the storage tiers first and designing for the delete path second, which is the reverse of how most pipelines get written.
The repository, the RFC and the published lake are linked above. If you are building anything that falls under the GDPR, the deletion path is worth designing on day one rather than bolting on later.
Reach me at swadhinbiswas.cse@gmail.com or on GitHub and LinkedIn.
No comments yet.