Product Event Pipeline: from your emit call to a queryable table

Product Event Pipeline: from your emit call to a queryable table A data-flow diagram generated by Archify. 01 / Emit 02 / Collect 03 / Validate 04 / Store 05 / Use Your Service · emit call site · 01 / Emit · owns schema Your Service emit call site owns schema Client Apps · web + mobile · 01 / Emit · events Client Apps web + mobile events Collector · enrich + batch · 02 / Collect · at the edge Collector enrich + batch at the edge Schema Registry · contract check · 03 / Validate · blocking Schema Registry contract check blocking Event Stream · partitioned log · 03 / Validate · ordered Event Stream partitioned log ordered Quarantine · rejected events · 04 / Store · replayable Quarantine rejected events replayable Raw Tables · append-only · 04 / Store · source of truth Raw Tables append-only source of truth Modelled Marts · tested transforms · 04 / Store · curated Modelled Marts tested transforms curated Dashboards · product metrics · 05 / Use · SQL Dashboards product metrics SQL Experiment Readouts · A/B decisions · 05 / Use · derived Experiment Readouts A/B decisions derived typed events server-side client events device-side validate schema contract accepted events append-only schema drift rejected landed events immutable tested transforms scheduled batch metrics SQL read-only assignment joins derived Legend primary data policy / PII async batch data store data flow

The schema is the interface

  • • An event name and its fields are a published contract
  • • Adding a field is safe; renaming or retyping one is a breaking change
  • • The registry rejects drift at collection, not at query time

Rejected does not mean lost

  • • Quarantined events keep their payload and their reason
  • • Fix the emit site, then replay the quarantine window
  • • Silent drops are the failure mode this design exists to prevent

Who reads what you emit

  • • Raw tables are append-only and never edited in place
  • • Marts are the tested layer analysts and dashboards build on
  • • Experiment readouts join your events to assignment data