Data pipelines & ETL templates
Ingestion, transformation and warehouse loading diagrams. Every one opens as a real, editable diagram — restyle it, extend it and export it as your own. See also the data pipeline diagram tool.
All 37 data pipelines & etl templates
Nightly ELT pipeline
Three sources, one loader, a warehouse and a transform step.
MermaidKafka event streaming pipeline
Producers into topics, consumed by stream processing and storage.
MermaidMedallion (bronze/silver/gold) warehouse
The standard lakehouse layering pattern.
MermaidBatch ETL to a data warehouse
Classic extract-transform-load with a staging area.
MermaidChange data capture (CDC) replication
Database changes streamed to a warehouse in near real time.
MermaidML feature engineering pipeline
Raw data to a feature store used by training and serving.
MermaidData pipeline with quality gates
Validation steps that can halt the pipeline before bad data lands.
MermaidKappa architecture
One streaming path for everything: an immutable log, stream processors for all transformations, and reprocessing by replaying the log instead of a separate batch layer.
MermaidBig data architecture (Hadoop ecosystem)
The classic on-prem big-data stack: HDFS, YARN, Hive, Spark, HBase, Kafka, Sqoop and Oozie, with Ranger security.
MermaidSpark big data processing platform
Object storage, Spark on Kubernetes, table formats, a metastore, a query engine and orchestration — the cloud-native successor to Hadoop.
MermaidReal-time analytics (Kafka, Flink, Pinot)
Sub-second user-facing analytics: events into Kafka, Flink enrichment and aggregation, Apache Pinot or Druid for serving, and a lake for history.
MermaidAWS real-time analytics
Kinesis Data Streams, Managed Service for Apache Flink, Firehose, OpenSearch, Timestream and Redshift streaming ingestion.
MermaidAzure real-time analytics
Event Hubs and IoT Hub into Fabric Real-Time Intelligence (Eventstream, Eventhouse/KQL), Stream Analytics, Data Activator and Power BI.
MermaidGoogle Cloud real-time analytics
Pub/Sub, Dataflow streaming, BigQuery streaming and continuous queries, Bigtable for serving, and Looker.
MermaidModern data stack
Managed ELT, cloud warehouse, dbt, semantic layer, BI, reverse ETL, orchestration and observability — the typical 2020s analytics stack.
MermaidStreaming lakehouse (Apache Iceberg)
Streams and batch landing in the same Iceberg tables, queried by several engines through one open catalog.
MermaidData fabric architecture
A metadata-driven layer over distributed sources: active metadata, knowledge graph, virtualisation, governance and self-service delivery.
MermaidLakehouse platform (Delta Lake)
Batch and streaming ingestion into bronze/silver/gold Delta tables with Unity Catalog governance and BI, ML and SQL consumers.
MermaidReal-time analytics pipeline
Clickstream events to sub-second dashboards: collection, validation, stream processing and OLAP.
MermaidReverse ETL to business tools
Warehouse models synced back into CRM, marketing and support tools so teams act on the same data.
MermaidData mesh with domain data products
Domain teams own and publish data products on a self-serve platform, governed by federated policies.
MermaidOrchestrated daily pipeline (Airflow DAG)
A daily DAG with sensors, parallel extracts, transforms, tests, and SLA alerting — how most batch pipelines are scheduled.
MermaidMLOps end-to-end pipeline
Data validation, training, registry, deployment and monitoring with automatic retraining.
MermaidLog collection & analytics pipeline
Application and infrastructure logs shipped, parsed, enriched, stored in tiers and searched.
MermaidCustomer 360 & identity resolution
Stitching customer identities across web, app, CRM and support into one golden record.
MermaidIoT telemetry data pipeline
Millions of device readings per minute: ingestion, downsampling, hot and cold storage, and fleet analytics.
MermaidPrivacy-aware data pipeline
Classification, pseudonymisation, consent filtering and deletion propagation built into the data platform.
MermaidLegacy warehouse to cloud migration
Moving from an on-prem warehouse to a cloud platform with parallel runs and reconciliation.
MermaidStreaming fraud detection pipeline
Transactions scored in milliseconds with streaming features, a model server and a case management loop.
MermaidMarketing attribution pipeline
Ad spend, sessions and conversions joined into multi-touch attribution for channel ROI.
MermaidData contracts between producers and consumers
Schema contracts enforced in CI and at runtime so upstream changes cannot silently break downstream pipelines.
MermaidETL job with retries & dead letters
Resilient batch processing: idempotent loads, retries with backoff, poison records and restart from checkpoint.
MermaidSnowflake cross-organisation data sharing
Publishing governed data to partners through secure shares and a data marketplace listing.
MermaidLambda architecture (batch + speed layer)
Combining a batch layer for accuracy and a speed layer for freshness behind one serving layer.
MermaidDocument processing (IDP) pipeline
Invoices and forms captured, OCRed, classified, extracted, validated and posted with human-in-the-loop review.
MermaidGeospatial data pipeline
Addresses and GPS points geocoded, joined to boundaries and tiled for map visualisation.
MermaidData observability & incident flow
Monitoring freshness, volume, schema and distribution across tables, and routing data incidents to owners.
Mermaid