02 — Capability · BC-750.30
Data Engineering & Pipelines
Build and run the pipelines that collect, clean, transform and deliver data to where it is used, reliably enough that a missing load is an incident rather than a normal morning.
- Data Engineering
- ETL
- Data Integration
In scope
- Data ingestion from internal and external sources
- Transformation and modeling in the platform
- Pipeline scheduling and monitoring
- Data platform operations and cost
Out of scope
- Application-to-application integration (see BC-720)
- Infrastructure that hosts the platform (see BC-730)
Realized by · 3
- Snowflake · Data Warehousecore via BC-750.30.40
- Snowflake · Data Engineeringstrong via BC-750.30.20
- Snowflake · Data Sharing and Marketplacecore via BC-750.30.10
Used in · 0
- Not yet placed on a value stream.
Build it · 8
Decomposes into · 4
- BC-750.30.10Data IngestionPull or receive data from source systems, files, streams and partners into the platform with its lineage intact.
- BC-750.30.20Data TransformationClean, join, conform and aggregate raw data into the modeled datasets analysts and reports consume.
- BC-750.30.30Pipeline OperationsSchedule, monitor and recover data pipelines so freshness and completeness targets are met and failures are seen before the business sees them.
- BC-750.30.40Data Platform OperationsAdminister the analytical platform — workspaces, compute, access, cost — as a shared service with capacity and performance someone owns.