Data Engineering · Leadership
A Metadata-Driven ETL Framework That Scaled to 300+ Sources
When every new data source meant another hand-written pipeline, the team became the bottleneck. Here's how a config-over-code framework — and the team behind it — turned days of work into hours.
Problem
Every new data source meant hand-writing another bespoke pipeline — slow to build, inconsistent in quality, and a maintenance burden that grew with every addition. The data team had become the single bottleneck standing between the business and its data.
Business Context
Demand for new data far outstripped the team's capacity to build pipelines one at a time. Scaling by hiring linearly with sources was neither affordable nor fast enough; the delivery model itself had to change.
Architecture
I designed a framework where new pipelines are declared as configuration rather than written as code. A single, well-tested engine reads that config and handles ingestion, typing, quality checks, and loading generically — so the differences between sources became data, not logic.
Technical Decisions
Quality gates and structured logging were baked into the engine, so every pipeline inherited them identically. Because the differences between sources lived in config, a fix or improvement to the engine instantly benefited all 300-plus pipelines at once.
Trade-offs
A framework costs more up front than the first handful of pipelines it replaces — you build the factory before the first product. That only pays off at scale, which is exactly where this platform was heading, so the investment was deliberate rather than premature.
Implementation
I led the team to migrate existing pipelines onto the framework incrementally while onboarding new sources through it from day one. Crucially, the config-driven model let analysts onboard many sources themselves, moving work off the engineering critical path entirely.
Challenges
The framework had to be genuinely good enough that engineers trusted it over writing their own pipeline — a high bar. Getting the abstraction right, and bringing the team along so they championed it rather than worked around it, was as much a leadership task as a technical one.
Performance Improvements
New-pipeline delivery fell from days to hours — often minutes — and quality became consistent across the estate because it was enforced in one place rather than reimplemented per pipeline.
Business Value
The data team stopped being the bottleneck. Onboarding a source became a self-serve config change, throughput rose sharply without linear hiring, and the whole organisation got faster access to trusted data.
Future Enhancements
Layering agents on top of the framework so that onboarding a source can start from a natural-language request — the agent drafts the config and tests, and an engineer reviews the pull request.
Let's build something intelligent together.
Whether you're modernising a data platform or bringing agentic AI into production, I'd love to hear what you're working on.