There's a moment every growing data team hits: you're onboarding your fortieth data source, and you realise you've written forty nearly-identical pipelines. Each one ingests, types, validates, and loads. Each one is slightly different, slightly buggy, and impossible to change consistently.
This is the signal to stop writing pipelines and start writing a framework.
The insight
Most pipelines are 90% the same. The differences — source location, schema, quality rules, target table — are data, not logic. So represent them as data.
Instead of a Python file per pipeline, you have a config entry:
- name: policies
source: s3://landing/policies/
format: parquet
quality:
- not_null: [policy_id]
- unique: [policy_id]
target: silver.policies
A single, well-tested engine reads that config and does the rest.
What you gain
- Speed. New pipelines go from days to hours — often minutes.
- Consistency. Logging, quality, and error handling are identical everywhere, because they live in one place.
- Self-serve. Analysts can onboard sources by editing config, no engineering ticket required.
- Maintainability. Fix a bug once in the engine, and every pipeline benefits.
The trade-off
A framework costs more upfront than the first few pipelines it replaces. You're building a factory before you've built the first product. The payoff is exponential — but only if you actually reach scale. For a team with five pipelines, skip it. For a team heading toward five hundred, it's the difference between shipping and drowning.
The deeper lesson
The best abstractions don't just save keystrokes — they change who can do the work. When onboarding a data source becomes a config change, the bottleneck moves off the engineering team entirely. That's the real win.