Data Pipeline Modernisation Checklist: 12 Steps Before You Migrate
Legacy ETL estates are the riskiest part of most cloud programmes — not because the technology is hard, but because the knowledge is scattered. This checklist condenses what consistently separates migrations that land on time from those that do not, drawn from SSIS, Talend, Informatica, DataStage and SAP FI estates.
Phase 1 — Discovery
- Inventory all pipelines: source tool, version, owner, schedule, upstream and downstream dependencies.
- Classify by criticality: revenue-impacting, regulatory, operational, historical.
- Map every source and target system: databases, files, APIs, message queues.
- Document the business rules: transforms, lookups, slowly changing dimensions, custom scripts.
Phase 2 — Design
- Choose the target architecture: Databricks, Microsoft Fabric, dbt, Snowflake, plus an orchestrator.
- Define parity criteria per pipeline: row counts, aggregates, edge cases, null handling.
- Plan the orchestration layer: Airflow, Databricks Workflows, Azure Data Factory or Fabric pipelines.
Phase 3 — Migration
- Translate the logic: preserve semantics, not syntax.
- Generate parity tests from synthetic data plus real samples, including month-end edge cases.
- Route exceptions to human review: engineers validate whatever cannot be translated confidently.
Phase 4 — Cutover
- Run old and new in parallel for at least one full cycle.
- Keep the old system warm with a tested rollback plan until parity is proven.
Why the order matters
The phases are sequenced so that every expensive decision is made with the most information available at the cheapest moment. Discovery findings shape the target architecture; parity criteria defined during design drive the tests you build during migration; and the parallel run only works if rollback was planned before anyone needed it. Teams that reorder these — typically by starting builds during discovery — pay for it in UAT.
Three failure patterns this checklist prevents
- The scope explosion. Mid-migration, someone finds a pipeline nobody documented that feeds the board pack. Criticality classification in Phase 1 exists precisely so this happens in week two, not week twenty.
- The silent semantic drift. Row counts match after migration but a rounding rule changed, and finance figures shift by small amounts. Parity criteria defined up front — aggregates and edge cases, not just counts — catch this in testing, not in a reconciliation meeting.
- The one-way door. The legacy system is decommissioned before parity is proven, and a defect is found with no way back. The parallel-run and rollback steps are boring until the day they save the programme.
Where tooling helps — and where it does not
Tools are good at the mechanical middle: reading legacy artefacts, translating logic, generating parity tests. They do not decide which pipelines matter, what parity means for your figures, or when it is safe to switch things off. Those remain engineering and business decisions — which is why the checklist, not the tool, is the centre of the programme. If you want to see how the mechanical steps can be automated, LogicLift covers inventory, translation, parity testing and human-review routing; the checklist tells you when to use them.
Starting a modernisation programme?
Book a free 30-minute migration readiness assessment. We will walk your estate against this checklist and flag the gaps most likely to bite.
Frequently asked questions
What is the most common data pipeline modernisation mistake?
Starting to build before finishing discovery. Teams that skip the inventory and business-rule documentation phase discover the real complexity in UAT — the most expensive place to find it. The second most common: testing row counts instead of business figures.
How long should the discovery phase take?
For a mid-size estate (50–200 pipelines), two to four weeks with a dedicated engineer. It feels slow because there is little visible output — but it is the cheapest phase to fix mistakes in by an order of magnitude.
Do we need a data migration checklist if we use a migration tool?
Yes — a tool accelerates translation and testing, but the checklist governs the programme: criticality classification, parity criteria, parallel runs and rollback are decisions, not automations.
Which orchestrator should we standardise on?
The one your target platform makes easiest. Databricks shops should look at Databricks Workflows; Microsoft estates at Data Factory or Fabric pipelines; multi-platform teams at Airflow. Do not migrate pipelines and swap orchestrators in the same release if you can avoid it.