Data engineering for teams that answer for their numbers.
Pipelines that ingest from source, validate at every stage, keep the raw data intact for audit, and stop rather than publish a figure they cannot verify. Built by an engineer whose day job is investment data, where a wrong number reaches people who ask why.
Most reporting breaks in the same three ways
Almost every reporting problem I am asked to fix turns out to be one of three things, and none of them are solved by a better dashboard.
Nobody can explain the number. A figure is questioned, and answering means redoing the work. There is no record of which source file it came from, which rules were applied, or what changed between last month and this one. The report is trusted because it has always been there, not because it can be verified.
The logic lives in one person's head. A workbook has accreted rules over years. The person who built it can run it. Nobody else can, and when they are on leave the close waits. This is the single most common risk I find and the least often acknowledged.
Failures are silent. A source file arrives short by two thousand rows, or a column shifts, and the pipeline processes it anyway. The output looks normal. The problem is found weeks later, usually by someone outside the team. A pipeline that fails loudly is far safer than one that appears to work.
How the pipelines are built
Raw layer, never modified
Source data lands exactly as it arrived and is kept. Reprocessing never means asking anyone to resend a file, and any number can be traced back to the exact input that produced it.
Validation on ingestion
Schema checks, row counts, control totals against source, and explicit handling for the cases that should never occur. Anything unexpected raises a flag instead of flowing through.
Conformed layer
Cleaning, type coercion, deduplication, and business rules applied as documented, versioned logic rather than formulas buried in a workbook. The rules become readable, reviewable, and testable.
Reporting layer
Business-ready tables that dashboards and reports read from. One definition of each metric, so two people asking the same question get the same answer.
Parallel run before cutover
The new pipeline runs alongside the existing process until both agree line for line. Only then is the manual process retired. This step is not optional and it is what makes the handover safe.
Documentation and handover
Written rules, a runbook, and a live walkthrough. The objective is a system your team owns, not one that depends on me being available.
Typical engagements
Ingestion pipelines
Pulling from APIs, SFTP drops, email attachments, database extracts, and file shares on a schedule, with retry handling and alerting when a source does not arrive.
Data quality frameworks
Reusable validation rules applied consistently across sources, with a quarantine path for records that fail rather than silent dropping.
Document and PDF extraction
Structured data pulled out of unstructured statements, reports, and invoices, with arithmetic validation and confidence thresholds routing uncertain cases to review.
Consolidation and reporting
Multi-source, multi-entity consolidation producing one set of figures with a documented derivation for each.
Pipeline rescue
Inheriting a fragile, undocumented pipeline that fails intermittently. Instrumentation first so failures are visible, then refactoring once behaviour is understood.
Migration to a managed platform
Moving workbook-based or scheduled-script processes onto Databricks or the Power Platform, without losing the business logic embedded in the old process.
Tools, chosen to fit the problem
Python and SQL for transformation logic. Databricks and Spark where volume or lineage requirements justify it. Azure Data Factory, Power Automate, and Power Query where the work sits inside a Microsoft estate and needs to be maintainable by the team that owns it. Delta Lake for versioned, auditable storage. Streamlit and Power BI at the presentation end.
The platform choice follows the constraint, not the other way round. A finance team inside a locked down Microsoft 365 tenant is better served by a well-built Power Platform solution their IT department will actually approve than by a technically superior stack they cannot deploy. Most of the value in this work is in the validation and the documented rules, which are portable across all of it.
Explore further
Common questions
A data pipeline moves data from where it is created to where it is used, applying cleaning, validation, and transformation along the way, on a schedule and without anyone touching it. You need one when the same manual export-clean-paste sequence happens every week, when reports disagree because two people built them differently, or when nobody can explain how a number was derived. If your reporting depends on one person running a workbook, that is the signal.
Automation removes a manual step. Data engineering builds the structure underneath it, so the data arriving is validated, versioned, and traceable. A script that emails a report is automation. A pipeline that ingests raw source files, checks them against expected schema and control totals, keeps the raw copy intact for audit, and only then produces the report is data engineering. In finance the second matters, because you need to be able to prove where a figure came from.
Medallion architecture organises a pipeline into three layers. Bronze holds raw source data exactly as it arrived, never modified. Silver holds cleaned, conformed, validated data. Gold holds the business-ready tables that reporting reads from. The value is that when a number looks wrong you can trace it back through each layer to the original source file, and reprocessing does not mean re-requesting data from anyone.
Both. A large share of the work is inheriting a pipeline that has grown organically, is failing intermittently, and has no documentation, then stabilising it. That usually means adding validation and logging first so failures become visible, then refactoring the fragile parts once the behaviour is understood.
Validation at every stage rather than at the end. Schema checks on ingestion, control totals against source, row count reconciliation between layers, and explicit handling for the cases that should never happen. A pipeline should stop and raise a flag rather than publish a figure it cannot verify. Silent wrong numbers are the failure mode that actually costs money.
Have a pipeline that needs building — or rescuing?
A free 30-minute review will tell you what is actually wrong, what it would take to fix, and whether it is worth doing.