How much row-count variance is normal between deliveries?

It depends on the file and the cadence, which is why the baseline matters more than any universal threshold. Establish the normal weekly range over several deliveries, then set your alert against that observed range. A file that grows steadily by half a percent per week should raise an alert at a three percent jump, while a volatile feed may need a wider band.

Why hash the header instead of just checking the column names?

A hash catches ordering as well as naming, and it catches whitespace and encoding changes that a name comparison can miss. Column reordering is the dangerous case because it loads without error and shifts every downstream field, so the check needs to be sensitive to position and not only to the set of names present.

Should QC failures block the load or just warn?

Decide per check, in advance, and write it down. Structural failures such as a changed header or a missing segment should block, because loading them corrupts downstream data. Distribution drift is usually a warning that needs human judgment. The important part is that the decision exists before the first failure rather than being improvised while a client waits.

Where should the QC baseline live?

Next to the ingestion code, in version control, as a plain file. A baseline held in someone's notebook or memory cannot be diffed, reviewed, or handed over. Committing it means every change to what counts as normal is a reviewable event with an author and a date.

How to QC a Data File Before You Load It

Compare Against Last Week, Do Not Just Inspect This Week

The instinct on receiving a file is to open it and look. Looking finds almost nothing. What finds problems is comparison: this delivery against the last one, on a small number of stable measures. A row count on its own is neither good nor bad. The same row count is a normal weekly gain if last week was slightly lower, and an incident if last week was materially higher. The number carries no meaning until it sits next to its predecessor.

This is why the baseline matters more than the checklist. Write the measures to a file, commit it next to the ingestion code, and diff each delivery against it, which is the same discipline behind drift monitoring for production feeds. The checks themselves are simple. The discipline of keeping a comparable record is what makes them work.

The Six Checks

What to measure on every delivery

Check	What it catches	How to measure it
Row count delta	Truncated transfers, partial extracts, duplicated appends.	Count data rows, compare to the prior delivery, and state the expected range.
Header stability	Renamed, reordered, added, or dropped columns.	Hash the header row and compare the hash. A changed hash is a schema event.
Encoding and line endings	Mixed CRLF and LF, stray control bytes, an unexpected byte order mark.	Count line-ending types and any control characters outside the delimiter set.
Primary key uniqueness	Supplier-side join changes and accidental record duplication.	Count distinct keys against total rows. Investigate any first-ever duplicate.
Distribution drift	A field that silently changed meaning or stopped populating.	Track the distribution of the status and category fields you consume, week over week.
Segment completeness	A whole state, region, or category missing from an otherwise healthy file.	Assert the expected set is present, not merely that the count is close.

Hash the Header, Because Column Drift Is Silent

Of the six, header hashing repays its cost fastest. Row-count problems announce themselves. A reordered column does not. It loads cleanly, the pipeline reports success, and every downstream field is shifted by one position for as long as it takes somebody to notice that the city column contains postal codes. Hashing the header row turns that from an eventual discovery into an immediate failure.

Keep the expected header as an artifact rather than a constant in code, and keep its hash next to it. A supplier's data dictionary is where that expected shape should come from in the first place. When a supplier legitimately adds a column, the check fails, you look, you update the artifact deliberately, and the change is recorded. That is the behavior you want.

Completeness Is Not the Same as Count

A total row count can look entirely normal while a whole segment is missing, because another segment grew enough to mask it. This is the same failure a matched-sample pilot is designed to surface before production. If your file is supposed to contain every county, every state, or every product category, assert the set rather than the size. Enumerate what should be there, diff it against what is there, and fail on the difference.

This check is worth running even when nothing has ever gone wrong, because the first time it fires it will be for something no count-based check would have surfaced. The same reasoning applies to evaluating suppliers in the first place, which the bake-off checklist covers in more detail.

Write the Handling Rule Down

Every check needs a decision attached to it, agreed before the first failure rather than during it, the same way acceptance criteria are set in the evaluation checklists before a vendor round starts. Does a five percent row-count drop block the load or raise a warning? Does a first-ever duplicate key stop ingestion? Does a new column fail the run or get accepted with a note? Deciding under time pressure, with a client waiting, is how a one-off exception becomes the permanent standard.

Decide Block Versus Warn Per Check, in Advance

The useful split is structural against statistical. A structural failure means the file is not shaped the way the pipeline expects: a changed header, a missing segment, a duplicate primary key, an encoding shift. Loading those corrupts data downstream, so they block. A statistical failure means the file is shaped correctly and the numbers moved: a row-count swing, a distribution shift, a field that got sparser. Those need a human to look, because the same signal can mean a supplier problem or a real change in the world.

Write the rule where the check lives, not in a runbook nobody opens at eleven at night. A check whose failure mode is undocumented becomes a judgment call made by whoever is on shift, and two people will make it differently. That inconsistency is worse than either answer, because it means the same condition produces a load on Tuesday and a halt on Thursday, and nobody can explain to the client which one was correct.

There is one rule worth making absolute: never let a check that could not run report as a check that passed. Prove each check can fail, the way a seed match test is only meaningful when the seed is yours. A QC step that errors on a missing file, a permissions problem, or a bad connection must fail loudly. Folding could-not-measure into passed is how a pipeline reports green over a file it never actually examined, and it is the single most common way a QC suite becomes decorative while still being run on schedule.

Frequently Asked Questions

How much row-count variance is normal between deliveries?: It depends on the file and the cadence, which is why the baseline matters more than any universal threshold. Establish the normal weekly range over several deliveries, then set your alert against that observed range. A file that grows steadily by half a percent per week should raise an alert at a three percent jump, while a volatile feed may need a wider band.
Why hash the header instead of just checking the column names?: A hash catches ordering as well as naming, and it catches whitespace and encoding changes that a name comparison can miss. Column reordering is the dangerous case because it loads without error and shifts every downstream field, so the check needs to be sensitive to position and not only to the set of names present.
Should QC failures block the load or just warn?: Decide per check, in advance, and write it down. Structural failures such as a changed header or a missing segment should block, because loading them corrupts downstream data. Distribution drift is usually a warning that needs human judgment. The important part is that the decision exists before the first failure rather than being improvised while a client waits.
Where should the QC baseline live?: Next to the ingestion code, in version control, as a plain file. A baseline held in someone's notebook or memory cannot be diffed, reviewed, or handed over. Committing it means every change to what counts as normal is a reviewable event with an author and a date.