Data Evaluation Checklists (Pilot, Match, Governance)
High-intent buyer checklists for evaluating data feeds: match-rate targets, exclusions, delivery specs, and contract terms that prevent downstream rework.
How teams use these checklists
Copy each section into your vendor-comparison memo or RFP scorecard. Every check below names what to ask or test, what a pass looks like, and what a fail usually signals, so legal, privacy, data science, and procurement can score finalists against evidence rather than marketing claims. Pair the lists with a matched-sample pilot scoped on the pilot process page.
What to check first, and why
- 1
Governance knockouts first
Permitted use, sensitive-category exclusions, and onward-transfer terms can end an evaluation outright, and legal review is usually the slowest calendar to book. Clearing them first means nobody spends engineering time testing a feed your counsel will not approve.
- 2
Matched sample second
Once the license can cover your use case, test the data itself on your own seed, with thresholds written down in advance. This is where most of the effort goes, so it should only run on vendors that cleared step one.
- 3
Delivery third, across two cycles
A sample prepared by hand tells you little about production. Load a real delivery through the path you will use, then load the next one and compare the two.
- 4
Contract terms last
Write what the first three steps proved into acceptance criteria, deletion obligations, and exit terms, so the contract describes the feed you actually tested rather than the one in the deck.
Matched-sample checklist
Run this on a seed file you control. The goal is a result you could defend to someone who was not in the room, so every check names what you send, what comes back, and what you would accept.
Match keys and joins
- Ask or test:
- Which keys will the vendor match on (hashed email, postal address, phone, device ID), and which joins does your downstream model need?
- A pass looks like:
- The vendor states keys, normalization rules, and hashing method in writing before the sample runs, and the returned file joins to your tables without custom cleanup.
- A fail usually signals:
- Keys or normalization explained only after results come back, which usually means the match logic was tuned to the sample.
Thresholds per cohort
- Ask or test:
- What match and fill rates count as a pass for each geography, cohort, and channel you care about? Set a threshold per cohort before testing.
- A pass looks like:
- Results reported per cohort against the bar you set, including the cohorts that came in low.
- A fail usually signals:
- A single blended rate. A strong overall number can hide a cohort that matters to you and barely matched.
Known answers and planted non-matches
- Ask or test:
- Can you include records whose correct answer you already know, plus records that should not match anything?
- A pass looks like:
- Known records come back correct and the planted records stay unmatched.
- A fail usually signals:
- Planted records come back matched, which points to loose matching or appended fill that will not hold up in production.
Freshness and decay
- Ask or test:
- What do last_seen or event dates look like on matched records, and how does the vendor age out stale ones?
- A pass looks like:
- Dates are present and plausible, and the vendor can describe its refresh cadence and the rule that retires old records.
- A fail usually signals:
- Missing or identical dates across the file, which often means a snapshot reissued under a new date.
Opt-out propagation
- Ask or test:
- Submit a test suppression and ask how it reaches files that were already delivered.
- A pass looks like:
- A documented path, a stated turnaround, and evidence the suppressed record is gone from the next delivery.
- A fail usually signals:
- A general promise to honor opt-outs with no mechanism behind it, which leaves your team carrying the deletion risk.
Governance checklist
These items decide whether you are allowed to use the data the way you plan to. Most of them are knockouts, so score them before anyone builds a pipeline.
Permitted use
- Ask or test:
- Does the license name your activation and measurement use cases, including modeling and any sharing with agencies or partners?
- A pass looks like:
- Your exact uses appear in the order form or an exhibit.
- A fail usually signals:
- Broad marketing-purposes language that legal will have to interpret later, usually at the worst possible moment.
Sensitive-category exclusions
- Ask or test:
- Which categories are excluded at the source (health conditions, visits to sensitive locations, data on minors), and how is the exclusion enforced?
- A pass looks like:
- A written exclusion list plus the method, such as field removal or location suppression, tied to the data dictionary.
- A fail usually signals:
- Exclusions described as policy with no field-level evidence you can check.
Provenance
- Ask or test:
- How was the data collected, under what notice and consent, and can the vendor document the chain for the feed you are licensing?
- A pass looks like:
- Sourcing documentation you can file with the evaluation, consistent with the vendor's published methodology.
- A fail usually signals:
- Provenance answered only by the sales team, or deferred until after signature.
Retention and deletion
- Ask or test:
- How long may you keep the data and derived outputs, and how is deletion evidenced when the term ends or a consumer request arrives?
- A pass looks like:
- Retention and deletion obligations written into the contract, with an operational process someone at the vendor can walk you through.
- A fail usually signals:
- Deletion terms that exist in the DPA but that nobody on the vendor side can explain in practice.
Subprocessors and audit rights
- Ask or test:
- Who else handles the data, and what audit or assessment rights does the contract give you?
- A pass looks like:
- A current subprocessor list and DPA language that matches it.
- A fail usually signals:
- A list that differs from what your security review turns up.
Delivery checklist
Delivery problems rarely show up on a sample. They show up on the third load job, after the contract is signed. These checks pull them forward.
Delivery path
- Ask or test:
- Which path will production use (S3, SFTP, Snowflake share, or API), and can the sample ship through that same path?
- A pass looks like:
- The sample arrives through the production path with the same permissions model you will run.
- A fail usually signals:
- The sample arrives as an emailed file, so the real integration stays untested until after signature.
Schema stability and change control
- Ask or test:
- How are new fields, renamed columns, and new enum values announced, and how much notice do you get?
- A pass looks like:
- A change-control process with advance notice and versioned dictionaries.
- A fail usually signals:
- Changes discovered when a load job breaks.
Manifests and checksums
- Ask or test:
- Does each delivery carry a manifest with row counts and checksums?
- A pass looks like:
- Your loader can confirm completeness automatically before anything downstream reads the file.
- A fail usually signals:
- No way to tell a partial delivery from a small one.
Region and encryption
- Ask or test:
- Where is the data stored and transferred, and how is it encrypted at rest and in transit?
- A pass looks like:
- Answers that match your security policy and, for public-sector work, the required cloud region.
- A fail usually signals:
- Region or encryption details that change between the sample and the production quote.
Second delivery
- Ask or test:
- Load the next scheduled delivery and diff it against the first.
- A pass looks like:
- Same schema, comparable fill, and row-count movement you can explain.
- A fail usually signals:
- Drift on the second cycle, which is a preview of what production will look like.
Scoring finalists with the checklist
- Mark each check pass, partial, or fail, and attach the artifact that justifies the mark (the sample result, the manifest, the contract clause, the suppression evidence). A mark with no artifact behind it counts as a fail.
- Treat the governance items as knockouts. A fail on permitted use or sensitive-category exclusions removes a vendor no matter how well its data matched.
- Weight the remaining checks before any results arrive, and use the same weights for every finalist. Write the weights into the memo so the decision can be reconstructed later.
- Break ties on the second delivery rather than the sample. The vendor whose second cycle looked like its first is usually the cheaper one to run.
- Keep the scorecard for the vendors you rejected. When the category comes back up for renewal, the reasons a supplier lost are the first thing anyone asks for.
Why a checklist beats a questionnaire
A vendor questionnaire collects assertions. A checklist produces evidence. The difference shows up months later, when someone asks why a feed was approved and the only record is a spreadsheet of yes answers with no artifact behind them. Write each item so it names the thing that would be attached: a matched-sample result on your own seed, a delivery manifest, a signed exhibit, a screenshot of a suppression run. If an item cannot produce an artifact, it belongs in a conversation rather than in a control.
Run the same checklist on every vendor
Comparability is the point. Scoring one supplier on match rate and another on coverage tells you which questions each sales team preferred, not which feed is better. Fix the seed, fix the geography and cohort splits, fix the success thresholds, and hold all three constant across every vendor in the round. Decide the thresholds before you see any results, because a bar set after the fact is a description of the winner rather than a test. Record the ones that failed and why.
The items teams most often skip
- Opt-out and deletion propagation: how a suppression reaches an already-delivered file, and how long it takes.
- Second-delivery stability: everything looks clean on the sample a supplier prepared. Schema drift, fill-rate changes, and enum additions only appear on the next cycle.
- Onward transfer: what happens when your team shares derived output with an agency or a downstream partner, and whether the license actually permits it.
- Exit terms: what you keep, what you must delete, and how deletion is evidenced when the contract ends.
Vendor comparison framing
When stakeholders ask “GSDSI vs ___”, use the comparison pages as a rubric, then run the same checklist against every finalist so apples-to-apples scoring holds.
- Privacy & compliance hub: post-FTC diligence threads.
- Clean room measurement: governed join patterns.
- Trust Center: security program and subprocessors.
Suggested path
Start with the pilot process, then review sourcing methodology for provenance and exclusions. For measurement-specific workflows, see clean room measurement.
Frequently asked questions about data evaluation checklists
What should a matched-sample checklist cover?
Define seed keys, geography and cohort segments, match-rate thresholds, freshness filters (last_seen or event date), opt-out propagation proof, and the delivery path you will use in production. Document pass/fail before expanding volume.
What governance items belong in procurement diligence?
Confirm sensitive-category exclusions, permitted uses, retention and deletion SLAs, downstream sharing rules, subprocessors, and audit rights. Align activation and measurement use cases with the privacy policy before scale.
What delivery checks prevent rework?
Validate format (Parquet, CSV, JSON), partition strategy, manifest checksums, region, refresh SLA, and warehouse compatibility (Snowflake share, S3, SFTP, API). Run a small end-to-end load in the buyer environment before signing production volume.
How does this relate to the GSDSI pilot process?
They follow the workflow on /pilot-process with one change in order: governance knockouts come first, because a vendor that fails on permitted use or sensitive-category exclusions is not worth a matched sample. Then run the matched sample, confirm delivery over two cycles, and lock production terms. Use them as an internal memo template during vendor comparison.
Can federal buyers use the same checklists?
Yes. Add GovCloud or Azure Government delivery, CAI-aligned provenance, and socioeconomic verification requirements. Start from /federal-procurement and the federal procurement checklist linked from this page.