Article
How to Build a Seed List to Benchmark Email Verifiers (Step by Step)
By Unlimited Verifier Team ·

Summary
A seed list for benchmarking email verifiers is a labelled set of roughly 3,000 addresses whose true status you know independently, run through each verifier and scored against that ground truth. Judge tools mainly by false-valid rate, since marking a bad address as valid is the costly error, and use valid recall and over-flagging as secondary checks.A seed list for benchmarking email verifiers is a set of about 3,000 addresses whose true status you already know, run through each verifier and scored against that ground truth. A usable seed needs roughly 1,000 confirmed-invalid addresses alongside the valid ones, and you should judge verifiers mainly by their false-valid rate, not their headline accuracy.
Headline accuracy is the number vendors print on their pages, and it tells you little about bounce risk. Our accuracy pillar guide covers the wider context. This article shows you how to produce your own number from your own list.
Quick answer
- Build from three sources: confirmed opt-ins that still engage, hard bounces coded 5.1.1 that are at least 180 days old, and addresses you control for structural tests.
- Target about 3,000 addresses: roughly 1,500 known-valid, 1,000 known-invalid, 300 accept-all and 200 role or disposable.
- Score in this order: false-valid rate first, then valid recall, then over-flagging.
- Never mail the invalid set to "double-check" it, and never let a verifier's output become part of the ground truth.
- Re-run every 90 days for shortlisted vendors, because mailboxes get reactivated and closed.
What is a seed list in email verification benchmarking?
A seed list is a labelled test set. Each address carries a ground_truth value that you established independently of any verifier, and you compare every verifier's output against it.
Verifiers return a label per address, usually Valid, Risky or Invalid, though vendors use other names such as deliverable, unknown or undeliverable. Your seed adds the second column: what the address actually is. Put the two side by side and you get a confusion matrix, the same table used to score any classifier. Four cells matter:
- True valid: known good, labelled Valid. This is what you want.
- False valid: known bad, labelled Valid. This is the expensive error, because the address goes into a campaign and bounces.
- Over-flag: known good, labelled Risky or Invalid. This is the quiet cost, because you hold back a real customer.
- Correct rejection: known bad, labelled Risky or Invalid.
Overall accuracy blends the expensive error with the quiet one, which hides which of the two is larger for your purposes.
Bottom line: A seed list turns "is this verifier accurate?" into "how many known-bad addresses does it mark safe, and how many known-good addresses does it hold back?"
Why isn't a vendor's accuracy figure enough?
Accuracy figures usually arrive without the test set behind them. Unlimited Verifier states 99.5% accuracy for its deep verification. That is the company's own figure. It tells you nothing about how many invalid addresses were in the sample, or how many were caught. Ask any vendor, including us, for the class breakdown behind a figure like this. We explore the problem further in are vendor accuracy claims for email verifiers trustworthy and how accurate email verification really is.
Class imbalance is the trap. Suppose 5% of a list is invalid. A verifier that labels every address Valid scores 95% accuracy and catches zero bad addresses. A headline accuracy number rewards whichever tool says "valid" most often, which is the opposite of what you need.
Report three things instead: the false-valid rate on known-invalid addresses, the valid recall on known-valid addresses, and the share of each class sent to Risky. For a fuller treatment of why 99% claims are hard to interpret, see do email verifiers really have 99% accuracy.
Bottom line: Ignore the headline number until you have class-level results from your own seed.
Where should the seed addresses come from?
Each source gives you a different kind of ground truth, with different blind spots.
| Source | How the label is set | Main strength | Main weakness | Use it for |
|---|---|---|---|---|
| Addresses you control | You create a mailbox (valid), or a non-existent local part on a domain that is not catch-all (invalid) | Truth you can verify yourself | Does not behave like real consumer or corporate mailboxes | Checking basic logic |
| Confirmed opt-ins that still engage | Double opt-in plus a click, reply or purchase in the last 90 days | Strong evidence of a live, human-owned mailbox | Skews toward engaged people; mailboxes can close later | The valid class |
| Hard bounces coded 5.1.1, aged 180+ days | Provider's permanent failure, confirmed against the bounce text | Real invalid addresses from your own list | Some mailboxes get reactivated; some rejections are policy blocks | The invalid class |
| Domains configured to accept everything | Labelled accept-all; no true valid/invalid status exists | Shows how a verifier handles catch-alls | Cannot be scored as right or wrong | Reporting separately |
| Disposable and role addresses | Labelled by domain list or local-part rule (for example, info@) |
Tests whether flags fire | Disposable domain lists change constantly | Flag detection |
| Purchased or scraped lists | Unknown | Nothing you can rely on | Spam traps and consent gaps | Do not use |
Two practitioner points matter here. First, stratify the seed to mirror your own provider mix. If 40% of your contacts sit on Gmail and 12% on Microsoft 365, your seed should look similar, and you should report results by provider. Verifier behaviour can differ by mailbox provider, which is why we cover how accurate email verifiers are on Microsoft 365 addresses separately.
Second, B2B contacts often decay faster than consumer ones. If you run an email verification program for ecommerce and SaaS, weight your invalid class toward older business-domain bounces.
Most ESPs show the bounce reason text in their logs even when they don't expose raw SMTP codes, so check your export before you plan around the 5.1.1 rule.
Bottom line: Use bounces for the invalid class and confirmed engagement for the valid class. Build your own accept-all and role-address tests, and mirror your provider mix.
How many seed addresses do you need?
Sample size depends on what you are measuring. For a proportion at 95% confidence, the standard textbook formula is n = 1.96² × p(1−p) ÷ E², where p is the expected rate and E is the margin of error you can tolerate. These are standard statistics, not vendor figures. For the broader question of test size, see how many emails to use when testing a verifier.
| What you are estimating | Assumed rate | Target margin | Seeds needed |
|---|---|---|---|
| False-valid rate on invalid seeds | 2% | ±1 point | 753 |
| False-valid rate on invalid seeds | 2% | ±0.5 points | 3,012 |
| Valid recall on valid seeds | 97% | ±1 point | 1,118 |
| Valid recall on valid seeds | 99% | ±1 point | 380 |
The invalid class needs the most seeds, because its error rate is small and hard to pin down. You also rarely need half-point precision. Most real decisions come down to verifiers that differ by one to three points on false-valid.
At 1,000 invalid seeds and a 2% false-valid rate, the margin is about ±0.9 points, which separates most meaningful differences. That is why we recommend 1,500 valid, 1,000 invalid, 300 accept-all and 200 role or disposable addresses, for 3,000 in total.
Bottom line: Plan for about 1,000 invalid seeds at minimum. Smaller invalid sets produce margins too wide to act on.
How do you label ground truth without harming your sender reputation?
A seed file puts personal data and sending risk in the same spreadsheet, so follow four rules.
Never mail the invalid set. Sending to addresses that have already hard-bounced repeats the behaviour that gets senders flagged. Google's sender guidelines and M3AAWG's best practices both push bulk senders toward list hygiene. Your benchmark should generate zero new sends to the invalid class.
Read bounce codes carefully. RFC 3463 defines enhanced status codes. Code 5.1.1 means the destination mailbox is bad, which is your invalid signal. Code 5.2.2 means the mailbox is full, which does not make the address invalid. Codes in the 5.7.x range, such as 5.7.1 (delivery not authorised), usually reflect policy or reputation decisions. Exclude them from the invalid class.
Never label from verifier output. If you add addresses that Verifier A marked Valid to your valid class, the test becomes circular. Ground truth must come from your own evidence.
Minimise and protect personal data. Store the seed securely, limit access and delete it when the benchmark ends. The UK regulator publishes guidance on personal data, and the FTC's CAN-SPAM compliance guide sets the US baseline for commercial email. Check with counsel before reusing opt-in data for anything beyond the benchmark. Our broader guidance on email verification compliance and hygiene covers this in more detail.
Bottom line: Ground truth should come from bounce codes, engagement and addresses you own. It should never come from sending to the invalid set or from a verifier's own answer.
How do you build a seed list step by step?
Follow these eight steps in order. Each one prevents a specific failure that tends to show up in ad-hoc benchmarks.
- Define the decision first. Write down what each label will do in your system, for example: "Valid is mailed, Risky is held back, Invalid is suppressed." Scoring must reflect this, so decide it before any test.
- Measure your provider mix. Count the share of your current list on each major mailbox domain, plus a bucket for corporate domains.
- Harvest valid seeds. Pull contacts with a confirmed opt-in and engagement in the last 90 days. Sample them to match your provider mix, then dedupe.
- Harvest invalid seeds. Pull hard bounces coded 5.1.1 that are at least 180 days old. Exclude 5.2.2 and 5.7.x. Recheck each address, and if it has engaged since, remove it from the invalid class.
- Add structural seeds. Create mailboxes on your own domain for the valid class, non-existent local parts on a non-catch-all domain for the invalid class, a domain you configure to accept everything for accept-all, and a set of role and disposable addresses.
- Normalise and dedupe. Lowercase and trim every address. Remove any address that appears in more than one class, because each address can have only one ground truth.
- Freeze the file. Save a dated CSV with counts per class and per source, and do not edit it during the benchmark. If you tune thresholds, keep a separate holdout set you never tune against.
- Run every verifier on the same file within 24–48 hours. Verifier outputs change over time, so a common window keeps the comparison fair. Record each output exactly as received.
Bottom line: A seed list is only as good as its labels and its freeze. Map decisions and document provenance first, and the scoring becomes straightforward.
Which metrics should I use to score email verifiers?
Score the metrics that match the decision you defined in step 1. If you suppress Risky, your cost is over-flagging. If you mail Risky, your cost is false-valid.
| Metric | Calculation | What it tells you | Working guidance |
|---|---|---|---|
| False-valid rate | Known-invalid labelled Valid ÷ known-invalid | Expected bounces from addresses marked safe | Lower is better; we use under 2% as a working bar for a sending list |
| Valid recall | Known-valid labelled Valid ÷ known-valid | How many good contacts you keep | Compare verifiers at equal false-valid |
| Over-flag rate | Known-valid labelled Risky or Invalid ÷ known-valid | Good contacts you hold back | Weigh against the value of lost sends |
| Invalid catch rate | Known-invalid labelled Invalid ÷ known-invalid | How completely you clean | Read alongside false-valid, never alone |
| Accept-all Valid share | Accept-all labelled Valid ÷ accept-all | How aggressively catch-alls are promoted | Report it; do not score it as right or wrong |
Two details are easy to miss. Treat unknown or timeout results as their own named category and report the count. Silently counting them as Valid or Invalid inflates one metric. If your benchmark produces many of these, start with why your verifier returns so many unknowns.
Second, the verifiers you compare may disagree for structural reasons, such as different handling of catch-all domains or timeouts. Why different email verifiers give different results explains the common causes.
Bottom line: Lead with false-valid rate, then check valid recall and over-flagging before you choose.
How do you run the benchmark with Unlimited Verifier?
This walkthrough uses Unlimited Verifier, but the steps apply to any verifier that accepts a CSV upload or an API call. Benchmark the same mode you plan to use in production.
- Create the seed CSV with four columns:
email,ground_truth(valid, invalid, accept_all or flagged),label_sourceandlabel_date. Save a dated copy and do not touch it again. - Pilot with 150 rows. Sign up for a free account. The Free plan covers 25 lists a month of up to 150 emails each, plus 750 deep verifications a month, with no API access. Results are kept for only 24 hours, so export the same day.
- Upload the file on the upload and verify a list page. CSV and Excel files up to 10MB are accepted, and the email column is detected automatically. Check the first few rows to confirm the detection is correct.
- Run verification and export immediately as CSV, Excel or JSON. Save the raw output file alongside your seed file.
- Join on the email key. Lowercase both columns, then use XLOOKUP, a pivot table or a short pandas merge to attach each result to its ground truth.
- Write the mapping before you score. For example: Valid maps to Valid, Invalid maps to Invalid, Risky maps to flagged. Record how each catch-all lands, and remember that accept-all results are reported, not scored.
- Plan the full run. Pro is $29 a month, or $25 a month billed annually. It covers lists of up to 25,000 emails each, 25,000 deep verifications a month, and results kept for 30 days. A 3,000-address seed fits in one list. Compare costs across tiers on the email verification pricing page.
- Run the other verifiers on the same file through their own upload or API. For scripted runs, Unlimited Verifier's email verification API is included on Pro, with 100,000 emails a month and bulk batches of up to 10,000 per request. The platform has no native CRM, ESP or Zapier integrations, so you script the export, call and re-import yourself. The broader workflow is covered in email verification API and automation.
- Compute the confusion matrix with COUNTIFS in a spreadsheet: one row per ground truth, one column per label. Derive every metric from that table.
- Spot-check disagreements. Pick 20–30 addresses where verifiers disagree and check them one by one with the free email checker. Use this to understand the differences, not to score them.
- Archive everything: raw outputs, the mapping, the seed file and the date. Pro results expire after 30 days, so your archive is your only long-term record.
Bottom line: Pilot on the free tier, run the full seed on Pro, and archive raw outputs before the retention window closes.
What does a worked seed-list benchmark look like?
The composition and results below are hypothetical and illustrate the arithmetic. They are not measurements from any vendor.
Seed composition
| Class | Count | Source |
|---|---|---|
| Known valid | 1,500 | Confirmed opt-ins with engagement in 90 days, mirroring the provider mix |
| Known invalid | 1,000 | 5.1.1 hard bounces, 180+ days old, rechecked |
| Accept-all | 300 | Own accept-all test domain |
| Role or disposable | 200 | Domain and local-part rules |
| Total | 3,000 |
This fits in one Pro list, since Pro allows up to 25,000 emails per list. If you run deep verification on all 3,000 addresses, that uses 3,000 of the 25,000 monthly deep verifications, so one Pro month leaves room for several runs. On the Free plan, the 750 monthly deep verifications would not cover a full seed, and you would need 20 lists of 150 to upload it at all.
Raw results
| Ground truth | Verifier A: Valid / Risky / Invalid | Verifier B: Valid / Risky / Invalid |
|---|---|---|
| Known valid (1,500) | 1,462 / 21 / 17 | 1,488 / 6 / 6 |
| Known invalid (1,000) | 14 / 42 / 944 | 31 / 58 / 911 |
| Accept-all (300) | 96 / 204 / 0 | 45 / 255 / 0 |
Role and disposable results (flagged as Risky or Invalid): Verifier A flagged 180 of 200, and Verifier B flagged 185 of 200.
Metrics
| Metric | Verifier A | Verifier B |
|---|---|---|
| Valid recall (1,462 ÷ 1,500; 1,488 ÷ 1,500) | 97.5% | 99.2% |
| False-valid rate (14 ÷ 1,000; 31 ÷ 1,000) | 1.4% | 3.1% |
| Over-flag rate (38 ÷ 1,500; 12 ÷ 1,500) | 2.5% | 0.8% |
| Invalid catch rate (944 ÷ 1,000; 911 ÷ 1,000) | 94.4% | 91.1% |
| Accept-all labelled Valid (96 ÷ 300; 45 ÷ 300) | 32% | 15% |
Interpretation
On the headline, Verifier B looks better, with 99.2% valid recall against 97.5%. On bounce risk, B is worse. It lets 31 known-invalid addresses through per 1,000, against 14 for A, more than double.
The margins matter too. At 1,000 invalid seeds, A's false-valid rate sits roughly between 0.7% and 2.1%, and B's between 2.0% and 4.2%. Those intervals barely touch, so treat the result as directional. Doubling the invalid class to 2,000 would narrow the margins to about ±0.5 points for A and ±0.8 points for B.
What the trade-off costs at scale
Take a 500,000-contact list with an illustrative 10% invalid share: 450,000 valid and 50,000 invalid.
- Over-flagging: A holds back 450,000 × 2.5% = 11,250 good contacts. B holds back 450,000 × 0.8% = 3,600. B saves 7,650 good contacts.
- Bounce leakage: A lets through 50,000 × 1.4% = 700 invalid addresses. B lets through 50,000 × 3.1% = 1,550. B adds 850 extra false-valid addresses.
B wins on contacts retained and loses on bounces. If your sender reputation is healthy, the over-flag cost may matter more. If you are recovering from a bounce problem, false-valid dominates, and A is the safer choice. For context on expected bounce outcomes, see what bounce rate to expect after email verification.
Bottom line: Choose by false-valid rate first, and accept a higher rate only if the contacts you recover are worth the extra bounces.
How often should you refresh the seed list?
Labels decay. A hard-bounced address can be reactivated years later, and a valid account can close within months. Re-run the benchmark every 90 days for the vendors on your shortlist, and re-run immediately after a vendor announces an engine change. Each refresh should reclassify the invalid set: drop any address that has engaged or been reactivated, and add new 5.1.1 bounces that have aged past 180 days.
For a third-party comparison to set against your own results, see the independent email verifier accuracy benchmark for 2026. For the ongoing routine that keeps your live list close to the seed's quality, read email list maintenance best practices. If you want checks at signup rather than after the fact, real-time email validation covers that workflow.
Bottom line: Treat the seed as a living asset with a 90-day review cycle, not a one-off test.
What are the common mistakes in benchmarking email verifiers?
- Building the invalid set from one provider's bounces. A single provider's bounce codes skew the invalid class. Mix bounce sources across providers so the set reflects your real list.
- Reporting accuracy without class counts. A 97% figure means little if the invalid class was small.
- Letting verifier output leak into ground truth. Adding addresses a verifier called Valid to your valid class makes the test circular.
- Counting unknowns silently. Give unknown or timeout results their own category and report the count.
- Running verifiers on different days. Outputs shift, so run everything within the same 24–48 hour window.
- Mailing the invalid set to "confirm". You create new bounces and damage the reputation you are trying to protect.
- Benchmarking the Free plan for a full seed. Free caps lists at 150 emails and deep verifications at 750 a month. Use Free for the 150-row pilot, then move to Pro.
- Losing results on the Free plan. Free keeps results for only 24 hours, so export on the day you run.
- Testing a mode you won't buy. If production will use deep verification, benchmark deep verification.
- Editing the seed mid-benchmark. Freeze it at step 7 and keep the dated copy.
Bottom line: Most failed benchmarks break one of three rules: the ground truth was not independent, the classes were unbalanced, or the run was not frozen.
How we compared (and limitations)
This article is based on public documentation: RFC 3463 for enhanced status codes, the sender guidelines cited above, and the FTC and ICO guidance on personal data and commercial email. The sample-size formulas are standard statistics, and the worked-example numbers are hypothetical.
We have not run controlled tests against any verifier, and we are not reporting results from anyone's benchmark. Unlimited Verifier's 99.5% deep-verification figure is the company's own stated number. The open-source verification engine is on GitHub, so you can read how its checks are coded. It may not be identical to the hosted service's deep verification, so confirm that before you use it to explain a hosted result.
The method has limits. Your seed reflects your list and may not represent someone else's. Ground truth carries its own error, because a recent opt-in can close its account later. Use the results to compare verifiers on your data, not as a universal ranking. Once the benchmark narrows the field, our buyer's checklist for choosing an email verifier and the best email verification tools guide can help with the rest of the decision.
Bottom line: Use the seed to answer one question for your list: which verifier's Valid label you can trust.
Scoring worked example
Test set: 1,000 known-invalid and 1,500 known-valid addresses.
| Verifier | Known-invalid marked Valid | False-valid rate | Known-valid marked Risky/Invalid | Over-flag rate |
|---|---|---|---|---|
| Tool A | 40 | 4.0% | 120 | 8.0% |
| Tool B | 90 | 9.0% | 30 | 2.0% |
Tool A lets fewer bad addresses through but holds back more good customers. Tool B does the reverse. Scoring false-valid rate first tells you which error you can least afford for your sending program.
Frequently asked questions
How big should an email verifier seed list be?
A practical seed list is about 3,000 addresses, with roughly 1,500 known-valid, 1,000 known-invalid, 300 accept-all and 200 role or disposable addresses. This size gives enough invalid addresses to measure a false-valid rate with reasonable confidence, while staying cheap enough to re-run every 90 days.
What is a false-valid rate in email verification?
The false-valid rate is the share of known-invalid addresses that a verifier labels as Valid. It is the most important metric for benchmarking because these addresses go into campaigns and cause bounces. Lower is better, and it should be scored before valid recall or over-flagging.
Where do I get known-invalid addresses for a seed list?
Use hard bounces coded 5.1.1 (mailbox does not exist) that are at least 180 days old, since these are less likely to have been reactivated. Do not mail the invalid set to check it. Record the bounce code and date as your ground-truth source.
Where do known-valid addresses come from?
Use confirmed opt-in subscribers who have engaged recently, such as opening or clicking within a set window. Engagement is a strong signal the mailbox exists, though it is not proof, so treat this set as a practical ground truth rather than a certainty.
Why do accept-all domains need their own category?
Accept-all servers reply positively to every address, so a verifier cannot confirm a mailbox from the SMTP response alone. Keeping about 300 accept-all addresses in the seed shows how each tool handles this ambiguity, rather than hiding it inside an overall accuracy number.
What is over-flagging and why does it matter?
Over-flagging is when a known-valid address is labelled Risky or Invalid. It is the quiet cost of a verifier, because you hold back real customers from campaigns. Track it as a separate rate alongside false-valid rate.
How often should I re-run a verifier benchmark?
Re-run the benchmark about every 90 days for shortlisted vendors. Mailboxes get reactivated and closed over time, so ground truth drifts, and a tool that scored well last quarter may behave differently now.
Can I use a verifier's output to build the ground truth?
No. If a verifier's labels become part of the ground truth, you are grading it against itself and the results are circular. Ground truth must come from sources independent of every tool you test, such as bounce records and opt-in data.
Cut Bounces and Protect Your Sender Reputation With Unlimited Email Verification
Every invalid address you send to hurts your sender reputation and your inbox placement. Verifying your list before each send means fewer bounces, healthier deliverability, and a predictable cost. With Unlimited Verifier, standard verification is unlimited on every plan, with no per-email fees.
Unlimited Verifier checks syntax, domain, MX records and the SMTP mailbox. It also detects catch-all domains and flags disposable and role-based addresses. Export a CSV from your email platform or CRM, upload and verify the list, then re-import the clean results. You can also automate checks with the email verification API.
| Free | Pro | |
|---|---|---|
| Price | $0 | $29/month ($25/month billed annually) |
| Lists | 25 a month, up to 150 emails each | Unlimited, up to 25,000 emails each |
| Deep verifications | 750 a month | 25,000 a month |
| API access | No | Yes (100,000 emails a month) |
| Result retention | 24 hours | 30 days |
Why teams choose Unlimited Verifier
- Unlimited standard verification on all plans, including Free, with no per-email fees.
- Catch-all detection so you know which domains accept every address.
- Clear Valid / Risky / Invalid results, exportable as CSV, Excel or JSON.
- An open-source verification engine you can inspect on GitHub.
Unlimited Verifier states 99.5% accuracy for deep verification.
Create a free account and verify your next list before you hit send.
Want to test a single address first? Try the free email checker.