Article
How Many Emails to Test an Email Verifier: A 2,000-Address Method
By Unlimited Verifier Team ·

Summary
To test an email verifier, use about 2,000 randomly drawn addresses from your own list, split into four mailbox groups of roughly 500 each. That size gives about ±2.2% precision on the overall rate at 95% confidence. Smaller samples of 100 to 200 are only useful for checking exports, not for ranking tools.To test an email verifier, use about 2,000 addresses drawn at random from your own list, split into four mailbox groups of roughly 500 each, which gives about ±2.2% precision on the overall rate. Sample size depends on the precision you need and the smallest group you want to judge, not on the size of your list. Runs of 100 to 200 addresses are useful for checking files and exports, but they are too small to rank two tools.
Quick answer
- Default size: 2,000 addresses in four groups of about 500, for example Gmail, Microsoft consumer, other consumer, and corporate or custom domains.
- Smoke test only: 100–200 addresses catch formatting, column and export problems. At 100 addresses, a 50% rate carries a ±9.8% margin, so you cannot rank tools with that sample.
- Freeze your labels first: Assign each address a known outcome from bounce logs, clicks and test mailboxes before you upload anything.
- Score the confusion table, not the headline: Track false positives (Valid but actually bad) and false negatives (Invalid but actually good) separately, and treat Risky as its own bucket.
- Run both tools on the same day: Mailbox states change, so a result from last week is not comparable with one from today.
How many emails do you need to test an email verifier?
Use 2,000 addresses for a head-to-head comparison. At 95% confidence, a simple random sample of that size gives a margin of ±2.2% on an overall rate. The table below shows how the margin shrinks as the sample grows.
| Sample size | Margin at a 50% rate | Margin at a 5% rate | Best use |
|---|---|---|---|
| 100 | ±9.8% | ±4.3% | Checking exports and column detection only |
| 385 | ±5.0% | ±2.2% | A rough single overall rate |
| 1,067 | ±3.0% | ±1.3% | Overall comparison of two tools |
| 2,000 | ±2.2% | ±1.0% | Default head-to-head test |
| 10,000 | ±1.0% | ±0.4% | Large audit or repeat regression test |
The 50% column is the worst case, used when you don't know the true rate. The 5% column is closer to what many consumer lists show for invalid addresses, so it is a better guide when your question is "how many are bad?" These margins apply to overall rates. Per-group margins are wider, which is why the grouping in the next sections matters.
Bottom line: Use 2,000 addresses as the default. Go larger only when you need per-group precision or an audit of the full list.
Why do some guides say 385 or 1,067 is enough?
Those figures come from the same formula, applied at different margins. The standard sample-size formula for a proportion is n = z² × p(1 − p) ÷ E², where z = 1.96 for 95% confidence, p is the expected rate and E is the margin you accept. The NIST/SEMATECH e-Handbook of Statistical Methods covers the underlying statistics. Sources disagree on what counts as "enough" because they target different margins. 385 is the textbook figure for ±5%, and 1,067 is the figure for ±3%. For a tool comparison, we recommend 2,000.
Working through it with p = 0.5 and E = 3%:
n = 3.8416 × 0.25 ÷ 0.0009 = 1,067
Tightening the margin to 2% gives 3.8416 × 0.25 ÷ 0.0004 = 2,401. Going from ±3% to ±2% more than doubles the sample, which is why 2,000 is a practical compromise.
Finite population correction matters for small lists. If your list has N = 3,000 rows, the adjusted size is n = 1,067 ÷ (1 + 1,066 ÷ 3,000) ≈ 787. For a 50,000-row list the correction is negligible (about 1,045).
Why headline accuracy is hard to check with small samples. If a vendor's 99.5% accuracy figure were literal, the tool would be wrong on about 1 address in 200. A 400-address sample would expect about two errors, which cannot separate 99% from 99.5%. Unlimited Verifier states 99.5% accuracy for its deep verification. Treat that as the company's own figure, and let your labelled sample decide. For more on this, see do email verifiers really have 99% accuracy and are vendor accuracy claims for email verifiers trustworthy.
Bottom line: 1,067 gives ±3% and 2,401 gives ±2%. Small samples cannot resolve differences of a few tenths of a percent, so don't try.
How big should the sample be for each decision?
Match the sample to the decision you are making, not to a generic rule.
| Decision | Sample | Groups | Precision to aim for | Why |
|---|---|---|---|---|
| Does the file upload and export correctly? | 100–200 | 1 | Not applicable | Tests plumbing, not accuracy |
| Roughly how dirty is my list? | 385–1,067 | 1 | ±3–5% overall | Enough for a trend, not a ranking |
| Which of two tools should I use? | 2,000 | 4 × ~500 | ±2.2% overall, ±4.4% per group | The default head-to-head test |
| Is a bulk clean safe to send? | 2,000 from the cleaned output | 4 × ~500 | ±2.2% overall | Confirms the residual bad rate |
| Has my tool drifted over time? | 2,000 fresh draw each run | Same 4 groups | ±2.2% overall | Compares like with like across runs |
Bottom line: For choosing between tools, 2,000 in four groups is the minimum that gives you usable per-group numbers.
Is a 100-email test enough?
Yes for plumbing, no for accuracy. A 100-address run tells you whether the file uploads, the email column is detected, every address returns a result, and the export opens in Excel. It also shows whether catch-all and disposable domains are flagged the way you expect.
Smoke tests catch the problems that waste a full run. Common ones include a non-UTF-8 export that corrupts accented characters, a leading blank row that shifts the header, and a column named "Work email" that the tool does not detect as the email field. These are cheap to fix at 100 rows and expensive to discover after a 2,000-row run.
Bottom line: Run 100 addresses to validate the workflow, then run 2,000 to make the decision.
How do you build a sample that represents your list?
Use stratified sampling: split the list into mailbox groups first, then draw randomly within each group. Verifiers do not perform uniformly across domains. Results for consumer mailboxes can differ from results for corporate or catch-all domains, so a single pooled sample can hide problems. For group-specific behaviour, see email verifier accuracy on Gmail vs Outlook addresses and how accurate are email verifiers on catch-all domains.
Oversample the small groups. Suppose 10% of your list is corporate domains. A simple random sample of 2,000 would contain about 200 of them, a margin of ±6.9% for that group at a 50% rate. Drawing 500 from each of four groups brings every group to ±4.4%.
Then weight the overall rate back to your list mix. Suppose your list is 46% Gmail, 22% Microsoft, 14% other consumer and 18% corporate. The overall bad rate for a tool is:
Overall = (0.46 × Gmail rate) + (0.22 × Microsoft rate) + (0.14 × other rate) + (0.18 × corporate rate)
Report per-group rates for diagnosis and the weighted figure for the decision. Teams working on email verification for ecommerce and saas often find that their B2B segment behaves very differently from their consumer segment, which is why this split pays off.
Bottom line: Stratify first, oversample the small groups, then weight the overall number back to your real mix.
How do you label addresses so the test is fair?
A test is only as good as its labels. A verifier must never label its own test set, because that makes the result circular. Use outside evidence:
| Ground-truth source | How to label | Strength | Weakness |
|---|---|---|---|
| Hard bounces with SMTP code 5.1.1 | Invalid | A receiving server rejected the mailbox | Only covers addresses you have already mailed. 5.7.x codes are policy blocks, not missing mailboxes |
| Clicks in the last 90 days | Likely valid | Harder to produce by accident than opens | Silence proves nothing |
| Test mailboxes you control | Valid (live) or Invalid (deleted) | Known truth across several providers | Few providers, and setup takes time |
| Made-up names on large consumer domains | Usually invalid | Easy to create | Not guaranteed, so use as a control rather than most of the sample |
| Mailboxes you deleted | Invalid after a delay | A real outcome | Some deleted accounts keep resolving for a while, so wait before labelling |
Opens are not on this list. Since Apple's Mail Privacy Protection preloads remote images, many opens no longer reflect a person reading the message. Clicks are a stronger signal.
The practitioner insight most tests miss: bounce labels exist only for addresses you have already mailed, and the bad ones among them are the easiest for a verifier to catch, because a mail server has already rejected them. A sample drawn only from mailed addresses measures the easy cases. Add a never-mailed slice, labelled through test mailboxes you control, so the test covers addresses that have never bounced.
Two more rules matter. First, role (such as info@) and disposable flags are categories, not validity labels, so keep them out of your valid/invalid ground truth. Second, freeze the labels by saving a timestamped master file before you upload to any tool. If you label after seeing results, you will unconsciously favour whichever tool disagreed with you least. Store that file with your email verification compliance and hygiene records, since it documents how you checked the list.
The SMTP mailbox check relies on the RCPT TO command, defined in RFC 5321. The 5.1.1 "bad destination mailbox" code is defined in RFC 3463.
Bottom line: Label from bounces, clicks and your own mailboxes, never from the tools under test, and save the labels before any upload.
How do you score a verifier from the results?
Build a confusion matrix: compare each address's bucket (Valid, Risky or Invalid) with its ground-truth label. Then compute four numbers:
- False-positive rate: the share of Valid results that are actually bad. This is the costliest error for a sender, because it becomes a bounce.
- False-negative rate: the share of Invalid results that are actually good. This is the cost of lost contacts.
- Invalid catch rate: the share of truly bad addresses labelled Invalid. Bad addresses labelled Risky still need handling.
- Risky share: the portion of the list the tool could not settle. This is not an error, but it shapes your send plan.
Which error matters more depends on the job. Before a campaign, false positives cost you bounces and reputation. For lead generation, false negatives cost you real prospects. For more on what a false positive looks like in practice, see what is a false positive in email verification.
Worked example (hypothetical figures). A 2,000-address sample contains 1,780 known-good and 220 known-bad addresses. Two tools return these results:
| Verifier A | Verifier B | |
|---|---|---|
| Valid (good / bad) | 1,700 (1,686 / 14) | 1,600 (1,590 / 10) |
| Risky (good / bad) | 180 (84 / 96) | 230 (170 / 60) |
| Invalid (good / bad) | 120 (10 / 110) | 170 (20 / 150) |
| False-positive rate | 14 ÷ 1,700 = 0.8% | 10 ÷ 1,600 = 0.6% |
| False-negative rate | 10 ÷ 120 = 8.3% | 20 ÷ 170 = 11.8% |
| Invalid catch rate | 110 ÷ 220 = 50% | 150 ÷ 220 = 68% |
| Risky share | 180 ÷ 2,000 = 9.0% | 230 ÷ 2,000 = 11.5% |
Verifier B catches more bad addresses outright, but it also removes more good ones. The gaps are small: 14 versus 10 false positives, and 10 versus 20 good addresses wrongly rejected. At this sample size, neither gap is statistically clear. The honest conclusion is a trade-off, not a winner. The next step is a larger run, or a hand review of the disagreements.
Bottom line: Score both error types, not just the headline. Use a sample to spot large gaps and a bigger sample to settle close calls.
How do you run the test, step by step?
- Write the decision first. Are you choosing a tool, auditing your current one, or checking whether a bulk clean is safe to send? The decision determines which error type matters most.
- Tag every address with its group. Use the four groups above, or a finer split if your list warrants it.
- Draw a stratified sample. Pull about 500 random addresses per group and leave the rest of the list untouched.
- Label and freeze. Attach ground-truth labels from bounces, clicks and test mailboxes, including a never-mailed slice. Save a timestamped master file.
- Run both tools the same day on identical files, with the same column layout and encoding. Record the depth setting used for each group.
- Build the confusion matrix for each group, then compute the weighted overall rate.
- Hand-review 50–100 disagreements. Where the tools differ, check the address against bounce history or a live test. Keep the labelled file for future regression tests.
For shortlisting tools before you test them, start with our guide to best email verification tools and the buyer's checklist for choosing an email verifier.
Bottom line: Decide, stratify, label, freeze, run together, score, then review the disagreements.
Hands-on tutorial: running a 2,000-address test on Unlimited Verifier
Follow these steps to test Unlimited Verifier, or to compare it with another tool.
- Export your sample as CSV. Create two files: an upload copy with only the email column, and a master file with the email address plus your ground-truth label. Keeping labels out of the upload file prevents them from influencing the result.
- Sign up. The Free plan allows lists of up to 150 emails, so split the upload copy into 14 files (2,000 ÷ 150 = 13.3, rounded up to 14). That uses 14 of the 25 lists the Free plan includes each month.
- Upload and verify a list. CSV and Excel files up to 10MB are accepted, and the email column is detected automatically. Check that the first rows contain real addresses before you run the full set.
- Choose your depth and record it. Standard verification is unlimited on every plan. Deep verification draws on the monthly allowance: 750 on Free, 25,000 on Pro. On Free, a 2,000-address run will exceed the deep allowance, so run standard verification on all 2,000 and deep verification only on the group that matters most. Note which addresses received deep checks, and compare tools at the same depth.
- Export the results the same day as CSV, Excel or JSON. Free-plan results are kept for 24 hours and Pro-plan results for 30 days. Save each export next to your master file.
- Join the files on the email address and count the buckets against your labels. In a spreadsheet, COUNTIFS on the Valid/Risky/Invalid column against the label column gives you the confusion matrix directly.
- Spot-check edge cases with the free email checker. Review catch-all results separately. A catch-all domain accepts every address, so its mailboxes cannot be confirmed, and the catch-all email verification page explains how those results are handled.
- Repeat for the second tool on the same day, using the same upload copy.
If you need to run tests on a schedule or at volume, the email verification API supports single-email checks and bulk batches of up to 10,000 emails per request. API access is part of the Pro plan, not Free. Our guide to email verification API and automation covers how to wire it into your own scripts.
Bottom line: Split for the plan's list cap, verify, record your depth setting, export the same day, then join your labels and count.
Do you need a paid plan to run the test?
Not for the standard part of the test. The Free plan can run a 2,000-address comparison on standard verification, but its 750 deep checks a month fall short of a full deep run. Pro, at $29/month or $25/month billed annually, covers the full deep run and is the only plan with API access.
| Test size | Free plan | Pro plan |
|---|---|---|
| 100 | 1 list, within 750 deep checks | 1 list |
| 2,000 | 14 lists of up to 150 emails; deep coverage is partial (750 of 2,000) | 1 list; uses 2,000 of 25,000 monthly deep checks |
| 10,000 | Does not fit: 67 lists would be needed (10,000 ÷ 150), but Free allows 25 lists, or 3,750 emails, a month | 1 list (under the 25,000 cap), or split into batches for the API, which accepts up to 10,000 emails per request |
Compare plans in the context of email verification pricing across vendors, but base your budget on the test size, not the price alone. For a 10,000-address test, Free is not an option.
Bottom line: A 2,000-address standard test fits the Free plan. A full deep-check test of 2,000 or more, or any 10,000-address test, needs Pro.
How often should you repeat the test?
Run a full test before you switch tools, after you change verification settings, and at least once a quarter if you send regularly. This is a practitioner recommendation, not an industry standard. Between full runs, a fixed labelled set works well as a regression benchmark, because the same addresses show whether the tool's behaviour has changed. For an overall rate, draw a new sample each time, because mailbox status changes over time.
Testing is the audit. To keep bad addresses from entering the list, validate at the point of capture with real-time email validation and keep an eye on the email list maintenance best practices that prevent decay between tests.
Bottom line: Test before a switch, after a settings change, and quarterly. Use a fixed labelled set for drift, and a fresh draw for the overall rate.
What mistakes ruin an email verifier test?
- Testing a vendor's sample. Curated "clean" lists flatter every tool. Always draw from your own data.
- Declaring a winner from 100 addresses. At that size the margin is ±9.8%, wider than most differences between tools.
- Labelling with the tool's output. That makes the test circular.
- Using only mailed addresses. Bounce labels exist only for addresses you have already mailed, so the test measures the easy cases unless you add a never-mailed slice.
- Counting Risky as Valid. Risky results include catch-all and role accounts. Score them separately.
- Using opens as ground truth. Mail Privacy Protection inflates opens, so use clicks or bounces.
- Counting 5.7.x codes as invalid. Those codes mean a receiving system blocked the message. The mailbox may still exist.
- Testing on different days. Mailbox status changes, so both tools need the same snapshot.
- Reporting the unweighted rate. A per-group sample is correct for diagnosis, but the headline must be weighted back to your list mix.
- Sending a campaign to see what bounces. Mailing the sample harms your sender reputation. SMTP-level checks ask whether a mailbox exists without sending a message body.
- Letting Free-plan results expire. They are kept for 24 hours, so export on the day you run the test.
For context on why these choices affect delivery, see does a valid result mean the email will be delivered and why different email verifiers give different results.
Bottom line: Most failed tests come from unrepresentative samples, circular labels, or mis-scored Risky results.
How we compared (and limitations)
This guide draws on the sample-size formula for proportions, the NIST/SEMATECH handbook, the SMTP specifications in RFC 5321 and the enhanced status codes in RFC 3463, and Unlimited Verifier's published plan limits. It is based on public documentation. We have not run controlled tests against any verifier or against any provider's mailbox behaviour. The worked-example figures are hypothetical and exist only to show the arithmetic.
Confidence intervals assume simple random sampling and independent addresses. Real lists contain clusters, such as many staff sharing one company domain, so the true uncertainty can be wider. Unlimited Verifier's 99.5% deep-verification accuracy is the company's stated figure, not an independent benchmark. For independent comparisons, see our independent email verifier accuracy benchmark and the broader guide to email verifier accuracy. Before you set pass/fail thresholds for your own test, also read how accurate email verification really is and what a good unknown rate looks like for an email verifier.
Worked sample-size calculation
Target margin ±3% at a 50% expected rate, 95% confidence (z = 1.96):
n = z² × p(1 − p) ÷ E² n = 3.8416 × 0.25 ÷ 0.0009 = 1,067
Tighten to ±2%: n = 3.8416 × 0.25 ÷ 0.0004 = 2,401
Finite population correction for a 3,000-row list: n = 1,067 ÷ (1 + 1,066 ÷ 3,000) ≈ 787
The 2,000-address default sits between these, giving about ±2.2%.
Frequently asked questions
Is 100 emails enough to test an email verifier?
100 emails is enough only for a smoke test, such as checking that your file format, columns and exports work. At a 50% rate, 100 addresses carries a ±9.8% margin, which is too wide to tell two verifiers apart. For a head-to-head comparison, use about 2,000 addresses instead.
Does the size of my email list change how many to test?
Usually not. Test sample size depends on the precision you need and the smallest group you want to judge, not on total list size. Only very small lists need a finite population correction. For example, a 3,000-row list reduces the 1,067 target to about 787.
What sample size gives a ±2% margin of error?
At 95% confidence and a 50% expected rate, a margin of ±2% requires about 2,401 addresses. A sample of 2,000 gives about ±2.2%, which is the practical default for comparing two email verifiers on overall accuracy.
How do I label addresses so I know the correct answer?
Assign each address a known outcome before running any verifier. Use bounce logs for invalid addresses, recent clicks or opens for valid ones, and seeded test mailboxes where you control delivery. Freeze these labels first so the verifier results cannot influence them.
What is a false positive in email verification?
A false positive is an address the verifier marks as Valid when it is actually bad, such as one that bounces. False positives hurt sender reputation most, so track them separately from other errors. Report them in a confusion table alongside false negatives.
What is a false negative in email verification?
A false negative is an address the verifier marks as Invalid when it is actually deliverable. These cost you reachable contacts rather than reputation. Score them separately from false positives, because a verifier can look accurate overall while silently rejecting good addresses.
How should I handle Risky results when testing?
Treat Risky as its own bucket rather than forcing it into Valid or Invalid. Risky addresses often include catch-all domains, where the server accepts any address and cannot confirm a specific mailbox. Report the share of Risky results and how those addresses later performed.
Should I test two email verifiers on different days?
No. Mailbox states change constantly, so run both tools on the same day, ideally within a few hours, using the same addresses. Results from different days are not comparable because a mailbox may have been created, closed or filled in between runs.
How do I split a test list into mailbox groups?
Divide the sample by provider type, such as Gmail, Microsoft consumer, other consumer domains, and corporate or custom domains, aiming for about 500 addresses per group. Per-group margins are wider than overall margins, so use grouping to compare tools within provider types rather than to claim precision for each one.
What confidence level should an email verifier test use?
Use 95% confidence, which corresponds to a z-value of 1.96 in the standard sample-size formula for proportions. It is the most common convention in published statistics and gives a sensible balance between precision and sample cost for a verifier comparison.
Cut Bounces and Protect Your Sender Reputation With Unlimited Email Verification
Every invalid address you send to hurts your sender reputation and your inbox placement. Verifying your list before each send means fewer bounces, healthier deliverability, and a predictable cost. With Unlimited Verifier, standard verification is unlimited on every plan, with no per-email fees.
Unlimited Verifier checks syntax, domain, MX records and the SMTP mailbox. It also detects catch-all domains and flags disposable and role-based addresses. Export a CSV from your email platform or CRM, upload and verify the list, then re-import the clean results. You can also automate checks with the email verification API.
| Free | Pro | |
|---|---|---|
| Price | $0 | $29/month ($25/month billed annually) |
| Lists | 25 a month, up to 150 emails each | Unlimited, up to 25,000 emails each |
| Deep verifications | 750 a month | 25,000 a month |
| API access | No | Yes (100,000 emails a month) |
| Result retention | 24 hours | 30 days |
Why teams choose Unlimited Verifier
- Unlimited standard verification on all plans, including Free, with no per-email fees.
- Catch-all detection so you know which domains accept every address.
- Clear Valid / Risky / Invalid results, exportable as CSV, Excel or JSON.
- An open-source verification engine you can inspect on GitHub.
Unlimited Verifier states 99.5% accuracy for deep verification.
Create a free account and verify your next list before you hit send.
Want to test a single address first? Try the free email checker.