Article

How to Run a Side-by-Side Email Verifier Test: A Practical Guide

By Unlimited Verifier Team ·

Diagram of a side-by-side email verifier test showing one labeled sample of addresses sent to two verifiers and scored against ground truth for false positives

Summary

A side-by-side email verifier test sends one identical, label-free list to two or more tools and grades each result against addresses whose true status you already know. Use about 2,400 addresses for roughly ±2 points at 95% confidence, and score false positives and false negatives separately.

To run a side-by-side email verifier test, pull a labeled sample of about 2,400 addresses from your own list, submit the identical blind file to each verifier within the same 24 hours, and score every Valid, Risky and Invalid result against outcomes you already know. At that size, the margin of error is roughly ±2 percentage points at 95% confidence.

Vendor accuracy pages measure accuracy on the vendor's data, using the vendor's own definition of "correct." Your test measures accuracy on your addresses, which is the only number that predicts your bounce rate. For the wider picture, start with our email verifier accuracy pillar. This guide covers the full method: building ground truth, sizing the sample, choosing metrics, a spreadsheet walkthrough and a worked example you can adapt.

Quick answer

What is a side-by-side email verifier test?

A side-by-side test submits one identical list to two or more verifiers, then grades each tool's output against addresses whose real status you already know. It is the same holdout method data teams use to evaluate any classifier: label examples, predict, compare.

Most tools sort addresses into three buckets. Valid means safe to mail. Risky means uncertain, often because of catch-all, disposable or role-based addresses. Invalid means suppress. Vocabularies differ between products, so map each tool's labels onto these three buckets before you run anything. A label such as "unknown" or "accept-all" can mean different things in different tools.

Two error types matter most:

Bottom line: A side-by-side test grades verifiers on your data, which is the only dataset that predicts your bounces.

Why not just compare vendors' accuracy claims?

Vendor accuracy figures are measurements, but they come from each vendor's test data and definitions, so they are rarely comparable. Unlimited Verifier states 99.5% accuracy for its deep verification. That is the company's own figure, not an independent benchmark. Read do email verifiers really have 99% accuracy and are vendor accuracy claims for email verifiers trustworthy before you weigh any headline number.

Two tools can disagree sharply on the same list for reasons that have nothing to do with quality. SMTP probe timing, catch-all handling and role-account rules all differ. The mechanics are explained in why different email verifiers give different results. Only a side-by-side test shows which differences matter for your list.

Bottom line: A headline accuracy figure is a claim until you have measured it on your own addresses.

How do you build a ground-truth sample?

You build ground truth from outcomes you already hold, and you keep each source separate so you can see where it is weak. Three sources are usable.

Hard bounces. Your ESP's bounce reports include SMTP enhanced status codes. Under RFC 3463, class 5 codes are permanent failures, and 5.1.1 means the destination mailbox does not exist. Use these as Invalid labels. Exclude 5.7.x codes, which signal policy or reputation blocks: a mailbox can exist and still reject your mail. Leave out 4.x.x codes, which are temporary.

Recent clicks. An address that clicked a link in the last 60 to 90 days is very likely a live, reachable mailbox. Prefer clicks to opens, because many mail clients prefetch images. Apple Mail Privacy Protection is the best-known example, and it inflates open rates.

Catch-all domains. Some domains accept every address, so bounces never reveal which mailboxes are real. Label these "unknown truth" and score them separately. How accurate are email verifiers on catch-all domains explains why this group needs its own analysis.

Both label sources are biased in a predictable way. Clickers are your most engaged contacts, and past bounces are addresses that already failed on your sends. Accuracy measured on these groups can overstate how well a tool performs on your full, messier list. Treat the result as a comparison between tools, not an absolute accuracy figure, and report results by label source.

Ground-truth source Label it as Strength Blind spot
5.1.1 hard bounces Invalid The receiving server said no Some servers misreport, and old bounces may refer to a since-fixed mailbox
Clicks in last 60–90 days Probable valid The mailbox was live and used Engaged-only bias; silent subscribers drop out
5.7.x policy blocks Do not use No information about existence Reflects your sending reputation, not the address
Catch-all domains Unknown truth Honest about uncertainty Cannot be scored for accuracy

Never test accuracy by mailing unverified addresses to see what bounces. Every invalid address you mail creates a hard bounce and pushes your rate up. Work from data you already hold.

Bottom line: Use bounces and clicks as labels, keep catch-all addresses out of the accuracy score, and never test by sending.

How many addresses does the test need?

A sample of about 2,400 addresses gives a margin of roughly ±2 percentage points at 95% confidence. The formula is n ≈ 1.96² × 0.25 ÷ e², where e is the margin of error. At e = 0.02, n is about 2,401. At e = 0.03, n is about 1,068.

The catch is subgroups. If you need accuracy for a segment, such as corporate domains, that segment needs its own sample. A 300-address corporate slice carries roughly ±5.7 points of margin, which is too wide to separate two tools that differ by 3 points. Either enlarge that slice or report it as directional only.

Stratify by source: large consumer providers such as Gmail and Outlook.com, corporate domains, catch-all domains and smaller hosts. Mailbox providers differ in how much SMTP probing reveals, so read email verifier accuracy on Gmail vs Outlook addresses before you set your strata.

Bottom line: Plan for about 2,400 labeled addresses overall, and size every segment you intend to report.

What metrics should you score?

Score five metrics, each tied to a specific cost. Accuracy alone hides the split between error types, which is why two tools with the same headline number can behave very differently.

Metric Formula What it tells you Cost if bad
Accuracy Correct calls ÷ labeled addresses Overall agreement with truth Hides the error mix
False-positive rate Truly invalid marked Valid ÷ all truly invalid Dead addresses that get through Bounces and reputation damage
False-negative rate Truly valid marked Invalid ÷ all truly valid Reachable contacts you discard Lost reach
Risky share Risky calls ÷ all addresses How much you must hold back Slower ramp and manual review
Consistency Disagreement between two runs of one tool Whether answers are stable Decisions you cannot repeat

Consistency is the metric teams skip. Run each tool twice on the same file, a day apart, in the same verification mode both times. If a tool changes its answers for a meaningful share of addresses, its single-run score overstates what you will get, because SMTP responses can shift with server load and greylisting. For what a reasonable Risky share looks like, see what is a good unknown rate for an email verifier.

Bottom line: Score false positives and false negatives separately, and check consistency with a second run.

How do you run the test step by step?

Follow these eight steps in order. Skipping the blinding and mapping steps is the most common reason tests produce unusable results.

  1. Write the decision question and the cost of each error. For example: "We are choosing a verifier for a cold list of 60,000 addresses." Your answer determines which error you weight most.
  2. Pull and label the sample. Stratify by source and apply the labeling rules above. Store the labeled key file where no vendor can see it.
  3. Clean both files identically. Trim whitespace, lowercase domains, remove exact duplicates and drop malformed rows. Different preprocessing creates phantom disagreements.
  4. Create a blind input file. Keep only the email column. Strip labels, bounce codes and engagement fields, so the input cannot be shaped by what you already know.
  5. Pre-define the bucket mapping. Write down how each tool's categories map to Valid (send), Risky (hold) and Invalid (suppress) before you see any output.
  6. Run all tools within one 24-hour window. The window keeps mailbox states and server responses from drifting between runs. It is a practical choice, not an industry standard. Export results immediately, because retention differs by plan.
  7. Score against the key file. Use the formulas in the tutorial below.
  8. Review disagreements and re-run. Hand-check a sample of rows where tools disagree, then re-run the top candidates after a day to test consistency.

Bottom line: Define the cost of each error, blind the input, pre-define the mapping, then score. The order matters.

How do you run and score the test in a spreadsheet?

This tutorial uses Unlimited Verifier as one of the tools and a spreadsheet for scoring. Repeat the upload and export steps for each competitor. Unlimited Verifier suits this kind of test for one practical reason: standard verification is unlimited on every plan, including Free, with no per-email fees, so repeat runs don't add per-address cost.

  1. Create an account. Sign up for free. The Free plan includes 25 lists a month of up to 150 emails each, plus 750 deep verifications a month.
  2. Prepare the blind file as a CSV. Keep only the email column. Excel files also work, up to 10MB.
  3. Split the file to fit your plan. On Free, each list holds up to 150 emails, so a 2,400-row test becomes 16 lists. On Pro, a single list can hold up to 25,000 emails.
  4. Upload and verify. Use upload and verify a list. The email column is detected automatically. When the run finishes, Valid, Risky and Invalid counts appear.
  5. Export and name the file. Download the results as CSV, Excel or JSON. Name each file with the tool and date, such as toolA_2026-03-02.csv.
  6. Repeat for each competitor in the same window. Use the identical input file for every tool, and record the verification mode you used.
  7. Build the scoring sheet. Put the key file in column A (email) and column B (Truth: Valid, Invalid or Unknown). Match each tool's results to the same rows in columns C and D with XLOOKUP or VLOOKUP, using the email address as the key. Exports can reorder rows, so never match by row position.
  8. Count errors with COUNTIFS. For Tool A's false positives, use =COUNTIFS(B:B,"Invalid",C:C,"Valid"). For false negatives, use =COUNTIFS(B:B,"Valid",C:C,"Invalid"). Divide false positives by COUNTIF(B:B,"Invalid") and false negatives by COUNTIF(B:B,"Valid").
  9. Report Unknown rows separately. Filter column B for Unknown and report them on their own line. Dropping them silently inflates accuracy.
  10. Re-run for consistency. The email verification API can script the second run on Pro, with bulk batches of up to 10,000 emails per request. Keep the same mode and input file, then count how many rows changed answers. For scripted workflows, see email verification API and automation.
  11. Spot-check disputed rows. Use the free email checker on a handful of disagreements. Treat these checks as evidence, not ground truth.

Bottom line: A few matching formulas turn raw exports into a scorecard you can defend.

What does a worked example look like with real numbers?

The figures below are illustrative, not measured results. They show the arithmetic. Tool A and Tool B are placeholders for any two verifiers.

Setup: The labeled key file has 2,400 addresses: 1,200 probable-valid (clicked in the last 90 days) and 1,200 known-invalid (5.1.1 bounces, no 5.7.x codes). Both tools receive the same blind file.

Plan fit: In standard mode, the test fits Free. 2,400 rows make 16 lists of 150, within the 25-list monthly cap, and standard verification is unlimited. In deep mode, 2,400 checks exceed Free's 750 deep verifications, so you need Pro. On Pro, the file runs as one list under the 25,000-email cap and uses 2,400 of 25,000 deep verifications, or 9.6%. At full use, Pro works out to $1.16 per 1,000 deep verifications on the monthly plan ($29 ÷ 25) or $1.00 billed annually ($25 ÷ 25). When you compare email verification pricing, use total cost per 1,000 checks across vendors, not headline monthly prices.

Illustrative results:

Truth group Tool A Valid Tool A Risky Tool A Invalid Tool B Valid Tool B Risky Tool B Invalid
Probable-valid (1,200) 1,130 30 40 1,090 50 60
Known-invalid (1,200) 35 65 1,100 12 48 1,140

Scoring:

Scaling to your list: Error rates only matter once they meet your volumes. Suppose a 50,000-address cold list contains 10,000 truly invalid addresses. Tool A passes 10,000 × 35/1,200 ≈ 292 of them as Valid. Tool B passes 10,000 × 12/1,200 = 100. The gap of about 192 dead addresses lands on your next send.

Now take a 80,000-address warm list with 70,000 truly valid addresses. Tool A discards 70,000 × 40/1,200 ≈ 2,333 reachable contacts. Tool B discards 70,000 × 60/1,200 = 3,500, about 1,167 more.

Identical accuracy hides opposite error profiles. For the cold list, Tool B is the safer choice. For the warm list, where reach matters more, Tool A may be better. The decision rests on the cost of each error, not the headline score.

Bottom line: Identical accuracy can hide opposite error profiles. Scale each error rate to your own list before you choose.

Which error type should decide your choice?

The error that costs you more on your list deserves the most weight. Set that priority before you see results.

Your situation Costlier error What to prioritise Typical trade-off
Cold or purchased-style outreach list False positive Lowest false-positive rate Accept more Risky holds
Dormant list you are reactivating Both, weighted by list size Balanced; sample each segment Expect a larger Risky share
Warm, opted-in newsletter list False negative Lowest false-negative rate Tolerate a few more bounces
Signup forms or checkout False negative (blocks real customers) Fast, consistent single checks Use real-time email validation at entry
SaaS or ecommerce customer database Both Consistency across re-runs Re-test on a schedule

For SaaS and ecommerce teams, the email verification for ecommerce and SaaS approach often matters more than a one-off test, because your database changes weekly. A Valid result is also not a delivery guarantee. Read does a valid result mean the email will be delivered before you treat the Valid bucket as a promise.

Bottom line: Choose the tool whose dominant error costs least on your list, and record that reasoning in writing.

What should you check about each tool before the test?

Record each tool's published features before you run it, because they determine how you map and score results. If a vendor does not publish a value, write "not published" rather than guessing. The table uses Unlimited Verifier's published facts as a template you can copy for each competitor.

What to check Why it matters for the test Unlimited Verifier (published facts)
Result buckets Maps directly onto send, hold and suppress Valid, Risky and Invalid
Catch-all detection Lets you score catch-all domains as unknown truth Detects catch-all domains; see catch-all email verification
Disposable and role flags Lets you score these categories separately Flags disposable and role-based addresses
Input size Determines how you split the sample CSV or Excel up to 10MB; up to 150 emails per list on Free, 25,000 on Pro
Export formats Affects how quickly you can score CSV, Excel or JSON
API for re-runs Enables scheduled consistency checks API on Pro: 100,000 emails a month, bulk batches of up to 10,000 per request
Result retention Sets how long you can revisit a run Kept 24 hours on Free, 30 days on Pro
Transparency Lets you inspect the checks performed Open-source engine on GitHub

For ongoing email verification compliance and hygiene, keep the labeled key file so future tests reuse the same baseline.

Bottom line: Record each tool's published limits before testing, so mapping and scoring stay consistent.

What mistakes skew side-by-side results?

Most failed tests fail on preparation, not on the tools themselves. These are the errors that recur most often.

Bottom line: Most failed tests come down to preparation, not tool quality.

How we compared (and limitations)

This guide draws on public documentation: RFC 3463 for enhanced status codes, RFC 5321 for the SMTP RCPT TO step that mailbox checks rely on, the FTC's CAN-SPAM guidance, and Unlimited Verifier's published plan facts. Sample-size figures come from standard statistics, not from any vendor.

We have not run controlled tests against other vendors, and this article does not rank competing tools. The worked-example figures are illustrative. Unlimited Verifier's 99.5% deep-verification accuracy is the company's stated figure, not an independent result. For an outside view, compare against the independent email verifier accuracy benchmark for 2026, and read how accurate email verification really is before relying on any single number. The industry body M3AAWG publishes messaging best practices that are useful background for a list-hygiene routine.

To shortlist candidates, review the best email verification tools and use how to choose an email verifier to set your criteria. Then run this test on your own data, and schedule repeat tests as part of email list maintenance best practices.

Bottom line: Treat every headline number, including ours, as a claim to verify on your own list.

Side-by-side test steps

  1. Pull about 2,400 addresses from your own list, stratified by source and type.
  2. Label ground truth: Invalid = 5.1.1 hard bounces; Valid = recent clicks or replies; drop 5.7.x blocks.
  3. Strip labels and outcome columns; deduplicate; save one identical file.
  4. Submit the file to each verifier within 24 hours.
  5. Map every output to Valid, Risky or Invalid.
  6. Count false positives (invalid marked Valid) and false negatives (valid marked Invalid) per tool.
  7. Compare error rates, then choose by the cost of each error for your sending program.

Frequently asked questions

How many email addresses do I need for a verifier test?

About 2,400 addresses gives a margin of error of roughly ±2 percentage points at 95% confidence. If budget is tight, about 1,070 addresses gets you roughly ±3 points. Larger samples narrow the margin further but with diminishing returns. Make sure the sample reflects your real list mix, including role, catch-all and disposable addresses.

What is ground truth in email verification testing?

Ground truth is the known real status of each address in your test sample. Build it from outcomes you already have, such as hard bounces (SMTP 5.1.1 codes) for invalid addresses and recent clicks or replies for valid ones. Exclude 5.7.x policy blocks, because they do not prove a mailbox exists or not.

Why should I submit the same file to every verifier within 24 hours?

Mailbox status can change over time, as addresses are created, abandoned or filtered. Submitting the identical file to all tools within a short window keeps the comparison fair. Deduplicate the list first and remove labels or outcome columns so no tool sees the answers.

What is the difference between false positives and false negatives?

A false positive is an invalid address the tool marks Valid, which leads to bounces. A false negative is a valid address the tool marks Invalid, which causes you to lose reachable contacts. Score them separately, because two tools can have similar overall accuracy while one lets far more dead addresses through.

How do I choose a verifier based on test results?

Choose by the cost of each error type for your program. Cold outreach lists punish false positives because bounces damage sender reputation, so favor the tool with fewer false positives. Warm lists with high engagement punish false negatives more, so favor the tool that keeps more valid addresses. Weigh both against price and processing speed.

How should I handle Risky or catch-all results in the test?

Map each tool's uncertain output to a single Risky bucket before scoring, since labels such as unknown or accept-all vary by vendor. Report Risky results separately rather than counting them as correct or incorrect. Then decide how your sending policy treats that bucket, such as sending only to engaged contacts.

Why do two email verifiers give different results on the same list?

Differences come from SMTP probe timing, catch-all handling, role-account rules, retry logic and how each vendor defines Risky. These design choices can shift results even when both tools are well built. A side-by-side test on your own data shows which differences actually affect your bounce rate.

Can I trust vendor accuracy claims instead of running a test?

Vendor accuracy figures are useful context, but they come from each vendor's own data and definition of correct, so they are rarely directly comparable. A test on your addresses measures the number that predicts your bounces. Use vendor claims to shortlist tools, then confirm with your own sample.

Should I remove duplicates before running the test?

Yes. Deduplicate the list before submitting it, and normalize case and whitespace so the same address is not counted twice. Duplicates inflate some scores and make the sample look larger than its independent size, which distorts your margin of error.

Cut Bounces and Protect Your Sender Reputation With Unlimited Email Verification

Every invalid address you send to hurts your sender reputation and your inbox placement. Verifying your list before each send means fewer bounces, healthier deliverability, and a predictable cost. With Unlimited Verifier, standard verification is unlimited on every plan, with no per-email fees.

Unlimited Verifier checks syntax, domain, MX records and the SMTP mailbox. It also detects catch-all domains and flags disposable and role-based addresses. Export a CSV from your email platform or CRM, upload and verify the list, then re-import the clean results. You can also automate checks with the email verification API.

Free Pro
Price $0 $29/month ($25/month billed annually)
Lists 25 a month, up to 150 emails each Unlimited, up to 25,000 emails each
Deep verifications 750 a month 25,000 a month
API access No Yes (100,000 emails a month)
Result retention 24 hours 30 days

Why teams choose Unlimited Verifier

Unlimited Verifier states 99.5% accuracy for deep verification.

Create a free account and verify your next list before you hit send.

Want to test a single address first? Try the free email checker.