Choosing a filter

How to evaluate a spam filter during a trial or pilot

By iBlockSpam Team How we check our guides Updated Oct 10, 2026 10 min read 7 sources

Short answer

Record a baseline first: spam reaching inboxes, real mail wrongly held and quarantine volume. Then run the filter on real mail for at least two weeks, test authentication, outage queuing, quarantine release, digests, allow and block lists, headers and message history, time the vendor's support, write down your exit steps and score each product the same way.

On this page 9 sections

A feature list tells you what a spam filter can do. A trial tells you what it does to your mail, your users and your week. This checklist turns a pilot into a fair, written test, so you can compare products on evidence rather than impressions.

It builds on our spam filter buying guide, which covers what to look for and which questions to ask vendors. This guide is about proving the answers. Once you've chosen and switched, use how to test that a spam filter is working to sign off the live setup.

What should you decide before the trial starts?

Write these down before anyone changes a DNS record:

What problem are you solving? For example, "phishing reaches inboxes weekly" or "users can't find held invoices". You'll score against this.
Which domain will you pilot? See the next section.
How long will it run? At least two weeks of real mail, and longer if a busy period is coming up. Agree the dates with the vendor.
Who owns it? Name one person who checks the quarantine, keeps the log and makes the final call.
What does success look like? Set targets, such as "fewer spam messages in inboxes than our baseline, and no real invoice held for more than a few hours".
How will you leave? Write the exit plan now (see below), not when something goes wrong.

How do you choose a pilot domain or subset?

An MX record applies to the whole domain. You can't send only some users' mail through a gateway by MX alone. That leaves three realistic choices:

Pilot scope	How it works	Good for	Watch out for
A second domain you use	Point its MX at the new filter; leave the main domain alone	Low risk, real external mail	A quiet domain may see little spam, so results look better than they will be
A subdomain with its own mailboxes	Give a team addresses such as `team@ops.example.com` and its own MX	Testing with real people	Only works if outsiders actually write to those addresses
The main domain	Full cutover with a short TTL and rollback plan	The most honest results	Every user is in the pilot, so prepare them

On Microsoft 365, you can also limit one setting while you test. Microsoft recommends applying Enhanced Filtering for Connectors to "a small set of users" as an initial test, then the whole organization. Microsoft notes it only takes effect when every recipient of a message is on that list.

For the main domain, lower your MX TTL a day or two ahead and follow how to change MX records without losing email.

What baseline should you record first?

Without a baseline, you can't tell whether the new filter is better or just different. Measure your current setup for one or two weeks before the pilot:

Spam reaching inboxes. Ask a few users to forward or report every spam and phishing message that reaches them. Count them per week.
False positives. Count legitimate messages that landed in junk, were quarantined or bounced. Note who noticed and how long it took.
Quarantine volume. How many messages are held per day, and how many of those anyone ever looks at.
Admin time. Minutes per week spent releasing mail, answering "where's my email?" and editing lists.
Your DNS. Save your MX, SPF and DMARC records with the MX Lookup and Email Health Check.

Your mail platform can help. Microsoft 365 has message trace, and Google Workspace has Email Log Search. Our guides to Microsoft 365 message trace and Google Workspace email log search show how to use them.

Which tests should you run during the pilot?

Run each test early, then again near the end. Keep a simple log with the date, test, result and time spent.

1. Authentication

Send messages from a few outside providers, such as a personal Gmail or Outlook.com account. Open the headers in the Email Header Analyzer and check that SPF, DKIM and DMARC results look right. On Microsoft 365 with Enhanced Filtering on, Microsoft says an X-MS-Exchange-ExternalOriginalInternetSender header is added when skip listing worked, which shows Microsoft still sees the true source. If your own domain has a DMARC record (check with the DMARC Checker), ask the vendor how to safely test that a message forging your domain is held.

2. Outage spooling

Ask what happens when your mail server can't accept mail. Senders retry on their own schedule; RFC 5321 says the give-up time "generally needs to be at least 4-5 days", but each sender chooses. A gateway that queues mail removes that guesswork. If you run your own server, agree a short, planned test with the vendor: block delivery for an hour, send test mail, restore it and confirm the queued mail arrives. See what happens to email when your server is down.

3. Quarantine release

Find a held legitimate message and release it as a normal user, then as an admin. Time both. Check that releasing a message doesn't quietly allow the sender forever.

4. Digests

Turn on quarantine digests for pilot users. Do they arrive on schedule? Can people act on them without calling IT? Is the schedule something busy staff will read?

5. Allow and block lists

Add an allow entry for a regular supplier and a block entry for a persistent spammer. Then check whether an allow entry still applies when someone forges that supplier's address. Entries that require the sender to pass authentication are safer. Our false positives guide explains why.

6. Malware and attachments

Send the EICAR test file from an outside account. EICAR says it is "safe to pass around, because it is not a virus". Confirm it's blocked or held. Also send a large but legitimate attachment, and a file type you plan to block.

7. Header inspection

On delivered mail, confirm the filter is really in the path. Its servers should appear in the Received headers. If they don't, mail is reaching your mailbox some other way. Our guide on how to read email headers shows what to look for.

8. Message trace and history

Pick a message a user says never arrived. Can you find it in the filter's history, see the verdict and explain what happened in under five minutes? Note how many days of history the product keeps.

9. Direct-delivery bypass

Ask the vendor how to stop attackers sending straight to your mailbox provider and skipping the filter, then check that the lock-down works after cutover. On Google Workspace, the inbound gateway setting has an option to reject mail that doesn't come from your gateway's IPs. Our guide to blocking direct-to-tenant bypass covers Microsoft 365 and Google Workspace.

10. Outbound, if you need it

If the product also relays outbound mail, send through it and check that DKIM signs for your domain and that SPF still passes.

How do you measure admin effort?

Keep a running tally of minutes for each task: setup, release requests, list changes, alerts and support contacts. Divide by the number of weeks. Also note how many times a user had to ask IT for help, because each one is a small interruption that a feature list never shows.

Microsoft lists "more complexity" and a false-positive rate that rises "as more products are added" among the trade-offs of adding a non-Microsoft service (Microsoft Learn). The same page says a non-Microsoft service "should provide a user experience that doesn't require users to think about which quarantine to use". Test exactly that: during the pilot, do users know where to look?

How do you test support responsiveness?

Contact support at least twice during the pilot, through the channel you'd use in real life:

One setup question with a clear answer, such as the IP ranges to trust.
One real problem, such as a held message you can't explain.

Record the time to first reply, whether the answer was correct, and whether you had to ask twice. A vendor that's slow during a trial won't get faster after you've paid.

What should your exit plan include?

Write it before cutover and keep it next to the log:

Your saved MX records, with hosts, priorities and TTLs.
The platform changes you made, such as a connector, gateway setting or trusted IP list, and how to undo each one.
How long you keep access to the trial quarantine and history after you leave.
Who is allowed to call the rollback.
A test message to confirm mail arrives directly afterwards.

Our guide to switching spam filter providers covers overlaps, quarantine continuity and clean-up in detail.

How do you score the results?

Score each product from 1 (poor) to 5 (excellent) on the same criteria, multiply by the weight and add the totals. Adjust the weights to the problem you wrote down at the start.

Criterion	Weight	Evidence to use
Spam and phishing reaching inboxes, vs baseline	3	User reports per week
Legitimate mail held or delayed, vs baseline	3	Your false-positive count
Quarantine release and digests	2	Release times, user questions
Outage spooling	2	Test result or written answer
Allow and block lists, including spoof-safe entries	1	Test 5
Message history and trace	1	Time to explain a missing message
Admin minutes per week	2	Your tally
Support responsiveness	1	Time to first useful reply
Exit and rollback	1	How easily you could leave
Total cost for your real mailbox and domain counts	2	Yearly totals on the same billing term

The maximum score is 90. Don't let a small lead in one row hide a failure in another: a filter that holds payroll mail for a day fails, whatever its total. For pricing, see how much email security costs a small business.

Next steps

Copy the scoring table and fill in your baseline for one to two weeks before any pilot.
Choose a pilot scope and write your exit plan, then check your starting DNS with the Email Health Check.
If you'd like to include iBlockSpam in your evaluation, you can get started with one domain. There's no free trial, but a domain's first annual subscription has a 30-day money-back guarantee, and the portal demo shows the dashboard with sample data first. iBlockSpam's quarantine and digests, outage spooling (up to 5 days) and 30 days of message history map directly to tests 2 to 4 and 8, and our cutover plan includes rollback steps.

Related: free vs paid spam filters helps you decide whether a paid filter is needed at all, and per-domain vs per-user pricing shows how to compare quotes fairly.

See more buying guides.

Frequently asked questions

How many weeks of data do I need before judging a spam filter pilot?

At least two full weeks of real mail, and ideally a month that includes a busy period such as month-end invoicing. Spam volumes and senders vary from week to week, so a few days rarely shows how often real mail gets held. Agree the length with the vendor before you change MX, so the pilot doesn't stop halfway.

Can I pilot an MX-level spam filter on only some users?

Not on the same domain through MX alone, because the MX record covers every address on the domain. The usual options are a second domain or subdomain that receives real mail, or the whole domain with a short TTL and a written rollback plan. On Microsoft 365, Enhanced Filtering for Connectors can be tested on a small set of recipients first.

How do I measure false positives during a trial?

Count every legitimate message that was held, delayed or rejected, and note who noticed it and how. Check the quarantine daily for the first week, ask users to report anything they expected but didn't receive, and compare the count against your baseline from the current filter. One held invoice matters more than ten missed newsletters.

Is it safe to send test viruses through a spam filter?

Use the EICAR test file, not real malware. EICAR publishes it so people can test anti-malware software, and says it is safe because it isn't a virus and contains no viral code. Send it from an outside account, confirm it's blocked or held, and tell your IT contact first so nobody panics.

What should I do if the trial goes badly?

Use the exit plan you wrote before you started. Point MX back to your old service or mail platform, remove any connector or gateway restriction you added, confirm that test mail arrives directly, and release anything important from the trial quarantine before access ends. A short TTL set before cutover makes the rollback quick.

Sources

See how your domain measures up

Run a free Email Health Check: MX, SPF, DKIM, DMARC and transport security in one scan, with the raw records shown. No sign-up.

Keep reading