Bot detection misses cloud link-scanning security gateways (SafeLinks / Mimecast / Proofpoint / Barracuda)

Summary

I run Mautic 7.1.1 in production and noticed that opens and clicks were being heavily inflated for contacts whose employers sit behind a cloud email-security gateway. After digging in, the culprit is that BotRatioHelper can't see these scanners. It's wired in correctly on both the open path (EmailModel::hitEmail) and the click path (PageModel::hitPage), but none of its three signals fire for a modern link scanner, so every link it pre-fetches gets logged as genuine engagement. That then poisons segment membership, lead scoring and campaign reporting downstream.

I have a fix running locally and I'm happy to open a PR if the approach sounds reasonable.

How isHitByBot() decides today

BotRatioHelper::isHitByBot() scores three signals and flags a hit when points / 3 >= botRatioThreshold (default 0.6, so 2 of 3 must be true):

1. isUnderTimeThreshold — the hit landed < timeFromEmailThreshold seconds after the email's send time (default 2s);
2. isIpInIgnoreList — the source IP is in blockedIPAddresses;
3. isUserAgentInIgnoreList — the UA contains a string from blockedUserAgents.

Why scanners slip past all three

Gateways like Microsoft Defender SafeLinks, Mimecast, Proofpoint URL Defense and Barracuda fetch every link in a message at delivery time, from a rotating pool of datacenter IPs, with spoofed mainstream-browser user agents. A typical scanned send on our install looked like a single message producing ~23 link hits from 10 distinct IPs inside a ~15-second span, all from datacenter ranges, all with normal-looking browser UAs.

Walking the three signals against that:

1. Time signal misses. Scanners fire on delivery, not send, and delivery is routinely tens of seconds after send once the MTA relays. That's well outside the 2s window, and widening the window to cover delivery would start flagging genuinely fast human reads.
2. IP signal misses. The scanner IP pools are large, rotate, and aren't in any shipped or configured blocklist. Hardcoding them is unmaintainable and goes stale fast.
3. UA signal misses. Scanners present real browser UA strings, so neither blockedUserAgents nor a UA-based bot library catches them.

Net result: 0 of 3 signals fire and every scanned link is recorded as a real open/click.

Proposed fix: a list-free burst/velocity signal

I'd suggest a fourth signal that needs no maintained data, so it can't rot across releases: a per-message burst check. No human session generates hits on one message from several distinct IPs within seconds, but a distributed scanner farm always does.

If the same email_id + lead_id has already been hit from burstMaxDistinctIps or more distinct IP addresses within burstWindowSeconds, treat the hit as a bot.

This only leans on columns that already exist and are indexed — page_hits.email_id, page_hits.lead_id, page_hits.date_hit — and it's a single COUNT(DISTINCT ...) per hit.

I'd make it strictly opt-in so default behaviour is unchanged, with two new params that mirror the existing bot_helper_* ones:

• MAUTIC_BOT_HELPER_BURST_MAX_DISTINCT_IPS — default 0 (disabled). I run it at 3.
• MAUTIC_BOT_HELPER_BURST_WINDOW_SECONDS — default 30.

The core change is small. In isHitByBot(), short-circuit before the existing three-signal scoring:

public function isHitByBot(Stat $emailStat, \DateTimeInterface $emailHitDateTime, IpAddress $ipAddress, string $userAgent): bool
{
    if ($this->isBurstScanner($emailStat, $emailHitDateTime)) {
        return true;
    }

    $totalPoints = (int) $this->isUnderTimeThreshold($emailStat, $emailHitDateTime) +
        (int) $this->isIpInIgnoreList($ipAddress) +
        (int) $this->isUserAgentInIgnoreList($userAgent);

    return $totalPoints / 3 >= $this->botRatioThreshold;
}

private function isBurstScanner(Stat $emailStat, \DateTimeInterface $emailHitDateTime): bool
{
    if ($this->burstMaxDistinctIps <= 0) {
        return false; // disabled by default
    }

    $email = $emailStat->getEmail();
    $lead  = $emailStat->getLead();

    if (null === $email || null === $lead || null === $email->getId() || null === $lead->getId()) {
        return false;
    }

    // date_hit is persisted in UTC (UTCDateTimeType); new \DateTime('@<ts>') is
    // already UTC, so format it directly — do NOT shift it to the app timezone,
    // or the comparison silently matches nothing.
    $since = (new \DateTime('@' . ($emailHitDateTime->getTimestamp() - $this->burstWindowSeconds)))
        ->format('Y-m-d H:i:s');

    $prefix = (string) MAUTIC_TABLE_PREFIX;

    $distinctIps = (int) $this->connection->fetchOne(
        "SELECT COUNT(DISTINCT ph.ip_id)
         FROM {$prefix}page_hits ph
         WHERE ph.email_id = :emailId
           AND ph.lead_id = :leadId
           AND ph.date_hit >= :since",
        [
            'emailId' => $email->getId(),
            'leadId'  => $lead->getId(),
            'since'   => $since,
        ]
    );

    return $distinctIps >= $this->burstMaxDistinctIps;
}

The two new params get autowired alongside the existing ones, and the helper picks up a Doctrine\DBAL\Connection dependency. A matching ConfigType field and a BotRatioHelperTest case covering the burst path should go with it.

For what it's worth, on our install I've been running this as a decorator we wrote locally (so the three core signals still apply and the velocity check is purely additive via logical OR), and it's been reliably catching the scanner bursts that previously sailed through. Folding it into core as an opt-in signal feels like the cleaner home for it.

Known limitation

The check runs before the current hit is persisted, so it only flags once burstMaxDistinctIps distinct IPs are already on record — the first couple of hits in a burst still slip through. In practice that still discards the large majority of a multi-hit scan while keeping the implementation to one indexed COUNT(DISTINCT ...). A fully airtight version would need post-hoc reconciliation, which felt like more machinery than this is worth.

Environment

• Mautic 7.1.1
• Also relevant to earlier 5.x/6.x lines that ship BotRatioHelper

Happy to send the PR (patch, ConfigType field, test) if maintainers are open to it.