Skip to content

/ Crawler Documentation

ThatSEOAgentBot

That SEO Agent uses two distinct request identifiers depending on the operation being performed. This page documents both, explains when each is used, and provides configuration examples for allowing or blocking them.

User agent strings

That SEO Agent makes two types of requests to external URLs. Each uses a different user agent.

Site crawler
ThatSEOAgentBot/1.0 (+https://thatseoagent.com/en/seo-bot)

Used by the crawl_site tool. Performs a full BFS crawl of your site to detect broken links, deep pages, thin content, and duplicate metadata. Only runs when explicitly triggered by a signed-in user via their MCP client.

Page audit fetcher
ThatSEOAgentBot/1.0 (page-audit; +https://thatseoagent.com/en/seo-bot)

Used by page audits and all on-page checks (title, meta, schema, E-E-A-T, GEO score, crawlability, security headers). Sends a single GET or HEAD request per URL per tool call.

What the crawler checks

When crawl_site runs, ThatSEOAgentBot performs a BFS crawl starting from your homepage. For each page it visits, it collects:

  • 01HTTP status code and redirect chain
  • 02Page title and meta description (for duplicate detection across the site)
  • 03Canonical URL and noindex directive
  • 04H1 headings
  • 05Word count (HTML stripped)
  • 06All internal links (used to discover the next pages in the queue)
  • 07BFS depth (number of clicks from the homepage)

The bot does not execute JavaScript, render pages, or download assets. It parses raw HTML only.

Crawl behavior

Concurrency3 parallel requests maximum
Inter-batch delay300ms between batches to avoid overloading your server
Timeout per page12 seconds hard timeout (site crawler) / 30 seconds (page audit fetcher)
RedirectsFollowed automatically
Content typeHTML only — sends Accept: text/html, application/xhtml+xml
Robots.txtFetched and parsed before every request, crawls and page audits alike. Any URL disallowed for User-agent: * or User-agent: ThatSEOAgentBot is not fetched at all. One robots.txt read is cached per site.
In-memory cachePage audit responses are cached for 60 seconds within a single agent turn. Multiple tools checking the same URL share one HTTP request.
TriggerOn-demand only. Crawls and page audits only run when a signed-in user explicitly requests them via their MCP client.

How to allow ThatSEOAgentBot

If your server or firewall blocks unknown user agents, add an exception for both strings. Examples below.

robots.txt
# One token covers both the crawler and the page-audit fetcher
User-agent: ThatSEOAgentBot
Allow: /
Cloudflare WAF
# Firewall Rule: Skip for ThatSEOAgentBot
# Field: http.user_agent
# Operator: contains
# Value: ThatSEOAgentBot
# Action: Skip

# One rule is enough — both variants carry the same token
Nginx
# Inside your server block
if ($http_user_agent ~* "ThatSEOAgentBot") {
    # Remove any rate-limit or block rules
    set $skip_limit 1;
}
Apache
# In .htaccess or VirtualHost
<If "%{HTTP_USER_AGENT} =~ /ThatSEOAgentBot/">
    # Exempt from mod_evasive or rate limits
</If>

How to block ThatSEOAgentBot

If you do not want That SEO Agent to touch your site, add a disallow rule to your robots.txt. One rule covers everything: the same token is used by the crawler and by the page audit fetcher, and both check the file before requesting anything.

robots.txt
User-agent: ThatSEOAgentBot
Disallow: /

Page audit requests are user-initiated, but a disallow still stops them: we do not treat a signed-in user as permission to override your robots.txt.

Verifying the user agent

Legitimate requests from That SEO Agent always match one of the two user agent strings listed above exactly. Any request claiming to be ThatSEOAgentBot with a different string is not from this service.

  • —Site crawler requests include Accept: text/html, application/xhtml+xml
  • —Page audit requests use a standard GET or HEAD method
  • —All requests originate from Vercel serverless infrastructure (no fixed IP range)

Cryptographic verification (Web Bot Auth)

A user agent string is a claim, not proof: anyone can send it. Every request we make carries an Ed25519 signature you can verify instead, per draft-meunier-web-bot-auth on top of RFC 9421. Both the site crawler and the page audit fetcher sign.

Public key (JWKS)

https://thatseoagent.com/.well-known/http-message-signatures-directory
  • —Signature-Agent: "https://thatseoagent.com" — names who issued the request
  • —Signature-Input: sig1=("@authority" "@method" "@path" "signature-agent") plus created, expires, keyid, alg="ed25519" and tag="web-bot-auth"
  • —Signature: sig1=:<base64 Ed25519 signature>:
  • —keyid matches the kid published in the JWKS above
  • —Signatures expire 300 seconds after they are created, so a captured header cannot be replayed later

Requests to Google APIs, Wikipedia and Wikidata carry no signature on purpose: those already authenticate through OAuth or an API key, and a second identity would only add noise.

Questions or abuse reports

If you believe That SEO Agent is making requests in violation of these specifications, contact us at support@thatseoagent.com.

On-page SEO auditHow That SEO Agent worksMCP setup guide

/ ThatSEOAgentBot

Audit any page in under 2 minutes.

8 checks. Live data. No copy-pasting.

See pricing

Connect your site and run your first audit from any MCP-compatible AI.