DesearchBot
DesearchBot is the web crawler operated by Desearch. It fetches public web pages so that people and AI agents searching through Desearch can find them, and cites the source back in every answer. This page documents how to identify it, how to verify that a request really came from us, and how to control what it crawls.
Identity
| robots.txt token | DesearchBot |
|---|---|
| User-Agent | Mozilla/5.0 (compatible; DesearchBot/1.0; +https://www.desearch.ai/crawler) |
| Signature-Agent | https://www.desearch.ai/crawler |
| Key directory | https://www.desearch.ai/.well-known/http-message-signatures-directory |
| Contact | support@desearch.ai |
There is no IP list to allowlist. Identify us by the signature on the request, described below.
Verifying a request
Any script can put DesearchBot in its User-Agent header. That string proves nothing on its own, and you should not allowlist on it.
Every request we send is signed with an Ed25519 key using HTTP Message Signatures (RFC 9421). The signature covers the method, host and path, so it cannot be replayed against a different URL.
Signature-Agent: "https://www.desearch.ai/crawler"
Signature-Input: sig1=("@authority" "@method" "@path");
created=1760000000; keyid="..."; alg="ed25519"
Signature: sig1=:MEUCIQ...:To check a request:
- Read the
Signature-Agentheader and fetch the key directory it names. - Match the
keyidfromSignature-Inputagainst a key in that directory. - Verify the signature over the covered components.
A request that fails this check is not from us, whatever its User-Agent claims. Our keys are published at https://www.desearch.ai/.well-known/http-message-signatures-directory and rotate on a published schedule.
Crawl behaviour
| robots.txt | Fetched before we crawl a host and cached for up to 24 hours. We obey Disallow, Allow and Crawl-delay for our token, falling back to the * group. |
|---|---|
| Request rate | At most 1 request per second per host, and slower if you set Crawl-delay. We halve the rate after any 429 or 5xx and keep it there. |
| Rendering | A plain HTTP GET. No JavaScript execution and no headless browser. |
| Revisits | A page is re-fetched when its sitemap lastmod changes, and once about a day after publication to pick up edits. |
| Attribution | Every indexed page keeps its link, headline and source in Desearch results. |
What it never does
- Bypass paywalls, logins, or metered access.
- Solve or evade CAPTCHAs and bot challenges.
- Submit forms or perform any other write action.
- Ignore robots.txt, or crawl a path it disallows.
- Rotate residential IPs to disguise where it comes from.
Controlling what it crawls
Add any of the following to /robots.txt at the root of your host. You do not need to contact us, and you do not need an entry to allow us: with no rule for our token we follow your * group.
Block us completely
User-agent: DesearchBot
Disallow: /Block part of the site
User-agent: DesearchBot
Disallow: /members/
Disallow: /newsletters/Slow us down
Seconds to wait between our requests to your host.
User-agent: DesearchBot
Crawl-delay: 10Removing a page that is already indexed
robots.txt controls crawling, not what is already stored. To drop a page, serve X-Robots-Tag: noindex or add a noindex meta tag, and we remove it at the next fetch. For an urgent takedown, email us and we act manually.
Contact
Wrong crawl rate, unexpected traffic, a page that should not be indexed, or a crawler claiming to be us that fails signature verification: support@desearch.ai. We reply within two business days.