OctenBot

The web crawler behind Octen search.

Purpose

Octen is a search engine. OctenBot fetches publicly accessible web pages so they can be found and cited through Octen, sending people back to the original site.

Identification

OctenBot sends this user-agent:

Mozilla/5.0 (compatible; OctenBot/1.0; +https://crawler.octen.ai/)

Its robots.txt product token is OctenBot. User-agent strings can be spoofed, so verify requests cryptographically as described below.

Verification

Every OctenBot request is signed with HTTP Message Signatures (RFC 9421) following the Web Bot Auth scheme. Each request carries Signature, Signature-Input and Signature-Agent: "https://crawler.octen.ai" headers.

On Cloudflare and other providers that support Web Bot Auth, OctenBot is verified automatically at the edge. To verify it yourself, validate the request signature against our Ed25519 public keys:

https://crawler.octen.ai/.well-known/http-message-signatures-directory

Verification does not depend on the request's IP address.

Controlling OctenBot with robots.txt

OctenBot reads robots.txt before fetching. It follows the group addressed to OctenBot if there is one, otherwise the * group, and honors Crawl-delay. If robots.txt cannot be fetched because of a server error, or cannot be parsed, OctenBot does not crawl the site.

Block OctenBot from your whole site:

User-agent: OctenBot
Disallow: /

Block specific paths:

User-agent: OctenBot
Disallow: /private/

How OctenBot behaves

• Checks robots.txt before every fetch, including every redirect target.
• Signs every request so you can confirm it is genuinely us.
• Sends at most one request at a time to a site and visits each site only a few times a day.
• Backs off when asked: it honors Retry-After on 429 and 503 responses and does not retry 403 responses.
• Fetches public content only. It does not use proxies, does not attempt to bypass logins, paywalls or CAPTCHAs, and submits no forms.

Contact

Questions or concerns about OctenBot: crawler@octen.ai. Please include the affected domain, the time of the requests, and relevant log lines.