Inline AI Crawler

What the crawler does

Inline’s crawler ingests a publisher’s own content via direct integration, sitemap, RSS feed, or submitted URLs. Once a publisher has allowlisted Inline AI’s crawler user agent, it only fetches pages that are explicitly permitted. That indexed content is used solely to ground the assistant’s answers to readers on the publisher’s own site, not to train models or build a cross-publisher corpus. More information can be found on Inline AI’s terms of service and privacy policy.

User agent

Every request the crawler makes carries this user agent string:

The product token to match in robots.txt is Inline-AI-Crawler. To allow the crawler, add:

A user agent alone is not proof of identity. Anyone can copy this string. Use the IP list or the Web Bot Auth signature below to confirm that a request really came from Inline.

IP addresses

The crawler makes all of its requests from these addresses:

• 98.82.101.115 production
• 3.232.159.3 production
• 3.92.246.252 production
• 34.202.184.50 staging
• 98.91.52.16 staging

Both environments are listed. Our staging crawler sends the same user agent, so treat every address above as ours.

A machine-readable copy of this list is published at https://www.getinline.io/crawler/ips.json. We keep that file current, so treat it as the source of truth rather than copying the addresses by hand.

Web Bot Auth signature

The crawler signs its requests with Web Bot Auth, the HTTP Message Signatures scheme defined in RFC 9421. Signed requests carry three extra headers:

• Signature-Agent: the URL of our key directory, as a quoted string.
• Signature-Input: the covered components ("@authority" "signature-agent"), our key identifier, the algorithm ed25519, the created and expires timestamps, and the tag web-bot-auth.
• Signature: the Ed25519 signature itself.

Our Ed25519 public keys are published as a JSON Web Key Set at:

The Signature-Agent header always names https://www.getinline.io. The directory response is itself signed, so a mirror of it cannot pass as ours. Verify a request by fetching the directory, selecting the key whose thumbprint matches the keyid in Signature-Input, and checking the signature over the request authority.

How the crawler treats robots.txt

The crawler ingests content from two kinds of source, and it treats them differently. We state both here so that our disclosure matches our traffic.

Publisher-supplied sources

A sitemap, an RSS feed, a URL list, or a page submitted from the publisher dashboard is a direct instruction from the site owner to ingest that page. The crawler fetches those URLs. A publisher who wants a page excluded removes it from the source they gave us, or asks us to remove it.

Automatically discovered sources

When the crawler discovers a URL on its own rather than receiving it from the publisher, it reads that host’s robots.txt first and skips any URL that is disallowed for Inline-AI-Crawler. It caches robots.txt for at most six hours, so a new Disallow rule takes effect quickly.

Request behaviour

• The crawler fetches only pages on domains that the publisher has registered with us.
• It fetches over HTTP and HTTPS only, follows at most five redirects, and stops reading a response at 32 MiB.
• It fetches on a schedule set by the publisher, and it does not crawl a site continuously.
• It does not attempt to bypass a paywall, a login, or a bot challenge.

Blocking the crawler

Add a Disallow rule for Inline-AI-Crawler in robots.txt, or email us. If you are an Inline publisher, blocking the crawler stops the assistant on your site from answering with your content.

Contact

For any question about this crawler, including a request to slow it down or stop it, email support@getinline.io. We answer crawler reports first.