The crawler that reads your site for you.
If you found Supercurve-Crawler in your logs, this page explains exactly what it is doing and how to turn it off.
Every request we make — page reads, sitemap and robots.txt reads, asset size checks, and rendered pages — sends this user agent, and nothing else:
Supercurve-Crawler/2.0 (+https://supercurve.ai)
We never disguise our crawler as a browser. If you see a request claiming to be Chrome or Safari, it is not us.
A user agent is only a claim, so we also publish a public key. When request signing is enabled, our requests carry Signature-Agent, Signature-Input and Signature headers you can verify against:
https://crawler.supercurve.ai/.well-known/http-message-signatures-directory
What it requests
robots.txt, your sitemap, and the public HTML pages it lists. It also makes small ranged requests for images and other assets referenced by those pages, to measure their size. It never attempts to log in, never submits forms, and never requests anything behind authentication.
How often
Visits are incremental. Rather than re-reading a whole site, each visit fetches only pages that are new or have changed, plus a small rotating sample to catch edits that sitemaps do not report. Requests run a few at a time, with a per-visit page cap and a wall-clock limit.
robots.txt
We read robots.txt before crawling and follow it. If you disallow Supercurve-Crawler, we stop. We do not treat a missing or unreachable robots.txt as permission to ignore your preferences.
JavaScript rendering
Most sites are read as plain HTML. When a site turns out to be drawn in the browser and the HTML comes back nearly empty, we re-read that site with a headless Chromium so the content is legible. It announces itself with the same user agent.
Who asked for it
Supercurve is a product our customers use to understand their own websites. A site is crawled because someone connected it to a Supercurve workspace. We are not a search engine, we do not build a public index, and we do not use your pages to train models.
Signed requests
Our requests can carry an Ed25519 signature over the IETF HTTP Message Signatures drafts, so you can verify a request really came from us rather than from something copying our user agent. Our public key is published at the directory URL below.
Block it
Add this to your robots.txt. We re-read robots.txt on every visit, so it takes effect on the next one — no need to contact us first.
User-agent: Supercurve-Crawler Disallow: /
To keep us out of one section only, disallow that path instead of /.
Allow it
If you are a Supercurve customer and your own site is turning us away, the block is usually at your CDN rather than in robots.txt — a bot-protection rule challenging anything it does not recognise. Allowing verified bots, or adding a rule that skips our user agent, is enough.
Tell us what you are seeing and we will help you find the rule.
Tell us and we will stop.
If our crawler is taking too much of your bandwidth, requesting something it should not, or you simply want it gone and would rather not edit robots.txt, email us. We treat crawl complaints as incidents: we will confirm receipt, stop crawling the site while we look, and tell you what we found.