OnPagePilot Bots

Crawling infrastructure

Like a search engine, OnPagePilot relies on crawling — and because the web is now read by AI assistants as well as search engines, we crawl the sites our customers register to keep their SEO and AI-visibility analysis current.

OnPagePilot Crawler

The OnPagePilot Crawler is our primary site-crawling agent. It visits websites that have been added on our platform by our users and collects publicly available information such as page structure, meta tags, link relationships, and technical SEO signals to power our analysis tools.

OnPagePilot Bots

Once live, general web surfing will be an automated process in which we send software - often AI-driven agents - across the web to discover pages and interpret their content. These agents will map what exists, what’s new, and what’s changed, so that when you run a deep search, the system will already know the page and can surface it.

Cited-page verification

One Oppbot activity you may see in your logs today is cited-page verification. When an AI assistant answers a question and cites a page on your site as its source, and that answer concerns a brand one of our users tracks, we fetch that one page to read what it actually says about the brand. Nothing is inferred from your domain name; the verdict comes from the page text.

To opt out, block Oppbot in your robots.txt as shown under Controlling Bot Behaviour — blocking the crawler blocks cited-page verification too.

OnPagePilot Technical Auditor

The Technical Auditor performs targeted security and performance probes on URLs registered by our users in our IndexMonitor system. It does not crawl or discover new pages. It checks specific endpoints for correct HTTPS configuration, security headers, and runs Lighthouse audits to measure page performance and accessibility scores.

OnPagePilot Technology Scanner

The Technology Scanner is triggered when a user adds a URL monitor on our platform. It performs a single-page visit using a headless browser to detect the technology stack in use (CMS, frameworks, analytics tools, etc.) and captures a visual screenshot. It does not follow links or crawl additional pages beyond the monitored URL.

Verification

You can verify that a request is genuinely from the OnPagePilot by checking the User-Agent header. All legitimate crawler requests will contain:

Mozilla/5.0 (compatible; Oppbot/2.0 +https://onpagepilot.com/bot)

The Technical Auditor and Technology Scanner make targeted single-URL requests to domains registered by our users and do not carry a custom user-agent token.

If you have concerns about traffic claiming to be from OnPagePilot, please contact us.

Benefits for Site Owners

Our bots help site owners maintain and improve their online presence by providing data for:

• Technical SEO Audits: the Crawler identifies broken links, missing meta tags, redirect chains, duplicate content, and other on-page issues across your entire site.
• Site Structure Analysis: the Crawler maps internal linking patterns and site architecture to reveal navigation bottlenecks and orphan pages.
• Indexing Insights: crawl data is compared against search engine behaviour to highlight pages that may be under-indexed or excluded.
• Security & HTTPS Validation: the Technical Auditor verifies SSL certificates, checks for missing HSTS headers, and detects faulty HTTP-to-HTTPS redirects before they affect visitors.
• Performance & Accessibility Scoring: Lighthouse audits run by the Technical Auditor measure page speed, accessibility compliance, and Core Web Vitals.
• Technology Detection: the Technology Scanner identifies your CMS, JavaScript frameworks, analytics tools, and other technologies so you can benchmark your stack.

Policies and Commitments

• Respect for robots.txt: our Crawler fully complies with the Robots Exclusion Protocol (RFC 9309), including Disallow, Allow, and Crawl-delay directives.
• Minimal server impact: the Crawler enforces a default delay of 500 ms between requests and automatically backs off when a server responds slowly or returns errors. The Technical Auditor and Technology Scanner make infrequent, single-URL requests.
• User-initiated only: all three bots operate exclusively on domains that have been added on our platform by our users.
• No personal data collection: we process only publicly accessible page content. None of our bots attempt to access password-protected areas, submit forms, or collect personal data.
• Transparent identification: the Crawler identifies itself with a unique User-Agent string that includes a link back to this page. The Technical Auditor and Technology Scanner target only user-registered URLs.
• GDPR compliant: our data processing complies with the EU General Data Protection Regulation. See our Legal & Privacy Policy for details.

Controlling Bot Behaviour

The OnPagePilot Crawler respects standard robots.txt directives on every fetch it makes — site crawling and cited-page verification alike. You can control its behaviour using the Oppbot token; a rule that blocks the crawler blocks cited-page verification too.

Already using OnPagePilot? Those rules still work. We honour both tokens, so nothing you have already written will stop being obeyed. New rules should use Oppbot.

Block the OnPagePilot Crawler entirely

User-agent: Oppbot
Disallow: /

Block specific directories

User-agent: Oppbot
Disallow: /private/
Disallow: /staging/

Set a custom crawl delay

User-agent: Oppbot
Crawl-delay: 2

Changes to your robots.txt are typically picked up within 24 hours.

Technical Auditor and Technology Scanner

These two agents do not spider your site and are not governed by robots.txt. They only visit specific URLs that our users have registered on our platform. To stop these requests, remove the corresponding monitors from your OnPagePilot account or contact us.