OnPagePilot Bots

Crawling infrastructure

Like a search engine, OnPagePilot relies on crawling — and because the web is now read by AI assistants as well as search engines, we crawl the sites our customers register to keep their SEO and AI-visibility analysis current.

OnPagePilot Crawler

The OnPagePilot Crawler is our primary site-crawling agent. It visits websites that have been registered by their owners on our platform and collects publicly available information such as page structure, meta tags, link relationships, and technical SEO signals to power our analysis tools.

OnPagePilot Bots

Once live, general web surfing will be an automated process in which we send software - often AI-driven agents - across the web to discover pages and interpret their content. These agents will map what exists, what’s new, and what’s changed, so that when you run a deep search, the system will already know the page and can surface it.

OnPagePilot Technical Auditor

The Technical Auditor performs targeted security and performance probes on URLs registered by site owners in our IndexMonitor system. It does not crawl or discover new pages. It checks specific endpoints for correct HTTPS configuration, security headers, and runs Lighthouse audits to measure page performance and accessibility scores.

OnPagePilot Technology Scanner

The Technology Scanner is triggered when a user adds a URL monitor on our platform. It performs a single-page visit using a headless browser to detect the technology stack in use (CMS, frameworks, analytics tools, etc.) and captures a visual screenshot. It does not follow links or crawl additional pages beyond the monitored URL.

Verification

You can verify that a request is genuinely from the OnPagePilot by checking the User-Agent header. All legitimate crawler requests will contain:

OnPagePilot Crawler/2.0 (+https://onpagepilot.com/bot)

Oppbot/1.0 (+https://onpagepilot.com/bot)

The Technical Auditor and Technology Scanner make targeted single-URL requests to domains registered by their owners and do not carry a custom user-agent token.

If you have concerns about traffic claiming to be from OnPagePilot, please contact us.

Benefits for Site Owners

Our bots help site owners maintain and improve their online presence by providing data for:

• Technical SEO Audits: the Crawler identifies broken links, missing meta tags, redirect chains, duplicate content, and other on-page issues across your entire site.
• Site Structure Analysis: the Crawler maps internal linking patterns and site architecture to reveal navigation bottlenecks and orphan pages.
• Indexing Insights: crawl data is compared against search engine behaviour to highlight pages that may be under-indexed or excluded.
• Security & HTTPS Validation: the Technical Auditor verifies SSL certificates, checks for missing HSTS headers, and detects faulty HTTP-to-HTTPS redirects before they affect visitors.
• Performance & Accessibility Scoring: Lighthouse audits run by the Technical Auditor measure page speed, accessibility compliance, and Core Web Vitals.
• Technology Detection: the Technology Scanner identifies your CMS, JavaScript frameworks, analytics tools, and other technologies so you can benchmark your stack.

Policies and Commitments

• Respect for robots.txt: our Crawler fully complies with the Robots Exclusion Protocol (RFC 9309), including Disallow, Allow, and Crawl-delay directives.
• Minimal server impact: the Crawler enforces a default delay of 500 ms between requests and automatically backs off when a server responds slowly or returns errors. The Technical Auditor and Technology Scanner make infrequent, single-URL requests.
• Owner-initiated only: all three bots operate exclusively on domains that have been registered on our platform by their owners or authorised representatives. We never crawl, probe, or scan sites without owner consent.
• No personal data collection: we process only publicly accessible page content. None of our bots attempt to access password-protected areas, submit forms, or collect personal data.
• Transparent identification: the Crawler identifies itself with a unique User-Agent string that includes a link back to this page. The Technical Auditor and Technology Scanner target only owner-registered URLs.
• GDPR compliant: our data processing complies with the EU General Data Protection Regulation. See our Legal & Privacy Policy for details.

Controlling Bot Behaviour

The OnPagePilot Crawler respects standard robots.txt directives. You can control its behaviour using the OnPagePilot token:

Block the OnPagePilot Crawler entirely

User-agent: OnPagePilot
Disallow: /

Block Oppbot entirely

User-agent: Oppbot
Disallow: /

Block specific directories

User-agent: OnPagePilot
Disallow: /private/
Disallow: /staging/

User-agent: Oppbot
Disallow: /private/

Set a custom crawl delay

User-agent: OnPagePilot
Crawl-delay: 2

Changes to your robots.txt are typically picked up within 24 hours.

Technical Auditor and Technology Scanner

These two agents do not spider your site and are not governed by robots.txt. They only visit specific URLs that the site owner has registered on our platform. To stop these requests, remove the corresponding monitors from your OnPagePilot account or contact us.