SearchlyBot

SearchlyBot is the crawler behind Site Health, the part of Searchly that audits a website for technical, AEO and content issues. It only visits the websites of Searchly projects, and only for two reasons:

• An audit runs. Someone on the project chose pages to audit in Site Health, or started an audit through the API or the MCP server. SearchlyBot fetches the pages that audit covers.
• The daily check. Once a day, SearchlyBot fetches the site's robots.txt, llms.txt and sitemap.

It finds pages to audit in the sitemap (listed on Sitemap: lines in robots.txt, or at /sitemap.xml), among the site's pages that AI answers cite, and in links on pages it has already audited. It does not crawl other websites.

SearchlyBot identifies itself with this user agent:

Mozilla/5.0 (compatible; SearchlyBot/1.0; +https://docs.trysearchly.com/searchlybot)

Match the token SearchlyBot rather than the whole string, because the version number can change.

• It follows robots.txt: the group for User-agent: SearchlyBot when there is one, otherwise the * group.
• It honors Crawl-delay, up to 10 seconds between requests.
• It makes at most 3 requests at a time to a site.
• It reads the HTML each URL returns and does not run JavaScript. Content that only appears after JavaScript runs is invisible to it, as it is to AI crawlers that don't run JavaScript.

Add a group for SearchlyBot to your robots.txt. When that group exists, SearchlyBot follows only it and ignores the * group, so repeat in it any * rules you still want it to follow.

This example keeps SearchlyBot out of /admin/ and /checkout/ and asks it to wait 5 seconds between requests. Every path the group doesn't disallow stays open to it.

User-agent: *
Disallow: /admin/

User-agent: SearchlyBot
Disallow: /admin/
Disallow: /checkout/
Crawl-delay: 5

To keep SearchlyBot off the whole site, give its group the single rule Disallow: /. Site Health can't audit the pages SearchlyBot isn't allowed to fetch.