Cloudflare

Choose Cloudflare when your domain's DNS records are proxied through Cloudflare. Logpush is the recommended method.

Logpush (recommended)

A Logpush job sends your zone's HTTP request logs to Searchly. It needs no change to your site and sees every request Cloudflare handles, including requests that Cloudflare's security settings block, so it shows when Cloudflare is blocking an AI crawler.

1. In the Cloudflare dashboard, open your domain, go to Analytics & Logs, then Logpush, and create a job with an HTTP destination.
2. Paste the destination URL from Setup. It contains the token: right after you generate or rotate it, Setup fills the token in; otherwise replace <TOKEN> in the URL with the token you saved.
3. Choose the HTTP requests dataset, the fields Setup lists, and the filter Setup lists. The fields send the path without its query string and leave out referrers and cookies. The filter sends only requests whose user agent could name an AI crawler, so your visitors' requests stay in Cloudflare. It also lets some other bots through, such as SEO crawlers, and Searchly discards them.
4. Save the job. Cloudflare checks the destination with a small validation upload, which Searchly records as a delivery.

Setup also shows an API request that creates the same job and sets upload batch limits, which the dashboard can't set. Run it with your zone ID and a Cloudflare API token that can edit Logpush jobs. The request's destination_conf carries your Searchly token, not the Cloudflare API token: right after you generate or rotate it, Setup fills the token in; otherwise replace <TOKEN> in destination_conf with the token you saved.

Logpush is available on every Cloudflare plan and is billed by usage. If the HTTP requests dataset isn't offered for your zone, use the Worker. Logpush sends logs in batches, so a delivery can take a few minutes, and it sends nothing from before the job was created. It doesn't see hosts whose DNS records aren't proxied through Cloudflare.

Worker

A Cloudflare Worker on your site's routes reports each crawler GET and HEAD request after your site answers, and passes every other request to your site untouched. Use it when Logpush isn't available for your zone.

1. Create a Worker and replace its code with the code from Setup. If a Worker already runs on your site's routes, merge the code into that Worker instead, because only one Worker runs on a route.
2. In the Worker's settings, add a secret named SEARCHLY_CRAWLER_TOKEN with your token.
3. Add the routes Setup shows. For a domain such as example.com they are example.com/* and www.example.com/*; for any other hostname, such as shop.example.com, only that hostname. Add a route for every other hostname of your site that crawlers should be counted on, such as docs.example.com/*; *.example.com/* covers them all if no other Worker runs on those hostnames.
4. Set the routes' request limit failure mode to Fail open, so visitors still reach your site if your Workers plan's daily request limit is reached.

The Worker doesn't see requests that Cloudflare's security settings block before Workers run, such as a setting that blocks AI bots. On the Workers Free plan every request on the route counts toward the daily limit, and once it is reached, no requests are reported for the rest of that UTC day.