Hosts and CDNs

Vercel and Next.js

Send AI crawler requests with a Vercel log drain or Next.js middleware.

On this page

Connect Vercel or Next.js

#

On Vercel, choose a log drain or middleware, not both. For a Next.js site on any other host that runs Next.js as a server, choose Next.js; it uses the middleware method.

Log drain

A Vercel log drain sends your project's request logs to Searchly. It needs no code change and records each response's status. Drains need a Vercel Pro or Enterprise team, and Vercel bills them by the volume of every log line sent, including your visitors' requests, which Searchly discards in memory.

  1. In your Vercel team settings, open Drains and add a drain with the Logs data type.
  2. Choose only your site's project and the Production environment. Select every source except build, and add no sampling rules.
  3. Choose a custom endpoint with the NDJSON format and paste the Vercel drain URL from Setup. Under Custom Headers, add the line Setup shows: Authorization: Bearer followed by your token.
  4. Save. Vercel tests the endpoint, and Searchly records the test as a delivery.

If your team hides IP addresses in drains, requests can't be checked against operators' published ranges and are stored as declared.

Middleware

Next.js middleware reports crawler requests from inside your app, on any Vercel plan or on any other host that runs Next.js as a server. The matcher Setup generates runs it only for requests whose user agent names an AI crawler, so your visitors' requests don't invoke it.

  1. If your app already has a proxy.ts or middleware.ts, keep that file: save the helper from Setup next to it and call the helper first in your existing function. Keep the function's name and your matcher; the helper checks the user agent itself. Next.js 16 still runs middleware.ts, and its build fails when both files exist. If your app has neither file, create proxy.ts (Next.js 16 and later) or middleware.ts (Next.js 15 and earlier) at the root of your project, or in src, with the code from Setup.
  2. In your host's environment settings, add SEARCHLY_CRAWLER_TOKEN with your token for Production.
  3. Deploy, so the code and the variable take effect.

Middleware runs before your page responds, so it doesn't see the final status: its requests have no status and are left out of success and error rates. It doesn't see redirects defined in next.config, which run before it, and it sends no IP address, so its requests are declared. It never runs on a static export (output: 'export'), because no server runs it; use your host's or CDN's method, such as Cloudflare, or the HTTP API.

Crawler logs