Custom

HTTP API

Post AI crawler requests to Searchly from your own code or edge functions.

Send events from your own code

#

For any other platform, choose Other (HTTP API) and post request events from your own code to the HTTP API URL in Setup. Edge code on AWS CloudFront, Akamai, or Fastly can post this format too. Setup shows a full example and a curl command.

Send POST requests with the header Authorization: Bearer followed by your token, and Content-Type: application/json. The body is {"events":[…]}, a JSON array of events, or one event per line with Content-Type: application/x-ndjson, at most 4 MB per request.

FieldValue
timestampRequired. When the request happened: an ISO 8601 date and time with seconds and a time zone, such as 2026-10-16T09:30:00Z or 2026-10-16T11:30:00+02:00, or a Unix timestamp in seconds, milliseconds, microseconds, or nanoseconds. A time without a time zone, such as 2026-10-16T09:30:00, is not stored. observedAt is also accepted. Events older than 7 days, or more than 5 minutes in the future, are not stored.
host and pathThe requested host and path. Searchly removes any query string.
urlInstead of host and path, the absolute URL. Searchly removes the query string.
userAgentRequired. The request's full User-Agent.
methodOptional. GET when omitted.
statusOptional. The response's HTTP status. statusCode is also accepted.
ipOptional. The client IP address your CDN or platform reports, never a value read from X-Forwarded-For. It is used only to verify the request and is not stored.
requestIdOptional. Your platform's ID for the request, so a retried post stores no event twice.

Other fields are ignored. You can send every request or only crawler requests: Searchly keeps only requests whose user agent names an AI crawler it recognizes and discards the rest in memory. A successful post returns 202 with counts of what was received, stored, and set aside.

Crawler logs