AI crawler traffic

Analytics → Overview → Crawlers shows server requests from OpenAI, Anthropic and Perplexity. Filter by provider, request type and verification; inspect requested pages, HTTP failures and retained request evidence. Human visitors remain in the Visitors report.

Node.js

Install @saaspro/node and use the secret ingest key for the selected environment. Add public site origins in Application settings. Never send this key to the browser.

import { SaaSProMax } from "@saaspro/node";

const spm = new SaaSProMax({
  secretKey: process.env.SPM_SECRET_KEY!,
  host: "https://saaspro.dev",
});

const response = await yourHandler(request);
await spm.crawlers.capture(request, { status: response.status });
return response;

Use your framework's background-work hook so capture does not delay the page response. For a Next.js route handler, schedule after(async () => { await spm.crawlers.capture(request, { status: response.status }); }) using after from next/server. A request-only hook can omit status; the report shows Status unknown.

The SDK filters known GET/HEAD requests, strips query strings and fragments, bounds concurrent captures to eight, and returns false on failure without throwing. Delivery is best effort with a five-second maximum timeout; there is no durable retry queue. shutdown() cancels outstanding crawler captures.

Express

import { expressCrawlerMiddleware } from "@saaspro/node";

app.use(expressCrawlerMiddleware(spm, {
  origin: "https://example.com",
}));

Register before your routes. It observes the final status after the response finishes. The origin is configured explicitly rather than copied from an untrusted Host header.

Source IP and verification

Pass { sourceIp } to capture, or provide the Express adapter's sourceIp(request) callback, only when the address comes from a trusted network boundary. Do not blindly copy forwarded headers. Your proxy must overwrite them and prevent direct access that bypasses the proxy.

The collector classifies the user agent itself and compares the reported address with published provider CIDRs. IP evidence is therefore conditional on your server reporting the correct source. It is not a request signature or proof of content use.

Provider feeds refresh every six hours. Failed refreshes discard old verification data and retry after one minute. Redirects and oversized or malformed feeds are refused. The supported directory contains nine agent tokens; ambiguous or unknown agents are ignored.

Direct HTTP

POST /api/v1/crawlers/collect
Authorization: Bearer spm_sec_prod_...
Content-Type: application/json

{"requests":[{"id":"d9b83600-0514-4612-8132-655c514bc26b","timestamp":"2026-09-08T00:00:00Z","url":"https://example.com/docs","method":"GET","userAgent":"GPTBot/1.4","status":200}]}

Generate a fresh UUID and current timestamp for each observed request. Optional ip is the trusted source address. Do not supply provider, category or verification fields. A batch holds 1–100 requests, is limited to 256 KiB, and accepts observations up to 24 hours late with one minute of future clock tolerance.

HTTP 202 returns received, stored, duplicate, ignored and budget counts. Retry using the same UUID and content; conflicting reuse returns 409 and rolls back that batch. Public keys are refused. The reported page origin must match the application's allowed origins when configured.

Collection controls

Collection settings support pause/resume, request categories, a UTC daily budget and 1–30 day retention. The default is seven days and 100,000 accepted requests per day. Each environment stores at most 100,000 request records. Current report totals cover retained observations, not estimated missed traffic.

Shortening retention requires a reason and hides older evidence immediately; the worker physically removes expired records every five minutes in bounded batches. Clearing requests requires a reason and fresh passkey, preserves usage counters, and rejects observations captured before the clear. Usage counters expire after 90 UTC days.

Raw IPs, full user agents, request headers/bodies, cookies, query strings and fragments are never stored. Known sensitive path patterns are redacted. All page paths additionally require people.pii.read, because arbitrary application paths can contain personal data. Requests do not enter visitor, funnel, attribution or experiment datasets.

Management API routes under /api/v1/apps/{app}/analytics/crawlers expose the report and deletion; /settings exposes policy reads and updates. They retain organization membership, environment, module, permission and audit checks.

Check the installation

1. Confirm the chosen environment and allowed origins.
2. Observe a known crawler request on a public page. A manual request with a crawler user agent verifies delivery only; it does not prove crawler authenticity.
3. Check Stored today, Request evidence, HTTP status and verification state. An empty report may mean that no supported crawler has requested the page.

Crawler requests do not establish an AI citation, referral, training use or purchase. Agent purpose labels describe the provider's documented token, not an observed downstream outcome. See the official OpenAI crawler directory, Anthropic crawler guidance, and Perplexity crawler guidance.