PreviewChecks

Keep preview crawlers out of challenges

A page can look normal in your browser while a link-preview crawler receives a challenge, a 403 response or a login wall. Test the public response first, then change only the rule that blocks legitimate preview traffic.

What a preview crawler needs

  • A public URL that returns the intended HTML without a cookie or signed-in session.
  • Metadata in the server response, not only after client-side JavaScript runs.
  • Public image URLs with a successful response and an image MIME type.
  • A redirect chain that stays reachable under the same security policy.
  • No interactive challenge, consent wall or geofence in front of the metadata.

Diagnose the response in order

  1. 1. Start with a fresh regular request

    Fetch the public URL without browser cookies. Record the final URL, status, content type and response body. A 200 response is not enough if the body is a challenge page or an HTML error document.

  2. 2. Compare crawler-like requests

    Compare the same URL with common social crawler user agents. A changed status, title, body or redirect can reveal a user-agent rule, bot score, managed challenge or origin behavior that the browser path did not trigger.

  3. 3. Fetch every selected image

    Image hosts often have separate hotlink, WAF and cache rules. Follow redirects and verify the final status, TLS connection, MIME type and decoded dimensions for the exact URL in the metadata.

  4. 4. Find the matching security event

    Use your provider's event log to identify the product and rule that acted on the request. Do not disable several protections at once: that hides the cause and creates a wider exception than the preview needs.

Use verified identity where available

A user-agent string is a claim, not proof. Anyone can send a request that says it is a social crawler. Prefer provider-maintained known-bot or verified-bot signals that use published IP ranges, reverse DNS or signed identity.

Cloudflare exposes the known-bot field cf.client.bot on all plans and the more specific cf.bot_management.verified_bot with Bot Management. Its WAF guidance shows how to exclude known or verified bots from a challenge rule. Apply the exception before the rule that would block or challenge the request.

Cloudflare also documents an important limitation: Bot Fight Mode cannot be skipped by a custom rule. Super Bot Fight Mode and Ruleset Engine phases can be skipped selectively. Check which feature generated the event before assuming a custom allow rule will take effect. 

(not cf.client.bot and ip.src.country in {"US" "MX"})

This is Cloudflare's documented pattern for challenging selected traffic while excluding known bots. Adapt the countries, action and scope to your own policy; do not paste a rule without reading the matching security events.

Keep the exception narrow

A public marketing page and its preview image do not need the same controls as login, checkout or API routes. Scope an exception to verified traffic and only the phases or paths that block the public asset. Keep rate limits and application security in place where they are still useful.

Do not allow every request with facebook, twitter or linkedin in its user agent. Do not allow a broad IP range copied from an old article. Bot ranges and signatures change, and a stale list can both block the real crawler and admit unrelated traffic.

After changing the rule, rerun the regular and crawler-like checks, inspect the security log again, and use the platform's official inspector when one exists. A PreviewChecks simulation can expose a risky policy, but it does not originate from the platform's verified network and cannot prove that platform's private fetch path.

Release checklist

  • HTML and preview images work without cookies, JavaScript or authentication.
  • Regular and crawler-like requests receive the intended content type and body.
  • The exact blocking security event and rule are known.
  • The exception uses verified identity where the provider supports it.
  • The exception is limited by path, phase and action instead of disabling protection.
  • The result is retested from outside the browser session.
Compare crawler access

Primary references

Reviewed July 14, 2026.