Summary
Crawl4AI can be affected by a server-side request forgery issue when link previews are enabled for crawl requests. An API client that controls a referenced page may cause the service to fetch internal, private, or cloud-metadata URLs and return parsed response data.
Crawl4AI can be affected by a server-side request forgery issue when link previews are enabled for crawl requests. An API client that controls a referenced page may cause the service to fetch internal, private, or cloud-metadata URLs and return parsed response data.
Affected context
The issue may occur in Crawl4AI deployments that expose the crawl API, enable
link_preview_config, and allow request-controlled pages to use external-link
inclusion and pattern-based discovery. Risk is higher when the crawler runs in
an environment with network access to private services, localhost endpoints, or
cloud instance metadata systems.
The issue may occur in Crawl4AI deployments that expose the crawl API, enable
link_preview_config, and allow request-controlled pages to use external-link
inclusion and pattern-based discovery. Risk is higher when the crawler runs in
an environment with network access to private services, localhost endpoints, or
cloud instance metadata systems.
Exploit preconditions
- Attack surface: The Crawl4AI crawl API and its link-preview processing path
- Required access: Ability to submit crawl requests or otherwise influence pages processed by the service
- Trigger point: Link discovery and fetching performed by
AsyncUrlSeeder - Why exploitation is possible: Request-controlled link-preview behavior may permit server-side fetches to destinations that are not sufficiently restricted by network or URL validation controls
- Attack surface: The Crawl4AI crawl API and its link-preview processing path
- Required access: Ability to submit crawl requests or otherwise influence pages processed by the service
- Trigger point: Link discovery and fetching performed by
AsyncUrlSeeder - Why exploitation is possible: Request-controlled link-preview behavior may permit server-side fetches to destinations that are not sufficiently restricted by network or URL validation controls
Why this happens
The vulnerability reflects insufficient restriction of server-side outbound requests
generated from attacker-influenced crawl content. When external links and matching
patterns are accepted without robust destination validation, the crawler may treat
internal or sensitive addresses as ordinary URLs and return their parsed metadata.
This behavior maps to CWE-918: Server-Side Request Forgery, in which an
application makes network requests on behalf of an attacker.
The vulnerability reflects insufficient restriction of server-side outbound requests
generated from attacker-influenced crawl content. When external links and matching
patterns are accepted without robust destination validation, the crawler may treat
internal or sensitive addresses as ordinary URLs and return their parsed metadata.
This behavior maps to CWE-918: Server-Side Request Forgery, in which an
application makes network requests on behalf of an attacker.
Potential impact
- Internal services may be probed or identified
- Private network resources may be accessed from the crawler's network position
- Cloud instance metadata or temporary credentials could be exposed
- Response metadata may disclose sensitive host or application information
- Internal systems could be affected if fetched endpoints perform state-changing actions
- Internal services may be probed or identified
- Private network resources may be accessed from the crawler's network position
- Cloud instance metadata or temporary credentials could be exposed
- Response metadata may disclose sensitive host or application information
- Internal systems could be affected if fetched endpoints perform state-changing actions
General mitigation guidance
Mitigation generally focuses on reducing trust in request-controlled URLs and
restricting the crawler's outbound network access.
Common defensive patterns include:
- Allowing outbound requests only to explicitly approved schemes, hosts, and ports
- Blocking loopback, link-local, private, reserved, and cloud-metadata address ranges
- Resolving and validating destinations after DNS resolution, including protection against rebinding
- Applying network egress policies that limit crawler access to required destinations
- Separating crawling workloads from sensitive internal networks and metadata services
- Returning only the minimum response data needed for link previews
- Requiring authentication and authorization for crawl operations
Mitigation generally focuses on reducing trust in request-controlled URLs and
restricting the crawler's outbound network access.
Common defensive patterns include:
- Allowing outbound requests only to explicitly approved schemes, hosts, and ports
- Blocking loopback, link-local, private, reserved, and cloud-metadata address ranges
- Resolving and validating destinations after DNS resolution, including protection against rebinding
- Applying network egress policies that limit crawler access to required destinations
- Separating crawling workloads from sensitive internal networks and metadata services
- Returning only the minimum response data needed for link previews
- Requiring authentication and authorization for crawl operations
How teams detect issues like this
Organizations can look for SSRF risks through:
- Static analysis of URL-fetching code and redirect handling
- Code review signals such as attacker-controlled URLs passed directly to HTTP clients
- Tests covering private, loopback, link-local, and metadata address handling
- Monitoring outbound crawler traffic for unexpected internal destinations or unusual DNS resolution
- Egress policies and alerts that detect requests to prohibited address ranges
- Automated security review tools that flag server-side network requests lacking destination validation
Organizations can look for SSRF risks through:
- Static analysis of URL-fetching code and redirect handling
- Code review signals such as attacker-controlled URLs passed directly to HTTP clients
- Tests covering private, loopback, link-local, and metadata address handling
- Monitoring outbound crawler traffic for unexpected internal destinations or unusual DNS resolution
- Egress policies and alerts that detect requests to prohibited address ranges
- Automated security review tools that flag server-side network requests lacking destination validation
