Summary
A server-side request forgery vulnerability in PraisonAI's web crawler allows a crawl target to redirect from a public URL to loopback, private-network, or cloud metadata services. When an attacker can influence the initial crawl URL, the crawler may retrieve internal responses and expose them to the agent context, even when local crawling is disabled.
Affected context
The issue occurs in deployments using PraisonAI's web_crawl functionality when the initial URL is validated but redirect destinations are not checked again. It becomes risky when an external party can influence a crawl target and the crawler has network access to loopback interfaces, private services, or cloud metadata endpoints.
Exploit preconditions
- Attack surface: The web_crawl function and its HTTP client
- Required access: Ability to influence or submit a URL for crawling
- Trigger point: HTTP redirect processing during a crawl request
- Network condition: The crawler runtime can reach internal or metadata services
- Why exploitation is possible: Redirect targets are followed automatically without applying the original URL validation rules to each destination
Why this happens
The vulnerability results from incomplete server-side request validation. The application checks the initial URL but delegates redirect handling to an HTTP client configured to follow redirects, without revalidating the resulting destinations.
This behavior maps to CWE-918: Server-Side Request Forgery, because attacker-influenced input causes the server to make requests to destinations that may not be intended for external access. Disabling local crawling does not provide sufficient protection when redirect processing can reach restricted address ranges.
Potential impact
- Internal services may be accessed through the crawler
- Cloud instance metadata or temporary credentials could be exposed
- Private network topology and service responses could be disclosed to the agent
- Internal administrative endpoints may be queried
- Retrieved sensitive data could influence downstream agent actions
General mitigation guidance
Mitigation generally focuses on enforcing destination validation throughout the complete request lifecycle rather than validating only the first URL.
Common defensive patterns include:
- Revalidating every redirect destination before following it
- Resolving hostnames and blocking loopback, link-local, private, reserved, and metadata address ranges
- Restricting outbound traffic from crawler workloads with network egress policies
- Using explicit allowlists for permitted schemes, hosts, and ports where feasible
- Disabling automatic redirects unless redirect behavior is required
- Running crawler components with minimal network privileges and service permissions
- Treating fetched content as untrusted before placing it into agent context
How teams detect issues like this
Organizations can identify SSRF risks through:
- Static analysis rules for HTTP clients that follow redirects without destination validation
- Code review checks comparing validation behavior for initial URLs and redirect targets
- Tests covering loopback, private-network, link-local, metadata, alternate-encoding, and DNS-resolution cases
- Egress monitoring for crawler workloads contacting internal address ranges
- Network policies that alert on or block unexpected access to metadata and administrative services
- Automated security review tools that flag attacker-controlled URLs passed to outbound request functions

