Summary
Ray's WebDataset reader performs unsafe deserialization of .pkl, .pickle, .pt, and .pth entries from tar archives, enabling arbitrary code execution during dataset loading.
When remote workers call read_webdataset() on attacker-controlled archives, malicious payloads are executed across all workers processing the data.
Affected context
This vulnerability appears in Ray deployments that use the WebDataset reader for ingesting data packaged as tar archives.
It is most relevant when datasets can be influenced by external or semi-trusted parties, or when archives are loaded without strict provenance and integrity checks.
The issue surfaces at deserialization time, as Ray remote workers process .pkl, .pickle, .pt, and .pth files using generic, code-executing loaders.
Exploit preconditions
- Attack surface: Data ingestion path via the
read_webdataset()function in Ray. - Required access: Ability to supply or modify tar archives consumed by Ray jobs (e.g., dataset storage, object store, or file system used by the pipeline).
- Trigger point: Invocation of
read_webdataset()causing_default_decoder()to automatically deserialize matching entries. - Why exploitation is possible: The decoder blindly trusts file extensions and uses
pickle.loads()andtorch.load(weights_only=False)on untrusted content, both of which can run attacker-provided code during deserialization.
Why this happens
The underlying bug stems from treating data inputs as trusted serialization streams without isolating or constraining their behavior.
In _default_decoder(), tar entries with specific extensions are deserialized using Python's pickle and PyTorch's torch.load in their default, code-executing modes, directly mapping to CWE-502: Unsafe Deserialization.
This pattern effectively embeds executable logic in data, allowing arbitrary object graphs and constructors from untrusted sources to run inside Ray remote workers during routine dataset reading.
Potential impact
- Remote code execution on Ray worker processes handling the malicious dataset.
- Full compromise of affected workers, including access to in-memory data and credentials available to those processes.
- Lateral movement within the Ray cluster or underlying infrastructure, depending on worker privileges and network connectivity.
- Data corruption, exfiltration, or sabotage of ML training pipelines and inference workloads.
- Use of compromised workers as a foothold for broader supply-chain or platform-level attacks.
General mitigation guidance
Effective mitigation generally focuses on eliminating or strictly constraining unsafe deserialization of dataset content.
Common secure patterns include:
- Avoiding generic deserialization mechanisms (
pickle, unrestrictedtorch.load) for data from non-fully trusted sources. - Introducing non-executable, schema-driven formats (e.g., JSON, Parquet, protocol buffers) for datasets instead of arbitrary pickled objects.
- Applying explicit validation and whitelisting of allowed data structures before reconstruction, favoring simple, value-only representations.
- Running Ray workers with least privilege, network segmentation, and hardened runtime profiles to limit the blast radius if deserialization bugs occur.
- Treating all externally supplied datasets as untrusted by default and enforcing provenance, integrity checks, and review before ingestion.
How teams detect issues like this
Detection usually combines code-level analysis with operational monitoring:
- Static analysis rules to flag use of
pickle.loads,torch.loadwith executable states, and similar APIs in data ingestion paths. - Code review practices that scrutinize deserialization logic, especially wherever worker nodes parse user-supplied or externally sourced datasets.
- Policy-based controls that require schema-based formats and ban unsafe serializers in shared libraries and data loaders.
- Runtime monitoring for anomalous behavior in worker processes when new datasets are introduced, such as unexpected network calls or filesystem activity.
- Optional use of automated security scanning tools integrated into CI/CD to identify unsafe deserialization patterns across the Ray ecosystem.

