Summary
An unsafe deserialization issue in the HuggingFace Transformers Trainer class allows arbitrary code execution when loading RNG state checkpoints via torch.load() without weights_only=True.
When running with PyTorch versions prior to 2.6, an attacker who can provide or overwrite checkpoint files such as rng_state.pth may trigger code execution during training resumption or checkpoint loading.
Affected context
This vulnerability appears in environments using the HuggingFace Transformers Trainer with:
- Transformers versions that include
_load_rng_state()callingtorch.load()withoutweights_only=True - PyTorch versions < 2.6 (where
safe_globals()does not prevent arbitrary execution) - Workflows that load RNG or training state from disk, such as resuming training, distributed training, or running experiments on shared infrastructure
The risk is most relevant when checkpoint directories or files (for example rng_state.pth) can be influenced, uploaded, or modified by untrusted users or processes.
Exploit preconditions
- Attack surface: Filesystem-based model/checkpoint loading through the Transformers
TrainerAPIs. - Required access: Ability to create, upload, or replace RNG state/checkpoint files (e.g.,
rng_state.pth) in locations that the training process loads from. - Trigger point: Invocation of
_load_rng_state()insideTrainer, typically when resuming training, recovering from a previous run, or loading a training state. - Why exploitation is possible:
torch.load()is invoked withoutweights_only=Trueunder PyTorch versions < 2.6, allowing execution of pickled Python objects embedded in the checkpoint, and thesafe_globals()mechanism does not meaningfully constrain this behavior in those versions.
Why this happens
The underlying bug stems from using a general-purpose, pickle-based deserialization API (torch.load) on data that is not guaranteed to be trusted.
In the _load_rng_state() method, the training RNG checkpoint is loaded without the weights_only=True safeguard, which means arbitrary Python objects inside the file can be instantiated and executed as part of the unpickling process.
This aligns with CWE-502: Unsafe Deserialization, where attacker-controlled serialized data is treated as benign state rather than potentially executable content, and is deserialized in a security-sensitive context (a long-running training process with access to system resources).
Potential impact
- Execution of arbitrary Python code with the privileges of the training process
- Theft or exfiltration of training data, model parameters, or API keys accessible from the environment
- Manipulation or poisoning of models, datasets, or experiment artifacts
- Deployment of backdoored models into downstream inference or production pipelines
- Use of compromised training jobs as a foothold for lateral movement within shared compute clusters
General mitigation guidance
Mitigation commonly focuses on both safe deserialization patterns and tightening trust boundaries around checkpoints:
- Prefer deserialization modes that treat files as data only, such as using
torch.load(..., weights_only=True)when available, or equivalent APIs that do not execute arbitrary pickled objects. - Restrict where checkpoints and RNG state files can originate from; treat any user-supplied or cross-tenant checkpoint as untrusted.
- Separate responsibilities so that only trusted components can write or update training state files used by production or shared training jobs.
- Run training processes with least-privilege OS and cloud permissions, limiting the impact of a compromised process.
- Apply filesystem and storage controls (e.g., permissions, isolation, non-shared checkpoint paths) to prevent lower-privileged users from injecting or modifying checkpoints.
- Consider validating or signing checkpoints in high-assurance environments to ensure they come from expected, trusted sources.
How teams detect issues like this
Detection typically combines code-level analysis with runtime controls:
- Static analysis / SAST: Flag uses of
torch.load()and similar APIs that omit safe parameters (such asweights_only=True) when loading checkpoints or state from paths that are not hard-coded or are user-configurable. - Code review signals: Pay special attention to any code paths that deserialize training state, RNG state, or model artifacts, especially when reading from shared directories, object storage, or user-uploaded locations.
- Security policies: Establish internal guidelines requiring safe deserialization patterns and explicit documentation of trust assumptions for checkpoint sources.
- Runtime monitoring: Watch for anomalous behavior by training processes (unexpected network calls, filesystem writes outside expected directories, or process spawning) when checkpoints are loaded, using EDR, audit logs, or container security tooling.

