Summary
Kubeflow Pipelines versions before 2.18.0 allowed reachable callers to retrieve run or node logs without authorization checks against the selected run. By supplying known identifiers, an attacker could read sensitive workload output and, in some live-log cases, access logs from an unrelated pod.
Kubeflow Pipelines versions before 2.18.0 allowed reachable callers to retrieve run or node logs without authorization checks against the selected run. By supplying known identifiers, an attacker could read sensitive workload output and, in some live-log cases, access logs from an unrelated pod.
Affected context
The issue occurs in the Kubeflow Pipelines API server's /apis/v1alpha1/runs/{run_id}/nodes/{node_id}/log handler when the endpoint is reachable without an effective authentication and authorization layer. The handler does not verify that the caller may access the requested run or that a selected live-log pod belongs to that run. External authentication gates, network policies, and deployment-specific access controls may reduce exposure.
The issue occurs in the Kubeflow Pipelines API server's /apis/v1alpha1/runs/{run_id}/nodes/{node_id}/log handler when the endpoint is reachable without an effective authentication and authorization layer. The handler does not verify that the caller may access the requested run or that a selected live-log pod belongs to that run. External authentication gates, network policies, and deployment-specific access controls may reduce exposure.
Exploit preconditions
- Attack surface: Kubeflow Pipelines API server log retrieval endpoint - Required access: Network reachability to the backend; endpoint authentication
may be absent unless supplied by an external gate
- Required knowledge: A victim run identifier and a suitable node or pod
identifier
- Trigger point: A request to retrieve archived or live logs - Why exploitation is possible: The handler lacks run-level authorization and
does not reliably bind a caller-selected live-log pod to the requested run
- Attack surface: Kubeflow Pipelines API server log retrieval endpoint - Required access: Network reachability to the backend; endpoint authentication
may be absent unless supplied by an external gate
- Required knowledge: A victim run identifier and a suitable node or pod
identifier
- Trigger point: A request to retrieve archived or live logs - Why exploitation is possible: The handler lacks run-level authorization and
does not reliably bind a caller-selected live-log pod to the requested run
Why this happens
The underlying bug is a missing authorization check, mapped primarily to CWE-862. The log handler retrieves resources based on caller-supplied identifiers without first establishing that the caller is permitted to access the associated run. For live logs, accepting a node identifier as a pod selection without validating resource ownership also creates an object reference and tenant-boundary failure, contributing to CWE-200 exposure of sensitive information.
The underlying bug is a missing authorization check, mapped primarily to CWE-862. The log handler retrieves resources based on caller-supplied identifiers without first establishing that the caller is permitted to access the associated run. For live logs, accepting a node identifier as a pod selection without validating resource ownership also creates an object reference and tenant-boundary failure, contributing to CWE-200 exposure of sensitive information.
Potential impact
- Attackers may read confidential machine learning workflow logs.
- Exposed logs could contain credentials, tokens, inputs, outputs, or dataset details.
- Attackers could access archived logs from runs they are not authorized to view.
- A known run identifier may allow selection of an unrelated pod's main-container logs in affected live-log scenarios.
- Disclosure may support further compromise of workloads or connected services.
- Attackers may read confidential machine learning workflow logs.
- Exposed logs could contain credentials, tokens, inputs, outputs, or dataset details.
- Attackers could access archived logs from runs they are not authorized to view.
- A known run identifier may allow selection of an unrelated pod's main-container logs in affected live-log scenarios.
- Disclosure may support further compromise of workloads or connected services.
General mitigation guidance
Mitigation generally focuses on enforcing authorization at the API boundary and maintaining strict resource ownership checks. Secure-by-default designs should require authenticated requests, authorize access to the specific run and node before log retrieval, and verify that any live-log pod belongs to the requested workflow. Caller-controlled identifiers should be treated as untrusted references and resolved through server-side ownership relationships. Network policies and external authentication gates can reduce reachability, and least-privilege service accounts should limit the API server's ability to read unrelated workloads. Sensitive values should also be excluded or redacted from logs where practical.
Mitigation generally focuses on enforcing authorization at the API boundary and maintaining strict resource ownership checks. Secure-by-default designs should require authenticated requests, authorize access to the specific run and node before log retrieval, and verify that any live-log pod belongs to the requested workflow. Caller-controlled identifiers should be treated as untrusted references and resolved through server-side ownership relationships. Network policies and external authentication gates can reduce reachability, and least-privilege service accounts should limit the API server's ability to read unrelated workloads. Sensitive values should also be excluded or redacted from logs where practical.
How teams detect issues like this
Detection can include static analysis and code review for handlers that use caller-supplied run, node, namespace, or pod identifiers without an explicit authorization decision. Reviewers can look for missing tenant or ownership checks, direct object lookup before policy evaluation, and Kubernetes resource access that is not bound to the requested workflow. Runtime monitoring can alert on unusual log-access volume, cross-workflow access patterns, and requests lacking expected identity context. API policy enforcement and automated security review tools can help identify unauthenticated or under-authorized routes.
Detection can include static analysis and code review for handlers that use caller-supplied run, node, namespace, or pod identifiers without an explicit authorization decision. Reviewers can look for missing tenant or ownership checks, direct object lookup before policy evaluation, and Kubernetes resource access that is not bound to the requested workflow. Runtime monitoring can alert on unusual log-access volume, cross-workflow access patterns, and requests lacking expected identity context. API policy enforcement and automated security review tools can help identify unauthenticated or under-authorized routes.
