v36.0.0 regression: exec WebSocket returns 401 Unauthorized on AKS (works in v35.0.0)

What happened?

After upgrading the kubernetes Python client from 35.0.0 to 36.0.0, pod exec via kubernetes.stream.stream() consistently fails against an Azure AKS cluster with 401 Unauthorized during the WebSocket handshake. Regular REST API calls (v1.list_namespaced_pod, v1.read_namespaced_pod, etc.) continue to work with the same kubeconfig and same identity. Downgrading the library to 35.0.0 immediately restores exec functionality with no other changes.

What did you expect to happen?

stream(v1.connect_get_namespaced_pod_exec, ...) should succeed in v36.0.0 the same way it does in v35.0.0, against the same AKS cluster, with the same kubeconfig.

How can we reproduce it (as minimally and precisely as possible)?

AKS cluster (Private Link), kubectl exec works fine from the same host.
kubeconfig uses an exec: auth plugin (kubelogin with Azure AD).
Run the following Python script with kubernetes==36.0.0:

from kubernetes import client, config
from kubernetes.stream import stream

cfg = client.Configuration()
cfg.verify_ssl = False
config.load_kube_config(config_file="/path/to/aks/kubeconfig", client_configuration=cfg)
cfg.proxy = None
v1 = client.CoreV1Api(api_client=client.ApiClient(cfg))

# REST: works in both v35 and v36
print([p.metadata.name for p in v1.list_namespaced_pod("vault").items])

# Exec WebSocket: works in v35, fails in v36 with 401
resp = stream(
    v1.connect_get_namespaced_pod_exec,
    "hashicorp-vault-0", "vault",
    command=["/bin/sh", "-c", "vault status"],
    stderr=True, stdin=False, stdout=True, tty=False,
)
print(resp)

Anything else we need to know?

The traceback from v36.0.0:

File ".../kubernetes/stream/ws_client.py", line 552, in create_websocket
    websocket.connect(url, **connect_opt)
File ".../websocket/_handshake.py", line 158, in _get_resp_headers
    raise WebSocketBadStatusException(
websocket._exceptions.WebSocketBadStatusException: Handshake status 401 Unauthorized -+-+-
{'audit-id': '...', 'cache-control': 'no-cache, private', 'content-type': 'application/json',
 'date': '...', 'content-length': '129', 'connection': 'close'}
-+-+- b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure",
        "message":"Unauthorized","reason":"Unauthorized","code":401}\n'

The 401 response carries an audit-id, confirming the request reaches the AKS API server and is rejected at the auth layer (not a network/proxy issue).

Root cause (suspected)

PR #2486 (implement pod exec websockets v5) changed the advertised WebSocket subprotocols from a single value to a comma-separated list:

kubernetes/base/stream/ws_client.py — v35.0.0:

header.append("sec-websocket-protocol: v4.channel.k8s.io")

kubernetes/base/stream/ws_client.py — v36.0.0:

header.append("sec-websocket-protocol: %s,%s" % (V5_CHANNEL_PROTOCOL, V4_CHANNEL_PROTOCOL))
# → "sec-websocket-protocol: v5.channel.k8s.io,v4.channel.k8s.io"

The Authorization header handling is identical between the two versions, and the Authorization header is sent. However, when the AKS API server receives multiple subprotocols in Sec-WebSocket-Protocol, it appears to take a different auth code path (presumably the browser-style base64url.bearer.authorization.k8s.io.<token> lookup inside the subprotocol header) and fails to fall back to honoring the standard Authorization header, returning 401 instead of negotiating down to v4.channel.k8s.io.

This makes the regression appear specifically against cluster/control-plane versions that don't natively support the v5 subprotocol.

Workaround

Pinning kubernetes==35.0.0 resolves the issue.

Kubernetes client version

$ pip show kubernetes | grep Version
Version: 36.0.0

Server version (AKS)

*.privatelink.<region>.azmk8s.io (Azure AKS, control plane managed by Azure; happy to share exact Kubernetes minor if needed)

OS

Red Hat Enterprise Linux 9.x (containerized Python 3.12 environment)

Python version

Python 3.12.11

Python client version

kubernetes==36.0.0
websocket-client==1.9.0

Installation method

pip3 install kubernetes

Re: comma vs comma-space formatting

Monkey-patched the installed ws_client.py to use "sec-websocket-protocol: %s, %s" (with a space) and re-ran the reproducer:

Still 401 Unauthorized.

Same traceback / response headers as the original report. So the comma formatting isn't what AKS is reacting to.

Re: AKS Kubernetes version

Client Version: v1.33.7
Kustomize Version: v5.6.0
Server Version: v1.33.1

So this AKS control plane is on v1.33.1, which is well past 1.30 where the v5 channel protocol was introduced server-side. The server should be fully capable of negotiating v5.

So the situation is

kubectl --kubeconfig=… exec … works against this same cluster, same kubeconfig.
Python client v35.0.0 (advertises only v4.channel.k8s.io) → exec works.
Python client v36.0.0 (advertises v5.channel.k8s.io,v4.channel.k8s.io) → 401 Unauthorized at the WebSocket handshake.
REST calls (list_namespaced_pod, etc.) on v36 work fine with the same Configuration / token.

So the bearer token is valid and is being honored by the server for REST. Something specifically about the v5 advertisement (rather than the auth header propagation) is causing the API server to reject the WebSocket handshake on AKS 1.33.1.

A few possibilities that might be worth investigating:

v5 auth path expects the token in the subprotocol header (browser-style: Sec-WebSocket-Protocol: base64url.bearer.authorization.k8s.io.<base64-token>, v5.channel.k8s.io, v4.channel.k8s.io) rather than (or in addition to) the Authorization header. kubectl may be doing this; the Python client currently only sets Authorization.
AKS-specific webhook/RBAC interception on the exec subresource that triggers only when v5 is selected.
Some interaction with the kubelogin-issued AAD token that the v5 negotiation path doesn't handle the same way as v4.

Happy to test any specific patch you'd like me to try against this cluster (e.g., dropping the Authorization header and embedding the bearer token in Sec-WebSocket-Protocol, or making v5 advertisement opt-in via Configuration). Just tell me what to monkey-patch and I'll come back with the result.

could it be related to #2592 & #2591 & #2584

@aojea

Suggested fix

I won't be able to send a PR myself (corporate process is non-trivial), but the change should be very small. Maintainers — or anyone in the community — feel free to pick this up.

Option A (preferred): update `kube_config.py` to match v36's generated key name

In kubernetes/base/config/kube_config.py, change every line that writes to api_key['authorization'] to write to api_key['BearerToken'] instead.

- configuration.api_key['authorization'] = "Bearer " + self.token
+ configuration.api_key['BearerToken'] = "Bearer " + self.token

Quick way to find every occurrence:

rg "api_key\['authorization'\]" kubernetes/base/config/

I verified locally with a one-line sed against the installed v36 copy and REST + WebSocket exec both work end-to-end after the change.

Option B: make `auth_settings()` accept the legacy key too (backward-compat)

If there's concern about breaking users who set Configuration.api_key['authorization'] directly in their own code, the alternative is to extend the generated auth_settings() to fall back to 'authorization':

def auth_settings(self):
    auth = {}
    bearer_key = (
        'BearerToken' if 'BearerToken' in self.api_key
        else 'authorization' if 'authorization' in self.api_key
        else None
    )
    if bearer_key:
        auth['BearerToken'] = {
            'type': 'api_key',
            'in': 'header',
            'key': 'authorization',
            'value': self.get_api_key_with_prefix(bearer_key),
        }
    return auth

This is the safer option for downstream consumers since it doesn't change any externally observable contract. The downside is that configuration.py is auto-generated, so the patch would need to live in the templates or post-generation step.

Recommended test

A regression test in kube_config_test.py after load_kube_config():

self.assertIn('BearerToken', cfg.auth_settings())
self.assertTrue(cfg.auth_settings()['BearerToken']['value'])

If this had existed in the v36 cut, the bug would have been caught immediately.

Workaround for users on v36 in the meantime

Pin kubernetes==35.0.0, or apply this two-line shim after load_kube_config():

if 'authorization' in cfg.api_key and 'BearerToken' not in cfg.api_key:
    cfg.api_key['BearerToken'] = cfg.api_key['authorization']

Happy to test any fix branch against the same AKS cluster.

cc: @aojea @yliaog @seans3

confirming this is affected EKS as well

Same with GKE, tested with 36.0.1 today and had to revert to 35.0.0

@bahag-klickst

@bahag-klickst is 36.0.1 showing the same failure?

Based on comment #2595 (comment) it seems the problem is because of the change on the BearerToken header that was fixed in that version ... can you provide a repro and the logged errors?

@aojea

Hej @aojea

have tested it today with 36.0.1 and received the following error which does not happen with 35.0.0

Exception when calling BatchV1Api->create_namespaced_job: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Audit-Id': '8c72d720-b643-4b85-adee-db0a067a2c7e', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Wed, 27 May 2026 06:08:53 GMT', 'Content-Length': '129'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

The code itself is quite simple

credentials, project_id = google.auth.default()
auth_req = google.auth.transport.requests.Request()
credentials.refresh(auth_req)
cluster_manager_client = ClusterManagerClient(credentials=credentials)
cluster=f"projects/{project_id}/locations/us-central1-a/clusters/clus-1337"
cluster = cluster_manager_client.get_cluster(name=cluster)
# set kubernetes config
config = client.Configuration()
config.host = f"https://{cluster.endpoint}"
config.debug = False
with NamedTemporaryFile(delete=False) as ca_cert:
    ca_cert.write(base64.b64decode(cluster.master_auth.cluster_ca_certificate))
    config.ssl_ca_cert = ca_cert.name
config.verify_ssl = True
config.api_key = {"authorization": "Bearer " + credentials.token}
client.Configuration.set_default(config)
k8s_batch_api = client.BatchV1Api()
runner_dict = {
    'apiVersion': 'batch/v1',
    'kind': 'Job',
    'metadata': {
        'name': 'test-1337',
        'namespace': 'nsp-1337',
        'annotations': {
            'cluster-autoscaler.kubernetes.io/safe-to-evict': 'false'
        },
        'labels': {
            'name': 'test-1337'
        },
    },
    'spec': {
        'template': {
            'metadata': {
                'labels': {
                    'is-busy': 'false',
                }
            },
            'spec': {
                'serviceAccountName': 'github-runner',
                'containers': [{
                    'name': 'test-1337',
                    'image': 'europe-west1-docker.pkg.dev/proj-1337/image-repo/image:stable',
                    'imagePullPolicy': 'Always',
                }],
                'restartPolicy': 'Never',
            },
        },
        'backoffLimit': 3,
        'ttlSecondsAfterFinished': 5,
    }
}
k8s_batch_api.create_namespaced_job(namespace=k8s_namespace, body=runner_dict)

Hope this helps.

Cheers,
Tim

@jicuss

Glad to see Option A landed in 36.0.1 — load_kube_config() users (the AKS case in this issue) are now unblocked, thanks.

The recent reports from @jicuss and @bahag-klickst are exactly the case Option B was meant to cover:

config.api_key = {"authorization": "Bearer " + credentials.token}

This is a perfectly reasonable pattern used by GCP/GKE and EKS examples in the wild, and was working in v35.0.0 and earlier. Since auth_settings() only checks 'BearerToken', these users still get the silent token-drop on 36.0.1.

Would you (@aojea / @yliaog) be open to extending Configuration.auth_settings() to fall back to 'authorization' for backward compatibility? It's a 3-line generated-code change and would close the regression for all users, not just the load_kube_config() path. Suggested patch:

def auth_settings(self):
    auth = {}
    bearer_key = (
        'BearerToken' if 'BearerToken' in self.api_key
        else 'authorization' if 'authorization' in self.api_key
        else None
    )
    if bearer_key:
        auth['BearerToken'] = {
            'type': 'api_key',
            'in': 'header',
            'key': 'authorization',
            'value': self.get_api_key_with_prefix(bearer_key),
        }
    return auth

Alternatively, a release note flagging the change in api_key key name from 'authorization' → 'BearerToken' as a breaking change for direct consumers would also help — users could then update their own code consciously rather than silently lose auth.

cc: @aojea @yliaog

We have the following release note to document the breaking change. Given users may take time to adapt their code, I think it is good to provide backward compatibility for certain period of time. Could someone help to create a PR?

https://github.com/kubernetes-client/python/blob/release-36.0/CHANGELOG.md#breaking-change-from-upgrading-openapi-generator-to-v660

Configuration auth uses 'BearerToken' instead of 'authorization' in api_key.

https://github.com/kubernetes-client/python/pull/2585/commits initially had written both authorization and BearerToken, I think if we were to provide the backward compatibility for a certain period of time, it is better to implement that in the handwritten code like what the PR did, than in the auto generated code with a patch.

any thoughts?

@yliaog

Thanks @yliaog that makes sense, the handwritten layer is a much cleaner place for transition logic than patching generated code.

One thing I want to make sure we're on the same page about: the dual-key write in kube_config.py (per PR #2585) only catches users who go through load_kube_config(). The GKE/EKS reports above (e.g. @bahag-klickst's example) construct Configuration directly and set api_key = {"authorization": "Bearer ..."} themselves ,that path never touches kube_config.py, so a fix there won't unblock them.

Is the intended scope of the transition shim only load_kube_config() users, with the direct-api_key users being expected to migrate per the release note? Or do we want to cover both? If the latter, I don't see a handwritten chokepoint in the request path, the migration would need to happen on read inside Configuration.auth_settings() (auto-generated) or via a Configuration subclass somewhere.

Whichever scope you settle on, happy to test against the AKS reproducer here, and would assume the GKE/EKS folks could test their setups too.

cc: @bahag-klickst @jicuss

Good point, in that case, I'm open to carry a patch on Configuration.auth_settings() for certain period of time. If someone could help create a PR, that would be great!

@GK-07

@GK-07
Sorry for the late reply, of course I would be willing to test a patch on our GKE instance.
In the meantime, I have moved to using load_kube_config_from_dict as follows, which seems to work with v36.0.1

def init_k8s_config(gke_resource_path: str, gke_cluster: str):
    credentials, project_id = google.auth.default()
    auth_req = google.auth.transport.requests.Request()
    credentials.refresh(auth_req)
    cluster_manager_client = ClusterManagerClient(credentials=credentials)
    cluster = f"projects/{project_id}/{gke_resource_path}/clusters/{gke_cluster}"
    cluster = cluster_manager_client.get_cluster(name=cluster)
    # set kubernetes config
    kube_config = {
        "apiVersion": "v1",
        "kind": "Config",
        "clusters": [
            {
                "name": gke_cluster,
                "cluster": {
                    "server": f"https://{cluster.endpoint}",
                    "certificate-authority-data": cluster.master_auth.cluster_ca_certificate,
                }
            }
        ],
        "contexts": [
            {
                "name": gke_cluster,
                "context": {
                    "cluster": gke_cluster,
                    "user": gke_cluster,
                }
            }
        ],
        "current-context": gke_cluster,
        "users": [
            {
                "name": gke_cluster,
                "user": {
                    "token": credentials.token,
                }
            }
        ]
    }
    local_config = client.Configuration()
    config.load_kube_config_from_dict(kube_config, client_configuration=local_config)
    scoped_api_client = client.ApiClient(configuration=local_config)
    return project_id, scoped_api_client

https://github.com/kubernetes-client/python/releases/tag/v36.0.2 has been released, please check if it helps fix the problem.

@GK-07

@GK-07 / @yliaog
I have spun up a new dev environment on our end and deployed the code base that did not yet have the refactorings in it I have mentioned here (#2595 (comment)), so still using the old auth mechanism of constructing Configuration directly.

The error is now gone, so the fix 36.0.2 should help other people relying on this as well, at least on GKE.

Thanks for the swift implementation and fixing.

@bahag-klickst

thanks @bahag-klickst for confirming the fix

build: publish dependency ranges, assert the FoundationDB series

fix(AAP-93615): send Bearer prefix on K8s transport auth for kubernet…

feat(AAP-93615): Kubernetes worker manager for Execution Plane

feat(AAP-93615): cold-start Kubernetes worker manager for Execution P…

Navigation Menu

Repository navigation

v36.0.0 regression: exec WebSocket returns 401 Unauthorized on AKS (works in v35.0.0) #2595

What happened?

What did you expect to happen?

How can we reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

Root cause (suspected)

Workaround

Kubernetes client version

Server version (AKS)

OS

Python version

Python client version

Installation method

Related

Re: comma vs comma-space formatting

Re: AKS Kubernetes version

So the situation is

Suggested fix

Option A (preferred): update `kube_config.py` to match v36's generated key name

Option B: make `auth_settings()` accept the legacy key too (backward-compat)

Recommended test

Workaround for users on v36 in the meantime

6 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

v36.0.0 regression: exec WebSocket returns 401 Unauthorized on AKS (works in v35.0.0) #2595

Description

What happened?

What did you expect to happen?

How can we reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

Root cause (suspected)

Workaround

Kubernetes client version

Server version (AKS)

OS

Python version

Python client version

Installation method

Related

Activity

aojea commented on May 25, 2026

GK-07 commented on May 25, 2026

Re: comma vs comma-space formatting

Re: AKS Kubernetes version

So the situation is

faust64 commented on May 25, 2026

GK-07 commented on May 26, 2026

Suggested fix

Option A (preferred): update kube_config.py to match v36's generated key name

Option B: make auth_settings() accept the legacy key too (backward-compat)

Recommended test

Workaround for users on v36 in the meantime

jicuss commented on May 26, 2026

bahag-klickst commented on May 27, 2026

aojea commented on May 27, 2026

bahag-klickst commented on May 27, 2026

GK-07 commented on May 27, 2026

yliaog commented on May 27, 2026

yliaog commented on May 27, 2026

GK-07 commented on May 28, 2026

yliaog commented on May 28, 2026

bahag-klickst commented on May 29, 2026

6 remaining items

yliaog commented on Jun 1, 2026

bahag-klickst commented on Jun 1, 2026

yliaog commented on Jun 1, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Option A (preferred): update `kube_config.py` to match v36's generated key name

Option B: make `auth_settings()` accept the legacy key too (backward-compat)