internal server error from secret-operator pod trying to convert secret classes

[maxgruber19]

Upgrading from 25.7.0 to 25.11.0 leads to follwing errors in the secret operator pod. also currently its not possible to start any stackable pod because no secret can be provided

2026-01-13T12:20:04.314195Z ERROR kube_client::client::builder: failed with status 500 Internal Server Error
2026-01-13T12:20:04.314283Z ERROR stackable_operator::logging::controller: Failed to reconcile object controller.name="truststore.secrets.stackable.tech" error=event queue error error.sources=[failed to per
form initial object list: ApiError: conversion webhook for secrets.stackable.tech/v1alpha1, Kind=SecretClass failed: Post "https://secret-operator.stackable.svc:8443/convert/secretclasses.secrets.stackable.
tech?timeout=30s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers):  (ErrorResponse { status: "Failure", message: "conversion webhook for secrets.sta
ckable.tech/v1alpha1, Kind=SecretClass failed: Post \"https://secret-operator.stackable.svc:8443/convert/secretclasses.secrets.stackable.tech?timeout=30s\": net/http: request canceled while waiting for conn
ection (Client.Timeout exceeded while awaiting headers)", reason: "", code: 500 }), ApiError: conversion webhook for secrets.stackable.tech/v1alpha1, Kind=SecretClass failed: Post "https://secret-operator.s
tackable.svc:8443/convert/secretclasses.secrets.stackable.tech?timeout=30s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers):  (ErrorResponse { statu
s: "Failure", message: "conversion webhook for secrets.stackable.tech/v1alpha1, Kind=SecretClass failed: Post \"https://secret-operator.stackable.svc:8443/convert/secretclasses.secrets.stackable.tech?timeou
t=30s\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)", reason: "", code: 500 }), conversion webhook for secrets.stackable.tech/v1alpha1, Kind=Sec
retClass failed: Post "https://secret-operator.stackable.svc:8443/convert/secretclasses.secrets.stackable.tech?timeout=30s": net/http: request canceled while waiting for connection (Client.Timeout exceeded
while awaiting headers): ]

Any idea what I'm missing out? Could be some networking thing but i already checked on that and didn't find something special

[Techassi]

Hey, a few quick questions to narrow this down:

• How was the upgrade performed? I assume via Helm? Can you provide the exact command used?
• Does the operator start up correctly, or are there any errors logged regarding the conversion webhook?
• Does the secret-operator Service exist with the correct ports?

[maxgruber19]

Hi @Techassi, upgrades are done via argo cd. We upgrade the chart version (25.7.0 to 25.11.0) and let it run a sync then. worked like a charm for all prior releases.

The operator ist running, it's just logging the lines from above, nothing else

Service "secret-operator" is there with port 8443

[maxgruber19]

What i noticed as well: I cannot list secretclasses anymore, it seems like k9s and kubectl timeout when trying to access them- Probably all that comes from the resource versioning

[Techassi]

worked like a charm for all prior releases.

There were a few breaking changes in this release for secret-operator, see the section for the secret-operator in the release notes.

The operator ist running, it's just logging the lines from above, nothing else

There should now be two pods running: One which is the actual operator (which contains the conversion webhook) and one pod which contains the CSI related containers.

To get more detailed log output, could you deploy the secret operator with a higher verbosity level? See the console log documentation.

[maxgruber19]

I have seen these breaking changes, but was not aware of any differences in deployment of that release

Yes i have the secret-operator pod and multiple secret-operator-csi-node-driver pods running at the same time.

I will try the log level configuration

[maxgruber19]

2026-01-13T14:13:14.249303Z ERROR stackable_operator::logging::controller: Failed to reconcile object controller.name="truststore.secrets.stackable.tech" error=event queue error error.sources=[error returned by apiserver during watch: too old resource version: 507719771 (507719830): Expired, too old resource version: 507719771 (507719830): Expired]
2026-01-13T14:13:15.050695Z DEBUG kube_runtime::utils::stream_backoff: Backoff complete, waking up deadline=Instant { tv_sec: 887779, tv_nsec: 504562808 }
2026-01-13T14:13:15.054262Z DEBUG tower::buffer::worker: service.ready=true processing request
2026-01-13T14:13:15.054492Z DEBUG HTTP{http.method=GET http.url=https://172.16.0.1/apis/secrets.stackable.tech/v1alpha1/truststores?&limit=500 otel.name="list" otel.kind="client"}: kube_client::client::builder: requesting
2026-01-13T14:13:15.054525Z DEBUG HTTP{http.method=GET http.url=https://172.16.0.1/apis/secrets.stackable.tech/v1alpha1/truststores?&limit=500 otel.name="list" otel.kind="client"}: hyper_util::client::legacy::pool: reuse idle connection for ("https", 172.16.0.1)
2026-01-13T14:13:15.063252Z DEBUG HTTP{http.method=GET http.url=https://172.16.0.1/apis/secrets.stackable.tech/v1alpha1/truststores?&limit=500 otel.name="list" otel.kind="client"}: hyper_util::client::legacy::pool: pooling idle connection for ("https", 172.16.0.1)
2026-01-13T14:13:15.069848Z DEBUG tower::buffer::worker: service.ready=true processing request
2026-01-13T14:13:15.070207Z DEBUG HTTP{http.method=GET http.url=https://172.16.0.1/apis/secrets.stackable.tech/v1alpha1/truststores?&watch=true&timeoutSeconds=290&allowWatchBookmarks=true&resourceVersion=507719867 otel.name="watch" otel.kind="client"}: kube_client::client::builder: requesting
2026-01-13T14:13:15.070238Z DEBUG HTTP{http.method=GET http.url=https://172.16.0.1/apis/secrets.stackable.tech/v1alpha1/truststores?&watch=true&timeoutSeconds=290&allowWatchBookmarks=true&resourceVersion=507719867 otel.name="watch" otel.kind="client"}: hyper_util::client::legacy::pool: reuse idle connection for ("https", 172.16.0.1)
2026-01-13T14:13:15.081290Z DEBUG HTTP{http.method=GET http.url=https://172.16.0.1/api/v1/configmaps?&limit=500&continue=eyJ2IjoibWV0YS5rOHMuaW8vdjEiLCJydiI6NTA3NzE5NzczLCJzdGFydCI6Ii9tZXNoLXByb2R1Y3QtcHJvdHJhbnMva3JiNS1jb25mXHUwMDAwIn0 otel.name="list" otel.kind="client" http.status_code=200}: hyper_util::client::legacy::pool: pooling idle connection for ("https", 172.16.0.1)
2026-01-13T14:13:15.149265Z DEBUG tower::buffer::worker: service.ready=true processing request
2026-01-13T14:13:15.149354Z DEBUG HTTP{http.method=GET http.url=https://172.16.0.1/api/v1/configmaps?&limit=500&continue=eyJ2IjoibWV0YS5rOHMuaW8vdjEiLCJydiI6NTA3NzE5NzczLCJzdGFydCI6Im1lc2gtcHJvZHVjdC10ZWxlbWF0aWsvc3BhcmstYXBwbGljYXRpb24tYW5vbnltaXphdGlvbi10cC1sb29zZS10YWdzLW1hbmlmZXN0XHUwMDAwIn0 otel.name="list" otel.kind="client"}: kube_client::client::builder: requesting
2026-01-13T14:13:15.149393Z DEBUG HTTP{http.method=GET http.url=https://172.16.0.1/api/v1/configmaps?&limit=500&continue=eyJ2IjoibWV0YS5rOHMuaW8vdjEiLCJydiI6NTA3NzE5NzczLCJzdGFydCI6Im1lc2gtcHJvZHVjdC10ZWxlbWF0aWsvc3BhcmstYXBwbGljYXRpb24tYW5vbnltaXphdGlvbi10cC1sb29zZS10YWdzLW1hbmlmZXN0XHUwMDAwIn0 otel.name="list" otel.kind="client"}: hyper_util::client::legacy::pool: reuse idle connection for ("https", 172.16.0.1)
2026-01-13T14:13:15.246099Z DEBUG HTTP{http.method=GET http.url=https://172.16.0.1/api/v1/configmaps?&limit=500&continue=eyJ2IjoibWV0YS5rOHMuaW8vdjEiLCJydiI6NTA3NzE5NzcxLCJzdGFydCI6Im1lc2gtcHJvZHVjdC10ZWxlbWF0aWsvdHAtY2xlYW4tYW5kLXRyYW5zZm9ybS1zY29yZXMtbWFuaWZlc3QtZHJpdmVyLXBvZC10ZW1wbGF0ZVx1MDAwMCJ9 otel.name="list_metadata" otel.kind="client" http.status_code=200}: hyper_util::client::legacy::pool: pooling idle connection for ("https", 172.16.0.1)
2026-01-13T14:13:15.350203Z DEBUG tower::buffer::worker: service.ready=true processing request
2026-01-13T14:13:15.350286Z DEBUG HTTP{http.method=GET http.url=https://172.16.0.1/api/v1/configmaps?&watch=true&timeoutSeconds=290&allowWatchBookmarks=true&resourceVersion=507719771 otel.name="watch_metadata" otel.kind="client"}: kube_client::client::builder: requesting
2026-01-13T14:13:15.350319Z DEBUG HTTP{http.method=GET http.url=https://172.16.0.1/api/v1/configmaps?&watch=true&timeoutSeconds=290&allowWatchBookmarks=true&resourceVersion=507719771 otel.name="watch_metadata" otel.kind="client"}: hyper_util::client::legacy::pool: reuse idle connection for ("https", 172.16.0.1)
2026-01-13T14:13:15.548037Z DEBUG HTTP{http.method=GET http.url=https://172.16.0.1/api/v1/configmaps?&limit=500&continue=eyJ2IjoibWV0YS5rOHMuaW8vdjEiLCJydiI6NTA3NzE5NzczLCJzdGFydCI6Im1lc2gtcHJvZHVjdC10ZWxlbWF0aWsvc3BhcmstYXBwbGljYXRpb24tYW5vbnltaXphdGlvbi10cC1sb29zZS10YWdzLW1hbmlmZXN0XHUwMDAwIn0 otel.name="list" otel.kind="client" http.status_code=200}: hyper_util::client::legacy::pool: pooling idle connection for ("https", 172.16.0.1)
2026-01-13T14:13:15.649449Z DEBUG tower::buffer::worker: service.ready=true processing request
2026-01-13T14:13:15.649550Z DEBUG HTTP{http.method=GET http.url=https://172.16.0.1/api/v1/configmaps?&limit=500&continue=eyJ2IjoibWV0YS5rOHMuaW8vdjEiLCJydiI6NTA3NzE5NzczLCJzdGFydCI6Im1lc2gtcHJvZHVjdC10ZWxlbWF0aWsvdHAtY2xlYW4tYW5kLXRyYW5zZm9ybS1zY29yZXMtbWFuaWZlc3QtZHJpdmVyLXBvZC10ZW1wbGF0ZVx1MDAwMCJ9 otel.name="list" otel.kind="client"}: kube_client::client::builder: requesting
2026-01-13T14:13:15.650029Z DEBUG HTTP{http.method=GET http.url=https://172.16.0.1/api/v1/configmaps?&limit=500&continue=eyJ2IjoibWV0YS5rOHMuaW8vdjEiLCJydiI6NTA3NzE5NzczLCJzdGFydCI6Im1lc2gtcHJvZHVjdC10ZWxlbWF0aWsvdHAtY2xlYW4tYW5kLXRyYW5zZm9ybS1zY29yZXMtbWFuaWZlc3QtZHJpdmVyLXBvZC10ZW1wbGF0ZVx1MDAwMCJ9 otel.name="list" otel.kind="client"}: hyper_util::client::legacy::pool: reuse idle connection for ("https", 172.16.0.1)
2026-01-13T14:13:16.147442Z DEBUG HTTP{http.method=GET http.url=https://172.16.0.1/api/v1/configmaps?&limit=500&continue=eyJ2IjoibWV0YS5rOHMuaW8vdjEiLCJydiI6NTA3NzE5NzczLCJzdGFydCI6Im1lc2gtcHJvZHVjdC10ZWxlbWF0aWsvdHAtY2xlYW4tYW5kLXRyYW5zZm9ybS1zY29yZXMtbWFuaWZlc3QtZHJpdmVyLXBvZC10ZW1wbGF0ZVx1MDAwMCJ9 otel.name="list" otel.kind="client" http.status_code=200}: hyper_util::client::legacy::pool: pooling idle connection for ("https", 172.16.0.1)
2026-01-13T14:13:16.355091Z DEBUG tower::buffer::worker: service.ready=true processing request
2026-01-13T14:13:16.355390Z DEBUG HTTP{http.method=GET http.url=https://172.16.0.1/api/v1/configmaps?&watch=true&timeoutSeconds=290&allowWatchBookmarks=true&resourceVersion=507719773 otel.name="watch" otel.kind="client"}: kube_client::client::builder: requesting
2026-01-13T14:13:16.355440Z DEBUG HTTP{http.method=GET http.url=https://172.16.0.1/api/v1/configmaps?&watch=true&timeoutSeconds=290&allowWatchBookmarks=true&resourceVersion=507719773 otel.name="watch" otel.kind="client"}: hyper_util::client::legacy::pool: reuse idle connection for ("https", 172.16.0.1)
2026-01-13T14:13:16.649363Z DEBUG HTTP{http.method=GET http.url=https://172.16.0.1/api/v1/secrets?&limit=500 otel.name="list" otel.kind="client" http.status_code=200}: hyper_util::client::legacy::pool: pooling idle connection for ("https", 172.16.0.1)
2026-01-13T14:13:17.048346Z DEBUG tower::buffer::worker: service.ready=true processing request
2026-01-13T14:13:17.048506Z DEBUG HTTP{http.method=GET http.url=https://172.16.0.1/api/v1/secrets?&limit=500&continue=eyJ2IjoibWV0YS5rOHMuaW8vdjEiLCJydiI6NTA3NzE5Nzc2LCJzdGFydCI6Ii9tb25pdG9yaW5nLXByb21ldGhldXMtZ3JhZmFuYS9ncmFmYW5hLXBvc3RncmVzLWRiLWFwcFx1MDAwMCJ9 otel.name="list" otel.kind="client"}: kube_client::client::builder: requesting
2026-01-13T14:13:17.048540Z DEBUG HTTP{http.method=GET http.url=https://172.16.0.1/api/v1/secrets?&limit=500&continue=eyJ2IjoibWV0YS5rOHMuaW8vdjEiLCJydiI6NTA3NzE5Nzc2LCJzdGFydCI6Ii9tb25pdG9yaW5nLXByb21ldGhldXMtZ3JhZmFuYS9ncmFmYW5hLXBvc3RncmVzLWRiLWFwcFx1MDAwMCJ9 otel.name="list" otel.kind="client"}: hyper_util::client::legacy::pool: reuse idle connection for ("https", 172.16.0.1)
2026-01-13T14:13:17.454726Z DEBUG HTTP{http.method=GET http.url=https://172.16.0.1/api/v1/secrets?&limit=500&continue=eyJ2IjoibWV0YS5rOHMuaW8vdjEiLCJydiI6NTA3NzE5Nzc2LCJzdGFydCI6Ii9tb25pdG9yaW5nLXByb21ldGhldXMtZ3JhZmFuYS9ncmFmYW5hLXBvc3RncmVzLWRiLWFwcFx1MDAwMCJ9 otel.name="list" otel.kind="client" http.status_code=200}: hyper_util::client::legacy::pool: pooling idle connection for ("https", 172.16.0.1)
2026-01-13T14:13:17.653590Z DEBUG tower::buffer::worker: service.ready=true processing request
2026-01-13T14:13:17.653758Z DEBUG HTTP{http.method=GET http.url=https://172.16.0.1/api/v1/secrets?&watch=true&timeoutSeconds=290&allowWatchBookmarks=true&resourceVersion=507719776 otel.name="watch" otel.kind="client"}: kube_client::client::builder: requesting
2026-01-13T14:13:17.653793Z DEBUG HTTP{http.method=GET http.url=https://172.16.0.1/api/v1/secrets?&watch=true&timeoutSeconds=290&allowWatchBookmarks=true&resourceVersion=507719776 otel.name="watch" otel.kind="client"}: hyper_util::client::legacy::pool: reuse idle connection for ("https", 172.16.0.1)
2026-01-13T14:13:43.057919Z DEBUG HTTP{http.method=GET http.url=https://172.16.0.1/apis/secrets.stackable.tech/v1alpha2/secretclasses?&limit=500 otel.name="list" otel.kind="client"}: hyper_util::client::legacy::pool: pooling idle connection for ("https", 172.16.0.1)
2026-01-13T14:13:43.058291Z ERROR HTTP{http.method=GET http.url=https://172.16.0.1/apis/secrets.stackable.tech/v1alpha2/secretclasses?&limit=500 otel.name="list" otel.kind="client" http.status_code=500 otel.status_code="ERROR" otel.status_code="ERROR" http.status_code=500}: kube_client::client::builder: failed with status 500 Internal Server Error
2026-01-13T14:13:43.058344Z DEBUG kube_client::client: Unsuccessful: ErrorResponse { status: "Failure", message: "conversion webhook for secrets.stackable.tech/v1alpha1, Kind=SecretClass failed: Post \"https://secret-operator.stackable.svc:8443/convert/secretclasses.secrets.stackable.tech?timeout=30s\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", reason: "", code: 500 }
2026-01-13T14:13:43.058365Z DEBUG kube_runtime::watcher: watch list error: Api(ErrorResponse { status: "Failure", message: "conversion webhook for secrets.stackable.tech/v1alpha1, Kind=SecretClass failed: Post \"https://secret-operator.stackable.svc:8443/convert/secretclasses.secrets.stackable.tech?timeout=30s\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", reason: "", code: 500 })
2026-01-13T14:13:43.058381Z DEBUG kube_runtime::utils::stream_backoff: Error received, backing off deadline=Instant { tv_sec: 887809, tv_nsec: 113682642 } duration=1.600000024s

[maxgruber19]

It's a networking thing, so it seems. I'll get back in touch with you tomorrow providing some more details