What's the best way to give on-call engineers temporary access to production Kubernetes clusters?

•

Product Marketing Engineer

•

Standing kubeconfigs are the single biggest unforced error in Kubernetes access control. Grant access when the page fires, scope it to the engineer's identity, and let it expire. See the Twingate Privileged Access for Kubernetes docs for the mechanics.

In April 2026, Palo Alto Networks' Unit 42 published its latest cloud threat research. One line stood out: Kubernetes attack attempts were up 282% year over year. Deeper in the report was a case study on a North Korean group that didn't exploit a CVE or breach the control plane. They phished a developer, waited until that developer authenticated into a production cluster, and used the active privileged session to deploy a malicious pod.

The cluster wasn't compromised. The developer was. And because the developer held standing cluster-admin, the attackers inherited everything the developer had.

Most on-call rotations are set up exactly this way. A shared kubeconfig lives in 1Password. A break-glass service account has permanent cluster-admin. Someone added a wildcard RoleBinding during an incident in 2023 and nobody took it out. When the page fires at 3am, the engineer grabs the credential and goes. The credential doesn't care who they are, and it doesn't go away when the incident ends.

This is the thing to fix.

The actual problem with standing credentials

A production Kubernetes cluster is a high-value target for the same reason it's useful: it runs your workloads, holds your secrets, and talks to your databases. If an attacker gets exec into a pod with a mounted service account token, they can usually pivot to the control plane, the cloud provider, or both.

Standing credentials multiply this risk in three ways:

  1. They outlive the reason they were granted. An engineer rotates off on-call. The kubeconfig doesn't.

  2. They detach access from identity. A shared credential tells your audit log what happened but not who did it.

  3. They're a phishing payload. Unit 42's North Korea case worked because the credential was active and persistent. A session that expires in 30 minutes with no refresh is a much less interesting target.

The counterargument is usually "but we need to move fast during an incident." That's true. It's also the reason just-in-time access exists. The goal isn't to make access slower. It's to make the credential disappear after the incident is over.

What good on-call Kubernetes access looks like

Four properties, in order of importance:

Property

What it means in practice

Identity-bound

Access is tied to the engineer's SSO identity, not a shared account or service token. RBAC decisions reference the human.

Just-in-time

No access until an incident or approved request. Engineer requests, request is approved (automatically or by a peer), access is granted.

Time-bound

Access expires on its own. No cleanup script, no manual revocation, no "I'll remove it tomorrow."

Recorded

Every session produces a replayable log that lives on infrastructure you control.

If your current setup fails any of these, you're trusting the credential more than you're trusting the person holding it. That's the inversion the Unit 42 report keeps showing up in.

Why Kubernetes RBAC alone isn't enough

Kubernetes RBAC is the right primitive for the authorization decision: a RoleBinding against the engineer's identity, scoped to the namespaces or verbs they need. The gap is everything else around RBAC:

  • How does the engineer's SSO identity get to the API server? The default answer is OIDC configured on the kube-apiserver, which works but requires every engineer to run kubectl with a configured OIDC plugin and refresh tokens. In practice, most teams fall back to static kubeconfigs because the OIDC path is fragile.

  • How is the cluster reachable? If the API server is exposed to the internet (even behind an IP allowlist), you've widened the attack surface. If it's not, your engineers need a VPN, which brings its own standing-access problem.

  • How do you record a kubectl exec session? Native Kubernetes audit logs capture the API call. They don't capture what happened inside the shell.

  • How does access expire? RoleBindings don't have a TTL. You have to build the expiration yourself.

Each of these can be solved individually with custom tooling. Most teams don't. That's how you end up with the shared kubeconfig in 1Password.

How Twingate handles this

Twingate Privileged Access for Kubernetes sits between the engineer's kubectl and the cluster API, passing the engineer's SSO identity through the Twingate Gateway so that Kubernetes RBAC applies per person without issuing a separate Kubernetes credential to each user.

The architecture, briefly:

  • The cluster API server stays private. No public endpoint, no inbound firewall rules. A Twingate Connector runs inside the cluster (or on the network with access to it) and makes outbound-only connections.

  • The engineer authenticates to Twingate via your IdP (Okta, Entra, Google, etc.).

  • When the engineer runs kubectl, the request routes through the Twingate Client to the Connector, which proxies to the API server with the engineer's identity attached.

  • RBAC on the cluster references the engineer's identity directly. A RoleBinding for alice@company.com applies when Alice runs commands, not when Bob does.

  • Sessions are recorded. The logs land in storage you own (your S3 bucket, your SIEM) not Twingate's.

Combine this with just-in-time access requests and the credential problem mostly goes away. Alice has no standing access to the production cluster. She gets paged. She requests access through Slack or the Twingate Admin Console. The request is auto-approved based on her being on the on-call rotation (or peer-approved if you want a human in the loop). She gets access for the duration of the incident, scoped to the namespaces she needs. When the TTL expires, the RoleBinding is gone.

If she's phished during that window, the attacker inherits an access grant that evaporates in 30 minutes and produces a recorded session the SOC can watch back.

A concrete pattern

A workable on-call setup looks something like this:

  1. Baseline RBAC: On-call engineers have view access to production namespaces and nothing else. This covers 90% of what you need during triage: reading logs, describing pods, checking events.

  2. Incident RBAC: A separate incident-responder ClusterRole with exec, delete pod, rollout, and whatever else your runbooks need. No binding exists by default.

  3. JIT grant on page: When PagerDuty fires, the acknowledging engineer requests the incident role through Twingate. Auto-approve if they're the current on-call. Grant for 60 minutes with option to extend.

  4. Expiration is automatic: The RoleBinding goes away. The Twingate access policy for the engineer reverts.

  5. Session recording is on by default: Every exec session, every port-forward, logged to your bucket with the engineer's identity attached.

The friction is one Slack command when the page fires. The payoff is that your standing production access is a view role on a cluster that isn't reachable from the public internet.

What this changes about your blast radius

Think about the Unit 42 case again. The attacker's play was to use an active privileged session. In the setup above:

  • There's no persistent privileged session to hijack. The engineer only holds elevated access during an active incident.

  • The window is bounded. 60 minutes, not 60 days.

  • The cluster API isn't on the internet. An attacker who steals the engineer's laptop still has to defeat device posture and SSO before they can reach anything.

  • The session is recorded. If something malicious did happen, you have the tape.

None of this makes Kubernetes magically secure. A determined attacker who compromises an engineer during an active incident window can still do damage. But you've turned a persistent, high-value credential into a brief, scoped, auditable one. That's the actual meaning of least privilege for on-call: not fewer permissions in the abstract, but fewer permissions at rest.

What to do this week

If you're running shared kubeconfigs or permanent cluster-admin for on-call, the shortest path forward is:

  • Inventory every ClusterRoleBinding with cluster-admin or wildcard verbs. Most will have been added for a specific reason that no longer applies.

  • Identify who actually needs standing write access to production. The answer is usually "no one, continuously."

  • Replace shared credentials with identity-bound access, even if the first step is just OIDC on the API server with short-lived tokens.

  • Add session recording. If you can't replay what happened during an incident, you can't investigate when something looks wrong afterward.

The 282% figure in the Unit 42 report isn't surprising. Kubernetes is where the valuable workloads live, and the standard access patterns were designed for a world where the operator and the attacker were obviously different people. That assumption doesn't hold anymore.

Closing

For the details on how Twingate's Kubernetes support works, including Connector deployment, RBAC passthrough, and session recording configuration, check the Twingate documentation.

New to Twingate? You can use Twingate for free for up to 5 users, request a personalized demo, or reach out to the team over on the Twingate subreddit.

Rapidly implement a modern Zero Trust network that is more secure and maintainable than VPNs.

/

On-call Access

What's the best way to give on-call engineers temporary access to production Kubernetes clusters?

•

Product Marketing Engineer

•

Standing kubeconfigs are the single biggest unforced error in Kubernetes access control. Grant access when the page fires, scope it to the engineer's identity, and let it expire. See the Twingate Privileged Access for Kubernetes docs for the mechanics.

In April 2026, Palo Alto Networks' Unit 42 published its latest cloud threat research. One line stood out: Kubernetes attack attempts were up 282% year over year. Deeper in the report was a case study on a North Korean group that didn't exploit a CVE or breach the control plane. They phished a developer, waited until that developer authenticated into a production cluster, and used the active privileged session to deploy a malicious pod.

The cluster wasn't compromised. The developer was. And because the developer held standing cluster-admin, the attackers inherited everything the developer had.

Most on-call rotations are set up exactly this way. A shared kubeconfig lives in 1Password. A break-glass service account has permanent cluster-admin. Someone added a wildcard RoleBinding during an incident in 2023 and nobody took it out. When the page fires at 3am, the engineer grabs the credential and goes. The credential doesn't care who they are, and it doesn't go away when the incident ends.

This is the thing to fix.

The actual problem with standing credentials

A production Kubernetes cluster is a high-value target for the same reason it's useful: it runs your workloads, holds your secrets, and talks to your databases. If an attacker gets exec into a pod with a mounted service account token, they can usually pivot to the control plane, the cloud provider, or both.

Standing credentials multiply this risk in three ways:

  1. They outlive the reason they were granted. An engineer rotates off on-call. The kubeconfig doesn't.

  2. They detach access from identity. A shared credential tells your audit log what happened but not who did it.

  3. They're a phishing payload. Unit 42's North Korea case worked because the credential was active and persistent. A session that expires in 30 minutes with no refresh is a much less interesting target.

The counterargument is usually "but we need to move fast during an incident." That's true. It's also the reason just-in-time access exists. The goal isn't to make access slower. It's to make the credential disappear after the incident is over.

What good on-call Kubernetes access looks like

Four properties, in order of importance:

Property

What it means in practice

Identity-bound

Access is tied to the engineer's SSO identity, not a shared account or service token. RBAC decisions reference the human.

Just-in-time

No access until an incident or approved request. Engineer requests, request is approved (automatically or by a peer), access is granted.

Time-bound

Access expires on its own. No cleanup script, no manual revocation, no "I'll remove it tomorrow."

Recorded

Every session produces a replayable log that lives on infrastructure you control.

If your current setup fails any of these, you're trusting the credential more than you're trusting the person holding it. That's the inversion the Unit 42 report keeps showing up in.

Why Kubernetes RBAC alone isn't enough

Kubernetes RBAC is the right primitive for the authorization decision: a RoleBinding against the engineer's identity, scoped to the namespaces or verbs they need. The gap is everything else around RBAC:

  • How does the engineer's SSO identity get to the API server? The default answer is OIDC configured on the kube-apiserver, which works but requires every engineer to run kubectl with a configured OIDC plugin and refresh tokens. In practice, most teams fall back to static kubeconfigs because the OIDC path is fragile.

  • How is the cluster reachable? If the API server is exposed to the internet (even behind an IP allowlist), you've widened the attack surface. If it's not, your engineers need a VPN, which brings its own standing-access problem.

  • How do you record a kubectl exec session? Native Kubernetes audit logs capture the API call. They don't capture what happened inside the shell.

  • How does access expire? RoleBindings don't have a TTL. You have to build the expiration yourself.

Each of these can be solved individually with custom tooling. Most teams don't. That's how you end up with the shared kubeconfig in 1Password.

How Twingate handles this

Twingate Privileged Access for Kubernetes sits between the engineer's kubectl and the cluster API, passing the engineer's SSO identity through the Twingate Gateway so that Kubernetes RBAC applies per person without issuing a separate Kubernetes credential to each user.

The architecture, briefly:

  • The cluster API server stays private. No public endpoint, no inbound firewall rules. A Twingate Connector runs inside the cluster (or on the network with access to it) and makes outbound-only connections.

  • The engineer authenticates to Twingate via your IdP (Okta, Entra, Google, etc.).

  • When the engineer runs kubectl, the request routes through the Twingate Client to the Connector, which proxies to the API server with the engineer's identity attached.

  • RBAC on the cluster references the engineer's identity directly. A RoleBinding for alice@company.com applies when Alice runs commands, not when Bob does.

  • Sessions are recorded. The logs land in storage you own (your S3 bucket, your SIEM) not Twingate's.

Combine this with just-in-time access requests and the credential problem mostly goes away. Alice has no standing access to the production cluster. She gets paged. She requests access through Slack or the Twingate Admin Console. The request is auto-approved based on her being on the on-call rotation (or peer-approved if you want a human in the loop). She gets access for the duration of the incident, scoped to the namespaces she needs. When the TTL expires, the RoleBinding is gone.

If she's phished during that window, the attacker inherits an access grant that evaporates in 30 minutes and produces a recorded session the SOC can watch back.

A concrete pattern

A workable on-call setup looks something like this:

  1. Baseline RBAC: On-call engineers have view access to production namespaces and nothing else. This covers 90% of what you need during triage: reading logs, describing pods, checking events.

  2. Incident RBAC: A separate incident-responder ClusterRole with exec, delete pod, rollout, and whatever else your runbooks need. No binding exists by default.

  3. JIT grant on page: When PagerDuty fires, the acknowledging engineer requests the incident role through Twingate. Auto-approve if they're the current on-call. Grant for 60 minutes with option to extend.

  4. Expiration is automatic: The RoleBinding goes away. The Twingate access policy for the engineer reverts.

  5. Session recording is on by default: Every exec session, every port-forward, logged to your bucket with the engineer's identity attached.

The friction is one Slack command when the page fires. The payoff is that your standing production access is a view role on a cluster that isn't reachable from the public internet.

What this changes about your blast radius

Think about the Unit 42 case again. The attacker's play was to use an active privileged session. In the setup above:

  • There's no persistent privileged session to hijack. The engineer only holds elevated access during an active incident.

  • The window is bounded. 60 minutes, not 60 days.

  • The cluster API isn't on the internet. An attacker who steals the engineer's laptop still has to defeat device posture and SSO before they can reach anything.

  • The session is recorded. If something malicious did happen, you have the tape.

None of this makes Kubernetes magically secure. A determined attacker who compromises an engineer during an active incident window can still do damage. But you've turned a persistent, high-value credential into a brief, scoped, auditable one. That's the actual meaning of least privilege for on-call: not fewer permissions in the abstract, but fewer permissions at rest.

What to do this week

If you're running shared kubeconfigs or permanent cluster-admin for on-call, the shortest path forward is:

  • Inventory every ClusterRoleBinding with cluster-admin or wildcard verbs. Most will have been added for a specific reason that no longer applies.

  • Identify who actually needs standing write access to production. The answer is usually "no one, continuously."

  • Replace shared credentials with identity-bound access, even if the first step is just OIDC on the API server with short-lived tokens.

  • Add session recording. If you can't replay what happened during an incident, you can't investigate when something looks wrong afterward.

The 282% figure in the Unit 42 report isn't surprising. Kubernetes is where the valuable workloads live, and the standard access patterns were designed for a world where the operator and the attacker were obviously different people. That assumption doesn't hold anymore.

Closing

For the details on how Twingate's Kubernetes support works, including Connector deployment, RBAC passthrough, and session recording configuration, check the Twingate documentation.

New to Twingate? You can use Twingate for free for up to 5 users, request a personalized demo, or reach out to the team over on the Twingate subreddit.

Rapidly implement a modern Zero Trust network that is more secure and maintainable than VPNs.

What's the best way to give on-call engineers temporary access to production Kubernetes clusters?

•

Product Marketing Engineer

•

Standing kubeconfigs are the single biggest unforced error in Kubernetes access control. Grant access when the page fires, scope it to the engineer's identity, and let it expire. See the Twingate Privileged Access for Kubernetes docs for the mechanics.

In April 2026, Palo Alto Networks' Unit 42 published its latest cloud threat research. One line stood out: Kubernetes attack attempts were up 282% year over year. Deeper in the report was a case study on a North Korean group that didn't exploit a CVE or breach the control plane. They phished a developer, waited until that developer authenticated into a production cluster, and used the active privileged session to deploy a malicious pod.

The cluster wasn't compromised. The developer was. And because the developer held standing cluster-admin, the attackers inherited everything the developer had.

Most on-call rotations are set up exactly this way. A shared kubeconfig lives in 1Password. A break-glass service account has permanent cluster-admin. Someone added a wildcard RoleBinding during an incident in 2023 and nobody took it out. When the page fires at 3am, the engineer grabs the credential and goes. The credential doesn't care who they are, and it doesn't go away when the incident ends.

This is the thing to fix.

The actual problem with standing credentials

A production Kubernetes cluster is a high-value target for the same reason it's useful: it runs your workloads, holds your secrets, and talks to your databases. If an attacker gets exec into a pod with a mounted service account token, they can usually pivot to the control plane, the cloud provider, or both.

Standing credentials multiply this risk in three ways:

  1. They outlive the reason they were granted. An engineer rotates off on-call. The kubeconfig doesn't.

  2. They detach access from identity. A shared credential tells your audit log what happened but not who did it.

  3. They're a phishing payload. Unit 42's North Korea case worked because the credential was active and persistent. A session that expires in 30 minutes with no refresh is a much less interesting target.

The counterargument is usually "but we need to move fast during an incident." That's true. It's also the reason just-in-time access exists. The goal isn't to make access slower. It's to make the credential disappear after the incident is over.

What good on-call Kubernetes access looks like

Four properties, in order of importance:

Property

What it means in practice

Identity-bound

Access is tied to the engineer's SSO identity, not a shared account or service token. RBAC decisions reference the human.

Just-in-time

No access until an incident or approved request. Engineer requests, request is approved (automatically or by a peer), access is granted.

Time-bound

Access expires on its own. No cleanup script, no manual revocation, no "I'll remove it tomorrow."

Recorded

Every session produces a replayable log that lives on infrastructure you control.

If your current setup fails any of these, you're trusting the credential more than you're trusting the person holding it. That's the inversion the Unit 42 report keeps showing up in.

Why Kubernetes RBAC alone isn't enough

Kubernetes RBAC is the right primitive for the authorization decision: a RoleBinding against the engineer's identity, scoped to the namespaces or verbs they need. The gap is everything else around RBAC:

  • How does the engineer's SSO identity get to the API server? The default answer is OIDC configured on the kube-apiserver, which works but requires every engineer to run kubectl with a configured OIDC plugin and refresh tokens. In practice, most teams fall back to static kubeconfigs because the OIDC path is fragile.

  • How is the cluster reachable? If the API server is exposed to the internet (even behind an IP allowlist), you've widened the attack surface. If it's not, your engineers need a VPN, which brings its own standing-access problem.

  • How do you record a kubectl exec session? Native Kubernetes audit logs capture the API call. They don't capture what happened inside the shell.

  • How does access expire? RoleBindings don't have a TTL. You have to build the expiration yourself.

Each of these can be solved individually with custom tooling. Most teams don't. That's how you end up with the shared kubeconfig in 1Password.

How Twingate handles this

Twingate Privileged Access for Kubernetes sits between the engineer's kubectl and the cluster API, passing the engineer's SSO identity through the Twingate Gateway so that Kubernetes RBAC applies per person without issuing a separate Kubernetes credential to each user.

The architecture, briefly:

  • The cluster API server stays private. No public endpoint, no inbound firewall rules. A Twingate Connector runs inside the cluster (or on the network with access to it) and makes outbound-only connections.

  • The engineer authenticates to Twingate via your IdP (Okta, Entra, Google, etc.).

  • When the engineer runs kubectl, the request routes through the Twingate Client to the Connector, which proxies to the API server with the engineer's identity attached.

  • RBAC on the cluster references the engineer's identity directly. A RoleBinding for alice@company.com applies when Alice runs commands, not when Bob does.

  • Sessions are recorded. The logs land in storage you own (your S3 bucket, your SIEM) not Twingate's.

Combine this with just-in-time access requests and the credential problem mostly goes away. Alice has no standing access to the production cluster. She gets paged. She requests access through Slack or the Twingate Admin Console. The request is auto-approved based on her being on the on-call rotation (or peer-approved if you want a human in the loop). She gets access for the duration of the incident, scoped to the namespaces she needs. When the TTL expires, the RoleBinding is gone.

If she's phished during that window, the attacker inherits an access grant that evaporates in 30 minutes and produces a recorded session the SOC can watch back.

A concrete pattern

A workable on-call setup looks something like this:

  1. Baseline RBAC: On-call engineers have view access to production namespaces and nothing else. This covers 90% of what you need during triage: reading logs, describing pods, checking events.

  2. Incident RBAC: A separate incident-responder ClusterRole with exec, delete pod, rollout, and whatever else your runbooks need. No binding exists by default.

  3. JIT grant on page: When PagerDuty fires, the acknowledging engineer requests the incident role through Twingate. Auto-approve if they're the current on-call. Grant for 60 minutes with option to extend.

  4. Expiration is automatic: The RoleBinding goes away. The Twingate access policy for the engineer reverts.

  5. Session recording is on by default: Every exec session, every port-forward, logged to your bucket with the engineer's identity attached.

The friction is one Slack command when the page fires. The payoff is that your standing production access is a view role on a cluster that isn't reachable from the public internet.

What this changes about your blast radius

Think about the Unit 42 case again. The attacker's play was to use an active privileged session. In the setup above:

  • There's no persistent privileged session to hijack. The engineer only holds elevated access during an active incident.

  • The window is bounded. 60 minutes, not 60 days.

  • The cluster API isn't on the internet. An attacker who steals the engineer's laptop still has to defeat device posture and SSO before they can reach anything.

  • The session is recorded. If something malicious did happen, you have the tape.

None of this makes Kubernetes magically secure. A determined attacker who compromises an engineer during an active incident window can still do damage. But you've turned a persistent, high-value credential into a brief, scoped, auditable one. That's the actual meaning of least privilege for on-call: not fewer permissions in the abstract, but fewer permissions at rest.

What to do this week

If you're running shared kubeconfigs or permanent cluster-admin for on-call, the shortest path forward is:

  • Inventory every ClusterRoleBinding with cluster-admin or wildcard verbs. Most will have been added for a specific reason that no longer applies.

  • Identify who actually needs standing write access to production. The answer is usually "no one, continuously."

  • Replace shared credentials with identity-bound access, even if the first step is just OIDC on the API server with short-lived tokens.

  • Add session recording. If you can't replay what happened during an incident, you can't investigate when something looks wrong afterward.

The 282% figure in the Unit 42 report isn't surprising. Kubernetes is where the valuable workloads live, and the standard access patterns were designed for a world where the operator and the attacker were obviously different people. That assumption doesn't hold anymore.

Closing

For the details on how Twingate's Kubernetes support works, including Connector deployment, RBAC passthrough, and session recording configuration, check the Twingate documentation.

New to Twingate? You can use Twingate for free for up to 5 users, request a personalized demo, or reach out to the team over on the Twingate subreddit.