Just-in-Time Access

Most platform tools I have worked with had the same access model at some point. A few people had permanent admin rights, and scripts and CI jobs used long-lived personal API keys. Both are convenient, and both are risky. An attacker can target a permanent admin account on any day, not only on the day someone needs it. A personal API key stays valid after the reason for it is gone, and usually after the person changes role.

My view now is that no human should hold admin rights by default, and no automation should use a human's key.

The model

Least privilege means each user or process gets the minimum access needed for its task. Just-in-time (JIT) access adds a time limit. You get the elevated role only for the task, and it expires automatically.

The setup I would aim for:

1. Every human is a non-admin by default. Admin roles are eligible, not active.
2. Elevation is time-boxed. A request names the role, the scope, a justification and a duration (for example one or four hours). The role is removed automatically when the time ends.
3. The owning team approves. The team that owns the system approves requests for it, not a central queue that does not know the context. The requester cannot approve their own request.
4. Every grant is logged. Record who requested, who approved, the justification, the scope and the start and end time. Keep those logs where the requester cannot edit them.
5. No personal tokens for automation. CI jobs and services authenticate with OpenID Connect (OIDC) or workload identity. Both issue short-lived credentials to the workload itself. No long-lived key is stored in a CI secret, so there is no key to leak or rotate.

Cloud providers sell this model as a product. Their features are a useful reference: Microsoft Entra Privileged Identity Management (eligible roles, time-bound activation, approval, audit history) and Google Cloud Privileged Access Manager (entitlements with a maximum grant duration, optional approvers, automatic revocation and Cloud Audit Logs). The AWS IAM best practices say the same for humans and for workloads: use temporary credentials.

Lessons

1. Keep the JIT tool off the incident critical path

During an incident, the on-call engineer needs access in seconds. The JIT tool can be slow or down. It can also depend on the system that is failing, for example the identity provider, the chat tool, or the cluster you are trying to fix. In these cases the JIT tool becomes part of the outage.

What to do:

1. Map what the JIT tool depends on, and list the incidents in which those dependencies would also be down.
2. Create a break-glass path for those cases. One option is a small number of emergency accounts whose credentials are sealed and stored offline. Another is a pre-approved on-call role that activates without approval but alerts the owning team.
3. Alert on every break-glass use, and review each use within a working day.
4. Test the break-glass path on a schedule, for example every quarter. Use the same person who would use it during an incident. A break-glass path that is not tested often fails when you need it.

2. Verify the identity behind chat approvals

Approving a request with a button in a chat tool is convenient. A button click can only be trusted as much as the link between the chat account and the person's real identity.

What to do:

1. Verify that the approval request came from the chat platform, for example by checking Slack request signatures, before acting on it.
2. Map the chat user ID to an identity in your identity provider, and check group membership there. Do not use the display name, because users can change it.
3. Reject approvals where the approver is the requester, or is not in the owning team at the time of approval.
4. For the most sensitive roles, require the approver to confirm in the JIT tool itself, behind single sign-on and multi-factor authentication, instead of in chat.

3. Define what "admin" means for each system

"Admin" means something different in every tool. In one system it lets you change settings. In another it lets you read every secret, impersonate users or delete audit logs. If you do not write this down, you cannot check whether "remove all admins" is done.

What to do:

1. For each system, list the permissions that count as privileged: managing users and roles, reading secrets, changing audit settings, deleting data, and editing deployment processes for production.
2. Create smaller roles for the common tasks, for example "restart a workload" or "approve a production deployment". Then most requests do not need full admin.
3. Some tools require an owner account. For those, keep a small number of non-human owner accounts. Store their credentials with your break-glass credentials, and alert on their use.

Checklist

• Every human admin assignment is converted to an eligible, time-boxed role.
• The owning team approves requests, and self-approval is blocked.
• Grant logs include requester, approver, justification, scope and times, and are stored outside the requester's control.
• Personal API keys used by automation are replaced with OIDC or workload identity, then revoked.
• Remaining long-lived credentials are listed with an owner and a removal date.
• The JIT tool's dependencies are mapped, and a break-glass path exists for each failure case.
• Break-glass use alerts the owning team and is tested on a schedule.
• Chat approvals are verified against the identity provider.
• Each system has a written definition of its privileged permissions.

References

1. Least privilegehttps://csrc.nist.gov/glossary/term/least_privilege
2. OpenID Connect (OIDC)https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/about-security-hardening-with-openid-connect
3. workload identityhttps://docs.cloud.google.com/kubernetes-engine/docs/concepts/workload-identity
4. Microsoft Entra Privileged Identity Managementhttps://learn.microsoft.com/en-us/entra/id-governance/privileged-identity-management/pim-configure
5. Google Cloud Privileged Access Managerhttps://docs.cloud.google.com/iam/docs/pam-overview
6. AWS IAM best practiceshttps://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#bp-users-federation-idp
7. for workloadshttps://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#bp-workloads-use-roles
8. Slack request signatureshttps://docs.slack.dev/authentication/verifying-requests-from-slack