harmony/adr/006-secret-management.md

Architecture Decision Record: Keycloak for Secret Management

Status

Proposed

TODO #3:

Before accepting this proposal we need to run a POC to validate this potential issue :

Keycloak Misuse: Using Keycloak primarily as a secrets manager is inappropriate, as it's designed for identity and access management (IAM), not secrets management. This creates scalability and functionality limitations.

Context

Our infrastructure orchestration requires a robust secret management system as part of our automation project to support secure transitions from development to production environments. Key considerations include:

1. User lifecycle management: In enterprise settings, developers and administrators frequently join and leave projects, creating security risks if their access to secrets isn't properly revoked.

2. Security limitations with file-based solutions: Traditional encrypted file-based approaches (like SOPS) present challenges with user revocation, as departed users can retain local copies of encrypted files and continue accessing secrets indefinitely.

3. Authentication integration: Our solution needs to integrate with existing identity providers to leverage centralized authentication systems already in place.

4. Diverse user base: Our solution must work well for both enterprise teams and small organizations/individual developers without imposing excessive complexity.

5. Operational simplicity: The solution should minimize the cognitive overhead required to manage secrets securely.

Decision

We will implement Keycloak as our secret management solution with the following workflow:

1. Projects will contain only configuration metadata indicating where secrets are stored and which identity provider to use.

2. When a developer runs the project locally, they will be automatically prompted to authenticate via SSO (browser-based or TOTP).

3. Upon successful authentication, the application will use the developer's credentials to retrieve appropriate secrets from the centralized Keycloak server.

4. When a developer leaves the organization, their SSO access is revoked, automatically removing their ability to access secrets.

For smaller organizations and individual developers, we will provide a fully managed Keycloak instance with:

• A free tier supporting a limited number of secrets or API calls
• Simplified setup that hides complexity through our automation tooling
• Potential for paid tiers as usage scales

Why keycloack

We wanted a solution that met these criterias

• Fully open source
• Mature
• Integrates with various SSO providers
• Supports secrets
• As easy as possible to deploy on our existing K8s infrastructure
• Supports any kind of secret, not just K8s

Other considered options :

• Vault : not fully pen source
• SOPS : no SSO integration, makes user lifecycle harder
• AWS Secrets : vendor lock-in and cost
• Bitwarden : SSO feature not fully open source

Consequences

Positive

1. Improved security posture: Secret access is tied directly to centralized identity management, reducing the risk of leaked credentials from former team members.

2. Simplified developer onboarding: New team members can immediately access appropriate secrets without manual sharing of encrypted files or keys.

3. Transparent authentication: SSO integration creates a seamless experience that leverages existing organizational authentication systems.

4. Centralized audit capability: All secret access can be logged and monitored in a single location.

5. Scalable for different organization sizes: The managed service option makes enterprise-grade secret management accessible to smaller teams.

Challenges

1. Network dependency: Developers require network connectivity to access secrets, which could complicate offline development scenarios.

2. Operational overhead: While hidden from most users, we will need to maintain a reliable managed Keycloak service.

3. Migration complexity: Existing projects using file-based secret solutions will require migration assistance.

4. Potential for clipboard leakage: While more difficult than with file-based solutions, determined users could still manually copy secrets they've legitimately accessed.

5. Service availability concerns: Dependency on the centralized secret service creates a potential single point of failure.

6. Implementation complexity: Integrating with various SSO providers and creating a seamless developer experience will require significant initial engineering effort.