ADR : Choose a default runtime for zero config deployments #7

New Issue

johnride · 2025-03-18T02:30:01Z

johnride commented

2025-03-18 02:30:01 +00:00

Here we are leaning towards K3D as a default runtime. See the ADR for full context and requirements.

Here is a GPT 4.5 analysis of why k3d could make sense over k3s and other alternatives along with potential challenges.

Recommended Homework to Validate k3d Reliability:

Before committing fully to k3d, it is prudent to perform due diligence:

1. Evaluate Stability & Community Activity:

Check GitHub issues and resolution speed.
Look into recent releases and frequency of updates.
Observe the size and activity of the k3d community (contributors, maintainers, users).

2. Run Realistic Test Workloads:

Set up representative workloads (similar complexity to your production workloads) within k3d clusters.
Validate that common Kubernetes features (ingress, persistent volumes, load balancing, networking, etc.) behave correctly.

3. Validate Performance & Resource Consumption:

Measure cluster startup time, memory usage, CPU overhead.
Compare with direct k3s installation to quantify overhead.

4. Investigate Integration & Compatibility:

Verify integration with IDEs, CI/CD pipelines, and existing workflows/tools.
Check compatibility with Podman Desktop (as Docker Desktop alternative) to anticipate future scenarios.

1. Clarifying the Difference between k3s and k3d:

k3s is a lightweight, certified Kubernetes distribution packaged as a single binary. It runs directly on the host OS and uses containerd internally.
k3d is a wrapper around k3s that runs k3s clusters inside Docker containers. It leverages Docker or compatible container runtimes (e.g., potentially Podman).

2. Benefits of Standardizing on k3d Everywhere:

Uniformity:
- Having a single runtime solution simplifies development, support, and documentation.
- Consistency across platforms makes troubleshooting more straightforward.
- Simplified automation scripts and tooling.
Cleanliness & Isolation:
- k3d clusters are ephemeral and cleanly isolated within containers.
- Easy teardown (just remove containers/volumes), leaving minimal residual footprint on the host.
Ease of Management:
- Docker/container runtimes are ubiquitous and widely adopted in developer environments.
- Easier integration into existing Docker-based workflows and CI/CD pipelines.
Reduced Host System Dependencies:
- No need to manage separate binaries/services directly on the host.
- Easier to manage multiple clusters simultaneously without conflicts.

3. Potential Challenges & Risks of k3d Everywhere:

Docker Dependency:
- Requires Docker (or compatible CRI) on all platforms, which might not always be desirable or possible.
- Docker Desktop licensing constraints could be a concern for enterprise environments.
Performance & Resource Overhead:
- Running k3s inside containers may introduce slight overhead compared to native k3s, though generally minimal and acceptable for development workloads.
Maturity and Stability of k3d:
- While k3d is actively maintained and popular, it's fair to perform a thorough evaluation to ensure it is robust enough for your workloads.

Here we are leaning towards K3D as a default runtime. See the ADR for full context and requirements. Here is a GPT 4.5 analysis of why k3d could make sense over k3s and other alternatives along with potential challenges. --- ## Recommended Homework to Validate k3d Reliability: Before committing fully to k3d, it is prudent to perform due diligence: **1. Evaluate Stability & Community Activity:** - Check GitHub issues and resolution speed. - Look into recent releases and frequency of updates. - Observe the size and activity of the k3d community (contributors, maintainers, users). **2. Run Realistic Test Workloads:** - Set up representative workloads (similar complexity to your production workloads) within k3d clusters. - Validate that common Kubernetes features (ingress, persistent volumes, load balancing, networking, etc.) behave correctly. **3. Validate Performance & Resource Consumption:** - Measure cluster startup time, memory usage, CPU overhead. - Compare with direct k3s installation to quantify overhead. **4. Investigate Integration & Compatibility:** - Verify integration with IDEs, CI/CD pipelines, and existing workflows/tools. - Check compatibility with Podman Desktop (as Docker Desktop alternative) to anticipate future scenarios. --- ### 1. Clarifying the Difference between k3s and k3d: - **k3s** is a lightweight, certified Kubernetes distribution packaged as a single binary. It runs directly on the host OS and uses containerd internally. - **k3d** is a wrapper around k3s that runs k3s clusters inside Docker containers. It leverages Docker or compatible container runtimes (e.g., potentially Podman). ### 2. Benefits of Standardizing on k3d Everywhere: - **Uniformity**: - Having a single runtime solution simplifies development, support, and documentation. - Consistency across platforms makes troubleshooting more straightforward. - Simplified automation scripts and tooling. - **Cleanliness & Isolation**: - k3d clusters are ephemeral and cleanly isolated within containers. - Easy teardown (just remove containers/volumes), leaving minimal residual footprint on the host. - **Ease of Management**: - Docker/container runtimes are ubiquitous and widely adopted in developer environments. - Easier integration into existing Docker-based workflows and CI/CD pipelines. - **Reduced Host System Dependencies**: - No need to manage separate binaries/services directly on the host. - Easier to manage multiple clusters simultaneously without conflicts. ### 3. Potential Challenges & Risks of k3d Everywhere: - **Docker Dependency**: - Requires Docker (or compatible CRI) on all platforms, which might not always be desirable or possible. - Docker Desktop licensing constraints could be a concern for enterprise environments. - **Performance & Resource Overhead**: - Running k3s inside containers may introduce slight overhead compared to native k3s, though generally minimal and acceptable for development workloads. - **Maturity and Stability of k3d**: - While k3d is actively maintained and popular, it's fair to perform a thorough evaluation to ensure it is robust enough for your workloads.

johnride added the

RFC

Architecture Decision Record

labels 2025-03-18 02:30:01 +00:00

jvtrudel changed title from ~~ADR : Choose a runtime for zero config deployments~~ to ADR : Choose a default runtime for zero config deployments

2025-12-09 15:49:49 +00:00

jvtrudel commented

2025-12-09 15:55:39 +00:00

Lots of effort is put to choose an implementation. This is good. It ensure that the decision are not random and that we have high quality engineering.

But, we should keep in mind that, thanks to our Hexagonal Architecture, we could "easilly" implement different cases, compare them on benchmarks and take data-driven decisions.

Also, I don't know if it exists, but we should have a standard, clear and easy mecanism to set default implementations of capabilities.

Lots of effort is put to choose an implementation. This is good. It ensure that the decision are not random and that we have high quality engineering. But, we should keep in mind that, thanks to our Hexagonal Architecture, we could "easilly" implement different cases, compare them on benchmarks and take data-driven decisions. Also, I don't know if it exists, but we should have a standard, clear and easy mecanism to set default implementations of capabilities.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: NationTech/harmony#7