NationTech/harmony
6
0
Fork 1
Code Issues 32 Pull Requests 6 Actions 7 Packages Projects Releases 1 Wiki Activity

feat: select k3d for cross-platform Kubernetes development

Browse Source 
Select k3d as the primary solution for running Kubernetes clusters on Windows and macOS, alongside native k3s on Linux, to achieve a consistent "zero setup" experience across platforms while considering resource usage, complexity, and long-term alternatives like WebAssembly.
This commit is contained in:
Jean-Gabriel Gill-Couture 2025-03-17 22:36:07 -04:00
parent 3d6f646460
commit 2433c02de9
1 changed files with 40 additions and 54 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

94
adr/007-default-runtime.md
View File

@ -1,79 +1,65 @@
# Architecture Decision Record: Default Runtime for Managed Workloads** ## Architecture Decision Record: Default Runtime for Managed Workloads
## Status ### Status
Proposed Proposed
## Context ### Context
Our infrastructure orchestrator manages workloads that require a runtime environment for execution. Our infrastructure orchestrator manages workloads requiring a Kubernetes-compatible runtime environment.
**Requirements** **Requirements**
- Cross platform - Cross-platform (Linux, Windows, macOS)
- Supports running complex workloads (similar level of complexity to kubernetes) - Kubernetes compatibility
- Easy to install and setup - Lightweight, easy setup with minimal dependencies
- Low footprint - Clean host teardown and minimal residue
- Well maintained - Well-maintained and actively supported
- K8s compatibility a big plus
When a user launches a project orchestrated by harmony it should run. End of story. ### Decision
This means that we need to be able to automatically install a runtime appropriate for our core use cases. As our context is mostly enterprise applications deployed on various flavors of kubernetes clusters, it is natural to move towards lightweight k8s distributions such as k3s, microk8s, minikube, etc. We select **k3d (k3s in Docker)** as our default runtime environment across all supported platforms (Linux, Windows, macOS).
Now, the main use case where we want it to "just work" is when a developer (or user) clones a repository or downloads an application and wants to launch it locally with zero setup. This means that there is no container runtime installed, no kubernetes cluster, no assumption of libraries or tools installed. ### Rationale
## Decision - **Consistency Across Platforms:**
One solution for all platforms simplifies development, supports documentation, and reduces complexity.
We selected **k3s as the default runtime** for managed workloads on Linux platforms and **k3d on Windows and MacOS**. Other platforms that do not support either k3s or docker are deemed out of scope. - **Simplified Setup and Teardown:**
k3d runs Kubernetes clusters in Docker containers, allowing quick setup, teardown, and minimal host residue.
## Rationale - **Leveraging Existing Container Ecosystem:**
Docker/container runtimes are widely adopted, making their presence and familiarity common among users.
- **Lightweight and Easy to Install:** K3s is designed to be a lightweight Kubernetes distribution, making it easy to install and run on resource-constrained environments, which aligns with our "zero setup" goal. K3d provides a similar experience for MacOS and Windows, leveraging Docker to create lightweight k3s clusters. - **Kubernetes Compatibility:**
- **Kubernetes Compatibility:** K3s is a certified Kubernetes distribution, ensuring compatibility with standard Kubernetes manifests and tools. This allows users to easily migrate workloads between our platform and other Kubernetes environments. K3d also provides this compatibility by running k3s in Docker containers. k3s (within k3d) is fully Kubernetes-certified, ensuring compatibility with standard Kubernetes tools and manifests.
- **Low Resource Consumption:** K3s has a small footprint, minimizing the resource overhead on the host system. This is crucial for local development environments where resources are often limited. K3d shares this advantage as it runs k3s in containers, allowing for efficient resource utilization.
- **Well-Maintained:** Rancher (now SUSE) actively maintains K3s, providing regular updates and security patches. This ensures the stability and security of our platform. K3d benefits from the active k3s community and its own dedicated maintainers.
- **Suitable for Edge Computing:** While not our primary focus *now*, K3s's design makes it suitable for edge computing scenarios, providing a potential future expansion path.
- **Single Binary:** K3s is packaged as a single binary, simplifying installation and management.
- **Docker Dependency on Windows/MacOS acceptable:** While not ideal, the ubiquity of Docker Desktop on these platforms makes k3d a reasonable choice, as it leverages an existing dependency.
## Consequences - **Active Maintenance and Community:**
k3d and k3s both have active communities and are well-maintained.
### Positive ### Consequences
- **Simplified User Experience:** Users can quickly launch projects without needing to install and configure a complex runtime environment. #### Positive
- **Increased Developer Productivity:** Developers can focus on writing code rather than managing infrastructure.
- **Consistent Environment:** Ensures a consistent runtime environment across different development machines.
- **Reduced Support Burden:** Automating runtime setup reduces the support burden on the platform team.
- **Future-Proofing:** Kubernetes compatibility provides a clear migration path to other Kubernetes environments.
- **Enables local testing that mirrors production:** By using a k8s distribution locally, developers can test how their application will behave in a production k8s cluster.
### Negative - **Uniform User Experience:** Users have a consistent setup experience across all platforms.
- **Reduced Support Overhead:** Standardizing runtime simplifies support, documentation, and troubleshooting.
- **Clean Isolation:** Containerization allows developers to easily clean up clusters without affecting host systems.
- **Facilitates Multi-Cluster Development:** Easy creation and management of multiple clusters concurrently.
- **Platform Dependency:** K3s is primarily designed for Linux, requiring a different solution (k3d) for Windows and macOS. This adds complexity to our platform. #### Negative
- **Docker Dependency on Windows/MacOS:** k3d relies on Docker being installed. Users without Docker will need to install it. This could be a barrier to entry for some users.
- **Potential Resource Conflicts:** While K3s is lightweight, it still consumes resources. Running multiple K3s instances (especially via k3d) can lead to resource contention on developer machines.
- **Security Considerations:** As with any runtime environment, K3s and k3d introduce potential security vulnerabilities. We need to ensure that we keep them up-to-date with the latest security patches.
- **Abstraction Leakage:** While we aim for a "zero setup" experience, users may still need to understand basic Kubernetes concepts to troubleshoot issues.
## Alternatives considered - **Docker Dependency:** Requires Docker (or compatible runtime) on all platforms.
- **Potential Overhead:** Slight performance/resource overhead compared to native k3s.
- **Docker Licensing Considerations:** Enterprise licensing of Docker Desktop could introduce additional considerations.
- **Minikube:** A popular option, but can be more resource-intensive than K3s. Also, the driver model can be complex to manage across different operating systems. ### Alternatives Considered
- **MicroK8s:** Another lightweight Kubernetes distribution, but K3s has a stronger focus on simplicity and ease of use.
- **Docker Compose:** Simpler than Kubernetes, but lacks the orchestration capabilities needed for complex workloads. Also, doesn't align with our Kubernetes-centric approach.
- **Podman Desktop:** An increasingly popular alternative to Docker Desktop, but k3d does not fully support it yet.
- **Kind (Kubernetes in Docker):** Similar to k3d, but specifically designed for testing Kubernetes itself, rather than running general workloads.
- **Relying on User-Provided Runtime:** This would require users to install and configure their own runtime environment, which goes against our "zero setup" goal. Increases support burden and leads to inconsistent environments.
- **Virtual Machines (VMs):** Using VMs would provide isolation, but would be much more resource-intensive and slower to start than container-based solutions.
## Future work - **Native k3s (Linux) / k3d (Windows/macOS):** Original proposal. Rejected for greater simplicity and consistency.
- **Minikube, MicroK8s, Kind:** Rejected due to complexity, resource usage, or narrower use-case focus.
- **Docker Compose, Podman Desktop:** Rejected due to lack of orchestration or current limited k3d compatibility.
There is a hype cycle around WASM that holds some ground : WASM to replace containers makes sense in many ways. ### Future Work
To me, it feels just like Java with bytecode. If you can compile to bytecode, all you need to run your program is a JVM. Now, as long as you can compile to WASM, you can run your program on any WASM runtime. - Evaluate Podman Desktop or other container runtimes to avoid Docker dependency.
- Continuously monitor k3d maturity and stability.
This feels like a sensible iteration over both managing containers, that are just bundling a bunch of libraries to make sure your program has everything it needs. This brings major benefits that used to come with the JVM such as observability into heap allocation, garbage collection, etc. - Investigate WebAssembly (WASM) runtimes as emerging alternatives for containerized workloads.
The benefits won't be the exact same but it is reasonable to consider that this simplification and unification of runtime from containers to WASM will bring very significant improvements to the entire software delivery lifecycle.
Down the road, it will be interesting to consider WASM, but the maturity versus our target customer base is not there yet.