NationTech/harmony
5
0
Fork 2
Code Issues 40 Pull Requests 13 Actions Packages Projects Releases Wiki Activity

feat/drain_k8s_node #232

Merged
johnride merged 13 commits from feat/drain_k8s_node into master 2026-02-17 15:01:09 +00:00
Conversation 3 Commits 13 Files Changed 19 +3904 -1182
johnride commented 2026-02-16 03:04:34 +00:00
Owner
Copy Link
No description provided.
johnride added 6 commits 2026-02-16 03:04:34 +00:00
doc(opnsense): Add note that dnsmasq mac addresses will be droped when bea2a75882 
setting static host
feat: Add k8s client drain node functionality with tests and example e2fa12508f
feat(okd/network_manager): Add get_node_name_for_id and refactor" d8936a8307
wip: Copy files on k8s node via ephemeral pod and configmap 2e89308b82
wip: K8s copy file on node refactoring to extract helpers and add tests 2d26790c82
feat(k8s client): K8sClient module now holds the responsibility for the k8s distribution detection, add resource bundle useful for easy create and delete of a bunch of related resources. cf3050ce87 
First use case is creating a privileged pod allowing writing to nodes on
openshift family clusters. This requires creating the clusterrolebinding
and pod and other resources.
johnride commented 2026-02-16 03:10:36 +00:00
Author
Owner
Copy Link

Grosse PR a review attentivement :

  • Reboot node proprement via API k8s : sauvegarde le boot id, drain, reboot, attend, verifie que boot id a change, attend que la node redevienne Ready
  • Ecrire un fichier sur la node : cree un pod privilegie (avec la gestion de SCC sur OKD), monte le contenu du fichier dans un configmap et ecrit le avec les permissions specifiees
  • Executer une commande sur une node : cree un pod privilegie de la meme maniere, execute la commande et retourne l'output en string
  • Reconfigurer l'interface principale d'une node en bond : utilise les utilitaires ci-haut pour faire tout ca via l'API k8s proprement. Pas besoin de SSH.

On veut s'assurer a la review qu'on ne laisse rien trainer et essayer de penser en dehors de la boite un peu aux cas d'utilisation differents.

Grosse PR a review attentivement : - Reboot node proprement via API k8s : sauvegarde le boot id, drain, reboot, attend, verifie que boot id a change, attend que la node redevienne Ready - Ecrire un fichier sur la node : cree un pod privilegie (avec la gestion de SCC sur OKD), monte le contenu du fichier dans un configmap et ecrit le avec les permissions specifiees - Executer une commande sur une node : cree un pod privilegie de la meme maniere, execute la commande et retourne l'output en string - Reconfigurer l'interface principale d'une node en bond : utilise les utilitaires ci-haut pour faire tout ca via l'API k8s proprement. Pas besoin de SSH. On veut s'assurer a la review qu'on ne laisse rien trainer et essayer de penser en dehors de la boite un peu aux cas d'utilisation differents.
johnride force-pushed feat/drain_k8s_node from 462544c725 to 3257cd9569 2026-02-16 03:17:51 +00:00 Compare
stremblay approved these changes 2026-02-16 14:29:38 +00:00
stremblay left a comment
Owner
Copy Link

lgtm

lgtm
harmony/src/domain/topology/k8s/bundle.rs
@@ -0,0 +115,4 @@
/// Delete all resources in this bundle from the cluster.
/// Resources are deleted in reverse order to respect dependencies.
pub async fn delete(&self, client: &K8sClient) -> Result<(), Error> {
stremblay commented 2026-02-16 14:01:13 +00:00
Owner
Copy Link

Deleting resources in sequential reverse order can deadlock the deletion process. A solution idea is to launch all deletion in parallel

Deleting resources in sequential reverse order can deadlock the deletion process. A solution idea is to launch all deletion in parallel
johnride marked this conversation as resolved
wjro requested changes 2026-02-16 15:43:48 +00:00
harmony/src/domain/topology/k8s/helper.rs Outdated
@@ -0,0 +40,4 @@
pub fn build_privileged_pod(
config: PrivilegedPodConfig,
k8s_distribution: &KubernetesDistribution,
wjro commented 2026-02-16 14:12:26 +00:00
Owner
Copy Link

This is never used for building the privileged pod, I feel like this should be at least a match over the various distributions that are implemented. Right now it is silently only working for open-shift distros

This is never used for building the privileged pod, I feel like this should be at least a match over the various distributions that are implemented. Right now it is silently only working for open-shift distros
johnride marked this conversation as resolved
harmony/src/domain/topology/k8s/helper.rs
@@ -0,0 +147,4 @@
let namespace = config.namespace.clone();
// 1. On OpenShift, create RBAC binding to privileged SCC
if let KubernetesDistribution::OpenshiftFamily = k8s_distribution {
wjro commented 2026-02-16 14:14:53 +00:00
Owner
Copy Link

Maybe the above comment doesn't matter right now since this will only work if the distribution is OpenshiftFamily. It feels to me like this should also be a match over the KubernetesDistributions

Maybe the above comment doesn't matter right now since this will only work if the distribution is OpenshiftFamily. It feels to me like this should also be a match over the KubernetesDistributions
johnride commented 2026-02-17 14:40:33 +00:00
Author
Owner
Copy Link

There is just nothing to do on other distros than openshift family to get a privileged pod. No need for additional permissions mapping/binding like in openshift because of SCC.

There is just nothing to do on other distros than openshift family to get a privileged pod. No need for additional permissions mapping/binding like in openshift because of SCC.
harmony/src/domain/topology/k8s/mod.rs
@@ -0,0 +48,4 @@
#[derive(Clone)]
pub struct K8sClient {
client: Client,
k8s_distribution: Arc<OnceCell<KubernetesDistribution>>,
wjro commented 2026-02-16 14:35:53 +00:00
Owner
Copy Link

It feels like a problem to me to have KubernetesDistribution defined in K8sAnywhere when it is used in the crate. It seems like everything related to KubernetesDistribution should be extracted here.

It feels like a problem to me to have KubernetesDistribution defined in K8sAnywhere when it is used in the crate. It seems like everything related to KubernetesDistribution should be extracted here.
johnride added 1 commit 2026-02-17 01:54:51 +00:00
wip
Some checks failed
Run Check Script / check (pull_request) Failing after 54s
Details
111181c300
johnride added 1 commit 2026-02-17 04:04:23 +00:00
fix: reboot node now works with correct command
Some checks failed
Run Check Script / check (pull_request) Failing after 54s
Details
752526f831
johnride added 3 commits 2026-02-17 14:30:08 +00:00
chore: cleanup comments 9e185cbbd5
Merge remote-tracking branch 'origin/master' into feat/drain_k8s_node 913ed17453
feat: Reboot k8s node works, good logs and tests
All checks were successful
Run Check Script / check (pull_request) Successful in 1m3s
Details
4fba01338d
johnride added 1 commit 2026-02-17 15:00:16 +00:00
fix: dnsmasq now replaces mac address
All checks were successful
Run Check Script / check (pull_request) Successful in 1m2s
Details
c1d46612ac
johnride merged commit c677487a5e into master 2026-02-17 15:01:09 +00:00
johnride deleted branch feat/drain_k8s_node 2026-02-17 15:01:10 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
stremblay
wjro
Labels
Clear labels
Architecture Decision Record
Good first issue
RFC
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
johnride jvtrudel letian stremblay wjro
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: NationTech/harmony#232
No description provided.
Delete Branch "feat/drain_k8s_node"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?