johnride/nationtech-website
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Good progress on Elastic PVC storageClass blog

Browse Source
This commit is contained in:
Jean-Gabriel Gill-Couture 2023-06-13 00:52:03 -04:00
parent 600940c17e
commit d0ca28aa7c
4 changed files with 73 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
src/app.rs
View File

@ -23,7 +23,7 @@ pub fn App(cx: Scope) -> impl IntoView {
// content for this welcome page
<Router>
<main class="text-white bg-indigo-950 min-h-screen ">
<div class="max-w-3xl container mx-auto p-4 md:p-0">
<div class="max-w-3xl container mx-auto p-4 md:p-0 md:pb-14">
<nav class="py-4 flex space-x-4">
<a href="/">"Home"</a>
<a href="/blog">"Blog"</a>

8
src/components/code.rs Normal file
View File

@ -0,0 +1,8 @@
use leptos::*;
#[component]
pub fn Code(cx: Scope, children: Children) -> impl IntoView {
view! {cx,
<code class="bg-slate-700 p-1 px-2 mx-1 rounded-md inlinei break-word">{children(cx)}</code>
}
}

2
src/components/mod.rs
View File

@ -1,2 +1,4 @@
mod codeblock;
mod code;
pub use codeblock::*;
pub use code::*;

67
src/routes/blog/page.rs
View File

@ -4,6 +4,11 @@ use crate::components::*;
#[component]
pub fn Blog(cx: Scope) -> impl IntoView {
let es_health_green_short = [
"$ kubectl exec -n <namespace> <elasticsearch-pod> -it -- curl localhost:9200/_cluster/health?pretty | grep status",
r#" "status" : "green", # <<<<< Make sure this is green <<<<<"#,
].join("\n");
let es_health_green = [
"$ kubectl exec -n <namespace> <elasticsearch-pod> -it -- curl localhost:9200/_cluster/health?pretty",
"{",
@ -33,15 +38,63 @@ pub fn Blog(cx: Scope) -> impl IntoView {
<p class="">"Thanks Percona for this excellent article "
<a class="text-orange-400" href="https://www.percona.com/blog/change-storage-class-on-kubernetes-on-the-fly/">"Change Storage Class on Kubernetes on the Fly."</a>
</p>
<p class="">"It inspired most of this procedure. Here it is : "
<ol class="list-decimal list-inside">
<li class="pb-2">"Delete the statefulset with "<code class="inline-block">"kubectl delete -n <namespace> statefulset --cascade=orphan <statefulset_name>"</code>" the "<code class="">"orphan"</code>" option is key here and will allow Kubernetes to delete the statefulset without deleting the pods."</li>
<li class="pb-2">"Change the storage class in the helm chart to the new one"</li>
<li class="pb-2">"Upgrade the deployment with "<code>"helm upgrade"</code></li>
<li class="pb-2">"Make sure your cluster is green "
<p class="">"It inspired most of this procedure. Here it is : "</p>
<h4 class="text-2xl pb-2">"Make sure your cluster is green "</h4>
<p>
"This is obviously the first step, and one we will repeat many times throughout the upgrade procedure."
</p>
<CodeBlock>{es_health_green}</CodeBlock>
</li>
<p>
"If your cluster is not green, refer to "<a class="text-orange-400" href="https://www.elastic.co/guide/en/elasticsearch/reference/current/red-yellow-cluster-status.html">"Elastic's documentation."</a>
</p>
<h4 class="text-2xl">"Delete the statefulset with "<Code>"--cascade=orphan"</Code></h4>
<p>
"This is where things get interesting. We are combining strengths of K8s and Helm here. By using the "<Code>"Orphan"</Code>" flag, we are telling K8s to leave the running pods alone. This way we are working around the limitation that prevents us from directly editing the PVC definition within the StatefulSet."
</p>
<CodeBlock>"kubectl delete -n <namespace> statefulset --cascade=orphan <statefulset_name>"</CodeBlock>
<p>
"See the documentation excerpt for the "
<Code>"orphan"</Code>
" option."
</p>
<blockquote class="border-l-2 border-white px-4 py-2 mx-4">"By default, when you tell Kubernetes to delete an object, the controller also deletes dependent objects. You can make Kubernetes orphan these dependents using kubectl or the Kubernetes API, depending on the Kubernetes version your cluster runs. To check the version, enter kubectl version."
<br/>
<br/>
<a class="text-orange-400" href="https://kubernetes.io/docs/tasks/administer-cluster/use-cascading-deletion/#set-orphan-deletion-policy">"https://kubernetes.io/docs/tasks/administer-cluster/use-cascading-deletion/#set-orphan-deletion-policy"</a>
</blockquote>
<h4 class="text-2xl">"Change the storage class in the helm chart to the new one"</h4>
<p>
"In my case, this simply means setting "<Code>"global.storageClass: ceph-block"</Code>
" on line 19 of my values file. If you have defined the storage class for more specific values in the chart you may have to modify those instead."
</p>
<h4 class="text-2xl">"Upgrade the deployment with "<Code>"helm upgrade"</Code></h4>
<p>
"This will create a new StatefulSet to replace the old one that has been deleted previously. We are now leveraging the "
<a class="text-orange-400" href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#stable-network-id">"idempotency of a StatefulSet's pod names"</a>
". This will allow the StatefulSet to take ownership of the exact same pod as the previous StatefulSet. That is, unless you changed its name in the chart's configuration, which would break more than only this."
</p>
<p>
"However, since the underlying components (Pods/ReplicaSets) do not support an update to the "
<Code>"...volumeClaimTemplates[].spec.storageClassName"</Code>
"Nothing more will be done"
</p>
<h4 class="text-2xl">"Delete pods and PVCs"</h4>
<p>
"Now the scary part : deleting."
</p>
<p>
"These steps must be done carefully for each pod in the Elastic cluster."
</p>
<ol class="list-decimal list-inside">
<li>"Find the PVC name for the pod to be deleted with "<Code>"kubectl get -n <namespace> pvc"</Code></li>
<li>"Delete the PVC with "<Code>"kubectl delete -n [namespace] pvc [pvc]"</Code>"This command may hang until the pod is deleted to allow the PVC to be deleted"</li>
<li>"Validate that the PVC is being deleted "<Code>"kubectl get -n [namespace] pvc [pvc]"</Code></li>
<li>"Delete the pod associated with the PVC. "<Code>"kubectl delete -n namespace pod [pod]"</Code>" At this point, the StatefulSet will detect that one of his pods is missing and will recreate it with the new spec, that has the new storageClass setting."</li>
<li>"Now wait until your Elastic cluster comes back to "<Code>"green"</Code>" status"</li>
</ol>
<p>
<Code>"...volumeClaimTemplates[].spec.storageClassName"</Code>
"Nothing more will be done"
</p>
</div>
</div>