NationTech/harmony
3
0
Fork 2
Code Issues 39 Pull Requests 11 Actions Packages Projects Releases 1 Wiki Activity

Ceph RGW CPU Limit #351

New Issue
Open
opened 2026-10-19 20:34:59 +00:00 by johnride · 0 comments
johnride commented 2026-10-19 20:34:59 +00:00
Owner
Copy Link

We've had an occurence of a Ceph RGW instance having a memory leak of some sort that led to very high CPU usage (over 30 cores maxed out) which caused other workloads to tank on this node.

Setting a CPU limit to the RGW pod would have avoided the neighbors degrading.

This also triggered this alert :

[FIRING:1] openshift-monitoring (NodeMemoryMajorPagesFaults kube-rbac-proxy https [...] node-exporter platform node-exporter-[...] openshift-monitoring/k8s node-exporter warning)
Alerts Firing:
Labels:
alertname = NodeMemoryMajorPagesFaults
container = kube-rbac-proxy
endpoint = https
instance = [...]
job = node-exporter
namespace = openshift-monitoring
openshift_io_alert_source = platform
pod = [...]
prometheus = openshift-monitoring/k8s
service = node-exporter
severity = warning
Annotations:
description = Memory major pages are occurring at very high rate at [...], 500 major page faults per second for the last 15 minutes, is currently at 986.59.
Please check that there is enough memory available at this instance.

summary = Memory major page faults are occurring at very high rate.
We've had an occurence of a Ceph RGW instance having a memory leak of some sort that led to very high CPU usage (over 30 cores maxed out) which caused other workloads to tank on this node. Setting a CPU limit to the RGW pod would have avoided the neighbors degrading. This also triggered this alert : ``` [FIRING:1] openshift-monitoring (NodeMemoryMajorPagesFaults kube-rbac-proxy https [...] node-exporter platform node-exporter-[...] openshift-monitoring/k8s node-exporter warning) Alerts Firing: Labels: alertname = NodeMemoryMajorPagesFaults container = kube-rbac-proxy endpoint = https instance = [...] job = node-exporter namespace = openshift-monitoring openshift_io_alert_source = platform pod = [...] prometheus = openshift-monitoring/k8s service = node-exporter severity = warning Annotations: description = Memory major pages are occurring at very high rate at [...], 500 major page faults per second for the last 15 minutes, is currently at 986.59. Please check that there is enough memory available at this instance. summary = Memory major page faults are occurring at very high rate. ```
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Architecture Decision Record
Good first issue
RFC
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
johnride leokumojin reda sbernier stremblay
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: NationTech/harmony#351
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?