What to Do When ElasticSearch is Full?¶
When ElasticSearch memory is full, you can choose to either scale up or delete data to resolve the issue:
You can run the following command to check the resource usage of ES nodes.
kubectl get pod -n mcamel-system | grep common-es-cluster-masters-es | awk '{print $1}' | xargs -I {} kubectl exec {} -n mcamel-system -c elasticsearch -- df -h | grep /usr/share/elasticsearch/data
Scale Up¶
If the host still has available resources, scaling up is a common solution, which involves increasing the PVC capacity.
-
First, run the following command to get the PVC configuration of the
es-data-0
node. Use the actual environment's PVC as a reference. -
Then modify the following
storage
field (the storage class SC you are using should be scalable):- Adjust this value as needed.
Delete Data¶
When ElasticSearch memory is full, you can also delete index data to free up resources.
You can follow the steps below to access the Kibana page and manually delete data.
-
First, ensure that the Kibana Pod exists and is running normally:
-
If it does not exist, manually set the replica to 1 and wait for the service to run normally. If it exists, skip this step.
-
Modify the Kibana Service to be exposed as a NodePort for access:
kubectl patch svc -n mcamel-system mcamel-common-es-cluster-masters-kb-http -p '{"spec":{"type":"NodePort"}}' # After modification, check the NodePort. For example, if the port is 30128, the access URL will be https://{NodeIP in the cluster}:30128 [root@insight-master1 ~]# kubectl get svc -n mcamel-system | grep mcamel-common-es-cluster-masters-kb-http mcamel-common-es-cluster-masters-kb-http NodePort 10.233.51.174 <none> 5601:30128/TCP 108m
-
Retrieve the ElasticSearch Secret to log in to Kibana (username is
elastic
): -
Go to Kibana -> Stack Management -> Index Management and enable the Include hidden indices option to see all indexes. Based on the index sequence numbers, keep the indexes with larger numbers and delete the ones with smaller numbers.