KnowledgeBase/CII.md

@@ -2326,6 +2326,11 @@ The Elasticsearch database is used by many CII services, e.g. to store CII log m
 
 Some house-keeping is automated (e.g. ".monitoring-es" indices are automatically rolled over every few days), others may be automated in the future, but currently are not. Instead you should perform the tasks below at your own discretion.
 
+**SOS - Disk Full**  When disk usage (`df -h /var/lib/elastic`) goes above 95%, elasticsearch goes into read-only mode. You will see this reported in /var/log/messages and the elastic logs, and by getting exceptions from CII operations like *oldb.CreateDataPoint()*. This will prevent you from doing any clean-up operations on elasticsearch. First, bring disk usage below 95% (e.g. by removing elastic logs with `find /var/log/elasticsearch -type f -mtime +10 -delete`, or by temporarily moving some files from the full partition to another partition), then put elasticsearch back into read-write mode with this command:
+`curl -XPUT -H "Content-Type: application/json" localhost:9200/_all/_settings -d '{ "index.blocks.read_only_allow_delete": null }'`. After this, you can proceed normally with the house-keeping operations described next.
+
+
+
 1. Check which indices you have and how much memory they consume:
 
 ```plaintext
@@ -2346,12 +2351,25 @@ ela_purge_idx jaeger 10     # delete *jaeger* indices older than 10 days
 curl -X POST "localhost:9200/cii_log_default_index/_delete_by_query?pretty" -H 'Content-Type: application/json' -d' {"query": {"range" : {"@timestamp" : {"lte": "now-30d/d" } } } } '
 ```
 
-4. To free your disk from diagnostic logs older than 10 days, do (as root):
+4. Or brute-force, delete all CII log messages:
+```plaintext
+curl -X DELETE "localhost:9200/cii_log_default_index?pretty"
+curl -X PUT "localhost:9200/cii_log_default_index?pretty"
+```
+
+5. To free your disk from elastic logs older than 10 days, do (as root):
 
 ```plaintext
 find /var/log/elasticsearch -type f -mtime +10 -delete
 ```
 
+Finally, if you do not need CII logs stored in elasticsearch (= you don't use kibana), note that you can stop the log transport and log analysis engine. This way, elasticsearch will grow much slower.
+```
+sudo cii-services stop log
+```
+
+
+
 ---
 
 ---