KnowledgeBase/CII.md

@@ -1626,48 +1626,75 @@ log4cplus.logger.CiiOldbRemoteDataPointProvider=FATAL
 
 **Problem**
 
-Trying to create a datapoint, you get this error:
+Trying to create a datapoint, you get errors such as:
 
 ```plaintext
    Cannot save to zpb.rr://ciiconfservicehost:9116/configuration/service/clientApi
+
+   [ERROR][CiiOldb] Unknown error occurred ::elt::error::icd::CiiSerializableException
 ```
 
-resp. you get this error:
+This is often caused by low disk space (95% used) available for the oldb permanent store:
 
 ```plaintext
-[ERROR][CiiOldb] Unknown error occurred ::elt::error::icd::CiiSerializableException
+df -h /var/lib/elasticsearch
 ```
 
-and in the IntCfg-logs (as root: journalctl -u srv-config) you see:
+The disk space taken by the permanent store database itself is in the vast majority of cases dominated by the log records stored in it. The solutions aim at decreasing this space.
+
+**Solution 1**
 
 ```plaintext
-Cannot save to zpb.rr://ciiconfservicehost:9116/configuration/service/clientApi
-```
+# Remove old log files:
+find /var/log/elasticsearch -type f -mtime +30 -delete
 
-This can be caused by low disk space (< 5%) available for the oldb permanent store:
+# Put database back into read-write mode:
+curl -X PUT -H "Content-Type: application/json" localhost:9200/_all/_settings -d '
+  { "index.blocks.read_only_allow_delete": null }'
 
-```plaintext
-df -h /var/lib/elasticsearch
+# Remove old log records:
+curl -X POST "localhost:9200/cii_log_default_index/_delete_by_query?pretty" -H 'Content-Type: application/json' -d '
+  { "query": { "range" : { "@timestamp" : { "lte": "now-30d/d" } } } }'
+
+# Note: Removal is a background operation, and can take several minutes until it shows an effect.
+# Run the below command repeatedly to monitor the removal, the docs.count should be decreasing.
+
+# See number of logs stored in permanent store ("docs.count"):
+curl http://localhost:9200/_cat/indices/cii_log_default_index?v\&s=store.size
 ```
 
-**Solution**
+**Solution 2 (brute-force)**
+
+If you could not bring your disk usage below 95%, you can also remove all logs from the permanent store. In this case, you may also want to prevent permanent log storage in the future.
 
 ```plaintext
-# Remove old log files:
-	find /var/log/elasticsearch -type f -mtime +30 -delete
+# Prevent storing logs in the permanent store:
+sudo cii-services stop log
+
+# Remove all log records:
+curl -X DELETE "localhost:9200/cii_log_default_index?pretty"
 
 # Put database back into read-write mode:
-	curl -XPUT -H "Content-Type: application/json" localhost:9200/_all/_settings -d '
-	{ "index.blocks.read_only_allow_delete": null }'
+curl -X PUT -H "Content-Type: application/json" localhost:9200/_all/_settings -d '
+  { "index.blocks.read_only_allow_delete": null }'
 
-# Remove old log records:
-	curl -X POST "localhost:9200/cii_log_default_index/_delete_by_query?pretty" -H 'Content-Type: application/json' -d'
-	{ "query": { "range" : { "@timestamp" : { "lte": "now-30d/d" } } } }'
+# Recreate empty log index
+curl -X PUT "localhost:9200/cii_log_default_index?pretty"
 ```
 
 **Background**
 
-If disk usage is above 95%, elasticsearch goes into read-only mode, and creating new datapoints is not possible any more. To remove old content from the database, it is first necessary to create some free space on the disk (since the database needs space to perform deletetion-operations), then unlock the database, and then remove unnecessary old content from it.
+If disk usage is 95% or more, elasticsearch goes into read-only mode, and creating new datapoints is not possible any more. To remove old content from the database, it is first necessary to create some free space on the disk (since the database needs space to perform deletetion-operations), then unlock the database, and then remove unnecessary old content from it.
+
+To check the read-only status:
+```plaintext
+# Check read-only status:
+# If the output contains any "true" values, you are facing the problem.
+curl -XGET -H "Content-Type: application/json" localhost:9200/_all/_settings/ | jq -r '.[][][]["blocks"]'
+```
+
+
+
 
 ---