KnowledgeBase/CII.md

@@ -1128,7 +1128,7 @@ elt.log.CiiLogManager.set_log_level(name_or_logger: Union\[str, logging.Logger\]
 
 --------------------------------------------------------------------------
 --------------------------------------------------------------------------
--------------------------------------------------------------------------- 
+--------------------------------------------------------------------------
 ### PYBIND errors \[ICD waf build\]
 
 **Problem**
@@ -1204,87 +1204,135 @@ For more information, see also: [KB: PYBIND errors \[ICD waf build\]](onenote:#K
 --------------------------------------------------------------------------
 --------------------------------------------------------------------------
 --------------------------------------------------------------------------
-### g++: internal compiler error: Killed \[waf build\]
+### g++: internal compiler error, g++ fatal error \[ICD waf build\]
 
 **Problem**
 
-Trying to build ("waf build") your mal-application, the build fails and you get this error message:
+Trying to build an ICD-module or MAL-application, the build takes a long time, and eventually fails with an error message like this:
+
+```
+g++: fatal error: Killed signal terminated program cc1plus
+compilation terminated.
+```
 
 ```
 Software/CcsLibs/CcsTestData/python/bindings/src/ModCcstestdata.cpp:18:1: note: in expansion of macro ‘PYBIND11_MODULE’
  PYBIND11_MODULE(ModCcstestdata, modccstestdata) {
  ^
 g++: internal compiler error: Killed (program cc1plus)
-Please submit a full bug report,
-with preprocessed source if appropriate.
-See <http://bugzilla.redhat.com/bugzilla> for instructions.
 ```
 
-**Background**
 
-The cpp compiler runs out of memory and crashes.
+**Background**
 
-You can see the effect by running htop in a separate terminal:
+The cpp compiler runs out of memory and crashes. You can see the effect by running htop in a separate terminal, all memory (including swap space) is consumed by the g++ compiler, which consequently crashes.
 
-All memory (including swap space) is consumed by the g++ compiler, which consequently crashes.
+Memory needed for building a given ICD module: There is a base load that is the same for all ICD modules. On top of that,
+the ICD file contents determine how much memory is needed to build the module.
 
-The issue is under investigation: [ECII-109](https://jira.eso.org/browse/ECII-109)
+_Rule of Thumb_
 	
-**Solution**
+| MAL version	| Base Load	  | Mem per ICD-Struct |
+| ---           | ----            | ----               |
+| MAL 1.8	| 650 MB (2/3 GB) | 320 MB (1/3 GB)    |
+| MAL 2.0	| 650 MB (2/3 GB) | 110 MB (1/8 GB)    |
 
-Until the problem is solved, use the following workarounds (in increasing order of invasiveness):
+So building an ICD with 20 structs under MAL 1.8 will require around 7 GB of available free memory for the gcc compiler.
 
-1.  Make sure to not have elasticsearch running
+_Measuring_
 
+Record metrics of the ICD build with this time-command:
 ```
-systemctl stop elasticsearch
+$ alias time='TIME="real\t%E\nmem\t%Mk\ncpu\t%P\npf\t%F" time'
+$ time waf build
+
+[...]
+real    10:28.19
+mem     7206676k
+cpu     765%
+pf      166
 ```
 
-2.  Prevent waf from spawning parallel build processes:
+- If you see high page fault counts (`pf  1168635`), see next section.
+- If the build process crashes, the time-command's output will not be fully reliable (it will show lower statistics than what they are in reality). See next section.
 
-```
-waf -j1 build
-```
+More info is available at [ECII-109](https://jira.eso.org/browse/ECII-109)
 
-3.  Add temporary swap space to your machine
 
-As root:
+**Solution 1: Remove unnecessary middlewares**
 
+Example module-wscipt:
+```python
+from wtools import module
+# Disable OPCUA and DDS, since not part of this interface.
+module.declare_malicd(mal_opts={'opcua_disabled': True, 'dds_disabled': True})
 ```
-fallocate -l 8G /swapfile
-dd if=/dev/zero of=/swapfile bs=1024 count=8388608
-chmod 600 /swapfile
-mkswap /swapfile
-swapon /swapfile
 
-```
+**Solution 2: Reduce build parallelity**
 
-And to remove the swap space after building:
+Reduce parallelity of build  (default: all cores used).
 	
-```
-swapoff -v /swapfile
-rm -f /swapfile   
-```
+Example: Build with only 4 cores
 
-4.  One of the C++ things that take very long are the OPC MAL mappings.
+` $ time waf -j4 build `
 
-As of DevEnv 2.1.17, it is possible to suppress MAL mappings, with this modification the wscript of your ICD module:
+and try different numbers of cores. Use the time-command output
+to find an optimum between real, page faults, and cpu.
 
-In `wscript`
 
+**Solution 3: Change compiler flags**
+(_to be confirmed_)
+	
+Pass custom compiler flags for your ICD-module
+
+In the module wscript:
+~~~python
+# module-wscipt
+from wtools import module
+def configure(cnf):
+   cnf.env.CXXFLAGS_MALPYTHON = '-O2 -flto -pipe'
+module.declare_malicd()
+~~~
+
+or for a whole package:
+~~~python
+from wtools import package
+def configure(cnf):
+   cnf.env.CXXFLAGS_MALPYTHON = '-O2 -flto -pipe'
+package.declare_package(recurse='m4lcsif m4amlcsif m4cslcsif m4kslcsif')
+~~~
+
+
+**Solution 4: Refactor your ICD**
+	
+If the above optimisations were not enough, you can further reduce the memory need by splitting the ICD up into 2 or more smaller ICD modules.
+
+
+**Solution 5: Free available memory**
+
+Find RAM consumers and stop them, at least temporarily. For example, ElasticSearch uses a significant amount of RAM.
 ```
-declare_malicd(use='icds.base', mal_opts = { 'opcua_disabled': True } )
+sudo cii-services stop elasticsearch
 ```
 
-5.  Ask your system administrator to assign more RAM to your VM.
-
-By default, ECM VMs come with 4 GB, but you want
 
--   8 (if you don't run databases like elasticsearch on your host)
+**Solution 6: Add temporary swap space to your machine**
+```
+# As root:
+ fallocate -l 8G /swapfile
+ dd if=/dev/zero of=/swapfile bs=1024 count=8388608
+ chmod 600 /swapfile
+ mkswap /swapfile
+ swapon /swapfile
 
--   12 (if you run databases like elasticsearch on your host).
+# and to remove it:
+ swapoff -v /swapfile
+ rm -f /swapfile   
+```
 
+**Solution 7: Ask your system administrator to assign more RAM to your VM**
 
+Assess the necessary amount by using the "Rule Of Thumb" above.
 
 
 --------------------------------------------------------------------------