Skip to content

Changes

Page history
Introduced sections, and grouped articles into them authored by Marcus Schilling's avatar Marcus Schilling
Hide whitespace changes
Inline Side-by-side
KnowledgeBase/CII.md
View page @ d4a22076
[[_TOC_]]
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### How-to post-install CII services
......@@ -81,52 +85,6 @@ After this, you find the list of available demo apps in the (generated) README f
Most demo apps require CII Services be running on your host or on another host => Check that the related CII Services (e.g. config service for a a config demo app) are accessible: see [How-to start/stop CII Services](#how-to-start-stop-cii-services)
--------------------------------------------------------------------------
### ------------------------------------------------
--------------------------------------------------------------------------
### Cannot create Datapoints [OLDB]
**Problem**
Trying to create a datapoint, you get this error:
```
Cannot save to zpb.rr://ciiconfservicehost:9116/configuration/service/clientApi
```
resp. you get this error:
```
[ERROR][CiiOldb] Unknown error occurred ::elt::error::icd::CiiSerializableException
```
and in the IntCfg-logs (as root: journalctl -u srv-config) you see:
```
Cannot save to zpb.rr://ciiconfservicehost:9116/configuration/service/clientApi
```
This can be caused by low disk space (< 5%) available for the oldb permanent store:
```
df -h /var/lib/elasticsearch
```
**Solution**
```
# Remove old log files:
find /var/log/elasticsearch -type f -mtime +30 -delete
# Put database back into read-write mode:
curl -XPUT -H "Content-Type: application/json" localhost:9200/_all/_settings -d '
{ "index.blocks.read_only_allow_delete": null }'
# Remove old log records:
curl -X POST "localhost:9200/cii_log_default_index/_delete_by_query?pretty" -H 'Content-Type: application/json' -d'
{ "query": { "range" : { "@timestamp" : { "lte": "now-30d/d" } } } }'
```
**Background**
If disk usage is above 95%, elasticsearch goes into read-only mode, and creating new datapoints is not possible any more. To remove old content from the database, it is first necessary to create some free space on the disk (since the database needs space to perform deletetion-operations), then unlock the database, and then remove unnecessary old content from it.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
......@@ -178,9 +136,13 @@ We'll check the merge request, and your change can make it to the next release.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### ------------------------------------------------------
### [ICD]
### Failure to build ZPB from ICD \[ICD waf build\]
**Problem**
......@@ -242,217 +204,208 @@ declare_project(name='mytests',
recurse='myconfig mylsvsim icd tests')
```
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### Connecting to OLDB takes long, then fails \[cpp OLDB\]
### PYBIND errors \[ICD waf build\]
**Question**
My application blocks a long time on first OLDB access, and eventually fails with a timeout.
**Problem**
**Answer**
Trying to build your MAL Application, you get errors like below related to the PYBIND module.
Reconfigure the communication timeout (default: 60 seconds)
```
icd/python/bindings/src/ModProto-benchmark.cpp:18:25: error: expected initializer before ‘-’ token
PYBIND11_MODULE(ModProto-benchmark, modproto-benchmark) {
```
a) through an environment variable and a properties file
**Solution**
Check the name of your ICD file:
```
$ cat <<EOF >/tmp/cii_client.ini
connection_timeout = 5
EOF
$ export CONFIG_CLIENT_INI = /tmp/cii_client.ini
> find icd
icd
icd/wscript
icd/src
icd/src/proto-benchmark.xml
```
b) programmatically (available since CII 2.0/DevEnv 3.9)
The icd file name contains a minus, which is actually reflected in the above error message.
Rename the file to something like this:
```
CiiClientConfiguration config_client_ini = { .connection_timeout = 5, };
elt::config::CiiConfigClient::SetDevClientConfig (config_client_ini);
> find icd
icd
icd/wscript
icd/src
icd/src/protobenchmark.xml
```
**Background**
In general, due to the many code generation steps taking place, your freedom in ICD file naming is limited.
The actual stalling comes from a failed MAL-communication with the CII Internal Configuration System, which likely is not running. Setting the timeout for the CiiConfigClient is therefore the thing to do. Note that the properties file (aka. "deployment config") takes precedence and will, if they overlap, overrule the programmatic (aka. "developer config") settings.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### Mock OLDB for unit tests \[OLDB cpp python\]
**Question**
Is there already a faked OLDB that I can use in my unit tests in cpp?
**Answer**
(by D. Kumar)
You can create an in-memory OLDB providing a cached config oldb implementation and using the local file system for blob data.
The oldb-client cpp module is providing a
- pure (virtual = 0) interface elt::oldb::CiiOldbDataPointProvider<sup>\[1\]</sup>, and two implementations:
- in-memory data point provider storing data points to the memory
(this is an empty implementation which provides a minimal operational fake oldb)
- a redis data point provider storing data points to redis.
- a remote filesystem interface elt::oldb:impl::ciiOldbRemoteFileProvider.hpp<sup>\[2\]</sup>, and two implementations:
### multiple XMLs found \[ICD waf build\]
- S3 implementation
**Problem**
- local file system implementation: ciiOldbLocalFileProvider<sup>\[3\]</sup> *\[Note: not before DevEnv 3.4\]*
Trying to build your ICD module, you see this error:
Here are complete examples of unit tests showing the main use cases how to use oldb (with subscriptions) and metadata creation:
```
Waf: Entering directory \`/home/eltdev/repos/hlcc/build'
Error: multiple XMLs found, just one supported.
```
while in fact you have only one XML file in your ICD directory.
<https://gitlab.eso.org/cii/srv/cii-srv/-/blob/master/oldb-client/cpp/oldb/test/oldbInMemoryTest.cpp>
**Solution**
The same exists in python:
Check the file name of your ICD file:
Example in <https://gitlab.eso.org/ahoffsta/cii-srv/-/blob/oldb-in-memory-missing-python-binding/oldb-client/python/oldb/test/oldbInMemoryTest.py>
make sure it starts with an uppercase letter.
**Background**
References
The error message is misleading (will be improved, [ECII-426](https://jira.eso.org/browse/ECII-426)).
\[1\] <https://gitlab.eso.org/cii/srv/cii-srv/-/blob/master/oldb-client/cpp/oldb/src/include/ciiOldbDataPointProvider.hpp>
The code generator for malicd_topics fails when the ICD file name starts with lowercase.
\[2\] <https://gitlab.eso.org/cii/srv/cii-srv/-/blob/master/oldb-client/cpp/oldb/src/include/provider/ciiOldbRemoteFileProvider.hpp>
For more information, see also: [KB: PYBIND errors \[ICD waf build\]](onenote:#KB%20PYBIND%20errors%20[ICD%20waf%20build]&section-id={F524F9BE-F51D-4A01-9976-93359FCC4966}&page-id={0FEE4FB9-C58B-4E8A-A276-2EC4367CFB30}&end&base-path=https://europeansouthernobservatory.sharepoint.com/sites/ELT_Control/SiteAssets/ELT_Control%20Notebook/Documentation/ELT%20Control%20KnowledgeBase.one)
\[3\] <https://gitlab.eso.org/cii/srv/cii-srv/-/blob/master/oldb-client/cpp/oldb/src/provider/ciiOldbLocalFileProvider.hpp> *\[Note: not before DevEnv 3.4\]*
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### Catching API Exceptions \[Python\]
### g++: internal compiler error, g++ fatal error \[ICD waf build\]
**Problem**
My application contains a call to the CII Python API.
Trying to build an ICD-module or MAL-application, the build takes a long time, and/or fails with an error message like this:
When I ran it, it threw an exception with the following backtrace:
```
Top Level Unexpected exception:
Traceback (most recent call last):
File "/home/eltdev/MODULES/test/app.py", line 91, in instantiateDP
double_dp = self.oldb_client.create_data_point(uri, metadataInstName)
CiiOldbPyB.CiiOldbDpExistsException: The Data point cii.oldb:/root/test/xxxdp already exists.
g++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
```
Therefore, I added a corresponding try-catch around my call:
```
try:
...
except CiiOldbPyB.CiiOldbDpExistsException as e:
Software/CcsLibs/CcsTestData/python/bindings/src/ModCcstestdata.cpp:18:1: note:
in expansion of macro ‘PYBIND11_MODULE’
PYBIND11_MODULE(ModCcstestdata, modccstestdata) {
^
g++: internal compiler error: Killed (program cc1plus)
```
When I run it, the try-catch doesn't work.
Moreover, I now get two backtraces:
**Background**
```
Top Level Unexpected exception:
Traceback (most recent call last):
File "/home/eltdev/MODULES/test/app.py", line 91, in instantiateDP
double_dp = self.oldb_client.create_data_point(uri, metadataInstName)
CiiOldbPyB.CiiOldbDpExistsException: The Data point cii.oldb:/root/test/xxxdp already exists.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/eltdev/MODULES/test/app.py", line 108, in main
oldbCreator.instantiateOLDB_exception()
File "/home/eltdev/MODULES/test/app.py", line 81, in instantiateOLDB_exception
self.instantiateDP(double_dp_uri, double_dp_meta.get_instance_name())
File "/home/eltdev/MODULES/test/app.py", line 94, in instantiateDP
except CiiOldbPyB.CiiOldbDpExistsException:
NameError: name 'CiiOldbPyB' is not defined
```
The cpp compiler runs out of memory and crashes. You can see the effect by running htop in a separate terminal, all memory (including swap space) is consumed by the g++ compiler, which consequently crashes.
**Solution**
_Memory needed for building a given ICD module_
You were mislead by the first backtrace: the exception name in the backtrace is not what you should catch.
There is a base load that is the same for all ICD modules. On top of that, the ICD file contents determine how much memory is needed to build the module.
In your code, replace "**CiiOldbPyB**" with "**elt.oldb**":
_Rule of Thumb_
| MAL version | Base Load | Mem per ICD-Struct |
| --- | ---- | ---- |
| MAL 1.x | 650 MB (2/3 GB) | 320 MB (1/3 GB) |
| MAL 2.0 | 650 MB (2/3 GB) | 110 MB (1/8 GB) |
```
try:
....
except elt.oldb.CiiOldbDpExistsException as e:
```
Thus, if your biggest ICD contains 20 structs, building under MAL 1.x will require around 7 GB of available free memory.
For completeness - do no forget this statement:
_Measuring_
Record metrics of the ICD build with this time-command:
```
import elt.oldb
$ alias time='TIME="real\t%E\nmem\t%Mk\ncpu\t%P\npf\t%F" time'
$ time waf build
[...]
real 10:28.19
mem 7206676k
cpu 765%
pf 166
```
**Background**
- If the build crashes, the time-command's output will not be fully reliable (the real memory need is higher than what the output shows).
- High page fault counts (`pf 1168635`) generally indicate you should reduce the module's footprint, see Solutions below.
The CII Python API is mostly a binding to the CII C++ API.
More info is available at [ECII-109](https://jira.eso.org/browse/ECII-109)
The CiiOldbPyB.CiiOldbDpExistsException is the original binding class.
This binding class is re-exported under the name elt.oldb.CiiOldbDpExistsException.
**Solution 1: Decrease the module's footprint**
The elt.oldb module internally loads the C++ binding module CiiOldbPyB. So both are the same exception.
1. Remove unnecessary middlewares
Nonetheless, you should use the re-exported name, not the original name in your application. We discourage the use of the original name because the structure of the CiiOldbPyB module is more "chaotic" and not equivalent to elt.oldb.
Use the `xyz_disabled` options:
```python
from wtools import module
# Disable OPCUA and DDS, since not part of this interface.
module.declare_malicd(mal_opts={'opcua_disabled': True, 'dds_disabled': True})
```
Unfortunately, in the backtraces you will always see the original name instead of the re-exported name.
2. Reduce build parallelism
This question was originally asked in [ECII-422](https://jira.eso.org/browse/ECII-422).
By default the build system uses all cores on the host. Less parallelism means less memory consumers during the build. This is controlled by the waf `-j` option.
To build with only 4 cores:
` $ time waf -j4 build `
As a rough estimate, each waf build task will consume around 2 GB RAM, so on a 12 core host with 16 GB RAM, a parallelism of 8 may be a good choice. Try different numbers of cores and use the output from the time-command (see above) to find an optimum between real, page faults, and cpu.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### access_key empty (DevEnv 3.2.0) \[OLDB\]
**Problem**
3. Adjust the compiler flags
The default set of compiler flags applied by the build system consume significant memory.
We recommend using "-O2 -flto -pipe" (_to be confirmed_) instead. This is how you pass custom compiler flags for your ICD-module:
Trying to use the OLDB on DevEnv 3.2.0, I'm getting this error:
In your project wscript:
~~~python
from wtools import project
[...]
def configure(cnf):
cnf.env.CXXFLAGS_MALPYTHON = '-O2 -flto -pipe'
[...]
~~~
Unexpected exception occurred. What:Configuration invalid: access_key empty
4. Refactor your ICD
Reduce the memory need by splitting the big ICDs up into 2 or more smaller ICD modules.
**Solution**
Run the following commands (you will be asked for the root pw):
**Solution 2: Increase the available memory**
```
wget -q www.eso.org/~mschilli/download/cii/postinstall/cii-postinstall-20210610
cii-services stop config
su -c "bash cii-postinstall-20210610 schemas"
# If the script
#You should see the following output:
1. Find RAM consumers and stop them, at least temporarily. For example, ElasticSearch uses a significant amount of RAM: `sudo cii-services stop elasticsearch`
Password:
CII PostInstall (20210610)
schemas: applying fix ECII397
/home/eltdev/
schemas: populating elasticsearch
schemas: skipping telemetry
schemas: skipping alarms
cii-services start config
2. Add temporary swap space to your host
```bash
# As root:
fallocate -l 8G /swapfile
dd if=/dev/zero of=/swapfile bs=1024 count=8388608
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
# and to remove it:
swapoff -v /swapfile
rm -f /swapfile
```
**Background**
3. Add permanent memory to your VM
The OLDB settings coming with 3.2.0 are buggy.
Increase your RAM, respectively ask your system administrator to do it. Assess the necessary amount by using the "Rule Of Thumb" above.
The CII post-install procedure is able to hotfix the settings (ECII397).
The problem will be fixed in DevEnv 3.4.
......@@ -460,123 +413,95 @@ The problem will be fixed in DevEnv 3.4.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### Datapoint already exists \[OLDB\]
### Choose middlewares to build \[MAL ICD\]
**Problem**
My application tries to create an OLDB datapoint.
I am certain that my ICD will never be used over OPC UA. Nonetheless, the ICD-compilation builds OPC UA mappings for my ICD. This is unnecessarily extending the compilation time for my application.
This fails because the datapoint "already exists":
**Solution**
By default the ICD-compilation builds mappings for all middlewares. But it is possible to exclude certain middleware mappings from compilation, which will reduce compilation time. You do this by passing mal options to the icd-generator.
**Example**
wscript
```
2021-06-22T11:01:33.003+0000, ERROR, CiiOldbRedisDataPointProvider/140709706681216, Data point uri: cii.oldb:/tcs/hb/tempser3 in Redis already exists.
declare_malicd(use='icds.base', mal_opts = { 'opcua_disabled': True } )
```
In response, my application skips the creation step, and wants to use the reportedly existing datapoint.
However, when doing this, I get the error "Datapoint doesn't exist".
The available options are:
Likewise, when I run the oldb-gui database browser, it does not show this data point in the OLDB.
- opcua_disabled = if True, disable OPCUA middleware generation
**Variant 2 of the Problem**
- dds_disabled = if True, disable DDS middleware generation
I try to access an OLDB datapoint, and I see two errors like this:
- zpb_disabled = if True, disable ZEROMQ middleware generation
```
Target configuration does not exist: Failed to retrieve configuration from elastic search: Configuration
[…]
elt.oldb.exceptions.CiiOldbDpExistsException: Data point cii.oldb:/alarm/alarm/device/motor/input_int_dp_alarm already exisits.
```
Go directly to Solution 2 below.
**Variant 3 of the Problem**
I try to delete an OLDB datapoint and I see an error like this:
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### Variable Tracking exceeded [ICD]
**Problem**
When building an ICD using CII-MAL, you see this warning message:
```
CiiOldbPyB.CiiOldbException: De-serialization error:sizeof(T)\*count is greater then remaining
variable tracking size limit exceeded
```
Go directly to Solution 2 below.
**Background**
The two errors are contradicting.
The -fvar-tracking-assignments is automatically enabled by GCC when optimizations are enabled.
Datapoints are stored in two databases: a document-database (permanent store) for its metadata, and a key-value-database (volatile store) for its current value. The above symptoms indicate that the two databases are out-of-sync, meaning the datapoint exists only "half".
There is a limit on how many variables can be tracked by the compiler. The warning tells you that more vars would need to be tracked than what's supported.
**Solution 1**
You can disable the tracking manually with -fno-var-tracking-assignments.
There is a good chance you can delete the datapoint to clean up the situation (as of [ECII-303](https://jira.eso.org/browse/ECII-303)):
There are two easy ways to do it on the overall project.
**Solution A**
```
#!/usr/bin/env python
import elt.config
import elt.oldb
oldb_client = elt.oldb.CiiOldbFactory.get_instance()
elt.oldb.CiiOldbGlobal.set_write_enabled(True)
uri = elt.config.Uri("cii.oldb:/tcs/hb/tempser3")
oldb_client.delete_data_point(uri)
Export CXXFLAGS=-fno-var-tracking-assignments
```
**Solution 2**
And then rerun “waf configure” and continue with the build.
If the above didn't help, find out which "half" of the datapoint exists:
Note: you have to have the exported variable each time you do a “waf configure” as that is the point at which such flags are saved
1. The current value exists, and the metadata is missing. This is the case when upgrading DevEnv/CII without deleting the Redis cache.
**Solution B**
Run the following command:
If you want it to be inside the project then put the flags fixed inside your project, so in the top level wscript (where you define the project) add this configure section before the project declaration:
Note: enter a search-expression matching the datapoint URI, in the example this is "tempser3".
```
def configure(cnf):
cnf.env.append_value('CXXFLAGS', \['-fno-var-tracking-assignments'\])
```
# confirm that this is the cause of the problem
redis-cli KEYS *tempser3*
```
If the command is showing the problematic key, we run a similar command to remove that key from the store:
```
# if so, delete the offending datapoint
redis-cli --scan --pattern *tempser3* | xargs redis-cli del
```
2. The metadata exists, and the current value is missing
Update: As of [ECII-303](https://jira.eso.org/browse/ECII-303), this should no longer be necessary, since the OLDB will make an attempt to automatically re-instate the current value that is missing.
**Solution 3**
If none of the above helped, another possibility is to clean up the metadata.
WARNING: This is an invasive operation. It deletes all datapoints in the OLDB.
```
# Clean up the OLDB databases
config-initEs.sh
oldb-initEs
redis-cli flushall
sudo cii-services stop config
sudo cii-services start config
```
If you are dealing with a multi-user oldb ("role_groupserver", meaning it serves an OLDB to a team of developers), after executing the above commands you need to additionally execute (with privileges):
```
/elt/ciisrv/postinstall/cii-postinstall role_groupserver
```
### ------------------------------------------------------
### [MAL]
If you have doubts, please contact us.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### Req/Rep Connection Listeners \[MAL Python\]
### Req/Rep Connection Listeners [MAL Python]
**Problem**
......@@ -651,91 +576,7 @@ Remember to delete (assign None), to this object when closing the connection to
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### Command Line Tools and Python snippets \[OLDB\]
**Problem**
I need to inspect or modify the content of the OLDB from the command line or a shell script.
**Solution**
**cii-oldb-traversal-tool** for searching through the OLDB
```
$ cii-oldb-traversal-tool --file output --quality OK
$ cat output
cii.oldb:///root/trklsv/cfg/log/level|OK|WARNING|2020-09-11T15:25:08Z
cii.oldb:///root/trklsv/cfg/req/endpoint|OK|zpb.rr://localhost:44444/m1/TrkLsvServer|2020-09-11T15:25:08Z
cii.oldb:///root/trklsv/ctr/current/altaz/alt|OK|0.000000|2020-09-11T15:24:25Z
cii.oldb:///root/trklsv/ctr/current/altaz/az|OK|0.000000|2020-09-11T15:24:25Z
cii.oldb:///root/trklsv/ctr/current/radec/dec|OK|0.000000|2020-09-11T15:24:27Z
cii.oldb:///root/trklsv/ctr/current/radec/ra|OK|0.000000|2020-09-11T15:24:27Z
cii.oldb:///root/trklsv/ctr/poserr|OK|0.000000|2020-09-11T15:24:27Z
cii.oldb:///root/trklsv/ctr/status|OK|UNKNOWN|2020-09-11T15:23:55Z
cii.oldb:///root/trklsv/ctr/substate|OK|UNKNOWN|2020-09-11T15:23:49Z
cii.oldb:///root/trklsv/ctr/target/altaz/alt|OK|0.000000|2020-09-11T15:24:24Z
cii.oldb:///root/trklsv/ctr/target/altaz/az|OK|0.000000|2020-09-11T15:24:25Z
cii.oldb:///root/trklsv/ctr/target/radec/dec|OK|0.000000|2020-09-11T15:24:26Z
cii.oldb:///root/trklsv/ctr/target/radec/ra|OK|0.000000|2020-09-11T15:24:26Z
```
**oldb-cli** for reading, writing, subscribing to an OLDB-datapoint
```
$ oldb-cli read cii.oldb:///root/trklsv/cfg/req/endpoint
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/eelt/ciisrv/1.0-RC3-20201030/lib/srv-support-libs/slf4j-nop-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/eelt/mal/1.1.0-2.2.3-20201027/lib/mal-opcua/slf4j-nop-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.helpers.NOPLoggerFactory]
log4j:WARN No appenders could be found for logger (io.netty.util.internal.logging.InternalLoggerFactory).
log4j:WARN Please initialize the log4j system properly.
Timestamp: 2020-09-11T15:25:08.648Z
Quality: OK
Value: zpb.rr://localhost:44444/m1/TrkLsvServer
```
**oldb Python API** for creating, deleting an OLDB-datapoint
```
$ python
Python 3.7.6 (default, Jan 8 2020, 19:59:22)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import elt.oldb
>>> from elt.config import Uri
>>> oldb = elt.oldb.CiiOldbFactory.get_instance()
```
… and, to create a datapoint:
```
>>> elt.oldb.CiiOldbGlobal.set_write_enabled(True)
>>> oldb.create_data_point_by_value (Uri("cii.oldb:/ccs/tst/tmp1"), "my text")
```
… and, to read a datapoint:
```
>>> oldb.get_data_point(Uri("cii.oldb:/ccs/tst/tmp1")).read_value().get_value()
'my text'
```
… and, to delete a datapoint:
```
>>> elt.oldb.CiiOldbGlobal.set_write_enabled(True)
>>> oldb.delete_data_point (Uri("cii.oldb:/ccs/tst/tmp1"))
```
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### Latency on Pub/Sub \[MAL ZMQ\]
### Latency on Pub/Sub [MAL ZMQ]
**Problem**
......@@ -772,51 +613,21 @@ The problem lies in the ZMQ send queues. Default size is 1000 and with 144MB per
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### Choose middlewares to build \[MAL ICD\]
### Sending an array of unions [MAL CPP]
**Problem**
I am certain that my ICD will never be used over OPC UA. Nonetheless, the ICD-compilation builds OPC UA mappings for my ICD. This is unnecessarily extending the compilation time for my application.
**Solution**
By default the ICD-compilation builds mappings for all middlewares. But it is possible to exclude certain middleware mappings from compilation, which will reduce compilation time. You do this by passing mal options to the icd-generator.
**Example**
wscript
Trying to send a msg, I get the following error message from CII:
```
declare_malicd(use='icds.base', mal_opts = { 'opcua_disabled': True } )
[libprotobuf ERROR google/protobuf/message_lite.cc:121] Can't parse message of type "generated.zpb.fcfif.StdCmds_Request" because it is missing required fields: data.Setup.payload[0].piezoData.input
```
The available options are:
- opcua_disabled = if True, disable OPCUA middleware generation
- dds_disabled = if True, disable DDS middleware generation
- zpb_disabled = if True, disable ZEROMQ middleware generation
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### Sending an array of unions \[MAL cpp\]
**Problem**
Trying to send a msg, I get the following error message from CII:
```
[libprotobuf ERROR google/protobuf/message_lite.cc:121] Can't parse message of type "generated.zpb.fcfif.StdCmds_Request" because it is missing required fields: data.Setup.payload[0].piezoData.input
```
My ICD definition looks like this
My ICD definition looks like this
```xml
<enum name="PiezoInput">
......@@ -917,63 +728,10 @@ Your code does not work since you do not use union instance provided by the pare
This issue was first described in [ECII-154](https://jira.eso.org/browse/ECII-154)
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### config not found on remote db \[Conf\]
**Problem**
You are intending to read a config from the local config database ("localdb"), but you see an error message like this.
```
elt.config.exceptions.CiiConfigNoTcException: Target configuration does not exist: Failed to retrieve configuration from elastic search: Configuration cii.config://*/supervisoryapp/TrkLsvDeploy on the remote db was not found
at elt.config.client.ConfigRemoteDatabase.retrieveConfig(ConfigRemoteDatabase.java:191)
at elt.config.client.CiiConfigClient.retrieveConfig(CiiConfigClient.java:354)
at elt.config.client.CiiConfigClient.retrieveConfig(CiiConfigClient.java:310)
at trkLsv.DataContext.loadConfig(DataContext.java:324)
at trkLsv.DataContext.<init>(DataContext.java:190)
at trkLsv.TrkLsv.go(TrkLsv.java:72)
at trkLsv.TrkLsv.main(TrkLsv.java:41)
Caused by: elt.error.icd.CiiSerializableException
at elt.config.service.client.icd.zpb.ServiceClientApiInterfaceAsyncImpl.processRequest(ServiceClientApiInterfaceAsyncImpl.java:73)
at elt.mal.zpb.rr.ClientAsyncImpl.events(ClientAsyncImpl.java:261)
at org.zeromq.ZPoller.dispatch(ZPoller.java:537)
at org.zeromq.ZPoller.poll(ZPoller.java:488)
at org.zeromq.ZPoller.poll(ZPoller.java:461)
at elt.mal.zpb.ZpbMal.processThread(ZpbMal.java:459)
at elt.mal.zpb.ZpbMal.lambda$new$0(ZpbMal.java:119)
at java.lang.Thread.run(Thread.java:748)
```
**Solution A**
Your local file may be invalid (e.g. illegal format, or doesn't match the config class definition).
Look at the content of your local config database, e.g. with
```
$ find $INTROOT/localdb
```
and correct the file in place, or fix the source yaml and then redeploy it from source to the localdb.
**Background**
You may have a malformed json file in your local db, which the config service failed to read.
Because of the use the location wildcard "\*" in your code (in "cii.config://\*/supervisoryapp/TrkLsvDeploy"),
the config service has consequently tried to load the config from the remote config database, where no such config exists.
To that end, the error message is misleading, and should be improved (ticket [ECII-208](https://jira.eso.org/browse/ECII-208)).
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### Failed to send request, send queue full \[CII MAL\]
### Failed to send request, send queue full [MAL]
**Problem**
......@@ -995,7 +753,7 @@ Probably you have called close() on the CiiConfigClient instance somewhere, mayb
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### Getting More Logs \[MAL\]
### Getting More Logs [MAL Log]
**Problem**
......@@ -1166,39 +924,77 @@ Loggers for **mal-opcua**
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### Change Log Levels at Run-time
### More Frames expected [MAL ZMQ]
**Problem**
I want to modify the log levels of my application programmatically, without having to reload the full log configuration.
In your application you get errors like this:
**Solution**
```
Oct 16, 2019 11:26:21 AM elt.mal.zpb.ps.ZpbSubscriber events
With [ECII-282](https://jira.eso.org/browse/ECII-282), the CiiLogManager was extended in all three languages to allow dynamically change log levels.
WARNING: Remote data entity type hash does not match (1040672065 != 1708154137).
**C++** added methods:
Oct 16, 2019 11:26:21 AM elt.mal.zpb.ps.ZpbSubscriber events
```
void elt::log::CiiLogManager::SetLogLevel(const std::string logger_name, log4cplus::LogLevel level)
void elt::log::CiiLogManager::SetLogLevel(log4cplus::Logger logger, log4cplus::LogLevel level)
```
WARNING: Failed to process message.
**Java** added methods:
java.lang.RuntimeException: more frames expected
at elt.mal.zpb.ps.ZpbSubscriber.requireMoreFrames(ZpbSubscriber.java:82)
at elt.mal.zpb.ps.ZpbSubscriber.events(ZpbSubscriber.java:114)
```
void elt.log.CiiLogManager.setLogLevel(
final String loggerName, final org.apache.logging.log4j.Level level);
void elt.log.CiiLogManager.setLogLevel(
org.apache.logging.log4j.Logger logger, final org.apache.logging.log4j.Level level)
**Background**
The first warning message indicates that your client has received a piece of data (= MAL entity type) on a channel that should not carry such data. This means you are running two publishers, publishing different types of data, on the same channel.
**Example of topic definition with port-clash**
```xml
<pubsub_topic>
<topic_name>sm:current_pos</topic_name>
<topic_type>sm_current_pos</topic_type>
<address_uri>zpb.ps://134.171.2.220:57110/test</address_uri>
<qos latency_ms="1" deadline_ms="100"/>
<performance rate_hz="10" latency_ms="1" synchronous="false" />
<mal>
<zpb />
</mal>
</pubsub_topic>
<pubsub_topic>
<topic_name>hp:global_status</topic_name>
<topic_type>hp_global_status</topic_type>
<address_uri>zpb.ps://134.171.2.220:57110/test</address_uri>
<qos latency_ms="10" deadline_ms="100"/>
<performance rate_hz="1" latency_ms="10" synchronous="false" />
<mal>
<zpb />
</mal>
</pubsub_topic>
```
**Python** added methods:
The second warning and the error trace are just a consequence of the first warning.
**Solution**
Check your topics.xml file, and make sure each channel has its own exclusive topic name.
In the above example, e.g.:
```
elt.log.CiiLogManager.set_log_level(name_or_logger: Union\[str, logging.Logger\], level: logging.Level)
zpb.ps://134.171.2.220:57110/test1
```
and
```
zpb.ps://134.171.2.220:57110/test2
```
......@@ -1207,74 +1003,77 @@ elt.log.CiiLogManager.set_log_level(name_or_logger: Union\[str, logging.Logger\]
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### PYBIND errors \[ICD waf build\]
### Address in use [MAL ZMQ]
**Problem**
Trying to build your MAL Application, you get errors like below related to the PYBIND module.
Running your application, you see this error message:
```
icd/python/bindings/src/ModProto-benchmark.cpp:18:25: error: expected initializer before ‘-’ token
PYBIND11_MODULE(ModProto-benchmark, modproto-benchmark) {
ZMQException: Errno 48 : Address already in use
```
**Solution**
**Solution A**
Check the name of your ICD file:
Another instance of your application is still running.
```
> find icd
icd
icd/wscript
icd/src
icd/src/proto-benchmark.xml
```
**Solution B**
The icd file name contains a minus, which is actually reflected in the above error message.
Another instance of your application has non-gracefully terminated without freeing the network port.
Rename the file to something like this:
Ensure your application always performs a call to "mal.close()" on shutdown.
**Solution C**
This could be really a Usage Error due to wrong configuration.
The error message is in fact misleading.
**Example**
```
> find icd
icd
icd/wscript
icd/src
icd/src/protobenchmark.xml
eltcii33 [09:38:27] eeltdev:~/mschilli > mal-esotests-testclient1 pub sAddr=zpb://eltcii28:12333/Sample tSlow=100 nSamp=100
pub:sys: Available MAL Flavours loaded: [dds, opc, zpb]
pub:config: sAddr=zpb://eltcii28:12333/Sample
pub:config: nSamp=100
pub:config: tSlow=100
Internal Error: org.eso.elt.mal.MalException: org.zeromq.ZMQException: Errno 48 : Address already in use
```
In general, due to the many code generation steps taking place, your freedom in ICD file naming is limited.
**Reason**
The above code is trying, on host eltcii33, to publish with an endpoint eltcii28.
**Fix**
On eltcii33, the endpoint must be eltcii33.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### multiple XMLs found \[ICD waf build\]
### Scheme not supported [MAL]
**Problem**
Trying to build your ICD module, you see this error:
Running your application, you get this error message:
```
Waf: Entering directory \`/home/eltdev/repos/hlcc/build'
Error: multiple XMLs found, just one supported.
elt.mal.SchemeNotSupportedException: middleware not supported
```
while in fact you have only one XML file in your ICD directory.
**Solution**
Check the file name of your ICD file:
make sure it starts with an uppercase letter.
**Solution A**
**Background**
In your code, you've misspelled the middleware name, e.g. "opc" instead of "opcua"
The error message is misleading (will be improved, [ECII-426](https://jira.eso.org/browse/ECII-426)).
**Solution B**
The code generator for malicd_topics fails when the ICD file name starts with lowercase.
The middleware is supported in fact, but failed to load.
For more information, see also: [KB: PYBIND errors \[ICD waf build\]](onenote:#KB%20PYBIND%20errors%20[ICD%20waf%20build]&section-id={F524F9BE-F51D-4A01-9976-93359FCC4966}&page-id={0FEE4FB9-C58B-4E8A-A276-2EC4367CFB30}&end&base-path=https://europeansouthernobservatory.sharepoint.com/sites/ELT_Control/SiteAssets/ELT_Control%20Notebook/Documentation/ELT%20Control%20KnowledgeBase.one)
- E.g. in DDS, you are using a Qos profile xml file which has some illegal syntax inside.
......@@ -1282,249 +1081,365 @@ For more information, see also: [KB: PYBIND errors \[ICD waf build\]](onenote:#K
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### g++: internal compiler error, g++ fatal error \[ICD waf build\]
### Choosing a NIC [MAL DDS]
**Problem**
Trying to build an ICD-module or MAL-application, the build takes a long time, and/or fails with an error message like this:
I'm using MAL-DDS, and I have two network cards (NICs) installed. MAL uses the wrong one, i.e. my network traffic goes into the "office" network, but should go into the "control" network.
```
g++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
```
**Background**
```
Software/CcsLibs/CcsTestData/python/bindings/src/ModCcstestdata.cpp:18:1: note:
in expansion of macro ‘PYBIND11_MODULE’
PYBIND11_MODULE(ModCcstestdata, modccstestdata) {
^
g++: internal compiler error: Killed (program cc1plus)
```
As multicast addresses are by definition not associated with hardware (ie they map to MAC addresses which have no corresponding Ethernet card), there is no means for the OS to resolve which NIC the IGMP subscription should be sent down. Thus the NIC must be specified, or the default is used (which is the office network).
Therefore, DDS allows you to specify which NIC you want to use for outgoing traffic.
**Background**
Therefore, this boils down to configuring DDS.
The cpp compiler runs out of memory and crashes. You can see the effect by running htop in a separate terminal, all memory (including swap space) is consumed by the g++ compiler, which consequently crashes.
**Solution**
_Memory needed for building a given ICD module_
1. Get the XML file shown in Solution #1 on this page:
There is a base load that is the same for all ICD modules. On top of that, the ICD file contents determine how much memory is needed to build the module.
<https://community.rti.com/howto/control-or-restrict-network-interfaces-nics-used-discovery-and-data-distribution>
_Rule of Thumb_
| MAL version | Base Load | Mem per ICD-Struct |
| --- | ---- | ---- |
| MAL 1.x | 650 MB (2/3 GB) | 320 MB (1/3 GB) |
| MAL 2.0 | 650 MB (2/3 GB) | 110 MB (1/8 GB) |
2. Continue with this article:
Thus, if your biggest ICD contains 20 structs, building under MAL 1.x will require around 7 GB of available free memory.
[KB: Configuring DDS](https://gitlab.eso.org/ecs/eltsw-docs/-/wikis/KnowledgeBase/CII#configuring-dds-cii-mal-dds)
_Measuring_
Record metrics of the ICD build with this time-command:
```
$ alias time='TIME="real\t%E\nmem\t%Mk\ncpu\t%P\npf\t%F" time'
$ time waf build
[...]
real 10:28.19
mem 7206676k
cpu 765%
pf 166
```
- If the build crashes, the time-command's output will not be fully reliable (the real memory need is higher than what the output shows).
- High page fault counts (`pf 1168635`) generally indicate you should reduce the module's footprint, see Solutions below.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### Configuring DDS [MAL DDS]
More info is available at [ECII-109](https://jira.eso.org/browse/ECII-109)
**Problem**
Some of the middlewares usable through MAL offer a variety of configuration options.
**Solution 1: Decrease the module's footprint**
This article explains how to define and use configuration for the DDS middleware.
1. Remove unnecessary middlewares
**Background**
Use the `xyz_disabled` options:
```python
from wtools import module
# Disable OPCUA and DDS, since not part of this interface.
module.declare_malicd(mal_opts={'opcua_disabled': True, 'dds_disabled': True})
```
To configure DDS, the following 3 things are necessary:
2. Reduce build parallelism
i\) put the desired config into an external XML file
By default the build system uses all cores on the host. Less parallelism means less memory consumers during the build. This is controlled by the waf `-j` option.
To build with only 4 cores:
` $ time waf -j4 build `
ii\) define the FASTRTPS_DEFAULT_PROFILES_FILE (Connext: NDDS_QOS_PROFILES) environment variable, so DDS finds the XML file
As a rough estimate, each waf build task will consume around 2 GB RAM, so on a 12 core host with 16 GB RAM, a parallelism of 8 may be a good choice. Try different numbers of cores and use the output from the time-command (see above) to find an optimum between real, page faults, and cpu.
iii\) pass 2 keys to the MAL factory, so DDS finds the right profile in the XML file
*(For a discussion about making this easier, see [ECII-77](https://jira.eso.org/browse/ECII-77))*
3. Adjust the compiler flags
The default set of compiler flags applied by the build system consume significant memory.
We recommend using "-O2 -flto -pipe" (_to be confirmed_) instead. This is how you pass custom compiler flags for your ICD-module:
**Solution**
In your project wscript:
~~~python
from wtools import project
[...]
def configure(cnf):
cnf.env.CXXFLAGS_MALPYTHON = '-O2 -flto -pipe'
[...]
~~~
1. Create the XML file.
4. Refactor your ICD
Reduce the memory need by splitting the big ICDs up into 2 or more smaller ICD modules.
Examples are easily found in the "Fast DDS" documentation and user forums.
2. In your code, let MAL load the DDS-specific properties.
**Solution 2: Increase the available memory**
Example (C++):
1. Find RAM consumers and stop them, at least temporarily. For example, ElasticSearch uses a significant amount of RAM: `sudo cii-services stop elasticsearch`
```cpp
auto publisher = factory.getPublisher<elt::telem::Temperature>(uri, {
std::make_shared<mal::ps::qos::Latency>(std::chrono::milliseconds(100)),
std::make_shared<mal::ps::qos::Deadline>(std::chrono::seconds(1)) }, {
{"dds.domain", "100"}, { "dds.qos.profile.library", “test"}, { "dds.qos.profile.name", “UDPv4_properties"}} });
2. Add temporary swap space to your host
```bash
# As root:
fallocate -l 8G /swapfile
dd if=/dev/zero of=/swapfile bs=1024 count=8388608
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
In the case of Fast DDS, the "profile.library" value can be set to a dummy value, since it will not be used.
# and to remove it:
swapoff -v /swapfile
rm -f /swapfile
```
3. Add permanent memory to your VM
3. Before running your code:
```
FASTRTPS_DEFAULT_PROFILES_FILE=\<path of XML file>
```
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### Summary of OPC/UA MAL in C++
This article covers integration of OPC/UA in CII MAL specifically for OPC/UA Data Access and Subscription profiles. OPC/UA method invocation is also supported in CII MAL but is not described in this article, likewise details of the Python (and Java) support are not provided. Only C++ is considered here.
OPC/UA communication middleware is exposed in CII MAL as either Publish/Subscribe or Request/Reply APIs.
The XML ICD definition of types in CII is used to map sets of data points together that are read/written as a group.
Each attribute in the defined type is connected to a corresponding data point in the OPC/UA data space via a URI. Thus the CII URI for a complex type will contain specific addresses of multiple nodes in the OPC/UA data space.
Increase your RAM, respectively ask your system administrator to do it. Assess the necessary amount by using the "Rule Of Thumb" above.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### More Frames expected \[CII MAL ZMQ\]
#### Pub/Sub API for OPC/UA Clients:
**Problem**
The CII MAL Pub/Sub API utilizes OPC/UA Data Access Reads and Writes, as well OPC/UA subscription. A Publisher will directly trigger an OPC DA write, while a Subscriber will work in one of two ways, depending on the type associated with the subscriber:
In your application you get errors like this:
- If the subscribers CII URI contains only a single node (i.e. the XML ICD type contained only a single attribute) then the Subscriber will create an OPC/UA subscription on that data point. The subscription will trigger notification of updates to the data point node, which will then be queued for notification via the CII Subscriber API.
- If the Subscriber is using subscription, the opc.ps.outstandingPublishRequests property should not be zero (e.g. set it to 5), see the example code below. The reason is that the publish queue is used to store and send subscription notification events, and if the queue is small the even notifications may simply be dropped.
- If the subscriber CII URI contains multiple nodes (i.e. the XML ICD type contains multiple attributes) then the Subscriber launches a thread to perform periodic polling of data from the OPC/UA server. The rate is based on the properties passed in creating the subscriber. e.g.
```java
try {
subscriber = factory.getSubscriber\<T>(opcua_uri, ::elt::mal::ps::qos::QoS::DEFAULT,
{{"opc.ps.outstandingPublishRequests","5"},{"opc.asyncLoopExecutionPeriodMs","50"},
{"opc.asyncCallSubmitTimeoutMs","1000"},
{"opc.ps.pollingPeriodMs","20000"},
{"opc.asyncCallRetryPeriodMs","250"}});
}
catch(...) {
throw;
}
```
Oct 16, 2019 11:26:21 AM elt.mal.zpb.ps.ZpbSubscriber events
WARNING: Remote data entity type hash does not match (1040672065 != 1708154137).
Oct 16, 2019 11:26:21 AM elt.mal.zpb.ps.ZpbSubscriber events
WARNING: Failed to process message.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
#### Request/Reply API for OPC/UA Clients:
java.lang.RuntimeException: more frames expected
As OPC/UA Data Access read and write essentially follow a synchronous request/reply pattern, CII MAL also provides this interface for OPC/UA clients.
at elt.mal.zpb.ps.ZpbSubscriber.requireMoreFrames(ZpbSubscriber.java:82)
The ICD is termed "virtual" in CII nomenclature as it does not require definition as an XML ICD using the service syntax, rather the same types defined for the Pub/Sub API may be used with a CII MAL OPC/UA Request/Reply.
at elt.mal.zpb.ps.ZpbSubscriber.events(ZpbSubscriber.java:114)
This approach means OPC/UA Data Access read (and write) are synchronous, and may be called as needed by the application.
```cpp
namespace mal {
namespace rr {
namespace da {
class DataAccess : public ::elt::mal::rr::RrEntity {
public:
[...]
template <typename T>
void read(::elt::mal::ps::DataEntity\<T>& value) {
readUnsafe(&value);
}
template <typename T>
void write(const ::elt::mal::ps::DataEntity\<T>& value) {
writeUnsafe(&value);
}
```
**Background**
The first warning message indicates that your client has received a piece of data (= MAL entity type) on a channel that should not carry such data. This means you are running two publishers, publishing different types of data, on the same channel.
A test application showing its use is here:
**Example of topic definition with port-clash**
<https://gitlab.eso.org/cosylab/elt-cii/mal/mal-test/-/blob/develop/cpp/mal-test-performance/opcua/mal-opcua-da-speed/src/common.cpp>
### ------------------------------------------------------
### [OLDB]
```xml
<pubsub_topic>
<topic_name>sm:current_pos</topic_name>
<topic_type>sm_current_pos</topic_type>
<address_uri>zpb.ps://134.171.2.220:57110/test</address_uri>
<qos latency_ms="1" deadline_ms="100"/>
<performance rate_hz="10" latency_ms="1" synchronous="false" />
<mal>
<zpb />
</mal>
</pubsub_topic>
<pubsub_topic>
<topic_name>hp:global_status</topic_name>
<topic_type>hp_global_status</topic_type>
<address_uri>zpb.ps://134.171.2.220:57110/test</address_uri>
<qos latency_ms="10" deadline_ms="100"/>
<performance rate_hz="1" latency_ms="10" synchronous="false" />
<mal>
<zpb />
</mal>
</pubsub_topic>
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### Cannot create Datapoints [OLDB]
**Problem**
Trying to create a datapoint, you get this error:
```
Cannot save to zpb.rr://ciiconfservicehost:9116/configuration/service/clientApi
```
The second warning and the error trace are just a consequence of the first warning.
resp. you get this error:
```
[ERROR][CiiOldb] Unknown error occurred ::elt::error::icd::CiiSerializableException
```
and in the IntCfg-logs (as root: journalctl -u srv-config) you see:
```
Cannot save to zpb.rr://ciiconfservicehost:9116/configuration/service/clientApi
```
This can be caused by low disk space (< 5%) available for the oldb permanent store:
```
df -h /var/lib/elasticsearch
```
**Solution**
Check your topics.xml file, and make sure each channel has its own exclusive topic name.
```
# Remove old log files:
find /var/log/elasticsearch -type f -mtime +30 -delete
In the above example, e.g.:
# Put database back into read-write mode:
curl -XPUT -H "Content-Type: application/json" localhost:9200/_all/_settings -d '
{ "index.blocks.read_only_allow_delete": null }'
# Remove old log records:
curl -X POST "localhost:9200/cii_log_default_index/_delete_by_query?pretty" -H 'Content-Type: application/json' -d'
{ "query": { "range" : { "@timestamp" : { "lte": "now-30d/d" } } } }'
```
zpb.ps://134.171.2.220:57110/test1
```
and
**Background**
If disk usage is above 95%, elasticsearch goes into read-only mode, and creating new datapoints is not possible any more. To remove old content from the database, it is first necessary to create some free space on the disk (since the database needs space to perform deletetion-operations), then unlock the database, and then remove unnecessary old content from it.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### Exception while connecting to OLDB service
When starting an application using the OLDB API, the following exception is received:
```
zpb.ps://134.171.2.220:57110/test2
<date/time>, ERROR, CiiOldbFactory/140635105143296, Unexpected config exception occurred while retrieving configuration for cii.config://remote/oldb/configurations/oldbClientConfig What:Path oldb/configurations/oldbClientConfig, version -1 not found
terminate called after throwing an instance of 'elt::oldb::CiiOldbException'
what(): Unexpected config exception occurred while retrieving configuration for cii.config://remote/oldb/configurations/oldbClientConfig What:Path oldb/configurations/oldbClientConfig, version -1 not found
```
**Solution**
This can indicate that elasticSearch on the config/oldb server is not running, or has crashed.
Use cii-services command to check the status on the server where the (cii-internal) config is running.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### Address in use \[CII MAL ZMQ\]
### Connecting to OLDB takes long, then fails \[cpp OLDB\]
**Problem**
**Question**
My application blocks a long time on first OLDB access, and eventually fails with a timeout.
Running your application, you see this error message:
**Answer**
Reconfigure the communication timeout (default: 60 seconds)
a) through an environment variable and a properties file
```
ZMQException: Errno 48 : Address already in use
$ cat <<EOF >/tmp/cii_client.ini
connection_timeout = 5
EOF
$ export CONFIG_CLIENT_INI = /tmp/cii_client.ini
```
**Solution A**
b) programmatically (available since CII 2.0/DevEnv 3.9)
Another instance of your application is still running.
```
CiiClientConfiguration config_client_ini = { .connection_timeout = 5, };
elt::config::CiiConfigClient::SetDevClientConfig (config_client_ini);
```
**Solution B**
**Background**
Another instance of your application has non-gracefully terminated without freeing the network port.
The actual stalling comes from a failed MAL-communication with the CII Internal Configuration System, which likely is not running. Setting the timeout for the CiiConfigClient is therefore the thing to do. Note that the properties file (aka. "deployment config") takes precedence and will, if they overlap, overrule the programmatic (aka. "developer config") settings.
Ensure your application always performs a call to "mal.close()" on shutdown.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### Mock OLDB for unit tests \[OLDB cpp python\]
**Solution C**
**Question**
This could be really a Usage Error due to wrong configuration.
Is there already a faked OLDB that I can use in my unit tests in cpp?
The error message is in fact misleading.
**Answer**
**Example**
(by D. Kumar)
You can create an in-memory OLDB providing a cached config oldb implementation and using the local file system for blob data.
The oldb-client cpp module is providing a
- pure (virtual = 0) interface elt::oldb::CiiOldbDataPointProvider<sup>\[1\]</sup>, and two implementations:
- in-memory data point provider storing data points to the memory
(this is an empty implementation which provides a minimal operational fake oldb)
- a redis data point provider storing data points to redis.
- a remote filesystem interface elt::oldb:impl::ciiOldbRemoteFileProvider.hpp<sup>\[2\]</sup>, and two implementations:
- S3 implementation
- local file system implementation: ciiOldbLocalFileProvider<sup>\[3\]</sup> *\[Note: not before DevEnv 3.4\]*
Here are complete examples of unit tests showing the main use cases how to use oldb (with subscriptions) and metadata creation:
<https://gitlab.eso.org/cii/srv/cii-srv/-/blob/master/oldb-client/cpp/oldb/test/oldbInMemoryTest.cpp>
The same exists in python:
Example in <https://gitlab.eso.org/ahoffsta/cii-srv/-/blob/oldb-in-memory-missing-python-binding/oldb-client/python/oldb/test/oldbInMemoryTest.py>
References
\[1\] <https://gitlab.eso.org/cii/srv/cii-srv/-/blob/master/oldb-client/cpp/oldb/src/include/ciiOldbDataPointProvider.hpp>
\[2\] <https://gitlab.eso.org/cii/srv/cii-srv/-/blob/master/oldb-client/cpp/oldb/src/include/provider/ciiOldbRemoteFileProvider.hpp>
\[3\] <https://gitlab.eso.org/cii/srv/cii-srv/-/blob/master/oldb-client/cpp/oldb/src/provider/ciiOldbLocalFileProvider.hpp> *\[Note: not before DevEnv 3.4\]*
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### access_key empty (DevEnv 3.2.0) \[OLDB\]
**Problem**
Trying to use the OLDB on DevEnv 3.2.0, I'm getting this error:
Unexpected exception occurred. What:Configuration invalid: access_key empty
**Solution**
Run the following commands (you will be asked for the root pw):
```
eltcii33 [09:38:27] eeltdev:~/mschilli > mal-esotests-testclient1 pub sAddr=zpb://eltcii28:12333/Sample tSlow=100 nSamp=100
pub:sys: Available MAL Flavours loaded: [dds, opc, zpb]
pub:config: sAddr=zpb://eltcii28:12333/Sample
pub:config: nSamp=100
pub:config: tSlow=100
Internal Error: org.eso.elt.mal.MalException: org.zeromq.ZMQException: Errno 48 : Address already in use
wget -q www.eso.org/~mschilli/download/cii/postinstall/cii-postinstall-20210610
cii-services stop config
su -c "bash cii-postinstall-20210610 schemas"
# If the script
#You should see the following output:
Password:
CII PostInstall (20210610)
schemas: applying fix ECII397
/home/eltdev/
schemas: populating elasticsearch
schemas: skipping telemetry
schemas: skipping alarms
cii-services start config
```
**Reason**
**Background**
The above code is trying, on host eltcii33, to publish with an endpoint eltcii28.
The OLDB settings coming with 3.2.0 are buggy.
**Fix**
The CII post-install procedure is able to hotfix the settings (ECII397).
On eltcii33, the endpoint must be eltcii33.
The problem will be fixed in DevEnv 3.4.
......@@ -1532,55 +1447,207 @@ On eltcii33, the endpoint must be eltcii33.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### Scheme not supported \[CII MAL\]
### Datapoint already exists \[OLDB\]
**Problem**
Running your application, you get this error message:
My application tries to create an OLDB datapoint.
This fails because the datapoint "already exists":
```
elt.mal.SchemeNotSupportedException: middleware not supported
2021-06-22T11:01:33.003+0000, ERROR, CiiOldbRedisDataPointProvider/140709706681216, Data point uri: cii.oldb:/tcs/hb/tempser3 in Redis already exists.
```
**Solution A**
In response, my application skips the creation step, and wants to use the reportedly existing datapoint.
In your code, you've misspelled the middleware name, e.g. "opc" instead of "opcua"
However, when doing this, I get the error "Datapoint doesn't exist".
**Solution B**
Likewise, when I run the oldb-gui database browser, it does not show this data point in the OLDB.
The middleware is supported in fact, but failed to load.
**Variant 2 of the Problem**
- E.g. in DDS, you are using a Qos profile xml file which has some illegal syntax inside.
I try to access an OLDB datapoint, and I see two errors like this:
```
Target configuration does not exist: Failed to retrieve configuration from elastic search: Configuration
[…]
elt.oldb.exceptions.CiiOldbDpExistsException: Data point cii.oldb:/alarm/alarm/device/motor/input_int_dp_alarm already exisits.
```
Go directly to Solution 2 below.
**Variant 3 of the Problem**
I try to delete an OLDB datapoint and I see an error like this:
```
CiiOldbPyB.CiiOldbException: De-serialization error:sizeof(T)\*count is greater then remaining
```
Go directly to Solution 2 below.
**Background**
The two errors are contradicting.
Datapoints are stored in two databases: a document-database (permanent store) for its metadata, and a key-value-database (volatile store) for its current value. The above symptoms indicate that the two databases are out-of-sync, meaning the datapoint exists only "half".
**Solution 1**
There is a good chance you can delete the datapoint to clean up the situation (as of [ECII-303](https://jira.eso.org/browse/ECII-303)):
```
#!/usr/bin/env python
import elt.config
import elt.oldb
oldb_client = elt.oldb.CiiOldbFactory.get_instance()
elt.oldb.CiiOldbGlobal.set_write_enabled(True)
uri = elt.config.Uri("cii.oldb:/tcs/hb/tempser3")
oldb_client.delete_data_point(uri)
```
**Solution 2**
If the above didn't help, find out which "half" of the datapoint exists:
1. The current value exists, and the metadata is missing. This is the case when upgrading DevEnv/CII without deleting the Redis cache.
Run the following command:
Note: enter a search-expression matching the datapoint URI, in the example this is "tempser3".
```
# confirm that this is the cause of the problem
redis-cli KEYS *tempser3*
```
If the command is showing the problematic key, we run a similar command to remove that key from the store:
```
# if so, delete the offending datapoint
redis-cli --scan --pattern *tempser3* | xargs redis-cli del
```
2. The metadata exists, and the current value is missing
Update: As of [ECII-303](https://jira.eso.org/browse/ECII-303), this should no longer be necessary, since the OLDB will make an attempt to automatically re-instate the current value that is missing.
**Solution 3**
If none of the above helped, another possibility is to clean up the metadata.
WARNING: This is an invasive operation. It deletes all datapoints in the OLDB.
```
# Clean up the OLDB databases
config-initEs.sh
oldb-initEs
redis-cli flushall
sudo cii-services stop config
sudo cii-services start config
```
If you are dealing with a multi-user oldb ("role_groupserver", meaning it serves an OLDB to a team of developers), after executing the above commands you need to additionally execute (with privileges):
```
/elt/ciisrv/postinstall/cii-postinstall role_groupserver
```
If you have doubts, please contact us.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### Command Line Tools and Python snippets \[OLDB\]
**Problem**
I need to inspect or modify the content of the OLDB from the command line or a shell script.
**Solution**
**cii-oldb-traversal-tool** for searching through the OLDB
```
$ cii-oldb-traversal-tool --file output --quality OK
$ cat output
cii.oldb:///root/trklsv/cfg/log/level|OK|WARNING|2020-09-11T15:25:08Z
cii.oldb:///root/trklsv/cfg/req/endpoint|OK|zpb.rr://localhost:44444/m1/TrkLsvServer|2020-09-11T15:25:08Z
cii.oldb:///root/trklsv/ctr/current/altaz/alt|OK|0.000000|2020-09-11T15:24:25Z
cii.oldb:///root/trklsv/ctr/current/altaz/az|OK|0.000000|2020-09-11T15:24:25Z
cii.oldb:///root/trklsv/ctr/current/radec/dec|OK|0.000000|2020-09-11T15:24:27Z
cii.oldb:///root/trklsv/ctr/current/radec/ra|OK|0.000000|2020-09-11T15:24:27Z
cii.oldb:///root/trklsv/ctr/poserr|OK|0.000000|2020-09-11T15:24:27Z
cii.oldb:///root/trklsv/ctr/status|OK|UNKNOWN|2020-09-11T15:23:55Z
cii.oldb:///root/trklsv/ctr/substate|OK|UNKNOWN|2020-09-11T15:23:49Z
cii.oldb:///root/trklsv/ctr/target/altaz/alt|OK|0.000000|2020-09-11T15:24:24Z
cii.oldb:///root/trklsv/ctr/target/altaz/az|OK|0.000000|2020-09-11T15:24:25Z
cii.oldb:///root/trklsv/ctr/target/radec/dec|OK|0.000000|2020-09-11T15:24:26Z
cii.oldb:///root/trklsv/ctr/target/radec/ra|OK|0.000000|2020-09-11T15:24:26Z
```
**oldb-cli** for reading, writing, subscribing to an OLDB-datapoint
```
$ oldb-cli read cii.oldb:///root/trklsv/cfg/req/endpoint
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/eelt/ciisrv/1.0-RC3-20201030/lib/srv-support-libs/slf4j-nop-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/eelt/mal/1.1.0-2.2.3-20201027/lib/mal-opcua/slf4j-nop-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.helpers.NOPLoggerFactory]
log4j:WARN No appenders could be found for logger (io.netty.util.internal.logging.InternalLoggerFactory).
log4j:WARN Please initialize the log4j system properly.
Timestamp: 2020-09-11T15:25:08.648Z
Quality: OK
Value: zpb.rr://localhost:44444/m1/TrkLsvServer
```
**oldb Python API** for creating, deleting an OLDB-datapoint
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### Choosing a NIC \[CII MAL DDS\]
```
$ python
Python 3.7.6 (default, Jan 8 2020, 19:59:22)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import elt.oldb
>>> from elt.config import Uri
>>> oldb = elt.oldb.CiiOldbFactory.get_instance()
```
**Problem**
… and, to create a datapoint:
I'm using MAL-DDS, and I have two network cards (NICs) installed. MAL uses the wrong one, i.e. my network traffic goes into the "office" network, but should go into the "control" network.
```
>>> elt.oldb.CiiOldbGlobal.set_write_enabled(True)
>>> oldb.create_data_point_by_value (Uri("cii.oldb:/ccs/tst/tmp1"), "my text")
```
**Background**
… and, to read a datapoint:
As multicast addresses are by definition not associated with hardware (ie they map to MAC addresses which have no corresponding Ethernet card), there is no means for the OS to resolve which NIC the IGMP subscription should be sent down. Thus the NIC must be specified, or the default is used (which is the office network).
```
>>> oldb.get_data_point(Uri("cii.oldb:/ccs/tst/tmp1")).read_value().get_value()
'my text'
```
Therefore, DDS allows you to specify which NIC you want to use for outgoing traffic.
… and, to delete a datapoint:
Therefore, this boils down to configuring DDS.
```
>>> elt.oldb.CiiOldbGlobal.set_write_enabled(True)
>>> oldb.delete_data_point (Uri("cii.oldb:/ccs/tst/tmp1"))
```
**Solution**
1. Get the XML file shown in Solution #1 on this page:
<https://community.rti.com/howto/control-or-restrict-network-interfaces-nics-used-discovery-and-data-distribution>
2. Continue with this article:
[KB: Configuring DDS](https://gitlab.eso.org/ecs/eltsw-docs/-/wikis/KnowledgeBase/CII#configuring-dds-cii-mal-dds)
### ------------------------------------------------------
### [Log]
......@@ -1588,210 +1655,199 @@ Therefore, this boils down to configuring DDS.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### Configuring DDS \[CII MAL DDS\]
### Change Log Levels at Run-time [Log]
**Problem**
Some of the middlewares usable through MAL offer a variety of configuration options.
I want to modify the log levels of my application programmatically, without having to reload the full log configuration.
This article explains how to define and use configuration for the DDS middleware.
**Solution**
**Background**
With [ECII-282](https://jira.eso.org/browse/ECII-282), the CiiLogManager was extended in all three languages to allow dynamically change log levels.
To configure DDS, the following 3 things are necessary:
**C++** added methods:
i\) put the desired config into an external XML file
```
void elt::log::CiiLogManager::SetLogLevel(const std::string logger_name, log4cplus::LogLevel level)
void elt::log::CiiLogManager::SetLogLevel(log4cplus::Logger logger, log4cplus::LogLevel level)
```
ii\) define the FASTRTPS_DEFAULT_PROFILES_FILE (Connext: NDDS_QOS_PROFILES) environment variable, so DDS finds the XML file
**Java** added methods:
iii\) pass 2 keys to the MAL factory, so DDS finds the right profile in the XML file
```
void elt.log.CiiLogManager.setLogLevel(
final String loggerName, final org.apache.logging.log4j.Level level);
void elt.log.CiiLogManager.setLogLevel(
org.apache.logging.log4j.Logger logger, final org.apache.logging.log4j.Level level)
```
*(For a discussion about making this easier, see [ECII-77](https://jira.eso.org/browse/ECII-77))*
**Python** added methods:
**Solution**
```
elt.log.CiiLogManager.set_log_level(name_or_logger: Union\[str, logging.Logger\], level: logging.Level)
```
1. Create the XML file.
Examples are easily found in the "Fast DDS" documentation and user forums.
2. In your code, let MAL load the DDS-specific properties.
Example (C++):
```cpp
auto publisher = factory.getPublisher<elt::telem::Temperature>(uri, {
std::make_shared<mal::ps::qos::Latency>(std::chrono::milliseconds(100)),
std::make_shared<mal::ps::qos::Deadline>(std::chrono::seconds(1)) }, {
{"dds.domain", "100"}, { "dds.qos.profile.library", “test"}, { "dds.qos.profile.name", “UDPv4_properties"}} });
### ------------------------------------------------------
### [Lang]
In the case of Fast DDS, the "profile.library" value can be set to a dummy value, since it will not be used.
```
3. Before running your code:
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### Multithreading in Java [Lang Java]
```
FASTRTPS_DEFAULT_PROFILES_FILE=\<path of XML file>
```
Look at this document:
- [Java JMM Visibility](Documents/Java-JMMVisibility.pdf)
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### Variable Tracking exceeded \[CII MAL\]
### Catching API Exceptions [Lang Python]
**Problem**
When building an ICD using CII-MAL, you see this warning message:
My application contains a call to the CII Python API.
When I ran it, it threw an exception with the following backtrace:
```
variable tracking size limit exceeded
Top Level Unexpected exception:
Traceback (most recent call last):
File "/home/eltdev/MODULES/test/app.py", line 91, in instantiateDP
double_dp = self.oldb_client.create_data_point(uri, metadataInstName)
CiiOldbPyB.CiiOldbDpExistsException: The Data point cii.oldb:/root/test/xxxdp already exists.
```
**Background**
The -fvar-tracking-assignments is automatically enabled by GCC when optimizations are enabled.
There is a limit on how many variables can be tracked by the compiler. The warning tells you that more vars would need to be tracked than what's supported.
Therefore, I added a corresponding try-catch around my call:
You can disable the tracking manually with -fno-var-tracking-assignments.
```
try:
...
except CiiOldbPyB.CiiOldbDpExistsException as e:
```
There are two easy ways to do it on the overall project.
When I run it, the try-catch doesn't work.
**Solution A**
Moreover, I now get two backtraces:
```
Export CXXFLAGS=-fno-var-tracking-assignments
Top Level Unexpected exception:
Traceback (most recent call last):
File "/home/eltdev/MODULES/test/app.py", line 91, in instantiateDP
double_dp = self.oldb_client.create_data_point(uri, metadataInstName)
CiiOldbPyB.CiiOldbDpExistsException: The Data point cii.oldb:/root/test/xxxdp already exists.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/eltdev/MODULES/test/app.py", line 108, in main
oldbCreator.instantiateOLDB_exception()
File "/home/eltdev/MODULES/test/app.py", line 81, in instantiateOLDB_exception
self.instantiateDP(double_dp_uri, double_dp_meta.get_instance_name())
File "/home/eltdev/MODULES/test/app.py", line 94, in instantiateDP
except CiiOldbPyB.CiiOldbDpExistsException:
NameError: name 'CiiOldbPyB' is not defined
```
And then rerun “waf configure” and continue with the build.
**Solution**
Note: you have to have the exported variable each time you do a “waf configure” as that is the point at which such flags are saved
You were mislead by the first backtrace: the exception name in the backtrace is not what you should catch.
**Solution B**
In your code, replace "**CiiOldbPyB**" with "**elt.oldb**":
If you want it to be inside the project then put the flags fixed inside your project, so in the top level wscript (where you define the project) add this configure section before the project declaration:
```
try:
....
except elt.oldb.CiiOldbDpExistsException as e:
```
```
def configure(cnf):
For completeness - do no forget this statement:
cnf.env.append_value('CXXFLAGS', \['-fno-var-tracking-assignments'\])
```
import elt.oldb
```
**Background**
The CII Python API is mostly a binding to the CII C++ API.
The CiiOldbPyB.CiiOldbDpExistsException is the original binding class.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### Multithreading in Java
Look at this document:
This binding class is re-exported under the name elt.oldb.CiiOldbDpExistsException.
- [Java JMM Visibility](Documents/Java-JMMVisibility.pdf)
The elt.oldb module internally loads the C++ binding module CiiOldbPyB. So both are the same exception.
Nonetheless, you should use the re-exported name, not the original name in your application. We discourage the use of the original name because the structure of the CiiOldbPyB module is more "chaotic" and not equivalent to elt.oldb.
Unfortunately, in the backtraces you will always see the original name instead of the re-exported name.
This question was originally asked in [ECII-422](https://jira.eso.org/browse/ECII-422).
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### Summary of OPC/UA MAL in C++
This article covers integration of OPC/UA in CII MAL specifically for OPC/UA Data Access and Subscription profiles. OPC/UA method invocation is also supported in CII MAL but is not described in this article, likewise details of the Python (and Java) support are not provided. Only C++ is considered here.
OPC/UA communication middleware is exposed in CII MAL as either Publish/Subscribe or Request/Reply APIs.
The XML ICD definition of types in CII is used to map sets of data points together that are read/written as a group.
Each attribute in the defined type is connected to a corresponding data point in the OPC/UA data space via a URI. Thus the CII URI for a complex type will contain specific addresses of multiple nodes in the OPC/UA data space.
### ------------------------------------------------------
### [IntCfg]
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
#### Pub/Sub API for OPC/UA Clients:
The CII MAL Pub/Sub API utilizes OPC/UA Data Access Reads and Writes, as well OPC/UA subscription. A Publisher will directly trigger an OPC DA write, while a Subscriber will work in one of two ways, depending on the type associated with the subscriber:
- If the subscribers CII URI contains only a single node (i.e. the XML ICD type contained only a single attribute) then the Subscriber will create an OPC/UA subscription on that data point. The subscription will trigger notification of updates to the data point node, which will then be queued for notification via the CII Subscriber API.
- If the Subscriber is using subscription, the opc.ps.outstandingPublishRequests property should not be zero (e.g. set it to 5), see the example code below. The reason is that the publish queue is used to store and send subscription notification events, and if the queue is small the even notifications may simply be dropped.
### config not found on remote db [IntCfg]
- If the subscriber CII URI contains multiple nodes (i.e. the XML ICD type contains multiple attributes) then the Subscriber launches a thread to perform periodic polling of data from the OPC/UA server. The rate is based on the properties passed in creating the subscriber. e.g.
**Problem**
```java
try {
subscriber = factory.getSubscriber\<T>(opcua_uri, ::elt::mal::ps::qos::QoS::DEFAULT,
You are intending to read a config from the local config database ("localdb"), but you see an error message like this.
{{"opc.ps.outstandingPublishRequests","5"},{"opc.asyncLoopExecutionPeriodMs","50"},
{"opc.asyncCallSubmitTimeoutMs","1000"},
{"opc.ps.pollingPeriodMs","20000"},
{"opc.asyncCallRetryPeriodMs","250"}});
}
catch(...) {
throw;
}
```
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
#### Request/Reply API for OPC/UA Clients:
As OPC/UA Data Access read and write essentially follow a synchronous request/reply pattern, CII MAL also provides this interface for OPC/UA clients.
The ICD is termed "virtual" in CII nomenclature as it does not require definition as an XML ICD using the service syntax, rather the same types defined for the Pub/Sub API may be used with a CII MAL OPC/UA Request/Reply.
This approach means OPC/UA Data Access read (and write) are synchronous, and may be called as needed by the application.
```cpp
namespace mal {
namespace rr {
namespace da {
class DataAccess : public ::elt::mal::rr::RrEntity {
public:
[...]
template <typename T>
void read(::elt::mal::ps::DataEntity\<T>& value) {
readUnsafe(&value);
}
template <typename T>
void write(const ::elt::mal::ps::DataEntity\<T>& value) {
writeUnsafe(&value);
}
elt.config.exceptions.CiiConfigNoTcException: Target configuration does not exist: Failed to retrieve configuration from elastic search: Configuration cii.config://*/supervisoryapp/TrkLsvDeploy on the remote db was not found
at elt.config.client.ConfigRemoteDatabase.retrieveConfig(ConfigRemoteDatabase.java:191)
at elt.config.client.CiiConfigClient.retrieveConfig(CiiConfigClient.java:354)
at elt.config.client.CiiConfigClient.retrieveConfig(CiiConfigClient.java:310)
at trkLsv.DataContext.loadConfig(DataContext.java:324)
at trkLsv.DataContext.<init>(DataContext.java:190)
at trkLsv.TrkLsv.go(TrkLsv.java:72)
at trkLsv.TrkLsv.main(TrkLsv.java:41)
Caused by: elt.error.icd.CiiSerializableException
at elt.config.service.client.icd.zpb.ServiceClientApiInterfaceAsyncImpl.processRequest(ServiceClientApiInterfaceAsyncImpl.java:73)
at elt.mal.zpb.rr.ClientAsyncImpl.events(ClientAsyncImpl.java:261)
at org.zeromq.ZPoller.dispatch(ZPoller.java:537)
at org.zeromq.ZPoller.poll(ZPoller.java:488)
at org.zeromq.ZPoller.poll(ZPoller.java:461)
at elt.mal.zpb.ZpbMal.processThread(ZpbMal.java:459)
at elt.mal.zpb.ZpbMal.lambda$new$0(ZpbMal.java:119)
at java.lang.Thread.run(Thread.java:748)
```
**Solution A**
A test application showing its use is here:
Your local file may be invalid (e.g. illegal format, or doesn't match the config class definition).
<https://gitlab.eso.org/cosylab/elt-cii/mal/mal-test/-/blob/develop/cpp/mal-test-performance/opcua/mal-opcua-da-speed/src/common.cpp>
Look at the content of your local config database, e.g. with
```
$ find $INTROOT/localdb
```
and correct the file in place, or fix the source yaml and then redeploy it from source to the localdb.
**Background**
--------------------------------------------------------------------------
--------------------------------------------------------------------------
--------------------------------------------------------------------------
### Exception connecting to config service
You may have a malformed json file in your local db, which the config service failed to read.
When starting an application using the Config client API and the following exception is received:
```
<date/time>, ERROR, CiiOldbFactory/140635105143296, Unexpected config exception occurred while retrieving configuration for cii.config://remote/oldb/configurations/oldbClientConfig What:Path oldb/configurations/oldbClientConfig, version -1 not found
terminate called after throwing an instance of 'elt::oldb::CiiOldbException'
what(): Unexpected config exception occurred while retrieving configuration for cii.config://remote/oldb/configurations/oldbClientConfig What:Path oldb/configurations/oldbClientConfig, version -1 not found
```
Because of the use the location wildcard "\*" in your code (in "cii.config://\*/supervisoryapp/TrkLsvDeploy"),
*info*
This can indicate that elasticSearch on the config/oldb server is not running, or has crashed.
Use cii-services command to check the status on the server where the config is running.
the config service has consequently tried to load the config from the remote config database, where no such config exists.
To that end, the error message is misleading, and should be improved (ticket [ECII-208](https://jira.eso.org/browse/ECII-208)).