Skip to content

Changes

Page history
More cleanup of md from OneNote authored by Gianluca Chiozzi's avatar Gianluca Chiozzi
Hide whitespace changes
Inline Side-by-side
KnowledgeBase/CII.md
View page @ 322f3dc3
......@@ -26,61 +26,38 @@ declare_project(name='mytests',
recurse='myconfig mylsvsim icd tests')
```
### Mock OLDB for unit tests \[cpp\]
**Question**
Is there already a faked OLDB that I can use in my unit tests in cpp?
**Answer**
(by D. Kumar)
You can create an in-memory OLDB providing a cached config oldb implementation and using the local file system for blob data.
The oldb-client cpp module is providing a
- pure (virtual = 0) interface elt::oldb::CiiOldbDataPointProvider<sup>\[1\]</sup>, and two implementations:
- in-memory data point provider storing data points to the memory
> (this is an empty implementation which provides a minimal operational fake oldb)
(this is an empty implementation which provides a minimal operational fake oldb)
- a redis data point provider storing data points to redis.
<!-- -->
- a remote filesystem interface elt::oldb:impl::ciiOldbRemoteFileProvider.hpp<sup>\[2\]</sup>, and two implementations:
- S3 implementation
- local file system implementation: ciiOldbLocalFileProvider<sup>\[3\]</sup> *\[Note: not before DevEnv 3.4\]*
Here are complete examples of unit tests showing the main use cases how to use oldb (with subscriptions) and metadata creation:
<https://gitlab.eso.org/cii/srv/cii-srv/-/blob/master/oldb-client/cpp/oldb/test/oldbInMemoryTest.cpp>
References
\[1\] <https://gitlab.eso.org/cii/srv/cii-srv/-/blob/master/oldb-client/cpp/oldb/src/include/ciiOldbDataPointProvider.hpp>
......@@ -89,91 +66,78 @@ References
\[3\] <https://gitlab.eso.org/cii/srv/cii-srv/-/blob/master/oldb-client/cpp/oldb/src/provider/ciiOldbLocalFileProvider.hpp> *\[Note: not before DevEnv 3.4\]*
NOTE: For python, a MR was created to add the feature: <https://gitlab.eso.org/cii/srv/cii-srv/-/merge_requests/11>
Example in <https://gitlab.eso.org/ahoffsta/cii-srv/-/blob/oldb-in-memory-missing-python-binding/oldb-client/python/oldb/test/oldbInMemoryTest.py>
### Catching API Exceptions \[Python\]
**Problem**
My application contains a call to the CII Python API.
When I ran it, it threw an exception with the following backtrace:
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>Top Level Unexpected exception:</p><p>Traceback (most recent call last):</p><p>File "/home/eltdev/MODULES/test/app.py", line 91, in instantiateDP</p><p>double_dp = self.oldb_client.create_data_point(uri, metadataInstName)</p><p><strong>CiiOldbPyB.CiiOldbDpExistsException</strong>: The Data point cii.oldb:/root/test/xxxdp already exists.</p></th></tr></thead><tbody></tbody></table>
```
Top Level Unexpected exception:
Traceback (most recent call last):
File "/home/eltdev/MODULES/test/app.py", line 91, in instantiateDP
double_dp = self.oldb_client.create_data_point(uri, metadataInstName)
CiiOldbPyB.CiiOldbDpExistsException: The Data point cii.oldb:/root/test/xxxdp already exists.
```
Therefore, I added a corresponding try-catch around my call:
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>try:</p><p>...</p><p>except <strong>CiiOldbPyB.CiiOldbDpExistsException</strong> as e:</p></th></tr></thead><tbody></tbody></table>
```
try:
...
except CiiOldbPyB.CiiOldbDpExistsException as e:
```
When I run it, the try-catch doesn't work.
Moreover, I now get two backtraces:
```
Top Level Unexpected exception:
Traceback (most recent call last):
File "/home/eltdev/MODULES/test/app.py", line 91, in instantiateDP
double_dp = self.oldb_client.create_data_point(uri, metadataInstName)
CiiOldbPyB.CiiOldbDpExistsException: The Data point cii.oldb:/root/test/xxxdp already exists.
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>Top Level Unexpected exception:</p><p>Traceback (most recent call last):</p><p>File "/home/eltdev/MODULES/test/app.py", line 91, in instantiateDP</p><p>double_dp = self.oldb_client.create_data_point(uri, metadataInstName)</p><p><strong>CiiOldbPyB.CiiOldbDpExistsException</strong>: The Data point cii.oldb:/root/test/xxxdp already exists.</p><p> </p><p><strong>During handling of the above exception, another exception occurred:</strong></p><p>Traceback (most recent call last):</p><p>File "/home/eltdev/MODULES/test/app.py", line 108, in main</p><p>oldbCreator.instantiateOLDB_exception()</p><p>File "/home/eltdev/MODULES/test/app.py", line 81, in instantiateOLDB_exception</p><p>self.instantiateDP(double_dp_uri, double_dp_meta.get_instance_name())</p><p>File "/home/eltdev/MODULES/test/app.py", line 94, in instantiateDP</p><p>except CiiOldbPyB.CiiOldbDpExistsException:</p><p><strong>NameError: name 'CiiOldbPyB' is not defined</strong></p></th></tr></thead><tbody></tbody></table>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/eltdev/MODULES/test/app.py", line 108, in main
oldbCreator.instantiateOLDB_exception()
File "/home/eltdev/MODULES/test/app.py", line 81, in instantiateOLDB_exception
self.instantiateDP(double_dp_uri, double_dp_meta.get_instance_name())
File "/home/eltdev/MODULES/test/app.py", line 94, in instantiateDP
except CiiOldbPyB.CiiOldbDpExistsException:
NameError: name 'CiiOldbPyB' is not defined
```
**Solution**
You were mislead by the first backtrace -
the exception name in the backtrace is not what you should catch.
You were mislead by the first backtrace: the exception name in the backtrace is not what you should catch.
In your code, replace "**CiiOldbPyB**" with "**elt.oldb**":
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>try:</p><p>....</p><p>except <strong>elt.oldb.CiiOldbDpExistsException</strong> as e:</p></th></tr></thead><tbody></tbody></table>
> For completeness - do no forget this statement:
```
try:
....
except elt.oldb.CiiOldbDpExistsException as e:
```
| import elt.oldb |
|-----------------|
For completeness - do no forget this statement:
```
import elt.oldb
```
**Background**
The CII Python API is mostly a binding to the CII C++ API.
The CiiOldbPyB.CiiOldbDpExistsException is the original binding class.
This binding class is re-exported under the name elt.oldb.CiiOldbDpExistsException.
......@@ -182,246 +146,157 @@ The elt.oldb module internally loads the C++ binding module CiiOldbPyB. So both
Nonetheless, you should use the re-exported name, not the original name in your application. We discourage the use of the original name because the structure of the CiiOldbPyB module is more "chaotic" and not equivalent to elt.oldb.
Unfortunately, in the backtraces you will always see the original name instead of the re-exported name.
This question was originally asked in [ECII-422](https://jira.eso.org/browse/ECII-422).
### access_key empty (DevEnv 3.2.0) \[OLDB\]
**Problem**
Trying to use the OLDB on DevEnv 3.2.0, I'm getting this error:
Unexpected exception occurred. What:Configuration invalid: access_key empty
**Solution**
Run the following commands (you will be asked for the root pw):
```
wget -q www.eso.org/~mschilli/download/cii/postinstall/cii-postinstall-20210610
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>wget -q <a href="https://www.eso.org/~mschilli/download/cii/postinstall/cii-postinstall-20210610">www.eso.org/~mschilli/download/cii/postinstall/cii-postinstall-20210610</a></p><p> </p><p>cii-services stop config</p><p> </p><p>su -c "bash cii-postinstall-20210610 schemas"</p><p> </p><p># If the script</p><p> </p><p>#You should see the following output:</p><table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>Password:</p><p>CII PostInstall (20210610)</p><p>schemas: applying fix ECII397</p><p>/home/eltdev/</p><p>schemas: populating elasticsearch</p><p>schemas: skipping telemetry</p><p>schemas: skipping alarms</p></th></tr></thead><tbody></tbody></table><p> </p><p>cii-services start config</p><p> </p></th></tr></thead><tbody></tbody></table>
cii-services stop config
su -c "bash cii-postinstall-20210610 schemas"
# If the script
#You should see the following output:
Password:
CII PostInstall (20210610)
schemas: applying fix ECII397
/home/eltdev/
schemas: populating elasticsearch
schemas: skipping telemetry
schemas: skipping alarms
cii-services start config
```
**Background**
The OLDB settings coming with 3.2.0 are buggy.
The CII post-install procedure is able to hotfix the settings (ECII397).
The problem will be fixed in DevEnv 3.4.
### Datapoint already exists \[OLDB\]
**Problem**
My application tries to create an OLDB datapoint.
This fails because the datapoint "already exists":
```
2021-06-22T11:01:33.003+0000, ERROR, CiiOldbRedisDataPointProvider/140709706681216, Data point uri: cii.oldb:/tcs/hb/tempser3 in Redis already exists.
```
> 2021-06-22T11:01:33.003+0000, ERROR, CiiOldbRedisDataPointProvider/140709706681216, Data point uri: cii.oldb:/tcs/hb/tempser3 in Redis already exists.
In response, my application skips the creation step, and wants to use the reportedly existing datapoint.
However, when doing this, I get the error "Datapoint doesn't exist".
Likewise, when I run the oldb-gui database browser, it does not show this data point in the OLDB.
**Variant 2 of the Problem**
I try to access an OLDB datapoint, and I see two errors like this:
```
Target configuration does not exist: Failed to retrieve configuration from elastic search: Configuration
\[…\]
[…]
elt.oldb.exceptions.CiiOldbDpExistsException: Data point cii.oldb:/alarm/alarm/device/motor/input_int_dp_alarm already exisits.
```
Go directly to Solution 2 below.
**Variant 3 of the Problem**
I try to delete an OLDB datapoint and I see an error like this:
> CiiOldbPyB.CiiOldbException: De-serialization error:sizeof(T)\*count is greater then remaining
```
CiiOldbPyB.CiiOldbException: De-serialization error:sizeof(T)\*count is greater then remaining
```
Go directly to Solution 2 below.
**Background**
The two errors are contradicting.
Datapoints are stored in two databases: a document-database (permanent store) for its metadata, and a key-value-database (volatile store) for its current value. The above symptoms indicate that the two databases are out-of-sync, meaning the datapoint exists only "half".
**Solution 1**
There is a good chance you can delete the datapoint to clean up the situation (as of [ECII-303](https://jira.eso.org/browse/ECII-303)):
```
#!/usr/bin/env python
import elt.config
import elt.oldb
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>#!/usr/bin/env python</p><p>import elt.config</p><p>import elt.oldb</p><p> </p><p>oldb_client = elt.oldb.CiiOldbFactory.get_instance()</p><p>elt.oldb.CiiOldbGlobal.set_write_enabled(True)</p><p> </p><p>uri = elt.config.Uri("cii.oldb:/tcs/hb/tempser3")</p><p>oldb_client.delete_data_point(uri)</p><p> </p></th></tr></thead><tbody></tbody></table>
oldb_client = elt.oldb.CiiOldbFactory.get_instance()
elt.oldb.CiiOldbGlobal.set_write_enabled(True)
uri = elt.config.Uri("cii.oldb:/tcs/hb/tempser3")
oldb_client.delete_data_point(uri)
```
**Solution 2**
If the above didn't help, find out which "half" of the datapoint exists:
1. The current value exists, and the metadata is missing. This is the case when upgrading DevEnv/CII without deleting the Redis cache.
>
>
> Run the following command:
>
> Note: enter a search-expression matching the datapoint URI, in the example this is "tempser3".
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p># confirm that this is the cause of the problem</p><p>redis-cli KEYS *tempser3*</p></th></tr></thead><tbody></tbody></table>
Run the following command:
>
>
> If the command is showing the problematic key,
>
> we run a similar command to remove that key from the store:
Note: enter a search-expression matching the datapoint URI, in the example this is "tempser3".
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p># if so, delete the offending datapoint</p><p>redis-cli --scan --pattern *tempser3* | xargs redis-cli del</p></th></tr></thead><tbody></tbody></table>
```
# confirm that this is the cause of the problem
redis-cli KEYS *tempser3*
```
>
If the command is showing the problematic key, we run a similar command to remove that key from the store:
```
# if so, delete the offending datapoint
redis-cli --scan --pattern *tempser3* | xargs redis-cli del
```
2. The metadata exists, and the current value is missing
> Repair the current value:
| |
|-----|
> Update: As of [ECII-303](https://jira.eso.org/browse/ECII-303), this should no longer be necessary, since the OLDB will make an attempt to automatically re-instate the current value that is missing.
Update: As of [ECII-303](https://jira.eso.org/browse/ECII-303), this should no longer be necessary, since the OLDB will make an attempt to automatically re-instate the current value that is missing.
**Solution 3**
If none of the above helped, another possibility is to clean-up the metadata.
This is a potentially very invasive operation. Please contact us for instructions.
### Req/Rep Connection Listeners \[MAL Python\]
**Problem**
MAL includes a callback registration method to allow monitoring of the MAL service connection status.
This is done with the method "*registerConnectionListener()*".
This is done with the method `registerConnectionListener()`.
Given the example below, this does not work, and the "*listenerMethod()"* is never called.
Given the example below, this does not work, and the `listenerMethod()` is never called.
```
......@@ -464,220 +339,155 @@ rtn.wait()
print( str(rtn.get() ))
```
*From \<<https://jira.eso.org/browse/ECII-212>\>*
**Solution**
The developer must keep a reference to the returned Object from the "*registerConnectionListener()*" invocation.
The developer must keep a reference to the returned Object from the `registerConnectionListener()` invocation.
```
stdcmds = factory.getClient(uri, StdCmdsAsync, qos=mal.rr.qos.ReplyTime(THREE_SECONDS))
listenerRegistration = stdcmds.registerConnectionListener(listenerMethod)
```
**Background**
The documentation states a Return value, but is the responsibility of the developer to keep the reference. Otherwise, the object will be deleted when exiting the block of code.
Remember to delete (assign None), to this object when closing the connection to the MAL service.
*From \<<https://jira.eso.org/browse/ECII-212>\>*
### Command Line Tools and Python snippets \[OLDB\]
**Problem**
I need to inspect or modify the content of the OLDB from the command line or a shell script.
**Solution**
**cii-oldb-traversal-tool**
> for searching through the OLDB
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>$ cii-oldb-traversal-tool --file output --quality OK</p><p>$ cat output</p><p>cii.oldb:///root/trklsv/cfg/log/level|OK|WARNING|2020-09-11T15:25:08Z</p><p>cii.oldb:///root/trklsv/cfg/req/endpoint|OK|zpb.rr://localhost:44444/m1/TrkLsvServer|2020-09-11T15:25:08Z</p><p>cii.oldb:///root/trklsv/ctr/current/altaz/alt|OK|0.000000|2020-09-11T15:24:25Z</p><p>cii.oldb:///root/trklsv/ctr/current/altaz/az|OK|0.000000|2020-09-11T15:24:25Z</p><p>cii.oldb:///root/trklsv/ctr/current/radec/dec|OK|0.000000|2020-09-11T15:24:27Z</p><p>cii.oldb:///root/trklsv/ctr/current/radec/ra|OK|0.000000|2020-09-11T15:24:27Z</p><p>cii.oldb:///root/trklsv/ctr/poserr|OK|0.000000|2020-09-11T15:24:27Z</p><p>cii.oldb:///root/trklsv/ctr/status|OK|UNKNOWN|2020-09-11T15:23:55Z</p><p>cii.oldb:///root/trklsv/ctr/substate|OK|UNKNOWN|2020-09-11T15:23:49Z</p><p>cii.oldb:///root/trklsv/ctr/target/altaz/alt|OK|0.000000|2020-09-11T15:24:24Z</p><p>cii.oldb:///root/trklsv/ctr/target/altaz/az|OK|0.000000|2020-09-11T15:24:25Z</p><p>cii.oldb:///root/trklsv/ctr/target/radec/dec|OK|0.000000|2020-09-11T15:24:26Z</p><p>cii.oldb:///root/trklsv/ctr/target/radec/ra|OK|0.000000|2020-09-11T15:24:26Z</p></th></tr></thead><tbody></tbody></table>
**oldb-cli**
> for reading, writing, subscribing to an OLDB-datapoint
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>$ oldb-cli read cii.oldb:///root/trklsv/cfg/req/endpoint</p><p>SLF4J: Class path contains multiple SLF4J bindings.</p><p>SLF4J: Found binding in [jar:file:/eelt/ciisrv/1.0-RC3-20201030/lib/srv-support-libs/slf4j-nop-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]</p><p>SLF4J: Found binding in [jar:file:/eelt/mal/1.1.0-2.2.3-20201027/lib/mal-opcua/slf4j-nop-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]</p><p>SLF4J: See <a href="http://www.slf4j.org/codes.html#multiple_bindings">http://www.slf4j.org/codes.html#multiple_bindings</a> for an explanation.</p><p>SLF4J: Actual binding is of type [org.slf4j.helpers.NOPLoggerFactory]</p><p>log4j:WARN No appenders could be found for logger (io.netty.util.internal.logging.InternalLoggerFactory).</p><p>log4j:WARN Please initialize the log4j system properly.</p><p>Timestamp: 2020-09-11T15:25:08.648Z</p><p>Quality: OK</p><p>Value: zpb.rr://localhost:44444/m1/TrkLsvServer</p></th></tr></thead><tbody></tbody></table>
**oldb Python API**
**cii-oldb-traversal-tool** for searching through the OLDB
```
$ cii-oldb-traversal-tool --file output --quality OK
$ cat output
cii.oldb:///root/trklsv/cfg/log/level|OK|WARNING|2020-09-11T15:25:08Z
cii.oldb:///root/trklsv/cfg/req/endpoint|OK|zpb.rr://localhost:44444/m1/TrkLsvServer|2020-09-11T15:25:08Z
cii.oldb:///root/trklsv/ctr/current/altaz/alt|OK|0.000000|2020-09-11T15:24:25Z
cii.oldb:///root/trklsv/ctr/current/altaz/az|OK|0.000000|2020-09-11T15:24:25Z
cii.oldb:///root/trklsv/ctr/current/radec/dec|OK|0.000000|2020-09-11T15:24:27Z
cii.oldb:///root/trklsv/ctr/current/radec/ra|OK|0.000000|2020-09-11T15:24:27Z
cii.oldb:///root/trklsv/ctr/poserr|OK|0.000000|2020-09-11T15:24:27Z
cii.oldb:///root/trklsv/ctr/status|OK|UNKNOWN|2020-09-11T15:23:55Z
cii.oldb:///root/trklsv/ctr/substate|OK|UNKNOWN|2020-09-11T15:23:49Z
cii.oldb:///root/trklsv/ctr/target/altaz/alt|OK|0.000000|2020-09-11T15:24:24Z
cii.oldb:///root/trklsv/ctr/target/altaz/az|OK|0.000000|2020-09-11T15:24:25Z
cii.oldb:///root/trklsv/ctr/target/radec/dec|OK|0.000000|2020-09-11T15:24:26Z
cii.oldb:///root/trklsv/ctr/target/radec/ra|OK|0.000000|2020-09-11T15:24:26Z
```
**oldb-cli** for reading, writing, subscribing to an OLDB-datapoint
```
$ oldb-cli read cii.oldb:///root/trklsv/cfg/req/endpoint
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/eelt/ciisrv/1.0-RC3-20201030/lib/srv-support-libs/slf4j-nop-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/eelt/mal/1.1.0-2.2.3-20201027/lib/mal-opcua/slf4j-nop-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.helpers.NOPLoggerFactory]
log4j:WARN No appenders could be found for logger (io.netty.util.internal.logging.InternalLoggerFactory).
log4j:WARN Please initialize the log4j system properly.
Timestamp: 2020-09-11T15:25:08.648Z
Quality: OK
Value: zpb.rr://localhost:44444/m1/TrkLsvServer
> for creating, deleting an OLDB-datapoint
```
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>$ python<br />
Python 3.7.6 (default, Jan 8 2020, 19:59:22)</p><p>[GCC 7.3.0] :: Anaconda, Inc. on linux</p><p>Type "help", "copyright", "credits" or "license" for more information.</p><p>&gt;&gt;&gt;</p><p>&gt;&gt;&gt; import elt.oldb</p><p>&gt;&gt;&gt; from elt.config import Uri</p><p>&gt;&gt;&gt; oldb = elt.oldb.CiiOldbFactory.get_instance()</p></th></tr></thead><tbody></tbody></table>
**oldb Python API** for creating, deleting an OLDB-datapoint
```
$ python Python 3.7.6 (default, Jan 8 2020, 19:59:22)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import elt.oldb
>>> from elt.config import Uri
>>> oldb = elt.oldb.CiiOldbFactory.get_instance()
```
… and, to create a datapoint:
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>&gt;&gt;&gt; elt.oldb.CiiOldbGlobal.set_write_enabled(True)</p><p>&gt;&gt;&gt; oldb.create_data_point_by_value (Uri("cii.oldb:/ccs/tst/tmp1"), "my text")</p></th></tr></thead><tbody></tbody></table>
```
>>> elt.oldb.CiiOldbGlobal.set_write_enabled(True)
>>> oldb.create_data_point_by_value (Uri("cii.oldb:/ccs/tst/tmp1"), "my text")
```
… and, to delete a datapoint:
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>&gt;&gt;&gt; elt.oldb.CiiOldbGlobal.set_write_enabled(True)</p><p>&gt;&gt;&gt; oldb.delete_data_point (Uri("cii.oldb:/ccs/tst/tmp1"))</p></th></tr></thead><tbody></tbody></table>
```
>>> elt.oldb.CiiOldbGlobal.set_write_enabled(True)
>>> oldb.create_data_point_by_value (Uri("cii.oldb:/ccs/tst/tmp1"), "my text")
```
… and, to read a datapoint:
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>&gt;&gt;&gt; oldb.get_data_point(Uri("cii.oldb:/ccs/tst/tmp1")).read_value().get_value()</p><p>'my text'</p></th></tr></thead><tbody></tbody></table>
>
```
>>> oldb.get_data_point(Uri("cii.oldb:/ccs/tst/tmp1")).read_value().get_value()
'my text'
```
### Latency on Pub/Sub \[MAL ZMQ\]
**Problem**
We see latencies of 400-1000ms in MAL ZMQ pub/sub communication, for sending a 12k x 12k image blob.
For the 2 first transmissions, there is somewhere between 400 and 500ms latency on the publisher side between just before calling the "publish" method and when we see the first packets on the wire.
This is for all messages, not only for the first messages. When we let the program run several minutes, during that period, all messages have a consistent delay when arriving on subscriber side.
**Solution**
From MAL 1.0.4 on,
MAL supports the below MAL specific property to limit queue size for large message publishers.
```
mal::Mal::Properties m_malProperties1;
m_malProperties1["zpb.ps.zmq.sndhwm"] = "1";
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th>mal::Mal::Properties m_malProperties1;<br />
m_malProperties1["zpb.ps.zmq.sndhwm"] = "1";<br />
<br />
auto publisher = factory.getPublisher&lt;mal::example::Sample&gt;(uri,<br />
{ std::make_shared&lt;mal::ps::qos::Latency&gt;(<br />
std::chrono::milliseconds(100)),<br />
std::make_shared&lt;mal::ps::qos::Deadline&gt;(<br />
std::chrono::seconds(1)) }, m_malProperties1);</th></tr></thead><tbody></tbody></table>
auto publisher = factory.getPublisher<mal::example::Sample>(uri,
{ std::make_shared<mal::ps::qos::Latency>(
std::chrono::milliseconds(100)),
std::make_shared<mal::ps::qos::Deadline>(
std::chrono::seconds(1)) }, m_malProperties1);
```
For more information how to use MAL specific properties, see MAL Binding Manual.
**Background**
The issue has been first reported in [ECII-159](https://jira.eso.org/browse/ECII-159).
The problem lies in the ZMQ send queues. Default size is 1000 and with 144MB per message (in this case) this means 144GB. Limiting this to 1 (for a test) a publisher can handle 20 subscribers (tested) without any problems. The solution is to reconfigure the send-queue size appropriately.
### Choose middlewares to build \[MAL ICD\]
**Problem**
I am certain that my ICD will never be used over OPC UA. Nonetheless, the ICD-compilation builds OPC UA mappings for my ICD. This is unnecessarily extending the compilation time for my application.
**Solution**
By default the ICD-compilation builds mappings for all middlewares. But it is possible to exclude certain middleware mappings from compilation, which will reduce compilation time. You do this by passing mal options to the icd-generator.
**Example**
wscript
| declare_malicd(use='icds.base', mal_opts = { 'opcua_disabled': True } ) |
|-------------------------------------------------------------------------|
```
declare_malicd(use='icds.base', mal_opts = { 'opcua_disabled': True } )
```
The available options are:
......@@ -687,200 +497,156 @@ The available options are:
- zpb_disabled = if True, disable ZEROMQ middleware generation
>
### Sending an array of unions \[MAL cpp\]
**Problem**
Trying to send a msg, I get the following error message from CII:
```
[libprotobuf ERROR google/protobuf/message_lite.cc:121] Can't parse message of type "generated.zpb.fcfif.StdCmds_Request" because it is missing required fields: data.Setup.payload[0].piezoData.input
```
\[libprotobuf ERROR google/protobuf/message_lite.cc:121\] Can't parse message of type "generated.zpb.fcfif.StdCmds_Request" because it is missing required fields: data.Setup.payload\[0\].piezoData.input
My ICD definition looks like this
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>&lt;enum name="PiezoInput"&gt;<br />
&lt;enumerator name="PIEZO_INPUT_SETMODE" /&gt;<br />
&lt;enumerator name="PIEZO_INPUT_MOVE" /&gt;<br />
&lt;/enum&gt;</p><p> </p><p>&lt;enum name="PiezoMode"&gt;<br />
&lt;enumerator name="PIEZO_MODE_1" /&gt;<br />
&lt;enumerator name="PIEZO_MODE_2" /&gt;<br />
&lt;/enum&gt;</p><p> </p><p>&lt;struct name="PiezoModeStruct"&gt;<br />
&lt;member name="mode" type="nonBasic" nonBasicTypeName="PiezoMode" /&gt;<br />
&lt;/struct&gt;</p><p> </p><p>&lt;enum name="PiezoMove"&gt;<br />
&lt;enumerator name="PIEZO_MOVE_1" /&gt;<br />
&lt;enumerator name="PIEZO_MOVE_2" /&gt;<br />
&lt;/enum&gt;<br />
<br />
&lt;union name="PiezoUnion"&gt;<br />
&lt;discriminator type="nonBasic" nonBasicTypeName="PiezoInput" /&gt;<br />
&lt;case&gt;<br />
&lt;caseDiscriminator value ="PIEZO_INPUT_SETMODE"/&gt;<br />
&lt;member name="piezoModeData" type="nonBasic" nonBasicTypeName="PiezoModeStruct" /&gt;<br />
&lt;/case&gt;<br />
&lt;case&gt;<br />
&lt;caseDiscriminator value ="PIEZO_INPUT_MOVE"/&gt;<br />
&lt;member name="piezoMoveData" type="nonBasic" nonBasicTypeName="PiezoMove" /&gt;<br />
&lt;/case&gt;<br />
&lt;/union&gt;<br />
<br />
&lt;struct name="Piezo"&gt;<br />
&lt;member name="id" type="string" /&gt;<br />
&lt;member name="input" type="nonBasic" nonBasicTypeName="PiezoUnion" /&gt;<br />
&lt;/struct&gt;</p><p> </p><p>&lt;interface name="PiezoTest"&gt;<br />
&lt;method name="test" returnType="void"&gt;<br />
&lt;argument name="arr" type="nonBasic" nonBasicTypeName="Piezo" arrayDimensions="(10)" /&gt;<br />
&lt;/method&gt;<br />
&lt;/interface&gt;</p><p> </p></th></tr></thead><tbody></tbody></table>
```
<enum name="PiezoInput">
<enumerator name="PIEZO_INPUT_SETMODE" />
<enumerator name="PIEZO_INPUT_MOVE" />
</enum>
<enum name="PiezoMode">
<enumerator name="PIEZO_MODE_1" />
<enumerator name="PIEZO_MODE_2" />
</enum>
<struct name="PiezoModeStruct">
<member name="mode" type="nonBasic" nonBasicTypeName="PiezoMode" />
</struct>
<enum name="PiezoMove">
<enumerator name="PIEZO_MOVE_1" />
<enumerator name="PIEZO_MOVE_2" />
</enum>
<union name="PiezoUnion">
<discriminator type="nonBasic" nonBasicTypeName="PiezoInput" />
<case>
<caseDiscriminator value ="PIEZO_INPUT_SETMODE"/>
<member name="piezoModeData" type="nonBasic" nonBasicTypeName="PiezoModeStruct" />
</case>
<case>
<caseDiscriminator value ="PIEZO_INPUT_MOVE"/>
<member name="piezoMoveData" type="nonBasic" nonBasicTypeName="PiezoMove" />
</case>
</union>
<struct name="Piezo">
<member name="id" type="string" />
<member name="input" type="nonBasic" nonBasicTypeName="PiezoUnion" />
</struct>
<interface name="PiezoTest">
<method name="test" returnType="void">
<argument name="arr" type="nonBasic" nonBasicTypeName="Piezo" arrayDimensions="(10)" />
</method>
</interface>
```
My code looks like this
```
[...]
auto piezo = mal->createDataEntity<::fcfif::Piezo>();
piezo->setId("foo");
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>[...]</p><p>auto piezo = mal-&gt;createDataEntity&lt;::fcfif::Piezo&gt;();</p><p>piezo-&gt;setId("foo");</p><p> </p><p>auto input = piezo-&gt;getInput();</p><p>auto mode = input-&gt;getPiezoModeData();</p><p>mode-&gt;setAction(::fcfif::ActionPiezoMode::SET_AUTO);</p><p> </p><p>auto union = mal-&gt;createDataEntity&lt;::fcfif::FcsUnion&gt;();</p><p>union-&gt;setPiezoData(piezo);</p><p> </p><p>[...]</p></th></tr></thead><tbody></tbody></table>
auto input = piezo->getInput();
auto mode = input->getPiezoModeData();
mode->setAction(::fcfif::ActionPiezoMode::SET_AUTO);
auto union = mal->createDataEntity<::fcfif::FcsUnion>();
union->setPiezoData(piezo);
[...]
```
**Solution**
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>auto p = factory.getClient&lt;::fcfif::PiezoTestSync&gt;(uri,<br />
{std::make_shared&lt;mal::rr::qos::ReplyTime&gt;<br />
(std::chrono::seconds(3))},<br />
{});</p><p>auto mal = p-&gt;getMal();</p></th></tr></thead><tbody></tbody></table>
```
auto p = factory.getClient<::fcfif::PiezoTestSync>(uri,
{std::make_shared<mal::rr::qos::ReplyTime>
(std::chrono::seconds(3))},
{});
auto mal = p->getMal();
auto piezo = mal->createDataEntity<::fcfif::Piezo>();
piezo->setId("foo");
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>auto piezo = mal-&gt;createDataEntity&lt;::fcfif::Piezo&gt;();<br />
piezo-&gt;setId("foo");</p><p>auto input = piezo-&gt;getInput();</p><p>auto piezoModeStruct = mal-&gt;createDataEntity&lt;::fcfif::PiezoModeStruct&gt;();<br />
piezoModeStruct-&gt;setMode(::fcfif::PiezoMode::PIEZO_MODE_1);</p><p> </p><p>input-&gt;setPiezoModeData(piezoModeStruct);</p></th></tr></thead><tbody></tbody></table>
auto input = piezo->getInput();
auto piezoModeStruct = mal->createDataEntity<::fcfif::PiezoModeStruct>();
piezoModeStruct->setMode(::fcfif::PiezoMode::PIEZO_MODE_1);
input->setPiezoModeData(piezoModeStruct);
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>auto piezo2 = mal-&gt;createDataEntity&lt;::fcfif::Piezo&gt;();<br />
piezo2-&gt;setId("foo2");</p><p> </p><p>auto input2 = piezo2-&gt;getInput();</p><p> </p><p>input2-&gt;setPiezoMoveData(::fcfif::PiezoMove::PIEZO_MOVE_1);</p></th></tr></thead><tbody></tbody></table>
auto piezo2 = mal->createDataEntity<::fcfif::Piezo>();
piezo2->setId("foo2");
auto input2 = piezo2->getInput();
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>std::vector&lt;std::shared_ptr&lt;::fcfif::Piezo&gt;&gt; sa;<br />
sa.push_back(piezo);<br />
sa.push_back(piezo2);</p><p>p-&gt;test(sa);</p></th></tr></thead><tbody></tbody></table>
input2->setPiezoMoveData(::fcfif::PiezoMove::PIEZO_MOVE_1);
std::vector<std::shared_ptr<::fcfif::Piezo>> sa;
sa.push_back(piezo);
sa.push_back(piezo2);
p->test(sa);
```
**Background**
Your code does not work since you do not use union instance provided by the parent structure. You need to obtain nested structure/union via accessors and do not try to created your own (detached) instance.
This issue was first described in [ECII-154](https://jira.eso.org/browse/ECII-154)
### config not found on remote db \[Conf\]
**Problem**
You are intending to read a config from the local config database ("localdb"), but you see an error message like this.
elt.config.exceptions.CiiConfigNoTcException: Target configuration does not exist: Failed to retrieve configuration from elastic search: Configuration cii.config://\*/supervisoryapp/TrkLsvDeploy on the remote db was not found
```
elt.config.exceptions.CiiConfigNoTcException: Target configuration does not exist: Failed to retrieve configuration from elastic search: Configuration cii.config://*/supervisoryapp/TrkLsvDeploy on the remote db was not found
at elt.config.client.ConfigRemoteDatabase.retrieveConfig(ConfigRemoteDatabase.java:191)
at elt.config.client.CiiConfigClient.retrieveConfig(CiiConfigClient.java:354)
at elt.config.client.CiiConfigClient.retrieveConfig(CiiConfigClient.java:310)
at trkLsv.DataContext.loadConfig(DataContext.java:324)
at trkLsv.DataContext.\<init>(DataContext.java:190)
at trkLsv.DataContext.<init>(DataContext.java:190)
at trkLsv.TrkLsv.go(TrkLsv.java:72)
at trkLsv.TrkLsv.main(TrkLsv.java:41)
Caused by: elt.error.icd.CiiSerializableException
at elt.config.service.client.icd.zpb.ServiceClientApiInterfaceAsyncImpl.processRequest(ServiceClientApiInterfaceAsyncImpl.java:73)
at elt.mal.zpb.rr.ClientAsyncImpl.events(ClientAsyncImpl.java:261)
at org.zeromq.ZPoller.dispatch(ZPoller.java:537)
at org.zeromq.ZPoller.poll(ZPoller.java:488)
at org.zeromq.ZPoller.poll(ZPoller.java:461)
at elt.mal.zpb.ZpbMal.processThread(ZpbMal.java:459)
at elt.mal.zpb.ZpbMal.lambda$new$0(ZpbMal.java:119)
at java.lang.Thread.run(Thread.java:748)
```
**Solution A**
Your local file may be invalid (e.g. illegal format, or doesn't match the config class definition).
Look at the content of your local config database, e.g. with
```
$ find $INTROOT/localdb
```
and correct the file in place, or fix the source yaml and then redeploy it from source to the localdb.
**Background**
You may have a malformed json file in your local db, which the config service failed to read.
Because of the use the location wildcard "\*" in your code (in "cii.config://\*/supervisoryapp/TrkLsvDeploy"),
......@@ -889,50 +655,22 @@ the config service has consequently tried to load the config from the remote con
To that end, the error message is misleading, and should be improved (ticket [ECII-208](https://jira.eso.org/browse/ECII-208)).
### Failed to send request, send queue full \[CII MAL\]
**Problem**
You are intending to update a config, but you get an IllegalStateException where the last line is
```
elt.mal.zpb.rr.ClientAsyncImpl:183
```
> throw new MalException("Failed to send request, send queue full");
throw `new MalException("Failed to send request, send queue full");`
**Solution**
Probably you have called close() on the CiiConfigClient instance somewhere, maybe also implicitly during a try-with-resource block.
### Getting More Logs \[MAL\]
**Problem**
......@@ -1635,7 +1373,9 @@ namespace da {
writeUnsafe(&value);
}
```
```
A test application showing its use is here:
......