|
|
|
[[_TOC_]]
|
|
|
|
|
|
|
|
### KB: Mock OLDB for unit tests \[cpp\]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Question**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Is there already a faked OLDB that I can use in my unit tests in cpp?
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Answer**
|
|
|
|
|
|
|
|
(by D. Kumar)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
You can create an in-memory OLDB providing a cached config oldb implementation and using the local file system for blob data.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The oldb-client cpp module is providing a
|
|
|
|
|
|
|
|
- pure (virtual = 0) interface elt::oldb::CiiOldbDataPointProvider<sup>\[1\]</sup>, and two implementations:
|
|
|
|
|
|
|
|
- in-memory data point provider storing data points to the memory
|
|
|
|
|
|
|
|
> (this is an empty implementation which provides a minimal operational fake oldb)
|
|
|
|
|
|
|
|
- a redis data point provider storing data points to redis.
|
|
|
|
|
|
|
|
<!-- -->
|
|
|
|
|
|
|
|
- a remote filesystem interface elt::oldb:impl::ciiOldbRemoteFileProvider.hpp<sup>\[2\]</sup>, and two implementations:
|
|
|
|
|
|
|
|
- S3 implementation
|
|
|
|
|
|
|
|
- local file system implementation: ciiOldbLocalFileProvider<sup>\[3\]</sup> *\[Note: not before DevEnv 3.4\]*
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Here are complete examples of unit tests showing the main use cases how to use oldb (with subscriptions) and metadata creation:
|
|
|
|
|
|
|
|
<https://gitlab.eso.org/cii/srv/cii-srv/-/blob/master/oldb-client/cpp/oldb/test/oldbInMemoryTest.cpp>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
References
|
|
|
|
|
|
|
|
\[1\] <https://gitlab.eso.org/cii/srv/cii-srv/-/blob/master/oldb-client/cpp/oldb/src/include/ciiOldbDataPointProvider.hpp>
|
|
|
|
|
|
|
|
\[2\] <https://gitlab.eso.org/cii/srv/cii-srv/-/blob/master/oldb-client/cpp/oldb/src/include/provider/ciiOldbRemoteFileProvider.hpp>
|
|
|
|
|
|
|
|
\[3\] <https://gitlab.eso.org/cii/srv/cii-srv/-/blob/master/oldb-client/cpp/oldb/src/provider/ciiOldbLocalFileProvider.hpp> *\[Note: not before DevEnv 3.4\]*
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
NOTE: For python, a MR was created to add the feature: <https://gitlab.eso.org/cii/srv/cii-srv/-/merge_requests/11>
|
|
|
|
|
|
|
|
Example in <https://gitlab.eso.org/ahoffsta/cii-srv/-/blob/oldb-in-memory-missing-python-binding/oldb-client/python/oldb/test/oldbInMemoryTest.py>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### KB: Catching API Exceptions \[Python\]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Problem**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
My application contains a call to the CII Python API.
|
|
|
|
|
|
|
|
When I ran it, it threw an exception with the following backtrace:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>Top Level Unexpected exception:</p><p>Traceback (most recent call last):</p><p>File "/home/eltdev/MODULES/test/app.py", line 91, in instantiateDP</p><p>double_dp = self.oldb_client.create_data_point(uri, metadataInstName)</p><p><strong>CiiOldbPyB.CiiOldbDpExistsException</strong>: The Data point cii.oldb:/root/test/xxxdp already exists.</p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Therefore, I added a corresponding try-catch around my call:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>try:</p><p>...</p><p>except <strong>CiiOldbPyB.CiiOldbDpExistsException</strong> as e:</p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
When I run it, the try-catch doesn't work.
|
|
|
|
|
|
|
|
Moreover, I now get two backtraces:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>Top Level Unexpected exception:</p><p>Traceback (most recent call last):</p><p>File "/home/eltdev/MODULES/test/app.py", line 91, in instantiateDP</p><p>double_dp = self.oldb_client.create_data_point(uri, metadataInstName)</p><p><strong>CiiOldbPyB.CiiOldbDpExistsException</strong>: The Data point cii.oldb:/root/test/xxxdp already exists.</p><p> </p><p><strong>During handling of the above exception, another exception occurred:</strong></p><p>Traceback (most recent call last):</p><p>File "/home/eltdev/MODULES/test/app.py", line 108, in main</p><p>oldbCreator.instantiateOLDB_exception()</p><p>File "/home/eltdev/MODULES/test/app.py", line 81, in instantiateOLDB_exception</p><p>self.instantiateDP(double_dp_uri, double_dp_meta.get_instance_name())</p><p>File "/home/eltdev/MODULES/test/app.py", line 94, in instantiateDP</p><p>except CiiOldbPyB.CiiOldbDpExistsException:</p><p><strong>NameError: name 'CiiOldbPyB' is not defined</strong></p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Solution**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
You were mislead by the first backtrace -
|
|
|
|
|
|
|
|
the exception name in the backtrace is not what you should catch.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
In your code, replace "**CiiOldbPyB**" with "**elt.oldb**":
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>try:</p><p>....</p><p>except <strong>elt.oldb.CiiOldbDpExistsException</strong> as e:</p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
> For completeness - do no forget this statement:
|
|
|
|
|
|
|
|
| import elt.oldb |
|
|
|
|
|-----------------|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Background**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The CII Python API is mostly a binding to the CII C++ API.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The CiiOldbPyB.CiiOldbDpExistsException is the original binding class.
|
|
|
|
|
|
|
|
This binding class is re-exported under the name elt.oldb.CiiOldbDpExistsException.
|
|
|
|
|
|
|
|
The elt.oldb module internally loads the C++ binding module CiiOldbPyB. So both are the same exception.
|
|
|
|
|
|
|
|
Nonetheless, you should use the re-exported name, not the original name in your application. We discourage the use of the original name because the structure of the CiiOldbPyB module is more "chaotic" and not equivalent to elt.oldb.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Unfortunately, in the backtraces you will always see the original name instead of the re-exported name.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This question was originally asked in [ECII-422](https://jira.eso.org/browse/ECII-422).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### KB: access_key empty (DevEnv 3.2.0) \[OLDB\]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Problem**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Trying to use the OLDB on DevEnv 3.2.0, I'm getting this error:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Unexpected exception occurred. What:Configuration invalid: access_key empty
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Solution**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Run the following commands (you will be asked for the root pw):
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>wget -q <a href="https://www.eso.org/~mschilli/download/cii/postinstall/cii-postinstall-20210610">www.eso.org/~mschilli/download/cii/postinstall/cii-postinstall-20210610</a></p><p> </p><p>cii-services stop config</p><p> </p><p>su -c "bash cii-postinstall-20210610 schemas"</p><p> </p><p># If the script</p><p> </p><p>#You should see the following output:</p><table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>Password:</p><p>CII PostInstall (20210610)</p><p>schemas: applying fix ECII397</p><p>/home/eltdev/</p><p>schemas: populating elasticsearch</p><p>schemas: skipping telemetry</p><p>schemas: skipping alarms</p></th></tr></thead><tbody></tbody></table><p> </p><p>cii-services start config</p><p> </p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Background**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The OLDB settings coming with 3.2.0 are buggy.
|
|
|
|
|
|
|
|
The CII post-install procedure is able to hotfix the settings (ECII397).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The problem will be fixed in DevEnv 3.4.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### KB: Datapoint already exists \[OLDB\]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Problem**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
My application tries to create an OLDB datapoint.
|
|
|
|
|
|
|
|
This fails because the datapoint "already exists":
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
> 2021-06-22T11:01:33.003+0000, ERROR, CiiOldbRedisDataPointProvider/140709706681216, Data point uri: cii.oldb:/tcs/hb/tempser3 in Redis already exists.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
In response, my application skips the creation step, and wants to use the reportedly existing datapoint.
|
|
|
|
|
|
|
|
However, when doing this, I get the error "Datapoint doesn't exist".
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Likewise, when I run the oldb-gui database browser, it does not show this data point in the OLDB.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Variant 2 of the Problem**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
I try to access an OLDB datapoint, and I see two errors like this:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Target configuration does not exist: Failed to retrieve configuration from elastic search: Configuration
|
|
|
|
|
|
|
|
\[…\]
|
|
|
|
|
|
|
|
elt.oldb.exceptions.CiiOldbDpExistsException: Data point cii.oldb:/alarm/alarm/device/motor/input_int_dp_alarm already exisits.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Go directly to Solution 2 below.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Variant 3 of the Problem**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
I try to delete an OLDB datapoint and I see an error like this:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
> CiiOldbPyB.CiiOldbException: De-serialization error:sizeof(T)\*count is greater then remaining
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Go directly to Solution 2 below.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Background**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The two errors are contradicting.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Datapoints are stored in two databases: a document-database (permanent store) for its metadata, and a key-value-database (volatile store) for its current value. The above symptoms indicate that the two databases are out-of-sync, meaning the datapoint exists only "half".
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Solution 1**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
There is a good chance you can delete the datapoint to clean up the situation (as of [ECII-303](https://jira.eso.org/browse/ECII-303)):
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>#!/usr/bin/env python</p><p>import elt.config</p><p>import elt.oldb</p><p> </p><p>oldb_client = elt.oldb.CiiOldbFactory.get_instance()</p><p>elt.oldb.CiiOldbGlobal.set_write_enabled(True)</p><p> </p><p>uri = elt.config.Uri("cii.oldb:/tcs/hb/tempser3")</p><p>oldb_client.delete_data_point(uri)</p><p> </p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Solution 2**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
If the above didn't help, find out which "half" of the datapoint exists:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1. The current value exists, and the metadata is missing. This is the case when upgrading DevEnv/CII without deleting the Redis cache.
|
|
|
|
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> Run the following command:
|
|
|
|
>
|
|
|
|
> Note: enter a search-expression matching the datapoint URI, in the example this is "tempser3".
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p># confirm that this is the cause of the problem</p><p>redis-cli KEYS *tempser3*</p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> If the command is showing the problematic key,
|
|
|
|
>
|
|
|
|
> we run a similar command to remove that key from the store:
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p># if so, delete the offending datapoint</p><p>redis-cli --scan --pattern *tempser3* | xargs redis-cli del</p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2. The metadata exists, and the current value is missing
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
> Repair the current value:
|
|
|
|
|
|
|
|
| |
|
|
|
|
|-----|
|
|
|
|
|
|
|
|
> Update: As of [ECII-303](https://jira.eso.org/browse/ECII-303), this should no longer be necessary, since the OLDB will make an attempt to automatically re-instate the current value that is missing.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Solution 3**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
If none of the above helped, another possibility is to clean-up the metadata.
|
|
|
|
|
|
|
|
This is a potentially very invasive operation. Please contact us for instructions.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### KB: Req/Rep Connection Listeners \[MAL Python\]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Problem**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
MAL includes a callback registration method to allow monitoring of the MAL service connection status.
|
|
|
|
|
|
|
|
This is done with the method "*registerConnectionListener()*".
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Given the example below, this does not work, and the "*listenerMethod()"* is never called.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
import sys
|
|
|
|
import datetime
|
|
|
|
import signal
|
|
|
|
import traceback
|
|
|
|
import logging
|
|
|
|
|
|
|
|
import elt.pymal as mal
|
|
|
|
from pymalcpp import TimeoutException
|
|
|
|
from ModTrk.Trk.StdCmds import StdCmdsSync
|
|
|
|
from ModTrk.Trk.StdCmds import StdCmdsAsync
|
|
|
|
from ModTrk.Trk import TelPosition
|
|
|
|
from ModTrk.Trk import AxesPosition
|
|
|
|
|
|
|
|
THREE_SECONDS = datetime.timedelta(seconds=3)
|
|
|
|
MINUTE = datetime.timedelta(seconds=60)
|
|
|
|
MY_SERVER_URL='localhost'
|
|
|
|
MY_SERVER_PORT='44444'
|
|
|
|
|
|
|
|
def listenerMethod(state):
|
|
|
|
print("listenerMethod: registerConnectionListener() response :" + str(state))
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
uri = 'zpb.rr://' + MY_SERVER_URL + ':' + str(MY_SERVER_PORT) + '/m1/' + 'TrkLsvServer'
|
|
|
|
print('MAL URI: ' + uri)
|
|
|
|
zpbMal = mal.loadMal('zpb', {})
|
|
|
|
factory = mal.CiiFactory.getInstance()
|
|
|
|
factory.registerMal('zpb', zpbMal )
|
|
|
|
stdcmds = factory.getClient(uri, StdCmdsAsync, qos=mal.rr.qos.ReplyTime(THREE_SECONDS))
|
|
|
|
stdcmds.registerConnectionListener(listenerMethod)
|
|
|
|
|
|
|
|
connectionFuture = stdcmds.asyncConnect()
|
|
|
|
|
|
|
|
connectionFuture.wait_for(THREE_SECONDS)
|
|
|
|
rtn = stdcmds.Status()
|
|
|
|
rtn.wait()
|
|
|
|
print( str(rtn.get() ))
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
*From \<<https://jira.eso.org/browse/ECII-212>\>*
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Solution**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The developer must keep a reference to the returned Object from the "*registerConnectionListener()*" invocation.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
stdcmds = factory.getClient(uri, StdCmdsAsync, qos=mal.rr.qos.ReplyTime(THREE_SECONDS))
|
|
|
|
listenerRegistration = stdcmds.registerConnectionListener(listenerMethod)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Background**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The documentation states a Return value, but is the responsibility of the developer to keep the reference. Otherwise, the object will be deleted when exiting the block of code.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Remember to delete (assign None), to this object when closing the connection to the MAL service.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
*From \<<https://jira.eso.org/browse/ECII-212>\>*
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### KB: Command Line Tools and Python snippets \[OLDB\]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Problem**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
I need to inspect or modify the content of the OLDB from the command line or a shell script.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Solution**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**cii-oldb-traversal-tool**
|
|
|
|
|
|
|
|
> for searching through the OLDB
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>$ cii-oldb-traversal-tool --file output --quality OK</p><p>$ cat output</p><p>cii.oldb:///root/trklsv/cfg/log/level|OK|WARNING|2020-09-11T15:25:08Z</p><p>cii.oldb:///root/trklsv/cfg/req/endpoint|OK|zpb.rr://localhost:44444/m1/TrkLsvServer|2020-09-11T15:25:08Z</p><p>cii.oldb:///root/trklsv/ctr/current/altaz/alt|OK|0.000000|2020-09-11T15:24:25Z</p><p>cii.oldb:///root/trklsv/ctr/current/altaz/az|OK|0.000000|2020-09-11T15:24:25Z</p><p>cii.oldb:///root/trklsv/ctr/current/radec/dec|OK|0.000000|2020-09-11T15:24:27Z</p><p>cii.oldb:///root/trklsv/ctr/current/radec/ra|OK|0.000000|2020-09-11T15:24:27Z</p><p>cii.oldb:///root/trklsv/ctr/poserr|OK|0.000000|2020-09-11T15:24:27Z</p><p>cii.oldb:///root/trklsv/ctr/status|OK|UNKNOWN|2020-09-11T15:23:55Z</p><p>cii.oldb:///root/trklsv/ctr/substate|OK|UNKNOWN|2020-09-11T15:23:49Z</p><p>cii.oldb:///root/trklsv/ctr/target/altaz/alt|OK|0.000000|2020-09-11T15:24:24Z</p><p>cii.oldb:///root/trklsv/ctr/target/altaz/az|OK|0.000000|2020-09-11T15:24:25Z</p><p>cii.oldb:///root/trklsv/ctr/target/radec/dec|OK|0.000000|2020-09-11T15:24:26Z</p><p>cii.oldb:///root/trklsv/ctr/target/radec/ra|OK|0.000000|2020-09-11T15:24:26Z</p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**oldb-cli**
|
|
|
|
|
|
|
|
> for reading, writing, subscribing to an OLDB-datapoint
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>$ oldb-cli read cii.oldb:///root/trklsv/cfg/req/endpoint</p><p>SLF4J: Class path contains multiple SLF4J bindings.</p><p>SLF4J: Found binding in [jar:file:/eelt/ciisrv/1.0-RC3-20201030/lib/srv-support-libs/slf4j-nop-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]</p><p>SLF4J: Found binding in [jar:file:/eelt/mal/1.1.0-2.2.3-20201027/lib/mal-opcua/slf4j-nop-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]</p><p>SLF4J: See <a href="http://www.slf4j.org/codes.html#multiple_bindings">http://www.slf4j.org/codes.html#multiple_bindings</a> for an explanation.</p><p>SLF4J: Actual binding is of type [org.slf4j.helpers.NOPLoggerFactory]</p><p>log4j:WARN No appenders could be found for logger (io.netty.util.internal.logging.InternalLoggerFactory).</p><p>log4j:WARN Please initialize the log4j system properly.</p><p>Timestamp: 2020-09-11T15:25:08.648Z</p><p>Quality: OK</p><p>Value: zpb.rr://localhost:44444/m1/TrkLsvServer</p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**oldb Python API**
|
|
|
|
|
|
|
|
> for creating, deleting an OLDB-datapoint
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>$ python<br />
|
|
|
|
Python 3.7.6 (default, Jan 8 2020, 19:59:22)</p><p>[GCC 7.3.0] :: Anaconda, Inc. on linux</p><p>Type "help", "copyright", "credits" or "license" for more information.</p><p>>>></p><p>>>> import elt.oldb</p><p>>>> from elt.config import Uri</p><p>>>> oldb = elt.oldb.CiiOldbFactory.get_instance()</p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
… and, to create a datapoint:
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>>>> elt.oldb.CiiOldbGlobal.set_write_enabled(True)</p><p>>>> oldb.create_data_point_by_value (Uri("cii.oldb:/ccs/tst/tmp1"), "my text")</p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
… and, to delete a datapoint:
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>>>> elt.oldb.CiiOldbGlobal.set_write_enabled(True)</p><p>>>> oldb.delete_data_point (Uri("cii.oldb:/ccs/tst/tmp1"))</p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
… and, to read a datapoint:
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>>>> oldb.get_data_point(Uri("cii.oldb:/ccs/tst/tmp1")).read_value().get_value()</p><p>'my text'</p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### KB: Latency on Pub/Sub \[MAL ZMQ\]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Problem**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
We see latencies of 400-1000ms in MAL ZMQ pub/sub communication, for sending a 12k x 12k image blob.
|
|
|
|
|
|
|
|
For the 2 first transmissions, there is somewhere between 400 and 500ms latency on the publisher side between just before calling the "publish" method and when we see the first packets on the wire.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This is for all messages, not only for the first messages. When we let the program run several minutes, during that period, all messages have a consistent delay when arriving on subscriber side.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Solution**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
From MAL 1.0.4 on,
|
|
|
|
|
|
|
|
MAL supports the below MAL specific property to limit queue size for large message publishers.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th>mal::Mal::Properties m_malProperties1;<br />
|
|
|
|
m_malProperties1["zpb.ps.zmq.sndhwm"] = "1";<br />
|
|
|
|
<br />
|
|
|
|
auto publisher = factory.getPublisher<mal::example::Sample>(uri,<br />
|
|
|
|
{ std::make_shared<mal::ps::qos::Latency>(<br />
|
|
|
|
std::chrono::milliseconds(100)),<br />
|
|
|
|
std::make_shared<mal::ps::qos::Deadline>(<br />
|
|
|
|
std::chrono::seconds(1)) }, m_malProperties1);</th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
For more information how to use MAL specific properties, see MAL Binding Manual.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Background**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The issue has been first reported in [ECII-159](https://jira.eso.org/browse/ECII-159).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The problem lies in the ZMQ send queues. Default size is 1000 and with 144MB per message (in this case) this means 144GB. Limiting this to 1 (for a test) a publisher can handle 20 subscribers (tested) without any problems. The solution is to reconfigure the send-queue size appropriately.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### KB: Choose middlewares to build \[MAL ICD\]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Problem**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
I am certain that my ICD will never be used over OPC UA. Nonetheless, the ICD-compilation builds OPC UA mappings for my ICD. This is unnecessarily extending the compilation time for my application.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Solution**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
By default the ICD-compilation builds mappings for all middlewares. But it is possible to exclude certain middleware mappings from compilation, which will reduce compilation time. You do this by passing mal options to the icd-generator.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Example**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
wscript
|
|
|
|
|
|
|
|
| declare_malicd(use='icds.base', mal_opts = { 'opcua_disabled': True } ) |
|
|
|
|
|-------------------------------------------------------------------------|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The available options are:
|
|
|
|
|
|
|
|
- opcua_disabled = if True, disable OPCUA middleware generation
|
|
|
|
|
|
|
|
- dds_disabled = if True, disable DDS middleware generation
|
|
|
|
|
|
|
|
- zpb_disabled = if True, disable ZEROMQ middleware generation
|
|
|
|
|
|
|
|
>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### KB: Sending an array of unions \[MAL cpp\]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Problem**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Trying to send a msg, I get the following error message from CII:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\[libprotobuf ERROR google/protobuf/message_lite.cc:121\] Can't parse message of type "generated.zpb.fcfif.StdCmds_Request" because it is missing required fields: data.Setup.payload\[0\].piezoData.input
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
My ICD definition looks like this
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p><enum name="PiezoInput"><br />
|
|
|
|
<enumerator name="PIEZO_INPUT_SETMODE" /><br />
|
|
|
|
<enumerator name="PIEZO_INPUT_MOVE" /><br />
|
|
|
|
</enum></p><p> </p><p><enum name="PiezoMode"><br />
|
|
|
|
<enumerator name="PIEZO_MODE_1" /><br />
|
|
|
|
<enumerator name="PIEZO_MODE_2" /><br />
|
|
|
|
</enum></p><p> </p><p><struct name="PiezoModeStruct"><br />
|
|
|
|
<member name="mode" type="nonBasic" nonBasicTypeName="PiezoMode" /><br />
|
|
|
|
</struct></p><p> </p><p><enum name="PiezoMove"><br />
|
|
|
|
<enumerator name="PIEZO_MOVE_1" /><br />
|
|
|
|
<enumerator name="PIEZO_MOVE_2" /><br />
|
|
|
|
</enum><br />
|
|
|
|
<br />
|
|
|
|
<union name="PiezoUnion"><br />
|
|
|
|
<discriminator type="nonBasic" nonBasicTypeName="PiezoInput" /><br />
|
|
|
|
<case><br />
|
|
|
|
<caseDiscriminator value ="PIEZO_INPUT_SETMODE"/><br />
|
|
|
|
<member name="piezoModeData" type="nonBasic" nonBasicTypeName="PiezoModeStruct" /><br />
|
|
|
|
</case><br />
|
|
|
|
<case><br />
|
|
|
|
<caseDiscriminator value ="PIEZO_INPUT_MOVE"/><br />
|
|
|
|
<member name="piezoMoveData" type="nonBasic" nonBasicTypeName="PiezoMove" /><br />
|
|
|
|
</case><br />
|
|
|
|
</union><br />
|
|
|
|
<br />
|
|
|
|
<struct name="Piezo"><br />
|
|
|
|
<member name="id" type="string" /><br />
|
|
|
|
<member name="input" type="nonBasic" nonBasicTypeName="PiezoUnion" /><br />
|
|
|
|
</struct></p><p> </p><p><interface name="PiezoTest"><br />
|
|
|
|
<method name="test" returnType="void"><br />
|
|
|
|
<argument name="arr" type="nonBasic" nonBasicTypeName="Piezo" arrayDimensions="(10)" /><br />
|
|
|
|
</method><br />
|
|
|
|
</interface></p><p> </p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
My code looks like this
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>[...]</p><p>auto piezo = mal->createDataEntity<::fcfif::Piezo>();</p><p>piezo->setId("foo");</p><p> </p><p>auto input = piezo->getInput();</p><p>auto mode = input->getPiezoModeData();</p><p>mode->setAction(::fcfif::ActionPiezoMode::SET_AUTO);</p><p> </p><p>auto union = mal->createDataEntity<::fcfif::FcsUnion>();</p><p>union->setPiezoData(piezo);</p><p> </p><p>[...]</p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Solution**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>auto p = factory.getClient<::fcfif::PiezoTestSync>(uri,<br />
|
|
|
|
{std::make_shared<mal::rr::qos::ReplyTime><br />
|
|
|
|
(std::chrono::seconds(3))},<br />
|
|
|
|
{});</p><p>auto mal = p->getMal();</p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>auto piezo = mal->createDataEntity<::fcfif::Piezo>();<br />
|
|
|
|
piezo->setId("foo");</p><p>auto input = piezo->getInput();</p><p>auto piezoModeStruct = mal->createDataEntity<::fcfif::PiezoModeStruct>();<br />
|
|
|
|
piezoModeStruct->setMode(::fcfif::PiezoMode::PIEZO_MODE_1);</p><p> </p><p>input->setPiezoModeData(piezoModeStruct);</p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>auto piezo2 = mal->createDataEntity<::fcfif::Piezo>();<br />
|
|
|
|
piezo2->setId("foo2");</p><p> </p><p>auto input2 = piezo2->getInput();</p><p> </p><p>input2->setPiezoMoveData(::fcfif::PiezoMove::PIEZO_MOVE_1);</p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>std::vector<std::shared_ptr<::fcfif::Piezo>> sa;<br />
|
|
|
|
sa.push_back(piezo);<br />
|
|
|
|
sa.push_back(piezo2);</p><p>p->test(sa);</p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Background**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Your code does not work since you do not use union instance provided by the parent structure. You need to obtain nested structure/union via accessors and do not try to created your own (detached) instance.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This issue was first described in [ECII-154](https://jira.eso.org/browse/ECII-154)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### KB: config not found on remote db \[Conf\]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Problem**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
You are intending to read a config from the local config database ("localdb"), but you see an error message like this.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
elt.config.exceptions.CiiConfigNoTcException: Target configuration does not exist: Failed to retrieve configuration from elastic search: Configuration cii.config://\*/supervisoryapp/TrkLsvDeploy on the remote db was not found
|
|
|
|
|
|
|
|
at elt.config.client.ConfigRemoteDatabase.retrieveConfig(ConfigRemoteDatabase.java:191)
|
|
|
|
|
|
|
|
at elt.config.client.CiiConfigClient.retrieveConfig(CiiConfigClient.java:354)
|
|
|
|
|
|
|
|
at elt.config.client.CiiConfigClient.retrieveConfig(CiiConfigClient.java:310)
|
|
|
|
|
|
|
|
at trkLsv.DataContext.loadConfig(DataContext.java:324)
|
|
|
|
|
|
|
|
at trkLsv.DataContext.\<init>(DataContext.java:190)
|
|
|
|
|
|
|
|
at trkLsv.TrkLsv.go(TrkLsv.java:72)
|
|
|
|
|
|
|
|
at trkLsv.TrkLsv.main(TrkLsv.java:41)
|
|
|
|
|
|
|
|
Caused by: elt.error.icd.CiiSerializableException
|
|
|
|
|
|
|
|
at elt.config.service.client.icd.zpb.ServiceClientApiInterfaceAsyncImpl.processRequest(ServiceClientApiInterfaceAsyncImpl.java:73)
|
|
|
|
|
|
|
|
at elt.mal.zpb.rr.ClientAsyncImpl.events(ClientAsyncImpl.java:261)
|
|
|
|
|
|
|
|
at org.zeromq.ZPoller.dispatch(ZPoller.java:537)
|
|
|
|
|
|
|
|
at org.zeromq.ZPoller.poll(ZPoller.java:488)
|
|
|
|
|
|
|
|
at org.zeromq.ZPoller.poll(ZPoller.java:461)
|
|
|
|
|
|
|
|
at elt.mal.zpb.ZpbMal.processThread(ZpbMal.java:459)
|
|
|
|
|
|
|
|
at elt.mal.zpb.ZpbMal.lambda$new$0(ZpbMal.java:119)
|
|
|
|
|
|
|
|
at java.lang.Thread.run(Thread.java:748)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Solution A**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Your local file may be invalid (e.g. illegal format, or doesn't match the config class definition).
|
|
|
|
|
|
|
|
Look at the content of your local config database, e.g. with
|
|
|
|
|
|
|
|
$ find $INTROOT/localdb
|
|
|
|
|
|
|
|
and correct the file in place, or fix the source yaml and then redeploy it from source to the localdb.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Background**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
You may have a malformed json file in your local db, which the config service failed to read.
|
|
|
|
|
|
|
|
Because of the use the location wildcard "\*" in your code (in "cii.config://\*/supervisoryapp/TrkLsvDeploy"),
|
|
|
|
|
|
|
|
the config service has consequently tried to load the config from the remote config database, where no such config exists.
|
|
|
|
|
|
|
|
To that end, the error message is misleading, and should be improved (ticket [ECII-208](https://jira.eso.org/browse/ECII-208)).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### KB: Failed to send request, send queue full \[CII MAL\]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Problem**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
You are intending to update a config, but you get an IllegalStateException where the last line is
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
elt.mal.zpb.rr.ClientAsyncImpl:183
|
|
|
|
|
|
|
|
> throw new MalException("Failed to send request, send queue full");
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Solution**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Probably you have called close() on the CiiConfigClient instance somewhere, maybe also implicitly during a try-with-resource block.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### KB: Getting More Logs \[MAL\]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> **Problem**
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> The MAL seems to misbehave. How can I get more log messages from the MAL used in my application?
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> **Solution A**
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> Java
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> From MAL 1.1.0, edit the MAL log4j config xml
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> and specify the MAL log levels:
|
|
|
|
|
|
|
|
| \<Logger name="elt.mal" level="TRACE" /> |
|
|
|
|
|------------------------------------------|
|
|
|
|
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> Cpp
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> Example for **Zpb.** For other middlewares, see below
|
|
|
|
>
|
|
|
|
>
|
|
|
|
|
|
|
|
1. Put a log-config file into your file system:
|
|
|
|
|
|
|
|
>
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 78%" /><col style="width: 21%" /></colgroup><thead><tr class="header"><th><p>log4cplus.rootLogger=TRACE, stdout</p><p> </p><p>log4cplus.logger.malDds=TRACE, MyFileAppender</p><p>log4cplus.logger.malDdsBasePubSub=TRACE, MyFileAppender</p><p>log4cplus.logger.malDdsPublisher=TRACE, MyFileAppender</p><p>log4cplus.logger.malDdsInstancePublisher=TRACE, MyFileAppender</p><p>log4cplus.logger.malDdsSubscriptionManager=TRACE, MyFileAppender</p><p>log4cplus.logger.malDdsSubscriptionReaderListener=TRACE, MyFileAppender</p><p>log4cplus.logger.malDdsSubscriber=TRACE, MyFileAppender</p><p>log4cplus.logger.malDdsMrvSubscriber=TRACE, MyFileAppender</p><p>log4cplus.logger.malDdsRequesterImpl=TRACE, MyFileAppender</p><p> </p><p>log4cplus.additivity.malDds=True</p><p>log4cplus.additivity.malDdsBasePubSub=True</p><p>log4cplus.additivity.malDdsPublisher=True</p><p>log4cplus.additivity.malDdsInstancePublisher=True</p><p>log4cplus.additivity.malDdsSubscriptionManager=True</p><p>log4cplus.additivity.malDdsSubscriptionReaderListener=True</p><p>log4cplus.additivity.malDdsSubscriber=True</p><p>log4cplus.additivity.malDdsMrvSubscriber=True</p><p>log4cplus.additivity.malDdsRequesterImpl=True</p><p> </p><p> </p><p>log4cplus.logger.malZpb=TRACE, MyFileAppender</p><p>log4cplus.logger.malZpbBasePubSub=TRACE, MyFileAppender</p><p>log4cplus.logger.malZpbPublisher=TRACE, MyFileAppender</p><p>log4cplus.logger.malZpbInstancePublisher=TRACE, MyFileAppender</p><p>log4cplus.logger.malZpbSubscriber=TRACE, MyFileAppender</p><p>log4cplus.logger.malZpbMrvSubscriber=TRACE, MyFileAppender</p><p>log4cplus.logger.malZpbServer=TRACE, MyFileAppender</p><p>log4cplus.logger.malZpbClientAsyncImpl=TRACE, MyFileAppender</p><p> </p><p>log4cplus.additivity.malZpb=True</p><p>log4cplus.additivity.malZpbBasePubSub=True</p><p>log4cplus.additivity.malZpbPublisher=True</p><p>log4cplus.additivity.malZpbInstancePublisher=True</p><p>log4cplus.additivity.malZpbSubscriber=True</p><p>log4cplus.additivity.malZpbMrvSubscriber=True</p><p>log4cplus.additivity.malZpbServer=True</p><p>log4cplus.additivity.malZpbClientAsyncImpl=True</p><p> </p><p> </p><p>log4cplus.logger.malOpcua=TRACE, MyFileAppender</p><p>log4cplus.logger.malOpcuaBasePubSub=TRACE, MyFileAppender</p><p>log4cplus.logger.malOpcuaPublisher=TRACE, MyFileAppender</p><p>log4cplus.logger.malOpcuaInstancePublisher=TRACE, MyFileAppender</p><p>log4cplus.logger.malOpcuaSubscriber=TRACE, MyFileAppender</p><p>log4cplus.logger.malOpcuaMrvSubscriber=TRACE, MyFileAppender</p><p>log4cplus.logger.malOpcuaMrvDataMonitor=TRACE, MyFileAppender</p><p>log4cplus.logger.malOpcuaDataPoller=TRACE, MyFileAppender</p><p>log4cplus.logger.malOpcuaDataMonitor=TRACE, MyFileAppender</p><p>log4cplus.logger.malOpcuaClient=TRACE, MyFileAppender</p><p>log4cplus.logger.malOpcuaClientEventLoop=TRACE, MyFileAppender</p><p> </p><p>log4cplus.additivity.malOpcua=True</p><p>log4cplus.additivity.malOpcuaBasePubSub=True</p><p>log4cplus.additivity.malOpcuaPublisher=True</p><p>log4cplus.additivity.malOpcuaInstancePublisher=True</p><p>log4cplus.additivity.malOpcuaSubscriber=True</p><p>log4cplus.additivity.malOpcuaMrvSubscriber=True</p><p>log4cplus.additivity.malOpcuaMrvDataMonitor=True</p><p>log4cplus.additivity.malOpcuaDataPoller=True</p><p>log4cplus.additivity.malOpcuaDataMonitor=True</p><p>log4cplus.additivity.malOpcuaClient=True</p><p>log4cplus.additivity.malOpcuaClientEventLoop=True</p><p> </p><p> </p><p>log4cplus.appender.stdout=log4cplus::ConsoleAppender</p><p>log4cplus.appender.stdout.layout=log4cplus::PatternLayout</p><p>log4cplus.appender.stdout.layout.ConversionPattern=%5p [%t] (%F:%L) - %m%n</p><p> </p><p>log4cplus.appender.MyFileAppender=log4cplus::RollingFileAppender</p><p>log4cplus.appender.MyFileAppender.File=/tmp/elt-mal-cpp-trace.log</p><p>log4cplus.appender.MyFileAppender.layout=log4cplus::PatternLayout</p><p>log4cplus.appender.MyFileAppender.layout.ConversionPattern=[%-5p][%D{%Y/%m/%d %H:%M:%S:%q}][%-l][%t] %m%n</p></th><th><p> </p><p><<mal-log4cplus.conf>></p><p> </p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
|
|
|
|
2. Pass the path to the log-config to MAL:
|
|
|
|
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> MAL logging is initialized from a configuration file, the path of which is read from mal properties with key mal::PROP_LOG_CONFIG_FILENAME. When loading mal, use set mal::PROP_LOG_CONFIG_FILENAME in mal properties.
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> For example:
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>auto zpbMal = mal::loadMal("zpb",</p><p>mal::Mal::Properties{{mal::PROP_LOG_CONFIG_FILENAME,"/path/to/mal-log4cplus.conf"}}</p><p>);</p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> or in python:
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>import elt.pymal as mal</p><p>zpbMal = mal.loadMal ("zpb",</p><p>{"zpb.log4cplus.filename":"/path/to/mal-log4cplus.conf"})</p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> **Solution B**
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> With [ECII-246](https://jira.eso.org/browse/ECII-246), it is possible to change the log levels of the MAL loggers at run-time via a method call:
|
|
|
|
>
|
|
|
|
> ::elt::mal::util::logging::setLogLevelForLoggers( ... )
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> The allowed logger names are listed below.
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> **Background**
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> MAL does not use the Cii Logging System (CiiLogManager etc.) directly. Instead, MAL expects logging to be initialized from a configuration file. Note that the format of the log-config differs per programming language.
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> List of logger names available in the **Cpp MAL / Python MAL:**
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> Loggers for mal-zpb
|
|
|
|
|
|
|
|
- malZpbInstancePublisher
|
|
|
|
|
|
|
|
- malZpbPublisher
|
|
|
|
|
|
|
|
- malZpbClientAsyncImpl
|
|
|
|
|
|
|
|
- malZpbServer
|
|
|
|
|
|
|
|
- malZpb
|
|
|
|
|
|
|
|
- malZpbSubscriber
|
|
|
|
|
|
|
|
- malZpbMrvSubscriber
|
|
|
|
|
|
|
|
- malZpbBasePubSub
|
|
|
|
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> Loggers for mal-dds
|
|
|
|
|
|
|
|
- malDds
|
|
|
|
|
|
|
|
- malDdsSubscriptionManager
|
|
|
|
|
|
|
|
- malDdsSubscriptionReaderListener
|
|
|
|
|
|
|
|
- malDdsSubscriber
|
|
|
|
|
|
|
|
- malDdsPublisher
|
|
|
|
|
|
|
|
- malDdsBasePubSub
|
|
|
|
|
|
|
|
- malDdsMrvSubscriber
|
|
|
|
|
|
|
|
- malDdsRequesterImpl
|
|
|
|
|
|
|
|
- malDdsInstancePublisher
|
|
|
|
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> Loggers for mal-opcua
|
|
|
|
|
|
|
|
- malOpcua
|
|
|
|
|
|
|
|
- malOpcuaBasePubSub
|
|
|
|
|
|
|
|
- malOpcuaMrvSubscriber
|
|
|
|
|
|
|
|
- malOpcuaMrvDataMonitor
|
|
|
|
|
|
|
|
- malOpcuaInstancePublisher
|
|
|
|
|
|
|
|
- malOpcuaPublisher
|
|
|
|
|
|
|
|
- malOpcuaSubscriber
|
|
|
|
|
|
|
|
- malOpcuaDataPoller
|
|
|
|
|
|
|
|
- malOpcuaDataMonitor
|
|
|
|
|
|
|
|
- malOpcuaClient
|
|
|
|
|
|
|
|
- malOpcuaClientEventLoop
|
|
|
|
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### KB: Change Log Levels at Run-time
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Problem**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
I want to modify the log levels of my application programmatically,
|
|
|
|
|
|
|
|
without having to reload the full log configuration.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Solution**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
With [ECII-282](https://jira.eso.org/browse/ECII-282), the CiiLogManager was extended in all three languages.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
C++ added methods:
|
|
|
|
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> void elt::log::CiiLogManager::SetLogLevel(const std::string logger_name, log4cplus::LogLevel level)
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> void elt::log::CiiLogManager::SetLogLevel(log4cplus::Logger logger, log4cplus::LogLevel level)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Java added methods:
|
|
|
|
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> void elt.log.CiiLogManager.setLogLevel(
|
|
|
|
>
|
|
|
|
> final String loggerName, final org.apache.logging.log4j.Level level);
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> void elt.log.CiiLogManager.setLogLevel(
|
|
|
|
>
|
|
|
|
> org.apache.logging.log4j.Logger logger, final org.apache.logging.log4j.Level level)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Python added methods:
|
|
|
|
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> elt.log.CiiLogManager.set_log_level(name_or_logger: Union\[str, logging.Logger\], level: logging.Level)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### KB: PYBIND errors \[ICD waf build\]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Problem**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Trying to build your MAL Application, you get errors like below related to the PYBIND module.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>icd/python/bindings/src/ModProto-benchmark.cpp:18:25: error: expected initializer before ‘-’ token</p><p>PYBIND11_MODULE(ModProto-benchmark, modproto-benchmark) {</p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Solution**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Check the name of your ICD file:
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>> find icd</p><p>icd</p><p>icd/wscript</p><p>icd/src</p><p>icd/src/proto-benchmark.xml</p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
The icd file name contains a minus, which is actually reflected in the above error message.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Rename the file to something like this:
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>> find icd</p><p>icd</p><p>icd/wscript</p><p>icd/src</p><p>icd/src/protobenchmark.xml</p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
In general, due to the many code generation steps taking place, your freedom in ICD file naming is limited.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### KB: multiple XMLs found \[ICD waf build\]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Problem**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Trying to build your ICD module, you see this error:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
> Waf: Entering directory \`/home/eltdev/repos/hlcc/build'
|
|
|
|
> Error: multiple XMLs found, just one supported.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
while in fact you have only one XML file in your ICD directory.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Solution**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Check the file name of your ICD file:
|
|
|
|
|
|
|
|
make sure it starts with an uppercase letter.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Background**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The error message is misleading (will be improved, [ECII-426](https://jira.eso.org/browse/ECII-426)).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The code generator for malicd_topics fails when the ICD file name starts with lowercase.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
For more information, see also: [KB: PYBIND errors \[ICD waf build\]](onenote:#KB%20PYBIND%20errors%20[ICD%20waf%20build]§ion-id={F524F9BE-F51D-4A01-9976-93359FCC4966}&page-id={0FEE4FB9-C58B-4E8A-A276-2EC4367CFB30}&end&base-path=https://europeansouthernobservatory.sharepoint.com/sites/ELT_Control/SiteAssets/ELT_Control%20Notebook/Documentation/ELT%20Control%20KnowledgeBase.one)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### KB: g++: internal compiler error: Killed \[waf build\]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Problem**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Trying to build ("waf build") your mal-application, the build fails and you get this error message:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>Software/CcsLibs/CcsTestData/python/bindings/src/ModCcstestdata.cpp:18:1: note: in expansion of macro ‘PYBIND11_MODULE’</p><p>PYBIND11_MODULE(ModCcstestdata, modccstestdata) {</p><p>^</p><p>g++: internal compiler error: Killed (program cc1plus)</p><p>Please submit a full bug report,</p><p>with preprocessed source if appropriate.</p><p>See <<a href="http://bugzilla.redhat.com/bugzilla">http://bugzilla.redhat.com/bugzilla</a>> for instructions.</p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Background**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The cpp compiler runs out of memory and crashes.
|
|
|
|
|
|
|
|
You can see the effect by running htop in a separate terminal:
|
|
|
|
|
|
|
|
All memory (including swap space) is consumed by the g++ compiler, which consequently crashes.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The issue is under investigation: [ECII-109](https://jira.eso.org/browse/ECII-109)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Solution**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Until the problem is solved, use the following workarounds (in increasing order of invasiveness):
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1. Make sure to not have elasticsearch running
|
|
|
|
|
|
|
|
| systemctl stop elasticsearch |
|
|
|
|
|------------------------------|
|
|
|
|
|
|
|
|
>
|
|
|
|
|
|
|
|
2. Prevent waf from spawning parallel build processes:
|
|
|
|
|
|
|
|
| waf -j1 build |
|
|
|
|
|---------------|
|
|
|
|
|
|
|
|
>
|
|
|
|
|
|
|
|
3. Add temporary swap space to your machine
|
|
|
|
|
|
|
|
> As root:
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th>fallocate -l 8G /swapfile<br />
|
|
|
|
dd if=/dev/zero of=/swapfile bs=1024 count=8388608<br />
|
|
|
|
chmod 600 /swapfile<br />
|
|
|
|
mkswap /swapfile<br />
|
|
|
|
swapon /swapfile</th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
> And to remove the swap space after building:
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th>swapoff -v /swapfile<br />
|
|
|
|
rm -f /swapfile </th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
>
|
|
|
|
|
|
|
|
4. One of the C++ things that take very long are the OPC MAL mappings.
|
|
|
|
|
|
|
|
> As of DevEnv 2.1.17, it is possible to suppress MAL mappings, with this modification the wscript of your ICD module:
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> wscript
|
|
|
|
|
|
|
|
| declare_malicd(use='icds.base', mal_opts = { 'opcua_disabled': True } ) |
|
|
|
|
|-------------------------------------------------------------------------|
|
|
|
|
|
|
|
|
>
|
|
|
|
|
|
|
|
5. Ask your system administrator to assign more RAM to your VM.
|
|
|
|
|
|
|
|
> By default, ECM VMs come with 4 GB, but you want
|
|
|
|
|
|
|
|
- 8 (if you don't run databases like elasticsearch on your host)
|
|
|
|
|
|
|
|
- 12 (if you run databases like elasticsearch on your host).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### KB: More Frames expected \[CII MAL ZMQ\]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Problem**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
In your application you get errors like this:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Oct 16, 2019 11:26:21 AM elt.mal.zpb.ps.ZpbSubscriber events
|
|
|
|
|
|
|
|
WARNING: Remote data entity type hash does not match (1040672065 != 1708154137).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Oct 16, 2019 11:26:21 AM elt.mal.zpb.ps.ZpbSubscriber events
|
|
|
|
|
|
|
|
WARNING: Failed to process message.
|
|
|
|
|
|
|
|
java.lang.RuntimeException: more frames expected
|
|
|
|
|
|
|
|
at elt.mal.zpb.ps.ZpbSubscriber.requireMoreFrames(ZpbSubscriber.java:82)
|
|
|
|
|
|
|
|
at elt.mal.zpb.ps.ZpbSubscriber.events(ZpbSubscriber.java:114)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Background**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The first warning message indicates that your client has received a piece of data (= MAL entity type) on a channel that should not carry such data. This means you are running two publishers, publishing different types of data, on the same channel.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
> **Example of topic definition with port-clash**
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p> <pubsub_topic></p><p> <topic_name>sm:current_pos</topic_name></p><p> <topic_type>sm_current_pos</topic_type></p><p> <address_uri>zpb.ps://134.171.2.220:57110/test</address_uri></p><p> <qos latency_ms="1" deadline_ms="100"/></p><p> <performance rate_hz="10" latency_ms="1" synchronous="false" /></p><p> <mal></p><p> <zpb /></p><p> </mal></p><p> </pubsub_topic></p><p> </p><p> <pubsub_topic></p><p> <topic_name>hp:global_status</topic_name></p><p> <topic_type>hp_global_status</topic_type></p><p> <address_uri>zpb.ps://134.171.2.220:57110/test</address_uri></p><p> <qos latency_ms="10" deadline_ms="100"/></p><p> <performance rate_hz="1" latency_ms="10" synchronous="false" /></p><p> <mal></p><p> <zpb /></p><p> </mal></p><p> </pubsub_topic> </p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The second warning and the error trace are just a consequence of the first warning.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Solution**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Check your topics.xml file, and make sure each channel has its own exclusive topic name.
|
|
|
|
|
|
|
|
In the above example, e.g.:
|
|
|
|
|
|
|
|
> zpb.ps://134.171.2.220:57110/test1
|
|
|
|
>
|
|
|
|
> and
|
|
|
|
>
|
|
|
|
> zpb.ps://134.171.2.220:57110/test2
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### KB: Address in use \[CII MAL ZMQ\]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Problem**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Running your application, you see this error message:
|
|
|
|
|
|
|
|
ZMQException: Errno 48 : Address already in use
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Solution A**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Another instance of your application is still running.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Solution B**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Another instance of your application has non-gracefully terminated without freeing the network port.
|
|
|
|
|
|
|
|
Ensure your application always performs a call to "mal.close()" on shutdown.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Solution C**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This could be really a Usage Error due to wrong configuration.
|
|
|
|
|
|
|
|
The error message is in fact misleading.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
> **Example**
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>eltcii33 [09:38:27] eeltdev:~/mschilli > mal-esotests-testclient1 pub sAddr=zpb://eltcii28:12333/Sample tSlow=100 nSamp=100</p><p>pub:sys: Available MAL Flavours loaded: [dds, opc, zpb]</p><p>pub:config: sAddr=zpb://eltcii28:12333/Sample</p><p>pub:config: nSamp=100</p><p>pub:config: tSlow=100</p><p>Internal Error: org.eso.elt.mal.MalException: org.zeromq.ZMQException: Errno 48 : Address already in use</p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
> **Reason**
|
|
|
|
>
|
|
|
|
> The above code is trying, on host eltcii33, to publish with an endpoint eltcii28.
|
|
|
|
>
|
|
|
|
> **Fix**
|
|
|
|
>
|
|
|
|
> On eltcii33, the endpoint must be eltcii33.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### KB: Scheme not supported \[CII MAL\]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Problem**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Running your application, you get this error message:
|
|
|
|
|
|
|
|
elt.mal.SchemeNotSupportedException: middleware not supported
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Solution A**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
In your code, you've misspelled the middleware name, e.g. "opc" instead of "opcua"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Solution B**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The middleware is supported in fact, but failed to load.
|
|
|
|
|
|
|
|
- E.g. in DDS, you are using a Qos profile xml file which has some illegal syntax inside.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### KB: Choosing a NIC \[CII MAL DDS\]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Problem**
|
|
|
|
|
|
|
|
I'm using MAL-DDS, and I have two network cards (NICs) installed. MAL uses the wrong one, i.e. my network traffic goes into the "office" network, but should go into the "control" network.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Background**
|
|
|
|
|
|
|
|
As multicast addresses are by definition not associated with hardware (ie they map to MAC addresses which have no corresponding Ethernet card), there is no means for the OS to resolve which NIC the IGMP subscription should be sent down. Thus the NIC must be specified, or the default is used (which is the office network).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Therefore, DDS allows you to specify which NIC you want to use for outgoing traffic.
|
|
|
|
|
|
|
|
Therefore, this boils down to configuring DDS.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Solution**
|
|
|
|
|
|
|
|
1. Get the XML file shown in Solution #1 on this page:
|
|
|
|
|
|
|
|
> <https://community.rti.com/howto/control-or-restrict-network-interfaces-nics-used-discovery-and-data-distribution>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
|
|
|
|
2. Continue with this article:
|
|
|
|
|
|
|
|
> [KB: Configuring MAL-DDS](onenote:#KB%20Configuring%20MAL-DDS§ion-id={35F80A9B-72FF-4F78-8793-5C85A09EA1FD}&page-id={3DBEC40D-B2DB-4226-917F-23EAE6A4B01C}&end&base-path=https://europeansouthernobservatory.sharepoint.com/sites/CCS/SiteAssets/CCS%20Notebook/General.one)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### KB: Configuring DDS \[CII MAL DDS\]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Problem**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Some of the middlewares usable through MAL offer a variety of configuration options.
|
|
|
|
|
|
|
|
This article explains how to define and use configuration for the DDS middleware.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Background**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
To configure DDS, the following 3 things are necessary:
|
|
|
|
|
|
|
|
i\) put the desired config into an external XML file
|
|
|
|
|
|
|
|
ii\) define the NDDS_QOS_PROFILES environment variable, so DDS finds the XML file
|
|
|
|
|
|
|
|
iii\) pass 2 keys to the MAL factory, so DDS finds the right profile in the XML file
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
*(For a discussion about making this easier, see [ECII-77](https://jira.eso.org/browse/ECII-77))*
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Solution**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1. Create the XML file.
|
|
|
|
|
|
|
|
> Examples are often easily found in the Connext documentation and user forums.
|
|
|
|
>
|
|
|
|
>
|
|
|
|
|
|
|
|
2. In your code, let MAL load the DDS-specific properties.
|
|
|
|
|
|
|
|
> Example (C++):
|
|
|
|
|
|
|
|
<table><colgroup><col style="width: 100%" /></colgroup><thead><tr class="header"><th><p>auto publisher = factory.getPublisher<elt::telem::Temperature>(uri, {</p><p> std::make_shared<mal::ps::qos::Latency>(std::chrono::milliseconds(100)),</p><p> std::make_shared<mal::ps::qos::Deadline>(std::chrono::seconds(1)) }, {</p><p> {"dds.domain", "100"}, { "dds.qos.profile.library", “test"}, { "dds.qos.profile.name", “UDPv4_properties"}} });</p></th></tr></thead><tbody></tbody></table>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
3. Before running your code:
|
|
|
|
|
|
|
|
| export NDDS_QOS_PROFILES=\<path of XML file> |
|
|
|
|
|----------------------------------------------|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### KB: Variable Tracking exceeded \[CII MAL\]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Problem**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
When building an ICD using CII-MAL,
|
|
|
|
|
|
|
|
> you see this warning message:
|
|
|
|
>
|
|
|
|
> variable tracking size limit exceeded
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Background**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The -fvar-tracking-assignments is automatically enabled by GCC when optimizations are enabled.
|
|
|
|
|
|
|
|
There is a limit on how many variables can be tracked by the compiler. The warning tells you that
|
|
|
|
|
|
|
|
more vars would need to be tracked than what's supported.
|
|
|
|
|
|
|
|
You can disable the tracking manually with -fno-var-tracking-assignments.
|
|
|
|
|
|
|
|
There are two easy ways to do it on the overall project.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Solution A**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Export CXXFLAGS=-fno-var-tracking-assignments
|
|
|
|
|
|
|
|
And then rerun “waf configure” and continue with the build.
|
|
|
|
|
|
|
|
Note: you have to have the exported variable each time you do a “waf configure” as that is the point at which such flags are saved
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Solution B**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
If you want it to be inside the project then put the flags fixed inside your project, so in the top level wscript (where you define the project) add this configure section before the project declaration:
|
|
|
|
|
|
|
|
def configure(cnf):
|
|
|
|
|
|
|
|
cnf.env.append_value('CXXFLAGS', \['-fno-var-tracking-assignments'\])
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### KB: Multithreading in Java
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The document is available on the web:
|
|
|
|
|
|
|
|
> <http://www.eso.org/~mschilli/JmmVisibility.rtf>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<img src="media/image1.png" style="width:8.5625in;height:11.0625in" alt="Machine generated alternative text: JMM Visibility, Marcus Schilling, November 2012. All rights reserved. Introduction This page is meant to raise awareness on thread-safety issues in multi-threaded Java applications. It is essential that you ensure your code contains proper memory barriers (= synchronization points), especially on multi-core machines. For more details, see Chapter 17 of the Java Language Specification (and section 17.4.5. Happens-before in particular) · Examples of non-thread-safe java code · (Almost complete list of) Ways to code a memory barrier in Java · Examples of thread-safe java code · Performance considerations Examples of non-thread-safe java code In the example scenario there are 2 threads A and B . First, Thread B instantiates class Bad. Then, a few seconds later, Thread A executes method a() . Then, a few seconds later, Thread B executes b() . class Bad { String name = "bob"; void a() {name = "alice";} void b() {print(name);} } ==> Thread B will print "bob" or "alice". class Bad { String[] names = {"alice", "bob"}; void a() {names[0] = "albert";} void b() {print(names[0]);} } ==> Thread B will print "alice" or "albert". class Bad { volatile String[] names = {"alice", "bob"}; void a() {names[0] = "albert";} void b() {print(names[0]);} } ==> Thread B will print "alice" or "albert". class Bad { volatile List names = new ArrayList(); void a() {names.add("alice");} void b() {print(names.size());} } ==> Thread B will print "0" or "1". " /><img src="media/image2.png" style="width:8.5625in;height:11.0625in" alt="Machine generated alternative text: class Bad { class Data { int x; } volatile Data data = new Data (); void a() {data.x = 1;} void b() {print(data.x);} } ==> Thread B will print "0" or "1". ========================= The above examples may seem surprisingly trivial. You need to be aware that the problem is not about concurrent modification of shared data. Except the example with the ArrayList, all the read and write operations in the examples are atomic (per se, because reference access is atomic). So, there are no race conditions in the above examples. And it's even more obvious from the fact that our threads run only every few seconds. The problem is the visibility in terms of the java memory model (called jmm visibility hereafter): There is no guarantee that Thread B will see the modifications done by Thread A . Each thread has its own copy of the main memory that needs to be refreshed from main memory and flushed to main memory. The flushing and refreshing is triggered when a thread passes a memory barrier. To have a jmm visibility guarantee, both threads must pass a memory barrier (or synchronization point). Next, it is CRUCIAL to be aware that the amount of refresh (= the changeset, = which subset of the dirty memory gets refreshed) fully depends on which memory barrier you use, there is no such thing like one single all-in-one refresh that brings your reader thread fully up-to-date. This means in order to have a jmm visibility guarantee, both threads must pass the same memory barrier, namely the one that includes all modifications needed by the reader . class Bad { List one = new ArrayList(); List two = new ArrayList(); void a() { synchronized(one) {one.add("alice");} synchronized(two) {two.add("bob"); } } void b() { synchronized(one) {print(one.size() + two.size()); } } } ==> Thread B will print "1" or "2". class Bad { class Data { int x; } volatile List list = new Vector(); void a() { Data data = new Data(); synchronized(data) { list.add(data); data.x = 1; } } void b() { print (list.get(0).x); } } ==> Thread B will print "0" or "1". In big classes with much code, this particular problem can be difficult and tedious to find. The code may look thread-safe, but in reality there are two threads passing different memory barriers, and thus are actually not synchronizing their data! Another remark: The jmm visibility has nothing to do with the access modifiers public , protected , " /><img src="media/image3.png" style="width:8.5625in;height:11.0625in" alt="Machine generated alternative text: or private . You can put any of them onto your variables without any impact on the jmm visibility . The only connection to our problem is this: Depending on how you coded your memory barrier, making a field private can prevent another Thread C from bypassing your memory barrier. Example: class Unstable { public int x; synchronized void a() {x=1;} synchronized void b() {print(x);} } ==> Thread C will see "x == 0" or "x == 1". ========================= (Almost complete list of) Ways to code a memory barrier in Java Flush (by Thread A) Refresh (by Thread B) Note Write access to volatile variable Read Access to volatile variable Both threads must, see above, access the same variable End of a synchronized section Begin of a synchronized section Both threads must, see above, synchronize on the same variable Write access to an AtomicInteger (etc.) Read Access to an AtomicInteger (etc.) Both threads must, see above, access the same variable Examples of thread-safe java code class Okay { volatile String name = "bob"; void a() {name = "alice";} void b() {print(name);} } class Okay { String name = "bob"; synchronized void a() {name = "alice";} synchronized void b() {print(name);} } class Okay { volatile String[] names = {"alice", "bob"}; synchronized void a() {names[0] = "albert";} synchronized void b() {print(names[0]);} } class Okay { volatile List names = Collections.synchronizedList(new ArrayList()); void a() {names.add("alice");} " /><img src="media/image4.png" style="width:8.5625in;height:11.0625in" alt="Machine generated alternative text: void b() {print(names.size());} } class Okay { class Data { volatile int x; } volatile Data data = new Data (); void a() {data.x = 1;} void b() {print(data.x);} } ========================= Performance considerations 1) volatile is cheaper than synchronize 2) BUT this is no longer true if you have loads of volatiles . 3) THUS, a single synchronize is cheaper than multiple volatiles . class Slower { volatile int x; volatile int y; volatile int z; void a() {x=1; y=1; z=1} void b() {print (x); print (y); print(z);} } class Faster { int x; int y; int z; synchronized void a() {x=1; y=1; z=1} synchronized void b() {print (x); print (y); print(z);} } 4) HOWEVER, code protected by synchronize is mutex, it cannot be executed in parallel. THUS, massive use of synchronize has massive impact on the scalability of your code. AND, synchronize has the power to create deadlocks, which volatile cannot. With these pros and cons in mind, you might conclude that you'll prefer some performance penalty with volatile over potential deadlocking and loss of scalability with synchronize . And I tend to agree with you. So, forget about synchronize ? You still need it to avoid race conditions , that is when one thread i) makes a modification to a shared data structure and ii) this modification should look atomic to another thread, but iii) cannot be performed atomically per se (so for any data structure beyond references, simple data types, and Atomic variables). This is something that volatile cannot give you. 5) THEREFORE, you will need to find a compromise between the use of synchronize and the use of volatile . Use synchronized sections but keep them short. Use volatile fields but not too often. 6) final is cheaper than volatile - if you can, use final instead of volatile class Slower { volatile String[] names = {"alice", "bob"}; " /><img src="media/image5.png" style="width:8.5625in;height:11.0625in" alt="Machine generated alternative text: synchronized void a() {names[0] = "albert";} synchronized void b() {print(names[0]);} } class Faster { final String[] names = {"alice", "bob"}; synchronized void a() {names[0] = "albert";} synchronized void b() {print(names[0]);} } " />
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### KB: Summary of OPC/UA MAL in C++
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This article covers integration of OPC/UA in CII MAL specifically for OPC/UA Data Access and Subscription profiles. OPC/UA method invocation is also supported in CII MAL but is not described in this article, likewise details of the Python (and Java) support are not provided. Only C++ is considered here.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
OPC/UA communication middleware is exposed in CII MAL as either Publish/Subscribe or Request/Reply APIs.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The XML ICD definition of types in CII is used to map sets of data points together that are read/written as a group.
|
|
|
|
|
|
|
|
Each attribute in the defined type is connected to a corresponding data point in the OPC/UA data space via a URI. Thus the CII URI for a complex type will contain specific addresses of multiple nodes in the OPC/UA data space.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
#### Pub/Sub API for OPC/UA Clients:
|
|
|
|
|
|
|
|
The CII MAL Pub/Sub API utilizes OPC/UA Data Access Reads and Writes, as well OPC/UA subscription. A Publisher will directly trigger an OPC DA write, while a Subscriber will work in one of two ways, depending on the type associated with the subscriber:
|
|
|
|
|
|
|
|
- If the subscribers CII URI contains only a single node (i.e. the XML ICD type contained only a single attribute) then the Subscriber will create an OPC/UA subscription on that data point. The subscription will trigger notification of updates to the data point node, which will then be queued for notification via the CII Subscriber API.
|
|
|
|
|
|
|
|
- If the Subscriber is using subscription, the opc.ps.outstandingPublishRequests property should not be zero (e.g. set it to 5), see the example code below. The reason is that the publish queue is used to store and send subscription notification events, and if the queue is small the even notifications may simply be dropped.
|
|
|
|
|
|
|
|
- If the subscriber CII URI contains multiple nodes (i.e. the XML ICD type contains multiple attributes) then the Subscriber launches a thread to perform periodic polling of data from the OPC/UA server. The rate is based on the properties passed in creating the subscriber. e.g.
|
|
|
|
|
|
|
|
> try {
|
|
|
|
>
|
|
|
|
> subscriber = factory.getSubscriber\<T>(opcua_uri, ::elt::mal::ps::qos::QoS::DEFAULT,
|
|
|
|
>
|
|
|
|
> {{"opc.ps.outstandingPublishRequests","5"},{"opc.asyncLoopExecutionPeriodMs","50"},
|
|
|
|
>
|
|
|
|
> {"opc.asyncCallSubmitTimeoutMs","1000"},
|
|
|
|
>
|
|
|
|
> {"opc.ps.pollingPeriodMs","20000"},
|
|
|
|
>
|
|
|
|
> {"opc.asyncCallRetryPeriodMs","250"}});
|
|
|
|
>
|
|
|
|
> }
|
|
|
|
>
|
|
|
|
> catch(...) {
|
|
|
|
>
|
|
|
|
> throw;
|
|
|
|
>
|
|
|
|
> }
|
|
|
|
>
|
|
|
|
>
|
|
|
|
|
|
|
|
#### Request/Reply API for OPC/UA Clients:
|
|
|
|
|
|
|
|
As OPC/UA Data Access read and write essentially follow a synchronous request/reply pattern, CII MAL also provides this interface for OPC/UA clients.
|
|
|
|
|
|
|
|
The ICD is termed "virtual" in CII nomenclature as it does not require definition as an XML ICD using the service syntax, rather the same types defined for the Pub/Sub API may be used with a CII MAL OPC/UA Request/Reply.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This approach means OPC/UA Data Access read (and write) are synchronous, and may be called as needed by the application.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
> namespace mal {
|
|
|
|
>
|
|
|
|
> namespace rr {
|
|
|
|
>
|
|
|
|
> namespace da {
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> class DataAccess : public ::elt::mal::rr::RrEntity {
|
|
|
|
>
|
|
|
|
> public:
|
|
|
|
>
|
|
|
|
> \[...\]
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> template \<typename T>
|
|
|
|
>
|
|
|
|
> void read(::elt::mal::ps::DataEntity\<T>& value) {
|
|
|
|
>
|
|
|
|
> readUnsafe(&value);
|
|
|
|
>
|
|
|
|
> }
|
|
|
|
>
|
|
|
|
>
|
|
|
|
>
|
|
|
|
> template \<typename T>
|
|
|
|
>
|
|
|
|
> void write(const ::elt::mal::ps::DataEntity\<T>& value) {
|
|
|
|
>
|
|
|
|
> writeUnsafe(&value);
|
|
|
|
>
|
|
|
|
> }
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
A test application showing its use is here:
|
|
|
|
|
|
|
|
<https://gitlab.eso.org/cosylab/elt-cii/mal/mal-test/-/blob/develop/cpp/mal-test-performance/opcua/mal-opcua-da-speed/src/common.cpp>
|
|
|
|
|
|
|
|
|
|
|
|
|
