FAQ

Supported Java Version

ECP 4.7.2 and older support Java 8 (both Oracle Java and Oracle OpenJDK are supported) ECP 4.7.1 and older are not compatible with Java 1.8.0_272 and newer.

ECP 4.8.0 and newer are compatible only with java 11 There is a compatibility issue with java 11.0.16. For more information see the ECCOSP-226 ticket.

Supported Oracle version

The latest versions of ECCo SP (ECP 4.9.0 and EDX 1.11.0) currently supports Oracle 12c Release 2 and Oracle 19c.

Nevertheless, the extended support for Oracle 12c Release 2 ended on 31.07.2022, so Oracle 19c is recommended.

For ECP 4.6.1/EDX 1.7.1 and older, only Oracle 12c Release 2 is supported.

Sock Proxy Config

Ticket: 9332633

There seems to be an issue with the usage of a SOCKS 5 – proxy in the ECP4 – environment, I’d like to report to you.

1. Setup

a. ECP4 is installed on Windows Server 2016

b. The connection to ecp.entsoe.eu is directed via SOCKS v5 – proxy (CISCO Ironport)

c. The Proxy – Settings are configured in ecp.properties

ConfigSetting
ecp.security.proxy.enabled=true
ecp.security.proxy.host=hostname
ecp.security.proxy.port=port
ecp.security.proxy.nonProxyHosts=IP of ECP4 - Server
ecp.security.proxy.username=username for proxy
ecp.security.proxy.password=WINDOWS Active Directory Password for Proxy-User !

2. Observed behaviour

a. Synchronisation of Component Directory (HTTPS) works without issues

b. The communication via AMQPS (Ports 5671, 5672) is not stable. That is, the connection gets lost and can’t be re-established for many hours. If the connection is established, it stay’s established for an arbitrary time (some hours up to 2-3 days).

c. Even if we can’t upload files to broker 10V1001C—000438 and 10V1001C—000446, we still can receive data via these brokers (ENTSOE-PSD messages).

3. Changes necessary to get the connection stable

a. Configure SOCKS Proxy to allow the complete server to access ecp.entsoe.eu on the given ports

b. Omit ecp.security.proxy.username and ecp.security.proxy.password from ecp.properties

4. Conclusion

a. In the setup shown above, the communication via SOCKS Proxy isn’t stable for AMQPS if the proxy – permissions are granted via User – Credentials (maybe especially, if this user is authenticated with Windows Active Directory or ECP4 / Tomcat is running on Windows Server))

b. Maybe the documentation of how to use a SOCKS – Proxy in ECP-environment isn’t complete in the sense that OS-specific information is missing

Disconnection Issues

Disconnection issue (9334178):

The disconnection issue was reported by APG in a ticket no. 9334178. After the issue analysis, it was found that EDX is not able to connect to the ECP properly. According to logs, ECP was working properly.

On EDX there was exhausted persistent storage for messages in ActiveMQ and the recovery does not finish properly.

Please perform the following procedure:

  1. stop EDX Toolbox
  2. delete folder “db” by default it is in C:\Program Files\EDX Toolbox
  3. delete folder “edx-activemq-data” in E:\opde\edx
  4. start EDX Toolbox
EDX Status Document questions

Question 1: Does the “StatusDocument” include the original message metadata (The Pull message notification)? If it is not the case, is there any way to retrieve the original message metadata ?

Status document contains the following metadata:

  • originalmessageID v=””/
  • receiveTimestamp v=””/
  • status v=””/
  • changeTimestamp v=””/

Question 2: Where can we find the XSD schema/documentation of a “StatusDocument” ?

XSD schemas could be found in EDXInterfaces-1.7.0.834.zip, which should be stored to https://extra.entsoe.eu/

Question 3: Is it possible to change/re-configure the max retries to download a pull message content ?

The reconfiguration could be done by adding the following lines into edx.properties file:

  • edx.toolbox.pull.downloadRetryCount= retry count
  • edx.toolbox.pull.retryJobDelay= delay of retry job in seconds
Generation of Root & Integrated Certificate for Windows

Document created by Swissgrid about it : selfSigned_jks.docx

A guide for the generation process of root and integrated certificate for Windows and KeyStore Explorer:

  1. Start KeyStore Explorer application
  2. Select “Create a new KeyStore” and type “JKS”
  3. Creation of root (global) certificate
Click on Tools -> Generate Keypair
Generate Key Pair screen and Select RSA Algorithm with key size 4096, then click OK

4 Creation of integrated certificate (for ECP CD)

   Right-click on global certificate and select Sign -> Sign New Key Pair
   Generate Key Pair screen
      - Select RSA Algorithm with key size 4096, then click OK
   Generate Key Pair Certificate screen
      - Select Version 3, SHA-256 with RSA, validity period e.g. 15 years and click Apply
   Fill the name fields - email, common name, organization, locality name and country
      - ECP4.4 : Common name has to be vCode of the ECP CD
   Add Extensions
      - Authority Key Identifier - Key Identifier 160-bit hash
      - Basic Constraints with Critical flag - Mark "Subject is a CA"
      - Key Usage - Select "Digital Certificate", "Certificate Signing", "CRL Signing"
      - Subject Key Identifier : Key Identifier 160-bit hash
   Enter Alias "integrated"
   Enter new password "password"

5 Save the created keystore and place it on safe place - this is the keystore which can be used for generating certificates for another ECP CDs

6 Right click on global certificate and select Export -> Export Certificate Chain

 Head Only, X.509, PEM
 Save it as global.cer

7 Right-click on integrated certificate and select Export -> Export Key Pair

 Save it as integrated.p12

8 Compilation of keystore for ECP CD

 Click on File -> New and select JKS
 Click on Tools -> Import Trusted Certificate and select file global.cer
    - Fill Alias "global"

9 Click on Tools -> Import Key Pair, select PKCS #12 and select file integrated.p12

 Fill Alias "integrated" and password "password"

10 Save the created keystore as ecp_module_cd.jks

11 After installation of ECP CD, use ecp_module_cd.jks for registration of ECP CD

How to access Derby DB

Please note that connection to the embedded Apache Derby database is not a common administrator task, so it is not described in ECP Administration/Installation Guide.

The connection via client DBeaver is described in ECP & EDX Migration guide, chapter ”3.1.1 Connect to embedded database” as follows:

  1. Stop ECP Endpoint
  2. Copy database from ECP server to location, where DBeaver client is installed.

Apache Derby database is stored in the folder db on path specified in ECP configuration file ecp.properties – dataDirectory value.

Default location of the embedded database:

  • Linux: /var/lib/ecp-endpoint/db
  • Windows: /db

3. Start DBeaver client, open ECP database

  • Open DBeaver application, go to the top menu,
  • select File > New > Database connection > Expand folder DBeaver > Database Connection > Expand folder Derby > Embedded >
  • Click on the button “Browse” and traverse to “db” directory,
  • fill in credentials ecp/ecp-password > Next > Finish​

4. In the left menu, go to General > Connections > locate your ECP database (name is by default “Derby - db”) > ECP > Tables ​​- ECP Messages are stored in the table MESSAGE, the content of unprocessed messages in table CONTENT_STORAGE.

  • After Table data modification, save the changes and upload the database directory back to the ECP Endpoint server. Please note that the default ECP retention period for messages is 14 days, you can change retention period in order to keep less data (ECP Administration Guide, chapter 5.16):

Property: ecp.messagebox.retentionPeriod

  • Defines the retention period of the message box in milliseconds. When the retention period of the archived messages expires, they will be deleted from the message box.
  • 1000x60x60x24x14 (14 days)

Property: ecp.messagebox.messageDeletingRunPeriod

  • Define the period of how often (in milliseconds) the message-deleting routine will be run.
  • 10006060 (one hour)

For testing purposes, the fast cleanup option could be to delete directories db and internal broker from ECP data directory after ECP Endpoint is stopped. Please note that new registration of ECP Endpoint is required when data directories are deleted, a new empty database is created automatically on the next startup.

How to clear messages in ECP

ECP implements a message deleting functionality to free the Message Store from the old processed messages. The message retention period is configurable and can be adapted to the traffic on specific ECP installation. The ECP instances with high traffic will set the period to a lower interval (e.g., several hours) than the instances with low traffic (e.g., several days). By default, the retention period is configured to two weeks.

ECP implements a mechanism to prevent the contentions and slow performance. When the amount of the stored messages exceeds a threshold, ECP emits a warning message advising to reduce the message retention period. When the amount of the stored messages exceeds a critical threshold, the messaging functions of ECP are paused until the messages amount drops. In such case, the user is warned via an information message and is advised to immediately execute the deleting process. Both thresholds are configurable to adjust the limits in case of emergencies.

There are several configuration properties that influence message box behaviour. See descriptions below:

PropertyDescriptionDefault value
ecp.messagebox.retentionPeriodDefines the retention period of the message box in milliseconds. When the retention period of the archived messages expires, they will be deleted from the message box.1209600000 = 100060602414 (14 days)
ecp.messagebox.messageDeletingRunPeriodDefine period of how often (in milliseconds) the message deleting routine will be run.3600000 = 10006060 (1 hour)

Alternative way

Another possibility is delete messages directly from database using following SQL commands:

DELETE FROM ECP.MESSAGE;

DELETE FROM ECP.SENT_MESSAGE_REGISTER;

How to clean messages in EDX

By default, EDX Toolbox runs deleting job every two minutes, which deletes messages from the database and contents from DMS, which are older than 168 hours. Periods for deleting from the database and from DMS can be configured independently, so it is possible to delete DMS contents and keep the messages in the database for some time. Because it is not possible to find DMS content after the message record is removed from the database, removing of the database record also deletes DMS content if it is still present in DMS. Following properties can be used for deleting job configuration:

Alternative way

Another possibility is delete messages directly from database using following SQL commands:

DELETE FROM EDX.TOOLBOX_MESSAGE WHERE PARENT_MESSAGE_ID IS NOT NULL;

DELETE FROM EDX.TOOLBOX_MESSAGE;

DELETE FROM EDX.PULL_MESSAGE;

DELETE FROM EDX.INFLIGHT_EXTERNAL_PROCESSING;

DELETE FROM EDX.TOOLBOX_MESSAGE_ECP_DELIVERY;

DELETE FROM EDX.TOOLBOX_MESSAGE_LOG;

If necessary, files on DMS (/usr/share/edx-toolbox/temp/edx-dms/edx-01/) can be deleted by command: find /usr/share/edx-toolbox/temp/edx-dms/edx-01/ -mtime +3 -delete.

Please note that the parameter for age (3 days) should be equal to edx.toolbox.deleting.deleteOlderThan configuration (72h).

Files to this directory: /opt/opdm-data/edx-pull/in/ are incoming to your EDX Toolbox from RSC (Central components). These files are then processed by OPDM Client.

However, OPDM Client in version 2.7.1 does not clean up the files after being processed. This will be implemented in the new version of the OPDM Client.

You may use the “find /opt/opdm-data/edx-pull/in/ -mtime +3 -delete” command in cooperation with Cron, to handle the count of files in this directory.

Receiver Handler Issues

We have found out the root cause of the issue, the ECP4 Endpoint runtime configuration was not loaded properly.

There was the following parameter configured:

  • ecp.endpoint.sendHandler[0].className

As a workaround we have removed all the parameters from runtime configuration, uploaded the clean runtime configuration and then we have added again following parameters necessary for integration with EDX Toolbox: 

  • ecp.endpoint.sendHandler[0].beanName=amqpApiSendHandler
  • ecp.endpoint.sendHandler[0].typeName=*
ECP4 Certificate Renewal Issues Summary

The propose of the document is to summarize issues during certificate renewal on ECP Endpoint through various application versions.

ECP 4.3.x

ISSUE

  • In ECP 4.3.x there is an issue in selecting a proper certificate in case that more certificates of the same type exist (e.g. signing, encryption).
  • The ECP Endpoint of this version selects as default the certificate with minimal “valid from” date and does not take into account the expiration date of the certificate.

This issue was resolved in ECP 4.4.x.

  • In case there are ECP 4.3 and 4.4 in-network, which is valid for TP. Then it is necessary to delete old certificates on ECP 4.4 as well. (TP ECP Endpoint has 4.4 and after certificate renewal, the DP with ECP 4.3 was not able to communicate.)

WORKAROUND

  • After certificate renewal, it is necessary to delete old certificates and push their configuration to ECP CD. These actions can be done from GUI.

-—————-

ECP 4.4.x

ISSUE

  • During certificate renewal ECP Endpoint remains in registering state in some cases.
  • Certificates are renewed and pushed, but the state of application causes the messaging issue

The other issue reported is that ECP Endpoint does not push renewed certificates into ECP CD. It can be revealed by finding the own ECP Endpoint in the Component list on ECP Endpoint GUI.

WORKAROUND

  • Connect to database, in table “application_properties” change value of “ecp.internal.status” to ACTIVE.
  • Manually push configuration from GUI (Settings -> Push Configuration)

-—————-

ECP 4.6.x + ECP 4.7.x

The issues from the previous version are resolved. There are no reported issues with certificate renewal.

During FAT the behaviour has been retested on ECP 4.7.2 and certificates were renewed properly.

VersionIssue ReportedAutomatic Renewal Working
4.3.xYesNo
4.4.xYesNo
4.6.xNoYes (Since 11/2019, by default automatic renewal disabled)
4.7.0 and aboveNoYes (since 07/2020, automatic renewal is enabled by default)
Endpoint login using REST interface

ISSUE Having upgraded the ecp-endpoints to v4.6, you can no longer login using REST.

Performing the same procedure carried out on a v4.6 endpoint, returns:

{ “timestamp”: “2020-01-03T11:39:58.198+0000”, “status”: 403, “error”: “Forbidden”, “message”: “Invalid CSRF Token ‘’ was found on the request parameter ‘_csrf’ or header ‘X-XSRF-TOKEN’.”, “path”: “/ECP_MODULE/login” }

Why is the endpoint expecting a CSRF token on login?

RESPONSE ECP Endpoint does not have a separate REST interface, it was intended to access only by dashboard interface. That is the reason that CSRF protection is always enabled. But as a workaround for access by REST, you could try to call GET request on the login page, get the CSRF token from the response and use it for the POST request. 

______________________________________________________________________________________

ISSUE When trying to execute the messaging connectivity check, using the same CSRF token as in the first check, I get a failure:

status=403; error=Forbidden; message=Invalid CSRF Token ‘e22b2862-c21c-489f-a42c-99d7f77db8bd’ was found on the request parameter ‘_csrf’ or header ‘X-XSRF-TOKEN’.; path=/ECP_MODULE/settings/connectivityCheck

I’ve also tried to skip the CD connectivity check, but the messaging connectivity check still fails. So, how do I get the CSRF token for the messaging connectivity check, if not using a GET on the /login URL?

RESPONSE CSRF token changes after login, but then it will remain the same until logout. So after login you will need another token, which can be extracted from the response of the GET request to the application root (*/ECP_MODULE/).

Register Toolbox to SC from Toolbox GUI

I’ve a problem with edx-toolbox (V. 1.8.2). The toolbox has to be registered to the respective Service Catalog (10V1001C-00273W in this case), but the button “+Register Toolbox to SC” is deactivated (see screenshot).

-> Thank you for the question. I investigated a little bit and I realized that it is not straightforward. I think a pop-up is present when you put your mouse on the button.

The principle is that a toolbox needs to be first registered in the Service Catalogue manually. After, the toolbox can register itself in other Service Catalogue through that button.

Please request to the Service Catalogue admin or project manager to add your toolbox in the SC.

--> Issue fixed by user: This issue has been resolved. Is was a typo in the ServiceCatalog EI-Code.

ECP certificates outdated - what to do?

If you are running an old verison (before 4.8) of ECP the automatic renewal doesn’t work correctly.

One way to overcome the issue is register the endpoint with a new Keystore provided by CD admin. Otherwise, register again the same V-code, once approved, it will over-write the previous entry in the CD !!! Be aware that “configuration push” might be needed from the endpoint! Depending on the version used.

EDX - Failed to connect to remote at amqps

Error message:

  • Failed to connect to remote at: <amqps://xxx>

This is usually causes by the endpoint being done (not running) and not accessible.

  • ECP Endpoint is not running
  • ECP Endpoint does not „trust“ the EDX Toolbox

    • ECP EP and EDX TB should have certificates with the same root CA. If the root CAs are different, ECP EP can reject the connection
    • Toolbox user and group must be defined on ECP EP in user.properties and groups.properties. For more information see the EDX Installation Guide, chapter 6.1.4. General prerequisites before installing EDX Toolbox
EDX - Sent message is marked as failed

Possible causes:

  • TTL of ECP Endpoint is expired - ECP EP has not been able to synchronize with CD for more than 28 days
  • Receiver code doesn’t exist in the ECP Network - there is probably a typo in the receiver code or receiver EP has been revoked
  • EDX Toolbox has not been able to find the pull message content – the content has not been copied to the correct pull.root.out directory or the content path in the notification document is invalid
  • In case that embedded pull is used, message payload exceeded the maximum message size
  • Receiver Toolbox is not in the same EDX Network as sender Toolbox – sender and receiver are not registered in the same Service Catalogue
ECP - How to activate hawtio?

This is how to deploy hawtio to your ECP Endpoint:

  1. Download Hawtio web war from MAVEN Central Repository (https://mvnrepository.com/artifact/io.hawt/hawtio-web/1.5.11)
  2. Rename downloaded hawtio-web-1.5.11.war to hawtio.war
  3. Drop the hawtio.war file to\tomcat\webapps
  4. Set property spring.jmx.enabled in your ecp.properties to true
  5. Restart the Endpoint – the plugin should be deployed
  6. You can then access the hawtio GUI by opening address https://:/hawtio in your web browser.
ECP - How to manage queue in hawtio with encrypted messages

Issue: ECP Endpoint was not able to decrypt incoming messages.

We will thus need to move away the messages in queue ecp.endpoint.download as they contain messages that your Endpoint is unable to decrypt. To do this, could you, please, follow these instructions:

  1. Open hawtio
  2. Click on the ecp.endpoint.download in the menu on the left side
  3. Click on the Operations tab in the upper bar
  4. Click on the operation called moveMatchingMessagesTo(java.lang.String,java.lang.String)
  5. Leave Selector field blank
  6. Fill ecp.endpoint.download_backup in Destination name
  7. Click on Execute in the right upper corner
  8. Restart the ECP Endpoint

After this had been done, we need to make sure the Endpoint is able to decrypt new incoming messages.

ECP - SOCKS Proxy implementation
  1. Does the AMQPS messaging protocol natively support SOCKS or is a ‘Socksifier’ required to make this work?

    • AMQPS supports SOCKS natively.
  2. Does traffic coming via socks support QoS?

    • QoS is not supported.
  3. I’m assuming SOCKS 5 will be used for authentication? Where will authentication handoff reside? On the F5/LDAP?

    • SOCKS5 has possibility to fill in credentials (username/password) for connection to SOCKS5 server. Username and password is defined in the ECP configuration file.
  4. How can we guarantee against nefarious use of SOCKS here?

    • SOCKS proxy is used only for outgoing connection, you can define which local server can connect to SOCKS server and which local server will be forwarded to remote host. The connection to SOCKS server can be secured by user/password.
  5. Support for SSL on the AMQPS.

    • SSL is always enabled when AMPQS is used.
How to limit publications on EDX

High Level Description

  • No implementation necessary
  • Services and Domains can be used for publication limitation.

Affected Components

  • EDX Toolbox
  • EDX Service Catalogue

Services and Domains can be used for publication limitation.

Toolbox will automatically consume all services in its domains, this means that toolbox will be able to subscribe all publications with defined service in its domain. Toolboxes from different domains will not be able to subscribe.

Note: When publication is created without service (only domain is defined), all toolboxes across all domains (within same SC) will be able to subscribe the publication.

Example scenario:

There are three Toolboxes registered in the same Service Catalogue and there will be two publications, Pub-TSO-only for TSO1 (as publication provider) and TSO2 only, and Pub-All for both TSOs and NEMO (as publication provider). The limitation can be done by following steps.

  1. On Service Catalogue create domain for TSO1 and TSO2 - e. g. TSOs.
  2. In domain TSOs create service e. g. TSOservice.
  3. On TSO1 create publicaton in domain TSOs and service TSOservice. This publication will be available only for TSO1 and TSO2 because NEMO is in different domain which means it will not consume TSOservice.
  4. Subscribe the publication on TSO1 and TSO2.
  5. On NEMO create publicaton in any domain and without service. This publication will be available for all Toolboxes across all domains.
  6. Subscribe the publication on TSOs and NEMO.
What are the login/password to access the broker AMQP GUI?

The credentials are located in the following path

  • (on Linux) /opt/ecp-broker/activemq/conf/jetty-realm.properties
  • (on Windows)\activemq\conf\jetty-realm.properties
How to allow connection to ECP Broker ActiveMQ GUI?

To allow connection to GUI from other locations (not just from the server where the ActiveMQ is installed) it is necessary to change the following configuration file:

  • (for Linux) /opt/ecp-broker/activemq/conf/jetty.xml
  • (for Windows)\activemq\conf\jetty.xml

Find element <property name=”host” value=”127.0.0.1”/> and change the IP address from localhost to the server’s IP address. Then restart ECP Broker.

After this, the ActiveMQ GUI will be available on http://:8161

GET THE MOST POWERFUL NEWSLETTER IN BRUSSELS