Yesterday we encountered the issue, that the docker image build by Bamboo was not able to run on the agent itself, but locally did. As with docker this should not happen, so we have to investigate. It turns out, that some of our JAXB dependencies we put into ${karaf.home}/lib fail while loading. Simply removing them solved at least the Smoke Test failures. The last test failing, is probably a flapper.

Today I spend half of the day documenting the adventure of Updating Karaf to Version 4.2.2/4.2.3. Besides that a bunch of KarafTestCaseITs were still failing. As the issue is not solvable for now I decided to @Ignore them. Another issue is, that some of the system tests, e.g. HealthCheckIT fail on Bamboo, but pass locally on my system. Usually if something fails on Bamboo it can be reproduced locally and then fixed.

I thought with all the issues I already fixed, now everything should work again. NOPE. Bamboo is telling me that NxosTelemetryIT is no longer working. Okay, that is easy to reproduce with my system-test.sh script, inspired by some version from Jesse White. Running the script with the Bamboo URL to the build, e.g. /system-tests.sh https://bamboo.opennms.org/browse/OPENNMS-ONMS2773-14 as an argument will download the RPMs and build the same docker images as Bamboo would.

Until now we have fixed most of our issues. However one of the remaining issues were, that locally I was not able to login to the Karaf Shell for all the docker containers of Minion/Sentinel/OpenNMS. I thought that this was the same issue as reported by Bamboo. But it turned out, that it was a local Docker on Mac VM issue. Resetting the Docker VM solved at least that problem.

Yesterday we spend all day fiddling with our KarafTestCases and finally disabled them. Today I thought we were going to fix some of our failed Smoke Tests, but instead we are going to do something different. Looking at the docker containers build for our System Tests, I encountered the following exception in the karaf.log. Caused by: java.lang.IllegalArgumentException: Not supported: http://javax.xml.XMLConstants/property/accessExternalDTD Consulting the Oracle pointed me to a topic on the Karaf Mailing List.

Yesterday we fixed all Vaadin related issue and were very positive to have fixed a lot of issues and pushed our changes to Bamboo to see what actually all is broken. Integration Test wise, all Karaf related tests failed. I investigated the issue and long story short: Hacking the startup.properties in our KarafTestCase is no longer working, instead the library must be provided otherwise. Executing the tests don’t work anymore and result in a ClassNotFoundException, bundle wiring is no longer valid

Yesterday we learned, that Vaadin UIs are no longer working as they should. The sympthoms were that the first Vaadin UI you visited were always presented to you, no matter which one you accessed afterwards. My first finding was, that this is caused by the way Vaadin implements the @PreserveOnRefresh functionality. See NMS-10601 for more details. However after I addressed/fixed that issue, the problem was not. Debugging some more showed, that only one OsgiUIProvider was present in the VaadinSession, but multiple should be.

Last week I started replacing our forked Http Bridge with a more OSGi friendly and generic approach. I got to a point where it somewhat worked, but Vaadin UIs seem to have a problem with the new structure. So today I am going to investigate the issue some more. Debugging the issue showed that the widgetset property, required for Vaadin UIs is not propagated properly, resulting in loading the DefaultWidgetSet which - for the Topology UI - is missing certain components.

Yesterday we started to to replace our custom Http Bridge with the official Http Felix Bridge. I started implementing a ProxyFilter to dispatch all requests to the OSGi world if it can be handled by any OSGi registered service to make /opennms/topology work instead of using /opennms/osgi/topology. While finalizing this initial work, I encountered a problem that service properties supported by the PAX WEB project are not officially supported by OSGi and with that not supported by the Apache Felix HTTP project.

Yesterday I found out, that we can no longer use our forked Http Service Bridge. This is good for once, as we can now finally get rid of it. But on the other hand this, will be a lot of work. So let’s get started. Which version to use As we can no longer use the pax-whiteboard bundle as we used to, we have to find another solution. I asked the Oracle and found out, that the Apache HTTP Felix project provides a "

Yesterday I learned, that the pax-whiteboard bundle is no longer starting. Investigating the issue some further revealed, that when activating the bundle an Exception java.lang.IllegalStateException: HttpService must be implementing Pax-Web WebContainer! is thrown. This is a known issue (with OpenNMS), but before Karaf 4.2.2 it just worked. My guess is, that the Exception was triggered AFTER the bundle was started. However now this happens when the bundle is still starting and therefore failing.

Last week I started upgrading Karaf to version 4.2.2. You can see the endevour started at Day 1 and continued at Day 2. At Day 2 we left off with three exception in the karaf.log. While two were something like "Cannot register an already registered Mbean" which are most likely caused by some code (inside Karaf) trying to register already registered Mbeans by Jetty itself. So the "real" Exception to investigate here is the issue with the plugin manager.

While yesterday I managed to fix One exception, a bunch of more showed up. I wanted to investigate those isuses today, but found a weird behaviour: After a restart of OpenNMS the web console was no longer available. Besides that, I got a ClassNotFoundException org.eclipse.jetty.jaas.JAASLoginService in the logs. First thing I did was to add the dependency <dependency> <groupId>org.eclipse.jetty</groupId> <artifactId>jetty-jaas</artifactId> <version>${jettyVersion}</version> </dependency> to OpenNMS and copy it to the lib directory.

We are attempting to get OpenNMS to run on Java 9 . The first step of this is to make it compile for Java 9, which we are working on actively . However making it run/compile on Java 9 requires a lot of dependencies to be upgrades as they are not Java 9 compatible as well. One of those dependency upgrades is to upgrade Apache Karaf to version 4.2.2, as you can see in the following figure.

In the very past OpenNMS was mostly driven by Jetty and Spring*. Probably a more flexible solution needed to be found, as with a big application like OpenNMS dependency issues are always present. For example dependency A requires dependency B in Version 1, but dependency C requires the same dependency B in Version 2. Modern containers - such as Apache Karaf (an OSGi container) - can solve these problems. Besides that OSGi offers a nice API and Implementation differentiation.

Today I wanted to wire in the new GraphService into OpenNMS and have a graph:list command show a GraphML graph I sent to the /opennms/rest/graphml endpoint. The idea was to leverage the GraphmlRepository to also store a configuration file for a new GraphMLGraphContainerProviderServiceFactory implementing a ManagedServiceFactory. Basically the same as we already do with the GraphMLTopologyFactory I wired everything together and was also able to install the features. However as soon as I send the graphml document to the rest endpoint via (of course OpenNMS should be started)

Sometimes it is required to show a (bundle) configuration from the Karaf Shell. This may be on OpenNMS, Minion or Sentinel. First of all a connection to the Karaf Shell must be made. Here is described on how to do this. The according ports can be found here. Now let’s assume we want to see the current values of the id, location and other settings on Minion. First of all we need to know in which cfg file the property is located.

Here I explained on how to connect to the OpenNMS Karaf Shell. However when connecting to Sentinel or Minion the ports are different. The following table shows the default ports for each system System Port OpenNMS 8101 Minion 8201 Sentinel 8301

OpenNMS uses Apache Karaf under the hood and in some cases (especially during development) it is required to connect to the OpenNMS Karaf Shell. First of all OpenNMS must be started to connect to the Karaf Shell. I usually start OpenNMS with the -v (verbose) and -t (debug) options: $OPENNMS_HOME/bin/opennms -vt start Afterwards I check with $OPENNMS_HOME/bin/opennms -v status until the Jetty Webserver is STARTED, as the Karaf Container is started by the Webapp itself.