Collectors

Decanter collectors harvest the monitoring data, and send this data to the Decanter appenders.

Two kinds of collector are available:

Event Driven Collectors react to events and "broadcast" the data to the appenders.
Polled Collectors are periodically executed by the Decanter Scheduler. When executed, the collectors harvest the data and send to the appenders.

Log

The Decanter Log Collector is an event driven collector. It automatically reacts when a log occurs, and sends the log details (level, logger name, message, etc) to the appenders.

The decanter-collector-log feature installs the log collector:

karaf@root()> feature:install decanter-collector-log

The log collector doesn’t need any configuration, the installation of the decanter-collector-log feature is enough.

Note

The Decanter log collector is using osgi:DecanterLogCollectorAppender appender. In order to work, your Apache Karaf Pax Logging configuration should contain this appender.

The default Apache Karaf etc/org.ops4j.pax.logging.cfg configuration file is already fine:

log4j.rootLogger = DEBUG, out, osgi:*

If you want, you can explicitly specify the DecanterLogCollectorAppender appender:

log4j.rootLogger = DEBUG, out, osgi:DecanterLogCollectorAppender, osgi:VmLogAppender

Optionally, you can select the log categories you want to exclude or include in the logger.

In the etc/org.apache.karaf.collector.log.cfg configuration file, you can set excluded.categories (comma separated list of loggers):

excluded.categories=ignored.logger

With this property, the log collector will ignore any log events/messages coming from ignored.logger logger.

On the other hand, you can include only some logger with the included.categories (comma separated list of loggers):

included.categories=my.logger

With this property, the log collector will only consider log events/messages coming from included.categories logger, other loggers will be ignored.

CXF Logging feature integration

The CXF message logging nicely integrates with Decanter. Simply add the org.apache.cxf.ext.logging.LoggingFeature to your service.

This will automatically log the messages from all clients and endpoints to slf4j. All meta data can be found in the MDC attributes. The message logging can be switched on/off per service using the org.ops4j.pax.logging.cfg.

When using with Decanter make sure you enable the log collector to actually process the message logs.

Log Socket

The Decanter Log Socket Collector is an event driven collector. It creates a socket, waiting for incoming event. The expected events are log4j LoggingEvent. The log4j LoggingEvent is transformed as a Map containing the log details (level, logger name, message, …). This Map is sent to the appenders.

The collector allows you to remotely use Decanter. For instance, you can have an application running on a different platform (spring-boot, application servers, …). This application can use a log4j socket appender that send the logging events to the Decanter log socket collector.

The decanter-collector-log-socket feature install the log socket collector:

karaf@root()> feature:install decanter-collector-log-socket

This feature installs the collector and a default etc/org.apache.karaf.decanter.collector.log.socket.cfg configuration file containing:

#
# Decanter Log/Log4j Socket collector configuration
#

#port=4560
#workers=10

the port property defines the port number where the collector is bound and listen for incoming logging event. Default is 4560.
the workers properties defines the number of threads (workers) which can deal with multiple clients in the same time.

File

The Decanter File Collector is an event driven collector. It automatically reacts when new lines are appended into a file (especially a log file). It acts like the tail Unix command. Basically, it’s an alternative to the log collector. The log collector reacts to local Karaf log messages, whereas the file collector can react to any file, including log files from other systems to Karaf. It means that you can monitor and send collected data for any system (even if it is not Java based).

The file collector deals with file rotation, file not found.

The decanter-collector-file feature installs the file collector:

karaf@root()> feature:install decanter-collector-file

Now, you have to create a configuration file for each file that you want to monitor. In the etc folder, you have to create a file with the following format name etc/org.apache.karaf.decanter.collector.file-ID.cfg where ID is an ID of your choice.

This file contains:

type=my
path=/path/to/file
any=value

type is an ID (mandatory) that allows you to easily identify the monitored file
path is the location of the file that you want to monitor
all other values (like any) will be part of the collected data. It means that you can add your own custom data, and easily create queries bases on this data.

You can also filter the lines read from the file using the regex property.

For instance:

regex=(.*foo.*)

Only the line matching the regex will be sent to the dispatcher.

For instance, instead of the log collector, you can create the following etc/org.apache.karaf.decanter.collector.file-karaf.cfg file collector configuration file:

type=karaf-log-file
path=/path/to/karaf/data/log/karaf.log
regex=(.*my.*)
my=stuff

The file collector will tail on karaf.log file, and send any new line matching the regex in this log file as collected data.

Parser

By default, the collector use the org.apache.karaf.decanter.impl.parser.IdentityParser parser to parse the line into a typed Object (Long, Integer or String) before send it to the EventDispatcher data map.

Identity parser

The identity parser doesn’t actually parse the line, it just passes through. It’s the default parser used by the file collector.

Split parser

The split parser splits the line using a separator (, by default). Optionally, it can take keys used a property name in the event.

For instance, you can have the following etc/org.apache.karaf.decanter.parser.split.cfg configuration file:

separator=,
keys=first,second,third,fourth
useDefaultKey=false

If the parser gets a line (collected by the file collector) like this,is,a,test, the line will be parsed as follows (the file collector will send the following data to the dispatcher):

first->this
second->is
third->a
fourth->test

If the keys configuration is not set and useDefaultKey is set to true, then key-0, key-1, etc will be used. When the useDefaultKey is set to false and the line doesn’t match the keys configuration the data will not be sent.

To use this parser in the file collector, you have to define it in the parser.target configuration (in etc/org.apache.karaf.decanter.collector.file-XXXX.cfg):

parser.target=(parserId=split)

Regex parser

The regex parser is similar to the split parser but instead of using a separator, it uses regex groups.

The configuration contains the regex and the keys in the etc/org.apache.karaf.decanter.parser.regex.cfg configuration file:

regex=(t.*t)

If the parser gets a line (collected by the file collector) like a test here, the line will be parsed as follows (the file collector will send the following data to the dispatcher):

key-0->test

It’s also possible to use keys to identify each regex group.

To use this parser in the file collector, you have to define it in the parser.target configuration (in etc/org.apache.karaf.decanter.collector.file-XXXX.cfg):

parser.target=(parserId=regex)

Custom parser

You can write your own parser by implementing the org.apache.karaf.decanter.api.parser.Parser interface and declare it into the file collector configuration file:

parser.target=(parserId=myParser)

EventAdmin

The Decanter EventAdmin Collector is an event-driven collector, listening for all internal events happening in the Apache Karaf Container.

Note	It’s the perfect way to audit all actions performed on resources (features, bundles, configurations, …) by users (via local shell console, SSH, or JMX). We recommend to use this collector to implement users and actions auditing.

The decanter-collector-eventadmin feature installs the eventadmin collector:

karaf@root()> feature:install decanter-collector-eventadmin

By default, the eventadmin collector is listening for all OSGi framework and Karaf internal events.

You can specify additional events to trap by providing a `etc/org.apache.karaf.decanter.collector.eventadmin-my.cfg' configuration file, containing the EventAdmin topics you want to listen:

event.topics=my/*

Note	By default, the events contain timestamp and subject. You can disable this by modifying `etc/org.apache.felix.eventadmin.impl.EventAdmin` configuration file: org.apache.felix.eventadmin.AddTimestamp=true org.apache.felix.eventadmin.AddSubject=true

JMX

The Decanter JMX Collector is a polled collector, executed periodically by the Decanter Scheduler.

The JMX collector connects to a JMX MBeanServer (local or remote), and retrieves all attributes of each available MBeans. The JMX metrics (attribute values) are send to the appenders.

In addition, the JMX collector also supports the execution of operations on dedicated ObjectName that you configure via cfg file.

The decanter-collector-jmx feature installs the JMX collector, and a default configuration file:

karaf@root()> feature:install decanter-collector-jmx

This feature brings a etc/org.apache.karaf.decanter.collector.jmx-local.cfg configuration file containing:

#
# Decanter Local JMX collector configuration
#

# Name/type of the JMX collection
type=jmx-local

# URL of the JMX MBeanServer.
# local keyword means the local platform MBeanServer or you can specify to full JMX URL
# like service:jmx:rmi:///jndi/rmi://hostname:port/karaf-instance
url=local

# Username to connect to the JMX MBeanServer
#username=karaf

# Password to connect to the JMX MBeanServer
#password=karaf

# Object name filter to use. Instead of harvesting all MBeans, you can select only
# some MBeans matching the object name filter
#object.name=org.apache.camel:context=*,type=routes,name=*

# Several object names can also be specified.
# What matters is that the property names begin with "object.name".
#object.name.system=java.lang:*
#object.name.karaf=org.apache.karaf:type=http,name=*
#object.name.3=org.apache.activemq:*

# You can also execute operations on some MBeans, providing the object name, operation, arguments (separated by ,)
# and signatures (separated by ,) for the arguments (separated by |)
#operation.name.rootLogger=org.apache.karaf:type=log,name=root|getLevel|rootLogger|java.lang.String

This file harvests the data of the local MBeanServer:

the type property is a name (of your choice) allowing you to easily identify the harvested data
the url property is the MBeanServer to connect to. "local" is a reserved keyword to specify the local MBeanServer. Instead of "local", you can use the JMX service URL. For instance, for Karaf version 3.0.0, 3.0.1, 3.0.2, and 3.0.3, as the local MBeanServer is secured, you can specify service:jmx:rmi:///jndi/rmi://localhost:1099/karaf-root. You can also polled any remote MBean server (Karaf based or not) providing the service URL.
the username property contains the username to connect to the MBean server. It’s only required if the MBean server is secured.
the password property contains the password to connect to the MBean server. It’s only required if the MBean server is secured.
the object.name prefix is optional. If this property is not specified, the collector will retrieve the attributes of all MBeans. You can filter to consider only some MBeans. This property contains the ObjectName filter to retrieve the attributes only of some MBeans. Several object names can be listed, provided the property prefix is object.name..
any other values will be part of the collected data. It means that you can add your own property if you want to add additional data, and create queries based on this data.
the operation.name prefix is also optional. You can use it to execute an operation. The value format is objectName|operation|arguments|signatures.

You can retrieve multiple MBean servers. For that, you just create a new configuration file using the file name format etc/org.apache.karaf.decanter.collector.jmx-[ANYNAME].cfg.

JMXMP

The Karaf Decanter JMX collector by default uses RMI protocol for JMX. But it also supports JMXMP protocol.

The features to install are the same: decanter-collector-jmx.

However, you have to enable the jmxmp protocol support in the Apache Karaf instance hosting Karaf Decanter.

You can download the jmxmp protocol provider artifact on Maven Central: [http://repo.maven.apache.org/maven2/org/glassfish/external/opendmk_jmxremote_optional_jar/1.0-b01-ea/opendmk_jmxremote_optional_jar-1.0-b01-ea.jar]

The opendmk_jmxremote_optional_jar-1.0-b01-ea.jar file has to be copied in the lib/boot folder of your Apache Karaf instance.

Then, you have to add the new JMX remote packages by editing etc/config.properties, appending the following to the org.osgi.framework.system.packages.extra property:

org.osgi.framework.system.packages.extra = \
    ...
    javax.remote, \
    com.sun.jmx, \
    com.sun.jmx.remote, \
    com.sun.jmx.remote.protocol, \
    com.sun.jmx.remote.generic, \
    com.sun.jmx.remote.opt, \
    com.sun.jmx.remote.opt.internal, \
    com.sun.jmx.remote.opt.security, \
    com.sun.jmx.remote.opt.util, \
    com.sun.jmx.remote.profile, \
    com.sun.jmx.remote.profile.sasl, \
    com.sun.jmx.remote.profile.tls, \
    com.sun.jmx.remote.socket, \
    javax.management, \
    javax.management.remote, \
    javax.management.remote.generic, \
    javax.management.remote.jmxmp, \
    javax.management.remote.message

Then, you can create a new Decanter JMX collector by creating a new file, like etc/org.apache.karaf.decanter.collector.jmx-mycollector.cfg containing something like:

type=jmx-mycollector
url=service:jmx:jmxmp://host:port
jmx.remote.protocol.provider.pkgs=com.sun.jmx.remote.protocol

You can see: * the url property contains an URL with jmxmp instead of rmi. * in order to support jmxmp protocol, you have to set the protocol provider via the jmx.remote.protocol.provider.pkgs property (by default, Karaf Decanter JMX collector uses the rmi protocol provider)

ActiveMQ (JMX)

The ActiveMQ JMX collector is just a special configuration of the JMX collector.

The decanter-collector-activemq feature installs the default JMX collector, with the specific ActiveMQ JMX configuration:

karaf@root()> feature:install decanter-collector-jmx-activemq

This feature installs the same collector as the decanter-collector-jmx, but also adds the etc/org.apache.karaf.decanter.collector.jmx-activemq.cfg configuration file.

This file contains:

#
# Decanter Local ActiveMQ JMX collector configuration
#

# Name/type of the JMX collection
type=jmx-activemq

# URL of the JMX MBeanServer.
# local keyword means the local platform MBeanServer or you can specify to full JMX URL
# like service:jmx:rmi:///jndi/rmi://hostname:port/karaf-instance
url=local

# Username to connect to the JMX MBeanServer
#username=karaf

# Password to connect to the JMX MBeanServer
#password=karaf

# Object name filter to use. Instead of harvesting all MBeans, you can select only
# some MBeans matching the object name filter
object.name=org.apache.activemq:*

This configuration actually contains a filter to retrieve only the ActiveMQ JMX MBeans.

Camel (JMX)

The Camel JMX collector is just a special configuration of the JMX collector.

The decanter-collector-jmx-camel feature installs the default JMX collector, with the specific Camel JMX configuration:

karaf@root()> feature:install decanter-collector-jmx-camel

This feature installs the same collector as the decanter-collector-jmx, but also adds the etc/org.apache.karaf.decanter.collector.jmx-camel.cfg configuration file.

This file contains:

#
# Decanter Local Camel JMX collector configuration
#

# Name/type of the JMX collection
type=jmx-camel

# URL of the JMX MBeanServer.
# local keyword means the local platform MBeanServer or you can specify to full JMX URL
# like service:jmx:rmi:///jndi/rmi://hostname:port/karaf-instance
url=local

# Username to connect to the JMX MBeanServer
#username=karaf

# Password to connect to the JMX MBeanServer
#password=karaf

# Object name filter to use. Instead of harvesting all MBeans, you can select only
# some MBeans matching the object name filter
object.name=org.apache.camel:context=*,type=routes,name=*

This configuration actually contains a filter to retrieve only the Camel Routes JMX MBeans.

Camel Tracer & Notifier

Decanter provides a Camel Tracer Handler that you can set on a Camel Tracer. It also provides a Camel Event Notifier.

Camel Tracer

Decanter Camel Tracer is actually an intercept strategy.

It catches all exchanges in the route. If you enable the tracer on a Camel route, all events (exchanges on each step of the route) are sent to the dispatcher.

The decanter-collector-camel feature provides the Decanter Camel intercept strategy:

karaf@root()> feature:install decanter-collector-camel

You just have to register Decanter intercept strategy in your Camel Context:

DecanterInterceptStrategy decanterCamelTracer = new DecanterInterceptStrategy();
decanterCamelTracer.setDispatcher(eventAdmin);
camelContext.addInterceptStrategy(decanterCamelTracer);

This is an example of a complete route with Decanter Camel intercept strategy:

<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0">

    <reference id="dispatcher" interface="org.osgi.service.event.EventAdmin"/>

    <bean id="decanterInterceptor" class="org.apache.karaf.decanter.collector.camel.DecanterInterceptStrategy">
        <property name="dispatcher" ref="dispatcher"/>
    </bean>

    <camelContext xmlns="http://camel.apache.org/schema/blueprint">
       <interceptStrategy>
          <bean ref="decanterInterceptor"/>
        </interceptStrategy>
        <route id="test">
            <from uri="timer:fire?period=10000"/>
            <setBody><constant>Hello World</constant></setBody>
            <to uri="log:test"/>
        </route>
    </camelContext>

</blueprint>

You can extend the Decanter event with any property using a custom DecanterCamelEventExtender:

public interface DecanterCamelEventExtender {

    void extend(Map<String, Object> decanterData, Exchange camelExchange);

}

You can inject your extender using setExtender(myExtender) on the DecanterInterceptStrategy. Decanter will automatically call your extender to populate extra properties.

Camel Event Notifier

Decanter also provides DecanterEventNotifier implementing a Camel event notifier: http://camel.apache.org/eventnotifier-to-log-details-about-all-sent-exchanges.html

It’s very similar to the Decanter Camel Tracer. You can control the camel contexts and routes to which you want to trap events.

System (oshi)

The oshi collector is a system collector (polled) that periodically retrieve all details about the hardware and the operating system.

This collector gets lot of details about the machine.

The decanter-collector-oshi feature installs the oshi system collector:

karaf@root()> feature:install decanter-collector-oshi

This feature installs a default etc/org.apache.karaf.decanter.collector.oshi.cfg configuration file containing:

################################################################################
#
#    Licensed to the Apache Software Foundation (ASF) under one or more
#    contributor license agreements.  See the NOTICE file distributed with
#    this work for additional information regarding copyright ownership.
#    The ASF licenses this file to You under the Apache License, Version 2.0
#    (the "License"); you may not use this file except in compliance with
#    the License.  You may obtain a copy of the License at
#
#       http://www.apache.org/licenses/LICENSE-2.0
#
#    Unless required by applicable law or agreed to in writing, software
#    distributed under the License is distributed on an "AS IS" BASIS,
#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#    See the License for the specific language governing permissions and
#    limitations under the License.
#
################################################################################

#
# Decanter oshi (system) collector
#

# computerSystem=true
# computerSystem.baseboard=true
# computerSystem.firmware=true
# memory=true
# processors=true
# processors.logical=true
# displays=true
# disks=true
# disks.partitions=true
# graphicsCards=true
# networkIFs=true
# powerSources=true
# soundCards=true
# sensors=true
# usbDevices=true
# operatingSystem=true
# operatingSystem.fileSystems=true
# operatingSystem.networkParams=true
# operatingSystem.processes=true
# operatingSystem.services=true

By default, the oshi collector gets all details about the machine. You can filter what you want to harvest in this configuration file.

System (script)

The system collector is a polled collector (periodically executed by the Decanter Scheduler).

This collector executes operating system commands (or scripts) and send the execution output to the appenders.

The decanter-collector-system feature installs the system collector:

karaf@root()> feature:install decanter-collector-system

This feature installs a default etc/org.apache.karaf.decanter.collector.system.cfg configuration file containing:

#
# Decanter OperationSystem Collector configuration
#

# This collector executes system commands, retrieve the exec output/err
# sent to the appenders
#
# You can define the number of thread to use for parallelization command calls:
# thread.number=1
#
# The format is command.key=command_to_execute
# where command is a reserved keyword used to identify a command property
# for instance:
#
# command.df=df -h
# command.free=free
#
# You can also create a script containing command like:
#
#   df -k / | awk -F " |%" '/dev/{print $8}'
#
# This script will get the available space on the / filesystem for instance.
# and call the script:
#
# command.df=/bin/script
#
# Another example of script to get the temperature:
#
#   sensors|grep temp1|awk '{print $2}'|cut -b2,3,4,5
#

You can add the commands that you want to execute using the format:

command.name=system_command

The collector will execute each command described in this file, and send the execution output to the appenders.

For instance, if you want to periodically send the free space available on the / filesystem, you can add:

command.df=df -k / | awk -F " |%" '/dev/{print $8}'

Network socket

The Decanter network socket collector listens for incoming messages coming from a remote network socket collector.

The decanter-collector-socket feature installs the network socket collector:

karaf@root()> feature:install decanter-collector-socket

This feature installs a default etc/org.apache.karaf.decanter.collector.socket.cfg configuration file containing:

# Decanter Socket Collector

# Port number on which to listen
#port=34343

# Binding address on which to listen
#host=0.0.0.0

# Number of worker threads to deal with
#workers=10

# Protocol tcp(default) or udp
#protocol=tcp

# Unmarshaller to use
# Unmarshaller is identified by data format. The default is json, but you can use another unmarshaller
unmarshaller.target=(dataFormat=json)

# The maximum request size (in bytes). Set to -1 not to put any limit on the request size.
max.request.size=100000

the port property contains the port number where the network socket collector is listening
the host property contains the host name/interface address where the network socket collector is listening
the workers property contains the number of worker threads the socket collector is using for the connection
the protocol property contains the protocol used by the collector for transferring data with the client
the unmarshaller.target property contains the unmarshaller used by the collector to transform the data sent by the client.
the max.request.size is the size (in bytes) that the collector can accept. By default, it’s bounded to prevent high memory consumption. You can use -1 to disable any limit.

JMS

The Decanter JMS collector consumes the data from a JMS queue or topic. It’s a way to aggregate collected data coming from (several) remote machines.

The decanter-collector-jms feature installs the JMS collector:

karaf@root()> feature:install decanter-collector-jms

This feature also installs a default etc/org.apache.karaf.decanter.collector.jms.cfg configuration file containing:

######################################
# Decanter JMS Collector Configuration
######################################

# Name of the JMS connection factory
connection.factory.name=jms/decanter

# Name of the destination
destination.name=decanter

# Type of the destination (queue or topic)
destination.type=queue

# Connection username
# username=

# Connection password
# password=

the connection.factory.name is the name of the ConnectionFactory OSGi service to use
the destination.name is the name of the queue or topic where to consume messages from the JMS broker
the destination.type is the type of the destination (queue or topic)
the username and password properties are the credentials to use with a secured connection factory

MQTT

The Decanter MQTT collector receives collected messages from a MQTT broker. It’s a way to aggregate collected data coming from (several) remote machines.

The decanter-collector-mqtt feature installs the MQTT collector:

karaf@root()> feature:install decanter-collector-mqtt

This feature also installs a default etc/org.apache.karaf.decanter.collector.mqtt.cfg configuration file containing:

#######################################
# Decanter MQTT Collector Configuration
#######################################

# URI of the MQTT broker
server.uri=tcp://localhost:61616

# MQTT Client ID
client.id=decanter

# MQTT topic name
topic=decanter

the server.uri is the location of the MQTT broker
the client.id is the Decanter MQTT client ID
the topic is the MQTT topic pattern where to receive the messages

Kafka

The Decanter Kafka collector receives collected messages from a Kafka broker. It’s a way to aggregate collected data coming from (several) remote machines.

The decanter-collector-kafka feature installs the Kafka collector:

karaf@root()> feature:install decanter-collector-kafka

This feature also installs a default etc/org.apache.karaf.decanter.collector.kafka.cfg configuration file containing:

###############################
# Decanter Kafka Configuration
###############################

# A list of host/port pairs to use for establishing the initial connection to the Kafka cluster
#bootstrap.servers=localhost:9092

# An id string to identify the group where the consumer belongs to
#group.id=decanter

# Enable auto commit of consumed messages
#enable.auto.commit=true

# Auto commit interval (in ms) triggering the commit
#auto.commit.interval.ms=1000

# Timeout on the consumer session
#session.timeout.ms=30000

# Serializer class for key that implements the Serializer interface
#key.serializer=org.apache.kafka.common.serialization.StringSerializer

# Serializer class for value that implements the Serializer interface.
#value.serializer=org.apache.kafka.common.serialization.StringSerializer

# Name of the topic
#topic=decanter

# Security (SSL)
#security.protocol=SSL

# SSL truststore location (Kafka broker) and password
#ssl.truststore.location=${karaf.etc}/keystores/keystore.jks
#ssl.truststore.password=karaf

# SSL keystore (if client authentication is required)
#ssl.keystore.location=${karaf.etc}/keystores/clientstore.jks
#ssl.keystore.password=karaf
#ssl.key.password=karaf

# (Optional) SSL provider (default uses the JVM one)
#ssl.provider=

# (Optional) SSL Cipher suites
#ssl.cipher.suites=

# (Optional) SSL Protocols enabled (default is TLSv1.2,TLSv1.1,TLSv1)
#ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1

# (Optional) SSL Truststore type (default is JKS)
#ssl.truststore.type=JKS

# (Optional) SSL Keystore type (default is JKS)
#ssl.keystore.type=JKS

# Security (SASL)
# For SASL, you have to configure Java System property as explained in http://kafka.apache.org/documentation.html#security_ssl

The configuration is similar to the Decanter Kafka appender. Please, see the Kafka collector for details.

Rest Servlet

The Decanter Rest Servlet collector registers a servlet on the OSGi HTTP service (by default on /decanter/collect).

It listens for incoming collected messages on this servlet.

The decanter-collector-rest-servlet feature installs the collector:

karaf@root()> feature:install decanter-collector-rest-servlet

REST

The Decanter REST collector periodically requests a REST service and returns the result (with all HTTP details).

The decanter-collector-rest feature installs the collector:

karaf@root()> feature:install decanter-collector-rest

This feature also installs etc/org.apache.karaf.decanter.collector.rest.cfg configuration file where you can setup the REST service request:

#
# Decanter REST collector
#

url=http://localhost:8080
paths=metrics

#request.method=GET (possible values are GET, POST, PUT, DELETE, default is GET)
#request=foo (request payload)
#header.foo=bar (header passed, prefixed with header.)
#user=user (used for basic authentication)
#password=password (used for basic authentication)
#topic=decanter/collector/rest (Decanter dispatcher topic name to use)

# Possible to wrap exception as HTTP response, where you can define the HTTP response code
# exception.as.http.response=true
# exception.http.response.code=501

# Unmarshaller to use
unmarshaller.target=(dataFormat=json)

The exception.as.http.response property allows you to "wrap" any connection exception as a HTTP message. If true, any exception is catched and we send kind of HTTP message in the Decanter dispatcher.

It’s also possible to define a HTTP response code (thanks to exception.http.response.code property) when an exception is catched.

SOAP

The Decanter SOAP collector periodically requests a SOAP service and returns the result (the SOAP Response, or error details if it failed).

The decanter-collector-soap feature installs the collector:

karaf@root()> feature:install decanter-collector-soap

This feature also installs etc/org.apache.karaf.decanter.collector.soap.cfg configuration file where you can setup the URL of the service and the SOAP request to use:

#
# Decanter SOAP collector
#

url=http://localhost:8080/cxf/service
soap.request=

The collector sends several collected properties to the dispatcher, especially:

soap.response property contains the actual SOAP response
error is only populated when the service request failed, containing the error detail
http.response.code contains the HTTP status code of the service request

Dropwizard Metrics

The Decanter Dropwizard Metrics collector get a MetricSet OSGi service and periodically get the metrics in the set.

The decanter-collector-dropwizard feature installs the collector:

karaf@root()> feature:install decanter-collector-dropwizard

As soon as a MetricSet (like MetricRegistry) service will be available, the collector will get the metrics and send to the Decanter dispatcher.

JDBC

The Decanter JDBC collector periodically executes a query on a database and sends the query result to the dispatcher.

The decanter-collector-jdbc feature installs the JDBC collector:

karaf@root()> feature:install decanter-collector-jdbc

The feature also installs the etc/org.apache.karaf.decanter.collector.jdbc.cfg configuration file:

#######################################
# Decanter JDBC Collector Configuration
#######################################

# DataSource to use
dataSource.target=(osgi.jndi.service.name=jdbc/decanter)

# SQL Query to retrieve data
query=select * from TABLE

This configuration file allows you to configure the datasource to use and the SQL query to perform:

the datasource.name property contains the name of the JDBC datasource to use to connect to the database. You can create this datasource using the Karaf jdbc:create command (provided by the jdbc feature).
the query property contains the SQL query to perform on the database and retrieve data.

This collector is periodically executed by the Karaf scheduler.

ConfigAdmin

The Decanter ConfigAdmin collector listens for any configuration change and send the updated configuration to the dispatcher.

The decanter-collector-configadmin feature installs the ConfigAdmin collector:

karaf@root()> feature:install decanter-collector-configadmin

Prometheus

The Decanter Prometheus collector is able to periodically (scheduled collector) read Prometheus servlet output to create events sent in Decanter.

The decanter-collector-prometheus feature installs the Prometheus collector:

karaf@root()> feature:install decanter-collector-prometheus

The feature also installs the etc/org.apache.karaf.decanter.collector.prometheus.cfg configuration file containing:

prometheus.url=http://host/prometheus

The prometheus.url property is mandatory and define the location of the Prometheus export servlet (that could be provided by the Decanter Prometheus appender for instance).

Redis

The Decanter Redis collector is able to periodically (scheduled collector) read Redis Map to get key/value pairs. You can filter the keys you want thanks to key pattern.

The decanter-collector-redis feature installs the Redis collector:

karaf@root()> feature:install decanter-collector-redis

The feature also installs the etc/org.apache.karaf.decanter.collector.redis.cfg configuration file containing:

address=localhost:6379

#
# Define the connection mode.
# Possible modes: Single (default), Master_Slave, Sentinel, Cluster
#
mode=Single

#
# Name of the Redis map
# Default is Decanter
#
map=Decanter

#
# For Master_Slave mode, we define the location of the master
# Default is localhost:6379
#
#masterAddress=localhost:6379

#
# For Sentinel model, define the name of the master
# Default is myMaster
#
#masterName=myMaster

#
# For Cluster mode, define the scan interval of the nodes in the cluster
# Default value is 2000 (2 seconds).
#
#scanInterval=2000

#
# Key pattern to looking for.
# Default is *
#
#keyPattern=*

You can configure the Redis connection (depending of the topology) and the key pattern in this configuration file.

Elasticsearch

The Decanter Elasticsearch collector retrieves documents from Elasticsearch periodically (scheduled collector). By default, it harvests all documents in the given index, but you can also specify a query.

The decanter-collector-elasticsearch feature installs the Elasticsearch collector:

karaf@root()> feature:install decanter-collector-elasticsearch

The feature also install etc/org.apache.karaf.decanter.collector.elasticsearch.cfg configuration file containing:

# HTTP address of the elasticsearch nodes (separated with comma)
addresses=http://localhost:9200

# Basic username and password authentication (no authentication by default)
#username=user
#password=password

# Name of the index to request (decanter by default)
#index=decanter

# Query to request document (match all by default)
#query=

# Starting point for the document query (no from by default)
#from=

# Max number of documents retrieved (no max by default)
#max=

# Search timeout, in seconds (no timeout by default)
#timeout=

addresses property is the location of the Elasticsearch instances. Default is http://localhost:9200.
username and password properties are used for authentication. They are null (no authentication) by default.
index property is the Elasticsearch index where to get documents. It’s decanter by default.
query property is a search query to use. Default is null meaning all documents in the index are harvested.
from and max properties are used to "square" the query. They are null by default.
timeout property limits the query execution. There’s no timeout by default.

Pax Web Jetty Handler

Pax Web Jetty Handler collector "intercepts" all HTTP exchanges with the Pax Web Jetty container running in Apache Karaf.

The decanter-collector-jetty feature installs the Pax Web Jetty container:

karaf@root()> feature:install decanter-collector-jetty

The collector automatically registers in the Pax Web Jetty container and then all HTTP requests/responses data will be sent to the appenders.

SNMP

Decanter SNMP Collector allows you to trap or poll metrics from SNMP.

The decanter-collector-snmp feature installs the SNMP collector:

karaf@root()> feature:install decanter-collector-snmp

Actually, this collector provides two kind of collectors: poller or trap.

Trap

If you want to use SNMP trap, you have to create etc/org.apache.karaf.decanter.collector.snmp.trap.cfg configuration file containing:

address=127.0.0.1:161
protocol=tcp

address is the listening SNMP trap address
protocol is the transport protocol (TCP or UDP).

Poll

If you want to use SNMP poller (periodically getting SNMP metrics), you have to create etc/org.apache.karaf.decanter.collector.snmp.poll.cfg configuration file containing:

address=127.0.0.1:161
protocol=tcp
retries=2
timeout=1500
treelist=false
oids=first,second
snmp.version=3
security.level=3
security.name=security
authentication.protocol=MD5
authentication.passphrase=pass
privacy.protocol=DES
privacy.passphrase=pass
#snmp.context.engine.id=foo
#snmp.context.name=foo
snmp.community=public

address is the SNMP service address
protocol is the transport protocol (tcp or udp)
retries is the number of attempts
timeout is the SNMP request timeout (in ms)
treelist is true to request SNMP tree, false else
oids is the list of SNMP OIDs to request
snmp.version is the SNMP version to use (3 by default)
security.level is the level of security expected by the SNMP version (3 == AUTH_PRIV by default)
security.name is the security name alias
authentication.protocol is the password protocol (MD5 or SHA1)
authentication.passphrase is the password/passphrase to use
privacy.protocol is DES, TRIDES, AES128, AES192, AES256
private.passphrase is the password/passphrase to use
snmp.context.engine.id is optional
snmp.context.name is optional
snmp.community is the community to use, public by default

Druid

Karaf Decanter Druid collector schedules execution of queries on Apache Druid broker.

It allows to easily create analytics on data and schedule execution with Decanter. Then, the query result is sent to the dispatcher.

The karaf-collector-druid feature installs the Druid collector:

karaf@root()> feature:install decanter-collector-druid

The location of the Apache Druid broker and Druid queries to execute are configured in etc/org.apache.karaf.decanter.collector.druid.cfg configuration file:

#
# Druid broker query API location
#
druid.broker.location=http://localhost:8888/druid/v2/sql/

# Druid queries set, using syntax: query.id
query.foo=select sum_operatingSystem_threadCount from decanter

# Unmarshaller to use
unmarshaller.target=(dataFormat=json)

The druid.broker.location is the URL of the Apache Druid broker.

Then, we can add the Druid queries using the format query.ID=QUERY.

Decanter Druid collector schedules the queries and send the results in the dispatcher.

OpenStack

Karaf Decanter OpenStack collector periodically request OpenStack API to get details about openstack ecosystem.

The karaf-collector-openstack feature installs the OpenStack collector:

karaf@root()> feature:install decanter-collector-openstack

This feature also installs etc/org.apache.karaf.decanter.collector.openstack.cfg configuration file:

#
# Decanter Openstack collector
#
# Openstack services API locations
#

#openstack.identity=http://localhost/identity
#openstack.project=2c7be0bac05c4144a328d4ab3dfac379
#openstack.username=admin
#openstack.password=secret
#openstack.domain=default

#openstack.compute.enabled=true
#openstack.compute=http://localhost/compute/v2.1
#openstack.block.storage.enabled=true
#openstack.block.storage=http://localhost/volume/v3
#openstack.image.enabled=true
#openstack.image=http://localhost/image
#openstack.metric.enabled=true
#openstack.metric=http://localhost/metric

# Unmarshaller to use
unmarshaller.target=(dataFormat=json)

You have to define the locations of the OpenStack APIs and if you enabled requesting the APIs or not.

Target dispatcher topics

All collectors use a default Decanter dispatcher topic name. However, you can change the topic name with the one of your choice.

For that, you have to set event.topics property in the collector configuration.

For instance:

event.topics=my/topic/name

Customizing properties in collectors

You can add, rename or remove properties collected by the collectors before sending it to the dispatcher.

In the collector configuration file (for instance etc/org.apache.karaf.decanter.collector.jmx-local.cfg for the local JMX collector), you can add any property. By default, the property is added to the data sent to the dispatcher.

You can prefix the configuration property with the action you can perform before sending:

fields.add. adds a property to the data sent. The following add property hello with value world:
```
----
fields.add.hello=world
----
```
fields.remove. removes a property to the data sent:
```
----
fields.remove.hello=
----
```
fields.rename. rename a property with another name:
```
----
fields.rename.helo=hello
----
```