root
API
Generated with
root/org.eso.ias/org.eso.ias.kafkaneo/org.eso.ias.kafkaneo.consumer

org.eso.ias.kafkaneo.consumer

package org.eso.ias.kafkaneo.consumer

Members list

Type members

Classlikes

class Consumer[K, V](val id: String, val groupId: String, val kafkaServers: String, val keyDeserializer: String, val valueDeserializer: String, val startReadingPos: StartReadingPos, val kafkaProperties: Map[String, String]) extends Thread, ConsumerRebalanceListener

Kafka consumer to get events from the BSDB.

Kafka consumer to get events from the BSDB.

This consumer allows to get events from several topics: compared to consumers in the org.eso.ias.kafkautils package, a process can instantiate only one consumer to get events for more topics (for example IASIOs and commands).

Features:

  • more listeners for each topic
  • collection of events are delivered to listeners

'''Limitations''':

  • the consumer uses only one group.id as it belongs to only one group; this is subject to kafka strategy on assigning consumers to topic partitions depending on the group (and other consumers in the same group). In short if you want your consumer to get all events from topic 1 and all events from topic 2 (most common case in IAS) then you have to ensure that the consumer is the only one consumer in the group for both topics.
  • the serializer/deserializer is the same for all the topics for example a StringSerialize/StringDeserializer (for example the event is a JSON string, then it must be parsed to build the java object like a command or a IASIO)

Type parameters

K

The type of the key

V

The type of the value

Value parameters

groupId

the group.id property for the kafka consumer

id

the id of the consumer for the kafka consumer

kafkaProperties

Other kafka properties to allow the user to pass custom properties

kafkaServers

the string of servers and ports to connect to kafka servers

keyDeserializer

the name of the class to deserialize the key

startReadingPos

The position in the kafka stream to start reading message, default to the end of the stream

valueDeserializer

the name of the class to deserialize the value

Attributes

Constructor

create a new kafka consumer

Since

13.0

Companion
object
Supertypes
trait ConsumerRebalanceListener
class Thread
trait Runnable
class Object
trait Matchable
class Any
Show all
object Consumer

Attributes

Companion
class
Supertypes
class Object
trait Matchable
class Any
Self type
Consumer.type
object ConsumerHelper

Helper for handling Kafka Consumer

Helper for handling Kafka Consumer

Attributes

Supertypes
class Object
trait Matchable
class Any
Self type
trait ConsumerListener[K, V](val iasTopic: IasTopic, val maxCacheSize: Int)

The listener of Kafka events

The listener of Kafka events

The listener must be enabled to get events. When paused, it cache up to maxCacheSize items: when the cache is full, oldest events are removed to leave room for new events

Type parameters

K

The type of the key

V

The type of the value

Value parameters

maxCacheSize:

max number of events cached when the listener is paused

Attributes

Companion
object
Supertypes
class Object
trait Matchable
class Any
Known subtypes
class ConsumerListenerJ[K, V]
class ListenerForTestJ[K, V]
class ListenerForTest[K, V]
class MockListener
object ConsumerListener

Attributes

Companion
trait
Supertypes
class Object
trait Matchable
class Any
Self type
abstract class ConsumerListenerJ[K, V](topic: IasTopic, sz: Int) extends ConsumerListener[K, V]

The consumer of Kafka events specialized for java consumers

The consumer of Kafka events specialized for java consumers

This consumer provides the same features of ConsumerListener but converts scala list of events into java list of events before sending the events to the listener

Type parameters

K

The type of the key

V

The type of the value

Attributes

Supertypes
trait ConsumerListener[K, V]
class Object
trait Matchable
class Any
Known subtypes
class ListenerForTestJ[K, V]
class ConsumersStatistics(val id: String, val timerPeriod: Int, val logLevel: Level) extends TimerTask

Build and publish the statistics for the consumers in the form of log messages.

Build and publish the statistics for the consumers in the form of log messages.

The ConsumersStatistics collects the number of events read from the topics and periodically publish log messages with the statistics. The period to collect data and publish statistics and the log level to use are customizable.

ConsumersStatistics.start must be invoked to activate the timer task that periodically publishes the logs. ConsumersStatistics.stop) must be invoked to stop publishing the statistics. Once stopped the ConsumersStatistics cannot be started again.

In case of overflow incrementing the number of the events read from the topics, the numbers are reset to 0. For synchronization, ConsumersStatistics uses AtomicLong whose values are reset to 0 when there is an overflow. The operation of (increase -> checkOverflow -> reset) is not atomic so it could happen that the statistics are not reporting useful values in that case but such event is so rare that it is not worst to add more complexity.

Value parameters

id

The id for the consumer

logLevel:

the level of the logs published for the statistics

timerPeriod

the number of minutes to push statistics

Attributes

Companion
object
Supertypes
class TimerTask
trait Runnable
class Object
trait Matchable
class Any
object ConsumersStatistics

Attributes

Companion
class
Supertypes
class Object
trait Matchable
class Any
Self type
enum StartReadingPos extends Enum[StartReadingPos]

The position to start reading messages when a partition is assigned to a consumer

The position to start reading messages when a partition is assigned to a consumer

Attributes

Supertypes
trait Enum
trait Product
trait Equals
class Enum[StartReadingPos]
trait Serializable
trait Comparable[StartReadingPos]
trait Constable
class Object
trait Matchable
class Any
Show all
Generated with