Kafka to Cassandra Filter Transform Application

Summary

This application template demonstrates continuous big data preparation while reading data messages from a configured Kafka topic. This data is considered to be in JSON format, which is further filtered, transformed based on configurable properties. Finally, this prepared message is written to a Cassandra .

Required Properties

End user must specify the values for these properties.

Property Type Example Notes
Cassandra Node Name String localhost Host name for cassandra server
Filter Condition For Tuples String ({$}.getPremium() >= 50000) Quasi java expression
Key Space On Cassandra String testdb Keyspace to be used for output
Table Name String test_input Name of the table on cassandra store
Json Parser Field Info String {"policyNumber":"LONG", "customerName":"STRING", "premium":"LONG"} JSON map with key indicating input field. Value indicating data type for the field.
Kafka Broker List String
  • localhost:9092
  • node1.corp1.com:9092, node2.corp1.com:9092
Comma seperated list of kafka brokers
Kafka Topic Name String transactions Topic names on Kakfa
Transform Expression Info String {"customerName":"{$} .getCustomerName() .toUpperCase()"} JSON map with key indicating output field. Value indicating expression to be used for calculating its value
Transform Output field Info String {"policyNumber":"LONG", "customerName":"STRING", "premium":"LONG"} JSON map with key indicating output field. Value indicating data type for the field.

Advanced Properties (optional)

Property Default Type Example Notes
Initial Offset Of Topic For Kafka Consumer LATEST String
  • EARLIEST
  • LATEST
  • APPLICATION_OR_EARLIEST
  • APPLICATION_OR_LATEST
Whether to read from beginning or read from current offset.