How to customize an app-template

This document describes how to customize an app-template.

Steps to customize the application

  1. Make sure you have following utilities installed on your machine and available on PATH in environment variable:

  2. Use following command to clone the repository:

    git clone git@github.com:DataTorrent/moodI.git

  3. Change directory to containing app-template you wish to customize:

    cd moodI/app-templates/kafka-to-hdfs-filter-transform/

  4. Import this maven project in your favorite IDE (e.g. eclipse).

  5. Change the source code as per your requirements.

  6. Make respective changes in the test case and properties.xml based on your environment.

  7. Compile this project using maven:

    mvn clean package

    This will generate the application package file with .apa extension in the target directory.

  8. Go to DataTorrent UI Management console on web browser. Click on the Develop tab from the top navigation bar. Develop tab

  9. Click on upload package button and upload the generated .apa file. Upload

  10. Application package page is shown with the listing of all packages. Click on the Launch button for the uploaded application package. Follow the steps for launching an application.

Note on customization

  1. Application.java defines the Application class which defines the application pipeline. Application is represented as directed acyclic graph (DAG). Vertices of the graph represents operators or computational units. Edges represents data flow or streams.

  2. Application class implements StreamingApplication interface by defining implementation for populateDAG method .

  3. populateDAG methods receives an argument with an instance of DAG object. App developers can add operators, streams to this dag object to define the pipeline based on the need.

  4. Second argument to populateDAG method is an instance of Configuration object. All the properties specified in properties.xml will be available through this configuration object.

  5. Most of the commonly used functionality, connectors are available in moodI, malhar operator library. If required functionality is not available in the operator library; one can implement own operators for custom computations.

  6. Add the operators to the DAG using dag.addOperator() API and connect the operators with upstream, downstream operators using dag.addStream() API.

    For example, suppose one needs to modify kafka-to-hdfs-filter-transform such that count of tuples discarded by the filter operator should be displayed on the console. This can be achived by adding following code to populateDAG method in Application.java:


Counter counter = dag.addOperator("counter", Counter.class);
ConsoleOutputOperator console = dag.addOperator("console", ConsoleOutputOperator.class);
dag.addStream("FilteredOut", filterOperator.falsePort, counter.input);
dag.addStream("FilteredOutToConsole", counter.output, console.input);

Follow the customization steps for applying your changes.