S3 to HDFS Sync App
Ingest and backup Amazon S3 data to hadoop HDFS for data download from Amazon to hadoop. This application copies files from the configured S3 location to the destination path in HDFS. The source code is available at: https://github.com/DataTorrent/app-templates/tree/master/s3-to-hdfs-sync.
Please send feedback or feature requests to: email@example.com
This document has a step-by-step guide to configure, customize, and launch this application.
Click on the AppFactory tab from the top navigation bar.
Page listing the applications available on AppFactory is displayed. Search for S3 to see all applications related to S3. Click on import button for
S3 to HDFS Sync App.
Notification is displayed on the top right corner after application package is successfully imported.
Click on the link in the notification which navigates to the page for this application package.
Detailed information about the application package like version, last modified time, and short description is available on this page. Click on launch button for
S3-to-HDFS-Syncapplication. In the confirmation modal, click the Configure button.
For example, suppose we wish to copy from all files in
/user/appuser/outputon the host cluster (on which app is running). Properties should be set as follows:
name value Access Key For S3Input ACCESS_KEY_ID Bucket Name For S3Input BUCKET_NAME Input Directory Or File Path On S3 /directory1 Output Directory Path On HDFS /user/appuser/output/directory1 Secret Key For S3Input SECRET_KEY
Details about configuration options are available in Configuration options section.
When you are finished inputting application configuration properties, click on the
savebutton on the top right corner of the page to save the configuration.
Click on the
launchbutton at the top right corner of the page to launch the application. A notification will be displayed on the top right corner after application is launched successfully and includes the Application ID which can be used to monitor this instance and to find its logs.
Click on the
Monitortab from the top navigation bar.
A page listing all running applications is displayed. Search for the current application based on name or application id or any other relevant field. Click on the application name or id to navigate to application instance details page.
Application instance details page shows key metrics for monitoring the application status.
logicaltab shows application DAG, Stram events, operator status based on logical operators, stream status, and a chart with key metrics.
Click on the
physicaltab to look at the status of physical instances of the operator, containers etc.
End user must specify the values for these properties (these properties are all strings).
|HDFS path for destination directory||
|Access URL for S3 source||
There are pre-saved configurations based on the application environment. Recommended settings for datatorrent sandbox are in
sandbox-memory-conf.xml and for a cluster environment in
|dt.operator.S3InputModule.prop.maxReaders||Maximum number of BlockReader partitions for parallel reading.||int||16||1|
|dt.operator.S3InputModule.prop.blocksThreshold||Rate at which block metadata is emitted per second||int||16||1|
Steps to customize the application
Make sure you have following utilities installed on your machine and available on
PATHin environment variable:
Use following command to clone the examples repository:
git clone firstname.lastname@example.org:DataTorrent/app-templates.git
Change directory to
Import this maven project in your favorite IDE (e.g. eclipse).
Change the source code as per your requirements. This application is for copying files from source to destination. Thus,
Application.javadoes not involve any processing operator in between.
Make respective changes in the test case and
properties.xmlbased on your environment.
Compile this project using maven:
mvn clean package
This will generate the application package with
.apaextension in the
Go to DataTorrent UI Management console on web browser. Click on the
Developtab from the top navigation bar.
Application Packagesfrom the list.
upload packagebutton and upload the generated
Application package page is shown with the listing of all packages. Click on the
Launchbutton for the uploaded application package. Follow the steps for launching an application.