Apache Nifi Data Flow

Apache NiFi is a dataflow system based on the concepts of flow-based programming. It supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. NiFi has a web-based user interface for design, control, feedback, and monitoring of dataflows. It is highly configurable along several dimensions of quality of service, such as loss-tolerant versus guaranteed delivery, low latency versus high throughput, and priority-based queuing. NiFi provides fine-grained data provenance for all data received, forked, joined cloned, modified, sent, and ultimately dropped upon reaching its configured end-state.
Put simply NiFi was built to automate the flow of data between systems. While the term dataflow is used in a variety of contexts, we use it here to mean the automated and managed flow of information between systems. This problem space has been around ever since enterprises had more than one system, where some of the systems created data and some of the systems consumed data.
This blog will describes a simple data flow from Local file system to HDFS using Apache Nifi.
Before we get into exercise first we must to aware of the basic components of Apache Nifi.
NiFi Components:
NiFi provides several extension points to provide developers the ability to add functionality to the application to meet their needs. The following list provides a high-level description of the most common extension points:
Processor
The Processor interface is the mechanism through which NiFi exposes access to FlowFiles, their attributes, and their content. The Processor is the basic building block used to comprise a NiFi dataflow. This interface is used to accomplish all of the following tasks:
Create FlowFiles
Read FlowFile content
Write FlowFile content
Read FlowFile attributes
Update FlowFile attributes
Ingest data
Egress data
Route data
Extract data
Modify data
ReportingTask
The ReportingTask interface is a mechanism that NiFi exposes to allow metrics, monitoring information, and internal NiFi state to be published to external endpoints, such as log files, e-mail, and remote web services.
ControllerService
A ControllerService provides shared state and functionality across Processors, other ControllerServices, and ReportingTasks within a single JVM. An example use case may include loading a very large dataset into memory. By performing this work in a ControllerService, the data can be loaded once and be exposed to all Processors via this service, rather than requiring many different Processors to load the dataset themselves.
FlowFilePrioritizer
The FlowFilePrioritizer interface provides a mechanism by which FlowFiles in a queue can be prioritized, or sorted, so that the FlowFiles can be processed in an order that is most effective for a particular use case.
AuthorityProvider
An AuthorityProvide is responsible for determining which privileges and roles, if any, a given user should be granted.
Steps to Workout:
1.Login to Nifi
2.Drag Processor GetFile
3.Drag Processor Put HDFS
4.Configuring GetFile processor
5.Configuring HDFS Processor
6.Required input file path
7.Make a connection
8.Start the processors
9.Check Output File Location
Logging In
Clicking the login link will open the log in page. If the user is logging in with their username/password they will be presented with a form to do so.

Building a DataFlow:
A DFM is able to build an automated dataflow using the NiFi UI. Simply drag components from the toolbar to the canvas, configure the components to meet specific needs, and connect the components together.
Adding Components to the Canvas

Processor :

The Processor is the most commonly used component, as it is responsible for data ingress, egress, routing, and manipulating. There are many different types of Processors. In fact, this is a very common Extension Point in NiFi, meaning that many vendors may implement their own Processors to perform whatever functions are necessary for their use case. When a Processor is dragged onto the canvas, the user is presented with a dialog to choose which type of Processor to use:
Clicking the Add button or double-clicking on a GetFile Processor Type will add the selected Processor to the canvas at the location that it was dropped.
Clicking the Add button or double-clicking on a PutHDFS Processor Type will add the selected Processor to the canvas at the location that it was dropped.
Once you have dragged a Processor onto the canvas, you can interact with it by right-clicking on the Processor and selecting an option from the context menu.
Configure:
This option allows the user to establish or change the configuration of the Processor.
Start or Stop:
This option allows the user to start or stop a Processor; the option will be either Start or Stop, depending on the current state of the Processor.
Stats:
This option opens a graphical representation of the Processor’s statistical information over time.
Data provenance:
This option displays the NiFi Data Provenance table, with information about data provenance events for the FlowFiles routed through that Processor
Usage:
This option takes the user to the Processor’s usage documentation.
Change color:
This option allows the user to change the color of the Processor, which can make the visual management of large flows easier.
Center in view:
This option centers the view of the canvas on the given Processor.
Copy:
This option places a copy of the selected Processor on the clipboard, so that it may be pasted elsewhere on the canvas by right-clicking on the canvas and selecting Paste. The Copy/Paste actions also may be done using the keystrokes Ctrl-C (Command-C) and Ctrl-V (Command-V).
Delete:
This option allows the DFM to delete a Processor from the canvas.
Configuring a GetFile Processor:

To configure a processor, right-click on the Processor and select the Configure option from the context menu. The configuration dialog is opened with four different tabs, each of which is discussed below. Once you have finished configuring the Processor, you can apply the changes by clicking the Apply button or cancel all changes by clicking the Cancel button.

Note that after a Processor has been started, the context menu shown for the Processor no longer has a Configure option but rather has a View Configuration option. Processor configuration cannot be changed while the Processor is running. You must first stop the Processor and wait for all of its active tasks to complete before configuring the Processor again.

Settings Tab
This tab contains several different configuration items. First, it allows the DFM to change the name of the Processor. The name of a Processor by default is the same as the Processor type. This tab also includes other configuration settings like auto terminate relationship,penalty duration, yield duration, bulletin level(refer Apache User Guide for more detail).
Scheduling Tab
The second tab in the Processor Configuration dialog is the Scheduling which contains the different types of scheduling such as Time Driven , Event Driven and Cron Driven.
Properties Tab
The Properties Tab provides a mechanism to configure Processor-specific behavior. There are no default properties. Each type of Processor must define which Properties make sense for its use case.

Here we have to configure properties for a GetFile and PutHDFS Processors.
GetFile Processor Properties:
Description:
Creates FlowFiles from files in a directory. NiFi will ignore files it doesn’t have at least read permissions for.

Here we configure Input Directory location (**/home/hduser/details_console) of the file where it is stored on the local disk. Other required properties we can override as per our requirements. Here i leave it as default configurations.

PutHDFS Processor Properties:

Here we configure Hadoop Configurations Resources directory (**/home/hduser/hadoop-2.7.1/etc/hadoop/core-site.xml and hdfs-site.xml) and destination directory (**nifi/nifi_put_hdfs_ex) where to store the file in HDFS. We can override other configuration properties, here i configured with default configurations properties.
Auto-terminate relationships:

Each of the Relationships that is defined by the Processor is listed here, along with its description. In order for a Processor to be considered valid and able to run, each Relationship defined by the Processor must be either connected to a downstream component or auto-terminated.
If a Relationship is auto-terminated, any FlowFile that is routed to that Relationship will be removed from the flow and its processing considered complete. Any Relationship that is already connected to a downstream component cannot be auto-terminated.
Here we checked the success and failure checkboxes for auto terminate the data flow whether the flow may get success or failure.
Comments Tab
The last tab in the Processor configuration dialog is the Comments tab. This tab simply provides an area for users to include whatever comments are appropriate for this component.
Connecting Components
Once processors and other components have been added to the canvas and configured, the next step is to connect them to one another so that NiFi knows what to do with each FlowFile after it has been processed. This is accomplished by creating a Connection between each component. When the user hovers the mouse over the center of a component, a new Connection icon appears
The user drags the Connection bubble from one component to another until the second component is highlighted. When the user releases the mouse, a Create Connection dialog appears. This dialog consists of two tabs: ‘Details’ and ‘Settings’.
Here we configured create connection dialog box as per the default details and settings configurations.

 
Processor Validation
Before trying to start a Processor, it’s important to make sure that the Processor’s configuration is valid. A status indicator is shown in the top-left of the Processor. If the Processor is invalid, the indicator will show a red Warning indicator with an exclamation mark indicating that there is a problem.
In this case, hovering over the indicator icon with the mouse will provide a tooltip showing all of the validation errors for the Processor. Once all of the validation errors have been addressed, the status indicator will change to a Stop icon, indicating that the Processor is valid and ready to be started but currently is not running.
Command and Control of the DataFlow
When a component is added to the NiFi canvas, it is in the Stopped state. In order to cause the component to be triggered, the component must be started. Once started, the component can be stopped at any time. From a Stopped state, the component can be configured, started, or disabled.
Starting a Component
In order to start a component, the following conditions must be met:
The component’s configuration must be valid.
All defined Relationships for the component must be connected to another component or auto-terminated.
The component must be stopped.
The component must be enabled.
The component must have no active tasks.
Components can be started by selecting all of the components to start and then clicking the Start icon ( ) in the Actions Toolbar or by right-clicking a single component and choosing Start from the context menu.
Starting GetFile Processor:

Starting PutHDFS Processor:

Input File in Local Ubuntu:

HDFS State before start Data Flow:

Local File Directory in ubuntu after starting Data Flow:

HDFS State after starting Data Flow:


Stopping a Component
A component can be stopped any time that it is running. A component is stopped by right-clicking on the component and clicking Stop from the context menu, or by selecting the component and clicking the Stop icon ( ) in the Actions Toolbar.
Save