Unique Rows

The Unique rows step removes duplicate rows from the input stream(s).

Important: Make sure that the input stream is sorted; otherwise, only consecutive double rows are evaluated correctly.

Options

The table below contains descriptions of all options for the Unique rows step:

Option	Description
Step name	Name of the step; this name has to be unique in a single transformation
Add counter to output?
Fields to compare table	Specify the field names on which you want to force uniqueness or click Get to insert all fields from the input stream(s) You can choose to ignore case by setting the Ignore case flag to Y. For example: Kettle, KETTLE, kettle are the same if the compare is performed as case-insensitive. In this instance, the first occurrence (Kettle) is passed to the next step(s).

Strings Cut

This step allows you to cut a portion of a substring. If the designated field is out of range, it returns blank.

Option	Definition
Step name	The name of this step as it appears in the transformation workspace.
In stream field	Name of the field whose substring to cut.
Out stream field	Rename the field upon output.
Cut from	Designate where to begin cutting the substring.
Cut to	Designate where to end cutting the substring.

Split Fields

Apply operations, such as trimming, padding, and others to the string value.

Option	Definition
Step name	The name of this step as it appears in the transformation workspace.
In stream field	Designate the field to transform.
Out stream field	Designate the name of the field to be created.
Trim type	Designate the trim type: none, left, right, or both.
Lower/Upper	Designate upper or lowercase.
Padding	Designate left or right padding.
Pad char	Designate the padding character.
Pad Length	Designate how long the padding will be.
InitCap	Transform to initial capitalization.
Escape
Digits	Designate whether to return remove, or do nothing to digits.
Remove Special character	Designate a special character to remove.

String Operations

Apply operations, such as trimming, padding, and others to the string value.

Option	Definition
Step name	The name of this step as it appears in the transformation workspace.
In stream field	Designate the field to transform.
Out stream field	Designate the name of the field to be created.
Trim type	Designate the trim type: none, left, right, or both.
Lower/Upper	Designate upper or lowercase.
Padding	Designate left or right padding.
Pad char	Designate the padding character.
Pad Length	Designate how long the padding will be.
InitCap	Transform to initial capitalization.
Escape
Digits	Designate whether to return remove, or do nothing to digits.
Remove Special character	Designate a special character to remove.

Split Fields

Description

The Field splitter step allows you to split fields based on delimiter information.

Options

The table below contains options configuring the Field splitter step:

Option	Description
Step name	Name of the step; this name has to be unique in a single transformation
Field to split	The name of the field you want to split
Delimiter	Delimiter that determines the end of a field
Fields table	This table is where you define the properties for each new field created by the split. For each new field, you must define the field name, data type, and other properties.

Split Field to Rows

Description

This step allows you to split a row containing a delimited field into multiple new rows, one per split value.
The original row is not delivered to following steps. The original field is left in each new row to assist in debugging or error handling.  It can be removed via a Select Values step if needed.
The original field must be a string field.
A new field name must be specified.
If the value type/format of the new row is not specified, the new field will use the same type/format as the original field.

Options

Option	Description
Step name	Name of the step. Note: this name has to be unique in a single transformation
Field to split	The field you want to split
Delimiter	The delimiter or separator used
New name field

Sort Rows

The Sort rows step sorts rows based on the fields you specify and on whether they should be sorted in ascending or descending order.

Notes:

• Kettle has to sort rows using temporary files when the number of rows exceeds the specified sort size (default 1 million rows). When you get an out of memory exception (OOME), you need to lower this size of change your available memory.
• When you use multiple copies of the step in parallel (on the local JVM with "Change number of copies to start" or in a clustered environment using Carte) each of the sorted blocks need to be merged together to ensure the proper sort sequence. This can be done, be adding the Sorted Merge step afterwards (on the local JVM without multiple copies to start or in the cluster on the master).

Options

The following table describes the options associated with the Sort step:

Option	Description
Step name	Name of the step;this name has to be unique in a single transformation.
Sort directory	The directory in which the temporary files are stored in case when needed; the default is the standard temporary directory for the system
TMP-file prefix	Choose an easily recognized prefix so you can identify the files when they show up in the temp directory.
Sort size	The more rows you store in memory, the faster the sorting process because fewer temporary files must be used and less I/O is generated.
Free memory threshold (in %)	If the sort algorithm finds that it has less available free memory than the indicated number, it will start to page data to disk. Note: This is not exact science, because: This is checked every 1000 rows. Depending on the row size and other steps within complex transformations this could still lead to an OutOfMemoryError. In a Java Virtual Machine it's not possible to know the exact amount of free memory.  As such we recommend you don't use this for very complex transformations with other steps and processes that use up a lot of memory.
Compress TMP Files	Compresses temporary files when they are needed to complete the sort.
Only pass unique rows?	Enable if you want to pass unique rows only to the output stream(s).
Fields table	Specify the fields and direction (ascending/descending) to sort. You can specify whether to perform a case sensitive sort (optional)
Get Fields	Click to retrieve a list of all fields coming in on the stream(s).