Version: v2.5 
DropDuplicateRows
This node creates a new DataFrame by dropping duplicate rows
Type
transform
Input
A DataFrame to filter rows on duplicate.
Output
A DataFrame containing no duplicate row on the selected columns.
Fields
Name | Type | Description |
---|---|---|
COLUMNS | Seq of columns | Seq of columns to check the duplicate condition.(By default duplication checked on all the columns) |
Examples
Input
name | age | height |
---|---|---|
Alice | 5 | 80 |
Alice | 5 | 80 |
Alice | 10 | 80 |
Parameters1 ----------
Name | Value |
---|---|
COLUMNS | name,height |
Output1
name | age | height |
---|---|---|
Alice | 5 | 80 |
Parameters2 ----------
Name | Value |
---|---|
COLUMNS |
Output2
name | age | height |
---|---|---|
Alice | 5 | 80 |
Alice | 10 | 80 |