PipePython2

This node runs any given Python code. It pipes the incoming DataFrame through pipe to the Python Script. Output back to Spark has to be written out using print.

Input

It pipes the incoming DataFrame through pipe to the Python Script. It also passes the Schema of the DataFrame to the Python script through the command line argument - argv[1]

Output

Output back to Spark has to be written out using print.

Type

transform

Fields

Name	Title	Description
codeHeader	Pipe Header Code	Header part of the Python code to be run. It receives each record as a string
codeBody	Pipe Body Code	Body part of the Python code to be run.
codeFooter	Pipe Footer Code	Footer part of the Python code to be run. It should write out each resulting record back as a string.
outputColNames	Output Column Names	Output Schema of Pipe Python Processor
outputColTypes	Output Column Types	Data Type of the Output Columns
outputColFormats	Output Column Formats	Format of the Output Columns