UDFUtils

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def arrayColumn(value: String, values: String*): Column

Function to take variable number of values and create an array column out of it.
Function to take variable number of values and create an array column out of it.
value
input value
values
variable number of input values.
returns
an array of column.
val array_value: UserDefinedFunction

UDF to find and return element in arr sequence at passed index.
UDF to find and return element in arr sequence at passed index. If no element found then null is returned.
final def asInstanceOf[T0]: T0

Definition Classes
Any
val call_rest_api: UserDefinedFunction

Spark UDF that makes a single blocking rest API call to a given url.
Spark UDF that makes a single blocking rest API call to a given url. The result of this udf is always produced, contains a proper error if it failed at any stage, and never interrupts the job execution (unless called with invalid signature).
The default timeout can be configured through the spark.network.timeout Spark configuration option.
Parameters:
- method - any supported HTTP1.1 method type, e.g. POST, GET. Complete list: httpMethods.
- url - valid url to which a request is going to be made
- headers - an array of "key: value" headers that are past with the request
- content - any content (by default, the supported rest api content type is application/json)
Response - a struct with the following fields:
- isSuccess - boolean, whether a successful response has been received
- status - nullable integer, status code (e.g. 404, 200, etc)
- headers - an array of name: value response headers (e.g. [Server: akka-http/10.1.10, Date: Tue, 07 Sep 2021 18:11:47 GMT])
- content - nullable string, response back
- error - nullable string, if the parameters passed are valid or the system failed to make a call, this field contains an error message
Definition Classes
RestAPIUtils
def castDataType(sparkSession: SparkSession, df: DataFrame, column: Column, dataType: String, replaceColumn: String): DataFrame

Function to add new typecasted column in input dataframe.
Function to add new typecasted column in input dataframe. Newly added column is typecasted version of passed column. Typecast operation is supported for string, boolean, byte, short, int, long, float, double, decimal, date, timestamp
sparkSession
spark session
df
input dataframe
column
input column to be typecasted
dataType
datatype to cast column to.
replaceColumn
column name to be added in dataframe.
returns
new dataframe with new typecasted column.
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@HotSpotIntrinsicCandidate() @throws( ... )
def createLookup(name: String, df: DataFrame, spark: SparkSession, keyCols: List[String], rowCols: String*): UserDefinedFunction

Function registers 4 different UDFs with spark registry.
Function registers 4 different UDFs with spark registry. UDF for lookup_match, lookup_count, lookup_row and lookup functions are registered. This function stores the data of input dataframe in a broadcast variable, then uses this broadcast variable in different lookup functions.
lookup : This function returns the first matching row for given input keys lookup_count : This function returns the count of all matching rows for given input keys. lookup_match : This function returns 0 if there is no matching row and 1 for some matching rows for given input keys. lookup_row : This function returns all the matching rows for given input keys.
This function registers for upto 10 matching keys as input to these lookup functions.
name
UDF Name
df
input dataframe
spark
spark session
keyCols
columns to be used as keys in lookup functions.
rowCols
schema of entire row which will be stored for each matching key.
returns
registered UDF definitions for lookup functions. These UDF functions returns different results depending on the lookup function.
def createRangeLookup(name: String, df: DataFrame, spark: SparkSession, minColumn: String, maxColumn: String, valueColumns: String*): UserDefinedFunction

Method to create UDF which looks for passed input double in input dataframe.
Method to create UDF which looks for passed input double in input dataframe. This function first loads the data of dataframe in broadcast variable and then defines a UDF which looks for input double value in the data stored in broadcast variable. If input double lies between passed col1 and col2 values then it adds corresponding row in the returned result. If value of input double doesn't lie between col1 and col2 then it simply returns null for current row in result.
name
created UDF name
df
input dataframe
spark
spark session
minColumn
column whose value to be considered as minimum in comparison.
maxColumn
column whose value to be considered as maximum in comparison.
valueColumns
remaining column names to be part of result.
returns
registers UDF which in turn returns rows corresponding to each row in dataframe on which range UDF is called.
def dropColumns(sparkSession: SparkSession, df: DataFrame, columns: Column*): DataFrame

Function to drop passed columns from input dataframe.
Function to drop passed columns from input dataframe.
sparkSession
spark session
df
input dataframe.
columns
list of columns to be dropped from dataframe.
returns
new dataframe with dropped columns.
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
Annotations
@HotSpotIntrinsicCandidate()
def hashCode(): Int

Definition Classes
AnyRef → Any
Annotations
@HotSpotIntrinsicCandidate()
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
lazy val logger: Logger

Attributes
protected
Definition Classes
LazyLogging
def lookup(lookupName: String, cols: Column*): Column

By default returns only the first matching record
def lookup_count(lookupName: String, cols: Column*): Column
def lookup_last(lookupName: String, cols: Column*): Column

Returns the last matching record
def lookup_match(lookupName: String, cols: Column*): Column

returns
Boolean Column
def lookup_nth(lookupName: String, cols: Column*): Column
def lookup_range(lookupName: String, input: Column): Column
def lookup_row(lookupName: String, cols: Column*): Column
def lookup_row_reverse(lookupName: String, cols: Column*): Column
def measure[T](fn: ⇒ T)(caller: String = findCaller()): T
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
Annotations
@HotSpotIntrinsicCandidate()
final def notifyAll(): Unit

Definition Classes
AnyRef
Annotations
@HotSpotIntrinsicCandidate()
def registerProphecyUdfs(spark: SparkSession): Unit
def replaceString(sparkSession: SparkSession, df: DataFrame, outputCol: String, inputCol: String, replaceWith: String, value: String, values: String*): DataFrame

Function to add new column in passed dataframe.
Function to add new column in passed dataframe. Newly added column value is decided by the presence of value corresponding to inputCol in array comprised of value and values. If inputCol is found then value of replaceWith is added in new column otherwise inputCol value is added.
sparkSession
spark session.
df
input dataframe.
outputCol
name of new column to be added.
inputCol
column name whose value is searched.
replaceWith
value with which to replace searched value if found.
value
element to be combined in array column
values
all values to be combined in array column for searching purpose.
returns
dataframe with new column with column name outputCol
def replaceStringNull(sparkSession: SparkSession, df: DataFrame, outputCol: String, inputCol: String, replaceWith: String, value: String, values: String*): DataFrame

Function to add new column in passed dataframe.
Function to add new column in passed dataframe. Newly added column value is decided by the presence of value corresponding to inputCol in array comprised of value and values and null. If inputCol is found then value of replaceWith is added in new column otherwise inputCol value is added.
sparkSession
spark session.
df
input dataframe.
outputCol
name of new column to be added.
inputCol
column name whose value is searched.
replaceWith
value with which to replace searched value if found.
value
element to be combined in array column
values
all values to be combined in array column for searching purpose.
returns
dataframe with new column with column name outputCol
def replaceStringWithNull(sparkSession: SparkSession, df: DataFrame, outputCol: String, inputCol: String, value: String, values: String*): DataFrame

Function to add new column in passed dataframe.
Function to add new column in passed dataframe. Newly added column value is decided by the presence of value corresponding to inputCol in array comprised of value and values and null. If inputCol is found then value of null is added in new column otherwise inputCol value is added.
sparkSession
spark session.
df
input dataframe.
outputCol
name of new Column to be added.
inputCol
column name whose value is searched.
value
element to be combined in array column.
values
all values to be combined in array column for searching purpose.
returns
dataframe with new column with column name outputCol
val replace_string: UserDefinedFunction

UDF to find str in input sequence toBeReplaced and return replace if found.
UDF to find str in input sequence toBeReplaced and return replace if found. Otherwise str is returned.
val replace_string_with_null: UserDefinedFunction

UDF to find str in input sequence toBeReplaced and return null if found.
UDF to find str in input sequence toBeReplaced and return null if found. Otherwise str is returned.
def splitIntoMultipleColumns(sparkSession: SparkSession, df: DataFrame, colName: String, pattern: String, prefix: String = null): DataFrame

Function to split column with colName in input dataframe using split pattern into multiple columns.
Function to split column with colName in input dataframe using split pattern into multiple columns. If prefix name is provided each new generated column is prefixed with prefix followed by column number, otherwise original column name is used.
sparkSession
spark session.
df
input dataframe.
colName
column in dataframe which needs to be split into multiple columns.
pattern
regex with which column in input dataframe will be split into multiple columns.
prefix
column prefix to be used with all newly generated columns.
returns
new dataframe with new columns where new column values are generated after splitting original column colName.
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
val take_last_nth: UserDefinedFunction

UDF to return nth element from last in passed array of elements.
UDF to return nth element from last in passed array of elements. In case input sequence has less number of elements than n then first element is returned.
val take_nth: UserDefinedFunction

UDF to take Nth element from beginning.
UDF to take Nth element from beginning. In case input sequence has less element than N then exception is thrown.
def toString(): String

Definition Classes
AnyRef → Any
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Deprecated Value Members

def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@Deprecated @deprecated @throws( classOf[java.lang.Throwable] )
Deprecated
(Since version ) see corresponding Javadoc for more information.

Related Doc: package libs

trait UDFUtils extends RestAPIUtils with Serializable with LazyLogging

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

def arrayColumn(value: String, values: String*): Column

val array_value: UserDefinedFunction

final def asInstanceOf[T0]: T0

val call_rest_api: UserDefinedFunction

def castDataType(sparkSession: SparkSession, df: DataFrame, column: Column, dataType: String, replaceColumn: String): DataFrame

def clone(): AnyRef

def createLookup(name: String, df: DataFrame, spark: SparkSession, keyCols: List[String], rowCols: String*): UserDefinedFunction

def createRangeLookup(name: String, df: DataFrame, spark: SparkSession, minColumn: String, maxColumn: String, valueColumns: String*): UserDefinedFunction

def dropColumns(sparkSession: SparkSession, df: DataFrame, columns: Column*): DataFrame

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

final def getClass(): Class[_]

def hashCode(): Int

final def isInstanceOf[T0]: Boolean

lazy val logger: Logger

def lookup(lookupName: String, cols: Column*): Column

def lookup_count(lookupName: String, cols: Column*): Column

def lookup_last(lookupName: String, cols: Column*): Column

def lookup_match(lookupName: String, cols: Column*): Column

def lookup_nth(lookupName: String, cols: Column*): Column

def lookup_range(lookupName: String, input: Column): Column

def lookup_row(lookupName: String, cols: Column*): Column

def lookup_row_reverse(lookupName: String, cols: Column*): Column

def measure[T](fn: ⇒ T)(caller: String = findCaller()): T

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

def registerProphecyUdfs(spark: SparkSession): Unit

def replaceString(sparkSession: SparkSession, df: DataFrame, outputCol: String, inputCol: String, replaceWith: String, value: String, values: String*): DataFrame

def replaceStringNull(sparkSession: SparkSession, df: DataFrame, outputCol: String, inputCol: String, replaceWith: String, value: String, values: String*): DataFrame

def replaceStringWithNull(sparkSession: SparkSession, df: DataFrame, outputCol: String, inputCol: String, value: String, values: String*): DataFrame

val replace_string: UserDefinedFunction

val replace_string_with_null: UserDefinedFunction

def splitIntoMultipleColumns(sparkSession: SparkSession, df: DataFrame, colName: String, pattern: String, prefix: String = null): DataFrame

final def synchronized[T0](arg0: ⇒ T0): T0

val take_last_nth: UserDefinedFunction

val take_nth: UserDefinedFunction

def toString(): String

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

final def wait(): Unit

Deprecated Value Members

def finalize(): Unit

Inherited from LazyLogging

Inherited from Serializable

Inherited from Serializable

Inherited from RestAPIUtils

Inherited from AnyRef

Inherited from Any

Ungrouped