Converts 1 digit julian year to 4 digits julian year.
Converts 1 digit julian year to 4 digits julian year.
date in Julian in "YJJJ" format
date in "yyyyMMdd" format
a date in "YYYYJJJ"
Method used for abinitio's reinterpret_as function to read necessary bytes from byteArray for input data and convert into struct format as per provided in typeInfo sequence.
Method used for abinitio's reinterpret_as function to read necessary bytes from byteArray for input data and convert into struct format as per provided in typeInfo sequence.
TypeInfo can have multiple entries, each could be either decimal or string type. Depending on the argument passed within decimal or string bytes are read from input byte array.
If decimal or string argument has some integer then that many bytes are read from input byte array or if decimal or string has some string delimiter as its argument then from the current position bytes are read until string delimiter is found in input byte array.
Method to read values from inputData and create dataframe with column name as columnName and column type as columnType for the values in inputData delimiter by delimiter.
Returns the internal representation of a date resulting from adding (or subtracting) a number of months to the specified date.
Returns the internal representation of a date resulting from adding (or subtracting) a number of months to the specified date.
in yyyy-MM-dd format
Computes number of days between two specified dates in "yyyyMMdd" format
Computes number of days between two specified dates in "yyyyMMdd" format
input date
input date
number of days between laterDate and earlierDate or null if either one is null
Returns the internal representation of a timestamp resulting from adding (or subtracting) a number of months to the specified timestamp.
Returns the internal representation of a timestamp resulting from adding (or subtracting) a number of months to the specified timestamp.
timestamp in yyyy-MM-dd HH:mm:ss.SSSS format
Returns the number of hours between two specified dates in standard format yyyy-MM-dd HH:mm:ss.SSSS.
Method uses a java regex to identify decimal numbers from input string.
Method uses a java regex to identify decimal numbers from input string. This decimal number could be of 3 types 1. Simple integral number. e.g. 013334848. This part is identified by regex. 2. decimal number with explicit decimal point. e.g. 123456.90. This part is identified by combination of [0-9]+(\$$decimal_point_char)[0-9]+ and (0\$$decimal_point_char)[0-9]+ regex
After extracting decimal number this code checks if length of decimal number is more than len parameter or not. If length is more than len parameter then it simply returns this extracted decimal number. Otherwise it first left pad decimal number with char_to_pad_with to make its length equal to len parameter and then adjusts minus sign (-) to left most part of decimal number.
input string.
length of characters.
character to left pad with. default value is "0"
A string that specifies the character that represents the decimal point.
a decimal string of the specified length or longer, left-padded with a specified character as needed and trimmed of leading zeros.
Method uses a java regex to identify decimal numbers from input string.
Method uses a java regex to identify decimal numbers from input string. This decimal number could be of 3 types 1. Simple integral number. e.g. 013334848. This part is identified by combination of [1-9][0-9]*[0-9] and [1-9]+ regex 2. decimal number with explicit decimal point. e.g. 123456.90. This part is identified by combination of [1-9][0-9]*(\\\$$decimal_point_char)[0-9]+ and (0\\\$$decimal_point_char)[0-9]*[0-9] regex
After extracting decimal number this code checks if length of decimal number is more than len parameter or not. If length is more than len parameter then it simply returns this extracted decimal number. Otherwise it first left pad decimal number with char_to_pad_with to make its length equal to len parameter and then adjusts minus sign (-) to left most part of decimal number.
input string.
length of characters.
character to left pad with. default value is "0"
A string that specifies the character that represents the decimal point.
a decimal string of the specified length or longer, left-padded with a specified character as needed and trimmed of leading zeros.
Function returns a value which is rounded down to right_digits number of digits to the right of decimal point.
Returns a number rounded up to a specified number of places to the right of the decimal point.
Function uses a java regex to identify decimal numbers from input string.
Function uses a java regex to identify decimal numbers from input string. This decimal number could be of 3 types 1. Simple integral number. e.g. 013334848. This part is identified by combination of [1-9][0-9 ]*[0-9] and [1-9]+ regex 2. decimal number with explicit decimal point. e.g. 123456.90. This part is identified by combination of [1-9][0-9]*(\$$decimal_point_char)[0-9 ]+ and (0\$$decimal_point_char)[0-9 ]*[0-9] regex
After extracting decimal number this code looks for minus sign before extracted number in input and appends it with decimal number if found minus sign.
In the end it replaces all whitespaces with empty string in the final resultant decimal number.
input string
A string that specifies the character that represents the decimal point.
a decimal from a string that has been trimmed of leading zeros and non-numeric characters.
UDF to get record of type decode_datetime_type.
UDF to get record of type decode_datetime_type. This record will have all its fields populated with corresponding entries in input date/timestamp.
Returned record will have following schema.
integer(8) year; integer(8) month; integer(8) day; integer(8) hour; integer(8) minute; integer(8) second; integer(8) microsecond;
Note: Supported Input time is in yyyy-MM-dd HH:mm:ss.SSSSSS or yyyy-MM-dd HH:mm:ss or yyyy-MM-dd formats only. Additional handling is done to support timestamp retrieved from now() function call.
integer values specifying days relative to January 1, 1900.
integer values specifying days relative to January 1, 1900. This function returns the internal representation of a date given the year, month, and date. encode_date returns the internal representation of the date specified by the year 1998, the month 5, and the day 18:encode_date(1998, 5, 18) = 35931
Returns true if string columns ends with given suffix
Method to return the result of evaluating a string expression in the context of a specified input column.
Method to return the result of evaluating a string expression in the context of a specified input column. Here input column could be struct type record, simple column, array type etc. Here expr could be reference to nested column inside input column or any expression which requires values from input column for its evaulation.
Note: Current implementation only supports scenerio where input column is of struct type and expr is simply dot separated column reference to input struct.
UDF to get file information for passed input file path.
Method to identify and return first non null expression.
Method to create dataframe with single column containing increasing sequence id from start to end.
Function to create sequence of array between two passed numbers
Function to create sequence of array between two passed numbers
starting point of generated sequence
terminating point of generated sequence.
column containing sequence of integers.
UDF to generate column with sequence of integers between two passed start and end columns.
UDF to get last Byte from ByteArray of input data.
Computes number of days in February month in a given year
Computes number of days in February month in a given year
year whose number of days in February needs to be calculated
number of days
Method to get field at specific position from struct column
UDF to get integer comprising of last 4 Bytes from ByteArray of input data.
UDF to get long comprising of last 8 Bytes from ByteArray of input data.
UDF to get long comprising of last 8 Bytes from ByteArray of input data.
UDF to get short comprising of last 2 Bytes from ByteArray of input data.
Method to check if current column is null or has empty value.
Checks if a string is ascii
Checks if a string is ascii
column to be checked
true if the input string is ascii otherwise false
Method to identify if input string is a blank string or not.
Method to identify if input string is a blank string or not.
input string.
return 1 if given string contains all blank character or is a zero length string, otherwise it returns 0
Tests whether an object is composed of all binary zero bytes.
Tests whether an object is composed of all binary zero bytes. This function returns: 1. 1 if obj contains only binary zero bytes or is a zero-length string 2. 0 if obj contains any non-zero bytes 3. NULL if obj is NULL
Checks if an input string contains only ascii code and numbers
Checks if an input string contains only ascii code and numbers
string to be checked
true if input string contains only ascii code and numbers or null if input is null
Method to identify if passed input column is a valid expression after typecasting to passed dataType.
Method to identify if passed input column is a valid expression after typecasting to passed dataType. Also while typecasting if len is present then this function also makes sure the max length of input column after typecasting operation is not greater than len.
input column expression to be identified if is valid.
datatype to which input column expression must be typecasted. If datatype is a string then it is treated as timestamp format. If it is a list of string then it is treated as having current timestamp format and and new timestamp format to which input column needs to be typecasted.
max length of input column after typecasting it to dataType.
0 if input column is not valid after typecasting or 1 if it is valid.
Validates date against a input format
Validates date against a input format
A pattern such as yyyy-MM-dd
or yyyy-MM-dd HH:mm:ss.SSSS
or dd.MM.yyyy
Input date to be validated
true if the input date is valid otherwise false
UDF to return a flag for each character if it is present or not in input String.
Method to create array of size "size" containing seedVal as each entry
Method to create array of size "size" containing seedVal as each entry
UDF to get multifile information for passed input file path.
UDF for murmur hash generation for any column type
Method to get current timestamp.
Method to get current timestamp.
current timestamp in YYYYMMddHHmmssSSSSSS format.
udf to group input decimal into multiple groups separated by separator
Returns the first string in a target string that matches a regular expression.
UDF wrapper over re_index function.
Returns the first string in a target string that matches a regular expression.
Replaces all substrings in a target string that match a specified regular expression.
Replaces all substrings in a target string that match a specified regular expression.
A string that the function searches for a substring that matches pattern_expr.
regular expression
replacement string
Number of characters, from the beginning of str, to skip before searching.
a replaced string in which all substrings, which matches a specified regular expression, are replaced.
Replaces only the first regex matching occurrence in the target string.
Replaces only the first regex matching occurrence in the target string.
A string that the function searches for a substring that matches pattern_expr.
regular expression
replacement string
a replaced string in which first substring, which matches a specified regular expression, is replaced.
UDF to split input string via pattern string and remove all empty subtrings.
Method removes any non-digit characters from the specified string column.
Method removes any non-digit characters from the specified string column.
input String Column
Cleaned string column or null
Method to replace String Columns with Empty value to Null.
Adds an explicit sign to the number.
Adds an explicit sign to the number. E.g. 2 -> +2; -004 -> -004; 0 -> +0
UDF to break input string into multiple string via delimiter.
UDF to break input string into multiple string via delimiter. Number of strings after split are adjusted as per passed width parameter. If number of strings are less then empty strings are added otherwise in case of more number of strings, first width number of entries are picked and remaining are discarded.
Returns true if string columns starts with given prefix
Method to return character code of character at index position in inputStr string.
Method to return character code of character at index position in inputStr string.
input string
location of character to get code.
integer column.
This implementation is incorrect.
Converts a string from one character set to another, replacing inconvertible characters with a specified string.
Method which returns string of characters present in both of the strings in the same order as appearing in first string
Compares two input strings, then returns characters that appear in one string but not in the other.
UDF to find index of seekStr in inputStr.
UDF to find index of seekStr in inputStr. Returned index will be 1 based index.
UDF to find index of seekStr in inputStr from offset index onwards.
UDF to find index of seekStr in inputStr from offset index onwards. Returned string position is 1 based position.
Method which returns true if input string contains all alphabetic characters, or false otherwise.
Method which returns true if input string contains all numeric characters, or false otherwise.
Concatenates the elements of column using the delimiter.
Method to test whether a string matches a specified pattern.
Method to test whether a string matches a specified pattern. This function returns 1 if the input string matches a specified pattern, and 0 if the string does not match the pattern.
In abinitio version % character in pattern means to match zero or more characters and _ character means matches a single character.
Left-pad the input string column with pad_char to a length of len.
Left-pad the input string column with pad_char to a length of len. If length of input column is more than len then returns input column unmodified.
function trims the string and then pad the string with given character upto given length.
function trims the string and then pad the string with given character upto given length. if the length of trimmed string is equal to or greater than given length than it return input string
input string
length in number of characters.
A character used to pad input string to length len.
string of a specified length, trimmed of leading and trailing blanks and left-padded with a given character.
function pads input on the right with the character char_to_pad_with to make the string length len.
function pads input on the right with the character char_to_pad_with to make the string length len. If str is already len or more characters long, the function returns input unmodified.
function trims the string and then pad the string on right side with given character upto given length.
function trims the string and then pad the string on right side with given character upto given length. if the length of trimmed string is equal to or greater than given length than it return input string
input string
length in number of characters.
A character used to pad input string to length len.
string of a specified length, trimmed of leading and trailing blanks and left-padded with a given character.
Function to replace occurrence of seekStr with newStr string in input string after offset characters from first character.
Function to replace occurrence of seekStr with newStr string in input string after offset characters from first character.
input string on which to perform replace operation.
string to be replaced in input string.
string to be used instead of seekStr in input string.
number of characters to skip from begining in input string before performing string_replace operation.
modified string where seekStr is replaced with newStr in input string.
Returns the index of the first character of the last occurrence of a seek string within another input string.
Returns the index of the first character of the last occurrence of a seek string within another input string. Returned index is 1 based.
UDF to find index of seekStr in inputStr from end of inputStr skipping offset number of characters from end.
UDF to find index of seekStr in inputStr from end of inputStr skipping offset number of characters from end. Offset index is number of characters, from the end of str, to skip before searching. Returned string position is 1 based position.
UDF to split input string via delimiter string.
UDF to split input string via delimiter string and remove all empty subtrings.
Method to find substring of input string.
Method to find substring of input string.
string on which to find substring.
1 based starting position to find substring from.
total length of substring to be found.
substring of input string
UDF to identify the number of characters in inputStr which are present in charFlag
Method to convert
Method to return integer value representing number of days to today from “1-1-1990”.
Method to return integer value representing number of days to today from “1-1-1990”.
integer value
UDF to return a string in the native character set made up of bytes from the given map.
UDF to return a string in the native character set made up of bytes from the given map. Each byte of the result is the value of map indexed by the character code of the corresponding byte of the input string str. The function returns NULL if any argument is NULL.
UDF to truncate microseconds part of timestamp.
UDF to truncate microseconds part of timestamp. This is needed as abinitio and spark has some incompatibility in microseconds part of timestamp format.
Converts yyyyyMMdd to YYYYJJJ
Converts yyyyyMMdd to YYYYJJJ
date in yyyyMMdd format
a date converted to YYYYJJJ
Method to zip two arrays with first one having event_type and second one having event_text
(Since version ) see corresponding Javadoc for more information.
Library of all spark functions which implements different abinitio functions used in abinitio workflows.