ConversionUtils

Type Members

case class UriComponents(scheme: String, host: String, port: Integer, path: Option[String], query: Option[String], fragment: Option[String]) extends Product with Serializable

Simple case class wrapper around the components of a URI.

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
def booleanToJByte(bool: Boolean): Byte

Helper to convert a Boolean value to a Byte.
Helper to convert a Boolean value to a Byte. Does not require any validation.
bool
The Boolean to convert into a Byte
returns
0 if false, 1 if true
def byteToBoolean(b: Byte): Validation[String, Boolean]

Helper to convert a Byte value (1 or 0) into a Boolean.
Helper to convert a Byte value (1 or 0) into a Boolean.
b
The Byte to turn into a Boolean
returns
the Boolean value of b, or an error message if b is not 0 or 1 - all boxed in a Scalaz Validation
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
def decodeBase64Url(field: String, str: String): Validation[String, String]

Decodes a URL-safe Base64 string.
Decodes a URL-safe Base64 string.
For details on the Base 64 Encoding with URL and Filename Safe Alphabet see:
http://tools.ietf.org/html/rfc4648#page-7
field
The name of the field
str
The encoded string to be decoded
returns
a Scalaz Validation, wrapping either an an error String or the decoded String
val decodeString: (String, String, String) ⇒ ValidatedString

Decodes a String in the specific encoding, also removing: * Newlines - because they will break Hive * Tabs - because they will break non-Hive targets (e.g.
Decodes a String in the specific encoding, also removing: * Newlines - because they will break Hive * Tabs - because they will break non-Hive targets (e.g. Infobright)
IMPLDIFF: note that this version, unlike the Hive serde version, does not call cleanUri. This is because we cannot assume that str is a URI which needs 'cleaning'.
TODO: simplify this when we move to a more robust output format (e.g. Avro) - as then no need to remove line breaks, tabs etc
returns
a Scalaz Validation, wrapping either an error String or the decoded String
def doubleDecode(field: String, str: String): ValidatedString

Decode double-encoded percents, then percent decode
Decode double-encoded percents, then percent decode
field
The name of the field
str
The String to decode
returns
a Scalaz Validation, wrapping either an error String or the decoded String
def encodeBase64Url(str: String): String

Encodes a URL-safe Base64 string.
Encodes a URL-safe Base64 string.
For details on the Base 64 Encoding with URL and Filename Safe Alphabet see:
http://tools.ietf.org/html/rfc4648#page-7
str
The string to be encoded
returns
the string encoded in URL-safe Base64
def encodeString(enc: String, str: String): String

Encodes a string in the specified encoding
Encodes a string in the specified encoding
enc
The encoding to be used
str
The string which needs to be URLEncoded
returns
a URL encoded string
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def explodeUri(uri: URI): UriComponents

Explodes a URI into its 6 components pieces.
Explodes a URI into its 6 components pieces. Simple code but we use it in multiple places
uri
The URI to explode into its constituent pieces
returns
The 6 components in a UriComponents case class
def extractQuerystring(uri: URI, encoding: String): Validation[String, Map[String, String]]

Attempt to extract the querystring from a URI as a map
Attempt to extract the querystring from a URI as a map
uri
URI containing the querystring
encoding
Encoding of the URI
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
def fixTabsNewlines(str: String): Option[String]

Replaces tabs with four spaces and removes newlines altogether.
Replaces tabs with four spaces and removes newlines altogether.
Useful to prepare user-created strings for fragile storage formats like TSV.
str
The String to fix
returns
The String with tabs and newlines fixed.
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def hashCode(): Int

Definition Classes
AnyRef → Any
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
def makeTsvSafe(str: String): String

Quick helper to make sure our Strings are TSV-safe, i.e.
Quick helper to make sure our Strings are TSV-safe, i.e. don't include tabs, special characters, newlines etc.
str
The string we want to make safe
returns
a safe String
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def singleEncodePcts(str: String): String

On 17th August 2013, Amazon made an unannounced change to their CloudFront log format - they went from always encoding % characters, to only encoding % characters which were not previously encoded.
On 17th August 2013, Amazon made an unannounced change to their CloudFront log format - they went from always encoding % characters, to only encoding % characters which were not previously encoded. For a full discussion of this see:
https://forums.aws.amazon.com/thread.jspa?threadID=134017&tstart=0#
On 14th September 2013, Amazon rolled out a further fix, from which point onwards all fields, including the referer and useragent, would have %s double-encoded.
This causes issues, because the ETL process expects referers and useragents to be only single-encoded.
This function turns a double-encoded percent (%) into a single-encoded one.
Examples: 1. "page=Celestial%25Tarot" - no change (only single encoded) 2. "page=Dreaming%2520Way%2520Tarot" -> "page=Dreaming%20Way%20Tarot" 3. "loading 30%2525 complete" -> "loading 30%25 complete"
Limitation of this approach: %2588 is ambiguous. Is it a: a) A double-escaped caret "ˆ" (%2588 -> %88 -> ^), or: b) A single-escaped "%88" (%2588 -> %88)
This code assumes it's a).
str
The String which potentially has double-encoded %s
returns
the String with %s now single-encoded
val stringToBoolean: (String, String) ⇒ Validation[String, Boolean]

Converts a String of value "1" or "0" to true or false respectively.
Converts a String of value "1" or "0" to true or false respectively.
returns
True for "1", false for "0", or an error message for any other value, all boxed in a Scalaz Validation
val stringToBooleanlikeJByte: (String, String) ⇒ Validation[String, Byte]

Extract a Java Byte representing 1 or 0 only from a String, or error.
Extract a Java Byte representing 1 or 0 only from a String, or error.
returns
a Scalaz Validation, being either a Failure String or a Success Byte
val stringToDouble: (String, String) ⇒ Validation[String, Double]

Converts a String to a Double.
Converts a String to a Double. Takes a field name and a string value and return a validated float.
val stringToDoublelike: (String, String) ⇒ ValidatedString

Convert a String to a String containing a Redshift-compatible Double.
Convert a String to a String containing a Redshift-compatible Double.
Necessary because Redshift does not support all Java Double syntaxes e.g. "3.4028235E38"
Note that this code does NOT check that the value will fit within a Redshift Double - meaning Redshift may silently round this number on load.
returns
a Scalaz Validation, being either a Failure String or a Success String
val stringToJInteger: (String, String) ⇒ Validation[String, Integer]

Extract a Scala Int from a String, or error.
Extract a Scala Int from a String, or error.
returns
a Scalaz Validation, being either a Failure String or a Success JInt
def stringToMaybeDouble(field: String, str: String): Validation[String, Option[Double]]

Convert a String to a Double
Convert a String to a Double
field
The name of the field we are validating. To use in our error message
str
The String which we hope contains a Double
returns
a Scalaz Validation, being either a Failure String or a Success Double
val stringToTwoDecimals: (String, String) ⇒ Validation[String, Double]

Converts a String to a Double with two decimal places.
Converts a String to a Double with two decimal places. Used to honor schemas with multipleOf 0.01. Takes a field name and a string value and return a validated double.
def stringToUri(uri: String): Validation[String, Option[URI]]

Parses a string to create a URI.
Parses a string to create a URI. Parsing is relaxed, i.e. even if a URL is not correctly percent-encoded or not RFC 3986-compliant, it can be parsed.
uri
String containing the URI to parse.
returns
Validation wrapping the result of the parsing:
- Success with the parsed URI if there was no error or with None if the input was null.
- Failure with the error message if something went wrong.
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
AnyRef → Any
def truncate(str: String, length: Int): String

Truncates a String - useful for making sure Strings can't overflow a database field.
Truncates a String - useful for making sure Strings can't overflow a database field.
str
The String to truncate
length
The maximum length of the String to keep
returns
the truncated String
val validateInteger: (String, String) ⇒ ValidatedString

returns
a Scalaz ValidatedString containing either the original String on Success, or an error String on Failure.
val validateUuid: (String, String) ⇒ ValidatedString

Validates that the given field contains a valid UUID.
Validates that the given field contains a valid UUID.
returns
a Scalaz ValidatedString containing either the original String on Success, or an error String on Failure.
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Doc: package utils

object ConversionUtils

Type Members

case class UriComponents(scheme: String, host: String, port: Integer, path: Option[String], query: Option[String], fragment: Option[String]) extends Product with Serializable

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

def booleanToJByte(bool: Boolean): Byte

def byteToBoolean(b: Byte): Validation[String, Boolean]

def clone(): AnyRef

def decodeBase64Url(field: String, str: String): Validation[String, String]

val decodeString: (String, String, String) ⇒ ValidatedString

def doubleDecode(field: String, str: String): ValidatedString

def encodeBase64Url(str: String): String

def encodeString(enc: String, str: String): String

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def explodeUri(uri: URI): UriComponents

def extractQuerystring(uri: URI, encoding: String): Validation[String, Map[String, String]]

def finalize(): Unit

def fixTabsNewlines(str: String): Option[String]

final def getClass(): Class[_]

def hashCode(): Int

final def isInstanceOf[T0]: Boolean

def makeTsvSafe(str: String): String

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

def singleEncodePcts(str: String): String

val stringToBoolean: (String, String) ⇒ Validation[String, Boolean]

val stringToBooleanlikeJByte: (String, String) ⇒ Validation[String, Byte]

val stringToDouble: (String, String) ⇒ Validation[String, Double]

val stringToDoublelike: (String, String) ⇒ ValidatedString

val stringToJInteger: (String, String) ⇒ Validation[String, Integer]

def stringToMaybeDouble(field: String, str: String): Validation[String, Option[Double]]

val stringToTwoDecimals: (String, String) ⇒ Validation[String, Double]

def stringToUri(uri: String): Validation[String, Option[URI]]

final def synchronized[T0](arg0: ⇒ T0): T0

def toString(): String

def truncate(str: String, length: Int): String

val validateInteger: (String, String) ⇒ ValidatedString

val validateUuid: (String, String) ⇒ ValidatedString

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from AnyRef

Inherited from Any

Ungrouped