util

Type Members

class AbstractScalaRowIterator[T] extends Iterator[T]

Shim to allow us to implement scala.Iterator in Java.
Shim to allow us to implement scala.Iterator in Java. Scala 2.11+ has an AbstractIterator class for this, but that class is private[scala] in 2.10. We need to explicitly fix this to Row in order to work around a spurious IntelliJ compiler error. This cannot be an abstract class because that leads to compilation errors under Scala 2.11.
class ArrayBasedMapData extends MapData
abstract class ArrayData extends SpecializedGetters with Serializable
case class BadRecordException(record: () ⇒ UTF8String, partialResult: () ⇒ Option[InternalRow], cause: Throwable) extends Exception with Product with Serializable

Exception thrown when the underlying parser meet a bad record and can't parse it.
Exception thrown when the underlying parser meet a bad record and can't parse it.
record
a function to return the record that cause the parser to fail
partialResult
a function that returns an optional row, which is the partial result of parsing this bad record.
cause
the actual exception about why the record is bad and can't be parsed.
class CaseInsensitiveMap[T] extends Map[String, T] with Serializable

Builds a map in which keys are case insensitive.
Builds a map in which keys are case insensitive. Input map can be accessed for cases where case-sensitive information is required. The primary constructor is marked private to avoid nested case-insensitive map creation, otherwise the keys in the original map will become case-insensitive in this scenario.
class GenericArrayData extends ArrayData
abstract class MapData extends Serializable

This is an internal data representation for map type in Spark SQL.
This is an internal data representation for map type in Spark SQL. This should not implement equals and hashCode because the type cannot be used as join keys, grouping keys, or in equality tests. See SPARK-9415 and PR#13847 for the discussions.
sealed trait ParseMode extends AnyRef
class QuantileSummaries extends Serializable

Helper class to compute approximate quantile summary.
Helper class to compute approximate quantile summary. This implementation is based on the algorithm proposed in the paper: "Space-efficient Online Computation of Quantile Summaries" by Greenwald, Michael and Khanna, Sanjeev. (http://dx.doi.org/10.1145/375663.375670)
In order to optimize for speed, it maintains an internal buffer of the last seen samples, and only inserts them after crossing a certain size threshold. This guarantees a near-constant runtime complexity compared to the original algorithm.
class StringKeyHashMap[T] extends AnyRef

Value Members

object ArrayBasedMapData extends Serializable
object ArrayData extends Serializable
object CaseInsensitiveMap extends Serializable
object CompressionCodecs
object DateTimeUtils

Helper functions for converting between internal and external date and time representations.
Helper functions for converting between internal and external date and time representations. Dates are exposed externally as java.sql.Date and are represented internally as the number of dates since the Unix epoch (1970-01-01). Timestamps are exposed externally as java.sql.Timestamp and are stored internally as longs, which are capable of storing timestamps with microsecond precision.
object DropMalformedMode extends ParseMode with Product with Serializable

This mode ignores the whole corrupted records.
object FailFastMode extends ParseMode with Product with Serializable

This mode throws an exception when it meets corrupted records.
object NumberConverter
object ParseMode extends Logging
object PermissiveMode extends ParseMode with Product with Serializable

This mode permissively parses the records.
object QuantileSummaries extends Serializable
object StringKeyHashMap

Build a map with String type of key, and it also supports either key case sensitive or insensitive.
object StringUtils
object TypeUtils

Helper functions to check for valid data types.
def benchmark[A](f: ⇒ A): A
def fileToString(file: File, encoding: String = "UTF-8"): String
def quietly[A](f: ⇒ A): A

Silences output to stderr or stdout for the duration of f
def quoteIdentifier(name: String): String
def resourceToBytes(resource: String, classLoader: ClassLoader = Utils.getSparkClassLoader): Array[Byte]
def resourceToString(resource: String, encoding: String = "UTF-8", classLoader: ClassLoader = Utils.getSparkClassLoader): String
def sideBySide(left: Seq[String], right: Seq[String]): Seq[String]
def sideBySide(left: String, right: String): Seq[String]
def stackTraceToString(t: Throwable): String
def stringOrNull(a: AnyRef): String
def stringToFile(file: File, str: String): File
def toPrettySQL(e: Expression): String
def usePrettyExpression(e: Expression): Expression

package util

Type Members

class AbstractScalaRowIterator[T] extends Iterator[T]

class ArrayBasedMapData extends MapData

abstract class ArrayData extends SpecializedGetters with Serializable

case class BadRecordException(record: () ⇒ UTF8String, partialResult: () ⇒ Option[InternalRow], cause: Throwable) extends Exception with Product with Serializable

class CaseInsensitiveMap[T] extends Map[String, T] with Serializable

class GenericArrayData extends ArrayData

abstract class MapData extends Serializable

sealed trait ParseMode extends AnyRef

class QuantileSummaries extends Serializable

class StringKeyHashMap[T] extends AnyRef

Value Members

object ArrayBasedMapData extends Serializable

object ArrayData extends Serializable

object CaseInsensitiveMap extends Serializable

object CompressionCodecs

object DateTimeUtils

object DropMalformedMode extends ParseMode with Product with Serializable

object FailFastMode extends ParseMode with Product with Serializable

object NumberConverter

object ParseMode extends Logging

object PermissiveMode extends ParseMode with Product with Serializable

object QuantileSummaries extends Serializable

object StringKeyHashMap

object StringUtils

object TypeUtils

def benchmark[A](f: ⇒ A): A

def fileToString(file: File, encoding: String = "UTF-8"): String

def quietly[A](f: ⇒ A): A

def quoteIdentifier(name: String): String

def resourceToBytes(resource: String, classLoader: ClassLoader = Utils.getSparkClassLoader): Array[Byte]

def resourceToString(resource: String, encoding: String = "UTF-8", classLoader: ClassLoader = Utils.getSparkClassLoader): String

def sideBySide(left: Seq[String], right: Seq[String]): Seq[String]

def sideBySide(left: String, right: String): Seq[String]

def stackTraceToString(t: Throwable): String

def stringOrNull(a: AnyRef): String

def stringToFile(file: File, str: String): File

def toPrettySQL(e: Expression): String

def usePrettyExpression(e: Expression): Expression

Inherited from AnyRef

Inherited from Any

Ungrouped