org.apache.spark.sql.hive

HiveContext

class HiveContext extends SQLContext

An instance of the Spark SQL execution engine that integrates with data stored in Hive. Configuration for Hive is read from hive-site.xml on the classpath.

Self Type
HiveContext
Linear Supertypes
SQLContext, Serializable, Serializable, UDFRegistration, ExpressionConversions, CacheManager, SQLConf, Logging, AnyRef, Any
Known Subclasses
LocalHiveContext, TestHive, TestHiveContext
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. HiveContext
  2. SQLContext
  3. Serializable
  4. Serializable
  5. UDFRegistration
  6. ExpressionConversions
  7. CacheManager
  8. SQLConf
  9. Logging
  10. AnyRef
  11. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new HiveContext(sc: SparkContext)

Type Members

  1. implicit class DslAttribute extends AnyRef

    Definition Classes
    ExpressionConversions

  2. implicit class DslExpression extends ImplicitOperators

    Definition Classes
    ExpressionConversions

  3. implicit class DslString extends ImplicitOperators

    Definition Classes
    ExpressionConversions

  4. implicit class DslSymbol extends ImplicitAttribute

    Definition Classes
    ExpressionConversions

  5. abstract class ImplicitAttribute extends ImplicitOperators

    Definition Classes
    ExpressionConversions

  6. abstract class QueryExecution extends HiveContext.QueryExecution

    Extends QueryExecution with hive specific features.

    Extends QueryExecution with hive specific features.

    Attributes
    protected[org.apache.spark.sql]

  7. class SparkPlanner extends SparkStrategies

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext

Value Members

  1. final def !=(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  4. def abs(e: Expression): Abs

    Definition Classes
    ExpressionConversions

  5. def analyze(tableName: String): Unit

    Analyzes the given table in the current database to generate statistics, which will be used in query optimizations.

    Analyzes the given table in the current database to generate statistics, which will be used in query optimizations.

    Right now, it only supports Hive tables and it only updates the size of a Hive table in the Hive metastore.

  6. lazy val analyzer: Analyzer { val extendedRules: List[org.apache.spark.sql.catalyst.rules.Rule[org.apache.spark.sql.catalyst.plans.logical.LogicalPlan]] }

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    HiveContext → SQLContext

  7. def applySchema(rowRDD: RDD[Row], schema: StructType): SchemaRDD

    Definition Classes
    SQLContext
    Annotations
    @DeveloperApi()

  8. def approxCountDistinct(e: Expression, rsd: Double): ApproxCountDistinct

    Definition Classes
    ExpressionConversions

  9. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  10. def avg(e: Expression): Average

    Definition Classes
    ExpressionConversions

  11. implicit def baseRelationToSchemaRDD(baseRelation: BaseRelation): SchemaRDD

    Definition Classes
    SQLContext

  12. implicit def bigDecimalToLiteral(d: BigDecimal): Literal

    Definition Classes
    ExpressionConversions

  13. implicit def binaryToLiteral(a: Array[Byte]): Literal

    Definition Classes
    ExpressionConversions

  14. implicit def booleanToLiteral(b: Boolean): Literal

    Definition Classes
    ExpressionConversions

  15. implicit def byteToLiteral(b: Byte): Literal

    Definition Classes
    ExpressionConversions

  16. def cacheTable(tableName: String): Unit

    Definition Classes
    CacheManager

  17. lazy val catalog: HiveMetastoreCatalog with OverrideCatalog

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    HiveContext → SQLContext

  18. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  19. def count(e: Expression): Count

    Definition Classes
    ExpressionConversions

  20. def countDistinct(e: Expression*): CountDistinct

    Definition Classes
    ExpressionConversions

  21. def createParquetFile[A <: Product](path: String, allowExisting: Boolean, conf: Configuration)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A]): SchemaRDD

    Definition Classes
    SQLContext
    Annotations
    @Experimental()

  22. implicit def createSchemaRDD[A <: Product](rdd: RDD[A])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A]): SchemaRDD

    Definition Classes
    SQLContext

  23. def createTable[A <: Product](tableName: String, allowExisting: Boolean = true)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A]): Unit

    Creates a table using the schema of the given class.

    Creates a table using the schema of the given class.

    A

    A case class that is used to describe the schema of the table to be created.

    tableName

    The name of the table to create.

    allowExisting

    When false, an exception will be thrown if the table already exists.

  24. implicit def dateToLiteral(d: Date): Literal

    Definition Classes
    ExpressionConversions

  25. val ddlParser: DDLParser

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext

  26. implicit def decimalToLiteral(d: Decimal): Literal

    Definition Classes
    ExpressionConversions

  27. implicit def doubleToLiteral(d: Double): Literal

    Definition Classes
    ExpressionConversions

  28. def dropTempTable(tableName: String): Unit

    Definition Classes
    SQLContext

  29. lazy val emptyResult: RDD[Row]

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext

  30. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  31. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  32. def executePlan(plan: LogicalPlan): QueryExecution

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    HiveContext → SQLContext

  33. def executeSql(sql: String): HiveContext.QueryExecution

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext

  34. var extraStrategies: Seq[Strategy]

    Definition Classes
    SQLContext

  35. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  36. def first(e: Expression): First

    Definition Classes
    ExpressionConversions

  37. implicit def floatToLiteral(f: Float): Literal

    Definition Classes
    ExpressionConversions

  38. lazy val functionRegistry: HiveFunctionRegistry with OverrideFunctionRegistry

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    HiveContext → SQLContext

  39. def getAllConfs: Map[String, String]

    Definition Classes
    SQLConf

  40. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  41. def getConf(key: String, defaultValue: String): String

    Definition Classes
    SQLConf

  42. def getConf(key: String): String

    Definition Classes
    SQLConf

  43. def hashCode(): Int

    Definition Classes
    AnyRef → Any

  44. val hivePlanner: SparkPlanner with HiveStrategies

  45. lazy val hiveconf: HiveConf

    SQLConf and HiveConf contracts:

    SQLConf and HiveConf contracts:

    1. reuse existing started SessionState if any 2. when the Hive session is first initialized, params in HiveConf will get picked up by the SQLConf. Additionally, any properties set by set() or a SET command inside sql() will be set in the SQLConf *as well as* in the HiveConf.

    Attributes
    protected[org.apache.spark.sql.hive]

  46. implicit def intToLiteral(i: Int): Literal

    Definition Classes
    ExpressionConversions

  47. def isCached(tableName: String): Boolean

    Definition Classes
    CacheManager

  48. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  49. def isTraceEnabled(): Boolean

    Attributes
    protected
    Definition Classes
    Logging

  50. def jsonFile(path: String, samplingRatio: Double): SchemaRDD

    Definition Classes
    SQLContext
    Annotations
    @Experimental()

  51. def jsonFile(path: String, schema: StructType): SchemaRDD

    Definition Classes
    SQLContext
    Annotations
    @Experimental()

  52. def jsonFile(path: String): SchemaRDD

    Definition Classes
    SQLContext

  53. def jsonRDD(json: RDD[String], samplingRatio: Double): SchemaRDD

    Definition Classes
    SQLContext
    Annotations
    @Experimental()

  54. def jsonRDD(json: RDD[String], schema: StructType): SchemaRDD

    Definition Classes
    SQLContext
    Annotations
    @Experimental()

  55. def jsonRDD(json: RDD[String]): SchemaRDD

    Definition Classes
    SQLContext

  56. def last(e: Expression): Last

    Definition Classes
    ExpressionConversions

  57. def log: Logger

    Attributes
    protected
    Definition Classes
    Logging

  58. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  59. def logDebug(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  60. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  61. def logError(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  62. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  63. def logInfo(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  64. def logName: String

    Attributes
    protected
    Definition Classes
    Logging

  65. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  66. def logTrace(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  67. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  68. def logWarning(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  69. implicit def logicalPlanToSparkQuery(plan: LogicalPlan): SchemaRDD

    Definition Classes
    SQLContext
    Annotations
    @DeveloperApi()

  70. implicit def longToLiteral(l: Long): Literal

    Definition Classes
    ExpressionConversions

  71. def lower(e: Expression): Lower

    Definition Classes
    ExpressionConversions

  72. def max(e: Expression): Max

    Definition Classes
    ExpressionConversions

  73. def min(e: Expression): Min

    Definition Classes
    ExpressionConversions

  74. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  75. final def notify(): Unit

    Definition Classes
    AnyRef

  76. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  77. lazy val optimizer: Optimizer

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext

  78. lazy val outputBuffer: OutputStream { ... /* 4 definitions in type refinement */ }

    Attributes
    protected

  79. def parquetFile(path: String): SchemaRDD

    Definition Classes
    SQLContext

  80. def parseSql(sql: String): LogicalPlan

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext

  81. val planner: SparkPlanner with HiveStrategies

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    HiveContext → SQLContext

  82. val prepareForExecution: RuleExecutor[SparkPlan] { val batches: List[this.Batch] }

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext

  83. def registerFunction[T](name: String, func: Function22[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration

  84. def registerFunction[T](name: String, func: Function21[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration

  85. def registerFunction[T](name: String, func: Function20[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration

  86. def registerFunction[T](name: String, func: Function19[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration

  87. def registerFunction[T](name: String, func: Function18[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration

  88. def registerFunction[T](name: String, func: Function17[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration

  89. def registerFunction[T](name: String, func: Function16[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration

  90. def registerFunction[T](name: String, func: Function15[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration

  91. def registerFunction[T](name: String, func: Function14[_, _, _, _, _, _, _, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration

  92. def registerFunction[T](name: String, func: Function13[_, _, _, _, _, _, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration

  93. def registerFunction[T](name: String, func: Function12[_, _, _, _, _, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration

  94. def registerFunction[T](name: String, func: Function11[_, _, _, _, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration

  95. def registerFunction[T](name: String, func: Function10[_, _, _, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration

  96. def registerFunction[T](name: String, func: Function9[_, _, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration

  97. def registerFunction[T](name: String, func: Function8[_, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration

  98. def registerFunction[T](name: String, func: Function7[_, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration

  99. def registerFunction[T](name: String, func: Function6[_, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration

  100. def registerFunction[T](name: String, func: Function5[_, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration

  101. def registerFunction[T](name: String, func: Function4[_, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration

  102. def registerFunction[T](name: String, func: Function3[_, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration

  103. def registerFunction[T](name: String, func: Function2[_, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration

  104. def registerFunction[T](name: String, func: Function1[_, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration

  105. def registerRDDAsTable(rdd: SchemaRDD, tableName: String): Unit

    Definition Classes
    SQLContext

  106. def runHive(cmd: String, maxRows: Int = 1000): Seq[String]

    Execute the command using Hive and return the results as a sequence.

    Execute the command using Hive and return the results as a sequence. Each element in the sequence is one row.

    Attributes
    protected

  107. def runSqlHive(sql: String): Seq[String]

    Runs the specified SQL query using Hive.

    Runs the specified SQL query using Hive.

    Attributes
    protected[org.apache.spark.sql]

  108. lazy val sessionState: SessionState

    SQLConf and HiveConf contracts:

    SQLConf and HiveConf contracts:

    1. reuse existing started SessionState if any 2. when the Hive session is first initialized, params in HiveConf will get picked up by the SQLConf. Additionally, any properties set by set() or a SET command inside sql() will be set in the SQLConf *as well as* in the HiveConf.

    Attributes
    protected[org.apache.spark.sql.hive]

  109. def setConf(key: String, value: String): Unit

    Definition Classes
    HiveContext → SQLConf

  110. def setConf(props: Properties): Unit

    Definition Classes
    SQLConf

  111. val settings: Map[String, String]

    Attributes
    protected[org.apache.spark]
    Definition Classes
    SQLConf

  112. implicit def shortToLiteral(s: Short): Literal

    Definition Classes
    ExpressionConversions

  113. val sparkContext: SparkContext

    Definition Classes
    SQLContext

  114. def sql(sqlText: String): SchemaRDD

    Definition Classes
    HiveContext → SQLContext

  115. val sqlParser: SparkSQLParser

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext

  116. def sqrt(e: Expression): Sqrt

    Definition Classes
    ExpressionConversions

  117. implicit def stringToLiteral(s: String): Literal

    Definition Classes
    ExpressionConversions

  118. def sum(e: Expression): Sum

    Definition Classes
    ExpressionConversions

  119. def sumDistinct(e: Expression): SumDistinct

    Definition Classes
    ExpressionConversions

  120. implicit def symbolToUnresolvedAttribute(s: Symbol): UnresolvedAttribute

    Definition Classes
    ExpressionConversions

  121. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  122. def table(tableName: String): SchemaRDD

    Definition Classes
    SQLContext

  123. implicit def timestampToLiteral(t: Timestamp): Literal

    Definition Classes
    ExpressionConversions

  124. def toString(): String

    Definition Classes
    AnyRef → Any

  125. def uncacheTable(tableName: String): Unit

    Definition Classes
    CacheManager

  126. def upper(e: Expression): Upper

    Definition Classes
    ExpressionConversions

  127. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  128. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  129. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def hiveql(hqlQuery: String): SchemaRDD

    Annotations
    @deprecated
    Deprecated

    (Since version 1.1)

  2. def hql(hqlQuery: String): SchemaRDD

    Annotations
    @deprecated
    Deprecated

    (Since version 1.1)

Inherited from SQLContext

Inherited from Serializable

Inherited from Serializable

Inherited from UDFRegistration

Inherited from ExpressionConversions

Inherited from CacheManager

Inherited from SQLConf

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped