Class

org.apache.spark.sql.execution.datasources

InsertIntoHadoopFsRelationCommand

Related Doc: package datasources

Permalink

case class InsertIntoHadoopFsRelationCommand(outputPath: Path, partitionColumns: Seq[Attribute], bucketSpec: Option[BucketSpec], fileFormat: FileFormat, refreshFunction: () ⇒ Unit, options: Map[String, String], query: LogicalPlan, mode: SaveMode) extends LeafNode with RunnableCommand with Product with Serializable

A command for writing data to a HadoopFsRelation. Supports both overwriting and appending. Writing to dynamic partitions is also supported. Each InsertIntoHadoopFsRelationCommand issues a single write job, and owns a UUID that identifies this job. Each concrete implementation of HadoopFsRelation should use this UUID together with task id to generate unique file path for each task output file. This UUID is passed to executor side via a property named spark.sql.sources.writeJobUUID.

Different writer containers, DefaultWriterContainer and DynamicPartitionWriterContainer are used to write to normal tables and tables with dynamic partitions.

Basic work flow of this command is:

  1. Driver side setup, including output committer initialization and data source specific preparation work for the write job to be issued. 2. Issues a write job consists of one or more executor side tasks, each of which writes all rows within an RDD partition. 3. If no exception is thrown in a task, commits that task, otherwise aborts that task; If any exception is thrown during task commitment, also aborts that task. 4. If all tasks are committed, commit the job, otherwise aborts the job; If any exception is thrown during job commitment, also aborts the job.
Linear Supertypes
Serializable, Serializable, RunnableCommand, Command, LeafNode, LogicalPlan, Logging, QueryPlan[LogicalPlan], TreeNode[LogicalPlan], Product, Equals, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. InsertIntoHadoopFsRelationCommand
  2. Serializable
  3. Serializable
  4. RunnableCommand
  5. Command
  6. LeafNode
  7. LogicalPlan
  8. Logging
  9. QueryPlan
  10. TreeNode
  11. Product
  12. Equals
  13. AnyRef
  14. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new InsertIntoHadoopFsRelationCommand(outputPath: Path, partitionColumns: Seq[Attribute], bucketSpec: Option[BucketSpec], fileFormat: FileFormat, refreshFunction: () ⇒ Unit, options: Map[String, String], query: LogicalPlan, mode: SaveMode)

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  4. lazy val allAttributes: AttributeSeq

    Permalink
    Definition Classes
    QueryPlan

  5. def analyzed: Boolean

    Permalink
    Definition Classes
    LogicalPlan

  6. def apply(number: Int): LogicalPlan

    Permalink
    Definition Classes
    TreeNode

  7. def argString: String

    Permalink
    Definition Classes
    TreeNode

  8. def asCode: String

    Permalink
    Definition Classes
    TreeNode

  9. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  10. val bucketSpec: Option[BucketSpec]

    Permalink

  11. lazy val canonicalized: LogicalPlan

    Permalink
    Definition Classes
    LogicalPlan → QueryPlan

  12. final def children: Seq[LogicalPlan]

    Permalink
    Definition Classes
    Command → LeafNode → TreeNode

  13. def childrenResolved: Boolean

    Permalink
    Definition Classes
    LogicalPlan

  14. lazy val cleanArgs: Seq[Any]

    Permalink
    Attributes
    protected
    Definition Classes
    QueryPlan

  15. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  16. def collect[B](pf: PartialFunction[LogicalPlan, B]): Seq[B]

    Permalink
    Definition Classes
    TreeNode

  17. def collectFirst[B](pf: PartialFunction[LogicalPlan, B]): Option[B]

    Permalink
    Definition Classes
    TreeNode

  18. def collectLeaves(): Seq[LogicalPlan]

    Permalink
    Definition Classes
    TreeNode

  19. lazy val constraints: ExpressionSet

    Permalink
    Definition Classes
    QueryPlan

  20. lazy val containsChild: Set[TreeNode[_]]

    Permalink
    Definition Classes
    TreeNode

  21. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  22. final def expressions: Seq[Expression]

    Permalink
    Definition Classes
    QueryPlan

  23. def fastEquals(other: TreeNode[_]): Boolean

    Permalink
    Definition Classes
    TreeNode

  24. val fileFormat: FileFormat

    Permalink

  25. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  26. def find(f: (LogicalPlan) ⇒ Boolean): Option[LogicalPlan]

    Permalink
    Definition Classes
    TreeNode

  27. def flatMap[A](f: (LogicalPlan) ⇒ TraversableOnce[A]): Seq[A]

    Permalink
    Definition Classes
    TreeNode

  28. def foreach(f: (LogicalPlan) ⇒ Unit): Unit

    Permalink
    Definition Classes
    TreeNode

  29. def foreachUp(f: (LogicalPlan) ⇒ Unit): Unit

    Permalink
    Definition Classes
    TreeNode

  30. def generateTreeString(depth: Int, lastChildren: Seq[Boolean], builder: StringBuilder, verbose: Boolean, prefix: String): StringBuilder

    Permalink
    Definition Classes
    TreeNode

  31. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  32. def getNodeNumbered(number: MutableInt): LogicalPlan

    Permalink
    Attributes
    protected
    Definition Classes
    TreeNode

  33. def getRelevantConstraints(constraints: Set[Expression]): Set[Expression]

    Permalink
    Attributes
    protected
    Definition Classes
    QueryPlan

  34. def hashCode(): Int

    Permalink
    Definition Classes
    TreeNode → AnyRef → Any

  35. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  36. def innerChildren: Seq[LogicalPlan]

    Permalink
    Attributes
    protected
    Definition Classes
    InsertIntoHadoopFsRelationCommand → QueryPlan → TreeNode

  37. def inputSet: AttributeSet

    Permalink
    Definition Classes
    QueryPlan

  38. final def isDebugEnabled: Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  39. final def isInfoEnabled: Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  40. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  41. def isStreaming: Boolean

    Permalink
    Definition Classes
    LogicalPlan

  42. final def isTraceEnabled: Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  43. def jsonFields: List[JField]

    Permalink
    Attributes
    protected
    Definition Classes
    TreeNode

  44. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  45. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  46. def logDebug(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  47. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  48. def logError(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  49. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  50. def logInfo(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  51. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  52. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  53. def logTrace(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  54. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  55. def logWarning(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  56. def makeCopy(newArgs: Array[AnyRef]): LogicalPlan

    Permalink
    Definition Classes
    TreeNode

  57. def map[A](f: (LogicalPlan) ⇒ A): Seq[A]

    Permalink
    Definition Classes
    TreeNode

  58. def mapChildren(f: (LogicalPlan) ⇒ LogicalPlan): LogicalPlan

    Permalink
    Definition Classes
    TreeNode

  59. def mapProductIterator[B](f: (Any) ⇒ B)(implicit arg0: ClassTag[B]): Array[B]

    Permalink
    Attributes
    protected
    Definition Classes
    TreeNode

  60. def maxRows: Option[Long]

    Permalink
    Definition Classes
    LogicalPlan

  61. def missingInput: AttributeSet

    Permalink
    Definition Classes
    QueryPlan

  62. val mode: SaveMode

    Permalink

  63. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  64. def nodeName: String

    Permalink
    Definition Classes
    TreeNode

  65. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  66. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  67. def numberedTreeString: String

    Permalink
    Definition Classes
    TreeNode

  68. val options: Map[String, String]

    Permalink

  69. val origin: Origin

    Permalink
    Definition Classes
    TreeNode

  70. def otherCopyArgs: Seq[AnyRef]

    Permalink
    Attributes
    protected
    Definition Classes
    TreeNode

  71. def output: Seq[Attribute]

    Permalink
    Definition Classes
    Command → QueryPlan

  72. val outputPath: Path

    Permalink

  73. def outputSet: AttributeSet

    Permalink
    Definition Classes
    QueryPlan

  74. val partitionColumns: Seq[Attribute]

    Permalink

  75. def prettyJson: String

    Permalink
    Definition Classes
    TreeNode

  76. def printSchema(): Unit

    Permalink
    Definition Classes
    QueryPlan

  77. def producedAttributes: AttributeSet

    Permalink
    Definition Classes
    LeafNode → QueryPlan

  78. val query: LogicalPlan

    Permalink

  79. def references: AttributeSet

    Permalink
    Definition Classes
    QueryPlan

  80. def refresh(): Unit

    Permalink
    Definition Classes
    LogicalPlan

  81. val refreshFunction: () ⇒ Unit

    Permalink

  82. def resolve(nameParts: Seq[String], input: Seq[Attribute], resolver: Resolver): Option[NamedExpression]

    Permalink
    Attributes
    protected
    Definition Classes
    LogicalPlan

  83. def resolve(nameParts: Seq[String], resolver: Resolver): Option[NamedExpression]

    Permalink
    Definition Classes
    LogicalPlan

  84. def resolve(schema: StructType, resolver: Resolver): Seq[Attribute]

    Permalink
    Definition Classes
    LogicalPlan

  85. def resolveChildren(nameParts: Seq[String], resolver: Resolver): Option[NamedExpression]

    Permalink
    Definition Classes
    LogicalPlan

  86. def resolveExpressions(r: PartialFunction[Expression, Expression]): LogicalPlan

    Permalink
    Definition Classes
    LogicalPlan

  87. def resolveOperators(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan

    Permalink
    Definition Classes
    LogicalPlan

  88. def resolveQuoted(name: String, resolver: Resolver): Option[NamedExpression]

    Permalink
    Definition Classes
    LogicalPlan

  89. lazy val resolved: Boolean

    Permalink
    Definition Classes
    LogicalPlan

  90. def run(sparkSession: SparkSession): Seq[Row]

    Permalink
    Definition Classes
    InsertIntoHadoopFsRelationCommand → RunnableCommand

  91. def sameResult(plan: LogicalPlan): Boolean

    Permalink
    Definition Classes
    QueryPlan

  92. lazy val schema: StructType

    Permalink
    Definition Classes
    QueryPlan

  93. def schemaString: String

    Permalink
    Definition Classes
    QueryPlan

  94. def simpleString: String

    Permalink
    Definition Classes
    QueryPlan → TreeNode

  95. def statePrefix: String

    Permalink
    Attributes
    protected
    Definition Classes
    LogicalPlan → QueryPlan

  96. def statistics: Statistics

    Permalink
    Definition Classes
    LogicalPlan

  97. def stringArgs: Iterator[Any]

    Permalink
    Attributes
    protected
    Definition Classes
    TreeNode

  98. def subqueries: Seq[LogicalPlan]

    Permalink
    Definition Classes
    QueryPlan

  99. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  100. def toJSON: String

    Permalink
    Definition Classes
    TreeNode

  101. def toString(): String

    Permalink
    Definition Classes
    TreeNode → AnyRef → Any

  102. def transform(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan

    Permalink
    Definition Classes
    TreeNode

  103. def transformAllExpressions(rule: PartialFunction[Expression, Expression]): InsertIntoHadoopFsRelationCommand.this.type

    Permalink
    Definition Classes
    QueryPlan

  104. def transformChildren(rule: PartialFunction[LogicalPlan, LogicalPlan], nextOperation: (LogicalPlan, PartialFunction[LogicalPlan, LogicalPlan]) ⇒ LogicalPlan): LogicalPlan

    Permalink
    Attributes
    protected
    Definition Classes
    TreeNode

  105. def transformDown(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan

    Permalink
    Definition Classes
    TreeNode

  106. def transformExpressions(rule: PartialFunction[Expression, Expression]): InsertIntoHadoopFsRelationCommand.this.type

    Permalink
    Definition Classes
    QueryPlan

  107. def transformExpressionsDown(rule: PartialFunction[Expression, Expression]): InsertIntoHadoopFsRelationCommand.this.type

    Permalink
    Definition Classes
    QueryPlan

  108. def transformExpressionsUp(rule: PartialFunction[Expression, Expression]): InsertIntoHadoopFsRelationCommand.this.type

    Permalink
    Definition Classes
    QueryPlan

  109. def transformUp(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan

    Permalink
    Definition Classes
    TreeNode

  110. def treeString(verbose: Boolean): String

    Permalink
    Definition Classes
    TreeNode

  111. def treeString: String

    Permalink
    Definition Classes
    TreeNode

  112. def validConstraints: Set[Expression]

    Permalink
    Attributes
    protected
    Definition Classes
    QueryPlan

  113. def verboseString: String

    Permalink
    Definition Classes
    QueryPlan → TreeNode

  114. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  115. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  116. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  117. def withNewChildren(newChildren: Seq[LogicalPlan]): LogicalPlan

    Permalink
    Definition Classes
    TreeNode

Inherited from Serializable

Inherited from Serializable

Inherited from RunnableCommand

Inherited from Command

Inherited from LeafNode

Inherited from LogicalPlan

Inherited from Logging

Inherited from QueryPlan[LogicalPlan]

Inherited from TreeNode[LogicalPlan]

Inherited from Product

Inherited from Equals

Inherited from AnyRef

Inherited from Any

Ungrouped