Packages

o

org.apache.spark.sql.execution.arrow

KyuubiArrowConverters

object KyuubiArrowConverters extends SQLConfHelper with Logging

Linear Supertypes
Logging, SQLConfHelper, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. KyuubiArrowConverters
  2. Logging
  3. SQLConfHelper
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. type Batch = (Array[Byte], Long)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. def conf: SQLConf
    Definition Classes
    SQLConfHelper
  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  9. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  10. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  13. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  14. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  15. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  16. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  17. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  18. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  19. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  20. def logError(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  21. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  22. def logInfo(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  23. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  24. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  25. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  26. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  27. def logWarning(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  28. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  29. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  30. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  31. def slice(schema: StructType, timeZoneId: String, bytes: Array[Byte], start: Int, length: Int): Array[Byte]

    this method is to slice the input Arrow record batch byte array bytes, starting from start and taking length number of elements.

  32. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  33. def takeAsArrowBatches(collectLimitExec: CollectLimitExec, maxRecordsPerBatch: Long, maxEstimatedBatchSize: Long, timeZoneId: String): Array[Batch]

    Forked from org.apache.spark.sql.execution.SparkPlan#executeTake(), the algorithm can be summarized in the following steps: 1.

    Forked from org.apache.spark.sql.execution.SparkPlan#executeTake(), the algorithm can be summarized in the following steps: 1. If the limit specified in the CollectLimitExec object is 0, the function returns an empty array of batches. 2. Otherwise, execute the child query plan of the CollectLimitExec object to obtain an RDD of data to collect. 3. Use an iterative approach to collect data in batches until the specified limit is reached. In each iteration, it selects a subset of the partitions of the RDD to scan and tries to collect data from them. 4. For each partition subset, we use the runJob method of the Spark context to execute a closure that scans the partition data and converts it to Arrow batches. 5. Check if the collected data reaches the specified limit. If not, it selects another subset of partitions to scan and repeats the process until the limit is reached or all partitions have been scanned. 6. Return an array of all the collected Arrow batches.

    Note that: 1. The returned Arrow batches row count >= limit, if the input df has more than the limit row count 2. We don't implement the takeFromEnd logical

  34. def toBatchIterator(rowIter: Iterator[InternalRow], schema: StructType, maxRecordsPerBatch: Long, maxEstimatedBatchSize: Long, limit: Long, timeZoneId: String): ArrowBatchIterator

    Different from org.apache.spark.sql.execution.arrow.ArrowConverters.toBatchIterator, each output arrow batch contains this batch row count.

  35. def toString(): String
    Definition Classes
    AnyRef → Any
  36. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  37. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  38. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Logging

Inherited from SQLConfHelper

Inherited from AnyRef

Inherited from Any

Ungrouped