Value Members

1.  final def !=(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

2.  final def ##(): Int

   Definition Classes

   AnyRef → Any

3.  final def ==(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

4.  final def asInstanceOf[T0]: T0

   Definition Classes

   Any

5.  def clone(): AnyRef

   Attributes

   protected[lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( ... ) @native()

6.  def conf: SQLConf

   Definition Classes

   SQLConfHelper

7.  final def eq(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

8.  def equals(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

9.  def finalize(): Unit

   Attributes

   protected[lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( classOf[java.lang.Throwable] )

10.  def fromBatchIterator(arrowBatchIter: Iterator[Array[Byte]], schema: StructType, timeZoneId: String, context: TaskContext): Iterator[InternalRow]
11.  final def getClass(): Class[_]

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

12.  def hashCode(): Int

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

13.  def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

    Attributes

    protected

    Definition Classes

    Logging

14.  def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Attributes

    protected

    Definition Classes

    Logging

15.  final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

16.  def isTraceEnabled(): Boolean

    Attributes

    protected

    Definition Classes

    Logging

17.  def log: Logger

    Attributes

    protected

    Definition Classes

    Logging

18.  def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

19.  def logDebug(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

20.  def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

21.  def logError(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

22.  def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

23.  def logInfo(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

24.  def logName: String

    Attributes

    protected

    Definition Classes

    Logging

25.  def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

26.  def logTrace(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

27.  def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

28.  def logWarning(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

29.  final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

30.  final def notify(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

31.  final def notifyAll(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

32.  def slice(schema: StructType, timeZoneId: String, bytes: Array[Byte], start: Int, length: Int): Array[Byte]

    this method is to slice the input Arrow record batch byte array bytes, starting from start and taking length number of elements.

33.  final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes

    AnyRef

34.  def takeAsArrowBatches(collectLimitExec: CollectLimitExec, maxRecordsPerBatch: Long, maxEstimatedBatchSize: Long, timeZoneId: String): Array[Batch]

    Forked from org.apache.spark.sql.execution.SparkPlan#executeTake(), the algorithm can be summarized in the following steps: 1.

    Forked from org.apache.spark.sql.execution.SparkPlan#executeTake(), the algorithm can be summarized in the following steps: 1. If the limit specified in the CollectLimitExec object is 0, the function returns an empty array of batches. 2. Otherwise, execute the child query plan of the CollectLimitExec object to obtain an RDD of data to collect. 3. Use an iterative approach to collect data in batches until the specified limit is reached. In each iteration, it selects a subset of the partitions of the RDD to scan and tries to collect data from them. 4. For each partition subset, we use the runJob method of the Spark context to execute a closure that scans the partition data and converts it to Arrow batches. 5. Check if the collected data reaches the specified limit. If not, it selects another subset of partitions to scan and repeats the process until the limit is reached or all partitions have been scanned. 6. Return an array of all the collected Arrow batches.

    Note that: 1. The returned Arrow batches row count >= limit, if the input df has more than the limit row count 2. We don't implement the takeFromEnd logical

35.  def toBatchIterator(rowIter: Iterator[InternalRow], schema: StructType, maxRecordsPerBatch: Long, maxEstimatedBatchSize: Long, limit: Long, timeZoneId: String): ArrowBatchIterator

    Different from org.apache.spark.sql.execution.arrow.ArrowConverters.toBatchIterator, each output arrow batch contains this batch row count.

36.  def toString(): String

    Definition Classes

    AnyRef → Any

37.  final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

38.  final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

39.  final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... ) @native()