object PythonForeachWriter extends Serializable
- Alphabetic
- By Inheritance
- PythonForeachWriter
- Serializable
- Serializable
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Type Members
-
class
UnsafeRowBuffer extends Logging
A buffer that is designed for the sole purpose of buffering UnsafeRows in PythonForeachWriter.
A buffer that is designed for the sole purpose of buffering UnsafeRows in PythonForeachWriter. It is designed to be used with only 1 writer thread (i.e. JVM task thread) and only 1 reader thread (i.e. PythonRunner writing thread that reads from the buffer and writes to the Python worker stdin). Adds to the buffer are non-blocking, and reads through the buffer's iterator are blocking, that is, it blocks until new data is available or all data has been added.
Internally, it uses a HybridRowQueue to buffer the rows in a practically unlimited queue across memory and local disk. However, HybridRowQueue is designed to be used only with EvalPythonExec where the reader is always behind the writer, that is, the reader does not try to read n+1 rows if the writer has only written n rows at any point of time. This assumption is not true for PythonForeachWriter where rows may be added at a different rate as they are consumed by the python worker. Hence, to maintain the invariant of the reader being behind the writer while using HybridRowQueue, the buffer does the following - Keeps a count of the rows in the HybridRowQueue - Blocks the buffer's consuming iterator when the count is 0 so that the reader does not try to read more rows than what has been written.
The implementation of the blocking iterator (ReentrantLock, Condition, etc.) has been borrowed from that of ArrayBlockingQueue.
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()