Intersection

Value Members

1. final def !=(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

2. final def !=(arg0: Any): Boolean

   Definition Classes

   Any

3. final def ##(): Int

   Definition Classes

   AnyRef → Any

4. def -->(targets: ConnectionTarget[DataFrame, SparkRuntime]*): Unit

   Definition Classes

   ConnectionSource

5. def -->(tgt: MultiInputStep[DataFrame, SparkRuntime]): tgt.type

   Definition Classes

   ConnectionSource

6. def -->(tgt: ConnectionTarget[DataFrame, SparkRuntime]): tgt.type

   Definition Classes

   ConnectionSource

7. final def ==(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

8. final def ==(arg0: Any): Boolean

   Definition Classes

   Any

9. def addStepListener(listener: StepListener[DataFrame, SparkRuntime]): Unit

   Registers a new listener.

   Registers a new listener.

   Definition Classes

   Step

10. val allInputsRequired: Boolean

    Specifies if the step should throw an error if one of the inputs is not connected.

    Specifies if the step should throw an error if one of the inputs is not connected.

    Definition Classes

    Intersection → Step

11. final def asInstanceOf[T0]: T0

    Definition Classes

    Any

12. def buildSchema(index: Int)(implicit runtime: SparkRuntime): StructType

    Returns the output schema of the step at a given index by calling output() in preview mode and retrieving the schema from the DataFrame.

    Returns the output schema of the step at a given index by calling output() in preview mode and retrieving the schema from the DataFrame. Since the default implementation of this method calls output(), invoking it from output() directly or indirectly will result in infinite cycle.

    Attributes

    protected

    Definition Classes

    Intersection → FrameStep

13. def clone(): AnyRef

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

14. def compute(args: IndexedSeq[DataFrame])(implicit runtime: SparkRuntime): DataFrame

    Attributes

    protected

    Definition Classes

    Intersection → Merger

15. def compute(index: Int)(implicit runtime: SparkRuntime): DataFrame

    Computes a step output value at the specified index.

    Computes a step output value at the specified index. This method is invoked from output() and can safely throw any exception, which will be wrapped into ExecutionException.

    index

    the output value index.

    Attributes

    protected

    Definition Classes

    Merger → Step

16. def ctx(implicit runtime: SparkRuntime): SQLContext

    Returns the implicit SQLContext.

    Returns the implicit SQLContext.

    Attributes

    protected

    Definition Classes

    FrameStep

17. final def eq(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

18. final def evaluate(implicit runtime: SparkRuntime): IndexedSeq[DataFrame]

    Evaluates all step's outputs and returns a list of results.

    Evaluates all step's outputs and returns a list of results.

    Definition Classes

    Step

    Annotations

    @throws( classOf[ExecutionException] )

19. def finalize(): Unit

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

20. final def getClass(): Class[_]

    Definition Classes

    AnyRef → Any

21. val in: LazyArray[InPort]

    Definition Classes

    MultiInputStep

22. val index: Int

    Definition Classes

    SingleOutputStep → ConnectionSource

23. val inputCount: Int

    The maximum number of input ports.

    The maximum number of input ports.

    Definition Classes

    FrameMerger → Step

24. def inputs(implicit runtime: SparkRuntime): IndexedSeq[DataFrame]

    Definition Classes

    MultiInputStep

25. final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

26. final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

27. final def notify(): Unit

    Definition Classes

    AnyRef

28. final def notifyAll(): Unit

    Definition Classes

    AnyRef

29. def optLimit(df: DataFrame, preview: Boolean): DataFrame

    Takes one row from the data frame is preview is true.

    Takes one row from the data frame is preview is true.

    Attributes

    protected

    Definition Classes

    FrameStep

30. final def outSchema(implicit runtime: SparkRuntime): StructType

    Returns the output schema of the step at index 0.

    Returns the output schema of the step at index 0.

    Definition Classes

    FrameStep

31. final def outSchema(index: Int)(implicit runtime: SparkRuntime): StructType

    Returns the output schema of the step at a given index.

    Returns the output schema of the step at a given index.

    Definition Classes

    FrameStep

32. def outbounds(index: Int): SingleOutputStep[DataFrame, SparkRuntime]

    Definition Classes

    SingleOutputStep

33. final def output(implicit runtime: SparkRuntime): DataFrame

    Shortcut for output(0).

    Shortcut for output(0). Computes a step output at index 0.

    Definition Classes

    Step

    Annotations

    @throws( classOf[ExecutionException] )

    Exceptions thrown

    ExecutionException

    in case of an error, or if the step is not connected.

34. final def output(index: Int)(implicit runtime: SparkRuntime): DataFrame

    Computes a step output value at the specified index.

    Computes a step output value at the specified index.

    index

    the output value index.

    Definition Classes

    Step

    Annotations

    @throws( classOf[ExecutionException] )

    Exceptions thrown

    ExecutionException

    in case of an error, or if the step is not connected.

35. val outputCount: Int

    The number of output ports.

    The number of output ports.

    Definition Classes

    SingleOutputStep → Step

36. def removeStepListener(listener: StepListener[DataFrame, SparkRuntime]): Unit

    Unregisters a listener.

    Unregisters a listener.

    Definition Classes

    Step

37. def resetCache(predecessors: Boolean, descendants: Boolean): Unit

    Clears the cache of this step and optionally that of its predecessors and descendants.

    Clears the cache of this step and optionally that of its predecessors and descendants.

    Definition Classes

    Step

38. def sc(implicit runtime: SparkRuntime): SparkContext

    Returns the implicit SparkContext.

    Returns the implicit SparkContext.

    Attributes

    protected

    Definition Classes

    FrameStep

39. val step: SingleOutputStep[DataFrame, SparkRuntime]

    Definition Classes

    SingleOutputStep → ConnectionSource

40. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes

    AnyRef

41. def to(targets: ConnectionTarget[DataFrame, SparkRuntime]*): Unit

    Definition Classes

    ConnectionSource

42. def to(tgt: MultiInputStep[DataFrame, SparkRuntime]): tgt.type

    Definition Classes

    ConnectionSource

43. def to(tgt: ConnectionTarget[DataFrame, SparkRuntime]): tgt.type

    Definition Classes

    ConnectionSource

44. def toJson: JValue

    Definition Classes

    Intersection → JsonExport

45. def toXml: Elem

    Definition Classes

    Intersection → XmlExport

46. def value(implicit runtime: SparkRuntime): DataFrame

    Definition Classes

    SingleOutputStep → ConnectionSource

47. final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

48. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

49. final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

50. final def wrap[U](body: ⇒ U): U

    Wraps exceptions into ExecutionException instances.

    Wraps exceptions into ExecutionException instances.

    Attributes

    protected

    Definition Classes

    AbstractStep