AvroInOut

Value Members

1. final def !=(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

2. final def ##(): Int

   

   Definition Classes

   AnyRef → Any

3. final def ==(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

4. final def asInstanceOf[T0]: T0

   

   Definition Classes

   Any

5. def clone(): AnyRef

   

   Attributes

   protected[java.lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( ... )

6. final def eq(arg0: AnyRef): Boolean

   

   Definition Classes

   AnyRef

7. def equals(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

8. def finalize(): Unit

   

   Attributes

   protected[java.lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( classOf[java.lang.Throwable] )

9. final def getClass(): Class[_]

   

   Definition Classes

   AnyRef → Any

10. def hashCode(): Int

    

    Definition Classes

    AnyRef → Any

11. final def isInstanceOf[T0]: Boolean

    

    Definition Classes

    Any

12. final def ne(arg0: AnyRef): Boolean

    

    Definition Classes

    AnyRef

13. final def notify(): Unit

    

    Definition Classes

    AnyRef

14. final def notifyAll(): Unit

    

    Definition Classes

    AnyRef

15. def read[T <: GenericRecord](path: String, withCount: Boolean = false, maxPartitionCount: Int = 200, deepCopy: Boolean = false, persist: Boolean = true)(implicit sc: SparkSession, ct: ClassTag[T]): Option[RDD[T]]

    

    This method reads avro records stored in directory/tablename/date into an RDD of generic/specific records and then repartition to create copy rather than explicit deep copy

    This method reads avro records stored in directory/tablename/date into an RDD of generic/specific records and then repartition to create copy rather than explicit deep copy

    T

    Generic/specific record type that records that should read as.

    path

    Full path to where the avro records are stored

    withCount

    Whether or not to use a counter accumulator

    maxPartitionCount

    max number of partitions

    deepCopy

    Whether or not to deep copy the avro records instead of repartitioning the RDD (SPARK-1018)

    persist

    Whether or not to persist the rdd

    sc

    Spark session

    ct

    data class tag

16. def readPathSeq[T <: GenericRecord](path: String, withCount: Boolean = false, maxPartitionCount: Int = 200, deepCopy: Boolean = false, persist: Boolean = true)(implicit sc: SparkSession, ct: ClassTag[T]): RDD[T]

    

    This method reads avro records stored in directory/tablename/date into an RDD of specific records.

    This method reads avro records stored in directory/tablename/date into an RDD of specific records. Allows user to pass in path of form "place1,place2,place3" and then repartition to create copy rather than explicit deep copy

    T

    Specific record type that records that should read as.

    path

    Full path to where the avro records are stored

    withCount

    Whether or not to use a counter accumulator

    maxPartitionCount

    max number of partitions

    deepCopy

    Whether or not to deep copy the avro records instead of repartitioning the RDD (SPARK-1018)

    persist

    Whether or not to persist the rdd

    sc

    Spark session

    ct

    data class tag

17. final def synchronized[T0](arg0: ⇒ T0): T0

    

    Definition Classes

    AnyRef

18. def toString(): String

    

    Definition Classes

    AnyRef → Any

19. final def wait(): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

20. final def wait(arg0: Long, arg1: Int): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

21. final def wait(arg0: Long): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )