FieldValidation

Value Members

1. final def !=(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

2. final def ##(): Int

   

   Definition Classes

   AnyRef → Any

3. final def ==(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

4. final val CARDINALITIES: Array[String]

   

5. final def asInstanceOf[T0]: T0

   

   Definition Classes

   Any

6. def clone(): AnyRef

   

   Attributes

   protected[java.lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( ... )

7. final def eq(arg0: AnyRef): Boolean

   

   Definition Classes

   AnyRef

8. def equals(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

9. final val fieldNames: Array[String]

   

10. def finalize(): Unit

    

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

11. final def getClass(): Class[_]

    

    Definition Classes

    AnyRef → Any

12. def hashCode(): Int

    

    Definition Classes

    AnyRef → Any

13. final def isInstanceOf[T0]: Boolean

    

    Definition Classes

    Any

14. final def ne(arg0: AnyRef): Boolean

    

    Definition Classes

    AnyRef

15. final def notify(): Unit

    

    Definition Classes

    AnyRef

16. final def notifyAll(): Unit

    

    Definition Classes

    AnyRef

17. def restrictFieldsBasedOnCardinality(fields: Array[String], cardinalityType: String, cardinalityLimit: Long, precision: Double = 0.05): Array[String]

    

    Method for filtering out any fields that are above a certain cardinality threshold to protect against creating unmanageably large feature vectors or computationally extreme StringIndexed values

    Method for filtering out any fields that are above a certain cardinality threshold to protect against creating unmanageably large feature vectors or computationally extreme StringIndexed values

    fields

    Fields to validate cardinality for

    cardinalityType

    The mode of cardinality checking [either "approx" for approximate distinct or "exact"]

    cardinalityLimit

    The limitation above which any field's cardinality will cause the field to be culled from the collection of fields to perform an operation on

    precision

    The precision set point for approx_distinct calculations for expected high cardinality fields or large data sets.

    returns

    Array[String] of column names whose cardinality is below the threshold specified by cardinalityLimit

    Since

    0.5.2

18. final val schema: StructType

    

19. final def synchronized[T0](arg0: ⇒ T0): T0

    

    Definition Classes

    AnyRef

20. def toString(): String

    

    Definition Classes

    AnyRef → Any

21. def validateCardinality(fields: Array[String], cardinalityType: String, cardinalityLimit: Long, precision: Double = 0.05): Array[String]

    

    Validation method for ensuring that the fields specified have a cardinality below a set threshold

    Validation method for ensuring that the fields specified have a cardinality below a set threshold

    fields

    Fields to test as an Array of Column Names

    cardinalityType

    The type of distinct check to perform to calculate the cardinality [either 'exact' or 'approx']

    cardinalityLimit

    The limit, above which, the check will fail.

    Annotations

    @throws( classOf[AssertionError] )

    Since

    0.5.2

    Exceptions thrown

    AssertionError if the cardinality of a field exceeds the threshold

22. final def wait(): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

23. final def wait(arg0: Long, arg1: Int): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

24. final def wait(arg0: Long): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )