Object

io.univalence.sparktest

SchemaComparison

Related Doc: package sparktest

Permalink

object SchemaComparison

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SchemaComparison
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. final case class AddField(dataType: DataType) extends FieldModification with Product with Serializable

    Permalink

  2. sealed trait ApplyModificationError extends Exception

    Permalink

  3. case class ApplyModificationErrorWithSource(error: ApplyModificationError, sc: StructType, schemaModification: SchemaModification) extends Exception with Product with Serializable

    Permalink

  4. final case class ChangeFieldType(from: DataType, to: DataType) extends FieldModification with Product with Serializable

    Permalink

  5. case class DuplicatedField(name: String) extends Exception with ApplyModificationError with Product with Serializable

    Permalink

  6. sealed trait FieldModification extends AnyRef

    Permalink

  7. case class NotFoundField(name: String) extends Exception with ApplyModificationError with Product with Serializable

    Permalink

  8. final case class RemoveField(dataType: DataType) extends FieldModification with Product with Serializable

    Permalink

  9. final case class SchemaModification(path: Path, fieldModification: FieldModification) extends Product with Serializable

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  4. object UnreachablePath extends Exception with ApplyModificationError with Product with Serializable

    Permalink

  5. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  6. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  7. def compareSchema(sc1: StructType, sc2: StructType): Seq[SchemaModification]

    Permalink

    Compare two schemas returning differences between both

    Compare two schemas returning differences between both

    Example : val sc1 = struct("number" -> int, "name" -> int) val sc2 = struct("rebmun" -> int, "name" -> double) compareSchema(sc1, sc2)

    Result : Seq( SchemaModification(path"number", RemoveField(int)), SchemaModification(path"name", ChangeFieldType(int, double)), SchemaModification(path"rebmun", AddField(int)) )

    sc1

    The entry schema

    sc2

    Schema to compare to

    returns

    Rhe sequence of schema modification from sc1 to s2

  8. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  9. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  10. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  11. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  12. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  13. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  14. def modifySchema(sc: StructType, schemaModification: SchemaModification): Try[StructType]

    Permalink

    Apply a modification to a schema

    Apply a modification to a schema

    Example : val sc = struct("number" -> int) val sm = SchemaModification(path"rebmun", AddField(int)) modifySchema(sc, sm)

    Result : Success(struct("number" -> int, "rebmun" -> int)

    sc

    The entry schema

    schemaModification

    A schema modication that must be applied to the entry schema

    returns

    The schema after the modification

  15. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  16. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  17. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  18. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  19. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any

  20. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  21. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  22. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from AnyRef

Inherited from Any

Ungrouped