ScioContextOps

Value Members

1. final def !=(arg0: Any): Boolean

   

   Definition Classes

   Any

2. final def ##(): Int

   

   Definition Classes

   Any

3. final def ==(arg0: Any): Boolean

   

   Definition Classes

   Any

4. final def asInstanceOf[T0]: T0

   

   Definition Classes

   Any

5. def avroFile[T <: SpecificRecord](path: String)(implicit arg0: ClassTag[T], arg1: Coder[T]): SCollection[T]

   

   Get an SCollection of type SpecificRecord for an Avro file.

6. def avroFile[T](path: String, schema: Schema)(implicit arg0: Coder[T], impl: AvroFileImpl[T]): SCollection[T]

   

7. def getClass(): Class[_ <: AnyVal]

   

   Definition Classes

   AnyVal → Any

8. final def isInstanceOf[T0]: Boolean

   

   Definition Classes

   Any

9. def objectFile[T](path: String)(implicit arg0: Coder[T]): SCollection[T]

   

   Get an SCollection for an object file using default serialization.

   Get an SCollection for an object file using default serialization.

   Serialized objects are stored in Avro files to leverage Avro's block file format. Note that serialization is not guaranteed to be compatible across Scio releases.

10. def parseAvroFile[T](path: String)(parseFn: (GenericRecord) ⇒ T)(implicit arg0: Coder[T]): SCollection[T]

    

    Get an SCollection of type T for data stored in Avro format after applying parseFn to map a serialized GenericRecord to type T.

    Get an SCollection of type T for data stored in Avro format after applying parseFn to map a serialized GenericRecord to type T.

    This API should be used with caution as the parseFn reads from a GenericRecord and hence is not type checked.

    This is intended to be used when attempting to read GenericRecords without specifying a schema (hence the writer schema is used to deserialize) and then directly converting to a type T using a parseFn. This avoids creation of an intermediate SCollection[GenericRecord] which can be in efficient because Coder[GenericRecord] is inefficient without a known Avro schema.

    Example usage: This code reads Avro fields "id" and "name" and de-serializes only those two into CaseClass

    val sColl: SCollection[CaseClass] =
      sc.parseAvroFile("gs://.....") { g =>
        CaseClass(g.get("id").asInstanceOf[Int], g.get("name").asInstanceOf[String])
      }

    Annotations

    @experimental()

11. def protobufFile[T <: Message](path: String)(implicit arg0: ClassTag[T], arg1: Coder[T]): SCollection[T]

    

    Get an SCollection for a Protobuf file.

    Get an SCollection for a Protobuf file.

    Protobuf messages are serialized into Array[Byte] and stored in Avro files to leverage Avro's block file format.

12. def toString(): String

    

    Definition Classes

    Any

13. def typedAvroFile[T <: HasAvroAnnotation](path: String)(implicit arg0: ClassTag[T], arg1: scala.reflect.api.JavaUniverse.TypeTag[T], arg2: Coder[T]): SCollection[T]

    

    Get a typed SCollection from an Avro schema.

    Get a typed SCollection from an Avro schema.

    Note that T must be annotated with AvroType.fromSchema, AvroType.fromPath, or AvroType.toSchema.