sparkey

Main package for Sparkey side input APIs. Import all.

import com.spotify.scio.extra.sparkey._

To save an SCollection[(String, String)] to a Sparkey file:

val s = sc.parallelize(Seq("a" -> "one", "b" -> "two"))

// temporary location
val s1: SCollection[SparkeyUri] = s.asSparkey

// specific location
val s1: SCollection[SparkeyUri] = s.asSparkey("gs:////")

The result SCollection[SparkeyUri] can be converted to a side input:

val s: SCollection[SparkeyUri] = sc.parallelize(Seq("a" -> "one", "b" -> "two")).asSparkey
val side: SideInput[SparkeyReader] = s.asSparkeySideInput

These two steps can be done with a syntactic sugar:

val side: SideInput[SparkeyReader] = sc
  .parallelize(Seq("a" -> "one", "b" -> "two"))
  .asSparkeySideInput

An existing Sparkey file can also be converted to a side input directly:

sc.sparkeySideInput("gs:////")

SparkeyReader can be used like a lookup table in a side input operation:

val main: SCollection[String] = sc.parallelize(Seq("a", "b", "c"))
val side: SideInput[SparkeyReader] = sc
  .parallelize(Seq("a" -> "one", "b" -> "two"))
  .asSparkeySideInput

main.withSideInputs(side)
  .map { (x, s) =>
    s(side).getOrElse(x, "unknown")
  }

Linear Supertypes

AnyRef, Any

Type Members

implicit class RichStringSparkeyReader extends Map[String, String]

Enhanced version of SparkeyReader that mimics a Map.
implicit class SparkeyPairSCollection[K, V] extends AnyRef

Enhanced version of SCollection with Sparkey methods.
implicit final class SparkeySCollection extends AnyVal

Enhanced version of SCollection with Sparkey methods.
implicit final class SparkeyScioContext extends AnyVal

Enhanced version of ScioContext with Sparkey methods.
trait SparkeyUri extends Serializable

Represents the base URI for a Sparkey index and log file, either on the local or a remote file system.
Represents the base URI for a Sparkey index and log file, either on the local or a remote file system. For remote file systems, basePath should be in the form 'scheme://<bucket>/<path>/<sparkey-prefix>'. For local files, it should be in the form '/<path>/<sparkey-prefix>'. Note that basePath must not be a folder or GCS bucket as it is a base path representing two files - <sparkey-prefix>.spi and <sparkey-prefix>.spl.
sealed trait SparkeyWritable[K, V] extends Serializable

Value Members

implicit val ByteArraySparkeyWritable: SparkeyWritable[Array[Byte], Array[Byte]]
implicit val stringSparkeyWritable: SparkeyWritable[String, String]

package sparkey

Type Members

implicit class RichStringSparkeyReader extends Map[String, String]

implicit class SparkeyPairSCollection[K, V] extends AnyRef

implicit final class SparkeySCollection extends AnyVal

implicit final class SparkeyScioContext extends AnyVal

trait SparkeyUri extends Serializable

sealed trait SparkeyWritable[K, V] extends Serializable

Value Members

implicit val ByteArraySparkeyWritable: SparkeyWritable[Array[Byte], Array[Byte]]

implicit val stringSparkeyWritable: SparkeyWritable[String, String]

Inherited from AnyRef

Inherited from Any

Ungrouped