p

lamp

data

package data

Ordering
  1. Alphabetic
Visibility
  1. Public
  2. Protected

Package Members

  1. package bytesegmentencoding

    Greedy contraction of consecutive n-grams

  2. package distributed
  3. package schemas

Type Members

  1. trait BatchStream[+I, S, C] extends AnyRef

    A functional stateful stream of items

    A functional stateful stream of items

    lamp's training loops work from data presented in BatchStreams.

    An instance of BatchStream is an description of the data stream, it does not by itself allocates or stores any data. The stream needs to be driven by an interpreter. lamp.data.IOLoops and the companion object BatchStream contain those interpreters to make something useful with a BatchStream.

    See the abstract members and the companion object for more documentation.

    I

    the item type , the stream will yield items of this type

    S

    the state type, the stream will carry over and accumulate state of this type

    C

    type of accessory resources (e.g. buffers), the stream might need an instance of this type for its working. The intended use for fixed, pre-allocated pinned buffer pairs to facilitate host-device copies. See lamp.Device.toBatched and lamp.BufferPair.

  2. trait Codec extends AnyRef

    An abstraction around byte to token encodings.

  3. trait CodecFactory[T <: Codec] extends AnyRef

    An abstraction around byte to token encodings.

  4. sealed trait LoopState extends AnyRef
  5. case class NonEmptyBatch[I](batch: I) extends StreamControl[I] with Product with Serializable
  6. case class Peek(label: String) extends Module with Product with Serializable
  7. case class SWALoopState(model: Seq[STen], optimizer: Seq[STen], epoch: Int, lastValidationLoss: Option[Double], minValidationLoss: Option[Double], numberOfAveragedModels: Int, averagedModels: Option[Seq[Tensor]], learningCurve: List[(Int, Double, Option[Double])]) extends LoopState with Product with Serializable
  8. case class SimpleLoopState(model: Seq[STen], optimizer: Seq[STen], epoch: Int, lastValidationLoss: Option[Double], minValidationLoss: Option[Double], minValidationLossModel: Option[(Int, Seq[Tensor])], learningCurve: List[(Int, Double, Option[(Double, Double)])]) extends LoopState with Product with Serializable
  9. case class SimpleThenSWALoopState(simple: SimpleLoopState, swa: Option[SWALoopState]) extends LoopState with Product with Serializable
  10. sealed trait StreamControl[+I] extends AnyRef
  11. trait TrainingCallback extends AnyRef
  12. trait ValidationCallback extends AnyRef

Value Members

  1. object BatchStream
  2. object BufferedImageHelper
  3. object DataParallel
  4. case object EmptyBatch extends StreamControl[Nothing] with Product with Serializable
  5. case object EndStream extends StreamControl[Nothing] with Product with Serializable
  6. object GraphBatchStream
  7. object IOLoops

    Contains a training loops and helpers around it

    Contains a training loops and helpers around it

    The two training loops implemented here are:

  8. object IdentityCodec extends Codec
  9. object IdentityCodecFactory extends CodecFactory[IdentityCodec.type]
  10. object Reader
  11. object SWA
  12. object StateIO

    Helpers to read and write training loop state

  13. object StreamControl
  14. object Text
  15. object TrainingCallback
  16. object ValidationCallback
  17. object Writer

    Serializes tensors

    Serializes tensors

    This format is similar to the ONNX external tensor serialization format, but it uses JSON rather then protobuf.

    Format specification

    Sequences of tensors are serialized into a JSON descriptor and a data blob. The schema of the descriptor is the case class lamp.data.schemas.TensorList. The location field in this schema holds a path to the data blob. If this is a relative POSIX path then it is relative to the file path where the descriptor itself is written. Otherwise it is an absolute path of the data blob file.

    The descriptor may be embedded into larger JSON structures.

    The data blob itself is the raw data in little endian byte order. Floating point is IEEE-754. The descriptor specifies the byte offset and byte length of the tensors inside the data blob. As such, the data blob contains no framing or other control bytes, but it may contain padding bytes between tensors.

Ungrouped