lamp.data
Members list
Packages
Greedy contraction of consecutive n-grams
Greedy contraction of consecutive n-grams
Attributes
Data loader and inference utilities for the language model module in lamp.nn.langaugemodel
Data loader and inference utilities for the language model module in lamp.nn.langaugemodel
Attributes
Type members
Classlikes
Attributes
- Companion
- object
- Supertypes
-
class Objecttrait Matchableclass Any
- Self type
-
Attributes
- Companion
- trait
- Supertypes
-
class Objecttrait Matchableclass Any
- Self type
-
BatchStream.type
Attributes
- Supertypes
-
class Objecttrait Matchableclass Any
- Self type
-
BufferedImageHelper.type
An abstraction around byte to token encodings.
An abstraction around byte to token encodings.
Attributes
- Supertypes
-
class Objecttrait Matchableclass Any
- Known subtypes
-
class ByteSegmentCodecobject IdentityCodec.type
An abstraction around byte to token encodings.
An abstraction around byte to token encodings.
Attributes
- Supertypes
-
class Objecttrait Matchableclass Any
- Known subtypes
-
class ByteSegmentCodecFactoryobject IdentityCodecFactory.type
Attributes
- Supertypes
-
class Objecttrait Matchableclass Any
- Self type
-
DataParallel.type
Attributes
- Supertypes
-
trait Singletontrait Producttrait Mirrortrait Serializabletrait Producttrait Equalstrait StreamControl[Nothing]class Objecttrait Matchableclass AnyShow all
- Self type
-
EmptyBatch.type
Attributes
- Supertypes
-
trait Singletontrait Producttrait Mirrortrait Serializabletrait Producttrait Equalstrait StreamControl[Nothing]class Objecttrait Matchableclass AnyShow all
- Self type
-
EndStream.type
Attributes
- Supertypes
-
class Objecttrait Matchableclass Any
- Self type
-
GraphBatchStream.type
Contains a training loops and helpers around it
Contains a training loops and helpers around it
The two training loops implemented here are:
- lamp.data.IOLoops.epochs
- lamp.data.IOLoops.withSWA implements Stochastic Weight Averaging
Attributes
- Supertypes
-
class Objecttrait Matchableclass Any
- Self type
-
IOLoops.type
Attributes
- Supertypes
- Self type
-
IdentityCodec.type
Attributes
- Supertypes
- Self type
-
IdentityCodecFactory.type
Attributes
- Supertypes
-
class Objecttrait Matchableclass Any
- Known subtypes
Attributes
- Supertypes
-
trait Serializabletrait Producttrait Equalstrait StreamControl[I]class Objecttrait Matchableclass AnyShow all
Attributes
- Supertypes
-
trait Serializabletrait Producttrait Equalstrait GenericModule[Variable, Variable]class Objecttrait Matchableclass AnyShow all
Attributes
- Supertypes
-
trait Serializabletrait Producttrait Equalstrait LoopStateclass Objecttrait Matchableclass AnyShow all
Attributes
- Supertypes
-
trait Serializabletrait Producttrait Equalstrait LoopStateclass Objecttrait Matchableclass AnyShow all
Attributes
- Supertypes
-
trait Serializabletrait Producttrait Equalstrait LoopStateclass Objecttrait Matchableclass AnyShow all
Helpers to read and write training loop state
Helpers to read and write training loop state
Attributes
- Supertypes
-
class Objecttrait Matchableclass Any
- Self type
-
StateIO.type
Attributes
- Companion
- object
- Supertypes
-
class Objecttrait Matchableclass Any
- Known subtypes
Attributes
- Companion
- trait
- Supertypes
-
trait Sumtrait Mirrorclass Objecttrait Matchableclass Any
- Self type
-
StreamControl.type
Attributes
- Companion
- object
- Supertypes
-
class Objecttrait Matchableclass Any
Attributes
- Companion
- trait
- Supertypes
-
class Objecttrait Matchableclass Any
- Self type
-
TrainingCallback.type
Attributes
- Companion
- object
- Supertypes
-
class Objecttrait Matchableclass Any
Attributes
- Companion
- trait
- Supertypes
-
class Objecttrait Matchableclass Any
- Self type
-
ValidationCallback.type
Serializes tensors
Serializes tensors
This format is similar to the ONNX external tensor serialization format, but it uses JSON rather then protobuf.
==Format specification==
Sequences of tensors are serialized into a JSON descriptor and a data blob. The schema of the descriptor is the case class lamp.data.schemas.TensorList. The location field in this schema holds a path to the data blob. If this is a relative POSIX path then it is relative to the file path where the descriptor itself is written. Otherwise it is an absolute path of the data blob file.
The descriptor may be embedded into larger JSON structures.
The data blob itself is the raw data in little endian byte order. Floating point is IEEE-754. The descriptor specifies the byte offset and byte length of the tensors inside the data blob. As such, the data blob contains no framing or other control bytes, but it may contain padding bytes between tensors.
Attributes
- Supertypes
-
class Objecttrait Matchableclass Any
- Self type
-
Writer.type