lamp-data
API
Generated with
lamp-data/lamp.data/BatchStream

BatchStream

lamp.data.BatchStream
See theBatchStream companion object
trait BatchStream[+I, S, C]

A functional stateful stream of items

lamp's training loops work from data presented in BatchStreams.

An instance of BatchStream is an description of the data stream, it does not by itself allocates or stores any data. The stream needs to be driven by an interpreter. lamp.data.IOLoops and the companion object BatchStream contain those interpreters to make something useful with a BatchStream.

See the abstract members and the companion object for more documentation.

Type parameters

C

type of accessory resources (e.g. buffers), the stream might need an instance of this type for its working. The intended use for fixed, pre-allocated pinned buffer pairs to facilitate host-device copies. See lamp.Device.toBatched and lamp.BufferPair.

I

the item type , the stream will yield items of this type

S

the state type, the stream will carry over and accumulate state of this type

Attributes

Companion
object
Graph
Supertypes
class Object
trait Matchable
class Any
Self type
BatchStream[I, S, C]

Members list

Value members

Abstract methods

def allocateBuffers(target: Device): Resource[IO, C]

Allocation of a resource needed during the lifetime of the stream The intended use is for transfer buffer pairs

Allocation of a resource needed during the lifetime of the stream The intended use is for transfer buffer pairs

Attributes

def init: S

Initial value of the State

Initial value of the State

Attributes

def nextBatch(device: Device, buffers: C, state: S): IO[(S, Resource[IO, StreamControl[I]])]

Returns the resource of the next item and the next state suspended in an effect.

Returns the resource of the next item and the next state suspended in an effect.

Returned values are wrapped in StreamControl type which signals the interpreter whether the stream is finished.

May be called from different threads, but always in serial State should be carried over in the state parameter and return type

Attributes

Concrete methods

def concat[I2 >: I, S2](other: BatchStream[I2, S2, C]): BatchStream[I2, (Boolean, Either[S, S2]), C]

Returns a stream which is the concatenation of this stream and an other.

Returns a stream which is the concatenation of this stream and an other.

Attributes

def drainIntoSeq(device: Device): Resource[IO, Vector[I]]

Drives the stream and returns all the items from it in a scala collection.

Drives the stream and returns all the items from it in a scala collection.

Attributes

def everyNth(n: Int, offset: Int): BatchStream[I, (Long, S), C]

Takes only batches where (i % n == offset), i being the number of batch counted from 0

Takes only batches where (i % n == offset), i being the number of batch counted from 0

Attributes

def foldLeft[B](zero: B, device: Device, stateZero: S)(f: (B, I) => IO[B]): IO[B]

Folds a function from an initial value over this stream

Folds a function from an initial value over this stream

Attributes

def map[I2](f: I => Resource[IO, StreamControl[I2]]): BatchStream[I2, S, C]

Maps f over the elements of this stream

Maps f over the elements of this stream

Attributes

def repeatOrTake(requiredLength: Long): BatchStream[I, (Long, S), C]

ensures the length of the stream is fixed. Either repeats an element or truncates

ensures the length of the stream is fixed. Either repeats an element or truncates

Attributes

def take(n: Long): BatchStream[I, (Long, S), C]

Returns a stream wich contains only the first n elements of this stream

Returns a stream wich contains only the first n elements of this stream

Attributes

def withoutEmptyBatches: BatchStream[I, S, C]

Returns a new stream with EmptyBatches filtered out

Returns a new stream with EmptyBatches filtered out

Attributes

Generated with