Packages

trait Op extends AnyRef

Represents an operation in the computational graph

Short outline of reverse autograd from scalar values

y = f1 o f2 o .. o fn

One of these subexpression (f_i) has value w2 and arguments w1. We can write dy/dw1 = dy/dw2 * dw2/dw1. dw2/dw1 is the Jacobian of f_i at the current value of w1. dy/dw2 is the Jacobian of y wrt to w2 at the current value of w2.

The current value of w1 and w2 are computed in a forward pass. The value dy/dy is 1 and from this dy/dw2 is recursed in the backward pass. The Jacobian function of dw2/dw1 is computed symbolically and hard coded.

The anonymous function which Ops must implement is dy/dw2 => dy/dw2 * dw2/dw1. The argument of that function (dy/dw2) is coming down from the backward pass. The Op must implement dy/dw2 * dw2/dw1.

The shape of dy/dw2 is the shape of the value of the operation (dy/dw2). The shape of dy/dw2 * dw2/dw1 is the shape of the parameter variable with respect which the derivative is taken, i.e. w1 since we are computing dy/dw1.

How to implement an operation
// Each concrete realization of the operation corresponds to an instance of an Op
// The Op instance holds handles to the input variables (here a, b), to be used in the backward pass
// The forward pass is effectively done in the constructor of the Op
// The backward pass is triggerd and orchestrated by [[lamp.autograd.Variable.backward]]
case class Mult(scope: Scope, a: Variable, b: Variable) extends Op {

// List all parameters which support partial derivatives, here both a and b
val params = List(
 // partial derivative of the first argument
 a.zipBackward { (p, out) =>
  // p is the incoming partial derivative, out is where the result is accumated into
  // Intermediate tensors are released due to the enclosing Scope.root
  Scope.root { implicit scope => out += (p * b.value).unbroadcast(a.sizes) }
  },
 // partial derivative of the second argument ..
 b.zipBackward { (p, out) =>
  Scope.root { implicit scope => out += (p * a.value).unbroadcast(b.sizes) }

  }
)
//The value of this operation, i.e. the forward pass
val value = Variable(this, a.value.*(b.value)(scope))(scope)

}
See also

https://en.wikipedia.org/wiki/Automatic_differentiation#Reverse_accumulation

http://www.cs.cmu.edu/~wcohen/10-605/notes/autodiff.pdf

Linear Supertypes
AnyRef, Any
Known Subclasses
Add, ArcTan, ArgMax, Assign, AvgPool2D, BatchNorm, BatchNorm2D, BatchedMatMul, BinaryCrossEntropyWithLogitsLoss, CappedShiftedNegativeExponential, CastToPrecision, Concatenate, ConstAdd, ConstMult, Conv1D, Conv2D, Conv2DTransposed, Cos, Cross, Diag, Div, Dropout, Embedding, EqWhere, EuclideanDistance, Exp, ExpandAs, Flatten, Gelu, IndexAdd, IndexFill, IndexSelect, Inv, L1Loss, LeakyRelu, Log, Log1p, LogDet, LogSoftMax, MaskFill, MaskSelect, MatMul, MaxPool1D, MaxPool2D, Mean, Minus, MseLoss, Mult, NllLoss, OneHot, Pow, PowConst, Relu, RepeatInterleave, Reshape, ScatterAdd, Select, Sigmoid, Sin, Softplus, SparseFromValueAndIndex, SquaredFrobeniusMatrixNorm, Stack, Sum, Tan, Tanh, ToDense, Transpose, Variance, View, WeightNorm
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Op
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract val params: List[(Variable, (STen, STen) ⇒ Unit)]

    Implementation of the backward pass

    Implementation of the backward pass

    A list of input variables paired up with an anonymous function computing the respective partial derivative. With the notation in the documentation of the trait lamp.autograd.Op: dy/dw2 => dy/dw2 * dw2/dw1. The first argument of the anonymous function is the incoming partial derivative (dy/dw2), the second argument is the output tensor into which the result (dy/dw2 * dw2/dw1) is accumulated (added).

    If the operation does not support computing the partial derivative for some of its arguments, then do not include that argument in this list.

    See also

    The documentation on the trait lamp.autograd.Op for more details and example.

  2. abstract val value: Variable

    The value of this operation

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  12. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  13. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  14. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  15. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  16. def toString(): String
    Definition Classes
    AnyRef → Any
  17. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  18. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  19. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from AnyRef

Inherited from Any

Ungrouped