For single instance and weights calculates gradient and loss.
For single instance and weights calculates gradient and loss. Depending on direction adds gradient and loss to the accumulated data.
Weights to evaluate gradient at
Featrues of instance to evaluate gradient at
Labels of the instance to evaluate gradient at
Update term to store gradient at
Loss vector to record resulting loss values.
Given labels info and weights matrice create appropriate ML models.
Given labels info and weights matrice create appropriate ML models.
Evaluates weight distance based on old and new weights images.
Evaluates weight distance based on old and new weights images.
Weights from the previous epoch
Weights from the current epoch.
Label to check for convergence.
Distance between old and new weights.
Adds L2 regularization part to the gradient and loss.
Adds L2 regularization part to the gradient and loss.
Apply L1 shrinkage to the updated weights.
Apply L1 shrinkage to the updated weights.
Given L1 regularization config create a vector with per-label reg param (by default - constant).
Given L1 regularization config create a vector with per-label reg param (by default - constant).
Given L2 regularization config create a vector with per-label reg param (by default - constant).
Given L2 regularization config create a vector with per-label reg param (by default - constant).
Utility used to split weights matrice into label -> vector map
Utility used to split weights matrice into label -> vector map
Extracts a single row from a matrice.
Extracts a single row from a matrice.
Extracts summary blocks from iterations loss history.
Extracts summary blocks from iterations loss history.
Extracts not converged labels based on actual and previous weights and on the loss history.
Extracts not converged labels based on actual and previous weights and on the loss history.
Evaluates loss difference simply as relative change
Merges weights from the new epoch with overal weights.
Merges weights from the new epoch with overal weights. Dimensions of weights matrices might be different when part of labels are already converged and do not participate in descend.
Used to preserve only active (not yet converged) labels into a vector
Used to preserve only active (not yet converged) labels into a vector
Used to preserve only active (not yet converged) labels into a matrix
Used to preserve only active (not yet converged) labels into a matrix
Single epoch of the descend
Single epoch of the descend
Data with features and labels
Weghts matrix to start with.
Average weights among walked during previous epoch.
Average gradient among seen during previous epoch.
Vector with the strength of L1 regularization (null if disabled)
Vector with the strength of L2 regularization (null if disabled)
Number of epoch
State with weights, averages and loss from this epoch
Updates the weights given update term and current value.
Updates the weights given update term and current value.
Evaluates weight norm for a given label.
Evaluates weight norm for a given label.
Weights matrix
Label to evaluate weights
Weights norm.
Created by dmitriybugaichenko on 10.11.16.
Implementation of a distributed version of Stochastic Variance Reduced Gradient Descent. The idea is taken from https://arxiv.org/abs/1512.01708 - input dataset is partitioned and workers performs descent simultaneously updating own copy of the weights at each random point (following SGD schema). At the end of epoche data from all workers are collected and aggregated. Variance reduction is achieved by keeping average gradient from previous iterations and evaluating gradient at one extra point (average of all weights seen during previous epoche). The update rule is:
w_new = w_old − η (∇f_i(w_old) − ∇f_i(w_avg) + g)
TODO: Other variance reduction and step size tuning techniques might be applied.
Requires AttributeGroup metadata for both labels and features, supports elastic net regularization and multiple parallel labels training (similar to MatrixLBFGS).