Example with external loss is nonfunctional
It would be great to have an actual working example rather that this one with random generated error. No PR because I could not make it work, this is my proposal:
- calculate error from model output, maybe this for convergence to 0.4 :
List<INDArray> activations = model.feedForward(true, false);
INDArray output = activations.get(activations.size()-1);
INDArray target = Nd4j.ones(n_batch, nIn).mul(0.4);
INDArray externalError = output.sub(target);
externalError.muli(externalError);
externalError = externalError.mean() // this code suggests that the mean should also be divided by minibatch size https://github.com/deeplearning4j/deeplearning4j/blob/4c22ac5fe4a8350d05d224e7f4499429f7f69c93/deeplearning4j/deeplearning4j-nn/src/main/java/org/deeplearning4j/nn/layers/BaseOutputLayer.java#L83Multiply error by activations, as suggested by https://github.com/deeplearning4j/deeplearning4j/blob/4c22ac5fe4a8350d05d224e7f4499429f7f69c93/deeplearning4j/deeplearning4j-nn/src/main/java/org/deeplearning4j/nn/multilayer/MultiLayerNetwork.java#L1875
Pair<Gradient, INDArray> p = model.backpropGradient(error.mul(output), null);The above is not enough to get convergence, other things that might be needed after looking at the source code a) maybe flatten the gradient before updating model :
INDArray g = gradient.gradient(); INDArray fullGrad = g.reshape(g.length());b) maybe add rather than subtract, since updater has a step of 1 not -1 (Nd4j.getBlasWrapper().level1().axpy(model.params().length(), 1.0, fullGrad, model.params());):model.params().addi(fullGrad);
Again, the above changes are not enough to make the example work.
Source: deeplearning4j/deeplearning4j