Non-gradient methods

Gradient-based algorithms are the most used of all learning algorithms for DTRNN. But there are also some interesting non-gradient-based algorithms, most of which rely on weight perturbation schemes. Of those, two batch learning algorithms are worth mentioning:

• Alopex (Unnikrishnan and Venugopal, 1994) is a batch learning algorithm that biases random weight updates according to the observed correlation between previous updates of each learnable parameter and the change in the total error for the learning set. It does not need any knowledge about the network's particular structure; that is, it treats the network as a black box, and, indeed, it may be used to optimize parameters of systems other than neural networks; this makes it specially attractive when it comes to test a new architecture for which derivatives have not been derived yet. Alopex has only found limited use so far in connection with DTRNN (but see Forcada and Ñeco (1997) or Ñeco and Forcada (1997)).
• The algorithm by Cauwenberghs (1993) (see also (Cauwenberghs, 1996)) uses a related learning rule: the change effected by a random perturbation ${\bf\pi}$ of the weight vector ${\bf W}$ on the total error $E({\bf W})$ is computed and weights are updated in the direction of the perturbation so that the new weight vector is ${\bf W}-\mu E({\bf W}+{\bf\pi})-E({\bf W})){\bf\pi}$, where $\mu$ acts as a learning rate.. Cauwenberghs (1993) shows that this algorithm performs gradient descent on average when the components of the weight perturbation vector are mutually uncorrelated with uniform auto-variance, with error decreasing in each epoch for small enough ${\bf\pi}$ and $\mu$, and with a slowdown with respect to gradient descent proportional to the square root of the number of parameters.