Recurrent cascade correlation

Fahlman (1991) has recently proposed a learning algorithm that establishes a mechanism to grow a DTRNN during training by adding hidden state units which are trained separately so that their output does not affect the operation of the DTRNN. Training starts with an architecture without hidden state units,

$$y_i[t]=g\left(\sum_{j=1}^{n_U} W_{ij}^{yu} u_j[t] + W^y_i \right)\;, i=1\ldots n_Y,$$ (4.29)

and a pool of $n_C$ candidate hidden units with local feedback which are connected to the inputs are trained to follow the residual error of the network:

$$x_i[t]=g\left(\sum_{j=1}^{n_U} W_{ij}^{xu} u_j[t] + W_{ii}^{xx} x_i[t-1] + W^x_i \right)$$ (4.30)

with $i=1\ldots n_C$. Training adds the best candidate unit to the network in a process called tenure. If there are already $n_H$ tenured hidden units, the state of candidate $i$ is

$$x_i[t]=g\left(\sum_{j=1}^{n_U} W_{ij}^{xu} u_j[t] + W_{ii}^{... ...i[t-1] + \sum_{j=1}^{n_H} W_{ij}^{xx'} x_j[t] + W^x_i \right)$$ (4.31)

(the prime in $W_{ij}^{xx'}$ meaning that it weights state values at time $t$, not $t-1$ as usual). Tenure adds the best of the candidates to the network as a hidden unit labeled $n_H+1$ (where $n_H$ is the number of existing hidden units), its incoming weights are frozen and connections are established with the output units and subsequently trained. Therefore, hidden units form a lower-triangular structure in which each of the units receives feedback only from itself (local feedback)and the output is computed from the input and each of the hidden units:

$$y_i[t]=g\left(\sum_{j=1}^{n_U} W_{ij}^{yu} u_j[t] + \sum_{j=1}^{n_H} W_{ij}^{yx} x_j[t] + W^y_i \right)\;, i=1\ldots n_Y.$$ (4.32)

Recurrent cascade correlation networks have recently been shown to be incapable of recognizing certain classes of regular languages (see section 4.2.3).