Pollack (1991):

This paper (http://www.dlsi.ua.es/~mlf/nnafmc/papers/pollack91induction.pdf) deals with the training of a class of second-order DTRNN (see section 3.2.1) to behave as language recognizers (Pollack (1991) uses the name dynamical recognizers, and defines them in a way parallel to the definition of deterministic finite automata, see section 2.3.3). The DTRNN is trained to recognize the seven languages in Tomita (1982) using a gradient-descent algorithm. One of the main emphases of the paper is in the cognitive implications of this process. Pollack (1991) also shows that, as learning progresses, the DTRNN undergoes a sudden change similar to a phase transition. He also formulates a tentative hypothesis as to the classes of languages that may be recognized by a dynamical system such as a DTRNN and its relation to the shape of the area visited by the network as strings get longer and longer (the attractor) and the way it is cut by the decision function used to determine grammaticality. Pollack (1991) studies then the nature of the representations learned by the DTRNN, first by examining the labels given by the networks to all strings up to length 9 (to find that the labelings are not completely consistent with the languages), and then by looking at the state space of the DTRNN, either graphically or by studying its fractal dimension.