Goudreau et al. (1994),

These authors (http://www.dlsi.ua.es/~mlf/nnafmc/papers/goudreau94first.pdf) study whether simple single-layer DTRNN (neural Moore machines, see section 3.2.2) with threshold linear units (TLU) are capable of representing finite-state recognizers or DFA. Their study shows that,

• when the output ${\bf y}[t]$ of the neural Moore machine is simply a projection of the state ${\bf x}[t]$, first-order DTRNN are not capable of representing some DFA, whereas second-order DTRNN may represent any DFA, and
• when the output is computed by a single-layer feedforward neural network (as in Elman (1990) nets), then first-order DTRNN may represent any DFA provided that some of the states (not all of them) in the DFA are represented by more than one state unit in the DTRNN, which Goudreau et al. (1994) call state-splitting (see section 4.2.2).

Complete splitting would lead to $n_X=\vert Q\vert\vert\Sigma\vert$ state units, as in the construction by Minsky (1967) , but the authors show an example where less units are used. With second-order networks, state-splitting is not necessary and $n_X=\vert Q\vert$ units are sufficient.