Omlin and Giles (1996b)

These authors http://www.dlsi.ua.es/~mlf/nnafmc/papers/omlin96stable.pdfset out to prove whether there exists a way to choose weights in a DTRNN based on real sigmoids (in particular, the logistic function $g(x)=1/(1+\exp(-x))$), therefore allowing real ranges of outputs and state values, so that the DTRNN behaves as a deterministic finite-state automaton (see section 2.3.3). They propose a way to choose the weights of a second-order DTRNN that guarantees that the language accepted by the DTRNN and that accepted by the DFA are identical; all weights and biases are simple multiples of a single value $H$ ($H$, $-H$, $-H/2$ and 0).^5.6 A careful worst-case analysis of the fixed points and bounds of repeated applications of the next-state function defines the actual value of $H$, which is always greater than $4$ and grows roughly as $\log(n_X)$. The experimental values of $H$ found by the authors seem however to be constant for a set of large random DFA.