Emergent Latent-State Computation under Stochastic Volatility

This paper studies how sequence models infer hidden stochastic dynamics when they observe only noisy returns. Using a controlled multivariate stochastic-volatility benchmark with researcher-accessible latent states, the authors find evidence for a two-stage computation: hidden representations encode information about the next latent volatility state, while the output head converts that representation into squared-return forecasts. In Transformers, the stage where latent-state information becomes decodable depends on the volatility period. For long cycles, the computation resembles a learned linear projection followed by ℓ² normalization. Output-head replacement suggests that some MSE degradation reflects readout misalignment rather than failed representations.