In quantum dynamics of observables in quadratic systems, information about the initial state is retained in long-time averages, as captured by the generalized Gibbs ensemble. Here, we show that this memory extends beyond the averages to the temporal fluctuations around them. By generating a manifold of non-Gaussian initial states and quantifying their non-Gaussianity through the binary entropy, we find that the magnitude of temporal fluctuations is a smooth function of the initial-state non-Gaussianity. This relation explains the qualitatively different scaling of fluctuations observed after quantum quenches: exponentially small fluctuations for non-Gaussian initial states and polynomially small fluctuations for Gaussian initial states. Our results establish non-Gaussianity as a measure of initial-state memory that persists in long-time dynamics.
