Scalable implementations of ensemble filtering algorithms for data assimilation

Example implementations of the parallel algorithm are described for environments with low (high) communication latency and cost. Hybrids of these implementations and the traditional sequential ensemble filter can be designed to optimize performance for a variety of parallel computing environments. For large models on machines with good communications, it is possible to implement the parallel algorithm to scale efficiently to thousands of processors while bit-wise reproducing the results from a single processor implementation.

Timing results on several Linux clusters are presented from an implementation appropriate for machines with low-latency communication. Most ensemble Kalman filter variants that have appeared in the literature differ only in the details of how a prior ensemble estimate of a scalar observation is updated given an observed value and the observational error distribution. These details do not impact other parts of either the sequential or parallel filter algorithms here, so a variety of ensemble filters including ensemble square root and perturbed observations filters can be used with all the implementations described.