The Need for Speed

In the current updates to the NE-CAT beamlines, we are improving all aspects of the beamline performance.  So far, we have upgraded the detector systems on 24-ID-C to the XE version of the EIGER2 to allow 500Hz data collection.  We have improved the performance of raster scanning and migrated data capture to use NeXus / HDF5 (NXmx) in place of the existing CBF system to support high-frame rate data collection. This casts the processing speed into sharp relief: if data collection can be performed in a few seconds, data processing taking minutes is no longer acceptable. In response, we have developed MXI, a DIALS-like data processing software package written with AI assistance (Claude Opus 5.5).  MXI accelerates data processing through optimized use of computing hardware to a timeframe comparable with a typical data acquisition cycle and runs on commonly-available hardware.

A concrete example of this can be seen with an example data set from Zenodo: taking the first sweep of this data set, 3,600 EIGER 2XE 16M frames, the processing with a minimal DIALS script takes around 9 minutes on an M4 Max MacBook Pro. For NE-CAT users familiar with XDS, in this case, data processing occurs in approximately 3 minutes, 40 seconds on the same hardware.  The same data set with version 1.3.1 of MXI only requires 35s, with comparable quality results. This performance is achieved by (i) optimizing the algorithms by reducing unnecessary overhead and (ii) optionally making use of GPU hardware (Apple Metal and NVIDIA CUDA supported) for the more compute-intensive steps. This timeframe allows decisions to be made on how to proceed while the sample is still on the goniometer, and certainly before the next sample has been centered.

The software is deliberately open. The software, the tutorials, and precompiled binaries are all available from the GitHub page above with an open-source license (recognizing that the MXI software itself is derived from open prior art).  All typical platforms are supported: macOS (Apple Silicon), Linux with and without CUDA (x64 and arm64 – yes, it will run on a Raspberry Pi 4/5 or NVIDIA Grace Hopper) and Windows (x64). Included in the Python release bundle are a GUI (mxi-ui) and image viewer (mxi-view), however, the fundamental package itself is a single binary of a few megabytes.

The design is explicitly limited in scope, processing only rotation data and only reads NXmx-format HDF5 files, but the capabilities are broad, being able to tackle all the usual DIALS tutorial data sets (e.g.  🐮, 🐷, and 🤠, and small molecule data sets). We expect this to form the basis for our future fast data processing on NE-CAT beamlines; so welcome user feedback.  As we improve the speed of our beamlines, we want to ensure we maintain the excellent data quality our users have come to expect in both instrumentation and data processing.

The software is made free to the community (and even our competition).  Its development used public funding from NIGMS and is based on the existing open-source packages DIALS.  For anyone interested the history is available online. While we recognize the environmental impact of AI in developing software, in this case it is measured against a vast reduction in the material and energy cost of processing data, which will pay back for years to come.