Scientists in Japan have developed a powerful new way to compress X-ray imaging data in real time, reducing file sizes by more than 8,000 times while preserving the scientific information needed for accurate research.
The breakthrough was led by researchers at the RIKEN SPring-8 Center and could help solve a growing problem facing modern science: how to manage the enormous amounts of data generated by advanced imaging experiments.
Today’s X-ray and electron detectors are far more powerful than those of the past.
They can capture incredibly detailed images and measurements at extremely high speeds.
While this allows researchers to study materials and biological systems in unprecedented detail, it also creates huge amounts of data.
In some cases, experiments generate data streams approaching terabits per second.
These rates are so high that traditional systems for transferring, storing, and analyzing data struggle to keep up.
As a result, scientists often face a bottleneck where valuable information is produced faster than it can be handled.
To address this challenge, the research team developed a real-time compression system and tested it at Japan’s SPring-8 synchrotron radiation facility, one of the world’s most advanced X-ray research centers.
The system was connected to an X-ray detector that continuously generated data at a rate of 216 gigabits per second. Over a full day, this would amount to approximately 2.3 petabytes of data—enough to fill millions of smartphone storage devices.
Despite these enormous data volumes, the new system operated reliably and compressed the data by more than 8,000 times. Importantly, it achieved this without losing the detailed X-ray intensity information required for scientific analysis.
This is especially significant because even small distortions in scientific data can lead to inaccurate results. High compression rates are common for photographs or videos where some loss of detail is acceptable, but scientific experiments require much greater precision. The new method successfully balances both needs: dramatically reducing file sizes while maintaining data quality.
A key part of the system is specialized hardware known as a Field-Programmable Gate Array, or FPGA. Unlike ordinary computer processors that run software on fixed hardware, FPGAs can be customized to perform specific tasks directly in hardware. This allows many calculations to be carried out simultaneously, making them ideal for processing large streams of data in real time.
The researchers programmed the most demanding parts of the compression process into the FPGA hardware, enabling the system to keep pace with the rapid flow of incoming data.
The technology could have important applications beyond X-ray science. According to the researchers, it may also be useful for experiments and inspection systems that use electrons, gamma rays, neutrons, protons, heavy ions, and other forms of radiation.
By making it easier to handle massive data streams, the new approach could accelerate research in fields ranging from materials science and electronics to biology and medicine.
It may also help scientists gain new insights into how atoms, molecules, and nanoscale structures behave, opening the door to faster discoveries and more advanced technologies in the future.
