A Distributed Data Processing Scheme Based on Hadoop for Synchrotron Radiation Experiments

Additionally, we develop a resource scheduling system for the cluster using Hadoop YARN. Furthermore, we design and develop a distributed automated data processing pipeline(Spark-DIALS) for crystallography by combining Hadoop Spark and DIALS. The dials_spot_finder.py and dials_integrate_run.py contain the source code transforming the spots finding and integrate of original DIALS.Moreover, the solution utilizes FastAPI to deploy each functional module in a distributed microservice architecture.There are primarily microservices related to Spark distributed automatic processing jobs and HBase data table operations(sparkJobApi.py&hbaseApi.py).