2026-10-15T03:37:15Z https://www.scidb.cn/oai

10.57760/sciencedb.j00186.00312 2023-11-06T10:31:25Z

2023-11-06 A Distributed Data Processing Scheme Based on Hadoop for Synchrotron Radiation Experiments doi:10.57760/sciencedb.j00186.00312 en This  research presents a case study on synchrotron radiation biomolecular crystallography to illustrate a beamline distributed data processing scheme based on the Hadoop ecosystem.We build a distributed file storage system for experimental crystallography data based on Hadoop HDFS. Additionally, we develop a resource scheduling system for the cluster using Hadoop YARN. Furthermore, we design and develop a distributed automated data processing pipeline(Spark-DIALS) for crystallography by combining Hadoop Spark and DIALS. The dials_spot_finder.py and dials_integrate_run.py contain the source code transforming the spots finding and integrate of original DIALS.Moreover, the solution utilizes FastAPI to deploy each functional module in a distributed microservice architecture.There are primarily microservices related to Spark distributed automatic processing jobs and HBase data table operations(sparkJobApi.py&hbaseApi.py). source code; code; Data Processing; Hadoop ding zhang PUBLIC https://api.github.com/licenses/mit dataset Science Data Bank