Data management in microscopy: outcomes of the ACCS project

Progress in technique and instrumentation in microscopy has paved the way for unprecedented scientific advances. The ever-increasing data volumes produced, the greater acquisition speed for data capture, the increasing requirements in storage and computing power for data processing as well as the lack of standardisation in data formats and metadata have however posed challenges for microscopy research facilities and researchers to organise and manage their workflows from data capture through to storage.

Microscopy Australia, in partnership with the Australian Research Data Commons (ARDC), The University of Queensland, Monash University, the University of Wollongong and AARNet, have developed a range of tools and guidelines to address these challenges. The work was completed under the Australian Characterisation Commons at Scale (ACCS) project (Work Package “Big-data electron and correlative microscopy from instrument to publication”).

The publications describing the outcomes of this work are listed below:

1. The landscape in informatics and data management at microscopy research facilities:

• Poger, D., van Schyndel, J., Nguyen, H., Silver, J., & Goscinski, W. J. (2021). Orchestration and management of data generated by big-data electron microscopy instruments: A Discovery report. Zenodo. http://doi.org/10.5281/zenodo.4744876 
• Poger, D., Yen, L., & Braet, F. (2023). Big data in contemporary electron microscopy: challenges and opportunities in data transfer, compute and management. Histochemistry and Cell Biology, Advance online publication. https://doi.org/10.1007/s00418-023-02191-8 

2. A higher-level service to transfer data:

• van Schyndel, J., Silver, J., & Poger, D. (2021). Investigation, prototyping and deployment of a higher-level service for data transport. Zenodo. https://doi.org/10.5281/zenodo.5124450

3. Optimisation and automation of workflows for data movement:

• Silver, J. (2023). Optimisation and automation of workflows from data capture to data processing in electron microscopy. Zenodo. https://doi.org/10.5281/zenodo.7039561

4. Reproducible baselining and benchmarking of file-transfer performance:

• Myers, C., & Poger, D. (2022). Reproducible baselining and benchmarking of network performance with perfSONAR. Zenodo. https://doi.org/10.5281/zenodo.7018190

5. Real-time data processing in cryo-electron microscopy:

• Silver, J. (2023). Real-time processing of microscopy data. Zenodo. https://doi.org/10.5281/zenodo.7039664

6. Automation of repetitive tasks in data processing and analysis:

• Nguyen, H. A., & White, A. (2023). Automation of particle counting and size characterisation in microscopy images. Zenodo. https://doi.org/10.5281/zenodo.7686749

7. Making EM software more accessible to users:

• van Schyndel, J. (2022). The Electron Microscopy Data Processing Portal: A new capability for data-intensive research. Zenodo. https://doi.org/10.5281/zenodo.7302408
• The Electron Microscopy Data Processing Portal. http://micro.org.au/EMDPP
• Motamen, S., van Schyndel, J., Villa, J., & Poger, D. (March 2023). Electron microscopy data processing portal. Australian Microscopy & Microanalysis Newsletter (Issue 157), 36

8. A data-management software for the big-data EM community:

• Nguyen, H. A. (2023). Pitschi: A Clowder-based end-to-end data management tool. Zenodo. https://doi.org/10.5281/zenodo.7183431
• Pitschi. https://cmm.centre.uq.edu.au/pitschi

9. Guidelines for data retention and disposal:

• Poger, D. (2023). Research data retention and disposal: Overview and guidelines. Zenodo. https://doi.org/10.5281/zenodo.7964322

10. Guidelines for metadata collection:

• Poger, D. (2023). Metadata collection at microscopy research facilities: Overview and recommendations. Zenodo. https://doi.org/10.5281/zenodo.7949932