FileSender addresses the challenge of moving large datasets around for researchers participating in the Joint Remote Sensing Research Program.
For environmental researchers, the volume of data generated by remote sensing technologies can be massive and challenging to manage. The Joint Remote Sensing Research Program (JRSRP) is a leader in this field, involving experts from academia and government to push the boundaries of what is possible in environmental observation. Their mission is to develop methods to track environmental changes over time, and to inform decisions on land use, agriculture and environmental management.
Addressing data management challenges
Navigating the complexities of data movement is a challenge for the JRSRP, a collaboration that involves multiple institutions, including the New South Wales Department of Climate Change, Energy, the Environment and Water(NSW DCCEEW), the University of New South Wales, the University of New England, as well as The University of Queensland and the Queensland Department of Environment, Science and Innovation.
JRSRP researchers rely on a suite of remote sensing instruments—among them, Sentinel-2 satellites and terrestrial laser scanners (TLS), to collect detailed data on vegetation and land use patterns. A typical research project can generate substantial amounts of data. For example, for a recent project, Sentinel-2 satellite imagery amounts to several gigabytes (GB) of data, and TLS scans reached 5.5 GB per site.
Dr Jim Watson, Principal Scientist for remote sensing at NSW DCCEEW said the cross-institute file transfer methods they were using had limitations, so they needed to find a better way to move these large datasets around quickly and securely.
“With the assistance of AARNet, we tried FileSender, which proved to be the robust data transfer solution we required for the task,” he said.
Leveraging the high-bandwidth AARNet network
With all the JRSRP partner organisations and institutions connected to the high-speed AARNet network, AARNet’s FileSender web application provided an optimal solution. FileSender, with its ease of use and reliability, also offers end-to-end encryption and flexible access control, ensuring data security and integrity.
“FileSender worked better than I initially expected because it benefits from the AARNet network’s ability to handle a high volume of data and scale up and down on demand without any degradation in performance. Also, being able to share and receive data from collaborators without requiring them to create an account significantly simplifies the sharing process,” explained Dr Watson.
Dr Watson and his team have now used FileSender for various projects, including successfully transferring large satellite and field datasets quickly and easily from the NSW DCCEEW high-performance computing facility to the Australian Research Data Commons Nectar Research Cloud, which they use for analysis.
They are in the process of testing FileSender for the purpose of distributing field data visualisations, a task that involves transferring more than 40 GB of drone-based LiDAR (Light Detection and Ranging) data to the Nectar Cloud. With FileSender, they can complete these large data transfers in just a few hours, significantly faster than the time it took with their former methods of data transfer.
Optimising research data transfers
Adopting FileSender allows program participants to enjoy hassle-free transfers without using up the storage quotas of their corporate cloud services, since it operates independently of their internal systems. In addition, FileSender streamlines the management of user accounts by allowing the use of existing institutional login credentials, eliminating the need to create a new account for the service.
The JRSRP’s integration of AARNet’s FileSender into their data movement workflow demonstrates the vital role robust and reliable digital infrastructure plays in data-intensive scientific collaborations.
The particle energies required for this testing were so high that only the capabilities of HIAF could meet the challenge. “We could not have performed the testing at any other facility in Australia,” says Jacob Cook, Instrumentation Engineer for the Binar Space Program.
The testing provided crucial data needed to understand how the detector would perform throughout the mission’s lifetime, ultimately determining the mission’s success.
Binar-2, 3 and 4 were successfully deployed from JAXA: Japan Aerospace Exploration Agency’s Kibo module of the International Space Station on August 29th at 7:20pm AWST, and contact was made by Curtin University with all 3 CubeSats on their very first pass overhead.
The information gathered from this mission not only aids in refining CSIRO’s radiation shielding material but also contributes to advancing our understanding of space radiation effects on satellite instrumentation.