Using a data trickler and a secure gigabit link to the Australian Academic Research Network (AARNet) each night’s scan produces about 0.7 terabytes of data, which is transferred from Coonabarabran to the NCI National Facility in Canberra.
The data is then stored using a hierarchical storage management system, which mixes disk and a large robotic tape library system to help preserve the data in a regular, categorised form and duplicate it in two separate locations for backup purposes.
(Check out CIO's SkyMapper slideshow here.)
To increase the usability and accessibility of the data, the project will also shrink the raw data down to the actual numbers that are most important to researchers -- the shapes, sizes and brightness of the billion objects in the southern sky.
"We are going to image those one billion-odd objects in 36 images spaced in time over five years, Keller says. “In that way we can look at objects that vary across the sky, objects such as pulsating stars and moving asteroids.
“Then we reduce the data and end up with a database of about 30 terabytes, which we then make available via the Web. As far as we know it will be Australia’s largest database.”
Automation
Needing to identify and catalogue around a billion objects, and scan for five years, SkyMapper relies on a high level of intelligent automation, Keller says.
Using an automation and scheduling application, SkyMapper is capable of independently assessing night sky conditions -- whether the moon is out, how bright the stars are, whether there are clouds -- and then progress through the most suitable scientific program.
The cataloguing of one billion objects across more than 4000 survey fields is also automated, meaning that SkyMapper is able to discern objects based on factors such as brightness and shape, Keller says.
“We can cleanly extract all the stars and measure their brightness,” he says. “Galaxies are a bit harder as they can have spiral arms on them but we can still easily find the interesting objects that lie in that data set.”
Open Source
According to Keller, the SkyMapper project is heavily based on open source software, largely because of its low cost.
“We are on a very tight budget, so any expertise we can draw on in a shared way is extremely valuable to us,” Keller says.
“Most of our pipeline is comprised of components written by astronomers elsewhere in the world over many decades. We draw together those little units and basically script them all up together with Perl and Python, and that makes for an efficient coding process.”
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.