|
|
Dear Fellow Communicators,
Sometimes the craft of outreach takes us into unknown territory. Today I’d like to tell the tale of a project that led us into astronomical and technological terra incognita. This little case study is about the production of a mosaic of the central parts of the Milky Way made with ESO’s VISTA telescope as part of the VVV survey (VISTA Variables in the Via Lactea) and released as ESO press release eso1242. Our system was tested to destruction by the attempted production of a 108k x 82k mosaicked image (108 250 x 82 067 pixels) — big enough to print a 9 x 7 square metre image (!) in 300 DPI. There were ten several significant challenges in this project made in collaboration between Olivier Hainaut, Martin Kornmesser, Richard Hook, Mathias Andre, Davide de Martin, Kaspar Nielsen, and myself:
- Astronomical processing: For a field as large as the VVV centre field — 30 x 20 degrees, there are significant projection and distortion effects. Team member Ignacio Toledo corrected most of the problems. A much smaller residual misalignment of a few pixels between the 3 exposures, was eventually corrected in Photoshop.
- Transport of the raw data: With the support of ESO’s helpdesk we expanded the FTP available storage and could finally move the vast quantities of data (the FITS files amount to a total of 166 gigabytes) back and forth between Toledo and ESO over several iterations.
- Liberation: Our normal dynamic range compression (conversion from FITS to tiff files) failed as the ESO/ESA/NASA FITS Liberator could not create tiff files above 2 GB. Since the dynamic range was not extremely large in the dataset, we reverted to using a less interactive method, using STIFF, that can create BigTIFF files of almost unlimited size. Meanwhile the FITS Liberator has been updated to write BigTiff files, and version 3.02 is about to be released on the website.
- Photoshop version: Only the latest Photoshop CS6, released just weeks before this project, can read BigTIFF files. An upgrade was performed and the individual greyscale exposures were opened in Photoshop and downconverted to 8 bits.
- Memory usage: The main problem of working on three layers of 9-gigapixel exposures with corresponding adjustment layers in Photoshop is memory. During the work it took up to 600 gigabytes of memory (!). First, an upgrade to a 64-bit Operating System was necessary (Windows 7, done earlier) and naturally the amount of RAM was significantly less than the needed memory. A solution to perform the swapping on state-of-the-art solid-state disks (SSD) gave a workable solution of providing a pseudo-memory delivering up to 1000 MB/s in real throughput.
- Rendering capacity: A modern graphics card was necessary to provide the necessary speed and on-board memory for the on-screen navigation in Photoshop.
- Cosmetic cleaning: For the cosmetic cleaning to remove instrumental artifacts and other blemishes of a non-cosmic origin in our astronomical images we normally we use an extremely solid outsourcing company, but in this case the transport of the individual layers was impractical, and also would have been too costly (as the cleaning is paid per megapixel). A less thorough, but sufficient in-house cleaning was performed. The one small by-product of this was that a few cosmetic defects were not cleaned and subsequently reported by dozens of hopeful “customers” as possible planets etc.
- Final file format: Unfortunately Photoshop CS6 does not write BigTIFF, so we looked for alternatives. JPEG can for instance at most save images in up to 65k x 65k resolution. We used the only format available above 65 kpixels — Photoshop’s proprietary PSB format. This is naturally a compromise as it is much less used than, e.g., TIFF, and hence less “portable”. This format was integrated into our web Digital Assets Management system (our own Djangoplicity), so that the PSB file could be served to the users. The final PSB file is 24.6 gigabytes — 108 250 x 82 067 pixels (~9 gigapixels). Also several new intermediate size formats were created in the system: 40k, 25k, 10k TIFFs.
- Zoomable images: Since very few users can actually use the full 9-gigapixel image, the main vehicle for delivering the experience to the end users is a zoomable image. Zoomify, our usual tool for creating zoomable products, did not work with PSB files. A good alternative was found: Krpano and Panotour Pro from Kolor, producing this beautiful zoomable version.
- Data serving: The final challenge was to actually serve the large individual files and the panorama. The news of the image spread like wildfire and it has so far been our most successful press release. To date we have had more than 600 000 visitors to this press release alone — more than double our previous record for a single release. The success nearly melted our servers during the peak load — they were sluggish for days and at times the number of “slots” were filled, which undoubtedly left some visitors disappointed. We had to move the large 24.6 GB PSB file first to another server and finally to a bit-torrent distribution as it took too many slots for too long. After a couple of weeks we went back to the normal system and are still serving hundreds of PSBs on a daily basis.
In the aftermath, we can summarise the lessons learned, and there are at least half as many as there were challenges:
- New paradigm: The data volumes of the new VISTA and VST surveys is a new paradigm for the creation of astronomical colour images.
- Investments needed: Big survey-type images take a significant investment in designing new pipelines, and in hardware and software.
- Never underestimate the needs of the target group: The users, more often than one might think, want the largest files. And it seems they actually can use them, and thereby help you disseminate your messages.
- Content delivery: Bit torrent is an excellent way to do distributed content delivery, if you (due to the large quantities of data) cannot afford Akamai or Amazon Cloudfront.
- Teamwork: Without an amazing team of experts in the areas of astronomy, technology and graphics we could not have overcome these individual challenges.
To those of you who have survived reading this far I’d like to mention that our 2013 calendar is available in the ESOshop as a free pdf for printing or for sale. Each month showcases one of ESO’s best vistas of the southern sky from 2012. These also include impressive images of ESO’s numerous telescopes, silhouetted against the remote Chilean landscape, that are guaranteed to catch the eye of visitors.The stock is limited so hurry up!
Let’s reach new heights in astronomy together,
Lars Lindberg Christensen (lars@eso.org)
Head, ESO education and Public Outreach Department (ePOD)
|
16 November 2012: This year, ESO is celebrating its 50th anniversary. To mark this occasion the organisation has produced several books and multimedia releases and hosted public events and workshops, culminating in a ...
|
| Read more |
|
14 November 2012: The ALMA Board announced today, 14 November 2012, that Pierre Cox has been appointed as the next Director of ALMA, the Atacama Large Millimeter/submillimeter Array. Cox will take up duty ...
|
| Read more |
|
13 November 2012: A new video compilation of freshly filmed footage of the Atacama Large Millimeter/submillimeter Array (ALMA) is now available. Under construction on the Chajnantor Plateau in the Chilean Andes, ALMA is ...
|
| Read more |
|
9 November 2012: Today, at ESO’s premises in Santiago, the European Southern Observatory and the Pontificia Universidad Católica de Chile launched the book Vistas de la Galaxia (Vistas of the Galaxy), by astronomers ...
|
| Read more |
|
2 November 2012: Building on the success of past years, ESO has produced a 2013 edition of the ESO Calendar. This was one of ESO’s most appreciated print products in the past and ...
|
| Read more |
|
25 October 2012: The latest edition of ESO’s quarterly journal, The Messenger, is now available online. Find out the latest news from ESO on topics ranging from new instruments to the latest science ...
|
| Read more |
|
30 October 2012: British New Wave rock band Ultravox were given a guided tour of ESO's Garching Headquarters on 28 October 2012. The band, consisting of Midge Ure (lead singer, guitarist, and frontman), ...
|
| Read more |
|
15 November 2012: This episode of the ESOcast introduces a new type of ESOcasts called "Chile Chill". These ESOcasts offer a calm experience of the Chilean night sky and ESO's observatory sites, undisturbed ...
|
| View and/or download |
|
|
|
|
|
|
|
|
