A serverless tool for streaming scientific datasets directly from the cloud, for exploration and engagement.
Acknowledgements
This dataset is a collaboration between Murdoch University and the WA Department of Water and Environmental Regulation (DWER).
Compute and storage infrastructure is provided by the Pawsey Supercomputing Research Centre.
Note: this is not yet an official product of any of these organisations, simply a proof of concept, demonstrating the potential of these approaches for data sharing and exploration.
Explore WRF regional climate model output (maximum and minimum temperatures, rainfall) streamed directly from Pawsey Acacia — no download required.
Interactively explore climate model output streamed directly from Pawsey Acacia object storage. Same architecture, bigger datasets, less polish.
What it can do
Climate model output is large, and traditionally doing anything useful with it has meant either getting an account on an HPC system, or sitting through a multi-gigabyte download — before writing a single line of code. Neither is great if a scientist, policy analyst, or curious member of the public just wants to see what the data looks like, or what the results mean for them.
This tool lets you explore climate datasets directly in the browser — no download, no login, no Python environment to configure. Select a variable, scroll through 40+ years of monthly output, and the relevant data streams on demand from Pawsey's object storage. You never pull more than the slice you're actually looking at, making real time exploration viable for the first time ever.
The goal is to lower the barrier to engagement with research outputs: making it easier for collaborators to sanity-check model runs, for stakeholders to explore projections, and for the public to connect with the science — without needing to involve software/data engineers, learn to code, or get access to special infrastructure.
Technical details
The key insight is that the original NetCDF files never need to move or be converted. Instead, VirtualiZarr reads the internal structure of each file — where each variable's chunks live on disk, their byte offsets and lengths — and builds a lightweight virtual Zarr store. Nothing is copied or duplicated; you end up with a reference catalogue that says "chunk [t,y,x] of tasmax is bytes 171,279,300–172,408,502 of this file on S3". The catalogue for the entire SWWA dataset fits in a small JSON file — a trivial overhead compared to the data itself.
That JSON is written in Kerchunk reference format, a simple spec that maps Zarr chunk keys to [url, offset, length] triples. Any Zarr-aware client that can issue HTTP range requests can consume it directly — no special server required.
In the browser, zarrita.js handles Zarr I/O backed by a range-request store. When you move the time slider, it resolves which chunks are needed, fires off a handful of HTTP Range requests to the object storage endpoint, decompresses the data in-browser using a WASM codec, and passes the result to the map renderer. A single monthly field for the SWWA domain requires only a few hundred kilobytes of compressed data — fetched on demand from an archive that is orders of magnitude larger.
The architecture is completely serverless: there is no backend, no tiling service, and no bespoke data pipeline running on a VM. The only infrastructure is the object storage bucket (which already exists for model output) and this static site.