Data Clouds to Prevent Pyro Clouds

The Challenge

The Department of Environment, Land, Water and Planning (DELWP) in Victoria, now DEECA, manages bushfire risk through the "Residual Risk" metric, using modelling to calculate the likelihood and severity of bushfires based on fuel levels on public land. After four years, the metric needed cloud technology to improve accuracy. DELWP sought AWS cloud-based infrastructure and an ML framework for a next-generation risk modelling platform.

The context was devastating. During the 2019/2020 bushfire season, fires burnt more than 18 million hectares, more than 30 lives were lost, billions of animals were killed, and damage bill estimates exceeded $200 million. PHOENIX RapidFire, the fire behaviour simulator that puts Australia at the forefront of bushfire analysis, needed to be modernised.

The existing system suffered from outdated data layers and models, post-processing bottlenecks preventing scalability, manual processes requiring human intervention, a PostgreSQL bottleneck in post-processing, and limited spatial resolution at 5km squared detail when 1km squared was needed. The platform needed to handle six times more ignition points and double the simulation resolution. It had to be accessible by both scientists and fire chiefs.

The Approach

The project ran from December 2020 to March 2021. The team built a robust, scalable AWS cloud data platform to deliver bushfire prediction models via an intuitive and interactive user interface.

The architecture centred on replacing bottlenecks with scalable AWS services. Amazon Glue replaced the PostgreSQL post-processing bottleneck with PySpark, running post-processing steps concurrently instead of serially. While individual jobs were not dramatically faster (15 minutes versus 20 minutes), the ability to execute concurrently with independent scaling was transformative. The SQL changes required to work in Spark SQL were minimal. Amazon Glue Crawlers and Athena enabled queries on transformed data in S3 directly without loading into a database.

Amazon SageMaker Batch Transform integrated university-led research models from the R ecosystem using custom containers for existing model code. Batch inference ran on Phoenix outputs without significant development effort, scaling compute as required without ongoing server costs.

Amazon Lambda performed automatic scaling of the Phoenix cluster, compiled data inputs, populated job queues, monitored batch job status, converted output, and initiated Glue Crawlers. Amazon AppSync provided a fully managed GraphQL API backing the user interface, with Amazon Aurora as the database. Step Functions connected Glue Jobs and SageMaker Batch Transform in a state machine pipeline.

The architecture followed core principles: scalability through auto-scaling to accommodate increased data size and variety; automation to remove human intervention and reduce time; cloud-native design with AWS professionally certified DevOps engineers and data scientists; removal of post-processing bottlenecks through big data pipeline best practices; cost effectiveness through spinning down unused resources, serverless compute, and optimised storage; and performance visibility through a web interface for job management and status.

Designers composed an accessible user interface that displayed prediction knowledge and management advice with accurate maps, enabling fast communication between teams during times of high pressure.

The Results

The platform delivers a fully scalable, cost-effective AWS cloud platform for bushfire risk assessments. The rich user interface enables users to quickly create scenarios, upload data, and repeat previous runs with modifications. The fully automated cloud-native solution removes scaling constraints with automated data pipelines and serverless processes. Amazon Glue enables concurrent post-processing and SageMaker enables automated inference with ML capabilities.

The platform handles six times the ignition points and double the simulation resolution compared to the previous system, with an estimated 2.75 million executions per model run. It maintains interoperability with existing mapping systems (ArcGIS), prediction models, and decision workflows.

Processing time for post-processing tasks dropped significantly. While individual Glue jobs ran in approximately 15 minutes (compared to 20 minutes in the legacy system), the ability to run dozens of jobs concurrently rather than serially compressed total batch processing from hours to minutes. Scientists and fire chiefs can now access the same interface, running scenario comparisons that previously required manual coordination between technical teams and operational staff. The combination of automated data pipelines, serverless compute, and scalable storage means the platform can accommodate future increases in data volume and model complexity without architectural changes.

Looking Forward

The new data platform assists State and Territory firefighting services to be more prepared for longer, more intense fire seasons. The integration of university-led research models through SageMaker means that advances in fire behaviour science can be incorporated into the platform without significant re-engineering. The scalable platform enables more sophisticated data analysis, machine-learning-based fire prediction models, and automated fire response mechanisms including robotics and drones. The architecture's interoperability with ArcGIS and existing prediction models ensures it fits within established emergency management workflows rather than requiring agencies to change how they operate.

As climate conditions drive longer and more unpredictable fire seasons, the platform's capacity to ingest new data sources (including higher-resolution satellite imagery and additional IoT sensor networks) positions it as a foundation for the next generation of bushfire preparedness. The serverless architecture means that operational costs remain proportional to usage, making the platform viable for agencies of varying size and budget. The work established a replicable pattern for applying cloud data platforms and machine learning to natural disaster prediction, applicable beyond bushfires to flood, storm, and drought modelling.