HazEvalHub

Overview¶

The HazEvalHub is where hazard models get scored. It holds the metrics, the validation protocols, and the held-out data that decide whether a prediction is good enough to act on.

We are developing two tracks for evaluating model performance:

• Extreme event monitoring: classification, regression, segmentation (e.g., detection of landslides, floods, earthquakes)

• Surrogate of Physical Models trained on physics-based simulations, a framework based on the AI Institute for Dynamical System Common Task Framework (collab with Nathan Kutz and Kaggle) with fair evaluation and hidden data set.

• Leaderboard: a set of hazard-relevant evaluation metrics when using geospatial and terrestrial networks in collaboration with AI2.

🔴 Live prototype — FrugalMind EvalHub¶

▶ Open the live eval board

FrugalMind EvalHub is the first working prototype of HazEvalHub. It is a live evaluation board for scientific AI agents in geoscience, and it establishes the design the full HazEvalHub generalizes to hazard tasks.

Every submission is scored on three questions:

The board. A cost-versus-performance scatter: each model appears twice — without domain skills (hollow marker) and with them (filled) — joined by a line showing the skill lift. Systems in the upper-left (high performance, low cost) win. Hover for model version, weights, and exact metrics; export to CSV/PNG.

Tasks. Document-based (literature review, retrieval-augmented QA, multimodal interpretation), software-agent (writing a detector, executing a pipeline, producing data), and research-workflow (orchestration and trajectory scoring). Concrete benchmarks include dv/v parameter choice, STA/LTA code generation, and ObsPy function usage. Validation splits are public; test splits are hidden to prevent memorization.

An early result. Free local 7B models (qwen2.5:7b, llama3.1:8b) reach perfect scores on configuration tasks once given domain skills — but fail at numerical code generation, where only cloud models succeed (~0.56 base, rising to 0.76 with skills). Domain skills lift small, cheap models to frontier parity on some task classes and not others; knowing which is the point of the board.

Where it goes next. Adopt its JSON scoring spec as the shared gaia-eval scorecard schema, carry the cost/frugality axis into the hazard metrics below, and extend the task taxonomy from agent tasks to the full pillar × hazard grid — keeping the agent tasks as the “research-workflow” track.

Evaluation Framework (TBD)¶

Components¶

1. Performance Metrics: Standardized metrics for model evaluation

2. Validation Protocols: Rigorous testing procedures

3. Benchmarking: Comparison against established baselines

4. Uncertainty Quantification: Assessment of prediction confidence

5. Operational Testing: Real-world performance evaluation

Evaluation Metrics¶

Validation Protocols¶

Cross-Validation¶

Temporal Cross-Validation¶

For time-dependent hazard data:

Spatial Cross-Validation¶

For spatially-correlated data.

Hold-out Testing¶

• Geographic hold-out: Testing on different regions

• Temporal hold-out: Testing on future time periods

• Event-based hold-out: Testing on specific hazard events

Benchmarking¶

Baseline Models¶

We will provide standard baselines for comparison (e.g., statistical baselines, classic ML models).

Performance Comparison¶

Uncertainty Quantification¶

Probabilistic Evaluation¶

Case Studies¶

Quality Assurance¶

Model Validation Checklist¶

Before deployment, models must pass:

• Cross-validation on training data

• Hold-out test performance meets thresholds

• Spatial/temporal generalization verified

• Uncertainty properly quantified

• Edge cases and failure modes identified

• Computational efficiency acceptable

• Documentation complete

Operational Evaluation¶

Real-time Monitoring¶

Feedback Integration¶

• Collect user feedback on predictions

• Incorporate new observations for continuous evaluation

• Update models based on operational experience

Evaluation Standards¶

We follow established standards:

• WMO Guidelines: For meteorological hazards

• USGS Standards: For seismic hazards

• ISO Standards: For risk assessment

• ML Best Practices: For model evaluation

Contributing¶

Help improve our evaluation framework:

1. Suggest new metrics for specific hazard types

2. Contribute validation datasets

3. Share evaluation protocols from your research

4. Report issues or limitations

Resources¶

• [Metrics API Reference]({{ github_org_url }}/{{ book_repo }}/wiki/metrics-api)

• Validation Datasets

Future Developments¶

Planned enhancements:

• Automated model selection based on evaluation metrics

• Multi-model ensembling with performance-weighted aggregation

• Continual learning with online evaluation

• Explainable AI techniques for model interpretability