Seismic event catalog workflows — the QuakeScope benchmarks

Why catalog workflows need their own evaluation¶

A phase picker is one stage. What a hazard user consumes is the catalog that a whole workflow emits — detection, phase picking, association, location, magnitude — and an improvement at one stage can be undone at the next. Scoring the picker in isolation measures a component; scoring the workflow measures the product.

The distinction matters concretely for GAIA. The fifteen-year Mount Rainier surface-event catalog is produced by running a classifier over a continuous archive and locating what it finds Kharita, 2025, and the choice of classifier was itself settled by a comparison of machine-learning methods on roughly 200,000 waveforms from an AI-curated regional dataset Kharita et al., 2026Ni et al., 2023. Every one of those stages has its own failure mode, and only the catalog at the end is what a landslide study actually uses.

The picker benchmarks¶

QuakeScope compares sets of model weights against reference arrivals across five study regions, consolidating five benchmark notebooks into one report.

Full report at seisscoped.org/QuakeScope/benchmark_summary.html; project at seisscoped.org/QuakeScope; source at github.com/SeisSCOPED/QuakeScope.

What is compared¶

Four sets of model weights — quakescope2026, jma_wc, original and instance — against five sets of reference arrivals:

The headline metric is recall — the fraction of reference picks recovered within 0.5 seconds — reported per phase for P and S.

A threshold is not an operating point¶

This is the transferable idea on the page, and it generalises well beyond phase picking.

Two models that both emit a probability do not emit the same probability. A confidence of 0.3 from one picker and 0.3 from another can correspond to completely different pick budgets, so comparing recall at a shared threshold silently compares one model being permissive against another being conservative. The model that emits more picks recovers more reference arrivals, and has not necessarily done anything better.

The comparison that holds is recall against picks emitted: sweep the threshold, plot the curve, and read off what each model recovers at equal budget. Any benchmark whose leaderboard reports a single number at a fixed threshold is making this mistake, and most do.

What this track needs next¶

Against the nine rules on the hub page:

• No frozen task specification (R1). The benchmarks are a report, not an open task others can submit to.

• Nothing is hidden (R2). Every study runs against public bulletins, so the benchmark cannot distinguish generalisation from familiarity with a well-studied sequence.

• No trivial baseline (R4). An STA/LTA detector published alongside would give the recall numbers a floor.

• Splits are not DOI-archived with a datasheet (R7), so a result here is not reproducible after the underlying bulletins change.

• No per-row cost (R9). Picker throughput is a real operational constraint at archive scale and is currently invisible.

One rule this track already passes that the agent track does not: the benchmark is maintained by SeisSCOPED, not by GAIA (R8). We are scored here rather than scoring ourselves, which is the separability the agent board has to engineer deliberately.

The GAIA-specific next step is a temporally disjoint Mount Rainier test set — events drawn from after the training window of the models it scores, newly labelled — which turns this from a report into a benchmark a result could be cited from.

References¶