WorkspaceHow it worksARIA Researcher

The question

Can an AI agent beat the published WildfireIA benchmark for predicting initial-attack failure: which US wildfires escape the first firefighting response? The prediction may only use information available at discovery time. Published reference: 0.533 test AUPRC.

Data: 38,128 fires. Train on 2016–2018, select on 2019, and test exactly once on 2020.

The research loop

  1. 1
    Observe. ARIA reads the full research state: every score, error count, confusion matrix, feature importance and where the model fails (by state, month, fuel type, fire danger, satellite detection).
  2. 2
    Hypothesize. It writes one falsifiable hypothesis grounded in the measured numbers, with an expected result.
  3. 3
    Design. It proposes exactly one experiment from an allowed menu, usually changing one variable against the current best.
  4. 4
    Validate. The worker checks the proposal against a strict schema. Anything out of bounds or repeated is rejected with structured feedback, and ARIA must try again. The code never substitutes its own idea.
  5. 5
    Train & evaluate. The worker trains the model on 2016–2018 fires and scores it on the 2019 validation year, logging the run, metrics and artifacts to Weights & Biases.
  6. 6
    Decide. ARIA marks the experiment KEEP or REJECT with a reason and what it learned, then asks the next question.
  7. 7
    Freeze. When the budget is spent, ARIA recommends one configuration. It is scored once on the sealed 2020 test year.

Who does what

ARIA (the AI agent)

W&B's research agent. The only decision-maker: hypotheses, experiments, KEEP/REJECT and the final recommendation. Each cycle is a fresh conversation, so it receives the complete state every time.

Worker (not AI)

Python code that enforces the rules: validates proposals, trains models, measures results and publishes the next state. If ARIA is silent, it waits.

W&B Automations + Artifacts

When an experiment run finishes, an Automation wakes ARIA. State goes out and proposals come back as versioned artifacts, so every exchange has provenance.

Weave

Traces each step of the loop (receive proposal, validate, train, evaluate, record decision) for inspection.

What ARIA can change

Models
Logistic regression, XGBoost
Feature sets
Incident metadata, FIRMS satellite detections, weather, fuel, vegetation, topography, access, human factors, all, or all-but-one
Weather window
1–5 days before discovery
Hyperparameters
Bounded ranges, e.g. XGBoost trees 50–2000, depth 2–10, learning rate 0.005–0.3, regularisation, class weighting

Every experiment reports validation AUPRC (the selection metric), AUROC, precision, recall, F1, missed fires, false alarms, the train–validation gap and per-group error breakdowns.

Guardrails

  • ARIA can only choose from a closed, bounded menu. Its output is data and is never executed.
  • The 2020 test year is sealed until ARIA freezes its final choice.
  • Validation scores are never presented as comparable to the published test score.
  • A scripted fallback exists for offline tests only, and every session it drives is labelled “NOT ARIA”.

Results so far

Two of ARIA's final recommendations were scored on the sealed 2020 test year over five seeds: 0.5413 ± 0.0024 and 0.5437 ± 0.0017 test AUPRC, both above the published 0.533. These are recorded results, not a certified benchmark claim.

This deployment replays the five recorded sessions read-only. Open the Research journal to step through ARIA's reasoning experiment by experiment.