A comparative forecasting framework for Everglades wading bird populations using dynamic GAMs and traditional time series models.
This project implements sliding window cross-validation to compare multiple forecasting approaches for wading bird populations in the Florida Everglades. Models incorporate water management covariates and evaluate predictions using probabilistic numeric accuracy (CRPS), point-forecast accuracy (RMSE), and ordinal accuracy (RPS).
Species Options: The pipeline dynamically handles species filtering based on configuration:
top6(Default):- Great Blue Heron (gbhe)
- Great Egret (greg)
- Roseate Spoonbill (rosp)
- Snowy Egret (sneg)
- White Ibis (whib)
- Wood Stork (wost)
all: All identified species in the database.- Custom List: E.g.,
["anhi", "limp", "snki"]. - Unknowns: Option to retain unidentified counts (e.g., small white heron) via
include_unknowns. - System Totals: Option to aggregate all selected species into a single
Totalbiomass count for system-wide or regional capacity forecasting.
Spatial Hierarchy:
all
└── subregion (8 regions: 1, 2a, 2b, 3an, 3as, 3ase, 3b, inlandenp)
└── colony (33 viable colonies ≥15 years)
# Required packages
install.packages(c(
"config", # Configuration management
"conflicted", # Namespace conflict handling
"distributional",# Vectorised probability distributions
"dplyr", # Data manipulation
"fable", # Traditional time series models
"feasts", # Feature extraction
"ggplot2", # Figures
"glue", # String formatting
"remotes", # install from github
"scales", # for nice plotts
"tidyr", # Data structure
"tsibble", # Time series tibbles
"verification", # RPS scoring
"zoo", # Interpolation
"future", # Parallel processing
"furrr", # Parallel mapping
"digest" # Caching
))
# Install from GitHub
remotes::install_github("nicholasjclark/mvgam") # Dynamic GAMs
remotes::install_github("weecology/wader") # Bird count data
remotes::install_github("weecology/edenr") # Everglades water data
remotes::install_github("hendersontrent/fable.gam") #Fable GAMs.
├── main.R # Main execution script
├── config.yml # Configuration profiles & settings
├── data_functions.R # Data loading, aggregation, and caching
├── evaluation.R # Cross-validation and metrics
├── plotting.R # Visualization functions
├── generate_forecast_pdfs.R # Forecast PDF generation and .rds cleanup
├── cache/ # Auto-generated data/model cache folder
├── models/
│ ├── mvgam_baseline.R # Baseline longterm average model
│ ├── mvgam_ar.R # AR model with smooth water covariates
│ ├── mvgam_ar_exog.R # AR with linear covariates
│ ├── mvgam_ar_exog_plus.R # AR with tensor product interactions
│ ├── mvgam_species_specific.R # Species-specific smooth responses
│ ├── mvgam_trait.R # Trait-weighted covariate effects
│ ├── mvgam_trait2.R # Traits + guilds + body size
│ └── fable_models.R # ARIMA, TSLM, ARIMA-exog, GAM
└── results/
└── <run_folder>/
├── forecast_results.rds
├── metrics_only.rds
├── config.rds
├── forecast_pdf_manifest.csv
├── mvgam_crps_skill_over_time.png
├── mvgam_rps_skill_over_time.png
├── mvgam_best_model_counts.png
└── windows/
└── <location>_<test_year>/
├── train_data.rds
├── test_data.rds
└── mvgam/
└── <model>/
├── forecast.pdf # Native forecast plots
└── metrics.rds
#config.R choose
- dgam/fable models you want to use.
- run fable and dgam
- ordinal evaluation or not
- ordinal breaks
- training and testing windown size
main.R:
↓
if sliding_window_breaks = FALSE:
Compute breaks once from full dataset
↓
filter_ordinal_years(ordinal_years) → subset if needed
↓
precomputed_breaks = [species-specific quantiles]
else:
precomputed_breaks = NULL
↓
Pass to fit_sliding_window()
↓
For each window:
↓
Save train_data.rds and test_data.rds to disk
↓
For each model:
Fit model → save forecast.rds → generate native PDF → delete .rds
↓
if precomputed_breaks exists:
Use fixed breaks
else:
Compute from training window
↓
filter_ordinal_years(ordinal_years) → subset if needed
↓
Save metrics_only.rds (small, for cross-scale comparisons)
↓
generate_plots() → skill plots + forecast PDFs
Continuous Ranked Probability Score
↓
Measures: How close is predicted distribution to actual count?
Lower = Better
Root Mean Squared Error measures the difference between point forecasts and observed counts:
RMSE = sqrt(mean((prediction - observed_count)^2))
RMSE is expressed in the same units as the observed counts and penalizes large errors more strongly than small errors.
Lower = Better.
The pipeline calculates RMSE within each cross-validation test window, separately for each model and species, and each region where applicable.
Point forecasts differ between frameworks:
- mvgam: Uses the predictive median (
Estimate, extracted frompredQ50). - fable: Uses the predictive mean (
.mean).
This distinction should be considered when comparing RMSE across frameworks. For a one-year test window with one observation per series, RMSE equals the absolute forecast error.
RMSE is calculated independently of ordinal evaluation: disabling use_ordinal does not disable RMSE.
Ranked Probability Score
↓
Step 1: Define categories from training data quantiles
Low: < 33rd percentile
Medium: 33rd - 50th percentile
High: 50th - 67th percentile
Very High: > 67th percentile
↓
Step 2: Convert predictions to category probabilities
P(Low) = pnorm(threshold_low, mean=prediction, sd=pred_sd)
P(Medium) = pnorm(threshold_med, ...) - P(Low)
...
↓
Step 3: Score predicted probabilities vs actual category
Lower = Better
crps_skill = 1 - (model_crps / baseline_crps)
rmse_skill = 1 - (model_rmse / baseline_rmse)
rps_skill = 1 - (model_rps / baseline_rps)
> 0: Better than baseline
= 0: Same as baseline
< 0: Worse than baseline
A successful baseline forecast is required to calculate skill scores.
Missing baseline scores or baseline scores of zero produce NA
because the ratio cannot be calculated.
Four plots generated per model framework:
- CRPS Skill Over Time - Time series of skill scores by species
- RPS Skill Over Time - Ordinal accuracy evolution
- Combined Metrics - All metrics in faceted grid
- Best Model Counts - Winner frequency by species
The run_all_scales.R wrapper includes available CRPS, RMSE, and RPS
skill scores in:
- Per-model window skill plots by spatial scale.
- Per-model and cross-model window delta plots.
- Marginal distribution plots.
- Density, ECDF, and violin plots of skill deltas.
- Per-window CSV outputs and aggregated summaries.
RMSE appears under the RMSE Skill label.
delta_skill = system_skill - subregion_skill
Positive: System has higher skill relative to its baseline.
Negative: Subregion has higher skill relative to its baseline.
These deltas compare baseline-relative skill, not raw RMSE values.
All settings controlled via config.yml. Key options:
default:
spatial:
level: all # "all", "subregion", or "colony"
run_by_region: false # true = separate model per spatial unit
forecast_totals: false # true = sum counts to a single 'Total' category
include_species: "top6" # "top6", "all", or list e.g., ["wost", "whib"]
include_unknowns: true # true = keep 'unkn', 'smwh', etc. when using 'all'
min_years_required: 10 # Minimum years to include a spatial unit
models:
mvgam: [baseline, ar, ar_exog, ar_exog_plus, species_specific, trait, trait2]
fable: [baseline, arima, tslm, arima_exog, gam]
run_mvgam: true # Enable/disable mvgam framework
run_fable: true # Enable/disable fable framework
use_ordinal: true # true = CRPS + RPS, false = CRPS only
ordinal_breaks: [0.33, 0.50, 0.67] # Category boundaries (quantiles)
sliding_window_breaks: true # true = recompute per window
ordinal_years: All # Years used to define breaks
train_years: 20 # Training window size
test_years: 2 # Test window size
parallel:
enabled: true # Multicore processing
workers: null # null = auto-detect cores
cache:
data: true # Speed up reruns by caching data prep
models: false # Speed up reruns by caching fitted models
chains: 4 # MCMC chains
burnin: 1500 # MCMC warmup iterations
samples: 1500 # MCMC sampling iterationsInstall packages on a login node and run models on a compute node. Both use the shared R package library under /blue.
From the project directory:
source setup_env.sh
Rscript install_packages.R
setup_env.sh loads R/4.6 and sets the package library to /blue/ewhite/alexanderblochel/R/libs. Do not load a separate GDAL module; on this system, it replaces the R module.
srundev --time 2:00:00 --mem 32gb
From the project directory:
source setup_env.sh
Rscript run_all_scales.R
You do not need to reinstall packages on the compute node. If a package is missing, install it on the login node into the shared /blue library, then retry. If an external download fails on the compute node, download or cache that data from the login node first.
On HiPerGator, the final block of run_all_scales.R creates a .zip archive of scale_run_folder for Globus transfer and removes the original results folder. Check that the archive exists and contains the expected files. This archiving step is skipped when running locally.