Hierarchical demand prediction

[zmek]

Hierarchical demand prediction

Summary

This branch introduces a hierarchical demand prediction system that models hospital bed demand across organisational levels (e.g. subspecialty → reporting unit → division → board → hospital). Instead of predicting demand only at a flat level, the system builds a tree-structured hierarchy, predicts at the bottom (service/subspecialty) level, then aggregates upward using convolution of probability distributions — with statistical capping at each level to control distribution size.

The branch also introduces a flow-based architecture that separately models arrivals and departures (with net flow computed as their difference), and a transfer predictor that estimates inter-service patient transfers.

---

New Modules

File	Lines	Purpose
predict/hierarchy/structure.py	821	Defines Hierarchy, EntityType, HierarchyLevel — generic tree structure with prefixed IDs to avoid name collisions across levels. Supports YAML config loading and dataframe population.
predict/hierarchy/orchestrate.py	623	HierarchicalPredictor — orchestrates a 3-phase algorithm: (1) bottom-up stats with top-down capping, (2) bottom-level PMF prediction, (3) upward aggregation via convolution.
predict/hierarchy/calculate.py	224	Calculates hierarchical statistics (sum of means, combined variance, max_support caps) used in Phase 1 capping.
predict/hierarchy/__init__.py	250	Public API re-exporting all hierarchy components.
predict/service.py	1,393	build_service_data() — prepares per-service prediction inputs from trained models and patient snapshots. Defines flow types (ED current, ED yet-to-arrive, elective/emergency departures, transfers). compute_transfer_arrivals() converts departure PMFs into transfer arrival PMFs.
predict/demand.py	587	DemandPredictor — core mathematical engine. Handles convolution of multiple distributions, PMF truncation with renormalization, and aggregation of child predictions into parent predictions.
predict/distribution.py	334	Distribution — immutable discrete probability distribution with operations: convolution, binomial thinning, net flow (arrivals minus departures), clamping, and statistical queries.
predict/types.py	385	DemandPrediction (PMF + expectation/mode/percentiles), PredictionBundle (arrivals + departures + net flow), FlowSelection (configurable boolean flags for which flow families to include — supports elective-only, emergency-only, custom cohorts).
predictors/transfer_predictor.py	1,135	TransferProbabilityEstimator — sklearn-style estimator that learns transfer probabilities and destination distributions from patient movement data. Supports cohort-aware (elective vs emergency) and subgroup-aware (paediatric, adult male/female, young/senior) analysis.
predictors/subgroup_definitions.py	41	create_subgroup_functions() — 5 standard patient subgroup identifiers extracted from subgroup_predictor.py for reuse.

Modified Modules

• predict/emergency_demand.py — Corrected docstring for find_probability_threshold_index to properly describe P(X >= k) >= threshold semantics.
• predictors/subgroup_predictor.py — Extracted create_subgroup_functions() into subgroup_definitions.py and imported from there.
• predictors/__init__.py — Added exports for new predictors.

New Tests (~74 test cases)

File	Tests	Coverage
test_hierarchy.py	~45	DemandPrediction dataclass, DemandPredictor convolution/capping/net-flow, FlowSelection config, cohort filtering, hierarchy name collision handling, multi-level hierarchical prediction
test_service.py	~12	build_service_data() construction, empirical YTA integration, multi-subgroup predictors, compute_transfer_arrivals() with various transfer scenarios
test_transfer_predictor.py	~17	Fitting, destination distributions, visit deduplication, cohort functionality, transition matrices, error handling

Notebooks

• 4b_Predict_emergency_demand.ipynb — Major update to reflect refactored DemandPrediction class (added mode, expectation, min_beds_with_probability).
• 4c_Evaluate_emergency_demand_predictions.ipynb — Reduced/refactored (540 line diff, net deletion).
• 4d_Predict_emergency_demand_with_special_categories.ipynb — Extended with new evaluation plots.
• 4e_Generate_predictions_using_hierarchy.ipynb — New notebook (910 lines) demonstrating the full hierarchical prediction workflow end-to-end.

Configuration

• config.yaml — Added hierarchy level definitions (specialty → division → hospital).
• hierarchy_config.yaml — New example config for a deeper hierarchy (subspecialty → reporting unit → division → board → hospital).

---

Key Design Decisions

1. 3-phase algorithm — Separates statistical capping (top-down), bottom-level prediction, and upward aggregation for clarity and correctness.
2. Flow-based architecture — Models arrivals and departures independently with net flow as their convolution difference, enabling flexible cohort/flow selection.
3. Generic hierarchy — The tree structure is not hospital-specific; entity types and levels are configurable via YAML.
4. Prefixed IDs — Internal "entity_type:entity_name" keys prevent name collisions when different hierarchy levels share names, while the public API uses clean names.
5. Distribution algebra — Immutable Distribution class centralizes PMF operations (convolution, thinning, net flow) with proper truncation and renormalisation.
6. Transfer modelling — Inter-service transfers are predicted from historical patterns and folded into the flow-based system as arrival sources.