A hierarchical reinforcement learning approach to personalized decision-making for brain connectivity segmentation

초록

The automation of empirical decision-making in complex biological signals is a prerequisite for the development of autonomous expert systems that eliminate human-induced bias and promote consistent and reproducible segmentation across subjects. In Major Depressive Disorder (MDD) diagnostics using dynamic functional connectivity (dFC), a parameter-selection bottleneck occurs when fixed temporal scales fail to capture the high inter-subject variability and non-stationary nature of neural dynamics. To resolve this, we propose a domain-algorithm co-design that instantiates Hierarchical Reinforcement Learning (HRL) for personalized dFC segmentation, translating the clinical phenomena of temporal rigidity and transient connectivity transitions into a two-level decision architecture. The macro-agent selects window sizes that match each subject's characteristic state dwell-time, while the micro-agent selects step ratios that control the sampling density of transient state transitions, decomposing the combinatorially explosive joint (window & times; step) action space into a tractable coordinated hierarchy. Three clinically constrained design decisions collectively adapt the generic h-DQN to the dFC setting: an asymmetric reward that penalizes false positives more severely than false negatives to avoid MDD over-diagnosis, a ratio-based step parameterization that enforces scale invariance across acquisition protocols, and an asymmetric credit-assignment structure that aligns window-level strategic decisions with step-level tactical refinements. Validated on multi-site MDD datasets across three backbone classifiers, the framework consistently outperforms fixed-parameter and flat RL baselines: F1 improves by up to 5.56 percentage points over the best fixed-parameter baseline within site, by up to 3.2 percentage points in cross-site generalization (Site20 -> Site1/21), and by approximately 9 percentage points over flat RL variants. The framework further generalizes to independent cohorts, yielding +4.04 and +4.61 F1 gains on ABIDE I (ASD) and ADNI (MCI), respectively, and the learned policy transfers to independently trained classifiers with a +3.19 F1 gain over fixed-parameter baselines, indicating that the policy captures subject-adaptive temporal structure rather than classifier-specific inductive biases. Furthermore, the learned policies uncover distinct segmentation regimes in MDD patients that reflect temporal rigidity and reveal a transient sparse-connectivity state (State 5) preferentially occupied by healthy controls, providing explainable biomarkers without apriori hypotheses. These results demonstrate that the framework serves as a principled expert system for subject-adaptive segmentation and as a scalable template for personalized, adaptive time-series analysis in precision psychiatry.

키워드