RASA: Reliability-Aware Selective Adaptation via Hybrid Reliability Threshold

Abstract

Test-time adaptation (TTA) has emerged as a highly promising paradigm for adapting pre-trained models to target domains. However, in open-world scenarios, standard TTA often suffers from noise overfitting due to indiscriminate entropy minimization, especially when faced with severe semantic shifts and out-of-distribution (OOD) samples. To address these challenges, we propose RASA, a Reliability-Aware Selective Adaptation framework that employs a hybrid reliability threshold for selective test-time adaptation. The core of RASA is a novel HistoryAware Thresholding (HAT) module, which integrates hybrid prior information by combining theoretical class priors with running historical entropy statistics, and dynamically adjusts the reliability criteria as the adaptation process progresses. Furthermore, we introduce a dual-stream Transformer architecture that explicitly decouples semantic classification and OOD inference into different feature subspaces, significantly enhancing sensitivity to outliers while maintaining accuracy on in-distribution samples. Extensive experiments demonstrate that RASA consistently outperforms existing state-of-the-art methods, exhibiting superior stability and robustness in open-world environments with severe data perturbations and a high proportion of OOD samples.