학술

Prediction of a task-specific AI-reliance choice from facial expression and galvanic skin response signals

PLOS ONE
CC BY
이 매체는 공공·자유 라이선스로 본문을 직접 표시합니다.

Figures
Abstract
Background
Reliance on AI-supported health recommendations is consequential, but a single behavioral source-selection decision is not equivalent to multidimensional psychological trust. Biosignals may contain predictive information before a choice, although small-sample multimodal models require leakage-free evaluation and cautious interpretation.
Purpose
This study evaluated whether facial-expression probabilities and galvanic skin response (GSR) features predict a task-specific binary choice of an AI-labeled rather than doctor-labeled recommendation in a simulated ADHD assessment. The outcome is treated as a behavioral indicator of reliance rather than a direct measure of psychological trust. Method: This cross-sectional laboratory analysis included a cohort of 60 participants (40 AI-labeled and 20 doctor-labeled choices). Eight facial-expression probabilities and two GSR phasic features were evaluated in prespecified Early (6–3 s) and Proximal (3–0 s) windows. Ten repeats of participant-level stratified five-fold cross-validation were used. Scaling and SMOTETomek resampling occurred only within training folds, and the logistic stacking meta-learner was trained on strictly cross-fitted base-model probabilities. Model-specific 95% confidence intervals and paired AUC differences were estimated with 10,000 stratified participant bootstrap samples; five paired tests were Holm-adjusted.
Results
Early-window ROC-AUCs were 0.643 (95% CI 0.490–0.786) for Face, 0.430 (0.287–0.578) for GSR, and 0.526 (0.375–0.679) for stacking. Proximal-window ROC-AUCs were 0.550 (0.399–0.699), 0.381 (0.240–0.530), and 0.376 (0.229–0.529), respectively. Stacking did not outperform either modality. Early stacking was worse than Early Face (ΔAUC = −0.116, 95% CI −0.183 to −0.056; Holm-adjusted p = 0.002); the other adjusted paired comparisons were not significant.
Conclusion
Multimodal stacking did not improve prediction in this small exploratory sample. These results do not establish either a general advantage or a lack of value of multimodal fusion. They do not support deployment or direct trust detection. Larger studies with verified synchronization, repeated decisions, independent trust measures, and external validation are required.
Citation: Shamszare H, Choudhury A (2026) Prediction of a task-specific AI-reliance choice from facial expression and galvanic skin response signals. PLoS One 21(11): e0363436. https://doi.org/10.1371/journal.pone.0363436
Editor: Iftikhar Ahmed Khan, University of Lahore - Raiwind Road Campus: The University of Lahore, PAKISTAN
Received: May 11, 2026; Accepted: November 7, 2026; Published: November 20, 2026
Copyright: © 2026 Shamszare, Choudhury. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: West Virginia University Internal Seed Grant (#3086). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
AI-supported mental-health applications are being explored in response to unmet mental-health needs and limited access to professional care [1–3]. Their usefulness depends partly on how people respond to their recommendations. Accepting a recommendation may reflect confidence in its source, agreement with its content, or both. Understanding that distinction is necessary when studying reliance on AI-supported health information.
Occupational stress is one context in which mental-health support is relevant [4], and AI applications have been discussed in relation to provider workload and cognitive offloading [5,6]. The present study, however, evaluates a laboratory decision made by engineering students. It does not measure occupational stress, workplace mental-health outcomes, or effects on professional performance.
Public concern about AI-supported health care remains substantial. In a 2023 U.S. survey, 60% of adults reported that they would be uncomfortable if their provider relied on AI in their care [7]. Inappropriate reliance can also introduce risks [8]. These concerns motivate examining behavioral source selection while distinguishing the observed decision from the broader psychological construct of trust.
Trust refers here to a psychological attitude concerning a source’s competence and dependability under uncertainty [9–12]. Reliance refers to the observable decision to select the AI-labeled recommendation in this task. A source-selection choice may be related to trust, but it is not a validated measurement of trust intensity or calibration. Calibrated trust requires that reliance be appropriate to a system’s actual reliability [10,13]. The ADHD pathway provided a standardized questionnaire and a defined source-selection event, allowing signals to be summarized before one decision. The task was used to study this choice, rather than to evaluate ADHD diagnosis or occupational mental health [14–16].
One proposed approach is to estimate whether a user is likely to rely on an AI recommendation before behavioral commitment. Such estimates might eventually support adaptive explanations or confidence cues, but this possibility requires prospective validation and should not be equated with continuous measurement of psychological trust, [11,12,17–20].
Facial behavior and GSR have been investigated as correlates of cognitive and affective states relevant to human–AI interaction [21–23]. Combining modalities could, in principle, capture complementary information. Whether these signals predict a subsequent reliance choice, and whether fusion improves prediction, must be established under participant-independent, leakage-controlled evaluation.
Prior research leaves open whether facial and physiological signals recorded before a decision contain reproducible participant-level information about a later AI-versus-human source-selection choice. Existing work also provides limited evidence about the effect of temporal-window definition and the incremental value of multimodal fusion. The present study addresses these questions in a controlled simulated ADHD-assessment task while treating the single behavioral outcome as an indicator of reliance.
This study evaluated whether facial-expression probabilities and phasic GSR features contain information about a subsequent AI-labeled versus doctor-labeled source choice in a simulated ADHD assessment. It also evaluated whether cross-fitted multimodal stacking improves discrimination over either modality alone and whether results vary with the duration of the predecision window. The analysis concerns a single task-specific reliance choice and provides exploratory internal validation.
Methods
Ethics and study design
Application and decision task
My Friendly Mind is a custom web-based mental-health screening and education application containing separate depression and ADHD pathways [14,15]. The depression pathway administered the Patient Health Questionnaire-9 (PHQ-9), and the ADHD pathway administered the Adult ADHD Self-Report Scale (ASRS) [16]. Participants created an account, completed the relevant screening questionnaire, reviewed their screening results, and accessed brief educational material. The present analysis concerns the ADHD/ASRS source-selection decision. After completing the ASRS, participants viewed outcomes side-by-side, one labeled AI-generated and the other doctor-generated, and selected the source they trusted more. In both the PHQ-9 and ASRS pathways, the doctor-labeled outcome was concordant with the participant’s questionnaire score, whereas the AI-labeled outcome was randomly selected from plausible alternatives and therefore only occasionally matched the score-concordant outcome. Participants were blinded to this programmed manipulation. Because source label and outcome concordance were not independently randomized, the selection was analyzed as a task-specific behavioral indicator of reliance rather than as a validated measure of multidimensional psychological trust.
Participants and analytic cohort
Participants were recruited through a university-wide mass email in rural Appalachia. Eligibility for the broader My Friendly Mind study required active enrollment in the College of Engineering and self-reported signs of depression and/or ADHD [14,15]. Eligible participants attended an in-person session in a controlled on-campus laboratory and used the application for approximately 20–30 minutes. Eighty participants completed the broader experiment. Overall, 20/80 participants (25.0%) were excluded. Two lacked a linked choice-metadata, facial-video, and GSR record. Among the 78 linked records, GSR processing at cohort selection did not yield usable features for 17 participants; one further GSR-eligible record did not complete facial processing. The resulting fixed complete-case cohort comprised 60 participants with complete, processable information for every required modality. Among linked records, 18/78 (23.1%) were excluded. Missing or unusable modality features were not imputed. The same cohort was used for all primary and sensitivity analyses; no participant was selected according to outcome or model performance.
The analytic cohort comprised 29 participants recorded as male and 31 recorded as female in the demographic file, with a mean age of 22 years. Among the 78 linked records, the AI-labeled choice occurred in 40/60 included participants (66.7%) and 10/18 excluded participants (55.6%); a two-sided Fisher exact test did not detect a difference in outcome distribution (p = 0.413). Individual demographic data for excluded records were unavailable, so demographic differences between included and excluded records could not be evaluated. The nonsignificant outcome comparison does not establish equivalence of the included and excluded groups or exclude selection bias.
Signal acquisition and retained records
Facial video and GSR were recorded contemporaneously while participants completed the browser-based task in a quiet, controlled university laboratory. GSR was acquired continuously with the wireless Bitbrain Ring electrodermal-activity device; the exported data provided an effective sampling rate of 32 Hz [15]. The retained analysis materials comprise decision-relative facial-video clips, GSR comma-separated-value files containing sequence and conductance channels, and choice metadata. The dissertation identifies the GSR device and sampling rate but does not report the precise electrode placement, camera manufacturer and model, image resolution, frame rate, or acquisition-software versions. Those specifications are therefore not inferred. The Supporting Information enables computational reproduction from the retained deidentified feature data.
Sample-size consideration
The final sample size was determined by availability of complete, processable data across every required modality, not by the observed model performance. With one outcome per participant and internal validation only, the study remains exploratory. To reduce analytic flexibility and dimensionality, the primary analysis used eight prespecified facial probabilities, two prespecified GSR features, and one regularized logistic model for each base and meta stage; it did not search across classifiers, hyperparameters, feature subsets, or thresholds. Precision is conveyed through confidence intervals rather than a post hoc power calculation.
Temporal windows and synchronization
The decision-relative facial clips placed the selection moment at 10 s. The primary Early window used frames at 4, 5, and 6 s (6–3 s before selection), and the Proximal window used frames at 7, 8, and 9 s (3–0 s before selection). GSR decision time was derived from the metadata selection timestamp. Video-duration strings were parsed as minutes and seconds. The Bitbrain Ring export stored two GSR observations for each 16-Hz sequence packet; sample time was reconstructed as (sequence − first sequence)/16 plus 0 or 1/32 s within each packet, yielding the effective 32-Hz rate reported in the dissertation [15]. Consistent with the acquisition workflow, estimated GSR start lag was defined as 0.70 × (video duration − GSR duration), and the metadata decision time was shifted by that lag. This explicit heuristic was used for every participant and all reported analyses. Its uncertainty is addressed as a limitation rather than as verified clock synchronization. No shared synchronization pulse or independent clock comparison was retained. The 1–5-s duration analysis varies window length while retaining the same timing heuristic; it does not validate clock alignment or quantify sensitivity to a timing offset. The deposited feature-level table cannot support re-extraction after shifting the GSR timestamps, so an offset sensitivity analysis could not be conducted from that table. An alignment error could move samples into a different decision-relative interval and could weaken or alter the estimated predictive associations.
Facial feature extraction and quality control
One frame per second was decoded from each decision-relative clip with FFmpeg [24]; the acquisition-time FFmpeg version was not retained. UltraFace RFB-320 was the primary face detector in the prespecified image region (confidence ≥0.40) [25]. Detected boxes were padded by 12%, converted to 64 × 64 grayscale crops, and evaluated with the pretrained FERPlus model [26]. To retain the fixed cohort without using outcome information, a MediaPipe Face Mesh crop was used as a detector fallback for the one fixed-cohort participant for whom UltraFace did not produce a complete set of valid crops [27]. MediaPipe 0.10.21 and ONNX Runtime C++ 1.29.0 were used in feature extraction. Eight FERPlus probabilities (neutral, happiness, surprise, sadness, anger, disgust, fear, and contempt) were averaged within each prediction window. Quality-control rules and fallback status were determined without reference to the outcome.
GSR feature extraction and quality control
GSR values were converted to microsiemens. To prevent post-decision leakage, preprocessing used only samples available by the end of the relevant prediction window; full-record normalization was not performed. Up to 60 s of continuous pre-endpoint history was interpolated to a regular 32-Hz grid. NeuroKit2 0.2.12 cleaning and phasic processing [28] were reproduced using a fourth-order 3-Hz Butterworth low-pass filter followed by a second-order 0.05-Hz high-pass filter, both applied with zero-phase filtering. The two analyzed features were the area under the absolute phasic component within each window (computed using trapezoidal integration with a 1/32-s sample interval) and the maximum absolute phasic amplitude in that window. They correspond to gsr_phasic_abs_area and gsr_phasic_abs_max in S1 File and to the labels in Fig 4. These electrodermal measures index sympathetic arousal but are not specific to trust or any single emotion [29,30].
Predictive modeling
All splits were at the participant level. The primary evaluation used 10 repeats of stratified five-fold cross-validation (50 outer fits; master seed 20260825, with deterministic fold-derived seeds). Separate Face and GSR pipelines implemented MinMaxScaler, SMOTETomek with SMOTE k = 3, and L2-regularized logistic regression (C = 1, solver = lbfgs, max_iter = 3000) using scikit-learn 1.8.0 and imbalanced-learn 0.14.2 [31–33]. Scaling and resampling were fitted only to training participants. Resampling modified the training data only; the fitted scaler and model were applied to the untouched test fold. No data-dependent feature selection or hyperparameter tuning was performed. The decision threshold was prespecified at 0.50, with the AI-labeled choice as the positive class.
For stacking, a separate stratified five-fold cross-fit was performed inside each outer-training set. Each training participant’s two meta-features were base-model probabilities generated by models that had not been fitted on that participant. A StandardScaler and class-weighted L2 logistic meta-learner (C = 1, solver = lbfgs, max_iter = 3000) were fitted only to these cross-fitted outer-training meta-features. The two base pipelines were then fitted on the full outer-training set to generate Face and GSR probabilities for the outer-test participants; the training-fitted meta-feature scaler and meta-learner transformed those probabilities into the stacked outer-test prediction. Thus, neither preprocessing, resampling, base-model fitting, nor meta-model fitting used the outer-test participants.
Statistical analysis
Each participant received one out-of-fold probability in each repeat; the 10 probabilities were averaged for the primary participant-level estimate. We report ROC-AUC, accuracy, balanced accuracy, F1, positive predictive value, sensitivity, specificity, negative predictive value, Matthews correlation coefficient, and Brier score at the prespecified 0.50 threshold. Model-specific 95% percentile confidence intervals were obtained from 10,000 stratified participant bootstrap samples (seed 42). Paired ROC-AUC differences were calculated within the same bootstrap resamples from the same participant-level averaged out-of-fold probabilities. Two-sided empirical p-values used twice the smaller bootstrap tail probability and were capped at 1. Five prespecified comparisons were adjusted by the Holm procedure [34]. No significance test was used to select the analytic cohort, features, windows, or model. As a prespecified sensitivity analysis, the complete pipeline was repeated for 1-, 2-, 3-, 4-, and 5-s windows using the same fixed N = 60 cohort, cross-validation splits, preprocessing, resampling, and modeling rules. These bootstrap intervals summarize uncertainty conditional on the fitted out-of-fold predictions; they do not incorporate all uncertainty from participant recruitment, signal alignment, or repeating the complete training process in a new sample.
Reproducibility
S1 File contains the deidentified derived features, participant-level averaged and repeat-specific out-of-fold probabilities, primary metric intervals, paired comparisons, duration sensitivity results, coefficient summaries, and the executable statistical and figure-generation scripts. The package includes deterministic outer- and inner-fold assignments, a data dictionary, reference outputs, and a verification script. Raw recordings and a complete raw-signal extraction workflow are not included, so reproduction begins from the deposited derived features. The analysis dependencies are NumPy 2.3.5, pandas 2.2.3, SciPy 1.17.0, scikit-learn 1.8.0, imbalanced-learn 0.14.2, and Matplotlib 3.10.8, with Python 3.12. The complete reproducibility package is publicly available at https://github.com/avishekadamas-glitch/ai-reliance-multimodal-plos-one (version 2.0.0). Reporting was checked against the STROBE statement [35]; the checklist is supplied as S2 File.
Results
Participant flow and evaluation cohort
Of 80 participants who completed the broader experiment, 78 had linked choice metadata, facial video, and GSR files. At cohort selection, GSR processing did not yield usable features for 17 linked records, and one additional GSR-eligible record did not complete facial processing. The exclusions represent 25.0% of the 80 participants and 23.1% of the 78 linked records. The resulting fixed complete-case cohort was N = 60 (40 AI-labeled and 20 doctor-labeled choices) and was used unchanged for the primary and sensitivity analyses. Fig 1 summarizes this flow, and Table 1 reports the cohort characteristics.
Of 80 participants completing the broader experiment, 78 had linked choice metadata, facial video, and GSR records. Twenty were excluded sequentially for the reasons shown, leaving the fixed N = 60 cohort used for every analysis. GSR, galvanic skin response.
Among the 78 linked records, the proportion choosing the AI-labeled recommendation was 66.7% in the included cohort and 55.6% among the 18 excluded linked records (two-sided Fisher exact p = 0.413). This outcome-only comparison has limited sensitivity in a small sample and does not exclude selection bias. Demographic differences could not be tested because excluded-record demographics were unavailable.
Primary predictive performance
Tables 2–4 report discrimination and overall classification metrics from the participant-level probabilities averaged across repeated cross-validation. The Early Face model had the highest primary ROC-AUC (0.643), but its 95% confidence interval included 0.50. The GSR models showed moderate-to-high sensitivity but low specificity, demonstrating why accuracy and F1 alone were misleading under the 40:20 class imbalance. Neither stacked model improved on Face. ROC curves and thresholded confusion matrices are shown in Figs 2 and 3, respectively. A classifier that always predicts the AI-labeled choice would achieve 40/60 = 66.7% accuracy, exceeding all six primary model accuracies (48.3–63.3%). That rule would have balanced accuracy of 0.50 and no ability to discriminate between participants. ROC-AUC, balanced accuracy, MCC, and their intervals therefore guide interpretation rather than raw accuracy alone.
(A) Early window, 6–3 s before selection. (B) Proximal window, 3–0 s before selection. Curves use participant-level out-of-fold probabilities averaged over 10 repeats of five-fold cross-validation. The dashed diagonal denotes chance discrimination. AUC, area under the curve; GSR, galvanic skin response.
Panels show Early Face (A), Early GSR (B), Early Stacked (C), Proximal Face (D), Proximal GSR (E), and Proximal Stacked (F). Rows represent observed source choice and columns represent predicted source choice. Counts are based on participant-level probabilities averaged across the 10 repeated cross-validation partitions. AI-labeled choice was the positive class; GSR, galvanic skin response.
Paired AUC comparisons
Early stacking was worse than Early Face (ΔAUC = −0.116, 95% CI −0.183 to −0.056; Holm-adjusted p = 0.002). The Proximal stacked-versus-Face comparison was negative but did not remain significant after multiplicity adjustment (ΔAUC = −0.174; adjusted p = 0.190). Stacking did not differ significantly from GSR in either window. The Early-versus-Proximal stacked difference was also not significant after adjustment (ΔAUC = 0.150; adjusted p = 0.296). Thus, no comparison after multiplicity adjustment supported multimodal superiority or a reliable temporal advantage. The complete set of prespecified paired comparisons is reported in Table 5.
Coefficient summaries
Descriptive scaled logistic coefficients varied across folds, and their uncertainty was substantial; they are not interpreted as stable biomarkers or causal effects. Foldwise coefficient summaries are shown in Fig 4.
Panels show Early Face (A), Early GSR (B), Proximal Face (C), and Proximal GSR (D). Bars are mean coefficients across 50 outer fits and error bars are between-fit standard deviations. Coefficients are descriptive predictive associations and are not interpreted as stable biomarkers or causal effects. The two GSR predictors are absolute phasic area and maximum absolute phasic amplitude. GSR, galvanic skin response.
Temporal-window sensitivity analysis
Across the 1–5-s sensitivity analysis, the prespecified 3-s duration was not the best-performing duration. The highest stacked ROC-AUC was 0.635 for the 4-s Early window and 0.420 for the 4-s Proximal window. All stacked estimates were weak or, for the Proximal window, below chance. Complete results are reported in Table 6 and Fig 5. These descriptive duration comparisons do not provide an independent validation of the selected window or assess errors in cross-modality synchronization.
(A) Early-window and (B) Proximal-window ROC-AUC estimates for 1–5-s durations. The vertical dashed line identifies the prespecified 3-s primary duration; the horizontal dotted line denotes chance discrimination. ROC-AUC, area under the receiver-operating-characteristic curve; GSR, galvanic skin response.
Discussion
Principal findings
In this small exploratory cohort, multimodal stacking did not improve discrimination of the reliance choice. The strongest primary model was Early Face, but its confidence interval included chance discrimination. Early stacking performed worse than Early Face in the adjusted paired comparison. These findings do not demonstrate reliable predictive ability or generalizability, and they do not establish that fusion has no value. Limited sample size, signal quality, uncertain alignment, and the single-choice design may have restricted complementary information across modalities.
Class imbalance makes raw accuracy and F1 difficult to interpret. The majority-choice rule described in the Results exceeds every primary model’s accuracy, and the GSR models show poor specificity. ROC-AUC, balanced accuracy, MCC, and the class-specific intervals provide a clearer account of the weak and uncertain performance. Brier scores describe probabilistic error and do not, on their own, establish calibration.
Temporal interpretation
The Early stacked AUC was numerically higher than the Proximal stacked AUC, but the paired difference was not significant after Holm adjustment. The sensitivity analysis also showed that the prespecified 3-s duration was not uniquely supported: a 4-s Early window yielded the highest stacked AUC, whereas Proximal estimates remained below chance across durations. These windows should be understood as analytical intervals rather than validated stages of trust formation. GSR response latency and the absence of directly verified synchronization further limit physiological timing claims [29,30]. Varying window duration cannot resolve an unknown offset between recordings. Future acquisition should record a shared event marker and verify its alignment in both streams before any physiological timing interpretation.
Facial and physiological interpretation
The eight facial-expression probabilities offered modestly more predictive information than the two phasic GSR features, but the confidence intervals were wide and the foldwise coefficients were unstable. This is a within-sample observation rather than evidence of a general advantage of facial signals. Facial-expression model outputs should not be treated as direct readouts of emotion or trust [36,37], and phasic electrodermal features are nonspecific measures of sympathetic arousal [29,30]. Multimodal fusion can improve prediction when modalities contribute complementary and reliable information [38–42]; the present data do not establish whether that condition would hold with larger samples and improved measurements.
Construct validity and scope
The behavioral target was one source-selection decision in a simulated application. Because recommendation source and correctness were not independently randomized, the outcome cannot separate trust in the source from perceived correctness, plausibility, or response preference. It does not measure trust intensity, distrust, confidence, over-reliance, under-reliance, or calibration. Future studies should include an independent validated trust questionnaire as well as repeated behavioral decisions. Orthogonal manipulation of source label and objective reliability is needed to distinguish perceived correctness from trust and to examine within-person trust dynamics.
Limitations
The small, imbalanced convenience sample of engineering students and the lack of external validation restrict generalizability. Twenty of the 80 participants were excluded for absent or unusable multimodal records; demographic differences could not be assessed, and the outcome-only comparison cannot rule out selection bias. SMOTETomek was confined to training folds, but its stability is uncertain with only 20 doctor-labeled choices. Outer-training folds contain 16 minority observations, and inner stacking-training subsets contain only 12–13; synthetic sampling and Tomek-link removal may therefore depend strongly on a few observations. No resampling-method comparison was performed, so its benefit or stability at this sample size is not established. Repeated cross-validation does not substitute for independent replication. Each participant contributed one ADHD source-selection outcome, preventing estimation of within-person variability. The facial model was not validated for trust or for this population. The doctor-labeled outcome was questionnaire-concordant and the AI-labeled outcome was randomized, so source selection may reflect perceived correctness or personal relevance as well as reliance. Finally, alignment relied on an unverified heuristic. An unknown timing offset could weaken discrimination, change the apparent comparison between windows, or include samples outside the intended predecision interval. The magnitude and direction of this effect cannot be quantified from the deposited derived features.
The next steps are larger and more diverse cohorts, verified synchronization, independent trust measures, and repeated decisions under independently manipulated source labels and reliability. A locked model should then be evaluated in an external cohort. Practical applications should be considered only after these measurement and validation requirements are met; the present findings do not support clinical, occupational, real-time, or safety-critical deployment.
Conclusion
Using participant-level repeated cross-validation, training-fold-only preprocessing and resampling, and cross-fitted stacking, this exploratory study found no evidence that multimodal fusion improved prediction of one task-specific AI-reliance choice in N = 60 participants. Predictive performance was weak and uncertain, and the results establish neither a general advantage nor a lack of value of fusion. The observed choice is a behavioral indicator of reliance, not a direct measure of psychological trust. Larger studies with verified synchronization, repeated decisions, independent trust measures, and external validation are needed before practical use is considered.
AI disclosure
During preparation of the revised manuscript, the authors used generative AI tools to assist with language editing and code auditing. All analytical decisions, code, outputs, citations, and manuscript statements are reviewed and approved by the authors, who retain full responsibility for the work.
Supporting Information
S1 File. Deidentified derived data and executable statistical analysis code.
This ZIP archive contains the feature-level data, deterministic outer- and inner-fold assignments, participant-level out-of-fold predictions, reference results, verification code, figure-generation code, and documentation needed to reproduce the reported statistical results, tables, and figures from derived features. Raw-signal feature extraction is outside the scope of this package.
https://doi.org/10.1371/journal.pone.0363436.s001
(ZIP)
S2 File. Completed STROBE checklist.
The checklist identifies where each item for a cross-sectional observational study is reported in the manuscript.
https://doi.org/10.1371/journal.pone.0363436.s002
(DOCX)
Acknowledgments
We thank Dr. Yeganeh Shahsavar for designing and developing the My Friendly Mind application and for leading the broader Phase 1 study documented in her dissertation.
References
- 1.
WHO. Mental health. World Health Organization. Available from: https://www.who.int/health-topics/mental-health#tab=tab_2
- 2.
NIHCM. The behavioral health care workforce. NIHCM. 2023. Available from: https://nihcm.org/publications/the-behavioral-health-care-workforce-shortages-solutions
- 3. Babu A, Joseph AP. Artificial intelligence in mental healthcare: transformative potential vs. the necessity of human interaction. Front Psychol. 2024;15:1378904. pmid:39742049
- 4. Kelloway EK, Barling J. Job characteristics, role stress and mental health. Journal of Occupational Psychology. 1991;64(4):291–304.
- 5. Pavuluri S, Sangal R, Sather J, Taylor RA. Balancing act: the complex role of artificial intelligence in addressing burnout and healthcare workforce dynamics. BMJ Health Care Inform. 2024;31(1):e101120. pmid:39181545
- 6. Gerlich M. AI Tools in Society: Impacts on Cognitive Offloading and the Future of Critical Thinking. Societies. 2025;15(1):6.
- 7. Tyson A. 60% of Americans Would Be Uncomfortable With Provider Relying on AI in Their Own Health Care. Pew Research Center. 2023. Available from: https://www.pewresearch.org/science/2023/02/22/60-of-americans-would-be-uncomfortable-with-provider-relying-on-ai-in-their-own-health-care/
- 8.
Yu S, Carroll F, Bentley BL. Trust and risk: Psybersecurity in the AI era. Psybersecurity. CRC Press. 2024. p. 130–55.
- 9. Kostick-Quenet K, Lang BH, Smith J, Hurley M, Blumenthal-Barby J. Trust criteria for artificial intelligence in health: normative and epistemic considerations. J Med Ethics. 2024;50(8):544–51. pmid:37979976
- 10. Lee JD, See KA. Trust in automation: designing for appropriate reliance. Hum Factors. 2004;46(1):50–80. pmid:15151155
- 11. Yang R, Wibowo S. User trust in artificial intelligence: A comprehensive conceptual framework. Electron Markets. 2022;32(4):2053–77.
- 12.
McKnight HD, Chervany NL. Trust and distrust definitions: One bite at a time. Trust in cyber-societies: Integrating the human and artificial perspectives. Springer. 2001. p. 27–54.
- 13. Naiseh M, Cemiloglu D, Al Thani D, Jiang N, Ali R. Explainable Recommendations and Calibrated Trust: Two Systematic User Errors. Computer. 2021;54(10):28–37.
- 14. Shahsavar Y, Choudhury A. Assessing user acceptance of a mental health app & its impact on depression and attention deficit hyperactivity disorder related knowledge: A mixed method experimental study. Journal of Behavioral and Cognitive Therapy. 2024;34(4):100508.
- 15. Shahsavar Y. Leveraging physiological signal activity and self-report data to assess students’ trust in “My Friendly Mind” app and its impact on their mental health knowledge: a mixed-method phase 1 clinical trial focusing on depression and attention deficit hyperactivity disorder from a human-factors standpoint. Morgantown (WV): West Virginia University. 2025.
- 16. Kessler RC, Adler L, Ames M, Demler O, Faraone S, Hiripi E, et al. The World Health Organization Adult ADHD Self-Report Scale (ASRS): a short screening scale for use in the general population. Psychol Med. 2005;35(2):245–56. pmid:15841682
- 17. Okamura K, Yamada S. Adaptive trust calibration for human-AI collaboration. PLoS One. 2020;15(2):e0229132. pmid:32084201
- 18. Sakamoto T, Harada Y, Shimizu T. Facilitating Trust Calibration in Artificial Intelligence-Driven Diagnostic Decision Support Systems for Determining Physicians’ Diagnostic Accuracy: Quasi-Experimental Study. JMIR Form Res. 2024;8:e58666. pmid:39602469
- 19.
Ahmad M, Alzahrani A, Robinson S, Rahat A. Modelling Human Trust in Robots During Repeated Interactions. In: International Conference on Human-Agent Interaction, 2023. 281–90. https://doi.org/10.1145/3623809.3623892
- 20. He X, Zheng X, Ding H. Existing Barriers Faced by and Future Design Recommendations for Direct-to-Consumer Health Care Artificial Intelligence Apps: Scoping Review. J Med Internet Res. 2023;25:e50342. pmid:38109173
- 21. Akash K, Hu W-L, Jain N, Reid T. A Classification Model for Sensing Human Trust in Machines Using EEG and GSR. ACM Trans Interact Intell Syst. 2018;8(4):1–20.
- 22.
Alzahrani A, Nasir J, Tayeb AJ, André E, Ahmad MI. What do the face and voice reveal? Investigating trust dynamics during human-robot interaction. In: 2025.
- 23. Sajno E, Bartolotta S, Tuena C, Cipresso P, Pedroli E, Riva G. Machine learning in biosignals processing for mental health: A narrative review. Front Psychol. 2023;13:1066317. pmid:36710855
- 24.
FFmpeg Developers. FFmpeg multimedia framework. Available from: https://ffmpeg.org/
- 25.
Linzaer. Ultra-Light-Fast-Generic-Face-Detector-1MB: lightweight face detector and ONNX models [software]. GitHub repository. Available from: https://github.com/Linzaer/Ultra-Light-Fast-Generic-Face-Detector-1MB
- 26.
Barsoum E, Zhang C, Ferrer CC, Zhang Z. Training deep networks for facial expression recognition with crowd-sourced label distribution. In: Proceedings of the 18th ACM International Conference on Multimodal Interaction, 2016. 279–83. https://doi.org/10.1145/2993148.2993165
- 27.
Kartynnik Y, Ablavatski A, Grishchenko I, Grundmann M. Real-time facial surface geometry from monocular video on mobile GPUs. In: 2019. https://doi.org/10.48550/arXiv.1907.06724
- 28. Makowski D, Pham T, Lau ZJ, Brammer JC, Lespinasse F, Pham H, et al. NeuroKit2: A Python toolbox for neurophysiological signal processing. Behav Res Methods. 2021;53(4):1689–96. pmid:33528817
- 29. Boucsein W, Fowles DC, Grimnes S, Ben-Shakhar G, roth WT, Dawson ME, et al. Publication recommendations for electrodermal measurements. Psychophysiology. 2012;49(8):1017–34. pmid:22680988
- 30.
Boucsein W. Electrodermal activity. 2nd ed. New York: Springer. 2012.
- 31. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O. Scikit-learn: machine learning in Python. Journal of Machine Learning Research. 2011;12:2825–30.
- 32. Lemaître G, Nogueira F, Aridas CK. Imbalanced-learn: a Python toolbox to tackle the curse of imbalanced datasets in machine learning. Journal of Machine Learning Research. 2017;18(17):1–5.
- 33. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP. SMOTE: Synthetic Minority Over-sampling Technique. J Artif Intell Res. 2002;16:321–57.
- 34. Holm S. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics. 1979;6(2):65–70.
- 35. von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. PLoS Med. 2007;4(10):e296. pmid:17941714
- 36. Ekman P, Friesen WV, O’Sullivan M, Chan A, Diacoyanni-Tarlatzis I, Heider K, et al. Universals and cultural differences in the judgments of facial expressions of emotion. J Pers Soc Psychol. 1987;53(4):712–7. pmid:3681648
- 37. Barrett LF, Adolphs R, Marsella S, Martinez AM, Pollak SD. Emotional Expressions Reconsidered: Challenges to Inferring Emotion From Human Facial Movements. Psychol Sci Public Interest. 2019;20(1):1–68. pmid:31313636
- 38.
Bhatti A, Behinaein B, Rodenburg D, Hungler P, Etemad A. Attentive cross-modal connections for deep multimodal wearable-based emotion recognition. In: 2021 9th international conference on affective computing and intelligent interaction workshops and demos (ACIIW). IEEE. 2021.
- 39.
Zhang T. Multi-modal fusion methods for robust emotion recognition using body-worn physiological sensors in mobile environments. In: 2019 International Conference on Multimodal Interaction. 2019.
- 40. Ham S-M, Lee H-M, Lim J-H, Seo J. A Negative Emotion Recognition System with Internet of Things-Based Multimodal Biosignal Data. Electronics. 2023;12(20):4321.
- 41. D’mello SK, Kory J. A Review and Meta-Analysis of Multimodal Affect Detection Systems. ACM Comput Surv. 2015;47(3):1–36.
- 42. Zou C, Deng Z, He B, Yan M, Wu J, Zhu Z. Emotion classification with multi‐modal physiological signals using multi‐attention‐based neural network. Cognitive Comp and Systems. 2024;6(1–3):1–11.

관련 뉴스