학술

Self-Rewarded Reinforcement Learning for Better-Structured Medical Explanations in Small Language Models

arXiv CS.AI
CC BY
이 매체는 공공·자유 라이선스로 본문을 직접 표시합니다.

Abstract

Small language models (SLMs) can run on commodity hardware and support privacy-preserving, on-premises clinical deployment, but their medical explanations are often short and poorly structured compared with large language models.

Recent supervised approaches such as Meerkat transfer chain-of-thought reasoning into 7B-8B backbones, yet their training objective does not directly target the style or organization of the generated rationales.

Building on the instruction-tuned Mistral-7B backbone, we investigate whether a lightweight reinforcement learning (RL) step can improve explanation structure without human preference labels.

We apply Proximal Policy Optimization (PPO) with LoRA adaptation to 2000 Meerkat-derived medical prompts, guided by a simple self-supervised reward that favors longer, lexically diverse, and well-punctuated responses while penalizing very short ones.

Training for about 170 PPO steps on a single NVIDIA A100 GPU yields a stable increase in the average reward.

Quantitative and qualitative comparisons show that the RL-tuned model produces longer, more organized, multi-point explanations while preserving the underlying medical content.

We discuss the promise and the limitations of self-rewarded RL for shaping the style of medical explanations.

관련 뉴스