面向不可回答问题的大语言模型拒答能力综合评估OA

[Objective]This study systematically evaluated the ability of large language models(LLMs)to abstain from answering unanswerable questions-those that lack sufficient,reliable,or coherent information for a definitive response.The goal was to unify diverse unanswerable-question datasets and testing paradigms to examine how model scale,architecture,and prompting strategies influence abstention behavior across both factual and nonfactual scenarios.[Methods]Five representative datasets were categorized as either factual unanswerable or nonfactual unanswerable.Two task paradigms were defined:(1)a binary-classification task requiring explicit"Yes/No"judgments on answerability and(2)an open-domain generation task requiring natural language answers or an abstention token when appropriate.Two prompting strategies were compared-direct prompting and chain-of-thought(CoT)prompting,where CoT prompting required intermediate reasoning steps before a final judgment.Experiments were conducted in zero-shot settings with the temperature fixed at 0.Models evaluated included both open-source and proprietary LLMs spanning small to large parameter scales.Performance metrics included overall accuracy(Acc),accuracy on unanswerable items(AcU),accuracy on answerable items(AcA),and F1 score.Outputs were parsed using standardized rules to detect explicit abstentions and typical abstention-related phrases.[Results]The performance gap between large and small LLMs was limited on nonfactual unanswerable datasets.Larger models often produced more fluent but incorrect answers,reflecting a tendency to rely on linguistic fluency rather than true abstention capability.Conversely,the models performed better on factual unanswerable datasets:most achieved＞70％AcU on FalseQA and NEC,and larger models showed higher F1 scores with a balanced trade-off between AcA and AcU.However,the UAQFact dataset remained challenging-even GPT-4o achieved only a 72.03％F1 score,with notably lower AcA,indicating that multifact reasoning and temporal consistency still pose challenges.Prompting strategies also played a significant role.CoT prompting improved accuracy and stability for some models,such as Qwen2.5-7B,Qwen-Plus,and GPT-4o;but for others(e.g.,Llama2-7B and DeepSeek-v3),direct prompting yielded higher F1 scores,suggesting that solvability judgment can benefit from concise prompts,while CoT reasoning may introduce redundant steps that obscure decision boundaries.This study also suggests that the LLM performance generally improves with scale but not in a linear manner.Some larger LLMs prioritize answering ability at the expense of abstention capability,reducing robustness and safety.Version upgrades also do not consistently improve the F1 score,indicating limited gains from standard iteration.As per this study,the binary classification and open-domain tasks should be considered when using small LLMs.The results further demonstrate that binary classification does not necessarily make models more susceptible to abstention,ensuring that the evaluation framework does not overestimate model safety.[Conclusions]Under a unified evaluation framework,LLMs exhibited meaningful progress in refusing factual unanswerable questions but remained unreliable on nonfactual unanswerable items.Abstention capability was found to depend not only on scale but also on model alignment,instruction tuning,and prompt design substantially influence outcomes.CoT prompting is not universally beneficial and can help or harm the refusal behavior.These findings indicate that targeted training and evaluation methods are required to improve LLM reliability in real-world scenarios in which safe abstention is critical.