Quantifying Temperature Effects on LLM Output Diversity Using SBERT

초록

This study quantitatively evaluates the effect of temperature control in large language models (LLMs) on the semantic similarity and diversity of generated text. Multiple samples were generated across the temperature range 0.0 to 1.2 for three task groups (FACTUAL, BRAINSTORM, CREATIVE, each with 50 prompts). Metrics included SBERT cosine similarity (relative to baseline T=0 anchor, and pair similarity within the same temperature range), Self-BLEU, and Distinct-2. Experimental results confirmed that as temperature increases, semantic similarity to the T=0 baseline consistently decreases, while pair similarity within the same temperature range and Self-BLEU also decrease, indicating increased semantic and expressive diversity. Lexical diversity (Distinct-2) increased rapidly in the lowtemperature range but tended to saturate around T=0.8. Average length and latency showed minimal impact from temperature changes. By category, Creative responded most sensitively, while FACTUAL showed relatively smaller changes. Based on this, we propose practical guidelines: T=0.2 to 0.5 for FACTUAL queries, T=0.8 to 1.0 for idea generation, and T=1.0 to 1.3 for creative tasks. This study presents a simple evaluation pipeline combining SBERT with lightweight diversity metrics, statically characterizing the consistency-diversity tradeoff induced by temperature adjustment.

키워드