Prompt Variability Effects on LLM Code Generation

Andrei Paleyes; Radzim Sendyka; Diana Robinson; Christian Cabrera; Neil D. Lawrence

Back to publications

Prompt Variability Effects on LLM Code Generation

Andrei Paleyes, Radzim Sendyka, Diana Robinson, Christian Cabrera, Neil D. Lawrence

Third International Workshop on Large Language Models for Code (LLM4Code 2026), co-located with ICSE 2026, 2026.

Abstract

Large language models lower barriers to writing code, but the quality of generated programs is sensitive to prompt quality and the user’s background. We propose a synthetic evaluation pipeline and a systematic persona-based evaluation approach to quantify an LLM’s sensitivity to input variations, independent of specific programming tasks or models.

Links

Cite this Paper

BibTeX


@InProceedings{publications/prompt-variability-effects-on-llm-code-generation,
  title = 	 {Prompt Variability Effects on LLM Code Generation},
  author = 	 {Paleyes, Andrei and Sendyka, Radzim and Robinson, Diana and Cabrera, Christian and Lawrence, Neil D.},
  booktitle = 	 {Third International Workshop on Large Language Models for Code (LLM4Code 2026), co-located with ICSE 2026},
  year = 	 {2026},
  url = 	 {/publications/prompt-variability-effects-on-llm-code-generation.html},
  abstract = 	 {Large language models lower barriers to writing code, but the quality of generated programs is sensitive to prompt quality and the user's background. We propose a synthetic evaluation pipeline and a systematic persona-based evaluation approach to quantify an LLM's sensitivity to input variations, independent of specific programming tasks or models.}
}

Endnote

%0 Conference Paper
%T Prompt Variability Effects on LLM Code Generation
%A Andrei Paleyes
%A Radzim Sendyka
%A Diana Robinson
%A Christian Cabrera
%A Neil D. Lawrence
%B Third International Workshop on Large Language Models for Code (LLM4Code 2026), co-located with ICSE 2026
%D 2026	
%F publications/prompt-variability-effects-on-llm-code-generation
%U /publications/prompt-variability-effects-on-llm-code-generation.html
%X Large language models lower barriers to writing code, but the quality of generated programs is sensitive to prompt quality and the user's background. We propose a synthetic evaluation pipeline and a systematic persona-based evaluation approach to quantify an LLM's sensitivity to input variations, independent of specific programming tasks or models.

RIS


TY  - CPAPER
TI  - Prompt Variability Effects on LLM Code Generation
AU  - Andrei Paleyes
AU  - Radzim Sendyka
AU  - Diana Robinson
AU  - Christian Cabrera
AU  - Neil D. Lawrence
BT  - Third International Workshop on Large Language Models for Code (LLM4Code 2026), co-located with ICSE 2026
DA  - 2026/09/29	
ID  - publications/prompt-variability-effects-on-llm-code-generation
UR  - /publications/prompt-variability-effects-on-llm-code-generation.html
AB  - Large language models lower barriers to writing code, but the quality of generated programs is sensitive to prompt quality and the user's background. We propose a synthetic evaluation pipeline and a systematic persona-based evaluation approach to quantify an LLM's sensitivity to input variations, independent of specific programming tasks or models.
ER  -

APA


Paleyes, A., Sendyka, R., Robinson, D., Cabrera, C. & Lawrence, N.D.. (2026). Prompt Variability Effects on LLM Code Generation. Third International Workshop on Large Language Models for Code (LLM4Code 2026), co-located with ICSE 2026 Available from /publications/prompt-variability-effects-on-llm-code-generation.html.