pairwiseLLM: Pairwise Comparison Tools for Large Language Model-Based Writing Evaluation

Provides a unified framework for generating, submitting, and analyzing pairwise comparisons of writing quality using large language models (LLMs). The package supports live and/or batch evaluation workflows across multiple providers ('OpenAI', 'Anthropic', 'Google Gemini', 'Together AI', and locally-hosted 'Ollama' models), includes bias-tested prompt templates and a flexible template registry, and offers tools for constructing forward and reversed comparison sets to analyze consistency and positional bias. The package additionally supports adaptive pairing workflows that iteratively select comparisons based on model uncertainty to improve ranking efficiency. Results can be modeled using frequentist or Bayesian Bradley–Terry–Luce models (Bradley & Terry, 1952 <doi:10.2307/2334029>; see also Caron & Doucet, 2012 <doi:10.1080/10618600.2012.638220>) or Elo rating methods (see Clark et al., 2018 <doi:10.1371/journal.pone.0190393>) to derive writing quality scores. For information on pairwise comparisons and comparative judgement, see Thurstone (1927) <doi:10.1037/h0070288> and Heldsinger & Humphry (2010) <doi:10.1007/BF03216919>.

Version:	1.3.1
Depends:	R (≥ 4.1)
Imports:	cli, curl, dplyr, httr2, jsonlite, rlang, stats, tibble, tidyselect, tools, utils
Suggests:	BradleyTerry2, cmdstanr, EloChoice, future, future.apply, knitr, mockery, pkgload, purrr, readr, rmarkdown, sirt, stringr, testthat (≥ 3.0.0), tidyr, vctrs, withr
Published:	2026-09-07
DOI:	10.32614/CRAN.package.pairwiseLLM
Author:	Sterett H. Mercer [aut, cre, cph]
Maintainer:	Sterett H. Mercer <sterett.mercer at ubc.ca>
BugReports:	https://github.com/shmercer/pairwiseLLM/issues
License:	MIT + file LICENSE
Copyright:	see file COPYRIGHTS
URL:	https://github.com/shmercer/pairwiseLLM, https://shmercer.github.io/pairwiseLLM/
NeedsCompilation:	no
SystemRequirements:	CmdStan and a C++17 toolchain (optional; needed only for Bayesian and adaptive modeling)
Additional_repositories:	https://stan-dev.r-universe.dev/
Citation:	pairwiseLLM citation info
Materials:	README, NEWS
CRAN checks:	pairwiseLLM results

Documentation:

Reference manual:

pairwiseLLM.html , pairwiseLLM.pdf

Vignettes:

Design: Adaptive Linking (source, R code) Guide: Adaptive Linking (source, R code) Guide: Adaptive Pairing (source, R code) Advanced: Submitting and Polling Multiple Batches (source, R code) Standalone Bayesian BTL with CmdStan (source, R code) Data Schemas and Prompt Management (source, R code) Getting Started with pairwiseLLM (source, R code) Backends and Tested Model Configurations (source, R code) Prompt Template Positional Bias Testing (source, R code) Provider Controls and Recovery (source, R code) Design: Adaptive Pairing (source, R code)

Downloads:

Package source:	pairwiseLLM_1.3.1.tar.gz
Windows binaries:	r-devel: pairwiseLLM_1.3.1.zip, r-release: pairwiseLLM_1.1.0.zip, r-oldrel: pairwiseLLM_1.1.0.zip
macOS binaries:	r-release (arm64): pairwiseLLM_1.1.0.tgz, r-oldrel (arm64): pairwiseLLM_1.1.0.tgz, r-release (x86_64): pairwiseLLM_1.1.0.tgz, r-oldrel (x86_64): pairwiseLLM_1.1.0.tgz
Old sources:	pairwiseLLM archive

Linking:

Please use the canonical form https://CRAN.R-project.org/package=pairwiseLLM to link to this page.