1. Overview

This vignette demonstrates how to use pairwiseLLM for Batch API workflows (server-side batching), which are distinct from the live API calls described in the Getting Started with pairwiseLLM vignette.

Batch workflows are ideal for large-scale jobs because they:

Allow submitting thousands of pairs at once
Are often cheaper (e.g., discounted batch pricing on some providers)
Avoid client-side timeout and connection issues
Can be polled and resumed even if your local R session ends

Supported Batch API providers:

OpenAI (batch pipeline: run_openai_batch_pipeline())
Anthropic (batch pipeline: run_anthropic_batch_pipeline())
Gemini (batch pipeline: run_gemini_batch_pipeline())

Vertex AI Gemini API is intentionally excluded here. In this series, Vertex is supported on the live path only via backend = "vertex" with VERTEX_API_KEY, and generic batch wrappers reject Vertex explicitly.

Recommended approach: For multiple batches (e.g., templates × providers × models × forward/reverse), use:

llm_submit_pairs_multi_batch() to split + submit jobs (no polling; writes an optional registry CSV)

llm_resume_multi_batches() to poll + download + parse results (can resume from a registry on disk)

These helpers orchestrate the provider-specific pipelines without forcing you to write your own polling loops.

Note: Together.ai and Ollama do not currently support a native Batch API compatible with this workflow. For those providers, use the live API wrapper submit_llm_pairs() as described in Getting Started with pairwiseLLM.

In this vignette, we will cover:

Designing a grid of provider/model/thinking/direction combinations
Submitting many batch jobs using the multi-batch helpers
Polling and resuming safely via on-disk registries
Producing per-run and merged results tables

Note: All heavy API calls in this vignette are set to eval = FALSE so that the vignette remains CRAN-safe. You can enable them in your own project.

For basic function usage, see the companion vignette:

Getting Started with pairwiseLLM

For prompt evaluation and positional-bias diagnostics, see the companion vignette:

Prompt Template Positional Bias Testing

2. Setup and API Keys

library(pairwiseLLM)
library(dplyr)
library(tidyr)
library(purrr)
library(readr)
library(stringr)

Configure the environment variable only for each provider included in your run plan:

Provider	Environment Variable
OpenAI	`OPENAI_API_KEY`
Anthropic	`ANTHROPIC_API_KEY`
Gemini	`GEMINI_API_KEY`

This vignette only covers batch-capable providers. Vertex live requests use a separate VERTEX_API_KEY surface and are documented in Getting Started with pairwiseLLM.

Check which are set:

check_llm_api_keys()
#> All known LLM API keys are set: OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, VERTEX_API_KEY, TOGETHER_API_KEY.
#> # A tibble: 5 × 4
#>   backend   service              env_var           has_key
#>   <chr>     <chr>                <chr>             <lgl>  
#> 1 openai    OpenAI               OPENAI_API_KEY    TRUE   
#> 2 anthropic Anthropic            ANTHROPIC_API_KEY TRUE   
#> 3 gemini    Google Gemini        GEMINI_API_KEY    TRUE   
#> 4 vertex    Vertex AI Gemini API VERTEX_API_KEY    TRUE   
#> 5 together  Together.ai          TOGETHER_API_KEY  TRUE

3. Example Data and Prompt Template

We use the built-in writing samples and a single trait (overall_quality).

data("example_writing_samples", package = "pairwiseLLM")

td <- trait_description("overall_quality")
td
#> $name
#> [1] "Overall Quality"
#> 
#> $description
#> [1] "Overall quality of the writing, considering how well ideas are expressed,\nhow clearly the writing is organized, and how effective the language and\nconventions are."

Default prompt template:

tmpl <- set_prompt_template()
cat(substr(tmpl, 1, 400), "...
")
#> You are a debate adjudicator. Your task is to weigh the comparative strengths of two writing samples regarding a specific trait.
#> 
#> TRAIT: {TRAIT_NAME}
#> DEFINITION: {TRAIT_DESCRIPTION}
#> 
#> SAMPLES:
#> 
#> === SAMPLE_1 ===
#> {SAMPLE_1}
#> 
#> === SAMPLE_2 ===
#> {SAMPLE_2}
#> 
#> EVALUATION PROCESS (Mental Simulation):
#> 
#> 1.  **Advocate for SAMPLE_1**: Mentally list the single strongest point of evidence that makes SAMPLE_1 the  ...

Construct a modest number of pairs to keep the example light:

set.seed(123)

pairs_all <- example_writing_samples |>
  make_pairs()

n_pairs <- min(40L, nrow(pairs_all))

pairs_forward <- pairs_all |>
  sample_pairs(n_pairs = n_pairs, seed = 123) |>
  randomize_pair_order(seed = 456)

pairs_reverse <- sample_reverse_pairs(
  pairs_forward,
  reverse_pct = 1.0,
  seed        = 789
)

get_pairs_for_direction <- function(direction = c("forward", "reverse")) {
  direction <- match.arg(direction)
  if (identical(direction, "forward")) {
    pairs_forward
  } else {
    pairs_reverse
  }
}

4. Designing the Batch Grid

Suppose we want to test several prompt templates across:

Anthropic models (with/without “thinking”)
OpenAI models (with/without “thinking” for specific models)
Gemini models (with “thinking” enabled)

Here we define a small grid:

anthropic_models <- c(
  "claude-haiku-4-5-20251001"
)

gemini_models <- c(
  "gemini-3.8-flash"
)

openai_models <- c(
  "gpt-4.1",
  "gpt-5.6-luna"
)

thinking_levels <- c("no_thinking", "with_thinking")
directions <- c("forward", "reverse")

anthropic_grid <- tidyr::expand_grid(
  provider  = "anthropic",
  model     = anthropic_models,
  thinking  = "no_thinking",
  direction = directions
)

gemini_grid <- tidyr::expand_grid(
  provider  = "gemini",
  model     = gemini_models,
  thinking  = "with_thinking",
  direction = directions
)

openai_grid <- tidyr::expand_grid(
  provider  = "openai",
  model     = openai_models,
  thinking  = "no_thinking",
  direction = directions
)

batch_grid <- dplyr::bind_rows(
  anthropic_grid,
  gemini_grid,
  openai_grid
)

batch_grid
#> # A tibble: 8 × 4
#>   provider  model                     thinking      direction
#>   <chr>     <chr>                     <chr>         <chr>    
#> 1 anthropic claude-haiku-4-5-20251001 no_thinking   forward  
#> 2 anthropic claude-haiku-4-5-20251001 no_thinking   reverse  
#> 3 gemini    gemini-3.8-flash          with_thinking forward  
#> 4 gemini    gemini-3.8-flash          with_thinking reverse  
#> 5 openai    gpt-4.1                   no_thinking   forward  
#> 6 openai    gpt-4.1                   no_thinking   reverse  
#> 7 openai    gpt-5.6-luna              no_thinking   forward  
#> 8 openai    gpt-5.6-luna              no_thinking   reverse

We will also imagine multiple prompt templates have been registered. For simplicity, we use the same tmpl string, but in practice you would substitute different text:

templates_tbl <- tibble::tibble(
  template_id     = c("test1", "test2", "test3", "test4", "test5"),
  prompt_template = list(tmpl, tmpl, tmpl, tmpl, tmpl)
)

templates_tbl
#> # A tibble: 5 × 2
#>   template_id prompt_template
#>   <chr>       <list>         
#> 1 test1       <chr [1]>      
#> 2 test2       <chr [1]>      
#> 3 test3       <chr [1]>      
#> 4 test4       <chr [1]>      
#> 5 test5       <chr [1]>

5. Submitting Many Batches with the Multi‑Batch Helpers

The key idea is:

Each combination of (template_id, provider, model, thinking, direction) becomes a run
Each run writes its files into its own subdirectory (so file names never collide)
Within each run you can still split into multiple segments using batch_size or n_segments

5.1 Create a run plan and output directory

out_root <- "dev-output/advanced-multi-batch"
dir.create(out_root, recursive = TRUE, showWarnings = FALSE)

run_plan <- tidyr::crossing(
  templates_tbl |> tidyr::unnest(prompt_template),
  batch_grid
) |>
  mutate(
    run_id = paste(template_id, provider, model, thinking, direction, sep = "__"),
    run_id = gsub("[^A-Za-z0-9_.-]+", "-", run_id),
    run_dir = file.path(out_root, run_id)
  )

run_plan |> dplyr::select(run_id, template_id, provider, model, thinking, direction, run_dir)

5.2 Submit all runs (no polling)

Below we submit each run using llm_submit_pairs_multi_batch(). This returns a jobs list and writes a jobs_registry.csv under each run directory (because write_registry = TRUE).

Provider-specific options can be forwarded via .... In the example below we:

Enable “thinking” output where applicable
Ask providers to include raw outputs in addition to parsed tags (helpful for debugging)

submit_one_run <- function(template_id, prompt_template, provider, model, thinking, direction, run_dir) {
  pairs_use   <- get_pairs_for_direction(direction)
  is_thinking <- identical(thinking, "with_thinking")

  # Provider-specific knobs (passed through via ...)
  extra_args <- list()

  if (identical(provider, "openai")) {
    # Only request thoughts for models that support them in this workflow
    extra_args$include_thoughts <- is_thinking && grepl("^gpt-5", model)
    extra_args$include_raw      <- TRUE
  } else if (identical(provider, "anthropic")) {
    extra_args$reasoning        <- if (is_thinking) "enabled" else "none"
    extra_args$include_thoughts <- is_thinking
    extra_args$include_raw      <- TRUE
    # Optional: set deterministic temperature when not using reasoning
    if (!is_thinking) extra_args$temperature <- 0
  } else if (identical(provider, "gemini")) {
    extra_args$include_thoughts <- TRUE
    extra_args$thinking_level   <- "low"   # example
    extra_args$include_raw      <- TRUE
  }

  message(
    "Submitting: ", template_id, " | ", provider, " / ", model,
    " / ", thinking, " / ", direction
  )

  # Split strategy:
  # - For real jobs, use batch_size (e.g., 500–5000) or n_segments (e.g., 10–50)
  # - Here we keep it simple and submit a single segment per run
  do.call(
    llm_submit_pairs_multi_batch,
    c(
      list(
        pairs             = pairs_use,
        backend           = provider,
        model             = model,
        trait_name        = td$name,
        trait_description = td$description,
        prompt_template   = prompt_template,
        n_segments        = 1L,
        output_dir        = run_dir,
        write_registry    = TRUE,
        verbose           = TRUE
      ),
      extra_args
    )
  )
}

run_results <- purrr::pmap(
  run_plan |>
    dplyr::select(
      template_id, prompt_template, provider, model, thinking, direction,
      run_dir
    ),
  submit_one_run
)

# Store a lightweight manifest so you can resume later without rebuilding run_plan
manifest <- run_plan |>
  mutate(registry_path = file.path(run_dir, "jobs_registry.csv"))

manifest_path <- file.path(out_root, "run_manifest.csv")
readr::write_csv(manifest, manifest_path)

manifest_path

At this point, each run directory contains:

JSONL input/output placeholders (one per segment)
A jobs_registry.csv that records all batch IDs and file paths for that run

You can safely stop R or restart your machine after submission.

6. Polling, Downloading, and Parsing (Resumable)

To poll all runs, read the manifest and call llm_resume_multi_batches() for each run_dir. If you restart R, you can resume without keeping the jobs objects in memory by setting jobs = NULL and pointing to output_dir (the function will load jobs_registry.csv).

manifest_path <- file.path(out_root, "run_manifest.csv")
manifest <- readr::read_csv(manifest_path, show_col_types = FALSE)

poll_one_run <- function(run_dir) {
  llm_resume_multi_batches(
    jobs               = NULL,   # load from jobs_registry.csv in run_dir
    output_dir         = run_dir,
    interval_seconds   = 60,
    per_job_delay      = 2,
    write_results_csv  = TRUE,   # writes batch_XX_results.csv files
    write_registry     = TRUE,   # refreshes jobs_registry.csv with done flags
    keep_jsonl         = TRUE,
    verbose            = TRUE,
    write_combined_csv = TRUE,   # writes combined_results.csv inside run_dir
    combined_csv_path  = "combined_results.csv"
  )
}

polled <- purrr::map(manifest$run_dir, poll_one_run)

6.1 Building a single merged results table (all runs)

Each element of polled contains a combined tibble for that run (i.e., all segments bound together). We can attach run metadata (template/provider/model/thinking/direction) and then bind all runs into one master table.

combined_all <- purrr::map2_dfr(
  polled,
  seq_len(nrow(manifest)),
  function(res, i) {
    meta <- manifest[i, ]
    if (is.null(res$combined)) return(NULL)

    res$combined |>
      mutate(
        template_id = meta$template_id,
        provider    = meta$provider,
        model       = meta$model,
        thinking    = meta$thinking,
        direction   = meta$direction,
        run_id      = meta$run_id
      )
  }
)

combined_path <- file.path(out_root, "combined_all_runs.csv")
readr::write_csv(combined_all, combined_path)

combined_path

7. Resuming After Interruption

Resuming jobs is possible:

Submission writes jobs_registry.csv under each run directory
Polling can be restarted at any time by calling llm_resume_multi_batches(jobs = NULL, output_dir = <run_dir>)
If you keep a run_manifest.csv with run_dir paths, resuming all runs is just a loop

Example: resume only unfinished runs (based on each run’s registry):

manifest <- readr::read_csv(file.path(out_root, "run_manifest.csv"), show_col_types = FALSE)

needs_poll <- function(run_dir) {
  reg_path <- file.path(run_dir, "jobs_registry.csv")
  if (!file.exists(reg_path)) return(FALSE)
  reg <- readr::read_csv(reg_path, show_col_types = FALSE)
  any(!as.logical(reg$done))
}

unfinished_dirs <- manifest$run_dir[vapply(manifest$run_dir, needs_poll, logical(1))]

polled <- purrr::map(unfinished_dirs, poll_one_run)

8. Next Steps

Once you have per-run results CSVs (e.g., one per template × model × thinking × direction), you can:

Compute reverse consistency with compute_reverse_consistency()
Analyze positional bias with check_positional_bias()
Aggregate results by provider/model/template using standard dplyr pipelines
Fit Bradley–Terry models with build_bt_data() + fit_bt_model()
Fit Elo models with fit_elo_model() (when EloChoice is installed)

Advanced: Submitting and Polling Multiple Batches