Documentation

What is PRISM?

PRISM (Prompt Reliability Through Intelligent Semantic Multiplexing) is a benchmark designed to study the reliability of language models under semantically equivalent prompt formulations.

Rather than evaluating a model using a single fixed prompt, PRISM presents the same underlying question through multiple prompt conditions. The model's responses are then analyzed to determine whether changes in prompt formulation affect correctness, instruction compliance, and answer consistency.

The Problem

A model can produce the correct answer when given one formulation of a question and produce a different answer when the same question is expressed differently.

That raises an important question:

How sensitive is a model's behavior to the way an otherwise equivalent instruction is written?

Traditional benchmark accuracy does not necessarily expose this behavior.

Prompt	Formulation
Prompt A	Select the correct answer.
Prompt B	Choose the correct option from the choices below.
Prompt C	Determine which option is correct.

The underlying task is unchanged. PRISM asks whether the model's behavior remains stable.

What PRISM Does NOT Claim to Measure

Important limitations

PRISM does not determine whether a model is reliable in a general sense, and it does not guarantee trustworthy answers.

It evaluates behavior only under:

its specific benchmark conditions;
its fixed datasets;
its fixed question sample;
its fixed prompt formulations;
its parser and scoring methodology;
its specific local inference configuration.

Results are evidence about behavior under PRISM's experimental conditions, not a universal verdict on a model's reliability.

PRISM should not currently claim to measure:

general intelligence;
general trustworthiness;
factual reliability across arbitrary domains;
safety;
hallucination rate in general;
stochastic reliability across repeated runs;
behavior under arbitrary prompts.

How PRISM Works

PRISM evaluates the same underlying question through multiple prompt formulations and compares the model's answers.

For each frozen question, PRISM keeps:

the same question;
the same answer options;
one official answer;
one question identity.

The question is rendered into five prompts, one for each prompt condition.

The five responses are:

collected;
stored as raw experimental evidence;
parsed;
scored against the official answer;
compared at question level;
aggregated into model-level results.

The task remains fixed while the prompt formulation changes.

Prompt Conditions

PRISM currently evaluates five prompt formulation conditions. Each preserves the same question, options, underlying task, and output requirement. Only the instruction formulation changes.

P0: Minimal

The question and options are presented with the barest possible framing, with no additional instructional language beyond the output requirement.

P1:Direct

The question is preceded by a short, direct instruction to answer and select the correct option.

P2: Structured

The question is presented with explicit labeled sections such as task, question, options, and required output, giving the model a more scaffolded, form-like input.

P3: Role

The question is preceded by a short instruction establishing the model as a knowledgeable subject-matter expert before presenting the same question.

P4: Careful Analysis

The model is instructed to carefully analyze the question before selecting an option, inviting a more deliberate or reasoning-oriented response.

Across all five conditions, the closing instruction is identical:

Return only the letter of the correct option.

This constant output requirement helps keep the requested output format fixed while the surrounding prompt formulation changes.

Datasets

PRISM currently evaluates two multiple-choice question-answering datasets.

ARC-Challenge

ARC-Challenge is a science-question benchmark designed to be challenging for models relying on simple retrieval or superficial word matching. Questions contain four answer options and one official answer.

SciQ

SciQ is a crowdsourced collection of science exam questions covering areas including physics, chemistry, and biology. Questions are represented as four-option multiple-choice questions in the PRISM pipeline.

Dataset Sampling

For both datasets:

a fixed number of questions is sampled;
a fixed random seed is used;
the selected questions are frozen;
the same question set is used across models;
the same question set is used across prompt conditions.

This ensures observed differences are not caused by models receiving different questions.

Evaluation Pipeline

Dataset Preparation

A fixed-size sample is drawn using a fixed seed and frozen for the experiment.

Prompt Generation

Each frozen question is rendered into P0–P4, producing five distinct prompt texts while preserving the same task and answer space.

Model Inference

Each prompt is sent independently to the model through Ollama on the user's local hardware. Current benchmark execution uses deterministic inference: temperature = 0; one inference per question-model-prompt combination.

Raw Response Storage

The raw, unmodified model response is stored immediately together with metadata such as latency, timestamp, generation parameters, model identifier, and benchmark metadata. Raw responses are treated as immutable experimental evidence.

Response Parsing

The parser attempts to determine which option letter the model selected. The parser does not consult the correct answer during extraction. It separately records: whether an answer letter could be recovered; whether the requested output format was followed.

Scoring

Only after parsing is complete is the parsed answer compared with the official answer.

Question-Level Aggregation

The five prompt responses are compared to calculate metrics such as agreement, prompt sensitivity, majority correctness, unanimity, and prompt-invariant incorrectness.

Model/Dataset Aggregation

Question-level results are aggregated into model/dataset and model/prompt-condition summaries.

Results

Aggregated data forms the basis of the PRISM application dashboard and public benchmark results.

Metrics

Each metric below includes its definition, formula, an example where applicable, interpretation guidance, and caveats.

Accuracy / Prompt-Condition Accuracy

Definition: The proportion of responses for a prompt condition whose parsed answer matches the official answer.
Formula: Accuracy = N correct / N observed
Example: 74 correct out of 100 gives 74%.
Interpretation: Measures raw correctness for one prompt formulation.
Caveat: High accuracy under one formulation does not imply equal accuracy under another.

Question Majority Accuracy

Definition: Proportion of questions where the majority answer across the five prompt conditions matches the official answer.
Formula: Question Majority Accuracy = questions with correct majority / total questions
Interpretation: Combines evidence from the five formulations.
Caveat: Majority aggregation can hide disagreement between individual prompt conditions.

Conditional Accuracy

Definition: Accuracy among responses from which an answer letter could be recovered, excluding UNKNOWN responses.
Formula: Conditional Accuracy = N correct / N recovered (where N recovered = total responses − UNKNOWN responses)
Interpretation: Separates correctness from answer recoverability.
Caveat: High conditional accuracy with high UNKNOWN rate may still indicate poor practical reliability.

Answer Recovery Rate

Definition: Proportion of responses from which an answer letter could be recovered, regardless of formatting compliance.
Formula: Answer Recovery Rate = non-UNKNOWN responses / total responses
Interpretation: Measures whether PRISM can determine which option the model selected.
Caveat: Recovery does not imply compliance or correctness.

Unknown Rate

Definition: Proportion of responses from which no answer letter could be recovered.
Formula: Unknown Rate = 1 − Answer Recovery Rate
Interpretation: Measures failure to produce a usable answer.
Caveat: UNKNOWN is not an answer choice and should not automatically be treated as incorrect.

Instruction Compliance Rate

Definition: Proportion of responses that follow the benchmark's required output-format rule.
Formula: Instruction Compliance Rate = fully compliant responses / total responses
Example: "B" is compliant. "The correct answer is B because…" is not.
Interpretation: Measures output-format compliance independently from correctness.
Caveat: Compliance is not a proxy for correctness.

Agreement

Definition: For a question, the proportion of valid non-UNKNOWN responses matching the most common answer.
Formula: Agreement = count of most frequent valid answer / number of valid responses
Example: B, B, B, B, C gives 0.8 agreement.
Interpretation: Measures answer stability across prompt formulations.
Caveat: Agreement says nothing about whether the common answer is correct.

Prompt Sensitivity

Definition: Inverse of agreement.
Formula: Prompt Sensitivity = 1 − Agreement
Interpretation: Higher values indicate greater answer variation across the tested prompt formulations.
Caveat: It describes observed variation across PRISM's five conditions, not all possible prompts.

Answer Unanimous Rate

Definition: Proportion of questions where all five prompt conditions produced valid, identical answers.
Formula: Answer Unanimous Rate = unanimous questions / total questions
Interpretation: The strictest consistency measure currently used.
Caveat: A unanimous answer can still be wrong.

Prompt-Invariant Incorrect Rate

Definition: Proportion of questions where all five valid prompt-condition answers were identical but incorrect.
Formula: Prompt-Invariant Incorrect Rate = unanimous incorrect questions / total questions
Example: B, B, B, B, B when the official answer is D.
Interpretation: Identifies questions where changing among PRISM's five prompt formulations did not change an incorrect answer.
Caveat: With one deterministic temperature-zero run per prompt condition, this measures invariance across prompt formulation, not stochastic reliability across repeated runs.

Complete Questions

Definition: Number or proportion of questions for which all five prompt conditions produced recorded responses, including UNKNOWN responses.
Formula: Complete Questions = questions with five recorded responses / total questions attempted
Interpretation: Measures data completeness, not model behavior.
Caveat: Low completeness reduces confidence in downstream metrics.

Interpreting Results

Metrics should be interpreted together.

High accuracy + high agreement

Generally correct and stable.

High accuracy + high prompt sensitivity

Capable but prompt-dependent.

Low accuracy + high agreement

Consistently wrong; this is the pattern associated with prompt-invariant incorrectness.

Low accuracy + high prompt sensitivity

Inaccurate and unstable.

These are interpretive patterns, not formal model classifications.

Reproducibility

Meaningful reproduction or comparison requires documenting:

model and model version/tag;
dataset and dataset version;
question selection;
random seed;
prompt templates;
template version;
prompt conditions;
generation parameters;
temperature;
maximum output length;
parser version;
scorer version;
benchmark/protocol version.

Results produced under different configurations should not be treated as directly comparable without accounting for those differences.

Published result pages should expose relevant metadata.

Glossary

Term	Definition
Accuracy	Proportion of responses whose parsed answer matches the official answer.
Conditional Accuracy	Accuracy among responses from which an answer could be recovered.
Answer Recovery	Successful extraction of an answer letter regardless of formatting compliance.
Unknown Response	A response from which no answer letter could be recovered.
Instruction Compliance	Whether the response followed the required output format.
Prompt Condition	One of PRISM's five fixed formulations of the same underlying question.
Agreement	Proportion of valid responses matching the most common answer.
Prompt Sensitivity	Inverse of agreement; observed answer variation across prompt conditions.
Question Majority	Most frequently selected answer among the five prompt-condition responses.
Answer Unanimity	All five prompt conditions produced valid, identical responses.
Prompt-Invariant Incorrectness	All prompt conditions produced the same valid answer, but it was incorrect.
Observed Response	The model's raw, unmodified output.
Recovered Response	An observed response from which a specific answer letter was parsed.