LLM evaluation code

1.Normalize an answer

Write normalize_answer(text) for exact-match scoring. In this order: lowercase the text, delete every character in Python's string.punctuation, split on whitespace, drop the words a, an and the, and join the remaining words with single spaces. Return the result string ("" if nothing is left).

2.Exact match with aliases

Write exact_match(predictions, answers). answers[i] is a list of accepted strings for predictions[i]. Normalize every string: lowercase, delete string.punctuation characters, split on whitespace, drop a/an/the, join with single spaces. A prediction scores 1 if it equals any normalized accepted answer. Return the mean score, or 0.0 if there are no predictions.

3.Token-level F1

Write token_f1(prediction, reference). Lowercase both, delete string.punctuation characters, split on whitespace, drop the words a, an, the. common = sum over tokens of min(count in prediction, count in reference). If both token lists are empty return 1.0; if either is empty or common is 0 return 0.0. Else precision = common/len(pred), recall = common/len(ref); return 2*P*R/(P+R).

4.Best F1 over references

Write best_f1(prediction, references). Tokens are the lowercased text split on whitespace (no other cleanup). F1 per reference: common = multiset overlap count; both empty gives 1.0, either empty or common 0 gives 0.0, else 2*P*R/(P+R) with P = common/len(pred), R = common/len(ref). Return [best, index]; ties, even all 0.0, go to the earliest index. With no references return [0.0, -1].

5.ROUGE-L F score

Write rouge_l(candidate, reference). Lowercase both and split on whitespace. Let L be the length of the longest common subsequence of the two token lists (in order, gaps allowed). If L is 0 (including empty input) return 0.0. Else P = L/len(candidate tokens), R = L/len(reference tokens); return 2*P*R/(P+R).

6.Clipped n-gram precision

Write ngram_precision(candidate, references, n). Lowercase and whitespace-split every string. Count the candidate's n-grams (n consecutive tokens). Clip each n-gram's count to the largest number of times it appears in any single reference. Return sum of clipped counts / total candidate n-grams, or 0.0 if the candidate has fewer than n tokens.

7.Recall at k

Write recall_at_k(retrieved, relevant, k). retrieved is a ranked list of document ids (may repeat), relevant a list of ids. Take the distinct ids among the first k entries of retrieved and return how many are relevant divided by the number of distinct relevant ids. Return 0.0 if relevant is empty. k may exceed the list length.

8.Retrieval hit rate

Write hit_rate(queries, k). Each query is a dict with "retrieved" (ranked ids) and "relevant" (ids). A query is a hit if any of its first k retrieved ids is relevant. Return [hits / number of queries, list of indices of queries that missed in ascending order]. With no queries return [0.0, []].

9.Mean pass@k

Write mean_pass_at_k(results, k). Each result is [n, c]: n samples drawn for a problem, c of them correct. For each problem with n >= k, pass@k = 1 - C(n-c, k) / C(n, k) using math.comb (C(m, k) is 0 when m < k). Skip problems with n < k. Return the mean over the remaining problems, or 0.0 if none remain.

10.Self-consistency vote

Write majority_vote(answers) for self-consistency sampling. Normalize each answer by stripping surrounding whitespace and lowercasing; ignore answers that become empty. Return the normalized answer that occurs most often. On a tie, return the one whose first occurrence comes earliest in the list. Return None if no answers remain.

11.Pick option by log probability

Write pick_option(options). Each option is a dict with "label" and "logprobs", the per-token log probabilities of that answer. Score an option by the mean of its logprobs (sum / count). Skip options with no logprobs. Return the label with the highest score, ties to the earliest option; return None if every option is skipped.

12.Expected calibration error

Write calibration_error(confidences, correct, num_bins). confidences are in [0, 1]; correct[i] is true/false (or 1/0). Put each item in bin min(int(conf * num_bins), num_bins - 1). For each non-empty bin, gap = |accuracy in bin - mean confidence in bin|. Return the sum over bins of gap * (bin size / total items), or 0.0 with no items.

13.Pairwise win rate

Write win_rates(verdicts). Each verdict is [model_a, model_b, result] where result is "a", "b" or "tie" from an LLM judge. Every model in a verdict plays one game. A win counts 1, a tie 0.5 for both models, a loss 0. Return a dict of model name to points / games played for every model that appears.

14.Elo ratings from matches

Write elo_ratings(matches, k). Each match is [a, b, score_a] with score_a 1 (a wins), 0.5 (tie) or 0 (b wins). Every model starts at 1000. Process matches in order: E_a = 1 / (1 + 10 ** ((R_b - R_a) / 400)), then R_a += k * (score_a - E_a) and R_b += k * ((1 - score_a) - (1 - E_a)), both using the ratings from before this match. Return a dict of name to rating.

15.Fill a prompt template

Write fill_template(template, values). A slot is { followed by a name (letters, digits, underscore; not starting with a digit) and }. Replace each slot whose name is in values with str(value); leave other slots unchanged. Other braces are plain text. Return {"text": result, "missing": names of unfilled slots, each once, in order of first appearance}.

16.Few-shot selection by overlap

Write select_examples(query, pool, k). pool is a list of example strings. Overlap = number of distinct words shared by the query and the example, where words come from lowercasing and splitting on whitespace. Choose up to k examples with overlap > 0, highest overlap first, ties by lower pool index. Return their pool indices in that chosen order.

17.Extract the first JSON object

Write extract_json(text). Try each { in text from left to right as a start. Scan forward counting { and } but ignore braces inside double-quoted strings (a backslash inside a string escapes the next character). When the count returns to 0, json.loads that span; if it gives a dict, return it, else try the next start. Return None if nothing works.

18.Parse a tool call

Write parse_tool_call(text) for calls like f(q="x", k=3); ignore outer whitespace. Name/keys: letters, digits, _; not digit-first. No space before (. Args: none, or comma-separated key=value pairs, no trailing comma; spaces OK around keys, = and commas. Value: double-quoted string (backslash escapes next char), int (-2), float (0.5), True, False, None. Return {"name", "args"}, None if malformed.

19.Validate JSON output

Write validate_output(text, schema). schema maps field to "str", "int", "float", "bool" or "list". If json.loads(text) fails return ["invalid json"]; if it is not a dict return ["not an object"]. Else for each schema field in schema order add "missing: f" or "wrong type: f", then "extra: f" for each unknown key in object order. bool is not int; float accepts int or float, not bool.

20.Strip chain-of-thought

Write final_answer(text) to drop a model's reasoning. Find the last occurrence of "Answer:" (case-sensitive) and return the text after it up to the next newline, stripped of surrounding whitespace. If "Answer:" does not appear, return the last line that is not blank, stripped. Return "" if there is no such line.

21.Detect refusals

Write detect_refusals(responses, phrases). Lowercase each response and replace every run of whitespace with one space. It is a refusal if any phrase, lowercased, appears as a substring. Return [refusals / number of responses, indices of refusals in ascending order], or [0.0, []] if there are no responses.

22.Near-duplicate pairs

Write near_duplicates(docs, n, threshold). Shingles of a doc: the set of n-word sequences from its lowercased whitespace-split words; a doc with 1 to n-1 words has one shingle of all its words; an empty doc has none. Jaccard = size of intersection / size of union, 0.0 if both are empty. Return every [i, j] with i < j and Jaccard >= threshold, sorted by i then j.

23.Deduplicate a prompt set

Write dedupe_prompts(prompts). Key of a prompt: lowercase, delete string.punctuation characters, then join the whitespace-split words with single spaces. Keep the first prompt with each key (original text, original order). Prompts whose key is empty are also removed. Return {"kept": kept prompts, "removed": indices of removed prompts in ascending order}.

24.Test-set contamination check

Write contaminated_items(train_docs, test_items, n). Lowercase and whitespace-split every text; its n-grams are its n consecutive words. A test item is contaminated if any of its n-grams also appears in any training doc. Items with fewer than n words are never flagged. Return [indices of contaminated items ascending, flagged count / number of test items], with 0.0 if there are no test items.

25.Truncate a conversation

Write truncate_conversation(messages, budget). Each message is {"role", "content"}; its tokens = number of whitespace-split words. Always keep every message with role "system". Then walk the other messages from newest to oldest, keeping each while the total of all kept tokens stays <= budget; stop at the first one that does not fit. Return the kept messages in their original order.

26.Stratified first-k sample

Write stratified_sample(examples, k) to build a balanced eval set without randomness. Each example is a dict with a "label" key. Walk the list in order and keep an example if fewer than k examples with its label have been kept so far. Return the kept examples, unchanged, in their original order.

27.Mean reciprocal rank

Write mean_reciprocal_rank(queries). Each query is a dict with "retrieved" (ranked ids) and "relevant" (ids). Its score is 1 / r where r is the 1-based position of the first retrieved id that is relevant, or 0 if none is. Return the mean score over all queries, or 0.0 if there are no queries.

28.Accuracy by slice

Write slice_report(records). Each record is {"slice": name, "correct": bool}. micro = correct records / all records. Slice accuracy = correct / count within the slice; macro = mean of slice accuracies. worst = the slice with the lowest accuracy, ties to the alphabetically first name. Return {"micro", "macro", "worst"}; with no records return {"micro": 0.0, "macro": 0.0, "worst": None}.