Select the subset, model provider, and variant below to get automated analyses of the model's performance on NoveltyBench. Click the "Load Data" button to see the results, and see a guide to the results at the bottom.
Analyze one model's responses and partitions for each prompt.
Choose a subset first.
Choose a subset first.
Choose a model first.
Choose a subset and a model to load its per-prompt results.
Nothing loaded yet. Choose a subset and a model above, then select "Load data".
Two numbers describe each model: how many of its responses say something different, and how much a reader gains from reading them in order.
Partitions represent functionally equivalent classes of responses. We argue that true diversity should go beyond surface-level differences (like paraphrasing or minor wording changes), since such variations provide little utility to the user. Therefore, we mainly consider functional equivalence, which defines two generations to be different if and only if a user who has seen one generation would likely benefit from seeing the other.
Under v1.0 we annotated pairs of generations and used this labeled data to train a DeBERTa-v3-large model to predict binary functional equivalence between two generations. For each new generation, we compare it against a random generation from each existing partition—if it's functionally equivalent to any existing class, we assign it there; otherwise, we create a new partition.
Under v1.1 a language model reads all ten responses to a prompt in one call and returns the groups directly, so a response is compared against every other rather than one member of each group, and against its full text rather than its first 128 tokens.
The utility score measures the combined diversity and quality of a model's responses. It's based on a user model that assumes users have a "patience level" (p=0.8 in our experiments) — after seeing each generation, they might continue to request another or stop entirely. Naturally, we apply a geometric discount to the utility of later generations.
The cumulative utility score takes into account both novel and high-quality responses by assigning zero utility to responses that are functionally equivalent to those already seen. We calculate this by combining: (1) a quality score for each generation, (2) whether the generation belongs to a new partition, and (3) the diminishing probability that a user would actually see later generations. The result is a single number that represents how much cumulative value a user would get from multiple generations, with higher scores indicating better performance.
The quality score comes from a reward model calibrated to output values between 1-10 under v1.0, and from a language model judge scoring one response per partition under v1.1.