Submit to NoveltyBench Leaderboard

If you are interested in submitting your model to the NoveltyBench Leaderboard, please do the following:

1. Fork the official repository;
2. Clone your fork;
3. Under evaluation/, create a new folder with the submission date and your model name (e.g., 2025-03-27_gemini-1.5-pro);
4. Within the folder (evaluation/<date + name>/), please include the following required assets:
• Follow the instruction in the README of the official repository to get the following files for each subset NB-Curated and NB-WildChat:
• generations.jsonl
• v1.1/partitions.jsonl
• v1.1/scores.jsonl
• v1.1/summary.json
• Your final folder should look like this. Scores sit under a v1.1/ directory named for the metric version that produced them, so a submission can be rescored later without overwriting what it originally reported:

- evaluation/
    - <date + name>/
        - nb-curated/
            - generations.jsonl
            - v1.1/
                - partitions.jsonl
                - scores.jsonl
                - summary.json
        - nb-wildchat/
            - generations.jsonl
            - v1.1/
                - partitions.jsonl
                - scores.jsonl
                - summary.json

• Generate with the v1.1 protocol: ten responses per prompt at --max-tokens 2048, and reasoning models at --reasoning-effort low. Say which sampling protocol you used — regenerate for independent samples, or in-context if each sample was asked for in the same conversation after the previous ones.
5. Create a pull request to this repository with the new folder.
6. (Optional) To get attribution on the leaderboard, include in your PR description:

   {
     "paper": "https://arxiv.org/abs/...",
     "model": "https://huggingface.co/...",
     "authors": "LastName et al.",
     "submitted_by": "your GitHub handle"
   }

   Entries naming a submitted_by are marked Community in the leaderboard's Source column, which separates outside submissions from the systems in the paper and the ones we have run since.

The NoveltyBench team will:

• Review and merge your submission
• Update the leaderboard with your results

Submission policy

The leaderboard is for training and inference-time methods applied to publicly available models (open weights, or a public API with a named model version). Closed systems and results we can't reproduce are out of scope and will be closed without review. Anyone can run NoveltyBench on any system and publish the results, citing the benchmark. Not sure if your method is in scope? Open an issue first.