OpenCompass v0.1.3

OpenCompass keeps getting better! v0.1.3 brings a variety of enhancements, new features, and crucial fixes. Here’s a summary of what we've packed into this release:

🆕 Highlights:

Extended Dataset Support: OpenCompass now integrates a broader range of public datasets, including but not limited to adv_glue, codegeex2, Humanevalx, SEED-Bench, LongBench, and LEval. We aim to provide extensive coverage to cater to a variety of research needs.
Utility Additions: From the inclusion of multi-modal evaluations on MME benchmark to the Tree-of-Thought method, this release comes packed with functionality enhancements.
Bug Extermination: Your feedback helps us grow. We’ve squashed a series of bugs to improve your experience.
More Evaluation Benchmark for Multimodal Models. We support another 10 evaluation benchmarks for multimodal models, including COCO Caption and ScienceQA, and provide corresponding evaluation code.

Let's delve deeper into what's new:

🌟 New Features:

📦 Extended Dataset Support:

• Introduction of other public datasets (#206, #214).
• Support for adv_glue dataset focused on adversarial robustness (#205).
• Added codegeex2, Humanevalx (#210).
• Integration of SEED-Bench (#203).
• LongBench support (#236).
• Reconstruct LEval dataset (#266).
• Support another 10 public evaluation benchmarks for multimodal models (#214)

🛠 Utilities and Functionality:

• Launch script added for ease of operations (#222).
• Multi-modal evaluation on MME benchmark (#197).
• Support for visualglm and llava on MMBench evaluation (#211).
• Tree-of-Thought method introduced (#173).
• Introduction of llama2 native implementations (#235).
• Flamingo and Claude support added (#258, #253).

📝 Documentation:

• Navigation bar language type updated for better clarity (#212).
• News updates for keeping users informed (#241, #243).
• Summarizer documentation added (#231).

🛠️ Bug Fixes:

• Addressed an issue with multiple rounds of inference using mm_eval (#201).
• Miscellaneous fixes such as name adjustments, requirements, and bin_trim corrections (#223, #229, #237).
• Local runner debug issue fixed (#238).
• Resolved bugs for PeftModel generate (#252).

⚙ Enhancements and Refactors:

• Refactored instructblip for better performance and readability (#227).
• Improved crowspairs postprocess (#251).
• Optimization to use sympy only when necessary (#255).

🎉 New Contributors:

Thank you to all our contributors for this release, with a special shoutout to our new contributors:

@yyk-wew (First PR)
@fangyixiao18 (First PR)
@philipwangOvO (First PR)
@cdpath (First PR)

Thank you to our dedicated contributors for making OpenCompass even more comprehensive and user-friendly! 🙌 🎉

Remember to star 🌟 our GitHub repository if you find OpenCompass helpful! Your feedback and contributions are invaluable.

---

Change log

• [Fix] Fix bugs of multiple rounds of inference when using mm_eval by @yyk-wew in #201
• [Feature]: Add other public datasets by @YuanLiuuuuuu in #206
• [Doc] Update Navigation bar language type by @Ezra-Yu in #212
• [Feat] support adv_glue dataset for adversarial robustness by @yingfhu in #205
• [Feat] Add codegeex2 and Humanevalx by @Ezra-Yu in #210
• [Feature]: Add other public datasets config by @YuanLiuuuuuu in #214
• [Feature] Support SEED-Bench by @fangyixiao18 in #203
• [Feature]: Add launch script by @YuanLiuuuuuu in #222
• [Fix]: Fix name by @YuanLiuuuuuu in #223
• [Fix] requirements by @gaotongxiao in #229
• [Dataset] LongBench by @philipwangOvO in #236
• [Fix] bin_trim by @philipwangOvO in #237
• [Feat] Support multi-modal evaluation on MME benchmark. by @yyk-wew in #197
• [Feat] Support visualglm and llava for MMBench evaluation. by @yyk-wew in #211
• [Fix] fix local runner debug by @Leymore in #238
• Update News by @tonysy in #241
• [Doc]update news by @tonysy in #243
• Update run.py by @liushz in #247
• [Doc] Add summarizer doc by @Leymore in #231
• [Feature] Add llama2 native implements by @Leymore in #235
• [Feature] Add Tree-of-Thought method by @liushz in #173
• [Refactor] Refactor instructblip by @fangyixiao18 in #227
• [Enhancement] Update crowspairs postprocess by @gaotongxiao in #251
• [Fix] use sympy only when necessary by @gaotongxiao in #255
• Update .owners.yml by @tonysy in #261
• [Fix] Fix bugs for PeftModel generate by @LZHgrla in #252
• [Feature]: Add Flamingo by @YuanLiuuuuuu in #258
• [Feature] Add Claude support by @gaotongxiao in #253
• [Dataset] Reconstruct LEval by @philipwangOvO in #266
• [Feature]: Verify the acc of these public datasets by @YuanLiuuuuuu in #269
• • [Feat] Support public dataset of visualglm and llava. by @yyk-wew in #265
• [Fix] wrong path in dataset collections by @gaotongxiao in #272
• [Fix] update descriptions of tools by @cdpath in #270
• [Feature] Support model-bound prediction postprocessor, use it in Claude by @gaotongxiao in #268
• [Feature] Simplify entry script by @gaotongxiao in #204
• Update README.md by @tonysy in #262

For a complete list of changes, please refer to our Full Changelog.

OpenCompass v0.1.2

This release continues the evolution of OpenCompass, bringing a mix of new features, optimizations, documentation improvements, and bug fixes.

🆕Highlights

🏆 Leaderboard: The evaluation results of Qwen-7B, XVERSE-13B, LLaMA-2, and GPT-4 has been posted to our leaderboard. Now it's also possible to conduct model comparison online. We hope this feature offers deeper insights!

📊 Datasets: Introduction of Xiezhi, SQuAD2.0, ANLI, LEval datasets, and more for diverse applications. (#101, #192) Add datasets related to safety to collections. [#185]

🎭New modality: Support for MMBench is introduced, and the evaluation of multi-modal models is on the way! (#56 ,#161) Besides, Intern language model is introduced. (#51)

⚙️Enhancement: Several enhancements on OpenAI models, including key deprecation, temperature setting, etc. [#121] [#128] Supporting multiple tasks on one GPU, filtering messages by levels, and more. [#148] [#187]

📝 Documentation: Comprehensive updates and fixes across READMEs, issue templates, prompt docs, metric documentation, and more.

🛠️ Bug Fixes: Including seed fixes in HFEvaluator, addressing issues in AGIEval multiple choice questions, and more. [#122] [#137]

🎉 New Contributors

Thank you to all our contributors for this release, with a special shoutout to our new contributors:

@go-with-me000 (First Contribution)
@anakin-skywalker-Joseph (First Contribution)
@zhouzaida (First Contribution)
@dependabot (First Contribution)

Changelog

• [Feat] add auto assignee bot by @yingfhu in #105
• [Doc] Update Readme and Fix failed links by @Ezra-Yu in #108
• Doc: add twitter link by @vansin in #111
• Support intern lanuage model by @go-with-me000 in #51
• [Docs] Update issue templates for proper guidance to discussions by @gaotongxiao in #116
• [Feature] Allow explicitly setting the temperature for API model by @kennymckormick in #121
• [Fix] Fix seed in HFEvaluator by @kennymckormick in #122
• [Feature] Update SC by @Leymore in #126
• 说明文档标题修改 by @anakin-skywalker-Joseph in #125
• [Docs] Update prompt docs by @Leymore in #46
• [Enhancement] Update README.md by @tonysy in #119
• [DOC] Add metric doc by @Ezra-Yu in #118
• [Feature] Evaluating acc based on minimum edit distance, update SIQA by @gaotongxiao in #130
• [Feature] Several enhancements by @gaotongxiao in #142
• [Doc] update acknowledgements by @Leymore in #147
• Fix typo in readme by @zhouzaida in #152
• [Feature]: Use multimodal by @YuanLiuuuuuu in #73
• [Refine] Refine PR #122 by @kennymckormick in #123
• [Enhancement] Optimize OpenAI models by @gaotongxiao in #128
• Update pre-commit ignore-word list by @gaotongxiao in #162
• [Script] Add scripts to evaluate MMBench by @kennymckormick in #161
• [Doc] Update Readme by @tonysy in #165
• [Feature]: Add mm suport for local runner by @YuanLiuuuuuu in #169
• Calculate max_out_len without hard code for OpenAI model by @zhouzaida in #158
• [API] Refine OpenAI by @kennymckormick in #175
• [Fix] Use a copy of the config object in Task by @gaotongxiao in #174
• Bump requests from 2.28.1 to 2.31.0 by @dependabot in #178
• [Fix] Fix AGIEval multiple choice by @Leymore in #137
• [Feature]: Refactor input and output by @YuanLiuuuuuu in #176
• [Feature] Add Xiezhi SQuAD2.0 ANLI by @Leymore in #101
• [Feature] Support turbomind by @tonysy in #166
• [Enhancement] Add humaneval postprocessor for GPT models & eval config for GPT4, enhance the original humaneval postprocessor by @gaotongxiao in #129
• [Fix] Fix some sc errors by @liushz in #177
• Fix meta template & unit tests by @gaotongxiao in #170
• [Feature] Support CUDA_VISIBLE_DEVICES and multiple tasks on one GPU by @mzr1996 in #148
• [Docs] Enhance issue template by @gaotongxiao in #183
• Skip invalid keys to avoid requesting API by @zhouzaida in #184
• [Feature] update news by @tonysy in #186
• [Feature] Support filtering specified levels message by @zhouzaida in #187
• [Feat] add safety to collections by @yingfhu in #185
• [Docs] Update contribution guide & toc, improve user experience by @gaotongxiao in #188
• [Feature] add llama-oriented dataset configs by @Leymore in #82
• [Feat] update postprocessor to get first option more accurately by @yingfhu in #193
• [Feature] Add LEval datasets by @gaotongxiao in #192
• Bump version to 0.1.2 by @gaotongxiao in #190
• [Fix] fix bug for postprocessor by @yingfhu in #195
• [Doc] update readme by @Leymore in #196

Full Changelog: 0.1.1...0.1.2

v0.1.1

Add some more datasets.

• AGIEval
• anli
• cmmlu
• jigsawmultilingual
• realtoxicprompts
• SQuAD2.0
• TheoremQA
• triviaqa
• xiezhi
• Xsum

v0.1.0

First release with some datasets.

• ARC
• BBH
• ceval
• CLUE
• FewCLUE
• GAOKAO-BENCH
• LCSTS
• math
• mbpp
• mmlu
• nq
• summedits
• SuperGLUE