[Feature] Update SC #126
Merged
[Feature] Update SC #126
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
liushz
approved these changes
Jul 28, 2023
gaotongxiao
reviewed
Aug 1, 2023
| ``` | ||
|
|
||
| ```{note} | ||
| 注意，OpenCompass 默认使用默认使用 argmax 的方式采样下一个 token，因此若不指定采样参数，模型每次的推理结果将会是完全一致的，多轮评测将会失效。 |
| 注意，OpenCompass 默认使用默认使用 argmax 的方式采样下一个 token，因此若不指定采样参数，模型每次的推理结果将会是完全一致的，多轮评测将会失效。 | ||
| ``` | ||
|
|
||
| Where `SAMPLE_SIZE` is the number of reasoning paths in Self-Consistency, higher value usually outcome higher performance. The following figure from the paper demonstrates the relation between reasoning paths and performance in several reasoning tasks: |
There was a problem hiding this comment.
From which paper? We need to make a citation
There was a problem hiding this comment.
Also need to point out that the sample generation_kwargs only works for HuggingFace models.
Comment on lines
+70
to
+72
| Where `SAMPLE_SIZE` is the number of reasoning paths in Self-Consistency, higher value usually outcome higher performance. The following figure from the paper demonstrates the relation between reasoning paths and performance in several reasoning tasks: | ||
|  | ||
| From the figure, it can be seen that in different reasoning tasks, performance tends to improve as the number of reasoning paths increases. However, for some tasks, increasing the number of reasoning paths may reach a limit, and further increasing the number of paths may not bring significant performance improvement. Therefore, it is necessary to conduct experiments and adjustments on specific tasks to find the optimal number of reasoning paths that best suit the task. |
There was a problem hiding this comment.
A blank line between the paragraph and image makes layout better
|
|
||
| ## 3. Self-Consistency | ||
|
|
||
| The SC (Self-Consistency) method is proposed in [this paper](https://arxiv.org/abs/2203.11171), which will sample multiple reasoning paths for the question, and make majority voting to the generated answers for LLMs. This method displays remarkable proficiency among reasoning tasks with high accuracy but may consume more time and resources when inferencing, because of the majority voting strategy. In OpenCompass, you can simply set SC method in the dataset config like: |
There was a problem hiding this comment.
We should explicitly tell readers they have to replace GenInferencer with SCInferencer
| ) | ||
| ) | ||
| gsm8k_eval_cfg = dict(sc_size=SAMPLE_SIZE) | ||
| ``` |
There was a problem hiding this comment.
Need a link to the new gsm8k config for interested readers to follow
| sc_results.append(results) | ||
| sc_prediction = list(map(list, zip(*sc_results))) | ||
| generated = sc_prediction | ||
| print(generated) |
| save_every: Optional[int] = None, | ||
| fix_id_list: Optional[List[int]] = None, | ||
| sc_size: Optional[int] = 1, | ||
| infer_type: Optional[str] = '', |
There was a problem hiding this comment.
infer_type is not even used here.
Its implementation seems pretty close to GenInferencer. Consider employing inheritance to cut down on code redundancy and ease future maintenance.
| @@ -164,6 +186,14 @@ def _extract_role_pred(self, s: str, begin_str: Optional[str], | |||
|
|
|||
| return s[start:end] | |||
|
|
|||
| def _get_vote_out( | |||
There was a problem hiding this comment.
A short docstring is required here.
|
Don't forget to update the ToC of documentation |
go-with-me000
pushed a commit
to go-with-me000/opencompass
that referenced
this pull request
Oct 9, 2023
* add self-consistency * add CoT method Self-Consistency * fix typo error and update openicl_eval * add tydiQA-GoldP task * fix sc * rename gsm8k_sc * fix sc * add self-consistency doc * refine sc --------- Authored-by: liushz <[email protected]>
liuyaox
pushed a commit
to liuyaox/opencompass
that referenced
this pull request
Jun 26, 2024
* add self-consistency * add CoT method Self-Consistency * fix typo error and update openicl_eval * add tydiQA-GoldP task * fix sc * rename gsm8k_sc * fix sc * add self-consistency doc * refine sc --------- Authored-by: liushz <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The same as #57