confused with you two evaluation methods

[yunzqq]

I feel a little bit confused with you two evaluation methods.
May I know can I directly use the Minilongbench as the test samples for evaluating LLMs.
And we use minilongbench_scorer.py to obtain the final scores?
or any other post-processing is necessary?
It is really confused.
Are the final score stored in (one of the two methods) eval_data//example_minilongbench_scores.pkl?

[linggm3]

Hi! Thanks for your questions.

Yes, you can directly use MiniLongBench as the test samples to evaluate your LLMs. The evaluation process is as follows:

1. First, use minilongbench_scorer.py to obtain the individual scores for each of the 237 test samples.

2. Once you have these 237 scores, you can run either eval_by_pred.ipynb or eval_directly.ipynb to calculate the final score on the compressed set.

No additional post-processing is required. Hope this clarifies the process for you!