Evaluation

[xiaoxiao-zhi]

How do you evaluate the three benchmarks?
For example, benchmarks like LongBench v2 do not come with built-in chunks, but I found that the model still generates ref_chunks in the model responses under your eval_results folder. Does additional preprocessing (such as adding chunk tags to the original text) need to be performed during evaluation? If so, could you please provide the corresponding script? Thank you!

[guanzhchen]

Hi,
Thank you for your question. For the evaluation, we use the same setting as training. To reproduce, you can use this function to split the benchmark context as 512-token chunks, with chunk ID assigned. We use the same prompt as that in training data, but with the specific instructions from benchmarks (e.g., answer with A/B/C/D from LongBench v2) appended in the tail. (You may need to adjust the place of these instructions or further use a LLM judge to evaluate if the answer format is right.)

Sorry that we can't provide the original evaluation scripts, as we have no access to the original server now. I believe you can fully produce the evaluation following the steps above.

Thank you.