MOOSE-Chem2: Exploring LLM Limits in Fine-Grained Scientific Hypothesis Discovery via Hierarchical Search
MOOSE-Chem2 introduces the task of fine-grained scientific hypothesis discoveryโenabling LLMs to move beyond vague ideas toward detailed hypotheses with sufficient methodological specifics for lab testing.
This repository includes the dataset, method implementation, and analysis scripts for MOOSE-Chem2. It also can be used in a copilot setting where it takes in customized research question.
Create and activate a dedicated Conda environment, then install the required dependencies:
conda create -n msc python=3.10
conda activate msc
pip install -r requirements.txtOpen main.sh and set the following:
api_type(0: OpenAI; 1: Azure; 2: Gemini)api_keyeval_api_key(can be the same asapi_key)base_url
Set function_mode=5 in main.sh.
Choose one of the following two paths:
- Set
--if_load_from_moosechem_ranking_file 1 - Set
--moosechem_ranking_file_pathto the--output_dirfrom./Method/evaluate.pyin MOOSE-Chem - Open
./Preprocessing/custom_research_background_dumping.py, go tomoosechem_ranking_file_to_json()and manually fill inbackground_survey.
- Set
--if_load_from_moosechem_ranking_file 0 - In
./Preprocessing/custom_research_background_dumping.py, go toresearch_background_to_json()and manually set:research_questionbackground_surveycoarse_grained_hypothesis
Then run:
bash main.shIf your research question falls within chemistry or geophysics, go to ./Method/utils.py and import the appropriate prompt module:
from Method.discipline_specific_prompt_chemistry load xxx # or
from Method.discipline_specific_prompt_geophysics load xxxKeep only the one relevant to your domain and comment out the other.
If your research area is outside chemistry or geophysics:
- Create a new prompt file by mimicking the structure of
discipline_specific_prompt_chemistry.py - In
./Method/utils.py, comment out the existing chemistry and geophysics imports - Import your custom prompt module instead
This ensures the system uses discipline-specific guidance tailored to your research field.
Set the following in main.sh:
function_mode=0model_nameandeval_model_name(usually the same)if_hierarchical=1num_hierarchy=5if_use_custom_research_background_and_coarse_hyp=1(if use customized input, else set to 0)bkg_idโ index of the research input you created in Step 2, default is 0 (the first one)
Then run:
bash main.sh
โ ๏ธ This step may take up to 400 minutes per input bkg_id.
To display output, set in main.sh:
function_mode=6--start_bkg_idand--end_bkg_idto cover the range ofbkg_ids you processed
(e.g., if onlybkg_id=0was run, set both to 0)
If file loading errors occur:
- Open
./Preprocessing/display_hypothesis.py - Under
# used for match hypothesis file name, ensure the parameter settings match those in Step 4 (main.sh, underif [[ ${function_mode} -eq 0 ]]; then)
Then run:
bash main.sh- Download the checkpoint files from this link.
- Unzip the downloaded archive into the current directory.
- Change the
display_typein./Evaluation/display_result.pyto 1, then runpython ./Evaluation/display_result.py
- Change the
display_typein./Evaluation/display_result.pyto 2, then runpython ./Evaluation/display_result.py
- Change the
display_typein./Evaluation/display_result.pyto 3, then runpython ./Evaluation/display_result.py
Now a part of Table 3 is shown. Other parts of Table 3 can be shown by adjusting:
pairwise_eval_model_name, pairwise_if_multiple_llmmodel_name_1, eval_model_name_1, if_multiple_llm_1model_name_2, eval_model_name_2, if_multiple_llm_2
-
Open
./Evaluation/display_result.pyand set:display_type = 3
-
Comment out the line:
start_end_id_pairs = [[24, 35]]
and uncomment the line:
start_end_id_pairs = [[0, 4], [5, 13], [14, 23], [24, 35], [36, 50]]
-
Set the following variables:
pairwise_eval_model_name, pairwise_if_multiple_llm = ["gpt-4o-mini", 1] model_name_1, eval_model_name_1, if_multiple_llm_1 = ["gpt-4o-mini", "gpt-4o-mini", 0] model_name_2, eval_model_name_2, if_multiple_llm_2 = ["gpt-4o-mini", "gpt-4o-mini", 1]
-
Run the script:
python ./Evaluation/display_result.py
-
Open
./Evaluation/display_result.pyand setdisplay_type = 2. -
Comment out the following lines:
if_multiple_llm = 1 which_exp = [1, 1, 1]
-
Uncomment the following lines:
if_multiple_llm = 0 which_exp = [1, 0, 0]
-
Run the script:
python ./Evaluation/display_result.py
-
Open
main.shand set the following:function_mode=3- Provide values for
api_type,api_key, andbase_url.
-
Fill in the parameters located between:
# shared parameters for pairwise compare and f1 score compare (start) ... # shared parameters for pairwise compare and f1 score compare (end)
-
Set
pairwise_or_f1_compare_modeto:1to conduct pairwise evaluation between two local maximum hypotheses.2to compare a local maximum hypothesis with the ground truth hypothesis and calculate recall.
-
-
If
pairwise_or_f1_compare_mode=1:-
Fill in the section between:
# shared parameters for pairwise compare (start) ... # shared parameters for pairwise compare (end)
-
If
compare_mode=2, also fill in the parameters between:# parameters for pairwise compare (compare_mode 2) (start) ... # parameters for pairwise compare (compare_mode 2) (end)
-
-
If
pairwise_or_f1_compare_mode=2:-
Fill in the parameters between:
# parameters for f1 score compare (start) ... # parameters for f1 score compare (end)
-
-
Finally, run:
bash main.sh
If you found this repository useful, please consider ๐citing:
@inproceedings{yang2025moose2,
title = {MOOSE-Chem2: Exploring LLM Limits in Fine-Grained Scientific Hypothesis Discovery via Hierarchical Search},
author = {Yang, Zonglin and Liu, Wanhao and Gao, Ben and Liu, Yujie and Li, Wei and Xie, Tong and Bing, Lidong and Ouyang, Wanli and Cambria, Erik and Zhou, Dongzhan},
booktitle = {Advances in Neural Information Processing Systems},
year = {2025}
}
This project is licensed under the MIT License. See the LICENSE file for details.
