Question about subtask planning

[Kai-Sheng-caesar]

Hi, thanks for your great work and for releasing the code.

I have a question about the subtask planning module mentioned in the paper. The paper states that Qwen3-VL-8B is used for subtask planning from the instruction. However, I could not clearly locate the corresponding implementation in the released code.

1. Could you please point me to the specific file or function where Qwen3-VL-8B is used for subtask planning?
2. Does this subtask planning module require additional training or fine-tuning, or is Qwen3-VL-8B used in a zero-shot manner to generate the subtasks?

Thanks a lot for your help!

[ys-2020]

Hi, thanks for your interest in ForeAct! We use Qwen3-VL-8B-Instruct in zero-shot manner for subtask planning. We discussed implementation details in Section A.5 (Appendix) of the paper.

We used vLLM/SGLang for VLM serving. Here is also a sample pseudo code snippet for subtask planner integration:

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",    # Your VLM Server URL
    api_key="YOUR_VLM_SERVER_KEY"
)


InitPrompt = f"""
You are a robot controller. Please plan to finish the task in several steps. And give instruction for each step in a concise way.
The task is to "{Task}".

RULES
- During the job, I will continuously give you an observation image of the current state. 
- Based on the observation, please judge if the last instruction has been finished. 
  - If yes, give me the instruction for the next step. 
  - If no, repeat the instruction of the ongoing subtask. 
- You're not required to describe the observation. Only output the instruction for each subtask.

Now, you are only required to output instruction for the first step. {Hint}
"""

Followups = """Pay attention to the latest observation. Firstly, judge if the last instruction has been finished. Secondly, if yes, give me the instruction for the next step; if no, repeat the instruction of the ongoing subtask. 
Your answer should be concise and deterministic.
"""


prompt = InitPrompt if init_round else Followups

user_message = {
      "role": "user",
      "content": [
          {"type": "image_url", "image_url": {"url": data_url_from_pil(input_frame)}},
          {"type": "text", "text": prompt},
      ],
  }
conversation_history.append(user_message)

response = client.chat.completions.create(
    model="Qwen/Qwen3-VL-8B-Instruct",
    messages=conversation_history,
    temperature=0.2
).choices[0].message.content
     
assistant_message = {
    "role": "assistant",
    "content": [
        {"type": "text", "text": response}
    ]
}
conversation_history.append(assistant_message)