What can cause the pilot to the FAILED state?

[liyu886]

Describe the question
In the MolecularDynamics stage, the output is as follows.

Update: DeepDriveMD state: SCHEDULING
Update: DeepDriveMD.MolecularDynamics state: SCHEDULING
Update: DeepDriveMD.MolecularDynamics.task.0000 state: SCHEDULING
Update: DeepDriveMD.MolecularDynamics.task.0000 state: SCHEDULED
Update: DeepDriveMD.MolecularDynamics state: SCHEDULED
ok
submit:########################################################################
Update: DeepDriveMD.MolecularDynamics.task.0000 state: SUBMITTING
closing session re.session.xxxxxxxxxxxxx.019314.0000
close pilot manager
wait for 1 pilot(s)
0 ok
ok
session lifetime:31.9s ok
close task manager ok
wait for 1 pilot(s)
0 timeout
All components terminated

In pmgr.log, there is an ERROR : [Callback]: pilot 'pilot.0000' failed (exit). But I don't know the reason that cause pilot failed. Please help me, thank you.

[braceal]

Thank you for your question. At first glance, this error may come from the radical layer which could point to a configuration issue, rather than the application layer. @andre-merzky Do you have any thoughts on what could cause this error? Thanks!

If you can send us information on the system you are trying to run on, as well as a .tar.gz of the outputs (experiment directory as well as radical output files), the yaml configuration file, and the full stdout, it will help us debug the issue. @liyu886