Grid search behaviour


#1

Hi,
when I run grid search over some parameters, a few experiments are enqueued. If my first experiment fails, others still are in “Waiting” status, and I cannot run them. I think they should start automatically after the first experiment fails, am I right?
Cheers


#2

HI Andrzej,

From what I can see you are running your grid search locally (neptune run).

What happened is your Neptune CLI process died together with the first experiment process and failed to mark the rest of experiments as ‘Failed’.

Unfortunately, they will never ‘start’ and you can manually abort them from the web page.

Good news is, this doesn’t happen when using ‘neptune send’

It is definitely a bug on our side and we will try to improve this behaviour in future CLI releases.