When running a script under neptune, pickle stops working on any classes defined within the script (throwing an attribute lookup error). If I had to guess, something with the way you are calling the python script from within
neptune-cli is meaning that classes defined within the python script are no longer in
__main__ and so cannot be pickled.
Here is a minimally reproducible example:
import neptune import pickle ctx = neptune.Context() class TestClass: def __init__(self): self.x = [1, 2, 3, 4] cls = TestClass() ctx.channel_send('test', 0) s = pickle.dumps(cls) print(s)
Running this with
neptune run test.py leads to:
Traceback (most recent call last): File "/home/m09170/anaconda3/lib/python3.6/site-packages/neptune/job_wrapper.py", line 107, in <module> execute() File "/home/m09170/anaconda3/lib/python3.6/site-packages/neptune/job_wrapper.py", line 103, in execute execfile(job_filepath, job_globals) File "/home/m09170/anaconda3/lib/python3.6/site-packages/past/builtins/misc.py", line 82, in execfile exec_(code, myglobals, mylocals) File "test.py", line 12, in <module> s = pickle.dumps(cls) _pickle.PicklingError: Can't pickle <class '__main__.TestClass'>: attribute lookup TestClass on __main__ failed
Whereas when running just with
python test.py it works perfectly (although neptune integration is disabled).
As a workaround, I found that by doing
neptune run python test.py appears to work although I have to test it on more significant scripts.
I think this is quite a serious bug as it also breaks eg. PyTorch model saving and various other packages that rely on pickle.