Running with `neptune run` completely breaks pickle


#1

When running a script under neptune, pickle stops working on any classes defined within the script (throwing an attribute lookup error). If I had to guess, something with the way you are calling the python script from within neptune-cli is meaning that classes defined within the python script are no longer in __main__ and so cannot be pickled.

Here is a minimally reproducible example:

import neptune
import pickle

ctx = neptune.Context()

class TestClass:
    def __init__(self):
        self.x = [1, 2, 3, 4]

cls = TestClass()

ctx.channel_send('test', 0)
s = pickle.dumps(cls)
print(s)

Running this with neptune run test.py leads to:

Traceback (most recent call last):
  File "/home/m09170/anaconda3/lib/python3.6/site-packages/neptune/job_wrapper.py", line 107, in <module>
    execute()
  File "/home/m09170/anaconda3/lib/python3.6/site-packages/neptune/job_wrapper.py", line 103, in execute
    execfile(job_filepath, job_globals)
  File "/home/m09170/anaconda3/lib/python3.6/site-packages/past/builtins/misc.py", line 82, in execfile
    exec_(code, myglobals, mylocals)
  File "test.py", line 12, in <module>
    s = pickle.dumps(cls)
_pickle.PicklingError: Can't pickle <class '__main__.TestClass'>: attribute lookup TestClass on __main__ failed

Whereas when running just with python test.py it works perfectly (although neptune integration is disabled).

As a workaround, I found that by doing neptune run python test.py appears to work although I have to test it on more significant scripts.

I think this is quite a serious bug as it also breaks eg. PyTorch model saving and various other packages that rely on pickle.


#2

Hi @anokas,

Thanks for your report. We actually have a bug in our code and we will fix it as soon as possible. Especially thank you for reproducible example and a temporary workaround. It’s going to be helpful.


#3

Hi,

Just a quick update:

We found out what was causing this and we are currently testing the fix.

We should release it early next week

Regards,
Hubert


#4

We’ve just released a new version of out CLI (2.8.16) which fixes the issue

Once you update, your neptune run commands should work properly.

We’re going to roll out this update to our environments within the next hour and it’ll work with neptune send too afer that

Best regards,
Hubert