Neptune run: ModuleNotFoundError: No module named 'common.logger'


#1

I have the following neptune project structure:

source
└─common
└─logger.py
└─main.py
neptune.yaml

I’ve grouped all my source files under a source folder. Inside the source folder there is a main.py and a common folder containing logger.py

The neptune.yaml file is located in the root of the project.

In my main.py:

import os
import sys

# append common modules to sys.path
sys.path.append(os.path.join(os.path.dirname(sys.path[0]), '.'))
sys.path.append(os.path.join(os.path.dirname(sys.path[0]), '/project/software/tutorial/pytorch/pytorch-template/source'))
sys.path.append(os.path.join(os.path.dirname(sys.path[0]), '/project/software/tutorial/pytorch/pytorch-template/source/common/'))

import common.logger as logger

When I run the following command to run neptune locally, without uploading any sources:

neptune run source/main.py – train_pipeline --pipeline_name lenet5 --config neptune.yaml

I get the following error:

Calculated experiment snapshot size: 0 Bytes   
Sending sources to server: 0.00B [00:00, ?B/s]
['', '/tool/python/conda/env/gis36/lib/python3.6/site-packages/deepsense/neptune', '/tool/python/conda/env/gis36/lib/python36.zip', '/tool/python/conda/env/gis36/lib/python3.6', '/tool/python/conda/env/gis36/lib/python3.6/lib-dynload', '/tool/python/conda/env/gis36/lib/python3.6/site-packages', '/tool/python/conda/env/gis36/lib/python3.6/site-packages/aeosa', '/tool/python/conda/env/gis36/lib/python3.6/site-packages/html5lib-0.999999999-py3.6.egg', '/tool/python/conda/env/gis36/lib/python3.6/site-packages/torchvision-0.2.1-py3.6.egg', '.', '/project/software/tutorial/pytorch/pytorch-template/source', '/project/software/tutorial/pytorch/pytorch-template/source/common/', '/project/software/tutorial/pytorch/pytorch-template/source/pipeline']
Traceback (most recent call last):
  File "/tool/python/conda/env/gis36/lib/python3.6/site-packages/deepsense/neptune/job_wrapper.py", line 107, in <module>
    execute()
  File "/tool/python/conda/env/gis36/lib/python3.6/site-packages/deepsense/neptune/job_wrapper.py", line 103, in execute
    execfile(job_filepath, job_globals)
  File "/tool/python/conda/env/gis36/lib/python3.6/site-packages/past/builtins/misc.py", line 82, in execfile
    exec_(code, myglobals, mylocals)
  File "source/main.py", line 20, in <module>
    import common.logger as logger
ModuleNotFoundError: No module named 'common.logger'
Calculated experiment snapshot size: 0 Bytes   
Process exited with return code 1.

#2

You should add empty __init__.py file to your common and source directories to make them a python packages. Then, since a working directory is the one you executed neptune command from (not the one in which main.py file is), you should import source.common.logger.

You can find more helpful information here: https://stackoverflow.com/questions/2349991/python-how-to-import-other-python-files