Neptune send question


#1

Does it assume there is a main.py at the top of my project directory?

That seems restrictive. I would rather do something like

neptune send utils/run_my_experiment.py --hidden 20 --epochs 6

#2

Hi Prasad!

Please, take a look at our documentation at http://docs.neptune.ml/config/experiment/#executable

For your convenience, I’ll quote it below:

By default, neptune send and neptune run look for main.py file. If you want to use different file as entry point to your program, use executable setting.

In neptune.yaml:

executable: other_python_script.py

Via command line, you can pass it as executable argument after neptune send or neptune run:

neptune send --executable other_python_script.py

or even easier as a positional argument:

neptune send other_python_script.py

Best regards,
Piotr


#3

Ok I found that documentation, but still having trouble running a very simple executable.

I do have a minimal neptune.yaml in my project folder, but I get an error saying:

validation.invalid
 --- All found errors ---
["Key 'project' was not defined. Path: ''"]
Errors found but will not raise exception...
The provided job configuration {'local_path': '/Users/pchalasani/MediaMath/GitForks/predevent/neptune.yaml', 'global_path': '/Users/pchalasani/.neptune.yaml'} is invalid! Validation errors: 1. Key 'project' was not defined. Path: ''

my neptune.yaml file looks like this:

project-key: predevent
open-webbrowser: false

executable: utils/run_experiment.py
name: predevent
description: run predevent train/test
project: predevent
tags: [tag1, tag2, tag3, tag4]

exclude:
  - data
  - test/data
  - utils/data

#4

And does the exclude feature in the yaml support wildcards like in .gitignore files? This would be useful, to avoid uploading all kinds of junk files


#5

Looks like we have a bug in our documentation - thanks for finding it :slight_smile:

The project key in your neptune.yaml is not recognized by our parser. Removing it will help.

Exclude doesn’t support wildcards at this point, but this is a feature coming in the near future.

Best,
Piotr


#6

Ok removed it, but now it complains:

pchalasani@MM-MAC-3358 ~/D/M/G/predevent> neptune send
Project with key 'predevent' not found.

what am I supposed to use for project-key – it’s making me setup all kinds of unnecessary stuff, and I just want to run my experiment :slight_smile:


#7

Actually I also removed project-key and now it’s attempting to upload 21 GB of stuff, but I told it to ignore the data folder (20GB) in my neptune.yaml and it seems to be ignoring it


#8

And specifying --exclude on cmd line doesn’t work either -

pchalasani@MM-MAC-3358 ~/D/M/G/predevent> neptune send --exclude data
Calculated experiment snapshot size: 22.79 GB

#9

It works in my case - both through .yaml and command line. The only problem I found with .yaml config is that you cannot do it in this way:

exclude: file_or_dir

but you need to do it like that:

exclude:
  - file_or_dir

When it comes to command line after exclude you need to provide a list of file like here --exclude file1 file 2 file3

Are you sure that there are no other files or directories (maybe hidden .)?


#10

ok so I moved the big data folder to a different location, now the send command made a snapshot of only 45MB which is good, however my job failed, and now I don’t know how to look at the log output files.

This is the most basic functionality I’m looking for – run an experiment, look at my console log output files, forget about channels, etc. I just want to see the console log output. How do I do that?


#11

Hi Prasad,

Thank you again for your time last Friday. By COB tomorrow we should have all the issues we discussed fixed:

  • requirements.txt support
  • better docker images - where in particular cario library is installed in the OS (you will still need to install cairocffi via pip)
  • stderr & stdout captured by default and presented as channels
  • fixed exclude functionality

We are still thinking about .neptuneignore file. I have feeling that we need to decide .ignore files vs exclude (in command line and/or in neptune.yaml). What do you think?


#12

I like the idea of .neptuneignore, it’s more flexible

I like that in the .ignore file we can put wildcards, etc


#13

Just to add some more thoughts/feedback. I guess if you make your “exclude” syntax more flexible (allow wildcards, etc), then I’m indifferent to whether to use a .ignore file or “exclude”. However the other two platforms I’m looking at, floydHub, and cnvrg.io both offer a .ignore facility, and it works very well.


#14

We have 3 options:

  1. support both: then we user would be able to specify in neptune.yaml or via command line arguments “ignore file name”, and Neptune would use them as ignore files additionally to exclude option (then you would be able to use .gitingore just by one line in neptune.yaml
  2. only ignore files
  3. exclude with wildcard support

What do you think 1, 2 or 3?


#15

1 sounds best (“ignore file name” in yaml)


#16

I discussed it with the team and we also agree that 1) is the best option. I will let you know once it is done.


#17

I think you forgot to fix one more issue: my scripts take command-line arguments of their own, and neptune should not mess with those. E.g. I tried two ways and they failed. This works perfectly on other platforms.
It is important to support command-line arguments to my scripts. Other mechanisms are not going to be good, e.g. specifying them in yaml etc.

pchalasani@MM-MAC-3358 ~/D/M/G/predevent> neptune send utils/select_experiment.py --config 143209-780262-only-visitors-cfg1 --quick
Config file (143209-780262-only-visitors-cfg1) does not exist.
pchalasani@MM-MAC-3358 ~/D/M/G/predevent> neptune send "utils/select_experiment.py --config 143209-780262-only-visitors-cfg1 --quick"
Executable 'utils/select_experiment.py --config 143209-780262-only-visitors-cfg1 --quick' does not exist.

#18

Also, the neptune.yaml exclude functionality does not seem to work. This the entire contents of my neptune.yaml, and I tried running a script that does not take any command-line arguments (to avoid the problem mentioned above), and it calculated an experiment snapshot size of 800 MB ! I know my code is less than 15MB.

exclude:
- data
- .git
- .eggs
- floyd_requirements.txt
- .domino
- eggs
- lib
- lib64
- parts
- results
- docs
- my_runs
- test/data
- .cache
- .idea
- utils/my_runs
- utils/data
- utils/results
- runs

#19

You are right - we will fix it ASAP.


#20

Have you updated neptune-cli? CAn you please type neptune --version and let me know the output.