Kaggle access from notebook


#1

Hello, I would like to use the neptune.ml notebook as a complete Kaggle workbench. This would include making submissions.

In the notebook, I can !pip install kaggle with success, however when I import kaggle I get the error:


SystemExit: Unauthorized: you must download an API key from https://www.kaggle.com/<username>/account
Then put kaggle.json in the folder /root/.kaggle

I have my kaggle.json file with my API key locally and I have the Sandbox project active with a notebook called SAN-1 open.

From the neptune CLI I did neptune data upload kaggle.json. This puts the kaggle.json file in the top level SAN/uploads directory.

How do I put it in the Neptune folder /root/.kaggle/?

In the notebook, I can’t seem to access /SAN, only /SAN-1 which is one lower. When I do !ls -l ../I get

total 6698
drwxr-xr-x   2 root root       0 May  2 23:42 backup
drwxr-xr-x   1 root root    4096 Apr 26 07:58 bin
drwxr-xr-x   2 root root    4096 Apr 12  2016 boot
drwxr-xr-x   2 root root       0 Apr 23 10:32 cli
drwxr-xr-x   5 root root     380 May  2 23:42 dev
-rwxrwxr-x   1 root root      96 Apr 20 06:15 entrypoint.sh
drwxr-xr-x   1 root root    4096 May  2 23:42 etc
-rw-r--r--   1 root root 1642329 Apr 26 08:00 get-pip.py
drwxr-xr-x   2 root root    4096 Apr 12  2016 home
drwxr-xr-x   4 root root    4096 May  2 23:42 job
drwxr-xr-x   1 root root    4096 Apr 26 07:57 lib
drwxr-xr-x   2 root root    4096 Apr 12 10:26 lib64
drwxr-xr-x   2 root root    4096 Apr 12 10:26 media
drwxr-xr-x   2 root root    4096 Apr 12 10:26 mnt
drwxr-xr-x   3 root root 1025640 May  2 23:56 neptune
-rw-rw-r--   1 root root   75852 Apr 20 06:15 neptune-logo.png
drwxr-xr-x   3 root root    4096 Apr 26 11:48 neptune_contents_manager
drwxr-xr-x   2 root root    4096 Apr 12 10:26 opt
drwxr-xr-x   2 root root 1000307 May  2 23:42 output
drwxr-xr-x   2 root root       0 May  2 23:42 pip
dr-xr-xr-x 136 root root       0 May  2 23:42 proc
drwxrwxr-x  10 1002 1003 3012741 Mar 13 12:28 public
-rw-rw-r--   1 root root     353 Apr 19 13:31 requirements.txt
drwx------   1 root root    4096 May  2 23:42 root
drwxr-xr-x   1 root root    4096 May  2 23:42 run
-rwxrwxr-x   1 root root     420 Apr 20 06:15 run_notebook.sh
drwxr-xr-x   1 root root    4096 Apr 26 07:57 sbin
drwxr-xr-x   2 root root    4096 Apr 26 11:48 source
drwxr-xr-x   2 root root    4096 Apr 12 10:26 srv
dr-xr-xr-x  13 root root       0 May  2 23:55 sys
drwxrwxrwt   1 root root    4096 May  2 23:42 tmp
drwxr-xr-x   1 root root    4096 Apr 12 10:26 usr
drwxr-xr-x   1 root root    4096 Apr 12 10:27 var

I don’t see the uploads directory. Please advise!

Thank you.


Kaggle, TalkingData AdTracking Fraud Detection Challenge
#2

After you upload any file you need to add it to your notebook or experiment as input. You need to do it before you start notebook.

You can do it in two ways.

  1. By command line
neptune experiment send-notebook -i kaggle.json

then it will be available in your notebook in /input directory.
You can check it by typing

!ls -l /input
total 1
-rw-rw-r-- 1 root root 14 May  3 07:06 kaggle.json
  1. You can add input file or directory from website by pressing “Start Notebook” button and adding input in “Start new notebook” configurator

#3

Here is what has worked for me:

!mkdir /root/.kaggle
!cp main.html /root/.kaggle
!ls /root/.kaggle

I guess you can copy any file you want that way.

The /root folder is your home folder in docker. In our notebook environment you have root access in the container.