Fail to run local experiment


#1

Hello,

Today I have updated neptune-cli from version 2.8.6 to version 2.8.9. After that I have tried to run a local experiment with the command

neptune run --config <config_file> <python_script>

I receive the output:

Current project: xxxx/xxxxx

>

> Experiment enqueued, id: YCATMACRO-58

>

> To browse the experiment, follow:

> https://app.neptune.ml/-/dashboard/experiment/2d288971-89da-4508-9240-6378bfa04812?getStartedState=folded

>

Calculated experiment snapshot size: 1.23 MB

Sending sources to server: 100%|███████████| 1.23M/1.23M [00:59<00:00, 20.6kB/s]

2018-08-17 14:10:00,918 deepsense.neptune.common.api.utils ERROR utils.py:126 - log_exception() Failed to call finalize_experiment_upload

2018-08-17 14:10:00,919 deepsense.neptune.common.api.utils ERROR utils.py:127 - log_exception() status: 502, response_message: <html>

<head><title>502 Bad Gateway</title></head>

<body bgcolor="white">

<center><h1>502 Bad Gateway</h1></center>

<hr><center>nginx</center>

</body>

</html>

Traceback (most recent call last):

File "/usr/local/lib/python3.5/dist-packages/deepsense/neptune/common/api/utils.py", line 157, in func_wrapper

return func(*args, **kwargs)

File "/usr/local/lib/python3.5/dist-packages/deepsense/neptune/common/api/utils.py", line 194, in func_wrapper

return func(*args, **dict(kwargs, headers_handler=headers_handler))

File "/usr/local/lib/python3.5/dist-packages/deepsense/generated/swagger_client/apis/default_api.py", line 4286, in finalize_experiment_upload

headers_handler=params.get(‘headers_handler’))

File "/usr/local/lib/python3.5/dist-packages/deepsense/generated/swagger_client/api_client.py", line 333, in call_api

response_type, auth_settings, callback, headers_handler)

File "/usr/local/lib/python3.5/dist-packages/deepsense/generated/swagger_client/api_client.py", line 155, in __call_api

post_params=post_params, body=body)

File "/usr/local/lib/python3.5/dist-packages/deepsense/generated/swagger_client/api_client.py", line 369, in request

body=body)

File "/usr/local/lib/python3.5/dist-packages/deepsense/generated/swagger_client/rest.py", line 164, in POST

body=body)

File "/usr/local/lib/python3.5/dist-packages/deepsense/generated/swagger_client/rest.py", line 132, in request

raise ApiException(http_resp=r)

deepsense.generated.swagger_client.rest.ApiException: (502)

Reason: Bad Gateway

HTTP response headers: {‘Alt-Svc’: ‘clear’, ‘Date’: ‘Fri, 17 Aug 2018 14:10:00 GMT’, ‘Content-Length’: ‘166’, ‘Server’: ‘nginx’, ‘Content-Type’: ‘text/html’, ‘Via’: ‘1.1 google’}

HTTP response body: <html>

<head><title>502 Bad Gateway</title></head>

<body bgcolor="white">

<center><h1>502 Bad Gateway</h1></center>

<hr><center>nginx</center>

</body>

</html>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/usr/local/lib/python3.5/dist-packages/deepsense/neptune/common/api/utils.py", line 130, in func_wrapper

return func(*args, **kwargs)

File "/usr/local/lib/python3.5/dist-packages/deepsense/neptune/common/api/utils.py", line 173, in func_wrapper

raise NeptuneServerResponseErrorException(exc.status, exc)

deepsense.neptune.common.api.exceptions.NeptuneServerResponseErrorException: status: 502, response_message: <html>

<head><title>502 Bad Gateway</title></head>

<body bgcolor="white">

<center><h1>502 Bad Gateway</h1></center>

<hr><center>nginx</center>

</body>

</html>

status: 404, response_message: default backend - 404

On Neptune dashboard the experiment remains in State “Creating”.

I’m running neptune on Ubuntu 16.04 on an AWS EC2 instance.

Thank you for your help.

Riccardo


#2

Moreover, I’ve noticed that in general now Neptune takes a longer time to switch from state Creating to Running.


#3

Hi @tessellir,

The newer version of the CLI is not at fault here.
We know the root cause of this problem and are currently working to make it better.

Sorry for this!

Best regards,
Piotr