Http request to Neptune server failed

i started experimenting with neptune today, from an aws ec2 instance.

I had done 9 quick experiments for a project, just to see how neptune works, and it was working fine.

But now i get

Http request to Neptune server failed: RequestsFutureAdapterConnectionError(). Retry in 1 seconds.
Http request to Neptune server failed: RequestsFutureAdapterConnectionError(). Retry in 2 seconds.
Http request to Neptune server failed: RequestsFutureAdapterConnectionError(). Retry in 4 seconds.
^CTraceback (most recent call last):
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/urllib3/connection.py”, line 159, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/urllib3/util/connection.py”, line 57, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File “/usr/lib/python3.6/socket.py”, line 745, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/urllib3/connectionpool.py”, line 600, in urlopen
chunked=chunked)
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/urllib3/connectionpool.py”, line 343, in _make_request
self._validate_conn(conn)
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/urllib3/connectionpool.py”, line 839, in _validate_conn
conn.connect()
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/urllib3/connection.py”, line 301, in connect
conn = self._new_conn()
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/urllib3/connection.py”, line 168, in _new_conn
self, “Failed to establish a new connection: %s” % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7fa83ec9e4e0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/requests/adapters.py”, line 449, in send
timeout=timeout
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/urllib3/connectionpool.py”, line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/urllib3/util/retry.py”, line 398, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host=‘ui.neptune.ml’, port=443): Max retries exceeded with url: /api/backend/swagger.json (Caused by NewConnectionError(’<urllib3.connection.VerifiedHTTPSConnection object at 0x7fa83ec9e4e0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution’,))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/bravado/http_future.py”, line 124, in wrapper
return func(self, *args, **kwargs)
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/bravado/http_future.py”, line 290, in _get_incoming_response
inner_response = self.future.result(timeout=timeout)
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/bravado/requests_client.py”, line 392, in result
**settings
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/requests/sessions.py”, line 646, in send
r = adapter.send(request, **kwargs)
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/requests/adapters.py”, line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host=‘ui.neptune.ml’, port=443): Max retries exceeded with url: /api/backend/swagger.json (Caused by NewConnectionError(’<urllib3.connection.VerifiedHTTPSConnection object at 0x7fa83ec9e4e0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution’,))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/neptune/utils.py”, line 191, in wrapper
return func(*args, **kwargs)
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/neptune/internal/backends/hosted_neptune_backend.py”, line 888, in _get_swagger_client
http_client=self._http_client
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/bravado/client.py”, line 96, in from_url
spec_dict = loader.load_spec(spec_url)
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/bravado/swagger_model.py”, line 105, in load_spec
self.request_headers,
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/bravado/http_future.py”, line 270, in result
incoming_response = self._get_incoming_response(timeout)
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/bravado/http_future.py”, line 128, in wrapper
self.future._raise_connection_error(exception)
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/bravado/http_future.py”, line 91, in _raise_connection_error
self._raise_error(BravadoConnectionError, ‘ConnectionError’, exception)
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/bravado/http_future.py”, line 82, in _raise_error
sys.exc_info()[2],
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/six.py”, line 692, in reraise
raise value.with_traceback(tb)
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/bravado/http_future.py”, line 124, in wrapper
return func(self, *args, **kwargs)
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/bravado/http_future.py”, line 290, in _get_incoming_response
inner_response = self.future.result(timeout=timeout)
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/bravado/requests_client.py”, line 392, in result
**settings
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/requests/sessions.py”, line 646, in send
r = adapter.send(request, **kwargs)
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/requests/adapters.py”, line 516, in send
raise ConnectionError(e, request=request)
bravado.http_future.RequestsFutureAdapterConnectionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “main.py”, line 8, in
api_token=‘confidential’)
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/neptune/init.py”, line 133, in init
backend = HostedNeptuneBackend(api_token, proxies)
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/neptune/utils.py”, line 191, in wrapper
return func(*args, **kwargs)
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/neptune/internal/backends/hosted_neptune_backend.py”, line 70, in init
.format(self.api_address))
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/neptune/utils.py”, line 198, in wrapper
time.sleep(2 ** retry)
KeyboardInterrupt

it works from my local desktop, but stopped working from the ec2 server. Can someone please help

Hey @aunick2017, thanks for reaching out!

Behavior like: “Retry in N seconds” is fine. It is due to connection interruptions -> in such case Neptune client retries to log data. Thanks to this, your experiment is robust to connection issues.

There is no need to kill the process. Let it just continue as usual. Please try, it and let me know if it works :slight_smile:

Best,
Kamil

thanks for the reply kamil :slightly_smiling_face:

I left it overnight on the ec2 instance,

WARNING: It is not secure to place API token in your source code. You should treat it as a password to your account. It is strongly recommended to use NEPTUNE_API_TOKEN environment variable instead. Remember not to upload source file with API token to any public repository.
Http request to Neptune server failed: RequestsFutureAdapterConnectionError(). Retry in 1 seconds.
Http request to Neptune server failed: RequestsFutureAdapterConnectionError(). Retry in 2 seconds.
Http request to Neptune server failed: RequestsFutureAdapterConnectionError(). Retry in 4 seconds.
Http request to Neptune server failed: RequestsFutureAdapterConnectionError(). Retry in 8 seconds.
Http request to Neptune server failed: RequestsFutureAdapterConnectionError(). Retry in 16 seconds.
Http request to Neptune server failed: RequestsFutureAdapterConnectionError(). Retry in 32 seconds.
Http request to Neptune server failed: RequestsFutureAdapterConnectionError(). Retry in 64 seconds.
Http request to Neptune server failed: RequestsFutureAdapterConnectionError(). Retry in 128 seconds.
Http request to Neptune server failed: RequestsFutureAdapterConnectionError(). Retry in 256 seconds.
Http request to Neptune server failed: RequestsFutureAdapterConnectionError(). Retry in 512 seconds.
Http request to Neptune server failed: RequestsFutureAdapterConnectionError(). Retry in 1024 seconds.
Traceback (most recent call last):
File “main.py”, line 8, in
api_token=‘editted for security’)
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/neptune/init.py”, line 133, in init
backend = HostedNeptuneBackend(api_token, proxies)
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/neptune/utils.py”, line 191, in wrapper
return func(*args, **kwargs)
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/neptune/internal/backends/hosted_neptune_backend.py”, line 70, in init
.format(self.api_address))
File “/home/ubuntu/.virtualenvs/dl4cv/lib/python3.6/site-packages/neptune/utils.py”, line 218, in wrapper
raise ConnectionLost()
neptune.api_exceptions.ConnectionLost: Connection lost. Please try again.

it works from my desktop still.

Please advise, thanks!

the ec2 instances it was failing on were in the OREGON region. I tried in the SYDNEY region and it seems to work fine.

Still dont know why it doesnt work in teh ORGEON region because initially it was working

Hey @aunick2017,

Glad to hear that Neptune works on SYDNEY region. I need to consult our engineers regarding the OREGON region. Thanks for letting me know :slight_smile:

Just wanted to make sure: with SYDNEY region you can proceed with Neptune without issues, right?

Best,
Kamil

Hi @aunick2017 ,

I’m going to look into this, it might be possible that the IPs are blocked.

I’ll get back to you once I know more.

Best regards,
Hubert Jaworski

Hi,

We found out that this is a DNS issue and oregon region does not resolve our domain.

While we’re working on the issue, we can offer a quickfix that should help in the meantime:

As root, run the following command on the ec2 instance:

echo "35.186.223.74 ui.neptune.ml" >> /etc/hosts

After this command, run cat /etc/hosts and you should see an extra line with Neptune address added

[root@ip-172-31-26-196 ec2-user]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost6 localhost6.localdomain6
[root@ip-172-31-26-196 ec2-user]# echo "35.186.223.74 ui.neptune.ml" >> /etc/hosts
[root@ip-172-31-26-196 ec2-user]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost6 localhost6.localdomain6
35.186.223.74 ui.neptune.ml

And after that Your code should start working.
You can verify that by running:

curl https://ui.neptune.ml/api/backend/healthz

I’ll keep you updated on DNS progress.

Best regards,
Hubert

thanks for the reply!

Awesome. Im going to continue using the SYDNEY region for now, but if I need to use OREGON again, then i shall implement this quick fix.

Kind Regards,

Nick

Hi @hubert.jaworski.nept,

I start getting the same error (when I try to start a new experiment):
Http request to Neptune server failed: RequestsFutureAdapterConnectionError(). Retry in 1 seconds.
Http request to Neptune server failed: RequestsFutureAdapterConnectionError(). Retry in 2 seconds.

I think in my case the authorization is the problem. Without the ‘X-Neptune-Api-Token’ header the
https://ui.neptune.ml/api/backend/v1/authorization/oauth-token endpoint sends a 400 bad request (missing X-Neptune-Api-Token)
When I send the token I got 408 request timeout errors.

This started yesterday afternoon.

Thanks for your help,
Peter

Hi @pestipeti

I pinpointed the issue and took action that should’ve fixed it.

Let us know if this happens again.

Best regards,
Hubert

It is working, thank you.