Using Keras integration breaks training output

#1

When using the Keras integration with

ctx = neptune.Context()
ctx.integrate_with_keras()

and then running the script with neptune run, the carriage returns in Keras’s verbose output stop working, leading to every update to the output resulting in a newline being printed, which is very frustrating.
Essentially the output looks like this:

1306/5947 [=====>........................] - ETA: 1:11:23 - loss: 1.8453 - acc: 0.5650 - map3: 0.7496
1307/5947 [=====>........................] - ETA: 1:11:22 - loss: 1.8449 - acc: 0.5651 - map3: 0.7497
1308/5947 [=====>........................] - ETA: 1:11:21 - loss: 1.8445 - acc: 0.5652 - map3: 0.7497
1309/5947 [=====>........................] - ETA: 1:11:20 - loss: 1.8441 - acc: 0.5652 - map3: 0.7498
1310/5947 [=====>........................] - ETA: 1:11:19 - loss: 1.8437 - acc: 0.5653 - map3: 0.7499
1311/5947 [=====>........................] - ETA: 1:11:18 - loss: 1.8433 - acc: 0.5654 - map3: 0.7500
1312/5947 [=====>........................] - ETA: 1:11:17 - loss: 1.8430 - acc: 0.5655 - map3: 0.7500
1313/5947 [=====>........................] - ETA: 1:11:16 - loss: 1.8426 - acc: 0.5656 - map3: 0.7501

This does not occur when the script is just run with the python command, although in this case the neptune integration does not work, so the issue occurs somewhere in the Keras integration.

#2

Hi,

Neptune reads the standard output stream from your experiment and stores it as lines of text.

We investigated this and Keras prints out those lines like:

1309/5947 [=====>........................] - ETA: 1:11:20 - loss: 1.8441 - acc: 0.5652 - map3: 0.7498\r

Appending the \r to each line.

We’ll look into how to adjust out front-end for this.

Best regards,
Hubert

#3

In a similar issue, I am getting a problem with tqdm progressbars, where the progressbar is not printed at all until \n is written (i.e. the progressbar has completed), which kind of defeats the purpose of the progressbar in the first place…

(This is without using the keras integration, there might be an issue with the way you are capturing stdout/stderr in neptune). Once again the issue dissapears when the script is called with python instead of neptune run.

#4

Hi @hubert.jaworski.nept,

Do you have an update on this issue? Specifically relating to tqdm as in my previous comment, it is practically unusable when using neptune-cli (as is any other progressbar which does not print a newline on every update). A workaround would be to add a flag or something to stop neptune from capturing the stdout/stderr for that run (so that the output does not break).

Thanks

#5

Hi,

The root cause of this is how we pipe output from child process.

We are currently testing a fix for this and we expect to release it some time next week. Our goal is to make Neptune CLI as invisible to console as possible.

Best Regards,
Hubert

#6

Hi @anokas

Just a quick update:

Even if we deploy our fix, due to how Keras is written (here and here) it will still print out every progress bar update with a newline character (so our fix won’t do anything for this)

Our testing shows that import ipykernel as first line main.py helps keep the output tidy (in current version keras will behave the same as tqdm), I thought it might help You for now

Best Regards,
Hubert Jaworski

#7

Hi,

We released new CLI (2.8.23) with fixes to those bugs.

We also released new environments with tensorflow 1.12, keras 2.2.4 and pytorch 1.0 libs

Progressbars should now behave correctly, but as mentioned before, Keras has a mind of it’s own and will still print newlines for each progressbar update unless experiment has import ipykernel

Best regards,
Hubert