Using Keras integration breaks training output

#1

When using the Keras integration with

ctx = neptune.Context()
ctx.integrate_with_keras()

and then running the script with neptune run, the carriage returns in Keras’s verbose output stop working, leading to every update to the output resulting in a newline being printed, which is very frustrating.
Essentially the output looks like this:

1306/5947 [=====>........................] - ETA: 1:11:23 - loss: 1.8453 - acc: 0.5650 - map3: 0.7496
1307/5947 [=====>........................] - ETA: 1:11:22 - loss: 1.8449 - acc: 0.5651 - map3: 0.7497
1308/5947 [=====>........................] - ETA: 1:11:21 - loss: 1.8445 - acc: 0.5652 - map3: 0.7497
1309/5947 [=====>........................] - ETA: 1:11:20 - loss: 1.8441 - acc: 0.5652 - map3: 0.7498
1310/5947 [=====>........................] - ETA: 1:11:19 - loss: 1.8437 - acc: 0.5653 - map3: 0.7499
1311/5947 [=====>........................] - ETA: 1:11:18 - loss: 1.8433 - acc: 0.5654 - map3: 0.7500
1312/5947 [=====>........................] - ETA: 1:11:17 - loss: 1.8430 - acc: 0.5655 - map3: 0.7500
1313/5947 [=====>........................] - ETA: 1:11:16 - loss: 1.8426 - acc: 0.5656 - map3: 0.7501

This does not occur when the script is just run with the python command, although in this case the neptune integration does not work, so the issue occurs somewhere in the Keras integration.

#2

Hi,

Neptune reads the standard output stream from your experiment and stores it as lines of text.

We investigated this and Keras prints out those lines like:

1309/5947 [=====>........................] - ETA: 1:11:20 - loss: 1.8441 - acc: 0.5652 - map3: 0.7498\r

Appending the \r to each line.

We’ll look into how to adjust out front-end for this.

Best regards,
Hubert

#3

In a similar issue, I am getting a problem with tqdm progressbars, where the progressbar is not printed at all until \n is written (i.e. the progressbar has completed), which kind of defeats the purpose of the progressbar in the first place…

(This is without using the keras integration, there might be an issue with the way you are capturing stdout/stderr in neptune). Once again the issue dissapears when the script is called with python instead of neptune run.