Problem with running keras experiments off line


#1

I’m using neptune run to run my python script but I received the following error message:

124.194855 hist = model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs,callbacks=callbacks_list, validation_data=(x_val,y_val),shuffle = True, verbose=2)
124.195007 File “//anaconda/lib/python2.7/site-packages/keras/engine/training.py”, line 1712, in fit
124.19514 validation_steps=validation_steps)
124.195262 File “//anaconda/lib/python2.7/site-packages/keras/engine/training.py”, line 1132, in _fit_loop
124.195384 stateful_metrics=self.stateful_metric_names)]
124.195506 TypeError: monkey_patched_BaseLogger() takes no arguments (1 given)

I’m using keras 2.1.14 installed locally with python 2.7
Keras is using theano and training is only done on the CPU.

Thanks,
Frank


#2

Hi,

Can you link us the experiment that failed with this error message?


#3

At this moment we can only integrate with keras <=2.0 and tensorflow <= 1.4.
Tell me if you run your experiment with --ml-framework keras parameter. If you do so, please try to run experiment without that parameter.


#4

I was actually running without the ml-framework parameter. Running it with the parameter results in the same error message.

@rzralek, if I understand your comment correctly, I need to downgrade my system to Keras <=2.0 in order to integrate with neptune even though I’m running the experiments on my machine?


#5

@fschilder We support integration with Keras and Tensorflow only for the versions specified above (integration means including automatic channels and Tensorboard). However, you should be able to use any version of your libraries without Keras/Tensorflow integration.

Do you run ctx.integrate_with_keras() or ctx.integrate_with_tensorflow() in your code? If so, please remove these lines and let us know if it made your experiment run.


#6

I wasn’t using ctx.integrate_with_keras() or ctx.integrate_with_tensorflow() in my code, but I now managed to run my experiments by creating a conda environment where I installed the respective versions of keras and tensorflow by
pip install tensorflow==2.0
pip install keras==1.4

thanks,
Frank


#7

I was mistaken, I actually used ctx.integrate_with_keras(). Running experiments on my machine is also working then.
However, I ran into another problem, when I want to run the same experiments in the cloud. Tensorflow 1.4.0 throws an error, but the same programs runs fine in my original environment using Tensorflow 1.2.0:

78.714069 File “main.py”, line 665, in
78.714329 hist = model.fit([x_train, x_train_char], y_train, batch_size=batch_size, epochs=epochs,callbacks=callbacks_list,validation_split=0.2, shuffle = True)
78.714589 File “//anaconda/envs/neptune/lib/python2.7/site-packages/keras/engine/training.py”, line 1457, in fit
78.714848 self._make_train_function()
78.715108 File “//anaconda/envs/neptune/lib/python2.7/site-packages/keras/engine/training.py”, line 1001, in _make_train_function
78.715366 self.total_loss)
78.715624 File “//anaconda/envs/neptune/lib/python2.7/site-packages/keras/optimizers.py”, line 557, in get_updates
78.715882 p_t = p - self.lr * m_t_bar / (K.sqrt(v_t_prime) + self.epsilon)
78.71614 File “//anaconda/envs/neptune/lib/python2.7/site-packages/tensorflow/python/ops/math_ops.py”, line 885, in binary_op_wrapper
78.716376 y = ops.convert_to_tensor(y, dtype=x.dtype.base_dtype, name=“y”)
78.716554 File “//anaconda/envs/neptune/lib/python2.7/site-packages/tensorflow/python/framework/ops.py”, line 836, in convert_to_tensor
78.71673 as_ref=False)
78.716907 File “//anaconda/envs/neptune/lib/python2.7/site-packages/tensorflow/python/framework/ops.py”, line 926, in internal_convert_to_tensor
78.717083 ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
78.717259 File “//anaconda/envs/neptune/lib/python2.7/site-packages/tensorflow/python/framework/constant_op.py”, line 229, in _constant_tensor_conversion_function
78.717445 return constant(v, dtype=dtype, name=name)
78.717622 File “//anaconda/envs/neptune/lib/python2.7/site-packages/tensorflow/python/framework/constant_op.py”, line 208, in constant
78.717798 value, dtype=dtype, shape=shape, verify_shape=verify_shape))
78.717973 File “//anaconda/envs/neptune/lib/python2.7/site-packages/tensorflow/python/framework/tensor_util.py”, line 371, in make_tensor_proto
78.718148 raise ValueError(“None values not supported.”)
78.718322 ValueError: None values not supported.

#8

If you are installing keras and tensorflow yourself via pip requirements (https://docs.neptune.ml/config/experiment/#pip-requirements-file), you should use base or base-gpu environment. Alternatively, you can use tensorflow-1.2 environment and install keras yourself.


#9

That makes sense. Thanks for the advice.

Frank