NotFoundError (see above for traceback): Restoring from checkpoint failed. This is most likely due t

Caused by op 'save/RestoreV2', defined at:
  File "D:/tensorflow_learning/neural_network_based_slim/structure_of_the_model.py", line 59, in <module>
    log_every_n_steps = 200)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\contrib\slim\python\slim\learning.py", line 659, in train
    saver = saver or tf_saver.Saver()
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1094, in __init__
    self.build()
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1106, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1143, in _build
    build_save=build_save, build_restore=build_restore)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 787, in _build_internal
    restore_sequentially, reshape)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 406, in _AddRestoreOps
    restore_sequentially)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 854, in bulk_restore
    return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_io_ops.py", line 1465, in restore_v2
    shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\util\deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 3272, in create_op
    op_def=op_def)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1768, in __init__
    self._traceback = tf_stack.extract_stack()

NotFoundError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Key beta1_power not found in checkpoint
	 [[{{node save/RestoreV2}} = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT64], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

Traceback (most recent call last):
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1292, in _do_call
    return fn(*args)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1277, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1367, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.NotFoundError: Key beta1_power not found in checkpoint
	 [[{{node save/RestoreV2}} = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT64], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1538, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 887, in run
    run_metadata_ptr)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1110, in _run
    feed_dict_tensor, options, run_metadata)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1286, in _do_run
    run_metadata)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1308, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key beta1_power not found in checkpoint
	 [[{{node save/RestoreV2}} = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT64], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

Caused by op 'save/RestoreV2', defined at:
  File "D:/tensorflow_learning/neural_network_based_slim/structure_of_the_model.py", line 59, in <module>
    log_every_n_steps = 200)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\contrib\slim\python\slim\learning.py", line 659, in train
    saver = saver or tf_saver.Saver()
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1094, in __init__
    self.build()
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1106, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1143, in _build
    build_save=build_save, build_restore=build_restore)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 787, in _build_internal
    restore_sequentially, reshape)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 406, in _AddRestoreOps
    restore_sequentially)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 854, in bulk_restore
    return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_io_ops.py", line 1465, in restore_v2
    shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\util\deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 3272, in create_op
    op_def=op_def)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1768, in __init__
    self._traceback = tf_stack.extract_stack()

NotFoundError (see above for traceback): Key beta1_power not found in checkpoint
	 [[{{node save/RestoreV2}} = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT64], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1548, in restore
    names_to_keys = object_graph_key_mapping(save_path)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1822, in object_graph_key_mapping
    checkpointable.OBJECT_GRAPH_PROTO_KEY)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 359, in get_tensor
    status)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 526, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: Key _CHECKPOINTABLE_OBJECT_GRAPH not found in checkpoint

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:/tensorflow_learning/neural_network_based_slim/structure_of_the_model.py", line 59, in <module>
    log_every_n_steps = 200)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\contrib\slim\python\slim\learning.py", line 748, in train
    master, start_standard_services=False, config=session_config) as sess:
  File "D:\anaconda\envs\tensorflow\lib\contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\supervisor.py", line 1005, in managed_session
    self.stop(close_summary_writer=close_summary_writer)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\supervisor.py", line 833, in stop
    ignore_live_threads=ignore_live_threads)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\coordinator.py", line 389, in join
    six.reraise(*self._exc_info_to_raise)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\six.py", line 693, in reraise
    raise value
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\supervisor.py", line 994, in managed_session
    start_standard_services=start_standard_services)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\supervisor.py", line 731, in prepare_or_wait_for_session
    init_feed_dict=self._init_feed_dict, init_fn=self._init_fn)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\session_manager.py", line 281, in prepare_session
    config=config)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\session_manager.py", line 211, in _restore_checkpoint
    saver.restore(sess, ckpt.model_checkpoint_path)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1554, in restore
    err, "a Variable name or other graph key that is missing")
tensorflow.python.framework.errors_impl.NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Key beta1_power not found in checkpoint
	 [[{{node save/RestoreV2}} = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT64], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

Caused by op 'save/RestoreV2', defined at:
  File "D:/tensorflow_learning/neural_network_based_slim/structure_of_the_model.py", line 59, in <module>
    log_every_n_steps = 200)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\contrib\slim\python\slim\learning.py", line 659, in train
    saver = saver or tf_saver.Saver()
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1094, in __init__
    self.build()
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1106, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1143, in _build
    build_save=build_save, build_restore=build_restore)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 787, in _build_internal
    restore_sequentially, reshape)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 406, in _AddRestoreOps
    restore_sequentially)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 854, in bulk_restore
    return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_io_ops.py", line 1465, in restore_v2
    shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\util\deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 3272, in create_op
    op_def=op_def)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1768, in __init__
    self._traceback = tf_stack.extract_stack()

NotFoundError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Key beta1_power not found in checkpoint
	 [[{{node save/RestoreV2}} = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT64], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

ERROR:tensorflow:==================================
Object was never used (type <class 'tensorflow.python.framework.ops.Tensor'>):
<tf.Tensor 'init_ops/report_uninitialized_variables/boolean_mask/GatherV2:0' shape=(?,) dtype=string>
If you want to mark it as used call its "mark_used()" method.
It was originally created here:
  File "D:/tensorflow_learning/neural_network_based_slim/structure_of_the_model.py", line 59, in <module>
    log_every_n_steps = 200)  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\contrib\slim\python\slim\learning.py", line 791, in train
    should_retry = True  File "D:\anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\util\tf_should_use.py", line 189, in wrapped
    return _add_should_use_warning(fn(*args, **kwargs))
==================================

在训练并保存模型时遇到的错误,解决方法是在ckpt目录前面加个点就可以解决,如下图:

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

扫码关注云+社区

领取腾讯云代金券