问6GB的GPU内存可用，但tensorflow.keras只使用3

EN

22022-05-26 20:09:34.753023: W tensorflow/core/common_runtime/bfc_allocator.cc:290] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.32GiB with freed_by_count=0. The caller indicates that this is not a failure, but this may mean that there could be performance gains if more memory were available.
2022-05-26 20:09:34.753167: W tensorflow/core/common_runtime/bfc_allocator.cc:290] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.32GiB with freed_by_count=0. The caller indicates that this is not a failure, but this may mean that there could be performance gains if more memory were available.
2022-05-26 20:09:35.198911: W tensorflow/core/common_runtime/bfc_allocator.cc:290] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.56GiB with freed_by_count=0. The caller indicates that this is not a failure, but this may mean that there could be performance gains if more memory were available.
2022-05-26 20:09:35.199028: W tensorflow/core/kernels/gpu_utils.cc:50] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once.
2022-05-26 20:09:47.505510: W tensorflow/core/common_runtime/bfc_allocator.cc:479] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.55GiB (rounded to 1661337600)requested by op zero_dce/StatefulPartitionedCall/StatefulPartitionedCall/model/conv2d_2/Conv2D
If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation.
Current allocation summary follows.
Current allocation summary follows.
2022-05-26 20:09:47.505656: I tensorflow/core/common_runtime/bfc_allocator.cc:1027] BFCAllocator dump for GPU_0_bfc
2022-05-26 20:09:47.506718: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] Bin (256):  Total Chunks: 25, Chunks in use: 25. 6.2KiB allocated for chunks. 6.2KiB in use in bin. 960B client-requested in use in bin.
2022-05-26 20:09:47.508009: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] Bin (512):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-26 20:09:47.508351: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] Bin (1024):         Total Chunks: 2, Chunks in use: 1. 2.5KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2022-05-26 20:09:47.508696: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] Bin (2048):         Total Chunks: 1, Chunks in use: 1. 3.5KiB allocated for chunks. 3.5KiB in use in bin. 3.4KiB client-requested in use in bin.
2022-05-26 20:09:47.509250: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] Bin (4096):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-26 20:09:47.509749: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] Bin (8192):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-26 20:09:47.510290: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] Bin (16384):        Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-26 20:09:47.510665: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] Bin (32768):        Total Chunks: 3, Chunks in use: 3. 129.2KiB allocated for chunks. 129.2KiB in use in bin. 108.0KiB client-requested in use in bin.
2022-05-26 20:09:47.511011: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] Bin (65536):        Total Chunks: 4, Chunks in use: 3. 324.2KiB allocated for chunks. 216.2KiB in use in bin. 198.0KiB client-requested in use in bin.
2022-05-26 20:09:47.511337: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] Bin (131072):       Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-26 20:09:47.511696: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] Bin (262144):       Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-26 20:09:47.512012: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] Bin (524288):       Total Chunks: 1, Chunks in use: 0. 692.2KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-26 20:09:47.512342: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] Bin (1048576):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-26 20:09:47.512667: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] Bin (2097152):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-26 20:09:47.512996: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] Bin (4194304):      Total Chunks: 1, Chunks in use: 1. 5.72MiB allocated for chunks. 5.72MiB in use in bin. 5.72MiB client-requested in use in bin.
2022-05-26 20:09:47.513321: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] Bin (8388608):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-26 20:09:47.513648: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] Bin (16777216):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-26 20:09:47.513968: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] Bin (33554432):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-26 20:09:47.514280: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] Bin (67108864):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-26 20:09:47.514616: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] Bin (134217728):    Total Chunks: 4, Chunks in use: 3. 594.14MiB allocated for chunks. 445.61MiB in use in bin. 445.61MiB client-requested in use in bin.
2022-05-26 20:09:47.514937: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] Bin (268435456):    Total Chunks: 3, Chunks in use: 2. 4.25GiB allocated for chunks. 3.09GiB in use in bin. 3.09GiB client-requested in use in bin.
2022-05-26 20:09:47.515309: I tensorflow/core/common_runtime/bfc_allocator.cc:1050] Bin for 1.55GiB was 256.00MiB, Chunk State:
2022-05-26 20:09:47.515636: I tensorflow/core/common_runtime/bfc_allocator.cc:1056]   Size: 1.15GiB | Requested Size: 5.72MiB | in_use: 0 | bin_num: 20, prev:   Size: 1.55GiB | Requested Size: 1.55GiB | in_use: 1 | bin_num: -1
2022-05-26 20:09:47.516110: I tensorflow/core/common_runtime/bfc_allocator.cc:1063] Next region of size 5190451200
2022-05-26 20:09:47.516933: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e000000 of size 256 next 1
2022-05-26 20:09:47.517305: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e000100 of size 1280 next 2
2022-05-26 20:09:47.517655: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e000600 of size 256 next 3
2022-05-26 20:09:47.517962: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e000700 of size 256 next 4
2022-05-26 20:09:47.518281: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e000800 of size 256 next 5
2022-05-26 20:09:47.518611: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e000900 of size 256 next 6
2022-05-26 20:09:47.518920: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e000a00 of size 256 next 9
2022-05-26 20:09:47.519250: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e000b00 of size 256 next 10
2022-05-26 20:09:47.519577: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e000c00 of size 256 next 57
2022-05-26 20:09:47.519891: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e000d00 of size 256 next 55
2022-05-26 20:09:47.520227: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e000e00 of size 256 next 16
2022-05-26 20:09:47.520550: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e000f00 of size 256 next 18
2022-05-26 20:09:47.520874: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e001000 of size 256 next 19
2022-05-26 20:09:47.521505: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e001100 of size 256 next 58
2022-05-26 20:09:47.523840: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e001200 of size 256 next 24
2022-05-26 20:09:47.524446: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e001300 of size 256 next 25
2022-05-26 20:09:47.524756: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e001400 of size 256 next 26
2022-05-26 20:09:47.525078: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e001500 of size 256 next 28
2022-05-26 20:09:47.525408: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e001600 of size 256 next 29
2022-05-26 20:09:47.525755: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] Free  at 70e001700 of size 708864 next 40
2022-05-26 20:09:47.526092: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e0ae800 of size 256 next 61
2022-05-26 20:09:47.526434: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e0ae900 of size 256 next 62
2022-05-26 20:09:47.526775: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e0aea00 of size 256 next 51
2022-05-26 20:09:47.527092: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e0aeb00 of size 256 next 42
2022-05-26 20:09:47.527404: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e0aec00 of size 39936 next 44
2022-05-26 20:09:47.527730: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e0b8800 of size 256 next 43
2022-05-26 20:09:47.528052: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e0b8900 of size 36864 next 46
2022-05-26 20:09:47.528375: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e0c1900 of size 256 next 45
2022-05-26 20:09:47.528743: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e0c1a00 of size 256 next 48
2022-05-26 20:09:47.529089: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] Free  at 70e0c1b00 of size 110592 next 50
2022-05-26 20:09:47.529432: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e0dcb00 of size 73984 next 52
2022-05-26 20:09:47.529935: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e0eec00 of size 55552 next 54
2022-05-26 20:09:47.530519: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] Free  at 70e0fc500 of size 1280 next 59
2022-05-26 20:09:47.530843: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e0fca00 of size 3584 next 60
2022-05-26 20:09:47.531200: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e0fd800 of size 73728 next 41
2022-05-26 20:09:47.531535: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e10f800 of size 73728 next 47
2022-05-26 20:09:47.531847: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e121800 of size 6000128 next 56
2022-05-26 20:09:47.532183: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 70e6da600 of size 155750400 next 7
2022-05-26 20:09:47.532544: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 717b63600 of size 155750400 next 8
2022-05-26 20:09:47.532881: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 720fec600 of size 155750400 next 13
2022-05-26 20:09:47.533209: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] Free  at 72a475600 of size 155750400 next 12
2022-05-26 20:09:47.533554: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 7338fe600 of size 1661337600 next 11
2022-05-26 20:09:47.533885: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] InUse at 79695e600 of size 1661337600 next 15
2022-05-26 20:09:47.534201: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] Free  at 7f99be600 of size 1237588480 next 18446744073709551615
2022-05-26 20:09:47.534585: I tensorflow/core/common_runtime/bfc_allocator.cc:1088]      Summary of in-use Chunks by size:
2022-05-26 20:09:47.535048: I tensorflow/core/common_runtime/bfc_allocator.cc:1091] 25 Chunks of size 256 totalling 6.2KiB
2022-05-26 20:09:47.535368: I tensorflow/core/common_runtime/bfc_allocator.cc:1091] 1 Chunks of size 1280 totalling 1.2KiB
2022-05-26 20:09:47.535709: I tensorflow/core/common_runtime/bfc_allocator.cc:1091] 1 Chunks of size 3584 totalling 3.5KiB
2022-05-26 20:09:47.536064: I tensorflow/core/common_runtime/bfc_allocator.cc:1091] 1 Chunks of size 36864 totalling 36.0KiB
2022-05-26 20:09:47.536386: I tensorflow/core/common_runtime/bfc_allocator.cc:1091] 1 Chunks of size 39936 totalling 39.0KiB
2022-05-26 20:09:47.536885: I tensorflow/core/common_runtime/bfc_allocator.cc:1091] 1 Chunks of size 55552 totalling 54.2KiB
2022-05-26 20:09:47.538171: I tensorflow/core/common_runtime/bfc_allocator.cc:1091] 2 Chunks of size 73728 totalling 144.0KiB
2022-05-26 20:09:47.540404: I tensorflow/core/common_runtime/bfc_allocator.cc:1091] 1 Chunks of size 73984 totalling 72.2KiB
2022-05-26 20:09:47.540821: I tensorflow/core/common_runtime/bfc_allocator.cc:1091] 1 Chunks of size 6000128 totalling 5.72MiB
2022-05-26 20:09:47.541139: I tensorflow/core/common_runtime/bfc_allocator.cc:1091] 3 Chunks of size 155750400 totalling 445.61MiB
2022-05-26 20:09:47.541450: I tensorflow/core/common_runtime/bfc_allocator.cc:1091] 2 Chunks of size 1661337600 totalling 3.09GiB
2022-05-26 20:09:47.541770: I tensorflow/core/common_runtime/bfc_allocator.cc:1095] Sum Total of in-use chunks: 3.54GiB
2022-05-26 20:09:47.542088: I tensorflow/core/common_runtime/bfc_allocator.cc:1097] total_region_allocated_bytes_: 5190451200 memory_limit_: 5767168000 available bytes: 576716800 curr_region_allocation_bytes_: 11534336000
2022-05-26 20:09:47.542449: I tensorflow/core/common_runtime/bfc_allocator.cc:1103] Stats:
Limit:                      5767168000
InUse:                      3796291584
MaxInUse:                   3968822784
NumAllocs:                         253
MaxAllocSize:               2536791040
Reserved:                            0
PeakReserved:                        0
LargestFreeBlock:                    0

2022-05-26 20:09:47.542854: W tensorflow/core/common_runtime/bfc_allocator.cc:491] **********__*****************************************************************_______________________
2022-05-26 20:09:47.543195: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at conv_ops.cc:684 : RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[1,32,3120,4160] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
  File "C:\Users\slind\Documents\IT\Development\In development\Deep-Learning-Image-Enhancer-Using-CLAHE-and-Zero-DCE\image-processor.py", line 94, in <module>
    start()
  File "C:\Users\slind\Documents\IT\Development\In development\Deep-Learning-Image-Enhancer-Using-CLAHE-and-Zero-DCE\image-processor.py", line 47, in start
    main()
  File "C:\Users\slind\Documents\IT\Development\In development\Deep-Learning-Image-Enhancer-Using-CLAHE-and-Zero-DCE\image-processor.py", line 57, in main
    (DCE(preprocess(image))).save(outputdir + image.replace(inputdir, ''))
  File "C:\Users\slind\Documents\IT\Development\In development\Deep-Learning-Image-Enhancer-Using-CLAHE-and-Zero-DCE\image-processor.py", line 87, in DCE
    enhanced = dce_model.predict(expanded)
  File "C:\Users\slind\AppData\Local\Programs\Python\Python39\lib\site-packages\keras\utils\traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "C:\Users\slind\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\eager\execute.py", line 54, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.ResourceExhaustedError: Graph execution error:

Detected at node 'model/conv2d_2/Conv2D' defined at (most recent call last):
    File "C:\Users\slind\Documents\IT\Development\In development\Deep-Learning-Image-Enhancer-Using-CLAHE-and-Zero-DCE\image-processor.py", line 39, in <module>
      dce_model = tf.keras.models.load_model('./model/cast2')
    File "C:\Users\slind\AppData\Local\Programs\Python\Python39\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "C:\Users\slind\AppData\Local\Programs\Python\Python39\lib\site-packages\keras\saving\save.py", line 209, in load_model
      return saved_model_load.load(filepath_str, compile, options)
    File "C:\Users\slind\AppData\Local\Programs\Python\Python39\lib\site-packages\keras\saving\saved_model\load.py", line 141, in load
      loaded = tf.__internal__.saved_model.load_partial(
Node: 'model/conv2d_2/Conv2D'
OOM when allocating tensor with shape[1,32,3120,4160] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[{{node model/conv2d_2/Conv2D}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
 [Op:__inference_predict_function_2591]
Press any key to continue . . .