使用tensorflow进行全连接网络迭代过程出现failed to allocate memory的报错
具体情况如下：
D:\IDE\python\310\lib\site-packages\sklearn\preprocessing\_data.py:235: UserWarning: Numerical issues were encountered when centering the data and might not be solved. Dataset may contain too large values. You may need to prescale your features.
warnings.warn(
D:\IDE\python\310\lib\site-packages\sklearn\preprocessing\_data.py:254: UserWarning: Numerical issues were encountered when scaling the data and might not be solved. The standard deviation of the data is probably very close to 0.
warnings.warn(
2021-10-21 21:58:27.722550: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-10-21 21:58:29.628655: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3989 MB memory: -> device: 0, name: GeForce GTX 1660 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5
D:\IDE\python\310\lib\site-packages\sklearn\preprocessing\_data.py:235: UserWarning: Numerical issues were encountered when centering the data and might not be solved. Dataset may contain too large values. You may need to prescale your features.
warnings.warn(
D:\IDE\python\310\lib\site-packages\sklearn\preprocessing\_data.py:254: UserWarning: Numerical issues were encountered when scaling the data and might not be solved. The standard deviation of the data is probably very close to 0.
warnings.warn(
D:\IDE\python\310\lib\site-packages\sklearn\preprocessing\_data.py:235: UserWarning: Numerical issues were encountered when centering the data and might not be solved. Dataset may contain too large values. You may need to prescale your features.
warnings.warn(
D:\IDE\python\310\lib\site-packages\sklearn\preprocessing\_data.py:254: UserWarning: Numerical issues were encountered when scaling the data and might not be solved. The standard deviation of the data is probably very close to 0.
warnings.warn(
2021-10-21 21:59:06.543918: W tensorflow/core/common_runtime/bfc_allocator.cc:457] Allocator (GPU_0_bfc) ran out of memory trying to allocate 7.54GiB (rounded to 8100000000)requested by op Sub
If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation.
Current allocation summary follows.
Current allocation summary follows.
2021-10-21 21:59:06.544339: I tensorflow/core/common_runtime/bfc_allocator.cc:1004] BFCAllocator dump for GPU_0_bfc
2021-10-21 21:59:06.544465: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (256): Total Chunks: 26, Chunks in use: 24. 6.5KiB allocated for chunks. 6.0KiB in use in bin. 780B client-requested in use in bin.
2021-10-21 21:59:06.544711: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (512): Total Chunks: 1, Chunks in use: 0. 512B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-10-21 21:59:06.544986: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (1024): Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2021-10-21 21:59:06.545207: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (2048): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-10-21 21:59:06.545419: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-10-21 21:59:06.554742: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-10-21 21:59:06.555085: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-10-21 21:59:06.555385: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-10-21 21:59:06.555684: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (65536): Total Chunks: 2, Chunks in use: 2. 193.0KiB allocated for chunks. 193.0KiB in use in bin. 192.9KiB client-requested in use in bin.
2021-10-21 21:59:06.555973: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (131072): Total Chunks: 2, Chunks in use: 2. 352.0KiB allocated for chunks. 352.0KiB in use in bin. 351.6KiB client-requested in use in bin.
2021-10-21 21:59:06.556604: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (262144): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-10-21 21:59:06.556855: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (524288): Total Chunks: 1, Chunks in use: 1. 879.0KiB allocated for chunks. 879.0KiB in use in bin. 878.9KiB client-requested in use in bin.
2021-10-21 21:59:06.557106: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-10-21 21:59:06.557337: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-10-21 21:59:06.557573: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (4194304): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-10-21 21:59:06.558275: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-10-21 21:59:06.558524: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-10-21 21:59:06.560650: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-10-21 21:59:06.560917: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-10-21 21:59:06.561147: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-10-21 21:59:06.561888: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (268435456): Total Chunks: 1, Chunks in use: 0. 3.89GiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-10-21 21:59:06.562149: I tensorflow/core/common_runtime/bfc_allocator.cc:1027] Bin for 7.54GiB was 256.00MiB, Chunk State:
2021-10-21 21:59:06.562285: I tensorflow/core/common_runtime/bfc_allocator.cc:1033] Size: 3.89GiB | Requested Size: 64B | in_use: 0 | bin_num: 20, prev: Size: 176.0KiB | Requested Size: 175.8KiB | in_use: 1 | bin_num: -1
2021-10-21 21:59:06.562539: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Next region of size 4183621632
2021-10-21 21:59:06.563063: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70ba00000 of size 900096 next 1
2021-10-21 21:59:06.563196: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70badbc00 of size 1280 next 2
2021-10-21 21:59:06.563319: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70badc100 of size 97536 next 3
2021-10-21 21:59:06.563477: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70baf3e00 of size 100096 next 4
2021-10-21 21:59:06.563604: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0c500 of size 256 next 5
2021-10-21 21:59:06.563726: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0c600 of size 256 next 6
2021-10-21 21:59:06.563849: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0c700 of size 256 next 7
2021-10-21 21:59:06.564386: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0c800 of size 256 next 8
2021-10-21 21:59:06.564510: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0c900 of size 256 next 9
2021-10-21 21:59:06.564632: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0ca00 of size 256 next 10
2021-10-21 21:59:06.564788: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0cb00 of size 256 next 11
2021-10-21 21:59:06.564913: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0cc00 of size 256 next 12
2021-10-21 21:59:06.565035: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0cd00 of size 256 next 13
2021-10-21 21:59:06.565166: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0ce00 of size 256 next 14
2021-10-21 21:59:06.565859: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] Free at 70bb0cf00 of size 256 next 15
2021-10-21 21:59:06.565999: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0d000 of size 256 next 16
2021-10-21 21:59:06.566148: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0d100 of size 256 next 17
2021-10-21 21:59:06.566271: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0d200 of size 256 next 18
2021-10-21 21:59:06.566394: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] Free at 70bb0d300 of size 256 next 19
2021-10-21 21:59:06.566533: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0d400 of size 256 next 20
2021-10-21 21:59:06.566665: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0d500 of size 256 next 21
2021-10-21 21:59:06.566788: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0d600 of size 256 next 22
2021-10-21 21:59:06.566926: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0d700 of size 256 next 23
2021-10-21 21:59:06.576100: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0d800 of size 256 next 24
2021-10-21 21:59:06.576228: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0d900 of size 256 next 25
2021-10-21 21:59:06.576349: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0da00 of size 256 next 26
2021-10-21 21:59:06.576476: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0db00 of size 256 next 27
2021-10-21 21:59:06.576629: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0dc00 of size 256 next 28
2021-10-21 21:59:06.576754: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] Free at 70bb0dd00 of size 512 next 30
2021-10-21 21:59:06.577267: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0df00 of size 256 next 31
2021-10-21 21:59:06.577415: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0e000 of size 256 next 32
2021-10-21 21:59:06.577542: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0e100 of size 180224 next 29
2021-10-21 21:59:06.577680: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb3a100 of size 180224 next 33
2021-10-21 21:59:06.577831: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] Free at 70bb66100 of size 4182155008 next 18446744073709551615
2021-10-21 21:59:06.577984: I tensorflow/core/common_runtime/bfc_allocator.cc:1065] Summary of in-use Chunks by size:
2021-10-21 21:59:06.578122: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 24 Chunks of size 256 totalling 6.0KiB
2021-10-21 21:59:06.578255: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 1280 totalling 1.2KiB
2021-10-21 21:59:06.578387: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 97536 totalling 95.2KiB
2021-10-21 21:59:06.578520: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 100096 totalling 97.8KiB
2021-10-21 21:59:06.578655: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 2 Chunks of size 180224 totalling 352.0KiB
2021-10-21 21:59:06.578787: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 900096 totalling 879.0KiB
2021-10-21 21:59:06.578912: I tensorflow/core/common_runtime/bfc_allocator.cc:1072] Sum Total of in-use chunks: 1.40MiB
2021-10-21 21:59:06.579036: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] total_region_allocated_bytes_: 4183621632 memory_limit_: 4183621632 available bytes: 0 curr_region_allocation_bytes_: 8367243264
2021-10-21 21:59:06.579256: I tensorflow/core/common_runtime/bfc_allocator.cc:1080] Stats:
Limit: 4183621632
InUse: 1465600
MaxInUse: 1465600
NumAllocs: 157535
MaxAllocSize: 900096
Reserved: 0
PeakReserved: 0
LargestFreeBlock: 0
2021-10-21 21:59:06.579654: W tensorflow/core/common_runtime/bfc_allocator.cc:468] *___________________________________________________________________________________________________
2021-10-21 21:59:06.580231: W tensorflow/core/framework/op_kernel.cc:1680] Resource exhausted: failed to allocate memory
Traceback (most recent call last):
File "D:\works\python\TensorFlows\tensorflows.py", line 107, in <module>
optimizer.apply_gradients(zip(grads, [W, B]))
File "D:\IDE\python\310\lib\site-packages\tensorflow\python\ops\math_ops.py", line 1367, in binary_op_wrappe
return func(x, y, name=name)
File "D:\IDE\python\310\lib\site-packages\tensorflow\python\util\dispatch.py", line 206, in wrappe
return target(*args, **kwargs)
File "D:\IDE\python\310\lib\site-packages\tensorflow\python\ops\math_ops.py", line 548, in subtract
return gen_math_ops.sub(x, y, name)
File "D:\IDE\python\310\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 10643, in sub
_ops.raise_from_not_ok_status(e, name)
File "D:\IDE\python\310\lib\site-packages\tensorflow\python\framework\ops.py", line 6941, in raise_from_not_ok_status
six.raise_from(core._status_to_exception(e.code, message), None)
File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.ResourceExhaustedError: failed to allocate memory [Op:Sub]
进程完成，退出码 1
请问有什么解决办法？
回答
相似问题