首页
学习
活动
专区
工具
TVP
发布
社区首页 >问答首页 >使用tensorflow进行迭代过程出现failed to allocate memory的bug?

使用tensorflow进行迭代过程出现failed to allocate memory的bug?

提问于 2021-10-21 22:06:36
回答 1关注 0查看 2.4K

使用tensorflow进行全连接网络迭代过程出现failed to allocate memory的报错

具体情况如下:

D:\IDE\python\310\lib\site-packages\sklearn\preprocessing\_data.py:235: UserWarning: Numerical issues were encountered when centering the data and might not be solved. Dataset may contain too large values. You may need to prescale your features.

warnings.warn(

D:\IDE\python\310\lib\site-packages\sklearn\preprocessing\_data.py:254: UserWarning: Numerical issues were encountered when scaling the data and might not be solved. The standard deviation of the data is probably very close to 0.

warnings.warn(

2021-10-21 21:58:27.722550: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2021-10-21 21:58:29.628655: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3989 MB memory: -> device: 0, name: GeForce GTX 1660 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5

D:\IDE\python\310\lib\site-packages\sklearn\preprocessing\_data.py:235: UserWarning: Numerical issues were encountered when centering the data and might not be solved. Dataset may contain too large values. You may need to prescale your features.

warnings.warn(

D:\IDE\python\310\lib\site-packages\sklearn\preprocessing\_data.py:254: UserWarning: Numerical issues were encountered when scaling the data and might not be solved. The standard deviation of the data is probably very close to 0.

warnings.warn(

D:\IDE\python\310\lib\site-packages\sklearn\preprocessing\_data.py:235: UserWarning: Numerical issues were encountered when centering the data and might not be solved. Dataset may contain too large values. You may need to prescale your features.

warnings.warn(

D:\IDE\python\310\lib\site-packages\sklearn\preprocessing\_data.py:254: UserWarning: Numerical issues were encountered when scaling the data and might not be solved. The standard deviation of the data is probably very close to 0.

warnings.warn(

2021-10-21 21:59:06.543918: W tensorflow/core/common_runtime/bfc_allocator.cc:457] Allocator (GPU_0_bfc) ran out of memory trying to allocate 7.54GiB (rounded to 8100000000)requested by op Sub

If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation.

Current allocation summary follows.

Current allocation summary follows.

2021-10-21 21:59:06.544339: I tensorflow/core/common_runtime/bfc_allocator.cc:1004] BFCAllocator dump for GPU_0_bfc

2021-10-21 21:59:06.544465: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (256): Total Chunks: 26, Chunks in use: 24. 6.5KiB allocated for chunks. 6.0KiB in use in bin. 780B client-requested in use in bin.

2021-10-21 21:59:06.544711: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (512): Total Chunks: 1, Chunks in use: 0. 512B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.544986: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (1024): Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.

2021-10-21 21:59:06.545207: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (2048): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.545419: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.554742: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.555085: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.555385: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.555684: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (65536): Total Chunks: 2, Chunks in use: 2. 193.0KiB allocated for chunks. 193.0KiB in use in bin. 192.9KiB client-requested in use in bin.

2021-10-21 21:59:06.555973: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (131072): Total Chunks: 2, Chunks in use: 2. 352.0KiB allocated for chunks. 352.0KiB in use in bin. 351.6KiB client-requested in use in bin.

2021-10-21 21:59:06.556604: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (262144): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.556855: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (524288): Total Chunks: 1, Chunks in use: 1. 879.0KiB allocated for chunks. 879.0KiB in use in bin. 878.9KiB client-requested in use in bin.

2021-10-21 21:59:06.557106: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.557337: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.557573: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (4194304): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.558275: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.558524: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.560650: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.560917: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.561147: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.561888: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (268435456): Total Chunks: 1, Chunks in use: 0. 3.89GiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.562149: I tensorflow/core/common_runtime/bfc_allocator.cc:1027] Bin for 7.54GiB was 256.00MiB, Chunk State:

2021-10-21 21:59:06.562285: I tensorflow/core/common_runtime/bfc_allocator.cc:1033] Size: 3.89GiB | Requested Size: 64B | in_use: 0 | bin_num: 20, prev: Size: 176.0KiB | Requested Size: 175.8KiB | in_use: 1 | bin_num: -1

2021-10-21 21:59:06.562539: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Next region of size 4183621632

2021-10-21 21:59:06.563063: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70ba00000 of size 900096 next 1

2021-10-21 21:59:06.563196: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70badbc00 of size 1280 next 2

2021-10-21 21:59:06.563319: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70badc100 of size 97536 next 3

2021-10-21 21:59:06.563477: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70baf3e00 of size 100096 next 4

2021-10-21 21:59:06.563604: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0c500 of size 256 next 5

2021-10-21 21:59:06.563726: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0c600 of size 256 next 6

2021-10-21 21:59:06.563849: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0c700 of size 256 next 7

2021-10-21 21:59:06.564386: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0c800 of size 256 next 8

2021-10-21 21:59:06.564510: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0c900 of size 256 next 9

2021-10-21 21:59:06.564632: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0ca00 of size 256 next 10

2021-10-21 21:59:06.564788: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0cb00 of size 256 next 11

2021-10-21 21:59:06.564913: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0cc00 of size 256 next 12

2021-10-21 21:59:06.565035: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0cd00 of size 256 next 13

2021-10-21 21:59:06.565166: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0ce00 of size 256 next 14

2021-10-21 21:59:06.565859: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] Free at 70bb0cf00 of size 256 next 15

2021-10-21 21:59:06.565999: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0d000 of size 256 next 16

2021-10-21 21:59:06.566148: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0d100 of size 256 next 17

2021-10-21 21:59:06.566271: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0d200 of size 256 next 18

2021-10-21 21:59:06.566394: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] Free at 70bb0d300 of size 256 next 19

2021-10-21 21:59:06.566533: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0d400 of size 256 next 20

2021-10-21 21:59:06.566665: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0d500 of size 256 next 21

2021-10-21 21:59:06.566788: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0d600 of size 256 next 22

2021-10-21 21:59:06.566926: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0d700 of size 256 next 23

2021-10-21 21:59:06.576100: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0d800 of size 256 next 24

2021-10-21 21:59:06.576228: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0d900 of size 256 next 25

2021-10-21 21:59:06.576349: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0da00 of size 256 next 26

2021-10-21 21:59:06.576476: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0db00 of size 256 next 27

2021-10-21 21:59:06.576629: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0dc00 of size 256 next 28

2021-10-21 21:59:06.576754: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] Free at 70bb0dd00 of size 512 next 30

2021-10-21 21:59:06.577267: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0df00 of size 256 next 31

2021-10-21 21:59:06.577415: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0e000 of size 256 next 32

2021-10-21 21:59:06.577542: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0e100 of size 180224 next 29

2021-10-21 21:59:06.577680: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb3a100 of size 180224 next 33

2021-10-21 21:59:06.577831: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] Free at 70bb66100 of size 4182155008 next 18446744073709551615

2021-10-21 21:59:06.577984: I tensorflow/core/common_runtime/bfc_allocator.cc:1065] Summary of in-use Chunks by size:

2021-10-21 21:59:06.578122: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 24 Chunks of size 256 totalling 6.0KiB

2021-10-21 21:59:06.578255: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 1280 totalling 1.2KiB

2021-10-21 21:59:06.578387: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 97536 totalling 95.2KiB

2021-10-21 21:59:06.578520: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 100096 totalling 97.8KiB

2021-10-21 21:59:06.578655: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 2 Chunks of size 180224 totalling 352.0KiB

2021-10-21 21:59:06.578787: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 900096 totalling 879.0KiB

2021-10-21 21:59:06.578912: I tensorflow/core/common_runtime/bfc_allocator.cc:1072] Sum Total of in-use chunks: 1.40MiB

2021-10-21 21:59:06.579036: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] total_region_allocated_bytes_: 4183621632 memory_limit_: 4183621632 available bytes: 0 curr_region_allocation_bytes_: 8367243264

2021-10-21 21:59:06.579256: I tensorflow/core/common_runtime/bfc_allocator.cc:1080] Stats:

Limit: 4183621632

InUse: 1465600

MaxInUse: 1465600

NumAllocs: 157535

MaxAllocSize: 900096

Reserved: 0

PeakReserved: 0

LargestFreeBlock: 0

2021-10-21 21:59:06.579654: W tensorflow/core/common_runtime/bfc_allocator.cc:468] *___________________________________________________________________________________________________

2021-10-21 21:59:06.580231: W tensorflow/core/framework/op_kernel.cc:1680] Resource exhausted: failed to allocate memory

Traceback (most recent call last):

File "D:\works\python\TensorFlows\tensorflows.py", line 107, in <module>

optimizer.apply_gradients(zip(grads, [W, B]))

File "D:\IDE\python\310\lib\site-packages\tensorflow\python\ops\math_ops.py", line 1367, in binary_op_wrappe

return func(x, y, name=name)

File "D:\IDE\python\310\lib\site-packages\tensorflow\python\util\dispatch.py", line 206, in wrappe

return target(*args, **kwargs)

File "D:\IDE\python\310\lib\site-packages\tensorflow\python\ops\math_ops.py", line 548, in subtract

return gen_math_ops.sub(x, y, name)

File "D:\IDE\python\310\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 10643, in sub

_ops.raise_from_not_ok_status(e, name)

File "D:\IDE\python\310\lib\site-packages\tensorflow\python\framework\ops.py", line 6941, in raise_from_not_ok_status

six.raise_from(core._status_to_exception(e.code, message), None)

File "<string>", line 3, in raise_from

tensorflow.python.framework.errors_impl.ResourceExhaustedError: failed to allocate memory [Op:Sub]

进程完成,退出码 1

请问有什么解决办法?

回答

和开发者交流更多问题细节吧,去 写回答
相关文章

相似问题

相关问答用户
领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档