浅谈Tensorflow2对GPU内存的分配策略
一、问题源起
从以下的异常堆栈可以看到是BLAS程序集初始化失败，可以看到是执行MatMul的时候发生的异常，基本可以断定可能数据集太大导致memory不够用了。
2021-08-10 16:38:04.917501: E tensorflow/stream_executor/cuda/cuda_blas.cc:226] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2021-08-10 16:38:04.960048: E tensorflow/stream_executor/cuda/cuda_blas.cc:226] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2021-08-10 16:38:04.986898: E tensorflow/stream_executor/cuda/cuda_blas.cc:226] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2021-08-10 16:38:04.992366: E tensorflow/stream_executor/cuda/cuda_blas.cc:226] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2021-08-10 16:38:04.992389: W tensorflow/stream_executor/stream.cc:1455] attempting to perform BLAS operation using StreamExecutor without BLAS support
Traceback (most recent call last):
File "/home/mango/PycharmProjects/DeepLearing/minist_conv.py", line 32, in <module>
model.fit(train_images, train_labels, epochs=5, batch_size=64)
File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/keras/engine/training.py", line 1183, in fit
tmp_logs = self.train_function(iterator)
File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/eager/def_function.py", line 889, in __call__
result = self._call(*args, **kwds)
File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/eager/def_function.py", line 950, in _call
return self._stateless_fn(*args, **kwds)
File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/eager/function.py", line 3023, in __call__
return graph_function._call_flat(
File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/eager/function.py", line 1960, in _call_flat
return self._build_call_outputs(self._inference_function.call(
File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/eager/function.py", line 591, in call
outputs = execute.execute(
File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InternalError: Blas xGEMM launch failed : a.shape=[1,64,576], b.shape=[1,576,64], m=64, n=64, k=576
[[node sequential/dense/MatMul (defined at home/mango/PycharmProjects/DeepLearing/minist_conv.py:32) ]] [Op:__inference_train_function_993]
Function call stack:
train_function
二、开发环境
mango@mango-ubuntu:~$ /usr/local/cuda/bin/nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Wed_Jul_14_19:41:19_PDT_2021
Cuda~~ compilation tools, release 11.4, V11.4.100==
Build cuda_11.4.r11.4/compiler.30188945_0
mango@mango-ubuntu:~$ tail -n 10 /usr/include/cudnn_version.h
#ifndef CUDNN_VERSION_H_
#define CUDNN_VERSION_H_
#define CUDNN_MAJOR 8
#define CUDNN_MINOR 2
#define CUDNN_PATCHLEVEL 2
#define CUDNN_VERSION (CUDNN_MAJOR * 1000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)
#endif /* CUDNN_VERSION_H */
mango@mango-ubuntu:~$ python3 --version
Python 3.9.5
mango@mango-ubuntu:~$ nvidia-smi
Tue Aug 10 19:57:58 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:01:00.0 Off | N/A |
| N/A 54C P0 N/A / N/A | 329MiB / 2002MiB | 9% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1818 G /usr/lib/xorg/Xorg 186MiB |
| 0 N/A N/A 2002 G /usr/bin/gnome-shell 45MiB |
| 0 N/A N/A 3435 G ...AAAAAAAAA= --shared-files 75MiB |
| 0 N/A N/A 6016 G python3 13MiB |
+-----------------------------------------------------------------------------+
mango@mango-ubuntu:~$ python3
Python 3.9.5 (default, May 11 2021, 08:20:37)
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
2021-08-10 18:33:05.917520: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
>>> tf.__version__
'2.5.0'
>>>
三、Tensorflow针对GPU内存的分配策略
By default, TensorFlow maps nearly all of the GPU memory of all GPUs (subject to CUDA_VISIBLE_DEVICES) visible to the process. This is done to more efficiently use the relatively precious GPU memory resources on the devices by reducing memory fragmentation.
默认情况下，为了通过减少内存碎片更有效地利用设备上相对宝贵的GPU内存资源，TensorFlow进程会使用所有可见的GPU。
In some cases it is desirable for the process to only allocate a subset of the available memory, or to only grow the memory usage as is needed by the process. TensorFlow provides two methods to control this.
在某些情况下，进程只分配可用内存的一个子集，或者只根据进程的需要增加内存使用量。TensorFlow提供了两种方法来控制这种情况。
The first option is to turn on memory growth by calling tf.config.experimental.set_memory_growth, which attempts to allocate only as much GPU memory as needed for the runtime allocations: it starts out allocating very little memory, and as the program gets run and more GPU memory is needed, the GPU memory region is extended for the TensorFlow process. Memory is not released since it can lead to memory fragmentation. To turn on memory growth for a specific GPU, use the following code prior to allocating any tensors or executing any ops.
第一种选择是通过调用tf.config.experimental.set_memory_growth来打开内存增长，它尝试只分配运行时所需的GPU内存:它开始分配很少的内存，当程序运行时需要更多的GPU内存时，GPU内存区域会进一步扩展增大。内存不会被释放，因为这会导致内存碎片。为了打开特定GPU的内存增长，在分配任何张量或执行任何操作之前，使用以下代码。
gpus = tf.config.list_physical_devices('GPU')
if gpus:
try:
# Currently, memory growth needs to be the same across GPUs
for gpu in gpus:
tf.config.experimental.set_memory_growth(gpu, True)
logical_gpus = tf.config.list_logical_devices('GPU')
print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
except RuntimeError as e:
# Memory growth must be set before GPUs have been initialized
print(e)
Another way to enable this option is to set the environmental variable TF_FORCE_GPU_ALLOW_GROWTH to true. This configuration is platform specific.
启用该选项的另一种方法是将环境变量TF_FORCE_GPU_ALLOW_GROWTH设置为true。此配置是特定于平台的。
The second method is to configure a virtual GPU device with tf.config.experimental.set_virtual_device_configuration and set a hard limit on the total memory to allocate on the GPU.
This is useful if you want to truly bound the amount of GPU memory available to the TensorFlow process. This is common practice for local development when the GPU is shared with other applications such as a workstation GUI.
第二种方法是使用tf.config.experimental.set_virtual_device_configuration配置虚拟GPU设备,并设置GPU上可分配的总内存的硬限制。
如果你想真正将GPU内存的数量绑定到TensorFlow进程中，这是非常有用的。当GPU与其他应用程序(如工作站GUI)共享时，这是本地开发的常见做法。
gpus = tf.config.list_physical_devices('GPU')
if gpus:
# Restrict TensorFlow to only allocate 1GB of memory on the first GPU
try:
tf.config.set_logical_device_configuration(
gpus[0],
[tf.config.LogicalDeviceConfiguration(memory_limit=1024)])
logical_gpus = tf.config.list_logical_devices('GPU')
print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
except RuntimeError as e:
# Virtual devices must be set before GPUs have been initialized
print(e)
四、问题分析验证
通过上边对TensorFlow文档的分析，默认情况下会占用所有的GPU内存，但是TensorFlow提供了两种方式可以灵活的控制内存的分配策略；
我们可以直接设置GPU内存按需动态分配
import tensorflow as tf
physical_gpus = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_gpus[0], True)
通过以下命令可以看到执行过程中GPU内存的占用最高为697M
mango@mango-ubuntu:~$ while true; do nvidia-smi; sleep 0.2; done;
Tue Aug 10 20:30:58 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:01:00.0 Off | N/A |
| N/A 58C P0 N/A / N/A | 1026MiB / 2002MiB | 72% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1818 G /usr/lib/xorg/Xorg 186MiB |
| 0 N/A N/A 2002 G /usr/bin/gnome-shell 45MiB |
| 0 N/A N/A 3435 G ...AAAAAAAAA= --shared-files 73MiB |
| 0 N/A N/A 6016 G python3 13MiB |
| 0 N/A N/A 13829 C /usr/bin/python3.9 697MiB |
+-----------------------------------------------------------------------------+
我们也可以限制最多使用1024M的GPU内存
import tensorflow as tf
physical_gpus = tf.config.list_physical_devices('GPU')
tf.config.set_logical_device_configuration(physical_gpus[0], [tf.config.LogicalDeviceConfiguration(memory_limit=1024)])
同样通过命令可以看到执行过程中GPU内存的占用最高为1455M
mango@mango-ubuntu:~$ while true; do nvidia-smi; sleep 0.2; done;
Tue Aug 10 20:31:24 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:01:00.0 Off | N/A |
| N/A 58C P0 N/A / N/A | 1784MiB / 2002MiB | 74% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1818 G /usr/lib/xorg/Xorg 186MiB |
| 0 N/A N/A 2002 G /usr/bin/gnome-shell 46MiB |
| 0 N/A N/A 3435 G ...AAAAAAAAA= --shared-files 72MiB |
| 0 N/A N/A 6016 G python3 13MiB |
| 0 N/A N/A 13570 C /usr/bin/python3.9 1455MiB |
+-----------------------------------------------------------------------------+
五、GPU分配策略分析
通过四中的测试结果可得
- 默认的分配策略会占用所有的内存，并且执行中不会进行释放，如果训练数据量比较打很容易内存不够用；
- 限制最大使用内存，测试占用内存比设置的大，这个可能跟训练中间使用的模型和操作的复杂程度有关系，需要根据具体的业务场景设置合适的值；但是要注意不能设置大了，否则还是会报错，但是设置小了只是执行的慢一些罢了；
- 设置内存按需分配可能是一个相对比较中庸的方案，感觉可能是一个更好的方案，不知道TensorFlow为什么没有设置为默认值，留作一个问题，后续有新的认知的话再补充；
六、扩展
单GPU模拟多GPU环境
当我们的本地开发环境只有一个GPU，但却需要编写多GPU的程序在工作站上进行训练任务时，TensorFlow为我们提供了一个方便的功能，可以让我们在本地开发环境中建立多个模拟GPU，从而让多GPU的程序调试变得更加方便。以下代码在实体GPU GPU:0 的基础上建立了两个显存均为2GB的虚拟GPU。
gpus = tf.config.list_physical_devices('GPU')
if gpus:
# Create 2 virtual GPUs with 1GB memory each
try:
tf.config.set_logical_device_configuration(
gpus[0],
[tf.config.LogicalDeviceConfiguration(memory_limit=1024),
tf.config.LogicalDeviceConfiguration(memory_limit=1024)])
logical_gpus = tf.config.list_logical_devices('GPU')
print(len(gpus), "Physical GPU,", len(logical_gpus), "Logical GPUs")
except RuntimeError as e:
# Virtual devices must be set before GPUs have been initialized
print(e)
多GPU的数据并行
使用 tf.distribute.Strategy可以将模型拷贝到每个GPU上，然后将训练数据分批在不同的GPU上执行，达到数据并行。
tf.debugging.set_log_device_placement(True)
gpus = tf.config.list_logical_devices('GPU')
strategy = tf.distribute.MirroredStrategy(gpus)
with strategy.scope():
inputs = tf.keras.layers.Input(shape=(1,))
predictions = tf.keras.layers.Dense(1)(inputs)
model = tf.keras.models.Model(inputs=inputs, outputs=predictions)
model.compile(loss='mse',
optimizer=tf.keras.optimizers.SGD(learning_rate=0.2))
到此这篇关于浅谈Tensorflow2对GPU内存的分配策略的文章就介绍到这了,更多相关Tensorflow2 GPU内存分配内容请搜索脚本之家以前的文章或继续浏览下面的相关文章希望大家以后多多支持脚本之家！
相关文章
用Python的Tornado框架结合memcached页面改善博客性能
这篇文章主要介绍了用Python的Tornado框架结合memcached页面改善vLog性能,主要使用到了缓存来提升性能,需要的朋友可以参考下2015-04-04
这篇文章主要介绍了python之dlib包安装失败问题及解决方案，具有很好的参考价值，希望对大家有所帮助。如有错误或未考虑完全的地方，望不吝赐教2023-05-05
在本篇文章里小编给各位整理的是一篇关于python 函数嵌套及多函数共同运行知识点讲解，需要的朋友们可以学习下。2020-03-03
本文主要介绍了numpy数组重塑。所谓数组重塑就是更改数组的形状，文中通过示例代码介绍的非常详细，具有一定的参考价值，感兴趣的小伙伴们可以参考一下2021-09-09
这篇文章主要介绍了Anaconda使用IDLE的实现示例，文中通过示例代码介绍的非常详细，对大家的学习或者工作具有一定的参考学习价值，需要的朋友们下面随着小编来一起学习学习吧2020-09-09
Python实现string字符串连接的方法总结【8种方式】
这篇文章主要介绍了Python实现string字符串连接的方法,结合实例形式总结分析了Python实现字符串连接的8种常见操作技巧,需要的朋友可以参考下2018-07-07
使用简单的 for 循环在 python 中打印模式。第一个外循环用于处理行数， 而内嵌套循环用于处理列数。操作打印语句，可以打印不同的数字图案、字母图案或星形图案。本文将利用这些方法实现打印金字塔图案，需要的可以参考一下2022-09-09
这篇文章主要为大家介绍了python的定义函数，具有一定的参考价值，感兴趣的小伙伴们可以参考一下，希望能够给你带来帮助，希望能够给你带来帮助2021-11-11
这篇文章主要介绍了python使用arp欺骗伪造网关的方法,涉及Python伪造网关的相关技巧,需要的朋友可以参考下2015-04-04
使用 Python 快速实现 HTTP 和 FTP 服务器的方法
这篇文章主要介绍了使用 Python 快速实现 HTTP 和 FTP 服务器 的方法，本文给大家介绍的非常详细，具有一定的参考借鉴价值,需要的朋友可以参考下2019-07-07
最新评论