参考链接: copy_(src, non_blocking=False) → Tensor
参考链接: detach()
参考链接: detach_()
参考链接: clone() → Tensor
总结:
使用clone():解释说明: 返回一个原张量的副本,同时不破换计算图,它能够维持反向传播计算梯度,并且两个张量不共享内存.一个张量上值的改变不影响另一个张量.使用copy_():解释说明: 比如x4.copy_(x2), 将x2的数据复制到x4,并且会修改计算图,使得反向传播自动计算梯度时,计算出x4的梯度后再继续前向计算x2的梯度. 注意,复制完成之后,两者的值的改变互不影响,因为他们并不共享内存.使用detach():解释说明: 比如x4 = x2.detach(),返回一个和原张量x2共享内存的新张量x4,两者的改动可以相互可见, 一个张量上的值的改动会影响到另一个张量.返回的新张量和计算图相互独立,即新张量和计算图不再关联,因此也无法进行反向传播计算梯度.即从计算图上把这个张量x2拆卸detach下来,非常形象.使用detach_():解释说明: detach()的原地操作版本,功能和detach()类似.比如x4 = x2.detach_(),其实x2和x4是同一个对象,返回的是self,x2和x4具有相同的id()值.
以下是手动绘制的计算图(其中的x3部分暂时不用,请读者暂时忽略和x3相关的部分):
1.最简单的情况:x4 = x2
解释说明: 无需做过多解释
import torchtorch.manual_seed(seed=)x0 = torch.randn(3, 5, requires_grad=True)bias = torch.randn(3, 5, requires_grad=True)x3 = torch.randn(3, 5, requires_grad=False)x4 = torch.randn(3, 5, requires_grad=False)x1 = 3 * x0 + 8x2 = 4 * x1 ** 2 + 5 * x1 + 7## x3 = x2.clone()# x3 = x3.copy_(x2)# x3 = x2.detach()# x3 = x2.detach_()# x3 = x2## x4 = x2.clone()# x4 = x4.copy_(x2)# x4 = x2.detach()# x4 = x2.detach_()x4 = x2#x5 = 9 * x4 ** 3 + 7 * x4 ** 2 + 3 * x4 + 5 + biasloss = torch.mean(x5)print("loss =",loss)loss.backward()print("输出梯度信息:")print(x0.grad)
控制台输出结果:
Windows PowerShell版权所有 (C) Microsoft Corporation。保留所有权利。尝试新的跨平台 PowerShell https://aka.ms/pscore6加载个人及系统配置文件用了 899 毫秒。(base) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq> conda activate ssd4pytorch1_2_0(ssd4pytorch1_2_0) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq> & 'D:\Anaconda3\envs\ssd4pytorch1_2_0\python.exe' 'c:\Users\chenxuqi\.vscode\extensions\ms-python.python-.12.424452561\pythonFiles\lib\python\debugpy\launcher' '49618' '--' 'c:\Users\chenxuqi\Desktop\News4cxq\test4cxq\test25.py'loss = tensor(9.6161e+08, grad_fn=<MeanBackward0>)输出梯度信息:tensor([[5.4391e+07, 1.7363e+07, 1.3316e+08, 3.6766e+05, 2.4903e+07],[5.1519e+08, 1.5781e+08, 3.6020e+05, 1.7468e+08, 1.2224e+08],[4.6100e+07, 9.8575e+07, 6.9595e+05, 2.4009e+07, 1.6252e+07]])(ssd4pytorch1_2_0) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq>
2.使用clone()的情况:x4 = x2.clone()
解释说明: 返回一个原张量的副本,同时不破换计算图,它能够维持反向传播计算梯度.
import torchtorch.manual_seed(seed=)x0 = torch.randn(3, 5, requires_grad=True)bias = torch.randn(3, 5, requires_grad=True)x3 = torch.randn(3, 5, requires_grad=False)x4 = torch.randn(3, 5, requires_grad=False)x1 = 3 * x0 + 8x2 = 4 * x1 ** 2 + 5 * x1 + 7## x3 = x2.clone()# x3 = x3.copy_(x2)# x3 = x2.detach()# x3 = x2.detach_()# x3 = x2#x4 = x2.clone()# x4 = x4.copy_(x2)# x4 = x2.detach()# x4 = x2.detach_()# x4 = x2#x5 = 9 * x4 ** 3 + 7 * x4 ** 2 + 3 * x4 + 5 + biasloss = torch.mean(x5)print("loss =",loss)loss.backward()print("输出梯度信息:")print(x0.grad)
控制台输出结果:
Windows PowerShell版权所有 (C) Microsoft Corporation。保留所有权利。尝试新的跨平台 PowerShell https://aka.ms/pscore6加载个人及系统配置文件用了 901 毫秒。(base) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq> conda activate ssd4pytorch1_2_0(ssd4pytorch1_2_0) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq> & 'D:\Anaconda3\envs\ssd4pytorch1_2_0\python.exe' 'c:\Users\chenxuqi\.vscode\extensions\ms-python.python-.12.424452561\pythonFiles\lib\python\debugpy\launcher' '49739' '--' 'c:\Users\chenxuqi\Desktop\News4cxq\test4cxq\test25.py'loss = tensor(9.6161e+08, grad_fn=<MeanBackward0>)输出梯度信息:tensor([[5.4391e+07, 1.7363e+07, 1.3316e+08, 3.6766e+05, 2.4903e+07],[5.1519e+08, 1.5781e+08, 3.6020e+05, 1.7468e+08, 1.2224e+08],[4.6100e+07, 9.8575e+07, 6.9595e+05, 2.4009e+07, 1.6252e+07]])(ssd4pytorch1_2_0) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq>
3.使用copy_()的情况:x4 = x4.copy_(x2)或者x4.copy_(x2)
解释说明: 将x2的数据复制到x4,并且会修改计算图,使得反向传播自动计算梯度时,计算出x4的梯度后再继续前向计算x2的梯度.
import torchtorch.manual_seed(seed=)x0 = torch.randn(3, 5, requires_grad=True)bias = torch.randn(3, 5, requires_grad=True)x3 = torch.randn(3, 5, requires_grad=False)x4 = torch.randn(3, 5, requires_grad=False)x1 = 3 * x0 + 8x2 = 4 * x1 ** 2 + 5 * x1 + 7## x3 = x2.clone()# x3 = x3.copy_(x2)# x3 = x2.detach()# x3 = x2.detach_()# x3 = x2## x4 = x2.clone()# x4 = x4.copy_(x2)# x4.copy_(x3)x4.copy_(x2)# x4 = x2.detach()# x4 = x2.detach_()# x4 = x2#x5 = 9 * x4 ** 3 + 7 * x4 ** 2 + 3 * x4 + 5 + biasloss = torch.mean(x5)print("loss =",loss)loss.backward()print("输出梯度信息:")print(x0.grad)
控制台输出结果:
Windows PowerShell版权所有 (C) Microsoft Corporation。保留所有权利。尝试新的跨平台 PowerShell https://aka.ms/pscore6加载个人及系统配置文件用了 914 毫秒。(base) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq> conda activate ssd4pytorch1_2_0(ssd4pytorch1_2_0) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq> & 'D:\Anaconda3\envs\ssd4pytorch1_2_0\python.exe' 'c:\Users\chenxuqi\.vscode\extensions\ms-python.python-.12.424452561\pythonFiles\lib\python\debugpy\launcher' '49845' '--' 'c:\Users\chenxuqi\Desktop\News4cxq\test4cxq\test25.py'loss = tensor(9.6161e+08, grad_fn=<MeanBackward0>)输出梯度信息:tensor([[5.4391e+07, 1.7363e+07, 1.3316e+08, 3.6766e+05, 2.4903e+07],[5.1519e+08, 1.5781e+08, 3.6020e+05, 1.7468e+08, 1.2224e+08],[4.6100e+07, 9.8575e+07, 6.9595e+05, 2.4009e+07, 1.6252e+07]])(ssd4pytorch1_2_0) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq>
注意:如果将x4.copy_(x2)改成x4.copy_(x3),那么反向传播无法传递到x0,控制台输出的结果是None:
import torchtorch.manual_seed(seed=)x0 = torch.randn(3, 5, requires_grad=True)bias = torch.randn(3, 5, requires_grad=True)x3 = torch.randn(3, 5, requires_grad=False)x4 = torch.randn(3, 5, requires_grad=False)x1 = 3 * x0 + 8x2 = 4 * x1 ** 2 + 5 * x1 + 7## x3 = x2.clone()# x3 = x3.copy_(x2)# x3 = x2.detach()# x3 = x2.detach_()# x3 = x2## x4 = x2.clone()# x4 = x4.copy_(x2)x4.copy_(x3)# x4.copy_(x2)# x4 = x2.detach()# x4 = x2.detach_()# x4 = x2#x5 = 9 * x4 ** 3 + 7 * x4 ** 2 + 3 * x4 + 5 + biasloss = torch.mean(x5)print("loss =",loss)loss.backward()print("输出梯度信息:")print(x0.grad)
Windows PowerShell版权所有 (C) Microsoft Corporation。保留所有权利。尝试新的跨平台 PowerShell https://aka.ms/pscore6加载个人及系统配置文件用了 873 毫秒。(base) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq> conda activate ssd4pytorch1_2_0(ssd4pytorch1_2_0) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq> & 'D:\Anaconda3\envs\ssd4pytorch1_2_0\python.exe' 'c:\Users\chenxuqi\.vscode\extensions\ms-python.python-.12.424452561\pythonFiles\lib\python\debugpy\launcher' '49870' '--' 'c:\Users\chenxuqi\Desktop\News4cxq\test4cxq\test25.py'loss = tensor(16.7390, grad_fn=<MeanBackward0>)输出梯度信息:None(ssd4pytorch1_2_0) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq>
4.使用detach()函数情况:x4 = x2.detach()
解释说明: 返回一个和原张量共享内存的新张量,两者的改动可以相互可见, 返回的新张量和计算图相互独立,即新张量和计算图不再关联,因此也无法 进行反向传播计算梯度.
import torchtorch.manual_seed(seed=)x0 = torch.randn(3, 5, requires_grad=True)bias = torch.randn(3, 5, requires_grad=True)x3 = torch.randn(3, 5, requires_grad=False)x4 = torch.randn(3, 5, requires_grad=False)x1 = 3 * x0 + 8x2 = 4 * x1 ** 2 + 5 * x1 + 7## x3 = x2.clone()# x3 = x3.copy_(x2)# x3 = x2.detach()# x3 = x2.detach_()# x3 = x2## x4 = x2.clone()# x4 = x4.copy_(x2)# x4.copy_(x3)# x4.copy_(x2)x4 = x2.detach()print('x4.requires_grad:',x4.requires_grad)# x4 = x2.detach_()# x4 = x2#x5 = 9 * x4 ** 3 + 7 * x4 ** 2 + 3 * x4 + 5 + biasloss = torch.mean(x5)print("loss =",loss)loss.backward()print("输出梯度信息:")print(x0.grad)
控制台输出结果:
Windows PowerShell版权所有 (C) Microsoft Corporation。保留所有权利。尝试新的跨平台 PowerShell https://aka.ms/pscore6加载个人及系统配置文件用了 975 毫秒。(base) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq> conda activate ssd4pytorch1_2_0(ssd4pytorch1_2_0) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq> & 'D:\Anaconda3\envs\ssd4pytorch1_2_0\python.exe' 'c:\Users\chenxuqi\.vscode\extensions\ms-python.python-.12.424452561\pythonFiles\lib\python\debugpy\launcher' '50013' '--' 'c:\Users\chenxuqi\Desktop\News4cxq\test4cxq\test25.py'x4.requires_grad: Falseloss = tensor(9.6161e+08, grad_fn=<MeanBackward0>)输出梯度信息:None(ssd4pytorch1_2_0) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq>
5.使用detach_()的情况:x4 = x2.detach_()
解释说明: detach()的原地操作版本,功能和detach()类似.
import torchtorch.manual_seed(seed=)x0 = torch.randn(3, 5, requires_grad=True)bias = torch.randn(3, 5, requires_grad=True)x3 = torch.randn(3, 5, requires_grad=False)x4 = torch.randn(3, 5, requires_grad=False)x1 = 3 * x0 + 8x2 = 4 * x1 ** 2 + 5 * x1 + 7## x3 = x2.clone()# x3 = x3.copy_(x2)# x3 = x2.detach()# x3 = x2.detach_()# x3 = x2## x4 = x2.clone()# x4 = x4.copy_(x2)# x4.copy_(x3)# x4.copy_(x2)# x4 = x2.detach()# print('x4.requires_grad:',x4.requires_grad)x4 = x2.detach_()print('x4.requires_grad:',x4.requires_grad)# x4 = x2#x5 = 9 * x4 ** 3 + 7 * x4 ** 2 + 3 * x4 + 5 + biasloss = torch.mean(x5)print("loss =",loss)loss.backward()print("输出梯度信息:")print(x0.grad)
控制台输出结果:
Windows PowerShell版权所有 (C) Microsoft Corporation。保留所有权利。尝试新的跨平台 PowerShell https://aka.ms/pscore6加载个人及系统配置文件用了 915 毫秒。(base) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq> conda activate ssd4pytorch1_2_0(ssd4pytorch1_2_0) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq> & 'D:\Anaconda3\envs\ssd4pytorch1_2_0\python.exe' 'c:\Users\chenxuqi\.vscode\extensions\ms-python.python-.12.424452561\pythonFiles\lib\python\debugpy\launcher' '50131' '--' 'c:\Users\chenxuqi\Desktop\News4cxq\test4cxq\test25.py'x4.requires_grad: Falseloss = tensor(9.6161e+08, grad_fn=<MeanBackward0>)输出梯度信息:None(ssd4pytorch1_2_0) PS C:\Users\chenxuqi\Desktop\News4cxq\test4cxq>
如果觉得《详解PyTorch中的copy_()函数 detach()函数 detach_()函数和clone()函数》对你有帮助，请点赞、收藏，并留下你的观点哦！
