# Two functions of pytorch detach() .detach_ () functions and differences of

When we retrain the network, we may want to keep some network parameters unchanged and adjust only some of them; Or we can train some branch networks to prevent their gradients from affecting the gradients of the main network. At this time, we need to use the detach() function to cut off the gradients of some branches Back propagation

## 1, Tensor detach()

Returns a new tensor, which is separated from the current calculation diagram, but still points to the storage location of the original variable. The difference is only requires_grad is false. The resulting tensor never needs to calculate its gradient and does not have grad.

Even if it's required again later_ If grad is set to true, it will not have gradient grad

In this way, we will continue to use the new tensor for calculation. Later, when we conduct back propagation, the tensor calling detach() will stop and can no longer continue to propagate forward

be careful:

The tensor returned by detach and the original tensor share the same memory, that is, when one is modified, the other will change.

For example, a normal example is:

``` 1 import torch
2
3 a = torch.tensor([1, 2, 3.], requires_grad=True)
5 out = a.sigmoid()
6
7 out.sum().backward()
9 '''return:
10 None
11 tensor([0.1966, 0.1050, 0.0452])
12 '''```

1.1. When detach() is used to separate the tensor, but this tensor is not changed, backward() will not be affected:

``` 1 import torch
2
3 a = torch.tensor([1, 2, 3.], requires_grad=True)
5 out = a.sigmoid()
6 print(out)
7
9 c = out.detach()
10 print(c)
11
12 #Not right at this time c Make changes, so it will not affect backward()
13 out.sum().backward()
15
16 '''return:
17 None
19 tensor([0.7311, 0.8808, 0.9526])
20 tensor([0.1966, 0.1050, 0.0452])
21 '''```

It can be seen from the above that tensor c is separated from out, but I have not changed this c. There will be no error in deriving the original out at this time, that is

c. The difference between out and C is that C has no gradient and out has gradient. However, it should be noted that the following two cases are reported incorrectly,

1.2 when detach() is used to separate the tensor, and then the separated tensor is used to calculate the derivative, it will affect backward(), and errors will occur

``` 1 import torch
2
3 a = torch.tensor([1, 2, 3.], requires_grad=True)
5 out = a.sigmoid()
6 print(out)
7
9 c = out.detach()
10 print(c)
11
12 #Use newly generated Variable Back propagation
13 c.sum().backward()
15
16 '''return:
17 None
19 tensor([0.7311, 0.8808, 0.9526])
20 Traceback (most recent call last):
21   File "test.py", line 13, in <module>
22     c.sum().backward()
23   File "/anaconda3/envs/deeplearning/lib/python3.6/site-packages/torch/tensor.py", line 102, in backward
25   File "/anaconda3/envs/deeplearning/lib/python3.6/site-packages/torch/autograd/__init__.py", line 90, in backward
26     allow_unreachable=True)  # allow_unreachable flag
27 RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
28 '''```

1.3 when detach() is used to separate the tensor and this tensor is changed, even if the derivative of the original out is calculated again, backward() will be affected and errors will occur

If you make a change to c at this time, the change will be tracked by autograd in out An error will also be reported when sum() performs backward() because the gradient obtained by backward() is wrong:

``` 1 import torch
2
3 a = torch.tensor([1, 2, 3.], requires_grad=True)
5 out = a.sigmoid()
6 print(out)
7
9 c = out.detach()
10 print(c)
11 c.zero_() #use in place Function to modify it
12
13 #Will find c The modification of will also affect out Value of
14 print(c)
15 print(out)
16
17 #Right now c Make changes, so it will affect backward()，It can't be done at this time backward()，Will report an error
18 out.sum().backward()
20
21 '''return:
22 None
24 tensor([0.7311, 0.8808, 0.9526])
25 tensor([0., 0., 0.])
27 Traceback (most recent call last):
28   File "test.py", line 16, in <module>
29     out.sum().backward()
30   File "/anaconda3/envs/deeplearning/lib/python3.6/site-packages/torch/tensor.py", line 102, in backward
32   File "/anaconda3/envs/deeplearning/lib/python3.6/site-packages/torch/autograd/__init__.py", line 90, in backward
33     allow_unreachable=True)  # allow_unreachable flag
34 RuntimeError: one of the variables needed for gradient computation has been modified
35 by an inplace operation
36 '''```

2, Tensor detach_ ()
Separate a tensor from the graph that created it and set it as a leaf tensor

In fact, it is equivalent to that the relationship between variables is X - > m - > y, and the leaf tensor here is x, but m.detach is performed on m at this time_ () operation is actually two operations:

• Grad of M_ The value of FN is set to None, so that m will no longer be associated with the previous node X. The relationship here will become x, M - > y, and m will become a leaf node
• Then m's requirements will be_ Grad is set to False, so that the gradient of m will not be calculated when backward() is performed on y

Summary: actually detach() and detach_ () very similar. The difference between the two is detach_ () is a change to itself, and detach() generates a new tensor

For example, in X - > m - > y, if you detach m (), you can still operate the original calculation diagram if you want to go back later

But if detach is performed_ (), then the original calculation chart has also changed, so we can't go back