First of all, many blogs say that tf.variable_scope is a variable scope, which is not so strict, because the scope here is only used for naming and searching, but it does not have the function of local validity and global invalidity (such as function, object, file, etc.).

Let's start with a few more detailed examples, but they are not comprehensive enough.

Tensorflow Function Description (4) - variable_scope/name_scope

Reference resources: https://blog.csdn.net/qq_19918373/article/details/69499091
This paper mainly introduces the use of shared variables for tf.get_variable.

tf.get_variable is used differently from tf.variable. The former looks up the name when creating the variable. If the given name has been occupied by other variables before, it will report an error and will not create the corresponding variable. The latter does not check, if there is duplication, then automatically change the name, add numbers to distinguish. So from this point of view, sharing variables can't be done by calling tf. get_variable and tf. variable multiple times with the same name.

For example, the following code:

def my_image_filter(input_images):
    conv1_weights = tf.Variable(tf.random_normal([5, 5, 32, 32]),
        name="conv1_weights")
    conv1_biases = tf.Variable(tf.zeros([32]), name="conv1_biases")
    conv1 = tf.nn.conv2d(input_images, conv1_weights,
        strides=[1, 1, 1, 1], padding='SAME')
    relu1 = tf.nn.relu(conv1 + conv1_biases)
 
    conv2_weights = tf.Variable(tf.random_normal([5, 5, 32, 32]),
        name="conv2_weights")
    conv2_biases = tf.Variable(tf.zeros([32]), name="conv2_biases")
    conv2 = tf.nn.conv2d(relu1, conv2_weights,
        strides=[1, 1, 1, 1], padding='SAME')
    return tf.nn.relu(conv2 + conv2_biases)

In this function, we have four variables,'conv1_weights','conv1_biases','conv2_weights','conv2_biases'. If we reuse this function, multiple sets of variables will be generated, and the same variables will not be used, as follows:

# First call creates one set of variables.
result1 = my_image_filter(image1)
# Another set is created in the second call.
result2 = my_image_filter(image2)

In fact, two different filters are used to filter image1 and image2, although the same function is used. So, this creates a problem. Here's how to share variables.

We use with tf.variable_scope for sharing. For example, there is the following code:

def conv_relu(input, kernel_shape, bias_shape):
    # Create variable named "weights".
    weights = tf.get_variable("weights", kernel_shape,
        initializer=tf.random_normal_initializer())
    # Create variable named "biases".
    biases = tf.get_variable("biases", bias_shape,
        initializer=tf.constant_intializer(0.0))
    conv = tf.nn.conv2d(input, weights,
        strides=[1, 1, 1, 1], padding='SAME')
    return tf.nn.relu(conv + biases)
def my_image_filter(input_images):
    with tf.variable_scope("conv1"):
        # Variables created here will be named "conv1/weights", "conv1/biases".
        relu1 = conv_relu(input_images, [5, 5, 32, 32], [32])
    with tf.variable_scope("conv2"):
        # Variables created here will be named "conv2/weights", "conv2/biases".
        return conv_relu(relu1, [5, 5, 32, 32], [32])

To call my_image_filter twice and use the same variable, the following is done:

with tf.variable_scope("image_filters") as scope:
    result1 = my_image_filter(image1)
    scope.reuse_variables()
    result2 = my_image_filter(image2)

Use reuse_variables() to reuse variables. Notably, the following code explains how tf.get_variable works:

with tf.variable_scope("foo"):
    v = tf.get_variable("v", [1])
with tf.variable_scope("foo", reuse=True):
    v1 = tf.get_variable("v", [1])
assert v1 == v

If reuse is turned on, when the same name is checked, the variable with the same name is returned directly instead of redefining a duplicate value.

Here's what you need to pay attention to when using it

1. In variable_scope, variable_scope inherits the reuse value above, that is, the upper layer opens reuse, and the lower layer opens. However, reuse cannot be set to false artificially. Only by exiting variable_scope can reuse become false:

with tf.variable_scope("root"):
    # At start, the scope is not reusing.
    assert tf.get_variable_scope().reuse == False
    with tf.variable_scope("foo"):
        # Opened a sub-scope, still not reusing.
        assert tf.get_variable_scope().reuse == False
    with tf.variable_scope("foo", reuse=True):
        # Explicitly opened a reusing scope.
        assert tf.get_variable_scope().reuse == True
        with tf.variable_scope("bar"):
            # Now sub-scope inherits the reuse flag.
            assert tf.get_variable_scope().reuse == True
    # Exited the reusing scope, back to a non-reusing one.
    assert tf.get_variable_scope().reuse == False

2. When the name of another scope is used in a variable_scope, it is no longer bound by the hierarchical relationship here, but is directly the same as the name of the scope used:

with tf.variable_scope("foo") as foo_scope:
    assert foo_scope.name == "foo"
with tf.variable_scope("bar")
    with tf.variable_scope("baz") as other_scope:
        assert other_scope.name == "bar/baz"
        with tf.variable_scope(foo_scope) as foo_scope2:
            assert foo_scope2.name == "foo"  # Not changed.

3.name_scope is slightly different from variable_scope. Name_scope only affects the name of ops, not variables.

with tf.variable_scope("foo"):
    with tf.name_scope("bar"):
        v = tf.get_variable("v", [1])
        x = 1.0 + v
assert v.name == "foo/v:0"
assert x.op.name == "foo/bar/add"

Differences and Relations between tf.Variable() and tf.get_variable() in Tensorflow

Reference resources: https://blog.csdn.net/kevindree/article/details/86936476
From the names of the two methods, you can simply understand that Variable is the definition of variables, and get_variable is the acquisition of variables (just redefine a variable if it can not be obtained). If you follow this logic, you can basically understand the difference between the two.

Now let's take a deeper look at the differences and characteristics of the two through some code.

Look at the following code first (it's better not to look at the output in the comments below, but to think for yourself what it will be).

import tensorflow as tf
 
v1 = tf.Variable(1,name="V1")                     # First sentence
v2 = tf.Variable(2,name="V1")                     # Second sentence
v3 = tf.Variable(3,name="V1")                     # Third sentence
v4 = tf.Variable(4,name="V1_1")                   # Fourth sentence
 
print ("v1:",v1.name)
print ("v2:",v2.name)
print ("v3:",v3.name)
print ("v4:",v4.name)
 
v1 = tf.Variable(1,name="V1")                     # The fifth sentence
print ("v1:",v1.name)
 



### The output is ###
# v1: V1:0
# v2: V1_1:0
# v3: V1_2:0
# v4: V1_1_1:0
# v1: V1_3:0

This code indicates that a variable is defined and passed in the parameter name. If the name takes the same value, the system will automatically add "_n" after the last incoming name, and N will increase in order from the beginning.

For example, in the second sentence and the third sentence above, because the parameter name passed in when defining a variable takes the same value as the name in the first sentence (V1), we add "_1" and "_2" respectively on the basis of "V1". That is, although the same variable name is specified when defining variables, the system creates a new memory space and automatically generates a new name.

Although the value of parameter name in the fourth sentence above is different from that in the first three sentences, because the value of the parameter name in the fourth sentence is the same as that of the name automatically generated by the system in the second sentence v2, it will be added "_1" on the basis of the above-mentioned rule, so that v4.name is V1_1_1.

Looking at the fifth sentence, it is exactly the same as the first sentence, but the system will still create a new memory space according to the above rules, and automatically generate a new name. There are already V1, V1_1, V1_2, so the generated name is V1_3, because the variable is still v1, so the system points V1 to the new one. Memory space.

Let's see what happens if it's tf.get_variable. (Please create a new program to run, and don't interfere with the code you just wrote.)

import tensorflow as tf
 
v5 = tf.get_variable(name="V1",initializer=1)
v6 = tf.get_variable(name="V1",initializer=1)
print ("v5:",v5.name)
print ("v6:",v6.name)

You guessed right, the result would be wrong: ValueError: Variable V1 already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope?

Because get_variable does not automatically process the same name variable created by get_variable() without defining the variable scope variable_scope, and notice that the bold attributive get_variable() above me is created. That is to say, is it possible to create a variable ()? Let's try:

import tensorflow as tf
 
v1 = tf.Variable(1,name="V1")
print ("v1:",v1.name)
 
v5 = tf.get_variable(name="V1",initializer=1)
v6 = tf.get_variable(name="V1_1",initializer=1)
print ("v5:",v5.name)
print ("v6:",v6.name)





### The output is as follows ###
# v1: V1:0
# v5: V1_1:0
# v6: V1_1_1:0

Is it a little out of your imagination? The system does not report errors, but creates a new variable memory space for V5 according to the rules mentioned above, and automatically generates a variable name. For v6, why is there no error? Hasn't the system generated name=V1_1 for v5? Back to the above, let's look at the bold attributive get_variable() I gave you, because the name of V1_1 is automatically generated by the system, not passed in when using get_variable() method, so the system can still manage the rename and automatically add "_1" after it.

Maybe it's a little winding, but you can understand it with a good understanding. Understanding the above characteristics, when writing programs, you can avoid many strange problems.

Now let's talk about variable scope variable_scope and shared variables. Let's take a look at the code first.

import tensorflow as tf
 
with tf.variable_scope("scope1"):
    v1 = tf.Variable(1, name="V1")
    v2 = tf.get_variable(name="V2", initializer=1.0)
 
with tf.variable_scope("scope1", reuse=True):
    v3 = tf.Variable(1, name="V1")
    v4 = tf.get_variable(name="V2", initializer=1.0)
 
print(v1.name)
print(v2.name)
print(v3.name)
print(v4.name)
 
print(v1 is v3, v2 is v4)
 




### The output is ###
# scope1/V1:0
# scope1/V2:0
# scope1_1/V1:0
# scope1/V2:0
# False True

From the above code, we can see that the variables V1 and v3 defined by Variable() set the reuse parameter to True in the same scope. Because the incoming names are all V1, the system still creates a new memory space. Although the name of the variable remains unchanged, the scope is automatically modified. 1_1.

The variables V2 and V4 defined by get_variable() are in the same scope, and reuse parameter is set to True, and the incoming parameter name is V2, so the system does not create a new variable, but shares the variable, that is, v2==v4.

Hope these contents can be helpful to you all!

The above blog, in fact, I think the name and variable name do not matter, in fact, the essence of Python is the reference (scope conversion), and Tensorflow is a computational graph, so the above code is not difficult to understand. What puzzles me most is the last piece of code, which mixes tf. Variable and tf.get_variable under variable_scope. It's amazing for tf. Variable. It directly modifies the upper-level domain name, and tf.get_variable can also be used normally. So this also shows that variable_scope is essentially a naming restriction and does not have a complete concept of variable function. It can be adjusted according to the use of different functions. It is only a command method of Tensorflow, not scope. In fact, you can change reuse to false or even use tf. Variable to create a different Variable with different names. For tf. Variable, the name of variable_scope will be automatically changed. In fact, the official also said that the combination of variable_scope and tf.get_variable is effective, and the combination of variable_scope and tf. variable_scope is effective. Not the original intention of the design.

Nonsense, the above blog has been well written, though not complete. Similar blogs include:
https://blog.csdn.net/Jerr__y/article/details/70809528
https://blog.csdn.net/u012223913/article/details/78533910
https://blog.csdn.net/u012436149/article/details/53018924
https://www.cnblogs.com/MY0213/p/9208503.html
https://www.jianshu.com/p/ab0d38725f88

From a comprehensive point of view, there are many, but they are not careful enough. Many details are not in place, such as:
tf.AUTO_REUSE, get_variable_scope().reuse_variables(), etc.

Differentiation between tf.name_scope and tf.variable_scope

Reference resources: https://blog.csdn.net/daizongxue/article/details/84284007
This article is a summary of some answers and blog posts on the Internet.

To explain this problem clearly, we need to combine the two ways of creating variables in tensorflow, tf.get_variable() and tf.Variable().

Under tf.name_scope:

The variable name created by tf.get_variable() is not affected by tf.name_scope, that is, the name of the created variable has no prefix of the name_scope definition. Moreover, if the shared variable is not specified, an error will be reported if the Rename is repeated.

tf.Variable() automatically detects whether there is a variable rename or not, and if so, handles it on its own.

To share variables, you need to use tf.variable_scope()

Analysis 1: https://www.zhihu.com/question/54513728/answer/515912730

For the use of tf.Variable, tf.name_scope, like tf.variable_scope, prefixes variables, which is equivalent to classification management and modularization.
For tf.get_variable, tf.name_scope is invalid. That is to say, TF thinks that when you use tf.get_variable, you only belong to tf.variable_scope to manage sharing or not.
Let's take an example:

with tf.name_scope('name_sp1') as scp1:
    with tf.variable_scope('var_scp2') as scp2:
        with tf.name_scope('name_scp3') as scp3:
            a = tf.Variable('a')
            b = tf.get_variable('b')

Equivalent to:

with tf.name_scope('name_sp1') as scp1:
    with tf.name_scope('name_sp2') as scp2:
        with tf.name_scope('name_scp3') as scp3:
            a = tf.Variable('a')
 
with tf.variable_scope('var_scp2') as scp2:
        b = tf.get_variable('b')

It should be noted that:

with tf.variable_scope('scp', reuse=True) as scp:
    a = tf.get_varialbe('a') #Report errors

and

with tf.variable_scope('scp', reuse=False) as scp:    
     a = tf.get_varialbe('a')
    a = tf.get_varialbe('a') #Report errors

Error will be reported, because reuse=True is that get_variable will force sharing, if it does not exist, error will be reported; reuse=Flase, it will force creation, if it already exists, error will also be reported.

If you want to achieve the "Share if you have, build if you don't", you can:

with tf.variable_scope('scp', reuse=tf.AUTO_REUSE) as scp:    
     a = tf.get_variable('a') #Nothing, Creation
     a = tf.get_variable('a') #Yes, sharing

When you write a mapping module (e.g. LSTM_block(), you don't know if the user will share the module, so you can group the variables in the module with tf.variable_scope only, and use tf.get_variable to provide the possibility of sharing instead of tf.Variable.

For example: We want to build a New Auto Encoder that contains an encoder and two decoder s that are shared.

First, create a new Dense layer:

def Dense(x, x_dim, y_dim, name, reuse=None):
 
    with tf.variable_scope(name, reuse=reuse):
        w = tf.get_variable('weight', [x_dim, y_dim])
        b = tf.get_variable('bias', [y_dim])
        y = tf.add(tf.matmul(x, w), b)
    return y
 
 
def Encoder(x, name):
 
    with tf.variable_scope(name, reuse=None):
        x = tf.nn.relu(Dense(x, 784, 1000, 'layer1', reuse=False))
        x = tf.nn.relu(Dense(x, 1000, 1000, 'layer2', reuse=False))
        x = Dense(x, 1000, 10, 'layer3', reuse=False)
    return x
 
def Decoder(x, name, reuse=None):
 
    with tf.variable_scope(name, reuse=reuse):
        x = tf.nn.relu(Dense(x, 10, 1000, 'layer1', reuse=False))
        x = tf.nn.relu(Dense(x, 1000, 1000, 'layer2', reuse=False))
        x = tf.nn.sigmoid(Dense(x, 1000, 784, 'layer3', reuse=False))
    return x
 
def build_network(x):
    batchsz = 32
    x_ph = tf.placeholder(tf.float32, [batchsz, 784], name='input')
    z_ph = tf.placeholder(tf.float32, [1, 10], name='z')
 
    x = Encoder(x_ph, 'Encoder')
    x_hat = Decoder(x, 'Decoder1', reuse=None)
    x_hat2 = Decoder(z_ph, 'Decoder2', reuse=True)
 
    # ...

Interpretation: All modules are encapsulated using tf.variable_scope with name parameters, such as Dense(), Encoder(), Decoder(). For modules that are explicitly not shared, such as Encoder in this example, reuse parameters may not be provided.

Note: When multiple tf_variable_scope nests, if reuse=True is turned on in the middle layer, the inner layer automatically shares all, even if reuse=False is set in the inner layer. And once tf.get_variable_scope().reuse_variables() is used to open the current domain sharing, it cannot be closed!

Summary:

1. tf.variable_scope and tf.get_variable must be used together (except global scope) to support share.

2. tf.Variable can be used alone or in combination with tf.name_scope to name variables and modularize them.

3. tf.Variable and tf.variable_scope collocation is not the original intention of the designer.

TensorFlow Foundation: Shared Variables https://www.jianshu.com/p/ab0d38725f88

The resuse in scope defaults to False, and the calling function reuse_variables() can be set to True. Once set to True, it cannot return to False, and the subspace reuse in this scope is True. If you don't want to reuse variables, you can go back to the upper scope, which is equivalent to the current scope of exit, such as

with tf.variable_scope("root"):
    # At start, the scope is not reusing.
    assert tf.get_variable_scope().reuse == False
    with tf.variable_scope("foo"):
        # Opened a sub-scope, still not reusing.
        assert tf.get_variable_scope().reuse == False
    with tf.variable_scope("foo", reuse=True):
        # Explicitly opened a reusing scope.
        assert tf.get_variable_scope().reuse == True
        with tf.variable_scope("bar"):
            # Now sub-scope inherits the reuse flag.
            assert tf.get_variable_scope().reuse == True
    # Exited the reusing scope, back to a non-reusing one.
    assert tf.get_variable_scope().reuse == False

One scope can be used as a parameter for another new scope, such as:

with tf.variable_scope("foo") as foo_scope:
    v = tf.get_variable("v", [1])
with tf.variable_scope(foo_scope):
    w = tf.get_variable("w", [1])
with tf.variable_scope(foo_scope, reuse=True):
    v1 = tf.get_variable("v", [1])
    w1 = tf.get_variable("w", [1])
assert v1 is v
assert w1 is w

Regardless of how nested scopes are, when an existing scope is opened with tf.variable_scope(), it jumps to that scope.

with tf.variable_scope("foo") as foo_scope:
    assert foo_scope.name == "foo"
with tf.variable_scope("bar"):
    with tf.variable_scope("baz") as other_scope:
        assert other_scope.name == "bar/baz"
        with tf.variable_scope(foo_scope) as foo_scope2:
            assert foo_scope2.name == "foo"  # Not changed.

There are two more details here, but I don't think the technical details behind them are accurate enough, that is, the scope of variables, and the problems of tf.auto_reuse and variable_scope_reuse().
http://wiki.jikexueyuan.com/project/tensorflow-zh/how_tos/variable_scope.html
https://blog.csdn.net/hancoder/article/details/90053736

After that, I also read a lot of blogs, and I found these two blogs are very good, detailed and comprehensive:
https://blog.csdn.net/qq_22522663/article/details/78729029
https://www.cnblogs.com/MY0213/p/9208503.html
The most important sentence is:
When reuse is False or None (which is also the default), the variable names under the same tf.variable_scope cannot be the same; when reuse is True, tf.variable_scope can only retrieve variables that have been created.

OK, I believe that after reading the above blog, we can basically understand the relationship and difference between tf.Variable, tf.get_variable, tf.variable_scope and tf.name_scope. But the drawback is that we haven't solved tf.auto_resue and tf.get_variable_scope().reuse_variables(), which increase the flexibility of use.
But a few points need to be noted:

1. tf.auto_resue is a feature only available in higher versions, roughly TensorFlow (>= 1.4.1)
   This part refers to:
   https://blog.csdn.net/leitouguan8655/article/details/82869881
   https://blog.csdn.net/zbgjhy88/article/details/78960388
   https://blog.csdn.net/johnboat/article/details/84846628
2. tf.get_variable_scope().reuse_variables() should be used with relative caution and with variable_scope.
   This part refers to:
   https://blog.csdn.net/JNingWei/article/details/78124526
   https://www.cnblogs.com/Charles-Wan/p/6200446.html

Generally speaking, the regulations of tf.Variable, tf.get_variable, tf.variable_scope and tf.name_scope do facilitate the management of variable names, but they also bring a lot of unexpected troubles, especially if you are not good enough in style and have no practical experience.

For more information:
https://www.cnblogs.com/hunttown/p/6866586.html
https://www.cnblogs.com/gczr/p/7614821.html
https://www.jianshu.com/p/ab0d38725f88