Before we go into details, let's look at how the man document of socket(7) describes this parameter:
SO_REUSEADDR Indicates that the rules used in validating addresses supplied in a bind(2) call should allow reuse of local addresses. For AF_INET sockets this means that a socket may bind, except when there is an active listening socket bound to the address. When the listening socket is bound to INADDR_ANY with a spe‐ cific port then it is not possible to bind to this port for any local address. Argument is an integer boolean flag.
From this document we can know three things:
- After using this parameter, the bind operation can reuse local address. Note that the local address, i.e. the local address composed of ip plus port, means that two local addresses, if any ip or port part is different, can coexist in themselves and do not need to use this parameter.
- When local address are used by a socket in listen state, the address cannot be reused with this parameter.
- When the ip part of the local address of the socket listening in listen state is INADDR_ANY, that is to say, all the local IPS are monitored. Even with this parameter, the bind can no longer contain any local address of this port. This is exactly the same as described in 2.
Okay, let's look at a few examples.
As mentioned in Figure 1 above, as long as the local address is different (ip or port is different), even without this parameter, the two addresses can be used at the same time. Let's see if this is the case.
The following is the client test code:
#include <arpa/inet.h> #include <assert.h> #include <netinet/in.h> #include <stdio.h> #include <strings.h> #include <sys/socket.h> #include <sys/types.h> #include <unistd.h> static int tcp_connect(char *ip, int port) { int sfd, err; struct sockaddr_in addr; sfd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP); assert(sfd != -1); // First bind local address bzero(&addr, sizeof(addr)); addr.sin_family = AF_INET; addr.sin_addr.s_addr = inet_addr(ip); addr.sin_port = htons(port); err = bind(sfd, (struct sockaddr *)&addr, sizeof(addr)); assert(!err); // Reconnect to the target server bzero(&addr, sizeof(addr)); addr.sin_family = AF_INET; addr.sin_addr.s_addr = inet_addr("127.0.0.1"); addr.sin_port = htons(7777); err = connect(sfd, (struct sockaddr *)&addr, sizeof(addr)); assert(!err); return sfd; } int main(int argc, char *argv[]) { // bind local address: 127.0.0.1:8888 tcp_connect("127.0.0.1", 8888); // Bid local address: 192.168.3.187:8888 tcp_connect("192.168.3.187", 8888); printf("Successful simultaneous establishment of two connections\n"); sleep(100); return 0; }
In this code, the local address is bound first, and then the target server is connected. As can be seen from the above, the ip part of the local address of the two bind connections is different, so the two bind operations should be successful.
We simulate the server with the following ncat command:
$ ncat -lk4 7777
Use the ss command to view all socket status for port 7777:
$ ss -antp | grep 7777 LISTEN 0 10 0.0.0.0:7777 0.0.0.0:* users:(("ncat",pid=19208,fd=3))
As can be seen from above, only the ncat server is listening on port 7777 at this time, and there is no other connection.
Let's execute the above program, and then look at all socket status of port 7777 again:
$ ss -antp | grep 7777 LISTEN 0 10 0.0.0.0:7777 0.0.0.0:* users:(("ncat",pid=19208,fd=3)) ESTAB 0 0 127.0.0.1:7777 192.168.3.187:8888 users:(("ncat",pid=19208,fd=5)) ESTAB 0 0 127.0.0.1:7777 127.0.0.1:8888 users:(("ncat",pid=19208,fd=4)) ESTAB 0 0 192.168.3.187:8888 127.0.0.1:7777 users:(("a.out",pid=19340,fd=4)) ESTAB 0 0 127.0.0.1:8888 127.0.0.1:7777 users:(("a.out",pid=19340,fd=3))
As you can see from the above, the two connections have indeed been successfully established.
In the above command output, there are four ESTAB state connections, which is normal, because the output is from the server side and the client side respectively.
The first three lines are from the server's point of view, and the second two lines are from the client's point of view. This can also be seen from the process name behind.
For the client, it can bind different local addresses before connect ing, and then with a target, it is also possible for the server. Before listen ing, it can bind different local addresses without SO_REUSEADDR parameter. Because the program code is similar, we will not demonstrate it here.
Let's look at the same address of bind before connect ing. Here's the test code.
#include <arpa/inet.h> #include <assert.h> #include <netinet/in.h> #include <stdio.h> #include <stdlib.h> #include <strings.h> #include <sys/socket.h> #include <sys/types.h> #include <unistd.h> static int tcp_connect(char *ip, int port) { int sfd, err; char buf[1024]; struct sockaddr_in addr; sfd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP); assert(sfd != -1); // First bind local address bzero(&addr, sizeof(addr)); addr.sin_family = AF_INET; addr.sin_addr.s_addr = inet_addr("127.0.0.1"); addr.sin_port = htons(8888); err = bind(sfd, (struct sockaddr *)&addr, sizeof(addr)); if (err) { sprintf(buf, "bind(%s:%d)", ip, port); perror(buf); exit(-1); } // Reconnect to the target server bzero(&addr, sizeof(addr)); addr.sin_family = AF_INET; addr.sin_addr.s_addr = inet_addr(ip); addr.sin_port = htons(port); err = connect(sfd, (struct sockaddr *)&addr, sizeof(addr)); assert(!err); return sfd; } int main(int argc, char *argv[]) { // Connection destination address: 127.0.0.1:7777 tcp_connect("127.0.0.1", 7777); // Connection destination address: 127.0.0.1:7778 tcp_connect("127.0.0.1", 7778); printf("Successful simultaneous establishment of two connections\n"); sleep(100); return 0; }
Before connecting, the program will bind the local address to 127.0.0.1:8888, and then connect to the target address, which is 127.0.0.1:7777 and 127.0.0.1:7778, respectively.
Still use ncat to simulate the server, only this time to open two.
Server 7777:
$ ncat -lk4 7777
Server 7778:
$ ncat -lk4 7778
Run client code:
$ gcc client.c && ./a.out bind(127.0.0.1:7778): Address already in use
As can be seen from above, the second connection failed because 127.0.0.1:8888 local address has been used by the first connection.
At this point, adding SO_REUSEADDR parameter should be able to solve this problem.
#include <arpa/inet.h> #include <assert.h> #include <netinet/in.h> #include <stdio.h> #include <stdlib.h> #include <strings.h> #include <sys/socket.h> #include <sys/types.h> #include <unistd.h> static int tcp_connect(char *ip, int port) { int sfd, opt, err; char buf[1024]; struct sockaddr_in addr; sfd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP); assert(sfd != -1); // Set SO_REUSEADDR first opt = 1; err = setsockopt(sfd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt)); assert(!err); // Re bind Local Address bzero(&addr, sizeof(addr)); addr.sin_family = AF_INET; addr.sin_addr.s_addr = inet_addr("127.0.0.1"); addr.sin_port = htons(8888); err = bind(sfd, (struct sockaddr *)&addr, sizeof(addr)); if (err) { sprintf(buf, "bind(%s:%d)", ip, port); perror(buf); exit(-1); } // Then connect to the target server bzero(&addr, sizeof(addr)); addr.sin_family = AF_INET; addr.sin_addr.s_addr = inet_addr(ip); addr.sin_port = htons(port); err = connect(sfd, (struct sockaddr *)&addr, sizeof(addr)); assert(!err); return sfd; } int main(int argc, char *argv[]) { // Connection destination address: 127.0.0.1:7777 tcp_connect("127.0.0.1", 7777); // Connection destination address: 127.0.0.1:7778 tcp_connect("127.0.0.1", 7778); printf("Successful simultaneous establishment of two connections\n"); sleep(100); return 0; }
Compile again and execute:
$ gcc client.c && ./a.out Successful simultaneous establishment of two connections
As you can see from the above, both connections succeeded, and SO_REUSEADDR allows us to repeat the same local address as bind.
Careful students may find out why the target address of the two connections is different.
Let's try it the same way.
#include <arpa/inet.h> #include <assert.h> #include <netinet/in.h> #include <stdio.h> #include <stdlib.h> #include <strings.h> #include <sys/socket.h> #include <sys/types.h> #include <unistd.h> static int tcp_connect(char *ip, int port) { int sfd, opt, err; char buf[1024]; struct sockaddr_in addr; sfd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP); assert(sfd != -1); // Set SO_REUSEADDR first opt = 1; err = setsockopt(sfd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt)); assert(!err); // Re bind Local Address bzero(&addr, sizeof(addr)); addr.sin_family = AF_INET; addr.sin_addr.s_addr = inet_addr("127.0.0.1"); addr.sin_port = htons(8888); err = bind(sfd, (struct sockaddr *)&addr, sizeof(addr)); if (err) { sprintf(buf, "bind(%s:%d)", ip, port); perror(buf); exit(-1); } // Then connect to the target server bzero(&addr, sizeof(addr)); addr.sin_family = AF_INET; addr.sin_addr.s_addr = inet_addr(ip); addr.sin_port = htons(port); err = connect(sfd, (struct sockaddr *)&addr, sizeof(addr)); if (err) { sprintf(buf, "connect(%s:%d)", ip, port); perror(buf); exit(-1); } return sfd; } int main(int argc, char *argv[]) { // Connection destination address: 127.0.0.1:7777 tcp_connect("127.0.0.1", 7777); // Connection destination address: 127.0.0.1:7777 tcp_connect("127.0.0.1", 7777); printf("Successful simultaneous establishment of two connections\n"); sleep(100); return 0; }
At this point, execute the program, the command line will have the following output:
$ gcc client.c && ./a.out connect(127.0.0.1:7777): Cannot assign requested address
Why? Because these two connections are from 127.0.0.1:8888 to 127.0.0.1:7777, this is not allowed at the tcp level, even if the SO_REUSEADDR parameter is added.
The tuple consisting of local address and destination address determines only one tcp connection. The local address and destination address of the two connections in the upper process are the same, which violates the principle of uniqueness.
The corresponding check code for the corresponding kernel is as follows:
// net/ipv4/inet_hashtables.c static int __inet_check_established(struct inet_timewait_death_row *death_row, struct sock *sk, __u16 lport, struct inet_timewait_sock **twp) { struct inet_hashinfo *hinfo = death_row->hashinfo; struct inet_sock *inet = inet_sk(sk); __be32 daddr = inet->inet_rcv_saddr; __be32 saddr = inet->inet_daddr; ... const __portpair ports = INET_COMBINED_PORTS(inet->inet_dport, lport); unsigned int hash = inet_ehashfn(net, daddr, lport, saddr, inet->inet_dport); struct inet_ehash_bucket *head = inet_ehash_bucket(hinfo, hash); ... struct sock *sk2; ... sk_nulls_for_each(sk2, node, &head->chain) { ... if (likely(INET_MATCH(sk2, net, acookie, saddr, daddr, ports, dif, sdif))) { ... goto not_unique; } } ... not_unique: ... return -EADDRNOTAVAIL; }
If a tuple of local and destination addresses already exists, the error code EADDRNOTAVAIL is returned, which is interpreted as follows:
// include/uapi/asm-generic/errno.h #define EADDRNOTAVAIL 99 /* Cannot assign requested address */
Just like the error message from the above executor.
Let's return to the discussion of SO_REUSEADDR parameters.
In the code above, the two connect ions use the same local address, so long as the SO_REUSEADDR parameter is added, are the two listen s OK?
Look at the code:
#include <arpa/inet.h> #include <assert.h> #include <stdio.h> #include <stdlib.h> #include <strings.h> #include <sys/socket.h> #include <sys/types.h> #include <unistd.h> static int tcp_listen(char *ip, int port) { int lfd, opt, err; char buf[1024]; struct sockaddr_in addr; lfd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP); assert(lfd != -1); opt = 1; err = setsockopt(lfd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt)); assert(!err); bzero(&addr, sizeof(addr)); addr.sin_family = AF_INET; addr.sin_addr.s_addr = inet_addr(ip); addr.sin_port = htons(port); err = bind(lfd, (struct sockaddr *)&addr, sizeof(addr)); if (err) { sprintf(buf, "bind(%s:%d)", ip, port); perror(buf); exit(-1); } err = listen(lfd, 8); assert(!err); return lfd; } int main(int argc, char *argv[]) { tcp_listen("127.0.0.1", 7777); tcp_listen("127.0.0.1", 7777); return 0; }
The output of this code after execution is as follows:
$ gcc server.c && ./a.out bind(127.0.0.1:7777): Address already in use
As can be seen from the above, even with the SO_REUSEADDR parameter, two listen s are not feasible.
In fact, as stated in the original man document, as long as listen takes up a local address, no other operation can use this address anymore.
Let's look at the kernel source code:
// net/ipv4/inet_connection_sock.c static int inet_csk_bind_conflict(const struct sock *sk, const struct inet_bind_bucket *tb, bool relax, bool reuseport_ok) { struct sock *sk2; bool reuse = sk->sk_reuse; ... sk_for_each_bound(sk2, &tb->owners) { if (sk != sk2 && ...) { if ((!reuse || !sk2->sk_reuse || sk2->sk_state == TCP_LISTEN) && ...) { if (inet_rcv_saddr_equal(sk, sk2, true)) break; } ... } } return sk2 != NULL; }
This method is used to determine whether the local address can be reused code.
If sk2 is not null in the end, the error code EADDRINUSE will be returned to the user, that is, the error output after the execution of the above program.
Let's see when sk2 is not null.
When our new socket and sk2 have the same local address, if the new socket does not set SO_REUSEADDR parameter, or sk2 does not set SO_REUSEADDR parameter, or sk2 is listen state, sk2 will eventually not be null, that is to say, the local address of the new socket can not be reused in these cases.
It's basically the same as what the man document says.
So why do we add this parameter when we write the server in peacetime? We all shut down the server first, and then open it. The previous listen socket, and all sockets that were connected at that time, should have been shut down? Should there not be the same local address?
Why?
This brings us to the TIME_WAIT state of tcp.
We know that in a tcp connection, the end that initiates the closure request will eventually enter the TIME_WAIT state, and the end that passively closes the connection will directly enter the CLOSE state, that is, the socket and the resources it occupies will be directly destroyed.
Suppose that before we shut down the server, we shut down the client first and then the server. At this time, all socket s of the server go directly into the CLOSE state, and the local address they occupy is also available immediately. At this time, if we start the server immediately, there will be no Address already in use error. .
But when we have a client connection, we shut down the server directly. That is to say, for all existing tcp connections, the server actively initiates a shutdown request. At this time, these connections will enter the TIME_WAIT state, occupying the local address used by the server and not allowing subsequent operations to use.
In this case, if you turn on the server again, the Address already in use error will appear, which is also the error we often encounter when writing the server.
The solution to this problem is to set the SO_REUSEADDR parameter.
As can be seen from the above inet_csk_bind_conflict method, if the SO_REUSEADDR parameter is set, the reuse value of the new socket and the old socket will be true, while the old socket is in the TIME_WAIT state at this time, so the inet_rcv_saddr_equal method will not be invoked subsequently to determine whether the two addresses are the same.
In the end, sk2 will also be null, which means that the kernel allows the new socket to use this address.
Verify with code:
#include <arpa/inet.h> #include <assert.h> #include <stdio.h> #include <stdlib.h> #include <strings.h> #include <sys/socket.h> #include <sys/types.h> #include <unistd.h> static int tcp_listen(char *ip, int port) { int lfd, opt, err; char buf[1024]; struct sockaddr_in addr; lfd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP); assert(lfd != -1); opt = 1; err = setsockopt(lfd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt)); assert(!err); bzero(&addr, sizeof(addr)); addr.sin_family = AF_INET; addr.sin_addr.s_addr = inet_addr(ip); addr.sin_port = htons(port); err = bind(lfd, (struct sockaddr *)&addr, sizeof(addr)); if (err) { sprintf(buf, "bind(%s:%d)", ip, port); perror(buf); exit(-1); } err = listen(lfd, 8); assert(!err); return lfd; } int main(int argc, char *argv[]) { int lfd, cfd; lfd = tcp_listen("127.0.0.1", 7777); printf("5 The first time will be closed in seconds. listen Of socket，Please initiate once during this period. tcp Connect\n"); sleep(5); cfd = accept(lfd, NULL, NULL); assert(cfd != -1); close(cfd); close(lfd); tcp_listen("127.0.0.1", 7777); printf("The second time listen Successful operation\n"); return 0; }
According to the program prompt, initiate the tcp connection to the server, and the final output of the server is as follows:
$ gcc server.c && ./a.out The socket for the first listen will be closed in 5 seconds, during which time a tcp connection will be initiated The second listen operation succeeded
So, with the SO_REUSEADDR parameter, even if we close the tcp connection first, we can listen again.
Interested friends can remove the code to set SO_REUSEADDR parameters and then execute it. In theory, it will be wrong.
So far, all the relevant SO_REUSEADDR parameters have been discussed, I hope to help you.
Finish.
For more original articles, please pay attention to my Wechat Public Number:
