In the process of learning Socket, I always wanted to learn about the bottom layer, so I found many materials on the Internet and found that there was still little description of the bottom layer. Finally, I found an article on managing Tcp connection. Because we know that Socket encapsulates Tcp/ip protocol, Socket itself is not a protocol, but a call interface API
Relationship between Socket connection and TCP/IP connection
When creating a socket connection, you can specify the transport layer protocol to use. Socket can support different transport layer protocols (TCP or UDP).
When the TCP protocol is used for connection, the Socket connection is a TCP connection.
socket is the encapsulation and application of TCP/IP protocol (at the programmer level). It can also be said that TCP/IP protocol is a transport layer protocol, which mainly solves how to transmit data in the network, while HTTP is an application layer protocol, which mainly solves how to package data. About the relationship between TCP/IP and HTTP protocol, the network has an easy to understand introduction:
"When we transmit data, we can only use (transport layer) TCP/IP protocol, but in that case, if there is no application layer, the data content cannot be recognized. If you want to make the transmitted data meaningful, you must use the application layer protocol. There are many application layer protocols, such as HTTP, FTP, TELNET, etc. you can also define your own application layer protocol. The WEB uses HTTP protocol as the application layer protocol to encapsulate HTTP text information, and then uses TCP/IP as the transport layer protocol to send it to the network. "
What socket do we usually talk about most? In fact, socket is for TCP/IP
The encapsulation of the protocol, Socket itself is not a protocol, but a call interface (API).
Through socket, we can use TCP/IP protocol. In fact, socket is not necessarily related to TCP/IP protocol. When designing the socket programming interface, I hope it can also adapt to other network protocols. Therefore, the emergence of socket only makes it easier for programmers to use TCP/IP protocol stack. It is an abstraction of TCP/IP protocol, thus forming some of the most basic function interfaces we know, such as create, listen, connect, accept, send, read and write. The network has a statement about the relationship between socket and TCP/IP protocol, which is easy to understand:
"TCP/IP is just a protocol stack. Like the operating mechanism of the operating system, it must be implemented in detail and provide external operating interfaces. Just as the operating system will provide standard programming interfaces, such as win32 programming interfaces, TCP/IP should also provide interfaces for programmers to do network development, which is the Socket programming interface."
In fact, TCP in the transport layer is based on IP protocol in the network layer, while HTTP protocol in the application layer is based on TCP protocol in the transport layer. Socket itself is not a protocol. As mentioned above, it only provides an interface for TCP or UDP programming. Socket is a tool for port communication development, which is lower level
Server
Firstly, a global instance is maintained in the network module of linux kernel to store all tcp related data
socket:
// net/ipv4/tcp_ipv4.c struct inet_hashinfo tcp_hashinfo;
Secondly, the instance is divided into four hashtable s according to different socket types:
// include/net/inet_hashtables.h
struct inet_hashinfo {
// key is a quad consisting of local address, local port, remote address and remote port
// value is the socket that is establishing or has established a connection
// For example, when the kernel receives a tcp message, it first reads the address, port and other information from the message header
// Then use this information to get the corresponding socket in ehash
// Finally, the remaining tcp data is added to the recv buf of the socket for the user program to read
struct inet_ehash_bucket *ehash;
// key is the local port
// value is all socket s using this port
// For example, when we use socket to listen on a port, the socket is in bhash
// Similarly, the socket s corresponding to the connection established by the listening port are also here
// Because they all use the same local port
struct inet_bind_hashbucket *bhash;
// key is a binary of local address and port
// value is the corresponding socket in listen state
struct inet_listen_hashbucket *lhash2;
// key is the local port
// value is the corresponding socket in listen state
struct inet_listen_hashbucket listening_hash[INET_LHTABLE_SIZE];
};
At system startup, this global tcp_hashinfo instance will be initialized in the following method:
// net/ipv4/tcp.c
void __init tcp_init(void)
{
// Initialize TCP_ Four hashtable s in hashinfo
}
This TCP_ The hashinfo instance will also be assigned to the following TCP_ Corresponding fields in prot instance:
// net/ipv4/tcp_ipv4.c
struct proto tcp_prot = {
// Sk will be passed in struct sock_ The prot field references the tcp_prot instance
// In other words, if you get any struct sock instance
// You can use its sk_ Get TCP from prot field_ Prot instance
// Then you can get tcp_hashinfo instance
.h.hashinfo = &tcp_hashinfo,
};
EXPORT_SYMBOL(tcp_prot);
OK, the above is the global data structure used by the operating system to manage tcp connections. Next, let's look at some specific operations.
tcp programming is generally divided into client and server. Let's first look at the corresponding operations of the server.
First, if a socket wants to listen to a port, it must bind an address first, and then perform the listen operation.
The bind operation uses the above TCP_ The bhash field in the hashinfo instance is used to determine whether the port is occupied.
Look at the following code:
// net/ipv4/inet_connection_sock.c
int inet_csk_get_port(struct sock *sk, unsigned short snum)
{
// Call stack of this method:
// SYSCALL_DEFINE3(bind)
// __sys_bind
// inet_bind
// inet_csk_get_port
// The following hinfo is the global instance tcp_hashinfo
struct inet_hashinfo *hinfo = sk->sk_prot->h.hashinfo;
// Calculate the hash value according to the port, and then find the corresponding slot in the bhash according to this value
head = &hinfo->bhash[inet_bhashfn(net, port,
hinfo->bhash_size)];
// Traverse the linked list pointed to by slot and find the value corresponding to port
inet_bind_bucket_for_each(tb, &head->chain)
if (net_eq(ib_net(tb), net) && tb->l3mdev == l3mdev &&
tb->port == port)
goto tb_found;
// If it is not found, it means that no one is using this port, so create a new one
// The newly created instance will be put into bhash, indicating that I am using this port
tb = inet_bind_bucket_create(hinfo->bind_bucket_cachep,
net, head, port, l3mdev);
tb_found:
// If the owners field of tb is not empty, it indicates that someone is using this port
if (!hlist_empty(&tb->owners)) {
// If the port is occupied by others and cannot be shared, an error is returned to the user
if (inet_csk_bind_conflict(sk, tb, true, true))
goto fail_unlock;
}
// Omit a lot of irrelevant code
// At the end of the method, INET is called_ bind_ Hash method
// The contents of the method are described below
if (!inet_csk(sk)->icsk_bind_hash)
inet_bind_hash(sk, tb, port);
}
EXPORT_SYMBOL_GPL(inet_csk_get_port);
Let's look at inet_bind_hash method:
// net/ipv4/inet_hashtables.c
void inet_bind_hash(struct sock *sk, struct inet_bind_bucket *tb,
const unsigned short snum)
{
// Save binding port
inet_sk(sk)->inet_num = snum;
// tb is a value in the bhash obtained or created in the above method
// Its owners field stores all the sock s that use this port
// The following statement means to add the sock to the owner
// In this way, when others get the tb, they can know which sock s are using the port corresponding to the tb
sk_add_bind_node(sk, &tb->owners);
// Store the tb address in the icsk of the sock_ bind_ In the hash field
// In this way, when you want to know the value in the bhash corresponding to the sock in the future (for example, when removing owners)
// You can get it through the following fields
inet_csk(sk)->icsk_bind_hash = tb;
}
Well, the bind method involves TCP_ The hashinfo has been covered here. Let's take a look at the listen method:
// net/ipv4/inet_hashtables.c
int __inet_hash(struct sock *sk, struct sock *osk)
{
// Call stack of this method:
// SYSCALL_DEFINE2(listen)
// __sys_listen
// inet_listen
// inet_csk_listen_start
// inet_hash
// __inet_hash
// Hashinfo is the global instance tcp_hashinfo
struct inet_hashinfo *hashinfo = sk->sk_prot->h.hashinfo;
// According to the local port, find the sock et in listening_ slot in hash
ilb = &hashinfo->listening_hash[inet_sk_listen_hashfn(sk)];
// Add the sock to the linked list corresponding to the slot
if (IS_ENABLED(CONFIG_IPV6) && sk->sk_reuseport &&
sk->sk_family == AF_INET6)
hlist_add_tail_rcu(&sk->sk_node, &ilb->head);
else
hlist_add_head_rcu(&sk->sk_node, &ilb->head);
// Add the sock to TCP with the local port and address as the key_ lhash2 in hashinfo
inet_hash2(hashinfo, sk);
}
EXPORT_SYMBOL(__inet_hash);
The listen method involves TCP_ That's where hashinfo is.
These are the related operations of the server. Let's take a look at the client.
client
The first step for the client is to connect to the server, so let's look at the corresponding connect method:
// net/ipv4/inet_hashtables.c
int __inet_hash_connect(struct inet_timewait_death_row *death_row,
struct sock *sk, u32 port_offset,
int (*check_established)(struct inet_timewait_death_row *,
struct sock *, __u16, struct inet_timewait_sock **))
{
// Call stack of this method
// SYSCALL_DEFINE3(connect)
// __sys_connect
// inet_stream_connect
// __inet_stream_connect
// tcp_v4_connect
// inet_hash_connect
// __inet_hash_connect
// hinfo is global tcp_hashinfo instance
struct inet_hashinfo *hinfo = death_row->hashinfo;
// Generally speaking, we will not actively specify the local port for the connect operation
// But let the operating system help us choose freely
// The following method is used to obtain the range of local ports freely selected by the operating system
// The default range is [32768-60999]
// The current range can be viewed by the following command:
// $ cat /proc/sys/net/ipv4/ip_local_port_range
inet_get_local_port_range(net, &low, &high);
// Check the ports in the range in turn and find the first one that can be used
// First port to detect
port = low + offset;
for (i = 0; i < remaining; i += 2, port += 2) {
// Find the slot in the bhash corresponding to the port
head = &hinfo->bhash[inet_bhashfn(net, port, hinfo->bhash_size)];
// Traverse the linked list pointed to by the slot to see if anyone is already using the port
inet_bind_bucket_for_each(tb, &head->chain) {
if (net_eq(ib_net(tb), net) && tb->l3mdev == l3mdev && tb->port == port) {
// If the port is already used
// Then check whether there are connected socket s in the user
// And the tcp quadruple of the socket is completely consistent with the tcp quadruple of our socket (the tcp quadruple uniquely determines a tcp connection)
// If so, the port is not available
// If not, it is available
if (!check_established(death_row, sk,
port, &tw))
goto ok;
goto next_port;
}
}
// If the port is not used, we will create a new object in bhash to indicate that we want to use it
tb = inet_bind_bucket_create(hinfo->bind_bucket_cachep,
net, head, port, l3mdev);
goto ok;
next_port:
}
ok:
// As mentioned above, this method is mainly to associate the socket with the tb instance
// See above for details
inet_bind_hash(sk, tb, port);
if (sk_unhashed(sk)) {
// This method will be described in detail below
inet_ehash_nolisten(sk, (struct sock *)tw);
}
}
Let's look at the INET mentioned above_ ehash_ Nolisten method:
bool inet_ehash_insert(struct sock *sk, struct sock *osk)
{
// hashinfo is the global tcp_hashinfo instance
struct inet_hashinfo *hashinfo = sk->sk_prot->h.hashinfo;
// Calculate the hash value of the socket according to the local and remote address port information
// And save to sk_ In the hash field for subsequent use
sk->sk_hash = sk_ehashfn(sk);
// Find the corresponding slot in the ehash according to the hash value
head = inet_ehash_bucket(hashinfo, sk->sk_hash);
// Add the socket to the linked list pointed to by the slot
if (ret)
__sk_nulls_add_node_rcu(sk, list);
}
bool inet_ehash_nolisten(struct sock *sk, struct sock *osk)
{
bool ok = inet_ehash_insert(sk, osk);
}
As can be seen from above, TCP_ The function of hashinfo in the connect operation is to select a suitable local port for the connect operation according to the information in bhash and ehash (the use of the port will also be recorded in bhash), and then put the socket into ehash before sending the syn message to the server. In this way, when the kernel receives the server's response message, You can find the corresponding socket.
The connect operation will eventually send a syn message to the server, so let's take a look at how the server handles this syn message when it receives it.
Before that, let's talk about some groundwork.
When the operating system receives any tcp message, it will call the following method to find the socket to which the tcp message belongs, and then conduct subsequent processing according to the current state of the socket and the content of the tcp message:
// net/ipv4/tcp_ipv4.c
int tcp_v4_rcv(struct sk_buff *skb)
{
struct sock *sk;
// This method will start from TCP_ Try to find the corresponding socket in various hashtable s in hashinfo
// Th - > source is the local port of the sender
// Th - > dest is the local port of the receiver
sk = __inet_lookup_skb(&tcp_hashinfo, skb, __tcp_hdrlen(th), th->source,
th->dest, sdif, &refcounted);
}
Look again__ inet_lookup_skb method:
// include/net/inet_hashtables.h
static inline struct sock *__inet_lookup_skb(struct inet_hashinfo *hashinfo,
struct sk_buff *skb,
int doff,
const __be16 sport,
const __be16 dport,
const int sdif,
bool *refcounted)
{
const struct iphdr *iph = ip_hdr(skb);
return __inet_lookup(dev_net(skb_dst(skb)->dev), hashinfo, skb,
doff, iph->saddr, sport,
iph->daddr, dport, inet_iif(skb), sdif,
refcounted);
}
The method called again__ inet_lookup method:
static inline struct sock *__inet_lookup(struct net *net,
struct inet_hashinfo *hashinfo,
struct sk_buff *skb, int doff,
const __be32 saddr, const __be16 sport,
const __be32 daddr, const __be16 dport,
const int dif, const int sdif,
bool *refcounted)
{
u16 hnum = ntohs(dport);
struct sock *sk;
// This method uses local and remote address port information
// From TCP_ Find the corresponding socket in the hash of hashinfo
sk = __inet_lookup_established(net, hashinfo, saddr, sport,
daddr, hnum, dif, sdif);
// If the corresponding socket is not found in ehash, call the following method
// From TCP_ Find the corresponding socket in listen state in lhash2 of hashinfo
return __inet_lookup_listener(net, hashinfo, skb, doff, saddr,
sport, daddr, hnum, dif, sdif);
}
OK, the groundwork is over.
After receiving the syn packet from the client, the server will first find the socket with the corresponding listen state in lhash2 through the above method (the listen method puts the socket into lhash2), and then execute the following logic:
// net/ipv4/tcp_input.c
int tcp_conn_request(struct request_sock_ops *rsk_ops,
const struct tcp_request_sock_ops *af_ops,
struct sock *sk, struct sk_buff *skb)
{
// Call stack of this method
// tcp_v4_rcv
// tcp_v4_do_rcv
// tcp_rcv_state_process
// tcp_v4_conn_request
// The parameter sk of this method is the socket in listen state found above
// After receiving the syn message, the server does not directly create a struct sock
// Instead, create a struct request_ Socket, indicating that the socket is still in the process of tcp three handshakes
struct request_sock *req;
req = inet_reqsk_alloc(rsk_ops, sk, !want_cookie);
if (fastopen_sk) {
} else {
if (!want_cookie)
// This method uses local and remote address port information
// Send request_ Socket to TCP_ ehash in hashinfo
// In this way, subsequent messages can find the request_sock
inet_csk_reqsk_queue_hash_add(sk, req,
tcp_timeout_init((struct sock *)req));
}
}
After processing the syn message, the server will send a synack to the client. After receiving the synack message, the client will send an ack to the server again, and set the socket state of the client to TCP_ESTABLISHED.
Because the logic of the client processing the synack message does not involve tcp_hashinfo, so I won't go into detail here.
Look at the logic of the server after receiving the ack message.
After receiving the ack message, the server will first pass the above described__ inet_lookup_skb method, find the request just created_ Sock, and then execute the following logic:
// net/ipv4/tcp_ipv4.c
struct sock *tcp_v4_syn_recv_sock(const struct sock *sk, struct sk_buff *skb,
struct request_sock *req,
struct dst_entry *dst,
struct request_sock *req_unhash,
bool *own_req)
{
// Call stack of this method
// tcp_v4_rcv
// tcp_check_req
// tcp_v4_syn_recv_sock
// The ack message received from the client indicates that the tcp connection is successfully established
// The following method will be based on request_sock create a real struct sock
struct sock *newsk;
newsk = tcp_create_openreq_child(sk, req, skb);
// This method will put the newly created socket into TCP_ In bhash of hashinfo
// The corresponding port is the port used to listen to the socket
// At this time, the owners field in the value in the bhash corresponding to the port contains the listening socket and the newly created socket
if (__inet_inherit_port(sk, newsk) < 0)
goto put_and_exit;
// Create request above_ When sock, put it into TCP_ In the ehash of hashinfo
// Here, request_ The task of sock has been completed
// So the following method will put the request_ Remove sock from ehash
// And put the newly created socket into ehash
*own_req = inet_ehash_nolisten(newsk, req_to_sk(req_unhash));
}
Up to now, a complete tcp connection has been established. Let's re sort out the whole idea.
First, the socket of the server performs the bind operation and puts itself into tcp_hashinfo's bhash, then performs the listen operation and puts itself into TCP_ lhash2 of hashinfo.
Then, the client executes the connect method, puts the corresponding socket into bhash and ehash, and then sends a syn message to the server.
After receiving the syn, the server first finds the socket in the corresponding listen state from lhash2, and then creates a request according to the socket and syn message_ Sock, put it into ehash, and finally send synack to the client.
After receiving the synack, the client first finds the corresponding socket from the ehash, and then sets its state to TCP_ESTABLISHED. Finally, ack is returned to the server.
After receiving the ack, the server will first find the previously created request from the ehash_ Sock, and then according to the request_sock, create a real sock, and finally put the request_ Remove the sock from ehash and put the newly created sock into bhash and ehash.
At this point, a tcp connection is established successfully.
Then, it is the data transmission process of tcp connection. When the operating system receives the data sent by the other party, first find the corresponding socket from ehash according to the address port and other information in the tcp message header, and then add the data to the acceptance buffer of the socket, so that the user can obtain the data through read and other methods.
This is the logic in tcp after the tcp connection is established successfully_ Use of hashinfo.
Next, let's look at the closing process of tcp_ How hashinfo is used.
Suppose the client first calls the close method and actively closes the connection. See the corresponding code:
// net/ipv4/tcp.c
void tcp_close(struct sock *sk, long timeout)
{
// Call stack of this method
// SYSCALL_DEFINE1(close)
// __close_fd
// filp_close
// fput
// fput_many
// ____fput
// __fput
// sock_close
// __sock_release
// inet_release
// tcp_close
// The following method will set the socket state to TCP_FIN_WAIT1
} else if (tcp_close_state(sk)) {
// Send fin message to the other party
tcp_send_fin(sk);
}
}
The processing logic of the server upon receiving the fin packet is:
// net/ipv4/tcp_input.c
void tcp_fin(struct sock *sk)
{
// Call stack of this method
// tcp_v4_rcv
// tcp_v4_do_rcv
// tcp_rcv_established
// tcp_data_queue
// tcp_fin
switch (sk->sk_state) {
case TCP_ESTABLISHED:
tcp_set_state(sk, TCP_CLOSE_WAIT);
}
This method sets the state of the server socket to TCP_CLOSE_WAIT, and then return ack to the client.
Processing logic after the client receives the ack:
// net/ipv4/tcp_input.c
int tcp_rcv_state_process(struct sock *sk, struct sk_buff *skb)
{
// Call stack of this method
// tcp_v4_rcv
// tcp_v4_do_rcv
// tcp_rcv_state_process
switch (sk->sk_state) {
case TCP_FIN_WAIT1: {
tcp_set_state(sk, TCP_FIN_WAIT2);
break;
}
}
After receiving the ack, this method sets the state of the socket corresponding to the client to TCP_FIN_WAIT2.
At this time, it is assumed that the application layer of the server also calls the close method of socket, which will execute the following logic:
// net/ipv4/tcp.c
void tcp_close(struct sock *sk, long timeout)
{
// Call stack of this method
// SYSCALL_DEFINE1(close)
// __close_fd
// filp_close
// fput
// fput_many
// ____fput
// __fput
// sock_close
// __sock_release
// inet_release
// tcp_close
// The following method will set the socket state to TCP_LAST_ACK
} else if (tcp_close_state(sk)) {
// Send fin message to the other party
tcp_send_fin(sk);
}
}
Processing logic of fin message received by the client:
// net/ipv4/tcp_input.c
void tcp_fin(struct sock *sk)
{
// This method calls the stack
// tcp_v4_rcv
// tcp_v4_do_rcv
// tcp_rcv_state_process
// tcp_data_queue
// tcp_fin
switch (sk->sk_state) {
case TCP_FIN_WAIT2:
// Send ack to the other party
tcp_send_ack(sk);
// Enter time wait logic processing
tcp_time_wait(sk, TCP_TIME_WAIT, 0);
}
}
Keep looking_ time_ Wait method:
// net/ipv4/tcp_minisocks.c
void tcp_time_wait(struct sock *sk, int state, int timeo)
{
// It is similar to the request created by the server during the three handshakes_ sock
// An INET is also created based on the current sock_ timewait_ sock
// Corresponds to the socket in the time wait state
struct inet_timewait_sock *tw;
tw = inet_twsk_alloc(sk, tcp_death_row, state);
if (tw) {
// Copy all necessary data from the sock to inet_timewait_sock
// Perform time wait timing, and INET will be called after timeout_ twsk_ Kill method
// Set INET_ timewait_ Remove the sock from ehash and bhash
inet_twsk_schedule(tw, timeo);
// This method removes the sock from ehash
// Set INET_ timewait_ The sock is added to ehash and bhash
inet_twsk_hashdance(tw, sk, &tcp_hashinfo);
}
// This method removes the sock from the bhash and destroys it
tcp_done(sk);
}
OK, the logic of the client is complete. Let's look at the logic of the server.
When the server is in TCP_ LAST_ In ack status, after receiving the ACK message from the client, perform the following processing:
// net/ipv4/tcp_input.c
int tcp_rcv_state_process(struct sock *sk, struct sk_buff *skb)
{
// This method calls the stack
// tcp_v4_rcv
// tcp_v4_do_rcv
// tcp_rcv_state_process
switch (sk->sk_state) {
case TCP_LAST_ACK:
if (tp->snd_una == tp->write_seq) {
// This method and its underlying methods remove the sock from ebash and bhash
tcp_done(sk);
}
}
}
At this point, a tcp connection is completely closed, and the front and back-end socket s have been removed from tcp_hashinfo is removed from ehash and bhash.
Now the system returns to the state before the tcp connection, that is, only the socket of one server is in the listen state, and the socket is stored in tcp at the same time_ bhash, lhash 2 and listening of hashinfo_ Hash.
Let's look at the corresponding TCP when the socket in the listen state is closed_ Hashinfo processing:
// net/ipv4/tcp.c
void tcp_set_state(struct sock *sk, int state)
{
// Call stack of this method
// SYSCALL_DEFINE1(close)
// __close_fd
// filp_close
// fput
// fput_many
// ____fput
// __fput
// sock_close
// __sock_release
// inet_release
// tcp_close
// tcp_set_state
switch (state) {
case TCP_CLOSE:
// The following method will remove the socket from lhash2 and listening_ Remove from hash
sk->sk_prot->unhash(sk);
// The following method will remove the socket from the bhash
if (inet_csk(sk)->icsk_bind_hash &&
!(sk->sk_userlocks & SOCK_BINDPORT_LOCK))
inet_put_port(sk);
}
}
So far, all socket s have been closed and tcp_hashinfo returns to its original empty state.
A lot of space has been devoted to the various operations of tcp_ How hashinfo is used. In fact, looking back, tcp_ The function of hashinfo is very simple. Its main purpose is to assist the operating system to find the corresponding socket in various cases.
For example, when the syn message comes, TCP is used to find the socket in the corresponding listen state_ lhash2 in hashinfo.
For example, when all subsequent messages after the syn message come, TCP is used to find the corresponding message processing socket_ ehash in hashinfo.
For example, when binding a port or selecting a port, you need to use bhash to query whether the port is occupied.
OK, that's all. That's the end of the article.
Generally speaking, this article is based on tcp_hashinfo is a global instance. Let's take a look at how the operating system manages tcp connections.