Epoll is an event driven model, which is one of the important reasons why nginx can efficiently handle client requests. From the flow point of view, the use of epoll model is mainly divided into three steps: the creation of epoll handle, listening to the addition of file descriptors and the triggering of waiting events. This paper will introduce how nginx can efficiently handle client requests based on these three steps.
1. Introduction to epoll model
Before introducing the implementation principle of nginx, we need to introduce the basic usage of epoll model. Epoll can be used in three ways:
// Create epoll handle int epoll_create(int size);
// Add a file descriptor to the epoll handle for listening int epoll_ctl(int epfd,int op,int fd,struct epoll_event* event);
// Wait for the occurrence of the corresponding event on the file descriptor to be monitored int epoll_wait(int epfd,struct epoll_event* events,int maxevents,int timeout);
First of all, we will call the epoll create() method to create a handle to an epoll instance. The handle here can be understood as an eventpoll structure instance. In this structure, there is a red black tree and a queue. The red black tree mainly stores the file descriptors to be monitored, while the queue contains the specified events in the monitored file descriptors These events will be added to the queue, as shown in the figure below:
Generally speaking, there is only one epoll handle in the whole running cycle of the program. For example, each worker process of nginx maintains only one epoll handle. After the handle is created, every port our program listens to is essentially a file descriptor, on which the Accept event can occur, that is, the client request is received. Therefore, initially, we will add the file descriptor corresponding to the port to be listened to to the epoll handle through the epoll < ctl() method. After adding successfully, each listening file descriptor corresponds to a node in the red black tree of eventpoll. In addition, after the file descriptor is added by calling epoll < ctl(), it will be associated with the corresponding device (network card). When an event occurs in the device driver, the callback method of the current file descriptor, EP < poll < callback(), will be called back to generate an event, and the event will be added to the event queue of eventpoll. Finally, when we call the epoll UU wait() method, we will get the corresponding event from the epoll handle. In essence, we will check whether the event queue of eventpoll is empty. If there is an event, we will return it. Otherwise, we will wait for the event. In addition, for the use of epoll, the events obtained here are generally Accept events. When processing this event, the handle of the client's connection will be obtained. This handle is essentially a file descriptor. At this time, we will continue to add it to the current epoll handle through the epoll uuctl() method to continue to wait for its data to be read through the epoll uuwait() method Fetch and write events.
We can see here that in the process of using epoll, there will be two types of file descriptors: one is the file descriptor corresponding to the port we are listening to, which we usually listen to its Accept event to wait for the client connection, and the other is the file descriptor corresponding to each client connection, and here we have one Generally, it listens for its read-write events to receive and send data to the client.
2. epoll implementation in nginx
In the previous article, we explained how nginx initializes the event driven framework. One of the core modules of the event framework is defined as follows:
ngx_module_t ngx_event_core_module = { NGX_MODULE_V1, &ngx_event_core_module_ctx, /* module context */ ngx_event_core_commands, /* module directives */ NGX_EVENT_MODULE, /* module type */ NULL, /* init master */ // This method is mainly invoked during the startup of master process, and is used for initialization time module. ngx_event_module_init, /* init module */ // This method is called after every worker process starts. ngx_event_process_init, /* init process */ NULL, /* init thread */ NULL, /* exit thread */ NULL, /* exit process */ NULL, /* exit master */ NGX_MODULE_V1_PADDING };
Here we need to pay special attention to the NGX ﹣ event ﹣ process ﹣ init() method. We have said that this method is used to initialize each worker when it is created, which involves two very important calls: a. initialize the corresponding event model; b. listen to each port specified in the configuration file. Here are the main codes of these two steps:
static ngx_int_t ngx_event_process_init(ngx_cycle_t *cycle) { // Omit part of the code // In the events {} configuration block of nginx.conf configuration file, you need to use the use directive to specify the currently used event model, // At this time, the index number of the event model used will be stored in ECF - > use, and the following code will get the current value in this way // The module that corresponds to the event model is then called the actions.init() method of the module to initialize the event model. for (m = 0; cycle->modules[m]; m++) { if (cycle->modules[m]->type != NGX_EVENT_MODULE) { continue; } // ECF - > use stores the module serial number of the selected event model. Here is to find the module if (cycle->modules[m]->ctx_index != ecf->use) { continue; } // Module is the module corresponding to the selected event model module = cycle->modules[m]->ctx; // Call the initialization method of the specified event model if (module->actions.init(cycle, ngx_timer_resolution) != NGX_OK) { exit(2); } break; } // Omit part of code ls = cycle->listening.elts; for (i = 0; i < cycle->listening.nelts; i++) { #if (NGX_HAVE_REUSEPORT) if (ls[i].reuseport && ls[i].worker != ngx_worker) { continue; } #endif // This is to bind an NGX ﹣ connection ﹣ T structure for each port being monitored c = ngx_get_connection(ls[i].fd, cycle->log); if (c == NULL) { return NGX_ERROR; } rev = c->read; // Sock "stream indicates TCP, which is generally TCP, that is, after receiving the accept event from the client, // It will call the NGX? Event? Accept() method to handle the event rev->handler = (c->type == SOCK_STREAM) ? ngx_event_accept : ngx_event_recvmsg; if ((ngx_event_flags & NGX_USE_EPOLL_EVENT) && ccf->worker_processes > 1) { if (ngx_add_event(rev, NGX_READ_EVENT, NGX_EXCLUSIVE_EVENT) == NGX_ERROR) { return NGX_ERROR; } continue; } } return NGX_OK; }
The code here mainly completes the following parts:
-
First, find the event model module used, then call the init() method to initialize the model. This method mainly does two things. One is to create a epoll handle through the epoll_create() method, which is a foundation of the current worker process operation, and the other is to assign the global variable ngx_event_actions, that is:
// Here, the epoll related event operation method is assigned to NGX ﹣ event ﹣ actions, // That is to say, epoll related methods will be used for subsequent events ngx_event_actions = ngx_epoll_module_ctx.actions;The call of this assignment is very important. After the assignment, several method macros defined by nginx are all the methods specified in the epoll module. Here are the definition of several macros:
#define ngx_process_events ngx_event_actions.process_events #define ngx_done_events ngx_event_actions.done #define ngx_add_event ngx_event_actions.add #define ngx_del_event ngx_event_actions.del #define ngx_add_conn ngx_event_actions.add_conn #define ngx_del_conn ngx_event_actions.del_connHere, the structure of NGX ﹣ epoll ﹣ module ﹣ ctx.actions is defined as follows:
{ // Corresponds to the add method in NGX event actions ngx_epoll_add_event, /* add an event */ // Corresponding to del method in NGX event actions ngx_epoll_del_event, /* delete an event */ // It corresponds to the enable method in NGX event actions and is consistent with the add method ngx_epoll_add_event, /* enable an event */ // It corresponds to the disable method in NGX event actions and is consistent with del method ngx_epoll_del_event, /* disable an event */ // Corresponds to the add conn method in NGX event actions ngx_epoll_add_connection, /* add an connection */ // Corresponding to del conn method in NGX event actions ngx_epoll_del_connection, /* delete an connection */ #if (NGX_HAVE_EVENTFD) ngx_epoll_notify, /* trigger a notify */ #else NULL, /* trigger a notify */ #endif // Corresponds to the process event method in NGX event actions ngx_epoll_process_events, /* process the events */ // Corresponds to the init method in NGX? Event? Actions? T ngx_epoll_init, /* init the events */ // Corresponds to the done method in NGX event actions ngx_epoll_done, /* done the events */ }From this, we can see the excellent design method of nginx. Through the event model we selected, we can dynamically specify the implemented sub modules for the macro such as ngx_add_event().
-
The second main task of the above method is to traverse all listening ports, obtain their descriptors, and then add them to the epoll handle through the ngx_add_event() method to listen for their client connection events. From here, we can feel that it is more ingenious, because in the previous step, we just initialized the epoll module, and set the implementation method of the ngx_add_event() macro, and here we use the method set here, which is essentially to add the currently monitored socket descriptor to the epoll handle through the epoll_ctl() method;
-
Finally, when the above method traverses all listening ports, the callback method added for the accept event of each connection is NGX event accept(). Through the introduction of the usage of the epoll model, we can roughly understand that the main function of the NGX event accept() method here is to pass the handle of the client connection currently accepted to the epoll ctl() method Add to the current epoll handle to continue listening for its read and write events;
Let's first look at how the module - > actions.init (cycle, NGX ﹣ timer ﹣ resolution) method call initializes the epoll module. Because it's an epoll module, the init() method here points to the NGX? Epoll? Init() method. The source code of the method is as follows:
static ngx_int_t ngx_epoll_init(ngx_cycle_t *cycle, ngx_msec_t timer) { ngx_epoll_conf_t *epcf; // Get the structure of NGX ﹣ epoll ﹣ conf ﹣ t epcf = ngx_event_get_conf(cycle->conf_ctx, ngx_epoll_module); if (ep == -1) { // Create the eventpoll structure and return the created file descriptor ep = epoll_create(cycle->connection_n / 2); // ep==-1 means creation failed if (ep == -1) { ngx_log_error(NGX_LOG_EMERG, cycle->log, ngx_errno, "epoll_create() failed"); return NGX_ERROR; } } // If nevents is less than epcf - > events, the length of event list array is not enough, so you need to re apply memory space if (nevents < epcf->events) { if (event_list) { ngx_free(event_list); } // Reapply memory space for event list event_list = ngx_alloc(sizeof(struct epoll_event) * epcf->events, cycle->log); if (event_list == NULL) { return NGX_ERROR; } } // Update nevents to the size specified in the configuration file nevents = epcf->events; ngx_io = ngx_os_io; // Here, the epoll related event operation method is assigned to NGX ﹣ event ﹣ actions, that is to say, there will be subsequent events // Will use epoll related methods ngx_event_actions = ngx_epoll_module_ctx.actions; // Here, NGX? Use? Clear? Event refers to use ET mode to use epoll, and use ET mode by default, // And NGX ﹣ use ﹣ level ﹣ event means use LE mode to use epoll #if (NGX_HAVE_CLEAR_EVENT) ngx_event_flags = NGX_USE_CLEAR_EVENT #else ngx_event_flags = NGX_USE_LEVEL_EVENT #endif // NGX ﹣ use ﹣ gray ﹣ event indicates that each pull event is the one that tries to pull the most | NGX_USE_GREEDY_EVENT | NGX_USE_EPOLL_EVENT; return NGX_OK; }
As you can see, the main functions of NGX ﹣ epoll ﹣ init() method here are two: A. create an epoll handle through the epoll ﹣ create() method; b. set the implementation of the method pointed to by the NGX ﹣ event ﹣ actions property, so as to determine the implementation method of NGX ﹣ add ﹣ event(), etc. Let's take a look at how NGX ﹣ add ﹣ event() adds the file descriptors to be listened to to the epoll handle:
static ngx_int_t ngx_epoll_add_event(ngx_event_t *ev, ngx_int_t event, ngx_uint_t flags) { int op; uint32_t events, prev; ngx_event_t *e; ngx_connection_t *c; struct epoll_event ee; // Ev - > data stores the current corresponding NGX ﹣ connection ﹣ T during use. If it is free ﹣ connection, // The next node's pointer is stored c = ev->data; // Event type events = (uint32_t) event; // If it's a read event if (event == NGX_READ_EVENT) { e = c->write; prev = EPOLLOUT; #if (NGX_READ_EVENT != EPOLLIN | EPOLLRDHUP) events = EPOLLIN | EPOLLRDHUP; // Set read event type #endif } else { e = c->read; prev = EPOLLIN | EPOLLRDHUP; #if (NGX_WRITE_EVENT != EPOLLOUT) events = EPOLLOUT; // Set write event type #endif } // Determine whether it is an active event according to the active flag bit to decide whether to modify or add the event if (e->active) { op = EPOLL_CTL_MOD; // Type is modify event events |= prev; } else { op = EPOLL_CTL_ADD; // Type is add event } #if (NGX_HAVE_EPOLLEXCLUSIVE && NGX_HAVE_EPOLLRDHUP) if (flags & NGX_EXCLUSIVE_EVENT) { events &= ~EPOLLRDHUP; } #endif // Add the event specified by the flags parameter to the listening list ee.events = events | (uint32_t) flags; // This is to assign the last bit of the connection pointer to ev - > instance, and then assign it to the ptr attribute of the event, so as to detect whether the event expires ee.data.ptr = (void *) ((uintptr_t) c | ev->instance); ngx_log_debug3(NGX_LOG_DEBUG_EVENT, ev->log, 0, "epoll add event: fd:%d op:%d ev:%08XD", c->fd, op, ee.events); // Add an event to the epoll handle if (epoll_ctl(ep, op, c->fd, &ee) == -1) { ngx_log_error(NGX_LOG_ALERT, ev->log, ngx_errno, "epoll_ctl(%d, %d) failed", op, c->fd); return NGX_ERROR; } // Mark event as active ev->active = 1; #if 0 ev->oneshot = (flags & NGX_ONESHOT_EVENT) ? 1 : 0; #endif return NGX_OK; }
Here, the NGX ﹣ add ﹣ event() method is relatively simple in nature. It is to convert the current NGX ﹣ event ﹣ t to an epoll ﹣ event structure, and set the event type to be monitored in the structure, and then add the current epoll ﹣ event to the epoll handle through the epoll ﹣ ctl() method.
In the previous NGX ﹣ event ﹣ process ﹣ init() method, nginx adds descriptors of each listening port to the epoll handle through the NGX ﹣ add ﹣ event() method, and then starts to listen for accept connection events on these descriptors. If there is a client connection request, it will call back the NGX ﹣ event ﹣ accept() method to process the request. Let's see that the method is How to handle the connection request of the client:
/** * This method is called to handle the accept event when it arrives at the client */ void ngx_event_accept(ngx_event_t *ev) { socklen_t socklen; ngx_err_t err; ngx_log_t *log; ngx_uint_t level; ngx_socket_t s; ngx_event_t *rev, *wev; ngx_sockaddr_t sa; ngx_listening_t *ls; ngx_connection_t *c, *lc; ngx_event_conf_t *ecf; #if (NGX_HAVE_ACCEPT4) static ngx_uint_t use_accept4 = 1; #endif if (ev->timedout) { // If the current event times out, continue to add it to the epoll handle to listen for the accept event if (ngx_enable_accept_events((ngx_cycle_t *) ngx_cycle) != NGX_OK) { return; } ev->timedout = 0; } // Get and parse event core configuration structure ecf = ngx_event_get_conf(ngx_cycle->conf_ctx, ngx_event_core_module); if (!(ngx_event_flags & NGX_USE_KQUEUE_EVENT)) { ev->available = ecf->multi_accept; } lc = ev->data; ls = lc->listening; ev->ready = 0; do { socklen = sizeof(ngx_sockaddr_t); #if (NGX_HAVE_ACCEPT4) if (use_accept4) { s = accept4(lc->fd, &sa.sockaddr, &socklen, SOCK_NONBLOCK); } else { s = accept(lc->fd, &sa.sockaddr, &socklen); } #else // Here, LC - > FD points to the file handle listening, calls accept() to get the connection of the client, and stores it in sa.sockaddr s = accept(lc->fd, &sa.sockaddr, &socklen); #endif // Check whether the number of connections obtained by the current process exceeds 7 / 8 of the maximum number of available connections. If yes, no more connections will be received ngx_accept_disabled = ngx_cycle->connection_n / 8 - ngx_cycle->free_connection_n; // Get new connection c = ngx_get_connection(s, ev->log); // Direct return if get connection failed if (c == NULL) { if (ngx_close_socket(s) == -1) { ngx_log_error(NGX_LOG_ALERT, ev->log, ngx_socket_errno, ngx_close_socket_n " failed"); } return; } // Mark current as TCP connection c->type = SOCK_STREAM; // Create a connection pool for the current connection c->pool = ngx_create_pool(ls->pool_size, ev->log); if (c->pool == NULL) { ngx_close_accepted_connection(c); return; } // Update the length of socklen if (socklen > (socklen_t) sizeof(ngx_sockaddr_t)) { socklen = sizeof(ngx_sockaddr_t); } // Apply memory space for sockaddr, and copy the client connection address to C - > sockaddr c->sockaddr = ngx_palloc(c->pool, socklen); if (c->sockaddr == NULL) { ngx_close_accepted_connection(c); return; } ngx_memcpy(c->sockaddr, &sa, socklen); // Apply for memory space of NGX log structure log = ngx_palloc(c->pool, sizeof(ngx_log_t)); if (log == NULL) { ngx_close_accepted_connection(c); return; } /* set a blocking mode for iocp and non-blocking mode for others */ if (ngx_inherited_nonblocking) { if (ngx_event_flags & NGX_USE_IOCP_EVENT) { // Set the connection to blocking mode if (ngx_blocking(s) == -1) { ngx_log_error(NGX_LOG_ALERT, ev->log, ngx_socket_errno, ngx_blocking_n " failed"); ngx_close_accepted_connection(c); return; } } } else { if (!(ngx_event_flags & NGX_USE_IOCP_EVENT)) { // Set connection to non blocking mode if (ngx_nonblocking(s) == -1) { ngx_log_error(NGX_LOG_ALERT, ev->log, ngx_socket_errno, ngx_nonblocking_n " failed"); ngx_close_accepted_connection(c); return; } } } *log = ls->log; // Set basic properties of the connection c->recv = ngx_recv; c->send = ngx_send; c->recv_chain = ngx_recv_chain; c->send_chain = ngx_send_chain; c->log = log; c->pool->log = log; c->socklen = socklen; c->listening = ls; c->local_sockaddr = ls->sockaddr; c->local_socklen = ls->socklen; #if (NGX_HAVE_UNIX_DOMAIN) if (c->sockaddr->sa_family == AF_UNIX) { c->tcp_nopush = NGX_TCP_NOPUSH_DISABLED; c->tcp_nodelay = NGX_TCP_NODELAY_DISABLED; #if (NGX_SOLARIS) /* Solaris's sendfilev() supports AF_NCA, AF_INET, and AF_INET6 */ c->sendfile = 0; #endif } #endif rev = c->read; wev = c->write; wev->ready = 1; if (ngx_event_flags & NGX_USE_IOCP_EVENT) { rev->ready = 1; } if (ev->deferred_accept) { rev->ready = 1; #if (NGX_HAVE_KQUEUE || NGX_HAVE_EPOLLRDHUP) rev->available = 1; #endif } rev->log = log; wev->log = log; // Update connection usage c->number = ngx_atomic_fetch_add(ngx_connection_counter, 1); // Update network address to a string address if (ls->addr_ntop) { c->addr_text.data = ngx_pnalloc(c->pool, ls->addr_text_max_len); if (c->addr_text.data == NULL) { ngx_close_accepted_connection(c); return; } c->addr_text.len = ngx_sock_ntop(c->sockaddr, c->socklen, c->addr_text.data, ls->addr_text_max_len, 0); if (c->addr_text.len == 0) { ngx_close_accepted_connection(c); return; } } #if (NGX_DEBUG) { ngx_str_t addr; u_char text[NGX_SOCKADDR_STRLEN]; ngx_debug_accepted_connection(ecf, c); if (log->log_level & NGX_LOG_DEBUG_EVENT) { addr.data = text; addr.len = ngx_sock_ntop(c->sockaddr, c->socklen, text, NGX_SOCKADDR_STRLEN, 1); ngx_log_debug3(NGX_LOG_DEBUG_EVENT, log, 0, "*%uA accept: %V fd:%d", c->number, &addr, s); } } #endif // Add the current connection to the epoll handle for monitoring if (ngx_add_conn && (ngx_event_flags & NGX_USE_EPOLL_EVENT) == 0) { if (ngx_add_conn(c) == NGX_ERROR) { ngx_close_accepted_connection(c); return; } } log->data = NULL; log->handler = NULL; // Callback method after establishing a new connection ls->handler(c); if (ngx_event_flags & NGX_USE_KQUEUE_EVENT) { ev->available--; } } while (ev->available); }
Here, the establishment process of client connection can be divided into the following steps:
- First, call the accept() method to obtain the connection established by the current client, and save its address information to the structure sa;
- Next, we call the NGX get connection() method to obtain an NGX connection T structure to correspond to the currently acquired client connection, and initialize the properties of the structure;
- Call the ngx_add_conn() method to add the current method to the epoll handle. In essence, the adding process here is to add the file descriptor of the current client's connection to the epoll handle through the epoll_ctl() method to listen for its read and write events;
In this way, we explained the process from the creation of epoll handle to the listening of the specified port, then processing the client connection, and adding the file descriptor corresponding to the client connection to the epoll handle to listen for the read and write events. Let's continue to see how nginx waits for events on these listening handles, which is the driver of the entire event framework. For the handling of events by the worker process, it is mainly in the NGX ﹣ process ﹣ events ﹣ and ﹣ timers() method. The source code of this method is as follows:
void ngx_process_events_and_timers(ngx_cycle_t *cycle) { // Attempt to acquire shared lock if (ngx_trylock_accept_mutex(cycle) == NGX_ERROR) { return; } // Here we start to process the events. For the kqueue model, it points to the ngx_kqueue_process_uevents() method, // For the epoll model, it points to the NGX ﹣ epoll ﹣ process ﹣ events() method // The main function of this method is to get the event list in the corresponding event model, and then add the event to NGX ﹣ posted ﹣ accept ﹣ events // In the queue or NGX "posted" events queue (void) ngx_process_events(cycle, timer, flags); // Here, we start to process the accept event, which is handed over to the NGX event accept() method of NGX event accept. C; ngx_event_process_posted(cycle, &ngx_posted_accept_events); // Start to release lock if (ngx_accept_mutex_held) { ngx_shmtx_unlock(&ngx_accept_mutex); } // If you do not need to process in the event queue, process the event directly // For the event processing, if it is an accept event, it will be handled by the NGX event accept() method of NGX event accept. C; // If it is a read event, it will be handled by the NGX HTTP wait request handler() method of NGX HTTP request. C; // For the completed event, it will be finally handed over to the NGX HTTP keepalive handler() method of NGX HTTP request. C. // We start to process other events except the accept event ngx_event_process_posted(cycle, &ngx_posted_events); }
In this method, we omit most of the code, leaving only the main process. In short, it mainly realizes the following steps:
- Obtain the shared lock to obtain the access to the client connection;
- Call the NGX? Process? Events() method to listen for and process the events of the file descriptors in the epoll handle. As we mentioned earlier, when nginx calls the init() method of the epoll module, it initializes the value of the NGX ﹣ event ﹣ actions attribute and points it to the method implemented by the epoll module. Here, it includes the method corresponding to the macro of the NGX ﹣ process ﹣ events() method, that is, the NGX ﹣ epoll ﹣ process ﹣ events() method, which is essentially Call the epoll UU wait() method to wait for the event listening on the epoll handle;
- Handle the events in the NGX ﹣ posted ﹣ accept ﹣ events queue. These events are actually the events mentioned above for clients to establish a connection. After getting the events in the NGX ﹣ EPL ﹣ process ﹣ events() method, it will determine whether it is an accept event or a read-write event. If it is an accept event, it will be added to the NGX ﹣ posted ﹣ accept ﹣ events queue. If it is a read-write event, it will It is added to the NGX [posted] events queue;
- Release the shared lock so that other worker processes can acquire the lock and receive the client connection;
- Handle the events in the NGX ﹣ posted ﹣ events queue, that is, the read-write events of the client connection. From here, we can see a reason for the high performance of nginx. It puts the accept event and the read-write event into two different queues. The accept event must be handled inside the lock, while the read-write event can be asynchronous with the accept event, which improves the ability of nginx to handle client requests.
Let's take a look at how the NGX? Epoll? Process? Events() method handles the events in the epoll handle:
static ngx_int_t ngx_epoll_process_events(ngx_cycle_t *cycle, ngx_msec_t timer, ngx_uint_t flags) { int events; uint32_t revents; ngx_int_t instance, i; ngx_uint_t level; ngx_err_t err; ngx_event_t *rev, *wev; ngx_queue_t *queue; ngx_connection_t *c; /* NGX_TIMER_INFINITE == INFTIM */ ngx_log_debug1(NGX_LOG_DEBUG_EVENT, cycle->log, 0, "epoll timer: %M", timer); // Get the events through the epoll_wait() method. The obtained events will be stored in the event_list, and the number of events will be returned events = epoll_wait(ep, event_list, (int) nevents, timer); err = (events == -1) ? ngx_errno : 0; // Here, the NGX event timer alarm is triggered by a timer task, which will be set to 1 in the timer, // So as to realize the purpose of regularly updating the time of nginx cache if (flags & NGX_UPDATE_TIME || ngx_event_timer_alarm) { ngx_time_update(); } if (err) { if (err == NGX_EINTR) { if (ngx_event_timer_alarm) { ngx_event_timer_alarm = 0; return NGX_OK; } level = NGX_LOG_INFO; } else { level = NGX_LOG_ALERT; } ngx_log_error(level, cycle->log, err, "epoll_wait() failed"); return NGX_ERROR; } // The number of events obtained is 0 if (events == 0) { // If the current time type is not NGX ﹣ timer ﹣ infinite, it means that the get event has timed out, then it directly returns if (timer != NGX_TIMER_INFINITE) { return NGX_OK; } // This indicates that the time type is NGX ﹣ timer ﹣ infinite, but 0 events are returned, indicating that there is a problem with the epoll ﹣ wait() call ngx_log_error(NGX_LOG_ALERT, cycle->log, 0, "epoll_wait() returned no events without timeout"); return NGX_ERROR; } // Traverse events for (i = 0; i < events; i++) { // The connection object corresponding to the current event is stored in the data.ptr of each event c = event_list[i].data.ptr; // Get the value of the instance stored in the event instance = (uintptr_t) c & 1; // Get connection pointer address value c = (ngx_connection_t *) ((uintptr_t) c & (uintptr_t) ~1); // Get read event structure rev = c->read; // If the file descriptor of the current connection is - 1, the instance obtained is not equal to the instance of the current event, // If the connection has expired, the event will not be processed if (c->fd == -1 || rev->instance != instance) { /* * the stale event from a file descriptor * that was just closed in this iteration */ ngx_log_debug1(NGX_LOG_DEBUG_EVENT, cycle->log, 0, "epoll: stale event %p", c); continue; } // Get the type of current event listening revents = event_list[i].events; ngx_log_debug3(NGX_LOG_DEBUG_EVENT, cycle->log, 0, "epoll: fd:%d ev:%04XD d:%p", c->fd, revents, event_list[i].data.ptr); // If an error occurs in the event, print the corresponding log if (revents & (EPOLLERR | EPOLLHUP)) { ngx_log_debug2(NGX_LOG_DEBUG_EVENT, cycle->log, 0, "epoll_wait() error on fd:%d ev:%04XD", c->fd, revents); /* * if the error events were returned, add EPOLLIN and EPOLLOUT * to handle the events at least in one active handler */ revents |= EPOLLIN | EPOLLOUT; } #if 0 if (revents & ~(EPOLLIN|EPOLLOUT|EPOLLERR|EPOLLHUP)) { ngx_log_error(NGX_LOG_ALERT, cycle->log, 0, "strange epoll_wait() events fd:%d ev:%04XD", c->fd, revents); } #endif // If the current is a read event and the event is active if ((revents & EPOLLIN) && rev->active) { #if (NGX_HAVE_EPOLLRDHUP) if (revents & EPOLLRDHUP) { rev->pending_eof = 1; } rev->available = 1; #endif // Mark event as ready rev->ready = 1; // By default, the NGX? Post? Events switch is turned on if (flags & NGX_POST_EVENTS) { // If it is currently an accept event, add it to the NGX [posted] accept [events queue, // If it is a read-write event, add it to the NGX ﹣ posted ﹣ events queue queue = rev->accept ? &ngx_posted_accept_events : &ngx_posted_events; ngx_post_event(rev, queue); } else { // If you do not need to separate the accept and read-write events, handle the event directly rev->handler(rev); } } // Get write event structure wev = c->write; if ((revents & EPOLLOUT) && wev->active) { // If the file descriptor of the current connection is - 1, the instance obtained is not equal to the instance of the current event, // If the connection has expired, the event will not be processed if (c->fd == -1 || wev->instance != instance) { /* * the stale event from a file descriptor * that was just closed in this iteration */ ngx_log_debug1(NGX_LOG_DEBUG_EVENT, cycle->log, 0, "epoll: stale event %p", c); continue; } // Mark the current event as ready wev->ready = 1; #if (NGX_THREADS) wev->complete = 1; #endif // Because it is a write event and needs to be marked as NGX ﹣ post ﹣ events status, // Therefore, it will be added directly to the NGX ﹣ posted ﹣ events queue, otherwise the event will be processed directly if (flags & NGX_POST_EVENTS) { ngx_post_event(wev, &ngx_posted_events); } else { wev->handler(wev); } } } return NGX_OK; }
Here, the method of NGX ﹣ epoll ﹣ process ﹣ events() first calls the method of epoll ﹣ wait() to get the event of the listening handle, and then traverses the event. According to the type of event, if it is an accept event, it is added to the queue of NGX ﹣ posted ﹣ accept ﹣ events, if it is a read-write event, it is added to the queue of NGX ﹣ posted ﹣ events, and the processing of the event in the queue, In the NGX? Process? Events? And? Timers () method described above.
4. summary
This paper first explains the principle of epoll model, and then explains how nginx implements event driven mode based on epoll model from the source level.