Problem scenarios

We use Suricata for traffic analysis. Suricata is deployed on a 48-core physical machine with a multi-network card. Due to business needs, suricata's network card for monitoring traffic has been upgraded from 5 to 6, enp176s0f1 has been added, and several similar errors have been found in suricata's log:

<Error> - [ERRCODE: SC_ERR_PF_RING_OPEN(34)] - Failed to open enp176s0f1: pfring_open error. Check if enp176s0f1 exists and pf_ring module is loaded.

Analysis

From the log, it is due to the failure of pfring to listen on the new network card, either the problem of pfring or the problem of the new network card.So, in order to narrow down your goals, you need to first address who is the problem.

Investigation

1. Location is the method of pfring problem:
The suricata monitoring configuration retains the new network card enp176s0f1, changes the six network cards suricata monitors to less than or equal to five, and suricata is functioning normally.
2. Check the system log for any exceptions and find a pfring prompt

# dmesg -T
[Tue May 26 11:29:09 2020] [PF_RING] Exceeded the maximum number of list items

3. Use powerful search engines to start searching for errors, discoveries, and MAX_NUM_ring_The SOCKETS variable is related, so download the source code and start looking for the variable and the code associated with the error.It was found that more than 256 socket s listening on rings caused an error.

# grep -n MAX_NUM_RING_SOCKETS * -r
kernel/linux/pf_ring.h:34:#define MAX_NUM_RING_SOCKETS          256
kernel/linux/pf_ring.h:344:#define MAX_NUM_LIST_ELEMENTS MAX_NUM_RING_SOCKETS /* sizeof(bits_set) [see below] */
kernel/linux/pf_ring.h:998:#define MAX_NUM_ZC_BOUND_SOCKETS MAX_NUM_RING_SOCKETS

# grep  -n 'Exceeded the maximum number of list items' * -r
kernel/pf_ring.c:606:    printk("[PF_RING] Exceeded the maximum number of list items\n");

# vim kernel/pf_ring.c +606
 605   if(l->num_elements >= MAX_NUM_LIST_ELEMENTS) {
 606     printk("[PF_RING] Exceeded the maximum number of list items\n");
 607     return(-1); /* Too many */
 608   }

4, we Suricata runs in workers mode, so each thread handles all the logic (including using pfring packages).Combined with suricata's pfring monitoring network card configuration, we find that the threads per network card is auto, and the default auto is to use the core number of cpu as the number of threads.Therefore, the number of threads started by six network cards Suricata is 6*48=288, which exceeds the limit of 256, so the last network card pfring open failed.

pfring:
  - interface: enp175s0f0
    threads: auto
    cluster-id: 81
    cluster-type: cluster_flow
  - interface: enp175s0f1
    threads: auto
    cluster-id: 82
    cluster-type: cluster_flow
  - interface: enp24s0f1
    threads: auto
    cluster-id: 83
    cluster-type: cluster_flow
  - interface: enp24s0f0
    threads: auto
    cluster-id: 84
    cluster-type: cluster_flow
  - interface: enp176s0f1
    threads: auto
    cluster-id: 85
    cluster-type: cluster_flow
  - interface: enp134s0f1
    threads: auto
    cluster-id: 86
    cluster-type: cluster_flow

Solve

1, increase MAX_in source codeNUM_RING_SOCKETS value, then reproduce the rpm package
2. Adjust threads reasonably according to the traffic size of each network card.For network cards with small flow direction, threads should be adjusted to 8, and network cards with large flow should be adjusted to a larger point. The number of threads for all network cards should not exceed 256.(Recommendation)

Reference resources

https://groups.google.com/forum/#!topic/security-onion/uiR9zSmu9Zc
https://github.com/ntop/PF_RING