Fault analysis of an intelligent network card (mellanox)

Background: This is reproduced in centos 7.6.1810. There are many smart network cards at present
The network card on the ECS is standard configuration. In OPPO, it is mainly used in vpc and other scenarios. The code of the intelligent network card will follow
The complexity has been increasing due to the enhancement of functions. The driver bug has always been the big head of the kernel bug. When encountering similar problems, the kernel developers are not familiar with the driver code, so it will be difficult to troubleshoot. The background knowledge involved itself is dma_pool,dma_page,net_device,mlx5_core_dev device, device uninstall, uaf problem, etc. in addition, this bug has not been solved visually in the latest linux baseline. This article lists it separately because the uaf problem is relatively unique.
Here's how we troubleshoot and solve this problem.

1, Fault phenomenon

The OPPO cloud kernel team received the connectivity alarm and found that the machine was reset:

UPTIME: 00:04:16-------------The running time is very short
LOAD AVERAGE: 0.25, 0.23, 0.11
TASKS: 2027
RELEASE: 3.10.0-1062.18.1.el7.x86_sixty-four
MEMORY: 127.6 GB
PANIC: "BUG: unable to handle kernel NULL pointer dereference at           (null)"
PID: 23283
COMMAND: "spider-agent"
TASK: ffff9d1fbb090000  [THREAD_INFO: ffff9d1f9a0d8000]
CPU: 0

crash> bt
PID: 23283  TASK: ffff9d1fbb090000  CPU: 0   COMMAND: "spider-agent"
 #0 [ffff9d1f9a0db650] machine_kexec at ffffffffb6665b34
 #1 [ffff9d1f9a0db6b0] __crash_kexec at ffffffffb6722592
 #2 [ffff9d1f9a0db780] crash_kexec at ffffffffb6722680
 #3 [ffff9d1f9a0db798] oops_end at ffffffffb6d85798
 #4 [ffff9d1f9a0db7c0] no_context at ffffffffb6675bb4
 #5 [ffff9d1f9a0db810] __bad_area_nosemaphore at ffffffffb6675e82
 #6 [ffff9d1f9a0db860] bad_area_nosemaphore at ffffffffb6675fa4
 #7 [ffff9d1f9a0db870] __do_page_fault at ffffffffb6d88750
 #8 [ffff9d1f9a0db8e0] do_page_fault at ffffffffb6d88975
 #9 [ffff9d1f9a0db910] page_fault at ffffffffb6d84778
    [exception RIP: dma_pool_alloc+427]//caq: exception address
    RIP: ffffffffb680efab  RSP: ffff9d1f9a0db9c8  RFLAGS: 00010046
    RAX: 0000000000000246  RBX: ffff9d0fa45f4c80  RCX: 0000000000001000
    RDX: 0000000000000000  RSI: 0000000000000246  RDI: ffff9d0fa45f4c10
    RBP: ffff9d1f9a0dba20   R8: 000000000001f080   R9: ffff9d00ffc07c00
    R10: ffffffffc03e10c4  R11: ffffffffb67dd6fd  R12: 00000000000080d0
    R13: ffff9d0fa45f4c10  R14: ffff9d0fa45f4c00  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
#10 [ffff9d1f9a0dba28] mlx5_ alloc_ cmd_ MSG at ffffc03e10e3 [mlx5_core] / / modules involved
#11 [ffff9d1f9a0dba78] cmd_exec at ffffffffc03e3c92 [mlx5_core]
#12 [ffff9d1f9a0dbb18] mlx5_cmd_exec at ffffffffc03e442b [mlx5_core]
#13 [ffff9d1f9a0dbb48] mlx5_core_access_reg at ffffffffc03ee354 [mlx5_core]
#14 [ffff9d1f9a0dbba0] mlx5_query_port_ptys at ffffffffc03ee411 [mlx5_core]
#15 [ffff9d1f9a0dbc10] mlx5e_get_link_ksettings at ffffffffc0413035 [mlx5_core]
#16 [ffff9d1f9a0dbce8] __ethtool_get_link_ksettings at ffffffffb6c56d06
#17 [ffff9d1f9a0dbd48] speed_show at ffffffffb6c705b8
#18 [ffff9d1f9a0dbdd8] dev_attr_show at ffffffffb6ab1643
#19 [ffff9d1f9a0dbdf8] sysfs_kf_seq_show at ffffffffb68d709f
#20 [ffff9d1f9a0dbe18] kernfs_seq_show at ffffffffb68d57d6
#21 [ffff9d1f9a0dbe28] seq_read at ffffffffb6872a30
#22 [ffff9d1f9a0dbe98] kernfs_fop_read at ffffffffb68d6125
#23 [ffff9d1f9a0dbed8] vfs_read at ffffffffb684a8ff
#24 [ffff9d1f9a0dbf08] sys_read at ffffffffb684b7bf
#25 [ffff9d1f9a0dbf50] system_call_fastpath at ffffffffb6d8dede
    RIP: 00000000004a5030  RSP: 000000c001099378  RFLAGS: 00000212
    RAX: 0000000000000000  RBX: 000000c000040000  RCX: ffffffffffffffff
    RDX: 000000000000000a  RSI: 000000c00109976e  RDI: 000000000000000d---read File fd number
    RBP: 000000c001099640   R8: 0000000000000000   R9: 0000000000000000
    R10: 0000000000000000  R11: 0000000000000206  R12: 000000000000000c
    R13: 0000000000000032  R14: 0000000000f710c4  R15: 0000000000000000
    ORIG_RAX: 0000000000000000  CS: 0033  SS: 002b                

From the stack, it is a process reading a file that triggers a kernel state null pointer reference.

2, Fault phenomenon analysis

From the stack information:

1. At that time, the process opens the file with fd number 13, which can be seen from the rdi value.

2,speed_show and__ ethtool_ get_ link_ Kssettings indicates the rate at which the network card is being read
Let's see which file is open,

crash> files 23283
PID: 23283  TASK: ffff9d1fbb090000  CPU: 0   COMMAND: "spider-agent"
ROOT: /rootfs    CWD: /rootfs/home/service/app/spider
 FD       FILE            DENTRY           INODE       TYPE PATH
  9 ffff9d0f5709b200 ffff9d1facc80a80 ffff9d1069a194d0 REG  /rootfs/sys/devices/pci0000:3a/0000:3a:00.0/0000:3b:00.0/net/p1p1/speed---This is still there
 10 ffff9d0f4a45a400 ffff9d0f9982e240 ffff9d0fb7b873a0 REG  /rootfs/sys/devices/pci0000:5d/0000:5d:00.0/0000:5e:00.0/net/p3p1/speed---Note correspondence 0000:5e:00.0 corresponding p3p1
 11 ffff9d0f57098f00 ffff9d1facc80240 ffff9d1069a1b530 REG  /rootfs/sys/devices/pci0000:3a/0000:3a:00.0/0000:3b:00.1/net/p1p2/speed---This is still there
 13 ffff9d0f4a458a00 ffff9d0f9982e0c0 ffff9d0fb7b875f0 REG  /rootfs/sys/devices/pci0000:5d/0000:5d:00.0/0000:5e:00.1/net/p3p2/speed---Note correspondence 0000:5e:00.1 corresponding p3p2

Note the correspondence between the pci number and the network card name above, which will be used later.
Opening a file to read speed itself should be a very common process,
Let's start with exception RIP: dma_pool_alloc+427 further analyzes why NULL pointer dereference is triggered
Expand the specific stack as follows:

#9 [ffff9d1f9a0db910] page_fault at ffffffffb6d84778
    [exception RIP: dma_pool_alloc+427]
    RIP: ffffffffb680efab  RSP: ffff9d1f9a0db9c8  RFLAGS: 00010046
    RAX: 0000000000000246  RBX: ffff9d0fa45f4c80  RCX: 0000000000001000
    RDX: 0000000000000000  RSI: 0000000000000246  RDI: ffff9d0fa45f4c10
    RBP: ffff9d1f9a0dba20   R8: 000000000001f080   R9: ffff9d00ffc07c00
    R10: ffffffffc03e10c4  R11: ffffffffb67dd6fd  R12: 00000000000080d0
    R13: ffff9d0fa45f4c10  R14: ffff9d0fa45f4c00  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
    ffff9d1f9a0db918: 0000000000000000 ffff9d0fa45f4c00 
    ffff9d1f9a0db928: ffff9d0fa45f4c10 00000000000080d0 
    ffff9d1f9a0db938: ffff9d1f9a0dba20 ffff9d0fa45f4c80 
    ffff9d1f9a0db948: ffffffffb67dd6fd ffffffffc03e10c4 
    ffff9d1f9a0db958: ffff9d00ffc07c00 000000000001f080 
    ffff9d1f9a0db968: 0000000000000246 0000000000001000 
    ffff9d1f9a0db978: 0000000000000000 0000000000000246 
    ffff9d1f9a0db988: ffff9d0fa45f4c10 ffffffffffffffff 
    ffff9d1f9a0db998: ffffffffb680efab 0000000000000010 
    ffff9d1f9a0db9a8: 0000000000010046 ffff9d1f9a0db9c8 
    ffff9d1f9a0db9b8: 0000000000000018 ffffffffb680ee45 
    ffff9d1f9a0db9c8: ffff9d0faf9fec40 0000000000000000 
    ffff9d1f9a0db9d8: ffff9d0faf9fec48 ffffffffb682669c 
    ffff9d1f9a0db9e8: ffff9d00ffc07c00 00000000618746c1 
    ffff9d1f9a0db9f8: 0000000000000000 0000000000000000 
    ffff9d1f9a0dba08: ffff9d0faf9fec40 0000000000000000 
    ffff9d1f9a0dba18: ffff9d0fa3c800c0 ffff9d1f9a0dba70 
    ffff9d1f9a0dba28: ffffffffc03e10e3 
#10 [ffff9d1f9a0dba28] mlx5_alloc_cmd_msg at ffffffffc03e10e3 [mlx5_core]
    ffff9d1f9a0dba30: ffff9d0f4eebee00 0000000000000001 
    ffff9d1f9a0dba40: 000000d0000080d0 0000000000000050 
    ffff9d1f9a0dba50: ffff9d0fa3c800c0 0000000000000005 --r12 yes rdi ,ffff9d0fa3c800c0
    ffff9d1f9a0dba60: ffff9d0fa3c803e0 ffff9d1f9d87ccc0 
    ffff9d1f9a0dba70: ffff9d1f9a0dbb10 ffffffffc03e3c92 
#11 [ffff9d1f9a0dba78] cmd_exec at ffffffffc03e3c92 [mlx5_core]

Take the corresponding mlx5 from the stack_ core_ Dev is ffff9d0fa3c800c0

crash> mlx5_core_dev.cmd ffff9d0fa3c800c0 -xo
struct mlx5_core_dev {
  [ffff9d0fa3c80138] struct mlx5_cmd cmd;
crash> mlx5_cmd.pool ffff9d0fa3c80138
  pool = 0xffff9d0fa45f4c00------This is dma_pool,Students who write driver code often encounter

The code line number of the problem is:

crash> dis -l dma_pool_alloc+427 -B 5
/usr/src/debug/kernel-3.10.0-1062.18.1.el7/linux-3.10.0-1062.18.1.el7.x86_64/mm/dmapool.c: 334
0xffffffffb680efab <dma_pool_alloc+427>:        mov    (%r15),%ecx
 And the corresponding r15,From the stack above, it is null. 
    305 void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_flags,
    306                      dma_addr_t *handle)
    307 {
    315         spin_lock_irqsave(&pool->lock, flags);
    316         list_for_each_entry(page, &pool->page_list, page_list) {
    317                 if (page->offset < pool->allocation)---//caq: currently satisfied
    318                         goto ready;//caq: jump to ready
    319         }
    321         /* pool_alloc_page() might sleep, so temporarily drop &pool->lock */
    322         spin_unlock_irqrestore(&pool->lock, flags);
    324         page = pool_alloc_page(pool, mem_flags & (~__GFP_ZERO));
    325         if (!page)
    326                 return NULL;
    328         spin_lock_irqsave(&pool->lock, flags);
    330         list_add(&page->page_list, &pool->page_list);
    331  ready:
    332         page->in_use++;//caq: indicates that it is being referenced
    333         offset = page->offset;//Use it from the last place
    334         page->offset = *(int *)(page->vaddr + offset);//caq: the line number of the problem

From the above code, page - > vaddr is NULL and offset is 0. Page has two sources,

The first is the page from the pool_ List,

The second is from pool_alloc_page is a temporary application. Of course, after the application, it will be linked to the page in the pool_ list,

Let's take a look at this page_list.

crash> dma_pool ffff9d0fa45f4c00 -x
struct dma_pool {
  page_list = {
    next = 0xffff9d0fa45f4c80, 
    prev = 0xffff9d0fa45f4c00
  lock = {
      rlock = {
        raw_lock = {
          val = {
            counter = 0x1
  size = 0x400, 
  dev = 0xffff9d1fbddec098, 
  allocation = 0x1000, 
  boundary = 0x1000, 
  name = "mlx5_cmd\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000", 
  pools = {
    next = 0xdead000000000100, 
    prev = 0xdead000000000200

crash> list dma_pool.page_list -H 0xffff9d0fa45f4c00 -s dma_page.offset,vaddr
  offset = 0
  vaddr = 0x0
  offset = 0
  vaddr = 0x0

Slave DMA_ pool_ From the code logic of alloc function, pool - > page_ The list is indeed not empty and satisfied
If (page - > offset < pool - > allocation), so the first page should be ffff9d0fa45f4c80
That is, from the first case:

crash> dma_page ffff9d0fa45f4c80
struct dma_page {
  page_list = {
    next = 0xffff9d0fa45f4d00, 
    prev = 0xffff9d0fa45f4c80
  vaddr = 0x0, //caq: this exception will cause crash if it is referenced
  dma = 0, 
  in_use = 1, //caq: this flag is in use and conforms to page - > in_ use++;
  offset = 0

This is the end of the problem analysis, because DMA_ vaddr will be initialized after the page in the pool is applied,
Usually in the pool_ alloc_ How can it be NULL when initializing in page?
Then check this address:

crash> kmem ffff9d0fa45f4c80-------This is dma_pool Medium page
CACHE            NAME                 OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE
ffff9d00ffc07900 kmalloc-128//caq: note the length 128 8963 14976 234 8K
  SLAB              MEMORY            NODE  TOTAL  ALLOCATED  FREE
  ffffe299c0917d00  ffff9d0fa45f4000     0     64         29    35

ffffe299c0917d00 10245f4000                0 ffff9d0fa45f4c00  1 2fffff00004080 slab,head

Because similar dma functions have been used before, I am impressed with dma_page is not so big. Let's look at the second dma_ The page is as follows:

crash> kmem ffff9d0fa45f4d00
CACHE            NAME                 OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE
ffff9d00ffc07900 kmalloc-128              128       8963     14976    234     8k
  SLAB              MEMORY            NODE  TOTAL  ALLOCATED  FREE
  ffffe299c0917d00  ffff9d0fa45f4000     0     64         29    35

ffffe299c0917d00 10245f4000                0 ffff9d0fa45f4c00  1 2fffff00004080 slab,head

crash> dma_page ffff9d0fa45f4d00
struct dma_page {
  page_list = {
    next = 0xffff9d0fa45f5000, 
    prev = 0xffff9d0fa45f4d00
  vaddr = 0x0, -----------caq: Also null
  dma = 0, 
  in_use = 0, 
  offset = 0

crash> list dma_pool.page_list -H 0xffff9d0fa45f4c00 -s dma_page.offset,vaddr
  offset = 0
  vaddr = 0x0
  offset = 0
  vaddr = 0x0
  offset = 0
  vaddr = 0x0

It seems that it is not only the first DMA_ There is a problem with the page. All DMAS in the pool_ All units are the same,
Check DMA directly_ Normal size of page:

crash> p sizeof(struct dma_page)
$3 = 40

According to the principle, the length is only 40 bytes. Even if you apply for slab, it should be expanded to 64 bytes. How can it be like the DMA above_ Is page 128 bytes? To solve this puzzle, find a normal node and compare it with other nodes:

crash> net
ffff8f9e800be000  lo
ffff8f9e62640000  p1p1   
ffff8f9e626c0000  p1p2   
ffff8f9e627c0000  p3p1   -----//caq: take this as an example
ffff8f9e62100000  p3p2   

Then according to the code: net_device see mlx5e_priv: 

static int mlx5e_get_link_ksettings(struct net_device *netdev,
                    struct ethtool_link_ksettings *link_ksettings)
    struct mlx5e_priv *priv    = netdev_priv(netdev);

static inline void *netdev_priv(const struct net_device *dev)
	return (char *)dev + ALIGN(sizeof(struct net_device), NETDEV_ALIGN);

crash> px sizeof(struct net_device)
$2 = 0x8c0

crash> mlx5e_priv.mdev ffff8f9e627c08c0---Based on offset calculation
  mdev = 0xffff8f9e67c400c0

crash> mlx5_core_dev.cmd 0xffff8f9e67c400c0 -xo
struct mlx5_core_dev {
  [ffff8f9e67c40138] struct mlx5_cmd cmd;

crash> mlx5_cmd.pool ffff8f9e67c40138
  pool = 0xffff8f9e7bf48f80

crash> dma_pool 0xffff8f9e7bf48f80
struct dma_pool {
  page_list = {
    next = 0xffff8f9e79c60880, //caq: one of the dma_page
    prev = 0xffff8fae6e4db800
  size = 1024, 
  dev = 0xffff8f9e800b3098, 
  allocation = 4096, 
  boundary = 4096, 
  name = "mlx5_cmd\000\217\364{\236\217\377\377\300\217\364{\236\217\377\377\200\234>\250\217\217\377\377", 
  pools = {
    next = 0xffff8f9e800b3290, 
    prev = 0xffff8f9e800b3290
crash> dma_page 0xffff8f9e79c60880     //caq: check this dma_page
struct dma_page {
  page_list = {
    next = 0xffff8f9e79c60840, -------One of them dma_page
    prev = 0xffff8f9e7bf48f80
  vaddr = 0xffff8f9e6fc9b000, //caq: normal vaddr cannot be NULL
  dma = 69521223680, 
  in_use = 0, 
  offset = 0

crash> kmem 0xffff8f9e79c60880
ffff8f8fbfc07b00 kmalloc-64--Normal length    64     six hundred and sixty-seven thousand nine hundred and twenty-one    745024  11641  4 k
  SLAB              MEMORY            NODE  TOTAL  ALLOCATED  FREE
  ffffde5140e71800  ffff8f9e79c60000     0     64         64     0

ffffde5140e71800 1039c60000                0        0  1 2fffff00000080 slab

The above operation requirements are correct_ Device and mlx5 related driver code are familiar.
Compared with abnormal dma_page, normal dma_page is a 64 byte slab, so obviously,
Either this is a memory problem or a uaf(used after free) problem.
General questions: how can we quickly judge which type it is? Because these two problems involve memory disorder, they are generally difficult to check. At this time, we need to jump out. Let's take a look at other running processes and find a process as follows:

crash> bt 48263
PID: 48263  TASK: ffff9d0f4ee0a0e0  CPU: 56  COMMAND: "reboot"
 #0 [ffff9d0f95d7f958] __schedule at ffffffffb6d80d4a
 #1 [ffff9d0f95d7f9e8] schedule at ffffffffb6d811f9
 #2 [ffff9d0f95d7f9f8] schedule_timeout at ffffffffb6d7ec48
 #3 [ffff9d0f95d7faa8] wait_for_completion_timeout at ffffffffb6d81ae5
 #4 [ffff9d0f95d7fb08] cmd_exec at ffffffffc03e41c9 [mlx5_core]
 #5 [ffff9d0f95d7fba8] mlx5_cmd_exec at ffffffffc03e442b [mlx5_core]
 #6 [ffff9d0f95d7fbd8] mlx5_core_destroy_mkey at ffffffffc03f085d [mlx5_core]
 #7 [ffff9d0f95d7fc40] mlx5_mr_cache_cleanup at ffffffffc0c60aab [mlx5_ib]
 #8 [ffff9d0f95d7fca8] mlx5_ib_stage_pre_ib_reg_umr_cleanup at ffffffffc0c45d32 [mlx5_ib]
 #9 [ffff9d0f95d7fcc0] __mlx5_ib_remove at ffffffffc0c4f450 [mlx5_ib]
#10 [ffff9d0f95d7fce8] mlx5_ib_remove at ffffffffc0c4f4aa [mlx5_ib]
#11 [ffff9d0f95d7fd00] mlx5_detach_device at ffffffffc03fe231 [mlx5_core]
#12 [ffff9d0f95d7fd30] mlx5_unload_one at ffffffffc03dee90 [mlx5_core]
#13 [ffff9d0f95d7fd60] shutdown at ffffffffc03def80 [mlx5_core]
#14 [ffff9d0f95d7fd80] pci_device_shutdown at ffffffffb69d1cda
#15 [ffff9d0f95d7fda8] device_shutdown at ffffffffb6ab3beb
#16 [ffff9d0f95d7fdd8] kernel_restart_prepare at ffffffffb66b7916
#17 [ffff9d0f95d7fde8] kernel_restart at ffffffffb66b7932
#18 [ffff9d0f95d7fe00] SYSC_reboot at ffffffffb66b7ba9
#19 [ffff9d0f95d7ff40] sys_reboot at ffffffffb66b7c4e
#20 [ffff9d0f95d7ff50] system_call_fastpath at ffffffffb6d8dede
    RIP: 00007fc9be7a5226  RSP: 00007ffd9a19e448  RFLAGS: 00010246
    RAX: 00000000000000a9  RBX: 0000000000000004  RCX: 0000000000000000
    RDX: 0000000001234567  RSI: 0000000028121969  RDI: fffffffffee1dead
    RBP: 0000000000000002   R8: 00005575d529558c   R9: 0000000000000000
    R10: 00007fc9bea767b8  R11: 0000000000000206  R12: 0000000000000000
    R13: 00007ffd9a19e690  R14: 0000000000000000  R15: 0000000000000000
    ORIG_RAX: 00000000000000a9  CS: 0033  SS: 002b

Why do you pay attention to this process? Over the years, uaf problems caused by uninstalling modules have been checked no less than 20 times, sometimes reboots, sometimes unload s, and sometimes releases resources in work. Therefore, intuitively, I think it has a lot to do with this uninstallation. Let's analyze the operation in the reboot process.

2141 void device_shutdown(void)
   2142 {
   2143         struct device *dev, *parent;
   2145         spin_lock(&devices_kset->list_lock);
   2146         /*
   2147          * Walk the devices list backward, shutting down each in turn.
   2148          * Beware that device unplug events may also start pulling
   2149          * devices offline, even as the system is shutting down.
   2150          */
   2151         while (!list_empty(&devices_kset->list)) {
   2152                 dev = list_entry(devices_kset->list.prev, struct device,
   2153                                 kobj.entry);
   2178                 if (dev->device_rh && dev->device_rh->class_shutdown_pre) {
   2179                         if (initcall_debug)
   2180                                 dev_info(dev, "shutdown_pre\n");
   2181                         dev->device_rh->class_shutdown_pre(dev);
   2182                 }
   2183                 if (dev->bus && dev->bus->shutdown) {
   2184                         if (initcall_debug)
   2185                                 dev_info(dev, "shutdown\n");
   2186                         dev->bus->shutdown(dev);
   2187                 } else if (dev->driver && dev->driver->shutdown) {
   2188                         if (initcall_debug)
   2189                                 dev_info(dev, "shutdown\n");
   2190                         dev->driver->shutdown(dev);
   2191                 }

From the above code, we can see the following two points:

1. Kobj of each device The entry member is concatenated in devices_ Kset - > list.

2. The shutdown process of each device is from device_shutdown is serial.

From the stack of reboot, the process of unloading an mlx device includes the following:


mlx5_ detach_ The process branch of device is:

void dma_pool_destroy(struct dma_pool *pool)
        while (!list_empty(&pool->page_list)) {//caq: DMA in pool_ Page delete one by one
                struct dma_page *page;
                page = list_entry(pool->page_list.next,
                                  struct dma_page, page_list);
                if (is_page_busy(page)) {
                } else
                        pool_free_page(pool, page);//Per dma_page to release

        kfree(pool);//caq: release pool

static void pool_free_page(struct dma_pool *pool, struct dma_page *page)
        dma_addr_t dma = page->dma;

        memset(page->vaddr, POOL_POISON_FREED, pool->allocation);
        dma_free_coherent(pool->dev, pool->allocation, page->vaddr, dma);
        list_del(&page->page_list);//caq: after release, the page_list member poisoning

From the stack of reboot, view the corresponding information

 #4 [ffff9d0f95d7fb08] cmd_exec at ffffffffc03e41c9 [mlx5_core]
    ffff9d0f95d7fb10: ffffffffb735b580 ffff9d0f904caf18 
    ffff9d0f95d7fb20: ffff9d00ff801da8 ffff9d0f23121200 
    ffff9d0f95d7fb30: ffff9d0f23121740 ffff9d0fa7480138 
    ffff9d0f95d7fb40: 0000000000000000 0000001002020000 
    ffff9d0f95d7fb50: 0000000000000000 ffff9d0f95d7fbe8 
    ffff9d0f95d7fb60: ffff9d0f00000000 0000000000000000 
    ffff9d0f95d7fb70: 00000000756415e3 ffff9d0fa74800c0 ----mlx5_core_dev Equipment, corresponding to p3p1,
    ffff9d0f95d7fb80: ffff9d0f95d7fbf8 ffff9d0f95d7fbe8 
    ffff9d0f95d7fb90: 0000000000000246 ffff9d0f8f3a20b8 
    ffff9d0f95d7fba0: ffff9d0f95d7fbd0 ffffffffc03e442b 
 #5 [ffff9d0f95d7fba8] mlx5_cmd_exec at ffffffffc03e442b [mlx5_core]
    ffff9d0f95d7fbb0: 0000000000000000 ffff9d0fa74800c0 
    ffff9d0f95d7fbc0: ffff9d0f8f3a20b8 ffff9d0fa74bea00 
    ffff9d0f95d7fbd0: ffff9d0f95d7fc38 ffffffffc03f085d 
 #6 [ffff9d0f95d7fbd8] mlx5_core_destroy_mkey at ffffffffc03f085d [mlx5_core]

Note that reboot is releasing mlx5_core_dev is ffff9d0fa74800c0, which corresponds to net_device is:
p3p1, while the 23283 process is accessing mlx5_core_dev is ffff9d0fa3c800c0, corresponding to p3p2.

crash> net
ffff9d0fc003e000  lo
ffff9d1fad200000  p1p1   
ffff9d0fa0700000  p1p2   
ffff9d0fa00c0000  p3p1  Corresponding mlx5_core_dev yes ffff9d0fa74800c0
ffff9d0fa0200000  p3p2  Corresponding mlx5_core_dev yes ffff9d0fa3c800c0

Let's take a look at what's still left_ device in kset:

crash> p devices_kset
devices_kset = $4 = (struct kset *) 0xffff9d1fbf4e70c0
crash> p devices_kset.list
$5 = {
  next = 0xffffffffb72f2a38, 
  prev = 0xffff9d0fbe0ea130

crash> list -H -o 0x18 0xffffffffb72f2a38 -s device.kobj.name >device.list

We found p3p1 And p3p2 None device.list In,

[root@it202-seg-k8s-prod001-node-10-27-96-220]# grep 0000:5e:00.0 device.list //caq: not found. This is p3p1. The current reboot process is unloading.
[root@it202-seg-k8s-prod001-node-10-27-96-220]# grep 0000:5e:00.1 device.list //caq: not found. This is p3p2, which has been uninstalled
[root@it202-seg-k8s-prod001-node-10-27-96-220]# grep 0000:3b:00.0  device.list //caq: this mlx5 device has not been unload ed yet
  kobj.name = 0xffff9d1fbe82aa70 "0000:3b:00.0",
[root@it202-seg-k8s-prod001-node-10-27-96-220]# grep 0000:3b:00.1 device.list //caq: this mlx5 device has not been unload ed yet
  kobj.name = 0xffff9d1fbe82aae0 "0000:3b:00.1",

Because p3p2 and p3p1 are not in device List, and according to PCI_ device_ In the serial uninstall process of shutdown, p3p1 is currently being unloaded, so it is certain that the 23283 process accesses the unloaded cmd_pool, according to the unloading process described above:
At this time, the pool has been released, and DMA in the pool_ All pages are invalid.

Then try the bug corresponding to google, and see that RedHat has encountered a similar problem, which is very similar to the current phenomenon: https://access.redhat.com/solutions/5132931

However, in this link, red hat thinks that the problem of uaf has been solved, and the combined patch is:

commit 4cca96a8d9da0ed8217cfdf2aec0c3c8b88e8911
Author: Parav Pandit <parav@mellanox.com>
Date:   Thu Dec 12 13:30:21 2019 +0200

diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index 997cbfe..05b557d 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -6725,6 +6725,8 @@ void __mlx5_ib_remove(struct mlx5_ib_dev *dev,
                      const struct mlx5_ib_profile *profile,
                      int stage)
+       dev->ib_active = false;
        /* Number of stages to cleanup */
        while (stage) {

Knock on the blackboard three times:
This integration cannot solve the corresponding bug, such as the following Concurrency:
Let's use a simple diagram to show concurrent processing:

    CPU1                                                            CPU2
    pci_device_shutdown                                            speed_show
                     clear_bit(MLX5E_STATE_OPENED, &priv->state);---Only this was cleaned bit
                                                  rtnl_trylock---After locking successfully
                                                  netif_running Just judgment net_device.state Lowest order of
          mlx5_cmd_cleanup---clear dma_pool                                                       
                                                      dma_pool_alloc---visit cmd.pool,trigger crash


So if you want to really solve this problem, you need netif_ device_ Cleaning in detach__ LINK_ STATE_ Bit of start, or in speed_ Judge in show__ LINK_STATE_PRESENT bit? If you consider the scope of influence and do not want to move the public process, you should
In mlx5e_ get_ link_ Judge from ksettings__ LINK_STATE_PRESENT.
This is left to the students who like to deal with the community to improve it.

static void mlx5e_nic_disable(struct mlx5e_priv *priv)
	if (netif_running(priv->netdev))
  //caq: add cleaning__ LINK_STATE_PRESENT bit 

3, Fault recurrence

1. The competition problem can create a competition scenario similar to cpu1 and cpu2 in the figure above.

4, Fault avoidance or resolution

Possible solutions are:

1. Don't follow red hat https://access.redhat.com/solutions/5132931 Upgrade like that.

2. Patch separately.

Introduction to the author


At present, I am responsible for linux kernel, container, virtual machine and other virtualization in OPPO hybrid cloud

For more exciting content, please scan code to focus on [OPPO technology] official account.

Keywords: Back-end

Added by premiso on Fri, 14 Jan 2022 03:33:46 +0200