linux Process / thread / CO process series 2: system commands of process / thread


Recently, when learning autopilot system, I met the concept of co operation. The process and thread have been lost, and there is another collaborative process. I believe many friends have the same idea as the author. After reading a lot of materials, they decided to write a summary to summarize the connections and differences between the three, and finally come down to the application of collaborative process in automatic driving. The goal of junior programmers is to clarify the concepts of the three and apply them to practice, while senior engineers consider the performance and cost of realizing the three at the system level. Until now, the implementation of the three is still one of the modules that are continuously updated and improved by the Linux kernel and various programming languages. Therefore, it is a compulsory course for advanced programmers to clarify the relationship and application of the three. If you are also vague about this knowledge, this article will be helpful to you. The purpose of this paper is to make a comprehensive summary of a series of complex concepts and knowledge points of process / thread / collaborative process. At the same time, try to make the knowledge points precise, detailed and comprehensive, distinguish fuzzy concepts, give consideration to the source code and programming implementation, and finally come down to the collaborative process implementation of Apollo.

The text of this plan is as concise as possible, but it can not achieve the purpose of refinement. Therefore, the positioning of this series of articles is a detailed summary of complex knowledge points, and on this basis, it should be as concise as possible. Because I have consulted a lot of materials and spent a lot of energy, although it is not perfect, I hope your little hand can support the author and make a one click triple, because the author's subsequent articles can help you keep exploring.

This series of articles is divided into seven parts: the first part: the evolution from process to collaborative process, involving the progress of program history and the knowledge of computer system structure. Chapter 2: summarize the system commands related to process / thread, so that you can have a perceptual understanding of them at the beginning, not just raw words. Chapter 3: how to view the source code of linux system? First-hand knowledge is the foundation. Only when you have access to first-hand information can you become a big fan in the industry. The source code is undoubtedly the most important first-hand information. Don't worry. I'll teach you here. Part IV: process knowledge crosstalk. Because processes, threads and collaborative processes come down in one continuous line, most of the process knowledge is also applicable to threads and collaborative processes. If you have a thorough understanding of the process, the difficulties of threads and collaborative processes will be solved. The fifth part: the nature discrimination and realization difference of the three. At present, most articles start with the concept, especially the nature comparison of the three, and the concept is easy to be confused. Therefore, the author adopts the mode of telling things in detail, with the foreshadowing in front, I believe you will feel inspired when you see the comparison of nature again. After analyzing the nature, we will also make a simple comparison on the implementation code of the three. Chapter 6: the scheduling of the three in memory. Memory scheduling can better reflect the process of program execution. Here, we go deep into the replacement of process memory scheduling to take readers to appreciate the charm of memory scheduling. Chapter 7: comparison and summary of various current collaborative processes, from the boost library to various collaborative libraries libgo/ntyco/libco based on the boost library, as well as the current implementation of Python/Java/Go; The author thinks that there are few examples of libgo's application and implementation in the current article. However, the author thinks that there are few examples of libgo's application and implementation in the current article; Chapter 9: the application of cooperative process in Apollo, analyzes the source code, advantages and disadvantages of Apollo cooperative process.


This chapter is divided into two sections: the first section explains the process related system commands and field parsing, and the second section is thread related. Since the cooperative process is generally a call library, there are no relevant system commands. Later, we will choose a specific mature library libgo for comparison and explanation, which will not be mentioned here for the time being.

1. Process related commands

1.1 graphic view command

  1. gnome graphics viewing:

The display interface is as follows, including process information, system resources and file system. The PID of process information is the process ID, which is unique. Can you understand other fields:

2. pstree
The pstree (display a tree of processes) command displays all processes in a tree view. The tree view will take pid (if specified) or init as the root. If a user id is specified, the tree view will only display the processes owned by the user. The command parameters are as follows:
-p develop process ID
-t displays the full name of the thread
-n sort according to PID
-h highlight the current process and its parent process
-A displays the complete instructions and parameters of the process. If it is a process replaced by memory, parentheses will be added
-c if there are repeated process names, they will be listed separately (the default value is to add *)

It can display the relationship between processes, and can also display the sub processes and threads with specified ID (such as browser). You can also see the web page thread you open:

//Show inter process relationships
$ pstree -apnh 
systemd,1 splash
  │   └─{systemd-timesyn},925
//pgrep can find the currently running process and list the qualified process ID s
pgrep firefox
//Displays the child processes and threads with the specified ID
$ pstree -p 2779 -t
firefox(2779)─┬─Privileged Cont(3078)─┬─{HTML5 Parser}(3119)
              │                       ├─{IPC I/O Child}(3084)
              │                       ├─{ImageBridgeChld}(3101)
              │                       ├─{ImageIO}(3090)
              │                       ├─{JS Watchdog}(3086)

1.2 static view command ps

ps is process status. Its command parameters are listed as follows:

  • -A ## view all processes
  • a ## view all foreground processes
  • x ## view all background processes
  • l ## view details, where
  • f ## displays the parent-child relationship of the process
  • e ## display process details (call of system resources)
  • – sort ## process sorting
  • -o [parameter] ## displays the process designation information, and the optional parameters are as follows:
    comm ## process name
    user ## process owner
    Group ## process group
    %cpu ## process cpu utilization
    %mem ## process memory usage
    pid ## process pid
    nice ## process priority, range - 20 ~ 19.

nice can be used as a command to set priorities, such as:

//View priority
$ ps ax -o pid,nice,comm,user | head -5 
    1   0 systemd         root
    2   0 kthreadd        root
    3 -20 rcu_gp          root
    4 -20 rcu_par_gp      root

//Set priority: nice -n priority number 	 Process name
nice -n -4 vim &	##Start vim and specify the process priority as - 4, & indicating background operation

ps common combinations:

  • ps aux ## displays all processes in the system and the process users
  • ps ef ## displays process details and parent-child relationships
  • PS aux -- sort% CPU /% MEM ## displays all processes in the system and the process users, and sorts them according to cpu/mem utilization

For example, the most commonly used combination of aux and axl is shown as follows:

$ ps aux | head -10
root         1  0.0  0.0 226088  9676 ?        Ss    2021   8:23 /sbin/init splash
root         2  0.0  0.0      0     0 ?        S     2021   0:01 [kthreadd]
root         3  0.0  0.0      0     0 ?        I<    2021   0:00 [rcu_gp]
root         4  0.0  0.0      0     0 ?        I<    2021   0:00 [rcu_par_gp]
root         9  0.0  0.0      0     0 ?        I<    2021   0:00 [mm_percpu_wq]
ps axl | head -5
4     0     1     0  20   0 226088  9676 -      Ss   ?          8:46 /sbin/init splash
1     0     2     0  20   0      0     0 -      S    ?          0:01 [kthreadd]
1     0     3     2   0 -20      0     0 -      I<   ?          0:00 [rcu_gp]
1     0     4     2   0 -20      0     0 -      I<   ?          0:00 [rcu_par_gp]

Description of fields in ps aux/axl: USER - USER, PID/PPID/SPID - process ID / parent process ID / system process ID,% CPU - CPU utilization,% MEM - memory utilization, VSZ - virtual memory utilization, RSS - actual memory utilization, TTY - terminal, STAT - status, START - START TIME of process, TIME - total TIME of process occupying CPU, COMMAND - directory or file name, F - process flag or permission (process flags, 4 for root, 1 for fork only), PRI - priority (- 100... 39), WCHAN: whether the process is running (- represents running, others represent the name of the kernel function that the process is sleeping; the name of the function is obtained from the / root/ file).

Where the value in STAT (process status) indicates:

  • S ## can interrupt sleep (sleep, blocked, waiting for a condition to form or receiving a signal)
  • l ## there is locking space in memory
  • N ## priority low
  • < ## priority high
  • +## foreground operation
  • s ## top level process
  • D ## non interruptible sleep (usually during IO operation) does not wake up and cannot run after receiving the signal. The process must wait until an interrupt occurs
  • R ## is running or runnable (in the run queue)
  • T ## stopped processes stop after receiving SIGSTOP, SIGSTP, SIGTIN and SIGTOU signals
  • W ## no resident pages, changing pages (valid before 2.6. Kernel)
  • X ## dead process (not started)
  • P ## waiting for swap page
  • Zombie (a defunct process), the process has been terminated, but the process descriptor exists until it is released after the parent process calls wait4() system call
  • l multithreading (using CLONE_THREAD, similar to pthreads of NPTL)

1.3 dynamic view command top

View the top dynamically, like the task manager of windows, as follows:

$ top
top - 14:34:05 up 28 days, 18:03,  1 user,  load average: 0.41, 0.29, 0.20
Tasks: 380 total,   5 running, 271 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.5 us,  0.7 sy,  0.0 ni, 97.6 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem : 32485348 total, 14727192 free, 10751656 used,  7006500 buff/cache
KiB Swap: 15999996 total, 15999996 free,        0 used. 19731244 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                        
 1708 shaw      20   0 5212864 533048 156940 R   7.3  1.6 111:16.31 gnome-shell                                                                                                                                    
 1421 shaw      20   0 25.122g 200084 154944 R   5.0  0.6 388:14.29 Xorg                                                                                                                                           
 2779 shaw      20   0 61.363g 0.013t 0.010t S   4.7 43.9   1741:16 firefox                                                                                                                                        
30446 shaw      20   0 4025532 693656 233384 S   4.3  2.1 520:34.93 Web Content                                                                                                                                    
 1777 shaw      20   0 3171592  21096  16636 S   2.0  0.1 605:03.78 pulseaudio                                                                        

Where: VIRT: virtual memory usage virtual memory (Applied memory), RES: resident memory usage resident memory (occupied memory), SHR: shared memory shared memory (shared library size).
During top operation, the display mode of the process can be controlled through the internal command of top. Internal commands are as follows:

  • h – display help
  • s – change the screen update frequency
  • l – turn off or turn on the top information in the first line of the first part
  • t – close or open the representation of Tasks in the second line and Cpus in the third line of the first part
  • m – turn off or on the representation of Mem in the fourth line and Swap in the fifth line of the first part
  • N – indicates the process list in the order of PID size
  • P – arrange the process list in the order of CPU utilization
  • M – sort the process list in order of memory usage
  • n – sets the number of processes displayed in the process list
  • < forward
  • >Backward
  • z color
  • Z set color (use digital adjustment)
  • q exit top

1.4 send signal to process

Use kill to send signals to the process, and use kill -l to view the signals supported by kill:

$ kill -l
//Kill a process
kill -9 PID 

Because there are so many signals, it's enough to remember some commonly used ones (kill - number / signal):

  • 1) SIGHUP reload configuration
  • 2) SIGINT keyboard interrupt ctrl+c
  • 3) SIGQUIT keyboard exits ctrl +, similar to SIGIN
  • 9) SIGKILL forced termination, unconditional
  • 15) SIGTERM termination (normal end), default signal
  • 18) SIGCOUNT continue
  • 19) SIGSTOP keyboard pause ctrl+z

1.5 pseudo file system / proc

Proc file system is a pseudo file system, which only exists in memory and does not occupy external memory space. It provides an interface for accessing system kernel data in the way of file system. Users and applications can get relevant information through proc and change some parameters of the kernel. Because the information of the system changes dynamically, when the user or application reads the proc file, the proc file system dynamically reads the required information from the system kernel and submits it. According to the usage of / proc, this section is divided into three parts: system state, process state and how to evaluate process memory

1.5.1 system status information

/The system information provided by various files under the proc file system is not specific to a specific process, but can be used in the context of the whole system. Not all of the files or subfolders listed below exist in your system, depending on your kernel configuration and loaded modules. In addition, there are three important directories under proc: net, scsi and sys. Sys directory is writable and can be used to access or modify kernel parameters, while net and scsi depend on kernel configuration. For example, if the system does not support scsi, the scsi directory does not exist. Here are some examples.

  • /proc/cmdline: system startup information, including kernel image, the location of the first block of the root directory file of the system, startup parameter information, etc. As shown below, where BOOT_IMAGE: the system starts the image; Root: the address of the root directory file, which can be replaced by / dev / sda1 (if the root file system is in the first partition), but cannot be replaced by root=(hd0,msdos1), because it is not a legal parameter type; ro: mount the root file system in read-only mode at startup; quiet: do not display the progress of instruction execution; Splash: display startup animation; vt.handoff=7: tty7 will be used in the graphical interface. This item is used to prohibit splash from occupying tty7. Therefore, if the splash parameter is removed, this item will become invalid. The above configuration is in / boot / grub / grub In CFG.
$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.4.0-89-generic root=UUID=bef418fa-4202-4513-b39b-cde6a5d9753f ro quiet splash vt.handoff=7
  • /proc/cpuinfo: this file provides a variety of information about the system CPU. This information is obtained from the CPU test code in the kernel. The document lists the general model of CPU (386486586686, etc.) and more specific information available (manufacturer, model and version). The file also contains the processor speed expressed in bogomips, and if multiple features or bug s of the CPU are detected, the file will also contain corresponding flags. The format of this file is: the file is composed of multiple lines, and each line includes a field name, a colon and a value.
  • /proc/meminfo: displays the information of memory status, such as the total amount of free memory, used physical memory and swap memory in the system. It also shows the total amount of shared memory and buffers used by the kernel. The format of this information is similar to the result displayed by the free command.
  • /proc/stat: this file contains information about CPU utilization, disks, memory pages, memory swapping, all interrupts, contact switches, and reward bootstrap time (seconds since January 1, 1970).
  • /proc/devices: the main device number of character and block devices and the device name assigned to these device numbers.
  • /proc/mounts: this file gives the file system information installed on the current system in the format of / etc/mtab file. This file can also reflect any file system that was manually installed so that it was not included in the / etc/mtab file.
  • /proc/sys: there are many subdirectories under this directory. Many items in this directory can be used to adjust the performance of the system. This directory contains too much information to cover all of it. Only some files in the directory are shown in the example.
  • /proc/net: the files in this directory describe or modify the behavior of networking code. Many of these special files can be set up or queried by using the arp, netstat, route, and ipfwadm commands.
  • proc/scsi: this directory contains a file that lists all detected SCSI devices, and provides a directory for each controller driver, and a subdirectory for each instance of the installed controller.
$ ls /proc/sys
debug  dev  fs  kernel  net  proc  sunrpc  vm
ls /proc/sys/fs
aio-max-nr   dentry-state       file-nr      lease-break-time  overflowgid
aio-nr       dir-notify-enable  inode-nr     leases-enable     overflowuid
binfmt_misc  file-max           inode-state  mqueue            quota
ls /proc/sys/net
core  ethernet  ipv4  ipv6  unix 

1.5.2. Process status information

In addition to the above, there are some directories named by numbers, which are process directories. Each process currently running in the system has a corresponding directory. Under proc, the PID of the process is the directory name. They are the interfaces to read process information. The self directory is the information interface for reading the process itself, which is a link. Other directories are as follows:

  • /proc/[pid]/cmdline: the complete command line information of the process runtime, including some parameters specified at runtime. For example, if it is mysqld, it includes basedir = =, datadir = =, port =, socket =, etc.
  • /proc/[pid]/cwd: current working directory is the working directory of the current process, including all sub directories.
  • /proc/[pid]/environ: this file saves the environment variables of the process. Each item is separated by a null character, and the end may also be a null character.
  • /exe: a symbolic connection of the actual executing program, pointing to the binary code to be executed.
  • /fd: each file opened by the process has a symbolic connection in the subdirectory, which is named after the file descriptor (number in parentheses). This name is actually a symbolic connection to the real file (the same as the exe record). For example, 0 is standard input, 1 is standard output, 2 is standard error, and so on. As follows:
  • /proc/[pid]/stack shows the kernel call stack information of the current process. Only config is opened during kernel compilation_ The stacktrace compilation option will generate this file.
  • /proc/[pid]/syscall displays the system call being executed by the current process. Config was turned on when the kernel was compiled_ HAVE_ ARCH_ The tracehook compilation option will generate this file.
  • /proc/[pid]/wchan displays the functions currently running in the kernel when the process sleep s.
  • /proc/[pid]/stat|status: the information to be represented by the two is the same. All CPU active information of the process, and all values in this file are accumulated from the start of system startup to the current time. But the latter is given in a format that is easy to read.
cat status
Name:	ibus-engine-lib
Umask:	0002
State:	S (sleeping)
Tgid:	3402
Ngid:	0
Pid:	3402
PPid:	1867
TracerPid:	0
Uid:	1000	1000	1000	1000
Gid:	1000	1000	1000	1000
FDSize:	64
Groups:	4 24 27 30 46 116 126 127 1000 
NStgid:	3402
NSpid:	3402
NSpgid:	1867
NSsid:	1419
VmPeak:	  295844 kB
VmSize:	  295844 kB
VmLck:	       0 kB
VmPin:	       0 kB
VmHWM:	    9984 kB
VmRSS:	    9984 kB
RssAnon:	    1100 kB
RssFile:	    8884 kB
RssShmem:	       0 kB
VmData:	   25960 kB
VmStk:	     132 kB
VmExe:	     780 kB
VmLib:	   14280 kB
VmPTE:	     200 kB
VmSwap:	       0 kB
HugetlbPages:	       0 kB
CoreDumping:	0
THP_enabled:	1
Threads:	4
SigQ:	0/126699
SigPnd:	0000000000000000
ShdPnd:	0000000000000000
SigBlk:	0000000000000000
SigIgn:	0000000001001000
SigCgt:	0000000180004002
CapInh:	0000000000000000
CapPrm:	0000000000000000
CapEff:	0000000000000000
CapBnd:	0000003fffffffff
CapAmb:	0000000000000000
NoNewPrivs:	0
Seccomp:	0
Speculation_Store_Bypass:	thread vulnerable
Cpus_allowed:	ffff
Cpus_allowed_list:	0-15
Mems_allowed:	00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list:	0
voluntary_ctxt_switches:	36
nonvoluntary_ctxt_switches:	0
  • /proc/[pid]/maps|smaps: all reflect the complete distribution of processes in memory at runtime. This is a complete list. The latter is given in a more detailed format, through which you can see all the memory information associated with the corresponding process (including shared and private), as follows:
$ sudo cat /proc/3402/smaps
55eed4630000-55eed46f3000 r-xp 00000000 103:0b 2366159                   /usr/lib/ibus/ibus-engine-libpinyin
Size:                780 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                 704 kB
Pss:                 704 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       704 kB
Private_Dirty:         0 kB
Referenced:          704 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
FilePmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:		0
VmFlags: rd ex mr mw me dw sd 

For the memory actually occupied by the process, / proc/[pid]/status, VmSize (the total memory currently occupied by the process, including applied unused and shared library memory) is exaggerated because it contains repeated statistics and unused memory, VmHWM / VmRss (maximum time / size of physical memory currently used by the application) is a relatively ideal basis for memory evaluation. To get a certain amount of memory usage, use all private in / proc/[pid]/smap_ Clean (non rewritten private page) and Private_Dirty (non rewritten private pages) adds up to a good solution.

In order to control the length, other fields will not be explained here. Interested partners can refer to appendix 11.

2. Thread related commands

There are few thread related system commands. Here are three for your reference.

2.1 ps related thread commands

  1. ps -T: in the ps command, the "- T" option enables thread viewing. The following command lists all threads created by the process with process number:
$ ps -T -p 2779 | head -5
 2779  2779 tty1     03:12:51 firefox
 2779  2785 tty1     00:00:00 gmain
 2779  2786 tty1     00:00:43 gdbus
 2779  2791 tty1     06:57:27 IPC I/O Parent

The "SPID" column indicates the thread ID, while the "CMD" column shows the thread name.

  1. ps -Lf pid: view the lwp number of the specified process (light weight process).
$ ps -Lf 2779 | head -5
shaw      2779     1  2779  0  239  2021 tty1     Sl+  263:29 /usr/lib/firefox/firefox -ne
shaw      2779     1  2785  0  239  2021 tty1     Sl+    0:00 /usr/lib/firefox/firefox -ne
shaw      2779     1  2786  0  239  2021 tty1     Sl+    1:11 /usr/lib/firefox/firefox -ne
shaw      2779     1  2791  1  239  2021 tty1     Sl+  561:43 /usr/lib/firefox/firefox -ne
shaw      2779     1  2900  0  239  2021 tty1     Sl+   11:56 /usr/lib/firefox/firefox -ne
  1. ps xH: you can view all existing threads, and each line represents a thread.
  2. ps -mp: This allows you to view the number of threads in a process.

2.2 top -H display thread

Add the - H option when starting top, and a thread is displayed in the top line. Otherwise, it shows a process on one line. Of course, you can also specify the pid display thread:

top - 21:48:03 up 31 days,  1:17,  1 user,  load average: 0.45, 0.56, 0.54
Threads: 253 total,   0 running, 253 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2.4 us,  0.6 sy,  0.0 ni, 96.9 id,  0.0 wa,  0.0 hi,  0.2 si,  0.0 st
KiB Mem : 32485348 total, 13764432 free, 11553884 used,  7167032 buff/cache
KiB Swap: 15999996 total, 15999996 free,        0 used. 18968152 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                                                                                                                         
 2791 shaw      20   0 59.682g 0.013t 0.010t S  1.7 43.3 417:41.50 IPC I/O Parent                                                                                                                                  
 2953 shaw      20   0 59.682g 0.013t 0.010t S  1.3 43.3 359:26.11 IPDL Background                                                                                                                                 
 2929 shaw      20   0 59.682g 0.013t 0.010t S  0.7 43.3 357:18.29 GLXVsyncThread                                                                                                                                  
 2779 shaw      20   0 59.682g 0.013t 0.010t S  0.3 43.3 193:28.88 firefox                                                                                                                                         
19477 shaw      20   0 59.682g 0.013t 0.010t S  0.3 43.3   1:37.65 Indexed~ #14341            

Note the Threads in the second line, indicating that Threads are displayed at this time. When this is Tasks, it indicates the process interface.

2.3 top's upgraded version of htop

To be honest, I don't see how helpful the upgraded version is to the viewing thread. The use methods listed here are for reference only. Display after installing htop. The command is as follows:

$ apt install htop
$ htop

For detailed usage of htop, please refer to appendix 12.

Summary: this chapter summarizes the process and thread related system commands. The next chapter goes further and explains the method of viewing the source code of process and thread system, and gives the source code of libgo, the selected collaborative library.


  1. ps command details
  2. How to evaluate the memory space occupied by a process
  3. Detailed explanation of files in / proc directory in Linux
  4. linux proc/pid / information description
  5. Linux system status command htop most detailed explanation (no one!)

Keywords: Linux Autonomous vehicles apollo ps top

Added by freebie on Wed, 02 Feb 2022 19:35:26 +0200