At the beginning of the establishment of the team, we used npm3 to manage project dependency. Later, we developed our own component library, chart library and tool library, and adopted monorepo management, Dependency management is also switched from npm3 to yarn (yarn workspace). Both npm3 and yarn adopt the flat node_modules folder to avoid the problems of too deep hierarchy and repeated versions of the same dependencies.

With the continuous growth of the company's business, more and more projects are supported by the team. Because the dependency follows the project, it takes up a lot of disk space.

For the above reasons, I began to try to use pnpm for management.

Save disk space

Pnpm dependencies will be stored in a global content addressable repository (${OS. Homedir}) /. pnpm store), the dependencies used in specific projects are hard linked rather than copied. Keep only one copy of each version of each module. For example, 10 local projects depend on the same vue version. If npm or yarn is used, the local disk needs to have 10 copies of vue; There is only one pnpm.

1. If you use different versions of a dependency, only the different files will be added to the warehouse (public warehouse).
2. All files are stored in the same location on the hard disk. When multiple package s are installed, all files will be hard linked from the same location without taking up additional disk space. This allows the same version of dependencies to be shared across projects.

$ pnpm install

Packages: +1585
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Packages are hard linked from the content-addressable store to the virtual store.
  Content-addressable store is at: /Users/ligang/.pnpm-store/v3
  Virtual store is at:             node_modules/.pnpm
Progress: resolved 1585, reused 1585, downloaded 0, added 1585, done

It can be found that:

• Content is addressable and stored in / users / Ligang / pnpm-store/v3
• Virtual storage directory node_modules/.pnpm
• downloaded 0, which greatly improves the install ation speed

ll node_modules

lrwxr-xr-x   1 ligang  staff    44B  9  1 17:59 deepmerge -> .pnpm/deepmerge@3.3.0/node_modules/deepmerge
lrwxr-xr-x   1 ligang  staff    72B  9  1 17:59 element-resize-detector -> .pnpm/element-resize-detector@1.2.2/node_modules/element-resize-detector
lrwxr-xr-x   1 ligang  staff    58B  9  1 17:59 element-ui -> .pnpm/element-ui@2.13.1_vue@2.6.12/node_modules/element-ui
lrwxr-xr-x   1 ligang  staff    39B  9  1 17:59 eslint -> .pnpm/eslint@5.16.0/node_modules/eslint

node_ All files in the modules directory are soft linked to the virtual storage path pnpm. . Pnpm / stores all packages in the form of tiles (Format:. pnpm/@/node_modules /). Packages in the. Pnpm directory will be hard linked to the global warehouse (/ users / Ligang /. Pnpm store / V3).

You can view "hard chain" and "soft chain" Part I Blog.

Take the dependent element UI in the project as an example:

cd node_modules

ls -li element-ui
8643474522 lrwxr-xr-x  1 ligang  staff  58  9  1 17:59 element-ui -> .pnpm/element-ui@2.13.1_vue@2.6.12/node_modules/element-ui

ls -li .pnpm/element-ui@2.13.1_vue@2.6.12/node_modules/
8643424956 drwxr-xr-x  13 ligang  staff  416  9  1 17:59 element-ui

node_ In the modules directory, the element UI is soft linked Pnpm corresponding directory element UI Under the pnpm directory, element UI is a hard link (link count 13).

Non flattened node_modules folder

Go back to node_modules structure history:

Phase I: npm@3 Previous version

node_modules
└─ foo
   ├─ index.js
   ├─ package.json
   └─ node_modules
      └─ bar
         ├─ index.js
         └─ package.json

• If the dependency tree is too deep, the directory path on Windows will be too long
• When the same package is needed in different dependencies, there will be multiple copies of the same package

Phase II: npm@3 Version, flattening

It mainly solves the above two problems

node_modules
├─ foo
|  ├─ index.js
|  └─ package.json
└─ bar
   ├─ index.js
   └─ package.json

Phase III: pnpm

Because the flattening algorithm is extremely complex, and there will be multiple items that depend on the same copy. pnpm gave up flattening node when trying to solve these problems_ Modules. Instead, hard chain + soft chain is adopted.

node_modules
├─ .pnpm
|  ├─ foo@1.0.0/node_modules/foo
|  |  └─ index.js
|  └─ bar@2.0.0/node_modules/bar
├─ foo -> .pnpm/foo@1.0.0/node_modules/foo
└─ bar -> .pnpm/bar@2.0.0/node_modules/bar

node_ The package in the modules root directory is just a symbolic link. require('foo ') will execute node_ modules/. pnpm/ foo@1.0.0 /node_ The file in modules / foo / indexjs (here is a hard link), not the file in node_modules/foo/index.js.

benefit

A great advantage of this layout structure is that only packages that are really in dependencies (package. JSON dependencies) can be accessed. Using the flattened node_modules structure, all promoted packages can be accessed.

npm@3/yarn manages nodes in a flat way_ modules

Example: in chokidar take as an example

"dependencies": {
  "chokidar": "^3.5.2"
}

The project relies on chokidar to monitor the changes of folder contents, and the post installation structure through npm

So many dependent packages come from flattening. chokidar dependent packages and their dependent dependent packages are extracted into the primary directory. In this way, packages that are not explicitly dependent can also be referenced.

const isNumber = require('is-number')
console.log(isNumber(123), isNumber('abc'))

The above can be referenced normally!

Reinstall with pnpm

When the above code is executed, an error will be reported: Error: Cannot find module 'is number'

problem

Flattened node_modules caused the above error. If this situation exists, how should we handle it if we need to switch to pnpm?

Option 1:

Add dependency through pnpm add

Option 2:

Add related dependencies through related hooks

.pnpmfile.cjs

module.exports = {
  hooks: {
    readPackage: (pkg) => {
      if (pkg.name === "inspectpack") {
        pkg.dependencies['babel-traverse'] = '^6.26.0'
      }
      return pkg
    }
  }
}

Option 3:

If there are too many dependencies missing, you can use the promotion option. This option is not officially recommended.

pnpm install --shamefully-hoist

Because cli3 does not fully support pnpm (it is fully supported in cli4), we adopt this method Issue

summary

Implementation essence of pnpm mode

1. Through the form of soft chain, require can be referenced normally; At the same time, the projects that are not really dependent are isolated (to avoid confusion of reference dependency)
2. . The existence of pnpm avoids the problems of circular reference and too deep level (both at the first level)
3. Hard chain makes different projects have the same dependency, and only one copy exists, reducing disk space

Reference link

• https://www.kochan.io/nodejs/pnpms-strictness-helps-to-avoid-silly-bugs.html
• https://www.kochan.io/nodejs/why-should-we-use-pnpm.html
• https://github.com/vuejs/vue-cli/issues/2703
• https://pnpm.io/zh/faq