yarn PnP support not working

[zsalzbank]

Describe the bug

I upgraded to v8.0.0 from v7.5.0. When I try to run node-pg-migrate I get the following error:

Error: node-pg-migrate tried to access dotenv, but it isn't declared in its dependencies; this makes the require call ambiguous and unsound.

Required package: dotenv (via "dotenv/package.json")
Required by: node-pg-migrate@virtual:e298dc21ee497008eb9a0ebbe4a77b90443e0bab900aeeee3d636fad7ae3e60b61bd41079ed9d02f79f9898511e9fbf35388a339162608abfefebd9574b5c251#npm:8.0.0 (via REDACTED/.yarn/__virtual__/node-pg-migrate-virtual-fb0f6e4d60/0/cache/node-pg-migrate-npm-8.0.0-609b1b273d-2e9b854683.zip/node_modules/node-pg-migrate/bin/node-pg-migrate.js)

    at makeError (REDACTED/.pnp.cjs:38584:34)
    at resolveToUnqualified (REDACTED/.pnp.cjs:40224:21)
    at Object.resolveToUnqualified (REDACTED/.pnp.cjs:40404:26)
    at resolve$1 (file://REDACTED/.pnp.loader.mjs:2031:31)
    at nextResolve (node:internal/modules/esm/hooks:866:28)
    at Hooks.resolve (node:internal/modules/esm/hooks:304:30)
    at MessagePort.handleMessage (node:internal/modules/esm/worker:196:24)
    at [nodejs.internal.kHybridDispatch] (node:internal/event_target:825:20)
    at MessagePort.<anonymous> (node:internal/per_context/messageport:23:28)

To get around this (and other similar errors for other dependencies), I had to add to my .yarnrc.yml file as follows:

packageExtensions:
  node-pg-migrate@*:
    dependencies:
      dotenv: '*'
      config: '*'
      'dotenv-expand': '*'

Steps to reproduce

1. Start a new project using yarn PnP.
2. Install node-pg-migrate.
3. Try to run it.

Logs

No response

System Info

System:
    OS: Linux 6.8 Ubuntu 22.04.5 LTS 22.04.5 LTS (Jammy Jellyfish)
    CPU: (14) x64 Intel(R) Core(TM) Ultra 7 165U
    Memory: 19.30 GB / 30.80 GB
    Container: Yes
    Shell: 5.1.16 - /bin/bash
  Binaries:
    Node: 20.13.1 - ~/.nvm/versions/node/v20.13.1/bin/node
    Yarn: 4.7.0 - ~/.nvm/versions/node/v20.13.1/bin/yarn
    npm: 10.5.2 - ~/.nvm/versions/node/v20.13.1/bin/npm

[brenoepics]

I still need to reproduce this, but I think it happens because dotenv is a devDependency and not a dependency, peerDependency or optionalDependency?

node-pg-migrate/package.json

Line 105 in e37aa21

"dotenv": "16.5.0",

[brenoepics]

Ok, I have reproduced the issue at https://github.com/brenoepics/yarn-pgm-poc

Error: node-pg-migrate tried to access dotenv, but it isn't declared in its dependencies; this makes the require call ambiguous and unsound.
Error: node-pg-migrate tried to access dotenv-expand, but it isn't declared in its dependencies; this makes the require call ambiguous and unsound.
Error: node-pg-migrate tried to access config, but it isn't declared in its dependencies; this makes the require call ambiguous and unsound.

Yarn Berry (Yarn 2+) enforces strict dependency declarations. If node-pg-migrate uses packages like config, dotenv, or dotenv-expand but doesn't declare them as dependencies, users must add them manually to their own package.json.

You can fix it by adding the required packages to your dependencies:

yarn add config dotenv dotenv-expand

https://yarnpkg.com/features/pnp
https://yarnpkg.com/features/pnp#how-can-i-fix-ghost-dependencies

Maybe it would help Yarn 2+ users if we either list them as dependencies, document or add to the known ghost dependencies list?

EDIT:

It looks like bun also got issues with this, while looking at #1397 I tried

bunx --bun migrate up

$ node-pg-migrate --config-file ./test.json --tsconfig ./tsconfig.json up
error: Cannot find package 'dotenv' from '/project/node_modules/node-pg-migrate/bin/node-pg-migrate.js'
error: script "migrate" exited with code 1

[zsalzbank]

You can fix it by adding the required packages to your dependencies

yarn add config dotenv dotenv-expand

If I remember correctly from when I was trying to fix it, I don't think this works. You need to explicitly add it as it says for the ghost dependencies.

[zsalzbank]

Or add them as actual dependencies in the node-pg-migrate package, which makes more sense to me since they truly are dependencies. The ghost dependencies file is just a workaround for packages that have broken definitions as far as I understand.

[brenoepics]

If I remember correctly from when I was trying to fix it, I don't think this works. You need to explicitly add it as it says for the ghost dependencies.

Yeah, likely works for npm run migrate but if you use programmatically u need to do ghost deps way.

Or add them as actual dependencies in the node-pg-migrate package, which makes more sense to me since they truly are dependencies. The ghost dependencies file is just a workaround for packages that have broken definitions as far as I understand.

I agree, having ghost dependencies are not good either since this makes it difficult to analyze the supply chain/dependency graph and we end up using something that is not in the dependencies, I think the ideal would be to declare these 3 dependencies as necessary at runtime since we actually use them.

[brenoepics]

e.g. in my PoC repository, I don't know that dotenv, dotenv-expand and config are transitive dependencies from node-pg-migrate

but I can see yargs as transitive.

[Shinigami92]

dotenv is a optional thing in the CLI

node-pg-migrate/bin/node-pg-migrate.ts

Line 263 in e37aa21

const dotenv = await tryImport<typeof import('dotenv')>('dotenv');

but yargs is a must have dependency in the CLI

node-pg-migrate/bin/node-pg-migrate.ts

Line 83 in e37aa21

const parser = yargs(process.argv.slice(2))

[brenoepics]

I see. What if we go with the optionalDependencies approach? Does that make sense?
To be honest, I’ve never used it before, so I’m not entirely sure.

@zsalzbank, do you think it helps with ghost dependencies? If I understood the docs correctly, it does.

Docs: https://docs.npmjs.com/cli/v11/configuring-npm/package-json#optionaldependencies

EDIT:
not sure if it makes sense but I also wanted to note this
https://docs.npmjs.com/cli/v11/configuring-npm/package-json#peerdependenciesmeta

[zsalzbank]

I tried the following in my .yarnrc.yml file and got the same problem, so not sure if this would solve it or not:

packageExtensions:
  node-pg-migrate@*:
    peerDependencies:
      dotenv: '*'
      config: '*'
      'dotenv-expand': '*'
    peerDependenciesMeta:
      dotenv:
        optional: true
      config:
        optional: true
      'dotenv-expand':
        optional: true

This seems to be how I would add optional dependencies according to https://yarnpkg.com/configuration/yarnrc#packageExtensions.

Error: node-pg-migrate tried to access dotenv (a peer dependency) but it isn't provided by your application; this makes the require call ambiguous and unsound.

Required package: dotenv (via "dotenv/package.json")
Required by: node-pg-migrate@virtual:e298dc21ee497008eb9a0ebbe4a77b90443e0bab900aeeee3d636fad7ae3e60b61bd41079ed9d02f79f9898511e9fbf35388a339162608abfefebd9574b5c251#npm:8.0.0

[Shinigami92]

@zsalzbank I'm bit confused be the "Error" message, as it is more like a correct warning. If you do not have the optional dependency installed in your package.json, then it will be not consumed in node-pg-migrate's CLI. And this is the wanted behavior. 🤔
So to me it looks like yarn is a bit to strict about it 🤔