https://pact.io logo
Powered by
#pact-js
Title
# pact-js
d

Dany Marques

03/01/2024, 7:16 AM
Hell all, Quick question, in the migration guide for v12 it's written that we don't need node-gyp anymore. But I don't get why during the installation of the package, I still see some node-gyp stuff going on. Any idea? Basically, my installation fails if I don't provide --ignore-scripts
Copy code
~/Documents/mobi_projects/defant-taskclient-rwc git:[feature/WV-2439-angular-17-migration]
npm i -D @pact-foundation/pact
npm ERR! code 1
npm ERR! path /Users/danymarques/Documents/mobi_projects/defant-taskclient-rwc/node_modules/@pact-foundation/pact-core
npm ERR! command failed
npm ERR! command sh -c node-gyp rebuild
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp info using node-gyp@10.0.1
npm ERR! gyp info using node@18.18.0 | darwin | arm64
npm ERR! gyp info find Python using Python version 3.9.6 found at "/Library/Developer/CommandLineTools/usr/bin/python3"
npm ERR! gyp info spawn /Library/Developer/CommandLineTools/usr/bin/python3
npm ERR! gyp info spawn args [
npm ERR! gyp info spawn args '/Users/danymarques/Documents/mobi_projects/defant-taskclient-rwc/node_modules/node-gyp/gyp/gyp_main.py',
npm ERR! gyp info spawn args 'binding.gyp',
npm ERR! gyp info spawn args '-f',
npm ERR! gyp info spawn args 'make',
npm ERR! gyp info spawn args '-I',
npm ERR! gyp info spawn args '/Users/danymarques/Documents/mobi_projects/defant-taskclient-rwc/node_modules/@pact-foundation/pact-core/build/config.gypi',
npm ERR! gyp info spawn args '-I',
npm ERR! gyp info spawn args '/Users/danymarques/Documents/mobi_projects/defant-taskclient-rwc/node_modules/node-gyp/addon.gypi',
npm ERR! gyp info spawn args '-I',
npm ERR! gyp info spawn args '/Users/danymarques/Library/Caches/node-gyp/18.18.0/include/node/common.gypi',
npm ERR! gyp info spawn args '-Dlibrary=shared_library',
npm ERR! gyp info spawn args '-Dvisibility=default',
npm ERR! gyp info spawn args '-Dnode_root_dir=/Users/danymarques/Library/Caches/node-gyp/18.18.0',
npm ERR! gyp info spawn args '-Dnode_gyp_dir=/Users/danymarques/Documents/mobi_projects/defant-taskclient-rwc/node_modules/node-gyp',
npm ERR! gyp info spawn args '-Dnode_lib_file=/Users/danymarques/Library/Caches/node-gyp/18.18.0/<(target_arch)/node.lib',
npm ERR! gyp info spawn args '-Dmodule_root_dir=/Users/danymarques/Documents/mobi_projects/defant-taskclient-rwc/node_modules/@pact-foundation/pact-core',
npm ERR! gyp info spawn args '-Dnode_engine=v8',
npm ERR! gyp info spawn args '--depth=.',
npm ERR! gyp info spawn args '--no-parallel',
npm ERR! gyp info spawn args '--generator-output',
npm ERR! gyp info spawn args 'build',
npm ERR! gyp info spawn args '-Goutput_dir=.'
npm ERR! gyp info spawn args ]
npm ERR! gyp: binding.gyp not found (cwd: /Users/danymarques/Documents/mobi_projects/defant-taskclient-rwc/node_modules/@pact-foundation/pact-core) while trying to load binding.gyp
npm ERR! gyp ERR! configure error 
npm ERR! gyp ERR! stack Error: `gyp` failed with exit code: 1
npm ERR! gyp ERR! stack at ChildProcess.<anonymous> (/Users/danymarques/Documents/mobi_projects/defant-taskclient-rwc/node_modules/node-gyp/lib/configure.js:271:18)
npm ERR! gyp ERR! stack at ChildProcess.emit (node:events:517:28)
npm ERR! gyp ERR! stack at ChildProcess._handle.onexit (node:internal/child_process:292:12)
npm ERR! gyp ERR! System Darwin 23.2.0
npm ERR! gyp ERR! command "/Users/danymarques/.nvm/versions/node/v18.18.0/bin/node" "/Users/danymarques/Documents/mobi_projects/defant-taskclient-rwc/node_modules/.bin/node-gyp" "rebuild"
npm ERR! gyp ERR! cwd /Users/danymarques/Documents/mobi_projects/defant-taskclient-rwc/node_modules/@pact-foundation/pact-core
npm ERR! gyp ERR! node -v v18.18.0
npm ERR! gyp ERR! node-gyp -v v10.0.1
npm ERR! gyp ERR! not ok

npm ERR! A complete log of this run can be found in: /Users/danymarques/.npm/_logs/2024-02-29T17_44_16_466Z-debug-0.log
m

Matt (pactflow.io / pact-js / pact-go)

03/01/2024, 7:23 AM
Any chance you could share a bit more on the Mac OS you're on? Looks like M1
Annoying as we have pre built the binaries so strange it wants to check if python exists. Maybe it's still a dependency of the prebuilds package during startup it does a preflight check
I'll try installing by removing python from my environment
d

Dany Marques

03/01/2024, 7:33 AM
Yes sure. I'm using a M3 Max with Mac Os Sonoma 14.2.1
Something I don't get, it only happens on our private registry.
We've 2 private registries. One in azure and a local nexus
And if I get the package from the cloud, it works
I also downloaded the packages manually from both sources, and they have exactly the same size
m

Matt (pactflow.io / pact-js / pact-go)

03/01/2024, 9:00 AM
What version of pact core is cached?
When you say cloud, do you mean npm?
It should only attempt a build if the pre-built binaries aren't there. Could your cached registry versions exclude those files? Could you please share a dir listing with scripts disabled? What does an install look like the verbose logging?
d

Dany Marques

03/01/2024, 10:34 AM
pact core 14.3.0
<When you say cloud, do you mean npm?> No, we have a private registry on Azure artifacts.
From where are the pre-built binaries coming? From a build like in a postinstall script or am I downloading them with npm?
What do you mean by that: Could you please share a dir listing with scripts disabled?
m

Matt (pactflow.io / pact-js / pact-go)

03/01/2024, 10:48 AM
From where are the pre-built binaries coming? From a build like in a postinstall script or am I downloading them with npm?
they should be in the npm package
I’ll show you what I mean. My hypothesis is that it’s falling back to 
node-gyp
because the prebuilt native extensions are not on the file system. Why aren’t the on the file system, I’m not sure yet
Copy code
npm i @pact-foundation/pact-core@latest
find ./node_modules/@pact-foundation/pact-core > files.txt
Then attach files.txt here
Or perhaps as a starter, run this command:
Copy code
find ./node_modules/@pact-foundation/pact-core | grep "\.node"
It should print:
Copy code
./node_modules/@pact-foundation/pact-core/prebuilds/linux-arm64/node.napi.node
./node_modules/@pact-foundation/pact-core/prebuilds/darwin-x64/node.napi.node
./node_modules/@pact-foundation/pact-core/prebuilds/win32-x64/node.napi.node
./node_modules/@pact-foundation/pact-core/prebuilds/darwin-arm64/node.napi.node
./node_modules/@pact-foundation/pact-core/prebuilds/linux-x64/node.napi.node
Those .node files are the native extensions pre built and packaged into the distro
y

Yousaf Nabi (pactflow.io)

03/01/2024, 11:42 AM
you can see the prebuilds folder in the code tab of npm with the prebuilt binaries https://www.npmjs.com/package/@pact-foundation/pact-core?activeTab=code is this a fresh install of pact-core or an upgrade? the binding.gyp file is removed from the published package, but might be retained if upgrading from an older version of pact-core which still contained it ( when users would have to build the bindings locally )
hmm i wonder if we need to suppress node gyp when users dont run with ignore scripts, node-gyp-build will try and execute its own install command. something like 
"install": "echo welcome to Pact-JS - This post install script supresses node-gyp rebuild as it is not required for this package"
odd how you only get the issue from your internal sources
d

Dany Marques

03/01/2024, 2:15 PM
So, after running with --ignore-scripts (Otherwise I've nothing in the node_modules), that's the output of the command:
Copy code
find ./node_modules/@pact-foundation/pact-core | grep "\.node" 
./node_modules/@pact-foundation/pact-core/prebuilds/linux-arm64/node.napi.node
./node_modules/@pact-foundation/pact-core/prebuilds/darwin-x64/node.napi.node
./node_modules/@pact-foundation/pact-core/prebuilds/win32-x64/node.napi.node
./node_modules/@pact-foundation/pact-core/prebuilds/darwin-arm64/node.napi.node
./node_modules/@pact-foundation/pact-core/prebuilds/linux-x64/node.napi.node
It is a fresh install
I'm clueless
That's the full log when I try to install it with the private registry Nexus
onprem.txt
onprem.txt
And that's the one when I do exactly the same but from another registry, also private.
cloud.txt
cloud.txt
y

Yousaf Nabi (pactflow.io)

03/01/2024, 2:37 PM
can you show the same pulled directly from npm? very odd, nothing is shouting out at me from the logs
d

Dany Marques

03/01/2024, 2:48 PM
npmjs.txt
npmjs.txt
Here we go
y

Yousaf Nabi (pactflow.io)

03/01/2024, 2:55 PM
the cloud.txt one looks ok
Copy code
added 75 packages in 45s

268 packages are looking for funding
run `npm fund` for details
npm verb exit 0
npm info ok
its from this source on on-prem
Copy code
npm http fetch GET 200 <https://nexus.mobicorp.ch:8443/nexus/repository/npm-mobi/@pact-foundation%2fpact> 219ms (cache hit)
which then does the node-gyp
Copy code
pm info run @pact-foundation/pact-core@14.3.0 install node_modules/@pact-foundation/pact-core node-gyp rebuild
npm info run @pact-foundation/pact-core@14.3.0 install { code: 1, signal: null }
npm verb stack Error: command failed
npm verb stack     at ChildProcess.<anonymous> (/Users/danymarques/.nvm/versions/node/v18.18.0/lib/node_modules/npm/node_modules/@npmcli/promise-spawn/lib/index.js:53:27)
npm verb stack     at ChildProcess.emit (node:events:517:28)
npm verb stack     at maybeClose (node:internal/child_process:1098:16)
npm verb stack     at ChildProcess._handle.onexit (node:internal/child_process:303:5)
npm verb pkgid @pact-foundation/pact-core@14.3.0
npm verb cwd /Users/danymarques/Documents/mobi_projects/defant-taskclient-rwc
npm verb Darwin 23.2.0
npm verb node v18.18.0
npm verb npm  v9.8.1
npm ERR! code 1
npm ERR! path /Users/danymarques/Documents/mobi_projects/defant-taskclient-rwc/node_modules/@pact-foundation/pact-core
npm ERR! command failed
npm ERR! command sh -c node-gyp rebuild
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp verb cli [
npm ERR! gyp verb cli '/Users/danymarques/.nvm/versions/node/v18.18.0/bin/node',
npm ERR! gyp verb cli '/Users/danymarques/Documents/mobi_projects/defant-taskclient-rwc/node_modules/.bin/node-gyp',
npm ERR! gyp verb cli 'rebuild'
npm ERR! gyp verb cli ]
npm ERR! gyp info using node-gyp@10.0.1
can you check the contents of that package? can you view the source in the private reg before pulling it?
d

Dany Marques

03/01/2024, 3:36 PM
I can yes
I mean, I can download it directly from the registry
With the browser
pact-12.3.0.tgz
pact-12.3.0.tgz
Any idea? @Matt (pactflow.io / pact-js / pact-go) @Yousaf Nabi (pactflow.io)
m

Matt (pactflow.io / pact-js / pact-go)

03/04/2024, 1:22 AM
That package looks OK to me. The issue would be in the 
pact-core
package. Any chance you could download it directly from the registries and upload that package? I’m also wondering if you have any relevant 
npm config
items that could impact this?
Otherwise I’m a bit stumped
d

Dany Marques

03/04/2024, 7:04 AM
Hi @Matt (pactflow.io / pact-js / pact-go)
Here we go
pact-core-14.3.0.tgz
pact-core-14.3.0.tgz
I don't find any "postinstall" script or something like thi
Where do you check if the binaries are already there otherwise you invoke node-gyp?
m

Matt (pactflow.io / pact-js / pact-go)

03/04/2024, 7:08 AM
I’d need to check, I think it’s in the 
node-gyp-build
package but perhaps it’s elsewhere.
hmm the package looks correct too
so it seems like something else is trying to do it
d

Dany Marques

03/04/2024, 7:14 AM
Do we still need de depend on node-gyp-build even though we don't compile anymore?
m

Matt (pactflow.io / pact-js / pact-go)

03/04/2024, 7:16 AM
Does this happen to just you, or other developers also? Does it only happen on Mac OSX or does it e.g. fail on CI linux/docker/etc.?
I’m also wondering if perhaps a different npm version resolves the issue
d

Dany Marques

03/04/2024, 7:22 AM
It also happens to others. My colleague is using WSL2 with Ubuntu and has the same issue.
I already tried with Node 18 and 20.
m

Matt (pactflow.io / pact-js / pact-go)

03/04/2024, 7:24 AM
It’s this line that is confusing me:
Copy code
npm info run @pact-foundation/pact-core@14.3.0 install node_modules/@pact-foundation/pact-core node-gyp rebuild
All I can assume is that 
npm
somehow thinks it needs to be re-built and calls out to its mate 
node-gyp
. Nowhere in our code do we have an 
install
or 
pre/post
script that calls this.
Any chance you could see what version of 
node-gyp
you have on your OS? Maybe that’s the problem (It comes with node()
d

Dany Marques

03/04/2024, 7:38 AM
With Node 18, I've node-gyp 10.0.1
And it looks like I don't have node-gyp with Node 20. Strange
image.png
But in the log it's written:
Copy code
'-Dnode_gyp_dir=/Users/danymarques/Documents/mobi_projects/lcs-cas-rwc/node_modules/node-gyp',
So to me, it tries to install node-gyp when installing pact, and it fails
Just tried couple of more things. I installed node-gyp and node-gyp-build directly in the project in order to check if these packages are in fault or if we have a problem with pact.
Both installations worked perfectly
Now I installed pact with --ignore-scripts
And then in the pact-core dir, I ran:
Copy code
sh -c "node-gyp rebuild"
And it fails with the same error
Will now check if this also happens with the other registry
Ok, same error.
So now it's clear that sometimes, we try to invoke
Copy code
node-gyp rebuild
and sometimes we don't.
Now we've two choices, either we try to find out why we are not able to rebuild pact with node-gyp, either we try to find out why we sometimes invoke node-gyp rebuild
WDYT @Matt (pactflow.io / pact-js / pact-go)?
m

Matt (pactflow.io / pact-js / pact-go)

03/04/2024, 8:52 AM
When you say if you install node gyp and the build variant directly and it "works perfectly" do you mean pact works or the install of the node gyp packages works? Because the latter is expected. They don't need to rebuild. It's only applicable for a project that has native extensions
(like Pact)
d

Dany Marques

03/04/2024, 9:18 AM
I mean the install of both packages worked
I tried to install directly pact-core and it fails
m

Matt (pactflow.io / pact-js / pact-go)

03/04/2024, 9:40 AM
so I think we need to: 1. Identify what are the triggers of that call - it’s not clear from the 
npm i
why that is 2. Once we know the triggers, understand what’s different about your environment that reliably triggers it
Just thinking, and probably it won’t help, but worth a shot. Could you please try a fresh install of 
@pact-foundation/pact-core
at loglevel 
silly
?
Now we’ve two choices, either we try to find out why we are not able to rebuild pact with node-gyp, either we try to find out why we sometimes invoke node-gyp rebuild
I think this problem is obvious - we don’t distribute 
binding.gyp
. @Yousaf Nabi (pactflow.io) might remember why we removed that, probably because we prebuild for most architectures (and maybe it was misfiring the node-gyp process. We could add that back, possibly. But that would make install slower, even though the prebuilds are there. It’s reliably a problem for your org, so my suspicion is there is something about your setup we can’t understand that’s causing it. 
npm config list
and 
npm config -g list
might be enlightening
Are there any 
.npmrc
files checked in that might impact this?
e.g. the following flag might be set for that registry
Copy code
--build-from-source
y

Yousaf Nabi (pactflow.io)

03/04/2024, 11:02 AM
if we store the binding.gyp in the published package of pact-js-core, we have to suppress node gyp from running, as it doesn’t detect the prebuilds and will always run unless ignore-scripts is set
you cant run node-gyp rebuild in the project as it doesn’t have the required files so that wont work tbh if the issue only exists in a private mirror then I would politely suggest the issue isn’t probably with Pact but with your mirror provider
your report is unwieldy now and hard to follow, and we are also unable to reproduce. i would raise an issue with clear steps of how one can reproduce this
Now we've two choices, either we try to find out why we are not able to rebuild pact with node-gyp, either we try to find out why we sometimes invoke node-gyp rebuild
the latter is relevant but only to your private reggo it seems. the former is a non starter.
node-gyp-build is only depended on in the published project to load the pre built bindings, see line 2 import https://github.com/pact-foundation/pact-js-core/blob/master/src/ffi/index.ts prior to the release with the prebuilds, users had to build the node bindings on their local machines, which used node-gyp for two steps, one to create the bindings and one to load them.
m

Matt (pactflow.io / pact-js / pact-go)

03/04/2024, 11:14 AM
Yeah. I can't see how it would want to compile at install time. It obviously doesn't need to at run time because when you install with ignore scripts set, it works (meaning the prebuilds are correct)
y

Yousaf Nabi (pactflow.io)

03/04/2024, 11:21 AM
i did think about vendoring the bindings function to avoid any prod dep on node gyp build. the other alternative is considered is pulling the arch platform specific binding from the gh release page on install, and allowing the user to provide their own. still gives the prebuild func, with smaller bundles, but wont work out the box for those in locked down envs but they can provide their own bindings release from gh
basically i love the prebuilds, dislike package size, still want to cater for those on locked down networks
d

Dany Marques

03/04/2024, 4:21 PM
I ran the install with silly loglevel
The only thing I can spot there is
Copy code
npm info run @pact-foundation/pact-core@14.3.0 install node_modules/@pact-foundation/pact-core node-gyp rebuild
I tried something different, I downloaded pact-core from my two private registries and installed both packages and both work.
So the error only happens when downloading and installing from one of the registries
I just cloned pact-core repo but I'm not able to run
Copy code
npm i
Is it normal?
y

Yousaf Nabi (pactflow.io)

03/04/2024, 4:56 PM
d

Dany Marques

03/04/2024, 4:57 PM
But again, why this triggers node-gyp rebuild?
That's exactly the same error I've in the project
y

Yousaf Nabi (pactflow.io)

03/04/2024, 4:59 PM
because there is a binding.gyp in that project
as per those instructions
it is removed from the end product, also stated in those docs
you can run npm install but you’d need to download the shared libs first (first step) which means the binding.gyp will reference and build them. This is done is a seperate prebuild step now. I really think the issue is with your private registry
d

Dany Marques

03/04/2024, 5:04 PM
Yes me too
But I don't get what can be tbh
y

Yousaf Nabi (pactflow.io)

03/04/2024, 5:04 PM
Raise a support ticket with them?
d

Dany Marques

03/04/2024, 5:12 PM
I already did.
They told me that the the registry is a simple mirror from npmjs
They cleaned the cache and resetted the indexes
They can't do more
y

Yousaf Nabi (pactflow.io)

03/04/2024, 5:13 PM
well this works
Copy code
docker run --rm -it node:20-slim sh -c 'cd /home && npm install @pact-foundation/pact-core'
as does this
Copy code
docker run --rm -it node:20-slim sh -c 'cd /home && npm install @pact-foundation/pact'
Not sure how we can do more, we don’t know what private registry you are using so we can’t replicate it. We don’t have access to your registry
If you can create a reproducible example, we can look to solve it, otherwise you’ll have to use 
--ignore-scripts
or retrieve from the working registry (you said you have two private, one works, one doesn’t)
d

Dany Marques

03/04/2024, 5:15 PM
I can reproduce but only on my machine as the registry is private
I also can't use --ignore-scripts because in the CI, we don't have this command
And I also can't use the other registry because it is not reachable from the CI
I just created a new project, with 0 dependencies. I just created the .npmrc file to set the registry and tried to install pact-core and it fails.
Gosh
I'm out of ideas
The thing is, the package in the registry is ok because when I download it manually and install it, it works.
y

Yousaf Nabi (pactflow.io)

03/04/2024, 5:25 PM
I’ve seen an issue noted about using an npmrc file https://github.com/pact-foundation/pact-js-core/issues/384 I’m not sure if its relevant
but the problem described, doesn’t seem the same, the only relevance is they have an npmrc and a failure to install (not that it was running node-gyp - this issue was pre prebuilds)
d

Dany Marques

03/04/2024, 5:27 PM
Hum, let me try something
Brb
That's not the issue. I kept the .npmrc but set the registry which works and installed again and it works
If you would like to check on my machine, we can have a quick call
m

Matt (pactflow.io / pact-js / pact-go)

03/05/2024, 12:50 AM
It probably got lost in the above, but the output of 
npm config list
and 
npm config -g list
might be enlightening Are there any 
.npmrc
files checked in that might impact this? e.g. the following flag might be set for that registry
Copy code
--build-from-source
Other than that, I’m out of ideas myself.
d

Dany Marques

03/05/2024, 5:50 AM
Copy code
~/Documents/mobi_projects/test_pact
npm config list
; "user" config from /Users/danymarques/.npmrc
; strict-ssl = false ; overridden by project

; "project" config from /Users/danymarques/Documents/mobi_projects/test_pact/.npmrc

//our.private.registry:8443/nexus/repository/npm-mobi/:_auth = (protected) 
always-auth = true 
registry = "<https://our.private.registry:8443/nexus/repository/npm-mobi/>" 
strict-ssl = false 

; node bin location = /Users/danymarques/.nvm/versions/node/v18.19.1/bin/node
; node version = v18.19.1
; npm local prefix = /Users/danymarques/Documents/mobi_projects/test_pact
; npm version = 10.2.4
; cwd = /Users/danymarques/Documents/mobi_projects/test_pact
; HOME = /Users/danymarques
; Run `npm config ls -l` to show all defaults.
m

Matt (pactflow.io / pact-js / pact-go)

03/05/2024, 6:43 AM
thx
nothing in the global config? Man, this is strange!
Honestly, I think we’ve exhausted the debugging we can do here in this thread. I’d suggest from here raising a tracking issue on Pact JS, ideally with a repro. Without a way of reproducing, I think you’re asking a bit much of this community to debug it any further - it’s just guessing and trial/error I’m afraid
d

Dany Marques

03/05/2024, 9:20 AM
That's really strange
I just compared all dependencies versions that are installed when I install pact-core from both registries
And they are exactly the same
m

Matt (pactflow.io / pact-js / pact-go)

03/05/2024, 9:36 AM
You didn’t share the global npm config, I don’t expect it to be different, but let’s just cross that off the list
I just compared all dependencies versions that are installed when I install pact-core from both registries
But yeah. I don’t think it’s that that’s the problem. something thinks Pact needs a rebuild, but I’m just not sure what. If only the logs indicated what is the initiator of that. I assume `silly` level logs didn’t reveal anything for you about the initiator?
d

Dany Marques

03/06/2024, 12:39 PM
The global npmrc looks like this:
Copy code
; begin auth token
//pkgs.dev.azure.com/diemobiliar/RWC/_packaging/npm-main/npm/registry/:email="xxx"
//pkgs.dev.azure.com/diemobiliar/RWC/_packaging/npm-main/npm/registry/:username="xxx"
//pkgs.dev.azure.com/diemobiliar/RWC/_packaging/npm-main/npm/registry/:_password="xxx"
//pkgs.dev.azure.com/diemobiliar/RWC/_packaging/npm-main/npm/:_password="xxx"
//pkgs.dev.azure.com/diemobiliar/RWC/_packaging/npm-main/npm/:username="xxx"
//pkgs.dev.azure.com/diemobiliar/RWC/_packaging/npm-main/npm/:email="xxx"
; end auth token

strict-ssl=false
registry="myregistry"
Here is the log of the install in silly loglevel
test.txt
I don't know if it helps, but pact-core@13.13.9 works
Starting from 14.0.0 it fails
Also opened an issue on node-gyp repo: https://github.com.mcas.ms/nodejs/node-gyp/issues/2988
Hello guys, I finally found the problem. Could we have a call @Yousaf Nabi (pactflow.io) or @Matt (pactflow.io / pact-js / pact-go)?
y

Yousaf Nabi (pactflow.io)

03/13/2024, 2:21 PM
Can you provide a recording? It is easier, as we are all in different timezones?
d

Dany Marques

03/13/2024, 3:00 PM
I would like to double check that my assumptions are correct
I reverse-engineered npm and found out how the different steps and whats happens inside them
And the problem is on the manifest you published to npmjs.com
I did this small script, inspired from the sources of npm, to check what you published:
Copy code
const pickManifest = require('npm-pick-manifest')

fetch('<https://registry.npmjs.org/@pact-foundation/pact-core').then(res> => {
  return res.json()
}).then(packument => {
  return pickManifest(packument, '^14.3.0')
}).then((x) => console.log(x.scripts))// get same manifest as npm would get if you `npm i npm-pick-manifest@^1.0.0`
If you run this (ofc you need to first install the 
npm-pick-manifest
dep), then you'll see that the install script is there
Now the question is why it works when I install this package from npmjs or from one of my private registries ?
npmjs and azure artifacts (one of my private registries) they somehow "mitigate" this issue by reading the manifest from the tarball and not the manifest sent to the registry hen publishing
But the other private registry (Nexus), does nothing to prevent the manifest confusion
So it uses the real manifest from npmjs and ofc, it fails!
I packed pact-core on my machine I the manifest looks good (it doesn't have the install script)
So I don't really get why this script appears on the manifest on npmjs
I'll provide the quick easy fix soon and raise a PR
Just have to pick up the train
Also just discovered this command: 
npm view @pact-foundation/pact-core@14.3.0 scripts
You can try with npmjs as a registry and you'll see that the install script is there 😉
y

Yousaf Nabi (pactflow.io)

03/13/2024, 4:25 PM
ahh that is a cheeky potential vuln! I wasn’t aware of that attack vector. given the pr a review. small tweak and we can get that merged
d

Dany Marques

03/13/2024, 4:25 PM
I found where the issue is in npm sources
Will raise a MR to npm also
Was an hard one
y

Yousaf Nabi (pactflow.io)

03/13/2024, 4:26 PM
legend! this is a blog post in the making dude
d

Dany Marques

03/13/2024, 4:26 PM
God took me so many hours
y

Yousaf Nabi (pactflow.io)

03/13/2024, 4:26 PM
@Liran Tal would love this
d

Dany Marques

03/13/2024, 4:26 PM
Sorry for the DS_Store, will fix now
y

Yousaf Nabi (pactflow.io)

03/13/2024, 4:27 PM
no worries man, thank you for the Herculean effort!
That is released now as 14.3.1 however this command
Copy code
npm view @pact-foundation/pact-core@14.3.1 scripts
still reveals 
install: 'node-gyp rebuild'
think up
Copy code
{
  clean: "rimraf '{src,test,bin,standalone}/**/*.{js,map,d.ts}' 'package.zip' '.tmp' 'tmp'",
  lint: 'eslint . --ext .ts --config .eslintrc',
  'lint:fix': 'npm run lint -- --fix',
  prebuild: 'npm run clean',
  'download-libs': 'npm run clean && bash script/download-libs.sh',
  'clean-libs': "rimraf standalone/*.{js,map,d.ts} standalone/{windows**,linux**,darwin**} 'ffi'",
  build: 'tsc --project tsconfig.build.json',
  prerelease: 'npm run snyk-protect',
  release: 'commit-and-tag-version',
  test: 'cross-env LOG_LEVEL=debug PACT_DO_NOT_TRACK=true mocha "{src,test,bin,standalone}/**/*.spec.ts"',
  'snyk-protect': 'snyk-protect',
  'format:base': 'prettier --parser typescript',
  'format:check': 'npm run format:base -- --list-different "{src,standalone,bin,test}/**/*.{ts,tsx}"',
  'format:fix': 'npm run format:base -- --write "{src,standalone,bin,test}/**/*.{ts,tsx}"',
  install: 'node-gyp rebuild'
}
d

Dany Marques

03/13/2024, 6:12 PM
😞
y

Yousaf Nabi (pactflow.io)

03/13/2024, 6:22 PM
I am going to try removing the binding.gyp file from the repo prior to publishing it seems that having it as part of the .npmignore is not enough https://github.com/npm/cli/issues/5234#issuecomment-1291139150
d

Dany Marques

03/13/2024, 6:34 PM
That should work but my suggestion also should
I'm trying to understand why it doesn't
The npm version used in your ci is the latest.. looks good
Just spotted the issue on npm
Will have a closer look tomorrow
I'm exhausted
y

Yousaf Nabi (pactflow.io)

03/13/2024, 6:42 PM
rest well my friend, we will get this sorted in the end
d

Dany Marques

03/13/2024, 6:42 PM
I got it!
Just found
It's weird in npm
Actually there are 2 bugs in npm
That's because the install script is empty
y

Yousaf Nabi (pactflow.io)

03/13/2024, 6:43 PM
d

Dany Marques

03/13/2024, 6:43 PM
Will fix that on npm tomorrow
Hope they release fast
y

Yousaf Nabi (pactflow.io)

03/13/2024, 6:44 PM
fingerscrossed what a win for private npm users everywhere, great work again
d

Dany Marques

03/13/2024, 7:09 PM
Thanks 🙂
Actually I've to raise 2 different PR on npm
y

Yousaf Nabi (pactflow.io)

03/13/2024, 7:15 PM
we can try another approach, which is vendoring parts of node-gyp-build, as we only need the binding resolver at runtime https://github.com/pact-foundation/pact-js-core/compare/master...deps/vendor_node-gyp-build still requires testing of the case of existence of the binding.gyp at the point of publishing (despite being ignored in .npmignore) I can dig into that side
d

Dany Marques

03/13/2024, 7:22 PM
Humm, I think this won't work
I'm working on a first PR on npm that should already fix our issue with the workaround of having an empty install script
After the second one, I hope we can remove the hacky empty install script and it should still work!
Not sure I'm finishing anything today though
j

James P

03/13/2024, 8:46 PM
FWIW, I too am on an M3, node 16 / npm 8 attempting a fresh 
npm i
and getting a similar error. What stands out as a little different for me is the error 
npm ERR! ModuleNotFoundError: No module named 'distutils'
m

Matt (pactflow.io / pact-js / pact-go)

03/13/2024, 10:40 PM
Wow - you really managed to get deep in the wormhole! 🤯 This deserves a write up on our blog next month Yousaf!
I’m working on a first PR on npm that should already fix our issue with the workaround of having an empty install script
👏 So I figure that we roughly know the problem now, and we have a few possible ways to fix. It sounds like the 
"install": ""
option doesn’t work, but removing 
binding.gyp
before publishing might be what resolve this?
d

Dany Marques

03/14/2024, 6:24 AM
You're right @Matt (pactflow.io / pact-js / pact-go). But, IMHO, npm should be smart enough to decide if it adds the "install" script based on the .npmignore file.
m

Matt (pactflow.io / pact-js / pact-go)

03/14/2024, 6:33 AM
Agree!
I expect 
npm
to take a while to release any changes. If we can fix your use case by simply removing the 
binding.gyp
that should be easy enough for us.
d

Dany Marques

03/14/2024, 6:39 AM
We can do this, and as soon as npm release, then we rollback to keep pact-js-core clean.
m

Matt (pactflow.io / pact-js / pact-go)

03/14/2024, 6:42 AM
I think it’s just a matter of removing the file prior to running 
publish.sh
. I think in this block, effectively: https://github.com/pact-foundation/pact-js-core/blob/master/script/ci/lib/publish.sh#L21 @Yousaf Nabi (pactflow.io) refactored the release process a little while back so probably knows better than I, but I can attempt it there. I’ll try it now
Does this look better now?
Copy code
npm view @pact-foundation/pact-core@latest scripts                                                                                                                                                                                       
{
  clean: "rimraf '{src,test,bin,standalone}/**/*.{js,map,d.ts}' 'package.zip' '.tmp' 'tmp'",
  lint: 'eslint . --ext .ts --config .eslintrc',
  'lint:fix': 'npm run lint -- --fix',
  prebuild: 'npm run clean',
  'download-libs': 'npm run clean && bash script/download-libs.sh',
  'clean-libs': "rimraf standalone/*.{js,map,d.ts} standalone/{windows**,linux**,darwin**} 'ffi'",
  build: 'tsc --project tsconfig.build.json',
  prerelease: 'npm run snyk-protect',
  release: 'commit-and-tag-version',
  test: 'cross-env LOG_LEVEL=debug PACT_DO_NOT_TRACK=true mocha "{src,test,bin,standalone}/**/*.spec.ts"',
  'snyk-protect': 'snyk-protect',
  'format:base': 'prettier --parser typescript',
  'format:check': 'npm run format:base -- --list-different "{src,standalone,bin,test}/**/*.{ts,tsx}"',
  'format:fix': 'npm run format:base -- --write "{src,standalone,bin,test}/**/*.{ts,tsx}"',
  install: ''
}
d

Dany Marques

03/14/2024, 8:28 AM
Looks perfect!
I just checked in the private registry and the script is not there anymore
Let me try to install now
Works!!
Thank you!
m

Matt (pactflow.io / pact-js / pact-go)

03/14/2024, 10:26 AM
WOOHOO!
seriously 🙇 to you my fiend
d

Dany Marques

03/14/2024, 10:28 AM
Thanks to you also for reactivity! 🙂
16 Views
Powered by