http://coldfusion.com logo
Join Slack
Powered by
Might be heresy to ask about this here, but is any...
# box-products
d
Might be heresy to ask about this here, but is anyone using CommandBox with private github packages? Curious as to what your workflow looks like with publishing changes and such (as compared to ForgeBox packages, which we’ve done). And is a github package any different than a normal github repo? Just dipping our toes into this a bit.
b
@danmurphy I don't know how github packages work, but if Github has basically written an online registry with some sort of API to interact with, I assume an endpoint could be written in CommandBox to integrate with it.
As you may have guessed, Ortus would prefer to look into our private ForgeBox packages 🙂
d
Ha, right. I totally understand that. We have teams using our org within Github for private npm packages and so this is something we wanted to explore. It might just work with the current github functionality, but figured I’d ask around first.
b
I see Github also has a feature where they can basically spoof the API of common registries like npm or Rubygems so the native CLI for that tool can just install the packages directly from Github. of course, CF/ForgeBox never get any sort of love/support for these sort of things
Sonatype allows you to add support for arbitrary registries( fomats, as they call them), but I have no idea if Github has made itself extensible like that.
but either way, some one has to write one of the following • A CommandBox installation endpoint that talks to the Github API to get pacakges • An integration into Github (if it's even possible) to present its registry in ForgeBox's REST API format so CommandBox can be pointed at it (thinking it's a forgebox instance)
We're actually looking to do bullet #2 right, but for Sonatypes artifact repository, but we have an Enterprise client using the tooling who is considering sponsoring the development of it.
Otherwise, we're not likely to give you a workaround to use enterprise features with our tool and give Github/etc $$ while paying us nothing 🙂
Kind of makes me wonder how much of npm Enterprise's business Github is sltealing 🤔 😆
d
So this is probably a really ignorant question, but what does all of that get you that the 
install <git://site.com/user/repo.git#v1.2.3>
doesn’t provide? (just reading about it in the git section of the CommandBox docs)
b
Mostly faster installions (git cloning is slow), proper version support (Git repos generally always show as "outdated" because there's no way to query the repo to ask what the latest version is), and you don't have to screw around with super annoying SSH keys or access tokens for private Git repos.
d
Ha, yeah, screwing around with authentication was literally on my agenda for the day, actually. 😄
b
So with ForgeBox, I can do proper semver ranges like
Copy code
install "coldbox@4.x || <= 6.2.*"
and it's super smart about finding the package whose version is the best fit, and updating that package in the future when new versions come out
Cloning a Git repo will always just give you the same thing, unless you're pointing to a branch which is changing
The
Copy code
outdated
and
Copy code
update
commands are far more useful with proper packages from forgeBox
Git is sort of "dumb" in that all it can do is clone the repo and checkout the commitish, but it really can't tell you what versions of the lib exist or if yours is out of date
d
So say we have a private Forgebox repo, how does it interact with our also private github repo? Do we have some of the same challenges?
b
Forgebox auth is baked in so you can
Copy code
forgebox login
and then we know who you are. ForgeBox is also getting org support as well have org-level packages and user-level packages. Then there's collaborators. public/private packages, etc. It's all built into our permission scheme so you can install whatever you have permissions to
ForgeBox doesn't have "repos", it has "packages" 🙂 Forgebox doesn't really care where the code for a package may or may not live. Assuming you're using the built in forgebox storage like npm works, each package version is just a zip file up in a private S3 bucket
So your project has a CI process which publishes new versions. For me, it's as simple as
Copy code
bump --minor
(which uses a package script to run the 
pubilish
command) -- and a couple seconds later, my new package version is live on ForgeBox
Github's package management solution is basically going to be the same thing except • they will have their own CLI • They will store the list of packages and their versions • They will host the binaries
☝️ I'm making a few assumptions there, but I'm pretty confident that's what they've done
d
But all of your Forgebox packages are pointing to public github repos? My point being, we want to keep our codebase in our private github repo. Just trying to compare what a private Forgebox package would look like in that scenario (vs whatever I’m going to screw around with in a github only solution).
b
all of your Forgebox packages are pointing to public github repos
Not at all. Where did you get that idea?
My point being, we want to keep our codebase in our private github repo.
Sure, go ahead, lol. Nothing stopping you. We have lots of clients doing this.
d
It was just a question.
b
Then the answer is no 🙂
It's worth noting that forgebox stores the metadata about a package (name, description, versions, etc). There are lots of options of how a package version can tell the CLI how to install it • it can point to an HTTP URL • it can point to a Git repo+commitish • It can point manually to an S3 file you've created • It can handle the file storage automatically for you (which ultimately uses S3) When you install package 
foo
in CommandBox, we look at the package metadata and then defer to the proper endpoint to get the actual code
👍 1
So that means, it's possible to directly install a git repo
Copy code
install <git://gitub.com>...
or you can install a package from forgebox
Copy code
install forgebox:mypackage
which may POINT to a Git repo if it wishes. In that case, the CLI defers back to the same Git endpoint anyway to finish resolving the package
d
So it sounds like you’ll have the same Github private key / access token finagling if you have a private ForgeBox package pointing to a private github repo as you probably would with a private github package/repo. Does that sound right?
b
In fact, when install from an HTTP URL • the http URL gets the zip downloaded • which defers to the local file endpoints to unzip it • which defers to the local folder endpoint to install it.
So in all cases, regardless of the endpoint (file, folder, http, Git, S3) the resource you're pointing to can be public or private by virtue of authentication or even network access. This is totally valid
Copy code
install file:\\\\myservershare/sharename/folder/file.zip
or
Copy code
install <https://local-intranet-server.dev/folder/file.zip>
and both of those would only work obviously on your computer where you have access to those resources\
So whether the actual artifact is public or not is entirely up to you and where you place it
The built in "forgebox storage" is inherently on a private S3 bucket, so the visibility is up to the package itself
So it sounds like you’ll have the same Github private key / access token finagling if you have a private ForgeBox package pointing to a private github repo as you probably would with a private github package/repo. Does that sound right?
Yes, but that would be largely pointless. The goal of using private Git repos is usually because people are trying to avoid private forgebox packages! If you have a private forgebox package, you would be better served just using S3 storage at that point and it makes no difference where the actual repo is.
Which is super nice, because now you can install your private packages anywhere in the world without being on some private company VPN or having SSH keys on your laptop. Just log into forgebox and you're good.
I can publish a package (public or private) to forgebox right now and not even have a Git repo!
Copy code
mkdir my-package --cd
touch coolStuff.cfm
package init
publish
and that's it
The folder of code (
box.json
, and 
coolStuff.cfm
) would be zipped up and whisked away to a private S3 bucket and the package published to ForgeBox.io
That's using "forgebox storage" as we call it. Which means you don't need to care about the Git repo at the point where you install it
Installing that package is just downloading the zip from the private S3 bucket using a secure, signed URL
So, in essence, by controlling who has access to the forgebox package metadata (private packages,etc), you can control who installs it. And it doesn't need to have any tie to the source code.
Really, 90% of the people using private Git repo cloning and SSH keys IMO are just people trying to avoid paying for a private package
d
Ok, I’m repeating to make sure I’m hearing you. I think I had some assumptions wrong. So I might work on my private package (that is in a private github repo, which sounds like it doesn’t really matter). I get to the point where I commit stuff for a version 1.5, for example. I can commit everything and push it to github just like I would with any other codebase in github. Then I can “publish” a 1.5 version in commandbox to a private forgebox package, which really just zips up artifacts on my local machine and sends it up to ForgeBox. Am I right so far?
b
Yep, that's 100% correct.
And technically, commiting/pushing is not a requirement to publish, but it's a good idea.
d
Ok, right.
b
Meaning, the publish command really just cares about what code happening to be sitting on your hard drive. if you want to commit, tag, push it, then that's on you (and can be automated as well) via CommandBox package scripts.
The one caveat to that ☝️ is if your project is a Git repo AND you have staged, but uncomitted files, the 
publish
command will yell at you since it assumes you forgot something.
We copied that behavior from npm
d
So my build process (the DEV/QA/PROD servers) won’t need to care about authenticating to github, but they will need to know how to authenticate to our private Forgebox package. Right?
b
The 
bump
command will increases your version number and tag your git repo too which is super handy.
Correct ✅
d
So if we’re doing a 
box install
within our Dockerfile, would we need to have some env variables or something with the ForgeBox credentials or something? (trying to see if this is addressed in the docs already…)
b
Yes
You can do it a couple different ways, and they both would likely use an env var or a docker secret
You'd either run
Copy code
box forgebox login user pass
box install
Or you could just directly set your API token with
Copy code
box config set endpoints.forgebox.APIToken="xxxxx"
box install
And the nice thing about the latter optoin, is you can skip the 
config set
entirely and just set this env var and be done
Copy code
ENV box_config_endpoints_forgebox_apiToken=XXXXX
CommandBox will just pick up that config setting by convention and use it when it runs
None of this is too different from what you're already doing to authenticate to npm or a docker registry
d
So today, you can have a public forgebox package point to a private github package 🤔 . That feels weird. Do folks do that?
b
Yes, you could do that. I can't think of a reason why you would, and no I've never seen anyone do that
ForgeBox doesn't really "validate" the download location you give it because it may not even be generally accessible
That wouldn't be much different than a typo in your git repo name. Can you do it, sure. Would you want to, no. Either way, it's up to you to fix
d
I guess if you wanted to obfuscate ownership of the github repo?
b
Not sure I follow you, if I created a public package on ForgeBox called 
brad
and its download location pointed to 
<http://github.com/privateorg/privatePackage|github.com/privateorg/privatePackage>
then • everyone would be able to see the package on forgebox.io and via the CLI • Everyone but you would get an error when they tried to install it • Everyone would still be able to see where it was trying to install from So it wouldn't be obfuscating anything, but it may make people question your sanity.
d
Ok, I’m repeating to make sure I’m hearing you. I think I had some assumptions wrong. So I might work on my private package (that is in a private github repo, which sounds like it doesn’t really matter). I get to the point where I commit stuff for a version 1.5, for example. I can commit everything and push it to github just like I would with any other codebase in github. Then I can “publish” a 1.5 version in commandbox to a private forgebox package, which really just zips up artifacts on my local machine and sends it up to ForgeBox. Am I right so far?
I guess I need to just play around some more. With this question ☝️ , it sounded like commandbox/forgebox didn’t actually talk to github at all when it installed a package if you are using forgebox storage. It just had the zipped up artifacts it needed in Forgebox (regardless of whether the forgebox package is private or public).
b
Yes, that's correct.
I was talking about cloning a private Git repo in my last message because you specifically asked a question about that scenario 🙂
Unless that's not what you mean when you said,
you can have a public forgebox package point to a private github package
When you said "point to", did you mean it literally installs from a repo, or did you simply mean to say that you publish a public package for the community to use that downloads an artifact from ForgeBox Storage (S3), but you keep the code internally for that package in a private repo for your own purposes?
@danmurphy
Because that second option means something totally different, lol
d
Yeah, I’m probably using “point to” as “cloning from” and “may reference but doesn’t need to talk to” synonymously, which doesn’t make sense.
I’ve just learned in this thread that there is a difference. I thought ForgeBox was really always having to communicate with GitHub.
b
I thought ForgeBox was really always having to communicate with GitHub.
Oh goodness, no 🙂 Back in the early days, about half the packages cloned git repos just for convenience and the rest pointed to zip files we would manually place up on our own S2 buckets in our build
But ever since we introduced Forgebox Storage, I've been converting all my projects over to it just because it's less moving parts and easier for me to release.
d
So in the forgebox storage situation, the github repo is there for reference but not for usage, if that makes sense?
b
Yeah, you don't need to have any git repo whatsoever just to publish a package (as I showed in my example commands above)
d
Right. This is good info.
b
Now, you can put a link to a Git repo in the box.json which because part of the package metadata and will create a handy link in the forgebox.io UI but that has nothing at all to with the actual downloading of the package. It's just purely informational
d
message has been deleted
This slack team really needs the 
/giphy
integration. 😄
b
lol
So this optional metadata here https://github.com/bdw429s/commandbox-fusionreactor/blob/master/box.json#L7-L10 creates this link in the GUI
Note in this case, the package is owned by the Ortus user on forgebox, but the code is still in a repo owned by myself
But none of that controls how the package is installed. The 
box.json
in that repo has no 
location
key which means it's using the default value of 
forgeboxStorage
So when I cut a version, I run
Copy code
CommandBox> bump --patch
which • increments the version • tags the repo • zips up the code on my hard drive and uploads it to S3 • runs this package script https://github.com/bdw429s/commandbox-fusionreactor/blob/master/box.json#L20 • That package script publishes the package to forgebox • then it runs this package script https://github.com/bdw429s/commandbox-fusionreactor/blob/master/box.json#L21 • which pushes my commits and tags for me
👏 2
I'm rather lazy, so I love having a single command that does all the work for me 🙂
d
@bdw429s - playing around with a workflow like you have above, and the 
bump --minor
command is giving me…
Copy code
Can't copy file [/Users/dan.murphy/code/scs-private-test/models/ModelUtility.cfc] to [/Users/dan.murphy/.CommandBox/temp/5BFBC4AD96745C2D16435F94B8D3D493/models/ModelUtility.cfc]

No such file or directory
My file definitely exists, I can click on it in the error and open it. It looksl ike 
/Users/dan.murphy/.CommandBox/temp/5BFBC4AD96745C2D16435F94B8D3D493/
exists even. But the 
/models
folder does not at that directory. Thoughts?
It is still bumping the minor version in the box.json, but it fails with this error each time I try.
So still getting the error on Publish. I can manually create the models directory in the temp folder. CommandBox then yells during the publish command that it can’t create the directory because it already exists. Which, well, is true at that point. But if I delete the 
5BFB…
directory completely, it is back to giving me that original error about 
No such file or directory
. Hmm.
b
Sorry, my weekend hit early and I'm camping out at the lake. I've never seen the error you're taking about and I have no clue what is happening, lol. I'll take a look after I'm back home.
I do think there were some changes to this logic in the latest release so it's possible you've found an edge case. I'll need to figure out how to reproduce.
@danmurphy Please provide instructions for how to reproduce this error. This logic did change in the last release, but I've reviewed it and it all looks correct. I've obviously been publishing packages for months on the new logic and it's been working fine. Perhaps it's related to a *nix file system?
When creating the temp zip • the temp folder is deleted and re-created, so any manual changes you are trying to do shouldn't matter • The folders are all created up front before any files are copied, so if the 
models
folder is not there, that's a clue, but I'm not sure how it would happen. • Are other folders being created? All folders are created from the shortest to the longest in that order so subdirs are all created last. • No files are copied until all the folders are created.
Do you have any custom 
ignore
settings in your 
box.json
?
d
I can check on Monday to be sure, but files in the same directory as the models folder were there, just not the models sub directory and the file inside of it. I'll check on settings Monday too. Just a note, it was creating the temp folder after a retry with the exact same name, even after deleting it. I just thought that was interesting.
b
Yes, that's because I didn't write that code, lol
I usually use 
createUUID()
, but Eric who did this part used a hash of the path
So it's the same folder every time,
but it does delete it every time too
Copy code
var tmpPath = tempDir & hash( arguments.path );
		if ( directoryExists( tmpPath ) ) {
			directoryDelete( tmpPath, true );
		}
Either way, ping me monday with details
If this is a regression, I'd like to get it fixed and there's several things we can do to debug
I'm curious if your ignore patters are somehow filtering the 
models
folder out of the list of things to copy but keeping files inside the folder.
I'm not sure how that would even be possible, but anything could happen
Note, the 
publish
command uses your box.json ignores as well as your 
.gitignore
file
d
I'm on a Mac, so yeah, maybe *nix related.
Huh, I'll check that out for sure then. I'll get in touch on Monday. Thanks for the follow up Brad!
b
There are people I know have been publishing packages in Ortus who use Mac or Linux, but maybe you've got something special going on 🙂
And if you want to poke at it in the mean time, the logic is near line 619 of 
cfml/system/endpoings/forgebox.cfc
Copy code
wirebox.getInstance( 'globber' )
			.inDirectory( arguments.path )
    		.setExcludePattern( ignorePatterns )
    		.loose()
			.copyTo( tmpPath );
Just starting with adding this debugging before that line
Copy code
systemoutput(
  wirebox.getInstance( 'globber' )
   .inDirectory( arguments.path )
   .setExcludePattern( ignorePatterns )
   .loose()
   .asArray(),
  1)
will tell us what it's trying to copy
d
Ok, I think this is what you were asking for @bdw429s…
Copy code
evaluateComponent("globber.models.Globber","b471c4dc34de00d4e57a0f33d69079b8",{},{"baseDir":"/Users/dan.murphy/code/scs-private-test/","LOOSE":true,"format":"array","sort":"type, name","MATCHQUERYARRAY":[],"EXCLUDEPATTERN":[".*.swp","._*",".DS_Store",".git/",".hg/",".svn/",".lock-wscript",".wafpickle-*","config.gypi","## General Ignores + IDE",".DS_Store","settings.xml","WEB-INF",".env",".vscode/*","*.code-workspace","*.sublime-workspace","## logs + tests","logs/**","/testbox","/tests/results/**","/tests/resources/app/coldbox","## npm","**/node_modules/*","npm-debug.log","yarn-error.log","yarn.lock","#### Ignored Dependencies","coldbox/*","testbox/*","modules/*","/includes/js","/includes/css","/includes/fonts","/includes/build","**/apidocs/**","**/.*","/test/","/tests/"],"NOTEXCLUDEPATTERN":[],"pathPatternMatcher":evaluateComponent("globber.models.PathPatternMatcher","1200e3ab33d203f37482b555cceb904a",{},{}),"PATTERN":[]})
b
Nope
That's just a serialized CFC, lol
What code do you have in place?
The code above should have returned the actual array of results
d
I just plopped that systemoutput line above the globber getInstance line and copied that from the console after.
b
Oh, I see the issue, hold on
Need to chain the 
.matches()
method to the end of the DSL
That's the one that actually returns the results
@danmurphy
d
Wait - I just got rid of the systemOutput line and it worked. No error. Huh?
The package is on Forgebox. I am really, really confused.
b
Just publish it again. Perhaps the behavior is randomized
Copy code
publish --force
You can run that as many times as you want. The 
--force
makes it re-zip the code even though the version hasn't changed
d
Worked 3 times in a row. 🤯
After not working 100 times in a row, ha.
b
What's the output of the debugging?
d
Sorry, let me put that back in…
b
If you wipe out the temp folder, does it change?
d
Is it safe to blow away everything in the temp folder?
(assuming it is since it is well, a temp folder)
b
yes
Also, did you adjust • the code in the folder • the ignores at all?
d
No - I put in the system output line, saved it (it failed again), took that out, saved it, and it worked.
Deleting everything in the temp folder and doing another 
publish --force
and it worked again.
Wow, this is strange.
b
Did you happen to perform an exorcism on your machine over the weekend? 😆
😂 1
d
So you’re saying the debug line should be this?
Copy code
systemoutput(
  wirebox.getInstance( 'globber' )
   .inDirectory( arguments.path )
   .setExcludePattern( ignorePatterns )
   .loose()
   .asArray()
   .matches(),
  1)
✅ 1
b
So what was the output of the debugging??
That's a start-- I mean I have like 30 other things we can debug. That was just the first one
d
I don’t need to do some kind of cache flush or something if I’m changing something in that Forgebox.cfc do I?
2 Views
Open in Slack
PreviousNext
Powered by