https://cloudposse.com logo
Join Slack
Powered by
Hi there, I have a Makefile from you guys in a pro...
# geodesic
c
Hi there, I have a Makefile from you guys in a project that has been managing our terraform for a few years. Running any of the Make targets is just hanging, not doing anything. For example, 
make all
or 
make run
. After typing the command and hitting enter, it just hangs at the prompt. Has anyone else experienced anything like this that you've heard? It is untouched in a long time, and I ran it as recently as last week.
I'm trying to manage infrastructure for a very large enterprise, and suddenly can't do anything, any help would be very appreciated
here is the makefile with the enterprise clients name obfuscated out with *
Copy code
export HOME_DIR ?= /home/code
export STORAGE_DIR ?= /mnt/storage
export PROJECTS_DIR ?= $(STORAGE_DIR)/projects
export DOCKER_ORG ?= grizzlyforce
export DOCKER_IMAGE ?= $(DOCKER_ORG)/***-atmos
export DOCKER_TAG ?= latest
export DOCKER_IMAGE_NAME ?= $(DOCKER_IMAGE):$(DOCKER_TAG)
export APP_NAME = ***-atmos
GEODESIC_INSTALL_PATH ?= /home/code/.local/bin
export INSTALL_PATH ?= $(GEODESIC_INSTALL_PATH)
export SCRIPT = $(INSTALL_PATH)/$(APP_NAME)
export ADR_DOCS_DIR = docs/adr
export ADR_DOCS_README = $(ADR_DOCS_DIR)/README.md
BUILD_HARNESS_EXTENSIONS_PATH := $(CURDIR)/.build-harness-extensions

-include $(shell curl -sSL -o .build-harness "<https://cloudposse.tools/build-harness>"; echo .build-harness)

.DEFAULT_GOAL := default

## Initialize build-harness, install deps, build docker container, install wrapper script and run shell
all: init deps build install run
	@exit 0

## Install dependencies (if any)
deps:
	@exit 0

## Build docker image
build:
	@make --no-print-directory docker/build

## Push docker image to registry
push:
	@$(call fail,Refusing to push $(DOCKER_IMAGE_NAME) to docker hub)

## Install wrapper script from geodesic container
install:
	@docker run --rm $(DOCKER_IMAGE_NAME) | bash -s $(DOCKER_TAG) || (echo "Try: sudo make install"; exit 1)

## Start the geodesic shell by calling wrapper script
run:
	docker run -ti --rm -v $(HOME_DIR):/localhost -v $(PROJECTS_DIR):/localhost/projects  -v $(STORAGE_DIR)/cody/.aws:/localhost/.aws -v $(STORAGE_DIR)/cody/.ssh:/localhost/.ssh grizzlyforce/***-atmos
e
@Jeremy White (Cloud Posse) can you share how you fixed it?
(it's related to the latest version of Make)
c
i had a feeling this was the case, thanks Erik. I'm on arch so i've rolled into the latest version i guess
j
Aloha, yeah, I had this happen. Can you share your make version with 
make --version
?
c
GNU Make 4.4
j
aha, I think this is the related issue: https://github.com/cloudposse/build-harness/issues/333
but I have a quick way to check
try running 
/usr/bin/make all
c
@Jeremy White (Cloud Posse) going to try this now, thank you for the info and patience, i got delayed on other things
I ran the 
make SHELL='sh -x' init
and did experience the infinite loop
is the only option for now to rollback make @Jeremy White (Cloud Posse)?
in the build harness, it looks like git submodule --init --recursive could be replaced with
Copy code
git submodule foreach --remote 'git fetch | 0'
git submodule foreach --remote 'git merge origin master | 0'
or something like this
j
to be clear, you tried with the old make binary? 
/usr/bin/make
?
c
i did not
but i was able to replicate the forever loop
from that github issue
/usr/bin/make
for me is also 4.4
j
we've been able to recreate the loop issue as well, so this is definitely a confirmed regression with newer make
c
lol very annoying
j
aha, okay, that complicates things.
c
yeah, in arch there is a project for rolling back packages, not sure what other issues that will cause. I could also just build and install an older version myself i guess
and symlink to it or something
j
well, there is also the chance you can replace the formula in brew with the make 4.3 script
c
oh i dont use brew, im not on a mac
its arch linux
j
also, if you are use nix, I can give you a nix-shell command to try
c
well, endeavour
hmmm, i dont use nix, but i suppose i could do that as well
no time like the present to get hooked on another tool like nix lol
j
nix is honestly the easiest way to have an old binary for things like this while bugs are squashed. the command would be 
nix-shell -p gnumake42
which would hopefully give you a make 4.2 binary in your PATH to quickly unblock
but if you don't have it set up, that can take time
c
i see, yeah, i've read plenty about it, but never attempted to set it up
haha šŸ˜… i can't remove make because dkms relies on it
so i think installing the downgrade beside it is my only option
j
of you don't want to remember the command, you can commit a file 
shell.nix
and put this inside it:
Copy code
{ pkgs ? import <nixpkgs> {} }:
pkgs.mkShell {
  nativeBuildInputs = with pkgs; [
    gnumake42
  ];
}
c
such a huge project to get held up on, just my luck haha
j
well, we're hoping to squash this bug soon
I just cannot say if it will be soon enough for you
c
makes sense, i provided a potential solution above. have you guys tried to approach this in any way yet?
j
also, have you tried the change you mentioned above? to alter the git submodule line?
c
i haven't, but i could, if i can figure out where that gets called
i dont entirely understand where i would change something to call a different build-harness
i could fork the harness though if you can tell me where that gets pulled in
j
the include line is in the makefile, around line 25
I'm going to try to find the line you speak of... if I can test it, I'll holler
c
i found the call for the build-harness, will attempt
Ok, I did the following: ā€¢ commented out the line in the Makefile that downloads the build-harness makefile from cloudposse.tools/build-harness ā€¢ altered the previously downloaded file to use my forked build-harness repo (grizzlyforce/build-harness) ā€¢ updated the forked repo to use git fetch / git merge ā€¢ ran 
make run
successfully
i can't run 
make all
or 
make build
though which tells me my solution is probably not a complete solution?
Copy code
āžœ make build
make[1]: *** No rule to make target 'docker/build'.  Stop.
make: *** [Makefile:30: build] Error 2
I think I need to host my own version of that cloudposse.tools/build-harness makefile
j
okay... can you link to any files you changed?
c
one sec, let me take this a step further, i just realised that file is actually in the repo, so i also have it hosted and can change to my version
j
gotcha
c
it still infinite loops, so it must be before my change
I searched geodesic and build-harness for 'submodule' and the only call i found was in build-harness/modules/git/Makefile
is there somewhere else that i'm missing?
j
I wasn't able to find it anywhere else either
but
I am suspect of the bootstrap.Makefile in the git module. It has a trailing 
endif
and I don't like the way that looks...
c
Copy code
āžœ make SHELL='sh -x' all
+ curl -sSL -o .build-harness <https://raw.githubusercontent.com/grizzlyforce/build-harness/master/templates/Makefile.build-harness>
+ echo .build-harness
+ pwd
+ pwd
+ '[' -d build-harness ']'
+ pwd
+ config --get remote.origin.url
+ sed 's/git@\(.*\):/https:\/\/\1\//g'
+ sed 's/\.git$//g'
+ show -s --format=%aN
+ show -s --format=%aN
+ config --get remote.origin.url
+ sed 's/\.git$//g'
+ sed 's/git@\(.*\):/https:\/\/\1\//g'
+ rev-parse --verify --short HEAD
+ config --get remote.origin.url
+ sed 's/git@\(.*\):/https:\/\/\1\//g'
+ sed 's/\.git$//g'
+ rev-parse --verify --short HEAD
+ rev-parse --verify --short HEAD
+ config --get remote.origin.url
+ sed 's/\.git$//g'
+ sed 's/git@\(.*\):/https:\/\/\1\//g'
+ rev-parse --verify --short HEAD
+ show -s --format=%aN
+ show -s --format=%aN
+ rev-parse --verify --short HEAD
+ rev-parse --verify --short HEAD
+ show -s --format=%aN
+ config --get remote.origin.url
+ sed 's/\.git$//g'
+ sed 's/git@\(.*\):/https:\/\/\1\//g'
+ show -s --format=%aN
+ rev-parse --verify --short HEAD
+ rev-parse --verify HEAD
+ config --get remote.origin.url
+ sed 's/\.git$//g'
+ sed 's/git@\(.*\):/https:\/\/\1\//g'
+ show -s --format=%aN
+ show -s --format=%aN
+ config --get remote.origin.url
+ sed 's/\.git$//g'
+ sed 's/git@\(.*\):/https:\/\/\1\//g'
+ rev-parse --verify HEAD
+ config --get remote.origin.url
+ sed 's/\.git$//g'
+ sed 's/git@\(.*\):/https:\/\/\1\//g'
+ rev-parse --verify HEAD
+ rev-parse --verify HEAD
+ config --get remote.origin.url
+ sed 's/\.git$//g'
+ sed 's/git@\(.*\):/https:\/\/\1\//g'
j
the issue by rb claims that the bootstrapper is what causes the loop
and if you comment out the include for the bootstrap.Makefile includes, it works. I'll test that now
confirmed that fixes it for me
can you too? just hop into the 
build-harness/Makefile
and comment out line 42
it should be something about including bootstrap files
c
I commented out that line, pushed into my repo, ran make all, and still get the infinite loop
I restored the modules/git/Makefile to original to test again, no difference
j
so, I noticed you said you 'pushed' the change. I didn't have to push. I did the edit directly in the infra folder... since the build-harness directory should be present after running the include lines
c
it seems to try to download the .build-harness file everytime
j
oh, not the file
the folder
there should be a 
build-harness/Makefile
c
oh, you're running make all in the build-harness project? sorry i was downstream in my own project
thats why i pushed
j
gotcha. I'm assuming you're doing a geodesic project, so I have a similar geodesic project
c
ok, i was able to run make init in the build-harness project with that line commented out.
i was not able to run make init while that line is still active
yeah, we use atmos for managing our terraform projects
so, geodesic in the background
j
okay, so I think the smoking gun here is the bootstrap.Makefiles
and I definitely see the `modules/git/bootstrap.Makefile`` as being especially suspect...
c
fair enough. i am unblocked in my geodesic project by commenting out the line that downloads the build-harness everytime
at least i can run terraform now šŸ¤¦ā€ā™‚ļø
an extremely large staffing company is white-labelling our software for their own use and i was in the middle of provisioning their infrastructure when everything stopped working
serious panic set in haha
i was on the verge of just building my own container and installing atmos
j
gotcha. you don't have to roll any further with this. I just hoped to unblock you all the same
c
thank you very very much for your help, i cant thank you enough!
if i can help test in any way, i'm happy to do so, you know i face this problem now
j
absolutely, I think once I have a fix in place, I'll mention the branch, and you should be able to set it in the harness or at least hop into the dir and 
git checkout
if there's any other way I can help, just holler or add it to the github issue
have a good thursday!
c
thank you sir! you as well
šŸ‘ 1
4 Views
Open in Slack
PreviousNext
Powered by