Hi @Andrey, I pulled your branch and I’m trying to run the

PulsarVeniceSinkTest

but I’m not able to… getting this output. I’ve also tried following the instructions here but I’m failing at the

docker compose -f tests/docker-images/docker-compose-pulsar-venice.yaml up -d

step. It’s telling me that:

Error response from daemon: pull access denied for pulsar/venice-test, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

I have been able to run

docker compose -f quickstart/docker-compose.yaml up -d

however. I have also done the docker login command, but it doesn’t help. There must be something wonky in my setup, but IDK what. Does any of this ring a bell for you?

Or perhaps I need to be granted permission to access

pulsar/venice-test

on the docker hub? IDK much about Docker, so I’m not too sure 😄 …

it takes pulsar image and add new sink there as a connector to mimic what's typically happens in prod

e.g. to clean rebuild everything I do:

# build Venice and sink
cd /Users/andreyyegorov/src/venice
./gradlew clean assemble

# build Venice's latest-dev docker images
cd /Users/andreyyegorov/src/venice/docker
./build-venice-docker-images.sh

# build latest-dev docker image for Pulsar to use in tests
cd /Users/andreyyegorov/src/venice/tests/docker-images/pulsar-sink
docker build --tag=pulsar/venice-test:latest-dev /Users/andreyyegorov/src/venice -f ./Dockerfile

# run the test
cd /Users/andreyyegorov/src/venice
./gradlew :tests:venice-pulsar-test:pulsarIntegrationTest -i

as it all runs in the docker you'll have a hard time attaching debugger. tbh I never tried, but it might be possible using remote debugger if you open some ports

to simplify access to logs I mount the logs to local disk, e.g. for broker:

volumes:
      - ./data/broker/logs:/pulsar/logs

Venice's logs go to other location so mount accordingly. You'll see the logs in tests/venice-pulsar-test/src/pulsarintegrationtest/resources/data/...

lazydocker https://github.com/jesseduffield/lazydocker works nicely with docker compose containers - you can easily jump into the command line/see logs etc I had so much better experience with Orb's docker replacement https://orbstack.dev/ than with the original docker. At least on my Intel Mac. Give it a shot.

In case some containers take to long to shut down youc an kill all het running containers lie

docker kill $(docker ps -q)

Sometimes docker runs out of space allocated for it to cache data etc, so I just run

docker system prune -f && docker volume prune -f

before running the test

this all is not as scary as it looks 🙂