I'm trying to run a custom transformation step and...
# advice-data-transformation
h
I'm trying to run a custom transformation step and airbyte is failing to checkout the repo with the code. The output i'm seeing the logs is this:
Copy code
docker run --rm --init -i -w /data/70/1/transform --log-driver none --name normalization-normalize-70-1-puevj --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e AIRBYTE_VERSION=0.37.0-alpha airbyte/normalization:0.1.77 configure-dbt --integration-type postgres --config destination_config.json --git-repo "${REPO}"
Running: git clone --depth 5 --single-branch  $GIT_REPO git_repo
Cloning into 'git_repo'...
Last 5 commits in git_repo:
7b6d2d8 Create destination_config.json
418171b first commit
/data/70/1/transform
Running: transform-config --config destination_config.json --integration-type postgres --out /data/70/1/transform
Namespace(config='destination_config.json', integration_type=<DestinationType.POSTGRES: 'postgres'>, out='/data/70/1/transform')
Traceback (most recent call last):
  File "/usr/local/bin/transform-config", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/normalization/transform_config/transform.py", line 335, in main
    TransformConfig().run(args)
  File "/usr/local/lib/python3.9/site-packages/normalization/transform_config/transform.py", line 20, in run
    original_config = self.read_json_config(inputs["config"])
  File "/usr/local/lib/python3.9/site-packages/normalization/transform_config/transform.py", line 305, in read_json_config
    with open(input_path, "r") as file:
FileNotFoundError: [Errno 2] No such file or directory: 'destination_config.json'
I've tried locally and the command does in fact fail:
Copy code
`
export REPO="<https://levanhai82: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@github.com/levanhai82/datam.git>"
docker run --rm --init -i -w /data/70/1/transform --log-driver none --name normalization-normalize-70-1-puevj --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e AIRBYTE_VERSION=0.37.0-alpha airbyte/normalization:0.1.77 configure-dbt --integration-type postgres --config destination_config.json --git-repo "${REPO}"
Running: git clone --depth 5 --single-branch $GIT_REPO git_repo
Cloning into 'git_repo'...
Last 5 commits in git_repo:
7b6d2d8 Create destination_config.json
418171b first commit
/data/70/1/transform
Running: transform-config --config destination_config.json --integration-type postgres --out /data/70/1/transform
Namespace(config='destination_config.json', integration_type=<DestinationType.POSTGRES: 'postgres'>, out='/data/70/1/transform')
Traceback (most recent call last):
File "/usr/local/bin/transform-config", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.9/site-packages/normalization/transform_config/transform.py", line 335, in main
TransformConfig().run(args)
File "/usr/local/lib/python3.9/site-packages/normalization/transform_config/transform.py", line 20, in run
original_config = self.read_json_config(inputs["config"])
File "/usr/local/lib/python3.9/site-packages/normalization/transform_config/transform.py", line 305, in read_json_config
with open(input_path, "r") as file:
FileNotFoundError: [Errno 2] No such file or directory: 'destination_config.json'
a
Hey @Hai Le Van could you please post on our forum ? We moved the community support there for troubleshooting.