https://linen.dev logo
Join Slack
Powered by
Hey there again! About normalization in snowflake:...
# announcements
a
Hey there again! About normalization in snowflake: I’m testing the Jira connector. The data gets synced correctly to the raw tables, however the normalization is not doing a great job with that data. I have found the DBT folders in the workspace but i am struggling to understand how the normalization is actually being set-up by airbyte and if it’s possible to modify it to be improved. Anyone can help me with this?
✅ 1
c
you can export it and run it yourself outside of airbyte but it’s not possible to submit your modifications back to airbyte for the moment
it would be nice to report the normalization error also so we can fix it and make it more robust
a
There’s no particular error happening, it’s more like the fields selected for the normalization are very limited compared to the amount of data available in the raw table
c
Are the missing data of type objects? (nested struct, array?)
a
let me check
no they are not objects
some of them are but not all of them
can you point me at the portion of the code in Airbyte that takes care of the normalization?
c
it’s here 
airbyte-integrations/bases/base-normalization/normalization/transform_catalog/transform.py
but i am currently heavily re-working it to handle nested objects
👀 1
a
Still on this topic: when i try to run 
docker run --rm -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/50/0/normalize --network host airbyte/normalization:0.1.10 run --integration-type snowflake --config target_config.json --catalog catalog.json
i get 
Error: No such container: airbyte/normalization:0.1.10
and in fact if i do 
docker container ls
it looks like the container is not there
c
if i do 
docker container ls
 it looks like the container is not there
It should be: 
docker images airbyte/normalization
?
a
oh you are right 😅
ok that works indeed
c
then how come your 
docker run
can’t find it then? 🤔
a
no if i run that command it actually works
but if i try to spawn the container with a shell it doesnt
but maybe i’m doing it wrong
docker run -it airbyte/normalization:0.1.10 /bin/bash
c
docker run --rm -i --entrypoint /bin/bash -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/50/0/normalize --network host airbyte/normalization:0.1.10
a
you savior 🙌
😎 1
btw @Chris (deprecated profile) about the Jira normalization again, the column i’m missing is indeed an object. I was wrong before because it looked like many fields in the json, but it is actually one containing all of them
c
👌
i’m going to fix that soon
a
will this be available at the destination (snowflake) in one column as an object type?
c
will try yes
🙌 1
3 Views
Open in Slack
PreviousNext
Powered by