https://datahubproject.io logo
Join Slack
Powered by
Hi team, I have enable table lineage and view line...
# all-things-deployment
c
Hi team, I have enable table lineage and view lineage still I am unable to see anything in my objects Please help @early-lamp-41924 and @big-carpet-38439
b
Hi @creamy-van-28626! Which source are you using?
c
Snowflake
b
@dazzling-judge-80093 Do you know why this might be? perhaps you can share the obfuscated recipe with us - it will help us triage
c
Let me share you
This recipe I am using
Lineage option is active but it has no data
b
got it - thank you. @square-activity-64562 Can you also take a pass? I don't see anything obvious - are you sure that Dataset->Dataset lineage exists in side of snowflake (just to confirm)
c
@square-activity-64562 can you please confirm on this ?
b
@creamy-van-28626 Do you expect to see Dataset->Dataset lineage for this table?
c
Yes for every object in every table I am unable to see lineage it's showing 0 upstream and 0 downstream
m
@creamy-van-28626 you may want to include 
ignore_start_time_lineage: True
for the initial load
b
^^ thank you Steve!
c
It not working my recipe.yaml is not running now
m
@creamy-van-28626 Can you paste your config here? And could you screen grab instead of taking photo of the screen, it is very hard to read.
c
My recipe.yaml
m
Which version of datahub do you use?
c
By adding ignore lineage here it is giving validation error
It is 0.8.31
m
That is strange because i also use 0.8.31
c
This error
m
If you can paste the recipe as text format, it would be better because i think you have indentation issue with the recipe
c
I am currently working in VDI unable to access datahub slack from there
m
Can you check if you have correct indentation?
c
These parameters of lineage should come under config Right?
d
yes
m
I dont know but from profiling section toward the end doesnt look like it lined up with database pattern section
c
m
You would need the 
ignore_start_time_lineage
for the initial load, otherwise it only looks back 1 day for object creation, you might miss a bunch
c
I have removed the pattern sections can you che k now
m
Can you 
:set noexpandtab
in your vim?
c
Now the validation error for all three
m
Your indentation is not right. You have to fix your file. You cannot mix tab and space in your yaml file.
As i said earlier, open the file in vim and 
:set noexpandtab
you can see where you have the wrong indentation
c
I am unable to set that
m
@creamy-van-28626 you have indentation issue with your yaml config file, You have to fix that. Unless you fix it, your ingestion will never work
c
I have checked the indentation only for ignore start time lineage
Validation error is there
m
@creamy-van-28626 if the validation error is still there, your config is still not properly formatted. How did you check your indentation?
c
I have checked by using same recipe in UI it is giving valid yaml
m
But the portion you added in doesnt have the right indentation. Maybe you can edit the recipe on the ui to add the configuration for lineage
s
Does it run successfully through the UI based ingestion? If yes, maybe you can use UI based ingestion. Is there a reason you cannot use UI based ingestion? Regarding the tabs issue in the file can you please share the reasoning why you cannot use the suggestion by @modern-artist-55754 about
Can you 
:set noexpandtab
in your vim?
Can you use 
sed
on that machine? https://stackoverflow.com/questions/31903432/using-sed-to-replace-tab-with-spaces Maybe just run this or something similar to replace them manually to spaces?
Have you verified that your snowflake instance is on Enterprise edition? Please see https://datahubproject.io/docs/metadata-ingestion/source_docs/snowflake/#caveats-1
c
Actually ingestion from UI is not working that's why we are ingesting from cron job itself by creating configmap
This error we are getting from UI ingestion
s
Is DataHub working on • on a k8s cluster (AWS EKS, GCP GKE etc). • in a air-gapped environment My guess is it is the second case. But please confirm.
If an air-gapped environment then the reason it is not working is because it is unable to connect to 
pypi
or whatever proxy there is to get the python package. You will need to connect to your company's infra team for this.
What is the image tag of 
actions
container which is being run?
c
Yes it's air gapped environment working in VDI
m
Copy code
source:
  type: snowflake
  config:
    env: $METADATA_ENV
    # Coordinates
    host_port: $SNOWFLAKE_ACCOUNT
    warehouse: $SNOWFLAKE_DEFAULT_WAREHOUSE

    # Credentials
    role: $SNOWFLAKE_DEFAULT_ROLE
    username: $SNOWFLAKE_DEFAULT_USER
    password: $SNOWFLAKE_DEFAULT_PASSWORD

    include_tables: true
    include_views: true

    include_table_lineage: true
    bucket_duration: DAY
    ignore_start_time_lineage: true

    table_pattern:
      deny:
        - .*dbt.*tmp
sink:
  type: "datahub-rest"
  config:
    server: $DATAHUB_URL
c
I have installed both snowflake and snowflake usage on actions image
m
you can copy and paste this config & replace with your parameters
i used this for my ingestion and it works fine.
c
It's 0.8.31
s
Maybe the container restarted after that and the installation got wiped out? If it was already installed then it should have used it from the pip cache.
Please try out what @modern-artist-55754 said with the recipe copy-paste. If you already have the package in cache then it should work via cron. Ideally the UI based ingestion should work if the package is installed there.
c
From cron ingestion is working only lineage is not set
Lineage option is active but it is showing 0 upstream and 0 downstream
I tried to you use same recipe as shared by Steve but only for ignore-start-time-lineage it is giving validation error apart from that all other attributes in recipe are working
m
I am using 0.8.31.6, maybe try that version
c
Okay
s
@creamy-van-28626 Have you verified that your snowflake instance is on Enterprise edition? Please see https://datahubproject.io/docs/metadata-ingestion/source_docs/snowflake/#caveats-1
c
Means you are saying I should try source type as snowflake usage for lineage attributes ?
s
Is your snowflake instance on Enterprise edition?
Lineage does not show up on standard edition
c
How can I check it is enterprise or standard one ?
s
Login to the UI. Click on the name in top right.
c
It is coming this way
s
You need to login to your snowflake instance
c
Enterprise edition
s
Did you try https://datahubspace.slack.com/archives/CV2UVAPPG/p1650448830534269?thread_ts=1650374602.905759&cid=CV2UVAPPG this with the latest CLI version. In the 
0.8.31.x
versions there were many fixes done on snowflake source
c
I am currently using 0.8.31 version of datahub
Need to upgrade that?
All other things are working but this ignore start time lineage is giving validation error
s
You can use this option https://datahubproject.io/docs/ui-ingestion#advanced-running-with-a-specific-cli-version to specify a new version of CLI. Suggest to use https://github.com/acryldata/datahub/releases/tag/v0.8.31.6 so server upgrade is not required.
c
I can change the version of acryl datahub from advanced tab right?
s
normally that would work. But given this is an air-gapped env you will need to install that version of CLI in your actions pod the same way you installed it earlier
c
Got it need to install 0.8.31.6 version datahub action image ?
s
There is no separate image of actions currently for each DataHub CLI. The CLI is installed inside it
You had mentioned earlier you installed it so I am assuming you know how to do that
c
I have pulled acryl datahub action of 0.8.31.6 version image from docker hub in my gitlab and over that image I have installed snowflake and snowflake usage package of 0.8.31 and that image I have to deploy kubernetes
s
No. As I said in earlier comment need to install https://github.com/acryldata/datahub/releases/tag/v0.8.31.6
c
Okay I will do that
One thing I am also facing I have deployed the same code in other environment as well but there I am getting this error while fetching the schema itself
Code 500 I am getting in this environment
s
Can we please create another thread for this so we don't mix two things and things get lost? Briefly this means you will need to check logs to see the exact error. 500 error means a server error. Without that stacktrace it isn't possible to help out
c
Hi @modern-artist-55754 and @big-carpet-38439 , I have added this ignore start time lineage as true in my recipe , it ran and ingested the data but still in UI it showing like this
s
Hi @creamy-van-28626 As mentioned in the DMs can you try this? • create a test database in snowflake • run these commands in them
Copy code
create or replace table base_table (date_utc timestamp, revenue INTEGER);
insert into base_table values  (CURRENT_TIMESTAMP(), 100);
create or replace table lineage_from_base as (select * from base_table);
• wait 3 hours. It might work sooner but just to be safe this is maximum time you have to wait. • run the ingestion on this test database in snowflake. • check whether you can see lineage
Please note the lineage tab will only be populated IF there is some lineage. It won't populate for all cases. You have to check a table which has actual lineage.
c
Okay let me check
s
@creamy-van-28626 Did this work out?
c
I was on leave last Friday will check today on this
It worked now my UI ingestion is also working
👍 1
b
Amazing!
Open in Slack
PreviousNext
Powered by