https://datahubproject.io logo
Join Slack
Powered by
hi guys, i try to ingest some matadata from biguqe...
# ingestion
f
hi guys, i try to ingest some matadata from biguqery and i got this er
Copy code
UserWarning: Cannot create BigQuery Storage client, the dependency google-cloud-bigquery-storage is not installed
. Another college try end he dosent have this error My setup
Copy code
DataHub CLI version: 0.8.19.1
Python version: 3.8.2 (default, Nov  4 2020, 21:23:28) 
[Clang 12.0.0 (clang-1200.0.32.28)]
google-api-core==2.3.2
google-api-python-client==2.33.0
google-auth==2.3.3
google-auth-httplib2==0.1.0
google-cloud==0.34.0
google-cloud-appengine-logging==1.1.0
google-cloud-audit-log==0.2.0
google-cloud-bigquery==2.31.0
google-cloud-bigquery-storage==2.10.1
google-cloud-core==2.2.1
google-cloud-logging==2.7.0
google-cloud-pubsub==2.9.0
google-cloud-secret-manager==2.8.0
google-cloud-storage==1.27.0
google-crc32c==1.3.0
google-resumable-media==2.1.0
googleapis-common-protos==1.54.0
grpc-google-iam-v1==0.12.3
His setup
Copy code
DataHub CLI version: 0.8.19.1
Python version: 3.9.9 (main, Nov 21 2021, 03:16:13) 
[Clang 13.0.0 (clang-1300.0.29.3)]
1:58
google-api-core==2.3.2
google-api-python-client==2.33.0
google-auth==2.3.3
google-auth-httplib2==0.1.0
google-cloud==0.34.0
google-cloud-appengine-logging==1.1.0
google-cloud-audit-log==0.2.0
google-cloud-bigquery==2.31.0
google-cloud-bigquery-storage==2.10.1
google-cloud-core==2.2.1
google-cloud-logging==2.7.0
google-crc32c==1.3.0
google-resumable-media==2.1.0
googleapis-common-protos==1.54.0
grpc-google-iam-v1==0.12.3
Its looks similaryl, idk why this is happaning. Thx a lot 🙏
d
I asked internally and this warning should not affect your ingestion. It is raised from the lib we use.
Please, can you confirm the ingestion works regardless this warning?
f
i will run an ingestion to confirm if i have the linage
the tables/views are in datahub but i think i dont have the linage (i will do a1 year ingestion. to confirm it)
d
thanks
f
this is the shink report
Copy code
{'records_written': 552,
 'warnings': [],
 'failures': [],
 'downstream_start_time': datetime.datetime(2022, 1, 21, 17, 24, 15, 856626),
 'downstream_end_time': datetime.datetime(2022, 1, 21, 17, 29, 52, 983898),
 'downstream_total_latency_in_seconds': 337.127272}
i have the tables/views pot i dont have the lineage
i also try to do it with k8s jobs and linkedin/datahub-ingestion:head. same error
and the datahub verison on k8s its 0.8.23
I can try other helm version, maybe it will work
any news? 😄
d
Do you see any lineage or nothing at all?
f
nothing 😞
i think i found the problem. If the view its create via interface, not query, the lineage dosent work.
😅
d
ahh, cool, yes we follow lineage based on queries
f
its posible to add a tichet for future feature?
Open in Slack
PreviousNext
Powered by