https://datahubproject.io logo
Join Slack
Powered by
Hi guys, i tried to inget some bigquery metadata b...
# ingestion
f
Hi guys, i tried to inget some bigquery metadata but i get this strange error
Copy code
{'error': 'Unable to emit metadata to DataHub GMS',
Info
2022-02-01 12:48:18.653 EET
metadata-ingest
 'info': {'exceptionClass': 'com.linkedin.restli.server.RestLiServiceException',
Info
2022-02-01 12:48:18.653 EET
metadata-ingest
 'stackTrace': "com.linkedin.restli.server.RestLiServiceException [HTTP Status:400]: Parameters of method 'ingest' failed "
also i got this
Copy code
Pipeline failed. Not closing the source to prevent bad commits."
s
It would be helpful to see the recipe or pipeline that you are using.
Also, can you give the output of
Copy code
python3 -c "import sys; print(sys.version); print(sys.executable); import datahub; print(datahub.__file__); print(datahub.__version__);"
f
This is the script
Copy code
source:
    type: bigquery
    config:
      project_id:<project>
      max_query_duration: 5
      include_table_lineage: True
      profiling:
        enabled: true
sink:
  type: "datahub-rest"
  config:
    server: <ip>
And i use linkedin/datahub-ingestion:head image. I think is becausei try to run 2 paralel ingestion
s
Can you try to use 
0.8.23
instead of 
head
?
f
yea
Copy code
Would it be a problem if I try to run 2 ingestion in parallel?
s
Two ingestions of bigquery in parallel? Or different sources?
Please share the other recipe that you are running in parallel
f
bigquery, but for different project
it the same recepie, just other bq project
s
I don't expect that to happen. Can you try to run 1 recipe with 
0.8.23
and let me know if the problem is still happening?
Then we can try with 2 recipes and see if it still happens
f
sure šŸ‘
oh, v0.8.23 found it
s
That is 
v0.8.22
. 
v0.8.23
seems to be missing. Try with 
v0.8.24
instead of head?
f
yea, its missing :))
i will try with v0.8.24
i got this error
Copy code
'(Background on this error at: <http://sqlalche.me/e/13/4xp6)']>},
Info
2022-02-01 14:53:44.229 EET
metadata-ingest
 'tables_scanned': 920,
Info
2022-02-01 14:53:44.229 EET
metadata-ingest
 'views_scanned': 907,
Info
2022-02-01 14:53:44.229 EET
metadata-ingest
 'entities_profiled': 931,
Info
2022-02-01 14:53:44.229 EET
metadata-ingest
 'filtered': [],
Info
2022-02-01 14:53:44.229 EET
metadata-ingest
 'soft_deleted_stale_entities': [],
Info
2022-02-01 14:53:44.229 EET
metadata-ingest
 'query_combiner': {'total_queries': 32045,
Info
2022-02-01 14:53:44.229 EET
metadata-ingest
 'uncombined_queries_issued': 14222,
Info
2022-02-01 14:53:44.229 EET
metadata-ingest
 'combined_queries_issued': 1966,
Info
2022-02-01 14:53:44.229 EET
metadata-ingest
 'queries_combined': 18907,
Info
2022-02-01 14:53:44.229 EET
metadata-ingest
 'query_exceptions': 16}}
Info
2022-02-01 14:53:44.229 EET
metadata-ingest
Sink (datahub-rest) report:
Info
2022-02-01 14:53:44.229 EET
metadata-ingest
{'records_written': 2290,
Info
2022-02-01 14:53:44.229 EET
metadata-ingest
 'warnings': [],
Info
2022-02-01 14:53:44.229 EET
metadata-ingest
 'failures': [],
Info
2022-02-01 14:53:44.229 EET
metadata-ingest
 'downstream_start_time': datetime.datetime(2022, 2, 1, 12, 26, 56, 51767),
Info
2022-02-01 14:53:44.229 EET
metadata-ingest
 'downstream_end_time': datetime.datetime(2022, 2, 1, 12, 53, 44, 53582),
Info
2022-02-01 14:53:44.229 EET
metadata-ingest
 'downstream_total_latency_in_seconds': 1608.001815}
Info
2022-02-01 14:53:44.229 EET
metadata-ingest
{}
Info
2022-02-01 14:53:44.229 EET
metadata-ingest
Pipeline finished with failures
s
Did you miss the exception itself? Don't see any error
f
(google.cloud.bigquery.dbapi.exceptions.DatabaseError) ' Info 2022-02-01 145344.229 EET metadata-ingest Ā ā€˜400 Cannot access field Styles on a value ā€™
and i think i have it for multple tables
s
f
hmm, i have
Copy code
BigQuery Admin
BigQuery Job User
Logging Admin
i can try to add all of the roles from the link
but normaly i have acces to dataset/table/logs
but i cant find bigquery.datasets.get for example in the iam
Copy code
Sink (datahub-rest) report:
Info
2022-02-01 16:01:03.884 EET
metadata-ingest
{'records_written': 3205,
Info
2022-02-01 16:01:03.884 EET
metadata-ingest
 'warnings': [],
Info
2022-02-01 16:01:03.884 EET
metadata-ingest
 'failures': [],
Info
2022-02-01 16:01:03.884 EET
metadata-ingest
 'downstream_start_time': datetime.datetime(2022, 2, 1, 13, 35, 32, 188984),
Info
2022-02-01 16:01:03.884 EET
metadata-ingest
 'downstream_end_time': datetime.datetime(2022, 2, 1, 14, 1, 2, 594138),
Info
2022-02-01 16:01:03.884 EET
metadata-ingest
 'downstream_total_latency_in_seconds': 1530.405154}
Info
2022-02-01 16:01:03.884 EET
metadata-ingest
{}
Info
2022-02-01 16:01:03.884 EET
metadata-ingest
Pipeline finished with failures
I found something wierd. Even if it says
Copy code
Pipeline finished with failures
the failure array is empty
Copy code
'failures': [],
s
Can you please send a screenshot of the whole output? There has to be some error even if not in the failures array
f
also i got this
Copy code
{datahub.ingestion.source.ge_data_profiler:770} - Profiling 2962 table(s) finished in 1085.831 seconds"

WARNING  {datahub.ingestion.run.pipeline:178} - Pipeline failed. Not closing the source to prevent bad commits."

 INFO {datahub.cli.ingest_cli:83} - Finished metadata ingestion
s
These are not complete logs
f
i also found this
Copy code
'tables_scanned': 1434,
Info
2022-02-01 16:01:03.884 EET 'views_scanned': 1239,
Info
2022-02-01 16:01:03.884 EET 'entities_profiled': 2962,
Info
2022-02-01 16:01:03.884 EET 'filtered': [],
Info
2022-02-01 16:01:03.884 EET 'soft_deleted_stale_entities': [],
Info
2022-02-01 16:01:03.884 EET 'query_combiner': {'total_queries': 18598,
Info
2022-02-01 16:01:03.884 EET 'uncombined_queries_issued': 13162,
Info
2022-02-01 16:01:03.884 EET 'combined_queries_issued': 3444,
Info
2022-02-01 16:01:03.884 EET 'queries_combined': 12513,
and a lot of those things
s
This is GCP, right? Can you search for ā€¢ exception ā€¢ error
f
yea its gcp
s
In the above there is "Cannot access"
so access is missing
f
hmmmm, i think i found something. I tried to make a query oon python using the datahub service account and i got this
Copy code
User does not have bigquery.jobs.create permission in project am-datahub
i think the service cant make a query, i will try to add it
oh, wrong service account šŸ˜‘
The erros is
Copy code
'Cannot access field colors on a value with type ARRAY<STRUCT<colors BOOL>> at '
so , i think its a problem with array<struct columns
3 Views
Open in Slack
PreviousNext
Powered by