https://datahubproject.io logo
Join Slack
Powered by
Howdy - seeing some inconsistent behavior with ing...
# troubleshoot
d
Howdy - seeing some inconsistent behavior with ingesting DBT via inline config and calling 
pipeline.run()
in a script vs. 
datahub ingest -c dbt.yml
. Using the datahub cli, we can ingest our dbt stuff fine with some warnings generated. When calling 
pipeline.run()
, we notice that it’s unable to emit data to GMS. This feels specific to DBT as we use the inline config for redshift -> datahub and it works fine
g
what error do you get when you ingest via pipeline.run()?
d
the 
errors
section of the JSON payload says:
Copy code
SinkReport(records_written=685, warnings=[], failures=[{'error': 'Unable to emit metadata to DataHub GMS'
@green-football-43791 ^
g
anything else? any errors in your gms logs?
d
ah let me look there
well I don’t see anything glaring, but I have a long stacktrace that’s in our airflow logs
Copy code
datahub.configuration.common.PipelineExecutionError: ('Sink reported errors', SinkReport(records_written=685, warnings=[], failures=[{'error': 'Unable to emit metadata to DataHub GMS', 'info': {'exceptionClass': 'com.linkedin.restli.server.RestLiServiceException', 'stackTrace': "com.linkedin.restli.server.RestLiServiceException [HTTP Status:500]: com.datahub.util.exception.RetryLimitReached: Failed to add after 3 retries\n\tat com.linkedin.metadata.restli.RestliUtil.toTask(RestliUtil.java:42)\n\tat com.linkedin.metadata.restli.RestliUtil.toTask(RestliUtil.java:50)\n\tat com.linkedin.metadata.resources.entity.EntityResource.ingest(EntityResource.java:178)\n\tat sun.reflect.GeneratedMethodAccessor255.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
I don’t know if that helps any
g
you’re sure theres no error reported on the gms side?
500s should also print out in the GMS logs- that may be helpful in debugging.
d
i’ll try running it again and see if I can correlate any error there
g
ok, thanks!
d
ah found stuff
Copy code
Caused by: io.ebean.DuplicateKeyException: Error when batch flush on sql: insert into metadata_aspect_v2 (urn, aspect, version, metadata, createdOn, createdBy, createdFor, systemmetadata) values (?,?,?,?,?,?,?,?)
dupes
Copy code
Caused by: java.sql.BatchUpdateException: Duplicate entry 'urn:li:dataset:(urn:li:dataPlatform:xxx' for key 'metadata_aspect_v2.PRIMARY'
g
ah, is there any chance, that you have two urns that are identical except for casing?
d
hmm, we aren’t generating the URNs so I don’t know of a good way of answering that, we theoretically shouldn’t in the sense that dbt and redshift should differentiate each other by platform
but it seems like we have multiple errors for duplicates
g
out of curiosity, when did you start up your datahub instance first?
d
and then in redshift, we shouldn’t have any dupe object names otherwise we’d hit collisions
two weeks ago
and as I mentioned, we don’t get the same error when using the CLI
g
oh-- what version of the pipeline are you using?
we had some casing issues w/ dbt & redshift fixed somewhat recently actually
this could be it
if you’re on different versions
d
Copy code
acryl-datahub            0.8.27.1
g
and then what about the pipeline class?
d
hmm easy way to find that out?
g
how are you importing it?
d
ooh
there it is
on our docker instance 
acryl-datahub
is 0.8.17.1
but on my localhost it’s 0.8.27.1
sounds like the culprit
g
ahhhh !
that is definitely it
8.17.1 had that redshift/dbt bug
d
unfortunately we have a bit of a deadlock situation:
Copy code
cker-local-runner-1  | The conflict is caused by:
docker-local-runner-1  |     apache-airflow[package-extra] 2.0.2 depends on markupsafe<2.0 and >=1.1.1
docker-local-runner-1  |     acryl-datahub[redshift,redshift-usage] 0.8.27.1 depends on markupsafe==2.0.1
docker-local-runner-1  |     The user requested (constraint) markupsafe==1.1.1
and we can’t easily upgrade airflow
g
Hmm… I see- @dazzling-judge-80093 do you know anything about this?
I know Tamas has been working on airflow compatability lately
d
I just validated that switching to airflow 2.2.2 won’t solve the issue, as it is still pinning 
MarkupSafe
to 1.1.1
interestingly markupsafe introduced a bug in 2.1.0 that caused airflow to be defensive and pin it to < 2: https://github.com/apache/airflow/blob/main/setup.cfg#L136-L141
e
Sounds like the easiest way to deal with this would be to change the requirement on datahub’s end to markupsafe >=1.1.1, markupsafe < 2.0
d
@green-football-43791 what do you think of ^
hey @green-football-43791 - just wondering if you got a chance to see this or our PR to try and resolve it? pr link: https://github.com/linkedin/datahub/pull/4388
g
cc @square-activity-64562 who has spent some time dealing w/ airflow compat issues
I don’t feel qualified to review this one 😬
d
appreciate it
s
Open in Slack
PreviousNext
Powered by