A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

image.png

Trying to connect to Redshift but getting 'Unable to emit metadata to DataHub GMS'. What would be the cause for this?

Hi - Could you try setting sink.config.max_threads: 1 in your config and try it again? this thread might be helpful
<https://datahubspace.slack.com/archives/CUMUWQU66/p1684170546906109?thread_ts=1682332234.663259&amp;cid=CUMUWQU66>

Thanks <@U04QRNY4ZHA>. Will try this
<@U05B84TDRUL> <@U05AYPD0527> FYI

<@U04QRNY4ZHA> where do we find the "sink.config.max_threads: 1" is it in datahub deployment file or anywhere else?
It's bit new for us

How are you deploying your datahub instance? is it quickstart or helm chart?

<@U04QRNY4ZHA> Trust you are well! Sorry but would it be possible to let us know where this setting needs to be made. Just FYI, we are ingesting metadata from Redshift

This settings would be made in the YAML recipe: <https://datahubproject.io/docs/metadata-ingestion/sink_docs/datahub#config-details>

<@U04QRNY4ZHA> Thanks. Sorry but where can we find this YAML recipe. Very new to Datahub

Hi <@U04QRNY4ZHA> we have added the max_thread=1 in YAML file and updated. But still we see that intermittently when we run ingestion source on Redshift; error as follows;
 {
          "error": "Unable to emit metadata to DataHub GMS: javax.persistence.PersistenceException: Error when batch flush on sql: update metadata_aspect_v2 set metadata=?, createdOn=?, createdBy=?, createdFor=?, systemmetadata=? where urn=? and aspect=? and version=?",
          "info": {
            "exceptionClass": "com.linkedin.restli.server.RestLiServiceException",
            "message": "javax.persistence.PersistenceException: Error when batch flush on sql: update metadata_aspect_v2 set metadata=?, createdOn=?, createdBy=?, createdFor=?, systemmetadata=? where urn=? and aspect=? and version=?",
            "status": 500,
            "id": "urn:li:dataset:(urn:li:dataPlatform:redshift,dev.aso_schema.${google_keyword},PROD)"
          }
        }