A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

image.png

Hi Everyone, I'm having issues with ingestion from Metabase. I'm receiving this errors printed bellow.

Did ingestion process stop after this error and other chart/dashboard metadata weren't load?

Can anyone know what I need to do to solve this?

Hello Paulo,
I imagine the user declared in the data returned by Metabase's API is unavailable. Maybe is a deleted user.

I develop new Metabase ingestion for more recent versions of Metabase and improvements with the BigQuery integration. I pretend to open source it until the end of the next month. These improvements include the solution to this problem.

For now, you can ignore the failures because, at the code level, this error doesn't impact the ingestion.

Hi <@U0398JJG49W> tks for your feedback. The User was *Deactivated*, I activate and worked.  Now the only issue is the query.

```'failures': {'metabase-query': ['Unable to retrieve lineage from query. Query: SELECT\\n'\n"
           "                                 'vChamadaSuporte_TempoReal_Receptivo_ACC.CodCliente,\\n'\n"
           "                                 'vChamadaSuporte_TempoReal_Receptivo_ACC.NProtocolo,\\n'\n"
           "                                 'CASE\\n'\n"
           '                                 "WHEN ([dbo].[FC_FlagEtapaCliente] (MAX(vChamadaEtapa_TempoReal.CodEtapaChamada)) = \'NULL\' OR "\n'
           '                                 "MAX(vChamadaEtapa_TempoReal.DataHoraFim) IS NULL) THEN \'Sem Interação\'\\n"\n'
           "                                 'ELSE [dbo].[FC_FlagEtapaCliente] (MAX(vChamadaEtapa_TempoReal.CodEtapaChamada))\\n'\n"
           '                                 "END AS \'Status\',\\n"\n'

 "Reason: '\n"
           '                                 "\'NoneType\' object has no attribute \'strip\' ",\n'```

<@U0398JJG49W> If I ignore, can I considere that all other metadata from Metabase was loaded?

Yes! The user not found error implies that some dashboards/cards will be created without owner and linage parser errors will create cards without upstream lineage. But I'm sure that all dashboards and cards will be ingested, even the personal collections, if the user has the permission to list.

datahub uses `sqllineage` to parse lineage, and I found the library quite buggy sometime. Essentially you are missing some lineage because datahub cannot parse the query. It shouldn’t be a big problem

If you’re comfortable sharing, it’d be great to know what queries sqllineage fails to parse. At some point we’re going to either improve sqllineage, switch to a different parser, or build our own - it’d be good to have these sql queries for reference to make sure we improve :slightly_smiling_face:

<@U01GZEETMEZ> This is the current issue i have. <https://github.com/reata/sqllineage/issues/278>

I ended up switching to <https://github.com/macbre/sql-metadata/> which fit my use case, not necessarily replace what sqllineage in datahub

<@U030H0S9QUV> Sorry, I didn't understand what you did. Did you change Datahub Metabase ingestion Python code to use the package sql-metadata?

<@U03F8GMUB7G> Mine was tableau, i added a bit extra to the tableau ingestion code and make a custom source. It’s too specific to my case so i didn’t want to raise a PR against the open source project