Hi, folks! Having trouble with this error:

Provided Schema does not match Table

when I edit 'select' section in meltano.yml. Is there a way to recreate one specific table on the next run of meltano elt with new settings and keep other tables and their incremental state intact?

@den_usenko Can you share which tap you’re using that’s giving you this error?

Is there a way to recreate one specific table on the next run of meltano elt with new settings and keep other tables and their incremental state intact?

Manually running

meltano elt <tap> <target> --select=<stream/table> --full-refresh --job_id=<job_id>

should do the trick. That run should reload

stream

, and subsequent runs should use the combined state of that run and the earlier incremental ones

Hello, Douwe. Thank you for your quick response. That's

pipelinewise-tap-mysql

and

target-bigquery

Can you share the

select

config in

meltano.yml

? Do the names you’re using there match what you see in

meltano select <tap> --list --all

?

I guess

select

config is fine and the problem is on the

target-bigquery

side. I've added few new fields to

select

(--list --all shows tables I need and the '*' symbol does the job right) and got error 400: Provided Schema does not match Table. Before your response I've deleted the table in question manually in BigQuery to get rid of mismatch situation. That helped. On the next elt run this table was created from scratch. Example of my selects:

select:

- '*a_users_orders.orders_id'

- '*a_users_orders.users_id'

- '*a_users_orders.orders_paid'

In my case, adding a few more lines like this caused an error. I expected full table reset (drop and create with new set of fields) in case of schema mismatch, but probably that's just out of the scope of elt job :)

Ah right, makes sense that it’s caused by a schema mismatch. I think target-bigquery could handle those better, by updating the destination table instead of failing, but that would need to be contributed to the target itself; there’s not much Meltano can do there