Hey, I'm a bit confused by the whether the upsert capability in the ElasticSearch destination is usable with my connection. When setting up the destination in the UI, the tooltip on the upsert toggle says:

Upsert Records

If a primary key identifier is defined in the source, an upsert will be performed using the primary key value as the elasticsearch doc id. Does not support composite primary keys.

The source for my connection is BigQuery, which doesn't have primary keys. Instead I construct a surrogate key field called

id

. However, I can't select a primary key in the connection because the ElasticSearch destination doesn't support

Incremental Sync - Deduped History

.

Hi Kevin! I'm not too familiar with this particular connector, let me ask the team and get back to you!

I was actually able to figure it out last night! Using octavia, I modified the primary key in the connection's yaml and applied the change to the server. Running

Full Refresh - Overwrite

mode, the primary key is properly picked up and used as the document

_id

in ElasticSearch. My only suggestion here would be to clarify the requirements for enabling the Upsert in the documentation - that if your source system doesn't have primary keys, you should set one yourself in the connection's configuration.yml, as it is not possible in the UI.

Full Refresh - Overwrite

mode is also required.

Great to hear, thank you for the details on how you worked it out!

Hi @Kevin Grismore. Your above research is quite helpful. Is it possible for incremental-append sync mode? appreciated any suggestions ..! Thanks Santos