Hello Team,
I am facing some weird issue while ov...
# ingestiona
astonishing-yak-92682
07/01/2021, 4:30 AMHello Team,
I am facing some weird issue while overriding data. in v0.8.1
Tried ingesting one dataset with
urn - urnlidataset:(urnlidataPlatform:kafka,SampleKafkaDataset,PROD)
using pushed based architecture, everything worked fine (proper entry in es and mysql)
now added one new searchable aspect in the same urn and then tried to ingest
expected -> new aspect should present in mysql and es
actual result -> new aspect in only in es but not in mysql
Same case if we delete any aspect -
expected -> deleted aspect should not be in mysql and es
actual result -> removed aspect got deleted from es but still present in mysql
I also verified above steps using quickstart and ingestion docker scripts
Is this something expected or its an issue ?
e
early-lamp-41924
07/01/2021, 5:36 AMJust to clarify, you created a new aspect that you added to the DatasetAspect union which has a @Searchable field right?
early-lamp-41924
07/01/2021, 5:37 AMCan we get a bit more information here? We have never encountered this issue before and would like to better understand it.
early-lamp-41924
07/01/2021, 5:37 AMAlso how did you delete an aspect?
a
astonishing-yak-92682
07/01/2021, 5:40 AMi removed that aspect from mce message
for example if my entity have aspects A,B,C,D, and
at first time ingestion if my mce message contains only A,B,C aspects.
and next time mce message for same urn contans A,B,C,D aspects
So ideally D should also have an entry in sql
Same in case of removing aspect
Fisrt time my mce message have asspect A,B,C
and if next time if have only A,B
So data of C aspect should not be in mysql
Right ?
astonishing-yak-92682
07/01/2021, 5:42 AMI will update the complete step to reproduce.
With mce messages
e
early-lamp-41924
07/01/2021, 5:49 AMThe first should be the case. But the latter should not. For instance, ingesting one aspect doesn’t delete all other aspects.
early-lamp-41924
07/01/2021, 5:50 AMLet me look into why the first case is happening
b
big-carpet-38439
07/01/2021, 1:03 PMIn the first case, you're righ t that ABCD should all be in there
big-carpet-38439
07/01/2021, 1:03 PMIf they are not, there is an issue here (though we haven't seen this in the wild before)
a
astonishing-yak-92682
07/01/2021, 1:08 PMIt went intermittent. May be i had done something wrong, i nuked containers and it worked.
Sorry my bad 😶