https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
r
I think it always restart the task, even if nothing changes, but if your ingestion is kafka based you should not lose any data, just a hiccups in querying if you don't have any replica for the MM/peon task
v
in general an ingestion spec is NOT idempotent
for example the ingestion spec could be set to append data (in native batch ingestion you could set 
appendToExisting: true
, in SQL based ingestion you could use 
INSERT
vs 
REPLACE
). In that case every ingestion spec will add data to your datasource (so COUNT(*) would increase)
also for batch data ingestion even if you are doing 
REPLACE
or 
appendToExisting: false
it will ingest the data at the time of ingestion, so if you have added or removed data from your source (like S3) between submitting the specs then different data will be ingested (I hope that is obvious and intuitive)
for streaming ingestion the supervisor spec defines the supervisor which then launches tasks (with their own specs). streaming ingestion (supervisor) specs are idempotent AFAIK. You can only have one supervisor per datasource
m
thanks for the quick response - so if I am understanding right: submitting a streaming ingestion spec reading from a kafka topic - submitting an identical spec later on will will create a new supervisor but there shouldn't be any duplicated or dropped messages into that data source
v
yeah if you resubmit a supervisor spec it just recreates the supervisor with a new version. It will shut down all the tasks made by the old version supervisor and make new ones that incorporate whatever changes you made to the spec. The entire operation should be seamless as it is all done in a rolling manner
d
ah, this is great to know. Thank you!
4 Views
Open in Slack
PreviousNext
Powered by