This message was deleted.
# generals
Slackbot
09/08/2023, 12:27 PMThis message was deleted.
d
David McHealy
09/08/2023, 1:07 PMEach ingestion writes to its own segment files its own columns, so they don't interfere with each other. Each query queries columns that exist on any segment within the range of the query intervals. If there are segments with the same column name of different types, that will query error any query that touches each different type of segment. You can go back and fix old segments. You can also just make the decision to ingest to a new column name, then reingest old data to convert data from the old column to the new.
s
Saurabh Pande
09/08/2023, 1:10 PMHi @David McHealy, I want to know more about whether directly updating ingestionSpec and submitting it to supervisor would start impacting ingestion tasks? will tasks start failing?
d
David McHealy
09/08/2023, 1:10 PMNo, it will kill the existing tasks, they will write whatever data they have outstanding, then new tasks are created and those upload new files.
s
Saurabh Pande
09/08/2023, 1:13 PMGot it, thanks, I also had similar understanding but we did such update in couple of our ingestionSpecs and some tasks started failing. Some were due to parsing exception and for some, there were no any specific logs as such so couldn’t find exact reason hence we reverted.
d
David McHealy
09/08/2023, 1:58 PMEven if the existing streaming tasks die unexpectedly, the new ones will rewind in kafka back to the point where those ones began, so you will not lose any data.
✅ 1
David McHealy
09/08/2023, 1:59 PMIf the new ones were dying due to parsing errors, my guess is that the new spec had some problem with the stream of data it was reading.
s
Saurabh Pande
09/08/2023, 2:03 PMYeah, so basically what we did is, we removed one metric and added new dimension. Now, there can be some data in kafka yet to be read which doesn’t have this newly added dimension but have that field on which we were creating metric(which is removed now)
Saurabh Pande
09/08/2023, 2:06 PMWe’ll recheck our updated ingestionSpec. Thanks for your help 🙌
Just a question, Is there any manual way exists that we can verify our ingestionSpec with the message that we have in kafka stream?
d
David McHealy
09/08/2023, 3:14 PMMy guess, the new task just started a little too early in the kafka stream and a field it expected to exist didn't exist in some of the messages yet. I usually let things run for awhile after I add data to make sure everything that is drawing gets the new field. No real trick to it really. Theoretically you could check the offsets mentioned in the stream to see where it is at and check using kcat that the event exists in those partitions at those offsets, but that's a bit manual.
David McHealy
09/08/2023, 3:15 PMAnd I think the default or at least our default is that each ingestion task spends an hour ingesting before it persists its segments, so I'm fine with waiting to see the tasks have all cycled before I update the ingestion.
David McHealy
09/08/2023, 3:15 PMOr you could just like, try and if it errors revert and wait and then eventually it won't error anymore.
s
Saurabh Pande
09/08/2023, 4:57 PMOhh, got it, thanks!
2 Views