Hi we just experienced an issue where a server restarted and Apache Pinot #troubleshooting

Hi, we just experienced an issue where a server re...

Elon

06/14/2021, 10:52 PM

Hi, we just experienced an issue where a server restarted, and when downloading a segment from gcs it threw an

java.io.IOException: Input is not in the .gz format

- we saw that the segment was just being written to gcs. Once I deleted the segment on the server and restarted it downloaded it without any issues. Has anyone ever experienced that before? I can create a github issue w some ideas for fixes...

Mayank

06/14/2021, 10:54 PM

Was it an existing segment being overwritten? Typically, the .tar.gz should be ready before server attempts to read it

Elon

06/14/2021, 10:54 PM

yep, it was existing

Elon

06/14/2021, 10:55 PM

And right after the error occurred I manually downloaded from gcs, and it was valid tar.gz

Elon

06/14/2021, 10:55 PM

Could it have been in the middle of overwritting the old and new versions? i.e. gcs is not like a posix filesystem, maybe there were some concurrency issues?

Mayank

06/14/2021, 10:56 PM

Yeah i think that is what it is. Is a move atomic in GCS?

Elon

06/14/2021, 10:57 PM

There is no move, it's copy and delete

Elon

06/14/2021, 10:57 PM

Since it uses streaming to write it, could it have been in the middle of writing or deleting?

Mayank

06/14/2021, 10:57 PM

If there are no atomic constructs to do so, I am unsure how to handle this.

Mayank

06/14/2021, 10:58 PM

Not that I have put much thought to it

Elon

06/14/2021, 10:58 PM

Maybe write some other "signal" file? or updated it (gcs has generation # in metadata) - i.e. just a pointer to the real file?

Elon

06/14/2021, 10:58 PM

yeah, I just put 1 thought into it right now 🤣

😁 1

Open in Slack

Previous Next