This message was deleted.

web console shows many stats around query execution. can you share those stats with us?

Hi Bharat, for Step #1 you have to create new segment(s) using valid time chunk(s). If those segments comprise many GB of data then 20 minutes may not be unreasonable to expect to do this operation. However you do not have to create new segments that are same size as the originals ... if you data is clustered around specific times then you only need re-ingest those times, using valid time partitions ... for example if the data all falls within a couple of one or two hour time chunks on a given day, then you can do a REPLACE/OVERWRITE using just those hours instead of the entire day. However if your delete is interspersed evenly throughout the data in the segments then you may have no other choice than to do the full replace as you are doing. It looks like Abhishek is asking for info that would relate to performance tuning here.

Hey @Abhishek Agarwal @John Kowtko Thanks for looking into the issue. @Abhishek Agarwal I am using sql API to trigger this query so I don't have any stats from console. Do we have any way to get stats when we trigger via apis? @John Kowtko total data size of these segments is ~800mb which is taking 20-30mins. Also, I are having the data on Day level. Also, i think we are doing the same thing that you are suggesting us. We are not updating whole segment, we are just replacing/overwriting a set of rows with respect to the column name.

@Bharat Kumar Singh when using the API to submit queries, you can still use the web console to get stats using the "attach query by ID" feature. It'll pop up a tab in the console with your SQL and a bunch of useful counters