Hi Team. Very happy with openobserve so far. I run into an issue with a pipeline we built. It is a very simple 1h aggregation pipeline and it works fine running every hour but it does not want to work with the backfill. I set up a backfill for the last 10 days and it processes but the data does not appear in the stream. is there an obvious way to fix that? thx! David.

Any error do you see for the backfill jobs on the UI? Also make sure you have used the _timestamp field (say, use

max(_timestamp) as _timestamp

) in the select clause in the pipeline query. Because if the _timestamp is not selected, o2 will add a _timestamp with the epoch of processing timestamp and not the timestamp the data is from.

thx for you help! no error as far as i can see. this is the query: SELECT histogram(_timestamp, '1 hour') AS _timestamp, account, COUNT(*) AS request_count, SUM(bytes_sent) AS bytes_sent_total FROM "cdn" WHERE account IS NOT NULL AND account != '' GROUP BY 1, 2 ORDER BY 1 ASC

for this query, o2 will still use the backfill processing time as the _timestamp value. o2 expects the _timestamp to be in epoch timestamp in microseconds. For

histogram(_timestamp,..) as _timestamp

will not work as expected because, the histogram returns the timestamp in human readable string (not epoch microseconds). Hence, o2 will replace the _timestamp with current timestamp. You can try something like -

SELECT
  max(_timestamp) AS _timestamp,
  account,
  COUNT(*) AS request_count,
  SUM(bytes_sent) AS bytes_sent_total
FROM "cdn"
WHERE account IS NOT NULL
  AND account != ''
GROUP BY
  1, 2, histogram(_timestamp, '1 hour')
ORDER BY
  1 ASC

changed it to:

SELECT
  MAX(bucket_timestamp) AS _timestamp,
  account,
  COUNT(*) AS request_count,
  SUM(bytes_sent) AS bytes_sent_total
FROM (
  SELECT
    CAST(FLOOR(_timestamp / 3600000000) AS BIGINT) * 3600000000 AS bucket_timestamp,
    account,
    bytes_sent
  FROM "cdn"
  WHERE account IS NOT NULL
    AND account != ''
) t
GROUP BY
  bucket_timestamp,
  account
ORDER BY
  bucket_timestamp ASC

But still no backfill working. the server logs suggest everything is working but it simply does not write the rows. Any more ideas?

Great 🚀