This message was deleted.

To add to this, after waiting a few minutes without making any changes to the database, the query returns the correct results. It's almost as if it is the last message to come in, it doesn't work but once another message comes in, it's fine.

Hmmm ... I just tried this and could not reproduce ... it works for me. Fyi I am on the latest Imply release so equivalent to Druid 27 I think ... and I have useDefaultValueForNulls=false, if that makes any difference here. Here is my ingest and query SQL if you want to try it out separate from your existing datasource:

INSERT INTO test_array_offset 
WITH tmp AS (
  SELECT * FROM TABLE(EXTERN(
    '{"type":"inline","data":"pam_unix(sshd:session): session opened for user root by (uid=0)\npam_unix(sshd:session): session opened for user root by (uid=0)\n"}',
    '{"type":"csv","findColumnsFromHeader":false,"columns":["message"]}',
    '[{"name":"message","type":"string"}]'
  ))
)
SELECT message from tmp
PARTITIONED BY ALL

... and query ...

select message, ARRAY_OFFSET(STRING_TO_ARRAY(message,' '),ARRAY_OFFSET_OF(STRING_TO_ARRAY(message,' '),'user')+1)
from "test_array_offset"

I am ingesting from a kafka stream, I wonder if that has anything to do with it. I am running 27 as well but not the imply release

Hi Forrest, yes if you are using streaming ingestion then the query is likely accessing the real-time segment ... there could be some behavior in querying the real-time segments with this particular array function that is suspect. I will ping someone who will know more than me about this ...

Thanks John, I really appreciate it!

Hi @Forrest Holifield, sorry this thread is old, I don't remember who I pinged about it. However, I was able to reproduce it just now in my test environment, setting up a streaming ingestion with a test data generator ... I can confirm that the "bad" value is produced when I am querying the most recent records, which I presume are from real-time segments ... but works fine on older data, which I presume are from historical segments. It may not even be all real-time ... could just be the data that is sitting in the row buffer prior to persist. I will report this internally ...

Awesome! Thank you! I noticed that same behavior in my non extensive testing. It seems to just be the latest entry.