https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
s
Hi Sebastian. is this running on a single docker instance or on many as a cluster? How is Deep Storage configured? A common mistake when expanding into a cluster, is that the default for a single server config is to use Deep Storage "local" which does not work across multiple nodes. The symptom is usually ingestions that succeed, but no data is visible because historicals look at their local storage for "deep storage segments" and they have actually been written to the local storage of the middle manager that ingested the data. Setting Deep Storage to something that all the nodes can access is a critical element of cluster deployment.
If this is what you are facing, here are the docs on configuring Deep Storage : https://druid.apache.org/docs/24.0.0/configuration/index.html#deep-storage
s
I followed this tutorial to run with Docker. It is running on many containers, though I assumed this wasn't an issue - I had it working just last week without changing the 
environment
file or anything, but I decided to reset the cluster for testing purposes (
pruned
the containers), and only today did the issue appear.
s
It should be configured correctly in the docker-compose.yml. I'll test the tutorial.
docker compose uses local Deep Storage, but it will work because all the containers are using the "druid_shared" volume mount for deep storage at 
/opt/shared
in each container pointing to the same volume. So disregard my comments about Deep Storage. Without changing the environment file, I did:
Copy code
docker-compose up
I then ingested some data, and all was good, including the datasources and segments views. I tried to create a reindex job from one of the datasource: kttm_simple which I had loaded with (one of the SQL ingestion example):
Copy code
REPLACE INTO "kttm_simple" OVERWRITE ALL
SELECT *
FROM TABLE(
  EXTERN(
    '{"type":"http","uris":["<https://static.imply.io/example-data/kttm-v2/kttm-v2-2019-08-25.json.gz>"]}',
    '{"type":"json"}',
    '[{"name":"timestamp","type":"string"},{"name":"agent_category","type":"string"},{"name":"agent_type","type":"string"},{"name":"browser","type":"string"},{"name":"browser_version","type":"string"},{"name":"city","type":"string"},{"name":"continent","type":"string"},{"name":"country","type":"string"},{"name":"version","type":"string"},{"name":"event_type","type":"string"},{"name":"event_subtype","type":"string"},{"name":"loaded_image","type":"string"},{"name":"adblock_list","type":"string"},{"name":"forwarded_for","type":"string"},{"name":"language","type":"string"},{"name":"number","type":"long"},{"name":"os","type":"string"},{"name":"path","type":"string"},{"name":"platform","type":"string"},{"name":"referrer","type":"string"},{"name":"referrer_host","type":"string"},{"name":"region","type":"string"},{"name":"remote_address","type":"string"},{"name":"screen","type":"string"},{"name":"session","type":"string"},{"name":"session_length","type":"long"},{"name":"timezone","type":"string"},{"name":"timezone_offset","type":"long"},{"name":"window","type":"string"}]'
  )
)
PARTITIONED BY ALL TIME
I tried the to use Load Data to create a Reindex from Druid job and selected the source kttm_simple and I got the same result as you: "no data returned from sampler". After investigating it I realized that the SQL ingestion had not specified a "__time" column, so all rows were loaded with __time = 1970-01-01 000000. I adjusted the Interval in the sampler page to start at 1970-01-01 and I got the data back. Not sure if this is what you are seeing. I fixed it by re-running the ingestion with:
Copy code
REPLACE INTO "kttm_simple" OVERWRITE ALL
SELECT TIME_PARSE("timestamp") as __time, *
FROM TABLE(
  EXTERN(
    '{"type":"http","uris":["<https://static.imply.io/example-data/kttm-v2/kttm-v2-2019-08-25.json.gz>"]}',
    '{"type":"json"}',
    '[{"name":"timestamp","type":"string"},{"name":"agent_category","type":"string"},{"name":"agent_type","type":"string"},{"name":"browser","type":"string"},{"name":"browser_version","type":"string"},{"name":"city","type":"string"},{"name":"continent","type":"string"},{"name":"country","type":"string"},{"name":"version","type":"string"},{"name":"event_type","type":"string"},{"name":"event_subtype","type":"string"},{"name":"loaded_image","type":"string"},{"name":"adblock_list","type":"string"},{"name":"forwarded_for","type":"string"},{"name":"language","type":"string"},{"name":"number","type":"long"},{"name":"os","type":"string"},{"name":"path","type":"string"},{"name":"platform","type":"string"},{"name":"referrer","type":"string"},{"name":"referrer_host","type":"string"},{"name":"region","type":"string"},{"name":"remote_address","type":"string"},{"name":"screen","type":"string"},{"name":"session","type":"string"},{"name":"session_length","type":"long"},{"name":"timezone","type":"string"},{"name":"timezone_offset","type":"long"},{"name":"window","type":"string"}]'
  )
)
PARTITIONED BY ALL TIME
Even tested restarting the containers, which I am not sure means the same thing as "prune" that you mentioned. Can you explain what that entailed. I did not see any adverse effects after the restart, the data was still there and I was able to run a Reindex on it.
s
Unfortunately, I don't think we are having the same error. When I said I was following the tutorial, I meant for setup and running the containers. I am actually ingesting from Kafka, and have a defined 
__time
column, that contains correct data. However, when inspecting the spec, I do see 
__time
included in 
dimensionExclusions
under the 
dimensionsSpec
. This appears to be intended behavior, but I'm not sure if this is what is causing the issue.
s
No. And you can verify that you have __time by selecting a few rows. Any other aspects of what you are doing that might help me reproduce the issue? • I'll start a kafka ingestion. • When you restarted the pods, was the ingestion running? Or did you pause it? Recovery should work either way. It is just cleaner if you pause the supervisors. • anything else?
s
Yes, I have verified 
__time
by selecting some rows, they are correct (not Jan 1 1970). I think the ingestion was running, though I did a hard reset (by prune, I meant running 
docker volume prune
after shutting down the containers, so all metadata and historical data would get wiped).
oh, one other thing I did was rename the postgres container, as I previously had another postgres container with the same name on my machine. I have since tried with both the original name 
postgres
and the new name, neither of which change anything.
s
Oh! prune actually wipes the volumes. That explains a lot. Given that the setup uses local shared storage for Deep Storage, you effectively wiped the data from the database. You might have also wiped some configuration files, not sure. Let me try...
Copy code
docker volume prune
WARNING! This will remove all local volumes not used by at least one container.
Are you sure you want to continue? [y/N] y
Total reclaimed space: 0B
Did you prune while the cluster was shutdown?
s
yeah, while it was shut down
s
so, your intent was to clean the cluster and start fresh?
s
yes
s
I stopped the cluster, but the volume claims still exist, so prune is still not doing anything. If I restart the cluster after this... I don't see the same effect as you. So, now I'll delete the cluster, then prune and try this again...
before I do that, what sequence did you follow. Delete cluster, prune, then 
docker-compose up
again?
s
yep
s
ok, that worked and there is no data anymore, but what doesn't make sense is that you still had any metadata about datasources
s
oh I think you misunderstand, I didn't at first
but when I tried ingesting again after that, that's when I ran into the issue
s
okay, so the full sequence would be, in the new cluster, run kafka ingestion, then wipe the cluster (delete, prune), then rebuild and restart the ingestion by submitting the ingestion spec again. The you try a reindex off of that datasource and that's when it fails?
s
yeah, that's the full sequence
s
I think I may know what's happening. But I'll need to test this. If you restarted the kafka ingestion, it could be that it had not published any segments at that point, so reindexing from it would return nothing from the sampler. I need to confirm whether reindex reads from the real-time portion of the data. What are you trying to achieve with this? What is the purpose of the reindex? into the same data source or into a different one?
s
the reindex is into a new datasource. I'm simply testing different ingest strategies, and one of them involves using one datasource with data from kafka, then reindexing into a new datasource with transforms and rollup
s
makes sense. I'll try this out and come back with my result. You may need to put a delay on the reindex such that you get stable data segments at reindex time. In other words, only reindex the stuff that is already in immutable files.
s
I see, and what about the 0s in the metadata? Thanks for your help so far btw
s
so, I did confirm that the real-time segments do not show up when trying to reindex, I'm going to wait for it to complete a cycle, so it publishes some segments (the cycle is defined by the kafka job's 
ioConfig.taskDuration
property which defaults to 1H. After 1H I should see segments published and the reindex job should pick them up. I also see a bunch of segments in the segments view that do not have row counts initially, but if you wait some time, you see the real-time segments start showing num rows values >0.
Now. You mentioned that you want to apply some roll-up after ingestion. So I'm guessing you want both the raw data in one data source and one or more aggregations in another. If you want both to be created and updated in real-time, you can use a different approach: • create another kafka ingestion spec from the same topic and do a different rollup, ◦ keeping only dimensions you want to group by ◦ setting query granularity to the time aggregation you want. ◦ adding any metrics you want in the rollup • This means that you will be using more task slots in the middle manager(s) to achieve this on a continuous basis • If you are processing with multiple tasks in each job, you'll still want to consolidate segments using auto-compaction
s
I see, yeah that is definitely something I could consider. Also, thanks for the info about 
taskDuration
, just looked into it and that information is kinda buried in the docs. Thank you so much for all the help! Hopefully I didn't take up too much of your time.
s
No worries, this is how I learn.
Just to close this out, I did confirm that the reindex task sees the data once it has been published to Deep Storage.
s
Yep I can also confirm this works after explicitly setting a low 
taskDuration
. Thanks again!
Open in Slack
PreviousNext
Powered by