This message was deleted.
# troubleshootings
Slackbot
09/08/2022, 10:11 AMThis message was deleted.
r
Renato Santos
09/08/2022, 10:12 AMI'm trying to 'fix' the datasource names bc of bad trino: https://github.com/trinodb/trino/issues/6850
h
Heiner
09/08/2022, 10:21 AMWhy don't you just re-ingest/re-index to a new datasource?
r
Renato Santos
09/08/2022, 10:25 AMI only have ~ last 5 days or so in kafka, 30 days used/loaded on the historicals, and a few months of data not loaded (unused), although it could be fine to keep the 'old' segment as historical only (but during the migration I would have to rewrite all queries to query both datasources, paginations would be hard)
Renato Santos
09/08/2022, 10:26 AMI think I gonna export to csv, import to s3 and load back to trino. I was doing the same thing, but download to local db, importing to postgres and then running some analytics with lots of joins where druid just can't beat pg with indexes
h
Heiner
09/08/2022, 10:27 AMI mean reindex from druid.
Copy code
{
"type": "index_parallel",
"spec": {
"ioConfig": {
"type": "index_parallel",
"inputSource": {
"type": "druid",
"dataSource": "OldDataSourceName",
"interval": "1970-01-01T00:00:00/9999-01-01T00:00:00"
}
},
...
"dataSchema": {
...
"dataSource": "newdatasourcename",
...
}
}
}Heiner
09/08/2022, 10:28 AMThis will leave the old datasource as-is and create a new datasource with a new name
Heiner
09/08/2022, 10:28 AMwhile at it, you can fine tune your segment size et cetera
Heiner
09/08/2022, 10:29 AMā¢ turn off kafka ingestion to old datasource
ā¢ run Druid reindex task
ā¢ check that it worked (at least compare numRows between old and new datasource)
ā¢ turn on kafka ingestion to new datasource
r
Renato Santos
09/08/2022, 10:31 AMoh! that's make sense, I can use this to reindex for the current data!
h
Heiner
09/08/2022, 10:31 AM(Disclaimer, I'm not a Druid developer, just a user, but I did just that, and it worked great (I had lots of small segments before))
r
Renato Santos
09/08/2022, 10:32 AMI forgot that this option exists!
h
Heiner
09/08/2022, 10:32 AMjust use the wizard under "Load Data" to create the spec
r
Renato Santos
09/08/2022, 10:33 AMyeah, I will start this job
I will not change the current ingestion right now, but I can run the analysis on a copy
Renato Santos
09/08/2022, 10:33 AMthanks!
h
Heiner
09/08/2022, 10:34 AMnp
k
kfaraz
09/08/2022, 10:45 AMYes, re-indexing is the only real way of renaming your datasource.
r
Renato Santos
09/08/2022, 4:21 PMeasy task but I'm unlucky today! First died with crazy error (Expected [700,958] bytes, only saw [615,989], potential corruption?)
Then I reduced merge tasks, try again, then out of space (can I set the path of this task only? I've setup only 11GB on the /var/druid for the MM {just to clean restarts} ,but I have 600gb local disks on the host VM)
the third I cancelled to remove some columns, tried again with just 2 days loaded, now I got out of memory ! haha
Renato Santos
09/08/2022, 5:02 PMyay!
š 1
g
Gian Merlino
09/09/2022, 12:04 AMyou should have better luck in the next druid release! we've implemented substantially more self tuning for the new SQL-based ingest feature
i have been enjoying using it in prerelease testing š
Gian Merlino
09/09/2022, 12:04 AMglad you got it done though
š 1