Hi folks, I'd like to ask for a bit of help here: ...
# troubleshootingd
Diogo Baeder
10/26/2021, 2:00 PMHi folks, I'd like to ask for a bit of help here: I'm still struggling to be able to run integration tests between my app and Pinot. What I need is to be able to start each test with a clean slate, while also making sure I have the tables I need created. I tried deleting the tables and recreating them, tried deleting all segments then creating the tables afterwards (hoping a new segment would automatically be created), and none of these methods are working. How can I accomplish what I need?
Diogo Baeder
10/26/2021, 2:08 PMAlso another problem I'm facing is, if I try to delete the table and then recreate it, and then add data to it, it seems like the table was never deleted, because in the "Incubator" (the Pinot admin UI) I can see all the previously created rows.
k
Kishore G
10/26/2021, 2:45 PM
Delete happens asynchronously.. you will have to wait in your code for query to return zero records
d
Diogo Baeder
10/26/2021, 2:53 PMSo, how would you put all of this together, in steps? Would this make sense?:
1. Delete all tables
2. Wait a few secs
3. Recreate all tables
4. Wait a few secs
5. Start running the test (publishing rows and querying them afterwards)
k
Kishore G
10/26/2021, 3:29 PM
I would not make it wait based on time
Kishore G
10/26/2021, 3:29 PM
Instead wait for count * queries to return 0
Kishore G
10/26/2021, 3:30 PM
See baseclusterintegrationtest in Pinot code
Kishore G
10/26/2021, 3:30 PM
It has everything you need
d
Diogo Baeder
10/26/2021, 4:11 PMNice, I'll look for that. Thanks man!
👍 1
m
Mayank
10/26/2021, 4:41 PM
Also, if your tests are table specific, then you can create different tables for different tests in the same cluster. That way one test doesn't have to wait for the previous test to complete and delete the table.
d
Diogo Baeder
10/26/2021, 6:20 PMGot it. It's going to be 2 tables total, actually, we're going to use it for 2 parts of our internal analytics, so I'll work on having them cleaned up before each test. Thanks!
Diogo Baeder
10/26/2021, 6:26 PMAlright, so, with this: https://github.com/apache/pinot/blob/7e9ca6a5a4afe0d4e283ac1307c45430e474cbf2/pino[…]apache/pinot/integration/tests/AdminConsoleIntegrationTest.java
If I understand correctly it starts all the applications and then tears them down around each test, also deleting the temp dir used to store the data afterwards. However this is not very efficient for me, I'd like to avoid having to start and stop the whole suite of apps on each test, if possible.
k
Kishore G
10/26/2021, 6:40 PM
you can use beforeSuite for setup then
Kishore G
10/26/2021, 6:40 PM
in each setup for testcase you can delete the segments if table exists and wait for the count to go to zero
d
Diogo Baeder
10/26/2021, 6:42 PMHmmm... not sure I understand that last part ("delete the table if it exists and wait for it go to zero"), but how about I:
1. Delete the table
2. Keep listing tables until my table doesn't exist
3. Create the table
4. Keep counting until I get result zero
k
Kishore G
10/26/2021, 6:46 PM
edited - in each setup for testcase you can delete the segments if table exists and wait for the count to go to zero (edited)
Kishore G
10/26/2021, 6:47 PM
I think your flow may not work because the segments might still be on the servers
Kishore G
10/26/2021, 6:47 PM
delete segments is asynchronous
d
Diogo Baeder
10/26/2021, 6:48 PMHmmm... got it... but after I delete all the segments, how do new ones get created? Last time I deleted all the segments in the table I got no new ones created...