This message was deleted.
# general
s
This message was deleted.
m
Hey @John Masvongo here are a few things to consider: • Postman test: Try building the same request manually in Postman to test if the API endpoint is accessible and the query format is correct. Alternative approaches: • Druid Python client libraries: Consider using official Python client libraries like
pydruid
or
druidis
for interacting with Druid. These libraries can handle common error handling and serialization tasks, simplifying your workflow.
I also suspect that you need to do this:
Copy code
payload = {"key1": "value1", "key2": 2}
json_payload = json.dumps(payload)
response = <http://session.post|session.post>(url, data=json_payload)
especially given the HTTP 400 you're getting.
I almost always use Postman as my first step in debugging for any Python API project I do.
g
You might also be getting a useful error message along with the response Try printing the response body, not just
response.status
, when you get an error, to see what it is
From looking at your query one issue is your identifiers are single-quoted but they should be double-quoted, like this:
"stocks_intervals"
and
"__time"
instead of
'stocks_intervals'
and
'__time'
m
Ya, good suggestion @Gian Merlino And I also saw the single quotes and wondered about those...
j
I have been able to partially resolve the error by changing from single-quoted to double. The query string that now works is: query_string = "SELECT * FROM \"stocks_intervals\" WHERE \"symbol\" = 'AAPL' AND \"__time\" > TIMESTAMP '2024-01-22 125000'" I however still have a problem when I use a Python f-string to dynamically change the "ticker" and "interval" as in the f-string below: query_string = f"SELECT * FROM \"stocks_intervals\" WHERE \"symbol\" = '{ticker}' AND \"__time\" > TIMESTAMP '{interval}'" Though I get a 200 success status response back the query does not return any data when I know for a fact that the data do exist because using the first non-dynamic version returns results. What am I getting wrong with the f-string?
The new asynchronous Python function that partially works is as follows, with the problematic lines in bold: async def query_druid_async(ticker, interval): druid_host = 'localhost' druid_port = 8888 # Adjust the port as needed druid_scheme = 'http' # or 'https' if using SSL url = f"{druid_scheme}//{druid host}{druid_port}/druid/v2/sql" headers = { 'Accept': 'application/json', 'Content-Type': 'application/json' } # Query does not work if I use the f-string to dynamically change "ticker" and "interval" as below *query_string = f"SELECT * FROM \"stocks_intervals\" WHERE \"symbol\" = '{ticker}' AND \"__time\" > TIMESTAMP '{interval}'"* # Query does work if I have static query parameters as below *query_string = "SELECT * FROM \"stocks_intervals\" WHERE \"symbol\" = 'AAPL' AND \"__time\" > TIMESTAMP '2024-01-22 125000'"* query = json.dumps({ "query": query_string, "header": 'false', "typesHeader": 'false', "sqlTypesHeader": 'false' }) async with aiohttp.ClientSession() as session: print(f"Query String: {query}") async with session.post(url, headers=headers, data=query) as response: print(f"Response status: {response.status}") if response.content_type == 'application/json': data = await response.json() #print(data) return pd.DataFrame(data) else: response_text = await response.text() print(f"Unexpected content type: {response.content_type}") print(f"Response text: {response_text}") return pd.DataFrame()
m
Hey @John Masvongo I'm pretty sure that you don't want the single quotes around the fields that you want to fill in for the format string, so:
Copy code
query_string = f"SELECT * FROM \"stocks_intervals\" WHERE \"symbol\" = '{ticker}' AND \"__time\" > TIMESTAMP '{interval}'"
should be:
Copy code
query_string = f"SELECT * FROM \"stocks_intervals\" WHERE \"symbol\" = {ticker} AND \"__time\" > TIMESTAMP {interval}"
try printing
query_string
in your Python code or inspecting that variable in your debugger.
j
If I do that then the query string that ends up being posted is as follows: "query": "SELECT * FROM \"stocks_intervals\" WHERE \"symbol\" = AAPL AND \"__time\" > TIMESTAMP 2024-01-22 125000" It won't have quotes around the fields and that generates a "Response status: 400" error.
m
Sorry, John, didn't have my coffee yet. What error do you get if you have the single quotes?
I just whipped up this query to run in the router against the example datasource
wikipedia
Copy code
SELECT * FROM "wikipedia" WHERE "page" = 'Salo Toraut' AND "__time" > TIMESTAMP '2016-06-01'
and that seems to work, so not sure what error you might be seeing with your example. I usually have POSTMAN set up to run queries and I heavily rely on the router to help with debugging, especially complex issues. I try to get something working, then make changes to isolate an issue.
j
I had another error in the code where I was passing a "None" value for "interval" that's why the query was not returning any records. So the following string is the one that works:
Copy code
query_string = f"SELECT * FROM \"stocks_intervals\" WHERE \"symbol\" = '{ticker}' AND \"__time\" > TIMESTAMP '{interval}'"
✅ 1
m
That is fantastic, @John Masvongo!! I believe that if you make a few mistakes and then resolve an issue, you learn a lot more. https://giphy.com/clips/viralhog-viral-hog-otter-claps-and-taps-for-clams-vfI3uwl4FMuDaLbZah
j
Is it possible to create 5 minutes open, high, low, close bars from stock market tick data streaming from a kafka using Druid rollup in a kafka indexing task?