https://www.getdaft.io logo
Join Slack
Powered by
no metrics when using prompt with UdfOptions: ```w...
# daft-dev
g
no metrics when using prompt with UdfOptions:
Copy code
with_prompts.with_column("coding_classification", prompt(
        messages=col("prompts"),
        provider=papyrus_provider,        
        model="o3",
        return_format=CodingClassification,
        udf_options=UDFOptions(concurrency=3, on_error="raise")
    ))
the run seems to just hang. with 0 rows out. no metrics or anything. cant seem to get any visibility or logs around whats going on. any direction on other ways of managing concurrency for the prompt(..) AI function?
👀 3
s
Hi @Giridhar Pathak, do you happen to have a small reproducer that I can try? It looks like you're using a custom provider (
papyrus_provider
). Have you seen the same problem with another provider, like one of the built-in ones?
g
if i set concurrency at all, i get stuck here. with out the concurrency setting, i at least can see something happen.
yes the papyrus provider is basically the Openai provider with diff urls
here is the provider:
Copy code
papyrus_provider = OpenAIProvider(
        name="papyrus-generator",
        base_url="<https://papyrusproxy-eastus2-hnbkf6cdctasdsff.b02.azurefd.net>",
        timeout=60,
        api_key=token,
        max_retries=3,
        default_headers={
            "papyrus-priority": "1",
            "papyrus-model-name": "o3",
            "papyrus-timeout-ms": "300000",
            "x-policy-id": "nil",
        }
    )
its just a hosted service i have setup in azure.
👍 1
also when i give it a concurrency value, it spins up python workers. that dont die after i kill the main process
have to kill them manually
so this should be reproduceable by just hitting open ai too
cause it works when i remove the 
udf_options
arg
but then i have no way to manage concurrency
s
Just curious, do you mind trying a smaller dataset first, such as including a 
limit(100)
. I just want to make sure that you see rows coming out of the prompt UDF in the metrics / progress bar
g
without the 
udf_options
i do see rows come out and it all works fine
s
Good, but can we try it with 
udf_options
and 
limit(100)
as well? I want to make sure the UDF worker process in not hanging at the end
g
ok trying
s
That would help narrow down the bug
g
so it doesnt give me any mertics or say if any rows came thru, but it did finish
the live metrics at the end disappeared so dont have a screen shot for you
but i got 0 indication that any prorgess was being made
s
just to make sure, it did return results right, just that the metrics didnt update step-by-step
g
no clue if im getting 429s or errors that are being retied etither.
yep did return results
just blank in terms of metrics or progress
trying w 1000 now,
s
and did you have to kill the python workers manually again?
g
its defi processed rows .. but no visible progress or errors.
and did you have to kill the python workers manually again?
nope, when it completes properly it seems to cleanup fine, just if i kill the main process it does not clean up
s
ok good, do you mind trying one more thing? can you modify your script from 
daft.read_json(...)
to 
daft.read_json(...).into_batches(100)
?
I believe whats going on is when concurrency is specified and we run 
prompt
on a separate process, we submit larger batches of data to the process and wait for all of those to complete. without concurrency, we use much smaller batches
my guess is the batches of inputs are too big that we're making too many requests at once, and everything is slowing down. specifying 
into_batches(100)
should help override this so we only process 100 prompts at a time
c
FYI, we're releasing 0.7.0 soon, which comes with opt-in dynamic batching (
daft.set_execution_config(enable_dynamic_batching=True)
) so you don't need to manually specify 
into_batches(100)
we'll handle adjusting the batch sizes and you'll continue seeing incremental updates in the terminal!
🙌 1
g
ill try the into_batches
its better with into_batches!
so, into_batches breaks up the data into 100 record batches and each batch is sent to 1 of the udf instances for prompt right? so if i have a concurrency of 3, ill be running 300 concurrent requests... right?
s
Yes exactly!
g
i was going over the OpenAIPrompter code, and i noticed that it awaits on every http call... it does not run a bunch of them in parallel (controlled by a semaphore). this would imply the parallelism is limited to number of processes spun up. i.e 3 in the case described above. @Srinivas Lade is this intentional? Given that its asyncio, any reason that this can not be done using limited number of processes but many asyncio tasks on a few processes managed by a semaphore per process?
s
to clarify, even though the code in OpenAIPrompter itself only has 1 await, in general we call async UDFs in parallel using something like https://docs.python.org/3/library/asyncio-task.html#asyncio.gather to run concurrently on multiple inputs
❤️ 1
Open in Slack
PreviousNext
Powered by