WhyLabs R2AI Community

Hi everyone! I was playing around with the Starter tier and found out that I’m only able to create models with daily frequency as the other options are disabled, is that right? Does that depend on the tier or am I missing something?

You're right <@U04V2KV1AAC>, thats a feature in different tier. You can check out the differences between free and paid tiers on the pricing page: <https://whylabs.ai/pricing>

<@U04V2KV1AAC> if you’d like to test anything that’s not in the free tier, we’d be happy to help you - please ping me directly at <mailto:maria@whylabs.ai|maria@whylabs.ai>. Cheers! 

Hi everyone! can we save the graphs generated in profile report as image?

Hi Akshay! While we don’t have a specific image save, you can download the html or screenshot. If you are referencing the whylogs notebook profile visualization reports these have a save method you can find at the end of this example: <https://github.com/whylabs/whylogs/blob/mainline/python/examples/basic/Notebook_Profile_Visualizer.ipynb|https://github.com/whylabs/whylogs/blob/mainline/python/examples/basic/Notebook_Profile_Visualizer.ipynb>

Or are you asking about the graphs on WhyLabs?

Thank-you for replying, I was looking for an image alternative to html download option as you have suggested in first part.

Is it possible to save constraint reports to whylabs? Or do I need to save those things myself somewhere else if I want them?

Hello <@U054RG0UCPN>! Are you looking to just save the visualization? Or are you looking to put the constraint logic in the platform?

I'd love to hear more about your use case!

I actually maybe have a whole slew of questions regarding best practices around the platform that would also involve this use case. Are you the best person to ask?

Hey <@U054RG0UCPN> :wave: I can help, it's usually best to get a call scheduled where we can go through them with you. Would that work?

That would be great. Are you available tomorrow?

Sure does 12pm PST work for you? If so please DM me your email and I'll send over an invite :+1:

Hello, in whylabs platform can we see the prompt for which the scores are shown

Yes, but this feature is currently beta-only for a small number of LLM customers

Hello, I am evaluating the Whylabs platform. But I struggle to see how the logs uploaded to the platform related to my prompts and responses so I can take action. The responses and prompts are not uploaded to the whylabs platform, hence making the work of linking metrics with the original data difficult
Anything I am missing here ?

Hi Francois! By default whylogs profiles in a privacy preserving way collecting summary statistics on the inputs/outputs without storing the raw data. What you see in WhyLabs are things like type counts, numeric distributions and cardinality estimation on extracted features.

Based on your reference to prompts and responses it sounds like you might be looking at LLM scenarios? Are you logging your data with LangKit?

These kinds of statistic are useful in detecting issues like drift (which might not be caused by a single data point): <https://whylabs.ai/blog/posts/data-drift-monitoring-and-its-importance-in-mlops>

We also have condition validators that can run in-line with profiling to look for problems on the individual records, which might be useful in scenarios like content moderation or other  LLM applications, and there is some content here with linked examples: <https://whylabs.ai/blog/posts/safeguard-monitor-large-language-model-llm-applications>

hey <@U029N32P150>, thanks for the answers. Actually I'm trying very simply to run some of your demonstration notebooks. Indeed, I am looking at condition validators. I wish there was a way while I am navigating a profile in the whylabs platform to identify the prompts which are creating a surge in some of the metrics (like toxicity) after the fact.

So far, I feel that I need to store my prompts and responses on my own and somehow link the whylogs metrics to the messages myself ?

Right, the profiles are aggregate statistics by design and might not contain the individual prompt/response records or their specific scores in WhyLabs. And then with condition validators you can conditionally store a prompt/response for a record that meets some criteria (e.g. toxicity score is high for example).

But there are other ways around the aggregation if thats not what you need

Well, if I see after the fact an interesting piece of data on a profile, I would like a way to investigate it and go back to the prompts and responses that triggered this data. That is a way for me to improve my model or my use case by looking at past data

(but maybe I'm not having the right mindset here :slightly_smiling_face: )

but I guess I understand, it is about me to store the interesting messages through condition validators when those numbers are raised rather than just looking at the whylabs dashboard and drilling down to the message

If you can store all your prompts and responses that can be a useful debug flow.

We also recently added support for DebugEvents which might help you store some useful excerpt context that helps you debug. They can be associated with a profile, queried by users, contain custom JSON supplied by your integration and are never aggregated.

The debug flow is something we are actively working on and trying to improve, so we are also open to ideas and trying to better understand what features would be most useful to people in the community

Example for debug events: <https://github.com/whylabs/whylogs/blob/a498105f870b7e3b1c8f17cff6771c35ec25d535/python/examples/experimental/Logging_with_Debug_Events.ipynb>

well drilling down from the whylab dashboard to the faulty message would be nice. Identifying a surge on some metrics and then identifying which messages/prompts caused it, all from the platform, would be a good time saver rather than having to have to manually develop it :slightly_smiling_face:

Hi Whylabs team. I'm currently tasked to evaluate if Whylabs will be great for our company's model monitoring needs. For example, I have a spam filtering model. For every input, the model returns a spam likelihood score. Is Whylabs capable of sending alerts if the score of a single sample exceeded a certain threshold? I am aware it is possible for a set of data but how about if it's just one sample?

Hi <@U0677G23BSM>! We have various features around debugging or constraint check that would work in this use case. Would it be useful to hop on a quick call to chat about the use case? Shoot me an email at <mailto:andy@whylabs.ai|andy@whylabs.ai> if you want to schedule a quick chat!

I am trying to push LLM monitoring metrics to Platform. I am facing this error even though langkit has been installed. Could you please help ? No module named : langkit.callback.handler. Lankit version is 0.0.2 and the error seems to be due to the code - whylabs = WhyLabsCallbackHandler.from_params()

Hi <@U063VU6KRCH>,

That error message looks like its from our LangChain integration?

LangKit 0.0.2 is too old for that integration, our LangChain integration was released after LangKit version 0.0.6. LangChain's WhyLabs callback handler requires LangKit version greater than 0.0.6. <https://github.com/langchain-ai/langchain/blob/5dca107621ce18dbbbda79d88016b1783d159f20/libs/cli/poetry.lock#L680>

I'd suggest you try the latest LangKit release and see if that resolves your error:
```pip install langkit==0.0.30```

Thanks Jamie. I was having challenges when I tried to change the Langchain version. I will continue to try this. thanks for your response.

To clarify, I don't think you need to change the langchain version, but you need to upgrade the langkit package. The langkit version referenced in your error: 0.0.2 is quite old and predates the LangChain integration. Out of curiosity how did you install LangKit in your setup?

I am trying to push llm metrics to the platform. While I am successfully able to push metrics for llama2 model using langchain ctransformer but not able to see metrics getting pushed for openAI or chat google API. Please help.

hi <@U063VU6KRCH> - Can you share details about how you are pushing metrics?  are you using the same langchain code that you used for the llama2 model and just swapping llm's?  Are you getting any errors?  Some code examples?