WhyLabs R2AI Community

Anyone use something similar to detect Hallucinations in generative models?
<https://huggingface.co/spaces/ml6team/post-processing-summarization>

I've been looking at getting started with some embedded ML projects using MCUs (Micro controller units), does anyone have a chip/dev board they like?

Anybody reading a good ML book or going through any good courses?
I just started the NLP &amp; huggingface transformer book. I like how hands on it is

^ if there is any interested in some sort of book club let me know!

Designing Machine Learning Systems by Chip Huyen is a good read

Do you have a current favorite data annotation tool for computer vision (bounding boxes, segmentation, etc)?

This tool has some cool NLP/string  features: <https://github.com/stanfordnlp/string2string>
&gt;  The string2string library is an open-source tool that offers a comprehensive suite of efficient algorithms for a broad range of string-to-string problems. It includes both traditional algorithmic solutions and recent advanced neural approaches to address various problems in pairwise string alignment, distance measurement, lexical and semantic search, and similarity analysis. Additionally, the library provides several helpful visualization tools and metrics to facilitate the interpretation and analysis of these methods.

Hey all - I posted in <#C02UY8ZECS2|events> but Capital One and I are organizing a Kubeflow (MLOps) meeting and we’d love to see you there! It’s in person and in NYC - check out the channel!

added an integration to this channel: <https://rsqrd-ai.slack.com/services/B056VA894LV|linen.dev>

hey! :wave: is anyone here using whylogs with mlflow for monitoring predictions? :slightly_smiling_face:

I’m a month late to answering this lol, but yes our company (Glassdoor) uses it!

no worries <@U04NQ8Q2CRZ> - also late lol and how are you doing to create your predictions profiles? are your monitored models typically  batch or online?
asking just because we have some things in the integrations roadmap for mlflow, but it would be great to understand what needs to be improved at the moment :slightly_smiling_face:

<@U039DGH8VGW> We have both realtime and batch use cases, although batch use cases definitely outnumber the realtime ones. For realtime, we worked with WhyLabs to make monitoring more efficient to allow us to meet our tight latency constraint (a blog post coming out on this very soon!). We haven’t too much trouble trying to integrate monitoring for our batch use cases.

Hey all, wasn't totally sure where to post this - I'm looking at ML observability products to potentially use and wondering if anyone can share thoughts/experience on why one might be better than another. I'm sure <@U03K99ZMJUE> may have some thoughts :slightly_smiling_face:

Currently looking at:
• <https://whylabs.ai/|WhyLabs>
• <https://arize.com/|Arize>
• <https://www.aporia.com/|Aporia>
• <https://www.fiddler.ai/|Fiddler>
• <https://www.evidentlyai.com/|Evidently>
Thanks for your help!

Well, I work for WhyLabs so I'm very biased :joy:
Would love to know more about your use case, like what type of models &amp; data you have!

I'd be happy to hop on a call and give a demo / answer any questions.

Currently prototyping a product that uses multiple layers LLM inference. When the product reaches the next stage of scale, we'll be fine tuning on text, tabular, and image data.

Do you have any kind of doc or page I can read comparing the products? I'll also DM you to set a time to chat

If you haven't looked into how we work yet, you generate privacy preserving telemetry in your environment with our <https://github.com/whylabs/whylogs|open source library>, and log those profiles in in WhyLabs. For LLM use cases we extract language metrics from prompts and responses with <https://github.com/whylabs/langkit|LangKit>.

I sent you a demo link! You can see a quick example with a hugging face model in this post, but we just launched a really cool new dashboard we can show you as well.  <https://medium.com/whylabs/hugging-face-and-langkit-your-solution-for-llm-observability-d00ab01ca036>

Super curious about this topic, too! Trying to research the field as I consult for an AI edtech (<https://www.atypicalai.com/>) and help select an approach for evals at both prompt eng time and in prod. I mentioned a few in <https://rsqrd-ai.slack.com/archives/C01RQ9SSFQR/p1695082694436369|my intro message> the other day.

Some commercial offerings in the space (some lean o11y, other lean toward supporting prompt eng):
• <https://promptlayer.com/>
• <https://www.magiklabs.app/>
• <https://www.helicone.ai/>
• <https://costly.ai/> (I know the founder, early stage but some interesting proprietary quality metric stuff. Initial value prop is mostly managing cost during the go-to-prod phase)
Some FOSS offerings - these tend more towards evals, less on retention, backtesting, dashboards, o11y lifecycle, etc. that the hosted offerings provide.
• Ragas: <https://github.com/explodinggradients/ragas> 
• Trulens: <https://www.trulens.org/> (from commercial offering <https://truera.com/>)
• LLM-guard: <https://github.com/laiyer-ai/llm-guard>  (from <https://laiyer.ai/>, presumably commercial offering forthcoming?)
• Langfuse <https://langfuse.com/>


So many awesome things coming out every week(or even day), It's an exciting space to be in right now! :)

<@U06GBFB2L7Q>
This channel - NIshiwaga Town Yuda

in  Amity the project I gave you. I deleted our prev for security

Can you tell me how I can contribute to this project?