Add user-configurable settings into the UX/frontend has been open for a while. I think it's a nice feature to have that allows the user to have more finer control of the LLM, but maybe there could be a better approach to the UI? Two questions: 1. Do we want to allow the user to modify these details? What are people's thoughts on this? Any additional things to consider like

config.toml

(cc @calm-artist-7206) conflicts or extending saas settings (cc @quick-window-91705 @straight-sundown-84319) that we might want to address before pulling this in? 2. Is there a better design we can come up with, e.g., that includes sliders and such to make it more compact? cc @breezy-kilobyte-39430

I'll review and put something together.

if these are changeable mid conversation/workstream, making note of that change will be important... if they're not, then it would, imo, make sense to have a 'post repo selection pre-conversation beginning phase wherein these changes could be performed

If I understand correctly, these are configurations we want at the conversation level or at the user/org level (applying to all conversations)?

They are stored for the user - changes are typically applied to new conversations but not existing ones.

There are 2 categories in the PR: LLM and Agent. I think they’re not the same: • LLM should be possible to change in the same conversation; and the new LLM will handle all history • Agent cannot be applied to old histories, IOW shouldn’t be changed within a conversation

if these are changeable mid conversation/workstream, making note of that change will be important...

Yes, currently we show a toast when the user changes settings that let's them know they may have to restart it in order for the changes to apply

If I understand correctly, these are configurations we want at the conversation level or at the user/org level (applying to all conversations)?

Good question. Let's assume for now it's just for a single user and that settings apply for every conversation until they change it, just as settings behave now

I think with nested conversations the application after start is less viable too

either way, I'd think that having the ability to review/alter post-repo-selection and before-engaging-conversation mechanism would be helpful to have available... especially if it's post 'auto-selection-of-microagents' in the context amalgamation process.... as that would then provide an intuitive spot to adjust that as well.... (not really relevant to THIS part of the conversation; admittedly, but IS relevant to the user-starts-an-interaction-workflow... which is why I mention it here)

ie: When you start a conversation, the settings get copied to the nested server - and restarting doesn't update them (At least at the moment)

but this is just my opinion which ... when combined with ~$11 can get you a coffee at most places 😉 (IOW: My opinion's not anything special 🙂 )

When you start a conversation, the settings get copied to the nested server

Ohh so we use to restart the runtime but not anymore with the nested runtimes change? Not even after long periods of inactivity?

At the moment this is the case

Ahh, interesting, thank you Tim. That’s a bit confusing, though, because we’re working to make them updatable in real time, eventually

We should probably add a settings update when resuming a conversation

it would be nice if , one would be able to fork a new conversation from a point-in-time of a previous conversation... or pick up from the end of $old conversation.... this would be especially useful, for example, for doing benchmarking of dif things, as the context branches would be identical

We have been looking at another paradigm shift in this regard - the ability to run multiple conversations in the same runtime - though we need to hash out how the API will reflect this

that's neat..... I like the concept.... sorta tying the conversation to a stream of work, rather than a step.

This would have ramifications for forking - like if a conversation resulted in some global install within the runtime, should a fork include this or not... 🤔

well, I mean.... you're talking about two very dif flavors of work that are both equally useful in dif contexts

For sure - and they are not mutually exclusive

DO NOT MIX THE RAPTORS WITH ANY OF THE OTHER BOXES

I dunno - depending on the number of raptors and tribbles it may balance out - but be messy

lulz 🙂

So, these are the new options that need to be added to the LLM page: •

temperature

(0.0-2.0) - Controls response creativity and randomness •

top_p

(0.0-1.0) - Nucleus sampling parameter for response diversity •

max_output_tokens

- Maximum tokens in LLM responses •

max_input_tokens

- Maximum tokens in input prompts •

max_message_chars

- Character limit for individual messages •

input_cost_per_token

- Cost tracking for input tokens •

output_cost_per_token

- Cost tracking for output tokens But I also see these: Created a new dedicated Agent Settings page with 10 configurable capabilities: • Tool Access: Web browsing, Jupyter notebooks, command execution • Editor Options: LLM-based editor vs. standard string replacement editor • Agent Behavior: Thinking steps, task completion, prompt extensions, history management • Microagent Control: Ability to disable specific microagents So my confusion is whether we are attempting to add all of these configurations to the LLM page or if there is a new Agent Settings page?

so hm... those settings relate to the llm layer which is beneath the agent layer which is beneath... or perhaps adjacent the microagent layer.... how/when would you expect a user to switch the LLM settings? When should OH support doing it, even if its atypical?

@breezy-kilobyte-39430 You’re right, these are probably best in an Agent tab of their own. Correct me if wrong, currently the LLM tab contains the selection of the Agent? I think that’s for… “historical reasons”. 😅 I think LLM configurability is important, and I know people have wanted more knobs there (as Wolf says) to tweak it. For Agent, it’s a bit different and maybe we should take a moment to discuss this - I’ll follow up with notes on those options

@calm-artist-7206 correct, here's snapshot:

On a side note, cc: @freezing-spoon-14104 the PR author also submitted a simpler PR with the most necessary LLM settings:

the microagent UI is only available if you know it exists, and type the path in directly, afaik there's no exposure to it organically (but wtf do I know? I'm just a monkey with 10 thumbs) ideally, the tools would be visible, and interactable, imo, in the chat modal somehow, or at least a link to them, such that one could turn them off, or on... (

think

is a good example of why (imo) it would be beneficial to have them easily accessible.... it's a thing one may wanna toggle on/off regularly within the context of a chat) I'd think of it like: there's 4 tiers of tools ( OHTools, ORGTools, USRTools, REPOTools ) I'd prolly use the tools pane, and have an 'agent management' tab, where one could adjust the active tools in the conversation, as well as see the stats from usage of each of them over time... as adding them to the chat ui might make things too cluttered... but ¯\_(ツ)_/¯

What about a simple 'collapsible' textarea allowing "key=value" entries, with a placeholder text containing examples for the most commonly used settings, and a link to a guide? So that you don't crowd the UI with many small scary settings users might have no idea how to set (the textarea will be empty by default = no confusion).