Ability to set the context length per chat/model
under review
Sam
It would be good to be able to set the context length per chat/model/prompt/assistant.
At present the only way seems to be to go into the main settings and set it on the whole API.
Perhaps a slider in the chat parameters ranging from something like 0-128K with a self-updating numeric field (e.g. double click the current showing number to enter a precise number).
Here's a quick and dirty mock
Daniel Nguyen
under review
Thanks. I need to find a better UX for showing these advanced configuration as the current UI is not really easy to do that.
Sam
Daniel Nguyen Hey Daniel is there any chance this is on the cards some time soon?
I find it really painful having to go into the settings and add a duplicate of each server/service I have for different context sizes - it also ends up being quite messy.
This screen shot is on just one of my laptops, and really I need to add another 2 or 3 context sizes to my Ollama and TabbyAPI server
Sam
Honestly I think it'd be great if this and other inference params could be in a panel on the right, e.g.
Daniel Nguyen
Sam Hmmm I actually stopped using this context length parameter for most AI services. Do you use custom endpoint mostly?
Sam
Daniel Nguyen Oh really! I use Ollama mostly, sometimes OpenAI compatible with TabbyAPI or text generation webui (for exl2 models)
Daniel Nguyen
Sam Yeah. I used to use this context_length to warn users if their context is too large. But now most LLMs support a large context window, it's not really useful.
Going to remove it soon :D
Daniel Nguyen
Sam Good idea. I will think more about this. Will need to relocate the search field though
Sam
Daniel Nguyen cool thanks. Basically the context size you set will not just limit how large your conversation can be but also the resources required when the LLM is started on the server.
If it is set too high you might not have enough vram and it might perform very slowly.
If it’s set too low you’ll get gibberish once you hit the limit.
If you set it dynamically each time your server will have to completely unload and reload the model each time it makes a request which is very time consuming.
Having the option to set it based on your needs is what I find most practical.
Daniel Nguyen
Sam Yes but it should be on the back-end side (Ollam, llama.cpp etc.). BoltAI doesn't have the ability to change this params. At least not as I know of.
Sam
Daniel Nguyen Ollama sets its context size from the API request, usually when making a request you provide num_ctx with the size you want and that along with the model name (and soon context cache quantisation) defines which model and how it is loaded when requested.
If you don’t provide it - it will default to a very small setting of just 2048 which is Ollamas fall back default.
Sam
Happy to provide some screenshots / example code or even do a screen share some time if that’s helpful
Daniel Nguyen
Sam: Ah right. I mean it's not a standard in the OpenAI-compatible API format, which BoltAI is using.
Sam
Daniel Nguyen Oh is it? I thought it was using the native API!
Sam
You're missing out on a lot of the useful options with it:
curl http://localhost:11434/api/generate -d '{
"model": "llama3",
"prompt": "Why is the sky blue?",
"stream": false,
"options": {
"num_keep": 5,
"seed": 42,
"num_predict": 100,
"top_k": 20,
"top_p": 0.9,
"min_p": 0.0,
"tfs_z": 0.5,
"typical_p": 0.7,
"repeat_last_n": 33,
"temperature": 0.8,
"repeat_penalty": 1.2,
"presence_penalty": 1.5,
"frequency_penalty": 1.0,
"mirostat": 1,
"mirostat_tau": 0.8,
"mirostat_eta": 0.6,
"penalize_newline": true,
"stop": ["\n", "user:"],
"numa": false,
"num_ctx": 1024,
"num_batch": 2,
"num_gpu": 1,
"main_gpu": 0,
"low_vram": false,
"f16_kv": true,
"vocab_only": false,
"use_mmap": true,
"use_mlock": false,
"num_thread": 8
}
}'
Sam
I'd really like to be able to set this and also other model level settings such as num_batch, top_p, repeat_last_n, num_keep etc....
R
Roger Kint
One workaround I did for this was to add new API for different models (local ollama in my case) and rename them and set context length on each one, then switch using API and not models during chats
Sam
Roger Kint good hack! thanks for the temporary workaround :)