Daniel Nguyen Hey Daniel is there any chance this is on the cards some time soon?

I find it really painful having to go into the settings and add a duplicate of each server/service I have for different context sizes - it also ends up being quite messy.

This screen shot is on just one of my laptops, and really I need to add another 2 or 3 context sizes to my Ollama and TabbyAPI server

Honestly I think it'd be great if this and other inference params could be in a panel on the right, e.g.

Sam Hmmm I actually stopped using this context length parameter for most AI services. Do you use custom endpoint mostly?

Daniel Nguyen Oh really! I use Ollama mostly, sometimes OpenAI compatible with TabbyAPI or text generation webui (for exl2 models)

Sam Yeah. I used to use this context_length to warn users if their context is too large. But now most LLMs support a large context window, it's not really useful.

Going to remove it soon :D

Sam Good idea. I will think more about this. Will need to relocate the search field though

Daniel Nguyen cool thanks. Basically the context size you set will not just limit how large your conversation can be but also the resources required when the LLM is started on the server.

If it is set too high you might not have enough vram and it might perform very slowly.

If it’s set too low you’ll get gibberish once you hit the limit.

If you set it dynamically each time your server will have to completely unload and reload the model each time it makes a request which is very time consuming.

Having the option to set it based on your needs is what I find most practical.

Sam Yes but it should be on the back-end side (Ollam, llama.cpp etc.). BoltAI doesn't have the ability to change this params. At least not as I know of.

Daniel Nguyen Ollama sets its context size from the API request, usually when making a request you provide num_ctx with the size you want and that along with the model name (and soon context cache quantisation) defines which model and how it is loaded when requested.

If you don’t provide it - it will default to a very small setting of just 2048 which is Ollamas fall back default.

Happy to provide some screenshots / example code or even do a screen share some time if that’s helpful

Sam: Ah right. I mean it's not a standard in the OpenAI-compatible API format, which BoltAI is using.

Daniel Nguyen Oh is it? I thought it was using the native API!

You're missing out on a lot of the useful options with it:

curl http://localhost:11434/api/generate -d '{

"model": "llama3",

"prompt": "Why is the sky blue?",

"stream": false,

"options": {

"num_keep": 5,

"seed": 42,

"num_predict": 100,

"top_k": 20,

"top_p": 0.9,

"min_p": 0.0,

"tfs_z": 0.5,

"typical_p": 0.7,

"repeat_last_n": 33,

"temperature": 0.8,

"repeat_penalty": 1.2,

"presence_penalty": 1.5,

"frequency_penalty": 1.0,

"mirostat": 1,

"mirostat_tau": 0.8,

"mirostat_eta": 0.6,

"penalize_newline": true,

"stop": ["\n", "user:"],

"numa": false,

"num_ctx": 1024,

"num_batch": 2,

"num_gpu": 1,

"main_gpu": 0,

"low_vram": false,

"f16_kv": true,

"vocab_only": false,

"use_mmap": true,

"use_mlock": false,

"num_thread": 8

}

}'