(Push?) notifications for slow responses
under review
D
Dmitry Matora
Right now local CPU running large models are not usable with Bolt, since it takes minutes for them to respond. If you look at the window, it's too long and irritating. If you switch app, you won't notice when it finishes generating response.
Having notifications for finishing generating response would make it usable just like any messaging app, where waiting for person response could take hours or days and everyone is fine with that.
D
Dmitry Matora
Daniel Nguyen now with chain of thoughts models like o1, this is a significant feature even for those those are not using local models. Should be quick to implement, have you had a chance to look into that?
Daniel Nguyen
under review
Daniel Nguyen
Interesting. I never thought of this case. I will think more about it
D
Dmitry Matora
Daniel Nguyen you can maybe use this LLM UI for inspiration https://www.reddit.com/r/LocalLLaMA/comments/1f3ozoz/yet_another_local_llm_ui_but_i_promise_its/
D
Dmitry Matora
But after testing it I have to say that's not the best way to deal with this. First of all it sends entire message to cloud unconditionally, which is not good for privacy. Plus it doesn't work reliably when running local (certificates issues, etc) so maybe needs significant architecture shift. Maybe url field for a webhook that you can play with and find something that works in your particular case, like local running server or telegram bot or something else.
D
Dmitry Matora
Ok I've made some adjustments to the app to make it send notifications offline, and now notifications work rock solid. It's desktop only solution, but that's where BoltAI shines
Daniel Nguyen
Dmitry Matora Yeah it doesn't require push notification in this case. Just local notification would work. I might add this soon
D
Dmitry Matora
Daniel Nguyen I guess this only requires notifications when app is not in focus. Or maybe better when response takes more than (let's say) 10 seconds to generate (if it takes 5 minutes you still want a sound in case you went to make a coffee). Maybe an option for how many seconds delay should trigger it (not if it will delay feature deployment)