Project Direction - Features, improvements, etc

[neodon]

I'm considering contributing to the project. I was wondering if you could provide some insights on where you see it going. Do you have particular features or improvements in mind? I have some ideas, but they're fairly specific to my workflows and quirks. I imagine there are a lot of use cases I'm unaware of.

Some things I've been looking for in particular:

• Chat interface with rich export options so I can share conversations (e.g. single HTML file, PDF, full-height SVG or PNG). Maybe at some point the ability to share a URL for a read-only view of a conversation.
• Auto-discovery of models available for providers that support it such as OpenRouter instead of needing to wait for new models to be added to the app. Perhaps not importing each provider's entire model list, but allowing a search by model name and importing just the ones I want. In addition to the curated model lists you have now if they are reasonable to maintain.
• Transparency of what's being sent and received for the API and providers.
• Responses API support for better interleaving of tool calls and thinking (affects output quality quite a bit - seems to be a more visible issue lately).
• Memory support via vector indexing and search. I want to be able to import the latest documentation and articles for things I use and expose them via tool calls. Right now my chats contain repetitive web searches for docs with results that fill up the context with irrelevant data. Context7 is closest to what I'm thinking, but the UX and quality of results aren't great. Also everything is stored on their servers.

I'd also be interested in your thoughts around the directions/features of Open Web UI and LLM Gateway (and any other projects you think of) compared to this project.

[mythz]

Do you have particular features or improvements in mind?

Plugin System

First thing I want to do is add is a plugin system (ala ComfyUI custom nodes) where you can clone a github repo to ~/.llms/plugins/ and restart the server to extend both server routes and serve custom UIs. I would like to keep the core ChatGPT like functionality baked into the core /ui and anything opinionated that goes beyond this (or anything requiring additional dependencies) in plugins so they can be easily disabled/replaced/etc.

It would also allow others be able to develop, prototype and share functionality easier.

First step is to design how plugin system would work.

Installation:

Should be able to copy anyones git repo into the plugins folder, e.g:

git clone https://github.com/username/repo.git ~/.llms/plugins/repo

So to remove the feature they'd only need to delete the folder.

Requirements, should be able to:

• extend the server aiohttp Web Application routes
• access to provider configuration, clients, etc
• serve their own static ui files
• override built-in components
• Make room in the UI for plugins/extensions (ala VS Code)
  • icons in toolbar (below Model/Prompt selector)
  • vertical icons on the right?
  • status bar?

Convention for storing user data outside the plugins folder, e.g ~/users/default or ~/users/<github-user>

File Storage

Add a file/sha storage feature for attachments, that you raised earlier so we can keep image previews / original attachments in the UI.

Streaming

It would be nice to show streaming of responses from providers that support it.

llama.cpp/llamaswap

Explicit support for llama.cpp and auto discovery of llama-swap models. Maybe vLLM as well if there's demand for it, although I've never used it personally.

Static Website

Adding docs for all features in 1 long README isn't feasible, and will eventually need a static docs website.

Tool call support

Nice to have as we don't have any, as I have no idea what that would look like - copying what someone who does this well would be a good start.

Gemini RAG

Other than that the only feature I would like to build personally atm (after implementing a plugin system) is support for File Search in Gemini where I can select a folder to upload all files to Gemini and be able to query my knowledge base. No idea how that would look/work yet, but it's a feature I'd like to have. It's aggressively priced so I expect this to be very popular for querying knowledge bases.

Chat interface with rich export options

With the export options, given we have optional GitHub OAuth support I think publishing to a gist makes the most sense. But what to export is unclear.

Auto-discovery of models available for providers that support it such as OpenRouter

Not sure about this, it currently uses user-defined model ids so requests can be cascaded to other configured providers if they fail. There's no universal API to discover models/pricing so this would be unique and specific to each provider. As such I think this would be good to add this as a OpenRouter plugin which we could bundle by default.

Transparency of what's being sent and received for the API and providers.

This is available via --verbose flag:

llms --serve 8000 --verbose

Responses API support for better interleaving of tool calls

Maybe something that could fit in Tool support? Not currently on my radar.

Memory support via vector indexing and search.

Similar to the feature I'd like to build with Gemini File Search, Should be a plugin. A lot of people are going to want an all-local/privacy solution.

I'd also be interested in your thoughts around the directions/features of Open Web UI and LLM Gateway

I don't spend anytime thinking about anything I don't use. Not interested in chasing features others are doing that I don't use personally. Happy to copy good ideas if I find them useful but I definitely don't want to be relying on anything that's VC funded which eventually leads into enshittification, as such happy to maintain my own tool that I have complete control of. I primarily want a good AI Assistance experience that I can use for local/cloud LLMs in the same UI. I'd like to keep the core functionality in a single file with minimal deps that's an easy drop-in any Python project (e.g. ComfyUI Custom Node) to access AI Features.

Any other features should be delivered via plugins that users can easily enable/disable.

[mythz]

This is now mostly implemented in v3:
v3 Release Notes: https://llmspy.org/docs/v3

• Plugin System - v3 is a full rewrite focused on extensibility with most features having been implemented as an extension
• Extension System - Support for UI and Server Extensions ala ComfyUI Extension folders from git repos
• File Storage - All generated Image / Audio / Files are saved to ~/.llms/cache using their SHA256 hash, all embedded content replaced with cached links
• SQLite storage - Browser client storage has been replaced with persistent SQLite storage on the server
• Streaming - Not fully implemented, but with server db storage the UI now allows multiple chat sessions in parallel, each tool usage is persisted to db and updated in the UI
• Static Website - The single README.md has been replaced with a fully documented website https://llmspy.org
• Tool call support - Added support for tool calling with Core Tools, add more with extensions

Despite its features the core is still only implemented in a single main.py where most features available from the CLI. All other features are implemented as extensions, so it could easily be re-packaged and built with only the features you want.

Still a lot more that can be done, but I'm happy that everything was rebuilt using extensions which makes it easier for everyone to contribute and develop their own.

[neodon]

Nice, lots of cool things I will check out soon.

[richardkiss]

Is there any interest in using package managers for plug-ins? I know right now adding an extension is doing a git clone in a certain directory. However, this is a bit non-standard, and requires knowing where the code lives, which can be a bit tricky if, for example, installing with uvx or pipx. Non-standard becomes a bigger problem as time goes by since it will likely confuse AI agents (not to mention humans) who may not have memorized that this particular project does things differently.

The uv package manager is excellent and it's become pretty standard for installing python mcp extensions, for example, and python packaging is mature enough to allow non-python artifacts like the tsx used in this project. I'm no expert on pluggy, but I suspect it could do the job for plug ins, and would still allow you to do stuff like uv tool install llms --with git+https://github.com/user/repo@commit_hash, so you wouldn't even have to rely on pypi.

[mythz]

This follows ComfyUI custom_notes / extensions which has one of the most vibrant Software extension ecosystems in Python.

Why add the extra steps, complexity and friction? Extensions are just a folder which you can maintain locally, clone a folder from GitHub or any other source. Note: llms .py detects if you have uv installed which it will use to install your extensions requirements.txt if it exists, otherwise falls back to using pip.

llms-py also supports MCP via the fast_mcp extension, but I ended up having to convert some of the core filesystem MCP because of the noticeable extra overheard and inability to configure it from Python. It's much easier to control its behavior, integration an performance when python code is run directly. The primary benefit of MCPs is being able to run tools implemented in other languages, but there's a performance, overhead, complexity and integration costs to using it.

MCPs are also becoming replaced by SKILLS + scripts, as popularized by openclaw and the success of its https://clawhub.ai ecosystem. If you wanted to use uvx tools I'd recommend adding a skill for it to tell the model when to use your package.