Skip to content

assistant.md

Latest commit

 

History

History
323 lines (260 loc) · 11.5 KB

assistant.md

File metadata and controls

323 lines (260 loc) · 11.5 KB
Raw

Assistant

The built-in assistant gives a model tools and a long-term memory. Name the tools in the assistant block of the configuration file, and chat with it:

assistant:
  mcp:
    - name: web
      command: [uvx, duckduckgo-mcp-server]
gmlx chat --assistant

This gives the model web search with no API key. Tools need the assistant extra, which Optional features shows how to install. Memory needs an embeddings service, as Memory describes.

The same assistant block works in three places:

Where How to start it Where tools run
Text gmlx chat --assistant On your Mac, as you
Voice gmlx talk with talk.brain: assistant, as The assistant by voice shows On your Mac, as you
API A served assistant id from server.assistants On the server host

The assistant suits short tasks: looking something up, chaining a few tool calls, writing a note or remembering a fact. A turn ends when the model answers. For longer coding work, connect a coding agent with gmlx launch.

Tools

Tools come from MCP servers: separate programs that offer tools to a model. Each entry in assistant.mcp runs a local command or connects to a url:

assistant:
  mcp:
    - name: files
      command: [npx, -y, "@modelcontextprotocol/server-filesystem", "/Users/me/notes"]
    - name: search
      url: http://127.0.0.1:8931/mcp

When the model asks for a tool, the assistant runs it, sends the result back and asks the model again, until the model answers in text. Each round is an ordinary chat request, so an answer that needs several tools takes longer than a plain reply. Models that handle tool calls well, such as Qwen3.8-27B, make fewer mistakes than small ones.

Two settings limit a turn:

Key Default Meaning
assistant.max_tool_rounds 8 Rounds of tool calls before the model must answer without tools.
assistant.tool_timeout_s 60 Seconds before a tool call returns a timeout error to the model.

A command server gets only a few variables from your environment, so pass any token it needs in env. ~ is not expanded in command or env, so write full paths. gmlx doctor checks that each command server's program is found.

Tool examples

Each example is a complete assistant block.

Web search without an API key

This community server searches with DuckDuckGo, which needs no key, and also fetches pages:

assistant:
  mcp:
    - name: web
      command: [uvx, duckduckgo-mcp-server]

Web search on your own SearXNG

A SearXNG instance that you run keeps your queries local. Add json to search.formats in its settings:

assistant:
  mcp:
    - name: search
      command: [npx, -y, mcp-searxng]
      env: {SEARXNG_URL: "http://127.0.0.1:8888"}

Web search with an API key

Brave's official server gives richer results with a free API key:

assistant:
  mcp:
    - name: brave
      command: [npx, -y, "@brave/brave-search-mcp-server"]
      env: {BRAVE_API_KEY: your-key-here}

Document search as a tool

The official Qdrant server, in local mode, stores and finds documents in a folder on disk. It embeds them with its own small model, so it does not need server.embeddings:

assistant:
  mcp:
    - name: docs
      command: [uvx, mcp-server-qdrant]
      env: {QDRANT_LOCAL_PATH: /Users/me/vectors, COLLECTION_NAME: notes}

Unlike memory, which recalls facts about you on every turn, a search tool works on documents you load, and the model decides when to use it. For document search in a chat app, see RAG pipelines.

Decisions as a tool

This tool sends a question with fixed options to /v1/systemone and returns the top answer with its probability, such as train, score 0.88. The model calls it when you ask it to choose between options or ask a yes or no question. Save the script as /Users/me/decide-tool.py. A gmlx checkout has the same file at examples/assistant-tools/decide-tool.py:

# /// script
# dependencies = ["mcp<2"]
# ///
"""A decide tool for the gmlx assistant: asks /v1/systemone and returns the
top answer with its score."""
import json
import os
import urllib.request

from mcp.server.fastmcp import FastMCP

URL = os.environ.get("GMLX_URL", "http://127.0.0.1:8080") + "/v1/systemone"
mcp = FastMCP("decide")


@mcp.tool()
def decide(question: str, options: list[str], context: str = "") -> str:
    """Pick the best of a fixed set of options with the decision model. For a
    yes or no question, pass the options ["yes", "no"]. Put any facts the
    user gave in context."""
    if sorted(o.lower() for o in options) == ["no", "yes"]:
        q = {"type": "noul", "instructions": question}
    else:
        q = {"type": "choice", "instructions": question,
             "criteria": {o: None for o in options}}
    body = {"state": context or question, "questions": {"q": q}}
    req = urllib.request.Request(URL, json.dumps(body).encode(),
                                 {"content-type": "application/json"})
    with urllib.request.urlopen(req, timeout=60) as r:
        a = json.load(r)["answers"]["q"]
    if a["type"] == "noul":
        top, score = ("yes", a["noul"]) if a["noul"] >= 0.5 else ("no", 1 - a["noul"])
    else:
        top, score = a["choice"], a["probabilities"][a["choice"]]
    return f"{top}, score {score:.2f}"


mcp.run()
assistant:
  mcp:
    - name: decide
      command: [uv, run, --script, /Users/me/decide-tool.py]

The server's decision model answers: the model in server.systemone.model, or the default model when that is not set. Serving a decision model compares the choices. The tool needs uv and reaches the server at http://127.0.0.1:8080. For another port, add env: {GMLX_URL: "http://127.0.0.1:PORT"} to its entry.

Memory

The assistant remembers facts across conversations. After each turn, it asks the model for up to three lasting facts from the exchange, such as "sister Ana, birthday March 12". Small talk stores nothing. Before each turn, it finds the stored facts closest to your message and adds them to that request only.

Memory needs server.embeddings. When the server also runs a reranker, it picks the best facts when many match. Without embeddings, the assistant prints a memory disabled warning and runs without memory.

gmlx talk and gmlx chat --assistant share one store, so a fact you say by voice is known in text chat too. Manage it from either client:

Command Effect
/memory Lists the newest 20 facts with their ids
/memory forget ID Removes one fact
/memory clear yes Removes every fact

The store is ~/.local/share/gmlx/assistant-memory.db by default, and assistant.memory.path moves it. top_k sets how many facts a turn gets, 4 by default. ttl_days and max_items limit the store's age and size.

Text chat

gmlx chat --assistant sends each turn through the server and starts the server if it is down. Name a served model id, or none for the server's default model:

gmlx chat --assistant
gmlx chat qwen3.8-27b-ud-q6 --assistant

Chat works as it does with a server model. While the model uses a tool, a status line shows [assistant] using NAME.... /retry and /undo go back over whole tool rounds, and /model <id> switches models and keeps the conversation.

Served assistants

A served assistant is a model id that runs the tool loop in the server. Clients that cannot run a tool loop, such as curl, Open WebUI or a phone app, choose the id as their model and get the tools. The server.assistants block defines each id:

server:
  model_dirs: [~/models]
  assistants:
    helper:                      # The served id, listed in /v1/models.
      model: qwen3.8-27b-ud-q6   # Required. The configured model that answers.
      memory: false              # With true, the id gets its own store.
      mcp: null                  # null uses assistant.mcp. [] gives no tools.

models:                          # The entry that gmlx pull wrote.
  qwen3.8-27b-ud-q6:
    path: unsloth__Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q6_K.gguf

assistant:
  mcp:
    - name: clock
      command: [uvx, mcp-server-time]
curl localhost:8080/v1/chat/completions -H 'content-type: application/json' -d '{
  "model": "helper",
  "messages": [{"role": "user", "content": "What time is it?"}]
}'

At start, the server prints a line such as [server] assistant 'helper' -> qwen3.8-27b-ud-q6 tools: ... for each id. /v1/models lists each id with "assistant": true, and alias_of names the model that answers.

  • Assistant ids work only on /v1/chat/completions.
  • A request that sends its own tools goes to the underlying model unchanged, so a client with its own tool loop keeps it. An empty tools: [] list counts as no tools, so the server runs the loop.
  • The server builds each turn from the messages that the client sends. The reply carries only the answer, so tool results from earlier turns are not kept.
  • response_format is not sent to the rounds, because it would stop tool calls. A request for JSON gets prose.
  • max_tokens limits each round, and defaults to 4096.
  • A streaming reply carries a comment line such as : assistant using NAME for each tool call.
  • A server runs at most 4 assistant turns at a time, and at most 2 for each launch container session. A request over either limit gets a 429 at once.

With memory: true, each id has its own store, ~/.local/share/gmlx/assistant-<id>.db, that every client of the id shares. That suits a personal server, not one with several users. The store uses top_k, extract, ttl_days and max_items from assistant.memory, and ignores enabled and path. Turns from a launch container session do not use memory.

Security

gmlx talk and gmlx chat --assistant run tools on your Mac, as you. A served assistant runs tools on the server host for anyone who can reach the server. On the default loopback address, that is only your Mac.

  • Tools come only from the configuration file. A request cannot add MCP servers or change tools.
  • On an address other than loopback, a server with server.assistants refuses to start unless server.assistant_allow_remote is true. It also needs an API key, unless server.no_auth is set.
  • With assistant_allow_remote: true and tools in assistant.mcp, each assistant must list its own mcp, with [] for no tools, so you choose each one's tools.
  • gmlx doctor warns when assistants are served beyond loopback, and says which tools each one has.