MCP Chat CLI

A terminal chat agent that runs an agentic, tool-calling loop over a local, free model. It implements both sides of the Model Context Protocol (MCP): an MCP server that exposes tools, resources, and prompts over an in-memory document store, and an MCP client that drives an agent loop where the model reads and edits those documents to answer you. The model runs on local Ollama by default (zero cost, offline, private), with Groq and GitHub Models as one-line cloud fallbacks.

A Claude like mcp cli chat application on simulated set of document profiles built on all free AI models. This is the foundation project that folio-mcp later extends into a document assistant with a web app.

📄 Live project page: https://shahrukh19s.github.io/ollama-mcp-chat-cli/

What it does

• Runs qwen2.5:7b locally through Ollama: zero cost, offline, private.
• Exposes an MCP server (mcp_server.py) with tools, resources, and prompts over a small in-memory set of document profiles.
• Runs an agentic tool-calling loop: the model can read and edit the documents to answer you, not just chat.
• Switches provider (Ollama, Groq, GitHub Models) by changing two .env variables, no code edits.
• Interactive CLI with @document mentions and /command prompts, plus autocompletion.

Why the design is meaningful

Piece	Why it matters
MCP server + client in one repo	You see both halves of the protocol: the server that publishes capabilities, and the client that consumes them and runs the agent loop.
Agentic tool-calling loop	The model does real work (read and edit documents), so tool calling has to actually function, not just produce chat text.
Local-first, provider-agnostic	The default path is a local model at zero cost; Groq and GitHub Models are drop-in fallbacks via LiteLLM's one OpenAI-shaped interface.

Architecture

  main.py  ->  MCP client (agent loop, holds model access via LiteLLM)
                  |  speaks MCP over stdio
                  v
              mcp_server.py  ->  tools, resources, prompts
                                 over an in-memory document store

The client runs the agent loop and calls the model; the server exposes the documents and tools. LiteLLM gives one interface over Ollama, Groq, and GitHub Models (routed by the model-string prefix).

Requirements

• (First) run and build the previous repository to setup local ollama + litellm tool-calling environment repo-here
• Python 3.10+
• uv
• Ollama running, with the model pulled: ollama pull qwen2.5:7b
• Node.js / npx (only needed for the optional MCP Inspector)
• (Optional) a Groq API key and/or a GitHub PAT for the cloud fallbacks

Demo

A full session on the local model (qwen2.5:7b) — @-mentions, /summarize, /format, a persisted edit_document (Monday → Tuesday), a clear "not found" error, and the bare-/command usage hint:

Exploring the server with the MCP Inspector (click to expand)

Launch it with uv run mcp dev mcp_server.py:

Resources — list_docs returns the document ids; the fetch_doc template returns one document:

Prompts — e.g. format:

Tools — read_document and edit_document:

Setup

1. Configure environment variables

Copy .env.example to .env and fill in the values:

cp .env.example .env

LLM_PROVIDER=ollama
LLM_MODEL=ollama_chat/qwen2.5:7b
OLLAMA_API_BASE=http://localhost:11434

# Optional cloud fallbacks:
GROQ_API_KEY=your-groq-api-key-here
GITHUB_MODELS_API_KEY=your-github-pat-with-models-read-scope
GITHUB_MODELS_API_BASE=https://models.github.ai/inference

USE_UV=1

.env is git-ignored — your keys never leave your machine.

2. Install dependencies

uv sync

3. Run

uv run main.py

Run in a real terminal (Windows Terminal / a normal shell). The CLI uses prompt_toolkit, which needs an interactive console.

Usage

• Basic chat: type a message and press Enter.
• Document retrieval: mention a document with @:

  > What is in @welcome.md?
  

• Commands: use / to run a server prompt (include the document id):

  > /summarize customer-feedback.txt
  

• Tools: ask the model to read or edit a document and it will call the MCP tools:

  > Using your tools, read equipment.pdf and tell me the recommended brew temperature.
  

Switching providers

Edit .env and restart:

Provider	LLM_PROVIDER	LLM_MODEL
Ollama (local, default)	ollama	ollama_chat/qwen2.5:7b
Groq (cloud fallback)	groq	groq/openai/gpt-oss-120b
GitHub Models (cloud fallback)	github	openai/gpt-4o-mini

Inspecting the MCP server

uv run mcp dev mcp_server.py

Opens the MCP Inspector in your browser to browse and call the server's tools, resources, and prompts.

First run: the Inspector is a Node tool (@modelcontextprotocol/inspector) that mcp dev launches via npx, so the first run prompts to install it — answer y. To skip the prompt next time, pre-install it once: npm install -g @modelcontextprotocol/inspector.

Tip: the document list is the resource docs://documents (returns the id list), while each document's content is the template docs://documents/{doc_id}. In the Inspector, click List Resources → list_docs for the id list; if a panel instead shows a single document's text, that's the {doc_id} template's result — hit Refresh or re-select list_docs.

Project structure

agent/
  llm_service.py     # the model call (LiteLLM, one interface over all providers)
  agent_loop.py      # the agentic tool-calling loop
  cli_agent.py       # wires the CLI to the agent
  terminal_ui.py     # prompt-toolkit UI: input, autocompletion, history
  tool_manager.py    # discovers and dispatches the MCP tools
main.py              # entry point
mcp_client.py        # the MCP client session
mcp_server.py        # the MCP server: tools, resources, prompts
docs/                # notes
live-demo/           # a recorded demo
.env.example         # blank placeholders (the real .env is git-ignored)
pyproject.toml       # dependencies (managed by uv)

Notes and limitations (honest)

• Local 7B is slower and less consistent than a large cloud model: expect a one-time model load, then steady generation. For fast, consistent answers, switch to a Groq or GitHub Models fallback.
• The document store is in-memory and small (a set of simulated document profiles). This repo is about proving the MCP server/client plus agentic tool-calling pattern cleanly, which folio-mcp then extends to real files with a web app.
• Secrets live only in the git-ignored .env; .env.example ships blank placeholders.

Tech stack

• MCP Python SDK (FastMCP): the server, the client session, and the transport.
• LiteLLM: one OpenAI-shaped API over Ollama, Groq, and GitHub Models (routes by the model-string prefix).
• prompt-toolkit: the interactive CLI prompt, autocompletion, and history.

Credits & acknowledgements

This project started from the starter scaffold in Anthropic's course. I rebuilt it to run on a free local model (Ollama) with Groq and GitHub Models cloud fallbacks via litellm, implemented the MCP server and client and the agentic tool-calling loop, and replaced the example content and the terminal UI with my own.

Thanks to Anthropic for the scaffold, and to Claude (via Claude Code) for pairing on the build — the commit history keeps the Co-Authored-By: Claude trailers for provenance.

License

Released under the MIT License — © 2026 Abdullah Ansari (Shahrukh19S).