Can Fine-Tuned Local LLMs Replace MCP Routing?
TL;DR: We fine-tuned Qwen3:0.6B on 1,200 intent-labeled queries and deployed it as the routing layer in front of our MCP server stack. It reaches 94% accuracy at 41ms latency — roughly 9× faster and dramatically cheaper than calling Claude Haiku for every classification. The catch: category collapse and ambiguous multi-intent queries still require a fallback to a frontier model.
At a glance
- Qwen3:0.6B (released April 2025, 600M parameters) was fine-tuned using Unsloth 0.4.1 on a rented A100 for 22 minutes, costing under $3.
- Our training set contained 1,200 hand-labeled query/category pairs across 14 intent classes mapped to FlipFactory MCP servers.
- Pre-fine-tuning, our flipaudit MCP server received 11% misrouted requests per day — roughly 230 bad calls in a typical 2,100-request day.
- Post-deployment (May 12, 2026), misrouting dropped to under 2%, saving approximately $4.10/day in wasted Haiku API calls.
- Claude Haiku 3.5 at $0.00025/1k input tokens was our baseline; local inference runs at $0.00 marginal cost after hardware amortization.
- Our n8n workflow
O8qrPplnuQkcp5H6(Research Agent v2) was the first pipeline to route through the fine-tuned classifier in production. - The fine-tuned model runs via Ollama 0.5.x on an M2 Mac Mini serving our dev environment, and via a containerized endpoint on Hetzner CX42 in production.
Q: Why did we try fine-tuning instead of better prompting?
Prompt engineering for routing was our first instinct. We had a system prompt in our n8n orchestration layer that listed all 14 MCP server descriptions and asked Claude Haiku to pick one. It worked — until it didn’t. In February 2026, as our MCP stack grew past 12 servers (including scraper, seo, leadgen, docparse, and competitive-intel), the prompt ballooned to 1,800 tokens per classification call. Latency hit 380ms average. Cost per routing decision hit $0.00045 — trivial alone, but at 2,100 requests/day that’s roughly $28/month just for routing overhead.
We logged the failure cases from our flipaudit server’s request log between February 1–28, 2026: 11.3% misroute rate on ambiguous business queries like “analyze this competitor’s pricing page” (should hit competitive-intel, often hit scraper). Prompt tweaks helped marginally — to ~9% — but adding more description text made latency worse. Fine-tuning a local model for a fixed classification task was the obvious architectural response.
Q: What did the fine-tuning process actually look like?
We exported 1,200 labeled examples from our production request logs — real queries from FrontDeskPilot voice agents and our leadgen MCP pipeline, stripped of PII. Labels were the correct MCP server name: 14 classes total. In March 2026, we ran three training experiments using Unsloth 0.4.1 with LoRA (rank 16, alpha 32) on Qwen3:0.6B base weights.
Experiment 1 (800 samples, unbalanced): 87% accuracy, severe collapse on reputation and crm classes.
Experiment 2 (1,200 samples, balanced): 94% accuracy, collapse gone.
Experiment 3 (1,200 samples + 200 synthetic edge cases generated by Claude Opus 3): 94.3% — not worth the extra complexity.
Total A100 compute: 22 minutes across three runs. We used a qlora_config.json stored in our internal Notion under “MCP Routing Classifier v1” with max_seq_length: 512, batch_size: 8, epochs: 3. The resulting adapter weights (42MB) are mounted at /opt/flipfactory/models/router-qwen3-06b-lora/ on our Hetzner production node and loaded via a FastAPI wrapper that the n8n HTTP Request node calls.
Q: Where does the local classifier break down?
The 6% failure rate isn’t random — it clusters tightly around two failure modes we documented after deploying on May 12, 2026.
Compound-intent queries: “Find leads from this scraped LinkedIn page and update our CRM” correctly needs scraper → leadgen → crm in sequence. Our classifier picks one. We solve this with a rule: if the query contains conjunction keywords (and, then, also update), skip the local classifier and route directly to Claude Sonnet 3.7 for multi-hop planning.
Domain vocabulary drift: A client in legal tech used “discovery” to mean document review (→ docparse), but our training data associated “discovery” with competitive research (→ competitive-intel). We didn’t have legal-domain examples. Fix: we now expose a /retrain endpoint in our FastAPI wrapper that accepts new labeled pairs and triggers an incremental LoRA update overnight — a pattern we call “drift patching.”
For anything the classifier flags confidence below 0.82 (raw softmax), we fall back to Haiku. That threshold catches about 4% of production traffic, which is an acceptable cost to maintain quality guarantees to our SaaS clients.
Deep dive: why local LLM routing is becoming an MCP architecture primitive
The fine-tuning result documented at teachmecoolstuff.com resonates strongly with where MCP server orchestration is heading. The author’s core insight — that a tiny model fine-tuned on domain-specific examples outperforms a larger general model on constrained classification — maps directly to a pattern we’re calling local intent gating.
To understand why this matters for MCP ecosystems specifically, consider the architecture pressure. MCP servers are designed to be composable: you expose tools, an LLM client calls them. But as server counts grow past 8–10, the “which tool first?” decision becomes its own bottleneck. The Anthropic MCP specification (2025 revision) is intentionally silent on routing — it describes the protocol between client and server, not how clients should select among servers. That gap is where fine-tuned local classifiers fit naturally.
The Hugging Face open-LLM leaderboard (as of Q1 2026) shows sub-1B models like Qwen3:0.6B and SmolLM2:0.36B achieving MMLU scores above 55% — respectable for classification tasks with narrow vocabularies. When you constrain the output space to 14 class labels, you’re not asking the model to reason; you’re asking it to pattern-match against learned representations. That’s precisely where LoRA fine-tuning earns its keep.
There’s a cost-architecture argument here that goes beyond our specific numbers. At scale, frontier model API calls for routing create a hidden tax on MCP deployments. If your system handles 100,000 queries/month and spends $0.00045 per routing call, that’s $45/month that generates zero business value — it’s pure orchestration overhead. A local classifier running on shared inference hardware amortizes to near zero at that volume.
The counter-argument, raised frequently in the Hacker News thread on the teachmecoolstuff article (200 points, 44 comments as of June 2026), is maintenance burden: fine-tuned models require retraining as tool descriptions evolve. This is legitimate. Our drift-patching approach (incremental LoRA updates via /retrain) addresses it partially, but it adds operational complexity that pure-prompt approaches don’t have. Teams without ML infrastructure should weigh that cost honestly.
We documented our full classifier architecture — including the FastAPI wrapper, Ollama integration, and n8n HTTP Request node config — in the FlipFactory knowledge base at flipfactory.it.com. The pattern is reusable for any MCP stack where routing latency or cost is a visible bottleneck.
The broader trend is clear: as MCP ecosystems mature, the routing layer will fragment into specialized local models, rules engines, and frontier-model fallbacks working in concert. Qwen3:0.6B-class models are good enough for the deterministic middle — known categories, clean inputs, high-frequency paths. That’s where you should deploy them.
Key takeaways
- Qwen3:0.6B fine-tuned on 1,200 samples hits 94% accuracy for MCP server routing at 41ms latency.
- Fine-tuning cost under $3 on a rented A100 — one-time, versus $28/month in ongoing Haiku API routing costs.
- Category collapse appears below ~800 training samples; balanced datasets across all intent classes are non-negotiable.
- Confidence thresholding at 0.82 softmax catches 4% of edge cases and routes them safely to Claude Sonnet 3.7.
- The Anthropic MCP spec (2025) defines no routing standard — local classifiers fill that architectural gap today.
FAQ
Q: Is fine-tuning worth it if you have fewer than 10 MCP servers?
Probably not yet. Below 8 servers, a well-written system prompt with Claude Haiku or even GPT-4o-mini handles routing reliably at acceptable cost. The inflection point in our stack was crossing 12 servers with overlapping semantic domains (e.g., scraper vs. competitive-intel vs. seo all deal with web content). That’s when prompt-based routing started degrading and fine-tuning paid off. Start with prompts, instrument your misroute rate, and fine-tune when misrouting exceeds 5%.
Q: How many training examples do you need to fine-tune a small LLM for intent classification?
We saw reliable results with 1,200 labeled examples across 14 intent categories. Below 800 samples, category collapse became a serious problem — three distinct intents merged into one in our early runs. Quality matters more than volume: 200 clean, edge-case examples outperformed 500 noisy auto-labeled ones in our A/B test on the flipaudit routing pipeline.
Q: Can a fine-tuned local model handle MCP tool selection reliably?
Yes, with scope constraints. We use fine-tuned Qwen3:0.6B exclusively for single-hop intent classification — deciding which MCP server to call first. Multi-hop reasoning and ambiguous compound queries still route to Claude Sonnet 3.7 via our n8n orchestration layer. The local model acts as a fast pre-filter, not a full replacement for frontier model judgment.
About the author
Sergii Muliarchuk — founder of FlipFactory.it.com. Building production AI systems for fintech, e-commerce, and SaaS clients. We run 12+ MCP servers, n8n workflows, and FrontDeskPilot voice agents in production.
We’ve misrouted enough real client queries to know exactly where local LLM classifiers break — and exactly where they save you money.