honestly i dont get why current multi-agent setups are built like this.
if you connect 4 or 5 mcp servers (github, postgres, filesystem, slack etc) you easily end up with 40-50+ tool schemas. right now the standard pattern in langchain or claude code is dumping all 4,000+ tokens of json schemas into the prompt on every single turn just so the llm can pick which function to run.
you end up paying token costs and waiting ~2 seconds of network latency just for what is basically an L7 routing lookup.
i spent the last few weeks testing if we can just decouple tool discovery completely from the llm using a local cpu vector mesh (all-MiniLM-L6-v2 + chromadb hnsw index) and a split execution daemon.
instead of embedding just the function name (which fails on synonyms), i combined the tool name, description, schema keys, and tags into a single profile vector. when a prompt comes in, it embeds the query on cpu and pulls the top match before ever touching the llm.
ran a cold-cache benchmark across 100,000 synthetic tool endpoints on my local machine (standard cpu, no gpu):
method top-1 acc avg latency p95 latency
---------------------------------------------------------------
naive keyword scan 38.1% 46.41 ms 57.60 ms
bm25 lexical index 40.4% 194.05 ms 247.25 ms
local vector mesh 70.7% 9.56 ms 11.44 ms
in-memory fast path n/a 0.082 ms 0.15 ms