Building a Graph-Based Tool Search Engine for LLM Agents

The author built a graph-based tool search engine called graph-tool-call to help LLM agents find the right tool chains for tasks, rather than just individual tools. The article discusses the limitations of vector search and the results of a competitive benchmark comparing different retrieval strategies.

Details

The author explains that LLM agents need access to a wide range of tools, but simply embedding tool descriptions and using vector search to find the closest match is not enough for real-world tasks that require a chain of tools. The author built graph-tool-call, a Python library that models tool relationships as a graph to retrieve execution chains rather than just individual tools. The article then describes a competitive benchmark the author ran, testing 6 different retrieval strategies across 9 datasets ranging from 19 tools to 1068 tools. The results showed that embedding-based approaches like vector search and hybrid vector+BM25 still outperformed the graph-based methods, achieving over 96% recall. The author also discovered several bugs in the graph-based approach that caused it to actually degrade performance in some cases, leading to a rethinking of the overall architecture.