I've been burned by LLM latency more times than I care to count. You fire off a query to a hosted model, praying the API gods smile upon you, and then you wait. And wait. Or worse—you route it to a local model that chokes on anything beyond a simple summarization. It's a guessing game. Until now.
Enter Wayfinder Router, an open-source project that does what every developer running hybrid LLM setups has been begging for: deterministic query routing between local and hosted models. No more prayers. No more guesswork. Just cold, hard logic deciding where each query goes.
What Wayfinder Actually Does
Wayfinder Router sits in front of your LLM infrastructure and examines every incoming query. It doesn't use fuzzy heuristics or machine learning—it uses deterministic rules. You define the conditions. A query contains sensitive data? Route to local. It's a complex code generation task? Send it to GPT-4. Simple summarization? Keep it local with Llama 3.
The project, hosted on GitHub under itsthelore/wayfinder-router, is refreshingly straightforward. There's no bloated config system. You write rules in a YAML file. That's it. The router parses each query, checks it against your rules, and forwards it to the right endpoint.
“The latency difference between a local model and a hosted API can be the difference between a user staying or leaving. Wayfinder gives you control over that trade-off.”
As of this writing, the repo has 34 points on Hacker News and exactly one comment—a sign of a project still in its infancy. But the idea is solid enough to warrant attention. The developer, who goes by 'itsthelore', has clearly been in the trenches of LLM deployment.
Why Deterministic Matters
The market is flooded with AI routing solutions that claim to be 'smart.' They use embeddings, cosine similarity, or some black-box model to decide where to send queries. That works—until it doesn't. A misclassification sends a private medical document to a hosted API in a foreign country. Or a simple 'what's the weather?' gets routed to a 70B-parameter model that costs you cents per query.
Wayfinder says no to that. Deterministic routing means you write explicit rules: if the query matches pattern X, go to endpoint Y. No surprises. No hallucinations in the routing layer. For compliance-heavy industries—healthcare, finance, legal—this isn't just convenient, it's essential.
The Setup: Minimal, Hacker-Friendly
I cloned the repo and had it running in about ten minutes. The config file is clean:
rules:
- when: query.contains('HIPAA')
route: local-llama3
- when: query.length > 500
route: gpt-4-turbo
- default: local-phi3
That's the whole thing. No dependencies beyond Python 3.10 and a couple of standard libraries. It supports OpenAI-compatible APIs for hosted models and Ollama for local ones. If you're already running Ollama, you're two commands away from integration.
Where It Falls Short
Let's be honest: this isn't production-ready for high-throughput environments. The router processes queries synchronously, and there's no load balancing or failover built in. If your local model goes down, the router doesn't automatically fall back to hosted. That's a feature request waiting to happen.
Also, the rule engine is basic. You get string matching and length checks. No regex yet (though it's in the issues), no semantic matching. For complex routing logic, you'll need to chain multiple rules or write custom preprocessors. The project is clearly v0.1—functional but rough around the edges.
The Verdict
Wayfinder Router isn't revolutionary. It's a simple tool that solves a specific pain point. But that's exactly why it's valuable. Every developer running hybrid LLM setups has hacked together something like this—a janky script that routes based on keywords, a fragile if-else chain in a Flask app. Wayfinder gives you a clean, shareable, and maintainable alternative.
Is it ready for your Fortune 500 company? No. But for a startup, a side project, or a compliance-conscious team, it's a solid foundation. The deterministic approach is a breath of fresh air in an AI landscape obsessed with 'intelligence.' Sometimes you don't need a smart router. You need a reliable one.
The Hacker News community gave it 34 points. One comment. That's not a slam dunk—it's a quiet nod from the people who build this stuff. And that might be worth more than a thousand hype-driven upvotes.
