G.A.C.A.: one bus for many AI models with automatic failover
An open-source proxy between your application and language model providers. It picks an available model, tracks rate limits and, when a provider does not respond, switches to the next one.
Industry: AI app builders, IT teams
In short
- Open source, MIT license
- 11 model providers, including Groq, Cerebras, Google AI, OpenRouter and Mistral
- Up to 30 failover attempts when a provider fails
- One REST endpoint: POST /api/complete
About the project
An application that relies on a single language model provider stops when that provider has an outage or runs out of rate limit. G.A.C.A. (Generative AI Cost Arbitrage) sits between the application and the providers and, for every request, picks a model that is available and ranks well.
The challenge
Many providers with different limits, formats and reliability had to be hidden behind one entry point, so the application sends a request and gets an answer without knowing who generated it.
Goals
- One endpoint for many model providers
- Automatic failover on error
- Rate limit tracking per provider and per model
- Model ranking by success rate and latency
Technical challenges
Limits on two levels
Every provider and every model has its own requests-per-minute and requests-per-day limits. The server tracks them separately, in an in-memory cache, and skips a model that has hit its limit.
A ranking that updates itself
A model's score is calculated automatically from success rate, latency and quality, so selection improves with every request.
Solution and implementation
Failover on outage
When a provider does not respond, the server tries further models, up to 30 attempts, before returning an error.
Costs and tokens
Token usage and estimated USD cost are recorded for every request.
Dashboard and REST API
A React dashboard for managing providers, models and the ranking, and a full REST API, including the completion endpoint and editable prompt templates.
Prompt templates and auto-update
System prompts are editable and accept variables. New models and providers added in code sync with the database automatically, without losing stored data.
Results
The app keeps working when a provider fails
The request goes to the next available model and the application still gets an answer.
One integration instead of many
Instead of handling every provider's API separately, the application talks to one server.
Takeaways
Independence from a single provider
A bus between the application and the models lets you change providers without changing application code.
Scope
- Model selection
- Automatic failover
- Limits and costs
- Admin dashboard
Technologies
- TypeScript
- Node.js
- Prisma
- React
Open-source project
Language: TypeScript, Node.js. License: MIT.
Installation
git clone https://github.com/gacabartosz/gaca-core.git
cd gaca-core
npm install
cp .env.example .env
npx prisma generate
npx prisma db push
npm run seed:force
npm run devUsage example
Send a request without choosing a model. The server picks the best available one.
curl -X POST http://localhost:3002/api/complete \
-H 'Content-Type: application/json' \
-d '{"prompt": "Explain quantum computing in one sentence"}'Frequently asked questions
What is G.A.C.A. for?
For sending requests to language models through one endpoint. The server picks the provider and model itself and switches when one fails.
Do I need paid API keys?
No. The README explains step by step how to create free keys with providers that offer free limits, with no credit card. Paid providers such as Anthropic and OpenAI can be added too.
How do I send a request?
With POST /api/complete and the prompt. The response includes the text, the model and provider name, tokens used and latency.
What does it run on?
Node.js with TypeScript, a database through Prisma and a React dashboard.
Can I add my own provider or model?
Yes. The dashboard and REST API let you manage providers and models, and the auto-update mechanism adds new entries from code without wiping existing settings and stats.