串接多模型的 LLM 基礎建設:API gateway、可觀測性、實驗追蹤怎麼搭(LiteLLM、MLflow)

At 2 a.m., a startup team developing an AI customer service platform received an alert: response times were slowing down, and error rates were skyrocketing. When the engineers opened the backend, they couldn't pinpoint the problem because they had integrated APIs from three different model providers, with some using supplier A and others using supplier B. The code was filled with if-else statements switching between them, and there was no single place to clearly see which provider was slowing down, which one was reporting errors, or how much money was being spent that month. They weren't incompetent programmers; they just lacked a foundational infrastructure.

This is the wall many AI teams have hit in the first half of 2026: models themselves aren't difficult to use, but when you need to use multiple models and deploy them to a production environment, the underlying engineering for "integration, observation, and experimentation" becomes a challenge. This article breaks down this foundational infrastructure into three parts to explain it clearly.

Why this matters now

Two years ago, most AI applications only connected to one model, and integrating a single API was enough to get started. However, in the past half year, I've seen teams moving towards using "multiple models": flagship models for high-difficulty reasoning, cheaper and faster small models for high-frequency simple tasks, and open-source self-hosted models for certain scenarios to land data. I also mentioned this in the article on coding agents - multiple model diversion is key to saving costs.

However, multiple models bring three practical problems. First, each API has different formats, parameters, and error handling, making your code filled with switching logic. Second, you can't see the overall picture - which requests are slow, which are reporting errors, where tokens are being spent, and how the monthly bill is generated, all scattered across different backend systems. Third, you don't know whether "switching to a different model or changing the prompt words will improve or worsen the effect" because there is no systematic record and comparison.

These three problems correspond to the three layers of LLM foundational infrastructure: API gateway (unified integration), observability (seeing the overall picture), and experiment tracking (knowing the pros and cons of changes). As the team scales up, these three layers will eventually need to be supplemented.

Main tools and differences

I'll explain each layer, what problems they solve, and some representative tools:

First layer: API gateway / unified integration
Allows you to use a unified interface to call different models, without needing to write a separate program for each provider.

• LiteLLM: The most commonly mentioned open-source solution for this layer. It helps you connect to multiple model providers with a consistent format, and also supports load balancing, setting up backups (automatically switching to another provider if one fails), and controlling usage and budget for each project. If you want to do multiple model diversion, it's usually the foundation.

Second layer: Observability
Allows you to see what happens to each request - latency, errors, tokens, costs, and even each step of the prompt and response.

• Langfuse: A observability platform specifically designed for LLM applications, which can track the complete call chain, record prompts and responses, and calculate costs. When problems occur, it can trace back to which step went wrong.
• Helicone: Also focuses on monitoring and cost analysis, known for being easy to access, suitable for teams that want to quickly change from "invisible" to "visible".

Third layer: Experiment tracking
Allows you to systematically record "what I changed this time, and what the result was", rather than relying on intuition to judge good or bad.

• MLflow: A veteran tool in the machine learning field, which has greatly enhanced its support for LLM and GenAI in the past two years. It can track experiments, manage versions, and evaluate results. If your team already has an ML background, it's a natural extension.
• Weights & Biases: Also a mainstream choice for experiment tracking and evaluation, with good visualization and convenient sharing of results for team collaboration.

Note that the boundaries between these three layers are becoming increasingly blurred in 2026 - many tools are starting to expand into each other's territory, and a single platform can do both observation and experimentation. So don't worry too much about classification, just recognize what you're missing.

How to use it (a gradual approach)

Not every team needs to start with the full suite. My suggestion is to progress gradually based on pain points:

1. Only one or two models, not yet deployed: Don't rush to build foundational infrastructure. Use makeshift methods and manual records, which are enough for now, and avoid over-engineering.
2. Starting to use multiple models: First, implement an API gateway. Use LiteLLM to unify all model calls to a single interface, making it easier to switch models or add backups later without modifying the code.
3. Deploying to a production environment, starting to have real users: Supplement observability. Record each request's latency, errors, and costs, so you can trace back when problems occur. You'll appreciate this layer when you're called up at 2 a.m.
4. Starting to seriously adjust effects: Supplement experiment tracking. Systematically record and compare each change, such as adjusting prompt words or switching models, using MLflow or Weights & Biases to turn "intuition" into "data-driven".
5. Linking the three layers together: In the mature stage, let the gateway's calls automatically carry observation, and let experiment results be compared to online performance, forming a closed loop.

Common pitfalls and suggestions

• Over-engineering is the biggest waste: If you're still verifying product direction and daily request volumes are in the tens, it's premature to implement full foundational infrastructure. Infrastructure should grow with pain points, not be done prematurely.
• The gateway will become a single point of failure: All traffic goes through this layer, and if it fails, everything fails. If self-hosted, make sure to do high availability, and don't put your lifeline on a single node without backup.
• Observation data may contain sensitive information: When recording complete prompts and responses, you may also store users' sensitive data. Think carefully about whether to mask it before recording, especially in regulated industries.
• Cost observation should be done early: Multiple models are most likely to lose control of bills. Waiting until the bill arrives to realize you've overspent is too late; include costs in observation from day one.
• Don't be intimidated by "big factory ML tools": Tools like MLflow may sound heavy, but you can use only the parts you need, without having to adopt the entire suite.

TheAI Academy's perspective

This foundational infrastructure isn't glamorous, doesn't have flashy demos, but it determines whether your AI product can survive in a production environment. I've seen too many teams put a lot of effort into models and prompt words, only to fail due to basic infrastructure holes like "not knowing why things go wrong after deployment, or burning through budgets without realizing it".

Comment: Models are the engine, and foundational infrastructure is the dashboard and gas tank - without it, you're racing without knowing how much gas you have left.

Specific suggestions for Taiwanese readers: Don't implement the full suite at once; progress gradually based on pain points. If you're an individual or small team doing experiments, you can skip these three layers for now; once you need to "use multiple models simultaneously", start with the LiteLLM gateway layer, which will make it easier to switch models or control costs later. When you really have users and start to fear nighttime issues, supplement observability. Think of this foundational infrastructure as insurance - you won't feel it during normal times, but it will save your life when problems occur. To see how these models are used in programming and auditing scenarios, read our coding agents landscape and AI code review tools guide.

Data sources

• LiteLLM official documentation: https://docs.litellm.ai
• MLflow official website: https://mlflow.org

This article is an explanatory summary of tool categories and architecture, and the actual capabilities and pricing of each tool are subject to change. Please refer to the official announcements for the latest information.