I noticed the response times growing longer as I continued to bolster more tools into my AI fitness app, Fitnis.

It was only when I was monitoring the raw requests with Braintrust that I saw the sheer size of the sent message, and realised that something had to change.

The problem

In Fitnis (my AI fitness coach), the agent has six categories of tools spanning roughly 20 individual functions (workout management, data queries, charts, photo processing, some generative UI stuff). When someone says "good morning," the model doesn't need access to process-workout-sheet. But it's sitting right there in the prompt. To clarify, that includes:

Tool title
Detailed tool description
Sample inputs/outputs
Input schema
Output schema

...so you can imagine that with ~20 tools, this really does begin to bloat the prompt(s).

Screenshot from Braintrust showing that the prompt is just way too long

The fix was dead simple: classify first, then load.

The solution: A cheap classifier up front

Before the main (expensive) chat model runs, I ask a fast, cheap model to classify the user's message into tool categories. That way, only the relevant ones get loaded.

Because I'm unsure if the user's requirements will change on a message-to-message basis, I'm currently running this after every message the user sends.

Here's a snippet of the implementation:

const ALWAYS_INCLUDED: ToolCategory[] = ["ui", "context"];
 
const DEPENDENCY_MAP: Partial<Record<ToolCategory, ToolCategory[]>> = {
  workout: ["ui"],
  photo: ["ui"],
  chart: ["query"],
};
 
const CLASSIFICATION_PROMPT = `You are a classifier that determines which tool categories a fitness assistant needs to respond to a user message.
 
Available categories:
- workout: Starting/finishing workouts, logging exercises, completing sets
- query: Querying workout history, analytics, statistics
- chart: Data visualization, graphs, exercise progression over time
- photo: Processing photos of workout sheets, extracting data from images
 
Rules:
- Only include categories that are clearly needed for the user's request.
- If the message is ambiguous, include all relevant ones.
- For general conversation or greetings, return an empty array.`;

The classifier uses Zod structured output via generateObject, so I always get back a valid array of categories; no parsing, no edge cases:

const categorySchema = z.object({
  categories: z.array(z.enum(classifiableCategories)),
  reason: z.string(),
});
 
const { object } = await generateObject({
  model: openrouter.chat("google/gemini-2.5-flash"),
  schema: categorySchema,
  system: CLASSIFICATION_PROMPT,
  prompt: latestUserMessage,
  abortSignal: controller.signal,
});

Handling dependencies

However - it's not that simple. Some categories depend on others.

For example, charts need data, so if the classifier picks chart, I also pull in query.

Photos need ui for presenting extracted results.

I handle this with a dependency map that gets resolved after classification:

const categories = new Set<ToolCategory>(ALWAYS_INCLUDED);
 
for (const cat of object.categories) {
  categories.add(cat);
  const deps = DEPENDENCY_MAP[cat];
  if (deps) {
    for (const dep of deps) {
      categories.add(dep);
    }
  }
}

The ALWAYS_INCLUDED array means foundational stuff like ui and context is always there, even when the classifier returns nothing for casual conversation.

What if it all goes wrong?

What happens when the classifier times out or throws?

Well, I just load everything.

The classifier's an optimisation, not a gate; the main chat flow should never break because classification failed.

const RESOLVE_TIMEOUT_MS = 10_000;
 
export async function resolveToolCategories(
  latestUserMessage: string,
  openrouter: OpenRouterProvider,
): Promise<ToolResolverResult> {
  const controller = new AbortController();
  const timeout = setTimeout(() => controller.abort(), RESOLVE_TIMEOUT_MS);
 
  try {
    // ... classification logic ...
    return { categories, reason: object.reason };
  } catch (error) {
    logger.warn("[ToolResolver] Classification failed, using all categories");
    return {
      categories: new Set<ToolCategory>(ALL_CATEGORIES),
      reason: `Fallback: classification failed`,
    };
  } finally {
    clearTimeout(timeout);
  }
}

Wiring it up

On the server side, each tool category maps to a factory function. Only the resolved categories actually get instantiated:

const toolBuilders: Record<ToolCategory, () => ToolSet> = {
  workout: () => createWorkoutTools(userId),
  query: () => createQueryTools(userId),
  ui: () => createUITools(),
  chart: () => createChartTools(userId),
  photo: () => createPhotoTools(userId),
  context: () => createContextTools(userId),
};
 
const allTools: ToolSet = {};
for (const category of resolvedCategories) {
  Object.assign(allTools, toolBuilders[category]());
}

What this ended up getting me

Fewer tokens per request because only relevant tool schemas get serialized
Better tool selection since the model picks from a narrower, more focused set instead of guessing across 20 functions
Lower latency from less prompt processing for the main model
Better logging because the classifier's reason field tells you exactly why tools were selected

The classifier adds ~200–500ms (I'm using Gemini Flash), but the reduction in main-model processing time more than makes up for it. For casual messages where no tools are needed, the savings are even bigger; the main model gets zero tool schemas.

TL;DR

If you've got a prompt with a lot of tools involved, consider adding a step that identifies the best tools for the conversation.

Takeaways

Structured output over free-form text. generateObject with a Zod schema guarantees valid categories. No parsing headaches.

Minimal classifier prompt. It only sees category descriptions and the user message; no conversation history, no system prompt. Keeps it fast.

Log the classification reason. The reason field is invaluable for debugging. When the agent picks the wrong tools, you can see exactly what the classifier was thinking (weirdly satisfying when it's right, extremely useful when it's not).

Don't over-optimise dependencies. If a category might need another, just add it. Loading one extra category is trivial compared to the cost of a failed tool call.

Cutting my agent's response time in half