Introducing Chat Summarization: Cut Your Token Costs by 95%

The Hidden Cost of Conversation History

Your chatbot or agent have been having a productive conversation with a user. They're 50 messages in, discussing a complex topic. Every new message requires sending the entire conversation history to the LLM for context.

The math:

With GPT-4o at $2.50 per million input tokens, a single power user having 10 long conversations per day can cost you $15-25/month — just in repeated context tokens. Scale that to 1,000 active users and you're looking at $15,000-25,000/month in avoidable costs.

How Chat Summarization Works

Mnexium's Chat Summarization uses a rolling summarization approach. Instead of sending your entire conversation history, we intelligently compress older messages into a concise summary while keeping recent messages verbatim.

The AI still has full context of what was discussed — project setup, decisions made, preferences expressed — but compressed into a fraction of the tokens. Recent messages are kept verbatim to maintain conversational flow.

Summarization Modes

Different applications have different needs. A customer support bot might need more context than a casual chatbot. We provide three pre-configured modes plus full customization:

mnx: {
  summarize_config: {
    start_at_tokens: 40000,
    chunk_size: 20000,
    keep_recent_messages: 12,
    summary_target: 900
  }
}

Non-Blocking Background Summarization

Summarization happens asynchronously in the background. When we detect that history exceeds the threshold, we trigger a summarization job but don't block the current request. The user gets their response immediately, and the summary is ready for the next request.

This architecture means zero added latency for your users. The first request after threshold might use full history, but subsequent requests benefit from the cached summary.

Full History Always Preserved

Summarization is purely an optimization — your complete conversation history is always saved. Every message, every response, every detail remains accessible in full fidelity. The summary is only used when injecting context into new LLM requests.

• Complete audit trail: Every message is stored verbatim for compliance and debugging
• History API unchanged: Fetch full conversation history anytime via the History API
• Dashboard access: View and search complete conversations in the Mnexium dashboard
• No data loss: Disable summarization anytime and your full history is still there

Think of summarization as a read-time optimization, not a write-time compression. Your data stays complete — we just send a smarter, smaller payload to the LLM.

The Cost Savings Math

Let's look at real numbers for a typical SaaS application:

Real-World Use Cases

How to Enable Summarization

Enabling summarization is a single parameter in your API request:

const response = await fetch("https://mnexium.com/api/v1/chat/completions", {
  method: "POST",
  headers: {
    "Authorization": "Bearer mnx_live_...",
    "x-openai-key": "sk-...",
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    model: "gpt-4o",
    messages: [{ role: "user", content: "Continue our discussion..." }],
    mnx: {
      subject_id: "user_123",
      chat_id: "chat_456",
      history: true,
      summarize: "balanced"  // "light" | "balanced" | "aggressive" | false
    }
  })
});

Custom Configuration

Need fine-grained control? Pass a custom configuration object:

mnx: {
  subject_id: "user_123",
  chat_id: "chat_456",
  history: true,
  summarize_config: {
    start_at_tokens: 40000,      // When to start summarizing
    chunk_size: 20000,           // Tokens to summarize at a time
    keep_recent_messages: 12,    // Always keep N recent messages
    summary_target: 900          // Target summary length in tokens
  }
}

What Gets Preserved in Summaries

Our summarization prompt is tuned to preserve the information that matters most for continued conversations:

• Key decisions made. "User chose PostgreSQL over MongoDB for better relational queries" — the AI remembers why, not just what.
• Important facts mentioned. Names, dates, numbers, technical specifications — anything that might be referenced later.
• Action items and tasks. "User needs to implement authentication next" — so the AI can follow up appropriately.
• User preferences expressed. Communication style, technical level, areas of expertise — for personalized responses.
• Technical details for reference. API endpoints discussed, code patterns established, architecture decisions — the context needed for coherent assistance.

Summarization + Memory: The Complete Picture

Chat Summarization works beautifully alongside Mnexium's memory system. Here's how they complement each other:

Use both together: summarize: "balanced" keeps your current conversation efficient, while learn: true andrecall: true ensure important facts persist across sessions.

Getting Started

Chat Summarization is available now for all Mnexium users. To get started:

1. 1Add the parameter: Include summarize: "balanced" in your mnx object.
2. 2Monitor in Activity Log: Watch for chat.summarize events to see compression ratios.
3. 3Tune as needed: Start with "balanced", adjust to "light" or "aggressive" based on your quality/cost tradeoff.
4. 4Track savings: Compare your token usage before and after — you'll see the difference immediately.

We're excited to help you build more cost-efficient AI applications. As always, we'd love to hear your feedback on how summarization is working for your use case.

Happy building! 🚀