[BugFix][Refactor] Modular Transformer Pipeline and Fix Gemini/Anthropic Empty Content Handling

[SongChiYoung]

Why are these changes needed?

This change addresses a compatibility issue when using Google Gemini models with AutoGen. Specifically, Gemini returns a 400 INVALID_ARGUMENT error when receiving a response with an empty "text" parameter.

The root cause is that Gemini does not accept empty string values (e.g., "") as valid inputs in the history of the conversation.

To fix this, if the content field is falsy (e.g., None, "", etc.), it is explicitly replaced with a single whitespace (" "), which prevents the Gemini model from rejecting the request.

• Gemini API compatibility: Gemini models reject empty assistant messages (e.g., ""), causing runtime errors. This PR ensures such messages are safely replaced with whitespace where appropriate.
• Avoiding regressions: Applying the empty content workaround only to Gemini, and only to valid message types, avoids breaking OpenAI or other models.
• Reducing duplication: Previously, message transformation logic was scattered and repeated across different message types and models. Modularizing this pipeline removes that redundancy.
• Improved maintainability: With future model variants likely to introduce more constraints, this modular structure makes it easier to adapt transformations without writing ad-hoc code each time.
• Testing for correctness: The new structure is verified with tests, ensuring the bug fix is effective and non-intrusive.

Summary

This PR introduces a modular transformer pipeline for message conversion and fixes a Gemini-specific bug related to empty assistant message content.

Key Changes

• [Refactor] Extracted message transformation logic into a unified pipeline to:

  • Reduce code duplication
  • Improve maintainability
  • Simplify debugging and extension for future model-specific logic

• [BugFix] Gemini models do not accept empty assistant message content.

  • Introduced _set_empty_to_whitespace transformer to replace empty strings with " " only where needed
  • Applied it only to "text" and "thought" message types, not to "tools" to avoid serialization errors

• Improved structure for model-specific handling

  • Transformer functions are now grouped and conditionally applied based on message type and model family
  • This design makes it easier to support future models or combinations (e.g., Gemini + R1)

• Test coverage added

  • Added dedicated tests to verify that empty assistant content causes errors for Gemini
  • Ensured the fix resolves the issue without affecting OpenAI models

---

Motivation

Originally, Gemini-compatible endpoints would fail when receiving assistant messages with empty content ("").
This issue required special handling without introducing brittle, ad-hoc patches.

In addressing this, I also saw an opportunity to modularize the message transformation logic across models.
This improves clarity, avoids duplication, and simplifies future adaptations (e.g., different constraints across model families).

---

📘 AutoGen Modular Message Transformer: Design & Usage Guide

This document introduces the new modular transformer system used in AutoGen for converting LLMMessage instances to SDK-specific message formats (e.g., OpenAI-style ChatCompletionMessageParam).
The design improves reusability, extensibility, and maintainability across different model families.

---

🚀 Overview

Instead of scattering model-specific message conversion logic across the codebase, the new design introduces:

• Modular transformer functions for each message type
• Per-model transformer maps (e.g., for OpenAI-compatible models)
• Optional conditional transformers for multimodal/text hybrid models
• Clear separation between message adaptation logic and SDK-specific builder (e.g., ChatCompletionUserMessageParam)

---

🧱 1. Define Transform Functions

Each transformer function takes:

• LLMMessage: a structured AutoGen message
• context: dict: metadata passed through the builder pipeline

And returns:

• A dictionary of keyword arguments for the target message constructor (e.g., {"content": ..., "name": ..., "role": ...})

def _set_thought_as_content_gemini(message: LLMMessage, context: Dict[str, Any]) -> Dict[str, str | None]:
    assert isinstance(message, AssistantMessage)
    return {"content": message.thought or " "}

---

🪢 2. Compose Transformer Pipelines

Multiple transformer functions are composed into a pipeline using build_transformer_func():

base_user_transformer_funcs: List[Callable[[LLMMessage, Dict[str, Any]], Dict[str, Any]]] = [
    _assert_valid_name,
    _set_name,
    _set_role("user"),
]

user_transformer = build_transformer_func(
    funcs=base_user_transformer_funcs,
    message_param_func=ChatCompletionUserMessageParam
)

• The message_param_func is the actual constructor for the target message class (usually from the SDK).
• The pipeline is ordered — each function adds or overrides keys in the builder kwargs.

---

🗂️ 3. Register Transformer Map

Each model family maintains a TransformerMap, which maps LLMMessage types to transformers:

__BASE_TRANSFORMER_MAP: TransformerMap = {
    SystemMessage: system_transformer,
    UserMessage: user_transformer,
    AssistantMessage: assistant_transformer,
}

register_transformer("openai", model_name_or_family, __BASE_TRANSFORMER_MAP)

• "openai" is currently required (as only OpenAI-compatible format is supported now).
• Registration ensures AutoGen knows how to transform each message type for that model.

---

🔁 4. Conditional Transformers (Optional)

When message construction depends on runtime conditions (e.g., "text" vs. "multimodal"), use:

conditional_transformer = build_conditional_transformer_func(
    funcs_map=user_transformer_funcs_claude,
    message_param_func_map=user_transformer_constructors,
    condition_func=user_condition,
)

Where:

• funcs_map: maps condition label → list of transformer functions

user_transformer_funcs_claude = {
    "text": text_transformers + [_set_empty_to_whitespace],
    "multimodal": multimodal_transformers + [_set_empty_to_whitespace],
}

• message_param_func_map: maps condition label → message builder

user_transformer_constructors = {
    "text": ChatCompletionUserMessageParam,
    "multimodal": ChatCompletionUserMessageParam,
}

• condition_func: determines which transformer to apply at runtime

def user_condition(message: LLMMessage, context: Dict[str, Any]) -> str:
    if isinstance(message.content, str):
        return "text"
    return "multimodal"

---

🧪 Example Flow

llm_message = AssistantMessage(name="a", thought="let’s go")
model_family = "openai"
model_name = "claude-3-opus"

transformer = get_transformer(model_family, model_name, type(llm_message))
sdk_message = transformer(llm_message, context={})

---

🎯 Design Benefits

Feature	Benefit
🧱 Function-based modular design	Easy to compose and test
🧩 Per-model registry	Clean separation across model families
⚖️ Conditional support	Allows multimodal / dynamic adaptation
🔄 Reuse-friendly	Shared logic (e.g., _set_name) is DRY
📦 SDK-specific	Keeps message adaptation aligned to builder interface

---

🔮 Future Direction

• Support more SDKs and formats by introducing new message_param_func
• Global registry integration (currently "openai"-scoped)
• Class-based transformer variant if complexity grows

---

Related issue number

Closes #5762

Checks

• I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://github.com/microsoft/autogen/blob/main/CONTRIBUTING.md to build and test documentation locally.
• I've added tests (if relevant) corresponding to the changes introduced in this PR.
• [ v ] I've made sure all auto checks have passed.

[SongChiYoung]

While preparing this fix, I noticed we currently don't have explicit tests validating empty-string handling for Gemini API.

Would it be helpful if I added a dedicated test case directly in this PR? Or should we handle the test separately in a follow-up PR/issue?

Please let me know what you prefer—happy to help either way!

[ekzhu]

Yes. It would be good to include the test as part of this PR.

Also, we can use model family to check if we need to make the white space adjustment, rather than doing this for all model API

[ekzhu]

Another thing is instead of modifying the returned content, we should be modifying the messages that got sent to the model api. So we need to modify to_oai_types function and enable option to use white space instead of empty string for empty content, if the model family is Gemini

[SongChiYoung]

@ekzhu
After thoroughly considering your feedback, I've become convinced that handling the empty-string issue before sending messages (at the request phase) indeed offers better robustness, maintainability, and extensibility—especially if similar compatibility issues arise with other models in the future.

Therefore, I'll implement a closure-based approach that:

• Defines a message-transformer function once at the API call preparation step.
• Passes this transformer down through the message-processing functions.
• Avoids repeated conditional checks, ensuring minimal overhead.

However, I still see some merit in the original approach (modifying responses), as it keeps changes minimal and simple. If you prefer simplicity and minimal impact on existing code over the extensibility provided by the closure approach, I'd be happy to revert to the original implementation.

I'll proceed with the closure-based solution for now, but please let me know if you'd prefer the simpler response-handling option instead. Happy to accommodate your preference!

here is sample code for closure case

# Define closure once at the top level
def create_content_transformer(model_family: ModelFamily):
    if model_family == ModelFamily.GEMINI:
        return lambda c: " " if not c else c
    else:
        return lambda c: c

# Usage at API request call-site
content_transformer = create_content_transformer(model_family)
oai_messages = [
    to_oai_type(msg, content_transformer=content_transformer)
    for msg in messages
]

# Usage at token counting
token_count_messages = [
    to_oai_type(msg, content_transformer=content_transformer)
    for msg in token_count_messages
]

# at to_oai_type
def to_oai_type(
    message: LLMMessage,
    prepend_name: bool = False,
    content_transformer: Callable[[str], str] = lambda x: x
) -> Sequence[ChatCompletionMessageParam]:

    if isinstance(message, SystemMessage):
        return [system_message_to_oai(message, content_transformer)]
    elif isinstance(message, UserMessage):
        return [user_message_to_oai(message, prepend_name, content_transformer)]
    elif isinstance(message, AssistantMessage):
        return [assistant_message_to_oai(message, content_transformer)]
    else:
        return tool_message_to_oai(message, content_transformer)

[ekzhu]

@SongChiYoung I think what you are doing is a useful step toward a more modular way to manage different message transformation logic for each model family. I think your new approach is in the right direction.

I would consider making it more modular and easier to maintain by having a global dictionary of separate transformation functions that goes from a list of LLMMessage to ChatCompletionMessageParam for each model family.

So we can then refactor the model client to use this dictionary. This will make it easier to solve other problem like this one #6034 easier.

As a next step in a different PR, we can address the parsing of Choice content using similar approach. This allows us to address issues like R1 model response parsing #5961

[SongChiYoung]

@ekzhu Thanks for your valuable feedback!
As you suggested, I'm planning to set up a global registry that maps each ModelFamily to a list of transformer functions, rather than a single transformer. A simple closure will combine these transformers into a single callable per ModelFamily.

Motivation:

• Extensibility: Easily handle future model feature combinations (e.g., Gemini + R1 or other mixed requirements).
• Maintainability: Clearly show each step of the transformation, making debugging and adding new transformers straightforward.

Additionally, I'll place this global registry within a dedicated transform package at the top level of autogen-ext. This cleanly separates the transformer logic from the existing codebase, enhancing discoverability and future maintainability.

Based on your previous comment, it seems like we're on the same page regarding this structural direction, but I just wanted to confirm once more before proceeding. Let me know if you agree or have additional suggestions—once confirmed, I'll try the implementation!

[ekzhu]

Additionally, I'll place this global registry within a dedicated transform package at the top level of autogen-ext. This cleanly separates the transformer logic from the existing codebase, enhancing discoverability and future maintainability.

I would keep this within the autoen_ext.models.openai namespace for now. This is for addressing the compatibility issues with "OpenAI-compatible" endpoints

[SongChiYoung]

@ekzhu Thanks again for your thoughtful feedback!

Please check my new changes of code.

• Global transformer registry by model family
• Removed duplicated transformation logic by unifying into a shared registry
• Gemini assistant fix for empty content / thought
• Added tests for Gemini tool calling edge cases

Details

• Introduced a global registry for message transformers mapped by ModelFamily
• Unified previously duplicated transformer logic (e.g., assistant conditionals) into centralized registry
• Applied Gemini-specific assistant transformers with fallback for empty content
• Fixed Gemini tool-calling issue caused by empty content or thought
• Added test cases to assert Gemini behavior on invalid assistant messages

I completely understand your point about keeping the registry within the `autogen_ext.models.openai` namespace for now, especially since the immediate goal is resolving compatibility issues with OpenAI-compatible endpoints.

Before finalizing the change, I just wanted to briefly share why I initially designed the global transformer registry at the top-level namespace (autogen-ext/transformation):

• Extensibility: This structure easily allows future integrations with other model families beyond OpenAI-compatible APIs (e.g., Gemini, R1, and any new emerging models).
• Maintainability: Clearly defined and separated transformer logic facilitates easier debugging, reviewing, and extending functionality in the long run.

I fully respect your initial suggestion and I'm happy to move forward by placing it under autogen_ext.models.openai as you recommended. But if there's still a possibility to reconsider placing it at the global level—even temporarily—I would greatly appreciate your thoughts once more.

Could you please let me know your final thoughts on this?

Thanks again for your time and patience!

[SongChiYoung]

Updated the PR title and description to better reflect the scope of changes. Let me know if anything else needs clarification!

[SongChiYoung]

@ekzhu As I mentioned in the recent issue I filed (#6083), Anthropic models also raise errors when given empty content — similar to the Gemini case. This applies to both OpenAI-compatible and native Anthropic SDKs.

To address this properly, I realized that the transformer logic needs to apply more broadly — not just under the OpenAI namespace.

I’m truly sorry to repeat this suggestion again, as I know I previously proposed a similar idea. However, this additional context (supporting Anthropic and potentially other models) makes it clearer that placing the transformer registry under a top-level transform package within autogen_ext would help avoid duplicated logic and keep things clean moving forward.

Please let me know what you think — and I really appreciate your patience reviewing this again.

[ekzhu]

They @SongChiYoung thanks for the update.

I understand that it is much more modular with the new approach with potential added benefit, but the PR is getting quite large, and it introduces a new architecture that maybe we as maintainers aren't extremely familiar with.

Consider this, you may stop contributing to the project after a few months with very good reason as you may start work on other things interest you. But we will still be left with the code we may not understand very well.

So, my suggestion is to make the changes incrementally and think about what the minimal design is needed to solve the problems related to OpenAIChatCompletionClient when used for different compatibility endpoints. I think the minimal design is simply a register dictionary with model family prefix and their respective to_oai_type functions. Just need to make sure the top-level to_oai_type function all have the same signature and we are good. This solves the urgent issue and leave a lot of room for future improvements.

Right now, I'd say beefing up the integration test cases for OpenAIChatCompletionClient is more important than creating a new architecture for message transformation. Right now, the tests only use OpenAI and Gemini models, but we hope to increase the modularity and expand to more providers.

[SongChiYoung]

Thank you @ekzhu — I really appreciate your thoughtful feedback, and your team’s dedication to maintaining the quality and long-term sustainability of AutoGen.

Your suggestion makes perfect sense, and I’ll proceed as you advised:

• I’ll simplify the current implementation to use a registry of ModelFamily -> to_oai_type functions.
• I’ll also keep everything scoped within the OpenAI namespace for now, without introducing broader architectural changes.

In addition, since the Anthropic SDK is specific to Anthropic models only, I’ll go ahead and apply the simple fix there as well — without needing any model check logic — and include that in this PR.

Thanks again, and I’ll update the PR shortly!

[ekzhu]

Thank you very much for your work!

[SongChiYoung]

@ekzhu
I've now completed the remaining updates for this PR. Here's a quick summary of what's been addressed:

1. ✅ As suggested, I moved the transformer registry and related logic from autogen-ext/transformation to autogen_ext/models/openai/. The registry now lives entirely within the OpenAI-compatible namespace, as you requested.

2. ✅ I reviewed PR #5989, which introduces reasoning tokens in streaming events. Since the Thought message type already exists and is fully integrated in the message pipeline, this PR ([BugFix][Refactor] Modular Transformer Pipeline and Fix Gemini/Anthropic Empty Content Handling #6063) does not require any change to support it. I also verified that all tests pass with Add a thought process analysis, and add a reasoning field in the ModelClientStreamingChunkEvent to distinguish the thought tokens. #5989 merged.

3. ✅ Regarding issue #6083: I applied a simple and safe fix for Anthropic models, which also reject empty assistant content. The fix ensures compatibility without relying on model checks, and I added dedicated test coverage for this case.

4. ✅ At this point, I believe the PR is feature-complete. All tests are passing, and the changes are now strictly scoped and modular. Please let me know if there's anything else you'd like me to revise or improve—happy to adjust!

5. 🛠️ As a follow-up, I plan to open a separate PR(FEAT: Add missing OpenAI-compatible models (GPT-4.5, Claude models) #6120) to:

   • Register Anthropic models (e.g., Claude 3) in _model_info.py
   • Add GPT-4.5 family to the model list for better test coverage and future support

Let me know what you think—appreciate your time and thoughtful guidance throughout this process!

[SongChiYoung]

Maybe, Solved all of your require changes

[SongChiYoung]

@ekzhu Done.

• Addressed all requested changes
• Resolved merge conflicts
• Added documentation for the new design at the top of this PR (If you want to add it, let me know if there's a better place for it)

[a-holm]

One potential issue identified during review:

In python/packages/autogen-ext/src/autogen_ext/models/openai/_message_transform.py, the code defines a __CLAUDE_TRANSFORMER_MAP which includes the _set_empty_to_whitespace fix (similar to the Gemini map). However, the registration loop later in the file assigns the __BASE_TRANSFORMER_MAP (which lacks the fix) to Claude models (for model in __claude_models: loop). This seems to prevent the fix from being applied to Claude models when accessed via the OpenAI client interface. Could you please double-check if this registration is intended or if it should use __CLAUDE_TRANSFORMER_MAP instead?

It might be on purpose, just pointing it out if it isn't.

[SongChiYoung]

@a-holm
Thanks a lot for catching this — that’s a really good observation.

You’re right: __CLAUDE_TRANSFORMER_MAP was defined separately with the _set_empty_to_whitespace fix (just like Gemini), but the registration loop mistakenly assigns __BASE_TRANSFORMER_MAP to Claude models. That wasn’t intentional — it’s a clear oversight on my part during the transformer registration cleanup.

I’ll fix the registration logic to correctly apply the Claude-specific map. Also planning to add a minimal test case to confirm the whitespace fallback behavior works properly when accessed via the OpenAI-compatible interface.

Really appreciate you pointing this out before it shipped. Saved me from an embarrassing bug down the line!

[SongChiYoung]

I found two other issues while fixing it:

1. Model family is broken when using Anthropic:
   Claude model prefix is claude-3-5-... instead of claude-3.5-..., so I changed them (both model_family and model_info).
   For the same reason, I also changed model_info inside openai.

I needed to change this in this PR, because I check whether a message is from Claude using model_family.startswith(...).
If you want, I could separate this into another PR.

2. Claude rejects empty or whitespace-only messages (not only '', but also ' ' and other all-whitespace strings),
   so I added logic to skip such messages and not send them to the server.

[codecov]

Codecov Report

Attention: Patch coverage is 83.93782% with 31 lines in your changes missing coverage. Please review.

Project coverage is 77.08%. Comparing base (7615c7b) to head (06ab3e6).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...rc/autogen_ext/models/openai/_message_transform.py	85.84%	15 Missing ⚠️
.../autogen_ext/models/anthropic/_anthropic_client.py	33.33%	10 Missing ⚠️
...ogen_ext/models/openai/_transformation/registry.py	91.66%	3 Missing ⚠️
...utogen-ext/src/autogen_ext/models/openai/_utils.py	71.42%	2 Missing ⚠️
...xt/src/autogen_ext/models/openai/_openai_client.py	91.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6063      +/-   ##
==========================================
+ Coverage   76.98%   77.08%   +0.10%     
==========================================
  Files         192      197       +5     
  Lines       13493    13636     +143     
==========================================
+ Hits        10387    10511     +124     
- Misses       3106     3125      +19     

Flag	Coverage Δ
unittests	77.08% <83.93%> (+0.10%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ekzhu]

Great work @SongChiYoung.

As next step (separate PR for each):

1. Mistral model
2. Add module-level documentation like what you have in this PR description to the source code: https://github.com/SongChiYoung/autogen/blob/06ab3e6c67ad663f65e101051e90029ce45df5ac/python/packages/autogen-ext/src/autogen_ext/models/openai/_message_transform.py#L1
3. Use MODEL_POINTERS to map model names for Claude models. Right now, if using claude-3-5-haiku here it will result in error:

https://github.com/SongChiYoung/autogen/blob/06ab3e6c67ad663f65e101051e90029ce45df5ac/python/packages/autogen-ext/tests/models/test_openai_model_client.py#L1623-L1624

[SongChiYoung]

@ekzhu
Haha Thanks a lot.

Mistral model

this issue (and also #6147 and #6145) seems addressable via the modular transformer design proposed in #6063.

• Update of @langchain/mistralai necessary for using Mistral API in Autogen Studio #6147
  The root cause appears to be the unfiltered inclusion of fields (e.g., name) incompatible with Mistral API. With a model-specific transformer map, this could be easily fixed by omitting name in the Mistral-specific branch.

• stream = True is required for the SelectorGroupChat #6145
  QwQ model always need "stream=True" with alibaba server API. With a model-specific transformer map, this could be easily fixed by omitting stream to True in the QwQ-specific branch.

• Issue
  I don’t have access to Mistral/QwQ API Key myself, but if helpful, I can draft a stub transformer (e.g., mistral_user_transformer) to show how it could be handled. Let me know.

Add module-level documentation like what you have in this PR description to the source code

Cool, I’ll get that in and open another PR shortly.

Use MODEL_POINTERS to map model names for Claude models. Right now, if using claude-3-5-haiku here it will result in error:

Haha, just to note, the model pointer for claude-3-5-haiku was actually already registered in this PR.

The error was due to a different underlying issue in AutoGen, not the missing pointer. I had tracked this separately as part of an internal analysis (including a follow-up Issue/PR), but decided to hold off to avoid adding too much surface area all at once — mainly to keep the community's cognitive load low.

I'll share the full context soon, once I package it cleanly.

[ekzhu]

I don’t have access to Mistral/QwQ API Key myself.

We don't have to work on every issue. :) I create a separate issue to address this for Mistral. #6147

The error was due to a different underlying issue in AutoGen, not the missing pointer. I had tracked this separately as part of an internal analysis (including a follow-up Issue/PR), but decided to hold off to avoid adding too much surface area all at once — mainly to keep the community's cognitive load low.

Oh nice. Thanks for investigating.