MultimodalWebSurfer with Gemini fails "Using cl100k_base encoding." when used in MagenticOneGroupChat & MagenticOne

[cattboy]

What happened?

Describe the bug
Running the example autogen.4 using Gemini-2.0-Flash model produces the error "Model gemini-2.0-flash not found. Using cl100k_base encoding." It doesns't ALWAYS fail, reading the console.stream the agent to agent handover of data fails at some point?

setup
vscode 1.98.1
nodejs 20.18.2
python 3.11.9
autogen-agentchat (0.4.9.2)
autogen-ext[magentic-one,openai] (0.4.9.2)
autogen-core==0.4.9.2

To Reproduce
Using the example with gemini .
The Surfer agent visits the webpage, MagenticOneOrchestrator askes to scroll down, back and forth a few times until error

import asyncio
from autogen_agentchat.teams import MagenticOneGroupChat
from autogen_agentchat.ui import Console
from autogen_ext.agents.web_surfer import MultimodalWebSurfer
# from autogen_ext.agents.file_surfer import FileSurfer
# from autogen_ext.agents.magentic_one import MagenticOneCoderAgent
# from autogen_agentchat.agents import CodeExecutorAgent
# from autogen_ext.code_executors.local import LocalCommandLineCodeExecutor
from autogen_ext.models.openai import OpenAIChatCompletionClient

async def main() -> None:
    model_client = OpenAIChatCompletionClient(
        model="gemini-2.0-flash",
        api_key=GOOGLE_API_KEY
    )
    surfer = MultimodalWebSurfer(
        "WebSurfer",
        model_client=model_client,
    )

    team = MagenticOneGroupChat([surfer], model_client=model_client)
    await Console(team.run_stream(task="What is the UV index in Melbourne today?"))

    # # Note: you can also use  other agents in the team
    # team = MagenticOneGroupChat([surfer, file_surfer, coder, terminal], model_client=model_client)
    # file_surfer = FileSurfer( "FileSurfer",model_client=model_client)
    # coder = MagenticOneCoderAgent("Coder",model_client=model_client)
    # terminal = CodeExecutorAgent("ComputerTerminal",code_executor=LocalCommandLineCodeExecutor())


asyncio.run(main())

callstack of model_client

<autogen_ext.models.openai._openai_client.OpenAIChatCompletionClient object at 0x000001B6C1CEE310>
SyntaxError('invalid syntax', ('<string>', 0, 0, '', 0, 0))
SyntaxError('invalid syntax', ('<string>', 0, 0, '', 0, 0))
SyntaxError('invalid syntax', ('<string>', 0, 0, '', 0, 0))
{'vision': True, 'function_calling': True, 'json_output': True, 'family': 'gemini-2.0-flash'}
None
None
'autogen_ext.models.openai.OpenAIChatCompletionClient'
'model'
1
{'vision': True, 'function_calling': True, 'json_output': True, 'family': 'gemini-2.0-flash'}
['component_config_schema', 'component_type']
SyntaxError('invalid syntax', ('<string>', 1, 1, '<_abc._abc_data object at 0x000001B6C1D86CC0>', 1, 2))
NameError("name 'RequestUsage' is not defined")
False
SyntaxError('invalid syntax', ('<string>', 1, 1, '<openai.AsyncOpenAI object at 0x000001B6C1D40090>', 1, 2))
{'model': 'gemini-2.0-flash'}
SyntaxError('invalid syntax', ('<string>', 1, 1, '<bound method BaseOpenAIChatCompletionClient._create_stream_chunks of <autogen_ext.models.openai._openai_client.OpenAIChatCompletionClient object at 0x000001B6C1CEE310>>', 1, 2))
SyntaxError('invalid syntax', ('<string>', 1, 1, '<bound method BaseOpenAIChatCompletionClient._create_stream_chunks_beta_client of <autogen_ext.models.openai._openai_client.OpenAIChatCompletionClient object at 0x000001B6C1CEE310>>', 1, 2))
SyntaxError('invalid syntax', ('<string>', 1, 1, "<bound method OpenAIChatCompletionClient._from_config of <class 'autogen_ext.models.openai._openai_client.OpenAIChatCompletionClient'>>", 1, 2))
SyntaxError('invalid syntax', ('<string>', 1, 1, "<bound method ComponentFromConfig._from_config_past_version of <class 'autogen_ext.models.openai._openai_client.OpenAIChatCompletionClient'>>", 1, 2))
False
{'vision': True, 'function_calling': True, 'json_output': True, 'family': 'gemini-2.0-flash'}
{'model': 'gemini-2.0-flash', 'api_key': 'nonononotforu'}
'gemini-2.0-flash'
SyntaxError('invalid syntax', ('<string>', 1, 1, '<bound method OpenAIChatCompletionClient._to_config of <autogen_ext.models.openai._openai_client.OpenAIChatCompletionClient object at 0x000001B6C1CEE310>>', 1, 2))
NameError("name 'RequestUsage' is not defined")

Expected behavior
Should return the UV Index as example shows

Terminal output

---------- MagenticOneOrchestrator ----------
Please search for "UV index Melbourne today" and report your findings, focusing on information from reputable sources like the Bureau of Meteorology Australia.
---------- WebSurfer ----------
I typed 'UV index Melbourne today Bureau of Meteorology Australia' into the browser search bar.

The web browser is open to the page [UV index Melbourne today Bureau of Meteorology Australia - Search](https://www.bing.com/search?q=UV+index+Melbourne+today+Bureau+of+Meteorology+Australia&FORM=QBLH).
The viewport shows 32% of the webpage, and is positioned at the top of the page
The following text is visible in the viewport:
---skipping copy/paste tons of json junk---
Here is a screenshot of the page.
<image>
---------- MagenticOneOrchestrator ----------
Please confirm today's date in Melbourne, Australia. Based on the search results, it seems the UV Index is predicted to reach 8 (Very High) with sun protection recommended from 10:10 am to 4:50 pm. Please confirm this information from the Bureau of Meteorology for today's date.
Model gemini-2.0-flash not found. Using cl100k_base encoding.

Which packages was the bug in?

Python AgentChat (autogen-agentchat>=0.4.0), Python Extensions (autogen-ext)

AutoGen library version.

Python dev (main branch)

Other library version.

No response

Model used

gemini-2.0-flash

Model provider

Google Gemini

Other model provider

No response

Python version

3.11

.NET version

.NET 8

Operating system

Windows

[ekzhu]

Thanks for reporting this issue. I tried to replicate this and got a different error:

---------- user ----------
What is the UV index in Melbourne today?
---------- MagenticOneOrchestrator ----------

We are working to address the following user request:

What is the UV index in Melbourne today?


To answer this request we have assembled the following team:

WebSurfer: A helpful assistant with access to a web browser. Ask them to perform web searches, open pages, and interact with content (e.g., clicking links, scrolling the viewport, filling in form fields, etc.). It can also summarize the entire page, or answer questions based on the content of the page. It can also be asked to sleep and wait for pages to load, in cases where the page seems not yet fully loaded.


Here is an initial fact sheet to consider:

Here is my response to the pre-survey:

1.  GIVEN OR VERIFIED FACTS
    *   The city is Melbourne.
    *   The time frame is "today".
    *   The data point of interest is the UV index.

2.  FACTS TO LOOK UP
    *   The current date. This information is required to determine what constitutes "today" for Melbourne, given potential time zone differences. Possible sources: A general-purpose search engine like Google ("current date") or a time-zone specific site.
    *   The UV index in Melbourne on the current date. Possible sources: Weather websites and apps specifically for Melbourne (e.g., the Bureau of Meteorology (Australia), AccuWeather, Weather Underground). It may be necessary to search specifically for "Melbourne UV index today" or "UV index Melbourne BOM".

3.  FACTS TO DERIVE
    *   The current date in Melbourne local time from the current date in my local time.

4.  EDUCATED GUESSES
    *   The UV index in Melbourne is likely to be higher during the middle of the day (solar noon) and lower in the early morning or late evening.
    *   Given Melbourne's location in the Southern Hemisphere, the UV index is likely to be higher during the summer months (December-February) and lower during the winter months (June-August).



Here is the plan to follow as best as possible:

Here's the plan:

*   **Use WebSurfer:** Ask WebSurfer to search for "Melbourne UV index today".
*   **Use WebSurfer:** Instruct WebSurfer to summarize the search results and identify the most authoritative sources (e.g., Bureau of Meteorology, reputable weather sites).
*   **Use WebSurfer:** If a reliable source is identified, instruct WebSurfer to visit the website and extract the current UV index for Melbourne.
*   **Present the UV index:** Report the UV index value found by WebSurfer.


---------- MagenticOneOrchestrator ----------
Please search for "Melbourne UV index today" and summarize the search results, identifying the most authoritative sources such as the Bureau of Meteorology or reputable weather websites.
---------- WebSurfer ----------
I typed 'Melbourne UV index today' into '0 characters out of 2000'.

The web browser is open to the page [Melbourne UV index today - Search](https://www.bing.com/search?q=Melbourne+UV+index+today&form=QBLH&sp=-1&lq=0&pq=&sc=0-0&qs=n&sk=&cvid=74F2F96A339A483E950D90D973B1BB52&ghsh=0&ghacc=0&ghpl=).
The viewport shows 30% of the webpage, and is positioned at the top of the page
The following text is visible in the viewport:

Skip to content
Melbourne UV index todayDeep search
All
Copilot
News
Images
Videos
Maps
Shopping
More
Tools
About 536,000 resultsMelbourne, Victoria, AustraliaUpdated over an hour ago
Share
Print
Feedback
56
°F
C64°50°
Wind: 9 MPH
Humidity: 63%
Mostly clear
Mon 17, 6:18 AM
7 AM
10 AM
1 PM
4 PM
7 PM
10 PM
1 AM
4 AM
1 AM
4 AM
7 AM
10 AM
1 PM
4 PM
7 PM
10 PM
Mon 1764°50°Tue 1873°55°Wed 1989°70°Thu 2081°60°Fri 2168°54°Sat 2270°56°Sun 2370°57°Mon 2472°61°Tue 2582°65°Wed 2688°61°57°
4%
7 AM
61°
3%
10 AM
62°
3%
1 PM
63°
2%
4 PM
61°
1%
7 PM
59°
1%
10 PM
57°
1%
1 AM
53°
4 AM
57°
1%
1 AM
53°
4 AM
50°
7 AM
60°
10 AM
68°
1 PM
73°
4 PM
70°
7 PM
62°
10 PM
Detailed forecast · Images by GettyShow background
UV index
No forecast
Gentle breeze
9 MPH, WSW
Sunrise
07:19 AM
Sunset
07:37 PM
WillyWeather
https://uv.willyweather.com.au › vic › melbourne › melbourne.html
Melbourne UV Index, VIC 3000 - WillyWeatherMelbourne UV Index updated daily. Detailed UV forecast charts, with today's UV radiation in real-time.
Launch MSN weather map
Temperature
Radar
Wind
More about Melbourne weather
MSN
Weather.com
WundergroundRow
Accuweather
Historical weather and averages
Temperature
Rain
Avg Low
Avg High
Rec Low
Rec High
Mar
56°
76°
44°(2005)103°(2016)Apr
51°
69°
36°(2009)94°(2005)May
48°
62°
35°(2008)81°(2002)See full year 
Related searches for 
Melbourne UV index todaycurrent uv index Melbourne

The following metadata was extracted from the webpage:

{
    "meta_tags": {
        "referrer": "origin-when-cross-origin",
        "SystemEntropyOriginTrialToken": "A5is4nwJJVnhaJpUr1URgj4vvAXSiHoK0VBbM9fawMskbDUj9WUREpa3JzGAo6xd1Cp2voQEG1h6NQ71AsMznU8AAABxeyJvcmlnaW4iOiJodHRwczovL3d3dy5iaW5nLmNvbTo0NDMiLCJmZWF0dXJlIjoiTXNVc2VyQWdlbnRMYXVuY2hOYXZUeXBlIiwiZXhwaXJ5IjoxNzUzNzQ3MjAwLCJpc1N1YmRvbWFpbiI6dHJ1ZX0=",
        "og:description": "Intelligent search from Bing makes it easier to quickly find what you\u2019re looking for and rewards you.",
        "og:site_name": "Bing",
        "og:title": "Melbourne UV index today - Bing",
        "og:url": "https://www.bing.com/search?q=Melbourne+UV+index+today&form=QBLH&sp=-1&lq=0&pq=&sc=0-0&qs=n&sk=&cvid=74F2F96A339A483E950D90D973B1BB52&ghsh=0&ghacc=0&ghpl=",
        "fb:app_id": "3732605936979161",
        "og:image": "http://www.bing.com/sa/simg/facebook_sharing_5.png",
        "og:type": "website",
        "og:image:width": "600",
        "og:image:height": "315"
    }
}

Here is a screenshot of the page.
<image>
---------- MagenticOneOrchestrator ----------
Please visit the WillyWeather page (https://uv.willyweather.com.au/vic/melbourne/melbourne.html) and extract the current UV index for Melbourne.
---------- WebSurfer ----------
I typed 'https://uv.willyweather.com.au/vic/melbourne/melbourne.html' into the browser address bar.

The web browser is open to the page [Melbourne UV Index, VIC 3000 - WillyWeather](https://uv.willyweather.com.au/vic/melbourne/melbourne.html).
The viewport shows 41% of the webpage, and is positioned at the top of the page
The following text is visible in the viewport:

74,485
1
6:18 am 
17 MarNow
0 Now 
Low
Max UV Today
7.4 High
High risk of harm from unprotected Sun exposure. Alert in effect for 
6 hours between 
11am–5pm.Weather
Wind
Rainfall
Sun
Moon
UV
Tides
Swell
More
Melbourne 
 UV IndexAustraliaVICMelbourne1-Day
3-Day
UV Index
Mon 17 MarTue 18 MarWed 19 MarThu 20 MarFri 21 Mar3+ Moderate
6+ High
8+ Very High
11+ Extreme

The following metadata was extracted from the webpage:

{
    "meta_tags": {
        "og:image": "https://cdnres.willyweather.com.au/images/facebook.png",
        "og:image:secure_url": "https://cdnres.willyweather.com.au/images/facebook.png",
        "apple-itunes-app": "app-id=592978502, app-argument=https://itunes.apple.com/au/app/weather-by-willyweather/id592978502?mt=8&uo=4&at=11lMKC%22",
        "og:url": "https://uv.willyweather.com.au/vic/melbourne/melbourne.html",
        "description": "Melbourne UV Index updated daily. Detailed UV forecast charts, with today's UV radiation in real-time",
        "viewport": "width=device-width,minimum-scale=1.0,maximum-scale=1.0"
    }
}

Here is a screenshot of the page.
<image>
---------- MagenticOneOrchestrator ----------
Find the current time in Melbourne, Australia, and report the UV index for Melbourne at that specific time using the WillyWeather page (https://uv.willyweather.com.au/vic/melbourne/melbourne.html).
---------- WebSurfer ----------
Web surfing error:

Traceback (most recent call last):
  File "/<path>/autogen/python/packages/autogen-ext/src/autogen_ext/agents/web_surfer/_multimodal_web_surfer.py", line 442, in on_messages_stream
    content = await self._generate_reply(cancellation_token=cancellation_token)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<path>/autogen/python/packages/autogen-ext/src/autogen_ext/agents/web_surfer/_multimodal_web_surfer.py", line 625, in _generate_reply
    return await self._execute_tool(message, rects, tool_names, cancellation_token=cancellation_token)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<path>/autogen/python/packages/autogen-ext/src/autogen_ext/agents/web_surfer/_multimodal_web_surfer.py", line 750, in _execute_tool
    return await self._summarize_page(question=question, cancellation_token=cancellation_token)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<path>/autogen/python/packages/autogen-ext/src/autogen_ext/agents/web_surfer/_multimodal_web_surfer.py", line 883, in _summarize_page
    page_markdown: str = await self._playwright_controller.get_page_markdown(self._page)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<path>/autogen/python/packages/autogen-ext/src/autogen_ext/agents/web_surfer/playwright_controller.py", line 569, in get_page_markdown
    res = self._markdown_converter.convert_stream(io.StringIO(html), file_extension=".html", url=page.url)  # type: ignore
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<path>/autogen/python/.venv/lib/python3.12/site-packages/markitdown/_markitdown.py", line 354, in convert_stream
    guesses = self._get_stream_info_guesses(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<path>/autogen/python/.venv/lib/python3.12/site-packages/markitdown/_markitdown.py", line 615, in _get_stream_info_guesses
    result = self._magika.identify_bytes(stream_bytes)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<path>/autogen/python/.venv/lib/python3.12/site-packages/magika/magika.py", line 129, in identify_bytes
    raise TypeError(
TypeError: Input content should be of type 'bytes', not <class 'str'>.

---------- MagenticOneOrchestrator ----------
Find the current time in Melbourne, Australia, and report the UV index for Melbourne at that specific time using the WillyWeather page (https://uv.willyweather.com.au/vic/melbourne/melbourne.html). Extract the information directly instead of summarizing the page.
---------- WebSurfer ----------
The current time is 6:18 am on Mar 17, and the UV index is 0.
---------- MagenticOneOrchestrator ----------
The UV index in Melbourne is currently 0, as of 6:18 am on March 17.

Model gemini-2.0-flash not found. Using cl100k_base encoding.

This warning message is to indicate that tiktoken doesn't have a built-in mapping for gemini-2.0-flash, so a default token counting mechanism is used.