ToolOutputImage not recognized when using LiteLLM (Azure provider) with Agents SDK

[UssuGit]

Please read this first

• Have you read the docs?Agents SDK docs
  Yes. the only mention of ToolOutputImage is here: https://openai.github.io/openai-agents-python/tools/

• Have you searched for related issues? Others may have faced similar issues.
  Yes. There were no issues involving ToolOutputImage usage with LiteLLM

Describe the bug

When using the OpenAI Agents SDK with OpenAI models directly (model="gpt-5.1"), an agent can successfully interpret tool outputs of type ToolOutputImage.
The reasoning engine correctly handles the Base64 data URL, and the model describes the image as expected.

However, when switching the model to:

model = LitellmModel(model="azure/responses/gpt-5.1", api_key=...)

…the agent no longer recognizes the tool output as an image.

Even though the tool returns:

ToolOutputImage(image_url="data:image/jpeg;base64,...", detail="high")

LiteLLM sends an empty function_call_output to the Azure OpenAI endpoint:

'input': [{'type': 'message', 'role': 'user', 'content': [{'type': 'input_text', 'text': 'Show me the image in path/to/image.jpeg. What do you see in this image?'}]}, {'type': 'function_call', 'call_id': 'call_mprvAO3bXx8xLbA2MfDsPzKA', 'name': 'retrieve_test_image', 'arguments': '{"path":"path/to/image.jpeg"}'}, {'type': 'function_call_output', 'call_id': 'call_mprvAO3bXx8xLbA2MfDsPzKA', 'output': []}],

Because of this, the agent receives no image content and cannot describe the image.

This works correctly when not using LiteLLM.

Debug information

• Agents SDK version: (e.g. v0.6.1)
• Python version (e.g. Python 3.10)

Repro steps

Minimal python script that can be run to reproduce the bug:

Without LiteLLM and working:

from agents import ToolOutputImage, function_tool, Agent, Runner,
from agents.extensions.models.litellm_model import LitellmModel
from typing import Union

import json
import base64
import litellm
litellm._turn_on_debug()

@function_tool
async def retrieve_test_image(path : str) -> Union[ToolOutputImage, str]:
    b64 = base64.b64encode(open(path, "rb").read()).decode()
    return ToolOutputImage(image_url=f"data:image/jpeg;base64,{b64}", detail="high")

test_agent = Agent(
    name="Image Test Agent",
    instructions="You retrieve and describe images.",
    model="gpt-5.1",
    tools=[retrieve_test_image]
)
result = await Runner.run(test_agent, "Show me the image in path/to/image.jpeg. What do you see in this image?")

With LiteLLM and not working:

from agents import ToolOutputImage, function_tool, Agent, Runner,
from agents.extensions.models.litellm_model import LitellmModel
from typing import Union

import json
import base64
import litellm
litellm._turn_on_debug()

@function_tool
async def retrieve_test_image(path : str) -> Union[ToolOutputImage, str]:
    b64 = base64.b64encode(open(path, "rb").read()).decode()
    return ToolOutputImage(image_url=f"data:image/jpeg;base64,{b64}", detail="high")

test_agent = Agent(
    name="Image Test Agent",
    instructions="You retrieve and describe images.",
    model=LitellmModel(model="azure/responses/gpt-5.1", api_key=settings.AZURE_API_KEY),
    tools=[retrieve_test_image]
)
result = await Runner.run(test_agent, "Show me the image in path/to/image.jpeg. What do you see in this image?")

Expected behavior

LiteLLM should pass the tool output exactly as provided (the Base64 data URL) to the Azure OpenAI Responses API so the agent can interpret it as an image, just like when using OpenAI directly.

[seratch]

Thanks for sharing this. The LiteLLM adapter is available for Chat Completions compatible format at this moment, so indeed using azure responses this way may not work. We recommend directly using OpenAI client for fully-compatible Responses API models but if anyone is interested in working on the enhancement to support this use case, we're happy to have such changes (that said, if the enhancement could make the internals of the LiteLLM adapter complicated, we may want to hold off having the changes).