.Net: feat(connectors): Support ImageContent in tool/function results

[Cozmopolit]

Motivation and Context

Fixes #13430

Currently, when a Semantic Kernel function returns ImageContent, it gets serialized to JSON - losing the binary image data and preventing multimodal-capable models from processing the image.

This PR enables ImageContent preservation in tool/function results, allowing connectors with multimodal capabilities (Gemini 3+, Anthropic) to pass images natively to the model. This is essential for agentic workflows where tools generate or process images that the model needs to analyze.

Description

FunctionCallsProcessor (shared infrastructure)

• Changed ProcessFunctionResult() return type from string to object
• Added early return for ImageContent to preserve it for multimodal-capable connectors
• Added ImageContentNotSupportedErrorMessage constant for consistent error messaging

Google Gemini Connector (native support)

• Extended FunctionResponsePart with Parts property for nested multimodal content
• Added FunctionResponsePartContent class with InlineData support
• Implemented CreateImageFunctionResponsePart() to convert ImageContent to Gemini's native inlineData format

OpenAI Connector (error handling)

• Added ImageContent check with clear error message (API does not support images in tool results)

OpenAI Agents (error handling)

• Added GetFunctionResultAsString() helper with ImageContent error handling

Amazon Bedrock Agents (error handling)

• Added GetFunctionResultAsString() helper with ImageContent error handling

New Unit Tests

Test	File
ItShouldPreserveImageContentWithoutSerialization	FunctionCallsProcessorTests.cs
FromChatHistoryImageContentInToolResultCreatesInlineDataPart	GeminiRequestTests.cs
FromChatHistoryImageContentWithoutDataThrowsInvalidOperationException	GeminiRequestTests.cs
FromChatHistoryImageContentWithoutMimeTypeThrowsInvalidOperationException	GeminiRequestTests.cs
VerifyAssistantMessageAdapterGetMessageWithImageContentInFunctionResult	AssistantMessageFactoryTests.cs

Notes

• No breaking changes: All changes are to internal APIs. The public FunctionResultContent.Result property is already object?.
• Gemini version detection: The connector does not check the model version. If an older Gemini model does not support functionResponse.parts, the API will return an appropriate error.
• URI-based ImageContent: Only ImageContent with binary data is supported. URI-based ImageContent will throw InvalidOperationException.
• MistralAI: Out of scope - has its own ProcessFunctionResult implementation.
• Related: Prepares infrastructure for PR .Net: feat: Add Anthropic Connector for Claude models #13419 (Anthropic Connector multimodal support)

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
• All unit tests pass, and I have added new tests where possible
• I didn't break anyone 😄

sk-pr-MultimodalToolResults.md

[rogerbarreto]

@Cozmopolit Thanks for the contribution!

With the latest merges into the Google connector this PR is now with a conflict, appreciate if you can take a look.

[Cozmopolit]

@rogerbarreto

Done, test file conflict resolved.

[RamType0]

Why is this staling?

[Copilot]

Pull request overview

This PR updates the .NET function-calling infrastructure so that kernel functions returning ImageContent can be preserved as native multimodal content (instead of being JSON-serialized), enabling connectors like Gemini (3+) to forward image tool results to multimodal models while returning a clear error for connectors/APIs that don’t support multimodal tool outputs.

Changes:

• Updated shared function-result processing to preserve ImageContent (changing ProcessFunctionResult from string to object) and introduced a shared “not supported” error message constant.
• Extended the Google Gemini request model to emit functionResponse.parts[].inlineData for ImageContent tool results, with validation and unit tests.
• Added OpenAI and agent-specific handling that converts ImageContent tool results into a clear error message (plus targeted unit tests).

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
dotnet/src/SemanticKernel.UnitTests/Utilities/AIConnectors/FunctionCallsProcessorTests.cs	Adds unit coverage verifying ImageContent is preserved rather than serialized.
dotnet/src/InternalUtilities/connectors/AI/FunctionCalling/FunctionCallsProcessor.cs	Preserves ImageContent in processed function results; introduces a shared “not supported” error message constant.
dotnet/src/Connectors/Connectors.OpenAI/Core/ClientCore.ChatCompletion.cs	Detects ImageContent in tool results and emits a connector-friendly error message instead of attempting to pass images.
dotnet/src/Connectors/Connectors.Google/Core/Gemini/Models/GeminiRequest.cs	Converts ImageContent tool results into Gemini’s native functionResponse.parts[].inlineData format with validation.
dotnet/src/Connectors/Connectors.Google/Core/Gemini/Models/GeminiPart.cs	Extends Gemini function-response modeling to support nested multimodal parts (parts).
dotnet/src/Connectors/Connectors.Google.UnitTests/Core/Gemini/GeminiRequestTests.cs	Adds tests for inlineData generation and required validation failures (missing data/mime type).
dotnet/src/Agents/UnitTests/OpenAI/Internal/AssistantMessageFactoryTests.cs	Adds a unit test ensuring OpenAI Assistants path returns the expected error text for ImageContent in function results.
dotnet/src/Agents/OpenAI/Internal/AssistantMessageFactory.cs	Adds helper to convert function results to string while handling ImageContent as “not supported”.
dotnet/src/Agents/Bedrock/Extensions/BedrockAgentInvokeExtensions.cs	Adds helper to convert function results to string while handling ImageContent as “not supported”.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.