From 4edebc250a6c6189ce7200e859dccc05b13312ed Mon Sep 17 00:00:00 2001 From: gmndrg Date: Wed, 19 Aug 2026 00:59:50 -0600 Subject: [PATCH 1/4] Modernize image serving example --- image-serving-example/image_serving.ipynb | 532 +++++++++++++--------- image-serving-example/requirements.txt | 1 + image-serving-example/sample.env | 38 +- 3 files changed, 340 insertions(+), 231 deletions(-) diff --git a/image-serving-example/image_serving.ipynb b/image-serving-example/image_serving.ipynb index 8de819821..5d4557d80 100644 --- a/image-serving-example/image_serving.ipynb +++ b/image-serving-example/image_serving.ipynb @@ -5,30 +5,31 @@ "id": "92ebfca4", "metadata": {}, "source": [ - "# Example: Image Serving with Azure Blob Knowledge Source\n", + "# Example: Agentic retrieval with image serving using Python\n", "\n", - "This notebook creates an Azure Blob-backed agentic retrieval pipeline on Foundry IQ (Azure AI Search) with **image serving** enabled.\n", - "Image serving delivers extracted image content (as hosted URLs) directly to the downstream model during retrieval,\n", - "so a multimodal chat model can reason over both text and images in one request.\n", + "This notebook creates an Azure Blob-backed agentic retrieval pipeline in Azure AI Search with image serving enabled. Managed ingestion uses Azure Content Understanding in Foundry Tools to create semantic chunks, preserve tables as Markdown, verbalize document-embedded figures, store extracted images in an asset-store blob container, and add `image_path` references to the generated index. During answer synthesis, Azure AI Search sends matching images to the multimodal model. The retrieve response contains image references, not image bytes, so the application downloads a referenced blob separately for rendering.\n", "\n", - "**Pre-requisites:**\n", - "1. Follow the steps in the [image serving documentation](https://learn.microsoft.com/azure/search/agentic-retrieval-how-to-image-serving) for permissions requirements prior to running this sample.\n", - "2. This sample uses token authentication, so it needs a registered application to generate the tokens. You can also modify for Foundry IQ keys.\n", + "**Prerequisites:**\n", + "\n", + "1. Complete the permissions and resource setup in the [image serving documentation](https://learn.microsoft.com/azure/search/agentic-retrieval-how-to-image-serving).\n", + "1. Create a Microsoft Foundry resource in a [region supported by Content Understanding](https://learn.microsoft.com/azure/ai-services/content-understanding/language-region-support), with Azure OpenAI embedding and multimodal chat model deployments. Use the resource endpoint in the `https://.services.ai.azure.com` format.\n", + "1. Assign the search service managed identity **Storage Blob Data Contributor** on the source and asset containers, and **Cognitive Services User** on the Foundry resource.\n", + "1. Assign the identity that runs this notebook **Search Service Contributor**, **Search Index Data Reader**, and **Storage Blob Data Reader** for the asset container.\n", + "1. Upload a PDF with embedded figures or another supported document type to the source container.\n", "\n", "**Flow:**\n", - "1. Create an `azureBlob` knowledge source with `content_extraction_mode=\"standard\"` — images are extracted via\n", - " Azure AI Services, persisted in a Blob asset-store container, and verbalized by a chat model at ingestion time.\n", - "2. Poll until the knowledge source completes its first synchronization.\n", - "3. Create a knowledge base with `enable_image_serving=True` on the knowledge-source reference.\n", - "4. Retrieve — pass `enable_image_serving=True` on `AzureBlobKnowledgeSourceParams` so the service delivers\n", - " image URLs alongside text in the response messages.\n", - "5. Inspect `ImageServingStatistics` from the activity records.\n", - "6. Clean up.\n", + "\n", + "1. Create an `azureBlob` knowledge source with `content_extraction_mode=\"standard\"` and an asset store.\n", + "1. Poll the generated indexer until managed ingestion succeeds.\n", + "1. Verify that the generated index contains a nonempty `image_path`.\n", + "1. Create a knowledge base with image serving enabled.\n", + "1. Retrieve with image serving disabled and enabled, and inspect `ImageServingStatistics`.\n", + "1. Parse an `image_path` reference and download the blob separately with `DefaultAzureCredential`.\n", + "1. Delete the knowledge base and knowledge source when you're finished.\n", "\n", "**SDK:** `azure-search-documents==12.1.0b1` (REST API `2026-05-01-preview`)\n", "\n", - "Save `sample.env` as `.env`, fill in the values, and create a virtual environment from `requirements.txt`\n", - "before running this notebook." + "Save `sample.env` as `.env`, fill in the nonsecret resource values, and create a virtual environment from `requirements.txt` before running this notebook.\n" ] }, { @@ -40,7 +41,7 @@ "\n", "Before you run this cell, save `sample.env` as `.env` and fill in the values.\n", "You should also create a virtual environment with `requirements.txt` as the dependency list\n", - "and select it as the notebook kernel." + "and select it as the notebook kernel.\n" ] }, { @@ -53,44 +54,36 @@ "import os\n", "import time\n", "\n", - "from azure.identity import ClientSecretCredential\n", + "from azure.identity import DefaultAzureCredential\n", "from azure.search.documents.indexes import SearchIndexClient\n", "from azure.search.documents.knowledgebases import KnowledgeBaseRetrievalClient\n", "from dotenv import load_dotenv\n", "\n", "load_dotenv(override=True)\n", "\n", - "endpoint = os.environ[\"AZURE_SEARCH_ENDPOINT\"]\n", - "client_id = os.environ[\"AZURE_CLIENT_ID\"]\n", - "client_secret = os.environ[\"AZURE_CLIENT_SECRET\"]\n", - "tenant_id = os.environ[\"AZURE_TENANT_ID\"]\n", - "ks_name = os.getenv(\"AZURE_SEARCH_KS_NAME\", \"image-serving-ks\")\n", - "kb_name = os.getenv(\"AZURE_SEARCH_KB_NAME\", \"image-serving-kb\")\n", - "blob_connection_string = os.environ[\"AZURE_BLOB_CONNECTION_STRING\"]\n", - "source_container = os.getenv(\"AZURE_BLOB_SOURCE_CONTAINER\", \"source-documents\")\n", - "asset_connection_string = os.getenv(\"AZURE_BLOB_ASSET_CONNECTION_STRING\", blob_connection_string)\n", - "asset_container = os.getenv(\"AZURE_BLOB_ASSET_CONTAINER\", \"image-assets\")\n", - "ai_services_endpoint = os.environ[\"AZURE_AI_SERVICES_ENDPOINT\"]\n", - "ai_services_api_key = os.getenv(\"AZURE_AI_SERVICES_API_KEY\")\n", - "openai_endpoint = os.environ[\"AZURE_OPENAI_ENDPOINT\"]\n", - "openai_embedding_deployment = os.getenv(\"AZURE_OPENAI_EMBEDDING_DEPLOYMENT\", \"text-embedding-3-small\")\n", - "openai_embedding_model = os.getenv(\"AZURE_OPENAI_EMBEDDING_MODEL\", \"text-embedding-3-small\")\n", - "chat_endpoint = os.environ[\"AZURE_OPENAI_CHAT_ENDPOINT\"]\n", - "chat_api_key = os.environ[\"AZURE_OPENAI_CHAT_API_KEY\"]\n", - "chat_deployment = os.getenv(\"AZURE_OPENAI_CHAT_DEPLOYMENT\", \"gpt-4o\")\n", - "chat_model = os.getenv(\"AZURE_OPENAI_CHAT_MODEL\", \"gpt-4o\")\n", - "\n", - "credential = ClientSecretCredential(\n", - " tenant_id=tenant_id,\n", - " client_id=client_id,\n", - " client_secret=client_secret,\n", + "endpoint = os.environ[\"AZURE_SEARCH_ENDPOINT\"]\n", + "ks_name = os.getenv(\"AZURE_SEARCH_KNOWLEDGE_SOURCE_NAME\", \"image-serving-ks\")\n", + "kb_name = os.getenv(\"AZURE_SEARCH_KNOWLEDGE_BASE_NAME\", \"image-serving-kb\")\n", + "storage_resource_id = os.environ[\"AZURE_STORAGE_RESOURCE_ID\"]\n", + "source_container = os.getenv(\"AZURE_BLOB_SOURCE_CONTAINER\", \"source-documents\")\n", + "asset_container = os.getenv(\"AZURE_BLOB_ASSET_CONTAINER\", \"image-assets\")\n", + "foundry_endpoint = os.environ[\"AZURE_FOUNDRY_ENDPOINT\"]\n", + "foundry_embedding_deployment = os.environ[\"AZURE_FOUNDRY_EMBEDDING_DEPLOYMENT\"]\n", + "foundry_embedding_model = os.environ[\"AZURE_FOUNDRY_EMBEDDING_MODEL\"]\n", + "foundry_chat_deployment = os.environ[\"AZURE_FOUNDRY_CHAT_DEPLOYMENT\"]\n", + "foundry_chat_model = os.environ[\"AZURE_FOUNDRY_CHAT_MODEL\"]\n", + "\n", + "managed_identity_connection = f\"ResourceId={storage_resource_id}\"\n", + "credential = DefaultAzureCredential()\n", + "index_client = SearchIndexClient(endpoint=endpoint, credential=credential)\n", + "kb_retrieval_client = KnowledgeBaseRetrievalClient(\n", + " endpoint=endpoint,\n", + " knowledge_base_name=kb_name,\n", + " credential=credential,\n", ")\n", - "index_client = SearchIndexClient(endpoint=endpoint, credential=credential)\n", - "kb_retrieval_client = KnowledgeBaseRetrievalClient(endpoint=endpoint, credential=credential)\n", "\n", - "print(f\"Endpoint: {endpoint}\")\n", "print(f\"Knowledge source: {ks_name}\")\n", - "print(f\"Knowledge base: {kb_name}\")" + "print(f\"Knowledge base: {kb_name}\")" ] }, { @@ -98,23 +91,17 @@ "id": "a9f56138", "metadata": {}, "source": [ - "## Create an Azure Blob knowledge source with image serving\n", - "\n", - "The knowledge source triggers an asynchronous ingestion pipeline that:\n", - "1. Crawls the source Blob container.\n", - "2. Extracts text and images from each document using Azure AI Services (OCR / Computer Vision).\n", - "3. Stores extracted images in the **asset-store** Blob container for later retrieval.\n", - "4. Optionally verbalizes images (generates text descriptions) using the chat model as a fallback\n", - " for models that do not support multimodal input.\n", - "5. Embeds document text using the Azure OpenAI embedding model.\n", - "\n", - "Key `KnowledgeSourceIngestionParameters` fields used here:\n", - "- `content_extraction_mode=\"standard\"` — full text + image extraction (required for image serving).\n", - "- `embedding_model` — vectorizes document text chunks.\n", - "- `chat_completion_model` — verbalizes images and is re-used for answer synthesis at retrieval time.\n", - "- `disable_image_verbalization=False` — keeps verbalization enabled as a fallback.\n", - "- `ai_services` — Azure AI Services used for OCR during ingestion.\n", - "- `asset_store` — target Blob container where extracted images are persisted." + "## Create an Azure Blob knowledge source with managed image extraction\n", + "\n", + "The knowledge source starts managed ingestion that:\n", + "\n", + "1. Reads PDF and image files from the source blob container.\n", + "1. Extracts text and images without an explicit `OcrSkill` or `normalized_images` pipeline.\n", + "1. Stores extracted images in the asset-store blob container.\n", + "1. Writes chunks, vectors, and `image_path` references to a generated index.\n", + "1. Optionally verbalizes images with the chat model during ingestion.\n", + "\n", + "The `ResourceId=` connection format tells Azure AI Search to use its managed identity. Don't put account keys or connection-string secrets in the notebook.\n" ] }, { @@ -127,51 +114,60 @@ "from azure.search.documents.indexes.models import (\n", " AzureBlobKnowledgeSource,\n", " AzureBlobKnowledgeSourceParameters,\n", + " AzureOpenAIVectorizerParameters,\n", " KnowledgeBaseAzureOpenAIModel,\n", ")\n", "from azure.search.documents.knowledgebases.models import (\n", - " KnowledgeSourceIngestionParameters,\n", - " KnowledgeSourceAzureOpenAIVectorizer,\n", " AIServices,\n", " AssetStore,\n", + " KnowledgeSourceAzureOpenAIVectorizer,\n", + " KnowledgeSourceIngestionParameters,\n", ")\n", "\n", - "ks = AzureBlobKnowledgeSource(\n", + "knowledge_source = AzureBlobKnowledgeSource(\n", " name=ks_name,\n", " azure_blob_parameters=AzureBlobKnowledgeSourceParameters(\n", - " connection_string=blob_connection_string,\n", + " connection_string=managed_identity_connection,\n", " container_name=source_container,\n", " ingestion_parameters=KnowledgeSourceIngestionParameters(\n", " content_extraction_mode=\"standard\",\n", " embedding_model=KnowledgeSourceAzureOpenAIVectorizer(\n", - " resource_uri=openai_endpoint,\n", - " deployment_id=openai_embedding_deployment,\n", - " model_name=openai_embedding_model,\n", + " azure_open_ai_parameters=AzureOpenAIVectorizerParameters(\n", + " resource_url=foundry_endpoint,\n", + " deployment_name=foundry_embedding_deployment,\n", + " model_name=foundry_embedding_model,\n", + " )\n", " ),\n", " chat_completion_model=KnowledgeBaseAzureOpenAIModel(\n", - " resource_uri=chat_endpoint,\n", - " api_key=chat_api_key,\n", - " deployment_id=chat_deployment,\n", - " model_name=chat_model,\n", + " azure_open_ai_parameters=AzureOpenAIVectorizerParameters(\n", + " resource_url=foundry_endpoint,\n", + " deployment_name=foundry_chat_deployment,\n", + " model_name=foundry_chat_model,\n", + " )\n", " ),\n", " disable_image_verbalization=False,\n", - " ai_services=AIServices(\n", - " uri=ai_services_endpoint,\n", - " api_key=ai_services_api_key,\n", - " ),\n", + " ai_services=AIServices(uri=foundry_endpoint),\n", " asset_store=AssetStore(\n", - " connection_string=asset_connection_string,\n", + " connection_string=managed_identity_connection,\n", " container_name=asset_container,\n", " ),\n", " ),\n", " ),\n", ")\n", "\n", - "result = index_client.create_or_update_knowledge_source(knowledge_source=ks)\n", - "print(f\"Knowledge source '{result.name}' created.\")\n", - "if getattr(result, \"created_resources\", None):\n", - " for r in result.created_resources:\n", - " print(f\" Created resource: {r}\")" + "result = index_client.create_or_update_knowledge_source(\n", + " knowledge_source=knowledge_source\n", + ")\n", + "if not isinstance(result, AzureBlobKnowledgeSource):\n", + " raise TypeError(\"Expected an Azure Blob knowledge source response.\")\n", + "created_resources = result.azure_blob_parameters.created_resources\n", + "if created_resources is None:\n", + " raise RuntimeError(\n", + " \"The knowledge source response didn't include generated resources.\"\n", + " )\n", + "generated_index_name = created_resources[\"index\"]\n", + "print(f\"Knowledge source '{result.name}' created or updated.\")\n", + "print(f\"Generated index: {generated_index_name}\")" ] }, { @@ -179,11 +175,9 @@ "id": "affc545d", "metadata": {}, "source": [ - "## Poll knowledge source sync status\n", + "## Wait for managed ingestion\n", "\n", - "The `azureBlob` ingestion pipeline is asynchronous. The cell below polls every 30 seconds until the\n", - "knowledge source reports a terminal status (`success` or `error`). Initial sync for a large container\n", - "can take several minutes; image extraction adds extra time compared to text-only pipelines." + "The Azure Blob knowledge source synchronizes asynchronously. Poll its status until the first synchronization completes. Treat failed items as an error instead of continuing with an incomplete index.\n" ] }, { @@ -193,61 +187,116 @@ "metadata": {}, "outputs": [], "source": [ - "POLL_INTERVAL_SECS = 30\n", - "POLL_TIMEOUT_SECS = 1800 # 30 minutes\n", - "\n", - "start = time.time()\n", - "sync_status = None\n", + "POLL_INTERVAL_SECONDS = 30\n", + "POLL_TIMEOUT_SECONDS = 1800\n", "\n", + "deadline = time.monotonic() + POLL_TIMEOUT_SECONDS\n", "while True:\n", - " ks_status = index_client.get_knowledge_source(ks_name)\n", - " sync_status = (\n", - " getattr(ks_status, \"last_sync_status\", None)\n", - " or getattr(ks_status, \"status\", None)\n", - " )\n", - " elapsed = int(time.time() - start)\n", - " print(f\"[{elapsed:4d}s] sync_status={sync_status}\")\n", - "\n", - " if sync_status in (\"success\", \"error\", \"transientFailure\"):\n", + " status = index_client.get_knowledge_source_status(ks_name)\n", + " current = status.current_synchronization_state\n", + " if current is not None:\n", + " print(\n", + " f\"Managed ingestion {status.synchronization_status}: \"\n", + " f\"{current.items_updates_processed} processed, \"\n", + " f\"{current.items_updates_failed} failed, \"\n", + " f\"{current.items_skipped} skipped.\"\n", + " )\n", + " if current.items_updates_failed:\n", + " messages = [error.error_message for error in current.errors or []]\n", + " raise RuntimeError(\n", + " \"Managed ingestion has failed items.\\n\" + \"\\n\".join(messages)\n", + " )\n", + " completed = status.last_synchronization_state\n", + " if completed is not None:\n", + " if completed.items_updates_failed:\n", + " raise RuntimeError(\n", + " \"Managed ingestion completed with \"\n", + " f\"{completed.items_updates_failed} failed item(s).\"\n", + " )\n", + " print(\n", + " \"Managed ingestion completed: \"\n", + " f\"{completed.items_updates_processed} item(s) processed.\"\n", + " )\n", " break\n", - " if time.time() - start > POLL_TIMEOUT_SECS:\n", - " print(\"Timed out waiting for sync. Proceeding anyway.\")\n", + " if time.monotonic() >= deadline:\n", + " raise TimeoutError(\"Timed out waiting for managed ingestion.\")\n", + " time.sleep(POLL_INTERVAL_SECONDS)" + ] + }, + { + "cell_type": "markdown", + "id": "fb4be8f4", + "metadata": {}, + "source": [ + "## Verify the generated index\n", + "\n", + "Managed ingestion creates the index schema and populates `image_path` for chunks associated with extracted images. Query the generated index and stop if no image reference is available.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e2979163", + "metadata": {}, + "outputs": [], + "source": [ + "from azure.search.documents import SearchClient\n", + "\n", + "search_client = SearchClient(\n", + " endpoint=endpoint,\n", + " index_name=generated_index_name,\n", + " credential=credential,\n", + ")\n", + "documents = list(\n", + " search_client.search(\n", + " search_text=\"*\",\n", + " select=[\"blob_url\", \"snippet\", \"image_path\"],\n", + " top=100,\n", + " )\n", + ")\n", + "image_path = None\n", + "for document in documents:\n", + " paths = document.get(\"image_path\") or []\n", + " if paths:\n", + " image_path = paths[0] if isinstance(paths, list) else paths\n", " break\n", - " time.sleep(POLL_INTERVAL_SECS)\n", "\n", - "print(f\"Final sync status: {sync_status}\")" + "if not image_path:\n", + " raise RuntimeError(\n", + " \"The generated index doesn't contain a nonempty image_path. \"\n", + " \"Verify that ingestion succeeded and the source contains extractable images.\"\n", + " )\n", + "\n", + "print(f\"Found image_path: {image_path}\")" ] }, { "cell_type": "markdown", - "id": "fb4be8f4", + "id": "38727fd7", "metadata": {}, "source": [ "## Create a knowledge base with image serving enabled\n", "\n", - "Setting `enable_image_serving=True` on `KnowledgeSourceReference` instructs the service to deliver\n", - "extracted images as hosted URLs in retrieval responses. The `gpt-4o` chat model configured on the\n", - "knowledge base will then receive both text and image URLs when synthesizing its answer." + "Set `enable_image_serving=True` on the blob knowledge-source reference. Azure AI Search uses matching asset-store images as multimodal input during answer synthesis, but the retrieve response doesn't return the image bytes.\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "e2979163", + "id": "ae7a2aa5", "metadata": {}, "outputs": [], "source": [ "from azure.search.documents.indexes.models import (\n", " KnowledgeBase,\n", - " KnowledgeBaseAzureOpenAIModel,\n", " KnowledgeSourceReference,\n", ")\n", "from azure.search.documents.knowledgebases.models import (\n", + " KnowledgeRetrievalMediumReasoningEffort,\n", " KnowledgeRetrievalOutputMode,\n", - " KnowledgeRetrievalMinimalReasoningEffort,\n", ")\n", "\n", - "kb = KnowledgeBase(\n", + "knowledge_base = KnowledgeBase(\n", " name=kb_name,\n", " knowledge_sources=[\n", " KnowledgeSourceReference(\n", @@ -256,90 +305,109 @@ " )\n", " ],\n", " output_mode=KnowledgeRetrievalOutputMode.ANSWER_SYNTHESIS,\n", - " retrieval_reasoning_effort=KnowledgeRetrievalMinimalReasoningEffort(),\n", + " retrieval_reasoning_effort=KnowledgeRetrievalMediumReasoningEffort(),\n", " models=[\n", " KnowledgeBaseAzureOpenAIModel(\n", - " resource_uri=chat_endpoint,\n", - " api_key=chat_api_key,\n", - " deployment_id=chat_deployment,\n", - " model_name=chat_model,\n", + " azure_open_ai_parameters=AzureOpenAIVectorizerParameters(\n", + " resource_url=foundry_endpoint,\n", + " deployment_name=foundry_chat_deployment,\n", + " model_name=foundry_chat_model,\n", + " )\n", " )\n", " ],\n", ")\n", "\n", - "index_client.create_or_update_knowledge_base(knowledge_base=kb)\n", + "index_client.create_or_update_knowledge_base(knowledge_base=knowledge_base)\n", "print(f\"Knowledge base '{kb_name}' created with image serving enabled.\")" ] }, - { - "cell_type": "markdown", - "id": "38727fd7", - "metadata": {}, - "source": [ - "## Retrieve with image serving\n", - "\n", - "Pass `enable_image_serving=True` on `AzureBlobKnowledgeSourceParams` to request image content in the\n", - "retrieval response. When the service matches documents that contain images, it injects image URLs into\n", - "the response messages alongside the text answer.\n", - "\n", - "`include_activity=True` attaches per-step diagnostics to the response, including `ImageServingStatistics`\n", - "on each Azure Blob activity record." - ] - }, { "cell_type": "code", "execution_count": null, - "id": "ae7a2aa5", + "id": "0f5885ba", "metadata": {}, "outputs": [], "source": [ "from azure.search.documents.knowledgebases.models import (\n", - " KnowledgeBaseRetrievalRequest,\n", + " AzureBlobKnowledgeSourceParams,\n", + " KnowledgeBaseAzureBlobActivityRecord,\n", " KnowledgeBaseMessage,\n", " KnowledgeBaseMessageTextContent,\n", - " AzureBlobKnowledgeSourceParams,\n", + " KnowledgeBaseRetrievalRequest,\n", ")\n", "\n", - "question = \"What do the charts and diagrams in these documents show?\"\n", - "\n", - "request = KnowledgeBaseRetrievalRequest(\n", - " messages=[\n", - " KnowledgeBaseMessage(\n", - " role=\"user\",\n", - " content=[KnowledgeBaseMessageTextContent(text=question)],\n", - " )\n", - " ],\n", - " knowledge_source_params=[\n", - " AzureBlobKnowledgeSourceParams(\n", - " knowledge_source_name=ks_name,\n", - " include_references=True,\n", - " include_reference_source_data=True,\n", - " enable_image_serving=True,\n", - " )\n", - " ],\n", - " output_mode=KnowledgeRetrievalOutputMode.ANSWER_SYNTHESIS,\n", - " retrieval_reasoning_effort=KnowledgeRetrievalMinimalReasoningEffort(),\n", - " include_activity=True,\n", + "query = os.getenv(\n", + " \"AZURE_SEARCH_QUERY\",\n", + " \"What information is shown in the diagrams and images?\",\n", ")\n", "\n", - "response = kb_retrieval_client.retrieve(\n", - " knowledge_base_name=kb_name,\n", - " body=request,\n", - ")\n", "\n", - "# Display response messages — may contain both text and image content\n", - "print(\"=== Response messages ===\")\n", - "for msg in (response.response or []):\n", - " role = getattr(msg, \"role\", \"unknown\")\n", - " for item in (msg.content or []):\n", - " text = getattr(item, \"text\", None)\n", - " if text:\n", - " print(f\"[{role}] {text}\")\n", - " image = getattr(item, \"image\", None)\n", - " if image:\n", - " url = getattr(image, \"url\", None)\n", - " if url:\n", - " print(f\"[{role}] \")" + "def retrieve(enable_image_serving):\n", + " request = KnowledgeBaseRetrievalRequest(\n", + " messages=[\n", + " KnowledgeBaseMessage(\n", + " role=\"user\",\n", + " content=[KnowledgeBaseMessageTextContent(text=query)],\n", + " )\n", + " ],\n", + " knowledge_source_params=[\n", + " AzureBlobKnowledgeSourceParams(\n", + " knowledge_source_name=ks_name,\n", + " enable_image_serving=enable_image_serving,\n", + " )\n", + " ],\n", + " include_activity=True,\n", + " output_mode=KnowledgeRetrievalOutputMode.ANSWER_SYNTHESIS,\n", + " )\n", + " return kb_retrieval_client.retrieve(request)\n", + "\n", + "\n", + "def answer_text(retrieval_response):\n", + " if not retrieval_response.response:\n", + " return \"\"\n", + " return \"\\n\".join(\n", + " content.text or \"\"\n", + " for message in retrieval_response.response\n", + " for content in message.content or []\n", + " if isinstance(content, KnowledgeBaseMessageTextContent)\n", + " )\n", + "\n", + "\n", + "def get_image_serving_totals(retrieval_response):\n", + " statistics = [\n", + " record.image_serving\n", + " for record in retrieval_response.activity or []\n", + " if isinstance(record, KnowledgeBaseAzureBlobActivityRecord)\n", + " and record.image_serving is not None\n", + " ]\n", + " return {\n", + " \"images_retrieved\": sum(item.images_retrieved or 0 for item in statistics),\n", + " \"images_sent_to_model\": sum(\n", + " item.images_sent_to_model or 0 for item in statistics\n", + " ),\n", + " \"total_image_size_bytes\": sum(\n", + " item.total_image_size_bytes or 0 for item in statistics\n", + " ),\n", + " \"verbalization_used\": any(\n", + " item.verbalization_used is True for item in statistics\n", + " ),\n", + " }\n", + "\n", + "\n", + "disabled_response = retrieve(enable_image_serving=False)\n", + "for attempt in range(1, 6):\n", + " response = retrieve(enable_image_serving=True)\n", + " if get_image_serving_totals(response)[\"images_sent_to_model\"] > 0:\n", + " break\n", + " print(f\"Image serving isn't ready (attempt {attempt} of 5). Retrying.\")\n", + " time.sleep(POLL_INTERVAL_SECONDS)\n", + "\n", + "disabled_answer = answer_text(disabled_response)\n", + "enabled_answer = answer_text(response)\n", + "print(\"Without image serving:\")\n", + "print(disabled_answer)\n", + "print(\"\\nWith image serving:\")\n", + "print(enabled_answer)" ] }, { @@ -351,8 +419,9 @@ "\n", "Each `KnowledgeBaseAzureBlobActivityRecord` in the response carries an `image_serving` field of type\n", "`ImageServingStatistics` when image serving is active. The statistics show how many images were\n", - "retrieved from the asset store, how many were passed to the model, and whether verbalization was used\n", - "as a fallback." + "retrieved from the asset store, how many were passed to the model, and the total image size.\n", + "The `verbalization_used` value reflects the knowledge source's indexing-time image verbalization\n", + "configuration and state; it isn't a retrieval fallback.\n" ] }, { @@ -362,27 +431,72 @@ "metadata": {}, "outputs": [], "source": [ - "from azure.search.documents.knowledgebases.models import (\n", - " ImageServingStatistics,\n", - " KnowledgeBaseAzureBlobActivityRecord,\n", - ")\n", + "disabled_totals = get_image_serving_totals(disabled_response)\n", + "enabled_totals = get_image_serving_totals(response)\n", + "print(f\"Image serving disabled: {disabled_totals}\")\n", + "print(f\"Image serving enabled: {enabled_totals}\")\n", + "\n", + "assert disabled_totals[\"images_sent_to_model\"] == 0\n", + "assert enabled_totals[\"images_retrieved\"] > 0\n", + "assert enabled_totals[\"images_sent_to_model\"] > 0\n", + "assert enabled_totals[\"total_image_size_bytes\"] > 0" + ] + }, + { + "cell_type": "markdown", + "id": "ca4a4242", + "metadata": {}, + "source": [ + "## Download a referenced image\n", "\n", - "print(\"=== Image serving statistics ===\")\n", - "found_stats = False\n", - "for record in (response.activity or []):\n", - " image_stats = getattr(record, \"image_serving\", None)\n", - " if image_stats is not None:\n", - " found_stats = True\n", - " record_type = getattr(record, \"type\", type(record).__name__)\n", - " print(f\"Activity record type : {record_type}\")\n", - " print(f\" images_retrieved : {getattr(image_stats, 'images_retrieved', 'n/a')}\")\n", - " print(f\" images_sent_to_model: {getattr(image_stats, 'images_sent_to_model', 'n/a')}\")\n", - " print(f\" verbalization_used : {getattr(image_stats, 'verbalization_used', 'n/a')}\")\n", - " print()\n", - "\n", - "if not found_stats:\n", - " print(\"No image serving statistics found in this response.\")\n", - " print(\"Ensure the knowledge source has documents with images and image serving is enabled.\")" + "The retrieve response doesn't contain image bytes. Use the application identity, which needs **Storage Blob Data Reader**, to download a blob referenced by `image_path`. This cell verifies that the blob is nonempty and has an image content type.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "78165d56", + "metadata": {}, + "outputs": [], + "source": [ + "from urllib.parse import unquote, urlparse\n", + "\n", + "from azure.storage.blob import BlobServiceClient\n", + "\n", + "storage_account_url = os.environ[\"AZURE_STORAGE_ACCOUNT_URL\"]\n", + "referenced_image_path = str(image_path).split(\";\", maxsplit=1)[0]\n", + "parsed_path = urlparse(referenced_image_path)\n", + "if parsed_path.scheme and parsed_path.netloc:\n", + " decoded_path = unquote(parsed_path.path).lstrip(\"/\")\n", + " container_prefix = f\"{asset_container}/\"\n", + " blob_name = (\n", + " decoded_path[len(container_prefix) :]\n", + " if decoded_path.lower().startswith(container_prefix.lower())\n", + " else decoded_path\n", + " )\n", + "else:\n", + " blob_name = referenced_image_path\n", + " if \":\" in blob_name:\n", + " blob_name = blob_name.split(\":\", maxsplit=1)[1]\n", + " blob_name = blob_name.lstrip(\"/\")\n", + "\n", + "blob_service_client = BlobServiceClient(\n", + " account_url=storage_account_url,\n", + " credential=credential,\n", + ")\n", + "blob_client = blob_service_client.get_blob_client(\n", + " container=asset_container,\n", + " blob=blob_name,\n", + ")\n", + "properties = blob_client.get_blob_properties()\n", + "image_bytes = blob_client.download_blob().readall()\n", + "content_type = properties.content_settings.content_type or \"\"\n", + "\n", + "assert image_bytes, \"The referenced image blob is empty.\"\n", + "assert content_type.startswith(\n", + " \"image/\"\n", + "), f\"Expected an image content type, but received '{content_type}'.\"\n", + "print(f\"Downloaded {len(image_bytes)} bytes with content type {content_type}.\")" ] }, { @@ -392,7 +506,7 @@ "source": [ "## Clean up\n", "\n", - "Delete the knowledge base first, then the knowledge source." + "Delete the knowledge base first, then the knowledge source. Skip these cells when you want to retain the generated Search pipeline. Deleting the Search resources doesn't delete source documents or projected image blobs. Delete those blobs separately only when no retained ingestion or retrieval pipeline still needs them.\n" ] }, { @@ -400,7 +514,7 @@ "id": "af68322a", "metadata": {}, "source": [ - "### Delete knowledge base" + "### Delete knowledge base\n" ] }, { @@ -410,8 +524,13 @@ "metadata": {}, "outputs": [], "source": [ - "index_client.delete_knowledge_base(kb_name)\n", - "print(f\"Knowledge base '{kb_name}' deleted.\")" + "from azure.core.exceptions import ResourceNotFoundError\n", + "\n", + "try:\n", + " index_client.delete_knowledge_base(kb_name)\n", + " print(f\"Knowledge base '{kb_name}' deleted.\")\n", + "except ResourceNotFoundError:\n", + " print(f\"Knowledge base '{kb_name}' doesn't exist; nothing to delete.\")" ] }, { @@ -419,7 +538,7 @@ "id": "4927c551", "metadata": {}, "source": [ - "### Delete knowledge source" + "### Delete knowledge source\n" ] }, { @@ -429,8 +548,11 @@ "metadata": {}, "outputs": [], "source": [ - "index_client.delete_knowledge_source(knowledge_source=ks_name)\n", - "print(f\"Knowledge source '{ks_name}' deleted.\")" + "try:\n", + " index_client.delete_knowledge_source(knowledge_source=ks_name)\n", + " print(f\"Knowledge source '{ks_name}' deleted.\")\n", + "except ResourceNotFoundError:\n", + " print(f\"Knowledge source '{ks_name}' doesn't exist; nothing to delete.\")" ] } ], diff --git a/image-serving-example/requirements.txt b/image-serving-example/requirements.txt index 1465f01b7..57d6cc81e 100644 --- a/image-serving-example/requirements.txt +++ b/image-serving-example/requirements.txt @@ -1,4 +1,5 @@ azure-search-documents==12.1.0b1 azure-identity +azure-storage-blob python-dotenv ipykernel diff --git a/image-serving-example/sample.env b/image-serving-example/sample.env index bd7fd2fb7..a4eb2f027 100644 --- a/image-serving-example/sample.env +++ b/image-serving-example/sample.env @@ -1,31 +1,17 @@ AZURE_SEARCH_ENDPOINT=https://.search.windows.net -AZURE_SEARCH_KS_NAME=image-serving-ks -AZURE_SEARCH_KB_NAME=image-serving-kb +AZURE_SEARCH_KNOWLEDGE_SOURCE_NAME=image-serving-ks +AZURE_SEARCH_KNOWLEDGE_BASE_NAME=image-serving-kb +AZURE_SEARCH_QUERY=What information is shown in the diagrams and images? -AZURE_CLIENT_ID= -AZURE_CLIENT_SECRET= -AZURE_TENANT_ID= - -# Azure Blob Storage — source documents container (documents to be indexed) -AZURE_BLOB_CONNECTION_STRING=DefaultEndpointsProtocol=https;AccountName=;AccountKey=;EndpointSuffix=core.windows.net +# Azure Blob Storage source and asset containers +AZURE_STORAGE_RESOURCE_ID=/subscriptions//resourceGroups//providers/Microsoft.Storage/storageAccounts/ +AZURE_STORAGE_ACCOUNT_URL=https://.blob.core.windows.net AZURE_BLOB_SOURCE_CONTAINER=source-documents - -# Azure Blob Storage — asset store container (extracted images are stored here) -# You can reuse the same connection string with a different container name -AZURE_BLOB_ASSET_CONNECTION_STRING=DefaultEndpointsProtocol=https;AccountName=;AccountKey=;EndpointSuffix=core.windows.net AZURE_BLOB_ASSET_CONTAINER=image-assets -# Azure AI Services — used for OCR and image understanding at ingestion time -AZURE_AI_SERVICES_ENDPOINT=https://.cognitiveservices.azure.com/ -AZURE_AI_SERVICES_API_KEY= - -# Azure OpenAI — embedding model (vectorises document text at ingestion time) -AZURE_OPENAI_ENDPOINT=https://.openai.azure.com/ -AZURE_OPENAI_EMBEDDING_DEPLOYMENT=text-embedding-3-small -AZURE_OPENAI_EMBEDDING_MODEL=text-embedding-3-small - -# Azure OpenAI — chat model (verbalises images and synthesises answers) -AZURE_OPENAI_CHAT_ENDPOINT=https://.openai.azure.com/ -AZURE_OPENAI_CHAT_API_KEY= -AZURE_OPENAI_CHAT_DEPLOYMENT=gpt-4o -AZURE_OPENAI_CHAT_MODEL=gpt-4o +# Microsoft Foundry resource and model deployments +AZURE_FOUNDRY_ENDPOINT=https://.services.ai.azure.com +AZURE_FOUNDRY_EMBEDDING_DEPLOYMENT=text-embedding-3-small +AZURE_FOUNDRY_EMBEDDING_MODEL=text-embedding-3-small +AZURE_FOUNDRY_CHAT_DEPLOYMENT=gpt-4o +AZURE_FOUNDRY_CHAT_MODEL=gpt-4o From 08ee850196b8c512164fe4bcbed9cb9507809a62 Mon Sep 17 00:00:00 2001 From: gmndrg Date: Wed, 19 Aug 2026 23:47:40 -0600 Subject: [PATCH 2/4] Clarify indexed image download workflow --- image-serving-example/image_serving.ipynb | 28 +++++++++++------------ 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/image-serving-example/image_serving.ipynb b/image-serving-example/image_serving.ipynb index 5d4557d80..d14bcb33d 100644 --- a/image-serving-example/image_serving.ipynb +++ b/image-serving-example/image_serving.ipynb @@ -7,7 +7,7 @@ "source": [ "# Example: Agentic retrieval with image serving using Python\n", "\n", - "This notebook creates an Azure Blob-backed agentic retrieval pipeline in Azure AI Search with image serving enabled. Managed ingestion uses Azure Content Understanding in Foundry Tools to create semantic chunks, preserve tables as Markdown, verbalize document-embedded figures, store extracted images in an asset-store blob container, and add `image_path` references to the generated index. During answer synthesis, Azure AI Search sends matching images to the multimodal model. The retrieve response contains image references, not image bytes, so the application downloads a referenced blob separately for rendering.\n", + "This notebook creates an Azure Blob-backed agentic retrieval pipeline in Azure AI Search with image serving enabled. Managed ingestion uses Azure Content Understanding in Foundry Tools to create semantic chunks, preserve tables as Markdown, verbalize document-embedded figures, store extracted images in an asset-store blob container, and add `image_path` references to the generated index. During answer synthesis, Azure AI Search sends matching images to the multimodal model. The retrieve response reports aggregate image-serving activity but doesn't define dedicated fields for the individual asset-store image paths or image bytes sent to the model. Separately, the notebook queries the generated index for an `image_path` and downloads that indexed asset to validate application access.\n", "\n", "**Prerequisites:**\n", "\n", @@ -21,10 +21,10 @@ "\n", "1. Create an `azureBlob` knowledge source with `content_extraction_mode=\"standard\"` and an asset store.\n", "1. Poll the generated indexer until managed ingestion succeeds.\n", - "1. Verify that the generated index contains a nonempty `image_path`.\n", + "1. Independently query the generated index for a nonempty `image_path`.\n", "1. Create a knowledge base with image serving enabled.\n", "1. Retrieve with image serving disabled and enabled, and inspect `ImageServingStatistics`.\n", - "1. Parse an `image_path` reference and download the blob separately with `DefaultAzureCredential`.\n", + "1. Download the independently selected indexed image asset with `DefaultAzureCredential`.\n", "1. Delete the knowledge base and knowledge source when you're finished.\n", "\n", "**SDK:** `azure-search-documents==12.1.0b1` (REST API `2026-05-01-preview`)\n", @@ -230,7 +230,7 @@ "source": [ "## Verify the generated index\n", "\n", - "Managed ingestion creates the index schema and populates `image_path` for chunks associated with extracted images. Query the generated index and stop if no image reference is available.\n" + "Managed ingestion creates the index schema and populates `image_path` for chunks associated with extracted images. Independently query the generated index and stop if no indexed image path is available.\n" ] }, { @@ -254,20 +254,20 @@ " top=100,\n", " )\n", ")\n", - "image_path = None\n", + "selected_image_path = None\n", "for document in documents:\n", " paths = document.get(\"image_path\") or []\n", " if paths:\n", - " image_path = paths[0] if isinstance(paths, list) else paths\n", + " selected_image_path = paths[0] if isinstance(paths, list) else paths\n", " break\n", "\n", - "if not image_path:\n", + "if not selected_image_path:\n", " raise RuntimeError(\n", " \"The generated index doesn't contain a nonempty image_path. \"\n", " \"Verify that ingestion succeeded and the source contains extractable images.\"\n", " )\n", "\n", - "print(f\"Found image_path: {image_path}\")" + "print(f\"Selected indexed image_path: {selected_image_path}\")" ] }, { @@ -447,9 +447,9 @@ "id": "ca4a4242", "metadata": {}, "source": [ - "## Download a referenced image\n", + "## Download an indexed image asset\n", "\n", - "The retrieve response doesn't contain image bytes. Use the application identity, which needs **Storage Blob Data Reader**, to download a blob referenced by `image_path`. This cell verifies that the blob is nonempty and has an image content type.\n" + "The retrieve response reports aggregate image-serving activity but doesn't define dedicated fields for individual asset-store image paths or image bytes sent to the model. The earlier wildcard index query selected `selected_image_path` independently of retrieval. Use the application identity, which needs **Storage Blob Data Reader**, to download that indexed asset. This cell verifies that the blob is nonempty and has an image content type.\n" ] }, { @@ -464,8 +464,8 @@ "from azure.storage.blob import BlobServiceClient\n", "\n", "storage_account_url = os.environ[\"AZURE_STORAGE_ACCOUNT_URL\"]\n", - "referenced_image_path = str(image_path).split(\";\", maxsplit=1)[0]\n", - "parsed_path = urlparse(referenced_image_path)\n", + "indexed_image_path = str(selected_image_path).split(\";\", maxsplit=1)[0]\n", + "parsed_path = urlparse(indexed_image_path)\n", "if parsed_path.scheme and parsed_path.netloc:\n", " decoded_path = unquote(parsed_path.path).lstrip(\"/\")\n", " container_prefix = f\"{asset_container}/\"\n", @@ -475,7 +475,7 @@ " else decoded_path\n", " )\n", "else:\n", - " blob_name = referenced_image_path\n", + " blob_name = indexed_image_path\n", " if \":\" in blob_name:\n", " blob_name = blob_name.split(\":\", maxsplit=1)[1]\n", " blob_name = blob_name.lstrip(\"/\")\n", @@ -492,7 +492,7 @@ "image_bytes = blob_client.download_blob().readall()\n", "content_type = properties.content_settings.content_type or \"\"\n", "\n", - "assert image_bytes, \"The referenced image blob is empty.\"\n", + "assert image_bytes, \"The selected indexed image blob is empty.\"\n", "assert content_type.startswith(\n", " \"image/\"\n", "), f\"Expected an image content type, but received '{content_type}'.\"\n", From 3e2a37fd18cd1a79341245d2f33c18561d0f248a Mon Sep 17 00:00:00 2001 From: haileytap <170486701+haileytap@users.noreply.github.com> Date: Thu, 20 Aug 2026 10:17:35 -0400 Subject: [PATCH 3/4] Content quality --- README.md | 5 +++-- image-serving-example/image_serving.ipynb | 6 ++---- 2 files changed, 5 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index e2ce61797..3ca0be652 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ This repository contains Python code samples used in Azure AI Search documentati If your configuration uses a search service managed identity for indexer connections, your search service must be on the Basic tier or higher. -## Day-one quickstarts and tutorials +## Day-one quickstarts | Sample | Description | |--------|-------------| @@ -15,7 +15,7 @@ If your configuration uses a search service managed identity for indexer connect | [Quickstart-Semantic-Ranking](Quickstart-Semantic-Ranking/semantic-ranking-quickstart.ipynb) | Extends the quickstart through modifications that invoke semantic ranking. This notebook adds a semantic configuration to the index and semantic query options that formulate the query and response. | | [Quickstart-Vector-Search](Quickstart-Vector-Search/quickstart-vector-search.ipynb) | Introduces vector search in Azure AI Search. This notebook demonstrates how to create, load, and query a vector index. | -## Deeper dive tutorials +## Deeper dive tutorials and examples | Sample | Description | |--------|-------------| @@ -23,6 +23,7 @@ If your configuration uses a search service managed identity for indexer connect | [azure-function-search](azure-function-search/readme.md) | An Azure Function that sends query requests to an Azure AI Search service. You can substitute this code to replace the contents of the `api` folder in the C# sample [azure-search-static-web-app](https://github.com/Azure-Samples/azure-search-static-web-app). | | [bulk-insert](bulk-insert/readme.md) | Create and load an index using the push APIs and sample data. You can substitute this code to replace the contents of the `bulk-insert` folder in the C# sample [azure-search-static-web-app](https://github.com/Azure-Samples/azure-search-static-web-app) | | [cmk-encryption](cmk-example/cmk-example.ipynb) | Encrypt content using customer-managed keys. | +| [image-serving-example](image-serving-example/image_serving.ipynb) | Runs managed ingestion, compares retrieval with image serving disabled and enabled, and independently queries and downloads an indexed image asset from Azure Blob Storage. | ## Archived samples diff --git a/image-serving-example/image_serving.ipynb b/image-serving-example/image_serving.ipynb index d14bcb33d..2492fe1f2 100644 --- a/image-serving-example/image_serving.ipynb +++ b/image-serving-example/image_serving.ipynb @@ -5,7 +5,7 @@ "id": "92ebfca4", "metadata": {}, "source": [ - "# Example: Agentic retrieval with image serving using Python\n", + "# Example: Image serving for agentic retrieval using Python\n", "\n", "This notebook creates an Azure Blob-backed agentic retrieval pipeline in Azure AI Search with image serving enabled. Managed ingestion uses Azure Content Understanding in Foundry Tools to create semantic chunks, preserve tables as Markdown, verbalize document-embedded figures, store extracted images in an asset-store blob container, and add `image_path` references to the generated index. During answer synthesis, Azure AI Search sends matching images to the multimodal model. The retrieve response reports aggregate image-serving activity but doesn't define dedicated fields for the individual asset-store image paths or image bytes sent to the model. Separately, the notebook queries the generated index for an `image_path` and downloads that indexed asset to validate application access.\n", "\n", @@ -27,9 +27,7 @@ "1. Download the independently selected indexed image asset with `DefaultAzureCredential`.\n", "1. Delete the knowledge base and knowledge source when you're finished.\n", "\n", - "**SDK:** `azure-search-documents==12.1.0b1` (REST API `2026-05-01-preview`)\n", - "\n", - "Save `sample.env` as `.env`, fill in the nonsecret resource values, and create a virtual environment from `requirements.txt` before running this notebook.\n" + "**SDK:** `azure-search-documents==12.1.0b1` (REST API `2026-05-01-preview`)" ] }, { From a5caa095aad790ec70aa1d787e438a407259179e Mon Sep 17 00:00:00 2001 From: gmndrg Date: Thu, 20 Aug 2026 20:34:54 -0600 Subject: [PATCH 4/4] Address image serving review feedback --- image-serving-example/image_serving.ipynb | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/image-serving-example/image_serving.ipynb b/image-serving-example/image_serving.ipynb index 2492fe1f2..8b987308d 100644 --- a/image-serving-example/image_serving.ipynb +++ b/image-serving-example/image_serving.ipynb @@ -5,17 +5,18 @@ "id": "92ebfca4", "metadata": {}, "source": [ - "# Example: Image serving for agentic retrieval using Python\n", + "# Example: Image serving for agentic retrieval (preview) using Python\n", "\n", - "This notebook creates an Azure Blob-backed agentic retrieval pipeline in Azure AI Search with image serving enabled. Managed ingestion uses Azure Content Understanding in Foundry Tools to create semantic chunks, preserve tables as Markdown, verbalize document-embedded figures, store extracted images in an asset-store blob container, and add `image_path` references to the generated index. During answer synthesis, Azure AI Search sends matching images to the multimodal model. The retrieve response reports aggregate image-serving activity but doesn't define dedicated fields for the individual asset-store image paths or image bytes sent to the model. Separately, the notebook queries the generated index for an `image_path` and downloads that indexed asset to validate application access.\n", + "This end-to-end notebook creates a blob knowledge source that uses managed ingestion with Azure Content Understanding in Foundry Tools to create semantic chunks, preserve tables, describe document-embedded figures, and extract images. Azure AI Search stores extracted images in an asset store, stores `image_path` references in the generated index, and supplies image content associated with matching results to the multimodal model during answer synthesis.\n", + "\n", + "The retrieve response reports aggregate image-serving statistics, but it doesn't guarantee an extracted-image `image_path` or image bytes. Separately from retrieval, the notebook runs an ordinary wildcard search against the generated index to select an indexed `image_path`, and then downloads that asset to validate application access. The selected path isn't demonstrably associated with a chunk that contributed to the retrieve response.\n", + "\n", + "This sample doesn't use an explicit `OcrSkill` or `normalized_images`. Those elements belong to the classic OCR enrichment pattern.\n", "\n", "**Prerequisites:**\n", "\n", - "1. Complete the permissions and resource setup in the [image serving documentation](https://learn.microsoft.com/azure/search/agentic-retrieval-how-to-image-serving).\n", - "1. Create a Microsoft Foundry resource in a [region supported by Content Understanding](https://learn.microsoft.com/azure/ai-services/content-understanding/language-region-support), with Azure OpenAI embedding and multimodal chat model deployments. Use the resource endpoint in the `https://.services.ai.azure.com` format.\n", - "1. Assign the search service managed identity **Storage Blob Data Contributor** on the source and asset containers, and **Cognitive Services User** on the Foundry resource.\n", - "1. Assign the identity that runs this notebook **Search Service Contributor**, **Search Index Data Reader**, and **Storage Blob Data Reader** for the asset container.\n", - "1. Upload a PDF with embedded figures or another supported document type to the source container.\n", + "- A Python virtual environment created from `requirements.txt` and selected as the notebook kernel.\n", + "- For required resources and permissions, see [Surface document-embedded images in agentic retrieval (preview)](https://learn.microsoft.com/azure/search/agentic-retrieval-how-to-image-serving#prerequisites).\n", "\n", "**Flow:**\n", "\n", @@ -27,7 +28,9 @@ "1. Download the independently selected indexed image asset with `DefaultAzureCredential`.\n", "1. Delete the knowledge base and knowledge source when you're finished.\n", "\n", - "**SDK:** `azure-search-documents==12.1.0b1` (REST API `2026-05-01-preview`)" + "**SDK:** `azure-search-documents==12.1.0b1` (REST API `2026-05-01-preview`)\n", + "\n", + "Save `sample.env` as `.env` and fill in the nonsecret resource values before running this notebook.\n" ] }, { @@ -205,7 +208,7 @@ " \"Managed ingestion has failed items.\\n\" + \"\\n\".join(messages)\n", " )\n", " completed = status.last_synchronization_state\n", - " if completed is not None:\n", + " if current is None and completed is not None:\n", " if completed.items_updates_failed:\n", " raise RuntimeError(\n", " \"Managed ingestion completed with \"\n",