Azure AI Content Understanding for Python: Setup, Usage & Best Practices

Azure AI Content Understanding provides multimodal semantic extraction from documents, images, audio, and video files through a Python SDK. This skill transforms unstructured media into structured, searchable content—markdown text, transcripts with timestamps, key frames, tables, and figures—optimized for RAG systems and automated workflows without manual preprocessing.

What This Skill Does

This SDK wraps Azure's Content Understanding service, which analyzes files and extracts semantic content suitable for downstream AI applications. Unlike simple OCR or transcription, it understands document structure (headings, paragraphs, tables, figures), video narrative flow (scenes, key frames, speaker changes), and semantic relationships between multimodal elements. The service returns everything as markdown or structured objects, making it trivial to index for vector search or feed into language models.

The API uses long-running operations: you submit analysis requests via begin_analyze(), which returns a poller that tracks progress. For documents and images, analysis completes in seconds to minutes. For audio and video, expect minutes to tens of minutes depending on file length. Results include contents (the primary output—markdown, transcript phrases, key frames) and optional fields when using custom analyzers with field schemas.

Prebuilt analyzers handle common scenarios: prebuilt-documentSearch for PDFs and Office docs, prebuilt-imageSearch for photos and diagrams, prebuilt-audioSearch for speech-to-text, and prebuilt-videoSearch for comprehensive video analysis. Custom analyzers let you define field schemas for domain-specific extraction—invoice line items, contract clauses, medical records—using the prebuilt analyzers as base models.

Getting Started

Install the SDK via pip:

pip install azure-ai-contentunderstanding

Configure environment variable:

export CONTENTUNDERSTANDING_ENDPOINT=https://<resource>.cognitiveservices.azure.com/

Create a client with DefaultAzureCredential:

import os
from azure.ai.contentunderstanding import ContentUnderstandingClient
from azure.identity import DefaultAzureCredential

endpoint = os.environ["CONTENTUNDERSTANDING_ENDPOINT"]
client = ContentUnderstandingClient(
    endpoint=endpoint,
    credential=DefaultAzureCredential()
)

This uses Azure identity for authentication—supports managed identities, Azure CLI login, and service principals.

Key Features

Multimodal Analysis: Process documents (PDF, Word, PowerPoint), images (JPEG, PNG), audio (MP3, WAV), and video (MP4, MOV) through a unified API.

Semantic Extraction: Get markdown-formatted content preserving structure. Headings become # headers, tables convert to markdown tables, lists maintain hierarchy.

Transcript with Timing: Audio and video analysis returns transcript phrases with start/end timestamps, enabling time-linked search and playback jump-to-timestamp features.

Key Frame Detection: Video analysis identifies significant frames with descriptions, useful for thumbnail generation, scene detection, or visual search.

Prebuilt Analyzers: Ready-to-use models for documents, images, audio, and video. No training required—just submit files and get results.

Custom Analyzers: Define field schemas to extract structured data (invoice totals, contract dates, medical codes) using prebuilt analyzers as base models.

Async Support: Full async client implementation (ContentUnderstandingClient from azure.ai.contentunderstanding.aio) for high-throughput scenarios.

Usage Examples

Analyze Document from URL:

from azure.ai.contentunderstanding.models import AnalyzeInput

poller = client.begin_analyze(
    analyzer_id="prebuilt-documentSearch",
    inputs=[AnalyzeInput(url="https://example.com/document.pdf")]
)
result = poller.result()

# Get markdown content
content = result.contents[0]
print(content.markdown)

Analyze Image:

poller = client.begin_analyze(
    analyzer_id="prebuilt-imageSearch",
    inputs=[AnalyzeInput(url="https://example.com/diagram.jpg")]
)
result = poller.result()
content = result.contents[0]
print(content.markdown)  # Extracted text and descriptions

Analyze Video with Transcript and Key Frames:

poller = client.begin_analyze(
    analyzer_id="prebuilt-videoSearch",
    inputs=[AnalyzeInput(url="https://example.com/video.mp4")]
)
result = poller.result()

content = result.contents[0]

# Print transcript with timestamps
for phrase in content.transcript_phrases:
    print(f"[{phrase.start_time} - {phrase.end_time}]: {phrase.text}")

# Print key frames
for frame in content.key_frames:
    print(f"Frame at {frame.time}: {frame.description}")

Analyze Audio for Transcription:

poller = client.begin_analyze(
    analyzer_id="prebuilt-audioSearch",
    inputs=[AnalyzeInput(url="https://example.com/podcast.mp3")]
)
result = poller.result()

content = result.contents[0]
for phrase in content.transcript_phrases:
    print(f"[{phrase.start_time}] {phrase.text}")

Create Custom Analyzer for Invoices:

# Define field schema
analyzer = client.create_analyzer(
    analyzer_id="invoice-extractor",
    analyzer={
        "description": "Extract invoice fields",
        "base_analyzer_id": "prebuilt-documentSearch",
        "field_schema": {
            "fields": {
                "vendor_name": {"type": "string"},
                "invoice_total": {"type": "number"},
                "line_items": {
                    "type": "array",
                    "items": {
                        "type": "object",
                        "properties": {
                            "description": {"type": "string"},
                            "amount": {"type": "number"}
                        }
                    }
                }
            }
        }
    }
)

# Use custom analyzer
poller = client.begin_analyze(
    analyzer_id="invoice-extractor",
    inputs=[AnalyzeInput(url="https://example.com/invoice.pdf")]
)
result = poller.result()

# Access structured fields
print(result.fields["vendor_name"])
print(result.fields["invoice_total"])

Async Client for High Throughput:

import asyncio
from azure.ai.contentunderstanding.aio import ContentUnderstandingClient
from azure.identity.aio import DefaultAzureCredential

async def analyze_documents():
    async with DefaultAzureCredential() as credential:
        async with ContentUnderstandingClient(endpoint, credential) as client:
            poller = await client.begin_analyze(
                analyzer_id="prebuilt-documentSearch",
                inputs=[AnalyzeInput(url="https://example.com/doc.pdf")]
            )
            result = await poller.result()
            return result.contents[0].markdown

asyncio.run(analyze_documents())

Best Practices

Use URL Sources When Possible: Uploading large files adds latency and bandwidth costs. If files are in Azure Blob Storage or publicly accessible, use URLs.

Poll Appropriately: Video and audio analysis can take minutes. Don't block user interfaces—run analysis in background workers and notify users when complete.

Access Results via contents[0]: Results are returned as lists even for single files. Always index with [0] for single-file analysis.

Use Prebuilt Analyzers First: Only create custom analyzers when you need structured field extraction. Prebuilt analyzers are faster and require no setup.

Handle Long-Running Operations: Use pollers (begin_analyze()) correctly. Call .result() to wait for completion, or .done() to check status without blocking.

Leverage Async for Batch Processing: If analyzing hundreds of files, use the async client with asyncio.gather() for concurrent processing.

Store Analyzer IDs: Custom analyzers persist in the service. Store IDs in your configuration to avoid recreating them on every deployment.

When to Use / When NOT to Use

Use this skill when:

• You're building RAG systems needing structured document indexing


• You need to extract content from mixed media (PDFs, videos, audio)


• You want transcripts with timestamps for audio/video search


• You're indexing large document repositories for semantic search


• You need automated content extraction without manual preprocessing


• You're building document summarization or Q&A systems


• You need key frame extraction from videos

Avoid this skill when:

• You only need simple OCR (use Azure AI Vision instead)


• You're processing real-time streaming media (this is for files)


• You need sub-second latency (long-running operations)


• You're working with specialized document types requiring custom models


• Your files are extremely large (>2GB)—consider chunking first


• You need on-premise processing without cloud dependencies

Related Skills

• azure-ai-document-intelligence-dotnet: Specialized document intelligence for forms and invoices
• azure-ai-vision-imageanalysis-py: General computer vision for images
• azure-ai-transcription-py: Batch transcription for audio files

Source

Maintained by Microsoft. View on GitHub