From 2df465a2ac38eb179469a557b88e7474edd0391f Mon Sep 17 00:00:00 2001 From: Theresa Iserman <98428787+theresa-i@users.noreply.github.com> Date: Tue, 23 Jun 2026 10:46:08 -0500 Subject: [PATCH 1/4] Migrate AI-05 lab from Azure OpenAI to Microsoft Foundry Changes from main: - Updated lab title and description to reference Microsoft Foundry - Replaced 'Provision an Azure OpenAI resource' section with 'Create a Microsoft Foundry resource and project', using the new Foundry resource creation portal link and Entra ID (access token) auth instead of API keys - Added step to create a project within the Foundry resource after deployment - Added Cloud Shell step to obtain a short-lived access token via `az account get-access-token --resource https://cognitiveservices.azure.com` - Narrowed region list to North Central US and Sweden Central (quota-safe) - Updated model deployment target from Azure OpenAI service to Microsoft Foundry deployments page, deploying gpt-4.1 (model version 2025-04-14) - Updated primarytopics metadata: replaced 'Azure OpenAI' with 'Microsoft Foundry' - Preserved all Databricks cluster, notebook, MLflow evaluation, and prompt-improvement content unchanged" --- .../Exercises/AI-05-Evaluate-models.md | 145 +++++++++++------- 1 file changed, 93 insertions(+), 52 deletions(-) diff --git a/Instructions/Exercises/AI-05-Evaluate-models.md b/Instructions/Exercises/AI-05-Evaluate-models.md index 3852438..628ccb7 100644 --- a/Instructions/Exercises/AI-05-Evaluate-models.md +++ b/Instructions/Exercises/AI-05-Evaluate-models.md @@ -1,6 +1,6 @@ --- lab: - title: Evaluate Large Language Models using Azure Databricks and Azure OpenAI + title: Evaluate Large Language Models using Azure Databricks and Microsoft Foundry description: You'll gain hands-on experience evaluating LLM outputs using MLflow's evaluation framework by defining custom evaluation criteria (guidelines and safety scorers) and running systematic evaluations against test datasets. You'll learn how to iteratively improve model performance by analyzing evaluation results, refining system prompts based on specific criteria like creativity and safety, and comparing evaluation runs to validate improvements. duration: 30 minutes level: 400 @@ -11,113 +11,154 @@ lab: - Microsoft Foundry --- -# Evaluate Large Language Models using Azure Databricks and Azure OpenAI +# Evaluate Large Language Models using Azure Databricks and Microsoft Foundry -Evaluating large language models (LLMs) involves a series of steps to ensure the model's performance meets the required standards. MLflow LLM Evaluate, a feature within Azure Databricks, provides a structured approach to this process, including setting up the environment, defining evaluation metrics, and analyzing results. This evaluation is crucial as LLMs often do not have a single ground truth for comparison, making traditional evaluation methods inadequate. +Unlike traditional software testing, evaluating large language models (LLMs) is challenging because there's rarely a single correct answer. A response might be technically accurate but unsafe, off-topic, or fail to meet your specific quality criteria. You need a systematic way to score outputs against criteria that matter to your use case. + +In this lab, you build a sentence-completion game app powered by gpt-4o, then evaluate its outputs using **MLflow's `genai.evaluate()` framework** in Azure Databricks. You define custom **Guidelines scorers** (checking that responses are funny, child-safe, and follow the template structure) alongside a built-in **Safety scorer**, and run them against a test dataset. After reviewing the results in the MLflow Experiment UI, you refine the system prompt and re-run the evaluation to verify the improvement. This lab will take approximately **30** minutes to complete. -> **Note**: The Azure Databricks user interface is subject to continual improvement. The user interface may have changed since the instructions in this exercise were written. +> **Note**: Azure user interfaces are subject to continual improvement. The user interface may have changed since the instructions in this exercise were written. ## Before you start You'll need an [Azure subscription](https://azure.microsoft.com/free) in which you have administrative-level access. -## Provision an Azure OpenAI resource +## Create a Microsoft Foundry resource and project + +If you don't already have one, create a Microsoft Foundry resource and project in your Azure subscription. -If you don't already have one, provision an Azure OpenAI resource in your Azure subscription. +> **Note**: Creating a Foundry resource only requires a subscription, resource group, region, and name. No Key Vault or Application Insights resources are needed. 1. Sign into the **Azure portal** at `https://portal.azure.com`. -2. Create an **Azure OpenAI** resource with the following settings: - - **Subscription**: *Select an Azure subscription that has been approved for access to the Azure OpenAI service* +2. Use the following link to open the Foundry resource creation page: `https://portal.azure.com/#create/Microsoft.CognitiveServicesAIFoundry` +3. On the **Create** page, provide the following information on the **Basics** tab: + - **Subscription**: *Select your Azure subscription* - **Resource group**: *Choose or create a resource group* - **Region**: *Make a **random** choice from any of the following regions*\* - - East US 2 - North Central US - Sweden Central - - Switzerland West - **Name**: *A unique name of your choice* - - **Pricing tier**: Standard S0 +4. Select **Review + create**, then select **Create** and wait for deployment to complete. + +> \* Foundry resources are constrained by regional quotas. The listed regions include default quota for the model type(s) used in this exercise. Randomly choosing a region reduces the risk of a single region reaching its quota limit in scenarios where you are sharing a subscription with other users. In the event of a quota limit being reached later in the exercise, there's a possibility you may need to create another resource in a different region. -> \* Azure OpenAI resources are constrained by regional quotas. The listed regions include default quota for the model type(s) used in this exercise. Randomly choosing a region reduces the risk of a single region reaching its quota limit in scenarios where you are sharing a subscription with other users. In the event of a quota limit being reached later in the exercise, there's a possibility you may need to create another resource in a different region. +5. Once deployment completes, go to the deployed resource. In the left pane, under **Resource Management**, select **Keys and Endpoint**, then copy the **Endpoint** — you will use it later in this exercise. -3. Wait for deployment to complete. Then go to the deployed Azure OpenAI resource in the Azure portal. +6. In the **Overview** page, select **Go to Microsoft Foundry** to open your resource in the Foundry portal (or navigate directly to `https://ai.azure.com`). -4. In the left pane, under **Resource Management**, select **Keys and Endpoint**. +7. In **Microsoft Foundry**, create a new **project** within your Foundry resource: + - Select the project name in the upper-left corner, then select **Create new project**. + - Enter a **Project name** and select **Create project**. + - Wait for the project to be created. + +8. Launch Cloud Shell and run the following command to get a temporary authorization token for API calls. Keep it together with the endpoint copied previously. + + ```bash + az account get-access-token --resource https://cognitiveservices.azure.com + ``` -5. Copy the endpoint and one of the available keys as you will use it later in this exercise. + >**Note**: You only need to copy the `accessToken` field value and **not** the entire JSON output. ## Deploy the required model -Azure provides a web-based portal named **Microsoft Foundry**, that you can use to deploy, manage, and explore models. You'll start your exploration of Azure OpenAI by using Microsoft Foundry to deploy a model. +Microsoft Foundry allows you to deploy, manage, and explore models. > **Note**: As you use Microsoft Foundry, message boxes suggesting tasks for you to perform may be displayed. You can close these and follow the steps in this exercise. -1. In the Azure portal, on the **Overview** page for your Azure OpenAI resource, scroll down to the **Get Started** section and select the button to go to **Microsoft Foundry**. - -1. In Microsoft Foundry, in the pane on the left, select the **Deployments** page and view your existing model deployments. If you don't already have one, create a new deployment of the **gpt-4o** model with the following settings: - - **Deployment name**: *gpt-4o* +1. In **Microsoft Foundry**, on the home page select **View deployments** (or select **Build** in the top navigation bar, then select **Deployments**). + +1. Select **Deploy** > **Deploy a base model**, search for and select **gpt-4.1**, then select **Deploy** > **Custom settings** to configure the deployment with the following settings: + - **Deployment name**: *gpt-4.1* - **Deployment type**: Standard - - **Model version**: *Use default version* - - **Tokens per minute rate limit**: 10K\* - - **Content filter**: Default + - **Model version**: *2025-04-14* + - **Model version upgrade policy**: Upgrade once new default version becomes available - **Enable dynamic quota**: Disabled - + - **Tokens per minute rate limit**: 10K\* + - **Guardrails**: DefaultV2 + > \* A rate limit of 10,000 tokens per minute is more than adequate to complete this exercise while leaving capacity for other people using the same subscription. +2. Wait for the deployment to complete. + +Now that your model is deployed, you'll use Azure Databricks to write Python code that calls it and evaluates its outputs using MLflow's `genai.evaluate()` framework. + ## Provision an Azure Databricks workspace > **Tip**: If you already have an Azure Databricks workspace, you can skip this procedure and use your existing workspace. -1. Sign into the **Azure portal** at `https://portal.azure.com`. -2. Create an **Azure Databricks** resource with the following settings: - - **Subscription**: *Select the same Azure subscription that you used to create your Azure OpenAI resource* - - **Resource group**: *The same resource group where you created your Azure OpenAI resource* - - **Region**: *The same region where you created your Azure OpenAI resource* - - **Name**: *A unique name of your choice* - - **Pricing tier**: *Premium* or *Trial* +1. In a new browser tab, return to the Azure portal at `https://portal.azure.com`. Search for **Azure Databricks** in the search bar, then select **Create** to create an **Azure Databricks** resource with the following settings: + - **Subscription**: *Select the same Azure subscription that you used to create your Foundry resource* + - **Resource group**: *The same resource group where you created your Foundry resource* + - **Workspace name**: *A unique name of your choice* + - **Region**: *Select any available region* + - **Pricing tier**: Premium (+ Role-based access controls) + - **Workspace type**: Hybrid + - **Managed Resource Group name**: *Leave blank* -3. Select **Review + create** and wait for deployment to complete. Then go to the resource and launch the workspace. + > **Note**: Azure Databricks does not need to be in the same region as your Foundry resource. If cluster creation fails due to quota limits in your chosen region, try deleting the workspace and creating a new one in a different region. -## Create a notebook and install required libraries +2. Select **Review + create**, and once validation succeeds, select **Create**. + +3. When deployment is complete, select **Go to resource**, then select **Launch Workspace** to open your Azure Databricks workspace in a new browser tab. -1. In the Azure portal, browse to the resource group where the Azure Databricks workspace was created. +## Create a cluster -1. Select your Azure Databricks Service resource. +Azure Databricks is a distributed processing platform that uses Apache Spark *clusters* to process data in parallel on multiple nodes. Each cluster consists of a driver node to coordinate the work, and worker nodes to perform processing tasks. In this exercise, you'll create a *single-node* cluster to minimize the compute resources used in the lab environment (in which resources may be constrained). In a production environment, you'd typically create a cluster with multiple worker nodes. -1. In the **Overview** page for your workspace, use the **Launch Workspace** button to open your Azure Databricks workspace in a new browser tab; signing in if prompted. +> **Tip**: If you already have a cluster with a 17.3 LTS **ML** or higher runtime version in your Azure Databricks workspace, you can use it to complete this exercise and skip this procedure. - > **Tip**: As you use the Databricks Workspace portal, various tips and notifications may be displayed. Dismiss these and follow the instructions provided to complete the tasks in this exercise. +1. In the sidebar on the left, select the **(+) New** task, select **More**, and then select **Cluster**. +1. In the **New Cluster** page, create a new cluster with the following settings: + - **Cluster name**: *User Name's* cluster (the default cluster name) + - **Policy**: Unrestricted + - **Machine learning**: Enabled + - **Databricks runtime**: 17.3 LTS + - **Use Photon Acceleration**: Unselected + - **Worker type**: Standard_D4ds_v5 + - **Single node**: Checked + - **Terminate after**: 30 minutes of inactivity +1. Select **Create** -1. In the Databricks workspace, go to the **Workspace** section. +2. Wait for the cluster to be created. It may take a minute or two. -1. Select **Create** and then select **Notebook**. +> **Note**: If your cluster fails to start, your subscription may have insufficient quota in the region where your Azure Databricks workspace is provisioned. See [CPU core limit prevents cluster creation](https://docs.microsoft.com/azure/databricks/kb/clusters/azure-core-limit) for details. If this happens, you can try deleting your workspace and creating a new one in a different region. -1. Name your notebook and select `Python` as the language. Select **Serverless** as the default compute. +## Create a notebook and install required libraries + +1. In the sidebar on the left, use the **(+) New** link to create a **Notebook**. + +1. Name your notebook and select `Python` as the language. In the **Connect** drop-down list, select your cluster if it is not already selected. If the cluster is not running, it may take a minute or so to start. 1. In the first code cell, enter and run the following code to install the necessary libraries: ```python - %pip install --upgrade "mlflow[databricks]>=3.1.0" openai "databricks-connect>=16.1" + %pip install --upgrade "mlflow[databricks]>=3.1.0" openai azure-identity "databricks-connect>=16.1" dbutils.library.restartPython() ``` -1. In a new cell, define the authentication parameters that will be used to initialize the OpenAI models, replacing `your_openai_endpoint` and `your_openai_api_key` with the endpoint and key copied earlier from your OpenAI resource: + > **Note**: You may see warnings that package versions are not pinned, or that core Python package versions changed. These are advisory only and won't affect the lab — the `dbutils.library.restartPython()` command restarts the Python environment to apply the updates. + +1. In a new cell, run the following code with the access information you copied earlier to assign persistent environment variables for authentication: ```python import os - os.environ["AZURE_OPENAI_API_KEY"] = "your_openai_api_key" - os.environ["AZURE_OPENAI_ENDPOINT"] = "your_openai_endpoint" - os.environ["AZURE_OPENAI_API_VERSION"] = "your_model_version_example: '2023-03-15-preview'" + os.environ["AZURE_OPENAI_ENDPOINT"] = "your_foundry_endpoint" + os.environ["COGNITIVE_SERVICES_TOKEN"] = "your_cognitiveservices_access_token" # from: az account get-access-token --resource https://cognitiveservices.azure.com ``` + > **Note**: The access token expires after approximately 60 minutes. If you encounter authentication errors during the lab, re-run the Cloud Shell command and update this cell. + ## Evaluate LLM with a custom function In MLflow 3 and above, `mlflow.genai.evaluate()` supports evaluating a Python function without requiring the model be logged to MLflow. The process involves specifying the model to evaluate, the metrics to compute, and the evaluation data. 1. In a new cell, run the following code to connect to your deployed LLM, define the custom function that will be used to evaluate your model, create a sample template for the app and test it: + > **Important**: When running code in Azure Databricks, `DefaultAzureCredential` authenticates as the **Databricks workspace managed identity**, not your signed-in user account. The code below uses the token you obtained via Cloud Shell, which authenticates as your own identity and avoids this issue. + ```python import json import os @@ -127,11 +168,11 @@ In MLflow 3 and above, `mlflow.genai.evaluate()` supports evaluating a Python fu # Enable automatic tracing mlflow.openai.autolog() - # Connect to a Databricks LLM using your AzureOpenAI credentials + # Connect to your deployed model using the token obtained from Cloud Shell client = AzureOpenAI( azure_endpoint = os.getenv("AZURE_OPENAI_ENDPOINT"), - api_key = os.getenv("AZURE_OPENAI_API_KEY"), - api_version = os.getenv("AZURE_OPENAI_API_VERSION") + azure_ad_token = os.getenv("COGNITIVE_SERVICES_TOKEN"), + api_version = "2024-08-01-preview" ) # Basic system prompt @@ -142,7 +183,7 @@ In MLflow 3 and above, `mlflow.genai.evaluate()` supports evaluating a Python fu """Complete a sentence template using an LLM.""" response = client.chat.completions.create( - model="gpt-4o", + model="gpt-4.1", messages=[ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": template}, @@ -174,7 +215,7 @@ In MLflow 3 and above, `mlflow.genai.evaluate()` supports evaluating a Python fu }, { "inputs": { - "template": "The ____ (adjective) ____ (job) shouted, “____ (exclamation)!” and ran toward the ____ (place)" + "template": "The ____ (adjective) ____ (job) shouted, "____ (exclamation)!" and ran toward the ____ (place)" } }, { @@ -207,7 +248,7 @@ In MLflow 3 and above, `mlflow.genai.evaluate()` supports evaluating a Python fu name="funny" ), Guidelines( - guidelines="Response must be appropiate for children", + guidelines="Response must be appropriate for children", name="child_safe" ), Guidelines( @@ -272,6 +313,6 @@ You can compare both runs in the Experiment UI and confirm that the revised prom ## Clean up -When you're done with your Azure OpenAI resource, remember to delete the deployment or the entire resource in the **Azure portal** at `https://portal.azure.com`. +When you're done, remember to delete the deployment or the entire Microsoft Foundry project in **Microsoft Foundry** at `https://ai.azure.com`. If you've finished exploring Azure Databricks, you can delete the resources you've created to avoid unnecessary Azure costs and free up capacity in your subscription. From e3d1e0a316d0508bc32c66ef715dd5e90a87cbaa Mon Sep 17 00:00:00 2001 From: Theresa Iserman <98428787+theresa-i@users.noreply.github.com> Date: Wed, 24 Jun 2026 15:40:09 -0500 Subject: [PATCH 2/4] Fix Cloud Shell step: specify Azure portal, improve token save instruction --- Instructions/Exercises/AI-05-Evaluate-models.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Instructions/Exercises/AI-05-Evaluate-models.md b/Instructions/Exercises/AI-05-Evaluate-models.md index 628ccb7..e398588 100644 --- a/Instructions/Exercises/AI-05-Evaluate-models.md +++ b/Instructions/Exercises/AI-05-Evaluate-models.md @@ -53,7 +53,7 @@ If you don't already have one, create a Microsoft Foundry resource and project i - Enter a **Project name** and select **Create project**. - Wait for the project to be created. -8. Launch Cloud Shell and run the following command to get a temporary authorization token for API calls. Keep it together with the endpoint copied previously. +8. In a new browser tab, return to the **Azure portal** at `https://portal.azure.com` and launch Cloud Shell. Run the following command to get a temporary authorization token for API calls. Copy the `accessToken` value and save it alongside the endpoint you copied in step 5. ```bash az account get-access-token --resource https://cognitiveservices.azure.com @@ -215,7 +215,7 @@ In MLflow 3 and above, `mlflow.genai.evaluate()` supports evaluating a Python fu }, { "inputs": { - "template": "The ____ (adjective) ____ (job) shouted, "____ (exclamation)!" and ran toward the ____ (place)" + "template": "The ____ (adjective) ____ (job) shouted, \"____ (exclamation)!\" and ran toward the ____ (place)" } }, { From 6d294dfecc47ab16573832bebfa72a2d5e4f8906 Mon Sep 17 00:00:00 2001 From: Theresa Iserman <98428787+theresa-i@users.noreply.github.com> Date: Wed, 24 Jun 2026 15:49:12 -0500 Subject: [PATCH 3/4] Add explicit Deploy button instruction after deployment settings list --- Instructions/Exercises/AI-05-Evaluate-models.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Instructions/Exercises/AI-05-Evaluate-models.md b/Instructions/Exercises/AI-05-Evaluate-models.md index e398588..9150e53 100644 --- a/Instructions/Exercises/AI-05-Evaluate-models.md +++ b/Instructions/Exercises/AI-05-Evaluate-models.md @@ -78,6 +78,8 @@ Microsoft Foundry allows you to deploy, manage, and explore models. - **Tokens per minute rate limit**: 10K\* - **Guardrails**: DefaultV2 + Then select **Deploy** at the bottom of the page. + > \* A rate limit of 10,000 tokens per minute is more than adequate to complete this exercise while leaving capacity for other people using the same subscription. 2. Wait for the deployment to complete. From 70d82418460692c5cc4911b53d77848b882bbd5a Mon Sep 17 00:00:00 2001 From: Theresa Iserman Date: Thu, 25 Jun 2026 18:01:52 -0400 Subject: [PATCH 4/4] Fix api_version to 2025-04-01-preview; use single-quote string to avoid confusing backslash escaping in template --- Instructions/Exercises/AI-05-Evaluate-models.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Instructions/Exercises/AI-05-Evaluate-models.md b/Instructions/Exercises/AI-05-Evaluate-models.md index 9150e53..e320b8c 100644 --- a/Instructions/Exercises/AI-05-Evaluate-models.md +++ b/Instructions/Exercises/AI-05-Evaluate-models.md @@ -174,7 +174,7 @@ In MLflow 3 and above, `mlflow.genai.evaluate()` supports evaluating a Python fu client = AzureOpenAI( azure_endpoint = os.getenv("AZURE_OPENAI_ENDPOINT"), azure_ad_token = os.getenv("COGNITIVE_SERVICES_TOKEN"), - api_version = "2024-08-01-preview" + api_version = "2025-04-01-preview" ) # Basic system prompt @@ -217,7 +217,7 @@ In MLflow 3 and above, `mlflow.genai.evaluate()` supports evaluating a Python fu }, { "inputs": { - "template": "The ____ (adjective) ____ (job) shouted, \"____ (exclamation)!\" and ran toward the ____ (place)" + "template": 'The ____ (adjective) ____ (job) shouted, "____ (exclamation)!" and ran toward the ____ (place)' } }, {