DRAG DROP (Drag and Drop is not supported)A team operates a generative AI-powered customer support assistant built on Microsoft Foundry. The application serves users globally and supports both real-time chat interactions and batch summarization jobs.The team must ensure that the application continues to meet defined service-level objectives (SLO) as usage increases.The team requires visibility into runtime behavior to identify performance regressions that affect the user experience and system capacity.You need to select the performance metrics that meet the requirements.Which performance metric should you monitor for each requirement? To answer, move the appropriate performance metrics to the correct requirements. You may use each performance metric once, more than once, or not at all. You may need to move the split bar between panes or scroll to view content.Note: Each correct selection is worth one point.Select and Place:
Answer(s): A
Box 1: Response time Detect when the system is slow as returning output.Response time is the best single metric to detect when the system is slow at returning output to the user.Response time measures the total duration from the moment a user sends a request until they receive the final output. Because this specific application handles real-time chat interactions, response time directly captures the exact slowness a user experiences. For generative AI applications, this often tracks critical user-centric metrics like Time to First Token (TTFT) and overall end-to-end generation time.Incorrect: Latency focuses strictly on the time spent on server-side processing or network transit. It often excludes components of the user experience, such as data rendering time or client-side delays, making it less comprehensive than response time for measuring end-to-end user-perceived slowness.Throughput measures the volume of data or number of requests processed over a specific period (e.g., tokens per second or requests per minute). While throughput is crucial for evaluating the capacity of your batch summarization jobs, a system can have high throughput while individual real-time users still suffer from slow, high-response-time delays.Box 2: Concurrency Identify when the system cannot sustain expect request volume.When a system cannot sustain expected request volumes, Concurrency acts as the single best leading metric to identify infrastructure bottlenecks.Active Load Exposure: Concurrency tracks the exact number of active, parallel requests currently being processed by the system at any single moment.Resource Exhaustion Warning: In generative AI applications, real-time chat interactions hold long-lived connections, while batch summarization jobs consume massive computational threads. If concurrency spikes or plateaus at your system's maximum thread/hosting limit, it serves as a direct indicator that the application cannot handle more parallel volume and will begin queueing or dropping incoming requests.Box 3: Latency Measure the duration of handling a request.Latency focuses strictly on the time spent on server-side processing or network transit. It often excludes components of the user experience, such as data rendering time or client-side delays, making it less comprehensive than response time for measuring end-to-end user-perceived slowness.
https://demiliani.com/2025/11/17/evaluating-llms-performances-for-your-ai-task-presenting-my-llmevaluator/ https://learn.microsoft.com/en-us/azure/foundry/openai/monitor-openai-reference
DRAG DROP (Drag and Drop is not supported)A team deploys a classification model to production and scores incoming customer data daily.After several weeks, business stakeholders report unexpected changes in prediction behavior, even though the endpoint remains healthy.You need to determine whether data drift is occurring and if it is, identify the appropriate actions.Which action should you perform for each observed signal? To answer, move the appropriate actions to the correct observed signals. You may use each action once, more than once, or not at all. You may need to move the split bar between panes or scroll to view content.Note: Each correct selection is worth one point.Select and Place:
Box 1: Analyze dataset drift metrics Input feature distributions differ from training data.The direct action you must take when input feature distributions differ from the training data is to retrain the model using the updated data distribution.This scenario describes a classic case of data drift (also known as covariate shift). While the endpoint infrastructure remains perfectly healthy, the statistical properties of the production data have shifted away from the baseline training data, leading to a degradation in predictive behavior.Box 2: Analyze dataset drift metrics Model accuracy drops without code changes.You should analyze dataset drift metrics to identify why the model accuracy dropped without any changes to the code.In production machine learning, dataset drift occurs when the statistical properties of the input data change over time. Even if your Azure Machine Learning endpoint remains perfectly healthy and functional, changes in the real-world data distribution will cause the model's predictive performance to degrade. This phenomenon is often referred to as data drift or concept drift.Box 3: Investigate scaling and infrastructure metrics Endpoint latency increases under load.You should immediately investigate infrastructure metrics, as latency spikes under load typically indicate resource exhaustion.
https://becominghuman.ai/microsoft-azure-machine-learning-x-udacity-lesson-3-notes-8498661210c4 https://www.evidentlyai.com/ml-in-production/data-drift
HOTSPOT (Drag and Drop is not supported)A team manages an Azure Machine Learning workspace to train and register machine learning models.Previous model versions must be retained for audit and rollback purposes but must not be used for new deployments.You need to manage model versions.What should you do for each requirement? To answer, select the appropriate options in the answer area.Note: Each correct selection is worth one point.Hot Area:
Box 1: Archive unused model versions Preserve Model history for audit purposes.Archiving unused model versions is highly appropriate and recommended for this scenario.Archiving hides unused versions from standard queries and clean user interfaces, while fully preserving the underlying artifacts and metadata required for compliance audits and production rollbacks.Preserves Audit Trails: Archiving does not delete the model. It perfectly retains the model files, lineage, run histories, and regulatory logs required for compliance audits.Enables Seamless Rollbacks: An archived model version remains active in your workflows. If a rollback becomes necessary, you can directly reference or restore the archived model to standard views via the AzureML CLI.Declutters the Registry: Standard commands like az ml model list ignore archived components. This simplifies your view to display only active models without disrupting underlying production pipelines.Box 2: Archive unused model versions Prevent outdate models from being deployed.Archiving hides unused versions from standard queries and clean user interfaces, while fully preserving the underlying artifacts and metadata required for compliance audits and production rollbacks.
https://docs.azure.cn/en-us/machine-learning/concept-model-management-and-deployment
HOTSPOT (Drag and Drop is not supported)You use Azure Machine Learning to implement hyperparameter tuning for an Azure ML Python SDK v2-based model training.Training runs must terminate when the primary metric is lowered by 25 percent or more compared to the best performing run.You need to configure an early termination policy to terminate training jobs.Which values should you use? To answer, select the appropriate options in the answer area.Note: Each correct selection is worth one point.Hot Area:
Box 1: Bandit The Bandit policy is the best termination policy type for this requirement.Within Azure Machine Learning (Python SDK v2), a Bandit Policy terminates ongoing training runs if the primary performance metric drops below a specified threshold relative to the top-performing run to date.Box 2: slack_factor To achieve this configuration, you should define the slack_factor parameter.In the Azure Machine Learning Python SDK v2, the BanditPolicy class allows you to specify a drop-off threshold using either a relative ratio or an absolute value: slack_factor: Specifies the allowable slack as a ratio or percentage relative to the best performing run.slack_amount: Specifies the allowable slack as an absolute distance or amount.Because your threshold requirement states that the primary metric cannot drop by a relative percentage (25% or more compared to the best run), you must use the ratio-based parameter.To configure a 25% allowable drop, you would set the slack_factor to 0.25.
https://learn.microsoft.com/en-us/Azure/machine-Learning/how-to-tune-hyperparameters
DRAG DROP (Drag and Drop is not supported)A data science team plans to evaluate multiple algorithms and hyperparameters for a classification problem without manually authoring separate training scripts.The team must run an automated machine learning (AutoML) job that compares models and identifies the best-performing configuration.You need to configure and run an AutoML training job.Which four actions should you perform in sequence? To answer, move the appropriate actions from the list of actions to the answer area and arrange them in the correct order.Select and Place:
To run an automated Azure Machine Learning (AutoML) job for classification without manual scripting, you must use the Azure ML Python SDK v2 to configure an automl.classification() job, submit it via the MLClient, and retrieve the best model.1. Set Up Your Workspace Connection [skip] Connect to your Azure Machine Learning workspace using your credentials.Step 1: Create an Azure Machine Learning compute target. 2. Prepare Data and Compute Resources Define your training data input and specify the compute cluster where the parallelized training iterations will execute.Step 2: Configure the AutoML Job with task type, primary metric, and training data. 3. Configure the AutoML Classification Job Use the automl.classification() function to define the target column, primary metric, limits, and training constraints.Step 3: Submit the AutoML Job to the Azure Machine Learning workspace. 4. Submit and Track the Job Submit the configured job to your workspace and wait for its execution to complete.Step 4: Review the completed runs and identify the best-performing model 5. Retrieve the Best Model Query the finished job properties to identify the pipeline execution that yielded the highest primary metric score.
https://oneuptime.com/blog/post/2026-02-16-how-to-use-automl-in-azure-machine-learning-to-find-the-best-classification-model/view
A team is validating a generative AI assistant for a company. The assistant generates responses by using internal knowledge sources.The company requires assurance that responses are accurate, supported by sources, and related to the user prompts before enabling production access.You need to implement quality metrics that confirm the assistant produces reliable and meaningful responses.Which two evaluation metrics should you use? Each correct answer presents part of the solution.Note: Each correct selection is worth one point.
Answer(s): A,B
To ensure your Azure Machine Learning generative AI assistant (which utilizes a Retrieval-Augmented Generation or RAG architecture) produces reliable and meaningful responses before going live, you should use the RAG Triad of built-in quality evaluation metrics.These primary automated metrics—Groundedness, Relevance, and Response Completeness—directly measure accuracy, connection to internal knowledge sources, and alignment with user prompts.The Primary Metrics (The RAG Triad) These metrics are evaluated on a 1-to-5 scale using Azure Machine Learning's built-in, AI-assisted "LLM-as-a-judge" evaluators: Groundedness: Measures how well the assistant's generated answer aligns only with the information retrieved from your internal knowledge sources. Even if a response is factually correct in the real world, it is penalized if the information cannot be verified inside the retrieved context document. This is your primary defense against hallucinations.Relevance: Assesses how pertinently the model's generated response directly addresses the user's specific prompt. This checks whether the system understood the user's intent or if it provided an off-topic or distracted response.Response Completeness: Focuses on the "recall" aspect of the assistant. It measures whether the generated text effectively answers all parts of the user prompt using the ground truth data, ensuring no critical insights or data points are omitted.
https://medium.com/thedeephub/a-deep-dive-into-evaluation-in-azure-prompt-flow-dd898ebb158c
DRAG DROP (Drag and Drop is not supported)You use Azure Machine Learning to deploy a model as a real-time web service.You need to create an entry script for the service that ensures that the model is loaded when the service starts and is used to score new data as it is received.Which functions should you include in the script? To answer, drag the appropriate functions to the correct actions. Each function may be used once, more than once, or not at all. You may need to drag the split bar between panes or scroll to view content.Note: Each correct selection is worth one point.Select and Place:
Box 1: init() Load the model when the service starts.To load an Azure Machine Learning model when a real-time web service starts, you must use the init() function in your entry script.Key Components of an Entry Script An Azure Machine Learning entry script (typically named score.py) requires two core functions to manage the web service lifecycle: init(): This function runs exactly once when the Docker container initializes. You use it to load the model from the registry into a global variable so it is cached and ready for incoming requests.run()/run(): This function executes every time the service receives a prediction request. It uses the model loaded by init() to process the input data and return a prediction.Box 2: run() Use the model to score new data.Use the run(data) function in your entry script to score new data as it is submitted to the deployed web service.
https://azure.github.io/azureml-sdk-for-r/articles/deploying-models.html
HOTSPOT (Drag and Drop is not supported)A regression model is deployed to a production online endpoint and is actively serving predictions.Business stakeholders report that the model’s prediction quality has changed.The endpoint remains available and responsive.You need to determine which metric to examine to validate each quality concern.Which metric should you review for each concern? To answer, select the appropriate options in the answer area.Note: Each correct selection is worth one point.Hot Area:
Box 1: Model performance metrics Prediction accuracy declines over time.To address a decline in model accuracy while the endpoint remains available, you should investigate model performance metrics, such as Data Drift and Prediction Drift.When a model's prediction quality declines over time without API or network failure, it is typically due to shifts in the real-world data (data drift) or concept drift.Box 2: Endpoint latency metrics Response times increase during peak usage.Use Endpoint latency metrics to investigate the increased response times during peak usage.Box 3: Prediction output distribution trends Predictions remain accurate, but business outcomes worsen.The most appropriate metric to look into further is Prediction output distribution trends (often analyzed as part of data drift or concept drift).When predictions remain statistically accurate (e.g., low mean squared error against actual target variables) but business outcomes degrade, it usually indicates a shift in the real-world environment. Monitoring the distribution of prediction outputs over time helps identify downstream concept drift or changes in user behavior.For example, if a pricing model accurately predicts lower market values during a recession, the model is "accurate," but the business loses revenue because the market itself has shifted.
https://www.nannyml.com/blog/3-common-causes-of-ml-model-failure-in-production https://medium.com/@jwang.ml/ml-model-monitoring-and-re-training-when-and-how-7fbfe3b64433
Post your Comments and Discuss Microsoft AI-300 exam prep with other Community members:
Our website is free, but we have to fight against AI bots and content theft. We're sorry for the inconvenience caused by these security measures. You can access the rest of the AI-300 content, but please register or login to continue.