On-Device AI Inference Changes the Architecture, Not the Risk
As organizations transform into Frontier Firms and integrate AI into more applications, cloud-hosted inference is not always the optimal architectural choice. Some applications operate with limited connectivity, require low-latency responses, or involve data that organizations prefer to process locally. As AI usage scales, token economics also becomes an architectural consideration as organizations look for greater predictability and control over cloud inference cost. By running AI inference directly on the device, applications can eliminate cloud round trip, support offline operations, avoid cloud token costs, and provide greater control over data during processing.
However, moving AI inference onto the device changes the application architecture, not the model’s exposure to behavioral risks. On-device language models can still encounter adversarial inputs intended to bypass safeguards, manipulate instructions, or elicit unsafe and unexpected responses. This makes systematic security and safety evaluation, including AI red teaming, an important part of adopting on-device AI.
AI red teaming evaluates an AI system from an adversarial perspective to uncover safety and security weaknesses before they are encountered in production. While functional testing asks whether a model works as intended under expected conditions, red teaming examines how it behaves when adversarial user intentionally attempts to make it fail or misbehave.
This article shows how organizations can use Microsoft technology to run AI on end-user devices while systematically evaluate model behavior against safety and security risks. To make this practical, I provide a sample solution that development teams can use to automate repeatable evaluations across models.
Running and Evaluating On-Device AI Security Risks
Microsoft addresses these on-device AI challenges with complementary solutions: Foundry Local runs end-to-end AI inference on the device, while AI Red Teaming Agent helps developers evaluate model behavior before release.
Foundry Local (generally available since April 2026), is a cross-platform runtime for integrating local language models directly into applications. It runs on Windows, macOS, and Linux, with SDKs for Python, JavaScript, C#, and Rust. The runtime manages the model lifecycle within the application: downloading, caching, loading, inference, and unloading.
Foundry Local is designed for single-user inference embedded within an application, rather than multi-user model serving through server-based inference stack. It provides a curated catalog of hardware-optimized models, while allowing developers to extend beyond the catalog using Bring Your Own Model (BYOM) in the ONNX format. To learn more, refer to the Foundry Local site.
AI Red Teaming Agent complements the solution by enabling developers to evaluate the safety and security risks in on-device language models before release. It combines Microsoft’s open-source Python Risk Identification Tool (PyRIT), with Foundry Risk and Safety Evaluations to generate adversarial probes, assess attack-response pairs, calculate metrics such as Attack Success Rate, and produce detailed reports.
The AI Red Teaming Agent supports adversarial testing of cloud models hosted in Microsoft Foundry, while its local scan workflow (in Public Preview) extends this capability to on-device models. Using the azure-ai-evaluation[redteam] package, developers can provide a model configuration or application callback as the scan target. The sample solution that follows uses this callback pattern to evaluate a model running through Foundry Local.
Try It Yourself: Red-Teaming a Foundry Local model
To make this article practical, I provided a sample solution in the foundry-local-eval GitHub repository. The sample solution lets you select and download a model from the Foundry Local curated catalog, runs it entirely on your machine, and evaluate its responses to adversarial prompts using the Azure AI Evaluation red teaming SDK. The Azure Developer CLI (azd) provisions the required Azure resources, helping you move from a fresh clone to a scored risk report seamlessly.
The sample uses a hybrid architecture, in which the target model runs locally through Foundry Local, while the Azure AI Evaluation red teaming SDK uses Microsoft Foundry project to generate adversarial prompts and asses the model’s responses. Model inference remains on the device, while attack generation and risk scoring use Azure services.
The scan probes the model across four risk categories: violence, hate and unfairness, sexual content, and self-harm. It applies a range of attack strategies and reports key measurements: Attack Success Rate (ASR) that you can track across models and tune your safeguards against. For more details, such as prerequisites, configuration, and step-by-step run instructions, refer to the repository.
Final Thoughts: Building On-Device AI with Trust from the Start
As organizations progress toward becoming Frontier Firms, bringing AI into more applications must be accompanied by deliberate governance and evaluation. On-device AI is no exception. In this article, I explored how Microsoft technologies can help organizations deploy AI models to end-user devices and conduct systematic, automated safety and security evaluations through AI red teaming. The sample solution provides a practical reference implementation that development teams can use as a starting point.
By incorporating repeatable red teaming into the development lifecycle, teams can compare results across candidate models, strengthen safeguards before deployment, and treat trust as a design requirement. As applications with on-device AI continue to evolve, this evaluation discipline can help organizations innovate with greater confidence while keeping safety and security central to their Frontier Firm transformation journey.