Azure Alerts can be used to proactively notify you when important conditions are found in your monitoring data. After setting up either metric alerts or log alerts for your workloads, specifically IaaS workloads, there may be times when you need to disable those alerts during a maintenance window.
Depending on the size of your environment and the number of alerts you’ve created, it might be quite a chore to go through each one to disable/enable.
The following will demonstrate how to setup an Azure Automation Runbook to quickly set the status of our IaaS Alerts to either Enabled or Disabled via a webhook. The webhook will allow us to execute the Azure Automation Runbook from anywhere, like an on-premises workstation, to set the alert status. The runbook will also take advantage of Azure Resource Graph as a mechanism to search for alerts across all of the available subscriptions.
- Automation Account
- Runbook – PowerShell
- PowerShell Modules
- The name of the alert must contain the server name
- Azure Resource Graph
- Used to query all subscriptions to find alerts based on server name
- Runbook – PowerShell
Step 1: Create a metric alert(s) for your IaaS Server(s) based on CPU Usage
If you already have an alert(s) defined with the server name in the alert rule, skip this step.
- Navigate to Alerts
- New alert rule
- Select resource
- Select a virtual machine
- Select a condition based on Percentage CPU
- Set the threshold value and leave the other options with their default value
- Select or create an Action Group (required)
- Fill in the remaining Alert rule details and include the server name in the Alert rule name
(For Step 2, chose either 2a or 2b for creating/deploying an Automation Account)
Step 2a: Create an Automation Account – ARM Template Method
- Deploying this ARM template [GitHub] will include the following:
- Azure Automation Account
- Import of PowerShell Modules (Az.Accounts, Az.Monitor, Az.ResourceGraph)
- Runbook (SetAzAlertsStatus-Webhook)
- Creation of the Automation Run As account is not supported when you’re using an ARM template.
- Create a Run As account in Azure portal
- Grant the run as account, at a minimum, the ability to manage Alerts. By default, the AAA run as account is granted contributor rights at the subscription it’s deployed into. In production, granting access to the AAA run as account at a Management Group is recommended.
Step 2b: Create an Automation Account – Manually Method
- Create an Azure Automation Account (AAA)
- Grant the AAA run as account, at a minimum, the ability to manage Alerts. By default, the AAA run as account is granted contributor rights at the subscription it’s deployed into. In production, granting access to the AAA run as account at a Management Group is recommended.
- Import PowerShell Gallery modules (Az.Accounts, Az.Monitor, Az.ResourceGraph) into the AAA
- Under Shared Resources, select Modules.
- Select Browse gallery, and then search the Gallery for a module.
- Select the module to import, and select Import.
- Select OK to start the import process.
- Create an Azure Automation runbook (PowerShell Runbook)
- In Create an Azure Automation runbook article, step #6, copy SetAzAlertsStatus-Webhook.ps1 from GitHub and paste it into the runbook.
Step 3: Create a Webhook for your Runbook
- Create a webhook for your Runbook.
- From the Runbooks page in the Azure portal, click the runbook that the webhook starts to view the runbook details. Ensure that the runbook Status field is set to Published.
- Click Webhook at the top of the page to open the Add Webhook page.
- Click Create new webhook to open the Create Webhook page.
- Fill in the Name and Expiration Date fields for the webhook and specify if it should be enabled. See Webhook properties for more information about these properties.
- Click the copy icon or press Ctrl+C to copy the URL of the webhook. Then record it in a safe place.
- Please save your webhook URL. Once you create the webhook, you cannot retrieve the URL again.
- Click Parameters, leave it blank, press OK.
- Click Create to create the webhook.
Step 4: Test your Automation Account Runbook via webhook
- Download the PowerShell script SetAzAlertsStatus-Webhook-Wrapper.ps1 and save it to your computer.
- Edit the script and update line 32 with your webhook URL:
- $uri = “<runbook webhook URL you saved earlier>”
- Execute the PowerShell script from your local computer.
With an Alert naming convention that includes your server name, this method works very well for quickly enabling or disabling Azure alerts.
I hope you have found this article helpful and thank you for taking the time to read this post.
The sample scripts are not supported under any Microsoft standard support program or service. The sample scripts are provided AS IS without warranty of any kind. Microsoft further disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample scripts and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Microsoft has been advised of the possibility of such damages.