PART 3 – Ingesting AWS GovCloud Microsoft Sentinel in Azure Commercial

In the first part of this series Ingesting Non-Microsoft Cloud Security Data into Microsoft Sentinel for Government & DIB Customers we looked into the compliance levels of the clouds and the architecture of the AWS connector.  In part 2 Ingesting Non-Microsoft Cloud Security Data into Microsoft Sentinel for Gov & DIB customers part 2 we walked thru setting up the connectors from AWS GovCloud and AWS into Microsoft Sentinel in Azure Government.  In this blog we'll discuss connecting AWS GovCloud into Microsoft Sentinel in Azure Commercial.  We recognize that many Government and DIB customers use Azure Commercial and AWS GovCloud, so we've included the ability to monitor and respond to those cloud environments. Most of the same steps will apply, this will be very similar to part 2 so I've added more trouble shooting information in this part. 

jeffsc_0-1715604126328.png

Installing the solution in Sentinel from Content Hub

 First, we will open the Content Hub in Sentinel and search for “Amazon” to find the solution.  The solution includes the S3 connector, 57 Analytic Rules, 36 Hunting Queries & 2 Workbooks. 

jeffsc_22-1715604283759.png

Click install and the solution will install the connector and other parts of the solution.  If you have already installed the Amazon Web Services solution, click on manage.  Once the solution is installed You will be able to select the S3 Connector

jeffsc_23-1715604330489.png

The connector uses Windows PowerShell and AWS CLI to automate the setup required in AWS.

Configuring the installation environment

Prior to installing the connector using the PS1 script, you must properly configure the system you are using.  If you have already done this skip forward to Running the ConfigueAwsConnector.ps1 Script section.

When running the script, ensure that you are using Windows PowerShell as there are Differences between Windows PowerShell  and PowerShell 7.x – PowerShell.

Here is the link for installing PowerShell Install PowerShell on Windows, Linux, and macOS – PowerShell

AWS CLI installation is required by the configuration script to create the connections in AWS. Here is the link from Amazon to setup AWS CLI, AWS CLI Installation instructions

The configuration script will confirm these settings when it runs. If this process returns any errors, check the *_accessKeys.csv file that you downloaded from AWS when you set up the access key, and ensure that the region is correct. In the AWS Console, the region displays US-Gov-West. You will need to update this to us-gov-west-1 when you run the AWS configure command or the script will throw an error.

jeffsc_24-1715604378038.png

Verified AWS access configuration

You will need to run the ConfigureAwsConnector.ps1 script. Depending on your organization's policies, you may not be able to run the PowerShell script without setting the Windows PowerShell execution policy (using Set-ExecutionPolicy) to allow running unsigned . You can check your policies by running.

PS C:> Get-ExecutionPolicy –List

 

If you need to change the policy run this command to allow running unsigned

              PS C:> Set-ExecutionPolicy -ExecutionPolicy Unrestricted

 

Once you are done installing the connections, you should set the execution policy back. More information about setting execution policies can be found on this page Execution Policies – PowerShell

Running the ConfigueAwsConnector.ps1 Script

To get started, download / extract the PowerShell script.

jeffsc_25-1715604430430.png

Make note of the log analytics workspace ID, this will be required when running the ConfigureAwsConnector.ps1 script. To further simplify this process, a Cloud Formation Template is in development. That information will be added to this blog series upon release.

In this example we are connecting VPC Flow logs.

The script will ask for the following information in order to setup the OpenID Connection ((OIDC)🙁

  • S3 bucket and will create one if needed. 
  • Azure Workspace ID
  • SQS queue to be created.

jeffsc_26-1715604466763.png

Make a note of the information listed under “Role Arn”, “Sqs Queue”, and “Destination Table” as you will need this to configure the connection in Sentinel.  On the connector page in Sentinel paste in the information

jeffsc_27-1715604543594.png

Enter the Role ARN, SQS URL & Destination table on the connector blade in Sentinel 

jeffsc_28-1715604597604.png

Then select add connection. The first time you connect, you may see an error (ID25).

jeffsc_29-1715604629275.png

This is due to a test message the new SQS queue sends on first connection.   The test message does not have the correct record that the connector is expecting and throws the error.  Microsoft is working on either clarifying the error message or suppressing the error.  When you click connect a second time and then the SQS queue sends the correct message, and the connection will be successful. 

jeffsc_30-1715604678565.png

You can repeat this process for the other data types in AWS GovCloud.  The connection process is the same for each data type.  Guard Duty & Cloud Trail data types will ask if you want to use KMS.  I used a different KMS for each data type just to keep things organized.  Here is an example of what I used in my lab

jeffsc_31-1715604726907.png

Once you have all the connections setup you will start seeing data flowing.  It might take a few minutes for data to come into Sentinel.  This depends on amount of data AWS generates and when Sentinel communicates with the SQS queue. You should see a list of the connection on the Amazon S3 connector page in Sentinel. 

jeffsc_32-1715604761183.png

Unique Scenario – Sending AWS Guard Duty data into both Azure Gov and Azure Commercial.

I found the scenario for ingesting one AWS account into two Sentinels due to the way my test lab and accounts were configured.  I don't think many customers will encounter this scenario, but I wanted to cover it just in case.  

jeffsc_33-1715604794073.png

Since Guard Duty Settings it can only use a single bucket / SQS queue, it is not possible to send Guard Duty logs to 2 instances of Sentinel “out-of-the-box”.  If you want to do this, you would have to add another Sentinel Workspace and possibly API path to the Trust Relationship Policy.

jeffsc_34-1715604843615.png

You will also have to find a way to handle the messages in the SQS queue.  Since each Sentinel instance will pick up the messages and the other Sentinel instance may miss the message to get the logs.

If you want to send single AWS account data into 2 Sentinel instances this will require using something like a message bus and custom code.  That discussion is beyond the scope of this blog.

the connector –

In the second blog entry Ingesting Non-Microsoft Cloud Security Data into Microsoft Sentinel for Gov & DIB customers part 2 we investigated verifying data flow and querying it once it has been imported to Microsoft Sentinel. In this section, we will tackle unexpected issues. There are several places that are helpful to check in AWS while or to ensure that the script ran correctly:

  • S3 Bucket – logs and event notifications
  • VPC – Flow Logs
  • SQS queues
  • Cloud Trail
  • Guard Duty

S3 Buckets – Here, we will be looking for logs and event notifications.  In the bucket you should see a folder called AWSLogs. Under that folder look for the folder with your account ID – you may also see a folder that starts with o-****.  The folder that with o-*** is where organization logs may be.  Look for the *.gz files, these are the actual logs.  

Also check the Event Notifications & SQS queues on the bucket to ensure they were created

jeffsc_35-1715604944666.png

Check the Bucket Policy and check the OIDC Role

jeffsc_36-1715604984230.png

VPC Flow Logs – verify the Destination Name is the correct S3 bucket in the VPC configuration on the AWS console.   Then you can verify that the folder mentioned above actually has *.gz files. 

SQS Queues – Look on the monitoring tab and verify traffic on the monitor tab of the queue(s). If you don't see any traffic here, then there might be an issue with the event notifications.  

jeffsc_37-1715605027300.png

Verify the Access Policy has the correct OIDC role and permissions

jeffsc_39-1715605116363.png

Cloud Trail – Verify Trail settings in the Cloud Trail console-  

  • Trail Log location under General Details.  and the IAM Role on the Trail page.
  • AWS KMS key is correct
  • IAM Role in the CloudWatch Logs

Guard Duty – verify the S3 Protection is on

jeffsc_40-1715605354318.png

the correct S3 bucket is set correctly in the console- Guard Duty Settings

jeffsc_41-1715605375635.png

jeffsc_42-1715605403784.png

Summary

In this part of this series, we've walked through setting up the CLI and required to establish the connection from AWS GovCloud into Microsoft Sentinel in Azure commercial and how can verify that the script setup the AWS resources correctly.  You're encouraged to explore the workbooks, analytics and hunting queries that are installed with the AWS Solution. These concepts can be explored in-depth with the helpful links below:

We just released the Multi Cloud Attack Essentials solution in the Sentinel Content Hub & GitHub.  Here is the link to the repository   Azure-Sentinel/Solutions/Multi Cloud Attack Coverage Essentials – Resource Abuse at master · Azure/A…

jeffsc_43-1715605436895.png

In our next blog, we will cover Google Cloud Platform (GCP) integration and the new Cloud Formation installation option for AWS.

 

This article was originally published by Microsoft's Sentinel Blog. You can find the original article here.