In August this year, research group Wiz discovered an unprecedented critical vulnerability within Microsoft Azure Cosmos DB, Azure’s flagship database. Dubbed ChaosDB, the vulnerability allows for remote account takeover of another customers Cosmos DB instance/s, by leveraging the Jupyter Notebook integration feature of Cosmos DB.
Once informed, Microsoft was quick to act on the notification and within 48 hours remediated the vulnerability. However, Wiz believes the vulnerability was active for months prior to remediation.
Fortunately, not all Cosmos DB customers would have been impacted by this finding and Microsoft actively contacted the 30% of total Cosmos DB customers whose accounts faced the highest risk.
No evidence has been discovered that the vulnerability was exploited or that customer data was breached. However, I strongly advise Cosmos DB customers – especially those with Jupyter integration enabled – immediately reset their security keys on all DB instances.
Jump ahead to our detailed recommendations >
The risk applies to any Azure Cosmos DB instance using the Jupyter Notebook integration. ChaosDB allowed for an external party to query the Jupyter service and retrieve the Primary Security Key for the Cosmos DB service, including the Jupyter Notebook compute and the Jupyter Notebook storage account. With these credentials, malicious actors can read, write and delete data in the target Cosmos DB.
Further reading on the research team’s findings and technical summary can be found here. The diagram below shows you how the attack works.
Source: Wiz, from chaosdb.wiz.io/
Microsoft were informed of the vulnerability on August 12, within two days by August 14, the vulnerable feature was disabled and the risk mitigated. Microsoft’s investigations have indicated that no customer data was accessed because of this vulnerability by third parties, or by the researchers who discovered the issue. Microsoft was prompt to notify approximately 30% of Cosmos DB customers whose keys may have been affected during the researcher’s activity, suggesting they regenerate their keys as a precaution.
For more details on Microsoft’s response and recommendations see Microsoft’s blog.
We advise all customers using Cosmos DB rotate all connection keys for each instance of Cosmos DB. The secondary read-write key, primary read-only key and secondary read-only key were not found to be vulnerable, but we recommended regenerating these as well to be on the safe side – even if Jupyter Notebook integration is not being used. See the step-by-step procedure below for help regenerating your security keys.
If you have diagnostic logs enabled, you can review the logs to double-check for unusual IP addresses accessing your Cosmos DB.
If you are running more than one instance of Cosmos DB, I’d suggest using PowerShell or the Azure Command Line Interface (Azure Cli) to rotate the security keys. You can also enable Diagnostic Logging or Azure Defender to periodically rotate your keys.
For a manual procedure for key regeneration follow the technical steps below:
1. Navigate to All Services within Microsoft Azure and then enter “Cosmos” into the filter.
2. For each deployed instance of Cosmos DB open the resource.
3. Navigate to the Keys configuration blade.
4. Select the ellipses on the right next to both the Primary and Secondary keys and select Regenerate. See the note bellow before you regenerate both keys at the same time.
Note: regenerating the Primary and Secondary Keys at the same time may cause service disruption for services that are still using the old connection string(s), ensure that this procedure includes updating any dependant services with the newly generated connection string.
To maintain service availability for mission critical workloads the Secondary Key should be regenerated first, then update the dependant application or service to use the Secondary Key. Once connections have been validated then regenerate the Primary Key and update the dependant service to use the new Primary Key. It is not recommended to leave the dependant application using the Secondary Key as it exists for this type of procedure.
5. Once you are ready to regenerate your keys, Microsoft will confirm you are sure, click OK.
6. Azure will now generate a new key.
7. After about 30 seconds a new key will be issued.
8. Dependant services can now be updated with the new connection string as this incorporates the new key that was generated.
9. Repeat the same steps for the read-only keys.
1. Customers using the native security features of Cosmos DB will most likely not have been impacted by this vulnerability. However, even with private networking enabled there is never a 100% guarantee a resource is safe. It’s always best practice to assume a breach and run an audit and access investigation. Follow the next steps for a basic review.
2. Use Metrics to review inbound connections to your database. Filtering by geographic region is one of the easiest ways to spot abnormal behaviour. Assess other metrics such as data consumption to validate normal and abnormal connections.
3. If your instance is onboarded into Azure Sentinel then further insights will be available to assist with assessing connection behaviour.
To mitigate future attacks against Cosmos DB it is recommended to enable the advanced protection features built into the platform. The following links share more on enabling advanced security features for Cosmos DB.
Hopefully, this information equips you to take immediate action in protecting your Azure environment. If you have any questions or require support securing your Azure environment, please reach out to your Account Manager or contact a security specialist here.