Monitoring Azure Databricks with Azure Monitor
AZURE
Azure Databricks provides comprehensive end-to-end diagnostic logs of activities performed by Azure Databricks users, allowing your enterprise to monitor detailed Azure Databricks usage patterns.
In this article, we're going to look at sending the logs of Azure Databricks workspace to log analytics workspace using diagnostics settings present in the Databricks workspace.
Here are the pre-requisites and steps to enable diagnostics setting for Azure Databricks
Pre-requisites:
- User with owner or contributor access where the Databricks workspace is deployed.
- Databricks workspace. The diagnostics logging for Azure Databricks service is available only for the Premium plan.
Steps:
- Login to Azure portal.
- Select the Databricks workspace.
3. Select the diagnostics settings.
4. Now click “+ Add Diagnostics Settings”.
5. Azure Databricks provides diagnostic logs for the following services:
- DBFS
- Clusters
- Pools
- Accounts
- Jobs
- Notebook
- SSH
- Workspace
- Secrets
- SQL Permissions
6. Here we are going to send the logs to the log analytics workspace.
7. Select all the logs you want and send them to log analytics. Here we’re sending cluster logs.
8. Click Save.
9. Allow some time to ingest the logs to log analytics workspace.
10. Now go to the log analytics workspace where the diagnostics are configured.
11. Select logs. Now using KQL we can query our data sent from the Databricks workspace.
12. The Databricks log tables are found under the LogManagement category.
Databricks Monitoring Dashboard
Here is the simple Databricks Monitoring dashboard we created for
- Cluster availability
- Failed job trend
- Success vs failed job trend
We hope this article helps you set up the right configurations to sending the logs of Azure Databricks workspace to log analytics workspace and build the Databricks monitoring dashboard.
Share this:
Take a look at the lastest aricles
In today's fast-paced enterprise world, the pressure is on to create workflows that are not just efficient, but truly intelligent and scalable. Gone are the days when clunky, form-based interfaces could keep up. They were rigid, often frustrating for users, and crucially, lacked the smarts needed to drive real productivity. But what if your forms […]
Are outdated HR processes holding your enterprise back? In today's hyper-competitive landscape, the efficiency of your human resources directly impacts your bottom line, employee satisfaction, and ability to attract top talent. Yet, many organizations are still grappling with manual, resource-intensive tasks that drain productivity and stifle growth. Imagine a world where: Crafting compelling job descriptions […]
In today's hyper-competitive digital landscape, delivering an exceptional user experience (UX) isn't just a nice-to-have – it's the bedrock of customer loyalty and business growth. But as customer behaviors constantly evolve and applications grow increasingly complex, a critical question emerges: How can organizations consistently measure, monitor, and elevate the user experience at scale, and in […]
Let’s shape your AI-powered future together.
Partner with CloudIQ to achieve immediate gains while building a strong foundation for long-term, transformative success.