Overview
Section 4 has five objectives, all about seeing what the platform is doing and reacting when something goes wrong. System tables in the system catalog expose billing, compute, access and Lakeflow data that you query with SQL for cost analysis, audit and workload monitoring. The REST API, Databricks CLI and SDK let scripts and external tools list runs, read run output and analyse pipeline updates.
Each Lakeflow pipeline writes an event log with flow progress, expectation results and cluster activity. Alerts evaluate a query on a schedule and notify when a condition is met, covering business metrics, data quality, cost, warehouse health, audit events and job outcomes. Finally, the Lakeflow Jobs UI and Jobs API show run status, durations and task timelines. The previous outline had six objectives here, including SQL Alerts and the Spark UI; the October 2026 version consolidates them.
How it's tested
Most items present a monitoring question from a stakeholder, such as who ran up cost last week or why a table has fewer rows, and ask which table, log, API or alert answers it. A few show a SQL query against a system table and ask what it returns or what is missing.
-
Find the right source
A cost, audit or reliability question is asked, and you pick the system table, event log or API that holds the answer.
-
Automate the check
Monitoring must run without people, and you choose an alert, a job notification or an SDK script.
-
Read the metric
A query or log extract is shown, and you interpret what it says about cost, quality or run health.
Finance asks for the dollar cost of each Lakeflow Job last month. The Jobs UI shows durations but not cost. system.billing.usage records DBU consumption tagged with job IDs, and joining it to system.billing.list_prices converts DBUs to list price. Grouping by the job identifier gives cost per job, so the billing system tables are the right source.
Objectives 5
Official objectives from the Databricks outline, in outline order.
4.1 Use Databricks system tables (billing, compute, access, lakeflow) for cost analysis, auditing and workload monitoringQuerying account-wide operational data with SQL.
- What it means
- System tables are Databricks-managed tables in the system catalog. The billing schema holds usage and list prices; compute holds clusters, node timelines and warehouse events; access holds audit logs and lineage; lakeflow holds jobs, job runs, task runs and pipeline information.
- What to understand
- Usage records include identity, SKU and custom tags, so cost can be grouped by job, pipeline, warehouse or team tag. Audit logs record actions such as permission changes and table reads. Lakeflow tables let you calculate run durations and failure rates across all workspaces in a region. Access must be granted by an admin, and the data arrives with some delay rather than in real time.
- How it shows up
- Questions ask which schema answers a cost, audit or reliability question, or how to join tables for a metric.
- Common confusion
- Expecting system tables to be real time; they are refreshed with a lag.
4.2 Use Databricks REST APIs / CLI and SDK for monitoring and analyzing jobs and pipelinesProgrammatic access to run and pipeline information.
- What it means
- The Jobs and Pipelines REST APIs expose run lists, run details, task outputs, pipeline updates and events. The Databricks CLI wraps them for shell scripts, and the Databricks SDK for Python offers typed clients such as WorkspaceClient for code.
- What to understand
- Use the SDK when monitoring logic lives in Python, the CLI for scripts and CI checks, and raw REST for other languages or tools. Authenticate with OAuth for a service principal rather than personal tokens. Runs can be filtered by job, status and time, and pipeline events can be paged. These interfaces also feed external observability tools.
- How it shows up
- A scenario wants an external tool to check job health, and you pick the API, CLI or SDK call.
- Common confusion
- Using a personal access token of a departed user for monitoring automation.
4.3 Use Pipeline event logs to monitor pipeline health and data qualityReading the structured log every pipeline produces.
- What it means
- Each Lakeflow pipeline records an event log covering updates, flow progress, data quality results, user actions and compute changes. It can be queried with the event_log table-valued function or published as a Unity Catalog table.
- What to understand
- flow_progress events carry row counts and, in their details, expectation pass and fail counts per rule. update_progress events show state changes such as failed or completed. Lineage of datasets appears in flow definitions. Querying the log lets you build dashboards and alerts on quality trends instead of reading the UI.
- How it shows up
- Questions ask where to find expectation metrics or why an update failed.
- Common confusion
- Looking for expectation results in system access tables instead of the pipeline event log.
4.4 Use Databricks Lakehouse alerts to monitor governed business metrics, data quality and anomaly signals, usage and cost, SQL warehouse and query health, audit and security events, AI agent quality, and Lakeflow Job branchingScheduled conditions that notify or branch workflows.
- What it means
- An alert runs a query on a schedule, compares a result to a threshold, and notifies destinations such as email, Slack or webhooks when the condition is met. Because the query can read any governed table, alerts cover business metrics, quality checks, system table cost, audit events and job outcomes.
- What to understand
- Alerts are only as good as their query, so base them on governed metric definitions and system tables. Choose the schedule to match the data's refresh rate. Alert results can drive Lakeflow Job logic, for example branching on a condition. Data quality monitoring adds anomaly detection on freshness and completeness for tables.
- How it shows up
- Expect a requirement to be notified automatically when a metric crosses a threshold, and you choose an alert.
- Common confusion
- Building a custom polling notebook when a scheduled alert on a query does the job.
4.5 Use Lakeflow Jobs UI and Jobs API to monitor job status and performance metricsWatching run state, durations and task timing.
- What it means
- The Jobs UI shows each run's status, duration, task graph and timeline, with a matrix view across runs. The Jobs API returns the same data for automation. Jobs also support notifications on start, success, failure and duration thresholds, plus streaming backlog health rules.
- What to understand
- Duration warnings and timeouts catch slow runs before they breach SLAs. The task timeline reveals which task dominates run time. Repair history and retries are visible per run. Notifications should go to shared destinations rather than individual inboxes.
- How it shows up
- Questions ask how to detect slow or failing runs and who gets notified.
- Common confusion
- Confusing a timeout, which stops a run, with a duration warning, which only notifies.
High-yield concepts
The terms, rules and formulas this domain turns into questions most often.
-
system.billing.usage
DBU consumption records with identity, SKU and tags.
- Remember
- Join with list_prices for cost.
- Tested as
- Cost questions.
- Trap
- Reading cost from the Jobs UI.
-
system.access.audit
Audit log of workspace and account actions.
- Remember
- Answers who did what and when.
- Tested as
- Audit questions.
- Trap
- Looking in lineage tables for permission changes.
-
system.lakeflow tables
Jobs, job runs, task runs and pipeline records.
- Remember
- Calculate durations and failure rates across workspaces.
- Tested as
- Reliability reporting.
- Trap
- Calling the API per workspace instead.
-
System table latency
Data lands with a delay.
- Remember
- Not suitable for second-level alerting.
- Tested as
- Design questions.
- Trap
- Treating them as streaming sources.
-
Databricks SDK WorkspaceClient
Typed Python client for workspace APIs.
- Remember
- Good for monitoring scripts.
- Tested as
- Programmatic monitoring.
- Trap
- Parsing CLI text output in Python.
-
event_log function
Table-valued function that reads a pipeline's event log.
- Remember
- Holds expectation metrics in flow_progress.
- Tested as
- Pipeline health questions.
- Trap
- Searching the audit log for quality data.
-
Alerts
Scheduled query conditions that notify destinations.
- Remember
- Quality depends on the underlying query.
- Tested as
- Automated notification questions.
- Trap
- Writing a custom polling notebook.
-
Duration threshold
Job setting that notifies when a run exceeds expected time.
- Remember
- Does not stop the run.
- Tested as
- SLA monitoring questions.
- Trap
- Confusing it with timeout.
-
Notification destinations
Shared targets such as Slack, email groups and webhooks.
- Remember
- Prefer team channels to personal addresses.
- Tested as
- Operational design.
- Trap
- Notifying only the job owner.
Side by side
Concepts candidates mix up, compared on the point that decides the answer.
Where each monitoring signal lives
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Billing system tables | DBU usage and list prices | Cost by job, warehouse or tag | Expecting cost in the Jobs UI |
| Audit system table | Recorded user and service actions | Security and compliance questions | Using it for data quality |
| Pipeline event log | Updates, flow progress and expectations | Per-pipeline health and quality | Expecting it to cover jobs |
| Jobs UI and API | Run state, duration and task timeline | Operational run monitoring | Using it for account-wide cost |
Timeout vs. duration warning vs. alert
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Timeout | Maximum run time | Cancels the run | Used when only notification was wanted |
| Duration warning | Expected run time threshold | Notifies, run continues | Expected to stop the run |
| Alert | Scheduled query condition | Works on any governed data | Used for in-run job events |
Exam traps
Each trap is an answer that looks right. Learn the clue that should change your mind.
-
Cost in the Jobs UI
- What candidates pick
- Read run durations and estimate cost manually.
- Why it looks right
- The Jobs UI is where jobs are viewed.
- Why it's wrong
- Only billing usage records consumption with SKUs; joining list prices gives accurate cost.
Clue to watch for The stakeholder asks for money or DBUs.
-
Audit for quality
- What candidates pick
- Query the audit table to count dropped rows.
- Why it looks right
- Audit sounds like checking data.
- Why it's wrong
- Expectation metrics are written to the pipeline event log, not the audit system table.
Clue to watch for The question mentions expectations or row quality.
-
Real-time system tables
- What candidates pick
- Alert on system tables for second-level reaction.
- Why it looks right
- They are queryable like any table.
- Why it's wrong
- System tables arrive with delay, so near-real-time needs job notifications or streaming metrics.
Clue to watch for The requirement demands immediate reaction.
-
Timeout for warning
- What candidates pick
- Set a timeout to learn about slow runs.
- Why it looks right
- Both relate to duration.
- Why it's wrong
- A timeout cancels the run; a duration threshold only notifies.
Clue to watch for The run must continue but someone must be told.
-
Personal token automation
- What candidates pick
- Use an engineer's token in a monitoring script.
- Why it looks right
- It works immediately.
- Why it's wrong
- Automation should authenticate as a service principal with OAuth so it survives staff changes and is auditable.
Clue to watch for The script runs unattended for the long term.
Answer framework
Use these steps when a Section 4 scenario asks how to observe or alert on something.
- Name the question being asked: cost, who did what, pipeline quality, job reliability or a business metric, since each has a home.
- Map it to the source: billing tables for cost, audit for actions, event log for pipeline quality, lakeflow tables or Jobs API for runs.
- Check freshness needs, because system tables lag while job notifications and pipeline events are closer to real time.
- Choose the automation: an alert on a query, a job notification or duration threshold, or an SDK or CLI script feeding another tool.
- Send notifications to shared destinations and run automation as a service principal so monitoring survives team changes.
Practice question
Want more like this? The Databricks DE Pro practice test has questions for every domain.
Checklist
0 of 10
Tick each one you can do without notes. Your ticks stay in this browser.
Study strategy
Learn this section by building a small monitoring kit. Start in the system catalog: write one query per schema, such as cost per job from billing, recent permission changes from audit, cluster uptime from compute, and failure rate from the Lakeflow job run tables. Save them, because writing them once fixes which schema holds what.
Next, open a pipeline you built earlier and query its event log. Find the flow_progress events and pull expectation metrics from the details column. Then trigger a failing update and look at update_progress events to see how failures are recorded.
For programmatic access, install the Databricks SDK for Python, authenticate as a service principal if you can, and list recent runs of one job. Repeat with the CLI so both interfaces are familiar.
Finally, create an alert on one of your system table queries and add duration warnings and failure notifications to a job. Keep a one-page map from question type to source, as most items in this section test that mapping.
FAQ
How much of the exam is Monitoring and Alerting?
Monitoring and Alerting is 10% of the scored content, roughly six questions out of 60.
Which system tables should I know?
Billing usage and list prices, compute tables for clusters and warehouses, access audit and lineage, and the Lakeflow jobs, runs and pipeline tables.
Where are expectation metrics stored?
In the pipeline event log, inside flow_progress event details. Query them with the event_log function or a published event log table.
Do I need to know the SDK code in detail?
Know what the SDK, CLI and REST API offer and when each fits. Recognising a WorkspaceClient call that lists runs is enough.
What are Databricks Lakehouse alerts on the exam?
The guide uses this name for alerts that evaluate governed data on a schedule and notify or influence workflows, covering metrics, quality, cost, audit and job outcomes.
Did the Spark UI move out of this section?
In the October 2026 guide, the Spark UI appears under Debugging and Deploying for diagnosis, while this section focuses on system tables, logs, APIs and alerts.
Other Data & Analytics certifications
Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.
Test your Domain 4 knowledge
You have covered system tables, APIs, event logs, alerts and job monitoring. Check whether you can map each monitoring question to its source quickly.
Keep going
- Training Training Built on your Weak areas
- Practice test Questions by domain, with explanations
- Mock exam Full length, timed like the real exam
- Readiness test Find your weakest domain in minutes
- Cheat sheet Key terms and formulas on one page
- Ask the Databricks DE Pro AI tutor Get any concept explained your way