Path 75 · The drift watch · Intermediate
An alert is a question with evidence.
A guided, practical route through model monitoring.
What you will be able to do
Create a local monitoring report for missing data, feature distribution and labeled prediction error, with explicit alert thresholds.
Plan about 8h 8m in total. Reading and practice are editorial estimates; video runtimes come from public provider metadata. Your pace will vary.
Before you begin
Check your starting point
What you need
Basic Python and percentages. Use CSV files and the standard library; no monitoring server or cloud account is required.
The selected public provider lessons are free to view. The original case can be completed with paper, a text editor or freely available local tools. Paid exams, certificates, subscriptions and optional provider software are not required for this project.
Completing this WikiFree path does not award a provider certificate. Any provider assessment, account requirement or credential must be checked separately on its official page.
Three readiness questions
Does feature drift prove performance decline?
No.
What denominator belongs to labeled error rate?
Only labeled records under the stated evaluation rule.
Should an alert automatically retrain a model?
Not without diagnosis and an authorized procedure.
Use a workable pace
At 5 hours per week, allow about 2 weeks. Start with Stage 1, continue in order and reserve 2h for the capstone. If a gate fails, use its repair step before advancing. At another weekly budget, divide 488 estimated minutes by your available weekly minutes and round up.
Keep a small evidence folder. A checked box means you reviewed your output; it is not a certificate or proof that a provider lesson was watched.
The working case
Watch the denominator
Two synthetic batches each contain 100 scored requests. The reference batch has 5 missing ages, 20 premium-category records and 10 errors among 80 labeled records. The current batch has 12 missing ages, 35 premium records and 18 errors among 60 labeled records. Labels are delayed and incomplete.
| Batch | Rows | Missing age | Premium category | Labeled rows | Errors |
|---|---|---|---|---|---|
| Reference | 100 | 5 | 20 | 80 | 10 |
| Current | 100 | 12 | 35 | 60 | 18 |
The table and explanations preserve the essential case information when the diagram is unavailable.
Your route
Five outputs, then a final handoff
- 01Define the monitoring contractMonitoring specification
About 1h 8m including assigned learning and practice. - 02Check data qualityData-quality result
About 1h 19m including assigned learning and practice. - 03Compare distribution and performanceDistribution and error report
About 1h 15m including assigned learning and practice. - 04Investigate before reactingAlert investigation runbook
About 1h 22m including assigned learning and practice. - 05Make the report reproducibleLocal monitoring pack
About 1h 5m including assigned learning and practice. - FINALCapstone reviewCombine the evidence into one inspectable handoff.
About 2h .
0 of 6 outputs checked. JavaScript adds browser-only progress saving.
The complete lesson and all downloads remain available. Use the printable project workbook to track your progress.
Use the minimum useful source material
The courses behind this route
DataTalks.Club
MLOps Zoomcamp: Free Course
Existing MLOps lessons introduce monitoring, data quality and debugging reports; the local standard-library implementation is original and independent of changing platform APIs.
Selected material only; this path does not claim completion of the entire provider course.
Course publication and selected public lesson metadata were checked on 13 October 2026. This is not a claim of full video playback, provider-course completion or permanent availability. Recheck the linked source if a lesson changes.
Learn, make, inspect
The five-stage workshop
Define the monitoring contract
Video 12m · reading budget 0m · original practice about 55m.
Monitoring needs a reference period, current window, metric definition and action owner. A threshold without a response plan creates noise. Distinguish data quality, distribution change and measured performance. The case is an offline report and does not run the provider’s monitoring platform or claim live telemetry.
Watch these selected lessons
Do the work
Define three metrics and a review owner for each.
Save this output
Monitoring specification
Pass the review gate
Every metric has a denominator, window and interpretation.
If you need another attempt
Split a vague model health score into inspectable measures.
Check data quality
Video 14m · reading budget 0m · original practice about 1h 5m.
Missingness can change because of collection, schema or user behavior. Compare rates and preserve counts. For this case, alert when current missing age exceeds 10%. The threshold is an exercise rule, not a universal standard. A missing value must not be silently replaced merely to make the alert disappear.
Watch these selected lessons
Do the work
Calculate missing-age rates and apply the rule.
Save this output
Data-quality result
Pass the review gate
Reference is 5%, current is 12%, so the current rule alerts.
If you need another attempt
Check strict greater-than versus greater-than-or-equal wording.
Compare distribution and performance
Video 0m · reading budget 0m · original practice about 1h 15m.
A category share change can indicate distribution shift without proving harm. Labeled error uses only the labeled subset, whose availability may be biased. State that limitation before comparing rates. Here alert for a category-share increase over 10 percentage points and labeled error over 20%.
Original application stage. Use your preceding outputs and the supplied case; there is no additional required provider lesson.
Do the work
Compute category-share change and both labeled error rates.
Save this output
Distribution and error report
Pass the review gate
Share rises 15 points; labeled error rises from 12.5% to 30%.
If you need another attempt
Do not divide 18 errors by all 100 current records.
Investigate before reacting
Video 17m · reading budget 0m · original practice about 1h 5m.
A report should help a reviewer distinguish a pipeline fault from genuine population change. Check schema, data arrival, labels and recent system changes before proposing retraining. The supplied aggregates cannot identify a root cause. Record hypotheses and the evidence required to test them.
Watch these selected lessons
Do the work
Write a triage checklist for the three triggered rules.
Save this output
Alert investigation runbook
Pass the review gate
No root cause is asserted without supporting evidence.
If you need another attempt
Replace automatic retrain with a bounded review decision.
Make the report reproducible
Video 0m · reading budget 0m · original practice about 1h 5m.
Version the thresholds and save the input counts with each report. Include boundary tests and a no-labels case so undefined performance is visible. The report should fail clearly on impossible counts, such as more errors than labeled records. A monitoring result is evidence for review, not proof of safe production behavior.
Original application stage. Use your preceding outputs and the supplied case; there is no additional required provider lesson.
Do the work
Run the reference checker and add a threshold-boundary test.
Save this output
Local monitoring pack
Pass the review gate
All alerts reconcile with the input, and zero labels gives unavailable rather than 0% error.
If you need another attempt
Validate count relationships before calculating percentages.
Make the result inspectable
Your final project
Deliver the metric contract, report script, threshold configuration, boundary checks and triage runbook. Add a batch with no labels and explain why its error rate cannot be evaluated.
The ZIP includes synthetic input data, a project workbook, worked reference, evidence log, scoring rubric and case-specific materials. Code projects include source and clearly stated local execution requirements.
Use one consistent rubric
| Criterion | Maximum | Evidence |
|---|---|---|
| Evidence and correctness | 30 | Calculations, source use and results match the supplied case; limitations are honest. |
| Required artifacts | 30 | All five stage outputs and the capstone handoff are present and inspectable. |
| Reasoning and decisions | 20 | Choices, assumptions, alternatives and the case-specific critical condition are explained. |
| Reproducibility | 10 | A reviewer can follow the stated steps and distinguish expected from observed results. |
| Clear communication | 10 | Labels, units, structure and evidence references make the work easy to review. |
Score at least 80/100, supply every required artifact, and satisfy the path-specific critical condition. Rework missing evidence before marking complete.
The critical condition
Unlabeled records are not treated as correct predictions and alerts do not claim a proven root cause.
Compare with a worked reference
Current missingness is 12/100 = 12%; category share rises from 20% to 35%, a 15-point shift. Labeled error is 10/80 = 12.5% versus 18/60 = 30%. All three case rules alert, but the aggregate data do not establish the cause.
Know the boundary
This is an offline CSV monitoring exercise, not a running dashboard, drift significance test or cloud deployment.
Questions before you finish
Do I need to finish every source course?
No. This path names selected lessons or reading sections. Complete the full provider course only if it supports your wider goal. The path does not claim equivalence to that course.
How do I know when to advance?
Use the output and review gate in each stage. If evidence is missing or a result conflicts with the case, follow the repair step and retry before advancing.
Can I do the project without paid software?
The selected public provider lessons are free to view. The original case can be completed with paper, a text editor or freely available local tools. Paid exams, certificates, subscriptions and optional provider software are not required for this project.
What does my checklist prove?
It records your own review in this browser only. It does not verify viewing, skill, a provider credential or a real-world deployment.
What if a lesson or tool changes?
Use the linked course and provider page to inspect current availability. Record the change and revisit the relevant assignment; do not invent a replacement lesson or claim an unavailable activity was completed.
Questions & discussion
Share a useful question or correction. Comments appear after moderation. Please avoid personal or sensitive information.