Create a local monitoring report for missing data, feature distribution and labeled prediction error, with explicit alert thresholds.
Two synthetic batches each contain 100 scored requests. The reference batch has 5 missing ages, 20 premium-category records and 10 errors among 80 labeled records. The current batch has 12 missing ages, 35 premium records and 18 errors among 60 labeled records. Labels are delayed and incomplete.
| Batch | Rows | Missing age | Premium category | Labeled rows | Errors |
|---|---|---|---|---|---|
| Reference | 100 | 5 | 20 | 80 | 10 |
| Current | 100 | 12 | 35 | 60 | 18 |
Monitoring needs a reference period, current window, metric definition and action owner. A threshold without a response plan creates noise. Distinguish data quality, distribution change and measured performance. The case is an offline report and does not run the provider’s monitoring platform or claim live telemetry.
Define three metrics and a review owner for each.
Monitoring specification
Record your result and evidence here or in a separate file.
Every metric has a denominator, window and interpretation.
Split a vague model health score into inspectable measures.
Missingness can change because of collection, schema or user behavior. Compare rates and preserve counts. For this case, alert when current missing age exceeds 10%. The threshold is an exercise rule, not a universal standard. A missing value must not be silently replaced merely to make the alert disappear.
Calculate missing-age rates and apply the rule.
Data-quality result
Record your result and evidence here or in a separate file.
Reference is 5%, current is 12%, so the current rule alerts.
Check strict greater-than versus greater-than-or-equal wording.
A category share change can indicate distribution shift without proving harm. Labeled error uses only the labeled subset, whose availability may be biased. State that limitation before comparing rates. Here alert for a category-share increase over 10 percentage points and labeled error over 20%.
Compute category-share change and both labeled error rates.
Distribution and error report
Record your result and evidence here or in a separate file.
Share rises 15 points; labeled error rises from 12.5% to 30%.
Do not divide 18 errors by all 100 current records.
A report should help a reviewer distinguish a pipeline fault from genuine population change. Check schema, data arrival, labels and recent system changes before proposing retraining. The supplied aggregates cannot identify a root cause. Record hypotheses and the evidence required to test them.
Write a triage checklist for the three triggered rules.
Alert investigation runbook
Record your result and evidence here or in a separate file.
No root cause is asserted without supporting evidence.
Replace automatic retrain with a bounded review decision.
Version the thresholds and save the input counts with each report. Include boundary tests and a no-labels case so undefined performance is visible. The report should fail clearly on impossible counts, such as more errors than labeled records. A monitoring result is evidence for review, not proof of safe production behavior.
Run the reference checker and add a threshold-boundary test.
Local monitoring pack
Record your result and evidence here or in a separate file.
All alerts reconcile with the input, and zero labels gives unavailable rather than 0% error.
Validate count relationships before calculating percentages.
Deliver the metric contract, report script, threshold configuration, boundary checks and triage runbook. Add a batch with no labels and explain why its error rate cannot be evaluated.
Unlabeled records are not treated as correct predictions and alerts do not claim a proven root cause.
| Criterion | Points | Evidence |
|---|---|---|
| Evidence and correctness | 30 | Calculations, source use and results match the supplied case; limitations are honest. |
| Required artifacts | 30 | All five stage outputs and the capstone handoff are present and inspectable. |
| Reasoning and decisions | 20 | Choices, assumptions, alternatives and the case-specific critical condition are explained. |
| Reproducibility | 10 | A reviewer can follow the stated steps and distinguish expected from observed results. |
| Clear communication | 10 | Labels, units, structure and evidence references make the work easy to review. |
Score at least 80/100, supply every required artifact, and satisfy the path-specific critical condition. Rework missing evidence before marking complete.
Current missingness is 12/100 = 12%; category share rises from 20% to 35%, a 15-point shift. Labeled error is 10/80 = 12.5% versus 18/60 = 30%. All three case rules alert, but the aggregate data do not establish the cause.
This is an offline CSV monitoring exercise, not a running dashboard, drift significance test or cloud deployment.