Overview
Before activating a health score configuration, test it on sample customers to ensure scores are accurate and predictive.What to Test
| Test | What to Look For |
|---|---|
| Score Accuracy | Do scores match your intuition about each customer? |
| Distribution | Are customers spread across tiers (not all in one)? |
| Category Balance | Is one category dominating the overall score? |
| Edge Cases | Do new customers and missing data handle gracefully? |
- Healthy (80-100): 30-40%
- Medium (60-79): 35-45%
- High Risk (40-59): 15-20%
- Critical (0-39): 5-10%
Creating a Test Set
Select 10-15 customers representing different scenarios:| Type | Count | Expected Score | What to Include |
|---|---|---|---|
| Healthy | 3-4 | 75-100 | High MRR, active usage, regular engagement |
| At-Risk | 3-4 | 30-60 | Declining usage, payment issues, low engagement |
| Medium | 2-3 | 60-75 | Steady but not growing |
| Recently Churned | 1-2 | 0-40 | Validates scores would have predicted churn |
| Edge Cases | 2-3 | Varies | New customers, missing data, seasonal accounts |
Analyzing Results
For each test customer, compare expected vs. actual scores:- Score matches - Configuration is working
- Score too high - Check if a category (often Revenue) is masking problems
- Score too low - Check if one bad metric is dragging down the score unfairly
Common Issues
| Issue | Likely Cause | Fix |
|---|---|---|
| All scores too high | Thresholds too lenient or missing data | Tighten thresholds, check integrations |
| All scores too low | Thresholds too strict | Loosen thresholds to match reality |
| Doesn’t predict churn | Wrong categories weighted | Analyze churned customers, adjust weights |
| One category dominates | Weight too high or no variance in others | Rebalance weights (none >40%) |
| Scores too volatile | Short time windows | Use 90-day windows, trend metrics |
Iteration Workflow
- Test - Run scores on your 10-15 customer set
- Identify issues - Note where actual vs. expected differ
- Make one change - Adjust one weight or threshold
- Retest - Compare to previous results
- Repeat - Typically 3-5 iterations needed
- Validate with CSMs - Get feedback before activating
Before Activating
- Scores align with expectations for all test customers
- Distribution looks reasonable across tiers
- No single category dominates
- Edge cases handled gracefully
- CSMs have reviewed and validated