文章作者：Tyan
博客：noahsnail.com | CSDN | 简书
1. Introduction to ML Strategy
1.1 Why ML Strategy?
Teach you ways of analyzing a machine learning problem that will point you in the direction of the most promising things to try.
1.2 Orthogonalization
Chain of assumptions in ML
- Fit training set well on cost function
- Fit dev set well on cost function
- Fit test set well on cost function
- Performs well in real world
Orthogonalization or orthogonality is a system design property that assures that modifying an instruction or a component of an algorithm will not create or propagate side effects to other components of the system. It becomes easier to verify the algorithms independently from one another, it reduces testing and development time.
When a supervised learning system is design, these are the 4 assumptions that needs to be true and orthogonal.
- Fit training set well in cost function
- If it doesn’t fit well, the use of a bigger neural network or switching to a better optimization algorithm might help. - Fit development set well on cost function
- If it doesn’t fit well, regularization or using bigger training set might help. - Fit test set well on cost function
- If it doesn’t fit well, the use of a bigger development set might help - Performs well in real world
- If it doesn’t perform well, the development test set is not set correctly or the cost function is not evaluating the right thing.
2. Setting up your goal
2.1 Single number evaluation metric
Set up a single real number evaluation metric for your problem.
Dev set + single number evaluation metric.
2.2 Satisficing and Optimiziong metric
multiple metrics
2.3 Train/dev/test distribution
Setting up the training, development and test sets have a huge impact on productivity. It is important to choose the development and test sets from the same distribution and it must be taken randomly from all the data.
- Guideline
Choose a development set and test set to reflect data you expect to get in the future and consider important to do well.
2.4 Size of dev and test sets
Set your dev set to be big enough to detect differences in algorithm/models you’re trying out.
Set your test set to be big enough to give high confidence in the overall performance of your system.
2.5 When to change dev/test sets and metrics
Orthogonalization for cat pictures: anti-porn
- So far we’ve only discussed how to define a metric to evaluate classifiers.
- Worry separately about how to do well on this metric.
3. Comparing to humanlevel performance
3.1 Why human-level performance?
Why compare to human-level performance?
Humans are quite good at a lot of tasks. So long as ML is worse than humans, you can:
- Get labeled data from humans.
- Gain insight from manual error analysis:
Why did a person get this right? - Better analysis of bias/variance.
