Which Model to Keep Based on Confusion Matrices
Question
You have just completed several runs of your ML experiment in Azure ML Studio.
You have run your multiclass classification experiments, trying several algorithms.
In order to determine which model gives the best result, you start evaluating the runs using the graphical tools provided by Studio.
First, you want to eliminate the models with the weakest performance.
By looking at the confusion matrices, which model should you keep?
Answers
Explanations
Click on the arrows to vote for the correct answer
A. B. C. D.Answer: D.
Option A is incorrect because for a high-accuracy model, the values in the confusion matrix are expected to center in the diagonal.
If most of the numbers are off-diagonal, then it is a sign of a weak model.
Option B is incorrect because for a high-accuracy model, the values in the confusion matrix are expected to center in the diagonal.
If most of the numbers are off-diagonal, then it is a sign of a weak model.
Option C is incorrect because for a high-accuracy model, the values in the confusion matrix are expected to center in the diagonal.
Zeros in the diagonal (with non-zeros out of diagonal) are signs of low model performance.
Option D is CORRECT because the confusion matrix visualizes the number of predicted labels compared to number of actual labels, i.e.
showing the model's accuracy.
With the actual labels on the Y axis and with the predictions on the X, for a high accuracy model the non-zero values must concentrate around the diagonal from top left.
Diagram:
/answer/imgbKJo503NhW906f2xqJgU9z6mvAItYSDX1KY2yYaLrAZ5mXj-D8g3d7RFxnGd5b_BYc-kSZNFL2J-c789m6Ua_nBypBqgnfvhdLsiX854RfI2xtwap-YRWPGKwchdyTzcY--1L1HD)
Reference:
To evaluate the performance of multiclass classification models, we can use confusion matrices. Confusion matrices are a table that allows us to visualize the performance of the model by comparing the actual labels of the test set with the predicted labels produced by the model.
In a confusion matrix, the predicted labels are represented in columns, while the actual labels are represented in rows. The diagonal of the matrix represents the cases where the predicted label matches the actual label.
Now, let's consider each answer option:
A. Where the non-zero values are concentrated in columns: If the non-zero values in the confusion matrix are concentrated in the columns, it means that the model is misclassifying many cases into certain predicted classes. This indicates that the model is biased towards those classes and may not be performing well overall. Therefore, we should eliminate this model.
B. Where the non-zero values are concentrated in rows: If the non-zero values in the confusion matrix are concentrated in the rows, it means that the model is misclassifying many cases from certain actual classes. This indicates that the model is biased against those classes and may not be performing well overall. Therefore, we should eliminate this model.
C. Where the zero values are found in the diagonal from top left: If the zero values are found in the diagonal from top left, it means that the model is performing perfectly on the test set. However, this scenario is unlikely in practice, and we should investigate whether the test set is too easy or whether the model is overfitting.
D. Where the non-zero values are in the diagonal from top left: If the non-zero values in the confusion matrix are concentrated in the diagonal from top left, it means that the model is correctly predicting many cases. This indicates that the model is performing well overall and should be kept.
In summary, we should keep the model where the non-zero values are in the diagonal from top left.