Determining Model Power: Evaluation Tools and Metrics

Answers: A and B.

Option A is CORRECT because the root mean squared error is a single value that summarizes the errors in the model.Its normalized version (RMSE divided by the range of the data) is one of the metrics typically used for regression problems

The closer its value to 0.0 the better.

Option B is CORRECT because one of the visualizations Azure ML provides for evaluating regression models is the Predicted vs.

True diagram.

It shows the relationship between a predicted value and its correlating true value.

It indicates good model performance if the predicted values are close to the y=x line.

Option C is incorrect because the ROC (Receiver Operating Characteristic) curve displays the correctly classified labels vs.

the incorrectly classified ones.

It actually indicates how “strong” the model is but it is only aApplicable for classification problems.

Option D is incorrect because Recall is a metric expressing the percent of correctly labeled elements of a certain class (the percent of the total amount of relevant instances that were actually found)

Applicable only for classification problems, not for regression.

Option E is incorrect because AUC (Area Under Curve) is a metric used for classification scenarios.

It shows the relationship between the true positives and the false positives, in a graphical form.

This is a good visual metric but not for regression models.

Reference:

The two evaluation tools/metrics that would help in determining the power of a regression model are Normalized root mean squared error (RMSE) and Predicted vs. True chart.

1. Normalized root mean squared error (RMSE): RMSE is a commonly used metric to measure the accuracy of a regression model. It calculates the difference between the predicted and actual values of the dependent variable, and takes the root mean square of that difference. The normalization of RMSE refers to scaling the error by the range of the dependent variable, making it more interpretable across different datasets. A lower RMSE score indicates better performance of the model, as it implies that the model's predictions are closer to the actual values.

2. Predicted vs. True chart: A Predicted vs. True chart helps to visualize the performance of the regression model by plotting the predicted values against the actual values. It provides a clear indication of how well the model is predicting the outcomes. In this type of chart, the ideal plot is a straight line, where the predicted values are identical to the actual values. The more scattered the plot, the poorer the model performance. The chart helps in understanding where the model is making the most errors, whether the errors are systematic or random, and how the model can be improved.

In contrast, the other metrics listed in the options, such as ROC chart, Recall, and AUC, are more suited for classification models. The ROC chart displays the tradeoff between true positive rate and false positive rate at different classification thresholds, while Recall measures the proportion of actual positive cases correctly identified by the model. AUC is a metric that measures the area under the ROC curve, which is useful in evaluating the overall performance of a classifier. However, in a regression problem, where the goal is to predict a continuous variable, these metrics do not provide meaningful insights into model performance.