Available Metrics for Selecting the Best Run
Question
You are responsible for training and deploying a classification model which can identify fraud attempts among banking transactions.
In order to minimize the time and effort, you decide to use Azure autoML services.
While configuring autoML, you need to select a primary metric which can be used by autoML to select the best run.
Which metrics can you choose from?
Answers
Explanations
Click on the arrows to vote for the correct answer
A. B. C. D.Answer: D.
Option A is incorrect because these are metrics for the Regression or Forecasting task types.
Option B is incorrect becausethese are metrics for the Regression or Forecasting task types.
Option C is incorrect because the list is a mix of metricsfor the Regression or Forecastingtask types.
Option D is CORRECT because these are the metrics you can use to score a Classification model.
Reference:
Azure AutoML is a machine learning tool that automates the process of model selection, training, and deployment. In order to select the best run, AutoML uses a primary metric to measure the performance of different models. The primary metric is selected based on the type of problem being solved, and the available data.
For a classification problem like identifying fraud attempts among banking transactions, the primary metric needs to be selected from metrics that measure the accuracy of the classification model. The higher the accuracy of the model, the better it is at identifying fraud attempts correctly.
Out of the options given, the best choice for a primary metric would be option D, which includes the following metrics:
AUC_weighted: The area under the curve (AUC) of the receiver operating characteristic (ROC) curve is a measure of how well the model can distinguish between positive and negative classes. The weighted AUC metric is useful when the dataset is imbalanced, which is often the case with fraud detection. A high AUC_weighted score indicates that the model is good at distinguishing between fraudulent and non-fraudulent transactions.
norm_macro_recall: The normalized macro recall is the average of the recall for all classes, weighted equally. Recall measures the fraction of positive cases that the model correctly identified as positive. A high norm_macro_recall score indicates that the model is good at identifying fraudulent transactions.
accuracy: The accuracy is the fraction of predictions that are correct. A high accuracy score indicates that the model is good at identifying both fraudulent and non-fraudulent transactions.
Therefore, D is the correct answer because it includes the most relevant and important metrics for a fraud detection problem. Options A, B, and C include metrics that are not relevant for a classification problem, such as r2_score and spearman_correlation, or do not provide a measure of the model's ability to identify fraudulent transactions, such as normalized_mean_absolute_error.
