You work as a machine learning specialist for a computer hardware component producer.
Your company produces individual components, such as processor chips, GPUs, etc.
and assembled computer peripherals such as monitors, external disk drives, among others.
You and your team have been tasked with building a machine learning model that predicts future product sales to improve supply chain management based on data from your semiconductor, transistor, and other base component suppliers as well as data from your sales department.
After training your model, you now need to evaluate it to determine whether its performance and accuracy will allow you to use it to predict future product sales accurately.
You have decided to perform an offline model evaluation of your model using your historical data.
You have split your validation dataset into 10 parts.
You then execute 10 training runs, which produces 10 models.
You then aggregate the 10 models to get your final evaluation.
Which model evaluation method are you using?
Click on the arrows to vote for the correct answer
A. B. C. D.Correct Answer: B.
Option A is incorrect.
Holdout validation uses one validation dataset to use in model validation.
Using this method, you evaluate your model using this one holdout set.
But you have split your validation dataset into 10 parts.
So you will have 10 training runs in your validation process.
Option B is correct.
You are using the k-fold cross-validation technique.
Using this validation method, you split the example dataset into k parts, in your case, k = 10
You then run 10 training using each of the 10 example datasets.
Finally, using this method, you aggregate the 10 run results to get your final evaluation.
Option C is incorrect.
There is no bayesian validation method.
Option D is incorrect.
There is no hierarchical validation method.
References:
Please see the Amazon SageMaker developer guide titled Validate a Machine Learning Model (https://docs.aws.amazon.com/sagemaker/latest/dg/how-it-works-model-validation.html),
The Towards Data Science article titled Validating your Machine Learning Model (https://towardsdatascience.com/validating-your-machine-learning-model-25b4c8643fb7)
The model evaluation method being used in this scenario is B. K-fold cross-validation.
Cross-validation is a technique used to assess the performance of a machine learning model. It involves dividing the data into training and validation sets, fitting the model on the training set, and then evaluating its performance on the validation set. Cross-validation helps to estimate the performance of the model on unseen data and helps to avoid overfitting.
In k-fold cross-validation, the data is divided into k subsets of equal size. The model is trained k times, each time using a different subset as the validation set and the remaining subsets as the training set. The final evaluation is then obtained by averaging the performance of the model on each of the k validation sets. This technique helps to reduce the variance of the evaluation by ensuring that each observation in the dataset is used for both training and validation.
In the given scenario, the dataset has been divided into 10 parts, and 10 training runs have been executed, which produces 10 models. This is an example of 10-fold cross-validation, where the dataset is divided into 10 subsets, and the model is trained 10 times, each time using a different subset as the validation set and the remaining subsets as the training set. The final evaluation is obtained by aggregating the performance of the 10 models.
This technique is useful for evaluating the performance of the model on a dataset with a limited number of observations, as it allows for a more accurate estimate of the model's performance. However, it can be computationally expensive, as it involves training and evaluating the model multiple times.
Therefore, the model evaluation method being used in this scenario is k-fold cross-validation, which helps to estimate the performance of the model on unseen data and avoid overfitting.