You work for an online retail company that is creating a visual search engine.
You have set up an end-to-end ML pipeline on Google Cloud to classify whether an image contains your company's product.
Expecting the release of new products in the near future, you configured a retraining functionality in the pipeline so that new data can be fed into your ML models.
You also want to use AI Platform's continuous evaluation service to ensure that the models have high accuracy on your test dataset.
What should you do?
Click on the arrows to vote for the correct answer
A. B. C. D.C.
As you are expecting the release of new products in the near future, you want to make sure that your image classification model continues to perform well even when the dataset is updated with new products. To achieve this, you have set up a retraining functionality in your ML pipeline so that new data can be fed into your ML models.
To ensure that the models have high accuracy on your test dataset, you want to use AI Platform's continuous evaluation service. This service can automatically evaluate your model on a specified test dataset and provide feedback on its performance. Based on this feedback, you can decide whether to keep the current model or retrain it with new data.
Now, let's consider the options provided in the question and their implications:
A. Keep the original test dataset unchanged even if newer products are incorporated into retraining.
This option is not ideal because the original test dataset may not represent the new products well. As a result, the model's performance may degrade when it encounters the new products. Keeping the original test dataset unchanged may lead to overfitting, where the model performs well on the original test dataset but poorly on new data.
B. Extend your test dataset with images of the newer products when they are introduced to retraining.
This option is a good approach as it allows the model to be evaluated on a test dataset that is representative of the current data distribution. By extending the test dataset with images of the newer products, the model can be evaluated on how well it generalizes to the new products.
C. Replace your test dataset with images of the newer products when they are introduced to retraining.
This option is not recommended because it completely discards the original test dataset. The original test dataset may still be relevant for evaluating the model's performance on older products. Replacing the test dataset with only images of the newer products may not provide a comprehensive evaluation of the model's performance.
D. Update your test dataset with images of the newer products when your evaluation metrics drop below a pre-decided threshold.
This option is also a good approach because it ensures that the model is evaluated on the most current test dataset when its performance drops below a pre-decided threshold. This approach helps to maintain the model's accuracy on new products and prevent overfitting.
In conclusion, option B or D would be the best approach. It is important to ensure that the test dataset is updated with new products to evaluate the model's performance on the most current data distribution.