Techniques to Correct Overfitting and Reduce Model Complexity
Question
You are a machine learning specialist working for a healthcare company where you are building a cancer detection model using a linear regression algorithm.
You have gathered your data of hundreds of thousands of patients with over 100 features.
However, when you train your model, you notice that it appears to be over-fitting your data. Which technique can you use to simultaneously correct the over-fitting and reduce your model complexity by removing less relevant features?
Answers
Explanations
Click on the arrows to vote for the correct answer
A. B. C. D.Answer: B.
Option A is incorrect.
The Ridge Regression approach would reduce the coefficients in your model but not all the way to 0
Therefore, it reduces complexity, but does not entirely eliminate any of the over 100 features in your data.
Option B is correct.
The Lasso Regression approach would reduce some of the coefficients in your model to zero, effectively eliminating some of the over 100 features in your data.
This will effectively reduce the complexity of your model.
Option C is incorrect.
The Stochastic Gradient Descent approach can use a regularization parameter, but it cannot be used to eliminate features from your dataset.
Option D is incorrect.
The Gaussian Process approach is used for regression problems, but it does not work well with high dimensional datasets, i.e., over a few dozen features.
Your dataset has over 100 features.
Also, it cannot be used to eliminate features from your dataset.
Reference:
Please see the Medium article titled Ridge and Lasso Regression: L1 and L2 Regularization (https://towardsdatascience.com/ridge-and-lasso-regression-a-complete-guide-with-python-scikit-learn-e20e34bcbf0b), and the SciKit Learn page titled 1
Supervised learning (https://scikit-learn.org/stable/supervised_learning.html#supervised-learning)
The correct answer is B. Use Lasso Regression.
Overfitting is a common problem in machine learning models where the model fits the training data too closely, resulting in poor generalization to new, unseen data. One of the main causes of overfitting is a high model complexity, which occurs when the model has too many features or too many parameters.
To reduce overfitting and simplify the model, you can use regularization techniques. Regularization adds a penalty term to the cost function that the model tries to minimize during training. This penalty term discourages the model from fitting the data too closely and can reduce the complexity of the model by shrinking the weights of less important features.
Ridge regression and Lasso regression are two popular regularization techniques. Ridge regression adds a penalty term proportional to the square of the L2 norm of the weight vector. This penalty term forces the weights to be small, but does not usually result in any weights being exactly zero. Lasso regression, on the other hand, adds a penalty term proportional to the absolute value of the L1 norm of the weight vector. This penalty term has a sparsity-inducing effect, which can result in some weights being exactly zero. Therefore, Lasso regression can be used to select features and simultaneously reduce overfitting and model complexity.
In this scenario, using Lasso regression would be the best technique to correct overfitting and reduce the model complexity by removing less relevant features. Since there are over 100 features, Lasso regression can select the most relevant features while shrinking the weights of less important features towards zero. This can improve the generalization performance of the model on new data.
