You are a machine learning specialist working for a polling company where you have been given the assignment of creating a machine learning model that predicts voter turnout in various voting districts across the voting population.
You have created a deep learning neural network model for your predictions.
The model performs well on the training data, 99% accuracy, but it performs poorly on the test data, 70% accuracy.
Which techniques can you leverage to address this overfitting situation with your deep learning model? (Select TWO)
Click on the arrows to vote for the correct answer
A. B. C. D. E.Correct Answers: C and D.
Option A is incorrect.
Your model is overfitting.
You should decrease the feature combinations (dimensionality) using a technique such as Principal Component Analysis (PCA) which will help with overfitting.
Increasing feature combinations, or increasing the feature dimensionality, would make the model have the opposite effect.
Option B is incorrect.
With an overfitting model, you need to increase your dropout rate to help the model reach generalization.
Decreasing dropout will have the opposite effect.
Option C is correct.
Increasing the dropout rate in your deep learning model is a proven technique to address the issue of training data overfitting.
Option D is correct.
Increasing regularization through L1 regularization, L2 regularization, or dropout helps lower the complexity of the model to help address overfitting.
Option E is incorrect.
Decreasing regularization will increase the complexity of your model and will not help address overfitting.
References:
Please see the Machine Learning Mastery article titled A Gentle Introduction to Dropout for Regularizing Deep Neural Networks (https://machinelearningmastery.com/dropout-for-regularizing-deep-neural-networks/),
The Towards Data Science article titled Regularization in Deep Learning - L1, L2, and Dropout (https://towardsdatascience.com/regularization-in-deep-learning-l1-l2-and-dropout-377e75acc036),
The Amazon SageMaker developer guide titled Principal Component Analysis (PCA) Algorithm (https://docs.aws.amazon.com/sagemaker/latest/dg/pca.html)
The situation described in the question is overfitting, which means that the model has learned the training data too well and is not able to generalize to new data. Overfitting is a common problem in deep learning models and can be addressed with several techniques, as listed below:
A. Increasing feature dimensionality: This option is unlikely to address overfitting. Increasing the feature dimensionality means adding more features to the model, which may result in better performance on the training data. However, it can also increase the complexity of the model and exacerbate overfitting. Therefore, this option is not the best choice for addressing overfitting.
B. Lowering the dropout rate: Dropout is a regularization technique used to prevent overfitting. It randomly drops out a fraction of the neurons in the model during training, which forces the remaining neurons to learn more robust features. Lowering the dropout rate can lead to overfitting since it allows the model to memorize the training data more easily. Therefore, this option is not recommended for addressing overfitting.
C. Increasing the dropout rate: As mentioned earlier, dropout is a regularization technique used to prevent overfitting. Increasing the dropout rate can help the model to learn more robust features and prevent it from memorizing the training data. Therefore, this option is a good choice for addressing overfitting.
D. Increasing L1 and/or L2 regularization amount: L1 and L2 regularization are techniques used to prevent overfitting by adding a penalty term to the loss function. L1 regularization encourages the model to learn sparse features by adding the absolute value of the weights to the loss function, while L2 regularization encourages the model to learn small weights by adding the squared weights to the loss function. Increasing the L1 and/or L2 regularization amount can help to prevent overfitting by constraining the model's weights. Therefore, this option is a good choice for addressing overfitting.
E. Decreasing L1 and/or L2 regularization amount: Decreasing the L1 and/or L2 regularization amount can lead to overfitting since it reduces the penalty on the weights, allowing the model to memorize the training data more easily. Therefore, this option is not recommended for addressing overfitting.
In summary, the two best techniques for addressing overfitting in a deep learning model are increasing the dropout rate and increasing the L1 and/or L2 regularization amount.