Handling Missing Categorical Data in Machine Learning Models
Question
You work for a financial services company where you are building a model to analyze equity futures prices to predict price movement for your firm's hedging strategy.
You receive several data feeds, some of which contain missing values for some data points.
The missing data points in your data feeds are of the categorical type, such as the expiration month or the exchange on which the futures contract is traded.
Which strategy should you employ to deal with the missing data point values while attempting to maximize the accuracy of your model without introducing bias into the model?
Answers
Explanations
Click on the arrows to vote for the correct answer
A. B. C. D.Answer: D.
Option A is incorrect because this approach will lead to the loss of data points with potentially useful information.
Option B is incorrect because, by definition, it can only be used with numeric data.
It is not advisable to use the Mean/Median approach with categorical data points.
Option C is incorrect because while working with categorical data, this method can introduce bias into your data.
Option D is correct.
Using a library such as the datawig python library, a deep learning approach uses deep neural networks to impute missing data values.
This is the most accurate strategy, in the list of given options, at imputing categorical values.
(See the datawig documentation: https://github.com/awslabs/datawig)
Reference:
Please see the article 6 Different Ways to Compensate for Missing Values In a Dataset (Data Imputation with examples)
When dealing with missing data in machine learning, it's essential to choose a strategy that maximizes the accuracy of the model while avoiding introducing bias.
In this scenario, the missing data points are of the categorical type, which means they are values that cannot be calculated or predicted based on other data in the dataset. Therefore, removing the observations that have missing data (Option A) is not the best strategy since it could lead to the loss of valuable information and the reduction of the dataset size, which can harm the model's accuracy.
Imputing the missing values using the Mean/Median strategy (Option B) is a viable option when dealing with continuous numerical data. However, since the missing values in this case are categorical, this strategy is not suitable.
Imputing the missing values using the Most Frequent strategy (Option C) is the recommended strategy when dealing with categorical data. This strategy involves replacing missing values with the most common value in the dataset. This approach ensures that the data distribution is maintained, and the model's accuracy is not affected. Additionally, it's an efficient way to deal with missing values in a large dataset, and it can be easily implemented using Python libraries like Scikit-learn.
Using a Deep Learning strategy (Option D) to impute missing values is an advanced technique that involves training a neural network to predict missing values. While this approach can be effective, it's not necessary in this scenario since there is a straightforward solution that can produce good results without introducing additional complexity to the model.
Therefore, the best strategy for dealing with the missing categorical data points in this scenario is to impute the missing values using the Most Frequent strategy (Option C).
