As a machine learning engineer, how do you avoid the curse of dimensionality in your designs?
To effectively answer this question, it is important to first understand what dimensionality means in reference to machine learning and how it can curse a project. As the number of features increases in comparison to the number of observations within a data set, some algorithms struggle in pulling correct data. Your job on this question is to talk about ways that you can avoid the curse in your designs. Some possible things that you may mention and describe are feature selection, correlation thresholds and variance thresholds.
"A common example that I use with people to explain complications in dimensionality is dropping a pin on a 10 foot straight line. This would be relatively simple to find. Next, if you dropped the pin in a 10 foot by 10 foot square, the task of finding the pin becomes more difficult. Adding a third dimension to make a 10 foot cubed area makes it all the more difficult to find the pin if placed within it. In bringing this back to machine learning, my job is to somehow make the three dimensional field that the machine will pull from easier to pull from. Last year, I was part of a team that developed a system for pulling public health data. We were able to set many variance thresholds that removed values that didn't change much from observation to observation. After careful testing, the system was able to pull information quickly and accurately based on these thresholds."
