As a machine learning engineer, how do you avoid the curse of dimensionality in your designs?

How to Answer
1st Answer Example
2nd Answer Example

To effectively answer this question, it is important to first understand what dimensionality means in reference to machine learning and how it can curse a project. As the number of features increases in comparison to the number of observations within a data set, some algorithms struggle in pulling correct data. Your job on this question is to talk about ways that you can avoid the curse in your designs. Some possible things that you may mention and describe are feature selection, correlation thresholds and variance thresholds.

"A common example that I use with people to explain complications in dimensionality is dropping a pin on a 10 foot straight line. This would be relatively simple to find. Next, if you dropped the pin in a 10 foot by 10 foot square, the task of finding the pin becomes more difficult. Adding a third dimension to make a 10 foot cubed area makes it all the more difficult to find the pin if placed within it. In bringing this back to machine learning, my job is to somehow make the three dimensional field that the machine will pull from easier to pull from. Last year, I was part of a team that developed a system for pulling public health data. We were able to set many variance thresholds that removed values that didn't change much from observation to observation. After careful testing, the system was able to pull information quickly and accurately based on these thresholds."

"As you can see from my resume, I've spent the last six years working in the electronics industry. Most of my machine learning work has focused audio data. To avoid the curse of dimensionality within the systems I've designed, autoencoders have been tremendous in pulling information. While a great amount of time and effort was needed to effectively train the systems, the work was well worth it in the end."

Next Question

How to Answer: As a machine learning engineer, how do you avoid the curse of dimensionality in your designs?

Advice and answer examples written specifically for an OutSystems job interview.

17. As a machine learning engineer, how do you avoid the curse of dimensionality in your designs?

How to Answer

To effectively answer this question, it is important to first understand what dimensionality means in reference to machine learning and how it can curse a project. As the number of features increases in comparison to the number of observations within a data set, some algorithms struggle in pulling correct data. Your job on this question is to talk about ways that you can avoid the curse in your designs. Some possible things that you may mention and describe are feature selection, correlation thresholds and variance thresholds.

1st Answer Example

"A common example that I use with people to explain complications in dimensionality is dropping a pin on a 10 foot straight line. This would be relatively simple to find. Next, if you dropped the pin in a 10 foot by 10 foot square, the task of finding the pin becomes more difficult. Adding a third dimension to make a 10 foot cubed area makes it all the more difficult to find the pin if placed within it. In bringing this back to machine learning, my job is to somehow make the three dimensional field that the machine will pull from easier to pull from. Last year, I was part of a team that developed a system for pulling public health data. We were able to set many variance thresholds that removed values that didn't change much from observation to observation. After careful testing, the system was able to pull information quickly and accurately based on these thresholds."

2nd Answer Example

"As you can see from my resume, I've spent the last six years working in the electronics industry. Most of my machine learning work has focused audio data. To avoid the curse of dimensionality within the systems I've designed, autoencoders have been tremendous in pulling information. While a great amount of time and effort was needed to effectively train the systems, the work was well worth it in the end."

Next Question