Comments on: Choosing the right features for Data Mining http://blog.markus-breitenbach.com/2007/06/01/choosing-the-right-features-for-data-mining/ AI, Data Mining, Machine Learning and other things Sun, 20 Jul 2008 23:40:39 +0000 http://wordpress.org/?v=2.2.1 By: Will Dwinnell http://blog.markus-breitenbach.com/2007/06/01/choosing-the-right-features-for-data-mining/#comment-2931 Will Dwinnell Mon, 11 Jun 2007 20:12:11 +0000 http://blog.markus-breitenbach.com/2007/06/01/choosing-the-right-features-for-data-mining/#comment-2931 This is indeed an interesting subject, both from intellectual and practical perspectives. I've been using two heuristics borrowed from Weiss and Indurkhya's "Predictive Data Mining". The first step excludes variables which seem obviously useless. Sometimes this step removes few variables, but sometimes it removes many, which helps reduce the computational workload down-stream. The second step uses a simple Mahalanobis-like distance between classes to gauge the predictive power of any given set of predictors. I hooked up a genetic algorithm to this heuristic, to search for optimal sets of predictors. This is indeed an interesting subject, both from intellectual and practical perspectives. I’ve been using two heuristics borrowed from Weiss and Indurkhya’s “Predictive Data Mining”.

The first step excludes variables which seem obviously useless. Sometimes this step removes few variables, but sometimes it removes many, which helps reduce the computational workload down-stream.

The second step uses a simple Mahalanobis-like distance between classes to gauge the predictive power of any given set of predictors. I hooked up a genetic algorithm to this heuristic, to search for optimal sets of predictors.

]]> By: Olivier Bousquet http://blog.markus-breitenbach.com/2007/06/01/choosing-the-right-features-for-data-mining/#comment-2770 Olivier Bousquet Sat, 02 Jun 2007 21:12:12 +0000 http://blog.markus-breitenbach.com/2007/06/01/choosing-the-right-features-for-data-mining/#comment-2770 Interesting post. However, I would tend to disagree on the last paragraph, for at least two reasons: 1) you can see a recommender system as a prediction problem with features: indeed, what you are trying to predict is whether someone will like a certain kind of music and your features are the id of the person and the style of music. So you can consider that your data consists of 2 features (which can take their values in large but finite sets). The specificity is that you use some sort of transitive relation to build your model (eg. if you have (user=123, music=jazz, +) and (user=123, music=classical, +) and (user=245, music=jazz, +) in your database, you can predict a + for (user=245, music=classical)) 2) you can probably do a better recommendation if you use features. Indeed, in the case you do not have an exact match, the recommender system may not do a good job, while if you had some additional features or information, you might solve the problem. Let me give an example: if instead of music style you try to predict whether someone likes a specific artist/band. You can find out that some people who like Mozart also like Duke Ellington, and you have a user who likes Charlie Parker. The question is whether this person likes Mozart. With a purely link-based recommender system, you cannot really answer, but if you have the additional feature "music style" you might recognize that Charlie Parker and Duke Ellington are jazz artists and thus draw a connection (of course if you had much more data you might also find out this connection automatically, but this is just an example of how data can substitute for prior knowledge). So the additional features (which introduce a structure or distance on your initial features [user, artist]) may provide useful information that helps you make better decisions with little data. 3) any prediction problem from features can be converted into a generalized recommender problem: indeed, if you have a problem with features X,Y,Z,T you can simply consider for example the pair (X,Y) as the 'user' and the pair (Z,T) as the 'product'. You may as well consider any other split into two groups of your features. You can also consider splits into 3 groups or more (thus generalizing the concept), and it is relatively easy to generalize some of the algorithms used in recommender systems to more than 2 'matrix dimensions'. Yet another way would be to split into user X, with feature Y and 'product' Z with feature T... So as a conclusion, I would really not distinguish recommender systems and vector-based prediction problems as they can be converted into one another, so I do not think there is any way to 'escape' the need of finding right features! Interesting post.
However, I would tend to disagree on the last paragraph, for at least two reasons:
1) you can see a recommender system as a prediction problem with features: indeed, what you are trying to predict is whether someone will like a certain kind of music and your features are the id of the person and the style of music. So you can consider that your data consists of 2 features (which can take their values in large but finite sets). The specificity is that you use some sort of transitive relation to build your model (eg. if you have (user=123, music=jazz, +) and (user=123, music=classical, +) and (user=245, music=jazz, +) in your database, you can predict a + for (user=245, music=classical))
2) you can probably do a better recommendation if you use features. Indeed, in the case you do not have an exact match, the recommender system may not do a good job, while if you had some additional features or information, you might solve the problem. Let me give an example: if instead of music style you try to predict whether someone likes a specific artist/band. You can find out that some people who like Mozart also like Duke Ellington, and you have a user who likes Charlie Parker. The question is whether this person likes Mozart. With a purely link-based recommender system, you cannot really answer, but if you have the additional feature “music style” you might recognize that Charlie Parker and Duke Ellington are jazz artists and thus draw a connection (of course if you had much more data you might also find out this connection automatically, but this is just an example of how data can substitute for prior knowledge). So the additional features (which introduce a structure or distance on your initial features [user, artist]) may provide useful information that helps you make better decisions with little data.
3) any prediction problem from features can be converted into a generalized recommender problem: indeed, if you have a problem with features X,Y,Z,T you can simply consider for example the pair (X,Y) as the ‘user’ and the pair (Z,T) as the ‘product’. You may as well consider any other split into two groups of your features. You can also consider splits into 3 groups or more (thus generalizing the concept), and it is relatively easy to generalize some of the algorithms used in recommender systems to more than 2 ‘matrix dimensions’. Yet another way would be to split into user X, with feature Y and ‘product’ Z with feature T…

So as a conclusion, I would really not distinguish recommender systems and vector-based prediction problems as they can be converted into one another, so I do not think there is any way to ‘escape’ the need of finding right features!

]]>