Abstract
Incorporating feature selection into a classification or regression method often carries a number of advantages. In this paper we formalize feature selection specifically from a discriminative perspective of improving classification/regression accuracy. The feature selection method is developed as an extension to the recently proposed maximum entropy discrimination (MED) framework. We describe MED as a flexible (Bayesian) regularization approach that subsumes, e.g., support vector classification, regression and exponential family models. For brevity, we restrict ourselves primarily to feature selection in the context of linear classification/regression methods and demonstrate that the proposed approach indeed carries substantial improvements in practice. Moreover, we discuss and develop various extensions of feature selection, including the problem of dealing with example specific but unobserved degrees of freedom -- alignments or invariants.
Keywords
Feature selectionDiscriminative modelArtificial intelligenceComputer sciencePrinciple of maximum entropyPattern recognition (psychology)Machine learningEntropy (arrow of time)RegressionFeature (linguistics)Support vector machineMathematicsStatistics
Affiliated Institutions
Related Publications
Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy
Feature selection is an important problem for pattern classification systems. We study how to select good features according to the maximal statistical dependency criterion base...
Feature selection: evaluation, application, and small sample performance
A large number of algorithms have been proposed for feature subset selection. Our experimental results show that the sequential forward floating selection algorithm, proposed by...
Feature selection for high-dimensional genomic microarray data
We report on the successful application of feature selection methods to a classification problem in molecular biology involving only 72 data points in a 7130 dimensional space. ...
Feature selection for multiclass discrimination via mixed-integer linear programming
We reformulate branch-and-bound feature selection employing L/sub /spl infin// or particular L/sub p/ metrics, as mixed-integer linear programming (MILP) problems, affording con...
Kernel Logistic Regression and the Import Vector Machine
The support vector machine (SVM) is known for its good performance in two-class classification, but its extension to multiclass classification is still an ongoing research issue...
Publication Info
- Year
- 2013
- Type
- article
- Pages
- 291-300
- Citations
- 76
- Access
- Closed
External Links
Social Impact
Altmetric
PlumX Metrics
Social media, news, blog, policy document mentions
Citation Metrics
76
OpenAlex
Cite This
Tony Jebara,
Tommi Jaakkola
(2013).
Feature Selection and Dualities in Maximum Entropy Discrimination.
arXiv (Cornell University)
, 291-300.
https://doi.org/10.48550/arxiv.1301.3865
Identifiers
- DOI
- 10.48550/arxiv.1301.3865