Abstract
We introduce a constructive, incremental learning system for regression problems that models data by means of locally linear experts. In contrast to other approaches, the experts are trained independently and do not compete for data during learning. Only when a prediction for a query is required do the experts cooperate by blending their individual predictions. Each expert is trained by minimizing a penalized local cross validation error using second order methods. In this way, an expert is able to find a local distance metric by adjusting the size and shape of the receptive field in which its predictions are valid, and also to detect relevant input features by adjusting its bias on the importance of individual input dimensions. We derive asymptotic results for our method. In a variety of simulations the properties of the algorithm are demonstrated with respect to interference, learning speed, prediction accuracy, feature detection, and task oriented incremental learning.
Keywords
Computer scienceBoosting (machine learning)Machine learningArtificial intelligenceDecision treeContrast (vision)Gradient boostingConstructiveMetric (unit)Field (mathematics)Feature (linguistics)RegressionData miningMathematicsStatisticsRandom forestProcess (computing)
Affiliated Institutions
Related Publications
Neural Network for Graphs: A Contextual Constructive Approach
This paper presents a new approach for learning in structured domains (SDs) using a constructive neural network for graphs (NN4G). The new model allows the extension of the inpu...
Chapter 18 Committees of decision trees
Many intelligent systems are designed to sift through a mass of evidence and arrive at a decision. Certain pieces of evidence may be given more weight than others, and this may ...
A Communication-Efficient Parallel Algorithm for Decision Tree
Decision tree (and its extensions such as Gradient Boosting Decision Trees and Random Forest) is a widely used machine learning algorithm, due to its practical effectiveness and...
Bagging, boosting, and C4.S
Breiman's bagging and Freund and Schapire's boosting are recent methods for improving the predictive power of classifier learning systems. Both form a set of classifiers that ar...
Assessing the limits of genomic data integration for predicting protein networks
Genomic data integration—the process of statistically combining diverse sources of information from functional genomics experiments to make large-scale predictions—is becoming i...
Publication Info
- Year
- 1995
- Type
- article
- Volume
- 8
- Pages
- 479-485
- Citations
- 231
- Access
- Closed
External Links
Citation Metrics
231
OpenAlex
Cite This
Harris Drucker,
Corinna Cortes
(1995).
Boosting Decision Trees.
Neural Information Processing Systems
, 8
, 479-485.