Abstract

The concept of maximum entropy can be traced back along multiple threads to Biblical times. Only recently, however, have computers become powerful enough to permit the widescale application of this concept to real world problems in statistical estimation and pattern recognition. In this paper, we describe a method for statistical modeling based on maximum entropy. We present a maximum-likelihood approach for automatically constructing maximum entropy models and describe how to implement this approach efficiently, using as examples several problems in natural language processing.

Keywords

Principle of maximum entropyComputer scienceEntropy (arrow of time)Maximum likelihoodMaximum-entropy Markov modelMaximum entropy spectral estimationArtificial intelligenceNatural languageNatural language processingMachine learningStatisticsMathematics

Affiliated Institutions

Related Publications

Black Holes and Entropy

Jacob D. Bekenstein

There are a number of similarities between black-hole physics and thermodynamics. Most striking is the similarity in the behaviors of black-hole area and of entropy: Both quanti...

1973 Physical review. D. Particles, fields... 6767 citations

Publication Info

Year
1996
Type
article
Volume
22
Issue
1
Pages
39-71
Citations
3120
Access
Closed

External Links

View on DOI.org

Social Impact

Altmetric
PlumX Metrics

Social media, news, blog, policy document mentions

Citation Metrics

3120
OpenAlex

Cite This

Adam Berger, Vincent J. Della Pietra, Stephen A. Della Pietra (1996). A maximum entropy approach to natural language processing. Computational Linguistics , 22 (1) , 39-71. https://doi.org/10.5555/234285.234289

Identifiers

DOI
10.5555/234285.234289