Looking for a Few Good Metrics: ROUGE and its Evaluation

Abstract

ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation. It includes measures to automatically determine the quality of a summary by comparing it to other (ideal) summaries created by humans. The measures count the number of overlapping units such as n-gram, word sequences, and word pairs between the computer-generated summary to be evaluated and the ideal summaries created by humans. This paper discusses the validity of the evaluation method used in the Document Understanding Conference (DUC) and evaluates five different ROUGE metrics: ROUGE-N, ROUGE-L, ROUGE-W, ROUGE-S, and ROUGE-SU included in the ROUGE summarization evaluation package using data provided by DUC. A comprehensive study of the effects of using single or multiple references and various sample sizes on the stability of the results is also presented.

Keywords

ROUGEAutomatic summarizationIdeal (ethics)Word (group theory)Computer scienceArtificial intelligenceNatural language processingMathematicsPhilosophy

Related Publications

Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation

David Powers

Commonly used evaluation measures including Recall, Precision, F-Measure and Rand Accuracy are biased and should not be used without clear understanding of the biases, and corre...

2020 arXiv (Cornell University) 4425 citations

An overlapping-feature-based phonological model incorporating linguistic constraints: Applications to speech recognition

Jiping Sun , Li Deng

Modeling phonological units of speech is a critical issue in speech recognition. In this paper, our recent development of an overlapping-feature-based phonological model that re...

2002 The Journal of the Acoustical Society... 71 citations

Exploring the Limits of Transfer Learning with a Unified Text-to-Text\n Transformer

Colin Raffel , Noam Shazeer , Adam Roberts +6 more

Transfer learning, where a model is first pre-trained on a data-rich task\nbefore being fine-tuned on a downstream task, has emerged as a powerful\ntechnique in natural language...

2019 arXiv (Cornell University) 8299 citations

Publication Info

Year: 2004
Type: article
Citations: 109
Access: Closed

External Links

Citation Metrics

109

OpenAlex

Cite This

APA Style

                            
                                    Chin-Yew Lin
                                
                            (2004). 
                            Looking for a Few Good Metrics: ROUGE and its Evaluation. 
                            
                            .