Parsing Natural Scenes and Natural Language with Recursive Neural Networks

Richard Socher; Cliff Chiung-Yu Lin; Christopher D. Manning; Andrew Y. Ng

Abstract

Recursive structure is commonly found in the inputs of different modalities such as natural scene images or natural language sentences. Discovering this recursive structure helps us to not only identify the units that an image or sentence contains but also how they interact to form a whole. We introduce a max-margin structure prediction architecture based on recursive neural networks that can successfully recover such structure both in complex scene images as well as sentences. The same algorithm can be used both to provide a competitive syntactic parser for natural language sentences from the Penn Treebank and to outperform alternative approaches for semantic scene segmentation, annotation and classification. For segmentation and annotation our algorithm obtains a new level of state-of-theart performance on the Stanford background dataset (78.1%). The features from the image parse tree outperform Gist descriptors for scene classification by 4%. 1.

Keywords

TreebankComputer scienceParsingArtificial intelligenceMargin (machine learning)Natural languageNatural language processingSegmentationSentenceImage segmentationSemantics (computer science)Artificial neural networkAnnotationTree (set theory)Recurrent neural networkPattern recognition (psychology)Machine learningProgramming language

Affiliated Institutions

Stanford University US

Related Publications

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Richard Socher , Alex Perelygin , Jean Y. Wu +4 more

Semantic word spaces have been very useful but cannot express the meaning of longer phrases in a principled way. Further progress towards understanding compositionality in tasks...

2013 6566 citations

DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding

Tao Shen , Tianyi Zhou , Guodong Long +3 more

Recurrent neural nets (RNN) and convolutional neural nets (CNN) are widely used on NLP tasks to capture the long-term and local dependencies, respectively. Attention mechanisms ...

2018 Proceedings of the AAAI Conference on... 729 citations

Scene labeling with LSTM recurrent neural networks

Wonmin Byeon , Thomas M. Breuel , Federico Raue +1 more

This paper addresses the problem of pixel-level segmentation and classification of scene images with an entirely learning-based approach using Long Short Term Memory (LSTM) recu...

2015 401 citations

Supervised Learning of Universal Sentence Representations from Natural\n Language Inference Data

Alexis Conneau , Douwe Kiela , Holger Schwenk +2 more

Many modern NLP systems rely on word embeddings, previously trained in an\nunsupervised manner on large corpora, as base features. Efforts to obtain\nembeddings for larger chunk...

2017 arXiv (Cornell University) 2038 citations

Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars

Luke Zettlemoyer , Michael J. Collins

This paper addresses the problem of mapping natural language sentences to lambda-calculus encodings of their meaning. We describe a learning algorithm that takes as input a trai...

2012 arXiv (Cornell University) 791 citations

Publication Info

Year: 2011
Type: article
Pages: 129-136
Citations: 1202
Access: Closed

External Links

Citation Metrics

1202

OpenAlex

Cite This

APA Style

                            
                                    Richard Socher, 
                                
                                    Cliff Chiung-Yu Lin, 
                                
                                    Christopher D. Manning
                                
                                et al.
                            
                            (2011). 
                            Parsing Natural Scenes and Natural Language with Recursive Neural Networks. 
                            
                            , 129-136.