An Overview of Neural NLP Milestones

 

 

Since the advent of word2vec, neural word embeddings have become a goto method for encapsulating distributional semantics in NLP applications. This series will review the strengths and weaknesses of using pre-trained word embeddings and demonstrate how to incorporate more complex semantic representation schemes such as Semantic Role Labeling, Abstract Meaning Representation and Semantic Dependency Parsing into your applications.

Neural Natural Language Processing Milestones

In the past five years the advent of deep learning has brought about some impressive milestones in natural language processing.

The first post of this series will highlight a sample of some of these milestones as well as links for getting started with them.

Co-Reference Resolution

 
Allen NLP Co-Reference resolution

Coreference resolution is the task of finding all expressions that refer to the same entity in a text. It is an important step for hardcore NLP tasks that involve natural language understanding such as document summarization, question answering, and information extraction. Two of the best available libraries for this task are huggingface and Allen NLP.

Sentiment Analysis

 
An example of the Azure Text Analytics service

These models are able to detect the sentiment of a given text. This capability is useful for detecting positive and negative sentiment in social media, customer reviews, and discussion forums. In cloud based services such as Azure Text Analytics content is provided by you; models and training data are provided by the service.

Trainable Named Entity Recognition and Span Tagging

 

Named entity recognition model extract entities such as people, locations, organizations, and miscellaneous from text. Azure Text Analytics provides powerful pre-trained models. For those who prefer to train their own models both Spacy and Allen NLP provide documentation for quickly training custom NER models that can be used with the Azure ML API and Azure Notebook Service.

Intent Classification

 

Designed to identify valuable information in conversations, Intent classification systems such as the LUIS cognitive service interprets user goals (intents) and distills valuable information from sentences (entities), for a high quality, nuanced language model. LUIS integrates seamlessly with the Azure Bot Service, making it easy to create a sophisticated bot.

Machine Reading Comprehension BiDAF

 
AllenNLP MC Model

Machine Comprehension (MC) answers natural language questions by selecting an answer span within an evidence text. The AllenNLP toolkit provides the following MC visualization, which can be used for any MC model in AllenNLP. For more information about machine reading comprehension see and getting started there is a great demo with Azure DLVM which contains implementations of some of the state of the art architectures such as BiDAF, SynNet, OpenNMT, and Document QA.

Natural Language Inference SLNI Decomposable Attention

 
Allen NLP NLI/TE model

Textual Entailment (TE) takes a pair of sentences and predicts whether the facts in the first necessarily imply the facts in the second one. Textual Entailment is a critical component of information extraction systems and is often used for filtering candidate answers by ensuring they entail a given extraction query. The AllenNLP toolkit provides the above TE visualization, which can be run for any TE model you develop as well as a pre-trained model that can be used in your applications.

Neural Translation NMT

 

Continuous improvements to translation are important. However, performance improvements have plateaued with SMT technology since the mid-2010s. By leveraging the scale and power of Microsoft’s AI supercomputer, tools like Microsoft Translator now offers neural based translation that enables a new decade of translation quality improvement.

Abstractive Single Document Summarization

 
Example of improvements in neural summarization Abigail See et. all

Neural NLP has also contributed to benchmarks in the task of abstractive single document summarization. This task enables users to generate short summaries of long documents traditonally this was done using extractive methods that generated a summary by taking direct quotes from the text but advances in neural NLP such as Abigail See’s pointer generator networks have enabled progress in creating comprehensible short abstractive summaries from long documents.

Call To Action

Below are some resources to get started with the tools and models above.

  • Azure Text Analytics — The Text Analytics API is a cloud-based service that provides advanced natural language processing over raw text, and includes four main functions: sentiment analysis, key phrase extraction, language detection, and entity linking.
  • Azure LUIS — A machine learning-based service to build natural language for custom intent classification and basic entity extraction.
  • Allen NLP — AllenNLP is an open-source NLP research library, built on PyTorch for developing state-of-the-art deep learning models on a wide variety of linguistic tasks.
  • Azure DLVM Machine Reading Comprehension- The Deep Learning Virtual Machine is a specially configured variant of the Data Science Virtual Machine (DSVM) to make it more straightforward to use GPU-based VM instances for training deep learning models. There is a great module for getting started with pyTorch on the DSVM that you can find here.
  • SpaCy — spaCy is a free open-source library for Natural Language Processing in Python. It features NER, POS tagging, dependency parsing, word vectors and more.
  • HuggingFace- Provide extensions of Spacy for models such as co-reference and sentiment analysis.

Next Post

The next post in this series will review how advancements in representing and modeling with word vectors, has contributed to the progress driving the improvements in NLP systems outlined above.

If you have any questions, comments, or topics you would like me to discuss feel free to follow me on Twitter if there is a milestone you feel I missed please let me know.

Article written by : Aaron (Ari) Bornstein (source)

About the Author

Aaron (Ari) Bornstein is an avid AI enthusiast with a passion for history, engaging with new technologies and computational medicine. As an Open Source Engineer at Microsoft’s Cloud Developer Advocacy team, he collaborates with Israeli Hi-Tech Community, to solve real world problems with game changing technologies that are then documented, open sourced, and shared with the rest of the world.