Syntactic Knowledge-Infused Transformer and BERT models

Dhanasekar Sundararaman*, Vivek Subramanian, Guoyin Wang, Shijing Si, Dinghan Shen, Dong Wang, Lawrence Carin

*Corresponding author for this work

Research output: Contribution to journalConference articlepeer-review

4 Scopus citations


Attention-based deep learning models have demonstrated significant improvement over traditional algorithms in several NLP tasks. The Transformer, for instance, is an illustrative example that generates abstract representations of tokens that are input to an encoder based on their relationships to all tokens in a sequence. While recent studies have shown that such models are capable of learning syntactic features purely by seeing examples, we hypothesize that explicitly feeding this information to deep learning models can significantly enhance their performance in many cases. Leveraging syntactic information like part of speech (POS) may be particularly beneficial in limited-training-data settings for complex models such as the Transformer. In this paper, we verify this hypothesis by infusing syntactic knowledge into the Transformer. We find that this syntax-infused Transformer achieves an improvement of 0.7 BLEU when trained on the full WMT'14 English to German translation dataset and a maximum improvement of 1.99 BLEU points when trained on a fraction of the dataset. In addition, we find that the incorporation of syntax into BERT fine-tuning outperforms BERTBASE on all downstream tasks from the GLUE benchmark, including an improvement of 0.8% on CoLA.

Original languageEnglish (US)
JournalCEUR Workshop Proceedings
StatePublished - 2021
Event2021 International Conference on Information and Knowledge Management Workshops, CIKMW 2021 - Gold Coast, Australia
Duration: Nov 1 2021Nov 5 2021


  • Semantics
  • Syntax
  • Transformers

ASJC Scopus subject areas

  • General Computer Science


Dive into the research topics of 'Syntactic Knowledge-Infused Transformer and BERT models'. Together they form a unique fingerprint.

Cite this