Affinitention nets: Kernel perspective on attention architectures for set classification with applications to medical text and images

David Dov, Serge Assaad, Shijing Si, Rui Wang, Hongteng Xu, Shahar Ziv Kovalsky, Jonathan Bell, Danielle Elliott Range, Jonathan Cohen, Ricardo Henao, Lawrence Carin

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Set classification is the task of predicting a single label from a set comprising multiple instances. The examples we consider are pathology slides represented by sets of patches and medical text data represented by sets of word embeddings. State-of-the-art methods, such as the transformer network, typically use attention mechanisms to learn representations of set data, by modeling interactions between instances of the set. These methods, however, have complex heuristic architectures comprising multiple heads and layers. The complexity of attention architectures hampers their training when only a small number of labeled sets is available, as is often the case in medical applications. To address this problem, we present a kernel-based representation learning framework that links learning affinity kernels to learning representations from attention architectures. We show that learning a combination of the sum and the product of kernels is equivalent to learning representations from multi-head multi-layer attention architectures. From our framework, we devise a simplified attention architecture which we term affinitention (affinity-attention) nets. We demonstrate the application of affinitention nets to the classification of the Set-Cifar10 dataset, thyroid malignancy prediction from pathology slides, as well as patient text-message triage. We show that affinitention nets provide competitive results compared to heuristic attention architectures and outperform other competing methods.
Original languageEnglish (US)
Title of host publicationACM CHIL 2021 - Proceedings of the 2021 ACM Conference on Health, Inference, and Learning
PublisherAssociation for Computing Machinery, Inc
Number of pages11
ISBN (Print)9781450383592
StatePublished - Apr 8 2021
Externally publishedYes


Dive into the research topics of 'Affinitention nets: Kernel perspective on attention architectures for set classification with applications to medical text and images'. Together they form a unique fingerprint.

Cite this