Optimizing Vision Transformers: Leveraging Max and Min Operations for Efficient Pruning

Philippe Bich*, Chiara Boretti, Luciano Prono, Fabio Pareschi, Riccardo Rovatti, Gianluca Setti

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

The research on Deep Neural Networks (DNNs) continues to enhance the performance of these models over a wide spectrum of tasks, increasing their adoption in many fields. This leads to the need of extending their usage also on edge devices with limited resources, even though, with the advent of Transformer-based models, this has become an increasingly complex task because of their size. In this context, pruning emerges as a crucial tool to reduce the number of weights in the memory-hungry Fully Connected (FC) layers. This paper explores the usage of neurons based on the Multiply-And-Max/min (MAM) operation, an alternative to the conventional Multiply-and-Accumulate (MAC), in a Vision Transformer (ViT). This enhances the model prunability thanks to the usage of Max and Min operations. For the first time, many MAM-based FC layers are used in a large state-of-the-art DNN model and compressed with various pruning techniques available in the literature. Experiments show that MAM-based layers achieve the same accuracy of traditional layers using up to 12 times less weights. In particular, when using Global Magnitude Pruning (GMP), the FC layers following the Multi-head Attention block of a ViT-B/16 model, fine-tuned on CIFAR-100, count only 560000 weights if MAM neurons are used, compared to the 31.4 million that remain when using traditional MAC neurons.

Original languageEnglish (US)
Title of host publication2024 IEEE 6th International Conference on AI Circuits and Systems, AICAS 2024 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages337-341
Number of pages5
ISBN (Electronic)9798350383638
DOIs
StatePublished - 2024
Event6th IEEE International Conference on AI Circuits and Systems, AICAS 2024 - Abu Dhabi, United Arab Emirates
Duration: Apr 22 2024Apr 25 2024

Publication series

Name2024 IEEE 6th International Conference on AI Circuits and Systems, AICAS 2024 - Proceedings

Conference

Conference6th IEEE International Conference on AI Circuits and Systems, AICAS 2024
Country/TerritoryUnited Arab Emirates
CityAbu Dhabi
Period04/22/2404/25/24

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Vision and Pattern Recognition
  • Hardware and Architecture
  • Electrical and Electronic Engineering
  • Instrumentation

Fingerprint

Dive into the research topics of 'Optimizing Vision Transformers: Leveraging Max and Min Operations for Efficient Pruning'. Together they form a unique fingerprint.

Cite this