Supervised Cross-Modal Factor Analysis for Multiple Modal Data Classification

Jingbin Wang, Yihua Zhou, Kanghong Duan, Jim Jing-Yan Wang, Halima Bensmail

Research output: Chapter in Book/Report/Conference proceedingConference contribution

16 Scopus citations


In this paper we study the problem of learning from multiple modal data for purpose of document classification. In this problem, each document is composed two different modals of data, i.e., An image and a text. Cross-modal factor analysis (CFA) has been proposed to project the two different modals of data to a shared data space, so that the classification of a image or a text can be performed directly in this space. A disadvantage of CFA is that it has ignored the supervision information. In this paper, we improve CFA by incorporating the supervision information to represent and classify both image and text modals of documents. We project both image and text data to a shared data space by factor analysis, and then train a class label predictor in the shared space to use the class label information. The factor analysis parameter and the predictor parameter are learned jointly by solving one single objective function. With this objective function, we minimize the distance between the projections of image and text of the same document, and the classification error of the projection measured by hinge loss function. The objective function is optimized by an alternate optimization strategy in an iterative algorithm. Experiments in two different multiple modal document data sets show the advantage of the proposed algorithm over other CFA methods.
Original languageEnglish (US)
Title of host publication2015 IEEE International Conference on Systems, Man, and Cybernetics
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Number of pages7
ISBN (Print)9781479986972
StatePublished - Jan 15 2016


Dive into the research topics of 'Supervised Cross-Modal Factor Analysis for Multiple Modal Data Classification'. Together they form a unique fingerprint.

Cite this