Mindstorms in natural language-based societies of mind

Mingchen Zhuge; Haozhe Liu; Francesco Faccio; Dylan R. Ashley; Robert Csordás; Anand Gopalakrishnan; Abdullah Hamdi; Hasan Abed Al Kader Hammoud; Vincent Herrmann; Kazuki Irie; Louis Kirsch; Bing Li; Guohao Li; Shuming Liu; Jinjie Mai; Piotr Piȩkos; Aditya A. Ramesh; Imanol Schlag; Weimin Shi; Aleksandar Stanić; Wenyi Wang; Yuhui Wang; Mengmeng Xu; Deng Ping Fan; Bernard Ghanem; Jurgen Schmidhuber

doi:10.26599/CVM.2025.9450460

Mindstorms in natural language-based societies of mind

Mingchen Zhuge, Haozhe Liu, Francesco Faccio, Dylan R. Ashley, Robert Csordás, Anand Gopalakrishnan, Abdullah Hamdi, Hasan Abed Al Kader Hammoud, Vincent Herrmann, Kazuki Irie, Louis Kirsch, Bing Li, Guohao Li, Shuming Liu, Jinjie Mai, Piotr Piȩkos, Aditya A. Ramesh, Imanol Schlag, Weimin Shi, Aleksandar StanićWenyi Wang, Yuhui Wang, Mengmeng Xu, Deng Ping Fan^*, Bernard Ghanem^*, Jurgen Schmidhuber^*

^*Corresponding author for this work

Research output: Contribution to journal › Article › peer-review

Abstract

Inspired by Minsky's Society of Mind, Schmidhuber's Learning to Think, and other more recent works, this paper proposes and advocates for the concept of natural language-based societies of mind (NLSOMs). We imagine these societies as consisting of a collection of multimodal neural networks, including large language models, which engage in a 'mindstorm' to solve problems using a shared natural language interface. Here, we work to identify and discuss key questions about the social structure, governance, and economic principles for NLSOMs, emphasizing their impact on the future of AI. Our demonstrations with NLSOMs - which feature up to 129 agents - show their effectiveness in various tasks, including visual question answering, image captioning, and prompt generation for text-to-image synthesis.

Original language	English (US)
Pages (from-to)	29-81
Number of pages	53
Journal	Computational Visual Media
Volume	11
Issue number	1
DOIs	https://doi.org/10.26599/CVM.2025.9450460
State	Published - 2025

Keywords

large language models (LLMs)
learning to think
mindstorm
multimodal learning
society of mind (SOM)

ASJC Scopus subject areas

Computer Vision and Pattern Recognition
Computer Graphics and Computer-Aided Design
Artificial Intelligence

Access to Document

10.26599/CVM.2025.9450460

Cite this

Zhuge, M., Liu, H., Faccio, F., Ashley, D. R., Csordás, R., Gopalakrishnan, A., Hamdi, A., Hammoud, H. A. A. K., Herrmann, V., Irie, K., Kirsch, L., Li, B., Li, G., Liu, S., Mai, J., Piȩkos, P., Ramesh, A. A., Schlag, I., Shi, W., ... Schmidhuber, J. (2025). Mindstorms in natural language-based societies of mind. Computational Visual Media, 11(1), 29-81. https://doi.org/10.26599/CVM.2025.9450460

@article{2e8ecf9e4a544abea7af974bdc9e881e,

title = "Mindstorms in natural language-based societies of mind",

abstract = "Inspired by Minsky's Society of Mind, Schmidhuber's Learning to Think, and other more recent works, this paper proposes and advocates for the concept of natural language-based societies of mind (NLSOMs). We imagine these societies as consisting of a collection of multimodal neural networks, including large language models, which engage in a 'mindstorm' to solve problems using a shared natural language interface. Here, we work to identify and discuss key questions about the social structure, governance, and economic principles for NLSOMs, emphasizing their impact on the future of AI. Our demonstrations with NLSOMs - which feature up to 129 agents - show their effectiveness in various tasks, including visual question answering, image captioning, and prompt generation for text-to-image synthesis.",

keywords = "large language models (LLMs), learning to think, mindstorm, multimodal learning, society of mind (SOM)",

author = "Mingchen Zhuge and Haozhe Liu and Francesco Faccio and Ashley, {Dylan R.} and Robert Csord{\'a}s and Anand Gopalakrishnan and Abdullah Hamdi and Hammoud, {Hasan Abed Al Kader} and Vincent Herrmann and Kazuki Irie and Louis Kirsch and Bing Li and Guohao Li and Shuming Liu and Jinjie Mai and Piotr Piȩkos and Ramesh, {Aditya A.} and Imanol Schlag and Weimin Shi and Aleksandar Stani{\'c} and Wenyi Wang and Yuhui Wang and Mengmeng Xu and Fan, {Deng Ping} and Bernard Ghanem and Jurgen Schmidhuber",

note = "Publisher Copyright: {\textcopyright} 2024 Tsinghua University Press.",

year = "2025",

doi = "10.26599/CVM.2025.9450460",

language = "English (US)",

volume = "11",

pages = "29--81",

journal = "Computational Visual Media",

issn = "2096-0433",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

number = "1",

}

Zhuge, M, Liu, H, Faccio, F, Ashley, DR, Csordás, R, Gopalakrishnan, A, Hamdi, A, Hammoud, HAAK, Herrmann, V, Irie, K, Kirsch, L, Li, B, Li, G, Liu, S, Mai, J, Piȩkos, P, Ramesh, AA, Schlag, I, Shi, W, Stanić, A, Wang, W, Wang, Y, Xu, M, Fan, DP, Ghanem, B & Schmidhuber, J 2025, 'Mindstorms in natural language-based societies of mind', Computational Visual Media, vol. 11, no. 1, pp. 29-81. https://doi.org/10.26599/CVM.2025.9450460

TY - JOUR

T1 - Mindstorms in natural language-based societies of mind

AU - Zhuge, Mingchen

AU - Liu, Haozhe

AU - Faccio, Francesco

AU - Ashley, Dylan R.

AU - Csordás, Robert

AU - Gopalakrishnan, Anand

AU - Hamdi, Abdullah

AU - Hammoud, Hasan Abed Al Kader

AU - Herrmann, Vincent

AU - Irie, Kazuki

AU - Kirsch, Louis

AU - Li, Bing

AU - Li, Guohao

AU - Liu, Shuming

AU - Mai, Jinjie

AU - Piȩkos, Piotr

AU - Ramesh, Aditya A.

AU - Schlag, Imanol

AU - Shi, Weimin

AU - Stanić, Aleksandar

AU - Wang, Wenyi

AU - Wang, Yuhui

AU - Xu, Mengmeng

AU - Fan, Deng Ping

AU - Ghanem, Bernard

AU - Schmidhuber, Jurgen

PY - 2025

Y1 - 2025

N2 - Inspired by Minsky's Society of Mind, Schmidhuber's Learning to Think, and other more recent works, this paper proposes and advocates for the concept of natural language-based societies of mind (NLSOMs). We imagine these societies as consisting of a collection of multimodal neural networks, including large language models, which engage in a 'mindstorm' to solve problems using a shared natural language interface. Here, we work to identify and discuss key questions about the social structure, governance, and economic principles for NLSOMs, emphasizing their impact on the future of AI. Our demonstrations with NLSOMs - which feature up to 129 agents - show their effectiveness in various tasks, including visual question answering, image captioning, and prompt generation for text-to-image synthesis.

AB - Inspired by Minsky's Society of Mind, Schmidhuber's Learning to Think, and other more recent works, this paper proposes and advocates for the concept of natural language-based societies of mind (NLSOMs). We imagine these societies as consisting of a collection of multimodal neural networks, including large language models, which engage in a 'mindstorm' to solve problems using a shared natural language interface. Here, we work to identify and discuss key questions about the social structure, governance, and economic principles for NLSOMs, emphasizing their impact on the future of AI. Our demonstrations with NLSOMs - which feature up to 129 agents - show their effectiveness in various tasks, including visual question answering, image captioning, and prompt generation for text-to-image synthesis.

KW - large language models (LLMs)

KW - learning to think

KW - mindstorm

KW - multimodal learning

KW - society of mind (SOM)

UR - http://www.scopus.com/inward/record.url?scp=105002705285&partnerID=8YFLogxK

U2 - 10.26599/CVM.2025.9450460

DO - 10.26599/CVM.2025.9450460

M3 - Article

AN - SCOPUS:105002705285

SN - 2096-0433

VL - 11

SP - 29

EP - 81

JO - Computational Visual Media

JF - Computational Visual Media

IS - 1

ER -

Mindstorms in natural language-based societies of mind

Abstract

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this