We present an approach to the accurate and efficient large-scale simulation of the complex dynamics of ferrofluids based on physical principles. Ferrofluids are liquids containing magnetic particles that react to an external magnetic field without solidifying. In this contribution, we employ smooth magnets to simulate ferrofluids in contrast to previous methods based on the finite element method or point magnets. We solve the magnetization using the analytical solution of the smooth magnets’ field, and derive the bounded magnetic force formulas addressing particle penetration. We integrate the magnetic field and force evaluations into the fast multipole method allowing for efficient large-scale simulations of ferrofluids. The presented simulations are well reproducible since our approach can be easily incorporated into a framework implementing a Fast Multipole Method and a Smoothed Particle Hydrodynamics fluid solver with surface tension. We provide a detailed analysis of our approach and validate our results against real wet lab experiments. This work can potentially open the door for a deeper understanding of ferrofluids and for the identification of new areas of applications of these materials.