ISAVS: Interactive scalable analysis and visualization system

Modern science is inundated with ever increasing data sizes as computational capabilities and image acquisition techniques con- tinue to improve. For example, simulations are tackling ever larger domains with higher fidelity, and high-throughput microscopy tech- niques generate larger data that are fundamental to gather biolog- ically and medically relevant insights. As the image sizes exceed memory, and even sometimes local disk space, each step in a sci- entific workflow is impacted. Current software solutions enable data exploration with limited interactivity for visualization and analytic tasks. Furthermore analysis on HPC systems often require complex hand-written parallel implementations of algorithms that suffer from poor portability and maintainability We present a software infrastructure that simplifies end-to-end visualization and analysis of massive data. First, a hierarchical stream- ing data access layer enables interactive exploration of remote data, with fast data fetching to test analytics on subsets of the data. Sec- ond, a library simplifies the process of developing new analytics algorithms, allowing users to rapidly prototype new approaches and deploy them in an HPC setting. Third, a scalable runtime sys- tem automates mapping analysis algorithms to whatever compu- tational hardware is available, reducing the complexity of develop- ing scaling algorithms. We demonstrate the usability and perfor- mance of our system using a use case from neuroscience: filtering, registration, and visualization of tera-scale microscopy data. We evaluate the performance of our system using a leadership-class supercomputer, Shaheen II.