Determining what abiotic and biotic factors affect the diversity and abundance of species through time and space is a basic goal of ecology and an integral step in predicting current and future distributions. Given the pervasive effect of humans worldwide, including anthropogenic factors when quantifying community dynamics is needed to understand discrete and emergent effects of humans on marine ecosystems, especially systems with economically important species. However, there are limited studies that combine a large-scale ecological survey with multiple natural and anthropogenic factors to determine the drivers of community dynamics of temperate reef systems. We combined data from a 24-year fish survey on temperate reefs along the Southeast United States coast with information on recreational and commercial fisheries landings, surface and bottom temperature, habitat characteristics, and climate indices to determine what factors may alter the community structure of fishes within this large marine ecosystem. We found that both abundance and richness of temperate reef fishes declined from 1990 to 2013. Climate indices and local temperature explained the greatest variation, and recreational fishing explained slightly more variation compared to commercial fishing in the temperate reef fish community over a multi-decadal scale. When including habitat characteristics in a 3 year analysis, depth, and local temperature explained the greatest variation in fish assemblage, while the influence of habitat was comparatively minimal. Finally, the interaction between predictor variables and fish traits indicated that bigger and longer-lived fishes were positively correlated with depth and winter temperature. Our findings suggest that lesser-studied anthropogenic impacts, such as recreational fishing, may influence communities throughout large ecosystems as much as other well-studied impacts such as climate change and commercial fishing. In addition, climate indices should be considered when assessing changes, natural or anthropogenic, to fish communities.