As sensing technologies become increasingly distributed and democratized, citizens and novice users are becoming responsible for the kinds of data collection and analysis that have traditionally been the purview of professional scientists and analysts. Leveraging this citizen engagement effectively, however, requires not only tools for sensing and data collection but also mechanisms for understanding and utilizing input from both novice and expert stakeholders. When successful, this process can result in actionable findings that leverage and engage community members and build on their experiences and observations. We explored this process of knowledge production through several dozen interviews with novice community members, scientists, and regulators as part of the design of a mobile air quality monitoring system. From these interviews, we derived design principles and a framework for describing data collection and knowledge generation in citizen science settings, culminating in the user-centered design of a system for community analysis of air quality data. Unlike prior systems, ours breaks analysis tasks into discrete mini-applications designed to facilitate and scaffold novice contributions. An evaluation we conducted with community members in an area with air quality concerns indicates that these mini-applications help participants identify relevant phenomena and generate local knowledge contributions.