There has been a tremendous increase in the amount and type of information that is available through the Internet and through various sensors that now pervade our daily lives. Consequentially, the field of context aware computing has also contributed significantly in providing new technologies to mine and use the available context data. We present Conspeakuous – an architecture for modeling, aggregating and using the context in spoken language conversational systems. Since Conspeakuous is aware of the environment through different sources of context, it helps in making the conversation more relevant to the user, and thus reducing the cognitive load on the user. Additionally, the architecture allows for representing learning of various user/environment parameters as a source of context. We built a sample tourist information portal application based on the Conspeakuous architecture and conducted user studies to evaluate the usefulness of the system.