We describe our experiences developing a system (EMBERS) to forecast significant societal events – such as protests, disease outbreaks, elections, domestic political crises – using a multitude of open source data feeds. Over the past three years, EMBERS has successfully forecast many international (and rare) events such as the “Brazilian Spring” (June 2013), Hantavirus outbreaks in Argentina and Chile (2013), student-led protests in Venezuela (Feb 2014), protests stemming from the kidnappings and killings of student-teachers in Mexico (Sep- Oct 2014), and protests in Paraguay (Feb 2015) against a new public-private partnership law. The system also correctly forecast the winners of presidential elections in Panama and Colombia, even as every major poll indicated other candidates would come out on top. We outline some of the lessons learned in operating a large-scale forecasting system, especially as pertaining to system architecture, model evaluation, and continual improvement to changing societal dynamics.