Lecture Notes in Computer Science, 2007, Volume 4365/2007, 50-62, DOI: 10.1007/978-3-540-73950-0_5

A Scalable Heterogeneous Solution for Massive Data Collection and Database Loading

Uri Shani, Aviad Sela, Alex Akilov, Inna Skarbovski and David Berk

View Related Documents

Abstract

Massive collection of data at high rates is critical for many industries. Typically, a massive stream of records is gathered from the business information network at a very high rate. Because of the complexity of the collection process, the classical database solution falls short. The high volume and rate of records involved requires a heterogeneous pipeline comprised of two major parts: a system that carries out massive collection and then uploads the information to a database, and a subsequent data analysis and management system consisting of an Extract Transform and Load component. We developed a massive collection and loading system, based on a highly scalable heterogeneous architecture solution. The solution has been applied successfully for Telco revenue assurance, and can be applied to other industrial areas. The solution was successful in scaling up a Telco client system to handle streams of records ten times larger than was previously possible.

Fulltext Preview

Image of the first page of the fulltext document