Low Latency, High Performance Data Stream Processing: Systems Architecture, Algorithms and Implementation

Data stream oriented applications are typically dealing with huge volumes of data. Storing data and performing off-line processing on this huge data set can be costly, time consuming and impractical. This work describes our research results while designing and implementing an efficient data management system for on-line and off-line processing of streaming data. We present major existing data stream processing engines, their internal architecture and how they compare to our platform, Global Sensor Network (GSN) middleware. In order to achieve high efficiency while processing large volumes of streaming data using window-based continuous queries, we present a set of optimization algorithms and techniques to intelligently group and process different types of continuous queries. Moreover we present an efficient query scheduling component which not only increases the performance at least by an order of magnitude but also, decreases the response time and memory requirements. Finally, we present techniques and algorithms to enable scalable delivery of streaming data for high data rate streams (e.g., Financial Ticks).