Complex Event Analytics Contributions

First we notice that existing techniques such as traditional online analytical processing (OLAP) systems are not designed for real-time pattern based operations, while state-of-the-art Complex Event Processing (CEP) systems designed for pattern matching tend to be limited in their expressive capability, and more importantly they do not support OLAP operations. Hence, in the context of event streams where the order and sequence of events are important, OLAP is clearly insufficient in supporting efficient event sequence analysis. Thus in this project, we design, develop and evaluate novel complex event analytics technology that achieves the above objectives. In particular, one key contribution to the discipline is the novel event analytics model that effectively leverages CEP and OLAP techniques for efficient multi-dimensional event pattern analysis at different abstraction levels.

The second key contribution to the discipline is that given a workload of CEP queries, our event analytics technology exploits interrelationships between CEP queries in terms of both concept and pattern refinement among these queries for optimized shared processing and maximal reuse of intermediate results thus saving critical computational and memory resources.

The third contribution is the development of a language and its semantics for expressing nested complex event patterns, including the treatment of sequencing, negation, and predicates. A methodology for correct execution of this pattern matching method has been designed and implemented.

The fourth innovation is the idea of continuous views for optimization nested CEP query processing. In particular, this includes lightweight data structures, along with algorithms for loading, checking, utilizing as well as continuously purging the continuous cache to keep the content up-to-date. The challenge of how to do so correctly, to allow for the contradictory demands of continuous updating and maintenance versus achieving real-time performance characteristics, has been examined and overcome.

The fifth contribution concerns the notion of integrating update actions into CEP stream processing, called active CEP. These actions may lead to events being omitted or system states changed, which in turn mayaffect the processing of event pattern matching. A best DBES award has been given for this innovation bythe Event Processing Technology Society. A patent has also been filed.

The sixth contribution concerns the novel innovation of supporting privacy in the context of CEP pattern matching via event suppression. Several optimization methods, purely type-based as well as instance-based runtime algorithms, are developed that tackle this optimization problem.

The seventh contribution is the innovation of sharing complex event patterns across multiple queries over time, not just by considering the shared subpattern structures at the syntax level but also by leveraging time-based event correlations across queries.

The eighth innovation is a methodology to push aggregation directly into the on-line sequence detection process so to scale up the stream processing. This is clever, reducing the costs of stream processing by several orders of magnitudes by avoiding the need to ever construct intermediate sequence results before counting them. Extensions cover various query features, such as, negation, local predicates, and join predicates. In addition, the MATTERS web-based analytics platform has been developed by Prof. Rundensteiner and WPI students, sponsored by and in collaboration with the Mass. High Technology Council, and other strategic partners. The students involved in this related effort can be found on the MATTERS WPI page.

MATTERS represents an important publicly-accessible web-based analytics platform for analyzing cost, talent, and economic indicators over time to assess economic competitiveness of US states. While MATTERS is not a primary data source, it is a valuable aggregator of published data and analysis from numerous seemingly disparate sources including federal and state governmental agencies, non-profit organizations and media outlets into one unified data hub. The MATTERS data explorer permits users to customize their experience and compare and contrast with ease one or more metrics from one or more states and across multiple years simultaneously. Data can be compared and contrasted visually by displaying them seemlessly in a variety of visualizations including tables, line charts, bar charts, and heatmaps.

As stated by Gary Beach, Editor Emeritus, CIO Magazine:
"Until MATTERS, enormously valuable data resided in disparate places - and in static form - effectively locked away from those who might leverage it the most to make informed decisions. By aggregating and injecting dynamism into those key data sets, MATTERS will equip policymakers, business leaders, advocates and researchers with a real-time data analytics tool that will help shape our public policy agenda, our debates and the outcomes of key decisions to be made in Massachusetts for years to come."

The MATTERS project has been an exciting opportunity for WPI students to work with partners across different organizations; which has garnered recognition and notice by external press as listed here.

These external notices about the tool include: