The recent advances in hardware and software have enabled the capture of different measurements of datain a wide range of fields. Applications that generate rapid, continuous and large volumes of event streams includereadings from sensors used in applications, such as physics, biology and chemistry experiments, weathe rsensors, health sensors, online auctions, and financial streams. Given these developments, the world is poised for a sea change in terms of variety, scale and importance of applications that can be envisioned based on the real-time analysis and exploitation of such event streams for decision making from dynamic traffic management, environmental monitoring to health care alike. Clearly, the ability to infer relevant patterns from these event streams in realtime to make near instantaneous yet informed decisions, henceforth called complex event analytics (CEA), is crucial for these mission critical applications and this isthe focus of our project work. We now list key contribution ideas next.
First we notice that existing techniques such as traditional online analytical processing (OLAP) systems are not designed for real time pattern based operations, while state-of-the-art Complex Event Processing (CEP) systems designed for pattern matching tend to be limited in their expressive capability and more importantly they do not support OLAP operations. Hence, in the context of event streams where the order and sequence of events areimportant, OLAP is clearly insufficient in supporting efficient event sequence analysis. Thus in this project, we design, develop and evaluate novel complex event analytics technology that achieves the above objectives. In particular, one key contribution to the discipline is the novel event analytics model that effectively leverages CEP and OLAPtechniques for efficient multi-dimensional event pattern analysis at different abstraction levels.
The second key contribution to the discipline is that given a workload of CEP queries, our event analytics technology exploits interrelationships between CEP queries in terms of both concept and pattern refinement among these queries for optimized shared processing and maximal reuse of intermediate results thus saving critical computational and memory resources.
The third contribution is the development of a language and its semantics for expressing nested complex event patterns, including the treatment of sequencing, negation and predicates. A methodology for correct execution of this pattern matching method has been designed and implemented.
The fourth innovation is the idea of continuous views for optimization nested CEP query processing. Inparticular, this includes lightweight data structures, along with algorithms for loading, checking, utilizing as well as continuously purging the continuous cache to keep the content up-to-date. The challenge of how to do so correctly, to allow for the contradictory demands of continuous updating and maintenance versus achieving real-time performance characteristics, has been examined and overcome.
The fifth contribution concerns the notion of integrating update actions into CEP stream processing, called active CEP. These actions may lead to events being omitted or system states changed, which in turn mayaffect the processing of event pattern matching. A best DBES award has been given for this innovation bythe Event Processing Technology Society. A patent has also been filed.
The sixth contribution concerns the novel innovation of supporting privacy in the context of CEP pattern matching via event suppression. Several optimization methods, purely typebased as well as instance-base runtime algorithms, are developed that tackle this optimization problem.
The seventh contribution is the innovation of sharing complex event patterns across multiple queries over time, not just by considering the shared subpattern structures at the syntax level but also by leveraging time-based event correlations across queries.
The eight innovation is a methodology to push aggregation directly into the on-line sequence detection process so to scale up the stream processing. This is clever, reducing the costs of stream processing by several orders of magnitudes by avoiding the need to ever construct intermediate sequence results before counting them. Extensions cover various query features, such as, negation, local predicates, and join predicates. In addition, the MATTERS web-based analytics platform has been developed by Prof. Rundensteiner and WPI students, sponsored by and in collaboration with the Mass. High Technology Council, and other strategic partners. The students involved in this related effort can be found on the MATTERS WPI page.
MATTERS represents an important publically-accessible web-based analytics platform for analyzing cost, talent, and economic indicators over time to assess economic competitiveness of US states. While MATTERS is not a primary data source, it is a valuable aggregator of published data and analysis from numerous seemingly disparate sources including federal and state governmental agencies, non-profit organizations and media outlets into one unified data hub. The MATTERS data explorer permits users to customize their experience and compare and contrast with ease one or more metrics from one or more states and across multiple years simultaneously. Data can be compared and contrasted visually by displaying them seemlessly in a variety of visualizations including tables, line charts, bar charts, and heatmaps.As stated by Gary Beach, Editor Emeritus, CIO Magazine:
"Until MATTERS, enormously valuable data resided in disparate places - and in static form - effectively locked away from those who might leverage it the most to make informed decisions. By aggregating and injecting dynamism into those key data sets, MATTERS will equip policymakers, business leaders, advocates and researchers with a real-time data analytics tool that will help shape our public policy agenda, our debates and the outcomes of key decisions to be made in Massachusetts for years to come."
The MATTERS project has been an exciting opportunity for WPI students to work with partners across different organizations; which has garnered recognition and notice by external press as listed here.These external notices about the tool include:
- New tool measures Massachusetts tech performance against other states, Boston Globe
- Mass. High Technology Council and Greater Boston Chamber of Commerce Announce Partnership to Benchmark State's Talent and Cost Competitiveness, Massachusetts High Technology Council
- Mass. High Technology Council and Greater Boston Chamber of Commerce Announce MATTERS Partnership, Massachusetts High Technology Council
- High-tech council creates data dashboard to keep state on right road, Boston Herald
- Mass. High Tech Council Launches Massachusetts' Technology, Talent and Economic Reporting System, Monster Goverment Solutions
- WPI Develops Groundbreaking Big Data Tool To Measure Massachusetts Technology Strengths, WPI Press Release
- WPI data tool to measure Mass. tech strength, Worcester Business Journals
- Diverse Student Team Develops Big Data Tool to Measure Tech Strength in Massachusetts, WPI Daily Herd
- Mass. High Tech Council Launches Massachusetts' Technology, Talent and Economic Reporting System, Massachusetts High Technology Council
- New online data analytics tool will show how Mass. ranks in talent, competitiveness, Boston Business Journal