Navigation

Dapprox: Improving cloud datacentre performance with new approximate analytics

 

To enhance user experience, datacentres monitor millions of resource usage series, resulting in big data to gather useful insights. Dapprox derives methods and tools to predict performance anomalies in real time by selecting a key subset of data and proposing solutions to better manage resources.

Portrait / project description (ongoing research project)

Dapprox is a set of methods and software tools for fast and approximate analyses of resource usage series in real time. The goal of Dapprox is to predict potential anomalies (and propose solutions) by simultaneously taking into account accuracy requirements, maximum delays and available resources. Dapprox first looks for characteristics that are common across servers over time, and then processes only subsets of “key” data in a way that does not sacrifice the accuracy of the results. Particularly, Dapprox can dynamically select and process the optimal amount of data, based on common structures that change over time. Dapprox comprises three work packages: dependency-aware predictive analytics for forecasting, approximate streaming analytics for live data and datacentre anomaly management.

Background

To ensure quality of service and system reliability, datacentres monitor and collect performance logs from many virtual and physical computing resources. The sheer quantity of data generated is so large that it is nearly impossible to always correctly analyse it in real time. Existing analyses tend to be unsophisticated and slow, which leads to delays in addressing performance anomalies and significantly degrades end-user experience.

Aim

Our goal is to analyse performance data to better manage computing resources in cloud datacentres and thus to enhance user experience. But rather than analysing all of the data, we will develop approximate analytics – i.e. methods and tools based on subsets of data – to predict complex patterns of resource usage series and so-called critical states. We will also create tools for real-time processing and anomaly analysis. Finally, we will propose anomaly management policies for cloud datacentres.

Relevance

The proposed research will develop practical solutions to exploit the value of big data in performance logs from today’s cloud datacentres, to efficiently and approximately process jobs on big data platforms and to enhance users’ computing experience in the cloud.

We expect Dapprox to benefit datacentre practitioners, researchers and users of big data analytics and cloud computing platforms. Because our approach is based on the generic structure of big data, our techniques should be widely applicable to different types of big data (e.g. data from Internet of Things devices) and to different system scenarios (e.g. energy-optimised datacentres).

Original title

Dapprox: Dependency-ware Approximate Analytics and Processing Platforms

Project leaders

Grantees

  • Dr. Lydia Yiyu Chen, IBM Research GmbH
  • Dr. Robert Birke, IBM Research GmbH

Project partner

  • Ass. Prof. Ce Zhang, Computer Science, ETH-Zurich

 

 

Further information on this content

 Contact

Dr. Lydia Yiyu Chen IBM Research GmbH Säumerstrasse 4 8803 Rüschlikon yic@zurich.ibm.com

On this Subject