Billions of records analyzed in real-time

The total system cost to analyze their 50 billion records utilizing Pivotbillions on AWS was easily less than 1/10 of conventional systems.

Mobile and IoT data has grown exponentially with the explosion of connected devices. Unfortunately, the cost and time associated to analyze this data has also grown exponentially as data volumes keep getting larger and larger. Companies that are collecting data from millions of devices are wrestling with the acute pains of this rapid data growth which regularly exceeds the capacity of most conventional data processing solutions.

AuriQ Systems in Pasadena, California has been at the forefront of trying to find creative and cost-effective solutions for their enterprise clients who are actively trying to analyze their IoT and Mobile data for customer insight and to apply it toward optimizing the customer journey. Their distributed data processing and analysis platform known as Pivotbillions, is uniquely capable of solving the prevalent issues of scalability, agility and cost that arises when dealing with large amounts of IoT and Mobile device information.

In a recent engagement, AuriQ used Pivotbillions to help their client analyze over 50 billion records of GPS data from a variety of devices. Specifically, the client wanted to be able to dynamically query all 50 billion records in real-time to identify the behavior patterns of their customers. That meant that analytic queries, including ad-hoc queries, against the entire data set had to be processed within seconds to allow data analysts to test their hypotheses quickly.

Pivotbillions is a massively parallel, in-memory data processing and analysis software solution that can run on Amazon Web Services. It does not require any special hardware and can run on inexpensive EC2 instances. The user interface of Pivotbillions is similar to an Excel-like spreadsheet that allows analysts to work with data immediately without any coding or steep learning curve.

Utilizing up to 500 concurrent Amazon EC2 instances, the client was able to process and load 6TB in 365 compressed files into PivotBillions in 30 minutes. Afterwards, running queries against all 50 billion records had response times of a few seconds to tens of seconds. This was in stark contrast to the client’s conventional system which took a few hours to days to process a query for even a partial sample of data.

The agility afforded by the instant query capabilities allowed analysts to parse and identify patterns of behavior from anonymized location data. In one analysis they parsed the flow of devices through a major airport and visualized the cities transited before and after the airport.

Ko Ikudome, CEO of AuriQ states that “The total system cost to analyze their 50 billion records utilizing Pivotbillions on AWS was easily less than 1/10 of conventional systems.” The end result was that the client was able to complete analysis and visualization tasks for various different requirements in just a few weeks at a lower cost with Pivotbillions compared to conventional solutions.

AuriQ is currently offering trials of the Pivotbillions solution for free. To find out more, users can go to the Pivot Billions website ( to request a demo or sign up for a free account.

About AuriQ Systems, Inc.

Since its inception in 1996, AuriQ has been creating high performance data analysis solutions to make our clients’ data more visible, accessible and actionable.

Share article on social media or email:

This site uses Akismet to reduce spam. Learn how your comment data is processed.