Introduction to Big Data Analytics

Big data analytics is where advanced analytic techniques operate on big data sets. Hence, big data analytics is really about two things—big data and analytics—plus how the two have teamed up to create one of the most profound trends in business intelligence (BI) today. Let’s start by defning advanced analytics, then move on to big data and the combination of the two.

Defning Advanced Analytics as a Discovery Mission

According to a 2009 TDWI survey, 38% of organizations surveyed reported practicing advanced analytics, whereas 85% said they would be practicing it within three years.1 Why the rush to advanced analytics? First, change is rampant in business, as seen in the multiple “economies” we’ve gone through in recent years. Analytics helps us discover what has changed and how we should react. Second, as we crawl out of the recession and into the recovery, there are more and more business opportunities that should be seized. To that end, advanced analytics is the best way to discover new customer segments, identify the best suppliers, associate products of afnity, understand sales seasonality, and so on. For these reasons, TDWI has seen a steady stream of user organizations implementing analytics in recent years. Te rush to analytics means that many organizations are embracing advanced analytics for the frst time, and hence are confused about how to go about it. Even if you have related experience in data warehousing, reporting, and online analytic processing (OLAP), you’ll fnd that the business and technical requirements are diferent for advanced forms of analytics. To help user organizations select the right form of analytics and prepare big data for analysis, this report will discuss new options for advanced analytics and analytic databases for big data so that users can make intelligent decisions as they embrace analytics. Note that user organizations are implementing specifc forms of analytics, particularly what is sometimes called advanced analytics. Tis is a collection of related techniques and tool types, usually including predictive analytics, data mining, statistical analysis, and complex SQL. We might also extend the list to cover data visualization, artifcial intelligence, natural language processing, and database capabilities that support analytics (such as MapReduce, in-database analytics, in-memory databases, columnar data stores).

Instead of “advanced analytics,” a better term would be “discovery analytics,” because that’s what users are trying to accomplish. (Some people call it “exploratory analytics.”) In other words, with big data analytics, the user is typically a business analyst who is trying to discover new business facts that no one in the enterprise knew before. To do that, the analyst needs large volumes of data with plenty of detail. Tis is often data that the enterprise has not yet tapped for analytics. For example, in the middle of the recent economic recession, companies were constantly being hit by new forms of customer churn. To discover the root cause of the newest form of churn, a business analyst would grab several terabytes of detailed data drawn from operational applications to get a view of recent customer behaviors. Te analyst might mix that data with historic data from a data warehouse. Dozens of queries later, the analyst would discover a new churn behavior in a subset of the customer base. With any luck, that discovery would lead to a metric, report, analytic model, or some other product of BI, through which the company could track and predict the new form of churn. Discovery analytics against big data can be enabled by diferent types of analytic tools, including those based on SQL queries, data mining, statistical analysis, fact clustering, data visualization, natural language processing, text analytics, artifcial intelligence, and so on. It’s quite an arsenal of tool types, and savvy users get to know their analytic requirements before deciding which tool type is appropriate to their needs. All these techniques have been around for years, many of them appearing in the 1990s. Te diference today is that far more user organizations are actually using them. Tat’s because most of these techniques adapt well to very large, multi-terabyte data sets with minimal data preparation. Tat brings us to big data.

Defning Big Data Via the Three Vs

Most defnitions of big data focus on the size of data in storage. Size matters, but there are other important attributes of big data, namely data variety and data velocity. Te three Vs of big data (volume, variety, and velocity) constitute a comprehensive defnition, and they bust the myth that big data is only about data volume. In addition, each of the three Vs has its own ramifcations for analytics. It’s obvious that data volume is the primary attribute of big data. With that in mind, most people defne big data in terabytes—sometimes petabytes. For example, a number of users interviewed by TDWI are managing 3 to 10 terabytes (TB) of data for analytics. Yet, big data can also be quantifed by counting records, transactions, tables, or fles. Some organizations fnd it more useful to quantify big data in terms of time. For example, due to the seven-year statute of limitations in the U.S., many frms prefer to keep seven years of data available for risk, compliance, and legal analysis. Te scope of big data afects its quantifcation, too. For example, in many organizations, the data collected for general data warehousing difers from data collected specifcally for analytics. Diferent forms of analytics may have diferent data sets. Some analytic practices lead a business analyst or similar user to create ad hoc analytic data sets per analytic project. Ten, there’s the entire enterprise, which in toto has its own, even larger scope of big data. Furthermore, each of these quantifcations of big data grows continuously. All this makes big data for analytics a moving target that’s tough to quantify.