Big Data, Data Mining and Data Analytics: Exam Notes

Every card swipe, toll payment, train booking and mobile recharge leaves a record behind, and the volume of such records has grown far beyond what a single computer or an ordinary database can handle. Big data is the name given to data of that size and speed, data mining is the work of finding useful patterns hidden inside it, and data analytics is the wider craft of turning those patterns into decisions. Banking, SSC, Railway and State papers now carry two or three questions from this chapter, almost always on the defining words rather than on programming.

What makes data big: the Vs

The three Vs were named by Doug Laney in 2001 and are still the standard answer. Volume is the sheer quantity, measured in terabytes and petabytes rather than megabytes. Velocity is the speed at which the data arrives and must be processed, as in a payment gateway that clears thousands of transactions a second. Variety is the mixture of forms: structured rows in a table, semi-structured files such as XML and JSON, and unstructured text, images, audio and video. Two more Vs were added later and complete the usual list of five: Veracity, meaning how trustworthy and clean the data is, and Value, meaning whether anything useful can be got out of it at all.

Hadoop and the tools of big data

Apache Hadoop is the open source framework that made big data practical. It was built by Doug Cutting and Mike Cafarella, is written mainly in Java, and takes its name from a toy elephant. Its idea is simple: instead of one huge machine, use a cluster of ordinary ones, break a file into blocks, keep several copies of each block on different machines, and send the computation to the machine that already holds the data. Its three core parts are the Hadoop Distributed File System or HDFS, which stores the blocks, MapReduce, which processes them in a map stage followed by a reduce stage, and YARN, which allots resources in the cluster. In the current versions the default HDFS block size is 128 megabytes. Around this core sit Hive, which lets a user query with SQL-like statements, Pig with its own Pig Latin language, the HBase database, Sqoop and Flume for moving data in, and Zookeeper for coordination. Apache Spark, which keeps data in memory, is much faster than MapReduce for repeated passes. Where tables will not do, NoSQL databases are used: MongoDB stores documents, Cassandra stores wide columns, Redis stores key-value pairs and Neo4j stores graphs.

Data warehouse, data mart and data lake

A data warehouse is a central store built for analysis rather than for daily work. Bill Inmon's classic definition calls it subject-oriented, integrated, time-variant and non-volatile: it is organised around subjects such as sales, it merges data from many systems, it keeps history, and old records are not overwritten. Data reaches it through ETL, that is Extract, Transform and Load. A data mart is a small warehouse serving one department, while a data lake holds raw data in its original form until someone needs it. The day-to-day system that records transactions is called OLTP, On-Line Transaction Processing: many small reads and writes, answered at once, always current. The analytical system built on the warehouse is OLAP, On-Line Analytical Processing: fewer but much heavier queries over a data cube, which the user can roll up, drill down, slice and dice.

Data mining and the kinds of analytics

Data mining is one step, the pattern-finding step, in the larger process called Knowledge Discovery in Databases or KDD, which also includes selection, cleaning, transformation and interpretation. Its standard tasks are classification, which puts a record into a known class, clustering, which groups similar records without knowing the classes in advance, regression, which predicts a number, association rule mining, which finds items bought together and is the basis of market basket analysis, and outlier detection, which is how banks flag a suspicious transaction. The Apriori algorithm is the textbook method for association rules; k-means is the textbook method for clustering; decision trees, naive Bayes and k-nearest neighbour are used for classification. Analytics itself is graded into four levels: descriptive analytics says what happened, diagnostic analytics says why it happened, predictive analytics says what is likely to happen next, and prescriptive analytics says what should therefore be done. R, Python, SAS, Tableau and Power BI are the tools named most often in question papers.