Computer Awareness Quiz: Big Data, Data Mining and Data Analytics

A data warehouse is exactly that, a central store into which data from many operational systems is extracted, cleaned and loaded so that it can be analysed for decision making.

On-Line Transaction Processing is the mode in which a bank counter, a railway booking window or a payment gateway works: each transaction is processed the moment it is entered, so the response to the user is immediate, and every committed transaction is written at once, so the database always reflects…

Virtualisation is the technique of running several logical machines on one physical machine and belongs to the chapter on cloud computing, not to the definition of big data.

It splits a large file into fixed-size blocks, 128 megabytes by default in current versions, scatters those blocks over the machines of the cluster and keeps more than one copy of each block so that the failure of a machine does not lose data.

In the process called Knowledge Discovery in Databases, or KDD, the stages run from selection of the relevant data, through cleaning and preprocessing, to transformation, then data mining, and finally interpretation and evaluation of what was found.

The Apriori algorithm first finds the sets of items that appear together often enough to pass a support threshold, using the rule that any subset of a frequent set must itself be frequent, and then turns those frequent sets into rules measured by confidence.

Storage units climb in powers of two, each unit being 1,024 times the one below it, so 1,024 bytes make a kilobyte, 1,024 kilobytes a megabyte, 1,024 megabytes a gigabyte, 1,024 gigabytes a terabyte and 1,024 terabytes a petabyte.

It uses historical data with statistical models and machine learning to estimate what will probably happen next, such as which loan accounts are likely to slip into default or how much electricity a city will draw next summer.

ETL names the three steps through which data enters a warehouse: it is extracted from the operational systems that produced it, transformed so that formats, units, codes and duplicates are reconciled into one consistent shape, and then loaded into the warehouse tables.

Clustering is unsupervised because the data carries no labels: the algorithm itself groups similar records, k-means doing so by placing k centres and moving them until each record sits with its nearest centre.

NoSQL databases drop the fixed table and row structure so that they can hold large and irregular data and spread it over many machines, and MongoDB is the best known document store, keeping records as documents in a format close to JSON.

Apache Hive puts a table-like view over files stored in HDFS and lets an analyst write queries in HiveQL, a language close to SQL, which Hive then turns into jobs that run on the cluster, so that no Java programming is needed.