Data Mining
... of industries - including retail, finance, heath care, and manufacturing transportation are already using data mining tools and techniques to take advantage of historical data. By using pattern recognition technologies and statistical and mathematical techniques to filter through warehoused information, data mining helps analysts recognize important facts, relationships, trends, patterns, exceptions and irregularities that might otherwise go unnoticed. For businesses, data mining is used to discover patterns and relationships in the data in order to help make better business decisions. Data mining can help spot sales trends, develop smarter marketing campaigns, and predict customer loyalty. Specific uses of data mining include market segmentation, trend analysis, fraud detection, and direct marketing. Data mining automates the process of finding predictive information in a large database. Questions that traditionally required extensive hands-on analysis can now be directly answered from the data. An example of a predictive problem is targeted marketing. Data mining uses data on past promotional mailings to identify the targets most likely to maximize return on investment in future mailings. Other predictive problems include forecasting bankruptcy and other forms of default, and identifying segments of a population likely to respond similarly to given events. How is data mining able to tell you important things that you didn't know or what is going to happen next? The technique that is used to perform these accomplishments is called modeling. Modeling is simply the act of building a model based on data from situations where the answer is known and then applying the model to other situations where the answers aren't known. Modeling techniques have been around for a while, but it is only recently that data storage and communication capabilities required collecting and store huge amounts of data, and the computational power to automate modeling techniques to work directly on the data, have been available. Data mining takes advantage of advances in the fields of artificial intelligence (AI) and statistics. Both disciplines have been working on problems of pattern recognition and classification. Both areas have made great contributions to the understanding of decision trees. Data mining does not replace traditional statistical techniques. It is an extension of statistical methods that is the result of a major change in the statistics community. The development of most statistical techniques was, until recently, based on elegant theory and analytical methods that worked quite well on the modest amounts of data being analyzed. The increased power of computers and their lower cost, coupled with the need to analyze large data sets with millions of rows, have allowed the development of new techniques based on a brute-force exploration of possible solutions. New techniques include relatively recent algorithms like decision trees and new approaches to older algorithms such as discriminate analysis. The increased computer power on the large volumes of available data, these techniques can approximate almost any functional form or interaction on their own. Traditional statistical techniques rely on the model to specify the functional form and interactions. The key point is that data mining is the application of these and other AI and statistical techniques to common business problems in a fashion that makes these techniques available to the skilled knowledge worker as well as the trained statistics professional. Data mining is a tool for increasing the productivity of people trying to build predictive models. One of the key things of data mining is the progress in hardware price and performance. The dramatic 99% drop in the price of computer disk storage in just the last few years has radically changed the economics of collecting and storing massive amounts of data. At $10/megabyte, one terabyte of data costs $10,000,000 to store. At 10¢/megabyte, one terabyte of data costs only $100,000 to store. This doesn’t even include the savings in real estate from greater storage capacities. The drop in the cost of computer processing has been equally dramatic. Each generation of chips greatly increases the power of the CPU, while allowing further drops on the cost curve. This is also reflected in the price of RAM, where the cost of a megabyte has dropped from hundreds of dollars to around a dollar in just a few years. Data mining is increasingly popular because of the considerable contribution it can make. It can be used to control costs as well as contribute to revenue increases. Many organizations are using data mining to help manage all phases of the customer life cycle, including acquiring new customers, increasing revenue from existing customers, and retaining good customers. By determining characteristics of good customers (profiling), a company can target prospects with similar characteristics. By profiling customers who have bought a particular product it can focus attention on similar customers who have not bought that product. By profiling customers who have left, a company can act to retain customers who are at risk for leaving because it is usually far less expensive to retain a customer than acquire a new one. Data mining offers value across a widespread variety of industries. Telecommunications and credit card companies are two of the leaders in applying data mining to detect fraudulent use of their services. Insurance companies and stock exchanges are also interested in using this technology to decrease fraud. Medical applications are another rewarding area of data mining; data mining can be used to predict the effectiveness of surgical procedures, medical tests and medications. Companies active in the financial markets use data mining to determine market and industry characteristics as well as to predict individual company and stock performance. Retailers are making more use of data mining to decide which products to stock in particular stores (and even how to place them within a store), as well as to assess the effectiveness of promotions and coupons. Pharmaceutical firms are mining large databases of chemical compou...