Data mining. Textbook
- -
- 100%
- +
Looking at the example of creating two datasets – one for implicit clustering and one for managed clustering – we can easily see the difference between the two methods. In the first example, the results may be the same in one case and different in another. But if the method is good for finding interesting relationships (as it usually is), it will give us useful information about the overall structure of the data. However, if the technique is not good at identifying relationships, then it will give us very little information.
Let’s say we are developing a system for determining the direction of a new product and want to identify similar products. Since it is not possible to measure the direction of a product outside the system, we will have to find relationships between products based on information about their names. If there is a good rule that we can use to establish relationships between similar products, then this information is very useful as it allows us to find interesting relationships (by identifying similar products that appear close to each other). However, if the relationship between two products isn’t very obvious, it’s likely that it’s just an unrelated relationship – which means the feature detection method we choose may not matter much. On the other hand, if the relationship is not very obvious but extremely useful (as in the example above), then we can start to learn how the product name is related to the process the product went through. This is an example of how different methods can produce very different results.
Unlike the characteristics of different methods, you also have different possible techniques. For example, when I say that my system uses image recognition, it doesn’t necessarily mean that the process the product goes through uses image recognition. If there are product images that we have taken in the past, or if we have captured some input from a product image, the resulting system will probably not use image recognition. It could be something completely different – something much more complex. Each of these methods is capable of identifying very different things. The result may depend on the characteristics of the actual data or on the data used. This means it’s not enough to look at a specific type of tool – we also need to look at what type of tool will be used for a particular type of process. This is an example of how data analysis should not be focused only on the problem being solved. Most likely, the system goes through many different processes, so we need to look at how different tools will be used to create a relationship between two points, and then decide which type of data to consider.
Often, we will be more concerned with how the method will be applied. For example, we might want to see what type of data is most likely to be useful for finding a relationship. We see that there is not much difference in how natural language processing is applied. This means that if we want to find a relationship, natural language processing is a good choice. However, natural language processing does not solve every possible relationship. Natural language processing is often useful when we want to take a huge number of small steps, but natural language processing does nothing when we want to go really deep. A look at natural language processing allows you to establish relationships between data that cannot be done using other methods. This is one of the reasons why natural language processing can be useful but not necessary.
However, natural language processing often doesn’t find as strong connections as image recognition because natural language processing focuses on simpler data whereas image recognition looks at very complex data. In this case, natural language processing is not very good, but can still be useful. Considering natural language processing is not always the best way to solve a problem. Natural language processing can be useful if the data is simple, but sometimes it is not possible to work with very complex data.
This example can be applied to many different types of data, but natural language processing is generally more useful for natural language data such as text files. For more complex data (such as images), natural language processing is often not enough. If there is a problem with natural language processing, it is important to consider other methods such as detecting words and determining what data is actually stored in an image. This data type will require a different data structure to find the relationship.
With the increasing complexity of technology, we often don’t have time to look at the data we’re looking at. Even if we look at the data, we may not find a good solution, because we have a large number of options, but not much time to consider them all. This is why many companies have a data scientist who can make many different decisions and then decide what works best for the data.
Classification
Classification is the task of generalizing a known structure to be applied to new data. For example, an email program might try to classify an email as «legitimate», or «spam», or maybe «deleted by the administrator», and if it does this correctly, it can mark the email as relevant to the user.
However, for servers, the classification is more complex because storage and transmission are far away from users. When servers consume huge amounts of data, the problem is different. The job of the server is to create a store and pass that store around so that servers can access it. Thus, servers can often avoid disclosing particularly sensitive data if they can understand the meaning of the data as it arrives, unlike the vast pools of data often used for email. The problem of classification is different and needs to be approached differently, and current classification systems for servers do not provide an intuitive mechanism for users to have confidence that servers are classifying their data correctly.
This simple algorithm is useful for classifying data in databases containing millions or billions of records. The algorithm works well, provided that all relationships in the data are sufficiently different from each other and that the data is relatively small in both columns and rows. This makes data classification useful in systems with relatively little memory and little computation, and therefore the classification of large datasets remains a major unsolved problem.
The simplest classification algorithm for classifying data is the total correlation method, also known as the correlation method. In full correlation, you have two sets of data and you are comparing data from one set to data from another set. This is easy to do for individual pieces of data. The next step is to calculate the correlation between the two datasets. This correlation of two sets of data tells you what percentage of the data is in each set. Thus, using this correlation, you can classify data as either one set or the other, indicating the parts of the data set that come from one set or the other.
This simple method often works well for data stored in simple databases with a small amount of data and slow data access speeds. For example, a database system may use a tree structure to store data, with the columns of a record representing fields in the structure. This structure did not allow data to be ranked because the data would be in two separate rows of the tree structure. This makes it impossible to make sense of the data if the data fits in only one tree structure. If the database has two data trees, you will need to compare each of the two trees. If there were a large number of trees, the comparison could be computationally expensive.
Therefore, full correlation is a poor classification method. Data correlation does not distinguish between relevant parts of the data, and the data is relatively small in both columns and rows. These problems make full correlation unsuitable for simple data classification systems and data storage systems. However, if the data is relatively large, full correlation can be applied. This example is useful for storage systems with a relatively high computational load.
Combining a data classification method with a data storage system improves both performance and usability. In particular, the size of the resulting classification algorithm is largely independent of the size of the data store. The detailed classification algorithm does not require a lot of memory to store data at all. It is often small enough to be buffered, and many organizations store their classification systems this way. Also, the performance characteristics of the storage system do not depend on the classifier. The storage system can handle data with a high degree of variability.
Why are classification systems not so good?
Most storage systems do not have a good classifier, and the data classification system is unlikely to get better over time. If your storage system does not have a good classifier, your classification system will have problems.
Most companies don’t think this way about their storage systems. Instead, they assume that the system can be fixed. They see it as something that can be improved over time based on future maintenance efforts. This belief also makes it easy to fix some of the problems that come from bad storage systems. For example, a storage system that doesn’t accept overly short or jumbled data can be improved over time if more people are involved in fixing it.
Конец ознакомительного фрагмента.
Текст предоставлен ООО «Литрес».
Прочитайте эту книгу целиком, купив полную легальную версию на Литрес.
Безопасно оплатить книгу можно банковской картой Visa, MasterCard, Maestro, со счета мобильного телефона, с платежного терминала, в салоне МТС или Связной, через PayPal, WebMoney, Яндекс.Деньги, QIWI Кошелек, бонусными картами или другим удобным Вам способом.