Challenges and Problems
- Error correction and loss of information: The most challenging problem within data cleansing remains the correction of values to remove duplicates and invalid entries. In many cases, the available information on such anomalies is limited and insufficient to determine the necessary transformations or corrections, leaving the deletion of such entries as the only plausible solution. The deletion of data, though, leads to loss of information; this loss can be particularly costly if there is a large amount of deleted data.
- Maintenance of cleansed data: Data cleansing is an expensive and time-consuming process. So after having performed data cleansing and achieving a data collection free of errors, one would want to avoid the re-cleansing of data in its entirety after some values in data collection change. The process should only be repeated on values that have changed; this means that a cleansing lineage would need to be kept, which would require efficient data collection and management techniques.
- Data cleansing in virtually integrated environments: In virtually integrated sources like IBM’s DiscoveryLink, the cleansing of data has to be performed every time the data is accessed, which considerably decreases the response time and efficiency.
- Data-cleansing framework: In many cases, it will not be possible to derive a complete data-cleansing graph to guide the process in advance. This makes data cleansing an iterative process involving significant exploration and interaction, which may require a framework in the form of a collection of methods for error detection and elimination in addition to data auditing. This can be integrated with other data-processing stages like integration and maintenance.
Read more about this topic: Data Cleansing
Famous quotes containing the words challenges and, challenges and/or problems:
“The approval of the public is to be avoided like the plague. It is absolutely essential to keep the public from entering if one wishes to avoid confusion. I must add that the public must be kept panting in expectation at the gate by a system of challenges and provocations.”
—André Breton (1896–1966)
“The approval of the public is to be avoided like the plague. It is absolutely essential to keep the public from entering if one wishes to avoid confusion. I must add that the public must be kept panting in expectation at the gate by a system of challenges and provocations.”
—André Breton (1896–1966)
“I respect guilt. It is a dangerous but sometimes useful beast. The guilt that made me want to solve all my children’s problems meant trouble. The guilt that made me question my role in our mother-daughter squabbles proved helpful. Yes, I care about my kids’ problems, and I long to make suggestions. But these days I wait for children to ask for help, and I give it sparingly. Some things can’t be fixed, and I tell them so.”
—Susan Ferraro (20th century)