The Life Cycle of Data Curation
For being the most critical element in data analysis and science, this topic is not talked about enough. Maintaining the data clean and ready for reuse is critical for any mission.
When I first joined the military, I lived in the barracks. One small room shared with another soldier.
The requirements were strict. My uniforms had to be hung a certain way, my civilian clothes in a separate stack. The shoes had to be lined up in a certain order. The room had to be clean at all times. Inspections were often and unannounced.
Images of how everything should be setup were taped into my cabinet. The standards were clear and expected to be followed.
Failing to comply with any of these requirements could result in taking all of the unattached furniture, clothes, and belongings outside and I would spend the rest of the day doing a deep cleaning.
Some of my peers were not really interested in keeping their rooms clean. They took the risk of being caught with trash cans overflowing or dirty laundry on the floor. I didn’t think it was worth it. I learned that maintaining a clean room was much easier.
Data curation is no different. Cleaning, maitaining, ensuring it passes inspection at any time of the day or night.
I learned that maintaining your data clean and ready for usage is one of the most critical roles in data analytics.
What is Data Curation?
Data curation is the ongoing process of collecting, organizing, cleaning, annotating, and preserving datasets so it can be reused.
Minamiyama et al. (2024) describe the main purpose of data curation as improving research efficiency by making research data more accessible while improving its utility in the data discovery process.
When effectively done, it allows for integrating metadata, documentation, and proper storage strategies, enabling researchers and analyst to locate relevant datasets quickly.
Curation goes beyond just storage. It enhances research efficiency and reproducibility, amplifying long-term scientific impact by ensuring proper documentation (Marsolek et al., 2023).
Curation Standards
Most organizations have data silos. Sales uses one software and Human Resources another. Those systems don’t talk to each other, which immediately creates a silos.
For someone trying to get an overview of the whole company, combining these systems can be problematic. To make matters worse, less than 40% of companies have data oversight.
When dealing with a lot of data, one of the critical elements is to create a standard. Much like the images attached to my cabinet in my barracks, standards should be available to anyone working with curation or data ingestion.
The decision on how to archive, manage, preserve, and retrieve all contribute to the integrity and usability of the data. Integrating these practices into a structured framework will optimize data discovery while promoting reproducibility and interoperability.
Core Tasks in Data Curation
Data Authentication
You want to ensure the data is credible. In the data authentication process, you are verifying the validity and provenance of data before it is used. Aside from credibility, integrity and attribution also fall within this task.
Techniques like checksum verification, digital signatures, and metadata validation are employed to authenticate the dataset. Authentication play a crucial role in preventing the use of compromised or innacurate data, ensuring reliability of the analysis.
Data Archiving
A lot of data can be reused. In some cases, data is required to be kept for a certain amount of time.
Properly archiving will also free up space in the hot data improving speed. The last thing you want to to have a data rot case making it unusable.
Archiving requires careful consideration in storage, formats, metadata, and accessibility. Common practices include the use of version control systems, cloud based repositories, and backup strategies.
The proper storage, preserving the historical context of the data avoids swamps creation.
Data Management
This is a big one. Managing the data often ends in finger pointing. Specially when multiple sources are involved and no one is fully managing them.
This task includes a wide range of activities: data collection, cleaning, integration, and monitoring.
Proper managing the data is like maintaining a barrack room. You have to organize, update to remain compliant. It also involves creating a data catalog, which also has to be maintained.
The idea is to make the data easily discoverable, and usable across the organization.
Data Preservation
Working in the cybersecurity industry, data comes cheap and the database is flooded with computer logs. Loosing a set of logs from one day might not make much of a difference. But that is not the same for other industries.
Arthur et al. (2025) claim that data preservation is crucial when dealing with irreplaceable research data. IN some cases, the replication of the lost data might be high cost or nearly impossible.
Imagine data being collected from a research on rare diseases.
The process of preservation requires bit-level preservation, and periodic data audits. This allows the data to remain accessible and interpretable for future research and decision-making.
Data Retrieval and Representation
Of course, data retrieval is critical in the process of curation. If you can’t pull the data, then what good is it? Retrival systems are designed to ensure users can locate relevant data quickly. When the data is optimized, the retrival minimizes downtime and enhances productivity in research and business environments.
Data representation is a combination of vizualization, formatting, and contextualization, enhancing interpretability and usability (Franciscani et al, 2014). In this stage the data needs to be presented in a way that is meaningful to the stakeholders.
Collection and Curating the Data
Data is being generated and collected daily. Perhaps not like Walmart, which collecting around 2.5 pentabytes an hour. But think about your phone. It has a GPS that is tracking you. Your daily calls and text messages. All of those things are collecting data and storing somewhere.
Often the data coming in might not be complete or meet the quality threshold. It doesn’t vanish. It sits there with the good data. Simply because the data was collected, it doesn’t mean it is usable.
Let’s take a small coffee shop for example. Let’s say the sales have been dropping and the owner is looking at the data for answers.
Naturally, the first place they will look is the POS. Often one data source doesn’t tell the whole story. It would only tell the people who actually had a transaction. Even for a small coffee shop, that is not all the data it generates.
Website traffic, wifi connection, phone and email logs are just some examples of other sources that could provide an insight into the problem.
In larger settings, a well-integrated system is more valuable than separate sources because it creates synergy, allowing for a deeper analytical insight and more accurate predictive modeling.
Conclusion
Effective data curation is critical. Data needs to remain accessible and reliable to be valuable.
The most critical element in curation is creating a standard where people can easily follow. As data continue to grow, building structure methodologies that safeguard data integrity maximizes utility.
Prioritizing rigorous curation practices becomes paramount as data continues to expand across disciplines. Specially when the focus becomes automation and machine learning.
References:
Arthur, B., Ofori, Y., Arthur, C., Afriyie, K., Agyekum, B. O., Dukper, K. B., Baidoo, D. K., Akotey, R., Filson, C. K., Taylor, J., Humphrey-Ackumey, S. a. Y., & Aviamu, Y. A. (2025). Data preservation, sharing and reuse among graduate students in Ghana. Information Development. https://doi.org/10.1177/02666669241313367
Marsolek, W., Wright, S. J., Luong, H., Braxton, S. M., Carlson, J., & Lafferty-Hess, S. (2023). Understanding the value of curation: A survey of researcher perspectives of data curation services from six US institutions. PLoS ONE, 18(11), e0293534. https://doi.org/10.1371/journal.pone.0293534
Minamiyama, Y., Takeda, H., Hayashi, M., Asaoka, M., & Yamaji, K. (2024). A study on formalizing the knowledge of data curation activities across different fields. PLoS ONE, 19(4), e0301772. https://doi.org/10.1371/journal.pone.0301772
Franciscani, G., Jr, Santos, R. L. T., Ottoni, R., Pesce, J. P., Meira, W., Jr, & Melo-Minardi, R. (2014). An Annotation Process for Data Visualization Techniques. The Third International Conference on Data Analytics, 24–32. https://www.thinkmind.org/download.php?articleid=data_analytics_2014_1_50_60144
No comments yet. Be the first.