Do we store raw data or only normalized data?

[cmutel]

There are at least three levels:

1. Store only harmonized and normalized data
2. Store raw input data in a way that can be accessed during the data pipeline
3. Archive raw input data but don't make it available to data pipelines

Arguments for storing raw data:

• Storing raw data can make it easier to go back and change our normalization and filtering strategies.
• It ensures that the original source is available.
• We are building on open data sources so storing raw data shouldn't raise legal questions.

Arguments against requirements to store raw data:

• Burns carbon and money
• Some data sources are database accessed via API, so storing raw data isn't an option.
• Data sources could be very large (e.g. large statistical datasets or remote sensing files), and we only need a small fraction of the data for our purposes.

[tngTUDOR]

Databases people would normally say that you don't need to "store" what can be calculated, but maybe the strongest argument here is that we need to provide as much transparency as possible.

One possible compromise would be to store raw data, only for a specific period of time, and discard it afterwards.

[jsvgoncalves]

For traceability and reproducibility we should find ways to store all raw data, but we can probably outsource e.g. to zenodo?
If a model as specific "translations" it needs to do, we should treat that as another model that other future models can reuse.
We can cache transformed data (even if at a physical level) if needed for performance reasons.
How do we make sure that when you point to a glossary entry, you are pointing to right version of that entry? Can we make parallels with git commits (maybe via a checksum) and git tags or even just versions?

[simontaurus]

Beside git, there are also versioning schemas in owl https://www.w3.org/TR/owl2-syntax/#Versioning_of_OWL_2_Ontologies, or more general referenced resources can be deployed at versioned api endpoints, e. g. https://my.domain.com/schema/v0.0.1/context.jsonld

[simontaurus]

Combining your arguments I would suggest to use a self-describing container format like https://www.researchobject.org/ro-crate/ that containes the processed data and optionally the raw data or at least references (URLs) to it. Storing it on Zenodo beside archived and versioned source code repos would provide reproducibility and long-term availability at no/low costs.