Split database population from database backend management

[cmutel]

Our current repo is a database schema, API, (mini-)frontend, and database population script. This isn't great for several reasons:

• It makes it much harder for people to contribute or understand the code base if they just want to add a new vocabulary
• It incentivizes data population scripts to get lazy and take shortcuts because the schema and interfaces are right there
• All the documentation and testing needs to cover many use cases

Therefore, I propose that we rename this repo sentier_glossary_BAR, where BAR is something like "backend", and shift the definitions of new datasets which populate the database to sentier_glossary_data. We should also change sentier_glossary to sentier_glossary_client (or maybe sentier_glossary_python_client)?

Pinging @sami-m-g @jsvgoncalves @tngTUDOR

[tngTUDOR]

BAR -> "builder" ? sentier_glossary_client_python <- so that we can add more client langs easily

…

________________________________ From: Chris Mutel ***@***.***> Sent: Thursday, May 23, 2024 14:34 To: Depart-de-Sentier/dds_glossary ***@***.***> Cc: Tomas NAVARRETE ***@***.***>; Mention ***@***.***> Subject: [Depart-de-Sentier/dds_glossary] Split database population from database backend management (Discussion #60) Our current repo is a database schema, API, (mini-)frontend, and database population script. This isn't great for several reasons: * It makes it much harder for people to contribute or understand the code base if they just want to add a new vocabulary * It incentivizes data population scripts to get lazy and take shortcuts because the schema and interfaces are right there * All the documentation and testing needs to cover many use cases Therefore, I propose that we rename this repo sentier_glossary_BAR, where BAR is something like "backend", and shift the definitions of new datasets which populate the database to sentier_glossary_data. We should also change sentier_glossary to sentier_glossary_client (or maybe sentier_glossary_python_client)? Pinging @sami-m-g<https://github.com/sami-m-g> @jsvgoncalves<https://github.com/jsvgoncalves> @tngTUDOR<https://github.com/tngTUDOR> — Reply to this email directly, view it on GitHub<#60>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACPXT63XSGJCUO5R2RNUSRTZDXO6DAVCNFSM6AAAAABIFR5OZOVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZWG4YTONZQGQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[cmutel]

so that we can add more client langs easily

Exactement

[cmutel]

"builder"

For the input data wrangling or the actual backend?

[tngTUDOR]

For the backend, right?

…

________________________________ From: Chris Mutel ***@***.***> Sent: Thursday, May 23, 2024 16:26 To: Depart-de-Sentier/dds_glossary ***@***.***> Cc: Tomas NAVARRETE ***@***.***>; Mention ***@***.***> Subject: Re: [Depart-de-Sentier/dds_glossary] Split database population from database backend management (Discussion #60) "builder" For the input data wrangling or the actual backend? — Reply to this email directly, view it on GitHub<#60 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACPXT62MPOEDVDM6QHAJDKDZDX4CNAVCNFSM6AAAAABIFR5OZOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TKMZVHAYDA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[sami-m-g]

@cmutel @jsvgoncalves @tngTUDOR
I think it makes sense to do the split of responsibilities, but I suggest the approach of a monolithic repo with multiple packages. Maintaining the CICD pipelines alone for a multi-repo approach would be a bit of a nightmare given the frequent changes we are undergoing. Most big companies (Google, Facebook, and Microsoft to name a few) use the monolithic approach.

[jsvgoncalves]

Thanks everyone for the comments; I have materialised our conversations of the past weeks in our first ADR which you can find on our meta repo: https://github.com/sentier-dev/sentier.dev/blob/main/Engineering/ADRs/ADR_1_MultiRepo.md.

I'll close this discussion as solved.