UDP support in a federation context?

[soxofaan]

(This ticket is a manual copy of https://github.com/openEOPlatform/architecture-docs/issues/368 from Sept 2023, which is too hidden in a private repo)

This is follow-up ticket for #90 where we added initial UDP support in the aggregator: all UDP requests are just forwarded to VITO. This means that UDPs can currently only be used in process graphs that are executed on VITO back-end, because other backends will not know how to resolve the UDP references.

The problem is now how to take the next step and support UDPs in a real federated manner. This is not trivial as already hinted in #90 (comment)

This probably needs some discussion and maybe some additional specifications/conventions to get it working.

[soxofaan]

To set the stage and initiate discussion, some possible next steps and approaches:

• Full "multicasting" of all UDP CRUD operations to all upstream back-ends. While this is probably the most logical option, it is not trivial as this requires a lot of synchronization (read/write) logic at the level of the aggregator to manage the distributed UDP storage, for example:
  • how to handle failures and inconsistencies? (e.g. after downtime of certain upstream back-end)
  • how to verify consistency across multiple UDP storage sources? And related: how to determine source of truth?
  • how to handle conflicts with existing UDPs on a back-end
• store UDPs in a central place (microservice that allows URL based references to UDP) and replace name based UDP references to URL based UDP references when passing a process graph upstream. This eliminates the synchronization challenges but requires all upstream back-ends to support URL based UDP references, which isn't even standardized. (FYI In VITO backend we have an implementation for this)
• store UDPs in a central place (microservice) and the aggregator replaces UDP "calls" with "inlined" version of the UDP before passing to upstream back-end. This approach is easy for the upstream backends as they never see the UDPs. It's however not clear if "inlining" UDPs in a process graph is feasible and what pitfalls this brings.
• extend process graph spec to allow appending UDPs to the "main" process graph, so that UDP references are fully internal in the process graph body. This would even be an interesting approach outside the federation, but it requires a non-trivial extension of the openEO API spec

[soxofaan]

related:

• Allow URLs as process namespace openeo-api#515

[jdries]

Note that the solution where UDP's are referenced by url is probably the way forward, given that this is what we also use in ESA APEx. If it's not yet standardized, we'll want to look into that.
What about using the 'validate' call to check if a process with UDP can be executed against a given backend?

[soxofaan]

Note that the solution where UDP's are referenced by url is probably the way forward

While I also prefer this "remote process definition" approach, I'm not talking about that kind of UDPs here,
but the openEO's CRUD sub-API to store/get/... UDPs (under /process_graphs).

Or are you suggesting to go for the solution:

• Explicitly not implement the /process_graphs sub-API in the aggregator and require all "UDP" usage to be done with the remote process definition extension?

[jdries]

Correct, I am proposing to focus on promoting the remote process definition extension.
To support storing of user UDP's, the second option that you described also seems interesting:

store UDPs in a central place (microservice that allows URL based references to UDP) and replace name based UDP references to URL based UDP references when passing a process graph upstream. This eliminates the synchronization challenges but requires all upstream back-ends to support URL based UDP references, which isn't even standardized. (FYI In VITO backend we have an implementation for this)

I would add that the 'validate' call would then allow to check which backends support a process graph that contains a UDP, avoiding that the aggregator would have to try and 'unroll' UDP's itself.