feat: adds reusable actions

[baywet]

This pull request adds action templates.

fixes #33
fixes #136
fixes #270

closes #238

This is another attempt to solve a scale limitation in the current specification. Action templates are better than the previous parameters proposal because:

• They do not introduce a matrix projection, people may use the template multiple times.
• They do not require "declaring" which environment variables to read in advance.
• They limit the string literal replacement to the templates, which does not impact how actions currently work.
• The template reference mechanism works like OpenAPI references.
• The template resolution mechanism can be isolated as a pre-processing step, and the resolved actions can then be fed to the existing code processing actions.
• It introduces a string literal replacement syntax which can be expanded upon for other sources or used at other places in the future.

The pull request is incomplete as it is, it's a draft, I want to collect feedback on the approach before making any further investments.

[ralfhandl]

I like the general direction.

[baywet]

Feedback from the meeting from both @lornajane and @ralfhandl :

• add a peer block to parameters for the environment variables, this allows to set defaults and "declare" which variables are used. But this declaration is optional.
• add a peer "actions" in the components, that's defined as everything except for the target. So people can specify the target on reference. Reusable actions may use action templates in turns.

[ralfhandl]

Mostly wording and capitalization

[baywet]

@ralfhandl @lornajane I pushed another update a couple of minutes ago. I wasn't happy about the whole reusable actions vs action templates kind of thing. After chatting with @mikekistler internally I realized we could simply define a an action template reference object as "you can override anything from the resolved template in the reference" like JSON schema does to some extent. And keep things extra simple.

Let me know what you think!

feedback implemented

[handrews]

A few minor comments and questions.

[lornajane]

I've got a lot of questions about the intent here (which I thought we'd already been through, so please accept my apologies for asking it again). I hadn't seen, or perhaps hadn't internalised, the use cases where target is getting rewritten in the reusable action as well as the action bit. I think this is beyond the scope of what I had previously seen and I'm uneasy about how approachable it is.

Also I think we use REQUIRED for fields that are required, rather than marking the ones that are not as Optional - so that needs to be consistent.

[lornajane]

I had thought it was not allowed to include the target. The target has to be with the action, but the action definition can be referred to as one of the reusable items from components. What's the use case for supporting action?

[baywet]

On this instance specifically, I think you've missed a few discussions, either directly here in the PR history, or on the Overlays call. I don't want to re-litigate the whole thing that late in the process. But at a high level:

• We believe that consistency is key, is there really any good reason to allow every field here but target?
• We believe that string interpolation + target are a powerful combination for advanced scenarios, that's actually lead to a huge reduction in my team's context "similar" actions.
• We believe we can also enable simpler scenarios (people who don't want to mess around with string interpolation), by allowing an override at the reference level.

Hopefully that's enough context, let me know if you still have questions or concerns.

[lornajane]

This change brings multiple features in at once, and I'm uneasy that we've missed the purpose of Overlays and leapt in a new "more is more" direction with this proposal which seems to bring a lot of complexity and also brings back many aspects that have previously been discarded as not a good fit - either for being too much change in a single leap, or just being out of scope for what Overlays is.

I'm supportive of actions being reusable to avoid repetition. I'm not in favour of configurable targets which would allow a matrix of difficult to diagnose or reason about operations. I've consistently expressed these views, I'm not sure it's fair to frame it as me having "missed" something.

[baywet]

I think I may have misunderstood your concern earlier. Based on your initial comment on the earlier proposal and from what I recall from our conversations, I had understood the main sticking point to be the matrix aspect. Essentially, I got a sense that "nobody wants to do math in overlays, it brings too much cognitive load". This is why I rewrote the thing entirely, to start with new design principles and avoid carrying this over.

I'll be honest — getting this feedback at this stage is frustrating, as the draft has been open for two months and this specific concern about target first came up 8 days ago. I recognize async communication is lossy, and I may have missed implicit signals, but it does make it harder to iterate when fundamental concerns surface this late.

Setting that aside, I'd like to better understand your concern about allowing string interpolation in target. Is it primarily about security, readability, invalid JSONPath after resolution, or overall spec complexity? If you can point to a concrete failure mode or example, that would help me think about alternatives.

The reason I've been pushing for interpolation in target/copy is that it enables patterns like the sidecar example. Another example would be cases where you have a dozen endpoints each with a different schema and want to introduce a status monitor for each, without relying on generic matching. I'm happy to update examples/test cases if they are not illustrative enough.

I'd also like to understand how you'd see this applying to copy, since I think consistency between the two matters.

I'm hoping this helps us narrow the disagreement and find a workable direction. :)

[lornajane]

I'm sorry I've been unable to schedule myself into the meetings but let me try to recap where I think we are:

• we're bringing in a few different features here
• one is reusable actions to avoid copy/pasting the same action for lots of target expressions
• we're also introducing parameters for those actions so that as well as allowing copy/paste, it's possible to adjust

All this is good so far, it's a big amount of change and does change the nature of Overlays as a project - but innovation is good and we have clear use cases and good examples that will help.

What I'm questioning here is the reusable targets. We've talked about interpolation in the target of an action object. I understood the reusable actions to be copy or updates with parameters, and parameters potentially being pulled into the description or target of the actual (not reusable) action. But seeing reusable targets, I had doubts and had assumed that further discussion was still welcome.

[baywet]

Hey @lornajane
Apologies. During the last Overlay meeting, I had committed to put together a concrete example of why I believe string interpolation support in targets is important. Health issues and other priorities came in the way of doing this.

Here is a gist that illustrates the difference in overlay experience, please have a look, and let me know whether it addresses your doubts regarding the usefulness of this aspect.

Essentially my concern is to enable reusable actions "components", where users only need to "fill in the parameters" and we get a consistent, easy to maintain, low thinking required experience from the users of the reusable actions.
I realized this maybe tailored to larger organizations, where large overlay documents are in use and where the people building the reusable actions and the one "referencing them" are not necessarily the same person. But I still think this is a genuine use case.

[lornajane]

What we have created here is a reasonably fully-featured overlay generator. It's quite a big departure from a lightweight format for describing changes to an API description. It adds overhead to Overlay tool implementations, and makes Overlays a machine-only, intermediate format rather than read/write for humans. I'd like us to discuss how this fits the overall project goals before bringing in any of the various features included here. I think it would also be better to introduce one feature at a time - I realise that won't be a popular suggestion after so much (really excellent standard of) work went into this, but I can't see that this proposal can be merged in its current form.

[baywet]

@lornajane we've been "stuck" for weeks now. I'm currently on the road and won't be able to attend the next few meetings either.
I believe your biggest objection is around string interpolation. I've tried to clarify the misunderstandings in this thread but I'm not sure yet we've reached a common understanding of the disagreement here? Can you please clarify with specifics?

Essentially: how do we get out of "we're stuck" loop? I've tried to provide examples, refine things etc...?

I was about to type an explanation of what it means to remove string interpolation (would lead to removal of parameters, environment variables), and ask why you have any opposition to string interpolation. But if we're not even on the same page about the scope of things, I'm not sure it's worth exploring yet?

I'd like us to discuss how this fits the overall project goals before bringing in any of the various features included here

This is going to be really difficult to do if we don't either create a dedicated issue on the topic to scope the project, or achieve being on the same meeting at the same time...

[lornajane]

The fundamental problem as I see it is that I do not think these features should be added to overlays all at once like this. However that feedback does not seem to be getting heard. Instead you seek to "clarify misunderstandings". Assuming that I did not understand is why we are stuck.

[baywet]

No, I wouldn't be that presemptuous, it's my confusion I'm trying to clarify.

So, if we break it down:

• would you be ok bringing reusable actions as is? (Most likely no?)
• would you be ok bringing them without string interpolation for copy and target? (Maybe?)
• would you be ok bringing reusable actions without any string interpolation at all? (And no parameters or environment variables)
• would you be ok bringing any kind of reusable action? (Most likely yes?)

In trying to get a good sense of where you draw the line so I can take actions from that or at least make a detailed case about any difference of opinion?

Did the example for interpolated targets help at all making a case for them?

[handrews]

@baywet @lornajane perhaps it would help to break down the "multiple features" involved and look at what is lower vs higher risk, and what the dependencies are? I'm going to try to list what I see, although not all options here are exactly as literally present in this PR.

• Actions vs Parametrized Actions: These could be distinct re-usable types, as long as non-parametrized Actions are worth re-using. Adding non-parametrized ones would be straightforward.
• Overriding References vs non-overriding: Orthogonal to the above. This could be a follow-on PR to either or (eventually) both of the above.
• String replacement, parameters vs environmental: String replacement syntax is always tricky and worth a lot of focus, although it does seem to have been discussed a lot already. Using parameters vs environment variables could be done separately, with either available first.

Don't take any of the above too literally in terms of suggestions, I'm just trying to provide another perspective on how this might be broken up, or what some intermediate things are that could go out in a 1.2 vs a 1.3. Doing multiple small releases (1.1 was only a few months ago) would be a good way to get feedback. The Overlay tool requirements are smaller which is a big benefit for rapid iteration. You can release more often than the OAS can (or even Arazzo, probably), so incrementalism is a much more viable path here.