Update orderless-escapable-split-on-space to support other characters

[dadinn]

Allows customising the escapable component separator character to be other than just a space. This feature was originally requested/discussed here.

Introduces a new customisable variable orderless-escapable-separator, which allows any character to be used as separator, not just the default space character (to be backwards compatible).

Example with corfu:

Example with vertico:

The above examples use the following orderless-matching-styles:

(setq orderless-matching-styles
   '(orderless-flex orderless-regexp))

[minad]

Thanks for the proposal. I am a bit skeptical about this addition, since I find it a bit too special. I am unsure if many users will need it, since it hasn't come up in discussion yet, except the linked issue. One can always write a custom splitting function if really needed. Can you show a real use case where we need this? In Corfu and in minibuffer completion in general the space separator works well.
@oantolin What do you think?

[minad]

It seems @oantolin had already rejected this feature request in #102 (comment). I think this still holds without further evidence.

[dadinn]

I found that company in particular does not work well with space as separator, at least in some of the languages I've tried.

IIRC, it happens when I type up the pattern first in the buffer, then try to use completion-at-point in the end. Most languages don't allow space in symbols. One might find this a bit contrived use-case, because if the separator character used is not a valid character for symbols, it causes a syntax error upfront. A syntax error then would prevent any completion-at-point in that case, which kind of defeats the purpose.

For example using a single | in Scheme when typing up the pattern fully first, causes syntax a syntax error.

This is not a problem in mini-buffer completion though (like with vertico).

Also, the component separator disappears after the completion is accepted, so any syntax errors due to a pattern are irrelevant afterwards.

[dadinn]

Also, the my implementation is supposed to be backwards compatible with the current spacey behaviour.

It only adds a new custom variable orderless-escapable-separator, which is by default the original space character.

[dadinn]

Actually, I would go even further.

In particular, with corfu the issue is even more glaring (because of its explicitly stated support for orderless multi-component matching). As stated above, with space as separator, one cannot just type up a pattern upfront, and then press completion-at-point. In most languages a space would terminate any symbol, therefore all components except the last would be ignored. Also, even | and , characters have the same problem. For example, none of these are valid characters for a symbol in lisp and scheme. They would either terminate the symbol (and pattern matching would ignore the preceding components), or it would be a syntax error, and block completion altogether.

I assume currently corfu only intends to work with components by entering them after the completion is already in progress. Instead, personally I often find myself typing out part of some multi-component pattern before pressing completion-at-point, which then I have to erase and try again from the beginning, and build it up step-by-step.

I've updated this PR, so that the orderless-escapable-separator can now be safely set to buffer-local character value . This would allow setting alternative characters for major-modes individually, for example in a project-specific .dir-locals.el file).

I've also created a PR for corfu, so that corfu-separator can be also set to a function, or a potentially buffer-local variable symbol, so as to dynamically look up the separator character:
minad/corfu#622

This allows the corfu-separator character to be retrieved dynamically, for example, by setting it to either (lambda () orderless-escapable-separator), or just 'orderless-escapable-separator directly.

One might consider this implementation a bit elaborate... but I and I likes it!
Check out this example:

(setq-default orderless-escapable-sepalator ?\|) ; custom global value
(setq corfu-separator 'orderless-escapable-separator)

(setq-local orderless-escapable-separator ?\-) ; buffer-local override

(let ((expired-count 42)
      (excitement nil))
   ex-\-c-nt$ ; assume the cursor now is after this dollar sign
)

At this point calling completion-at-point would auto-complete to expired-count.

[minad]

I assume currently corfu only intends to work with components by entering them after the completion is already in progress. Instead, personally I often find myself typing out part of some multi-component pattern before pressing completion-at-point, which then I have to erase and try again from the beginning, and build it up step-by-step.

This is certainly a good reason, but the use of alternative characters as separators also clashes somehow with the completion mechanism. I am still unsure if adding further complexity for this is justified (also in Corfu minad/corfu#622), but I will try these patches and then I will see.

EDIT: The problem is that the initial input must often be a prefix of the completion, due to the backends when they initially lookup candidates, so we won't win that much in many cases. While Orderless controls the filtering it does not control how completion is initiated - this is up to the backend.

[minad]

I've also created a PR for corfu, so that corfu-separator can be also set to a function, or a potentially buffer-local variable symbol, so as to dynamically look up the separator character:
minad/corfu#622

Why? This does not seem to be needed. You can simply set corfu-separator buffer locally and it will work. It doesn't seem problematic to set both variables in Orderless and Corfu, maybe except for file-local variables, if you want to keep them in sync?