Redefine command-buffer simultaneous-use

As discussed in KhronosGroup#891
the current definition of [simultaneous
use](https://registry.khronos.org/OpenCL/specs/3.0-unified/html/OpenCL_API.html#_command_buffers)
is hard for users to reason about.

Instead a better model is one simultaneous-use isn't a contraint of the
command-buffer submission, but it's execution. That is simultaneous-use
occurs when a command-buffer is enqueued for execution while any
previous submission of the command-buffer is still in-flight, and there
any no scheduling dependencies expressed to serialize execution.

This means that pipelined submissions of a command-buffer, where there
is an in-order queue, barrier, or cl_events to serialize execution, is
always valid usage without this optional simultaneous-use device
feature.

To avoid the runtime having to incur overheads from monitoring when
simultaneous-use occurs so it can throw an error, violating this
valid usage for non simultaneous-use command-buffers is UB.

Following from this related to KhronosGroup#1311
the pending state has also been removed from the command-buffer
lifecycle, and there is only the binary states of recording and
executable. This is because a user can use the existing OpenCL
mechanisms of host waits and event queries to inspect the state
of individual command-buffer enqueues to avoid simultaneous-use,
and having this stored as a command-buffer state incurs the runtime
overhead of tracking a previous submissions. The pending count
concept also now makes less sense.

The implications for updating a command-buffer is that the error
behavior is removed from update, so simultaneous-use is a property of
execution rather than enqueue. See CTS test ideas for how this could
work.

I think the CTS changes that are required for this as follows, but I could
create a separate CTS issue to track work once/if this PR merge. Or we
could try prototype the CTS changes to give confidence in the spec
change.
* Remove test for pending state query
* Either rework existing simulataneous use tests so that it tests the new definition,
  or delete them and create new more suitable tests without tech-debt
  from old definition.
* Add tests for pipelined submission of a command-buffer not created
with simultaneous-use. Ideally stressing indriect dependencies as well as direct
ones:
** in-order queue to express depdencies
** out-of-order queue with event dependencies to express depenencies
** barrier to express for dependencies
* Add cl_khr_command_buffer_mutable_dispatch tests for updating and
enqueueing pipelined submissions of a non-simultaneous use command-buffer with depdencies between each
enqueue, and only do a blocking wait at the end.
* Add cl_khr_command_buffer_mutable_dispatch tests for a
simultaneous-use command-buffer, where two invocations are scheduled
such that they can run concurrently, but the second invocation is
updated such that it uses different inputs/outputs to avoid race
conditions in the kernel.

Author: Ewan Crawford

api/cl_khr_command_buffer.asciidoc

@@ -4,15 +4,15 @@
 include::{generated}/meta/{refprefix}cl_khr_command_buffer.txt[]
 
 // *Revision*::
-//    0.9.6
+//    0.9.7
 // *Extension and Version Dependencies*::
 //     This extension requires OpenCL 1.2 or later.
 //     Buffering of SVM commands requires OpenCL 2.0 or later.
 
 === Other Extension Metadata
 
 *Last Modified Date*::
-    2024-12-13
+    2025-07-10
 *IP Status*::
     No known IP claims.
 *Contributors*::
@@ -126,11 +126,16 @@ from the sync-point values returned is implementation defined.
 
 ==== Simultaneous Use
 
-The optional simultaneous use capability was added to the extension so that
-vendors can support pipelined workflows, where command-buffers are repeatedly
-enqueued without blocking in user code. However, simultaneous use may result in
-command-buffers being more expensive to enqueue than in a sequential model, so
-the capability is optional to enable optimizations on command-buffer recording.
+The optional <<simultaneous-use, simultaneous use>> capability was added to the
+extension so that vendors could support concurrent execution of the same
+command-buffer. However, simultaneous use may result in command-buffers having
+a larger overhead to implement, so the capability is optional to enable
+optimizations when this usage isn't required by a user.
+
+Instead the goal for the base level of functionality provided by the extension
+is to support pipelined workflows, where a command-buffer is repeatedly
+enqueued, with each enqueue expressing depdencies on any previous submissions,
+without the enqueue calls blocking in user code.
 
 === Interactions With Other Extensions
 
@@ -257,7 +262,6 @@ features:
   * {cl_command_buffer_state_khr_TYPE}
   ** {CL_COMMAND_BUFFER_STATE_RECORDING_KHR}
   ** {CL_COMMAND_BUFFER_STATE_EXECUTABLE_KHR}
-  ** {CL_COMMAND_BUFFER_STATE_PENDING_KHR}
   * {cl_command_type_TYPE}
   ** {CL_COMMAND_COMMAND_BUFFER_KHR}
   * New Error Codes
@@ -470,3 +474,6 @@ features:
   * 0.9.7, 2024-12-13
   ** Refactor queue compatability between command-buffer creation and enqueue
      (experimental).
+  * 0.9.8, 2025-07-10
+  ** Rework simultaneous use definition and remove pending state
+     (experimental).

api/opencl_runtime_layer.asciidoc

@@ -14618,13 +14618,14 @@ on one or more command-queues without any application code interaction.
 Grouping the operations together allows efficient enqueuing of repetitive
 operations, as well as enabling driver optimizations.
 
-Command-buffers are _sequential use_ by default, but may also be set to
-_simultaneous use_ on creation if the device optionally supports this
-capability.
-A sequential use command-buffer must have a <<pending_count, Pending Count>>
-of 0 or 1.
-The simultaneous use capability removes this restriction and allows
-command-buffers to have a <<pending_count, Pending Count>> greater than 1.
+Upon creation a command-buffer is defined as being in the <<recording,
+Recording>> state, in order for the command-buffer to be enqueued
+it must first be finalized using {clFinalizeCommandBufferKHR} after which no
+further commands can be recorded. A command-buffer is submitted for execution
+on command-queues with a call to {clEnqueueCommandBufferKHR}. It is
+always valid to call {clEnqueueCommandBufferKHR} with a command-buffer that
+has previosuly been enqueued, provided the call doesn't violate the definition
+of <<simultaneous-use, simultaneous use>>.
 
 Command-buffers are created using an ordered list of command-queues that
 commands are recorded to and execute on by default. All these queue objects
@@ -14690,6 +14691,24 @@ If using layered extension {cl_khr_command_buffer_mutable_dispatch_EXT},
 usage>>.
 ====
 
+Simultaneous use is defined using the _prerequisite_ terminology from the
+<<_execution_model, execution model>>, and is an optional feature for devices
+to support concurrent executions of a command-buffer. A command-buffer must
+be created with {CL_COMMAND_BUFFER_SIMULTANEOUS_USE_KHR} to avoid undefined
+behavior if a simultaneous use usage pattern occurs.
+
+[[simultaneous-use]]
+Simultaneous Use:: When a command-buffer is submitted for
+execution without a prerequisite on all the previous submissions of the same
+command-buffer which are not in the {CL_COMPLETE} state.
+
+An example of simultaneous use would be two submissions of the same
+command-buffer to a single out-of-order queue, without any events or barriers
+used to express a dependency between the two enqueue calls. Using a single
+in-order queue, events, or barriers to express depdencies between submissions
+of the same command-buffer would each be ways to avoid simultaneous use and are
+valid usage of command-buffers created without the
+{CL_COMMAND_BUFFER_SIMULTANEOUS_USE_KHR} flag.
 
 ifdef::cl_khr_command_buffer_multi_device[]
 === Command-Buffers and Multiple Devices
@@ -14733,7 +14752,9 @@ endif::cl_khr_command_buffer_multi_device[]
 
 === Command-Buffer Lifecycle
 
-A command-buffer is always in one of the following states:
+A command-buffer is created in the recording state and transitions to the
+executable state when finalized, at which point it cannot move back to
+the recording state.
 
 [[recording]]
 Recording:: Initial state of a command-buffer on creation, where commands can be
@@ -14743,11 +14764,6 @@ recorded to the command-buffer.
 Executable:: State after command recording has finished with
 {clFinalizeCommandBufferKHR} and the command-buffer may be enqueued.
 
-[[pending]]
-Pending:: Once a command-buffer has been enqueued to a command-queue it enters
-the Pending state until completion, at which point it moves back to the
-<<executable, Executable>> state.
-
 // Image generated from the following mermaid diagram description using https://mermaid.live
 // Ideally we'd use the asciidoctor-diagram extension to generate the rendered diagram, but
 // there are issues installing the gem with ruby 2.3.3
@@ -14757,21 +14773,10 @@ the Pending state until completion, at which point it moves back to the
 // stateDiagram-v2
 //     [*] --> Recording: Create
 //     Recording -->Executable: Finalize
-//     Executable --> Pending: Enqueue
-//     Pending --> Executable: Completion
 // ....
 
 image::images/commandbuffer_lifecycle.png[align="center", title="Lifecycle of a command-buffer."]
 
-[[pending_count]]
-The Pending Count is the number of copies of the command
-buffer in the <<pending, Pending>> state.
-By default a command-buffer's Pending Count must be 0 or 1.
-If the command-buffer was created with
-{CL_COMMAND_BUFFER_SIMULTANEOUS_USE_KHR} then the command-buffer may have a
-Pending Count greater than 1.
-
-
 === Creating Command-Buffer Objects
 
 [open,refpage='clCreateCommandBufferKHR',desc='Create a command-buffer',type='protos']
@@ -14813,9 +14818,9 @@ include::{generated}/api/version-notes/CL_COMMAND_BUFFER_FLAGS_KHR.asciidoc[]
       | This is a bitfield and can be set to a combination of the following values:
 
         {CL_COMMAND_BUFFER_SIMULTANEOUS_USE_KHR_anchor} - Allow multiple
-        instances of the command-buffer to be submitted to the device for
-        execution.
-        If set, devices must support
+        instances of the command-buffer to be scheduled for execution on the
+        device in a usage pattern that exhibits <<simultaneous-use,
+        simultaneous use>>. If set, devices must support
         {CL_COMMAND_BUFFER_CAPABILITY_SIMULTANEOUS_USE_KHR}.
 
 include::{generated}/api/version-notes/CL_COMMAND_BUFFER_SIMULTANEOUS_USE_KHR.asciidoc[]
@@ -14898,16 +14903,6 @@ ifdef::cl_khr_command_buffer_multi_device[]
 |====
 endif::cl_khr_command_buffer_multi_device[]
 
-[NOTE]
-====
-Upon creation the command-buffer is defined as being in the
-<<recording, Recording>> state, in order for the command-buffer to be enqueued
-it must first be finalized using {clFinalizeCommandBufferKHR} after which no
-further commands can be recorded.
-A command-buffer is submitted for execution on command-queues with a call to
-{clEnqueueCommandBufferKHR}.
-====
-
 // refError
 
 {clCreateCommandBufferKHR} returns a valid non-zero command-buffer and
@@ -15089,9 +15084,6 @@ execution was successfully queued, or one of the errors below:
   * {CL_INVALID_COMMAND_BUFFER_KHR} if _command_buffer_ is not a valid
     command-buffer.
   * {CL_INVALID_OPERATION} if _command_buffer_ has not been finalized.
-  * {CL_INVALID_OPERATION} if _command_buffer_ was not created with the
-    {CL_COMMAND_BUFFER_SIMULTANEOUS_USE_KHR} flag and is in the <<pending,
-    Pending>> state.
   * {CL_INVALID_VALUE} if _queues_ is `NULL` and _num_queues_ is > 0, or
     _queues_ is not `NULL` and _num_queues_ is 0.
   * {CL_INVALID_VALUE} if _num_queues_ is > 0 and not the same value as
@@ -15125,6 +15117,10 @@ execution was successfully queued, or one of the errors below:
     required by the OpenCL implementation on the host.
 --
 
+Calling {clEnqueueCommandBufferKHR} in a usage pattern that exhbits
+<<simultaneous-use, simultaneous use>> when _command_buffer_ was not created
+with the {CL_COMMAND_BUFFER_SIMULTANEOUS_USE_KHR} flag results in undefined
+behavior.
 
 === Recording Commands to a Command-Buffer
 
@@ -16553,8 +16549,7 @@ include::{generated}/api/version-notes/clRemapCommandBufferKHR.asciidoc[]
   * _errcode_ret_ returns an appropriate error code.
     If _errcode_ret_ is `NULL`, no error code is returned.
 
-The returned command-buffer has the same state as the input command-buffer,
-unless the input command-buffer is in the <<pending, Pending>> state, in
+The returned command-buffer has the same state as the input command-buffer.
 which case the returned command-buffer has state <<executable, Executable>>.
 
 // refError
@@ -16682,10 +16677,6 @@ one of the errors below is returned:
   * {CL_OUT_OF_HOST_MEMORY} if there is a failure to allocate resources
     required by the OpenCL implementation on the host.
 
-Using this function when _command_buffer_ is in the <<pending, pending>>
-state and not created with the {CL_COMMAND_BUFFER_SIMULTANEOUS_USE_KHR} flag
-causes undefined behavior.
-
 [NOTE]
 ====
 Performant usage is to call {clUpdateMutableCommandsKHR} only when the
@@ -16903,18 +16894,10 @@ include::{generated}/api/version-notes/CL_COMMAND_BUFFER_STATE_KHR.asciidoc[]
 include::{generated}/api/version-notes/CL_COMMAND_BUFFER_STATE_RECORDING_KHR.asciidoc[]
 
         {CL_COMMAND_BUFFER_STATE_EXECUTABLE_KHR_anchor} is returned when
-        _command_buffer_ has been finalized and there is not a <<pending,
-        Pending>> instance of _command_buffer_ awaiting completion on a
-        command_queue.
+        _command_buffer_ has been finalized.
 
 include::{generated}/api/version-notes/CL_COMMAND_BUFFER_STATE_EXECUTABLE_KHR.asciidoc[]
 
-        {CL_COMMAND_BUFFER_STATE_PENDING_KHR_anchor} is returned when an
-        instance of _command_buffer_ has been enqueued for execution but not
-        yet completed.
-
-include::{generated}/api/version-notes/CL_COMMAND_BUFFER_STATE_PENDING_KHR.asciidoc[]
-
 | {CL_COMMAND_BUFFER_PROPERTIES_ARRAY_KHR_anchor}
 
 include::{generated}/api/version-notes/CL_COMMAND_BUFFER_PROPERTIES_ARRAY_KHR.asciidoc[]

images/commandbuffer_lifecycle.png

@@ -1360,7 +1360,6 @@ server's OpenCL/api-docs repository.
     <enums name="cl_command_buffer_state_khr" vendor="Khronos">
         <enum value="0"            name="CL_COMMAND_BUFFER_STATE_RECORDING_KHR"/>
         <enum value="1"            name="CL_COMMAND_BUFFER_STATE_EXECUTABLE_KHR"/>
-        <enum value="2"            name="CL_COMMAND_BUFFER_STATE_PENDING_KHR"/>
     </enums>
     <enums name="cl_mutable_dispatch_fields_khr" vendor="Khronos" type="bitmask">
         <enum bitpos="0"            name="CL_MUTABLE_DISPATCH_GLOBAL_OFFSET_KHR"/>
@@ -7284,7 +7283,7 @@ server's OpenCL/api-docs repository.
                 <enum name="CL_KERNEL_EXEC_INFO_DEVICE_PTRS_EXT"/>
             </require>
         </extension>
-        <extension name="cl_khr_command_buffer" revision="0.9.7" supported="opencl" depends="CL_VERSION_1_2" ratified="opencl" experimental="true">
+        <extension name="cl_khr_command_buffer" revision="0.9.8" supported="opencl" depends="CL_VERSION_1_2" ratified="opencl" experimental="true">
             <require>
                 <type name="CL/cl.h"/>
             </require>
@@ -7331,7 +7330,6 @@ server's OpenCL/api-docs repository.
             <require comment="cl_command_buffer_state_khr">
                <enum name="CL_COMMAND_BUFFER_STATE_RECORDING_KHR"/>
                <enum name="CL_COMMAND_BUFFER_STATE_EXECUTABLE_KHR"/>
-               <enum name="CL_COMMAND_BUFFER_STATE_PENDING_KHR"/>
             </require>
             <require comment="cl_command_type">
                 <enum name="CL_COMMAND_COMMAND_BUFFER_KHR"/>