math_brute_force: Optimize test execution time and improve coverage methodology

[rjodinchr]

TL;DR: Most math_brute_force tests compute 4 billion (1<<32) values, which results in excessive execution times, particularly on mobile GPUs. This issue proposes a strategy to reduce these workloads while maintaining a high level of confidence in the correctness of the implementations.

Current Implementation

1. Tests evaluating all 64K (1<<16) values:
(This behavior is correct and working as intended)

• unary_half
• unary_two_results_half
• unary_two_results_i_half
• i_unary_half
• macro_unary_half

2. Tests evaluating 4 billion (1<<32) values redundantly:
(These test the exact same inputs multiple times)

• unary_u_half

3. Tests evaluating 4 billion (1<<32) values using combinations:
(These test all possible combinations of special values, followed by combinations of randomly selected values)

• binary_double
• binary_float
• binary_half
• binary_i_double
• binary_i_float
• binary_i_half
• binary_operator_double
• binary_operator_float
• binary_operator_half
• macro_binary_double
• macro_binary_float
• macro_binary_half
• ternary_double
• ternary_float
• ternary_half

4. Tests evaluating 4 billion (1<<32) values using purely random combinations:
(These completely ignore special values)

• binary_two_results_i_double
• binary_two_results_i_float
• binary_two_results_i_half
• mad_double
• mad_float
• mad_half
• unary_u_double

5. Tests evaluating 4 billion (1<<32) values uniformly:
(These spread values uniformly across the range, but completely ignore special values)

• i_unary_double
• macro_unary_double
• unary_double
• unary_two_results_double
• unary_two_results_i_double

6. Tests evaluating all 4 billion (1<<32) values exhaustively:

• i_unary_float
• macro_unary_float
• unary_float
• unary_two_results_float
• unary_two_results_i_float
• unary_u_float

Identified Issues

• Redundancy: Group 2 tests the exact same values multiple times.
• Missing Edge Cases: Groups 4 and 5 completely ignore special values (e.g., NaN, Infinity, zero).
• Poor Coverage of Mixed Cases: Group 3 combines special values with other special values, but never tests special values against randomly selected values.
• Performance Bottleneck: Groups 3, 4, 5, and 6 evaluate 1<<32 values, which takes an excessively long time to execute.
• Code Duplication: Although these tests perform similar operations, they do not share common code. This leads to heavy duplication of test logic and redundantly copy-pasted special value arrays.

Proposal

• Standardize Unary Half Tests: Modify all unary half-precision tests to evaluate 64K values (specifically, fix unary_u_half).
• Consolidate Special Values:
  • Merge all FP32 special values into a single C++ array shared across all tests.
  • Merge all FP64 special values into a single C++ array shared across all tests.
• Revamp Unary Testing: For unary tests, use all special values and fill the rest of the buffer with randomly selected values spread uniformly across the range, up to a total of n values.
• Revamp Binary/Ternary Testing: Test n total combinations. To ensure comprehensive mixing of values:
  • We need m = n^(1/2) unique values for binary operations, and m = n^(1/3) unique values for ternary operations.
  • These m values will consist of s special values, plus r random values spread uniformly across the range (where r = m - s).
• Determine n: Figure out the optimal baseline size for n. Note that this value may vary depending on the number of inputs (unary/binary/ternary) and/or the data type (FP16/FP32/FP64).
• Retain Exhaustive Testing Option: Keep a command-line flag or option to run the full 1<<32 value suite.
  • Note: Running 1<<32 would now use this new dataset generation methodology, except for unary float tests where 1<<32 covers the entire exhaustive range (making special/random selection unnecessary).
• Update Execution Modes: "Wimpy" and "Embedded" modes will simply scale down the value of n.

Next Steps

1. Discuss the proposal.
2. Agree on an action plan.
3. Implement the agreed-upon plan.

---

Note: This is a long-standing issue as described in KhronosGroup/OpenCL-CTS#1054. This issue addresses only math_brute_force.

[bashbaug]

Thanks for writing this up and getting the discussion started!

There's one other item I think we should consider: in at least some cases (the data below was collected for the fp32 sin function), we're actually testing all 1<<32 fp32 values 6 times, one for each vector size (scalar, 2, 3, 4, 8, 16 components). Here is data from the OpenCL intercept layer, collected on a recent but relatively modest iGPU:

                                      Function Name,  Calls,     Time (ns), Time (%),  Average (ns),      Min (ns),      Max (ns)
 math_kernel SIMD32 REG128 GWS[ 32768 ] LWS[ NULL ], 131072,    3514954077,    6.98%,         26816,          6458,      10101458
math_kernel16 SIMD16 REG128 GWS[ 2048 ] LWS[ NULL ], 131072,   13604290165,   27.01%,        103792,         25052,      13030677
math_kernel2 SIMD32 REG128 GWS[ 16384 ] LWS[ NULL ], 131072,    3597632945,    7.14%,         27447,          5468,       6686093
math_kernel3 SIMD32 REG128 GWS[ 10923 ] LWS[ NULL ], 131072,    6374819137,   12.66%,         48636,          8072,      12206979
 math_kernel4 SIMD32 REG128 GWS[ 8192 ] LWS[ NULL ], 131072,    5434237172,   10.79%,         41459,          5989,      11565781
 math_kernel8 SIMD16 REG128 GWS[ 4096 ] LWS[ NULL ], 131072,    6976498481,   13.85%,         53226,          8593,       5564947

The vector size is described in the kernel name, "GWS" is the "global work size", and the number of times the kernel is enqueued is described by "calls". Note that in all cases, vector size x gws x calls is 1<<32.

Do we need to duplicate testing across all six vector sizes? If so, can we reuse the "reference" results computed on the host CPU for each of the vector sizes? If not, can we reduce the n factor even further, since we'll get additional coverage through each or the vector sizes?

[b-sumner]

I think we do need to test all vector sizes. We clearly can't test all 2^64 float2 unary function inputs, but I don't see anything wrong with testing more float2 inputs than float inputs, And similarly for the larger vector sizes. Reusing the reference results makes sense to me.

[rjodinchr]

It's a great idea to distribute the dataset across the vector size to prevent redundant computations.

Even if we halve the number of tests, the testing process should speed up by 12 times (2*6).

While I'd prefer to shrink the overall dataset even further, this approach would already yield significant improvements, in my opinion.