feat: Tabular SK analysis — pagination, auto-trim, return_columns forwarding, and 100 K handoff limit (#893)

[vivche]

Description

Summary

This PR delivers five enhancements to the Tabular Semantic Kernel analysis pipeline that together prevent silent data truncation on large Excel / CSV files and give the LLM finer control over what data it retrieves per tool call.

---

Change 1 — Pagination (start_row / max_rows / has_more / next_start_row)

Every analysis tool in TabularProcessingPlugin now accepts:

Parameter	Type	Default	Description
start_row	str	"0"	Zero-based first row to return
max_rows	str	"100"	Maximum rows to return in one call

Every tool response now includes:

Field	Description
has_more	true when rows were truncated by the page limit
next_start_row	Value to pass as start_row on the next call
total_matched	Total rows that matched the filter (when computable)

The LLM is instructed (via the @kernel_function docstrings) to call the same tool again with start_row = next_start_row when has_more is true.

---

Change 2 — Auto-trim (_auto_trim_df_for_output)

A new private helper estimates the serialised JSON size from a 20-row sample. When the estimated output would exceed the 50 K-char per-tool budget and return_columns was not supplied by the caller:

1. Phase 1 — Column drop: columns are dropped heaviest-first (by average serialised length) until the estimate fits within budget, preserving at least one column.
2. Phase 2 — Row truncation: if the estimate is still over budget after all but one column is retained, rows are truncated so that has_more = True fires and pagination picks up the remainder.

A trimmed_columns_excluded field in the tool response lists any dropped columns so the LLM knows what was omitted.

---

Change 3 — return_columns forwarding

The existing return_columns parameter was already accepted by some tools but not threaded through to the DataFrame slice. This PR ensures it is applied consistently across all eleven analysis tools and that _auto_trim_df_for_output is skipped whenever return_columns is supplied (the caller has already projected to the columns they need).

---

Change 4 — Handoff limit raised to 100 K

max_handoff_chars in route_backend_chats.py was raised from 24_000 to 100_000. A WARNING-level log_event is emitted when truncation still occurs, including the original and truncated lengths, so operators can detect and investigate cases where even 100 K is insufficient.

---

Change 5 — Elapsed timing logging

Each SK tabular analysis invocation now logs the wall-clock elapsed time at INFO level, making it straightforward to correlate slow responses with large files or high row counts.

---

Files Changed

File	Change
application/single_app/semantic_kernel_plugins/tabular_processing_plugin.py	Pagination params & response fields on all 11 analysis tools; _auto_trim_df_for_output; return_columns forwarding
application/single_app/route_backend_chats.py	max_handoff_chars 24 K → 100 K; truncation WARNING log; elapsed timing
docs/explanation/feature/v0.241.008/TABULAR_SK_PAGINATION_AUTOTRIM_AND_HANDOFF_TRUNCATION.md	Fix documentation

---

Testing

• Tested with a Excel file that has over 1,000 rows using gpt-5.1 models.
• Pagination confirmed: second filter_rows call with start_row = next_start_row returned the correct continuation page.
• Auto-trim confirmed: wide freetext columns were dropped on oversized results; trimmed_columns_excluded listed the omitted columns in the response.
• return_columns confirmed: tool response contained only the requested columns with no auto-trim applied.
• Handoff limit confirmed: analysis results previously truncated at 24 K now pass through at up to 100 K; truncation WARNING fires only for pathological cases.
• Elapsed timing log lines confirmed in dev server output.