Commit 3169ccd
block 4: observability & ops (version, health, metrics, access-log, log-format) (#74)
* feat(api): GET /api/version with ldflags + BuildInfo fallback
Moves the shared version-resolution logic out of cmd into a new
internal/buildinfo package so internal/api can serve the same data
without a cmd import cycle. Adds a public GET /api/version endpoint
that returns {version, commit, build_date, go_version, dirty, deps}
as JSON. Makefile LDFLAGS retargeted at the new package path.
bearerAuthMiddleware public bypass list extended with /healthz,
/readyz, and /api/version so upcoming probes and version endpoint
remain public.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(api): /healthz + /readyz probes with 10s cache
/healthz is dependency-free liveness. /readyz checks SQLite PingContext
and an LLM provider Complete(maxTokens=1) reach, caching the verdict
for 10 seconds to absorb Prometheus + Kubernetes probe loops. Nil
provider (config provider: none) reports llm.status=skipped and keeps
readiness green. Legacy /health route remains as a 200-returning alias
for older clients; /healthz is the canonical probe.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(obs): Prometheus metrics + workq stats
Replaces the ad-hoc text-format collector in internal/api/metrics.go
with the official prometheus/client_golang v1.20.5. Adds a new
internal/obs package hosting the Default registry and per-subject
collectors (HTTP, pipeline stages, embed latency, LLM tokens, workq
depth + rejections, build info).
Workq gains a Pool.Stats() snapshot accessor with rejectedTotal
counter; cmd/serve.go initialises obs and binds the live pool stats
provider. Pipeline IndexPath/IndexURL/Finalize are wrapped with
TimeStage for granular stage timings via a nil-safe helper.
Embedder observes per-batch provider latency; LLM Complete records
an approximate token count (bytes/4) until langchaingo usage data
is threaded through the Provider interface (tracked as follow-up).
HTTP recording moved out of loggingMiddleware's bespoke collector
into obs.HTTP.Observe; uses the Go 1.22 r.Pattern route to bound
label cardinality.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(api): structured access log with bytes_out + panic resilience
loggingMiddleware now emits one JSON/text slog line per request with
{req_id, method, path, route, status, duration_ms, bytes_out, auth,
project, panic}. The emission is deferred so a panic escaping
recoveryMiddleware still produces an access-log entry. auth is a
coarse label (bearer|cookie|anon) because docsiq uses a single shared
API key; there is no real user identity. responseWriter now tracks
bytes via an overridden Write and also proxies Flush for SSE/
streaming handlers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(log): log.format=json|text with emoji-strip production handler
New log.format config key (default "text"; DOCSIQ_LOG_FORMAT env).
Precedence is --log-format > env > config > default. The json handler
is wrapped in obs.NewProductionHandler, which strips a leading emoji
from slog Record.Message so log aggregators do not have to special-
case multi-byte sequences. The text handler keeps emoji for human
readers. Adds config-level defaults + env binding and three load-
level tests covering default, YAML, and env-var precedence.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(mock): add BatchCeiling() to satisfy post-rebase llm.Provider interface
Block 3 (merged to main) added BatchCeiling() int to llm.Provider.
After rebasing feat/block4-observability onto main, the mock provider
needed the new method to compile.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(api): bypass TimeoutHandler for SSE / streaming routes
http.TimeoutHandler buffers the response body and does not implement
http.Flusher. Block 4's responseWriter wrapper grew a Flush() method,
so SSE handlers now pass the Flusher type-assertion and enter their
streaming loop — but every Flush is a no-op because the underlying
timeoutWriter absorbs writes until the request completes. The client
times out reading the body at 30s (matches cfg.Server.RequestTimeout).
Carve GET /api/upload/progress and GET /mcp out of the timeout wrapper
so Flush propagates to the real net/http writer. SSE teardown still
runs via r.Context() cancellation on client disconnect or shutdown.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(health,test): real SSE stream consumption + decoupled probe ctx + lazy sq pinger
Three fixes bundled, all flagged by post-rebase CI or Codex review:
1. waitUploadDone now consumes the /api/upload/progress SSE stream
incrementally via bufio.Scanner instead of ReadAll. Prior code
worked on main only because Flush() was missing on responseWriter,
which made the handler return 500 immediately; with Block 4's
Flusher now wired through, ReadAll blocks until the handler closes
the stream, tripping the client's 30s timeout before "done" arrives.
2. /readyz SQLite probe no longer installs a no-op success fallback
when stores.Get fails at router build time. A lazy pinger resolves
the default store at probe time, so a genuine open failure
(permissions, corruption, disk) surfaces as 503 — and a store that
becomes available later flips readiness green without a restart.
3. readyzCache.check decouples probe context from the incoming request
context via context.WithoutCancel. Previously, a probing client
(Kubernetes, Prometheus, curl) disconnecting mid-probe would return
context.Canceled and pollute the 10-second cache for every
subsequent caller.
Coverage: new TestReadyz_ProbeCtxDecoupledFromRequestCtx exercises (3).
TestIsStreamingRoute_Classification and
TestRequestTimeoutMiddleware_StreamingRouteBypassesTimeout already
cover the Block-4-aware SSE bypass added in 8f9595b.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent 3a61148 commit 3169ccd
37 files changed
Lines changed: 2327 additions & 747 deletions
File tree
- cmd
- internal
- api
- buildinfo
- config
- embedder
- llm
- mock
- obs
- pipeline
- workq
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
7 | | - | |
8 | | - | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| 6 | + | |
6 | 7 | | |
| 8 | + | |
| 9 | + | |
7 | 10 | | |
8 | 11 | | |
9 | 12 | | |
| |||
28 | 31 | | |
29 | 32 | | |
30 | 33 | | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
10 | 11 | | |
11 | 12 | | |
12 | 13 | | |
| |||
41 | 42 | | |
42 | 43 | | |
43 | 44 | | |
44 | | - | |
| 45 | + | |
45 | 46 | | |
46 | 47 | | |
47 | 48 | | |
| |||
53 | 54 | | |
54 | 55 | | |
55 | 56 | | |
56 | | - | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
57 | 66 | | |
58 | 67 | | |
59 | 68 | | |
60 | 69 | | |
61 | | - | |
62 | | - | |
63 | | - | |
64 | | - | |
65 | | - | |
66 | | - | |
67 | | - | |
68 | | - | |
69 | | - | |
| 70 | + | |
| 71 | + | |
70 | 72 | | |
71 | 73 | | |
72 | 74 | | |
73 | 75 | | |
74 | 76 | | |
75 | 77 | | |
76 | 78 | | |
77 | | - | |
| 79 | + | |
78 | 80 | | |
79 | 81 | | |
80 | 82 | | |
81 | | - | |
82 | | - | |
83 | 83 | | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
84 | 91 | | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
| 18 | + | |
18 | 19 | | |
19 | 20 | | |
20 | 21 | | |
| 22 | + | |
21 | 23 | | |
22 | 24 | | |
23 | 25 | | |
| |||
154 | 156 | | |
155 | 157 | | |
156 | 158 | | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
157 | 172 | | |
158 | 173 | | |
159 | 174 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
5 | | - | |
6 | 5 | | |
| 6 | + | |
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
10 | | - | |
11 | | - | |
12 | | - | |
13 | | - | |
14 | | - | |
15 | | - | |
16 | | - | |
17 | | - | |
18 | | - | |
19 | | - | |
20 | | - | |
21 | | - | |
22 | | - | |
23 | | - | |
24 | | - | |
25 | | - | |
26 | | - | |
27 | | - | |
28 | | - | |
29 | | - | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | | - | |
36 | | - | |
37 | | - | |
38 | | - | |
39 | | - | |
40 | | - | |
41 | | - | |
42 | | - | |
43 | | - | |
44 | | - | |
45 | | - | |
46 | | - | |
47 | | - | |
48 | | - | |
49 | | - | |
50 | | - | |
51 | | - | |
52 | | - | |
53 | | - | |
54 | | - | |
55 | | - | |
56 | | - | |
57 | | - | |
58 | | - | |
59 | | - | |
60 | | - | |
61 | | - | |
62 | | - | |
63 | | - | |
64 | | - | |
65 | | - | |
66 | | - | |
67 | | - | |
68 | | - | |
69 | | - | |
70 | | - | |
71 | | - | |
72 | | - | |
73 | | - | |
74 | | - | |
75 | | - | |
76 | | - | |
77 | | - | |
78 | | - | |
79 | | - | |
80 | | - | |
81 | | - | |
82 | | - | |
83 | | - | |
84 | | - | |
85 | | - | |
86 | | - | |
87 | | - | |
88 | | - | |
89 | | - | |
90 | | - | |
91 | | - | |
92 | | - | |
93 | | - | |
94 | | - | |
95 | | - | |
96 | | - | |
97 | | - | |
98 | | - | |
99 | | - | |
100 | | - | |
101 | | - | |
102 | | - | |
103 | | - | |
104 | | - | |
105 | | - | |
106 | | - | |
107 | | - | |
108 | | - | |
109 | | - | |
110 | | - | |
111 | | - | |
112 | 10 | | |
113 | 11 | | |
114 | 12 | | |
115 | 13 | | |
116 | | - | |
| 14 | + | |
117 | 15 | | |
118 | | - | |
| 16 | + | |
119 | 17 | | |
120 | 18 | | |
121 | 19 | | |
122 | | - | |
| 20 | + | |
123 | 21 | | |
124 | 22 | | |
125 | 23 | | |
| |||
0 commit comments