Skip to content

Commit 4c0239e

Browse files
aksOpsaksOps
committed
chore(baseline): add consolidator and publish first BASELINE.md
1 parent 81a0e8f commit 4c0239e

2 files changed

Lines changed: 375 additions & 13 deletions

File tree

‎docs/superpowers/baselines/2026-04-17/BASELINE.md‎

Lines changed: 283 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,285 @@
11
# code-iq Baseline — 2026-04-17
22

3-
Status: in progress. Sections will be filled by Phase A tasks.
4-
5-
## Sections
6-
7-
- Toolchain
8-
- Maven build & tests
9-
- Coverage
10-
- Flaky tests
11-
- SpotBugs
12-
- OWASP dependency-check
13-
- Dependency tree
14-
- Frontend audit
15-
- Pipeline on seed repos
3+
This file is generated by `scripts/baseline/consolidate.sh`. Re-run after
4+
updating any capture script. Raw artifacts under `raw/` are gitignored.
5+
6+
## Toolchain
7+
8+
- Java: openjdk version "25.0.2" 2026-01-20 LTS
9+
- Maven: Apache Maven 3.8.7
10+
- Node: v24.15.0
11+
- npm: 11.12.1
12+
13+
## Maven build & tests
14+
15+
```json
16+
{
17+
"tests": 3059,
18+
"failures": 0,
19+
"errors": 0,
20+
"skipped": 31
21+
}
22+
```
23+
24+
## Coverage (JaCoCo)
25+
26+
```json
27+
{
28+
"inst_covered": 82247,
29+
"inst_missed": 10270,
30+
"br_covered": 5931,
31+
"br_missed": 2388,
32+
"line_covered": 16515,
33+
"line_missed": 1990,
34+
"inst_pct": 88.9,
35+
"br_pct": 71.29,
36+
"line_pct": 89.25
37+
}
38+
```
39+
40+
## Flaky tests
41+
42+
```json
43+
{
44+
"runs": 3,
45+
"failures_per_run": [
46+
0,
47+
0,
48+
0
49+
],
50+
"always_failing": [],
51+
"flaky": []
52+
}
53+
```
54+
55+
## SpotBugs
56+
57+
```json
58+
{
59+
"total_bugs": 1492,
60+
"by_priority": {
61+
"2": 1484,
62+
"1": 8
63+
},
64+
"by_category": {
65+
"STYLE": 546,
66+
"MALICIOUS_CODE": 203,
67+
"I18N": 1,
68+
"BAD_PRACTICE": 736,
69+
"MT_CORRECTNESS": 1,
70+
"PERFORMANCE": 4,
71+
"CORRECTNESS": 1
72+
},
73+
"top_patterns": [
74+
[
75+
"NM_METHOD_NAMING_CONVENTION",
76+
730
77+
],
78+
[
79+
"SF_SWITCH_NO_DEFAULT",
80+
448
81+
],
82+
[
83+
"EI_EXPOSE_REP2",
84+
77
85+
],
86+
[
87+
"MS_PKGPROTECT",
88+
60
89+
],
90+
[
91+
"BC_UNCONFIRMED_CAST",
92+
55
93+
],
94+
[
95+
"EI_EXPOSE_REP",
96+
46
97+
],
98+
[
99+
"NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE",
100+
26
101+
],
102+
[
103+
"MS_FINAL_PKGPROTECT",
104+
20
105+
],
106+
[
107+
"DLS_DEAD_LOCAL_STORE",
108+
5
109+
],
110+
[
111+
"SF_SWITCH_FALLTHROUGH",
112+
4
113+
],
114+
[
115+
"UC_USELESS_OBJECT",
116+
3
117+
],
118+
[
119+
"CT_CONSTRUCTOR_THROW",
120+
2
121+
],
122+
[
123+
"REC_CATCH_EXCEPTION",
124+
2
125+
],
126+
[
127+
"WMI_WRONG_MAP_ITERATOR",
128+
2
129+
],
130+
[
131+
"RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE",
132+
2
133+
],
134+
[
135+
"ES_COMPARING_STRINGS_WITH_EQ",
136+
2
137+
],
138+
[
139+
"DB_DUPLICATE_BRANCHES",
140+
1
141+
],
142+
[
143+
"DM_DEFAULT_ENCODING",
144+
1
145+
],
146+
[
147+
"UL_UNRELEASED_LOCK_EXCEPTION_PATH",
148+
1
149+
],
150+
[
151+
"UPM_UNCALLED_PRIVATE_METHOD",
152+
1
153+
]
154+
]
155+
}
156+
```
157+
158+
## OWASP dependency-check
159+
160+
```json
161+
{
162+
"status": "FAILED",
163+
"reason": "NVD DB update race: UpdateException (H2 lock) + NoDataException during first NVD sync. Re-run after clearing ~/.m2/repository/org/owasp/dependency-check-data/*.lock and optionally wiping the data dir.",
164+
"captured_log": "docs/superpowers/baselines/2026-04-17/raw/depcheck.log",
165+
"maven_exit_code": 1,
166+
"timestamp": "2026-04-17T08:06:05Z",
167+
"by_severity": {},
168+
"top_25": []
169+
}
170+
```
171+
172+
## Frontend
173+
174+
- Playwright:
175+
```json
176+
{
177+
"passed": 0,
178+
"failed": 575,
179+
"skipped": 0
180+
}
181+
```
182+
- Full logs: `raw/frontend/` (local only).
183+
184+
## Pipeline on seed repos
185+
186+
### spring-petclinic
187+
```json
188+
{
189+
"seed": "spring-petclinic",
190+
"timings": [
191+
"index duration=8s rc=0",
192+
"enrich duration=13s rc=0",
193+
"health=fail"
194+
],
195+
"stats": null,
196+
"health_ok": false
197+
}
198+
```
199+
200+
### realworld-express
201+
```json
202+
{
203+
"seed": "realworld-express",
204+
"timings": [
205+
"index duration=5s rc=0",
206+
"enrich duration=10s rc=0",
207+
"health=fail"
208+
],
209+
"stats": null,
210+
"health_ok": false
211+
}
212+
```
213+
214+
## Known gaps / issues
215+
216+
Ordered by severity. Each item cites the raw artifact it was derived from.
217+
218+
### Critical
219+
220+
- **OWASP dependency-check failed.** NVD initial sync hit `UpdateException: Unable to obtain exclusive lock on H2 database` followed by `NoDataException: No documents exist`. Maven exit 1 after 40 min. No CVE inventory captured. Must re-run (see §Re-run instructions below) before any security posture claim.
221+
- Raw: `raw/depcheck.log`, `raw/depcheck-summary.json` (stub, `status=FAILED`).
222+
223+
- **Playwright E2E: 0 passed / 575 failed.** 100% failure rate. Almost certainly environment/config rather than regressions — the audit script runs `npm run test:e2e` without starting the backend (`java -jar ... serve`), so any test that hits `/api/*` will fail. Needs a harness that spins up the server (or mocks it) before running Playwright, or a `webServer` entry in `playwright.config.ts`.
224+
- Raw: `raw/frontend/playwright.log`, `raw/frontend/playwright-summary.json`.
225+
226+
### High
227+
228+
- **Pipeline serve-smoke failed on both seed repos** (`health=fail`, `stats=null`). `index` and `enrich` succeeded (petclinic 8+13s, express 5+10s) but the 8-second sleep between starting `serve` and `curl /actuator/health` is at the low end of the documented 8–16s Spring Boot + embedded Neo4j cold-start window (see CLAUDE.md §Gotchas). Fix in Phase F hardening: poll `/actuator/health` with a retry budget instead of a fixed sleep.
229+
- Raw: `raw/pipeline/spring-petclinic/`, `raw/pipeline/realworld-express/`.
230+
231+
- **SpotBugs: 8 HIGH-priority findings (priority=1) + 1,484 at priority=2.** Total 1,492. HIGH findings must be triaged individually (read `raw/spotbugs.xml`). Noise-dominant rules (`NM_METHOD_NAMING_CONVENTION`=730, `SF_SWITCH_NO_DEFAULT`=448) should be filtered via a SpotBugs exclude file so real signal surfaces; real-concern patterns that deserve review now: `NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE` (26), `BC_UNCONFIRMED_CAST` (55), `UL_UNRELEASED_LOCK_EXCEPTION_PATH` (1), `WMI_WRONG_MAP_ITERATOR` (2), `ES_COMPARING_STRINGS_WITH_EQ` (2), `MT_CORRECTNESS` category (1).
232+
- Raw: `raw/spotbugs.xml`, `raw/spotbugs-summary.json`.
233+
234+
### Medium
235+
236+
- **Branch coverage 71.3% is notably below instruction coverage 89.0%.** Expected for a detector-heavy codebase, but targeted branch coverage on the enrichment / linker / LayerClassifier paths (which drive deterministic output) is worth a focused improvement pass in Phase E.
237+
- Raw: `raw/coverage-summary.json`, `raw/jacoco.csv`.
238+
239+
- **31 skipped tests.** Not investigated. Read surefire reports to confirm they're intentional (`@Disabled` / profile-gated) and not silently excluded.
240+
- Raw: `raw/surefire-reports.tar`.
241+
242+
### Low / noise
243+
244+
- `consolidate.sh` prints the Maven version with raw ANSI escape codes (`[1mApache Maven 3.8.7[m`). Strip with `sed 's/\x1b\[[0-9;]*m//g'` in a follow-up. Cosmetic only.
245+
246+
### Green
247+
248+
- **3,059 tests, 0 failures, 0 errors.** Clean.
249+
- **Flaky scan: 0 always-failing, 0 flaky across 3 runs.** Suite is stable.
250+
- **Instruction coverage 89.0%**, line coverage 89.25%. Strong baseline.
251+
- **`npm audit` + Vite build: no blocking issues** recorded in the capture.
252+
- **Pipeline `index` and `enrich` succeeded deterministically** on both seed repos.
253+
254+
## Re-run instructions (for blocked captures)
255+
256+
### OWASP dep-check
257+
```bash
258+
# 1. Stop any lingering dep-check processes
259+
pkill -f dependency-check 2>/dev/null
260+
# 2. Clear NVD locks (and optionally wipe the partial DB)
261+
rm -f ~/.m2/repository/org/owasp/dependency-check-data/11.0/*.lock
262+
# rm -rf ~/.m2/repository/org/owasp/dependency-check-data/11.0 # fallback if DB is corrupt
263+
# 3. Re-run
264+
./scripts/baseline/run-depcheck.sh
265+
```
266+
267+
### Pipeline serve-smoke
268+
Patch `scripts/baseline/run-pipeline.sh` to replace the `sleep 8` with a poll loop:
269+
```bash
270+
for _ in $(seq 1 60); do
271+
if curl -sf "http://127.0.0.1:$PORT/actuator/health" > "$OUT/health.json"; then break; fi
272+
sleep 2
273+
done
274+
```
275+
Then re-run `./scripts/baseline/run-pipeline.sh spring-petclinic` and `realworld-express`.
276+
277+
### Playwright E2E
278+
Add a `webServer` entry to `src/main/frontend/playwright.config.ts` that starts the code-iq server against a fixture repo, or supply a mock backend. Then re-run `./scripts/baseline/run-frontend-audit.sh`.
279+
280+
## Handoff to subsequent phases
281+
282+
- **Phase B (unified config)** — `codeiq.yml` smoke test against both seed repos; validation script gates CI.
283+
- **Phase D (MCP robustness)** — pipeline serve-smoke fix above is a prerequisite for any MCP contract test.
284+
- **Phase E (determinism)** — `index → enrich` reproducibility on the two seed repos above is the seed for graph-snapshot diffing; 31 skipped tests to triage.
285+
- **Phase F (ops & perf)** — Playwright harness + cold-start budgets.

‎scripts/baseline/consolidate.sh‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
#!/usr/bin/env bash
2+
# Build BASELINE.md from captured raw/ artifacts. Idempotent; safe to re-run.
3+
set -euo pipefail
4+
RAW="docs/superpowers/baselines/2026-04-17/raw"
5+
OUT="docs/superpowers/baselines/2026-04-17/BASELINE.md"
6+
7+
read_json() { [[ -f "$1" ]] && cat "$1" || echo "null"; }
8+
9+
TESTS=$(read_json "$RAW/test-counts.json")
10+
COV=$(read_json "$RAW/coverage-summary.json")
11+
FLAKY=$(read_json "$RAW/flaky.json")
12+
SB=$(read_json "$RAW/spotbugs-summary.json")
13+
DC=$(read_json "$RAW/depcheck-summary.json")
14+
FRONT_PW=$(read_json "$RAW/frontend/playwright-summary.json")
15+
PL_PC=$(read_json "$RAW/pipeline/spring-petclinic/summary.json")
16+
PL_RW=$(read_json "$RAW/pipeline/realworld-express/summary.json")
17+
18+
JAVA_V=$(java -version 2>&1 | head -n1)
19+
MVN_V=$(mvn -v 2>&1 | head -n1)
20+
NODE_V=$(node --version)
21+
NPM_V=$(npm --version)
22+
23+
cat > "$OUT" <<MD
24+
# code-iq Baseline — 2026-04-17
25+
26+
This file is generated by \`scripts/baseline/consolidate.sh\`. Re-run after
27+
updating any capture script. Raw artifacts under \`raw/\` are gitignored.
28+
29+
## Toolchain
30+
31+
- Java: $JAVA_V
32+
- Maven: $MVN_V
33+
- Node: $NODE_V
34+
- npm: $NPM_V
35+
36+
## Maven build & tests
37+
38+
\`\`\`json
39+
$TESTS
40+
\`\`\`
41+
42+
## Coverage (JaCoCo)
43+
44+
\`\`\`json
45+
$COV
46+
\`\`\`
47+
48+
## Flaky tests
49+
50+
\`\`\`json
51+
$FLAKY
52+
\`\`\`
53+
54+
## SpotBugs
55+
56+
\`\`\`json
57+
$SB
58+
\`\`\`
59+
60+
## OWASP dependency-check
61+
62+
\`\`\`json
63+
$DC
64+
\`\`\`
65+
66+
## Frontend
67+
68+
- Playwright:
69+
\`\`\`json
70+
$FRONT_PW
71+
\`\`\`
72+
- Full logs: \`raw/frontend/\` (local only).
73+
74+
## Pipeline on seed repos
75+
76+
### spring-petclinic
77+
\`\`\`json
78+
$PL_PC
79+
\`\`\`
80+
81+
### realworld-express
82+
\`\`\`json
83+
$PL_RW
84+
\`\`\`
85+
86+
## Known gaps / issues
87+
88+
Populate this section manually after reading the raw outputs. Examples to
89+
look for: failing tests, flaky set non-empty, SpotBugs HIGH priority bugs,
90+
CVEs ≥ CRITICAL, coverage hot spots.
91+
MD
92+
echo "Wrote $OUT"

0 commit comments

Comments
 (0)