Skip to content

Add CONTRIBUTING.md, PR validation CI, and eval framework#40

Closed
Copilot wants to merge 4 commits intomainfrom
copilot/add-contributing-guide-and-ci
Closed

Add CONTRIBUTING.md, PR validation CI, and eval framework#40
Copilot wants to merge 4 commits intomainfrom
copilot/add-contributing-guide-and-ci

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Apr 30, 2026

No contributor guide, no structural validation CI, and no eval framework for testing agent/skill behavior. Community PRs have no automated verification pipeline.

CONTRIBUTING.md

  • Skill contribution guide: directory structure, SKILL.md frontmatter schema, naming conventions
  • Agent changes are maintainer-curated (discussion-first model)
  • PR process and what CI validates
  • Microsoft Open Source Code of Conduct linked

PR Validation Workflow (.github/workflows/pr-validation.yml)

Active workflow running on every PR to main:

  • Structural validation (scripts/validate-structure.js): frontmatter required fields, name-directory consistency, kebab-case enforcement, SKILL.md presence, required sections (## When to Use, ## Procedure), agent cross-reference integrity, relative link resolution
  • Markdownlint with .markdownlint.json config

All checks pass on current repo state (section warnings are non-blocking for equivalent headings like ## Execution Playbook).

Eval Framework (evals/)

Decision: custom Node.js harness over openai/evals (wrong ecosystem, Python-only, model-coupled).

  • evals/README.md — investigation summary and decision rationale
  • evals/scenarios/deploy-function-app.json — core deployment path scenario
  • evals/run-eval.js — validates scenarios reference real agents/skills from repo definitions
{
  "id": "deploy-function-app",
  "expected_agent": "Git-Ape",
  "expected_skill_sequence": ["prereq-check", "azure-naming-research", "..."],
  "assertions": [
    { "type": "skill_order", "before": "azure-deployment-preflight", "after": "azure-integration-tester" }
  ]
}

- CONTRIBUTING.md with skill contribution guide, SKILL.md schema, and CoC link
- .github/workflows/pr-validation.yml for structural checks and markdownlint
- scripts/validate-structure.js for frontmatter, naming, cross-ref validation
- .markdownlint.json configuration
- evals/ directory with README, scenario, and runner

Agent-Logs-Url: https://github.com/Azure/git-ape/sessions/5fd523e4-5c2b-45f4-a8bd-fe51985855a3

Co-authored-by: sendtoshailesh <50418172+sendtoshailesh@users.noreply.github.com>
Comment thread scripts/validate-structure.js Fixed
Agent-Logs-Url: https://github.com/Azure/git-ape/sessions/5fd523e4-5c2b-45f4-a8bd-fe51985855a3

Co-authored-by: sendtoshailesh <50418172+sendtoshailesh@users.noreply.github.com>
Copilot AI changed the title [WIP] Add CONTRIBUTING.md and verification CI workflow Add CONTRIBUTING.md, PR validation CI, and eval framework Apr 30, 2026
Copilot AI requested a review from sendtoshailesh April 30, 2026 06:07
…tion or class'

Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
@sendtoshailesh
Copy link
Copy Markdown
Contributor

Closing in favor of a re-scoped PR. Issue #17 has been narrowed to CONTRIBUTING.md + structural PR validation CI, and the exploratory eval framework portion is now tracked in #61. The replacement PR (scoped to the reduced #17) is being opened next and carries forward the CONTRIBUTING.md, scripts/validate-structure.js, .github/workflows/pr-validation.yml, and .markdownlint.json from this branch. The evals/ artifacts here are not lost — #61 references them as prior art.

arnaudlh pushed a commit that referenced this pull request May 7, 2026
Implements scope of #17 (de-scoped):

- CONTRIBUTING.md: skill contribution guide, SKILL.md schema, PR process, Code of Conduct link

- .github/workflows/pr-validation.yml: structural validation + markdownlint on every PR

- scripts/validate-structure.js: frontmatter, naming, cross-reference, link, and section checks

- .markdownlint.json: shared lint config

The eval framework (originally section 3 of #17) is split out to #61 and is not part of this PR.

Supersedes #40.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CONTRIBUTING.md and contribution verification CI

2 participants