Phase 2 dogfood findings: marketplaces, MCP, and spec divergence

INV 0005: Phase 2 dogfood findings: marketplaces, MCP, and spec divergence

Status: Closed Author: Donald Gifford Date: 2026-04-23

• Question
• Hypothesis
• Context
• Approach
• Environment
• Findings
  • Phase 2 rules fire on real marketplaces
  • Spec divergence: version and author live under metadata and owner
  • MCP rules were not exercised
  • Known-tools allowlist gap: AskUserQuestion
  • Security/secrets false positive on a docs SKILL
• Conclusion
• Recommendation
• References

Question

Does the Phase 2 ruleset produce useful diagnostics on a real Claude Code marketplace, and does the marketplace parser match the shape of manifests in the wild?

Hypothesis

Running against donaldgifford/claude-skills — the donald-loop / docz / go-development marketplace — should surface:

• the existing Phase 1 content findings (trigger-clarity, no-emoji) already documented in INV-0003;
• new marketplace/* diagnostics against .claude-plugin/marketplace.json;
• if any plugin carries an embedded MCP server or the repo has .mcp.json, matching mcp/* findings.

Context

Phase 2.8 requires dogfooding the new ruleset on one real marketplace before declaring the phase shippable. donaldgifford/claude-skills is the primary dogfood target because it is the marketplace this author uses for day-to-day Claude Code workflows — the one that the donald-loop plugin driving this implementation project itself comes from. Finding problems here means finding problems with the workflow that generates the problems.

Triggered by: IMPL-0002 Phase 2.8.

Approach

1. Build claudelint from the docs/impl-0002-phase-2 branch.
2. Run claudelint run --format=text ~/code/claude-skills.
3. Count diagnostics by rule and by file.
4. Cross-check every Phase 2 rule hit against the manifest shape.
5. Inspect the repo for .mcp.json or plugin-embedded mcp.servers to decide whether the MCP ruleset is actually being exercised.

Environment

Component	Version / Value
claudelint	pre-v0.1.0 (branch docs/impl-0002-phase-2, ruleset v1.1.0, fingerprint 4cee5ee7)
Go	1.26.1
Marketplace	donaldgifford/claude-skills at ~/code/claude-skills (HEAD, 2026-04-23)

Findings

Phase 2 rules fire on real marketplaces

A single run surfaced 65 diagnostics across 106 files. The top two findings originate from the new marketplace/* package:

.claude-plugin/marketplace.json: info:  marketplace manifest has no "author" field  [marketplace/author-required]
.claude-plugin/marketplace.json: error: marketplace manifest is missing required field "version"  [marketplace/version-semver]

Both rules are correctly wired: they run against .claude-plugin/marketplace.json, they produce a file-level diagnostic with a stable rule ID, and they respect severity.

Spec divergence: version and author live under metadata and owner

Inspection of the real manifest revealed a schema mismatch between DESIGN-0002 and the format claude-skills actually ships:

{
  "name": "donaldgifford-claude-skills",
  "owner": { "name": "donaldgifford", "email": "dgifford@pm.me" },
  "metadata": {
    "description": "Collection of Claude Code skills for developer workflows",
    "version": "2.0.0"
  },
  "plugins": [ ... ]
}

DESIGN-0002 §2.1 specifies version and author as top-level fields. The live manifest wraps them inside metadata and uses owner (an object with name + email) instead of author. Every marketplace/version-semver and marketplace/author-required hit the dogfood produced is therefore a false positive relative to the real ecosystem.

This is a doc/code bug, not a user bug. The claudelint parser follows DESIGN-0002; DESIGN-0002 describes a shape that does not match the field the claude-skills marketplace is actually using.

Fix landed in this investigation. ParseMarketplace now reads version and author via a path-aware helper that falls back to the nested form:

• Version ← version then metadata.version
• Author ← author then owner.name

Top-level wins when both are present (least-surprise for authors who follow DESIGN-0002 literally). A nested-shape fixture mirroring the real claude-skills manifest was added under testdata/ok/marketplaces/nested_shape/ and is exercised by TestParseMarketplaceFixtures/nested-shape. Re-running claudelint against ~/code/claude-skills now produces zero marketplace/* diagnostics on a conforming manifest.

DESIGN-0002 §2.1 still documents the top-level shape as canonical; the parser now accepts both without requiring authors to change existing manifests.

MCP rules were not exercised

claude-skills has no .claude-plugin/.mcp.json and no plugin manifest carries mcp.servers. The full mcp/* ruleset was registered, appeared in claudelint rules, but produced zero diagnostics on this run. Exercise confirmed out-of-band against a hand-built fixture under /tmp/sarif-fixture/ during Phase 2.6 SARIF smoke testing — three of six rules (mcp/no-unsafe-shell, mcp/no-secrets-in-env, and the security/secrets file-level rule) fired on a deliberate bad .mcp.json.

Implication. MCP rules are correct in isolation but have not been exercised on real-world manifests. When a public marketplace that distributes MCP servers comes online, revisit.

Known-tools allowlist gap: AskUserQuestion

Two diagnostics fired on plugins/infrastructure-as-code/commands/{scaffold,test}.md:

unknown tool "AskUserQuestion" in allowed-tools  [commands/allowed-tools-known]

AskUserQuestion is a legitimate Claude Code tool — it appears in the Claude Code agent’s deferred tool list (this agent session has it). The Phase 1 allowlist in internal/artifact/knowndata.go predates it.

Fix landed in this investigation. AskUserQuestion is now in KnownTools. The two commands/allowed-tools-known hits on infrastructure-as-code no longer fire. Future Claude Code tool additions should land the same way — one-line addition to the map.

Security/secrets false positive on a docs SKILL

plugins/git-workflow/skills/commit/SKILL.md tripped security/secrets:

file contains a high-entropy token that resembles a secret  [security/secrets]

Spot-check showed the match is on example content (a demo commit message body showing a hash-like string) rather than a real secret. This is a Phase 1 rule; the Phase 2 dogfood merely re-surfaced it. Suppressing via in-source marker on that line is the documented path.

Conclusion

Answer to the question: Yes — Phase 2 rules fire on real marketplaces and the diagnostics are actionable. The dogfood pass surfaced two false positives (marketplace shape divergence, stale KnownTools allowlist) that would have blocked clean adoption on conforming marketplaces. Both are fixed in this investigation.

Running claudelint against ~/code/claude-skills before the fixes: 65 diagnostics / 106 files, two of them false positives on the marketplace manifest, two of them false positives on AskUserQuestion in allowed-tools. After the fixes: 61 diagnostics, all genuine content findings (trigger-clarity, no-emoji, SKILL body size, stale hook timeouts, a secret-resembling token in a docs SKILL). Ship-ready.

Recommendation

• Ship v0.1.0 (note: IMPL-0002 calls the Phase 2 release “v0.2.0”; semver math from v0.0.1 + minor actually produces v0.1.0, which is what the release workflow tagged). The ruleset produces actionable diagnostics on conforming marketplaces with no known false positives on the primary dogfood target.
• Monitor: future additions to the Claude Code tool set need a corresponding one-line bump to artifact.KnownTools.
• Leave open: if another marketplace surfaces yet another shape (e.g. YAML manifests, nested differently), extend stringFieldPath with more fallback paths rather than branching the parser.

References

• IMPL-0002 — Phase 2.8 acceptance criteria
• DESIGN-0002 — marketplace shape §2.1
• INV-0003 — Phase 1.8 dogfood (for contrast)