Codify your org standards with scorecards

Scorecards are one of the AgentOps certification methods. They turn your organization's standards into rules that Port evaluates continuously on every asset in your agentic registry.

Why codify your standards with scorecards

Today, standards for agentic assets live in a wiki page or a slide deck: every agent needs an owner, skills need a clear description, production agents can't run on personal tokens. Reviewers check them by hand when an asset is first approved, if at all.

With scorecards, each standard becomes a rule that Port evaluates continuously on every asset in the agentic registry. Gaps are visible as soon as they appear, and a workflow can propose the fix.

Pain point	Without Port	With Port
No way to know which agents to trust	There is no objective way to tell which agents are safe to put in production flows, so teams either block useful agents or trust ones that aren't ready.	Every agent has a score against your standards, so you know which agents are trustworthy enough for production flows.
Inconsistent reviews and drift	Each reviewer checks different things, so whether an asset reaches production depends on who reviewed it, and assets that met the bar last month quietly fall below it and keep running.	One set of rules is evaluated continuously on every asset, so the bar is the same no matter who builds or reviews it, and drift is caught the moment it happens.
Gaps nobody acts on	Nobody knows how many assets meet your standards, so audits turn into fire drills, and known problems sit on a backlog for weeks while the assets keep running.	Dashboards show compliance across all assets at any time, and a failed rule triggers a workflow that proposes the fix, so reporting takes minutes and owners only approve the fix.

Scorecard examples

Here are some ideas for scorecards you can define for the different types of assets in your agentic registry:

• Production readiness - whether the asset has what it needs to be relied on in production, such as an owner, a scoped identity and on-call coverage. It tells you which assets are safe to depend on.
• Discoverability - whether the asset's description is clear enough for agents and developers to find and use it. An asset nobody can find is wasted effort, and a vague one gets used for the wrong job.
• Uniqueness - whether the asset duplicates another, such as two skills for the same task or two agents that react to the same triggers. Duplicates waste effort and produce conflicting results.
• Composability - whether related assets are bundled so they can be installed together, such as skills and MCP servers grouped into a plugin. It keeps setup simple and consistent across teams.
• Blast radius - how much the asset could affect if it misbehaves, based on its permissions, tools and the systems it can reach. This applies to a skill that steers agents toward risky tools as much as to the agent itself.
• Approved access - whether the asset uses only approved MCP servers and tools, through an approved connector or gateway. It keeps agents away from systems nobody vetted.
• Predictability - whether the asset produces consistent results across similar sessions. Unpredictable assets can't be trusted with production work.

AI-evaluated rules for non-deterministic standards

Some standards can't be checked with a simple rule. "Has an owning team" is a yes-or-no question. "Is this skill's description clear enough to find?" is a judgment.

For these standards, an AI workflow evaluates the asset and writes its verdict to a property. The scorecard rule then checks that property like any other. The judgment is made by AI, but the rule, the threshold and the history stay visible and auditable in Port.

The certify skills to meet industry and org standards guide uses this pattern:

1. When a skill is published or updated, a certification workflow sends its description to an AI node.
2. The AI rates the description as good, too vague, too verbose or missing, and explains why.
3. The workflow records the rating, the reasoning and a recommendation to approve, iterate or reject the skill.
4. The discoverability scorecard reads the rating, so the skill's score updates automatically.
5. The skill's owner is notified when the skill needs work.

The same pattern works for any asset. For example, an AI workflow can review an agent's recent sessions to rate its predictability, or compare its triggers and purpose with other agents to rate its uniqueness.

From gap to fix

A scorecard that only reports gaps creates work. A scorecard connected to a workflow can also propose the fix:

1. A scorecard rule fails, for example a skill's description is rated too vague.
2. A workflow triggers on the failed rule and asks AI to draft a fix, such as a rewritten description.
3. The asset's owner reviews the proposed fix and approves, edits or rejects it.
4. On approval, the workflow applies the fix, for example by opening a pull request in the skill's repository.
5. The asset is re-scored once the change is merged.

Choosing what needs a human

Not every fix needs the same level of review. Deciding which fixes can be applied automatically is itself a standard, so you can codify it with scorecards too. A risk scorecard, built on signals such as blast radius, lifecycle stage and what the fix changes, rates each proposed fix, and the workflow routes it by that rating:

• Low risk: apply automatically - the fix is deterministic and the asset has a small blast radius, such as filling in a missing owner from the repository's code owners.
• Medium risk: owner approval - the fix changes behavior, such as rewriting a skill's instructions. The asset's owner approves it. As your scorecards show that a type of fix is consistently safe, you can move it to automatic.
• High risk: platform or security approval - the fix touches permissions, credentials or an asset with a large blast radius, such as a production agent. A platform or security reviewer approves it, and it never moves to automatic.

Related guides and docs

• Manage scorecards - create and update scorecards in Port.
• Self-heal scorecards with AI - use AI to remediate failed scorecard rules.
• Certify skills to meet industry and org standards - the AI-evaluated certification pattern for skills.