Skip to main content

Check out Port for yourself ➜ 

Set up pipeline reliability scorecards

Scorecards let you define maturity levels for your services and teams, giving engineering leaders a clear view of which areas meet reliability standards and which need attention. This guide walks you through setting up Pipeline Reliability scorecards using data from GitHub, GitLab, or Azure DevOps integrations, at different levels of the organization:

  • Service: Workflow failure rates and reliability trends for individual services.
  • Team: Same metrics aggregated across all services owned by each team.
  • Group: Executive-level reliability health for business units.
  • Organization: Top-level reliability view across the entire org.

Each scorecard evaluates entities against progressive maturity levels: Bronze, Silver, and Gold.

Common use cases​

  • Set clear, measurable reliability expectations across engineering teams.
  • Identify services with persistently high CI/CD failure rates before they impact delivery.
  • Spot reliability trends early to catch pipelines that are degrading before they become critical.
  • Give developers a self-service view of their pipeline health against organization-wide standards.

Prerequisites​

This guide assumes the following:

  • You have a Port account and have completed the onboarding process.
  • You have installed a GitHub, GitLab, or Azure DevOps integration.
  • You have completed the Measure delivery reliability and stability guide, which configures the required workflow run properties on the service, _team, and organization blueprints.
Branch and time scope

The example mappings used for reliability metrics often scope data to main, master, or production branches and the last 90 days so the scorecards focus on production-relevant CI/CD health. Exact defaults can vary by provider and by your current mapping.

To change the tracked branches or time window, go to Data Sources, open your GitHub, GitLab, or Azure DevOps integration, select the Mapping tab, and locate the reliability entity resource (githubWorkflowRun, gitlabJob, or azureDevopsBuild). Edit the selector.query field.

For example, this GitHub selector also includes develop:

selector:
  query: >-
    (.head_branch | IN("main", "master", "production", "develop")) and
    ((.created_at | fromdateiso8601) > (now - 7776000))

Add or remove branch names in the IN(...) list as needed. The 7776000 value is 90 days in seconds — change it to 2592000 for 30 days.

Maturity levels​

Port scorecards use four fixed maturity levels. An entity must pass all rules at a given level (and all levels below it) to achieve that level.

LevelProgressionDescription
Basic0 (default)Entity exists but meets no scorecard criteria
Bronze1Foundational hygiene — CI/CD is running and failure rate is below 30%
Silver2Good practices — failure rate under 15% and not trending worse
Gold3Excellence — failure rate under 5%

Create the scorecards​

Select the level you want to configure:

Blueprint: Service

#RuleLevelPropertyCondition
1Has workflow runs this monthBronzeworkflow_runs_30d> 0
2Workflow failure rate < 30%Bronzemonthly_workflow_failure_rate< 30
3Workflow failure rate < 15%Silvermonthly_workflow_failure_rate< 15
4Failure rate not degradingSilverfailure_rate_trend!= "Degrading"
5Workflow failure rate < 5%Goldmonthly_workflow_failure_rate< 5

Create the reliability health scorecard on the service blueprint

  1. Go to the Builder page of your portal.

  2. Search for the Service blueprint and select it.

  3. Click on the Scorecards tab.

  4. If the reliability_health scorecard does not exist, click + New Scorecard. If it already exists, open it instead.

  5. Click on the {...} Edit JSON button in the top right corner.

  6. Paste the following JSON configuration:

    Service Reliability Health scorecard (click to expand)
    {
      "identifier": "reliability_health",
      "title": "Reliability Health",
      "levels": [
        {
          "color": "paleBlue",
          "title": "Basic"
        },
        {
          "color": "bronze",
          "title": "Bronze"
        },
        {
          "color": "silver",
          "title": "Silver"
        },
        {
          "color": "gold",
          "title": "Gold"
        }
      ],
      "rules": [
        {
          "identifier": "has_workflow_runs",
          "title": "Has workflow runs this month",
          "description": "Service has at least one CI/CD workflow run in the last 30 days, confirming pipelines are active.",
          "level": "Bronze",
          "query": {
            "combinator": "and",
            "conditions": [
              {
                "operator": ">",
                "property": "workflow_runs_30d",
                "value": 0
              }
            ]
          }
        },
        {
          "identifier": "failure_rate_under_30",
          "title": "Workflow failure rate < 30%",
          "description": "Monthly CI/CD workflow failure rate is below 30%.",
          "level": "Bronze",
          "query": {
            "combinator": "and",
            "conditions": [
              {
                "operator": "<",
                "property": "monthly_workflow_failure_rate",
                "value": 30
              }
            ]
          }
        },
        {
          "identifier": "failure_rate_under_15",
          "title": "Workflow failure rate < 15%",
          "description": "Monthly CI/CD workflow failure rate is below 15%.",
          "level": "Silver",
          "query": {
            "combinator": "and",
            "conditions": [
              {
                "operator": "<",
                "property": "monthly_workflow_failure_rate",
                "value": 15
              }
            ]
          }
        },
        {
          "identifier": "failure_rate_not_degrading",
          "title": "Failure rate not degrading",
          "description": "Weekly failure rate is not worsening compared to the monthly average.",
          "level": "Silver",
          "query": {
            "combinator": "and",
            "conditions": [
              {
                "operator": "!=",
                "property": "failure_rate_trend",
                "value": "Degrading"
              }
            ]
          }
        },
        {
          "identifier": "failure_rate_under_5",
          "title": "Workflow failure rate < 5%",
          "description": "Monthly CI/CD workflow failure rate is below 5%.",
          "level": "Gold",
          "query": {
            "combinator": "and",
            "conditions": [
              {
                "operator": "<",
                "property": "monthly_workflow_failure_rate",
                "value": 5
              }
            ]
          }
        }
      ]
    }

  7. Click Save to create the scorecard.

Property availability

workflow_runs_30d, failed_workflow_runs_30d, monthly_workflow_failure_rate, and failure_rate_trend are aggregation and calculation properties configured by the reliability metrics setup for your provider. Newly ingested entities may show Basic until the integration completes its first sync. Once sync finishes, scorecard levels are recomputed automatically.

Visualize scorecard results​

Once the scorecards are in place, create a dashboard to monitor pipeline reliability maturity across services and teams.

Create the dashboard​

  1. Navigate to your software catalog.
  2. Click on the + button in the left sidebar.
  3. Select New folder.
  4. Name the folder Engineering Intelligence and click Create.
  5. Inside the Engineering Intelligence folder, click + again.
  6. Select New dashboard.
  7. Name the dashboard Pipeline Reliability Scorecards and click Create.

Add widgets​

You can populate the dashboard using either an API script or by manually creating each widget through the UI.

The fastest way to set up the dashboard is by using Port's API to create all widgets at once.

Get your Port API token

  1. In your Port portal, click on your profile picture in the top right corner.

  2. Select Credentials.

  3. Click Generate API token.

  4. Copy the generated token and store it as an environment variable:

    export PORT_ACCESS_TOKEN="YOUR_GENERATED_TOKEN"
    EU region

    If your portal is hosted in the EU region, replace api.port.io with api.port-eu.io in the dashboard creation command below.

Create the dashboard with widgets

Save the following JSON to a file named reliability_scorecards_dashboard.json:

Dashboard JSON payload (click to expand)
{
  "identifier": "reliability_scorecards",
  "title": "Pipeline Reliability Scorecards",
  "icon": "Alert",
  "type": "dashboard",
  "parent": "engineering_intelligence",
  "widgets": [
    {
      "id": "reliabilityScorecardsDashboardWidget",
      "type": "dashboard-widget",
      "layout": [
        {
          "height": 400,
          "columns": [
            {"id": "serviceLevelDistribution", "size": 4},
            {"id": "teamLevelDistribution", "size": 4},
            {"id": "servicesBelowBronze", "size": 4}
          ]
        },
        {
          "height": 400,
          "columns": [
            {"id": "serviceReliabilityTable", "size": 12}
          ]
        },
        {
          "height": 400,
          "columns": [
            {"id": "teamReliabilityTable", "size": 12}
          ]
        }
      ],
      "widgets": [
        {
          "id": "serviceLevelDistribution",
          "type": "entities-pie-chart",
          "title": "Service Reliability Levels",
          "icon": "Pie",
          "description": "Distribution of reliability scorecard levels across services",
          "blueprint": "service",
          "property": "scorecard#reliability_health",
          "dataset": {
            "combinator": "and",
            "rules": []
          }
        },
        {
          "id": "teamLevelDistribution",
          "type": "entities-pie-chart",
          "title": "Team Reliability Levels",
          "icon": "Pie",
          "description": "Distribution of reliability scorecard levels across teams",
          "blueprint": "_team",
          "property": "scorecard#team_reliability_health",
          "dataset": {
            "combinator": "and",
            "rules": [
              {
                "operator": "=",
                "property": "type",
                "value": "team"
              }
            ]
          }
        },
        {
          "id": "servicesBelowBronze",
          "type": "entities-number-chart",
          "title": "Services Below Bronze",
          "icon": "Alert",
          "description": "Number of services that have not reached Bronze reliability level",
          "blueprint": "service",
          "chartType": "countEntities",
          "calculationBy": "entities",
          "func": "count",
          "unit": "none",
          "dataset": {
            "combinator": "and",
            "rules": [
              {
                "operator": "=",
                "property": "scorecard#reliability_health",
                "value": "Basic"
              }
            ]
          }
        },
        {
          "id": "serviceReliabilityTable",
          "type": "table-entities-explorer",
          "displayMode": "widget",
          "title": "Service Reliability Scorecard Overview",
          "icon": "Table",
          "description": "Services with their reliability scorecard level and failure rate metrics",
          "blueprint": "service",
          "dataset": {"combinator": "and", "rules": []},
          "excludedFields": [],
          "blueprintConfig": {
            "service": {
              "groupSettings": {"groupBy": ["team"]},
              "propertiesSettings": {
                "order": ["$title", "team", "scorecard-property#reliability_health", "monthly_workflow_failure_rate", "weekly_workflow_failure_rate", "failure_rate_trend", "workflow_runs_30d", "failed_workflow_runs_30d"],
                "shown": ["$title", "team", "scorecard-property#reliability_health", "monthly_workflow_failure_rate", "weekly_workflow_failure_rate", "failure_rate_trend", "workflow_runs_30d", "failed_workflow_runs_30d"]
              },
              "filterSettings": {"filterBy": {"combinator": "and", "rules": []}},
              "sortSettings": {"sortBy": []}
            }
          }
        },
        {
          "id": "teamReliabilityTable",
          "type": "table-entities-explorer",
          "displayMode": "widget",
          "title": "Team Reliability Scorecard Overview",
          "icon": "Table",
          "description": "Teams with their reliability scorecard level and failure rate metrics",
          "blueprint": "_team",
          "dataset": {
            "combinator": "and",
            "rules": [
              {
                "operator": "=",
                "property": "type",
                "value": "team"
              }
            ]
          },
          "excludedFields": [],
          "blueprintConfig": {
            "_team": {
              "groupSettings": {},
              "propertiesSettings": {
                "order": ["$title", "scorecard-property#team_reliability_health", "monthly_workflow_failure_rate", "weekly_workflow_failure_rate", "failure_rate_trend", "workflow_runs_30d", "failed_workflow_runs_30d"],
                "shown": ["$title", "scorecard-property#team_reliability_health", "monthly_workflow_failure_rate", "weekly_workflow_failure_rate", "failure_rate_trend", "workflow_runs_30d", "failed_workflow_runs_30d"]
              },
              "filterSettings": {"filterBy": {"combinator": "and", "rules": []}},
              "sortSettings": {"sortBy": []}
            }
          }
        }
      ]
    }
  ]
}

Then run the following command to create the dashboard with all widgets:

curl -s -X POST "https://api.port.io/v1/pages" \
  -H "Authorization: Bearer $PORT_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d @reliability_scorecards_dashboard.json | python3 -m json.tool
Engineering Intelligence folder

The script assumes a folder with the identifier engineering_intelligence already exists in your catalog. If you haven't created it yet, follow steps 1–4 in the Create the dashboard section first. To find the exact identifier of an existing folder, open it in your catalog and check the URL — the identifier appears as the last path segment (e.g., app.getport.io/organization/catalog/my_folder_id). Replace engineering_intelligence in the JSON with your actual folder identifier if it differs.

Next steps​

  • Add automations: Use Port automations to notify teams via Slack when their service drops below Bronze, or create Jira tickets for services with persistently high failure rates.
  • Customize thresholds: The failure rate thresholds (30%, 15%, 5%) are starting points. Adjust them to match your organization's baselines.
  • Combine with Delivery Performance: Stack this scorecard alongside the Delivery Performance scorecard to get both reliability and velocity signals in one dashboard.