Scorecard Checks

Check Details

Service Metadata Checks

Application Association Count

Description: This check returns the number of applications associated with a service.

Operations: Equal, less than, less than or equal, greater than, greater than or equal

Threshold: integer number

Example: Application Association Count >= 1 passes if the service is associated with 1 or more applications

CI / CD Configured

Description: This check returns the number of CI / CD plugins configured for a service

Operations: Equal, less than, less than or equal, greater than, greater than or equal

Threshold: integer number

Example: CI / CD Configured >= 1 passes if the service has at least one configured CI / CD plugin

Code Inspection Configured

Description: This check returns the number of code inspection plugins a service has configured.

Operations: Equal, less than, less than or equal, greater than, greater than or equal

Threshold: integer number

Example: Code Inspection Configured >= 1 passes if the service has at least one plugin from the code inspection category configured

Environment Count

Description: This check returns the number of environments associated with a service.

Operations: Equal, less than, less than or equal, greater than, greater than or equal

Threshold: integer number

Example: Environment Count >= 2 passes if the service is associated with 2 or more environments

Environment-Specific Resource Count

Description: This check returns the number of resources associated with the specified environment for a service.

Parameters: Environment name (select one from dropdown)

Operations: Equal, less than, less than or equal, greater than, greater than or equal

Threshold: integer number

Example: Environment Specific Resource Count >= 2 passes if the service has 2 or more resources in the environment specified by the parameter

Filtered Environment-Specific Resource Count

Description: This check returns the number of resources associated with the specified environment for a service filtered by a query that checks properties inside that resource.

Parameters: Environment name (select one from dropdown), Resource type, Property to check, Value to check.

Operations: Equal, less than, less than or equal, greater than, greater than or equal

Threshold: integer number

Example: Let's say you want to check for the value LastUpdatedStatus inside the Configuration property of a Lambda Instance resource:

MemorySize
    128
PackageType
    Zip
Configuration
    CodeSha256 :HssmIMk6jmnU04in6oz8XsA63ZisBeqVoKHBVUCWJUU=
    CodeSize :4995
    Description :A starter AWS Lambda function.
    EphemeralStorage
    Size :512
    FunctionArn :arn:aws:lambda:eu-north-1:943404759651:function:serverlessrepo-hello-world-helloworld-YFTTGjlgiHIZ
    FunctionName :serverlessrepo-hello-world-helloworld-YFTTGjlgiHIZ
    Handler :index.handler
    LastModified :2021-11-03T15:25:07.462+0000
    LastUpdateStatus :Successful
    MemorySize :128

So you can create this check by selecting the parameters: Environment Name: Production

Resource Type: AWS:LAMBDA:Instance

Property to check: Configuration.LastUpdateStatus

Value to check: Successful

Operation: greater than or equal

Threshold: 1

So the check will return true if it finds 1 or more resources in that service and environment that match the Property and Value.

Issue Tracking Configured

Description: This check returns the number of issue-tracking category plugins a service has configured (Currently supported: JIRA).

Operations: Equal, less than, less than or equal, greater than, greater than or equal

Threshold: integer number

Example: Issue Tracking Configured >= 1 passes if the service has at least one plugin from the issue tracking category configured

Lifecycle Defined

Description: This check returns 1 if the service has a lifecycle defined.

Operations: Equal, less than, less than or equal, greater than, greater than or equal

Threshold: integer number

Example: Lifecycle defined = 1 passes if the service has its lifecycle field filled in

Description: This check returns the number of links of the specified type a service has. Link types supported include

  • Runbook

  • Metrics

  • Logs

  • Health Check

  • Documentation

  • Dashboard

  • API

  • Other

Operations: Equal, less than, less than or equal, greater than, greater than or equal

Threshold: integer number

Example: Links of selected type >= 1 pass if the service has at least one link of the specified type defined

Metadata Key Check

Description: This check returns 1 if the service has metadata of the specified name defined. You can look for name, type, or value properties.

Operations: Equal, less than, less than or equal, greater than, greater than or equal

Threshold: integer number

Example: Metadata Key Check of the selected value = 1 pass if the service has a metadata key/type/value defined of the specified value

Metadata Match

Description: The check looks at a specified metadata key, performs a JSON query, and enables text comparison of the result

Operations: Equal contains text; the text begins with, ends with, and matches regexp.

NB: regexp should be in the format without slashes as in literal notation.

Example: Meta tag name: category JSON Query: <empty> Operation: test matches regexp Threshold: [\/]

Threshold: the comparison value

Example: Let's see if a metadata key/value (category/landing) is present on a service.

Meta tag name: category JSON Query: <empty> Operation: equal Threshold: landing

Example: Let's look to see if a metadata key/value (config/{json}) has an address field that matches CC:22:3D:E3:CE:30. The example metadata on the service looks like this:

Meta tag name: config JSON Query: .address Operation: equal Threshold: CC:22:3D:E3:CE:30

HTTP Match

Description: The check requests to an HTTP endpoint. The request is made using the Axios HTTP client, which enables you to configure the request's body and headers richly. You then can use JQ to query the response and compare the result to a threshold. Currently, this metric supports text comparison.

To protect your tokens and passwords you should use our secrets storage feature to store the value and use it in the check with the {{secret.name}} notation. Check the secrets docs here.

Example: Let's call Github's status API to see if Github is operational.

URL: https://www.githubstatus.com/api/v2/summary.json JSON Query: .status.description Operation: equal Threshold: All Systems Operational

Example: Setting the URL to (https://dummyjson.com/products/1) or, in a more complex case where you want to specify the request more completely

{
    "url": "https://dummyjson.com/products/1",
    "method": "post",
    "auth": {
        "username": "janedoe",
        "password": "{{secret.YOUR_PASSWORD}}"
    }
}

or

{
  "url": "https://dummyjson.com/products/1", 
  "headers": {
    "Authorization": "{{secret.YOUR_SECRET_BEARER_TOKEN}}"
  } 
}

The data that you can pass as a JSON config for the request is based on AxiosRequestConfig; you can pass the following properties. For POST/PUT/PATCH/DELETE requests, use the data property, and for the GET request, use the param property.

Also, you can pass auth as a separate parameter or pass it in the headers field.

export interface AxiosRequestConfig<D = any> {
  url?: string;
  method?: Method | string;
  baseURL?: string;
  headers?: (RawAxiosRequestHeaders & MethodsHeaders) | AxiosHeaders;
  params?: any;
  data?: D;
  auth?: AxiosBasicCredentials;
}

To fetch example data, you can apply .title as the JSON query to extract the title property, "iPhone 9." You can then select an operation (equals, for example) and Threshold ("iPhone 9," for example), and the check will return true.

Example: Getting Contributors of a GitHub Repository:

URL: https://api.github.com/repos/USERNAME/REPOSITORY/contributors

JSON Query: .[].login

Operation: equal

Threshold: USERNAME

{
    "url": "https://api.github.com/repos/USERNAME/REPOSITORY/contributors",
    "method": "get",
    "auth": {
        "username": "USERNAME",
        "password": "{{secret.YOUR_GITHUB_PASSWORD}}"
    }
}

Example: Getting all contributors with a condition if none of them are Admin

URL: https://api.github.com/repos/USERNAME/REPOSITORY/contributors

JSON Query: .[].type

Operation: text matches regexp

Threshold: ^(?!.*\bAdmin\b).*$

Regex explanation:

  • `^` and `$` anchor the check to the beginning and end of the string, ensuring we examine the entire content.

  • `(?!.*\bAdmin\b)` is where the magic happens. It’s a negative lookahead assertion that tells the system to pass the check only if "Admin" is nowhere to be found in the string. The `\b` ensures we match only the whole word "Admin", not part of another word.

NB: Our documentation showcased different ways to use JSON Query, notably in contrast to our examples. Specifically, the last two examples start with .[], whereas the first example uses .status.description. This difference boils down to the type of response we're dealing with from the endpoint in question.

When the response is an array of objects, as seen in the last two examples, we use .[] to iterate over each item in the array. On the flip side, when dealing with a single, plain object (like in the first example), we directly access its properties, in this case, with .status.description.

It's crucial to pick the right JSON Query syntax that matches the structure of the endpoint's response. This ensures we're accurately querying and manipulating the data we receive.

The manual and tutorials are available on the official website for a more in-depth look. jq is incredibly powerful for JSON manipulation, and experimenting with it directly is a great way to learn its nuances.

On-Call Configured

Description: This check returns the number of on-call category plugins a service has configured (Currently supported: OpsGenie, PagerDuty).

Operations: Equal, less than, less than or equal, greater than, greater than or equal

Threshold: integer number

Example: On-Call Configured >= 1 passes if the service has at least one plugin from the on-call category configured

Owner Count

Description: This check returns the number of owners associated with a service.

Operations: Equal, less than, less than or equal, greater than, greater than or equal

Threshold: integer number

Example: Owner Count >= 1 passes if the service has 1 or more owners

Package Version

Description: This check returns 1 if the specified package meets the comparison condition specified

Operations: Equal, less than, less than or equal, greater than, greater than or equal

Threshold: version (1.5.0)

Example: Package Version for the specified package = 1 if the specified package meets the comparison condition specified

Repository Configured

Description: This check returns the number of on-call repositories a service has configured.

Operations: Equal, less than, less than or equal, greater than, greater than or equal

Threshold: integer number

Example: Repository configured >= 1 passes if the service has at least one repository configured

Repository Language Composition

Description: This check returns true if the selected language meets the comparison criteria specified.

Operations: Equal, less than, less than or equal, greater than, greater than or equal

Threshold: integer number

Example: Repository Language Composition for the selected language returns 1 if the operator condition is met.

Repository Language Detection

Description: This check returns true if the selected language is present in the service's repository

Operations: Equal, less than, less than or equal, greater than, greater than or equal

Threshold: integer number

Example: Repository Language Detection for the selected language returns 1 if the language is in the service's repository.

Repository Last Commit

Description: This check determines if a repository has received a commit within the specified number of days

Operations: Equal, less than, less than or equal, greater than, greater than or equal

Threshold: integer number - number of days

Example: Repository Last Commit (specified period) = 1 if the repository has a commit within the specified number of days

Repository Number of Required Approvals

Description: This check compares the number of required approvals for a repository against the specified value

Operations: Equal, less than, less than or equal, greater than, greater than or equal

Threshold: integer number - number of approvals

an Example: Repository Number of Required Approvals (a specified number) = 1 if the repository has at least the specified number of approvals

Repository Open PRs Count

Description: This check counts the number of PRs opened for a repository over the past N days.

Days to count: Number of days window you want to check the open PRs

Operations: Equal, less than, less than or equal, greater than, greater than or equal

Threshold: integer number - number of Open PRs on that day

Repository File Contents Matches Regular Expression

Description: This check looks at the contents of a specified file in a service's repository and determines if the contents match a specified regular expression

Example: Check the README.md file and see if "Troubleshooting" is present

File Name: README.MD Regexp: Troubleshooting Operation: Greater than or equal to Threshold: 1

Repository File Exists

Description: This check validates if a specific file or folder exists in the service repository.

filename: Case-sensitive filename with complete path available metrics folder; it should be specified like this. sub_folder/sub_file_name.xx

Operations: Equal, less than, less than or equal, greater than, greater than or equal

Threshold: integer number

JIRA Ticket Count

Description: This check returns the count of tickets matching the specified JQL Query. You can use any configure8 Service Macros in your query to help you identify service attributes for use in your JQL query.

Operations: Equal, less than, less than or equal, greater than, greater than or equal

Threshold: integer number

Example: (component = '{{service.metadata.JIRAComponentName}}' >= 1) if the query returns one or more tickets with the JIRAComponentName nickname metadata as the component name.

Dependable Vulnerability Count

Description: This check returns the count of Dependabot vulnerabilities by the selected Severity.

Operations: Equal, less than, less than or equal, greater than, greater than or equal

Threshold: integer number

SonarQube Metric Count

Description: This check returns the count of SonarQube metrics. You need to select a service so we can pull the list of metrics available. Later, it will fetch that metric for all applied services.

Operations: Equal, less than, less than or equal, greater than, greater than or equal

Threshold: integer number

SonarQube Metric Count from Repository

Description: This check returns the count of SonarQube metrics using the configured sonar.projectKey= property in a sonar-project.properties file placed at the root of the configured repository. Currently this check only works for GitHub repositories.

As we don't have access to the SonarQube Project while building the check, the autocomplete is not available for the metrics, so you have to type the metric as a string.

Operations: Equal, less than, less than or equal, greater than, greater than or equal

Threshold: integer number

Custom Data Check

Check the dedicated sub-page.

Pass on No Data

All scorecard checks allow you to have the check pass if there is no data if this option is selected. For example, if no MTTA data is available for a service and Pass on No Data is selected, the check will pass; otherwise, if there is data, the returned data will be used to determine if the check passes.

Level

The scorecard checks can be assigned a level to evaluate a service's maturity. Once a service passes all the checks in a level, it will move on to the next level. For example, a Silver level check is added for checking services for at least 1 owner and a Gold level check for owners with more than 3. If a service has 2 owners, it will be evaluated as Silver, and if it has 4 levels, it will be evaluated as Gold.

The levels are defined in the Scorecard Metric Levels and can be customized, added, or removed for each organization's evaluation structure.

Using Secrets in Scorecard Requests

Security is critical when setting up scorecard requests for our service, especially when sensitive information like passwords is involved. Instead of directly embedding passwords in your scorecard request parameters, our service allows using a predefined secret manager. This means you can replace hard-coded passwords with secure references to these secrets.

Here's how you make the switch:

Before (Not Recommended):

Embedding the password directly in the request:

{
  "url": "https://api.github.com/repos/USERNAME/REPOSITORY/contributors",
  "method": "get",
  "auth": {
    "username": "username",
    "password": "password"
  }
}

After (Recommended):

Using a secret reference for the password:

{
  "url": "https://api.github.com/repos/USERNAME/REPOSITORY/contributors",
  "method": "get",
  "auth": {
    "username": "username",
    "password": "{{secret.PASSWORD_IDENTIFIER}}"
  }
}

In the recommended method, {{secret.PASSWORD_IDENTIFIER}} is a placeholder for the actual password. This identifier corresponds to a secret stored securely in our secret manager. Doing this ensures that sensitive information is kept secure and not exposed in your configuration.

Learn More About Using Secrets

To learn about secrets, including creating and managing them, visit our detailed guide, Using Secrets in Configure8.

This guide covers everything you need to know about integrating secrets into your requests, offering an extra layer of security and peace of mind.

By following these practices, you can keep your scorecard service configurations secure and streamlined and avoid the pitfalls of hardcoding sensitive information.

Last updated

Copyright © 2023 configure8, Inc. All rights reserved.