Scorecard Checks
Check Details
Service Metadata Checks
Application Association Count
Description: This check returns the number of applications associated with a service.
Operations: Equal, less than, less than or equal, greater than, greater than or equal
Threshold: integer number
Example: Application Association Count >= 1 passes if the service is associated with 1 or more applications
CI / CD Configured
Description: This check returns the number of CI / CD plugins configured for a service
Operations: Equal, less than, less than or equal, greater than, greater than or equal
Threshold: integer number
Example: CI / CD Configured >= 1 passes if the service has at least one configured CI / CD plugin
Code Inspection Configured
Description: This check returns the number of code inspection plugins a service has configured.
Operations: Equal, less than, less than or equal, greater than, greater than or equal
Threshold: integer number
Example: Code Inspection Configured >= 1 passes if the service has at least one plugin from the code inspection category configured
Environment Count
Description: This check returns the number of environments associated with a service.
Operations: Equal, less than, less than or equal, greater than, greater than or equal
Threshold: integer number
Example: Environment Count >= 2 passes if the service is associated with 2 or more environments
Environment-Specific Resource Count
Description: This check returns the number of resources associated with the specified environment for a service.
Parameters: Environment name (select one from dropdown)
Operations: Equal, less than, less than or equal, greater than, greater than or equal
Threshold: integer number
Example: Environment Specific Resource Count >= 2 passes if the service has 2 or more resources in the environment specified by the parameter
Filtered Environment-Specific Resource Count
Description: This check returns the number of resources associated with the specified environment for a service filtered by a query that checks properties inside that resource.
Parameters: Environment name (select one from dropdown), Resource type, Property to check, Value to check.
Operations: Equal, less than, less than or equal, greater than, greater than or equal
Threshold: integer number
Example: Let's say you want to check for the value LastUpdatedStatus inside the Configuration property of a Lambda Instance resource:
So you can create this check by selecting the parameters:
Environment Name: Production
Resource Type: AWS:LAMBDA:Instance
Property to check: Configuration.LastUpdateStatus
Value to check: Successful
Operation: greater than or equal
Threshold: 1
So the check will return true if it finds 1 or more resources in that service and environment that match the Property and Value.
Issue Tracking Configured
Description: This check returns the number of issue-tracking category plugins a service has configured (Currently supported: JIRA).
Operations: Equal, less than, less than or equal, greater than, greater than or equal
Threshold: integer number
Example: Issue Tracking Configured >= 1 passes if the service has at least one plugin from the issue tracking category configured
Lifecycle Defined
Description: This check returns 1 if the service has a lifecycle defined.
Operations: Equal, less than, less than or equal, greater than, greater than or equal
Threshold: integer number
Example: Lifecycle defined = 1 passes if the service has its lifecycle field filled in
Links
Description: This check returns the number of links of the specified type a service has. Link types supported include
Runbook
Metrics
Logs
Health Check
Documentation
Dashboard
API
Other
Operations: Equal, less than, less than or equal, greater than, greater than or equal
Threshold: integer number
Example: Links of selected type >= 1 pass if the service has at least one link of the specified type defined
Metadata Key Check
Description: This check returns 1 if the service has metadata of the specified name defined. You can look for name, type, or value properties.
Operations: Equal, less than, less than or equal, greater than, greater than or equal
Threshold: integer number
Example: Metadata Key Check of the selected value = 1 pass if the service has a metadata key/type/value defined of the specified value
Metadata Match
Description: The check looks at a specified metadata key, performs a JSON query, and enables text comparison of the result
Operations: Equal contains text; the text begins with, ends with, and matches regexp.
NB: regexp should be in the format without slashes as in literal notation.
Example:
Meta tag name: category
JSON Query: <empty>
Operation: test matches regexp
Threshold: [\/]
Threshold: the comparison value
Example: Let's see if a metadata key/value (category/landing) is present on a service.
Meta tag name: category
JSON Query: <empty>
Operation: equal
Threshold: landing
Example: Let's look to see if a metadata key/value (config/{json}) has an address field that matches CC:22:3D:E3:CE:30. The example metadata on the service looks like this:
Meta tag name: config
JSON Query: .address
Operation: equal
Threshold: CC:22:3D:E3:CE:30
HTTP Match
Description: The check requests to an HTTP endpoint. The request is made using the Axios HTTP client, which enables you to configure the request's body and headers richly. You then can use JQ to query the response and compare the result to a threshold. Currently, this metric supports text comparison.
To protect your tokens and passwords you should use our secrets storage feature to store the value and use it in the check with the {{secret.name}} notation. Check the secrets docs here.
Example: Let's call Github's status API to see if Github is operational.
URL: https://www.githubstatus.com/api/v2/summary.json
JSON Query: .status.description
Operation: equal
Threshold: All Systems Operational
Example: Setting the URL to (https://dummyjson.com/products/1) or, in a more complex case where you want to specify the request more completely
or
To fetch example data, you can apply .title as the JSON query to extract the title property, "iPhone 9." You can then select an operation (equals, for example) and Threshold ("iPhone 9," for example), and the check will return true.
Example: Getting Contributors of a GitHub Repository:
URL: https://api.github.com/repos/USERNAME/REPOSITORY/contributors
JSON Query: .[].login
Operation: equal
Threshold: USERNAME
Example: Getting all contributors with a condition if none of them are Admin
URL: https://api.github.com/repos/USERNAME/REPOSITORY/contributors
JSON Query: .[].type
Operation: text matches regexp
Threshold: ^(?!.*\bAdmin\b).*$
Regex explanation:
`
^` and `$` anchor the check to the beginning and end of the string, ensuring we examine the entire content.`
(?!.*\bAdmin\b)` is where the magic happens. It’s a negative lookahead assertion that tells the system to pass the check only if "Admin" is nowhere to be found in the string. The `\b` ensures we match only the whole word "Admin", not part of another word.
NB: Our documentation showcased different ways to use JSON Query, notably in contrast to our examples. Specifically, the last two examples start with .[], whereas the first example uses .status.description. This difference boils down to the type of response we're dealing with from the endpoint in question.
When the response is an array of objects, as seen in the last two examples, we use .[] to iterate over each item in the array. On the flip side, when dealing with a single, plain object (like in the first example), we directly access its properties, in this case, with .status.description.
It's crucial to pick the right JSON Query syntax that matches the structure of the endpoint's response. This ensures we're accurately querying and manipulating the data we receive.
The manual and tutorials are available on the official website for a more in-depth look. jq is incredibly powerful for JSON manipulation, and experimenting with it directly is a great way to learn its nuances.
On-Call Configured
Description: This check returns the number of on-call category plugins a service has configured (Currently supported: OpsGenie, PagerDuty).
Operations: Equal, less than, less than or equal, greater than, greater than or equal
Threshold: integer number
Example: On-Call Configured >= 1 passes if the service has at least one plugin from the on-call category configured
Owner Count
Description: This check returns the number of owners associated with a service.
Operations: Equal, less than, less than or equal, greater than, greater than or equal
Threshold: integer number
Example: Owner Count >= 1 passes if the service has 1 or more owners
Package Version
Description: This check returns 1 if the specified package meets the comparison condition specified
Operations: Equal, less than, less than or equal, greater than, greater than or equal
Threshold: version (1.5.0)
Example: Package Version for the specified package = 1 if the specified package meets the comparison condition specified
Repository Configured
Description: This check returns the number of on-call repositories a service has configured.
Operations: Equal, less than, less than or equal, greater than, greater than or equal
Threshold: integer number
Example: Repository configured >= 1 passes if the service has at least one repository configured
Repository Language Composition
Description: This check returns true if the selected language meets the comparison criteria specified.
Operations: Equal, less than, less than or equal, greater than, greater than or equal
Threshold: integer number
Example: Repository Language Composition for the selected language returns 1 if the operator condition is met.
Repository Language Detection
Description: This check returns true if the selected language is present in the service's repository
Operations: Equal, less than, less than or equal, greater than, greater than or equal
Threshold: integer number
Example: Repository Language Detection for the selected language returns 1 if the language is in the service's repository.
Repository Last Commit
Description: This check determines if a repository has received a commit within the specified number of days
Operations: Equal, less than, less than or equal, greater than, greater than or equal
Threshold: integer number - number of days
Example: Repository Last Commit (specified period) = 1 if the repository has a commit within the specified number of days
Repository Number of Required Approvals
Description: This check compares the number of required approvals for a repository against the specified value
Operations: Equal, less than, less than or equal, greater than, greater than or equal
Threshold: integer number - number of approvals
an Example: Repository Number of Required Approvals (a specified number) = 1 if the repository has at least the specified number of approvals
Repository Open PRs Count
Description: This check counts the number of PRs opened for a repository over the past N days.
Days to count: Number of days window you want to check the open PRs
Operations: Equal, less than, less than or equal, greater than, greater than or equal
Threshold: integer number - number of Open PRs on that day
Repository File Contents Matches Regular Expression
Description: This check looks at the contents of a specified file in a service's repository and determines if the contents match a specified regular expression
Example: Check the README.md file and see if "Troubleshooting" is present
File Name: README.MD
Regexp: Troubleshooting
Operation: Greater than or equal to
Threshold: 1
Repository File Exists
Description: This check validates if a specific file or folder exists in the service repository.
filename: Case-sensitive filename with complete path available metrics folder; it should be specified like this. sub_folder/sub_file_name.xx
Operations: Equal, less than, less than or equal, greater than, greater than or equal
Threshold: integer number
JIRA Ticket Count
Description: This check returns the count of tickets matching the specified JQL Query. You can use any configure8 Service Macros in your query to help you identify service attributes for use in your JQL query.
Operations: Equal, less than, less than or equal, greater than, greater than or equal
Threshold: integer number
Example: (component = '{{service.metadata.JIRAComponentName}}' >= 1) if the query returns one or more tickets with the JIRAComponentName nickname metadata as the component name.
Dependable Vulnerability Count
Description: This check returns the count of Dependabot vulnerabilities by the selected Severity.
Operations: Equal, less than, less than or equal, greater than, greater than or equal
Threshold: integer number
SonarQube Metric Count
Description: This check returns the count of SonarQube metrics. You need to select a service so we can pull the list of metrics available. Later, it will fetch that metric for all applied services.
Operations: Equal, less than, less than or equal, greater than, greater than or equal
Threshold: integer number
SonarQube Metric Count from Repository
Description: This check returns the count of SonarQube metrics using the configured sonar.projectKey= property in a sonar-project.properties file placed at the root of the configured repository.
Currently this check only works for GitHub repositories.
As we don't have access to the SonarQube Project while building the check, the autocomplete is not available for the metrics, so you have to type the metric as a string.
Operations: Equal, less than, less than or equal, greater than, greater than or equal
Threshold: integer number
Custom Data Check
Check the dedicated sub-page.
Pass on No Data
All scorecard checks allow you to have the check pass if there is no data if this option is selected. For example, if no MTTA data is available for a service and Pass on No Data is selected, the check will pass; otherwise, if there is data, the returned data will be used to determine if the check passes.
Level
The scorecard checks can be assigned a level to evaluate a service's maturity. Once a service passes all the checks in a level, it will move on to the next level. For example, a Silver level check is added for checking services for at least 1 owner and a Gold level check for owners with more than 3. If a service has 2 owners, it will be evaluated as Silver, and if it has 4 levels, it will be evaluated as Gold.
The levels are defined in the Scorecard Metric Levels and can be customized, added, or removed for each organization's evaluation structure.
Using Secrets in Scorecard Requests
Security is critical when setting up scorecard requests for our service, especially when sensitive information like passwords is involved. Instead of directly embedding passwords in your scorecard request parameters, our service allows using a predefined secret manager. This means you can replace hard-coded passwords with secure references to these secrets.
Here's how you make the switch:
Before (Not Recommended):
Embedding the password directly in the request:
After (Recommended):
Using a secret reference for the password:
In the recommended method, {{secret.PASSWORD_IDENTIFIER}} is a placeholder for the actual password. This identifier corresponds to a secret stored securely in our secret manager. Doing this ensures that sensitive information is kept secure and not exposed in your configuration.
Learn More About Using Secrets
To learn about secrets, including creating and managing them, visit our detailed guide, Using Secrets in Configure8.
This guide covers everything you need to know about integrating secrets into your requests, offering an extra layer of security and peace of mind.
By following these practices, you can keep your scorecard service configurations secure and streamlined and avoid the pitfalls of hardcoding sensitive information.
Last updated
