Prometheus Metrics

A Prometheus query can be used to obtain measurements for analysis.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  args:
  - name: service-name
  metrics:
  - name: success-rate
    interval: 5m
    # NOTE: prometheus queries return results in the form of a vector.
    # So it is common to access the index 0 of the returned array to obtain the value
    successCondition: result[0] >= 0.95
    failureLimit: 3
    provider:
      prometheus:
        address: http://prometheus.example.com:9090
        # timeout is expressed in seconds
        timeout: 40
        headers:
        - key: X-Scope-OrgID
          value: tenant_a
        query: |
          sum(irate(
            istio_requests_total{reporter="source",destination_service=~"{{args.service-name}}",response_code!~"5.*"}[5m]
          )) /
          sum(irate(
            istio_requests_total{reporter="source",destination_service=~"{{args.service-name}}"}[5m]
          ))

The example shows Istio metrics, but you can use any kind of metric available to your prometheus instance. We suggest you validate your PromQL expression using the Prometheus GUI first.

See the Analysis Overview page for more details on the available options.

Authorization

Utilizing Amazon Managed Prometheus

Amazon Managed Prometheus can be used as the prometheus data source for analysis. In order to do this the namespace where your analysis is running will have to have the appropriate IRSA attached to allow for prometheus queries. Once you ensure the proper permissions are in place to access AMP, you can use an AMP workspace url in your provider block and add a SigV4 config for Sigv4 signing:

provider:
  prometheus:
    address: https://aps-workspaces.$REGION.amazonaws.com/workspaces/$WORKSPACEID
    query: |
      sum(irate(
        istio_requests_total{reporter="source",destination_service=~"{{args.service-name}}",response_code!~"5.*"}[5m]
      )) /
      sum(irate(
        istio_requests_total{reporter="source",destination_service=~"{{args.service-name}}"}[5m]
      ))
    authentication:
      sigv4:
        region: $REGION
        profile: $PROFILE
        roleArn: $ROLEARN

With OAuth2

You can setup an OAuth2 client credential flow using the following values:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  args:
  - name: service-name
  # from secret
  - name: oauthSecret  # This is the OAuth2 shared secret
    valueFrom:
      secretKeyRef:
        name: oauth-secret
        key: secret
  metrics:
  - name: success-rate
    interval: 5m
    # NOTE: prometheus queries return results in the form of a vector.
    # So it is common to access the index 0 of the returned array to obtain the value
    successCondition: result[0] >= 0.95
    failureLimit: 3
    provider:
      prometheus:
        address: http://prometheus.example.com:9090
        # timeout is expressed in seconds
        timeout: 40
        authentication:
          oauth2:
            tokenUrl: https://my-oauth2-provider/token
            clientId: my-cliend-id
            clientSecret: "{{ args.oauthSecret }}"
            scopes: [
              "my-oauth2-scope"
            ]
        query: |
          sum(irate(
            istio_requests_total{reporter="source",destination_service=~"{{args.service-name}}",response_code!~"5.*"}[5m]
          )) /
          sum(irate(
            istio_requests_total{reporter="source",destination_service=~"{{args.service-name}}"}[5m]
          ))

The AnalysisRun will first get an access token using that information, and provide it as an Authorization: Bearer header for the metric provider call.

Additional Metadata

Any additional metadata from the Prometheus controller, like the resolved queries after substituting the template's arguments, etc. will appear under the Metadata map in the MetricsResult object of AnalysisRun.

Skip TLS verification

You can skip the TLS verification of the prometheus host provided by setting the options insecure: true.

provider:
  prometheus:
    address: https://prometheus.example.com
    insecure: true
    query: |
      sum(irate(
        istio_requests_total{reporter="source",destination_service=~"{{args.service-name}}",response_code!~"5.*"}[5m]
      )) /
      sum(irate(
        istio_requests_total{reporter="source",destination_service=~"{{args.service-name}}"}[5m]
      ))