Skip to content

Troubleshooting

Tools

Monitoring

We have ping tests set up to notify about availability of each environment. Alerts go to #benefits-notify.

Logs

Azure Container App Logs

In the Azure Portal, in the Container App, you can find real time logs under Monitoring > Log stream and historical logs under Monitoring > Logs

For some pre-defined queries, click Queries, then Group by: Query type, and look under Query pack queries.

Live tail

After setting up the Azure CLI, you can use the following command to stream live logs:

az containerapp logs show --resource-group RG-CDT-PUB-VIP-CALITP-P-001 --name ca-cdt-pub-vip-calitp-p-web --tail 10 >&1 | grep -v /healthcheck

Sentry

Cal-ITP’s Sentry instance collects both errors (“Issues”) and app performance info.

Alerts are sent to #benefits-notify in Slack. Others can be configured.

You can troubleshoot Sentry itself by turning on debug mode and visiting /error/.

Specific issues

Terraform lock

General info

If Terraform commands fail (locally or in the Pipeline) due to an Error acquiring the state lock:

  1. Check the Lock Info for the Created timestamp. If it’s in the past ten minutes or so, that probably means Terraform is still running elsewhere, and you should wait (stop here).
  2. Are any GitHub action runs stuck? If so, cancel that build, and try re-running the Terraform command.
  3. Do any engineers have a Terrafrom command running locally? You’ll need to ask them. For example: They may have started an apply and it’s sitting waiting for them to approve it. They will need to (gracefully) exit for the lock to be released.
  4. If none of the steps above identified the source of the lock, and especially if the Created time is more than ten minutes ago, that probably means the last Terraform command didn’t release the lock. You’ll need to grab the ID from the Lock Info output and force unlock.

App fails to start

If the container fails to start, you should see a downtime alert. Assuming this app version was working in another environment, the issue is likely due to misconfiguration. Some things you can do:

Littlepay API issue

Littlepay API issues may show up as:

  • The monitor failing
  • The Connect your card button doesn’t work

A common problem that causes Littlepay API failures is that the certificate expired. To resolve:

  1. Reach out to support@littlepay.com
  2. Receive a new certificate
  3. Put that certificate into the configuration data and/or the GitHub Actions secrets

Eligibility Server

It would be unexpected, but a 403 error calling the Eligibility Server would indicate that the outbound IP address has changed, and that the new IP is not on the Eligibility Server firewall’s allowlist.

  1. Retrieve the IP address value from the Public IP address resource in the Azure Portal for the relevant environment.
  2. Update the IP address:
    1. Go to the Eligibility Server Pipeline
    2. Click Edit
    3. Click Variables
    4. Update the relevant variable with the new public IP address

Note there is nightly downtime as the Eligibility Servers restart and load new data.