Azure App Service Logs ¶
Logs for the environment you are interested in. The following tables are likely of interest:
stderrcoming from the container
AppServiceHTTPLogs: requests coming through App Service
AppServicePlatformLogs: deployment information
For some pre-defined queries, click
Group by: Query type, and look under
Query pack queries.
Live tail ¶
az webapp log tail --resource-group RG-CDT-PUB-VIP-CALITP-P-001 --name AS-CDT-PUB-VIP-CALITP-P-001 2>&1 | grep -v /healthcheck
You can troubleshoot Sentry itself by turning on debug mode and visiting
Specific issues ¶
This section serves as the runbook for Benefits.
Terraform lock ¶
If Terraform commands fail (locally or in the Pipeline) due to an
Error acquiring the state lock:
- Check the
Lock Infofor the
Createdtimestamp. If it’s in the past ten minutes or so, that probably means Terraform is still running elsewhere, and you should wait (stop here).
- Are any Pipeline runs stuck? If so, cancel that build, and try re-running the Terraform command.
- Do any engineers have a Terrafrom command running locally? You’ll need to ask them. For example: They may have started an
applyand it’s sitting waiting for them to approve it. They will need to (gracefully) exit for the lock to be released.
- If none of the steps above identified the source of the lock, and especially if the
Createdtime is more than ten minutes ago, that probably means the last Terraform command didn’t release the lock. You’ll need to grab the
Lock Infooutput and force unlock.
App fails to start ¶
- Check the logs
- Ensure the environment variables and configuration data are set properly.
- Turn on debugging
- Force-push/revert the environment branch back to the old version to roll back
Littlepay API issue ¶
Littlepay API issues may show up as:
- The monitor failing
Connect your cardbutton doesn’t work
A common problem that causes Littlepay API failures is that the certificate expired. To resolve:
- Reach out to firstname.lastname@example.org
- Receive a new certificate
- Put that certificate into the configuration data and/or the GitHub Actions secrets
Eligibility Server ¶
If the Benefits application gets a 403 error when trying to make API calls to the Eligibility Server, it may be because the outbound IP addresses changed, and the Eligibility Server firewall is still restricting access to the old IP ranges.
- Grab the
outputvalues from the most recent Benefit deployment to the relevant environment.
- Update the IP ranges
- Go to the Eligibility Server Pipeline
- Update the relevant variable with the new list of CIDRs
Note there is nightly downtime as the Eligibility Server restarts and loads new data.