Infrastructure

The infrastructure is configured as code via Terraform, for various reasons.

Architecture

System interconnections

flowchart LR
    benefits[Benefits application]
    style benefits stroke-width:5px
    recaptcha[Google reCAPTCHA]
    rider((User's browser))
    idg[Identity Gateway]
    elig_server[Eligibility Server]
    ac_data[(Agency Card data)]
    cookies[(Cookies)]

    benefits -->|Errors| sentry
    elig_server -->|Errors| sentry

    rider --> benefits
    rider -->|Credentials and identity proofing| Login.gov
    rider --> recaptcha
    rider -->|Payment card info| Littlepay
    rider -->|Events| Amplitude
    rider -->|Session| cookies

    benefits --> idg
    benefits <--> recaptcha
    benefits -->|Events| Amplitude
    benefits -->|Group enrollment| Littlepay
    benefits --> elig_server

    subgraph "Agency Cards (e.g. MST Courtesy Cards)"
    elig_server --> ac_data
    end

    idg --> Login.gov
    Login.gov -->|User attributes| idg
    idg -->|User attributes| benefits

Benefits application

flowchart LR
    internet[Public internet]
    frontdoor[Front Door]
    django[Django application]
    interconnections[Other system interconnections]

    internet --> Cloudflare
    Cloudflare --> frontdoor
    django <--> interconnections

    subgraph Azure
        frontdoor --> NGINX

        subgraph App Service
            subgraph Custom container
                direction TB
                NGINX --> django
            end
        end
    end

Front Door also includes the Web Application Firewall (WAF) and handles TLS termination. Front Door is managed by the DevSecOps team.

Ownership

The following things in Azure are managed by the California Department of Technology (CDT)’s DevSecOps (OET) team:

• Subcriptions
• Resource Groups
• Networking
• Front Door
• Web Application Firewall (WAF)
• Distributed denial-of-service (DDoS) protection
• IAM
• Service connections

Environments

Within the CDT Digital CA directory (how to switch), there are two Subscriptions, with Resource Groups under each. Each environment corresponds to a single Resource Group, Terraform Workspace, and branch.

Environment	Subscription	Resource Group	Workspace	Branch
Dev	CDT/ODI Development	RG-CDT-PUB-VIP-CALITP-D-001	dev	dev
Test	CDT/ODI Development	RG-CDT-PUB-VIP-CALITP-T-001	test	test
Prod	CDT/ODI Production	RG-CDT-PUB-VIP-CALITP-P-001	default	prod

All resources in these Resource Groups should be reflected in Terraform in this repository. The exceptions are:

• Secrets, such as values under Key Vault. prevent_destroy is used on these Resources.
• Things managed by DevSecOps

You’ll see these referenced in Terraform as data sources.

For browsing the Azure portal, you can switch your Default subscription filter.

Making changes

Terraform is plan‘d when code is pushed to any branch on GitHub, then apply‘d when merged to dev. While other automation for this project is done through GitHub Actions, we use an Azure Pipeline (above) for a couple of reasons:

• Easier authentication with the Azure API using a service connnection
• Log output is hidden, avoiding accidentally leaking secrets

Local development

1. Get access to the Azure account through the DevSecOps team.

2. Install dependencies:

3. Azure CLI

4. Terraform - see exact version in deploy.yml

5. Authenticate using the Azure CLI.

az login

1. Outside the dev container, navigate to the terraform/ directory.
2. Initialize Terraform. You can also use this script later to switch between environments.

./init.sh <env>

1. Make changes to Terraform files.
2. Preview the changes, as necessary.

terraform plan

1. Submit the changes via pull request.

For Azure resources, you need to ignore changes to tags, since they are automatically created by Azure Policy.

lifecycle {
  ignore_changes = [tags]
}

Naming conventions

The DevSecOps team sets the following naming convention for Resources:

<<Resource Type>>-<<Department>>-<<Public/Private>>-<<Project Category>>-<<Project Name>>-<<Region>><<OS Type>>-<<Environment>>-<<Sequence Number>>

Sample Names

• RG-CDT-PUB-VIP-BNSCN-E-D-001
• ASP-CDT-PUB-VIP-BNSCN-EL-P-001
• AS-CDT-PUB-VIP-BNSCN-EL-D-001

Resource Types

Use the following shorthand for conveying the Resource Type as part of the Resource Name:

Resource	Convention
App Service	AS
App Service Plan	ASP
Virtual Network	VNET
Resource Group	RG
Virtual Machine	VM
Database	DB
Subnet	SNET
Front Door	FD

Azure environment setup

The following steps are required to set up the environment, with linked issues to automate them:

• terraform apply
• Set up Slack notifications by creating a Slack email for the #benefits-notify channel, then setting it as a Secret in the Key Vault named slack-benefits-notify-email
• Set required App Service configuration and configuration by setting values in Key Vault (the mapping is defined in app_service.tf)
• Set up webhook from GitHub to App Service Deployment Center for the Packages event

This is not a complete step-by-step guide; more a list of things to remember. This may be useful as part of incident response.