Infrastructure ¶

The infrastructure is configured as code via Terraform, for various reasons.

Getting started ¶

Since the Benefits app is deployed into a Microsoft Azure account provided by the California Department of Technology (CDT)’s Office of Enterprise Technology (OET) team, you’ll need to request access from them to the CDT Digital CA directory so you can get into the Azure portal.

The Azure portal is where you can view the infrastructure resources for Benefits. We use GitHub actions to build and deploy those resources.

Environments ¶

Within the CDT Digital CA directory, there is one CDT/ODI Production Subscription, with Resource Groups for each deployed environment.

Each of our environments corresponds to a single Resource Group, Terraform Workspace, and branch.

Environment	Resource Group	Workspace
Dev	`RG-CDT-PUB-VIP-CALITP-D-001`	`dev`
Test	`RG-CDT-PUB-VIP-CALITP-T-001`	`test`
Prod	`RG-CDT-PUB-VIP-CALITP-P-001`	`default`

All resources in these Resource Groups should be reflected in Terraform in this repository. The exceptions are:

Secrets, such as values under Key Vault. prevent_destroy is used on these Resources.
Things managed by DevSecOps

Ownership ¶

The following things in Azure are managed by the California Department of Technology (CDT)’s DevSecOps (OET) team:

Subscriptions
Resource Groups
Front Door
- Web Application Firewall (WAF)
- Distributed denial-of-service (DDoS) protection
IAM
Service connections

You’ll see these referenced in Terraform as data sources, meaning they are managed outside of Terraform.

Architecture ¶

These diagrams show a high-level view of the architecture per environment, including some external systems (e.g. analytics, error monitoring, eligibility servers).

Benefits application ¶

flowchart LR
    internet[Public internet]
    frontdoor[Front Door]
    django[Django application]
    interconnections[Other system interconnections]

    internet --> Cloudflare
    Cloudflare --> frontdoor
    django <--> interconnections

    subgraph Azure
        frontdoor --> NGINX

        subgraph Container App
            direction TB
            NGINX --> django
        end
    end

Front Door also includes the Web Application Firewall (WAF) and handles TLS termination. Front Door is managed by the DevSecOps team.

System interconnections ¶

flowchart LR
    benefits[Cal-ITP Benefits]
    style benefits stroke-width:5px
    recaptcha[Google reCAPTCHA]
    rider((User's browser))
    idg[Identity Gateway]
    elig_server[Eligibility Server]
    ac_data[(Agency Card data)]
    cookies[(Cookies)]
    sentry[Sentry]
    login[Login.gov]
    fare_calc[Transit processor]
    amplitude[Amplitude]

    benefits -->|Errors| sentry
    elig_server -->|Errors| sentry

    rider --> benefits
    rider -->|Credentials and identity proofing| login
    rider --> recaptcha
    rider -->|Payment card info| fare_calc
    rider -->|Events| amplitude
    rider -->|Session| cookies

    benefits --> idg
    benefits <--> recaptcha
    benefits -->|Events| amplitude
    benefits -->|Group enrollment| fare_calc
    benefits --> elig_server

    subgraph "Agency Cards"
    elig_server --> ac_data
    end

    idg --> login
    login -->|User attributes| idg
    idg -->|User attributes| benefits

Naming conventions ¶

The DevSecOps team sets the following naming convention for Resources:

<<Resource Type>>-<<Department>>-<<Public/Private>>-<<Project Category>>-<<Project Name>>-<<Region>><<OS Type>>-<<Environment>>-<<Sequence Number>>

Sample Names ¶

RG-CDT-PUB-VIP-BNSCN-E-D-001
CAE-CDT-PUB-VIP-BNSCN-EL-P-001
CA-CDT-PUB-VIP-BNSCN-EL-D-001

Resource Types ¶

Use the following shorthand for conveying the Resource Type as part of the Resource Name:

Resource	Convention
Container Apps Environment	`CAE`
Container App	`CA`
Virtual Network	`VNET`
Resource Group	`RG`
Virtual Machine	`VM`
Database	`DB`
Subnet	`SNET`
Front Door	`FD`

Making changes ¶

Set up for local development ¶

Get access to the Azure account through the DevSecOps team.
Rebuild and launch the dev container
Navigate to the terraform/ directory.
Authenticate using the Azure CLI.
```
az login
```
Initialize Terraform. You can also use this script later to switch between environments.
```
./init.sh <env>
```
Create a local terraform.tfvars file (ignored by git) using the terraform.tfvars.sample file.
Create local env/[env].tfvars files (ignored by git) using env/dev.tfvars.sample for any environments you plan on interacting with.

Development process ¶

When configuration changes to infrastructure resources are needed, they should be made to the resource definitions in Terraform and submitted via pull request.

Make changes to Terraform files.

Preview the changes, as necessary.

./plan.sh # runs terraform plan with necessary variables hydrated

Submit the changes via pull request.

Azure tags

For Azure resources, you need to ignore changes to tags, since they are automatically created by an Azure Policy managed by CDT.

lifecycle {
    ignore_changes = [tags]
}

Infrastructure workflows ¶

When code is pushed to any branch, a GitHub actions workflow runs terraform plan. When the pull request is merged into main, another workflow runs terraform apply.

Azure environment setup ¶

These steps were followed when setting up our Azure deployment for the first time:

CDT team creates the resources that they own
terraform apply
Set up Slack notifications by creating a Slack email for the #notify-benefits channel, then setting it as a Secret in the Key Vault named slack-benefits-notify-email
Set required Container App configuration and configuration by setting values in Key Vault (the mapping is defined in app_service.tf)

This is not a complete step-by-step guide; more a list of things to remember. This may be useful as part of incident response.