Skip to content

Infrastructure

The infrastructure is configured as code via Terraform, for various reasons.

Getting started

Since the Benefits app is deployed into a Microsoft Azure account provided by the California Department of Technology (CDT)’s Office of Enterprise Technology (OET) team, you’ll need to request access from them to the CDT Digital CA directory so you can get into the Azure portal, and to the California Department of Technology directory so you can access Azure DevOps.

The Azure portal is where you can view the infrastructure resources for Benefits. Azure DevOps is where our infrastructure pipeline is run to build and deploy those infrastructure resources.

Environments

Within the CDT Digital CA directory, there are two Subscriptions, with Resource Groups under each. (Refer to Azure’s documentation for switching directories.)

Each of our environments corresponds to a single Resource Group, Terraform Workspace, and branch.

Environment Subscription Resource Group Workspace Branch
Dev CDT/ODI Development RG-CDT-PUB-VIP-CALITP-D-001 dev main
Test CDT/ODI Development RG-CDT-PUB-VIP-CALITP-T-001 test test
Prod CDT/ODI Production RG-CDT-PUB-VIP-CALITP-P-001 default prod

All resources in these Resource Groups should be reflected in Terraform in this repository. The exceptions are:

Ownership

The following things in Azure are managed by the California Department of Technology (CDT)’s DevSecOps (OET) team:

  • Subcriptions
  • Resource Groups
  • Networking
  • Front Door
    • Web Application Firewall (WAF)
    • Distributed denial-of-service (DDoS) protection
  • IAM
  • Service connections

You’ll see these referenced in Terraform as data sources, meaning they are managed outside of Terraform.

Architecture

These diagrams show a high-level view of the architecture per environment, including some external systems (e.g. analytics, error monitoring, eligibility servers).

Benefits application

flowchart LR
    internet[Public internet]
    frontdoor[Front Door]
    django[Django application]
    interconnections[Other system interconnections]

    internet --> Cloudflare
    Cloudflare --> frontdoor
    django <--> interconnections

    subgraph Azure
        frontdoor --> NGINX

        subgraph App Service
            subgraph Custom container
                direction TB
                NGINX --> django
            end
        end
    end

Front Door also includes the Web Application Firewall (WAF) and handles TLS termination. Front Door is managed by the DevSecOps team.

System interconnections

flowchart LR
    benefits[Benefits application]
    style benefits stroke-width:5px
    recaptcha[Google reCAPTCHA]
    rider((User's browser))
    idg[Identity Gateway]
    elig_server[Eligibility Server]
    ac_data[(Agency Card data)]
    cookies[(Cookies)]

    benefits -->|Errors| sentry
    elig_server -->|Errors| sentry

    rider --> benefits
    rider -->|Credentials and identity proofing| Login.gov
    rider --> recaptcha
    rider -->|Payment card info| Littlepay
    rider -->|Events| Amplitude
    rider -->|Session| cookies

    benefits --> idg
    benefits <--> recaptcha
    benefits -->|Events| Amplitude
    benefits -->|Group enrollment| Littlepay
    benefits --> elig_server

    subgraph "Agency Cards (e.g. MST Courtesy Cards)"
    elig_server --> ac_data
    end

    idg --> Login.gov
    Login.gov -->|User attributes| idg
    idg -->|User attributes| benefits

Naming conventions

The DevSecOps team sets the following naming convention for Resources:

<<Resource Type>>-<<Department>>-<<Public/Private>>-<<Project Category>>-<<Project Name>>-<<Region>><<OS Type>>-<<Environment>>-<<Sequence Number>>

Sample Names

  • RG-CDT-PUB-VIP-BNSCN-E-D-001
  • ASP-CDT-PUB-VIP-BNSCN-EL-P-001
  • AS-CDT-PUB-VIP-BNSCN-EL-D-001

Resource Types

Use the following shorthand for conveying the Resource Type as part of the Resource Name:

Resource Convention
App Service AS
App Service Plan ASP
Virtual Network VNET
Resource Group RG
Virtual Machine VM
Database DB
Subnet SNET
Front Door FD

Making changes

Set up for local development

  1. Get access to the Azure account through the DevSecOps team.

  2. Install dependencies:

  3. Authenticate using the Azure CLI.

    az login

  4. Outside the dev container, navigate to the terraform/ directory.

  5. Initialize Terraform. You can also use this script later to switch between environments.

    ./init.sh <env>

  6. Create a local terraform.tfvars file (ignored by git) from the sample; fill in the *_OBJECT_ID variables with values from the Azure Pipeline definition.

Development process

When configuration changes to infrastructure resources are needed, they should be made to the resource definitions in Terraform and submitted via pull request.

  1. Make changes to Terraform files.

  2. Preview the changes, as necessary.

    terraform plan

  3. Submit the changes via pull request.

Azure tags

For Azure resources, you need to ignore changes to tags, since they are automatically created by an Azure Policy managed by CDT.

lifecycle {
    ignore_changes = [tags]
}

Infrastructure pipeline

Build Status

When code is pushed to any branch on GitHub, our infrastructure pipeline in Azure DevOps runs terraform plan. When the pull request is merged into main, the pipeline runs terraform apply.

While other automation for this project is done through GitHub Actions, we use an Azure Pipeline for a couple of reasons:

  • Easier authentication with the Azure API using a service connnection
  • Log output is hidden, avoiding accidentally leaking secrets

Azure environment setup

These steps were followed when setting up our Azure deployment for the first time:

This is not a complete step-by-step guide; more a list of things to remember. This may be useful as part of incident response.