Helpful Links

Here are some resources data analysts have collected and referenced, that will hopefully help you out in your work. Have something you want to share? Create a new markdown file, add it to the example report folder, and message Amanda.

• Data Analysis

  • Python

  • Pandas

  • Summarizing

  • Merging

  • Dates

  • Monetary Values

• Visualizations

  • Charts

  • Maps

  • DataFrames

  • Ipywidgets

  • Markdown

Data Analysis

Python

• Composing Programs: comprehensive Python course

• Intermediate Python: tips for improving your programs

• Stop Python from executing code when a module is imported.

• Loop through 2 lists with zip in parallel.

• Find the elements that are in one list, but not in another list.

• What does += do?

Pandas

• Turn columns into dummy variables.

• Export multiple dataframes into their own sheets to a single Excel workbook.

• Display multiple dataframes side by side.

• Display all rows or columns of a dataframe in the notebook

Summarizing

• Groupby and calculate a new value, then use that value within your DataFrame.

• Explanation of the split-apply-combine paradigm.

• Pandas profiling tool: creates html reports from DataFrames.

  • Examples

Merging

• When working with data sets where the “merge on” column is a string data type, it can be difficult to get the DataFrames to join. For example, df1 lists County of Sonoma, Human Services Department, Adult and Aging Division, but df2 references the same department as: County of Sonoma (Human Services Department) .

  • Potential Solution #1: fill in a column in one DataFrame that has a partial match with the string values in another one.

  • Potential Solution #2: use the package fuzzymatcher. This will require you to carefully comb through for any bad matches.

  • Potential Solution #3: if you don’t have too many values, use a dictionary.

Dates

• Use shift to calculate the number of days between two dates.

df['n_days_between'] = (df['prepared_date'] - df.shift(1)['prepared_date']).dt.days

• Assign fiscal year to a date.

# Make sure your column is a date time object
df['financial_year'] = df['base_date'].map(lambda x: x.year if x.month > 3 else x.year-1)

Monetary Values

• Reformat values that are in scientific notation into millions or thousands.

  • Example in notebook.

    x=alt.X("Funding Amount", axis=alt.Axis(format="$.2s", title="Obligated Funding ($2021)"))

• Reformat values from 19000000 to $19.0M.

• Adjust for inflation.

# Must install and import cpi package for the function to work.
def adjust_prices(df):
    cols =  ["total_requested",
           "fed_requested",
           "ac_requested"]

    def inflation_table(base_year):
        cpi.update()
        series_df = cpi.series.get(area="U.S. city average").to_dataframe()
        inflation_df = (series_df[series_df.year >= 2008]
                        .pivot_table(index='year', values='value', aggfunc='mean')
                        .reset_index()
                       )
        denominator = inflation_df.value.loc[inflation_df.year==base_year].iloc[0]

        inflation_df = inflation_df.assign(
        inflation = inflation_df.value.divide(denominator)
        )

        return inflation_df

    ##get cpi table
    cpi = inflation_table(2021)
    cpi.update
    cpi = (cpi>>select(_.year, _.value))
    cpi_dict = dict(zip(cpi['year'], cpi['value']))


    for col in cols:
        multiplier = df["prepared_y"].map(cpi_dict)

        ##using 270.97 for 2021 dollars
        df[f"adjusted_{col}"] = ((df[col] * 270.97) / multiplier)
    return df

Visualization

Charts

Altair

• Manually concatenate a bar chart and line chart to create a dual axis graph.

• Adjust the time units of a datetime column for an axis.

• Label the lines on a line chart.

• Layer altair charts, lose color with no encoding, workaround to get different colors to appear on legend.

• Add regression line to scatterplot.

• Adjust scales for axes to be the min and max values.

• Resolving the error ‘TypeError: Object of type ‘Timestamp’ is not JSON serializable’

• Manually sort a legend.

• Add tooltip to chart functions.

def add_tooltip(chart, tooltip1, tooltip2):
    chart = (
        chart.encode(tooltip= [tooltip1,tooltip2]))
    return chart

Maps

• Examples of folium, branca, and color maps.

• Quick interactive maps with Geopandas.gdf.explore()

DataFrames

• Styling dataframes with HTML.

• After styling a DataFrame, you will have to access the underlying data with .data.

ipywidgets

Tabs

• Create tabs to switch between different views.

• Stack Overflow Help.

  • Notebook example.

  • Example on Ipywidgets docs page.

Markdown

• Create a markdown table.

• Add a table of content that links to headers throughout a markdown file.

• Add links to local files.

• Direct embed an image.