In this short tutorial we’ll learn how to quickly calculate the statistical variance of one or multiple columns in a Python Pandas DataFrame. Today. we will focus on several use cases:
- Variance of a Series or Pandas DataFrame column
- Variance of all columns in a Pandas DataFrame
- Variance of a Pandas Groupby object
- Pandas covariance
Create a DataFrame
As we typically do, we’ll start by importing the Pandas library into your favorite Data Analysis environment and then go ahead and create some example data. Feel free to use the DataFrame below to follow along this example.
import pandas as pd
#Define Dataframe columns
language = ['Go', 'Kotlin', 'Swift', 'Java']
first_interview = (76, 78, 84, 83)
second_interview = (51, 59, 58, 58)
third_interview = (15, 12, 19, 24)
# gather data in a dictionary
hr = dict(interview_1=first_interview, interview_2=second_interview, interview_3=second_interview)
# Construct the DataFrame from a dictionary
interviews = pd.DataFrame(hr, index=language)
print (interviews.head())
Here’s an output of our test data:
language | interview_1 | interview_2 | interview_3 | |
---|---|---|---|---|
0 | Go | 76 | 51 | 51 |
1 | Kotlin | 78 | 59 | 59 |
2 | Swift | 84 | 58 | 58 |
3 | Java | 83 | 58 | 58 |
Variance of a Pandas Column / Series
To calculate a Pandas column variant, we simply slice the column and use the var() Series method.
interviews['interview_1'].var().round(2)
Note that we used the round() function to minimize the trailing decimals.
Alternatively, we can define a new Series:
my_s = interviews['interview_1']
my_s.var().round(2)
Variance of all DataFrame columns
If we want to calculate the variance of all columns, we can use the DataFrame var() method, as shown below:
interviews.var().round(2)
This will render the following result:
interview_1 14.92 interview_2 13.67 interview_3 13.67 dtype: float64
You might also want to use the select_dtypes() DataFrame method, to subset the columns by data type:
interviews.select_dtypes(include='int64').var().round()
Calculate variance of some columns
You can use the Pandas ‘brackets’ notation to subset several columns and then apply the calculation.
my_df = interviews[['interview_2', 'interview_3']]
my_df.var().round(2)
This will result in:
interview_2 13.67 interview_3 13.67 dtype: float64
Pandas groupby variance
We’ll first add a new column to our DataFrame and use it group the data and calculate its variance.
interviews['area'] = ['Full_Stack','Full_Stack','Server','Server']
interviews.groupby('area').agg('var')
This will result in the following:
interview_1 | interview_2 | interview_3 | |
---|---|---|---|
area | |||
Full_Stack | 2.0 | 32.0 | 32.0 |
Server | 0.5 | 0.0 | 0.0 |
Get the Pandas covariance
In the same fashion we are able to calculate the covariance of a DataFrame columns
interviews.cov()
Or if we want to calculate the co-variance of a specific DataFrame subset:
interviews[['interview_1','interview_2']].cov()