I once had a data teacher told me, “You need to get intimate with your data.” One of the best ways to do this is through pandas describe.

pandas.DataFrame.describe()
pandas.Series.describe()

Pandas Describe does exactly what it sounds like, describe your data. Describe will return a series of descriptive information. This Series will tell you:

• The count of values
• The number of unique values
• The top (most frequent) value
• The frequency of your top value
• The mean, standard deviation, min and max values
• The percentiles of your data: 25%, 50%, 75% by default

Pseudo Code: With your Series or DataFrame, return a Series that tell us what the distribution of values looks like.

Pandas Describe

In order to evaluate a dataset, you need to get a feel for your data. This means you need to get an intuitive sense of how your data is distributed and what spectrum of values you have. This is the first step to launching a successful data analysis.

Often times the process of ‘getting to know your data’ is called Exploratory Data Analysis (EDA).

Pandas Describe Parameters

The standard deviation function is pretty standard, but you may want to play with a view items.

• percentiles = By default, pandas will include the 25th, 50th, and 75th percentile. However you can tell pandas whichever ones you want. Simply pass a list to percentiles and pandas will do the rest.
• include = You may want to ‘describe’ all of your columns, or you may just want to do the numeric columns. By default, pandas will only describe your numeric columns. Select ‘all’ to include all columns.
• exclude = The inverse of include, you can tell pandas which column data types you would like to exclude. Simply pass a list of datatypes you would like to exclude here.
• datetime_is_numeric: By default pandas will treat your datetimes as objects. Meaning, Pandas will not calculate things like ‘average time/date’. However, if you select datetime_is_numeric=True then pandas will apply the min, max, and percentiles to your datetimes.

Now the fun part, let’s take a look at a code sample

import pandas as pd

df = pd.read_csv('../data/Street_Tree_List.csv', parse_dates=['PlantDate'])
df = df[['TreeID', 'qSpecies', 'PlantDate', 'DBH']]
df.rename(mapper={'DBH':"tree_depth"}, axis=1, inplace=True)

df.head()

df.describe()

df.describe(include='all')

/opt/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:1: FutureWarning: Treating datetime data as categorical rather than numeric in `.describe` is deprecated and will be removed in a future version of pandas. Specify `datetime_is_numeric=True` to silence this warning and adopt the future behavior now.
  """Entry point for launching an IPython kernel.

df['PlantDate'].describe()

/opt/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:1: FutureWarning: Treating datetime data as categorical rather than numeric in `.describe` is deprecated and will be removed in a future version of pandas. Specify `datetime_is_numeric=True` to silence this warning and adopt the future behavior now.
  """Entry point for launching an IPython kernel.

count                   68911
unique                   8945
top       2000-06-23 00:00:00
freq                      314
first     1955-09-19 00:00:00
last      2020-07-30 00:00:00
Name: PlantDate, dtype: object

df['PlantDate'].describe(datetime_is_numeric=True)

count                            68911
mean     2000-12-02 22:19:59.122334464
min                1955-09-19 00:00:00
25%                1995-01-30 00:00:00
50%                2001-07-24 00:00:00
75%                2008-11-21 00:00:00
max                2020-07-30 00:00:00
Name: PlantDate, dtype: object

Link to code above

Check out more Pandas functions on our Pandas Page

Official Documentation