One of the Top 10 Pandas functions you must know is Pandas To Datetime. It a need-to-have in your data analysis toolkit. The wonderful thing about to_datetime() is it’s flexibility to read 95% of any dates you’ll throw at it.
Interested in my Top 10 Pandas Functions? Get em here.
Pandas To Datetime (.to_datetime()) will convert your string representation of a date to an actual date format. This is extremely important when utilizing all of the Pandas Date functionality like resample.
1. pd.to_datetime(your_date_data, format="Your_datetime_format")
If you walk away with anything from this post, make sure it’s an understanding of how to use format codes when converting dates. Check out the code sample below.
Pseudo code: Given format, convert a string into a datetime object.
Pandas To Datetime
To DateTime Parameters
.to_datetime() has a ton of parameters and they are all are important to understand. After you become familiar with them, you’ll need to understand date format codes below.
- arg: This is the ‘thing’ that you want to convert to a datetime object. Pandas gives you a ton of flexibility; you can pass a int, float, string, datetime, list, tuple, Series, DataFrame, or dict. That’s a ton of input options!
- format (Default=None): *Very Important* The format parameter will instruct Pandas how to interpret your strings when converting them to DateTime objects. The format must use the format codes below. See examples below.
- origin (Default=’unix’): An origin is simply a reference date. Where do you want to have your universe of timestamps to start? By default is is set to unix which is 1970-01-01.
'julian'is January 1, 4713 BC. You can even set your own origin. - unit: Say you pass an int as your
arg(like 20203939), withunit, you’ll be able specify what unit your int is is away from the origin. In the example here, if we set unit=’s’, this means pandas will interpret 20203939 as 20,203,939 seconds away from the origin. Available units are [D,s,ms,us,ns] - dayfirst: This parameter helps pandas understand if your ‘day’ is first in your format (ex: 01/02/2020 > 2020-02-01). I suggest playing with other parameters first before you try this one.
- yearfirst: Same as the dayfirst parameter above. This will help pandas parse your dates if your year is first. Try the format code options first.
- utc (Default=None): If you want to convert your DateTime objects to timezone-aware (meaning each datetime object also has a timezone) and you want that timezone to be UTC then set utc=True:
DateTime Format Codes
One extremely important concept to understand is DateTime format codes. This is how you instruct Pandas what format your DateTime string is in. It’s magic every time you see it work. In fact, I look forward to gross strings with dates in them just to parse. See documentation.
| Format Code | Description | Examples |
| %a | Weekday, abbreviated | Mon, Tues, Sat |
| %A | Weekday, full name | Monday, Tuesday, Saturday |
| %w | Weekday, decimal. 0=Sunday | 1, 2, 6 |
| %d | Day of month, zero-padded | 01, 02, 21 |
| %b | Month, abbreviated | Jan, Feb, Sep |
| %B | Month, full name | January, February, September |
| %m | Month number, zero-padded | 01, 02, 09 |
| %y | Year, without century, zero-padded | 02, 95, 99 |
| %Y | Year, with century | 1990, 2020 |
| %H | Hour (24 hour), zero padded | 01, 22 |
| %I | Hour (12 hour) zero padded | 01, 12 |
| %p | AM or PM | AM, PM |
| %M | Minute, zero-padded | 01, 02, 43 |
| %S | Second, zero padded | 01, 32, 59 |
| %f | Microsecond, zero-padded | 000001, 000342, 999999 |
| %z | UTC offset ±HHMM[SS[.ffffff]] | +0000, -1030, -3423.234 |
| %Z | Time zone name | ITC, EST, CST |
| %j | Day of year, zero-padded | 001, 365, 023 |
| %U | Week # of year, zero-padded. Sunday first day of week | 00, 01, 51 |
| %W | Week # of year, zero-padded. Monday first day of week | 00, 02, 51 |
| %c | Appropriate date and time | Monday Feb 01 21:30:00 1990 |
| %x | Appropriate Date | 02/01/90 |
| %X | Appropriate Time | 21:22:00 |
| %% | Literal '%' – Use this when you have a % sign in your format. | % |
Let’s run through each iteration of the above parameters
import pandas as pd
Pandas To Datetime¶
Pandas to datetime is a beautiful function that allows you to convert your strings into DateTimes. This is extremely useful when working with Time Series data.
Let's convert strings to datetimes:
- Basic conversion with scalar string
- Convert Pandas Series to datetime
- Convert Pandas Series to datetime w/ custom format
- Convert Unix integer (days) to datetime
- Convert integer (seconds) to datetime
The hardest part about this jupyter notebook will be creating the messy strings to convert. Forgive the plumming you'll see.
1. Basic Basic conversion with scalar string¶
To convert any string to a datetime, you'll need to start with .to_datetime(). This is called directly from the pandas library.
For this first one, I'll show the types of the variables to demonstrate going from a string to a datetime.
string_to_convert = '2020-02-01'
print ('Your string: {}'.format(string_to_convert))
print ('Your string_to_convert type: {}'.format(type(string_to_convert)))
print ()
# Convert your string
new_date = pd.to_datetime(string_to_convert)
print ('Your new date is: {}'.format(new_date))
print ('Your new type is: {}'.format(type(new_date)))
2. Convert Pandas Series to datetime¶
Instead of passing a single string, I usually pass a series of strings that need converting.
Then, I'll replace a DataFrame column with the new Datetime column
First I'll make my series
s = pd.Series(['2020-02-01',
'2020-02-02',
'2020-02-03',
'2020-02-04'])
s
s = pd.to_datetime(s)
s
3. Convert Pandas Series to datetime w/ custom format¶
Let's get into the awesome power of Datetime conversion with format codes. Say you have a messy string with a date inside and you need to convert it to a date. You need to tell pandas how to convert it and this is done via format codes.
Look how cool that is! We can pass any string along with a format and pandas will parse the dates
s = pd.Series(['My 3date is 01199002',
'My 3date is 02199015',
'My 3date is 03199020',
'My 3date is 09199204'])
s
s = pd.to_datetime(s, format="My 3date is %m%Y%d")
s
4. Convert Unix integer (days) to datetime¶
You can also convert integers into Datetimes. You'll need to keep two things in mind
- What is your reference point?
- What is the unit of your integer?
Reference point = What time do you want to start 'counting' your units from?
Unit = Is your integer in terms of # of days, seconds, years, etc.?
pd.to_datetime(14554, unit='D', origin='unix')
5. Convert integer (seconds) to datetime¶
More often, you'll have a unix timestamp that is expresses in seconds. As in seconds away from the default origin of 1970-01-01.
For example, at the time of this post, we are 1,600,355,888 seconds away from 1970-01-01. That's lot of seconds!
pd.to_datetime(1600355888, unit='s', origin='unix')
Bonus: 6. Change your origin or reference point¶
Say your dataset only has # of days after a certain time, but no datetimes. You could either add all of those days via a pd.Timedelta().
Or you could convert them to datetimes with a different origin. Let's check this out from 2020-02-01.
Below, we convert 160 into +160 days after 2020-02-01.
pd.to_datetime(160, unit='D', origin='2020-02-01')
Check out more Pandas functions on our Pandas Page
