Pandas Append – pd.DataFrame.append()

So you want to add more rows to your DataFrame? Sure, just use Pandas Append

Pandas DataFrame.append() will append rows (add rows) of other DataFrame, Series, Dictionary or list of these to another DataFrame.

1. YourDataFrame.append(other=Data_You_Want_To_Append)

I have two main uses for this function:

  • When I want to combine entire DataFrames together
  • When I simply want to add a single row to my DataFrame

Pseudo code: Take your new piece of data, and put it on the end (bottom) of your DataFrame

Pandas Append

Pandas Append - Append one dataframe to the end of another DataFrame

Above we are appending DataFrame2 to the end of DataFrame1. To ensure your index comes out clean, make sure to put ignore_index=True.

Pro Tip: If you have many rows to add, it’s computationally cheaper to first combine your data THEN add it to your new DataFrame.

Example: If you have 100s rows to add, instead of .append()-ing 100s times, first combine your 100s rows into a single DataFrame or list of dicts, then .append() once.

This is very similar to python’s regular append.

Append Parameters

  • other: The data that you want to append! This can be:
    • DataFrame: Add one DataFrame to the end of another DataFrame
    • Series: Add a series with index labels of the DataFrame your appending too. ignore_index must be true
    • Dictionary: Simply pass a dictionary who’s keys are the DataFrame columns you’re appending to. ignore_index must be true
  • ignore_index (Default: False): Here you tell pandas if you want to ignore the index of the data your appending. Generally I always select ignore_index=True. If you’re appending a Series or Dict then you’ll need to set this to true
  • verify_integrity (Default: False): This will check if your resulting DataFrame has any duplicate indexes. If it does, pandas will raise an error.
  • sort (Default: False): Sort the new DataFrames columns if the two combined columns do not align (aren’t the same). I generally don’t use this ever.

Here’s a Jupyter notebook showing how to set index in Pandas

In [3]:
import pandas as pd

Pandas Append

Pandas Append will add a piece of data to another DataFrame. This means adding data2 to data1 so you get data1+data2. This is very similar to regular python append.

Let's run through 4 examples:

  1. Appending multiple rows - Appending a DataFrame to a DataFrame
  2. Appending a single row - Appending a Series to a DataFrame
  3. Appending a single row - Appending a Dictionary to a DataFrame
  4. Appending a multiple rows - Appending a list of Dictionaries to a DataFrame

First, let's create 2 DataFrames

In [4]:
df1 = pd.DataFrame([('Foreign Cinema', 'Restaurant', 289.0),
                   ('Liho Liho', 'Restaurant', 224.0),
                   ('500 Club', 'bar', 80.5),
                   ('The Square', 'bar', 25.30)],
           columns=('name', 'type', 'AvgBill')
                 )

df2 = pd.DataFrame([('The Riddler', 'Restaurant', 29.02),
                   ('Smitten Ice Cream', 'Restaurant', 98.60),
                   ('Suppenkuche', 'Restaurant', 40.23),
                   ('Maven', 'bar', 45.20)],
           columns=('name', 'type', 'AvgBill')
                 )

print (df1)
print ()
print (df2)
             name        type  AvgBill
0  Foreign Cinema  Restaurant    289.0
1       Liho Liho  Restaurant    224.0
2        500 Club         bar     80.5
3      The Square         bar     25.3

                name        type  AvgBill
0        The Riddler  Restaurant    29.02
1  Smitten Ice Cream  Restaurant    98.60
2        Suppenkuche  Restaurant    40.23
3              Maven         bar    45.20

1. Appending multiple rows - Appending a DataFrame to a DataFrame

Here we are going to append df2 to df1. This means we will put df2 at the bottom of df1.

In [5]:
df1.append(df2)
Out[5]:
nametypeAvgBill
0Foreign CinemaRestaurant289.00
1Liho LihoRestaurant224.00
2500 Clubbar80.50
3The Squarebar25.30
0The RiddlerRestaurant29.02
1Smitten Ice CreamRestaurant98.60
2SuppenkucheRestaurant40.23
3Mavenbar45.20

Notice how the index from df2 transfered over to the resulting DataFrame (row 5 has an index label=0). If we wanted the regular index to continue, then we set ignore_index=True

In [6]:
df1.append(df2, ignore_index=True)
Out[6]:
nametypeAvgBill
0Foreign CinemaRestaurant289.00
1Liho LihoRestaurant224.00
2500 Clubbar80.50
3The Squarebar25.30
4The RiddlerRestaurant29.02
5Smitten Ice CreamRestaurant98.60
6SuppenkucheRestaurant40.23
7Mavenbar45.20

2. Appending a single row - Appending a Series to a DataFrame

Now in this case, I want to append a single row (instead of a whole DataFrame) to another DataFrame. In this example I'll use a pandas series. You'll need to make sure that your series index labels match up to the DataFame columns.

In [7]:
my_series = pd.Series(data=['SuppenKuche', 'Restaurant', 100.32], index=['name', 'type', 'AvgBill'])
my_series
Out[7]:
name       SuppenKuche
type        Restaurant
AvgBill         100.32
dtype: object
In [8]:
df1.append(my_series, ignore_index=True)
Out[8]:
nametypeAvgBill
0Foreign CinemaRestaurant289.00
1Liho LihoRestaurant224.00
2500 Clubbar80.50
3The Squarebar25.30
4SuppenKucheRestaurant100.32

3. Appending a single row - Appending a Dictionary to a DataFrame

Here, instead of a Series, I'll pass a python dictionary. The keys will need to match up to your DataFrame's column names

In [9]:
my_dictionary = {'name' : 'Nojo Tavern', 'type' : 'Restaurant', 'AvgBill' : 50.32}
my_dictionary
Out[9]:
{'name': 'Nojo Tavern', 'type': 'Restaurant', 'AvgBill': 50.32}
In [10]:
df1.append(my_dictionary, ignore_index=True)
Out[10]:
nametypeAvgBill
0Foreign CinemaRestaurant289.00
1Liho LihoRestaurant224.00
2500 Clubbar80.50
3The Squarebar25.30
4Nojo TavernRestaurant50.32

4. Appending a multiple rows - Appending a list of Dictionaries to a DataFrame

You can also pass a list of Series or a list of Dictionaries to append multiple rows.

In [11]:
my_dictionary_list = [{'name' : 'Fort Point', 'type' : 'Bar', 'AvgBill' : 20.42},
                      {'name' : 'The Gring', 'type' : 'Restaurant', 'AvgBill' : 14.32},
                      {'name' : 'Cafe Reveille', 'type' : 'Cafe', 'AvgBill' : 19.89}]
my_dictionary_list
Out[11]:
[{'name': 'Fort Point', 'type': 'Bar', 'AvgBill': 20.42},
 {'name': 'The Gring', 'type': 'Restaurant', 'AvgBill': 14.32},
 {'name': 'Cafe Reveille', 'type': 'Cafe', 'AvgBill': 19.89}]
In [12]:
df1.append(my_dictionary_list, ignore_index=True)
Out[12]:
nametypeAvgBill
0Foreign CinemaRestaurant289.00
1Liho LihoRestaurant224.00
2500 Clubbar80.50
3The Squarebar25.30
4Fort PointBar20.42
5The GringRestaurant14.32
6Cafe ReveilleCafe19.89

Bonus: When you pass column names that don't align with the original DataFrame

If you pass new column names in your DataFrame, Series, or Dict, then new columns will be made on your original DataFrame. I wish pandas had an option to exclude these new columns as an optional parameter, but that's life.

In [13]:
my_dictionary_new_columns = {'Place_name' : 'Nojo Tavern', 'type' : 'Restaurant', 'AvgBill' : 50.32}
my_dictionary_new_columns
Out[13]:
{'Place_name': 'Nojo Tavern', 'type': 'Restaurant', 'AvgBill': 50.32}
In [15]:
df1.append(my_dictionary_new_columns, ignore_index=True)
Out[15]:
nametypeAvgBillPlace_name
0Foreign CinemaRestaurant289.00NaN
1Liho LihoRestaurant224.00NaN
2500 Clubbar80.50NaN
3The Squarebar25.30NaN
4NaNRestaurant50.32Nojo Tavern

Link to code above

Check out more Pandas functions on our Pandas Page

Official Documentation