How to Drop Consecutive Duplicates from a Pandas DataFrame

Learn how to remove `consecutive duplicate rows` in a Pandas DataFrame, even when dealing with string columns.
---
This video is based on the question https://stackoverflow.com/q/64706563/ asked by the user 'abisko' ( https://stackoverflow.com/u/5120812/ ) and on the answer https://stackoverflow.com/a/64706600/ provided by the user 'Bill Huang' ( https://stackoverflow.com/u/3218693/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: Drop consecutive duplicates from DataFrame with multiple columns and with string

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
How to Drop Consecutive Duplicates from a Pandas DataFrame

When working with data in Python, using the Pandas library to manage DataFrames is a common practice. However, during data cleaning, you might encounter situations where consecutive duplicate rows exist in your dataset. For instance, you may have a DataFrame structured as below:

[[See Video to Reveal this Text or Code Snippet]]

This DataFrame looks like this:

[[See Video to Reveal this Text or Code Snippet]]

In this case, you may want to eliminate only the consecutive duplicates, which means you would like to drop the second row (index 1) in this example, resulting in:

[[See Video to Reveal this Text or Code Snippet]]

But, as you might have noticed, the diff() method alone won't work for string columns. So, what’s the best way to approach this?

The Solution

Instead of relying on diff() to find duplicates, you can compare each row with the row directly above it by using the shift() method. This method essentially shifts the rows up by one index, allowing for a straightforward comparison. Here's how to achieve the desired result:

Step-by-Step Explanation

Use the shift() Method: This will create a new DataFrame where each element of the DataFrame is shifted one position upwards.

Compare Original Rows with Shifted Rows: By performing an element-wise comparison between the original DataFrame and the shifted DataFrame, you can identify which rows are identical.

Apply the all() Method: Use the all(axis=1) function to check if all columns in a row are equal.

Filter Out Duplicates: Finally, negate the boolean results from the comparison to "drop" the consecutive duplicates.

The Implementation

Here’s the code you can use to drop consecutive duplicate rows effectively:

[[See Video to Reveal this Text or Code Snippet]]

Output

After executing the above code, the resulting DataFrame will be:

[[See Video to Reveal this Text or Code Snippet]]

Summary

By using the shift() method and comparing rows effectively, you can successfully eliminate consecutive duplicate rows in a Pandas DataFrame, even when dealing with string columns. This method is efficient and ensures that your data remains accurate and clean for further analysis.

Remember, proper data cleaning is essential in any data analysis task, and understanding how to handle duplicates is a significant part of this process.

So next time you face similar issues, you’ll know how to tackle them quickly and efficiently!