How to find unique occurrences in a Python list?

There are instances in which would like to remove duplicated items from your Python lists. Here are a few use cases that come to mind:

  • Counting the number of elements in the list.
  • Comparing different lists.
  • Preparing a list to be used in data analysis in pandas and visualization in matplotlib, or further post processing.
  • Analyzing the probability distribution of the unique list elements.

In this post, we’ll introduce three different techniques to identify if we have any repeated values in the list and obtain a list of unique values that you can later process as needed.

Option 1: Using a Python Set

The most straightforward way to drop list duplicates is to use a Python set object. Sets are data constructs that are by definition composed only of unique elements.

my_lst  = [1,2,3,4,4,5,3,2,1]
print (list( set (my_lst)))

This will return the following list:

[1, 2, 3, 4, 5]

Option 2: Using Pandas

If you are already using the python library for Data Analysis you can use the very handy drop_duplicates Series method to assure uniqueness of your list elements. If pandas is not installed in your Python environment,

import pandas as pd
my_lst  = [1,2,3,4,4,5,3,2,1]
my_unique_lst = list (pd.Series (my_lst).drop_duplicates())

Option 3: Using dict.fromkeys

In Python, We use the dict.fromkeys built in function to create a dictionary made of key value pairs from an iterable such a list. If we pass the dictionary to a list constructor, we filter out all duplicated rows.

my_lst  = [1,2,3,4,4,5,3,2,1]
my_unique_lst = list(dict.fromkeys(my_lst))

This will return our filtered list:

[1, 2, 3, 4, 5]

Option 4: Using a list comprehension and a set

There are several techniques we can use, but filtering out the duplicates using set is the most “pythonic”:

my_unique_lst = [e for e in set(my_lst)]