Conan smash mirrors! – Remove duplicates in a Python list

As most of the time – there are multiple valid ways of doing it.

Keeping the order

I prefer easy and readable, if it is not crucial for performance to go another way.
Create a new list and add only values to it that are not already included. You can do this with a simple list comprehension or a for loop. This removes the duplicates and keeps the former order.

original_list = [1, 2, 3, 5, 4, 4]
no_duplicates = []
[no_duplicates.append(x) for x in original_list if x not in no_duplicates]
print(no_duplicates)
# output is [1, 2, 3, 5, 4]

original_list = [1, 2, 3, 5, 4, 4]
no_duplicates = []
for i in original_list:
    if i not in no_duplicates:
        no_duplicates.append(i)
print(no_duplicates)

# output is [1, 2, 3, 5, 4]

You can also achieve it by (ab)using a the method fromkeys from a OrderedDict. Keys can exist only once, so duplicates get eliminated and converting the dict back to a list returns the list you want.

from typing import OrderedDict

original_list = [1, 2, 3, 5, 4, 4]
no_duplicates = list(OrderedDict.fromkeys(original_list))
print(no_duplicates)

# output is [1, 2, 3, 5, 4]

Loosing order

As sets do not allow duplicates, converting the list will also present you with the result. But the previous order is not ensured. Only use this if that is not important for your case.

original_list = [1, 2, 3, 5, 4, 4]
no_duplicates = list(set(original_list))
print(no_duplicates)

# output is [1, 2, 3, 4, 5]

With numpy you also have the possibility to remove duplicates directly, but you also lose the previous order. If numpy is not already installed, you would have to call “pip instal numpy”.

import numpy as np

original_list = [1, 2, 3, 5, 4, 4]
no_duplicates = np.unique(original_list).tolist()

print(no_duplicates)

# output is [1, 2, 3, 4, 5]