Literal
Basic solution with for-loop
The basic solution to remove the duplicates is to use for-loop. Prepare an empty list and add an item if the same value has not been added to the list.
from typing import List
duplicated_int_list = [1, 2, 3, 4, 2, 3, 4, 5]
def remove_with_for(data_list: List[int]):
result = []
for item in data_list:
if item not in result:
result.append(item)
return result
print(remove_with_for(duplicated_int_list))
# [1, 2, 3, 4, 5]This is simple and readable. Non-Python developers can also understand what it does.
Remove duplicates with set
I guess many languages have set class, so many developers can understand the code.
def remove_with_set(data_list: List[int]):
return list(set(data_list))
duplicated_int_list = [1, 2, 3, 4, 2, 3, 4, 5]
print(remove_with_set(duplicated_int_list))
# [1, 2, 3, 4, 5]The result is correct.
Remove duplicates with dict.fromKeys
I don’t like this way because the intention is not clear but it works.
def remove_with_from_keys(data_list: List[int]):
keys = dict.fromkeys(data_list)
# {1: None, 2: None, 3: None, 4: None, 5: None}
print(keys)
return list(keys)
duplicated_int_list = [1, 2, 3, 4, 2, 3, 4, 5]
print(remove_with_from_keys(duplicated_int_list))
# [1, 2, 3, 4, 5]Remove duplicates by using index func
This technique is often used in other languages like JavaScript. Search for the same value and the index. Once it’s found, compare the current index and the found index. If the index is the same, it is the first occurrence. Otherwise, it’s duplicated value.
def remove_with_comparing_index(data_list: List[int]):
result = []
for index, item in enumerate(data_list):
if index == data_list.index(item):
result.append(item)
return result
duplicated_int_list = [1, 2, 3, 4, 2, 3, 4, 5]
print(remove_with_comparing_index(duplicated_int_list))
# [1, 2, 3, 4, 5]This post explains it in more detail with a table.
Object
When it comes to an object list, it’s harder to remove duplicates than a literal list. I created the following simple class for the examples.
class Point:
def __init__(self, x, y) -> None:
self.x = x
self.y = y
class PointEx(Point):
def __eq__(self, __o: object) -> bool:
return self.x == getattr(__o, "x", None) and self.y == getattr(__o, "y", None)
def __hash__(self) -> int:
return hash(("x", self.x, "y", self.y))
def show_points(points: Union[List[Point], List[PointEx]]):
for point in points:
print(f"({point.x}, {point.y})")Comparing two objects returns false by default. PointEx improves it.
duplicated_points = [
Point(1, 1),
Point(2, 2),
Point(3, 3),
Point(4, 4),
Point(2, 2),
Point(6, 3),
Point(4, 4),
Point(5, 5),
]
print(Point(1, 1) == Point(1, 1)) # False
duplicated_pointExes = [
PointEx(1, 1),
PointEx(2, 2),
PointEx(3, 3),
PointEx(4, 4),
PointEx(2, 2),
PointEx(6, 3),
PointEx(4, 4),
PointEx(5, 5),
]
print(PointEx(4, 4).__eq__(PointEx(4, 4))) # True
print(PointEx(4, 4) == PointEx(4, 4)) # TrueRemove duplicates from an object list with for loop
The first solution for an object list is also for loop. If you don’t want to define __eq__ and __hash__ functions, this is the first choice.
def equal_point(a: Point, b: Point):
return a.x == b.x and a.y == b.y
def remove_object_for(points: List[Point]):
result: List[Point] = []
for current in points:
found = False
for stored in result:
if equal_point(current, stored):
found = True
if found is False:
result.append(current)
return result
show_points(remove_object_for(duplicated_points))
# (1, 1)
# (2, 2)
# (3, 3)
# (4, 4)
# (6, 3)
# (5, 5)Basically, the logic is the same as the literal version but it looks worse than the literal version.
If __eq__ function is defined in the class, the two objects is directly comparable.
def remove_object_for2(points: List[PointEx]):
result: List[PointEx] = []
for current in points:
found = False
for stored in result:
if current == stored:
found = True
if found is False:
result.append(current)
return resultBut this doesn’t improve the code. If __eq__ function is correctly implemented, it can be written in the same way as the literal version.
def remove_object_for3(points: List[PointEx]):
result: List[PointEx] = []
for current in points:
if current not in result:
result.append(current)
return resultOf course, the result is the same.
Remove duplicates from an object list with set
set can’t be used. To use set, both __eq__ and __hash__ need to be implemented.
print(Point(4, 4).__eq__(Point(4, 4))) # NotImplemented
show_points(list(set(duplicated_points)))
# (2, 2)
# (4, 4)
# (1, 1)
# (2, 2)
# (5, 5)
# (3, 3)
# (4, 4)
# (6, 3)
print(PointEx(4, 4).__eq__(PointEx(4, 4))) # True
show_points(list(set(duplicated_pointExes)))
# (6, 3)
# (3, 3)
# (1, 1)
# (4, 4)
# (5, 5)
# (2, 2)If the class has a lot of variables, the function length becomes long but if they are implemented in the class, the business logic side gets more readable.
Remove duplicates from an object list with index func
When using index, it’s necessary to define a function to return an index for the searched value. Once it’s implemented, the same logic as the literal version can be applied.
def remove_object_enumerate(points: List[Point]):
def index_where(data_list: List[Point], search: Point):
for index, item in enumerate(data_list):
if equal_point(item, search) is True:
return index
return -1
result: List[Point] = []
for index, current in enumerate(points):
if index == index_where(points, current):
result.append(current)
return result
show_points(remove_object_enumerate(duplicated_points))
# (1, 1)
# (2, 2)
# (3, 3)
# (4, 4)
# (6, 3)
# (5, 5)Partial comparison
If __eq__ and __hash__ are implemented, the same comparison is applied to all comparisons. However, sometimes two objects need to be compared with some properties only but not all the properties. How can we implement it if we want to change the condition?
Let’s define a new method in the class.
class PointEx(Point):
# ... __eq__ and __hash__ functions here
def equal_with(self, __o: object, *keys: str) -> bool:
if isinstance(__o, PointEx) is False:
return False
if len(keys) == 0:
return True
for key in keys:
if getattr(self, key) != getattr(__o, key):
return False
return TrueThis function allows us to use arbitrary properties for comparison. Let’s compare two PointEx object with y property.
duplicated_pointExes = [
PointEx(1, 1),
PointEx(2, 2),
PointEx(3, 3),
PointEx(4, 4),
PointEx(2, 2),
PointEx(6, 3),
PointEx(4, 4),
PointEx(5, 5),
]
def remove_object_for_partial(points: List[PointEx]):
result: List[PointEx] = []
for current in points:
found = False
for stored in result:
if current.equal_with(stored, "y"):
found = True
if found is False:
result.append(current)
return result
show_points(remove_object_for_partial(duplicated_pointExes))
# (1, 1)
# (2, 2)
# (3, 3)
# (4, 4)
# (5, 5)When passing both x and y
def remove_object_for_partial(points: List[PointEx]):
result: List[PointEx] = []
for current in points:
found = False
for stored in result:
if current.equal_with(stored, "x", "y"):
found = True
if found is False:
result.append(current)
return result
# (1, 1)
# (2, 2)
# (3, 3)
# (4, 4)
# (6, 3) <-- difference
# (5, 5)When no arguments are passed, it will be only one item.
def remove_object_for_partial(points: List[PointEx]):
result: List[PointEx] = []
for current in points:
found = False
for stored in result:
if current.equal_with(stored):
found = True
if found is False:
result.append(current)
return result
# (1, 1)
Comments