How to Merge Two Dictionaries Python

How to Merge Two Dictionaries in Python: A Senior Engineer’s Guide

To merge two Python dictionaries, the most Pythonic and efficient methods involve the dictionary unpacking operator (**, Python 3.5+) or the dictionary union operator (|, Python 3.9+). Both create a new dictionary, with values from the second dictionary overwriting those from the first for any shared keys. For in-place modification, use the dict.update() method.

The Senior Dev Hook: My Production Pitfall with Dictionary Merging

In my early days, before Python 3.5 introduced the elegant unpacking operator for dictionaries, I ran into a subtle but critical issue with dictionary merging in a backend service responsible for configuration aggregation. I needed to merge default settings with user-specific overrides. My initial approach, using dict.update(), seemed straightforward:

default_config = {"timeout": 30, "retries": 3, "log_level": "INFO"}
user_config = {"timeout": 60, "log_level": "DEBUG"}

# This modifies default_config in-place!
default_config.update(user_config)
# Merged config is now in default_config

The problem? default_config was a shared object, passed around to various functions. By mutating it directly, I inadvertently changed the “default” for other parts of the application that expected an immutable base. This led to intermittent, hard-to-debug issues where a function would suddenly pick up a user’s specific settings instead of the true defaults. The fix was to explicitly create a copy before updating, ensuring immutability for the original. This experience drilled into me the importance of understanding the mutability implications of each merging strategy.

Under the Hood: How Python Dictionaries Merge

A Python dictionary is fundamentally a hash table, where keys are hashed to quickly find their corresponding values. When you merge two dictionaries, Python needs to iterate through the items of one or both dictionaries and insert them into a new or existing dictionary structure. Here’s a breakdown of the common mechanisms:

1. Key Hashing and Collision Resolution: For each key-value pair being added, Python computes the hash of the key. If the hash bucket is empty, the item is inserted. If a collision occurs (another key hashes to the same bucket), Python uses an internal probe sequence to find the next available slot or a matching key.
2. Duplicate Key Handling: When merging, if a key from the second dictionary already exists in the first (or target) dictionary, the value associated with that key in the second dictionary will always overwrite the existing value. This is a crucial detail for data integrity.
3. Memory Allocation: When creating a new dictionary (e.g., using ** or |), Python allocates memory for the new hash table. The size of this allocation is typically proportional to the total number of unique items across both source dictionaries. For in-place updates (update()), if the dictionary needs to grow, it will reallocate memory and potentially rehash its existing contents, though Python’s dict implementation is highly optimized for this.
4. Optimized C Implementations: The underlying C implementation of Python’s dictionary operations is highly optimized for speed. Methods like update() and the union operator | are often implemented with direct C loops and memory operations, making them very efficient. The unpacking operator ** also leverages these internal optimizations.

Step-by-Step Implementation: Modern & Legacy Approaches

I advocate for modern Pythonic approaches for their clarity and efficiency. Let’s look at the primary methods.

Method 1: Using the Dictionary Unpacking Operator (**) – Python 3.5+

This is my go-to for creating a new merged dictionary. It’s concise and explicitly creates a new object, avoiding mutation side effects.

# config_manager.py
dict1 = {"name": "Alice", "age": 30, "city": "New York"}
dict2 = {"age": 31, "occupation": "Engineer", "city": "San Francisco"}

# Merging using ** (unpacking operator)
# Items from dict2 overwrite items from dict1 if keys overlap
merged_dict_unpack = {**dict1, **dict2} 

print("Original dict1:", dict1)
print("Original dict2:", dict2)
print("Merged dict (unpacking):", merged_dict_unpack)
# Expected output: {'name': 'Alice', 'age': 31, 'city': 'San Francisco', 'occupation': 'Engineer'}

# Demonstrating order of precedence
dict_a = {'x': 1, 'y': 2}
dict_b = {'y': 3, 'z': 4}
merged_dict_precedence = {**dict_a, **dict_b}
print("Merged dict (precedence A then B):", merged_dict_precedence) # y:3 wins

merged_dict_precedence_reverse = {**dict_b, **dict_a}
print("Merged dict (precedence B then A):", merged_dict_precedence_reverse) # y:2 wins

Explanation: The ** operator unpacks key-value pairs from a dictionary into the new dictionary literal. When multiple dictionaries are unpacked, items from later dictionaries in the sequence will override those from earlier ones if their keys clash. This explicitly creates a new dictionary, leaving the originals untouched.

Method 2: Using the Dictionary Union Operator (|) – Python 3.9+

This is the newest and arguably most semantic way to merge dictionaries, especially if you’re working with Python 3.9 or newer. It explicitly means “combine these.”

# data_aggregator.py
dict_base = {"id": 101, "status": "active", "timestamp": "2023-01-01"}
dict_updates = {"status": "inactive", "reason": "expired"}

# Merging using | (union operator)
# Items from dict_updates overwrite items from dict_base
merged_dict_union = dict_base | dict_updates

print("Original dict_base:", dict_base)
print("Original dict_updates:", dict_updates)
print("Merged dict (union operator):", merged_dict_union)
# Expected output: {'id': 101, 'status': 'inactive', 'timestamp': '2023-01-01', 'reason': 'expired'}

# In-place union (|=)
config = {"theme": "dark", "font_size": 14}
new_settings = {"font_size": 16, "language": "en"}
config |= new_settings # This modifies 'config' in-place
print("Config after in-place union:", config)
# Expected output: {'theme': 'dark', 'font_size': 16, 'language': 'en'}

Explanation: The | operator provides a clear, mathematical-set-like syntax for dictionary union. Like **, it returns a new dictionary. The |= operator is its in-place counterpart, directly modifying the left-hand dictionary. I prefer | for its explicit creation of a new object and better readability.

Method 3: Using dict.update() (In-place) – Python 2.x, 3.x

This method directly modifies an existing dictionary. Use it when you intentionally want to alter one of the original dictionaries.

# settings_overrider.py
base_settings = {"max_conn": 10, "timeout": 5, "debug_mode": False}
override_settings = {"timeout": 10, "debug_mode": True, "log_path": "/var/log"}

# Using update() - base_settings is modified
base_settings.update(override_settings)

print("Original override_settings (unchanged):", override_settings)
print("Base settings after update (modified):", base_settings)
# Expected output: {'max_conn': 10, 'timeout': 10, 'debug_mode': True, 'log_path': '/var/log'}

# If you need a new dictionary, copy first
original_defaults = {"user": "guest", "perm": "read"}
user_prefs = {"user": "admin", "theme": "light"}
# Create a copy, then update the copy
new_merged_dict_safe = original_defaults.copy()
new_merged_dict_safe.update(user_prefs)
print("Original defaults (still intact):", original_defaults)
print("New merged dict (safe update):", new_merged_dict_safe)

Explanation: The dict.update() method takes another dictionary (or an iterable of key-value pairs) and adds its contents to the calling dictionary. If keys overlap, the values from the argument dictionary overwrite those in the calling dictionary. Crucially, this operation is performed *in-place*. If you need a new dictionary, you must explicitly call .copy() on one of the dictionaries first, then .update() the copy.

What Can Go Wrong: Troubleshooting Edge Cases

1. Unexpected Mutation (The “Senior Dev Hook” Scenario):

   Problem: You use dict.update() expecting a new dictionary, but one of your original dictionaries gets modified, leading to subtle state bugs elsewhere in your application.

   global_defaults = {"theme": "dark", "loglevel": "INFO"}
   user_preferences = {"loglevel": "DEBUG"}
   
   # Mistake: directly updating a potentially shared global object
   global_defaults.update(user_preferences) 
   # Now global_defaults is {"theme": "dark", "loglevel": "DEBUG"} everywhere!
   

   Solution: Always explicitly create a copy if you intend to merge and produce a new dictionary without affecting the originals. Use {**dict1, **dict2} or dict1 | dict2 for new objects, or dict1.copy().update(dict2) for older Python versions.

2. Python Version Incompatibility:

   Problem: You try to use {**dict1, **dict2} on Python 3.4 or earlier, or dict1 | dict2 on Python 3.8 or earlier.

   # On Python < 3.5:
   d1 = {'a': 1}
   d2 = {'b': 2}
   # merged = {**d1, **d2} # SyntaxError: invalid syntax
   
   # On Python < 3.9:
   d1 = {'a': 1}
   d2 = {'b': 2}
   # merged = d1 | d2 # TypeError: unsupported operand type(s) for |: 'dict' and 'dict'
   

   Solution: Be mindful of your target Python version. For older versions, dict.update() (with a preceding .copy() if a new dict is needed) is the standard method.

3. Non-Hashable Keys:

   Problem: While not specific to *merging*, attempting to create or merge dictionaries where keys are mutable objects (e.g., lists, other dictionaries) will raise a TypeError: unhashable type: 'list'.

   # This would fail even before merging if d1 tried to define it
   d1 = {['key_list']: 1} # TypeError: unhashable type: 'list'
   

   Solution: Ensure all dictionary keys are immutable (strings, numbers, tuples, frozensets, etc.).

4. Unexpected Overwrites with Shared Keys:

   Problem: You expect a more complex merge strategy (e.g., merging nested dictionaries recursively, or appending values for shared keys), but standard merging methods simply overwrite.

   d1 = {'data': {'count': 1}}
   d2 = {'data': {'status': 'ok'}}
   merged = d1 | d2
   print(merged) # {'data': {'status': 'ok'}} -- d1['data'] was completely replaced, not recursively merged.
   

   Solution: For complex, recursive merges, you’ll need a custom function. Libraries like deepmerge or pydash can handle this, but it’s outside the scope of simple dictionary merging.

Performance & Best Practices

When selecting a dictionary merging strategy, consider performance, readability, and mutability.

When NOT to use a particular approach:

• Avoid dict.update() for creating a new dictionary: If your goal is a new dictionary and you want to preserve the originals, do not use dict.update() directly. While dict1.copy().update(dict2) works, it’s less concise and often marginally slower than {**dict1, **dict2} or dict1 | dict2 due to the two-step operation (copy then update).
• Avoid ** or | if you *must* modify in-place: If the requirement is explicitly to modify an existing dictionary, dict.update() or |= are the correct choices.
• Avoid ** or | on incompatible Python versions: As discussed, these are syntax or runtime errors on older Python versions.
• Avoid manual loops for simple merges: While possible to loop through one dictionary and add items to another, it’s less efficient and less readable than the built-in methods, which are implemented in C.

Alternative Methods (Legacy vs. Modern):

• Legacy (Pre-Python 3.5, for new dicts):

  new_dict = dict1.copy()
  new_dict.update(dict2)
  

  This explicitly creates a copy of dict1, then updates that copy with items from dict2. It’s safe for preserving dict1 but verbose.

• Using collections.ChainMap (Specific Use Case):

  The collections.ChainMap is not a merge in the traditional sense, but rather a way to group multiple dictionaries or other mappings together to create a single, updateable view. Lookups search the underlying mappings in order. Writes/deletes only affect the first dictionary in the chain. It’s excellent for chaining configurations without copying all data:

  from collections import ChainMap
  
  default_config = {"debug": False, "level": "INFO"}
  user_config = {"level": "DEBUG", "theme": "dark"}
  
  chained_map = ChainMap(user_config, default_config)
  
  print(chained_map["level"])  # Output: DEBUG (from user_config)
  print(chained_map["debug"])  # Output: False (from default_config)
  print(chained_map.maps)      # View of the underlying dictionaries
  
  # If you need a single merged dictionary from ChainMap:
  final_merged_dict = dict(chained_map)
  print(final_merged_dict)
  

  Information Gain: ChainMap offers O(1) space complexity for *creating* the chain, as it only stores references to the original dictionaries, not copies. Actual key lookups are O(k) where k is the number of dictionaries in the chain. Converting to a `dict(chained_map)` will then incur O(N+M) space and time complexity.

General Best Practices:

1. Prioritize Immutability: For clearer code and fewer side effects, lean towards methods that create new dictionaries (** or |) unless you have a specific, justifiable need for in-place modification.
2. Know Your Python Version: Always consider the Python version your project targets. This dictates which modern syntaxes you can safely use.
3. Readability: {**dict1, **dict2} and dict1 | dict2 are generally more readable and intent-revealing than dict1.copy().update(dict2).
4. Performance (General): For typical dictionary sizes, the performance differences between **, |, and .copy().update() are often negligible. For extremely large dictionaries (hundreds of thousands of elements), minor differences might emerge, but Python’s C-level implementations are highly optimized for all these scenarios. The | operator introduced in 3.9 might have marginal performance benefits due to being a dedicated operator for this task, potentially allowing for more direct C-level optimization.

For more on this, Check out more Python Basics Tutorials.

Author’s Final Verdict

As a senior backend engineer, I prioritize code clarity, maintainability, and predictability. For merging dictionaries, I strongly recommend adopting the dictionary union operator (|) if your project uses Python 3.9 or newer. Its syntax is clean, intuitive, and it explicitly creates a new dictionary, which is often the desired behavior to avoid unexpected side effects. If you’re on Python 3.5-3.8, the dictionary unpacking operator (**) is your best choice for the same reasons. Reserve dict.update() for scenarios where you genuinely intend to modify an existing dictionary in-place. Understanding the mutability implications of each method is paramount to writing robust and bug-free Python applications.