test_shelve.py crashes on WSL?

To reproduce, clone the repo, and run ./configure --with-pydebug && make -j $(nproc).

I'm on WSL (Ubuntu 24.04.1 LTS).

I get:

0:04:10 load avg: 1.81 [362/493/1] test_shelve failed (1 error)
test test_shelve failed -- Traceback (most recent call last):
  File "/home/lhott/cpython/Lib/test/test_shelve.py", line 205, in test_custom_serializer_and_deserializer
    s["bytearray_data"] = bytearray_data
    ~^^^^^^^^^^^^^^^^^^
  File "/home/lhott/cpython/Lib/shelve.py", line 138, in __setitem__
    self.dict[key.encode(self.keyencoding)] = serialized_value
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: gdbm mappings have bytes or string indices only

~~I had a few optional modules that were not found when I compiled Python, but if that was a problem, shouldn't test_shelve be skipped?~~

~~I assume _dbm is the problem but I'm not familiar with dbm modules.~~

Linked PRs

gh-137829: Fix shelve tests for backend compatibility #137879

I had a few optional modules that were not found when I compiled Python, but if that was a problem, shouldn't test_shelve be skipped?
I assume `_dbm` is the problem but I'm not familiar with `dbm` modules.

I installed _dbm then I ran ./configure --with-pydebug and make -j"$(nproc)" and I i still have the test that fails (no more _dbm missing when I compile Python though).

I had a few optional modules that were not found when I compiled Python, but if that was a problem, shouldn't test_shelve be skipped?

It most likely fell back to the Python implementation.

I had a few optional modules that were not found when I compiled Python, but if that was a problem, shouldn't test_shelve be skipped?

It most likely fell back to the Python implementation.

Yeah, reading the error message, I think the problem is this:

cpython/Lib/test/test_shelve.py

Lines 174 to 176 in 4e08a9f

    
           if isinstance(obj, (bytes, bytearray, str)): 
        
               if protocol == 5: 
        
                   return obj

Changing it to:

                 if protocol == 5:
+                    if isinstance(obj, bytearray):
+                        return bytes(obj)
                     return obj
                 return type(obj).__name__

Makes the test case pass.

@JasonMendoza2008

@JasonMendoza2008

Could you please run the following diagnostic script to gather additional information?

import dbm
import os
import glob
import shelve
import sys

print("=== Python Environment ===")
print(f"Python version: {sys.version}")
print(f"Platform: {sys.platform}")

print("\n=== Shelve Default Backend ===")
with shelve.open("myshelf", "c") as shelf:
    shelf["test"] = "data"
backend = dbm.whichdb("myshelf")
print(f"Shelve DBM backend: {backend}")
for file in glob.glob("myshelf*"):
    os.remove(file)

print("\n=== Individual Backend Functionality ===")
for backend_name in ["sqlite3", "gnu", "ndbm", "dumb"]:
    try:
        backend_module = __import__(f"dbm.{backend_name}", fromlist=[""])
        test_db = backend_module.open(f"test_{backend_name}", "c")
        test_db["test"] = b"data"
        test_db.close()
        print(f"dbm.{backend_name}: Functional")
    except Exception as e:
        print(f"dbm.{backend_name}: {e}")

for backend_name in ["sqlite3", "gnu", "ndbm", "dumb"]:
    for file in glob.glob(f"test_{backend_name}*"):
        os.remove(file)

=== Python Environment ===
Python version: 3.15.0a0 (heads/fix-issue-137759:207cccd106, Aug 15 2025, 19:34:35) [GCC 13.3.0]
Platform: linux

=== Shelve Default Backend ===
Shelve DBM backend: dbm.gnu

=== Individual Backend Functionality ===
dbm.sqlite3: No module named '_sqlite3'
dbm.gnu: Functional
dbm.ndbm: Functional
dbm.dumb: Functional

PS: It says heads/fix-issue-137759 but I had the problem before changing anything (and the CI tests run fine in the corresponding PR opened).

@JasonMendoza2008

@JasonMendoza2008

Actually two test are failing;

======================================================================
ERROR: test_custom_incomplete_serializer_and_deserializer (test.test_shelve.TestCase.test_custom_incomplete_serializer_and_deserializer)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/arf/Desktop/source-codes/cpython/Lib/test/test_shelve.py", line 239, in test_custom_incomplete_serializer_and_deserializer
    s["foo"] = "bar"
    ~^^^^^^^
  File "/home/arf/Desktop/source-codes/cpython/Lib/shelve.py", line 138, in __setitem__
    self.dict[key.encode(self.keyencoding)] = serialized_value
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: gdbm mappings have bytes or string indices only

======================================================================
ERROR: test_custom_serializer_and_deserializer (test.test_shelve.TestCase.test_custom_serializer_and_deserializer) (proto=5)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/arf/Desktop/source-codes/cpython/Lib/test/test_shelve.py", line 205, in test_custom_serializer_and_deserializer
    s["bytearray_data"] = bytearray_data
    ~^^^^^^^^^^^^^^^^^^
  File "/home/arf/Desktop/source-codes/cpython/Lib/shelve.py", line 138, in __setitem__
    self.dict[key.encode(self.keyencoding)] = serialized_value
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: gdbm mappings have bytes or string indices only

----------------------------------------------------------------------
Ran 356 tests in 1.130s

FAILED (errors=2, skipped=2)
test test_shelve failed
0:00:01 load avg: 1.36 [1/1/1] test_shelve failed (2 errors)

== Tests result: FAILURE ==

1 test failed:
    test_shelve

Total duration: 1.2 sec
Total tests: run=356 skipped=2
Total test files: run=1/1 failed=1
Result: FAILURE

I've implemented a fix for the issue. Thanks for the bug report.

Actually, the shelf automatically selects the dbm backend. I think it would be more useful if we had a feature that would specify the dbm backend when the shelf opens.

I'm confused. The TypeError appears to be because of the key being a string and then we try to encode it somehow and then use it. However the fix seems to be in the serializer, which acts on the value. Am I missing something? (I'm on mobile so hard to review)

I'm confused. The TypeError appears to be because of the key being a string and then we try to encode it somehow and then use it. However the fix seems to be in the serializer, which acts on the value. Am I missing something? (I'm on mobile so hard to review)

In the test_custom_serializer_and_deserializer, we did not anticipate encountering a negative case. As you may have noticed, this error occurs only with dbm.gnu. This is because our original serializer was not designed to support dbm.gnu. We have since updated the serializer to address this issue.

In the test_custom_incomplete_serializer_and_deserializer, incorrect behavior and type errors are expected. We were already suppressing these errors for sqlite, and have now also suppressed them for dbm.gnu.

@furkanonder

Changing it to:

                 if protocol == 5:
+                    if isinstance(obj, bytearray):
+                        return bytes(obj)
                     return obj
                 return type(obj).__name__

Makes the test case pass.

@furkanonder
This was enough to solve the problem for me, so I think I only had one test case that was failing on my system (and not two like you mentioned? I might misunderstand something, I didn't dig too deep)? I don't know shelve enough to understand your second change in the commit of your PR but I assume it fixes the second test case that was failing?

Changing it to:
if protocol == 5:
               if isinstance(obj, bytearray):
                   return bytes(obj)
               return obj
           return type(obj).__name__
Makes the test case pass.
@furkanonder This was enough to solve the problem for me, so I think I only had one test case that was failing on my system (and not two like you mentioned? I might misunderstand something, I didn't dig too deep)? I don't know shelve enough to understand your second change in the commit of your PR but I assume it fixes the second test case that was failing?

Could you try running the following snippet?

import shelve
import dbm.gnu

db = dbm.gnu.open("myshelf", "c")
db.close()


def serializer(obj, protocol=None):
    pass


def deserializer(data):
    return data.decode("utf-8")


with shelve.open("myshelf", serializer=serializer, deserializer=deserializer) as s:
    s["foo"] = "bar"

This reproduces the second case for me — it fails on my machine with the following traceback:

Traceback (most recent call last):
  File "/home/arf/Desktop/source-codes/cpython/repro.py", line 17, in <module>
    s["foo"] = "bar"
    ~^^^^^^^
  File "/home/arf/Desktop/source-codes/cpython/Lib/shelve.py", line 138, in __setitem__
    self.dict[key.encode(self.keyencoding)] = serialized_value
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: gdbm mappings have bytes or string indices only

So, I wanted to fix this test case too.

@furkanonder

@furkanonder I get a TypeError as well indeed.

I think I was confused because:
I got this:

lhott@JasonTower:~/cpython$ ./python Lib/test/test_shelve.py
....ss.s............................................................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 356 tests in 0.662s

With this change only:

lhott@JasonTower:~/cpython$ git diff
diff --git a/Lib/test/test_shelve.py b/Lib/test/test_shelve.py
index 64609ab9dd..9ca859f1da 100644
--- a/Lib/test/test_shelve.py
+++ b/Lib/test/test_shelve.py
@@ -173,6 +173,8 @@ def test_custom_serializer_and_deserializer(self):
         def serializer(obj, protocol):
             if isinstance(obj, (bytes, bytearray, str)):
                 if protocol == 5:
+                    if isinstance(obj, bytearray):
+                        return bytes(obj)
                     return obj
                 return type(obj).__name__
             elif isinstance(obj, array.array):

But I hadn't noticed that test_custom_incomplete_serializer_and_deserializer was skipped:
skipped "No module named '_sqlite3'"

Which is why I had always mentioned "the one" test case that was failing.

EDIT: reading through your commit, I guess you removed:

dbm_sqlite3 = import_helper.import_module("dbm.sqlite3")

so that the test case doesn't get skipped anymore? and allows tests for an incomplete serializer/deserialized no matter the backend selected.

Oh I actually now understand! the error is actually the same for both the keys and the values and the traceback actually only shows the [...] part without including the = part, and thus I assumed that the error occurred due to the key not the value. Ok, I now understand the issue.

gh-137829: Fix shelve tests for backend compatibility (#137879)

pythongh-137829: Fix shelve tests for backend compatibility (python#1…

Navigation Menu

Repository navigation

test_shelve.py crashes on WSL? #137829

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	if isinstance(obj, (bytes, bytearray, str)):
	if protocol == 5:
	return obj

test_shelve.py crashes on WSL? #137829

Description

Linked PRs

Activity

JasonMendoza2008 commented on Aug 15, 2025

StanFromIreland commented on Aug 15, 2025

JasonMendoza2008 commented on Aug 15, 2025

JasonMendoza2008 commented on Aug 15, 2025

furkanonder commented on Aug 16, 2025

JasonMendoza2008 commented on Aug 16, 2025

furkanonder commented on Aug 17, 2025

furkanonder commented on Aug 17, 2025

picnixz commented on Aug 17, 2025

furkanonder commented on Aug 17, 2025

JasonMendoza2008 commented on Aug 17, 2025

furkanonder commented on Aug 17, 2025

JasonMendoza2008 commented on Aug 18, 2025

picnixz commented on Aug 18, 2025

picnixz commented on Aug 18, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions