Repository navigation
Unclear error on passing connection with autocommit=True on creation #21
Description
Hi,
what I do:
- create connection with
autocommit=Trueand pass it to LiteQueue constructor
what I get:
File ".../lib/python3.12/site-packages/litequeue.py", line 235, in __init__
self.conn.execute("PRAGMA synchronous = NORMAL;")
sqlite3.OperationalError: Safety level may not be changed inside a transaction
what I expected to get:
error that connection has enabled autocommit field and it is unsupported by library
Thank you.
litequeue version 0.9
Activity
polyrand commentedon Jul 20, 2026
Thanks for reporting! I was actually working on removing the option to pass a custom sqlite3 connection object, and instead forcing the caller to use the internal connection settings done by
LiteQueue.Using
autocommit=Truemay interfere with the assumptions fromLiteQueue. I'm also trying to make it more thread-safe, and one of the things I was doing is splitting a singe "write" connection from read-only connections, to avoid other threads from reading uncommitted data started in a transaction by another thread.This would be incompatible with passing a custom
sqlite3.Connectionobject.I'm also removing the option to host more than one queue in a single DB file, and assume that
LiteQueuewould be used with isolated SQLite database files, one file per queue (instead of multiple queues in the same file). Mostly so that the code can be simplified, and I can work on polishing the code and making it more robust.I haven't released any new version yet, but I'm interested in knowing your use case to make sure it stays useful.
abcdenis commentedon Jul 20, 2026
Thank you!
I've noticed new "one queue per file" rule and split queues to dedicated files in my pet project today :)
I don't have much feedback because I use litequeue for a week.
polyrand commentedon Jul 20, 2026
Good, if one-queue per file works for you, I'll continue with the current changes. I'll probably release a
0.11version in a few days. I hadn't updated this project in a while, but now I'm reviewing some parts and noticed that it was a bit brittle in some ways.It supported multiple queues per file, but I wasn't pleased with the implementation and even had some bugs. I'm now making it just "a persistent queue backed by SQLite", and simplifying everything around that, even if I have to let some features go, or disallow certain behaviours.
abcdenis commentedon Jul 21, 2026
I plan to use litequeue as persistent queue for pet project: one process put data in queue and another process read data from it.
Will it be possible in future version? There were some announce like "only one writing thread and many reading threads" and it confuses me.
polyrand commentedon Jul 21, 2026
Yes, that use case will be possible, and it is one of LiteQueue’s intended uses.
Each process should create its own
LiteQueueinstance using the same queue name and folder, so both instances open the same SQLite queue file:Please don’t create and pass a custom
sqlite3.Connection. Let LiteQueue create and manage its own connections.The “one writer connection” wording describes LiteQueue’s internal connection management; it does not mean that only one application process can produce or consume messages. Each process has its own connections, while SQLite coordinates access to the shared database file and serializes writes when necessary. Claiming a message with
pop()is performed in a transaction, so two consumers should not receive the same message.The connection changes are primarily about thread safety and transaction isolation. If multiple threads share one SQLite connection, a read performed by one thread can become part of another thread’s active transaction. It may then see uncommitted changes that could later be rolled back. Similarly, another thread’s write could accidentally become part of that transaction and be committed or rolled back with it.
By managing the database file itself, LiteQueue can use a dedicated write connection and separate query-only read connections. The thread that owns a transaction can see its own uncommitted changes, while other threads use separate connections and continue seeing only committed data. This makes transaction boundaries much clearer and safer.
So your producer-process/consumer-process setup is supported: point both
LiteQueueinstances at the same queue file and let LiteQueue manage the SQLite connections.The tl;dr is that this is all for better thread safety.
Also notice that all this information is for an unreleased version of LiteQueue, which has breaking changes: mainly just allowing a single queue per database. (I'll release a new version tomorrow probably).
polyrand commentedon Jul 21, 2026
I just released version 0.11, which has all the recent changes and supports a single queue per file only. See the README for usage examples. Also I added an examples folder.
abcdenis commentedon Jul 21, 2026
So one queue will occupy not a file but entire folder?
I'll rather prefer specify filename (and queue name) and let litequeue manage it. Now it looks like "litequeue could change contract in the future and store something more in this folder".
abcdenis commentedon Jul 21, 2026
I forgot to mention uuid7 added in python 3.14!
https://docs.python.org/3/library/uuid.html#uuid.uuid7
polyrand commentedon Jul 21, 2026
I know, but I want to keep this backwards compatible up to Python 3.12, the
uuidv7function is actually copied from https://github.com/python/cpython/blob/f429fb36a16134a2a0e839d0f385eb8fee280de4/Lib/uuid.pypolyrand commentedon Jul 21, 2026
No, it just lets you specific where to create the SQLite file, and the sqlite file is based on the queue name.
I think you're right, I may be over-complicating the API. I was over-focusing on
queue_name. Now only the filename should be enough, since now each file can only have a single queue anyway, so file = queue. Just specifying the sqlite file location should be enough. I'll update this.polyrand commentedon Jul 21, 2026
Versio 0.12 now uses only a filename as input, the filename is the queue name, since a sqlite file can now only contain a single queue. The README, examples and tests are also updated.
abcdenis commentedon Jul 21, 2026
Thank you!