Skip to content

fix: sort sparse coordinates when building a SparseVector from SciPy - #160

Closed
chrikrah wants to merge 1 commit into
pgvector:masterfrom
chrikrah:chrikrah/sparsevec-sort-coords
Closed

chrikrah wants to merge 1 commit into
pgvector:masterfrom
chrikrah:chrikrah/sparsevec-sort-coords

Conversation

@chrikrah

@chrikrah chrikrah commented Oct 3, 2026

Copy link
Copy Markdown

_from_sparse now sorts the index/value pairs before assigning them, the way _from_dict at pgvector/sparsevec.py:97 already does. SciPy promises no coordinate order for a coo_array, and the constructor accepts any legal one, so unordered triplets reach the wire descending and the binary codec refuses them. The only signal a user gets is a Postgres DataException at insert time, a long way from the constructor that caused it. asyncpg has no text path, and psycopg's %s resolves to the binary dumper.

Before, against pgvector 0.8.7 on PostgreSQL 17:

$ python -c "
from scipy.sparse import coo_array
from pgvector import SparseVector
import psycopg; from pgvector.psycopg import register_vector
conn = psycopg.connect('postgresql://postgres:pw@127.0.0.1:54329/pgv', autocommit=True)
conn.execute('CREATE EXTENSION IF NOT EXISTS vector'); register_vector(conn)
v = SparseVector(coo_array(([3.0, 1.0, 2.0], ([4, 0, 2],)), shape=(6,)))
print('indices:', v.indices())
print(conn.execute('SELECT %s::sparsevec', (v,)).fetchone()[0])
"
indices: [4, 0, 2]
psycopg.errors.DataException: sparsevec indices must be in ascending order

After, same command against the same container:

indices: [0, 2, 4]
SparseVector({0: 1.0, 2: 2.0, 4: 3.0}, 6)

No issue in the tracker covers it, so there is no closing keyword.

Verification

$ pytest tests/test_bit.py tests/test_half_vector.py tests/test_sparse_vector.py tests/test_vector.py tests/test_psycopg.py tests/test_asyncpg.py -q
96 passed in 1.01s
$ pycodestyle . --ignore=E501,E704,W504 --exclude=.venv   # pycodestyle 2.15.0
$ mypy pgvector                                           # mypy 2.4.0
Success: no issues found in 40 source files
# not run: the Django, SQLAlchemy, SQLModel, Peewee, pg8000 and psycopg2 suites

Baseline on 99a6776 over the same six files is 95 passed, so the delta is test_coo_array_unordered. Keeping that test and restoring the old _from_sparse gives 1 failed, 95 passed, on assert [4, 0, 2] == [0, 2, 4].

@ankane, 2cff2f8 went the other way on allocation here, so I held this to the two lines _from_dict uses. Duplicate coordinates are the other half of a non-canonical COO and this patch leaves them alone: say the word and I will swap both for sum_duplicates() on a copy, guarded by has_canonical_format, which costs an ordered array nothing.

`tocoo()` returns coordinates in whatever order the source array holds them,
so a coo_array built from unordered triplets produced a SparseVector whose
indices descend. `sparsevec_recv` rejects that with "sparsevec indices must
be in ascending order", which is the error an asyncpg or psycopg user sees on
insert, and `_from_dict` already sorts for the same reason.
@ankane ankane closed this in fd74f78 Oct 3, 2026
@ankane

ankane commented Oct 3, 2026

Copy link
Copy Markdown
Member

Thanks @chrikrah, merged a version of this in the commit above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants