Repository navigation
Using async iterator in copy_records_to_table #689
Description
- asyncpg version: 0.22.0
- PostgreSQL version: 13
- Do you use a PostgreSQL SaaS? If so, which? Can you reproduce
the issue with a local PostgreSQL install?: no - Python version: 3.8, 3.9
- Platform: Windows, Linux
- Do you use pgbouncer?: no
- Did you install asyncpg with pip?: git
- If you built asyncpg locally, which version of Cython did you use?: 0.29.21
- Can the issue be reproduced under both asyncio and
uvloop?: yes
I want to stream data to base with COPY, but i have async code, that produce data. I can pass regular iterator to records, but it ends then Queue empty. So reconnect needed.
queue = asyncio.Queue(2<<10)
loop.create_task(feed(queue))
async def get_records(queue):
while True:
r = await queue.get()
yield r
self.q.task_done()
async with pg.acquire() as connection:
await connection.copy_records_to_table(
'radarlog2',
records = get_records(queue)
)
Reacted by Rob Haswell
Activity
[-]Using async iterator in [/-][+]Using async iterator in copy_records_to_table[/+]robhaswell commentedon Mar 8, 2021
I too was surprised that this wasn't supported.
It seems that
copy_to_tablesupports an async iterator but it must generate the Postgres binary protocol? I couldn't find a protocol formatter anywhere even though I assume thatasyncpgmust contain one somewhere in order to support this function.If anyone has some guidance on how to use
copy_to_tablethen that would be a good alternative, thanks.Add support for asynchronous iterables to copy_records_to_table()
elprans commentedon Mar 8, 2021
#713 adds support for async iterables in
copy_records_to_table.Add support for asynchronous iterables to copy_records_to_table()
robhaswell commentedon Mar 9, 2021
Thanks!
Add support for asynchronous iterables to copy_records_to_table()
Add support for asynchronous iterables to copy_records_to_table()
Add support for asynchronous iterables to copy_records_to_table()
Add support for asynchronous iterables to copy_records_to_table()
Add support for asynchronous iterables to copy_records_to_table()
Add support for asynchronous iterables to copy_records_to_table() (#713)