Skip to content
Navigation Menu
Sign in
Products
Platform
GitHub
Repository hosting, code review, and collaboration
Copilot
AI coding agent for tasks, projects and automated work
Build & Ship
Actions
CI/CD and automation
Code Quality
Quality analysis, fixes, and enforcement
Codespaces
Cloud development environments
Secure
Code Security
Vulnerability detection and remediation
Secret Protection
Secret scanning and leak prevention
All Features
Developers
Learn
Documentation
Training
Certifications
Build
Build with Copilot
Customize Copilot
Integrations
Explore
Trending repos
GitHub Sponsors
Developer articles
Changelog
Changelog
Your source for the latest features, improvements, and fixes across GitHub products, all in one place.
Businesses
By Stage
Enterprises
Team
Startups
Nonprofits
By Industry
Healthcare
Financial services
Manufacturing
Government
See all industries
Evaluate GitHub
Why GitHub
Customer Stories
Business insights
Trust Center
Add-ons
GitHub Advanced Security
Enterprise-grade code security
Copilot for business
Enterprise-grade AI features
Premium Support
Enterprise-grade 24/7 support
Resources
Learn & Connect
Documentation
Resource center
Blog
Events & webinars
Community Forum
Customer Stories
Integrations
Support
Customer support
Training
GitHub Status
Company
About
Partners
Changelog
GitHub Universe
GitHub Universe
The event for developers, by developers — talks, workshops, and what’s next for GitHub.
Pricing
Search
/
Sign in
Sign up
Appearance settings
You signed in with another tab or window.
Reload
to refresh your session.
You signed out in another tab or window.
Reload
to refresh your session.
You switched accounts on another tab or window.
Reload
to refresh your session.
Dismiss alert
{{ message }}
apache
iceberg-python
Repository navigation
Code
Issues
173
(173)
Pull requests
145
(145)
Actions
Projects
Security and quality
Insights
More
items
Improve upsert memory pressure
- #1995
#1995
Merged
Fokko
merged 20 commits into
apache:main
apache/iceberg-python:main
from
koenvo:feat/use-batchreader-in-upsert
koenvo/iceberg-python:feat/use-batchreader-in-upsert
Copy head branch name to clipboard
Jun 20, 2025
Conversation
Commits
20
(20)
Checks
Files changed
Merged
Improve upsert memory pressure
#1995
Fokko
merged 20 commits into
apache:main
apache/iceberg-python:main
from
koenvo:feat/use-batchreader-in-upsert
koenvo/iceberg-python:feat/use-batchreader-in-upsert
Copy head branch name to clipboard
Commits
Commits on May 13, 2025
Move actual implementation of upsert from Table to Transaction
koenvo
committed
7abfee9
View commit details
Copy full SHA for 7abfee9
Browse repository at this point
Fix some incorrect usage of schema
koenvo
committed
db334ae
View commit details
Copy full SHA for db334ae
Browse repository at this point
Write a test for upsert transaction
koenvo
committed
cebfda3
View commit details
Copy full SHA for cebfda3
Browse repository at this point
Add failing test for multiple upserts in same transaction
koenvo
committed
52fd35e
View commit details
Copy full SHA for 52fd35e
Browse repository at this point
Fix test
koenvo
committed
f336c0b
View commit details
Copy full SHA for f336c0b
Browse repository at this point
Add failing test
koenvo
committed
07890ac
View commit details
Copy full SHA for 07890ac
Browse repository at this point
Use Transaction.table_metadata when doing the data scan in upsert
koenvo
committed
ae0e60f
View commit details
Copy full SHA for ae0e60f
Browse repository at this point
Remove as it's resolved
koenvo
committed
ce8d9ef
View commit details
Copy full SHA for ce8d9ef
Browse repository at this point
Use to_arrow_batch_reader instead of to_arrow in upsert to prevent OOM when updating large tables
koenvo
committed
5bdb0b8
View commit details
Copy full SHA for 5bdb0b8
Browse repository at this point
Filter rows to insert on each iteration instead of keeping a list of all filter expressions. Prevents memory pressure due to large filters
koenvo
committed
65fe36d
View commit details
Copy full SHA for 65fe36d
Browse repository at this point
Merge branch 'main' into feat/use-batchreader-in-upsert
koenvo
committed
4614543
View commit details
Copy full SHA for 4614543
Browse repository at this point
Commits on Jun 2, 2025
Accept concurrent_tasks when fetching record_batches
koenvo
committed
88a4ad2
View commit details
Copy full SHA for 88a4ad2
Browse repository at this point
Commits on Jun 3, 2025
Use ExecutorFactory
koenvo
committed
f8acdb0
View commit details
Copy full SHA for f8acdb0
Browse repository at this point
Simplify to_arrow to use the optimized to_record_batches
koenvo
committed
119d92f
View commit details
Copy full SHA for 119d92f
Browse repository at this point
Fix for shutdown pool after doing a map
koenvo
committed
5d3a6aa
View commit details
Copy full SHA for 5d3a6aa
Browse repository at this point
minor
koenvo
committed
445845d
View commit details
Copy full SHA for 445845d
Browse repository at this point
Improve comments in to_record_batches
koenvo
committed
6b8dace
View commit details
Copy full SHA for 6b8dace
Browse repository at this point
Commits on Jun 14, 2025
Make sure the 'infer the types when reading (#1669)' works again
koenvo
committed
2cd2137
View commit details
Copy full SHA for 2cd2137
Browse repository at this point
Commits on Jun 20, 2025
Update pyiceberg/io/pyarrow.py
Show description for 8e6f5e9
koenvo
and
Fokko
authored
8e6f5e9
View commit details
Copy full SHA for 8e6f5e9
Browse repository at this point
Update pyiceberg/io/pyarrow.py
Show description for 6813061
koenvo
and
Fokko
authored
6813061
View commit details
Copy full SHA for 6813061
Browse repository at this point
You can’t perform that action at this time.
