Request for Async Support and Dependency Update #82

Activity

1. andrefurlan-db commented on Jan 17, 2023

   

   andrefurlan-db

   on Jan 17, 2023

   Contributor

   More actions

   Hi @devmessias, tudo bem?

   Just to clarify a bit, what exactly do you mean by async? Separate thread for parallelism/concurrency, or the ability to save operation handle and come back to it later in a potentially different process?
2. susodapop commented on Jan 18, 2023

   

   susodapop

   on Jan 18, 2023

   Contributor

   More actions

   Ditto Andre's questions.

   Separately:

   We have noticed that one dependency of the sqlalchemy (https://github.com/crflynn/sqlalchemy-databricks) dialect it is pyhive. The problem here is that pyhive is not being maintained anymore by dropbox. This raises security concerns. There are any additional considerations about changing that?

   sqlalchemy-databricks is a community project. We are building our own SQLAlchemy dialect which does not rely on Pyhive and will be available soon (there is an open pull request that implements it). We don't officially support the community project and once ours is available you will be able to connect with SQLAlchemy without installing any separate packages.

   sqlalchemy-databricks but it currently lacks support for async operations

   We don't have plans to implement the SQLAlchemy asynchronous API at this time but are open to considering this in the future. SQLAlchemy considers asyncio support to be a beta feature so I don't think it makes sense to prioritise it at this time.

   which is causing performance issues for us.

   I'm very curious to know what performance issues you're seeing. If you're interested we can share a dev build of databricks-sql-connector with you that includes the first-party SQLAlchemy connector -- this very well may solve the performance issues even without async.

   I would appreciate if you could let me know if there is anything I can do to help with these requests.

   The best thing we can get is feedback on our new connector :) #57
3. susodapop commented on Feb 22, 2023

   

   susodapop

   on Feb 22, 2023

   Contributor

   More actions

   SQLAlchemy support is now built-in to databricks-sql-connector as of v2.4.0.
4. susodapop

   closed this as completedon Feb 22, 2023