Repository navigation
ndarray data ownership question #28
Description
Hi,
Lets say I have a
boost::shared_ptrstd::vector vp;
What is the best way to construct a numpy array and return it?
I am currently using:
std::vector &vec(*vp);
np::from_data(vec.data(), np::dtype::get_builtin() , p::make_tuple(vec.size()), p::make_tuple(sizeof(int)), owner);
The question is: what should I put for owner? I'd like to be able to call vp.reset() in my C++ code [and therefore giving up ownership] and have python manage the memory as long as it needs to (and clean up if the python object is ever del'ed).
I am assuming having a p::object() on the stack, but I am guessing that it is probably not a good thing (as it will destruct when my function ends). So, I can create a "dummy" owner and have it linger around in my c++ code, but that'll keep objects around longer than they are needed (essentially leaking memory, w/o technically doing so).
Thanks in advance.
Activity
TallJimbo commentedon May 28, 2013
What you want to do here is create an opaque Python object that does nothing but hold your
shared_ptr, and destroy it when the Python object goes out of scope. Happily, you can do that with a PythonCObject(orCapsuleobject, I believe, in Python 3), so you don't have to go to all of the work of creating a Boost.Python class just to create an owner. The ndarray project, which is a higher-level Python/C++ array conversion library that uses Boost.NumPy to access the NumPy C-API, does something very much like this here:https://github.com/ndarray/ndarray/blob/master/include/ndarray/bp/Array.h#L33
It uses an
ndarray::Manager::Ptr(a typedef toboost::intrusive_ptr) instead of ashared_ptr, but the idea is the same. The most important part is the call toPyCObject_FromVoidPtr; the rest is mostly just an attempt to avoid cycles ofshared_ptrs that holdCObjects that holdshared_ptrs.NavreetGill commentedon May 28, 2013
OK, so essentially, the destroyManager function will get called when the objected is del'ed in python (or I guess when the GC decides to call it).
If I look closely, it seems like I will have to have different destroyManager functions depending on the type. I will probably use templates here.
I wish I had looked at your ndarray/Vector class earlier. It would've helped out.
I still think that the use case I mentioned might be good to add in the library (converting a
shared_array<T>orshared_ptr<vector<T> >to a ndarray and transferring ownership).TallJimbo commentedon May 30, 2013
Yeah, I agree that it'd be worthwhile to add this functionality to Boost.NumPy. I'll keep this issue open to remind me to do that.
7starsea commentedon Jan 11, 2018
My typical use case is:
namespace np = boost::python::numpy;
namespace bp = boost::python;
np::dtype dtype = np::dtype::get_builtin();
///creating np_shape, np_stride
data = new double[200];
arr = np::from_data( data_, dtype, np_shape, np_stride, bp::object() );_
and then the variable arr is returned to python and will be deleted in python, and I am experiencing a memory leak for the data_. How could I avoid the memory leak? @TallJimbo Thanks so much.
7starsea commentedon Jan 12, 2018
Never mind, I solved the problem. For anyone who are experiencing the same memory-leaking, here is my solution.
We need to add a deconstructor:
boost::python::handle<> h(::PyCapsule_New((void *)data, NULL, (PyCapsule_Destructor)&destroyManagerCObject));
arr = np::from_data( data_, dtype, np_shape, np_stride, bp::object(h) );_
where
inline void destroyManagerCObject(PyObject* obj) {
double * b = reinterpret_cast<double*>( PyCapsule_GetPointer(obj, NULL) );
if(b){ delete [] b; }
}
Hope it helps.
PhilippeCarphin commentedon Aug 8, 2019
It sure does help! This is an application of what 7starsea said to do boostorg/python#97 (comment)