Regression: "Out of host memory" on Nvidia ICD #196

Activity

1. matthiasdiener commented on Sep 21, 2022

   

   matthiasdiener

   on Sep 21, 2022

   Last edited by matthiasdiener

   Collaborator

   More actions

   I can reproduce it on bock (which has a Titan V), but interestingly, the tests run fine on koelsch with a K40.

   This code is helpful when debugging this:

   @pytest.fixture(autouse=True)
   def slow_down_tests():
       yield
       import time
       time.sleep(3)
2. matthiasdiener commented on Sep 21, 2022

   

   matthiasdiener

   on Sep 21, 2022

   Last edited by matthiasdiener

   Collaborator

   More actions

   It looks like Nvidia CL on bock does not like creating multiple CL contexts with SVM allocations. The following program quickly fills up the GPU's memory on Nvidia CL, but not with pocl-cuda or pocl-pthreads:

   import pyopencl as cl
   import time
   from pyopencl.tools import SVMAllocator, ImmediateAllocator
   
   
   while True:
       ctx = cl.create_some_context() # putting this before the loop is fine
       queue = cl.CommandQueue(ctx)
       allocator = SVMAllocator(queue.context, queue=queue)
       # allocator = ImmediateAllocator(queue)  # using normal allocations is fine
       x = allocator(2*1024*1024*1024)
       # x.release()
       x.enqueue_release()
       queue.finish()
       del queue
       del ctx
   
       # every iteration of this loop increases memory usage on the GPU by ~300 MByte (with Nvidia CL + SVM allocs)
       time.sleep(2)
3. inducer commented on Sep 22, 2022

   

   inducer

   on Sep 22, 2022

   OwnerAuthor

   More actions

   Thanks for digging into this! That's helpful as an observation. I'll look at a CL call trace to see if that looks OK, and if so... not sure that we can do much?
4. inducer commented on Sep 26, 2022

   

   inducer

   on Sep 26, 2022

   OwnerAuthor

   More actions

   I'm not sure I like this solution, but I don't see an alternative: Could you do a PR to turn off use of SVM on Nvidia CL, with a pointer to this issue?

   I've put together https://gitlab.tiker.net/inducer/arraycontext/-/merge_requests/6 to make sure we maintain at least some coverage of the SVM code paths.