Collect profiles from other threads. #16

Activity

1. kalyanac

   added

   enhancementNew feature or requestNew feature or request

   on Sep 26, 2019
2. ar-qun commented on Oct 31, 2023

   

   ar-qun

   on Oct 31, 2023

   More actions

   This would be good feature to have. Recently we had worked on a Flask app and were confused as to why none of our code showed up.
3. lunalucadou commented on Dec 6, 2023

   

   lunalucadou

   on Dec 6, 2023

   More actions

   Thirding this feature request. After upgrading a Google Cloud Function from 1st gen to 2nd gen, profiling no longer works because gen 2 Cloud Functions are concurrent by default. Although we're not sure how much extra performance it provides under heavy load due to the GIL, it does keep functions hot for longer, thus reducing initialization times, so we don't have a reason to disable it, either.

   We shouldn't have to choose between concurrency and profiling, especially when the Java and Go profilers don't have this limitation. [Unknown - No Python thread state] is where all our code is being executed, and that isn't a very helpful metric.

   
4. aabmass commented on Feb 9, 2024

   

   aabmass

   on Feb 9, 2024

   Collaborator

   More actions

   I'd like to clarify what is currently supported today:

   • The CPU profiler already supports profiling multiple python threads. Any non-python threads will be grouped together into a single frame with [Unknown - No Python thread state].
   • The Wall time profiler only profiles the python main thread

     cloud-profiler-python/googlecloudprofiler/pythonprofiler.py

     Lines 32 to 37 in 9952717

     	class WallProfiler:
     	"""Python2-compatible implementation for Wall time profiler.
     	This Wall profiler avoids dependency on any native code by using
     	Python signal module. It only profiles the main thread. It has the following
     	limitations:
5. aabmass commented on Feb 9, 2024

   

   aabmass

   on Feb 9, 2024

   Collaborator

   More actions

   [Unknown - No Python thread state] is where all our code is being executed, and that isn't a very helpful metric.

   @lunalucadou it looks like you're using the CPU profiler, so all of your python code running in any python thread should be showing in the profile. The frame with [Unknown - No Python thread state] is for threads that don't have a PyThreadState (non-python threads). Are you using a native extension that creates its own threads? What version of python are you running?
6. lunalucadou commented on Feb 9, 2024

   

   lunalucadou

   on Feb 9, 2024

   More actions

   @aabmass Python 3.11. We don't explicitly use any concurrency; any threads are just whatever Cloud Functions gen 2 does by default.
7. aabmass commented on Feb 14, 2024

   

   aabmass

   on Feb 14, 2024

   Last edited by aabmass

   Collaborator

   More actions

   I don't think this is the case but it would be interesting to know if profiles look different in Python 3.10. I made some changes to support 3.11 but i don't think it would cause this.

   We also don't technically support Cloud Functions, meaning I can't provide much support if its something specific to that platform. https://cloud.google.com/profiler/docs/about-profiler#environment_and_languages
8. aabmass commented on Feb 14, 2024

   

   aabmass

   on Feb 14, 2024

   Collaborator

   More actions

   @lunalucadou I tried it out and see the same issue with Cloud Functions gen 2 regardless of concurrency. I think what's happening is that gen 2 is using a post-fork server (I believe gunicorn) to fork worker processes and profiler is only set up on the original process. Similar to what is described for uWSGI in the docs https://cloud.google.com/profiler/docs/profiling-python#known-issues.

   I would reach out to Cloud Functions team or the functions-framework-python repo and see if they can make a general recommendation (Cloud Profiler team can't provide any support on this platform). One thing I tried is using os.register_at_fork() and that fixed the issue:

   GCF source code

   import functions_framework
   import os
   
   def setup_profiler():
       service_version = "1.0.1-postfork8"
       print("======== Setting up profiler in postfork process for {}".format(service_version))
       import googlecloudprofiler
   
       # Profiler initialization. It starts a daemon thread which continuously
       # collects and uploads profiles. Best done as early as possible.
       try:
           googlecloudprofiler.start(
               service="hello-profiler",
               service_version=service_version,
               # verbose is the logging level. 0-error, 1-warning, 2-info,
               # 3-debug. It defaults to 0 (error) if not set.
               verbose=3,
               # project_id must be set if not running on GCP.
               # project_id='my-project-id',
           )
       except (ValueError, NotImplementedError) as exc:
           print(exc)  # Handle errors here
   
   # initialize profiler in each subprocess after it forks
   os.register_at_fork(after_in_child=setup_profiler)
   
   @functions_framework.http
   def hello_http(request):
       """HTTP Cloud Function.
       Args:
           request (flask.Request): The request object.
           <https://flask.palletsprojects.com/en/1.1.x/api/#incoming-request-data>
       Returns:
           The response text, or any set of values that can be turned into a
           Response object using `make_response`
           <https://flask.palletsprojects.com/en/1.1.x/api/#flask.make_response>.
       """
       request_json = request.get_json(silent=True)
       request_args = request.args
   
       if request_json and 'name' in request_json:
           name = request_json['name']
       elif request_args and 'name' in request_args:
           name = request_args['name']
       else:
           name = 'World'
       return 'Hello {}! {}'.format(name, myotherfunc())
   
   def myotherfunc():
       def innerfunc():
           return 1 + 1
       
       out = 1
       for i in range(10 ** 4):
           out *= innerfunc() * innerfunc()
       return out % 1000000000000000

   

   Again the Cloud Functions team might have a better recommendation and this needs some tweaking to set up in the root process only once.

   👍React with 👍2Reacted by Luna Lucadou and Alex