Could operator.methodcaller be optimized using LOAD_METHOD? #89013

Activity

1. anntzer commented on Aug 6, 2021

   

   anntzermannequin

   on Aug 6, 2021

   MannequinAuthor

   More actions

   Currently, methodcaller is not faster than a plain lambda:

   In [1]: class T:
      ...:     a = 1
      ...:     def f(self): pass
      ...:     
   
   In [2]: from operator import *
   
   In [3]: %%timeit t = T(); mc = methodcaller("f")
      ...: mc(t)
      ...: 
      ...: 
   83.1 ns ± 0.862 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
   
   In [4]: %%timeit t = T(); mc = lambda x: x.f()
      ...: mc(t)
      ...: 
      ...: 
   81.4 ns ± 0.0508 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
   

   (on some machines, I find that it is even slower).

   Compare with attrgetter, which *is* faster:

   In [5]: %%timeit t = T(); ag = attrgetter("a")
      ...: ag(t)
      ...: 
      ...: 
   33.7 ns ± 0.0407 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
   
   In [6]: %%timeit t = T(); ag = lambda x: x.a
      ...: ag(t)
      ...: 
      ...: 
   50.1 ns ± 0.057 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
   

   Given that the operator module explicitly advertises itself as being "efficient"/"fast", it seems reasonable to try to optimize methodcaller. Looking at its C implementation, methodcaller currently uses PyObject_GetAttr followed by PyObject_Call; I wonder whether this can be optimized using a LOAD_METHOD-style approach to avoid the construction of the bound method (when applicable)?
2. anntzer

   added

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   on Aug 6, 2021
3. sweeneyde commented on Aug 7, 2021

   

   sweeneyde

   on Aug 7, 2021

   Member

   More actions

   Using _PyObject_GetMethod similarly to the way that LOAD_METHOD/CALL_METHOD does seems like a reasonable idea to me -- do you want to make a pull request? It would also be nice to see some microbenchmarks for the change once it's ready.
4. sweeneyde

   added

   3.11only security fixesonly security fixes

   performancePerformance or resource usagePerformance or resource usage

   on Aug 7, 2021
5. sweeneyde commented on Aug 7, 2021

   

   sweeneyde

   on Aug 7, 2021

   Member

   More actions

   For what it's worth, in my benchmarks on 3.11, methodcaller was already a bit faster than lambda:

   #################### Builtin calls ####################

   PS > .\python.bat -m pyperf timeit -s "from operator import methodcaller as mc" -s "reverse_it = mc('reverse')" -s "arr = []" "reverse_it(arr)"
   Running Release|x64 interpreter...
   .....................
   Mean +- std dev: 84.3 ns +- 1.9 ns

   PS >.\python.bat -m pyperf timeit -s "reverse_it = lambda x: x.reverse()" -s "arr = []" "reverse_it(arr)"
   Running Release|x64 interpreter...
   .....................
   Mean +- std dev: 95.5 ns +- 2.9 ns

   #################### Python calls ####################

   PS > .\python.bat -m pyperf timeit -s "from operator import methodcaller as mc" -s "reverse_it = mc('reverse')" -s "class A: reverse = lambda self: None" -s "arr=A()" "reverse_it(arr)"
   Running Release|x64 interpreter...
   .....................
   Mean +- std dev: 140 ns +- 4 ns
   PS > .\python.bat -m pyperf timeit -s "reverse_it = lambda x: x.reverse()" -s "class A: reverse = lambda self: None" -s "arr=A()" "reverse_it(arr)"
   Running Release|x64 interpreter...
   .....................
   Mean +- std dev: 159 ns +- 4 ns
6. markshannon commented on Aug 19, 2021

   

   markshannon

   on Aug 19, 2021

   Member

   More actions

   If the problem is that methodcaller is not faster than a lambda, then why not use a lambda?

   Lambdas are just Python functions and calling them will get faster.

   We aren't going to spend time optimizing calls to methodcaller, so you might as well use a lambda.
7. ezio-melotti

   transferred this issue fromon Apr 10, 2022
8. iritkatriel

   added

   3.12only security fixesonly security fixes

   and removed

   3.11only security fixesonly security fixes

   on Sep 7, 2022
9. corona10

   added a commit that references this issue on Aug 1, 2023

   gh-89013: Improve the performance of methodcaller (lazy version) (gh-…

   ...

   f7c9144
10. hugovk commented on Dec 11, 2023

    

    hugovk

    on Dec 11, 2023

    Member

    More actions

    The PR has been merged, I think we can close this now. Feel free to re-open if still needed.
11. hugovk

    closed this as completedon Dec 11, 2023
12. colesbury

    added a commit that references this issue on Dec 11, 2024

    gh-127065: Make methodcaller thread-safe and re-entrant (GH-127746)

    ...

    b0f278f
13. srinivasreddy

    added a commit that references this issue on Jan 8, 2025

    pythongh-127065: Make methodcaller thread-safe and re-entrant (python…

    ...

    05af03a