Backport Python 3.10 NewTypes #156

Activity

1. AlexWaygood commented on May 8, 2023

   

   AlexWaygood

   on May 8, 2023

   Member

   More actions

   Here you go: #157
2. effigies commented on May 8, 2023

   

   effigies

   on May 8, 2023

   Author

   More actions

   Amazing. Thanks for being so quick!

   🚀React with 🚀1Reacted by Alex Waygood
3. JelleZijlstra commented on May 8, 2023

   

   JelleZijlstra

   on May 8, 2023

   Member

   More actions

   @effigies may I ask why it's important to you that types are picklable? We have quite a lot of code to deal with pickling types and on first sight I don't really understand why people would care. This may be relevant to implementing new type system features in the future.
4. JelleZijlstra

   closed this as completedin #157on May 8, 2023
5. effigies commented on May 8, 2023

   

   effigies

   on May 8, 2023

   Author

   More actions

   Sure. It's not a principled requirement so much as practically it's causing me problems while trying to add static type checking to an existing code-base that uses type annotations at runtime.

   The context is a workflow engine where we use types to annotate inputs and outputs (basically attrs classes), for example:

   @attr.s
   class InputSpec:
       in_file: File = attr.ib(...)
   
   @attr.s
   class OutputSpec:
       out_file: File = attr.ib(...)
   
   class MyTask(Task):
       input_spec = InputSpec
       output_spec = OutputSpec
   
       def run(self, runtime):
           # run task
           return runtime

   This is a static example to give you the flavor. The MyTask class is sent to a runner along with a dictionary of inputs. We also have syntactic sugar for pure Python tasks:

   @task
   def mytask(in_file: File) -> File:
       ...

   Here, the input/output specs are dynamically constructed. In order to dispatch them to run in separate interpreters or on other servers, some form of pickling is required. It's possible that there's a way to rewrite our code base to avoid it, but I found that when adding a NewType I had passing tests on Python 3.10 but errors like the following in Python 3.7-3.9:

   concurrent.futures.process._RemoteTraceback: 
   """
   Traceback (most recent call last):
     File "/Users/runner/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/concurrent/futures/process.py", line 205, in _sendback_result
       exception=exception))
     File "/Users/runner/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/multiprocessing/queues.py", line 358, in put
       obj = _ForkingPickler.dumps(obj)
     File "/Users/runner/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
       cls(buf, protocol).dump(obj)
   AttributeError: Can't pickle local object 'NewType.<locals>.new_type'
   """
   

   To be honest, I'm not even sure if we'll end up making File/Directory into NewTypes, so I may not need this feature, but I think having consistent implementations across Python versions would at least help determine if NewType is the right tool for the job.
6. JelleZijlstra commented on May 8, 2023

   

   JelleZijlstra

   on May 8, 2023

   Member

   More actions

   Thanks, that's helpful!