Enum types are not picklable if defined in nested namespaces

[BCSharp]

This used to work on .NET:

import clr
import pickle

clr.AddReference("System.Text.Json")
clr.AddReference("Microsoft.Scripting")

import System.Text.Json
import Microsoft.Scripting

data = [
    System.DayOfWeek.Wednesday,  # still works
    System.Text.Json.JsonValueKind.Object,  # nested namespace
    Microsoft.Scripting.Severity.Warning,  # different namespace than System
]

for value in data:
    print(pickle.dumps(value))

It does not work on .NET 8.0 and higher.

It was relying on clr.Deserialize, which was included if IronPython was compiled with FEATURE_SERIALIZATION (on by default). The implementation was using BinaryFormatter, which is now unsupported. With clr.Deserialize and clr.Serialize gone, pickling relies on a generic object __reduce_ex__ implementation. It fails for types that are placed in nested namespaces, because __module__ attribute on those object is not the actual module as in imported sys.modules but the complete dotted-path namespace.

Two ideas come to mind on how to resolve this regression:

1. Tweak pickle.py in StdLib to be more mindful of IronPython namespaces; for instance a translation of (module, name) from e.g. (System.Text.Json, JsonValueKind) to (System, Text.Json.JsonValueKind) is simple and would do the trick, but it will require using pickle protocol >= 4, which allows for dotted (nested) type names.

2. Capture the resolution of __reduce_ex__ in OneOffRTesolver like it used to be when FEATURE_SERIALIZATION was defined, and redirect to EnumOps.SerializeReduce. This will handle all enums, but not other types that may have worked in case of option 1, OTOH, StdLib is untouched.

(or the combination of the two).

Both options make objects pickled by IronPython/.NET 6.0 not unpicklable (not loadable) on .NET 8.0 and higher, but that is the consequence of BinaryFormatter being gone. AFAIK, pickle file format is not meant to be binary-portable across Python versions, so one can argue that this is better than the alternative of not being able to pickle at all. Universal pickling of an arbitrary .NET object is probably not feasible, but it is feasible and desirable for simple built-in types. For user types for which pickle support is needed, the developer can implement their own __reduce_ex__.

See also #29

[BCSharp]

A variant of idea 1: instead of modifying pickle code in StdLib, modify the lookup within sys.modules. If the key is not found, treat it as a dotted-path namespace and retrieve the namespace object instead of the module. This would only be done for lookups, not keys enumeration.

[slozier]

Could this be solved by having the import mechanism add .NET modules to sys.modules? The pickle modules calls __import__, but the importer (for .NET modules) is bypassing the regular importlib mechanism which would add System.Text.Json to sys.modules. For example this doesn't blow up:

import sys
import pickle
import clr
clr.AddReference("System.Text.Json")
import System.Text.Json
sys.modules["System.Text.Json"] = System.Text.Json
pickle.dumps(System.Text.Json.JsonValueKind.Object)

[BCSharp]

Just to check that we are talking about the same thing: is what you call ".NET modules" the same as .NET namespaces?

>>> import clr
>>> clr.AddReference("System.Text.Json")
>>> import System.Text.Json
>>> type(System.Text.Json)     
<class 'namespace#'>

I think it will work just the same as modifying the lookup, but arguably in a more appropriate place. So it will still import System, and add both namespace System and namespace System.Text.Json to sys.modules.

[slozier]

Just to check that we are talking about the same thing: is what you call ".NET modules" the same as .NET namespaces?

Yep, same thing. I just said module because that's what it shows up as via repr:

>>> System.Text.Json
<module 'Json' (CLS module from System.Text.Json, Version=8.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)>

Anyway, my initial thought for the proper long term solution was to rip out the .NET namespace import machinery into a meta path finder. I'm not sure if that's enough to make things just work, but it seems like a step in the right direction to not make importing .NET namespaces special...