GH-49002: [Python] Fix array.to_pandas string type conversion for arrays with None

[AlenkaF]

Rationale for this change

The conversion from array with string type to pandas series, when array only has a None element, has been taking the old code path even with pandas 3.0.

What changes are included in this PR?

Always check dtype in the _array_like_to_pandas conversion and use pandas new default string dtype if available.

Are these changes tested?

Yes.

Are there any user-facing changes?

No, only bug fix.

• GitHub Issue: [Python] Wrong cast from StringArray to pandas 3 when element is None #49002

[raulcd]

Thanks @AlenkaF this looks good to me. Seems to be what was proposed by @jorisvandenbossche in the issue and in-line to what we have here:

arrow/python/pyarrow/pandas_compat.py

Lines 936 to 944 in d2315fe

	# for pandas 3.0+, use pandas' new default string dtype
	if _pandas_api.uses_string_dtype() and not strings_to_categorical:
	for field in table.schema:
	if field.name not in ext_columns and (
	pa.types.is_string(field.type)
	or pa.types.is_large_string(field.type)
	or pa.types.is_string_view(field.type)
	) and field.name not in categories:
	ext_columns[field.name] = _pandas_api.pd.StringDtype(na_value=np.nan)

I'll wait until end of day in case @jorisvandenbossche has time to take a look otherwise I'll merge.

[raulcd]

Oops, sorry, I've just realised there are several test failures which are related and we should fix. Should have checked CI before :)

[jorisvandenbossche]

I think it would be better to move this check a bit more above, together with the if/elif block checking for types_mapper. Because then we get to if hasattr(dtype, '__from_arrow__'): and will avoid actually converting the pyarrow memory to an numpy object-dtype array of strings

[AlenkaF]

Yes, that was my first idea but didn't think it well through before changing to what I have now. The thing is that I have hit this line https://github.com/AlenkaF/arrow/blob/6f1fda5ef1cfe7ee40ccd1ddefc3861c2718d920/python/pyarrow/array.pxi#L2319
and then the change of the dtype got reverted to None.

Will try putting the whole if/elif block further back, as suggested (if I do not break something else).

[jorisvandenbossche]

Suggested change

	elif _pandas_api.uses_string_dtype() and (
	elif _pandas_api.uses_string_dtype() and not strings_to_categorical and (

like we do in the other place with this logic (which Raul quoted), only that you will have to get the option value out of the options object

[AlenkaF]

Yes, will do!

[AlenkaF]

Thanks for quick reviews! Will go through the comments now - was going through all the tests I just broke =) Should have put the PR back to draft, will do so next time.