Skip to content

[Python] Consider pa.date32/64.to_pandas_dtype() returning datetime64[s] instead of datetime64[ms] #49168

@mroeschke

Description

@mroeschke

Describe the enhancement requested

In #35656, it appears that pa.Date32/64.to_pandas_dtype was specified to return datetime64[ms] instead of datetime[s] because of a limitation in parquet(?) xref #35656 (comment)

In [1]: import pyarrow as pa

In [2]: pa.__version__
Out[2]: '23.0.0'

In [3]: pa.date64().to_pandas_dtype()
Out[3]: dtype('<M8[ms]')

In [4]: pa.date32().to_pandas_dtype()
Out[4]: dtype('<M8[ms]')

I see that a TIMESTAMP with a SECONDS unit isn't still an official parquet type, https://parquet.apache.org/docs/file-format/types/logicaltypes/, but I was hoping that ideally this pandas conversion would ideally be de-coupled from this parquet limitation and return datetime64[s] for this API.

Component(s)

Python

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions