CLI flag for REST catalog's `{prefix}`

[DerGut]

Feature Request / Improvement

IIUC the pyiceberg CLI only supports configuration of a REST catalog's {prefix} path parameter via configuration file or environment variable PYICEBERG_CATALOG__DEFAULT__PREFIX. It would be great to have a flag for that (similar to --uri).

I could probably contribute this myself.

[kevinjqliu]

sounds like a great idea! thanks!

[kevinjqliu]

Taking another look at this, i think we might want to change this to --warehouse. The REST spec handshake expects a warehouse parameter for v1/config, https://github.com/apache/iceberg/blob/d06e6c22fecf00be81f2eb399f090fae320560a4/open-api/rest-catalog-open-api.yaml#L64-L78

iceberg-python/pyiceberg/catalog/rest/__init__.py

Lines 617 to 620 in ec0a451

	def _fetch_config(self) -> None:
	params = {}
	if warehouse_location := self.properties.get(WAREHOUSE_LOCATION):
	params[WAREHOUSE_LOCATION] = warehouse_location

for which the server then customize the prefix

[kevinjqliu]

cc @DerGut @mrutunjay-kinagi @Fokko

[mrutunjay-kinagi]

cc @DerGut @mrutunjay-kinagi @Fokko

@kevinjqliu Does this need a PR ?

[DerGut]

I just started to wonder the same thing, but I'm still a bit torn:
Doesn't the config response from the server still use the "prefix" configuration key to tell the client which {prefix} to use (via default or overrides)?

EDIT: from that point of view, we may need to distinguish between two configurations

1. prefix which is used to build the endpoints, but can be overridden from a catalog's /v1/config response
2. warehouse which can be used to influence the catalog's /v1/config response

[kevinjqliu]

Yea it is confusing. I realized this when looking at my .pyiceberg.yaml config file. We allow warehouse param to be configurable for IRC https://py.iceberg.apache.org/configuration/#rest-catalog

The flow / "handshake" is:

• client (pyiceberg) calls IRC's /v1/config endpoint with warehouse parameter
• IRC returns the prefix in the /v1/config response
• client uses the prefix for all subsequent requests to IRC

The main point here is that the client (pyiceberg) doesn't specify prefix directly, the server to configures the prefix for the client.

We technically could let the client (pyiceberg) to override prefix but doing so might be error prone

[Fokko]

Ah, I misunderstood the PR. I agree that passing in the prefix is less valuable. Often the prefix is used to distinguish between different workspaces/organisations, but not sure if that's valuable for users to pass in by hand.

[kevinjqliu]

i also got confused until i saw the warehouse param in my .pyiceberg.yaml file 😄

[Fokko]

I think warehouse makes more sense, but not in a REST context. In Hive/Hadoop this actually points to the location where the data lives. Yay for re-using parameter names :D