[doc/build] Fix iceberg quickstart to build on flink standard image #2555
Conversation
MehulBatra
changed the title
luoyuxia
left a comment
luoyuxia
left a comment
There was a problem hiding this comment.
@MehulBatra Hi, thanks for your update. I left minor comments. PTAL
| @@ -199,9 +222,8 @@ services: | |||
| datalake.iceberg.warehouse: /tmp/iceberg | |||
| volumes: | |||
| - shared-tmpfs:/tmp/iceberg | |||
There was a problem hiding this comment.
from flink 0.9, we will also need mount /tmp/fluss:
shared-tmpfs:/tmp/fluss
| @@ -220,9 +242,11 @@ services: | |||
| datalake.iceberg.warehouse: /tmp/iceberg | |||
| volumes: | |||
| - shared-tmpfs:/tmp/iceberg | |||
There was a problem hiding this comment.
from flink 0.9, we will also need mount /tmp/fluss:
shared-tmpfs:/tmp/fluss
There was a problem hiding this comment.
why is that, can you please help me understand?
There was a problem hiding this comment.
the tablet server will need to access remote file in getLatestLakeSnapshot method
luoyuxia
left a comment
luoyuxia
left a comment
There was a problem hiding this comment.
@MehulBatra Thanks for the update. Left minor comments. Otherwise, LGTM!
| entrypoint: ["sh", "-c", "cp -v /tmp/lib/*.jar /opt/fluss/plugins/iceberg/ && exec /docker-entrypoint.sh coordinatorServer"] | ||
|
|
||
| - shared-tmpfs:/tmp/fluss | ||
| - ./lib/fluss-lake-iceberg-$FLUSS_DOCKER_VERSION$.jar:/opt/fluss/plugins/iceberg/fluss-lake-iceberg.jar |
There was a problem hiding this comment.
nit:
| - ./lib/fluss-lake-iceberg-$FLUSS_DOCKER_VERSION$.jar:/opt/fluss/plugins/iceberg/fluss-lake-iceberg.jar | |
| - ./lib/fluss-lake-iceberg-$FLUSS_DOCKER_VERSION$.jar:/opt/fluss/plugins/iceberg/fluss-lake-iceberg-$FLUSS_DOCKER_VERSION$.jar |
?
|
|
||
| - shared-tmpfs:/tmp/fluss | ||
| - ./lib/fluss-lake-iceberg-$FLUSS_DOCKER_VERSION$.jar:/opt/fluss/plugins/iceberg/fluss-lake-iceberg.jar | ||
| - ./lib/hadoop-apache-3.3.5-2.jar:/opt/fluss/plugins/iceberg/hadoop-apache.jar |
There was a problem hiding this comment.
dito, to add version for the jar mounted in image?
| @@ -220,9 +242,11 @@ services: | |||
| datalake.iceberg.warehouse: /tmp/iceberg | |||
| volumes: | |||
| - shared-tmpfs:/tmp/iceberg | |||
There was a problem hiding this comment.
the tablet server will need to access remote file in getLatestLakeSnapshot method
| FLINK_PROPERTIES= | ||
| jobmanager.rpc.address: jobmanager | ||
| taskmanager.numberOfTaskSlots: 10 | ||
| taskmanager.memory.process.size: 2048m |
2 participants
Purpose
Linked issue: close #2543
Brief change log
To build on flink standard image, while tiering
Mkdirs failed error is coming while writing iceberg data. The core issue is that the standard Flink image drops privileges, and user: "flink:root" doesn't prevent that.
Custom images with pre-modified entrypoints. For the
Standard Flink image to work, we need to replicate that, so we added a docker file to create an image.
Tests
API and Format
Documentation