Argoverse 2 the provided eval_mask seems include ground points

[Kin-Zhang]

Hanqiu found it when we evaluated and reproduced Floxels in the Argoverse 2 validation set locally.

# for local and online eval mask from official repo
s5cmd --no-sign-request cp "s3://argoverse/tasks/3d_scene_flow/zips/*" .

The provided eval_mask at first 5-10 frames in each scenes will include ground points as evaluaiton and some flow annotation actually on the ground points.

Some discussions and images:

Maybe that's another reason behind why local validation set have lower performance than test set (leaderboard-v2 change eval_mask and eval frames so it didn't use the v1 one here). But it should be fair for all methods, since we set all ground points' relative flow=0 for all methods in opensf code. 😅

In this discussion thread, I will run all methods in argoverse 2 validation set and backup all methods in OpenSceneFlow so that we can easily get these correct results after masking out the ground points also.

The commit I ran is: Kin-Zhang/OpenSceneFlow@c5d2177. I will merge these changes in the next paper with fixed results here.

[Kin-Zhang]

Supervised Method

DeltaFlow:

SSF:

DeFlow:

---

Self-supervised Method

TeFlow:

SeFlow++:

SeFlow:

[Kin-Zhang]

And here are others' methods in our codebase also:

Flow4D:

VoteFlow:

FastNSFP (two version w. diff params)

NSFP

ICP-Flow (default params as shown in the Fig.):