Add support for XR_MSFT_hand_tracking_mesh and XR_ANDROID_hand_mesh

[keveleigh]

No change to rigged hands:

But support is added for platform hand meshes via XR_MSFT_hand_tracking_mesh:

Follow-up: #1020

[MaxPalmer-UH]

Made some comments based on the code review.

[MaxPalmer-UH]

Still need to test this but have made some comments based on reviewing the code.

[Cameron-Micka]

Hey @keveleigh, this is super cool! I think I see two issues with the android hand mesh. Both of these issues are because the existing hand mesh's vertex colors drive certain effects - and we don't have these vertex colors anymore.

(1) We lost the wrist fade - the hand is rendered in 3 steps (with 3 different shaders). If you look at the second step the wrist fade is controlled by the red channel of the hand model's vertex color (this line). The new hand model doesn't have this info. We could try and get an artist to supply it.

Or... I recently had to fix a similar issue and went down the programmatic approach.
What I did is created a component that passes a sphere into the shader at the wrist's location (if the hand isn't rigged and skinned this is even easier, you just pass a constant sphere position and radius into the shader) then in the shader do a fade based on pixel's distance from the sphere:

(2) We lost the outline around the hand - plus pinch outline animation. Similar issue to above, the outline is rendered in the third render pass and is controlled by the green channel of the vertex color (this line). This is a bit harder to fix. You could just go down the route of making the outline a constant width (ignore the vertex color). You lose the pinch animation but still get a nice looking outline. Or you will need to programmatically specify which vertices get outline more when pinching (make a sphere around the joints that pinch?) or have an artist paint in those vertex colors.

[MaxPalmer-UH]

Hi @keveleigh, I'm having some issues trying to test this. I've pulled your repo/branch and am not certain which non MRTK packages / versions I need to expose this functionality for both OpenXR features: XR_MSFT_hand_tracking_mesh and XR_ANDROID_hand_mesh. I have added an openxr_left/right_hand.prefab from the MRTK Input package and can see that it has a PlatformHandMesh GameObject, but the MeshFilter mesh is set to None. What's meant to set that up (should it be unassigned by default?) I feel I'm missing a step. Also, will be away next week so will get back to this the following week.

[MaxPalmer-UH]

Oh, also I can't see a Changelog entry for this change.

[keveleigh]

Hey @keveleigh, this is super cool! I think I see two issues with the android hand mesh. Both of these issues are because the existing hand mesh's vertex colors drive certain effects - and we don't have these vertex colors anymore.

Thank you so much for your feedback, @Cameron-Micka! I just went down your suggested programmatic path for fading the wrist based on a sphere (left hand is rigged, right hand is platform):

It's not identical, but it seems to work reasonably well, and we can continue iterating.
I'm currently looking into the outline issue.

[keveleigh]

Oh, also I can't see a Changelog entry for this change.

@MaxPalmer-UH Added!

[keveleigh]

Hey @keveleigh, this is super cool! I think I see two issues with the android hand mesh. Both of these issues are because the existing hand mesh's vertex colors drive certain effects - and we don't have these vertex colors anymore.

Thank you so much for your feedback, @Cameron-Micka! I just went down your suggested programmatic path for fading the wrist based on a sphere (left hand is rigged, right hand is platform):

It's not identical, but it seems to work reasonably well, and we can continue iterating. I'm currently looking into the outline issue.

I have now restored the outline! (once again, left hand is rigged, right hand is platform):

The pinch highlight is currently the entire hand, but it'll take some time to find a solution there:

[Cameron-Micka]

Hey @keveleigh, this is super cool! I think I see two issues with the android hand mesh. Both of these issues are because the existing hand mesh's vertex colors drive certain effects - and we don't have these vertex colors anymore.

Thank you so much for your feedback, @Cameron-Micka! I just went down your suggested programmatic path for fading the wrist based on a sphere (left hand is rigged, right hand is platform):

It's not identical, but it seems to work reasonably well, and we can continue iterating. I'm currently looking into the outline issue.

I have now restored the outline! (once again, left hand is rigged, right hand is platform):

The pinch highlight is currently the entire hand, but it'll take some time to find a solution there:

Awesome, looking great @keveleigh!

[whitetigerhelix]

lg2me

There are some comments by Max that still look unresolved. It would be good to get his approval on this PR also.