Here’s a Unity demo I created for a recent pitch demonstrating how we could use AR to try on jewelry. I used Vuforia cylinder targets and a simple script to sample and average skin tone in an attempt to cover up the paper tracker.

Along with what I believe is a bug with how Unity is handling macro focus mode (you’ll notice that autofocus is wreaking havoc)  I’m not entirely satisfied with how I am matching the skin tone. I’m trying to sample and average skin values along the edges of the marker. I’m using the surface normal of those sample positions to only include pixels in areas that are visible to the camera and that fall within a certain angle of view. It works ok, but with the variation in skin tone and how it changes with light angle there are color shifts that are far from convincing.  I think the better solution is to employ a fragment shader that displaces the live camera pixels – pinching them inward to obscure the tracker.

Hopefully, I’ll be able to revisit this soon so I can figure out how to get that working.