I’ve been working on a demo project that involves putting a pair of sneakers on a statue in a mobile AR app. This presented some significant challenges typical to this type of project – mainly registering AR tracking to the physical space and then precisely fitting the shoes around the feet and ensuring that they were accurately occluded by the legs.

First, here is a quick demo video. Apologies for the choppy frame rate. The screen recording process really hurts performance.

For this demo, I did not want to rely on a printed marker. Since the recent beta for ARKit 2 supports 3D trackable objects and a serializable world map, my initial hope was that I would be able to use the natural features in the environment for localization. But after a few tests, I could tell that this was not going to work well because the statue – cast in bronze, has minimal contrasting surface detail and a fairly reflective surface. Further, relocalizing to a saved point cloud of the surrounding environment was slow, did not work reliably in a dynamic outdoor environment, and lacked the precision required for this effect.

So less desirable but nonetheless reliable method of registering against a marker image was used. In this example, I’m using Vuforia VuMarks in Unity with extended ARKit tracking enabled.
So once we’ve registered the AR tracking to the physical space, there is still the matter of getting the shoes positioned correctly and then creating an occlusion object (3d mask) for the legs.

This would be a laborious task if we relied only on hand measurements and trial and error.
I covered photogrammetry – the process of reconstructing 3d objects from a series of photos, in a recent blog post.  This also happens to be a great technique for measurement and positional reference. By taking a number of photos of the base and legs of the statue I was able to generate a reference mesh that I could use to create an occlusion mesh for the legs and as a reference for fitting the sneakers.