Shoe Buying & Augmented Reality

Finding the right pair of shoes can be a nightmare. It doesn’t matter if you’re concerned more with style or just have foot-structure needs, the act of buying a good pair of shoes can seem like an endless game of roulette.

Zappos.com, the popular online shoe retailer, has helped make this easier on the shoe buyer. With hassle-free returns and free shipping, you can try on–and send back–an unlimited number of shoes until you find the right pair. Many orders get upgraded to one-day shipping. And you can print out shipping labels and mail back using the same box, making Zappos shopping a timesaving alternative to the hours spent in a shoe store.

Zappos encouragement of try-on-and-return has helped catapult them to success. Many people will tell you that Zappos doesn’t even sell shoes, it sells customer service.

So we won’t argue that Zappos’s gets a gold star for encouraging returns. But I’d like to point out this kind of service only addresses the question, “How do we help customers return shoes that they don’t like?” Perhaps they should perhaps be addressing the question, “How do we help customers find the right shoe, on the first try?”

It’s not personalized

Customers surveying this product said…
56% “Felt a half size larger than marked”
95% “Felt true to width”
53% “Excellent arch support”

These stats are the primary way customers on Zappos have to make a purchasing decision. The problem is that this data, even if aggregated from a large group, is still just based on everyone else’s perception of shoe fit.

The alternative ways to finding a good fit are equally constraining. A customer could judge against a brand they’re already familiar with and that has a consistent size range, but this is restrictive to out-of-brand purchases. They could look at photos, which work well for style, but doesn’t necessarily give a sense of fit compared to their own foot.

Green issues

With Zappos offering free shipping both ways, people are less inhibited to constrain numerous cycles of shipping and returning. The common upgrade to overnight shipping can mean that a 2.2 lb package of shoes will emit its own weight in CO2 on a one-way air cargo trip. For a customer who takes 3 separate attempts to find the right shoe, they could have created 5 times the weight of the final product in CO2 emissions. And this doesn’t even count the potential wasted packaging and printed packing slips!

Helping people find a good fit, up front

Ideally, we’d need a way to allow customers to judge a shoe against their own feet before the purchase flow even begins.

In a potential scenario, a customer is browsing the Zappos site and sees a pair of shoes that looks promising for her business-casual wardrobe. On first glance, the reviews look great. The pictures look great. But the customer needs to make sure that the toe box is wide enough to keep from compressing her toes, while at the same time making sure the shoe doesn’t look too chunky when worn with her work slacks. She’s used to buying and returning shoes from Zappos, but this time notices a “Try it on” tool and launches it.

With this tool, the customer is able to see a real-time composite of the pair of shoes against her own feet, right on the site. “Try it on” automatically displays the most likely size and width based on her feet, and the simulated shoes match her feet on the screen as she walks around and rotates her ankles. She really likes the style. But the simulator has noticed that at a few angles, the toe box looks a little snug, and warns the customer of this.

The customer tells “Try it on” to find similar styles with a wider toe box. The tool comes up with a list of options, and the customer toggles through each of them one-by-one until she finds a pair she really likes. She switches the color to brown instead of black, selects “Add to Cart,” and completes her order with free one-day shipping. When the shoes arrive a day later, they fit very well. The customer still would have had the option to return for free, but didn’t need to.

An Augmented Approach

The above scenario is achievable, if we take things into the realm of augmented reality (AR). Some people are familiar with the USPS box simulator, “See if it fits,” which helps customers determine what size box they need to ship an item. The customer prints out an emblem from their home printer, tapes it to the surface of an object, sets up their web cam, and can view Priority Mail boxes mapped around the object.

Although the USPS example doesn’t calculate true geometry, and requires the customer to pick the right fit box, it is a great visualization and decision-making tool. For Zappos customers, use of a similar AR tool could allow a more personalized decision-making process by allowing judgment of fit and style against the customer’s own feet instead of against historical data about other people’s feet.

Breaking this apart a bit, this is what we might need.

AR software

No matter what, there needs to be some way to composite what is happening in reality with what is virtual, ideally over an Internet connection. The company Total Immersion has created a pretty good piece of software that can recognize, track and render models.

Web camera

A web camera connected to the computer is required for the software to create a composite from reality. This may also require an extra plugin in order to talk to the tool; the USPS simulator uses Flash. Virtual models of all Zappos shoes, in all sizes and widths

You can’t have AR without something to simulate, which in this case would need to be in the form of 3D shoe models. Since Zappos touts over 1 million shoes, this is clearly the most difficult and expensive task.

The easiest method might be to create a simple texture-mapped box model using photos of the six sides of a shoe, which Zappos already has. Unfortunately, these models will be clunky and lack the nuances of shape that may be essential to determining good fit and style. Potentially, such simple texture maps could be paired with measurement data, such as a formula to calculate how the texture maps will stretch and scale off of a baseline size. At the very least, these simple models could make for a good, early prototype for testing.

That said, true 3D models are the key to creating a visually realistic and accurate true-to-fit experience. Perhaps Zappos could engage their passionate community to construct 3D models in all sizes, if they can’t be obtained from the manufacturers.

A way to detect the position and size of each foot

Most mainstream AR software requires the use of a marker to distinguish key items in a scene. In this case, we’d need 2 separate “left and right” markers for each foot, with a directional indication for the front and the back. We’d also want the markers to be mobile, since the customer might want to pick up their feet and rotate them.

One method could be to have users print two strips of paper, one marked with an “L”, and one marked with an “R”, both with an arrow. The customer would tape the strips around the middle of their feet with the arrow pointing toward their toes. Each shoe model would have a central axis in its true middle, which would map to the strip of paper as an anchor point. This is a cheap and easy method for the customer, but the tradeoff might be in some inaccuracies as a result of the assumed middle anchor point.

For models and AR software that might have more complex mapping capabilities, customers could send away for a pair of socks. The left sock would be covered in a pattern that is different from the right sock, and the software would use the variations of this pattern as it stretches, creases, and tapers with perspective, to create an accurate mapping. The tradeoff is that this causes more delay and hassle for the customer, especially the first time user. It also requires more complex rendering software and system capabilities.

As customer computers and AR apps get more powerful, and as cloud processing gets more mainstream, we’ll likely see applications able to recognize native shapes and geometry without the use of markers.

Taking things to the next level

All of this is great for simulation of a shoe against the visible, outer structures of a person’s foot. But how about detecting the structural issues in the customer’s feet as they stand and put pressure on them? We might think about combining the visual AR simulation with a Wii-like pad that can sense arch height and pronation and make suggestions. In fact, this isn’t too far off what Dr. Scholl’s has been doing with their “Footmapping” kiosks in local pharmacies; for our case, we would just be bringing this into the home.

Zappos can easily continue along with their current strategy, and keep most of their customers happy for a good, long time. But to really amp it up a notch, trying a few ways to prevent the returns that they are so good at expediting could save a lot of trees, a lot of money, and a good amount of time.

Resources

If you’re looking for some great information about Augmented Reality, I encourage you to watch Kevin Cheng’s presentation from Interaction 10.

Calculations for carbon emissions based off of Time For Change’s shipping goods emissions table.