Andrew Hsu: The Invention of TOUCH & What’s Next

Bill Gates stood on the stage at the (now-defunct) Comdex show in Las Vegas in 2000 with his schoolboy smile touting the new “tablet PC.”  Penned on the tablet in Bill’s handwriting was “Tablet PC is SUPER COOL!”  Behind the stage a backlit sign read “experience the evolution”.

Microsoft evolution never became a revolution because the company’s disparate and factional divisions failed to work together to vision and implement a turnkey experience.

The revolution happened in 2007 with the launch of the iPhone.

(As with most industries) evolution is often interrupted by black-swan revolutions. Sound (voice communications), touch (pinch and zoom navigation), sight (Heads Up Display [HUD]) all changed the way consumer used the phone and is one of the gating factors in technology adoption.

Knowing what technology will help us evolve and what technology revolutionizes is more of a human insight that a science. Ergonomics help us rearrange the digital furniture; however, changing the way we connect with this communication device is profoundly human. What is beyond touch, what is the next revolution?

A Short History of Touch

Although Gates told reporters off stage in Las Vegas that how excited everyone was in Redmond (Developers were checking the tablet out to play with – “a very good sign,” he said) 6 months later warehouses were still full of the Tablets. Q2 shipments had plummeted 25% with a meager 100,000 total units sold.  Mike Magee, technology writer for the Inquirer wrote despondently that “This is another classic case of IT firms thinking they know what technology people will like, and failing to take off the blinkers.”

Touch appeared back in 1971 over a ten year period began to appear in the form of  infrared technology (such as the Hewlett Packard 150) which show up in various military applications. The IR matrix of beams are used to detect a finger touching the screen.

But the IR technology was expensive and the technology gained more mainstream adoption was “resistive touch”.

It was a simple concept. Resistive touch screens were built using two layers of conductive material (Indium Tin Oxide). The two layers were separated by a small pocket of air. An action was triggered when a stylus, or other object, pressed the top layer into contact with the bottom layer.

The limitation was it was like a pin board. You could tell the device where you were move the point of contact. But it did not have multi-touch functionality essential to pitch and zoom navigation.

Mass-market adoption was not an option:

  1. The screen wore out
  2. Required a stylus pen for accuracy
  3. The air pocket made the screen appear hazy
  4. OEMs had to build a clunky hole in the casing (as the top of the resistive sensor had to be exposed to user’s input)

This is the technology that Bill Gates was holding up at Comdex in 2000*. The unit’s resistive touch stylus was used to input into clunky dialogue boxes to input text and commands. The entire project was “resistive”. The Office team refused to build for the unit adding to the painful UX.

*[A technogeek aside: Microsoft's Surface touch solution uses Frustrated Total Internal Reflection (FTIR)]

Meeting Andrew Hsu

In 2013, I ran an event on connect screens in New York. I wanted to tell a story about the importance of the screen in the evolution of mobile phone design and adoption. I invited Professor Donnell Walton from Corning Glass, as well as representatives from Microsoft’s Surface team, Google Glass and was looking to find a speaker to explain “touch”.  Maybe I could locate someone from the scuttled Apple Newton team?

I found, much to my surprise (like an anthropologist that finds that we did not evolve directly from monkeys) that the precursor to the 2007 Apple iPhone was a skunk works project headed up by an engineer called Andrew Hsu.

Andrew developed and patented a capacitive touchscreen suitable for mobile devices way back in 1999. He developed a system which computes the location of a user’s fingers based on how they change the capacitance values of an invisible matrix of electrodes.  The capacitive touchscreen did not suffer from the various user experience drawbacks of the resistive touchscreen – it does not wear out, it does not cloudy the underlying display, and it does not require a big hole to be cut into the device casing.  But most importantly, it enables natural finger input.

This capacitive touch is not a mouse click. It is not a data poke with a Stylus. Andrew Hsu’s touch allowed us to communicate in a very human way with pointing and pinching space.

Don Norman is often quoted about touch.

“We’ve lost something really big when we went to the abstraction of a computer with a mouse and a keyboard, it wasn’t real . . . swiping your hand across the page  . . . is more intimate. Think of it not as a swipe, think of it as a caress.”

While mobile success is almost always based on interface and usability, it took seven years for Andrew Hsu to convince the industry to adopt the technology. Revolutions come in simple packages: text messaging, Apple’s mobile application SDK, gesture-based gaming.

We talk about the consumerization of technology; touch was the humanization of technology. In a world where data appeared cerebral and uninviting, we suddenly can interface in this data and content as we do with real object. The physical world became extensible and less scary.

From Click to Pinch & Zoom

In 2006, handset manufacturer LG trialled launched capacitive touch with their designer Prada phone. The LG phone had all the correct ingredients – capacitive touchscreen for intuitive finger input, high resolution display, and one of the first graphics co-processors in a handset. Prada brought style to the table and LG brought the insight that touch that would ultimately inspire the new mobile consumer.

But we had to wait one more year.

When Jobs returned to Apple he shut down the Newton project.  This legacy 1993 technology had poor handwriting recognition and had little traction in the market. But Andrew Hsu’s capacitive touch appealed to Steve Jobs UI sensibilities.

As a post Newtonist, Jobs once said “we are born with five styluses on each hand”.

When he introduced the iPhone, we knew that being able to move large format data on a small screen with a pinch and zoom changed the way the consumer saw their mobile device.  Where Steve Job went further than touch was his insight in designing a full edge-to-edge screen that had the dimensions of a letter size piece of paper.  The screen called out to be touched, worked on and paged through.

Although touch revolutionize the phone and lines weaved around the block for new product releases of Apple new “human” interface, the consumer was still nose-to-screen, bumping into lamp posts while elegantly navigating data a hundred miles away.

“Bump” (the file exchange application recently acquired by Google) and other application including NFC payment extending this love of tactile interface by promote social touch between other phones and public devices such as POS.

Gesture: Moving Beyond the Cool?

While touch is an important sense, sight is essential for navigation. The next revolution is to make data come to live seamlessly in the real world.

When we talk about HUD, we think of the new Google Glass and the opportunity to integrate data into our line of site. In parallel, see the world and the data behind it. Integrated cyborg solutions like Google Glass and future visions of embedded epidermal circuit (seen in Total Recall).

Microsoft had the lead in a new HUD interface using gesture.  XBOX Kinects was the one product that Microsoft was seeing growth in the consumer sector. However, the leviathan was unable to make this a multiscreen strategy fast enough.

Moving gesture elegantly to PCs and window phones never happened. There is a Kinect for Windows but it lacks the software for controlling the interface.

The Leap motion controller is a step forward.  A small multiscreen sensor box not tied to console in the dean but with the ability to tether like a dongle to a wide variety of screens and deliver better sensitivity to Kinect. It has multiple commands down to finger level accuracy.

Andrew Hsu still believes that touch is less ambiguous on the consumer navigation intent. “How can you disambiguate between “accidental” and intentional gestures.  The beauty of touch interaction is that you basically get user intent for “free” – a user typically only touches the device when he/she wants to interact with it.  The cases of accidental activation are much lower and easier to reject.”

Arguably HUD is a solution looking for a problem. Like the inspired Seque cycles, the inventor’s goal was to develop an urban consumer transport vehicle but he failed to get significant adoption. The Segue has now found a home with urban tourist touring groups and airport police. Why? It provided an elevated view with minimal multitasking: Ideal for tourists and law enforcement.

Andrew agrees: “What these technologies really need to address is what sort of “problem” they are trying to solve.  That is, with capacitive touchscreens, there were certainly a number of value propositions that arguably were superior to the previous (resistive) solution that helped transform/enable touch input.  Natural gestures (HUD) is still looking for a compelling value proposition”

Google Glass is a platform without a certain home. Will “super cool” it has not inspired the consumer. We have not seen the “a-ha!” that Jobs brought to the touch. We know new more intuitive human interfaces are coming. But we need a Steve Jobs to take the technology and humanize it for intuitive consumption.

Gary Schwartz is the CEO of Impact Mobile. Having been at the frontlines of the mobile industry for over a decade, Gary is the author of two books, “The Impulse Economy: Understanding Mobile Shoppers” and “Fast Shopper. Slow Store: A Guide to Courting and Capturing the Mobile Consumers,” both of which highlight the current state of the mobile commerce space and chronicle the significant impact that mobile is having on consumers, retailers and brands. Gary is also a chair emeritus for the Interactive Advertising Bureau and the Mobile Entertainment Forum NA and global director of the Location Based Marketing Association.

SCREEN WARS (Digital Media Forum Keynote 2013)

In Dubai talking to agencies and brands about “digital velcro”. How linking content seamlessly between one screen plus other consumer screen equals a multiple of value for a brand.  20  mins – view here.

Samsung Abandoning Android? Tis Tizen?

The elephant in the room at the Galaxy S4 launch in New York last week was Android. Not one word about the Google OS. Is all well in Camelot?

Some industry pundits such as ABI Research analyst Aapo Markkanen say there is a clear and intentional distancing of Samsung from its existing OS partner, Android.  Does Samsung want to reduce its almost total dependence on the platform over the next few years? Samsung seems to quietly be building independently on top of the Android OS and may make a jump to a more neutral industry partner by 2016.

Window’s has not offered a compelling alternative to Android. What are other options?

Mozilla (Firefox) and Linux (Tizen) are going head to head to capture next generation developers with their web-based operating system. The Firefox and Tizen SDK and API allow developers to use web-based HTML5.

Tis Tizen

Tizen (a Linux Foundation initiative) may have the edge. Tizen is an open source, standards-based software platform for multiscreen devices (smartphones, tablets, netbooks, in-vehicle devices, and smart TVs). Like Firefox, it provides a cross-screen environment for application developers, based on HTML5.

Samsung abandoned its homegrown smartphone OS,  Bada, early this year and announced that it would start developing Tizen-based devices: ”We plan to release new, competitive Tizen devices within this year and will keep expanding the lineup depending on market conditions.”

The implications are significant to the connected screen economy and place application development in a more mature web main-frame on the device. The application store now can exist in a more manageable web environment with bookmark apps and not get lost in widget-design interface promoted by Apple.

Premature Technology Arousal (PTA) in Barcelona

To sum up Mobile World Congress 2013, I will borrow from Peter Marx, head of business development at Qualcomm Labs. Peter talks about a tendency for PTA or (for those in the know) Premature Technology Arousal in the mobile industry.

Much of the MWC 2013 floor area at the new Fira Gran Via venue exhibited PTA or Premature Technology Arousal. Solutions that are excited about being solutions. Solutions that are too early. Solutions that are missing reach and frequency. Things that are just not simple enough to drive adoption.

Even before 70 thousand executives hit the show floor, there were signs of “PTA”. From the Near Field Communications (NFC) show name tags that tried to emulated plastic (but that few used because you still needed to show the plastic) to tapping on Coke dispensers with cloud-base wallets that are many quarters away for mainstream adoption.

Booth after booth in this 1.01 million square feet techno-playground displayed incredible solutions and screens.  But the real story to follow was how each solution quietly added value to a given business ecosystem. There was an invisible hand playing connect the dots. Here are a few examples:

The Invisible Google Hand

Google was almost absent – unlike the MWC of 2011 and 2012 where Google groupies ran from partner booth to partner booth in search of cute Android pins. But Google was most definitely on the floor. This year the company is wisely playing “powered by Google”. They are the dark silent type. Turn left or right in every hall, Android is the fuel this industry is consuming.

The same holds for Qualcomm. They are the chip manufacture that is quietly taking the lion’s share of the revenue on each global handset. (Intel just cannot seem to create a competitive landscape.) Qualcomm Labs is building in consumer identity and credentials onto its “platform” hoping to not only power the connected device but also own the big data behind the user. When Qualcomm demos a vision of a home of the near future, they power many of the moving pieces.

The Samsung Show

While Qualcomm’s chip and Google’s OS were the main stories in Barcelona, another key and not so silent player is Samsung. (So much so that my hotel concierge asked me if I was attending that “Samsung” show that was in town.)  The word that floated above the white new-age Samsung booth was “innovation” but the innovation is not just the 3D camera or the ubiquity of the new S-Pen. The innovation was in their business model connecting their screen across the consumer journey. The 3D camera sells their tablet and television. The S-Pen and its SDK allows for ergonomic continuity across their new tablets and fablets.

Mozilla is other important story in Barcelona. Using the Firefox browser on lower-end ZTE devices to run the camera, map and  . . . oh ya the browser was a definite tech-turn on. Moving the developer and more importantly the consumer out of the (Apple-invented and dominated) app store into the real world super-app is an inevitable step and fundamental to our mobile evolution. The quicker the industry can move away for relying exclusively on industrial design and the app storefront as the sales tool, the faster we will grow.

2014 Screen Wars

The most important leitmotif was the screen. Not only the proliferation of devices with new form factor and appliance, but the realization that it is in the connecting of these screen that the we can accelerate business models. Samsung, ZTE, Motorola, Nokia all address the consumer journey across all screens and throughout their day. Nearly all marketing VPs had spent their last few months and budget trying to tell this consumer story.

Again while many products had indecent “PTA”, the most important insight was not what was happening on the screen but what companies were doing to connect them seamlessly. The new battle ground this year moving into MWC 2014 will be centered around who can best manage big data, wallet credentials and identity between the screens.

2012 “Top 10″ MOBILE trends & rankings

In a season where every second tweet and Google+ post is a look-back or forward at the “mobile” year, it is sometimes difficult to navigate all the insights. Just Google “Mobile Top 10 . . .” and you will find every blogger, publication and pundit providing their vision of what is newsworthy, trendworthy or simply rankable in mobile.

The proliferation of mobile “Top 10s” is a good thing. It is proof of all the many verticals areas that mobile has intersected. Health, IT, gaming, banking, retail, social, payment, fraud, security, privacy, patent trolling – all now have some form of a top 10 mobile list.

The explosion of the Mobile Top Ten list shows both how disruptive mobile has become and, also, what a massive audience it commands. Top 10-related news headlines seem to drive more hits and be retweeted more than other items.

(Twitter even tweets its top 10 most retweeted tweets. No. 2 being @LilTunechi Lil Wayne WEEZY F “aaaaaaahhhhhhmmmmm baaaaakkkkkkkkkk.” The 2011 top of the top 10 retweets was by Wendy’s restaurants “RT for a good cause. Each retweet sends 50c to help kids in foster care. #TreatItFwd.” Wendy’s raised $1.8 million.)

We seem to all need, what Perry Hoekstra in his blog calls an “Obligatory 2011 Top Ten Mobile Story List.” I presume it helps us simplify this expanding, complex world of mobile. If we can prioritize importance and cut off the discussion at ten, the list can help us make sense of mobile.

Lists also keep us honest. We never are right on half our predictions. Remember our 2010 trends and forecasts? They are out there in the blogosphere for all those with 20/20 hindsight to chuckle.

In December 2010, we all had an opinion on HP webOS-powered tablet plans – that never happened. And where on the list was Google’s purchase of Motorola, ostensibly as fodder for their IP wars against Apple? Missed that one.

But the biggest problem with these lists is their length. In the wasteland of eggnog and turkey dinners, I ask, where are the mobile Clifs Notes? Where is the super-list? Where is the list of lists?

Steve Yankovich, head of mobile for eBay, tells us that the mobile consumer’s attention span is roughly 15 seconds, nose-to-phone. EBay designs its mobile pages for the dwell time at an elevator or the time idling at the red light. Perhaps, in the spirit of mobile efficiency, we better design our year-end prognostications for this small attention-deficit window.

This holiday season, I add to the litany of lists my retrospect and forecast of mobile, but below in 100 words or less.

2011

  1. Facebook timeline set it back in history
  2. Google+ builds circle of trust
  3. Symbian sang swansong
  4. Apple continued to make data a free commodity
  5. Spectrum war showed its dark side
  6. All screens called mobile
  7. Amazon cloud disrupts the mall
  8. Consumer data became jewel in the crown
  9. CarrierIQ poster-child of privacy angst
  10. NFC wallet traded press, not payment

2012

  1. NFC proximity marketing, not proximity payment – yet
  2. X9 (ISO) delivers mobile security recommendations
  3. Apple says “I Do” to NFC
  4. Research In Motion exits stage right
  5. Cloud checkout is optimized and mainstream
  6. Super App = HTML5 browser
  7. Focus on prepaid market services
  8. LTE networks and spectrum become big issues
  9. Mobile privacy hype continues
  10. Microsoft and Nokia enter stage left

Not quite 100 words, but near enough.

Happy Mobile New Year.

Thank Google-ness, it’s Windows?

Mobility is a treacherous business. Cell phones are only used for an average of 18 months before dropping down toilets, falling off tables and general m-wear and tear drives the consumer back into the buyers’ market. For fashion and trend conscious consumer this cycle is probably shorter.

And mobile folk are a fickle bunch. While some may be diehard Apple or Android loyalist, the masses swap out the old for the new in rapid succession. Every time a screen smashes, a phone drops down a manhole or domiciles in a taxi (without you), there is a fresh opportunity for a handset manufacturer and an operating platform to woo the new shopper.

Of course, there is some stickiness. Consumers may be faithful to their Blackberry Messenger group, or the slick screen design of their iPhone, or the price point of their Android.

But when shoppers are back on the market they tend to be unchaste.

It is not just the device. Apple has made sure that the consumer is well aware that if you want APPs there is only one show in town. Developers worldwide are working hard to make sure that Apple and Android operating systems can market themselves and sell based on access to this wealth of content. RIM’s Playbook and Hewlett-Packard’s webOS never stood a chance without a preexisting content library.

Apple’s marketing genius continues to position the iPhone as the aspirational handheld, while Google’s open-android operating system on Samsung, HTC and Motorola continues to drive the bulk of smartphone sales. With Hewlett-Packard declaring defeat and Blackberry (RIM) and Nokia under siege, it looks like a two-horse race.

But recently things have become a little difficult to track.

Apple starts an IP war, litigating against Samsung and winning a stay of the release of its new TAB in Europe. Google’s purchase of Motorola, ostensibly for IP, suddenly makes the giant of the cloud a retail-bound cell phone manufacture.  This forces Samsung and HTC Android loyal hardware partners to reevaluate their strategy and possibly further diversify their operating systems.

The pressure of the looming patent war and Google’s unsettling role in the device market is opening the door for Windows reenter the market.

With the salvo of recent litigation, the Android operating system is turning into a liability.  Open-Android does not come bundled with an indemnification for the hardware vendors. Windows Phone’s does! This is becoming an expensive problem for handset manufacturers. Samsung and HTC have found themselves paying for their own legal defense and paying on any ensuing royalty settlement.

In the background, Windows has been quietly enforcing its IP rights globally. Goldman Sachs analysts estimated that Microsoft will make nearly half a billion dollars in 2012 on royalties of $3 to $6 per Android device from vendors like Samsung and HTC.

Microsoft’s “Android” royalties is estimated to be TRIPLE the revenue earned on Windows Phone licenses.

With all these mobile machinations, Windows 8 may be powering your new device pick on the shelf next year. Microsoft may allow for light computing with more robust utility than Apple or Android.

Next time your phone-du-jour falls down the stairwell or is left at the bus station, a phone powered by a Windows operating system maybe the new flavor of the day.