Convergence, Enterprise AR, and Mapping the Inside World with Charlie Fink

How will the convergence of 5G and AR and artificial intelligence lead to the creation of experiences that no one’s ever imagined? Join Forbes columnist and author Charlie Fink to find out!

Alan: Hello everybody. Today’s guest is one of my good friends and someone I consider a mentor. XR consultant, columnist, speaker, and author, Mr. Charlie Fink. Charlie Fink is a Forbes columnist and an author of two AR-enabled books – Charlie Fink’s Metaverse: A Guide to VR and AR, and Convergence: How the World Will Be Painted with Data. Charlie is a former Disney, AOL, and AG Interactive executive, famous for coming up with the idea that turned into The Lion King. Charlie was EVP and COO of VR Pioneer Virtual World Entertainment. He was an SVP of AOL Studios and president of the American Greetings Interactive. Charlie founded and exited to venture-backed startups and has produced over 30 award-winning Broadway musicals. Charlie is leading the way in XR for business by covering and reporting on everything XR-related. With that, let’s welcome Charlie Fink.

Charlie: Wow, I don’t know if I can live up to an introduction like that. Thank you Alan.

Alan: Oh you’re amazing. You’re amazing and thank you so much for being on the show. First of all I want to congratulate you on the launch of your new book Convergence, an AR-enabled book about AR. Maybe you can tell us a little bit about the book, and why it came to be.

Charlie: Well, we were looking at doing a second edition of the first book which sold very well, but it soon became apparent that the AR topic in particular had been sort of glossed over in the first book, which was largely about VR. So, I started to think about the idea of doing a new book, and I got a lot of support for it in the community. In fact, it is a sponsored book, meaning I got enough donations that I could afford to write the book and get it printed. It’s printed on premium paper and it’s priced high like a textbook at fifty dollars – of course it’s filled with an hour of animation – so it’s not a normal book and it doesn’t adhere to book economics. So to make it work I really needed to build a community of about 100 people around the book, many providing augmented scenes and many actually contributing thought leadership and reporting. The topic of the book kind of evolved because I, you know, originally I was thinking the title would be something like ‘the many modes of AR,’ or ‘how the world will be painted with data,’ which is a phrase I coined, which suggests a fully-scaleable AR cloud. Which would make the devices we use a whole lot more useful and interesting than they are today, where we use something called marker-based AR.

And as I got into the book and editing the work of my collaborators, it was clear everyone was talking about the same thing, which is this world of ubiquitous, wearable computing. What the final form factor will be we can debate, but this idea of a magic-verse based on a mirror world became the theme of the book, which is essentially the convergence of 5G and AR and artificial intelligence that would manage all of this data for you so that it is wanted and contextual. Otherwise, if it’s interrupting you, I don’t see how that’s possibly a welcome augmentation, right? It can’t be showing you advertising; it has to show you what you want when you want it or where you need it. You don’t need to see the weather in front of your eyes spatially, although it would be fun from time to time. But the truth is you know what the weather is.

So that became the title of the book, Convergence. And if you look at history you know there have been a number of moments of sort of platform innovation that launched heretofore unconceived ideas. So if, for example, look at GPS in a phone, right, there would be no Uber, no Seamless. There’s a whole range of services, and GPS is getting better now that it’s being married to computer vision, so you can get highly-localized Google directions – really accurate, down to a couple of feet.

So convergence is beginning, right? The combustion engine is an example of convergence. What did it enable? Look at steam: it enabled factories, it enabled transportation and railroads in the same way that GPS and the Internet has essentially created five of the biggest companies in the world. So that’s the topic of the book. And of course, it’s filled with augmentations, as opposed to the first book, which had kind of a funny cartoon series that was made by living pop-ups. This has 50 augmentations, mostly from the sponsors. There is a little living pop-ups layered in just because it’s so funny and so incredibly well-done.

Alan: It is amazing.

Charlie: On the topic of technology, so, it’s great to work with them and all the collaborators. We made this book together… although, of course, I had to do that horrible 12 weeks of doing nothing else and sleeping five hours a night in order to ultimately pull it all together with all the pictures; the book is beautifully designed. You know that’s an award-winning design that Porteramic came up with. So I’m feeling good about it, people seem to really like the book. It’s much denser than the first book. I think the first book was 40,000 words; this is 90,000 words. So I would say it’s the most in-depth book about what’s happening in AR and what the trends are today.

Alan: Awesome. So, Charlie, you know, your book really covers a lot. So it’s got in there everything from kind of social and social augmentations and things like this, but really, for the listeners of this podcast, we want to drill down into the business applications of this. In the book, you discuss military, telepresence, design and architecture, medical, retail, marketing, journalism, social media, entertainment, gaming, and education. Which of these do you feel right now holds the greatest business potential and why?

Charlie: Well, enterprise AR just in its infancy, but it transforms anyone who’s involved with either the manufacturing process or logistics, which is just about every company in the world. So AR is going to make the jobs that people are already doing much easier. There will be adoption issues. It’s quite inexpensive compared to other methods; it’s no more expensive than an RF gun, and it enables all sorts of new functionality like, as you mentioned, see what I see. So you can call a remote expert. And again, that’s an innovation that is being driven by the camera. You can also do that on a pad or a smartphone, and of course the use of the camera and the digitization of real environments spatially, you know, is going to enable a whole range of functionality. They’ve already got, you know, in your Apple phone, RF detection is already built in. That’s the beginning.

Alan: Interesting. So then I think one of the questions that people are going to have is, at what point do we go from a mobile device – so, a phone or a tablet – and then go to glasses, and where kind of is that divide, and is there a solution that basically bridges the gap between that, and what do you see as the uptake of glasses versus devices such as iPads?

Charlie: There are two things. One is, glasses allow workers to keep their hands free. Glasses allow workers to see schematics and the handwork in the same field of view, so that many fewer mistakes are made. When you’re wiring a jet, mistakes are…inspection and correction takes as long as the actual wiring of the jet. So you’ve got companies like Boeing and Airbus saving tens of millions of dollars per jet – I don’t know, Boeing makes 3,000 jets a year – so that’s quite extraordinary in terms of the amount of savings, and they have a system that they can use themselves and change out the schematics. The workers can take a picture of their work when they’re done, so that if a mistake is found, the picture could be reviewed, or their picture could be reviewed in real time.

Alan: So interesting.

Charlie: Many companies are adapting it for that reason. But, you know, companies should select the right tool for the job. Is this a job for Google Glass? Is this a job for the Hololens? Or is this something that we can just have a worker do on a smartphone?

Alan: It’s interesting that you talk about these different headsets. And for the listeners who don’t understand the difference between them, maybe you can walk us through kind of the beginning of it. So, you know, maybe a phone-based, right up and through to something like the Hololens, and kind of what are the differentiations between them?

Charlie: Well, first let me set this up by talking about the difference between AR and VR.

Alan: Sure.

Charlie: VR has a lot of fantastic applications in training and simulation. That is its most powerful use. And it is quite effective at those things. Of course it also has telepresence, so you could be virtually present in a place like Egypt – not in real time, but in space and simulation. And a lot of companies – anybody who’s got vehicle-based businesses, machine businesses – there was a fantastic example, up South by Southwest. The Raymond Corporation, which makes forklifts, has created a simulator module that you can clip onto a real forklift. You can have that in your warehouse, and whenever you’re training a new worker, you have them spend a few hours on the simulator.

Alan: That’s awesome.

Charlie: So that’s what VR is good for, right? It fully includes the world and creates a new reality in which you can be present. So that’s a very powerful quality that it has, and again, it’s a platform and people are innovating. But the success seems to be, right now, limited to both medical applications and training applications. There are some games and social experiences. I mean, there are millions of people going into Rec Room. So, it has some social benefits, you know, because that’s not a game – that’s an experience. Not to say that games can’t be experienced. But in my opinion, I don’t think the killer app of virtual reality is games.

But let’s now switch to the topic of augmented reality, which is really what this conversation is about. It’s much more broadly applicable to business than VR. They’re both technologies that that share some things in common, but largely are separate. Because you need the real world in order to augment it.

Alan: Absolutely. What you’re saying here is virtual reality is great for kind of those before you’re on the site – so, training, and then maybe remote collaboration for design, maybe? And then AR is more on the practical side of your everyday work.

Charlie: Yeah that’s one way to think about it. In other words, think about it on a spectrum of immersion… although that doesn’t really make sense. Let’s start out with the basic device, right? You can do a lot of the things we’re talking about – see what I see with remote experts, documentation, geo-located instructions – you can do all that with a smartphone, right? That’s cheapest, because the worker already has a pad or a smartphone with them. So no retraining required. This is not a worker who necessarily has to have their hands free in order to read the information that they need.

Charlie: So the cheapest and the simplest is, of course, in many cases, all that’s necessary. You have to look at the segment specific-elements of it that can be augmented. Not everything in the business process is going to be benefited by augmented reality. Of course, anybody who has a warehouse can be hugely benefited, because you take the functionality of that pad, the barcode, the RF gun, and you put it all in a head-mounted display. So that’s actually a fantastic use case, and there are a lot of companies focused on exactly that: warehousing and logistics. Because augmented reality is so effective, and they use something called the monocular micro display, which is really a tiny TV set hanging out there in your peripheral view, and you’re mostly looking past it, but you can focus on it when you need it.

Alan: Kind of like heads-up instructions.

Charlie: Yeah it’s Google Glass. You know, there are a lot of companies that probably people on this podcast have never heard of, nor do they need to know their names, that make monocular micro displays. Some of them, like Vuzix, are as popular as Google Glass. They make something called the M300, which is very popular. So, the next kind of augmented reality is called Reflective AR, and this is where you’re using your smartphone, but you’re balancing the image or the information on some kind of projection surface in front of your face. And that is actually similar to the way the Hololens works, except the Hololens is filled with sensors, and is aware of your environment, and it has advanced optics which can place 3D objects in your field of view.

Alan: So give us an example of why that would be important, where you need to be contextually aware and why you would need 3D data.

Charlie: Let’s go back to the wiring of the jet.

Alan: OK.

Charlie: And I’ll talk about something that Lockheed does. You put the Hololens on, and it can place on each peg what wire goes in it. Why can it do that? It has much better, more sophisticated sensors. So that’s a better tool for that job in a certain respect. But it’s much more intrusive, it’s heavier, it’s harder to wear for a long period of time. Whereas a monocular micro display can clip one to your helmet.

Alan: Interesting. So give us an example of, you know, maybe some companies – you said Lockheed Martin is using spatially-aware headsets to kind of overlay instructions and data on top of the real world. Give an example of maybe a company that you know of that’s this monocular display. You talked about Vuzix and I think there’s you know there’s other–

Charlie: Vuzix is one. Kopin is one, but they made a reference design that they’re hoping other companies like RealWear will adapt and manufacture and market.

Alan: Maybe describe RealWear to everybody.

Charlie: RealWear is a company that is associated with Kopin, and uses their reference design to create a monocular micro-display. But they house it… their trick is the way they house it, is industrial-grade. So it clips on to, for example, work glasses. Let’s say you were doing welding: you know, it incorporates easily, those kinds of practical uses.

Alan: I’ve actually tried it at CES, they had it in a work helmet.

Charlie: Yes, I saw that too, and also when they walked on stage at AWE last May, one of the guys was in a hazmat suit.

Alan: Interesting.

Charlie: So it’s a very versatile, rugged device that they made using somebody else’s technology, but their industry knowledge and their understanding of the customer needs.

Alan: It’s interesting, because I think we’re going to see a lot of this, where you have hardware manufacturers like Kopin, and for people who don’t know, Kopin a display manufacturer that makes micro displays for these types of glasses and headsets; I believe they make parts for the for the Vuzix Blade, and they make different parts for different people’s glasses. But there’s gonna be the makers of the of the actual hardware, but then there’s going to be solutions providers, like companies like Upscale, where they’re actually providing kind of the software back-end and other companies–

Charlie: They are Reflect. There’s a whole class of companies that I call “consulting integrators.”

Alan: Got it.

Charlie: So they’ve got a platform that they’re going to leave behind them. They’re going to work with the client to identify the need, the right tools for the job, and then they leave them with a SAS platform that’s object-oriented, meaning you don’t need to be a programmer to update the app.

Alan: Absolutely.

Charlie: Of course, then they get recurring revenue, and that’s really what everybody wants in media today.

Alan: So maybe you can, in the interest of the listeners and getting straight to the point of value creation, what are some of the companies – you mentioned a couple there – what are some of the companies that are offering this service? Because I know coming up in other episodes of this podcast we’ll be interviewing these different companies, so what are some of the companies that you are seeing that are–

Charlie: Depends on the vertical, right? They’re sort of dividing it up by vertical.

Alan: Got it.

Charlie: So you’ve got a company like Visualix out of Germany that is specifically looking at geo-location in warehouses; micro geo-location. So it’s using edge computing, it’s identifying a box down to six inches. So you’re driving the forklift and it’s giving you directions to the box and everything is automatic. And it tracks the changes in the environment. So that’s one example. You have Reflect, which is focused largely on the industrial aftermarket and large equipment and integrating CAD. Occipital does that as well. Scope AR, you know, is a broad integrator, like Upscale, that has clients in a lot of different verticals. And then of course the big daddy that really doesn’t get talked about as much is PTC, which is a 10-billion dollar, publicly-traded company focused, I would say, probably 65 per cent, two thirds, on augmented reality. And then they have a legacy CAD business that has different forms for them, and still is a lot of their revenue. Right? But a lot of what they did was say, ‘why can’t we bring these CADs into the real world?’ Think of the application for architecture and design on-site. To see their buildings in place before they break ground.

Alan: Interesting. We’ve talked a lot about kind of the industrial and enterprise applications, but you were just at South by Southwest for the launch of your book. Can you maybe talk about some of the the marketing applications, or some of the things that you saw there, that were maybe not as industrialised for listeners that are in marketing or in sales or in training or HR; what are some of the other aspects of this technology that are being used that you’ve seen?

Charlie: Anything that uses your camera is inherently social. The way companies market today is they get their customers to do it for them on social media. Obviously they do microtargeting and use Facebook and Twitter and Instagram to try and find people in context, but they also use different strategies for getting people to make content for them. And this is an example of the camera as a platform enabling things that were heretofore impossible.

Amazon Prime has a series coming up called Good Omens, which is about angels and devils among us in the days before the apocalypse. And it’s a comedy. And they set up a compound that people waited in line hours to get into, and it was sort of a Garden of Earthly Delights, if you will, where you could post it in different stations and take selfies, and you uploaded them to get prizes. They had a big leaderboard of all the social posts that were being made, and they made it as visual as possible. Including when you left, having a machine that would take a selfie and post it for you. And then, they were walking all around, had costumed characters as angels and devils all walking around the convention center, for some experiential marketing. It was fantastically well-done activation, it probably cost millions of dollars.

Alan: Is this for a new TV show, is that what it’s for?

Charlie: Yeah it’s it’s a series on Amazon.

Alan: Amazing.

Charlie: But companies don’t have, you know, TV commercials don’t really do it today, and they’re still very expensive, so that was a way to take 80,000 influencers, and people were active – you know, generally people who go to South by Southwest are thought leaders of some sort in their own social circle.

Alan: This is true.

Charlie: So you see a lot of big companies doing activation. There are always the Vox Media, and Vevo, and Dell, and Deloitte all have, you know, big sort of houses that they rent, both by the convention center and over on Rainey St., which is across the street from the convention center. There is a street full of bars, you know, a lot of little houses that were converted into bars, and those get rented by companies – probably, I’m going to guess, for about $25,000 a day. And they have events there, and they host parties there. Everyone is trying to, you know, get in front of the people who are there. And some of them are really popular, and they’re free, right? People wait in line to get their drink tickets and hear a band. The other thing about Southwest is you do hear a lot of amazing music by accident. I was at Flatstock, which is the paper print exhibition – posters and big bands. So it’s called Flatstock. And there was a Flatstock stage, where there was a band playing: Jonathan from Croatia.

Alan: Okay.

Charlie: He was really pretty good. So, those are the delights of South by Southwest. Those things that happen accidentally that weren’t on your schedule that you just stumble into.

Alan: It’s interesting.

Charlie: Brands will that too to establish themselves. Now, you asked about specifically marketing applications for AR, because all this started with with me saying it has to be social. So, Burger King has an app where it’s doing computer vision to recognize the advertising of a competitor, and sort of deface it in an amusing way.

Alan: I actually posted that the other day and it got 50,000 views in two days.

Charlie: Cool. So it’ll get out and, of course, people are using it socially. You know, it’s on their camera. They’re record it.

Alan: Really a fun one. I love that.

Charlie: They think it’s cool. So that’s a tremendous amount of marketing on Burger King’s behalf that is being done by its customers.

Alan: Absolutely. It’s interesting that you brought that one up because, you know, of all the crazy things we’ve seen people do with AR from, you know, selfie images and all of this, the AR – and for those of you who don’t know, Burger King, what they did was they created an app so that if you pointed it at their competitor’s advertising, whether it be a billboard or a poster or whatever, it burst into flames and gave you a free Whopper, a flame-grilled Whopper.

Charlie:   They’re using computer vision. It’s a really good application.

Alan: So smart. I’m assuming they’re just looking for the logos of their competitors and then blowing them up. So what a great–

Charlie: Or they are, you know, texture-mapping or depth-mapping the images so that they can trigger the hyper location, right? They know where they are physically; they need a kind of mesh to recognize it, if they’re not using a marker of some kind, or the logo. Some logos will scan better than others, depending on what technology they’re using.

Alan: Absolutely. So let me ask you a question: of all the things that you’ve seen – in your new book here, you’ve gotten military applications, you’ve got retail applications, you’ve got – what do you think is that so-called ‘killer app’ of augmented reality, what is that? And you know it’s funny, because I wrote an article about augmented reality’s first killer app, but I’m interested to know what your thought is on the first killer app of this, in business.

Charlie: I don’t think we…well, I mean, in business, remote experts and work instructions and wayfinding in warehouses are the three really big ones.

Alan: So maybe let’s let’s unpack that. So the first one is remote instruction.

Charlie: Well that’s where you, remote instructions, right? That was my wiring diagram from earlier.

Alan: So one of the things that–

Charlie: Any worker who’s looking at schematics can benefit from having the schematics in the field of view of the handwork.

Alan: One of the things that we saw early on was kind of a ‘see-what-I-see,’ being able to call in an expert and–

Charlie: Yeah, take a low-skill worker and turn them into a high-skill worker. So, you know, the low-skill workers is on the factory floor; the high-skill worker is in the office. So call your supervisor. He doesn’t have to walk 10 minutes to find you and, or, you know. So the see-what-you-see is powerful and useful.

Alan: I agree.

Charlie: And also the ability to document the work in a hands-free way.

Alan: That’s very important. So shifting gears slightly, you know, and this is interesting, it’s in your book as well: A recent study commissioned by Microsoft in the Harvard Business Review showed that 87 per cent of correspondants that they interviewed are currently exploring, piloting, or deploying mixed reality in their company workflows. What I want to know is, is this congruent with what you’re seeing and hearing in the field?

Charlie: At big companies that have innovation offices? It is true. And there are a lot of integrators out there. The big consulting companies that are evangelizing this in various companies, it’s starting to be something that they do.

Alan: So there’s, you know, the Deloittes and the Accentures of the world, is that you’re talking about?

Charlie: Correct. Correct. But they’re looking at, you know, because they have Fortune 500 clients. And their clients aren’t super cost-sensitive. Again, look at the savings that Boeing yielded working with Upscale. They liked them so much they invested in the company, because it was obvious to them how many applications there would be, and in fact they’re going factory-by-factory and figuring it out. But, you know, the pace of change in a company like Boeing is extraordinarily small…er, extraordinarily large! So that’s partly what the Innovation Office does, they kind of figure out what’s out there and try and evangelize it to colleagues. Sometimes on their own, sometimes with the help of the CIO, or just depends on how the company is approaching it. But I think the companies that Microsoft included in the survey were those large companies that have innovation offices. But, you know, you’re talking about, you know, one per cent of their workflow is being assisted by augmented reality. This kind of augmented reality that we’re largely talking about with micro displays are no longer, we’re no longer calling it ‘augmented reality’ – we’re calling that ‘assisted reality.’

Alan: Got it.

Charlie: The difference being it doesn’t go through the camera.

Alan: Got it. OK, so that’s just an assisted reality, a heads-up display. You know, interestingly, the person who wrote your forward for your new book Convergence, Jay Samit, he was the kind of chairman or–

Charlie: He’s still, right now, the non-executive vice chairman.

Alan: –Non-executive vice chairman of Deloitte Digital. And in their–

Charlie: Yeah, they deploy him when they need him.

Alan: Interesting.

Charlie: He’s like a weapon in their quiver, and he knows a lot of people. I would expect Jay is actually helping them with sourcing clients.

Alan: Absolutely. He’s a very, very well-known innovator.

Charlie: He also is a fantastic speaker and one of the nicest guys you’ll ever meet.

Alan: Yeah. I’m hoping to have him on the podcast.

Charlie: –who I used to work for years ago. He seems to be living a few years into the future, so it’s always exciting to get his perspective of what’s going on.

Alan: So speaking of looking into the future, somebody else who’s kind of looking to the future that you’ve been covering in your Forbes articles is Magic Leap, and their CEO, Rony. Maybe you can discuss kind of what their big vision is – because I know you’ve spent some quite a lot of time with them, and you’ve written several articles about their vision for the future – and how do you see them relating to kind of the business applications of XR technologies?

Charlie: Well, a few things about Magic Leap, just to describe it for your readers who don’t know: they use a multi-planer system to place objects in-depth. That is its most unique aspect of the optical technology that they have, right? They know how far away things are. That’s part of the beauty of their sensors. They haven’t yet–

Alan: Be honest; when you first put it on in and you hear a sound come from behind you, and it sounds like it should be 10 feet away, and you look and something is 10 feet away – it’s incredible.

Charlie: Right. You place digital objects and they remain anchored where you’ve placed them. It does games where they, you know, anchor on a wall and then things come through the hole in the wall. It’s a very, very clever system that took them over five years to pull together because there were so many issues that had to be solved. Along the way they raised over 3 billion dollars, which is quite remarkable since most of their plans were secret.

Alan: Let’s just kind of… you know, 3 billion dollars they raised as a startup.

Charlie: Right, and they raised it. And part of the story there is who they raised it from. Google put in 500 million dollars. Once they did that, it activated a lot of large investors, including Disney and Warner Brothers, and AT&T, and the sovereign wealth fund of Saudi Arabia. They also have the largest investment companies in the world investing in them and they don’t expect to make money for at least 10 years.

Alan: Incredible. So they’re taking the long route.

Charlie: They’re taking the Amazon route.

Alan: Do you think that…

Charlie: Amazon is basically still not profitable.

Alan: Do you think 3 billion is enough?

Charlie: Probably not. But they’re getting a lot of traction, so they’ll be able to raise more money at a more realistic valuation because so much more will be known by the market. But I think they could go several years on the money they have now. Magic Leap a lot of great people running off doing different things. They’re in education. They’re pushing hard for enterprise. They recognize that those kinds of applications in the near-term, because they’re more expensive, are going to be the ones used most frequently. In the long term they see a consumer platform that has some powerful A.I. in it and can function well in a fully-5G environment. I think the beauty of Magic Leap is this whole idea of spatial computing, which you describe very well a couple of minutes ago. They have, the past several years, been evangelizing this vision. It hasn’t always been clear to people what the heck they were talking about, but their vision is a 3D world in which digital objects and information cohabit the world with us. So imagine walking down Fifth Avenue and everybody walks past you as a cartoon animal and every building is, you know, like Toon Town, just to use an example. Those are the kinds of apps that Magic Leap wants to enable. But you need ubiquitous, zero-latency wearable computing to do it. So that’s their vision. And they’re thinking, the final form factor they’re thinking is very light and not terribly different from the glasses you’re already wearing. But they’re also thinking, it’s not necessarily the device, right? It’s the layers. The Magic-verse is where the value is. And you might project that from a watch onto a projection surface. We don’t know what the form factor is going to be. And they’re hedging, because they know they don’t know. But they’re using the optical system and the infrastructure that they’re building to build this 3D twin of the world in which content can be realistically anchored and persist for the next person who sees it.

Alan: It’s interesting you mention that, because, you know, recently Kevin Kelly released an article on Wired called ‘Mirrorworld,’ talking about how we would have an exact duplicate of the entire planet, and if you look at what Google’s done over the last 20 years, or 10 years I guess, they’ve really taken a depth map and full, comprehensive view of the entire external world that we live in…

Charlie:  Google Maps maybe one of the biggest digital assets in the world.

Alan: Absolutely. And the one thing that people don’t realize is that Google Maps has a map of the entire planet on the outside; they don’t have the inside world. So I think there’s gonna be a huge opportunity for companies like Magic Leap, and maybe Apple and Amazon and Google, to capture a point cloud or a version – a mirror world – of every part of our lives, from the inside of our house, to our offices, to buildings. What are your thoughts on that, and companies like 6d.ai that are kind of pioneering this as well?

Charlie: 6d.ai is an interesting company. They have an augment in the book and we’re one of the sponsors; the CEO and co-founder of the company Matt Miesnieks, a former Samsung executive who also is part of a well-known AR venture capital fund, Super Ventures. In his travels, he became exposed to this technology and started thinking about this notion of using everybody’s depth data and point clouds, and stitching them together on the back end to create a digital twin of the world, that mirror world, that you could place augmented reality on top of a within. So there are two things that we’re really talking about, right? One is the mirror world, and the other is the AR which gets anchored to it spatially.

Alan: Interesting.

Charlie: And Magic Leap calls that the Magic-verse, or Ian Barr calls it the Wikipedia for the real world, and I call it a world painted with data. All the data, which will be like anchoring pictures of food where you take them right? So this is the end of the news feed, which goes by at 100 miles an hour. Instead, it will be the age of contextual content. So you want to take a picture of food and you’ll add a little review a friend who’s in the same restaurant a year later.

Alan: Incredible. Absolutely.

Charlie: So it’s pretty exciting. But we’re… again, a mirror world is talking about the twin, right? It’s talking about infrastructure. The AR cloud, or the magic-verse, or the layers that Magic Leap refers to as the content of the services that sit on top of that. Again, let’s go back to the combustion engine enabling, you know, the car and the airplane. So the combustion engine is the thing – the person who made the combustion engine didn’t think, this is going to make air travel possible.

Alan: No, absolutely.

Charlie: Right? But but it was extremely disruptive new technologies that, you know, like the camera, like GPS, enabled fantastic new services.

Alan: One of the things that I think is really intriguing is this idea that, once you have the magic-verse, or this, you know, the capture of the mirror world, who actually owns that three dimensional space? And you know, it’s interesting, we’re doing a virtual and augmented reality association meet-up on the legal aspects, or the looking through the legal lens of this technology, and you know something like the Burger King example where Burger King is using their competitors’ advertisings to advertise their products. And I mean, what are the legal implications around using three dimensional space, and will it be more… give more agency to the end user, so that they can decide what they see versus what advertisers want them to see?

Charlie: Well anything that you can see in public is public domain, right? You can’t say, the front of my building is so unique that you may not photograph it, right?

Alan: That’s true, but I can’t go over it…

Charlie: When you get inside a building, now I think you’re in a legal gray zone, right? Is the inside of a mall a public place? Or does the mall owner own that data?

Alan: Interesting. That’s a really good point.

Charlie: Of course, in your private place, there’ll be all sorts of security protocols that you can opt in and out of. But anybody who walks through your house can make a map of some sort, so that’s a very different matter, right? I don’t think people are going to be comfortable with other people knowing where everything is in their house. But any kind of a magic-verse is going to be governed by a filter. That’s where the AI comes in. In order to see the burning McDonald’s sign, you have to activate that filter, say, ‘yes you can show me that.’

Alan: Interesting.

Charlie: But it has to be a permission-based system. Let’s say, I mean, there’s going to be a ton of graffiti. There will be graffiti apps and, you know, graffiti layers, if you will. Maybe people will turn them on or off for entertainment. They’ll be in a park. They’ll see it’s there. Right? There has to be a detection system that tells you when content is proximate to you and ask if you want to see it. But it would be so many questions, it would be so intrusive. The machine has to know; the computer has to know. So that’s artificial intelligence, where it’s integrating what it’s learning with what it knows.

Alan: So, Charlie, you’ve mentioned a couple of technologies. You’ve mentioned 5G, AR, and now artificial intelligence. How do they all work together? Because I think what people don’t realize is that, as we enter this kind of exponential age or phase of technology where they all kind of coalesce together, you’re gonna have all of these different technologies working together, and you can’t really look at augmented reality without 5G, without IoT sensors (or Internet of Things sensors)…

Charlie: Right, and what you’re talking about is what the technology people call a stack; what could be in the stack? Computer vision, geo-location… sensors, sensors, sensors. It’s all about sensors. And what I would say is, a sensor does not need to be head-mounted, right? A sensor could be on your lapel, and you could be wearing some Bose spacial sound glasses that have your prescription in them, and it could be telling you what you need to know rather than disrupting your field of view. And they’re quite extraordinary, because you’re the only one who hears them, even though you don’t have your earplugs in. And they are spatial, as we were describing the Hololens, or the Magic Leap device; the creator’s edition, which is their developer edition. So the current version of the Magic Leap is $2,300 and it’s focused on programmers and pro-sumers.

Alan: Interesting. Shifting focuses a little bit to something that I saw on the Hololens 2 launch that, I think, is personally going to be a really killer app for businesses, and that’s telecommunication or remote communication; something like Spatial – the company Spatial –  that kind of came out of stealth recently and introduced this amazing ability to have virtual people in your space. So I could have a meeting with you in virtual space and you and I could see each other, interact with each other, look into each other’s eyes because it now has eye tracking. So maybe talk to the business applications around Spatial and what they’re doing.

Charlie: You’re talking about a new startup called Spatial, which is well-funded and staffed by big technology company veterans – although they’re still quite young – and it creates an avatar for the user, no matter what device they’re using. And the avatar is generally based on your social media presence. So, they built my avatar, when I went to visit them, out of my Twitter picture. And then they use a real-time AI to kind of predict, based on that picture, what the rest of you looks like.

Alan: Amazing.

Charlie: So, a very uncanny resemblance – although it’s not photographic, it’s graphic. It’s cartoon. Just like Facebook does something similar in Facebook spaces for virtual reality.

Alan: But is it creepy? Because you mentioned uncanny, and for those of you who don’t know, there’s a term called…

Charlie: No, it’s not like, you know, those realistic Hanson robots, whatever that–

Alan: Yeah, what’s her name?

Charlie: That one is a little creepy, but it’s a really dumb person. They’re trying to make Robbie the Robot, right? Or that alien helper, or the cyborg helper in the Aliens movies. That’s never going to happen. That’s a technology… that’s alien technology. We’re so far away from that, you know, my grandchildren won’t see it.

Alan: So do you think Spatial, the solution that they’re providing, where you can have collaboratively–

Charlie: It’s very compelling. I mean, you do get a genuine sense of presence, no matter what device you’re using. Obviously it’s best in a more immersive headset like the Magic Leap or Hololens But even people who are using their pads, know, feel like they’re there. And of course, you then have the point-of-view of your avatar; you’re not seeing yourself. You can go into 3-D mode, but typically you’re yourself. You’re present. And it uses the tone of your voice to create gestures, and is generally pretty good as a remote collaboration platform. You could bring in YouTube, you could move the screens around, you could bring in 3D objects, you can walk around them, you can walk closer to someone. So it really does have a very, very strong sense of presence, especially for an augmented reality device where you’re still anchored in your home or office.

Alan: Do you think it’s going to replace travel for business? Or not replace, but maybe decrease–?

Charlie: It will augment it.

Charlie: What we’re seeing, right, is that it’s slowly creeping in to apps we use every day, such as social media filters, and people don’t say that it’s augmented reality. They say it’s face filters. They don’t say it’s facial recognition filters. Those are words that consumers don’t need to know. I mean, every extra word is explaining, and explaining is bad.

Alan: So, I know you, I know your opinion on this, but since you just hit on it, what do you think about this term ‘XR,’ as a catch-all phrase for virtual, augmented, and mixed reality?

Charlie: I think it’s useful, to be honest. I have my…objections to it, which is it just conflates all these technologies, which is super unhelpful, because there are many, many modes of AR. On the other hand, if you’re writing about it, or writing a book about it, it’s convenient to say XR instead of AR/VR. And is AR/VR even that accurate? I have been railing against it, because I really think there’s so much misunderstanding out there. Another XR, or another ‘R,’ I should say, and now we’ve included diminished reality, and as I said earlier, assisted reality. So I don’t like all these R’s, I don’t like ‘XR,’ but I’m done fighting about it? Ori Inbar, in his keynote at AWE 2018, said, ‘go XR or go home.’

Alan: Yeah, I read that in your book, actually.

Charlie: Once he said that, I mean, I kind of lost. I know other people feel as I do, so I’m not alone. I think I’m in a distinct minority, but it’s over. There seems to be a rear guard action I may have ignited. But that’s just going to lead to even more confusion.

Alan: Exactly. So–

Charlie: I don’t know about this, other than this is all entre nous. This is all inside baseball. Most people don’t even need to know that we’re arguing about this because, all they care about is what it does–

Alan: What it does for them, and I think that leads me into, kind of, the last part of this interview, which I really want to thank you so much for taking the time to enlighten our listeners here about the different business applications and what’s coming. The question that I have is, then, what do you see for the future of virtual, augmented, and mixed reality, or XR, as it pertains to business? Where do you see as, kind of, the ultimate future of this technology?

Charlie: We’d have to look at the development of the personal computer, which took 20 years. For a long time, people didn’t think they applied to them at work, and eventually you had to have them. And then people started to learn about the Internet and e-commerce, and they said, oh it’s it’s got an application that is for me personally. And don’t forget, in the mid 90s people started to feel that if they didn’t have email, and if they didn’t have a website if you were an entrepreneur, that you didn’t exist, or you feared you wouldn’t exist. So that was creating a lot of FOMO. But I think part of it was driven by email, and part of it was driven by e-commerce, and both of those enabled by consumer access to the Internet, through dial-up at first and then through broadband, which really came around very quickly; you know, by the end of the 90s, everybody had a cable modem.

Alan: It’s interesting, I was just mapping kind of the–

Charlie: By the way, the end of the 90s was 20 years .

Alan: Yeah! Oh my goodness.

Charlie: So this could easily take 20 years. I know people are saying five. And you know, we’ll see. But it’s going to take some convincing to get a lot of people wearing a head-mounted display. I don’t know what the benefit is yet, that is powerful enough to get people to do that.

Alan: I agree. I haven’t seen it yet either.

Charlie: But in business, of course, it’s a tool. You just take it off when you’re done. It’s not… I will also say this, both about consumer AR and AR for industry: AR is not a thing to like or dislike, or do or not do. AR as a tool. AR enables tools. AR augments tools, like the personal computer that has the schematics on it; well, AR would would make that a little better. You look at consumer apps like Google Maps – we talked about the combination of computer vision and geo-location, which enables local directions, walking directions, to be much more effective than they are right now, because it can tell you where to go but… it could tell you where you are, but not how to get there. So that’s going to change. Will people walk around saying, oh my god, computer vision and augmented reality have made Google Maps much better? Or will I just say, this is better Google Maps? Yeah that is cool.

Alan: Exactly.

Charlie: Because that’s how much thought they’re going to give it. So, you know, that’s an example of, you know, augmented reality taking things we’re doing everyday and making it better, and that’s how it integrates itself into your life. And so I think most of the growth of augmented reality, both in enterprise and on the consumer side – which will take much longer – you know, is going to happen without people calling it anything. It’s just going to be better computers, and better tools, and better apps. So augmented reality is a quality, or a tool that apps use to be better. So I’m not sure that there’s going to be a big parade for augmented reality.

Alan: Definitely not.

Charlie: It’ll be a big parade for better Google Apps.

Alan: It’s interesting…

Charlie: Again, a lot of this conversation is baseball. If you’re not in the industry, you don’t care.

Alan: It’s interesting because, you know, Sunder Pichai from Google said, you know, the idea is that the very idea of the device is going to fade away. So it’ll be less–

Charlie: Invisible computing, that’s true! The device becomes your glasses; you’re not even aware of them being special. That’s certainly something that our grandchildren will see.

Alan: So, speaking of our grandchildren and looking quite that far out, one last thing that I think is an interesting segue into the future is brain-computer interfaces, and it’s not something that people are talking about, but I think, you know…

Charlie: Elon Musk funded the company to the tune of 40 million dollars that’s doing that. We do have implants, right? People who have epilepsy and Parkinson’s are getting implants. People have been getting implants in our hearts for 30 years. So we already have implants now and we have some in the brain. There’s still not a lot known about the brain; it’s usually to treat a disease. But, ultimately, you know, there’s a company working on contact lenses. My understanding is right now they can put basically a tweet in contact lenses. So they may be a ways away…

Alan: Kind of like the North glasses.

Charlie: Well, North glasses are just a reflective system that tells you what the weather is and…

Alan: In a tweet.

Charlie: …But, my problem with North – and I love the form factor – is just, are going to want to be interrupted like that?

Alan: I have a pair and–

Charlie: I don’t see the benefit.

Alan: –they are a little distracting. I was walking down the street and I got a message and thought, how cool is this? I was reading my message, and I almost walked into somebody because, even though it’s monocular – meaning one eye – both eyes kind of look towards it and I guess over time your brain, kind of, you know, can can do both things? But I had just got them, and I’m walking down the street, and I’m looking at this message and I literally almost walked into some poor woman on the street. So I can see this as being something maybe dangerous in driving, especially the first little bit, until you get used to that.

Charlie: Yeah, well, I agree. The final form factor is that… that’s another argument for sound, by the way. I happen to think Apple’s stealth way into augmented reality is not going to be optical going to be based on airpods.

Alan: Yeah it’ll be interesting to see what Apple comes up with. They’ve hired a lot, a lot of people. They acquired Matteo and Nirvana and a number of other companies. I believe it was Eyefluence or one of the eye-tracking companies.

Charlie: Well I mean, my guess is what they’re going to do is going to be a spatial AR/VR inter-operable device. But the technology is really not quite there yet, in terms of miniaturization and form factor. And, you know, Microsoft is really leading the way with sensors, and I don’t think we can overemphasize the importance of sensors versus optics.

Alan: Really excited to actually try the new Azure Connect, which is a cloud-based connect system. So yeah, you’re–

Charlie: Of course, it’s a way for Microsoft to charge you extra money every month.

Alan: They love that.

Charlie: They love that Windows license fee level for enterprise.

Alan: Well, Charlie, thank you so, so much, and everybody, thank you for listening. This podcast was another amazing example of how XR technologies are revolutionizing business across every industry. You can learn more about Charlie and his new book by buying the book Convergence today, by going to www.ConvergenceAR.com. And Charlie, thank you again so much for being on the show. It’s been an amazing honor.

Charlie: Thank you.

Alan: Thank you so much.