Combining the Best of AR and VR with Varjo’s XR-1, featuring Niko Eiden

The human eye is a wonderful and complex thing, and it’s a technological feat just to even come close to its natural revolution. Well, the folks at Varjo have created something that is pretty darn good at it. Alan may have trouble remembering how to say their company’s name, but he can attest to the clarity of the XR-1’s display. Varjo CEO Niko Eiden comes by to give us a look behind the curtain of its creation.

Alan: Welcome to the XR for Business Podcast with your host, Alan Smithson. Today we have Niko Eiden, CEO of Varjo. He’s formerly held top product leadership positions at Microsoft and Nokia. At Nokia, Niko led a product program team in 2006-2007, and together with researchers from Nokia Research Center, his team developed the basis for the optical technology that later became the Microsoft Hololens. Niko has a Master’s in science in aeronautical engineering. Varjo is redefining reality by jump starting a new era in computing. Their hardware and software lets people seamlessly mix realities together, moving from the real world to extended reality into pure virtual reality, all with human eye resolution. Their new headset, XR-1, is a mixed reality developer device for engineers, researchers, and designers who are pioneering a new reality. With photorealistic visual fidelity, ultra-low latency, and integrated eye tracking, the XR-1 seamlessly merges virtual content with the real world for the first time ever. If you want to learn more about Varjo, you can visit varjo.com. And I want to welcome to the show, Niko. Thanks so much for joining me.

Niko: Thanks, Alan. Nice to be here.

Alan: It’s my absolute pleasure. We got a chance to meet at AWE this year and I got to try the XR-1, which was — oh my God — what an incredible experience. You put on the headset, the pass-through cameras were as if I didn’t have a headset on at all, I could just see the whole world. And then all of a sudden a car appeared in front of me; in the space I was in, there was a car. Then I got in the car, and the space around me disappeared, and I was in the car. And the one thing that really stuck out with me that blew my mind was. I was looking — it was a Volvo — and I remember looking at the steering wheel. And the little Volvo symbol in chrome was so crystal clear. It looked like a real car. Let’s talk about your technology and how you guys ended up at this place.

Niko: Sure. We really had the initial vision– the founding team, we had a long background in different augmented reality and VR devices. And we had always been talking about that video see-through type of devices could actually combine the best of both worlds of AR and VR devices. And every time you started talking about them, somebody kind of commented basically that don’t bother: the latency, the lag between what you’re seeing through the cameras, and what’s happening in reality is gonna be too long. Or the resolution is not going to be good enough, it’s not going to look great. But we were in the summer of 2016, we were looking at a demo and the demo was shown with the Hololens first gen device. And I got really thinking that this would be such a cool demo, but it’s really missing a big part of the experience, because it wasn’t able to show the image in a photorealistic fashion. It was a very high fidelity graphic scene that we were looking at with Hololens and we were just thinking that this would be so much better with a VR device. And combining the reality with a video see-through device could actually really work. We were a bit curious. We had some extra time at that point together with Urho [Konttori] — one of the other founders — and we built our first prototype in 24 hours, and the experience was really magical. So it was a very simple experience. But the main thing of that first experience was that we were able to dim an existing room and make it dark, add some fog into it, and there were some knights. So it was immediately something that you couldn’t do with any other device and you still can’t do with any other device, because with an AR device, you can only add light. You can’t add darkness. That demo was good enough to get us the initial funding and get us going, so it really proved the point pretty neatly, and that’s just how we got started.

Alan: It’s interesting, I actually tried– there was a company based in Toronto — and I can’t remember the name of them — but they had a VR device with pass-through cameras, and then they had this demo where a giant crashed through your ceiling and it picked you up. And so it was this blend of real world and then mixed reality, and then you were in full virtual reality. And that was my first foray into it. But to your point, the resolution just wasn’t there and there was such a lag in the latency that it kind of made me feel queasy, and I felt none of that with your headset. It was crystal clear, and not only crystal clear with the latency, I waved my hands. There’s actually a photo of me waving my hand in front of my face, because I just couldn’t believe that I was seeing my hand. Your secret sauce or your magic that’s behind it is, you’ve kind of created almost like a false foveated rendering, meaning if you look directly in front of your eyes, your fovea are 100 percent focused. And then as you move out from there, it gets less and less focused. So if you’re staring at your computer and you hold your hand out to the periphery, your hand is kind of blurry, you can’t really see it in detail. You guys have leveraged that by putting a higher resolution screen within a screen. Can you walk us through that?

Niko: Yeah. So that was actually the first pivot that we had to make. As I mentioned, the big vision was to do mixed reality with video see-through device. But in the beginning, we didn’t plan to set up a company that would produce VR headsets. So we actually were thinking that it would make sense to do a good accessory and equip the existing VR headsets with a very good accessory that would solve the latency problem. We had a few ideas how we could crack that problem all the way from the very beginning. But when we were testing it, fairly quickly we realized that we can’t leverage any of the existing VR headsets on the market still today, just because the resolution was not on par what we could achieve with the video see-through system. So fairly early on we decided that okay, even though we want to do the mixed reality with the video see-through, we have to park it for a while. We we have to work a bit in a serial fashion on this problem and we have to tackle the resolution problem on the VR headsets first. The idea that we came up with, as you mentioned, was exactly that. So we are utilizing four displays in a headset, so two displays per eye. And we are using more of a traditional setup that we call “context display” that provides you the full field of view and you get the periphery and that doesn’t have to be in such a high resolution because the way how your eye works, it’s impossible to see and in high detail for the periphery part. And in the center we have a second display that we’re overlaying with an optical mirror, and that way we are able to mix it almost borderlessly in front of the complex display we call “focused display.” And this display is able to match the human eye resolution by something that we call “60 pixels per one degree field of view” in that high resolution area. Very early on we had a target that we want to be able to read a newspaper in VR. That headset is able to perfectly allow you to read a newspaper in VR.

Alan: Now, for those who haven’t spent a lot of time in VR, that is something that is a real challenge for VR, is just reading text is a problem. And there’s companies like Monotype, who has created all the type fonts that you see, — Times New Roman and all these things — but they actually started creating type fonts that would be visible in AR and VR. Then you guys took a different approach and just said, well, the problem isn’t the text, the problem is the resolution.

Niko: Yeah. And then there were quite a lot of restrictions, and still are. I mean, one is the compute power available on desktop PC. So if everything would be high resolution VR, we would talking about not 4K display; we’d need, what, 16K, 20K displays that would be required to pull it off in a reasonable field of view. So that was not the track that we could take. I mean, the displays didn’t exist, the processing power didn’t exist. But with this way, by combining these four displays that existed on the market, — basically off-the-shelf components — and combining them in a smart way allowed us actually to do something that we can produce today, and that will work with the computers of today and provide you this region of high resolution. It would be bombastic if everything would be high resolution and you couldn’t distinguish anything anymore from resolution perspective, but unfortunately that we aren’t get there.

Alan: From a user’s perspective, when I put it on, I didn’t notice that the periphery wasn’t focused. As I looked at the steering wheel, I looked around the car, I looked at the leather stitching and actually my brain just totally ignored it. It wasn’t until the second part of the demo — I was looking at a whole area where there was a moose crossing, and it was really neat — but it wasn’t until then that I kind of realized that there was a little bit of a border around what I was looking at. The way you guys have done it, with the super high resolution in the middle and it kind of almost feathers or fades out to the other resolution. It’s almost imperceivable if you didn’t know what to look for. I think it’s really magical. You guys have a partnership with Volvo. How did that come about?

Niko: The Volvo partnership, we started off developing the VR-1, which is our headset that brings human eye resolution to the market and brings an eye tracker to the market, that also differentiates us from pretty much all of the other companies out there today. But the Volvo case was a bit special, and Volvo had a vision of stuff that they wanted to do in the mixed reality space, and our dream of doing mixed reality with a video see-through actually matched. And we were able to offer them first very early prototypes, so that they could start testing. And it resonated really well. So Volvo had a fantastic engineering team. So they had very good capabilities of actually doing work from an engineering perspective. And for us, it was a fantastic case to pressure test whether the vision, whether the product that we were actually signing made sense from an enterprise perspective, and in the case of Volvo it definitely did. So it was these kind of early discussions. It started around the VR-1, doing VR in high resolution. And we mentioned that we have this mixed reality prototype that we’re still working on, and we were able to show. It clicked from there on that, and we collaborated pretty tightly from there on. We were able to finalize and understand what an enterprise company would need and Volvo could get their hands on very early on something that simply wasn’t available anywhere else.

Alan: What are they using it for?

Niko: Volvo is planning to accelerate their design process for cars. The team that we’ve been working on is mainly working on the interior of future cars. And if they have a new idea, with our device they can actually test it before they have to build a single physical prototype. So what they can do, is they can replace the interior of an existing car and virtually replace it with a future car interior. And they can start testing the interior of the instruments, switch locations. And with the eye tracking, they can actually do proper testing so they can drive on a real road. And with our headset, we are able to segment the window so that the windows from inside the car show the reality. And that view is to our camera setup and then the rest of the interior is then the future car interior, which is completely synthetic and created graphics. And with the eye tracking, they can light up, for example, a warning light and they can check in a real driving condition how quickly a driver could, for example, see that light, just has a crude example of the stuff that they could do. Or head-up display system, mixed graphics and show– your virtual moose you mentioned, they can show it while driving. There doesn’t have to be a real moose, but they can simulate the moose and they can start testing driver reactions.

Alan: How are you capturing the… I guess the window views? Is that through a 360 camera setup or something like that?

Niko: No, we have the see-through cameras, obviously, and then we have a full model of the car. If you want to replace the interior of the car, you have to have a full model of the car, and then you just have to have good calibration so that the tracking of the headset and the head pose is synchronized and calibrated.

Alan: So, wait a second. Hold on. People are driving a real car wearing the headset?

Niko: Yes.

Alan: Oh, wow. So they’re getting in a car, putting on this headset — I’m assuming there’s a computer in the back seat or somewhere — and they’re driving around a closed track.

Niko: Yep. Officially, it is a closed track. Absolutely.

Alan: Wow. Oh, my God. That’s next level product testing.

Niko: And you don’t have to build a single thing in order to start testing. So this early on testing, fail fast type of stuff, completely new world from a design perspective.

Alan: So what are some of the– it’s almost silly to talk about ROI because they’re probably seeing a dramatic decrease in the design times. Do you have any data around how this is benefiting them?

Niko: Yeah, we have a discussion on this with Volvo. They didn’t want to go into detail about the ROI on that one, but it’s as you mentioned, it’s extremely obvious. Being able to create a faster design cycle and especially doing auto-validation of something that’s still virtual, just on a computer is pretty bombastic. The interior design is just one example. Car design in general, I guess you’ve seen pictures of these full-size wax models as well, where car designers go around a full-size car, built out of wax and modify the lines.

Alan: It seems so antiquated.

Niko: Yeah.

Alan: [laughs] Now that we have VR, it seems so antiquated to do because one, you have to have the physical model. Two, you have to have everybody physically there to look at it. And three, you can’t make any changes on the fly.

Niko: Yes.

Alan: So dramatic, dramatic decreases in design times, which as we’re entering into this area where AI, and VR, and XR, all these technologies are culminating in creating this exponential growth pattern. Every single efficiency that we can afford these companies is going to be snapped up and used really, really well. So this is a great way to do it.

Niko: It’s not just vehicle design. I mean, architecture, everything. The fact that you’re able to see photo-realistically a room, and see how the light plays inside the room is big. Previously you had to imagine that stuff and wait until the building is built. Now you can experience light inside a building before it’s been built.

Alan: There’s so much happening around ray tracing as well, having light shine in from different angles, reflect off of materials. There’s been so much research done on that, Unreal Engine and Unity are really pushing towards that. “How do we get photorealistic renders without blowing up your computer?” What are some other businesses that are using this, and how are they using your headset?

Niko: Well, apart from design, training is another really big focus for us. Easy to imagine is really high profile training of airplane pilots, for example, if you are able to reduce the simulator time required to train a pilot, for example, by allowing them to retrain in a virtual cockpit beforehand. Again, the business case is absolutely a no-brainer for even aiming for a high-end device like ours. But it’s not just the pilot training. It scales down to everything, to all professions where there’s something unusual might happen during your day of work, but it’s hard to prepare and train. So emergency rooms, control rooms, police, firefighters, those type of professions, I think they will benefit immensely from proper mixed reality and virtual reality training programs.

Alan: You know, people say, what’s the killer use case for VR? Well, I think training is the killer use case and design is a close second to that. Can you talk about some other specific examples of companies that are using your device, and how they’re using it?

Niko: We are still learning how companies plan to use this device, but it varies quite a bit. And for us, the key learning has been also we don’t know how a business or enterprise wants to use a mixed reality or VR device. Running a business and then, for example, designing something or training something that’s really the core of that company. We need to work in close collaboration to understand their specific use case. And for those companies now thinking whether they should deploy or think about using mixed reality or VR: for some specific cases, yes, there will be very clear places to read how you could use it for your own business, but in most cases you need to understand what you can do, and then you need to start dreaming. And the good news is there are thousands of companies out there willing to help, to program and create the software tools required to do stuff, especially in enterprise and business segment. But it needs to come from the inside out. And those companies who are kind of really advanced, like Volvo, like Audi, for example, a lot of the car companies that’re really advanced, they usually have a big team of specialists, who are able to support than those people who have the need and were able to dream the use case. And let’s say in an automotive company, they can use it for the accelerated design process that we discussed. They can use it to accelerate the design process of their car factories and they can use it to help configure a future car model that they selling to an end customer in a showroom. So there are plenty of different completely different use cases with completely different needs, even within just one large corporation. And to me, that’s super exciting. But it does require that those companies are pretty active themselves as well.

Alan: There’s really no part of any business– and I published a post on LinkedIn a couple weeks ago and I said: “Is there businesses that won’t be impacted by this technology?” Can you think of any?

Niko: No. I have a strong feeling that this will change fundamentally how we will be working in general, and I think everybody will be impacted one way or the other. This is kind of a pretty bold statement I’m making. But if you think about the tools with what we are working in general today, it’s a phone, it’s a PC, maybe in some cases it’s a tablet. That’s it. It’s very restricted, it’s very 2D. Once you start to have a device that can match the resolution of reality — from the VR or mixed reality perspective — you can move all of the existing tools and make them part of the virtual experience, the mixed reality experience. And that’s why I have a feeling that this transition, once these products start to be out there, is going to happen faster than we anticipate. And when we hit the first professions where using virtual reality or mixed reality will be part of their daily routine everyday and it’s a few hours, I think that’s when we start to see a really big transition happening also, from professional space all over.

Alan: I was at the PTC LiveWorx conference in Boston just recently. And some of the use cases that they’re bringing online now are not even really mixed reality so much as the RealWear headset where you can just see a little screen out in front of you and give you a heads up display. And what they’re able to do is capture the standard operating procedures that experts know intuitively because they’ve been doing the job for 20 years. As those experts start to retire, they need to capture that knowledge somehow, and then transfer that knowledge to younger workforces. And I think when people start wearing this a couple hours a day on a regular basis to help them with their work, and I think in some very small instances that’s happening already and the results across the board– Shelly Peterson is coming on the show and she’s talking about how Lockheed Martin is using augmented or mixed reality glasses for assembly. The original trials that they did had 85 percent reduction in task completion times. 85 percent reduction! So when you have these crazy reductions in speed-to-product or speed-to-development, if you can take a process that takes a year of designing a car and shrink that to six months, that is a massive savings to a company. Where I’m going with this is that, it offsets the cost. If, for example, it costs me $10,000 or $20,000 for a VR headset that shaves millions off of my design process, who cares what it cost at $20,000? It doesn’t matter. You guys have the premium headset in the market. Talk to us about the costs with assembling not just the headset, but the whole package. What would something like that cost to fully outfit one designer with a setup, the ideal setup?

Niko: It’s around the 10K mark and that includes already the state of the art workstation. It’s very similar than a few years ago, just to workstation alone from a pricing perspective. We haven’t seen the price be really the issue of adoption. It’s more about the day to day practicalities and how do you work. But I think one great example has been on the architecture and civil engineering side. For them, the benefits are huge. I feel that they are fairly slow in picking up new technologies in general. But if you think about what happened during the 90s, I was an engineer at that point. Then I started my university studies by drawing with a pencil to a big white paper. And when I ended, everybody had transition to a CAD system. And that happened all across the world, not just in the university, but all the companies transitioned as well. So civil and architect engineers — from my perspective — they are fairly slow to pick up, but once they pick up, it’s a huge snowball effect that everybody does it at once. And that’s something that I’m expecting and hoping that would happen as well, from a VR/mixed reality perspective fairly soon.

Alan: Your timing is probably very, very perfect because we’re also seeing this as well. You get these proof points out and it’s only recently in the last kind of six months where big companies are releasing their original pilot information. We’re just starting to come out of pilots with some companies. They’re going, “Hey, Lockheed Martin, we decreased our time to production by 85 percent.” Macy’s is using VR for sales in their stores and they– averaged across 110 stores, they’ve increased their sales by 45 percent and decreased their returns from 7 to 2 percent. So once you kind of start to see these use cases, if you’re going to remain competitive and your competitors are seeing these kind of transformational shifts, it’s only a matter of time before you have to do it. It’s not nice to have, it’s a must have. So what is your timeline around where most companies start to roll this out? Is it the 3, 5, 10 year rollout or what are we looking at?

Niko: We are seeing every Fortune 500 company looking at this stuff. Some are doing trials and pilots still fairly early on. Some are super advanced. It’s hard to say a timeline, but what I can say is that the transition has started, and this is going to happen. There’s been a lot of discussion, whether VR or mixture reality. Is it real? Will it take place? I mean, the consumer side didn’t work out as everybody was expecting a few years ago. But from an enterprise perspective, from the perspective of using this stuff to accelerate and make your work more efficient, I think that has already started and there’s no way to stop it anymore.

Alan: So what is the most important thing that businesses can do to get started?

Niko: Take it seriously. The minimum is, as I said, it’s very hard for externals. It needs to come from the inside because you need to have the understanding of how stuff works in the specific business in order to make it more efficient. And obviously, consultants can come in and do the interviews and then they provide you an answer. But it really has to come from inside. So put the few creative people, give them a mandate to study and really understand what mixed reality in the next few years can do. And I think it will be very easy to come up with three to five things that yes, could actually make our life a lot easier from a business perspective. And then just find a partner with whom to build a pilot. It’s an upfront investment that companies will need to do within their own context, but it’s going to be worthwhile.

Alan: I agree. You know, recently we started XR Ignite, which is a community hub based on the idea that startup studios, developers, they all have a role to play. And a lot of VC investment funds only look at platforms and products, and they’re looking for giant billion dollar unicorns. What we’re seeing is that there’s going to be a lot of acquisitions around the studios and developers, because as big companies start to realize that they spin up a small team, they go, “Hey, this is working for us, we need to start doing this at scale. We can either hire people, which is hard because not a lot of talent out there, or we can acquire a small team that’s already working together, that already has what we’re looking for.” And Deloitte just acquired a startup studio, rather, and a couple of other companies have acquired studios. What do you think the investment landscape is going to look like? Because VCs don’t typically invest in studios and content. They’re more invested in the platforms, where we took a very holistic approach and said, “well, as a community hub, we can all help each other get more clients, but also facilitate these smaller acquisitions, because there’s going to be a massive need for content soon.” How do you think we’re going to address that?

Niko: I think in general, the appetite from a VC perspective and the interest in VR and mixed reality, I think it’s turning, and it’s turning towards the positive side. I mean, the past few years have been fairly difficult for many startups working in this arena. It hasn’t been easy with the software, whether platforms, whether content to get funding, especially if it had a title VR in front of it. It’s a bit easier if you were somehow in the augmented reality domain in the past years. But I have a good feeling, and based on the discussions that I’ve had recently, I think the– there starts to be a lot of appetite, and this appetite is actually being fueled by these real world use cases: the companies who haven’t played a role in the VR/AR before actually showcasing. And I think the Volvo case is a fantastic example of one of those cases. You would definitely not put them into a VR/AR bracket, but hey, they are showing something that’s completely advanced and then blows everybody out of the water at the moment. And it’s coming from a company who you would position into a fairly traditional automotive sector and not into the latest stuff of computer graphics. So I think those showcases are fantastic, and it’s gonna be easier in the next few years to get funding because there is a clear need. There is– the business case is there, and demand is strong. And I think the training and simulator market is a fantastic example. I think there are thousands of companies — if you go to a trade show focusing on training and simulation — there are thousands of companies showcasing already a product. But they’re showcasing it with hardware that actually doesn’t match the needs of the product or is somehow inferior, but it doesn’t matter. Despite the fact that the hardware hasn’t been up to the standards that they would need actually, they have already created the product and somehow they found the investment to create that product. And timing wise, it’s from our perspective, it’s fantastic. We can give them now a product that fulfills many of their needs and they can go and promote it again with a complete new spin. I think it’s going to happen and I think we will be seeing more and more investment in the space as well.

Alan: I think we’re already starting to see it. I think the trough of disillusionment when people went, “Hey, we’re gonna– VR going to be huge! Everybody is going to have it, and every consumer is going to wear AR glasses!” Yes, this will all happen, but it will take 10 years. And I think that the consumer market is really fickle. It has to be perfect, cheap and lots of content, whereas the enterprise just needs to make an ROI. And we’ve already proven that across every industry — almost everything from oil and gas, to automotive, to training, to retail — every aspect of business is being impacted by this.

But, let’s take this in a different direction for a second. What problem in the world do you want to see solved using XR Technologies?

Niko: Change the way how we work. As I mentioned earlier, for me, that’s one of the big visions. In the beginning, we wanted to change and create the future displays. The transition from a 2D display — like a monitor or a cinema screen — into a more immersive display. That’s still something that I see as a transition perspective. But before that, for me, the big thing is to change the way how we work.

Alan: One last thought I was thinking, people buy all different VR headsets for different reasons, for consumer reasons now. If you buy a Varjo headset and you’re designing on it, can you still play Beat Saber on it?

Niko: Not yet.

Alan: Oh, you got to fix that.

[laughs]

Niko: Yeah, we’ll fix it. I mean, that’s the problem with that is, that all of the games, they’ve been optimized for the consumer grade headsets, which means very limited resolution and two displays. So in our case, you would have to drive four displays and you would have to have enough resolution, so it would make sense.

Alan: Do you guys have an SDK then, that helps people — developers, for example — create content specifically for this?

Niko: Absolutely, yes. So we have an SDK that plugs in into Unity, Unreal, all the game development engines are fairly straightforward. We’ve also been working quite extensively with the vertical engines like Lockheed Martin, for example, with a CAD system so that you can you can use our headset directly with those systems with which are not necessarily on a gaming platform or available and then those consumer type of stores. And then we are focusing quite a bit on the OpenXR standard. And at CIGREF we’re actually showing our XR device with a pure OpenXR demo as well. So I think OpenXR is going to unify quite a bit all the different standards and it’s going to make everybody’s life a lot easier.

Alan: I really, really hope so. Having the ability to pull this up on web, being able to do what you need to do and then switch between programs seamlessly, I think is gonna be where this takes off. If we can get it right with enterprise, it’s going to lay the groundwork and foundations for the consumers to just enjoy the best resolution, the best everything at the lowest cost possible. So, Niko, is there anything else you want to share with the listeners?

Niko: It’s been a pretty fantastic grind. And I mean, you experienced when we came out with XR at AWE. For us, that was a huge milestone. It was our initial vision to do a mixed reality device video see-through. We had to do a couple of other things before that. So we had to develop our human eye resolution. We had to do bombastic eye tracking technology. We had to do the tech, we had to productize it. But the fact that we are able to bring it to the market and create the product, that’s something I’m super proud of. And it’s always a big milestone, actually, instead of just dreaming and envisioning technology, but being able to make the technology available on a commercial market, that’s a fantastic achievement. And this is gonna be a fantastic year for us, and I hope everybody else as well.