Using XR to Enhance the Hardhat, with Trimble’s Jordan Lawver

It might seem like a small, even simple fix, to attach an AR device to a hardhat, but according to Trimble’s Mixed Reality expert, Jordan Lawver, such a simple fix exponentially expands the capabilities of folks working in heavy industry. He drops by to explain to Alan how that is, and how AR can take things even further as it goes hands-free.

Alan: Welcome to the XR for Business Podcast. Today’s show, we interview Jordan Lawver, head of mixed reality at Trimble. They’ve got a number of different solutions for the construction worker that leverage the Microsoft Hololens 2, the XR10, which is their new wonderful hardhat based augmented reality/mixed reality headset. So all that coming up. And more on the XR for Business Podcast.

Hey, everyone. My name’s Alan Smithson. Today, we’re speaking with Jordan Lawver. Jordan, welcome to the show, my friend.

Jordan: Hey, Alan, good morning. Thanks for having me.

Alan: Oh, it’s my absolute pleasure. I’m really excited. And I wish this was show and tell, because what do you have sitting right on your desk right now?

Jordan: So you just let the cat out the bag. We’ve got an XR10, sitting here right in front of me. And I promise it’s not the only one in the world. We’re starting to ramp these guys up and get them ready to go out onto a job site near you.

Alan: What is the XR10?

Jordan: So I imagine that most people that listen to your podcast are pretty familiar with the Hololens 2, that Microsoft has announced that they plan to start shipping later this year. So what we did is, is we hopped on board with Microsoft kind of from the start, maybe mid last year. And we wanted to find a way that we could take the Hololens 2, and adapt it for use out in kind of safety controlled environment. Our focus is on construction, but of course, there’s many mixed reality customers out there in oil and gas, and manufacturing, and other kinds of heavy industries that require PPE — Personal Protective Equipment — when they’re out on the site. So whether that’s safety glass, or hardhat protection, or chin straps, or earmuffs, we wanted to make an integration that took the Hololens 2 and all of its capabilities, and made it able to work with folks out in those industries. So we essentially OEMed the Hololens 2 components from Microsoft and we bolted it into a new form factor that slides down on top of kind of an industry-standard hardhat, and still enables you to use your over your hearing protection, chin straps, all that other type of gear that people need to keep them safe out on the site.

Alan: We have HL2 + PPE = XR10.

Jordan: Yeah, exactly. Yeah. It’s not our first go at this. We actually made a hardhat attachment for the first Hololens. You know, we weren’t there on the ground level from release — like we are at this time — but Hololens 1 came out, and a bunch of people ran with it and said “OK, what can actually use this for?” And as you know, most of the use cases that emerged were very enterprise-focused. And in many of those heavy industries that I mentioned, we were creating software from day one, first for architects with our SketchUp viewer app, but then moving out onto onsite construction with our Trimble Connect app. And we realized very quickly that we just weren’t going to sell any software, because no one could take a Hololens 1 and fit it under a hardhat out on the site. So at that time, we worked with Microsoft, and we basically built clips that would retrofit an off-the-shelf Hololens 1 up into a hardhat. And it sold like hotcakes, people were all over it. And then when Alex Kipman showed us Hololens 2 sometime last year, we quickly realized that because the new form factor — with them moving some of the device to the back of the head — there was just not going to be any way to retrofit a hardhat over top of it. It was just– there too much interference between the hardhat and the Hololens 2. So that was kind of where the decision was made to go full OEM integration on it.

Alan: For those people listening: Hololens, everybody gets it, it’s a mixed reality headset. You can see basically three-dimensional computing. It also has cameras in the front that detect where you are. How are people using this? Like, why would somebody buy this and go down to the trouble of putting a Hololens into a hardhat, and what are the productivity components to it?

Jordan: So I’ll speak to construction, which I guess is where most of my expertise is. Out on the construction site, you’ve seen a bit of a revolution over the last decade or two of BIM — Building Information Modeling — which started with CAD, Autodesk and AutoCAD and Trimble’s got a lot of authoring tools as well. Architects kind of spearheaded it and then it started moving into general contractors and even into the subtrades of modeling what it is they’re going to do, before they actually go out and do it. And in modern days, you have not only the CAD, but all the embedded information in that CAD, where you can click on a steel beam and it’ll tell you when is this due for install? Who’s in charge of it? How long is it? How much did it cost? What are the material components of it? So you have an industry — and construction is not the only one like this — that has undergone or is undergoing a digital transformation, and essentially creating digital twins of what it is they’re going to build before they do.

But still to this day, despite that investment in that kind of initial data creation, there’s this huge data disconnect in 3D, from building a 3D digital twin, going out on a construction site and building physical 3D. But somewhere in between, you walk in on a job site and you’ve got guys holding up 2D paper plans and iPads. There’s a lot getting lost in translation. There’s a lot of rework. Construction is, like, notoriously over budget, over time in almost every project. So that’s really what we’re out to solve by kind of combining those two worlds, using a mixed reality device like the XR10 to essentially be a wormhole between those two 3D worlds, so that you can walk out on-site, collaborate with others and literally see what it is you’re building as you build it.

Alan: The rework problem is a multi-billion dollar problem. We actually have an investment in a small startup that’s looking at overlaying the models on top of the real world. Very similar. But you guys have built a suite of tools around mixed reality. So you’ve got Trimble Connect, SketchUp Viewer, Connected Mine — which I assume is from mining — Trimble Sitevision and Trimble PULSE remote expert. You want to walk us through each of those solutions?

Jordan: Yeah, sure. So some of them touch on different industries, but they all play off that exact same idea of connecting all this digital content that’s being authored or are gathered, out to the real world. So SketchUp Viewer is the design phase of buildings and construction. It’s for your architects and your owners as they’re designing something and iterating through a design, so that they can better understand what it is that each is designing. You know, this has been done in VR for years and years and years, but mixed reality is kind of adding a new component. The ability to actually, you know, let’s say you’re retrofitting a bathroom, being able to go out on the existing bathroom and overlay what it’s going to look like for a customer. The Trimble Connect solution is then kind of that next step in the process. It’s being used by general contractors, plumbers, electricians, HVAC installers to actually go out on-site and monitor the construction as it’s going in, and then ensure that things are going in correctly after they’re done.

We’ve actually worked a little bit– it’s not on our website, but we’ve worked a little bit on the facility management side as well, which is kind of a third piece to that puzzle, which is the operations side. So that’s for the guy 30 years later walking into that operating building, and being able to see this hot water tank is due for repair in the next two months, not only having that information in some SQR database hidden away somewhere, but actually being able to wear a device and see it heads-up on top of that hot water tank. SiteVision is a really interesting one. That’s something that Trimble — believe it or not — we have been developing SiteVision since 1997. Go on YouTube if you don’t believe it. We, like many others, recognize that Hololens isn’t the ideal tool for — or really, any mixed reality heads-up display — isn’t the ideal tool for outdoor use, mostly just because of the sun washing out the sensors and the display.

And so we developed SiteVision, which is a tablet integrated with really Trimble’s bread and butter, which is high precision GPS technology. So if you’re a surveyor or a heavy civil contractor or a utility worker, and you have GIS data or buried utility data, you can walk out on site and see where it is in augmented reality overlaid on the real world to about centimeter level precision. It’s pretty incredible what these GPS receivers can do. The Connected Mine application, I actually used to work on in my last job. It’s really the same concept as what we’re doing in construction. You know, the whole push in the mining industry is getting people out of the mine. It’s an unsafe environment. And so mines, they set up shop for 100 years on a single open pit. So they have the ability to put a bunch of fixed cost investment in the technology for that mine. And many of the leading mines, they are essentially mapping in real-time — using radar and LiDAR and drones and photos — a real digital twin of the mine. And so mixed reality is allowing them to remotely — you could have an office in Toronto — remotely monitoring a mine in South Africa, sitting at their desk but wearing a device.

So that’s what Connected Mine does. And it’s not only overlaying the 3D CAD that that laser scanned at the mine, but also all the stockpile volumes and the movement of ore through the mine, and all of that kind of IoT information coming off of the different sensors. And then last but not least, the PULSE application, it’s built off of a field service application called Trimble Pulse that we have. And it’s essentially a remote expert solution. So if you have workers out in the field repairing electrical transmission towers or whatever it might be, giving them the ability through their phone to essentially call back home and have an expert back in the office — which are becoming kind of few and far between, many of them are retiring — so you essentially have that expert back in the office able to beam out to 100 different guys to help them, only when they need it.

Alan: That is a really complete suite of tools and they’re kind of across everything from the design phase or when you’re planning a project right through to servicing a project years later. What are some of the, I guess, improvements? What can people expect using the XR10 with Trimble Connect and these types of things? What are some of the results that people are getting? How do you measure that over what they’re currently using?

Jordan: Depending on the use case, there is– it’s easier or harder to quantify that return on investment. In the SketchUp world, on the architectural side, it’s a little more of a qualitative tool, right? There’s a ton of value in it, but it’s a little harder to map the value of having an owner and an architect on the same page. You can go downstream and recognize like, “OK, well we can show that X percentage of rework has been avoided, because the architect and the owner were on the same page about what the architect was designing and what the owner was expecting from the start, which a year later saved us from having to rip out his penthouse suite and redo it.” That’s a little bit longer of an ROI to map out. There is a great example I used for the Trimble Connect app, and it’s easy for me to talk about, because it’s such a closer return on investment.

We Trimble actually just built another building here on our Denver campus. And during the construction, we actually went out with a lot of our technology and kind of used it on the site. And so we went out with a general contractor and the HVAC sub, and it was right after the structural steel had been installed — so basically the decks and the columns and the beams were up — and we essentially took their BIM model of the HVAC and we walked the floor to essentially do as-built comparisons. We’re comparing the digital HVAC to the real world installed steel. And within about five minutes we found, I think, three or four different spots where the steel guys had gone off their plan a little bit. They had installed little support kickers, probably a completely necessary change, but they hadn’t essentially synched it back to what’s known as the coordinated model, that all these different subcontractors share to ensure that their puzzles are kind of fitting together. And because the steel guys hadn’t mapped that back — which is a very, very common thing — the HVAC guys were in the process of prefabricating all of those HVAC components somewhere in the Pacific Northwest.

And so they were expecting delivery about a week later for components that they were now realizing weren’t going to fit. And so without Hololens, those components would’ve just shown up and they would have had to either hack away to make them fit — which could’ve then had downstream impact on the plumber and the electrician — or they would have had a bunch of guys standing around not able to work while they wait for new components to come in. And so then that’s even more of a snowball effect. Through Hololens, they were able to very easily see that issue like right from the start. Like within five minutes they pulled it out of there, and they marked it up right in the Trimble Connect application. They sent what’s called a to-do up to their factory in the Pacific Northwest saying, hey, the steel guys made some changes. We need to make some changes as well, to ensure that the new HVAC routes around these structural components that they added. They made the change right there, and that was it. The components that they received the next week all fit, and they completely avoided that massive issue.

Alan: That’s amazing. So this was a test for you guys. You guys weren’t even expecting to do this, but you saw the error– not error, but the changes that needed to be done, and we’re able to annotate that directly back to the main designer, I guess, the main BIM model. That is gonna be more– I can’t imagine that in five years from now, any building will be built without that, it doesn’t make any sense.

Jordan: [chuckles] I sure hope so. It’s what we’re out here preaching. I think what we really need to do a great job of doing this time around — that we didn’t hone in on too much with Hololens 1 — is really mapping that quantitative value, because ultimately that’s what gets people to buy it. And speaking with the general contractor after that happened, you know, in full disclosure, we weren’t even out there doing it for real. We were out there doing a photoshoot that day for the first generation Hololens. And they were like “we might as well do it for real” and just happened to find all these issues. We asked the GC after the fact and they said that finding that potential rework before it happened saved them about $12,000. That was five minutes that more than paid for– that paid for an XR10 and seven years of a TCH license.

Alan: [laughs] When you put it like that, I guess. One error that you find will pay for itself 10 times over. That’s amazing.

Jordan: Yeah, exactly. It makes me think that I should be charging like a quarter of a million dollars for an XR10.

Alan: Well–

Jordan: I promise you I won’t.

Alan: The thing is, I’ve put this crazy number out there, I said “Virtual/augmented/mixed reality is going to create a trillion dollars in value in the next 10 years.” Now, if you think about it practically with solutions like Trimble Connect and Connected Mine and just the stuff you guys are doing, just in the money that you’re able to save customers in — just call it one specific aspect and that’s rework — over the next five years, just in your company alone, that could be in the hundreds of millions, if not billions of dollars in savings, just in one company. So you can imagine this, times all the companies in the world, times warehouses and sales and training. And if you factor in every type of way this technology can be used, I think that trillion dollars in value created by XR is probably closer to the five to seven-year mark. Because it’s just so– people are thinking in terms of how many headsets we’re going to sell. But like exactly what you said, the headsets, call it $10,000. Well, you’re saving that in one small area immediately. So the value created over the lifetime of five years of this headset is gonna be in the millions of dollars.

Jordan: Yeah. And fortunately, the XR10’s half of that, it’s under $5,000. So you’re under the cost of the Hololens 1. And it’s a much more capable device this time around. So it’s pretty impressive. I very much agree with you. I don’t know about the trillion-dollar, maybe, I don’t– I haven’t tried to run the numbers. But the way I think about it is the Internet was like a great unifier for the world, right? Like you had all these people, billions of people spread across the world, all with their own independent knowledge on billions of different topics, from how do you cook curry, to how do you build a house. And the Internet gave a platform to connect those people together, and all that knowledge together. When I think of XR technology to a lot of people, it’s still kind of seems like a gimmick. But once we get the mindshare out there of what the technology really is, I think they’ll come around. Because what it really is is you have all of this data which is properly connected, human to human through the Internet, but it’s not properly connected, human to world.

Jordan: And so that’s what AR is going to give you the ability to do, is to take all that information that is relevant on the real world and actually show it on the real world, rather than having some kind of middle man that is your phone or a tablet overlay or a piece of paper. I did a talk a couple of months ago and my first slide was a picture of a wormhole from Stephen Hawking’s book and kind of doing that relation to a wormhole. I also every once in a while use an analogy to Stranger Things — if you’re familiar with the show — and the idea of there being this kind of Underworld. It’s in the same X, Y, Z, and T, but you can’t see it, it’s in this fifth dimension and there being kind this portal that connects the two worlds together. That’s essentially all an AR device is, it’s a portal of connecting that kind of fifth dimension of a data world. In some cases, your five, six, seven D and connecting it out to the real world.

Alan: Even going to get more interesting as we start to have a lot more sensors in the world. They’re estimating billions of sensors being in everything. And construction is one of the easiest use cases to put sensors in. It’s fixed cost, sensors are low. You want to know how much water is going through a pipe at any given point in the pipe, how much gas is going through, how much airflow. These sensors are inexpensive, but collecting the data is great. But really we’re collecting so much data that it’s kind of useless, it’s overwhelming. Nobody can really deal with that. So using AI to make sense of the data and then using XR to project the data in ways that is relevant and contextual to what we need is very important.

Jordan: Yeah, I agree. Construction sites are actually a little tricky to set sensors up on. You know, like an open-pit mine is actually very easy, because — like I said — it gets open and they carve it for 150 years. Construction sites because they change so often. It’s a little bit difficult to get infrastructure that’s not going to immediately get blocked by a piece of drywall or something, right? What’s interesting about a solution like Hololens is, today most of the work that’s been done is, how can we take data that already exists and then use the Hololens to visualize it? But I think that’s only one part of a bigger story. And I think that in the very near future, we’re going to start exploring how Hololens can actually collect the data as it goes. So not only can it show you a ton of information, but it can collect it as it goes, as well. So that example that I use of our guys up on the construction site and being able to see that design versus as-built clash, there’s no reason why a Hololens couldn’t use artificial intelligence and machine learning to tell you that by itself, right?

Jordan: And if that’s the case, then you don’t have to rely on the human anymore to be able to identify that issue. Rather, you could have– you could put a cheap little occipital sensor on every single person’s hardhat — rather than everyone wearing Hololens — and walk around. And it’s the exact same concept as what Elon Musk is trying to do with the Tesla, or any of these folks out there doing autonomous vehicles. You have vehicles that are not only using a map and knowledge to kind of navigate themselves, but they’re also sending information back to it at all times. It’s like, “Oh, this business went out of business, send that back to Google Maps.” And so it’s this kind of constant data cycle.

And I think you’ll start to see the exact same thing out on the construction site. It could be as simple as like we’re developing a solution internally right now through our labor and equipment management group. It’s all about workplace safety and making sure you know where people are on site. And a Hololens and through spatial anchors and that world mesh can tell you at any given time where people are on your construction site, just from that outside-in tracking. And then it could use the camera to see Joe over there is not wearing a safety vest. And send a note back to the office and say, “Hey, Joe’s not wearing a safety vest out there.”.

Alan: Yeah. Or maybe just send a quick message to Joe, say, “Hey, Joe, you forgot your safety vest.”

Jordan: Yeah, exactly.

Alan: Yeah. The possibilities are endless. I mean, one of the early use cases of the Hololens was for the elevator company Thyssenkrupp. They are able to take a process that takes six, seven hours of measuring the stairs, and they just walk up the stairs with the Hololens because it’s able to capture a cloud map very quickly. And it went from six hours of collecting data with a tape measure to literally minutes. So, really cool.

Jordan: Yeah, we’ve worked pretty closely with the Thyssenkrupp guys. We actually had them here in our office at one point last year during a workshop with our developers, and they’ve done some amazing work over there. There’s definitely some good code that got shared that week.

Alan: Amazing. Amazing. And that’s what I think is very unique about the XR industry, is people are so collaborative. I wonder if that’s going to change in the future. But for now, it seems like everybody’s just willing to help each other, which is wonderful.

Jordan: Yeah, yeah, absolutely.

Alan: So speaking of helping people in the world, what is one problem in the world that you want to see solved using XR technologies?

Jordan: So I’ll hone in on construction again, because those are the big problems I’m trying to solve right now. But there is a really big concern in the construction industry right now, around the next generation coming in. You have a lot of very old knowledge that is retiring and a new crop of folks coming in that aren’t quite as educated, that knowledge transfer isn’t coming through. And just not enough people are going into the trades, to be frank. And so I hope that XR will kind of do two things. One is I hope it will let the younger class think construction’s cool. I grew up plumbing new homes with my dad, and now I’m obviously on the tech side of it. But, you know, I still think fondly back to that time spent with my dad. It’s a good hard day’s work. And there’s a lot of value in construction and a lot of folks to see it as dirty gross work. And I’m hoping that through technology, we can start to attract some of this fresher crop into the industry.

And I think in addition to that, it’s not only using it for the allure factor, bringing people in, but it’s also going to make their lives easier and make it so that they can do their jobs better through technology, in the same way that having a phone in your pocket has made our lives easier every day, having a mixed reality device attached to your hardhat should make your construction life easier, and allow you to go home earlier and get a job done better and just be a more efficient worker, and make more money and really have a good livelihood through construction.