The Right Displays for Challenging Tasks: XR on Oil Rigs, with Shell’s Michael Kaldenbach

Many scenarios that might be improved by an augmented reality heads-up display shouldn’t require an overly arduous selection process; most gizmos will do if you’re checking a weather app while jogging. The same can’t be said for picking out a device to help oil rig workers work safely and efficiently in the middle of the Permian Basin. Shell’s VR incubator lead Michael Kaldenbach talks with Alan about the things his team had to consider when selecting the right device for the job.

Alan: Today’s guest — Michael Kaldenbach – is an augmented, mixed, and virtual reality incubation lead at Shell, the global oil company. He is a driven, goal-oriented, resourceful, and creative person, who really understands the usefulness of this technology, and bringing how to bring it to the market. He’s chosen the family motto of Arctic explorer Sir Ernest Shackleton, as it accurately reflects how he approaches any challenge or goal: “victory through perseverance,” or “Fortitudine Vincimus.” He strives to apply entrepreneurial mindsets and thinking up out-of-the-box solutions and approaches when working in this technology. If you want to learn more about his work, you can visit Shell.com.

I want to welcome to the show, Michael Kaldenbach. Welcome to the show, Michael.

Michael: Hi, Alan. Thank you very much for having me on.

Alan: It’s my absolute pleasure. I’m really excited. I want to dig right in here, because I know you guys at Shell have been doing a ton of work in everything from kind of marketing and trade shows, right through to oil wells previsualization. So let’s talk about some of the ways that you and your team are using virtual/augmented reality right now.

Michael: So I think One of the better case studies we have is around augmented reality remote assistance, and I’m sure you’ve seen examples in the wider industry for that one. But for us at Shell, that means that we utilize a head-mounted display — in this case specifically, the Realwear — and it is used for our operators; for quick resolution, and to get remote expertise to be brought in.

I think it always helps if I provide a little story to set the scene; think of an offshore oil platform out there in the ocean. Typically, the most senior person is the control room operator, and there are more junior operators that are assisting the running and maintaining of these kind of assets. If in the control room, they see a deviation on one of the many dashboards they have, they send out a more junior operator to investigate — normally with a radio phone or walkie talkie — and then they guide them through, they get back to “what is the situation; what’s the sound the machine is making?” But where we really revolutionize that process is with a head-mounted display. It is as if the experienced operator has immediate eyes on the situation. So think about [it] — you see (or I see) what the junior operator is seeing, and thereby, I can use my years of expertise to resolve the issue, and get back to safe operations.

In a case where my expertise set is also not sufficient, we can quickly be joined by a remote expert who can be onshore — can be anywhere in the world — to join that same virtual room, so that a three-way conversation happens. Not only that: instead of having those conversations like, “I recognize the problem; you need to switch off the third button from the left, it’s kind of greenish on the left side, bottom side of the machine,” instead, we use something called “telestration,” and that’s the benefit of having a head-mounted display, whereby I — as the remote expert — can draw on my screen and the same visual is replicated to the junior operator, so in his line of sight, he gets an annotation — a circle or an arrow, whatever is helpful; it could also be a video — to resolve the situation. Thereby, we quickly resolve issues that might end on production deferment, or something even more serious. It is really helping to revolutionize our operations.

Alan: Just to kind of recap: for people the junior operator’s got a pair of glasses on. He goes on site, or she goes on site. The senior operator is sitting at a desk, being able to monitor multiple junior operators, I would expect.

Michael: Correct.

Alan: And that person is now able to remote see exactly what the junior operator is seeing, and not only that, but annotize on top of their vision. Is that correct?

Michael: Correct. And as you can imagine, the environments that we as an energy business operate in, they can be far-flung. They they can potentially be dangerous. So, you really want to have the knowledge with everyone out there in the field. This world is changing so fast, and it’s really difficult for everyone to keep up with all these changes. By having the ability to pool your resources — bring them via virtual reality into the situation — that is really helping us.

Alan: So this is real-time. So This isn’t like virtual reality, where you know you’re training in something — and then obviously, you can use that as well — But this is real-time. See-What-I-See interactions. Now, what about some of the lag issues? Or are you guys experimenting with 5G technologies to eliminate the lag?

Michael: As you can understand, connectivity is a struggle, whether it’s 3G, 4G, and the signal range when you’re out there in the Permian Basin, or out there in the North Sea — it’s difficult to get that signal. Wi-Fi is also very difficult. We tend to have a lot of steel environments, and line of sight obstructions. And in general, if you think about telecommunications – 3G, 4G – that’s pretty much centered around populated environments. So definitely, we struggle, and we look to all sorts of possible solutions for connectivity — whether that is low-part/wide-area network, whether that’s 5G, whether it’s low-orbit satellites — it’s always dependent on the actual business case, and what makes sense in that situation.

Alan: I always find it interesting: when we’re talking about connectivity, and you actually cut out right when you’re talking about 5G.

Michael: [both laugh] I mean that’s why we need 5G.

Alan: It always happens right around the connectivity conversation. “How do we get connectivity stable?” And you’re like, “oh-ah-erk-uhh.” Okay! [If] we can’t make a podcast recording work, what are we supposed to do in an oil site in the middle of the Congo?

Michael: Exactly. Exactly.

Alan: You guys are really solving massive problems, using this technology. You mentioned Realwear; maybe you can walk listeners through what the Realwear glasses are, and why you chose them over other pairs of glasses?

Michael: Well, again – it’s about the environments that we are in. (And let me know when I start cutting out again).

We are predominantly in something we call ATEX zones, which means that it’s specific safety regulations due to the substances we use in our production environments; they require our devices to be intrinsically safe. For that reason, we look towards the market. The hardware market, and identify the pieces of equipment that are licensed to operate in that environment.

When we started looking at these head-mounted displays about two years ago, we did try pretty much everything that’s out there in the market, ranging from the old-but-true Google Glass, to an ODG or Realwear. There used to be a fair number of opportunities for us, but again, we were sort of restricted by that ATEX certification requirement. We went through quite a rigorous phase of proof of concept. We tested connectivity – is this even going to work at our sites? We looked at ruggedization – if we drop it, does it immediately get damaged? And for us, the Realwear — which markets itself as a head-mounted tablet, which… uh, fair enough – it’s a seven inch screen. I don’t know if you want to call it a tablet, or just a phone-sized screen, but it came out on top. It’s very ruggedized, it comes with an ATEX certification, and it has all the essentials we need; a good camera, a good microphone, and a good screen, for us to look at the information that we need.

Alan: You mentioned good camera good microphone. I think these are essentials, and one of the things that I saw recently at CES was a company called Kopin, and they make the micro displays – I think they even make the one for Realwear – they make the actual micro displays. Something that they introduced me to was their Whisper technology, where you could – in very loud environments – you could talk to your device, and it would understand it, and it got rid of all of the external noise altogether. It was really quite amazing; somebody who is standing right next to me was trying to tell it the instructions, and it would only respond to my voice, which I think – as we move into these headsets that do more than just be able to bring them information, but they’re actually bringing up information using AI algorithms and stuff to give us real-time information – I think having a system that allows you to whisper or talk in really loud environments is key.

When you chose to go with Realwear, how do you then start developing for that? Are there off-the-shelf products? Are you looking to startups to to help develop these things? Are you building things in-house?

Michael: Well… if you don’t mind, let me get back to what you said, because you did trigger me a little bit. The reason why I’m passionate about this technology, and I’m passionate about the team within Shell, is because I firmly believe that we’re on the cusp of a paradigm shift. We went from using your mouse and keyboard to get information, to using the speed of your thumb, whereby we use our thumb to operate a mobile phone and tablet. And now we’re at the cusp of changing that technology to voice-driven.

So, you are absolutely right in saying it becomes exceedingly important for these kinds of voice recognition algorithms to not only work in noisy environments, but to then also accurately translate what you need, whether you have an accent or not. This journey started a couple years ago with the likes of Alexa coming to the market. We have other players like Cortana and Google Allo. But really, I’m so excited. I also don’t think that we should separate voice and natural language processing from the idea of the extended realities, whether it’s virtual or augmented reality. For me, they represents the foundational blocks upon which we build everything.

So, sorry. I wanted to leave that with you, because I feel so passionate about that.

Alan: I think that’s a really good point, and I think we should just dig into this a little. On my last interview, I interviewed Rori DuBoff from Accenture, and she mentioned XR – or extended realities – you’ve got virtual and augmented reality, mixed reality. So, virtual reality; you put on a headset, and you can change the whole environment you’re in for training and stuff like that. Augmented reality; overlays data. And then mixed reality; overlays data in context to the real world – when you’re looking at a machine, it can give you very specific data around how to turn the knobs and stuff like that. As an extension of that, you’ve got voice recognition, or human language processing (which I guess falls under the category of artificial intelligence), but then you also have computer vision, and machine learning, and big data analytics, and no one of these technologies on their own is a useful solution.

People need to understand that this is a continuum of a number of technologies coalescing together, and I think this – as somebody who studies the future of humanity – we’re reaching this exponential growth point, where 5G connects with artificial intelligence/computer vision/machine learning/human language processing/virtual and augmented reality, and all running on a block chain. If you take all that technology separately, they’re amazing; put them together, and now we’re really revolutionizing businesses.

Michael: I think that’s, for me also, what’s exciting about working for a larger enterprise. Because we do have all those individual components. We have dedicated teams working towards that, which allows the digital realites team that we’re leading – I’m just gonna put it out there; it’s going to stretch the mind a little bit – but we believe in this thing called “Digital First,” whereby sort of the digital twins and the various digital realities become a fundamental component of operations.

The idea there is that any action or operation is initiated through these kind of digital methods, with the physical reality only being an outcome of that digital reality. Whether it’s like collaboration in virtual reality in order to prepare for a complicated task with colleagues across the globe, or it’s about simulating the outcome of like a task with a digital twin, where you run through all the permutations. Or – one of my favorites – step-by-step guidance in augmented reality to really perfectly execute difficult procedures, so that we are much safer and much more efficient. That sort of digital-first mindset builds upon all those capabilities – and some people call it the “metaverse,” I like to term “the digitalverse” – but honestly, it’s an exciting time, and I hope everyone joins us.

Alan: “Metaverse,” for the people that are listening, the term was coined in a book called Snow Crash about, oh, maybe 15 years ago? The term refers to the world you’re in when you’re in virtual and augmented reality – you’re in the “metaverse” – and it’s kind of got this worldwide thought to it.

But I am a little partial to that word. [laughs]

Michael: No, no; fair. But, as I was sort of ranting on, I am reminded that what we’re doing, it’s more than just technology. It really is about people, and creating more agile ways of working, and for safety. And I don’t want people to forget that; all the amazing things that we can do, it is for the benefit of people. Just wanted to put that out there.

Alan: Let’s unpack this one by one, here; you talked about collaboration and VR. Whether that’s training or previsualization for something, what does that mean to people? What is the benefit to the people in your organization, being able to collaborate in VR?

Michael: Well, I think it comes, first, from a mindset that we recognize that we don’t know everything. They always said – I don’t know if I have this saying correctly – it takes a village to raise a child. In this case, it takes a large organization to tackle some of these big problems worth solving. In that sense, we really need to work together. But we’re in all places in the world. In order to really utilize the diversity that we have within the company, you want to bring them together, but at the same time, not be a burden to the earth, in terms of having everyone fly in from all over the world. Where virtual reality is definitely changing the game here is: it’s as simple as putting one of these headsets on, and then you meet your co-workers wherever they are.

Alan: What platform are you using for that type of thing in virtual reality? Did you build your own, or are you experimenting still?

Michael: For that, we are very much in the experimentation phase. The idea’s there – and I mean when I say that the idea’s there, the core idea has been around for ages – and when we first started this journey two years ago, we saw maybe three players out there in the market. In the last year, they have popped up like mushrooms.

Alan: Probably three coming every month now.

Michael: [Laughs] Exactly! So that makes it a little bit–

Alan: The ones we’ve tried are Roomi, Alt Space – obviously, is not really one for professional – but Alt Space, Roomi, VR Chat, High Fidelity. There’s one called Glu out of Finland. There’s a bunch that have come up. I tried the Glu one; I was really impressed with that one, actually. It was really good.

Michael: Well, it sort of reminds me – I might be betraying my age, now – but it reminds me of Second Life, and when they started, how amazed people were, and how it brought people together – not in the sense of a game, but in the sense of a community. You saw the community efforts build beautiful things, and functional things as well. We saw companies go to Second Life and open up their branch there, for customer support.

So when I say that, I see this sort of same moving happen out there in the market right now. But we’ve learned from Second Life, and these kind of relatively uncontrolled environments are difficult to manage for an enterprise such as ours – never mind the plethora of legacy systems that we have in operation that you would like to tie into these kind of environments. The current market, with the many, many players out there, is not conducive for us to select one. Right now, we’re very much testing the waters; finding what works for us, and what not does not work for us. In honesty, I think we’re going to have to go with one partner, and then help them achieve what we need them to achieve.

Alan: Yeah, I think that’s what other people are doing, as well. They find the solution that’s closest to what you need, and everybody rolls up their sleeves to just make it what you needed to be.

This is a great example – and it’s a great segue – because something that we’re about to announce… I’m not going to talk about it on the show right now, but something we’re going to announce is that collaboration between enterprises, like Shell, and startups that are building these technologies. Because you guys could probably build it in-house, but you’d have to find the people to do it; find people that are passionate. It’s better just to partner with a startup, give them some funding, and say, “here’s some funding so you don’t have to worry about paying your bills, and we’re going to be your client, and we’re going to tell you what we need, and help you build it for us. Then you can have it as a product after that.”

Michael: Correct. And in that sense, companies like Shell, we’re not – I have to be careful when they say it is a little bit – we’re not an IT company. We have a lot of IT components, and we spend a lot of money on IT systems; but in our nature, we are not a company that’re going to build these kind of platforms, and then sustain them, make sure that you evolve into the market. In that sense, we’ve adopted a platform as a service; software as a service preference. So, market standard, unless we have a severe gap in what the market is doing. In that way, we prefer working with startups. We prefer working with the market, so that the products that we use get also used by other parties, and that then evolves the ecosystem in ways that we could not have imagined if we were to come in heavy-handed and build it ourselves.

For the people out there: don’t be afraid of the larger corporations. I can only speak for Shell, of course, but we’re definitely open to these kind of collaborations, and sometimes, it’s only a five-man team that comes up with something brilliant. We definitely keep our eyes and ears open to the whole market.

Alan: Some of the best things that we’ve seen recently have been… it’s – again – that three-, four-, five-person team, because here’s the thing: with digital, as long as you have some talented people, and you have a vision, you can build something – pretty much anything! If you have a vision, and you have a strong team that can build it; it won’t take that long. A lot of these platforms have been built over the course of six months to a year, maybe a year and a half. And these are really robust systems.

I think one of the challenges that you mentioned is integrating with legacy systems. I think this is going to be the biggest challenge for any startup, building something; how do you build it to work with all the different, weird legacy systems that companies are using? That’s going to be a challenge, I know.

Michael: Oh definitely, definitely. But not insurmountable.

Alan: No, definitely not! And I think working with partners like yourself, that understand that working together is beneficial for everyone, is the key. For the businesses listening: it’s really vital that you… there’s some companies that I’ve heard, they’ll invite startups in, they sit down with them, they get as many ideas as possible, and then they build their own team and try to cut the startup out of it. I’ve seen it happen over and over again, and it’s not a good way of doing business in the long run. You’re going to miss out on opportunities by doing that. It may save some money in the short run, but by embracing these startups and really working with them, I think it’s really beneficial for everybody in the entire ecosystem as well. And it gives you the first foot in the door to acquire them, should you find that this is something you really want to have.

The second thing you mentioned was simulating outcomes using digital twins. Can you maybe unpack that a bit for people? What do you mean by simulating outcomes using digital twins?

Michael: Well, let me start by perhaps explaining what we mean with a “digital twin.” For us, the definition of a digital twin is “a digital representation of something physical,” and the reason why I say that “for us,” a digital twin can be as small as a screw that is critical to a process, but it can also be the size of a full-blown asset, or offshore plant. It takes a lot of these digital twins to work together to have logic behind them.

That then allows us to simulate certain scenarios. If you can imagine a situation where you want to perform a certain type of maintenance – whereby we have to take a compressor offline – a piece of equipment in the whole process. Then, the rest of the process usually continues, but everyone has to take a step up and run at 120-130 percent efficiency. Now, when you have these digital twins, you can perform the operation in – typically, we use virtual reality for these kinds of scenarios – whereby you simulate, “okay, if I take this compressor offline, then the other compressor is going to have to work at 130 percent. However, we also have quite detailed maintenance logs.” So, using very advanced analytics – and honestly, that’s not my field, so I won’t comment further on that – we are able to make probability statements around potential failures, what the probability of those failures are, and then, how do we visualize the impact of that failure? Does that result into a spill? Does that result into something more serious? All these variables allow us to be more prepared, before we do the actual operation. It’s all with the preparedness mindset where we use – again, usually virtual reality – but you can also use augmented reality to simulate, before you do something, “what am I going to expect in the future?”

Alan: Can you give an example? Like, a specific example.

Michael: Well, I mean, I just gave the example of the compressors.

Alan: When you’re talking about the compressors, though, what you mean? So, you watching a compressor, and then… what would that look like, when when you guys were doing it?

Michael: A good example is we had to update a piece of equipment in an asset recently, and it needed to be lifted out of the asset for the new one to be placed back in. What we did is, we had the digital twin in virtual reality, and by the power of being in a virtual reality where everything is rendered by computer, we were able to lift that piece of component up way into the sky, to detect whether it would hit any cross beams, or any other pipes that were still in the vicinity. And we lifted it up, no problem. It cleared everything. So that means that, if we were to use a crane, we could just lift it straight up and we could go ahead.

However, the new piece of equipment actually had a protrusion. The base was still the same, so the technical drawings, they looked to be okay, but one of the protrusions was about a meter out to the side, which – when we lowered it back in – we had a collision detection on the digital twin over our environment, with the new digital twin of the component. That allowed us to preemptively say, “okay, well, in this case, we’re gonna have to reroute certain pipelines, in order to get the new piece of equipment in.”

That kind of simulation is very powerful, because if you were to experience this in the real reality, you would have manpower there; it would constitute a potentially unsafe situation. And, when lowering the piece of machine, you would actually detect, “we can’t do this. You have to lift it back up.” Then you have to take action. So, these kind of processes get delayed, and we are working on the clock, with very specific permits there. So it’s a very costly affair. By having these digital twins available, you save a lot of cost and time.

Alan: Okay. Let’s just take that one specific example. What kind of cost savings do you think that created? Just a ballpark, I mean; like, is it in the tens of thousands or hundreds of thousands?

Michael: Yeah, I don’t like to go into specifics on that one.

Alan: But it far outweighs the costs of creating the digital twins, would be my… that’s what I’m…

Michael: Definitely, by far.

Alan: People think, “it’s expensive to create these scenarios,” or whatever, but the consequence of not creating the scenarios is exponentially more expensive.

Michael: Correct. I’ll subscribe to that statement, yes.

Alan: Perfect. And the last one you talked about – and we’ve kind of dug into this a bit more – is remote collaboration. Are there examples of how this has benefited… have you seen a time where remote collaboration has averted a downtime that would have costs millions of dollars to have downtime? Because I would assume for every hour of downtime of an oil-producing facility is going to be in the millions of dollars of lost revenues – or lost production, anyway. Do you a specific example of when remote collaboration saved the day?

Michael: I do not have, at the top of my mind, an example where we save the day. But we had a collaborative session last week, which is why it’s sort of at the front-and-center of my mind, where we were gearing up for a workshop to tackle a very difficult challenge.

We got into the virtual space, and we did a little bit of an intro, and one of the gentlemen who were joining us from… I believe he was somewhere in the Americas region, he explained his background and experience, and he mentioned offhand that it felt very similar to something he experienced before. So – by the power of being in these kind of virtual realities – we conjured up the documents, which were stored on a server somewhere, and we found out it’s actually quite similar and (what I was not looking forward to) what was going to be a three-day workshop turned into a 30-minute exercise, whereby we had a clear idea of how we were going to resolve and tackle the serious challenge.

For me, that’s also the power of this collaboration. It is utilizing the knowledge that might not be readily available – might not even be documented – but because this person was willing to join us in this virtual space, and we were able to connect – and not not via chat, but he was talking, we got to know each other – thereby, we linked on something that’s – well, again – not saving the day, but certainly saved a lot of time and money.

Alan: Well, yeah. I mean, if you take – I don’t know how many people were there – but let’s say five people, and you took something that would have taken three days to 30 minutes; that’s an enormous savings for any company, and just that one simple use case probably paid for all the technology investment that you made in those VR headsets. It’s crazy. The exponential savings and profitability from using these technologies cannot be ignored anymore.

Michael: Agreed. Yep.

Alan: What is the most important thing businesses can do to leverage the power of XR technologies right now? What is the thing that you would say – for a company that’s listening now that maybe hasn’t even experimented, doesn’t know really anything [about XR] – what’s the first step? What’s the most important thing that they can start to do?

Michael: This is a little bit going back to university for me, but honestly, ask the question “why” first. I’m very passionate about all the digital realities, but you have to take it into account; is it worth it for the business? Why are you doing it? And what is the differentiator?

Just to reflect back on the example I gave around collaboration; could we have done the same with a Skype call, rather than a virtual room that we were in? Make sure that you tell the story of why something is differentiating from a current capability. It might not always be down to, “you’re gonna save X amount,” but it is about the intangible aspects; you’re going to save time, or this is allows you to operate safer. But in a lot of cases, you really need to have the “why” clear, then the “how” sort of follows up on that. And the “how” typically describes which of the digital realities you would want to use, because the situation will lean towards one or the other. Again: preparedness is more virtual reality, when you’re out there in the field; augmented reality is preferred because… well, if you’re out there, you’re not going to put up a headset, and thereby lose your complete line of sight by doing virtual reality when you’re out there.

So, focus on the “why,” and then the others – the “hows,” and especially the “how much” – will tell itself.

Alan: My last question for you, Michael – and this is more of a personal one – what problem in the world do you want to see solved using XR technologies?

Michael: For me, I’m going to throw some jargon at this: it’s the idea of instant upskilling. Now, I’m going to explain that one a little bit, “instant upskilling.” What that means is – again, it’s a little bit of a jargon – but when we started this conversation, I told you that right now, the world is changing so fast, that you can learn everyday. You can learn every hour, and still something new will pop up. So there is a competency gap that is growing.

I see these kind of extended realities – whatever you want to call it – fill that void, where we have much more immediate access to expert knowledge, allowing us to go much broader and – when necessary – that you have it available to you. I don’t want to go too far into the future. I don’t want to go Matrix style, where we inject it into your brain, so you have it immediately. But think about augmented reality. You’re out there, you have to change some wiring. You may have done that specific wiring course three years ago – again, I’m just painting a picture. Then you can conjure up the step-by-step instructions of how you actually need to change that wire. Then, you can continue doing what you need to be doing, so that… I want to say “at your fingertips,” but that’s outdated – by talking to your voice-driven device, you can have that information available to you. That’s, for me, the big idea.

Alan: Amazing. I love the fact that you [said, “have all the information at your fingertips… wait a second – that is outdated!” That’s a crazy statement, that having all the information “at your fingertips” is now an outdated statement.

Michael: [Laughs] Exactly.

Alan: That was the catch that’s going to get people hooked on this podcast, for sure.

Well Michael, I want to thank you so much for your insights and your input; it’s been very valuable. I’m sure listeners – if they want to reach out to you – how can they find you?

Michael: The best ways to find me on LinkedIn. Again, I am very eager to connect to the wider market.

I really respect, Alan, what you’re trying to do. As a whole, it is about raising awareness about the potential, and then following through with it. So everyone, please reach out; follow us on LinkedIn. We tend to share what we do there, via press releases and blog posts, et cetera. So, happy to connect.