At the 15th annual Augmented World Expo (AWE 2024), Campfire CEO Jay Wright hosted a panel session with two long-time XR champions from the aerospace industry: James Cooper of Raytheon Technologies and Ryan Wheeler of Collins Aerospace.

With more than 4 decades of combined experience in XR between them, all three shared their perspectives on how the landscape of spatial collaboration has changed in recent years — and what that means for engineering and manufacturing companies going forward. 

Here’s the transcript of their session, lightly edited for clarity and conciseness. You can also watch the full session recording on AWE’s YouTube channel.

Introduction

JAY

Hi everybody and welcome. My name is Jay Wright. I'm a Co-founder and CEO at Campfire, and I’m joined today by two very special people — James Cooper and Ryan Wheeler. For most people here, you don't need a whole lot of introduction. But James, would you like to kick it off? 

JAMES

Yeah, sure. I'm James Cooper. I'm the Chief Technologist for Advanced Visualization at Raytheon. So basically, I’m responsible for knowing the use cases, needs, and opportunities across Raytheon — which is an aerospace and defense company — and looking at various solutions internally and externally, trying to connect the various dots.

RYAN 

And I'm Ryan Wheeler, Senior Technical Fellow of Advanced Visualization at Collins Aerospace, part of Applied Research and Technology, working with a team figuring out how to improve human-machine teaming and using XR as a major part of what we're doing.

JAY 

Quick background on me. I've been in this XR space for 16 years now, originally building a product and a business called Vuforia — developer tools for people building AR applications. Then I spent a few years running that group in XR at PTC, a CAD company.  

We're going to be talking about spatial collaboration today. First, maybe just a quick definition of what we mean by spatial collaboration, because it means a lot of different things in XR. What we're talking about specifically is the ability to share virtual content — mostly digital models of physical things, sometimes called digital twins — and to share it very easily, for a variety of different use cases.  

This is something that we struck out to work on with Campfire, and to focus on spatial collaboration. It's been something that's been very difficult to do in the past. Typically, you'd have to buy a number of different devices. And if you've tried to build these applications, you're familiar with how challenging it can be to build for various types of devices, take 3D content and 3D models, and make it all work together. Then you've got a whole other series of challenges when you want to try and deploy this in your enterprise environment.  

Today, we're talking about an inflection point that's the convergence of three different trends, each of which we're going to talk about. One is the changes that we've seen in devices. Another is changes that we've seen in software. And the third is about the cloud, and more specifically, security in the cloud. 

With that, I'm going to go to our first point!

Changes in the XR hardware landscape

JAY 

Devices over the past decade have evolved tremendously. Those of you that have been doing this since HoloLens 1, you know exactly what I'm talking about. There have been a lot of features that have been added, a lot of evolution.  

But one feature in particular I cannot overstate the importance of — is passthrough, and what that has done for overcoming a lot of the objections people have had to VR in the past. It's just been incredible. 

So I'd love for you guys to share, what do you think the importance of passthrough is? What problem does it solve? And what's this going to mean for devices in your organizations?

JAMES 

The story that immediately comes to mind for me was about probably about seven or eight years ago. I was doing a presentation, trying to show off virtual reality to an audience at Raytheon in our large event center. I'm up on the stage, getting ready, doing a little background. And with one of the old, tethered headsets that we had back then, I had marked out a guardian zone. And I was walking around the stage, practicing the talk — and suddenly people are shouting at me. 

So I take off the headset, my foot's still hovering in the air, and I look down and I realize that I was just about to take a stage dive. I would’ve hit the ground really hard, and it would’ve been really bad. And it turned out that guardian zone that I defined, it had shifted. 

But with things like mixed reality and how good it is today, the modern headsets and the improvements that are being made, distortion correction, things like that — that wouldn't have been an issue. I would’ve been in mixed reality mode, and I would’ve seen the real world around me. That would not have happened. 

Then I further extrapolate this thinking to our factories and our fields and customer sites and everywhere — and awareness, spatial awareness, is critical to safety. It’s critical to our customers’ mission. So these new devices make a huge difference in the ability for our customers and our employees to be able to do their jobs safely, effectively, and to come home after that.

RYAN 

Yeah, totally agree. I think that devices are finally reaching a point where they live up to expectations. I've been in this game since 2005, and so many times in the past — you see a YouTube video, you see marketing videos, and it’s really, really neat technology. Then you put on the technology, and it's a letdown. It's not anything like what it was going to be.  

I feel like the XR industry has kind of done itself a disservice in not being able to live up to its own hype. We all know what the Gartner hype cycles are. And in that way, we're kind of our own worst enemy a lot of the time. 

But if you've experienced some of these most recent generations of headsets — I've used them and I find myself thinking, okay, I want to find a place for this technology in my life. I don't necessarily know, because spatial computing is still kind of a new thing, but we've been talking about sports applications at this conference — and I could totally see myself being able to experience a sporting event sideline. Because the quality of the headset is now comparable to the quality of my television, but being spatially present in that event, that would be so much better. 

At the same time, you take work examples. This is a real-life example. I was flying, and somebody had their laptop open in front of me. And you glance, you know, you can't help it. We're human beings. Something catches our attention, and we glance. And when I glanced, I noticed that it was a PowerPoint slide that somebody had open about a merger and acquisition for their company. That's not something that you should have open on a plane. 

But this technology could now allow us to work privately in a public setting, and still be able to interact with the flight attendant as they come around with drinks and food and things like that. So it's finally getting to a point where it's comparable to the alternatives. And that's really the key thing, I think, when it comes to hardware.

JAMES 

Yeah, we've done a lot of demonstrations through our immersive design center, with a lot of people — a lot of adults, engineers, and business leaders. But we’ve also done a lot of things with students. We go to schools and support STEM events, with middle schoolers and elementary schoolers.  

And we've had the opportunity to run them through earlier generations of devices, up to the current generation with things like mixed reality and hand tracking and improved UIs. The sense of presence and immersion that they get — they get it so much faster, and they stay in it so much longer. They stay so much more engaged with these newer generations of technologies.

JAY  

As you think about people in your organizations that might have dismissed VR in the past — maybe they've got a headset that's sitting on a shelf, or even people that just aren't even willing to do a demo anymore and they say, ‘No, I've done that thing, I can't do it, I get sick or I'm uncomfortable.’ It's like, I know that was your experience before, it was actually my experience before, but try this. 

And going from where we were with everything else before, to Quest 3, it's a night and day experience. Lights go on in a way that they did not go on, people are more comfortable in a way that they were not comfortable, and it's really changing everything. I can tell you, as someone doing software and solutions for it, it's just been a night and day difference.

Changes in the XR software landscape

JAY 

Moving on to our next point, I'm going to hit on software. There's a lot of challenges in building software for these devices, a lot of challenges in making it really intuitive for users. 

I think a lot of the technology-focused folks, we like to come up with new and innovative interaction methods and cool things that we think are really, really neat. And then sometimes you put it in front of users and they're like, what the heck do I do here?  

It's just so important to adhere to basic principles of what people already know how to do. Could you speak about some of those challenges in the past, and the extent to which new software is overcoming those challenges?

JAMES 

I'll kick it off, then I'm going quickly hand it over to my colleague here, because he's the UI/UX guru. In aerospace and defense, we don't win awards for UX design. If anything, we get angry letters about our UX. 

So, this is something that's really important and valuable that has long required addressing. And I think we're starting to see people out there in the world — the Campfires and everybody else in the world — really starting to take it seriously, and it's a great thing.

RYAN 

The thing that's difficult about XR — and there are many things that are difficult about XR — but one of the big ones is the intersectionality of hardware and software. And practically all technologies have this intersection of hardware and software, but it is magnified by orders of magnitude in the XR space. 

So we can have these really cool devices, but if the software that runs on them is clunky, is kludgy, is confusing — then we might as well not even have the devices.  

You asked about how we’ve done this in the past. So, my team created a virtual prototyping application back in 2010, and it's still alive and well today within Collins Aerospace. And the thing that was the breakthrough for us was moving away from the handheld devices. Back in 2010, the handheld devices for XR, for virtual reality, were not that great. They were like 1950s, 1960s TV remotes. 

So we went to an entirely gesture-based interface. Your hand was tracked, and if you wanted to grab a tool, you would reach into the toolbox, and the tool that you touched was the one that would attach to your hand. Then if you wanted to set it down, you would simply touch it to the toolbox again. There was no finger movement or anything like that, but it was a very close analog to what you do in real life. 

As a result, we were able to bring new people into the environment, and they were using it and proficient with it, in a matter of less than a minute — even people who are self-proclaimed technophobes. 

I think that the same thing applies with MR. There are a lot of mental models, to use Don Norman terminology — a lot of natural interactions that we all have built in, we all have internalized, that we could bring to bear in augmented reality and mixed reality. 

I'll give a quick example. On day one of AWE, we heard the example of, ‘Hey Meta, tell me more information about this building’... said no one ever. That's not something that a lot of us would be comfortable saying out loud on the street in front of everybody else. But what Meta's doing there is a very good thing. It's just how do we get to that end? 

If we were to think of the world as a webpage: there's no buttons in the world, but there's things like hovering on things. If I were to set my sight on that building and dwell with my sight on that building for a couple of seconds, it would be the equivalent of a mouse over. And when I mouse over something that's interactive, a line appears around it. So that's a signifier to help it be discoverable to me that it's interactive. And as I'm keeping my eye on it for a couple of seconds, information about it pops up. That's a tool tip. 

Those are all equivalents to mental models that we have in our daily lives right now, that we don't have to expend a button on or speak to; it's just a natural way that we interact with digital technology right now, today. We can extrapolate those things. 

There are going to be some new paradigms, like Apple Vision Pro brought the pinch to wherever your hand happens to be, and that's a very clever way of going about doing that. Maybe that's going to be the new paradigm in the same way that Apple brought new paradigms to multi-touch interaction with iPhones and iPads. So it’s a brave new world, with a lot of really cool stuff going on!

JAY 

Yeah, the Vision Pro one is interesting. What they did with pinch and gaze really comes across as just an analog for a mouse in space, right? And everybody identifies with that model. I get it. This is how I move my mouse. This is how I click the button. It makes a ton of sense.

Designing XR for both headset and desktop users

JAY

Another aspect of software that I think is forgotten when we start talking about XR, is there's this tendency to think that the entire experience, the entire application, or the entire system is on the headset. And the reality is that the headset experience is super important, and it's got to be super, super easy to use — because my goodness, it better be, for me to go through the effort of putting that thing on my head and enduring whatever it is I've got to do. 

But there's a whole bunch of other people involved in this workflow that are not in headsets. And it might be because they're unwilling or unable, or it just doesn't make sense because they're sitting at a desk somewhere. So how important is it to be able to consider desktop — and what we call flat screen users — and have a really simple interface for those folks as well? 

RYAN 

It's absolutely crucial. And not brag on you too much, but that's one of the smart things that Campfire has done. When we created virtual prototype modeling, we had been trying to figure it out for about four years using commercial off-the-shelf technology. This is a long time ago, and most of these companies aren't around anymore, and things have improved a great deal as well. But we were trying to figure it out and we realized that there were three modalities that we needed.

We needed an immersive virtual reality interface for first-person, hands-on interactions. We needed a Powerwall virtual reality interface for when we had large audiences who needed to be able to understand the content quickly. And then we needed a desktop interface, because almost no one has a head-mounted display and almost no one has a Powerwall virtual reality system. 

And if you could make that desktop interface as user-friendly as anything else, then your ability to help the masses gets magnified by multitudes. That's why whenever I'm giving a demonstration or talking about virtual prototyping, we have those three modalities. And at the very bottom is the desktop interface, and I'll always say it's the unsung hero. It's 70% of the value that we get out of this technology.

JAMES 

I said earlier about going to various schools and STEM events and having other demos within our company. And initially, you know, with Campfire, it was their augmented reality headset that they offered back when we joined them in the pilot program years ago. But then afterwards, we were seeing them using an iPad at trade shows or using the computer interface. So we started saying, let's try that, too. Let's see what value that brings.  

And to us, it was helpful to be able to have that desktop or the iPads as part of it. But then we started realizing that other people, adults and children alike, wanted to see what was going on and they want to interact via these other modes, too. And of course, you see kids on their touch devices, and it is natural and enjoyable to them.  

So, having these different modes to interact with, and being able to interact with each other through these different modes — it’s been a real eye-opener how valuable of a tool it really is.

JAY 

And I think for us, the headset experience is the thing people get excited about initially. That's kind of the sizzle. But inevitably, most of what people are using in deployments — in most use cases, there's more users that are on flat screens than in headsets. And that's okay. That doesn't mean there's a value here. It all needs to work together.

Secure cloud-based collaboration 

JAY 

Let's switch over to our stickiest of topics, the cloud.  

Coming from a CAD company, I came to understand pretty quickly the importance of security with CAD. In software land, our source code is kind of our crown jewels. In physical products land, CAD is sort of the equivalent, and it's got to be kept private and secure. When it comes to defense and aerospace, it's literally a matter of national security. 

So, clouds are here. Everybody has this SaaS stuff. But it has to be secure. How big a problem is this, for you to adopt new SaaS solutions? I've heard you talk about it as various forms of ‘just say no.’ But, what do we do to solve it?   

RYAN 

Well, James hopefully will be able to speak to the solving it a little bit better than I will. I can certainly talk to the ‘just say no’ side of things. So, you know, aerospace and defense, it's essential that our technical data and our intellectual property are secured.  

A quick story. I read a story about a cologne called Eight & Bob that I had never heard about before, but it was about how it got named and how it became a marketed product as opposed to a specialty product. I read it out loud to my wife one morning.  

After I finished reading this story to her about this cologne, she opened up her phone and started scrolling through social media — and the first thing that she saw was an Eight & Bob ad. And she said, I've got those features turned off on my phone. This isn't something that just aerospace and defense should be concerned about. This is something that all enterprises should be concerned about. 

If there is a technology that you are streaming intellectual property through — your CAD models, your conversations about those designs, your interactions with those designs — if that's being recorded by somebody, then it could be very bad for your competitiveness as a company.

So it's a very difficult situation to be in for enterprise. We all need to go the direction of cloud. It's just the future. The question is, how do we do it responsibly?

JAMES 

I saw a graphic a few years back that was really interesting to me. It showed the iteration of when the United States would release some kind of defense technology and then when our kind of competitor states, these foreign actors, would release their version of it. And the time that it took between us releasing it and them releasing their iteration of it — it kept getting shorter and shorter over the years. A lot of it was actually increasing sophistication of their espionage abilities. And of course, having things on networks and servers, that has increased the ability for them to incur upon these systems. Some of it, of course, is local espionage actors. 

So, it is drilled into us. It is absolutely vital for us to be able to protect this information. National security, the security of our peers and partners out there. So in our companies, we have very strong policies and procedures and measures in place.  

On the other side, commercial cloud, when it came out — we understand the benefits of commercial cloud. Scalability, being able to spin up different resources, accessibility all over the place. There are many benefits to cloud out there. But commercial cloud just didn't meet the rigor that we require for these capabilities. 

But now there's GovCloud. I thank and sometimes apologize to Jay many times, because he's been through this long, painful journey with us where he said, ‘yeah, I'm going to support you. We're going to go on GovCloud. We're going to deploy this.’ So we've been working with Campfire since December 2020, and there's still a lot of rigor involved in this, but the government has basically said GovCloud is okay by us. It meets our standards, as long as you're also meeting your internal standards that you've had over the years. So that is a vital component, now, to us being able to use technologies like Campfire or other cloud-distributed technologies for aerospace and defense applications. And so, it's terrific because it's really opened up a lot of doors and capabilities to us.

JAY 

Yeah, it's been great. I think the GovCloud phenomenon has been huge. And then with SaaS vendors like us, providing hybrid cloud architecture where sensitive data can remain under a customer's own control and a customer's own GovCloud — that’s the answer. It’s so great to see that we're finally at a point where that can happen, and that you and your teams are leading the way.