On demand | Elevate 2022

KEYNOTE  •  THOUGHT LEADERSHIP

Machine learning and artificial intelligence have been at the forefront of optimization and efficiencies in the airline industry, from operations to dynamic pricing.

What can we expect from the next 10 years of innovation in AI, and how far can we take the technology? What will be the new data needs, and will machine learning drive stable profitability? 

Thomas: I will talk about machine learning and what it means for revenue management, and in particular the bottom line of the industry.  

This picture is an illustration of revenue optimization from machine learning. It's just to give a little teaser here, so we can see the magic of machine learning, and of course we wonder what will it mean for revenue management.  

So to predict the future, it's maybe useful to first start taking a look at how we got here, the history of revenue management. And what you see is the evolution over 40 years. You have from its inception in 1980s and all the way up to now and then beyond. It's shown in chronological order. You have the three lanes here, the business objectives of the airline, then the distribution, and then technology.  

So if we start in the 80s, revenue management really came out of deregulation. Airlines were certainly free to set their prices and schedule, and then they needed a revenue optimization tool. That was the leg-based revenue management system. And then if we move to the 90s, the airlines’ business model changed from point-to-point to hub-and-spoke, and consequently they needed a new revenue management system that could support them. So that is the network revenue management system.  

If we then move to the 2000s, that was the entry of the low-cost carriers. And the low-cost carriers, they were really focused on reducing the cost so they wanted to distribute their products over the Internet and the rules and restrictions that is associated with the fare structures, they essentially removed it. That was easier for them in order to distribute, and the legacy carriers follow through because they still wanted to be competitive. The issue, however, was that the revenue management system that was developed initially in the 80s and 90s, they did not really handle these fenceless fare structures. So there was a massive buy down and the airlines switched off the revenue management tools and instead became rule-based, which is, you can say, a kind word for Excel steering. And then it took maybe a decade or so to recover to develop the science and technology to optimize simplified fare structures, first with a single fare family and later with fare families and branded fares. This is what we call choice-based revenue management.  

This leads us up to today where we have dynamic pricing, and dynamic pricing is really interesting. It enables the airline to, in the shopping session, dynamically adjust the price in response to, you could say, the customer, the context of the customer, but also the competitors’ prices.  

And if we go even beyond, we see dynamic offers and now the dynamicity is beyond just the price, it's also on the product side. So how to tailor a specific product and price to a customer. That is what we call offer management. We will see the impact on that as well.  

Now let's take a look at the revenue. It's basically the same journey I just went through, but now from a revenue perspective. What you see on the left side is the Avenue index, 100%. It is the avenue of a leg-based system with fenced fare structures. And now if we move on 10 years with the network revenue management systems, you see a 2% revenue gain that is really sort of the industry-accepted value of bringing O&D control.  

And now look at what happens when we come to the 2000s with the entry of the low-cost carriers. You see a massive drop of 20%, and that is really when you use your legacy revenue management system in a fenceless environment. And then you see gradually the improvement that we made over the years. It took a very long time to get back with the fare families and finally dynamic pricing where we stand now. And the revenue index is 95%, you see. It's actually less than what it was in the 80s. But we also have to remember that the optimization problem is so much harder. And also I added in two new bars, machine learning and offer optimization, and I will unhide this little picture at the end of my intro so you will see what is the contribution of these two.  

Now let's move on to machine learning and artificial intelligence. And of course the time I have here I cannot go through all the history of machine learning as well. But it's interesting to see the different periods. You have first, initially general AI where we were concerned with human intelligence, how we could make machines do what humans can. And then data expert systems, that is where the programmer instructs the program to solve specific tasks. So it's very task-specific. And finally we come to the modern area of machine learning, where there's a lot of infusion of mathematics and statistics. And here the goal has changed to basically the machine will self-learn from data without being programmed to solve the specific topic. So you see all sorts of algorithms that has occurred as a result of that. And lately we have deep learning, which is maybe the preferred machine learning model today. And applications, it's quite wide. You have natural language processing, translation, self-driving cars, robotics, also, folding of proteins, plasma control, et cetera.  

It's really a marvel and therefore it's logical for us to think what would it really mean for our revenue management? The changes we could anticipate could be all the way from minor like modular, maybe you have revenue management, you will replace certain functions in the revenue management, for example, no show or cancellation computation.  

But we may also have hybrid models where analytical models that I will talk to in just a moment, and machine learning work hand-in-hand to come up with better forecast. We could also anticipate that machine learning could extend the scope of revenue management. So remember that revenue management only deals about optimizing the flight, while offer management would like to optimize the total offer, so flight and ancillaries, but it could also be entirely disruptive. I do not believe that that will happen, but you know.  

So to give you a little flavor of the two schools, we have here on the left hand side you see the traditional revenue management, and on the right hand side pure machine learning. And we are looking at a problem of single flight optimization. The equation you see on top is the so-called bellman equation, and what you notice already is that this equation is the same whether we solve it using the traditional revenue management approach or the machine learning approach. What you do in revenue management today or typically is that you on the right hand side you see the P, which is the transition probability, it's basically the forecast. And in an analytical model, you have a precise formula for the forecast. It's one given below the one we use, but the other expressions could of course also work. You have a volume, you have a day, a week, you have special peers like Christmas, Easter, and then seasonality, pick up curve and purchase probability with parameters you can interpret. So the user would be able to understand what comes out of these parameters and can intervene on them.  

On the machine learning side, you still want to solve the same equation, Q. Here is the expected revenue to go. And instead we think about it more broadly. We say, okay, Q is a function and any function can actually be modeled by a neural network provided it's complex enough, and therefore we can train the neural network based on observations. So what we tried, the prices we tried, and what happened, so the result is a function, it would be more like a black box because you have a lot of parameters and here I have written 100,000, but in reality it could be much larger. So these are really the two schools.  

To compare these two schools, I made a little grid, A two-by-two grid. So what you see on the left hand side is depending on domain knowledge. So how much do you know about the problem? A little or a lot? How much data do you have? A little or a lot? So let's start in the low corner here. You basically do not know too much and you don't have a lot of data, so there's not so much you can do either. And so, for example, you could do linear regression.  

If we go up to the pure machine learning, the domain here is that you don't know too much about the problem, but you do have a lot of data. That could, for example, be playing video games, where you could generate data by having the computer playing against itself.  

And if we go to the right hand lower corner with the analytical models, there you have a lot of knowledge, for example we looked at with the forecast, but you do not have in general too much data.  

Let's take a look on the right hand side where I have compared the analytical model and machine learning models on different axes, and you see that the analytical model has an advantage on the data side. So it can handle sparse data. It can extrapolate, for example, to prices that you have not seen, it can handle non-stationary data which you typically have in the airline, and the parameters were interpretable. While the machine learning model it basically has an advantage in terms of domain knowledge, you don't need to know so much about the problem. It can handle much higher dimensions. We can easily extend it to new data sources and also it typically have a higher accuracy if there's enough data.  

You see they have different pros and cons and therefore it's natural to think couldn't we really combine these two and take the best of the two worlds? And indeed we can.  

So what I'm showing you here is real airline data. Since it's a little corrupted, there's a little figure there on the top where you have the analytical model that feeds into the neural network. So basically we have the forecast model you saw before and then you feed the error of the forecast model into a neural network that is trained to predict or compensate the forecast model. What we see is that the machine learning model can actually identify a feature that was not included in the forecast model, which is the sales day of week. What was in the model was the departure day of week, but of course, we could include it in the forecast. What we also see is that the average correction that the machine learning model does is 8.5% and you have a histogram below where you can see the sizes of the adjustments, and it leads to improved forecast quality, and there's a little table to the right. You see between 2 and 7%, there are two different fare families and two different KPIs, root mean square error and mean absolute error. So if we take for example, the mean absolute error about 7%, we can convert that to revenue by dividing by about 5, So you get maybe 1.5% improvement from that.  

Now let's move on to offer management. Because our goal or end goal is really to optimize the total revenue. Again, remember that RMS concerns the flight revenue only and offer management is the total revenue. We can also draw the journey that I explained from the 80s and all the way up to now in offer space. This is the IATA capability matrix. You have price determination on the X axis with increased sophistication going from filed fares all the way to dynamic prices, and on the Y axis you have similar on the product side. You have static products and then all the way up to full dynamic bundling.  

And what you see is that we start in the lower corner 1D with RMS and then it took 20, 30 years to go from 1D to 1C, and only recently that we moved horizontally with dynamic pricing, and now we are maybe at 3C depending a little on vendor. And then the next leap we are going to make is going from 3C to 3A and therefore we may ask ourselves, is this really a disruption or can we rely on all the work that we have done over the last 40 years? And luckily we can.  

What I'm showing you here is how we believe that we can construct the optimal offer. This is sort of a conveyor belt where you have RMS starting with having a flight, it computes the bid price and then it sends it off to airline dynamic pricing that computes, it takes out of the bid price up, and then puts back the flight with a ticket on what is the price of the flight. And then we have a recommender system, for example, machine learning-driven, that would propose ancillaries to the customers. Ancillary dynamic pricing is the purple crane. It takes it up and pushes down the prices of the ancillaries and then offer optimization, the last step, it builds two offers, you see a bundle offer with flights, see them back, and then another offer with the flight alone, and all these pieces fit together.  

In the equation below you can see the profit of the offer set. Because there are two offers, it can be computed as a sum of offers and then you have the revenue of the offer, you have the price of the flight, the price of the ancillaries, and then minus the cost, minus the discount for the bundle, and then times the purchase probability of the offer and the offer set.  

The important thing to take away from this is that all the work that we have done over the last 40 years they're still useful also in this offer optimization context, and also machine learning will play an important role in, for example, the recommender system, even though it could be applied many other places as well.  

So that leads me to the slide where I will reveal what are the revenue benefit. I see this, a little more corruption with the arrows, you should see the revenue actually goes up, so you see the last one going from 95 to 99.7 and then all the way up to 103.7. So finally after 40 years or more, we are now above the revenue where we started. The way this assessment has been made is that for machine learning, I illustrated it with an improvement on the forecast where I assessed the revenue benefit of 1.5%, but we can actually do better by injecting other data sources, for example, competitive fares, and then we can have an even better forecast to inject into revenue management.  

And for the offer optimization, this was work done in collaboration with Kevin Rank, which was presented I think a couple of weeks ago in Agathos where we demonstrated that we can increase the revenue of the ancillaries by 20% and here we took the assumption that 20% of the spend is on ancillary, so 20% of 20%, that is the 4% you see here.  

So with that I will conclude and just say to reach this 5 to 8%, the benefit of machine learning and offer management, we have to master execution, data, and science, people and organization, and distribution. And then I will call on the panel where the idea is that we can maybe go deeper into these topics. Thank you.  

Tom: Thank you very much, Thomas. I hope that was helpful for everyone to put context on the evolution of the science side of the dynamic offers and revenue management, that's really what we're going to debate now here on the panel, how far the can the science take us.  

So let me introduce some of the industry experts in this space. First of all, I want to bring up on stage Richard Ratliff from Sabre, Michael Wu from PROS, and Sam Chamberlain from Flyr Labs. Welcome, all.  

I'm going to make a little joke before Michael. Here. There's, that's what was missing from mine. And if you saw the LinkedIn post, he said what's missing on his avatar, his picture, the hat. You can see that I never do this, but I have that on my avatar as well. So thank you all for joining.  

I really think we're not going to have much time for me to talk because we had a lunch break and all these guys have a lot of really interesting things to share. On the different aspects, I'm going to go into how Thomas left the area of opportunity or exploration. There's three main themes that I think we want to talk about here. And I'll start with the first one. As we think about big data, machine learning, AI, let's talk about data. What is the data that's really going to help advance us to the next generation of the science and where it's going to go? I'm going to ask maybe Richard to start out that question on data. What do you think about the data needs?  

Richard: Thank you, Tom. Well, first, good job, Thomas...  

Thomas: Thanks.  

Richard: ...for setting the stage with it. I think there's two things really appropriate to the data, two concerns. And really that they have been looking for. The first is in today's environment. With the airlines filing fares, it's relatively easy through the ATPCO forum for airlines to look where they stack up relative to their competition. But there's a trend towards more private fares and NDC is accelerating that. So with more and more private fares, I think it's making the competitor actions of the competitor fares more opaque. And that's exacerbated by, as Thomas had mentioned, the increasing complexity of the offers. Okay. So I really think that the airline pricing the way I did when I was in a pricing department is going to have to change going forward. There's a lot of valuable information that is being lost and that I think either the processes are going to have to change or we're going to have, as an industry, have to do something to fill in and complete those data gaps.  

I think the second big thing from a data perspective, the, as you've heard all the speakers talk about throughout the day, has been the emphasis on personalization and the trend towards personalization. So customer data becomes a really important aspect. And customer data is a lot more complex than most people realize because it's not just about keeping Michael's frequent flyer information and tracking his recent purchases that he's made, the kind of classic recency, frequency, monetary spend, and everything in a database. In today's environment, in Europe in particular, but also increasingly in the United States and Asia, with a global data privacy concerns, okay, at the consumer level, with private data privacy concerns, I think we have to find a way to make this personalization work. But make it work in a way where it's, I know this is an oxymoron, but it's one I like to use, where we have anonymous personalization, it's personalized but it's personalized at a cohort level, people who fit into this particular group have these particular preferences and respond well to these kinds of ancillaries. They have these sensitivity to price versus schedule, et cetera. I think we can make really sophisticated systems that operate well at that anonymous level, but they respect data privacy. I think that the union of taking the customer data, it's obviously, if we're using it in ways that are advantageous to the customer, having that customer, his specific customer history, is going to be really useful. But putting those together is something that's going to take time to evolve. And I think we have to make a lot of efforts to collect those data and use it because it's going to be important going forward.  

Tom: Good start, Richard, thank you. I want to maybe turn to Michael. I know you did a white paper on some data, different aspects of the data. Maybe you can talk a little bit on your thoughts on what can we do as an industry or what data needs do you think we have?  

Michael: Sure, I think certainly I do agree with a lot of what Richard has said, but I also like to offer a different perspective because I feel coming from the consumer world, I think yes, privacy is important. But consumers, especially the millennials, they don't have any problem giving up the data if they know what value they are changing for. So I think that it's good that we go down this personalization path. But ultimately if you look at the consumer world as an early indicator of where it might end up, this is basically going down to segments of smaller and smaller number. Eventually you're going to end up with a segment of one, that's what we call hyper personalization, hyper personalized to that one person. And in that case there's really no need to do any aggregation. It's actually easier from a technology perspective because you just match to that person's preference. It's not just matching to his preference nominally, but it's actually context sensitive, whether I'm traveling for business travel versus the personal travel, whether I'm traveling on vacation versus during the week, a workday, that's going to be different based on my past history of travel before. So I think there's ways that we can actually make these kind of very personalized offers attractive and without violating too much of the privacy by making sure the value proposition upfront that they're actually agreeing to opt into this. So there's definitely that aspect that I want to talk about.  

And certainly I think there's data about the consumer which I think will be key for a lot of the future airline offers, but there's also lots of data about the market. So I think that's something that every airline is trying to protect, but there is actually substantial value in sharing those data. Because we actually, during COVID had an initiative called a COVID task force where we actually help our airline customers build this model to predict recovery. And what we did is really unprecedented. Just like COVID, unprecedented. We actually pulled together over dozens of airlines booking and basically pull them together so that we use this data collectively to build a model, a joint model. And we were able to demonstrate that this joint model actually predicts better than each one of these individual models on average. The error reduction of the joint model was about 82%, which is actually very significant. So there is value in sharing data as well.  

Tom: Thank you. So we heard about competitive data, customer data, how do you, what's the right level of anonymous or personalized customer data and data sharing. Sam, any thoughts on data from Flyr Labs, what data are you thinking will help advance and move us forward?  

Sam: Yeah. First of all, I agree with the comments Michael made, but something that I've also found speaking to airlines over the last few years, they don't necessarily want to go to that most granular level of personalization. Now I have to deal with PII and GDPR issues and things like this that Richard talked about a little bit. So I think there's a level slightly above that that's maybe anonymous personalization that's abstracted a little bit. That's the right, happy medium for airlines to deal with and the obvious sources and where we go to most often are things like shopping data and personalization attributes like can identify loyalty status and things like this.  

But at Flyr, we're interested in all the different data sources that help set the context around which airline revenue management and pricing analysts have to make decisions. So that could be social media information, it can be event information, what's happening in the market, what's important that's going to impact the pricing decisions that you make. We're passionate about collecting data from a lot of different sources, but collecting data is only one step in that journey. I was at Aviation Festival last week, and I imagine a lot of people in this audience were, and somebody gave a presentation that said stop giving me data. Like, I have a lot of data, I have enough data. The next step that airlines and vendors need to take is to take meaningful insights from that data that they've got and start agreeing across the industry what are important metrics, what are important aggregations of that data, how are we going to use the data in a common way? And I think a lot of time and effort needs to be put into that as well, as well as collecting lots of different data sources.  

Tom: Last but not least, Thomas, do you have anything you want to add? You gave the 18-minute opening.  

Thomas: Well, I think that one of the important data sources would be competitive fares. We did not talk about, I mean, we talk about customer profiles, customer data, but collecting information about what is available at a given point in time from across the industry. That is really what we would like to have to make better forecasts. That drives a lot of the revenue. And I believe also that that would actually also be able to share that among the airlines because you rather prefer you know a clever competitor than a dumb competitor, essentially. And providing that we have a common data source, I think could actually also be helpful.  

Richard: I agree.  

Tom: And I'll kind of self-serve a little bit, ATPCO, we've done that well. We've been an airline clearing of data from different data sources. Do you think there's opportunity to go to different data types and do the same kind of thing if we want to talk about customer data or shopping data, we've done in the past that the industry with DDS and some of the MIDT and kind of shared across companies. Is that something you think that there's an appetite to approach that and go after that?  

Michael: I think so. I certainly think that would be a tremendous value to the industry. I think that. So I've been involved in several industry consortiums like organizations in my previous role and one of the huge movement is that this data about the customer is yours to keep, your customer is your customer and no one's going to, it depends on how you're going to use it. How you are going to serve your customer, that's up to you. But data about the market that will benefit other people. People pull together because everybody gained something from it. Even though the gain may be different, but everybody do have a gain, do realize some gain. And that's exactly what we see from our study as well, that 8.2% improvement in forecast accuracy. You know different airlines realized different gain, but everybody had a gain.  

Sam: I think if you're seeing measured results in sharing data, I think it's fair to assume that everybody participating this and still working together in a collaboration that creates datasets that different players, vendors, airlines have been involved with is still valuable and still important. The only thing I would say, and this is how we come at it from Flyr as well. We don't want things to slow down, so we don't want that collaboration in defining new datasets and coming up with datasets to slow down the pace of innovation in the industry. And I think you also have to be aware and cognizant of if there are players that have not participated in that, does that create a separation in playing this game or those that are contributing to the data and those that aren't that are going to behave differently or feel differently or feel at a disadvantage because they weren't part of this group of creating a common data set. So long as it's available to all and so long as it's not slowing the industry down, I think we would be completely supportable.  

Richard: I agree with that. I think beyond the research benefits and everybody benefits from the research part of it as well. I think we're in a situation, especially with NDC and the hyper personalization that Michael talked about, that the airlines are going to be experimenting with a lot of different things. And it's going to be really important to know whether those things are actually working or not. So having the competitor insights and how things are responding to customers I think is going to be really important. Without that, we'll just create a lot of churn in the industry. And we could end up actually in a worse place even than the one where we are now. I think if it's well thought out, we can, I genuinely believe that we can end up in a situation where it's better for the airlines and better for the customer, so excellent. 

Tom: Any other comments? No? I'm going to pivot to the next.  

Sam: There's been a couple of failed attempts by different players in the market at trying to consolidate and aggregate data for these purposes. And I think whoever wins at this, whoever is successful at it, really needs to have a win-win business model that everyone's going to benefit from. And it has to provide real blanket market coverage. It has to be applicable to 90+ percent of the market, for example. I think that's really important as well and those that have done that have been successful at providing data.  

Tom: You're smiling, Thomas.  

Thomas: Well, I'm smiling. It's not every airline that wants to participate. I mean, it's all, I mean, typically the low-cost carriers would maybe not like that but I think we can never expect I think full coverage, we have to sort of manage anyway.  

Michael: I think I had to start somewhere, right? It isn't a company like ATPCO, it doesn't start out with every single airline participate or not, not even close to 90% coverage, right? So yeah, but I think if you do have some big airline participating and people the big fear is often, what if I opt in then I don't get value and that becomes a competitive advantage for other people against me. So that's the big kind of fear there. But during this study, in this COVID task force study, we actually created a mechanism where all the airlines contribute their data, each one airline contribute there, but we pull together the data and then feed it through a model and then the airline, all the participating airlines, have access to the output of the model. They don't have access to the raw data themselves. But they have access to the output of those models that we built and those are the insights that we could provide to all the participating airlines and that's hopefully a lot safer mechanism of distributing these insights and data.  

Tom: So we said we're going to talk about three points. We've covered data. I'm going to go to the second point, science. Where is the science going to take us and maybe open the parentheses, not just the science in this machine learning and AI, but actually the industry transformation and the technology advances that we're going to do as part of that. So I don't know who wants to open that one. Sam, do you want to start us out on that?  

Sam: Yeah, we based our entire product offering and business model on data science intelligence and AI and ML. So we've kind of set that at the foundation of everything we're doing because we wholeheartedly believe that's what's going to drive the future of the travel industry, and pricing and revenue management in particular. I think there's a lot for us to learn still and especially across different industries. We don't need to restrict how we're evolving and how we're learning and how we're adopting data science to only what's happening in the travel industry. It's become a bit of a cliche over the years to say that airlines want to be like Amazon of the world, or things like this, but they are trying to become retailers and more sophisticated merchandisers and I think we can look at how data science has worked in different industries and start applying learnings from that into our own industry with a lot of effect.  

Tom: Maybe I'll go to Thomas. You kind of did this slide, you went from revenue management to offer management. Is there science evolution that has to happen to get us there?  

Thomas: Yes, we were scratching the surface, I would say on the offer side, I think what we presented at Agathos was sort of the first attempt where we were able to build offers at scale with ancillaries. But still this model has to be industrialized, you have to collect data, you have to try how they pan out and so on. So there's a lot to be done there.  

When you mentioned on the science side, I think that I brought it up with this convergence that is between analytical models and machine learning models, and I find that really interesting that it depends where you're coming from. But it seems like regardless of if you are born in the old school of analytical model and you want to inject machine learning, or you are born from the modern world with machine learning. Then you want to add in knowledge and analytical models to stabilize it and so on. So that's how I see it. These two worlds are really nearing each other.  

Richard: I think too that as systems become more complex, one of the other really important facets of it is that the results be interpretable and explainable. We talked over lunch about this at length, and I think one of the conclusions that we came to is that users don't want to have black box solutions. That's one of the dangers of just setting up a model that feeds it a lot of data and comes back with an answer without a lot of user input or user guidance. We really want to make sure that the models, and as researchers this is on us, to make sure that the results are really a glass box instead of a black box. It's important for folks to be able to understand that, and I think part of the reason for that is that we run the risk of looking at one particular application in isolation and over engineering, over engineering and over optimizing it to the exclusion of other areas. Like, say for example, focusing on revenue management without really taking into consideration pricing or advertising or all the other promotions and frequent loyalty and all the other aspects that come up. And so if the models are more interpretable, then it makes it easier for the users to understand all the different components and sort of put that together to make sure everything is working in harmony as opposed to working against each other. That's something I'm in particular really passionate about.  

Thomas: Yeah. So just to maybe complement a little. So that's really why I think you have to combine these analytical models with machine learning because you have an analytical model that is interpretable. It gives maybe 90% of the answer and then there's some randomness that is very hard to model and you let the machine do that. I have a mental picture of this. If you have a boat that sails, you can have the analytical model describing the trajectory or whatever, the path the boat takes, but then there's a lot of turbulence. I mean the ocean is complicated, you cannot possibly try to model all the currents and so on, and that is really where machine learning takes over. That you can sort of predict tiny adjustments to the boat's position. And that's how I see sort of them playing together.  

Tom: Michael, I didn't get to you, do you have any comments? Otherwise I'm going to jump to the very important one, I think is the last question.  

Michael: I think there's certainly some thoughts on that because we should talk about this glass box approach. I think that's actually the approach that we use at PROS trying to build some of these forecasting models, specifically, we're using shopping data to help us forecast demand and booking. And the idea is that, as Thomas has said before, you could use some interpreter model too in conjunction with this black box type of machine learning model. And so our approach is basically use a simple interpreter model in the beginning and then fit the data in successive stages of more and more complex models. So that each stage is actually trying to fit and explain the variance that is not accounted for by that stage, and then you fit the next stage, and at the very end you can even use a completely black box model, and the beauty of that is that now you know early on the many stages actually perfectly interpretable, even if your last stage is completely black box, because these type of model, the error actually decreases from, the residual decreases from stage for next, right? So the black box, even though it's unexplainable, it's fitting on very tiny residual, but it doesn't really matter, it doesn't contribute very much to the decision anyway, right? So it's actually a good way to look at that black box problem. So you fit in this glass box approach.  

And one more thing, I just want to say that this black box problem is really a misnomer because they're actually a huge effort in the AI community for explainable AI. Explainable AI is also a misnomer because it's not explainable AI, It's actually not AI, they're actually algorithms or technology that make black box AI algorithms more transparent. So there's actually a lot that's being developed there and I won't have time to share all of that in depth with you. But it has made a lot of these so-called black box models a lot more interpretable, which are major contributing factors to the decision and all that so we could score the attributes and all that. So there's a lot of stuff that can be done.  

Richard: That's the danger of having a bunch of scientists.  

Tom: Yeah. Yeah. Well, I'm going to go to the last question—  

Richard: Before you do, I got to bring in two other points, I think.  

Tom: Wow.  

Richard: But to power all this, everything needs to be data-driven. So it's kind of incumbent. I mean we talked about the importance of data and everything. But we want these models, we want fact-based, data-driven decisions and everything from the models.  

And I think another really important point is because the complexity of the offer generation processes, as Thomas very eloquently described, is increasing, that the automation is much more important. I think one big difference we've seen over the past few years is the rise of what's commonly referred to as machine learning ops or ML ops. Google, for example, has a great platform called Vertex AI, which has enabled us to take new data as we ingest new data, to be able to automatically recalibrate models based on the new data and then actually test it, make sure it's working better and put that in production and do all of that automatically. We can do all the testing and make sure that it's safe and everything before we actually go and do that, but that saves a lot of human effort and from a business perspective it's a lot more scalable. So whether you're using vendor solution or whether you've got an in-house solution, those kinds of techniques are very powerful. They take more effort to build and implement, but they're incredibly useful in practice, simply because of the overall effort that's being saved and the currency of the model results that are actually being used to provide this. 

Tom: Told you we'd have a lot of great conversation up here. Well, the last question I really want to ask. I'm an old pricing person. And as we advance the science, what's the role of an analyst? Are all the pricing analysts and all the revenue management going to go away and it's all going to be replaced by automation? Sam, do you want to kick us off?  

Sam: I don't think they're going away. And I think it's fair to say there's a bit of animosity or trepidation associated with artificial intelligence and machine learning because I think sometimes people do think, okay, this is going to replace me, the machine’s going to replace the analyst. I think what will happen in reality is these models, these algorithms and the harnessing and leveraging machine learning will unlock more potential within the analyst to go and do something slightly different. We want those models and the outputs of those models to empower analysts to make higher quality decisions and maybe to focus their decision making in a slightly different place where they've not been able to in the past because they haven't had the decision support to do that in the past. Richard referred to something the complexity of, there's a lot of things going on around pricing and revenue management and the decisions you make have implications across the board. We want models that are leveraging machine learning to allow you to go and think more holistically about what's happening across the board. So I definitely don't think the role of the analyst is going away. I do think there will be more and more of a convergence of pricing and revenue management as independent disciplines coming together over time. As we go through that journey of first steps in dynamic pricing, getting to continuous pricing and dynamically priced offers, but I think the analysts will be making slightly different decisions with higher quality inputs than they do today.  

Richard: Yeah, I think it was well said. I personally, I think that the job of the pricing revenue management analyst going forward is going to be a lot more interesting. There's going to be a lot more facets that people just sort of ignored because they didn't have the time to deal with it and everything before. But with better automation, we can identify where the hotspots are more easily so that the analysts can devote their time and energy to more like an exception processing basis too, while paradoxically the same time, they've got more data available to do really global kinds of things like major promotions and things like that. So I think they'll actually be able to do much more in an overall airline marketing content.  

Thomas: Yeah, I'm thinking that machine learning can automate a lot of processes that the analysts do today and hopefully they can do more value added job and also potentially there can be fewer analysts. I think that's also very possible that that they would do different job more than analyst type, I would say. Maybe it's a little controversial, but.  

Richard: Well, I don't know. Every time you think of it, the job just becomes more and more complex with more facets.  

Thomas: Yeah, that's true.  

Richard: So I think it's going to take quality people to do that.  

Michael: Well, I think that the job of the analyst will be different but I do think that, from what you say, it sounds like you basically the bar for analysts would be raised or something like that.  

Thomas: Well, I think they will have a different job.  

Michael: Yeah, I think it would be the other way around because I think technology actually enable us to make better decisions in general. It's like, I don't know, 30 years ago we don't know how to, don't have spreadsheets, but today everybody could use spreadsheets to make some decisions in their household or whatever, very easily. So technology enables us to handle more complexity easier. But then, also what that means is that I would say at the job, maybe 80% of what you did was maybe somewhat repetitive. So those may be be augmented by these AI/ML processes. But what that means is that you have more time now to focus on the exceptions, to focus on the difficult problem that you don't have, you've been putting aside, that you’ve been ignoring. That's something difficult. I'm not going to deal with it for now.  

Thomas: Yeah, right.  

Michael: But now we have time to deal with it. So it would be more definitely much more interesting.  

Thomas: Yeah, right.  

Michael: Definitely a lot more multifaceted as before, so it's less repetitive. So I think it's a good future.  

Tom: Well, we're out of time. I want to thank the panel. They did a great job. I'm just going to close and ask them to close with one sentence on what do you want the audience to take away with as far as what should we do moving forward? Thomas?  

Thomas: Well, I was thinking about it and going back to the evolution of revenue management over 40 years. And I think that it's maybe surprising how resilient revenue management has been. It has been disrupted multiple times and it has always come back stronger and being able to come up with methods that not only solve the new problems but also included the old ones. Like, for example, network optimization is more general than leg optimization. We can handle generalized fare structures as more general than simply whatever, like fenced structure and so on. Even the COVID crisis also made us creative building systems that are more adaptive and so on that we can still use even without a COVID situation and so on. So I think this is really a marvel, or beauty of I would call it, even, of revenue management, that we have seen this resiliency.  

Richard: I think in periods of great change, it's easy for people to stay frozen and wait and see, right? But I would encourage everybody not to be complacent. Really everyone in the audience here, you're the champions of this kind of technology and opportunities and everything for your respective airlines and agencies. And so I really encourage you to make efforts to be leaders and be forces of that change and everything in your companies.  

Tom: Michael?  

Michael: Well said. I think, one of the forces that that this was driving is potentially sharing data.  I think that will benefit everyone here in the room. I think, I deeply believe in that. And in terms of the science, I would say there will be, a future is coming that, where  the black box is no longer a thing. A black box is, those people who are afraid of using AI because of black box. I think black box would be a scapegoat in the future. And roles of analysts, I would say that AI and machine learning are there to help you work better and more, much more efficiently and to let you have more time to focus on the things that you want to do.  

Tom: Last but not least, Sam.  

Sam: Yeah, my sentiments follow a similar theme. I think users of products that have ML need to create a trusted partnership with those products, with those models. They need to embrace them, but don't base that just on theory, look at actual measured results of what ML is proving and providing and look at the small incremental steps of benefit that you can get from it. So similar to my peers here, just be willing to embrace, be open-minded about it and find ways. And we're here to also make sure that you can build a trusted partnership with a model and with a product that has AI and ML at its core.  

Tom: Thank you very much for all your time and all your insights. I think these guys are really the experts in the industry. Reach out to and talk to them over dinner, over the cocktail hours and pick their brains because I think we can learn a lot. Thank you very much.