ROI from MT deployment

Rodrigo Cristina 00:06 Okay, so say me how thank you for for joining the panel today. So this is this is a panel dedicated to each of our journeys in terms of return on investment for machine translation. As some of you know, I like to start our panels with, with greetings extensive to troll timezones. I feel that this might be a good morning, good afternoon, or good evening for some of us, but definitely not not for all. I think we're all connected. And our industrial localization industry is all about building bridges between people, communities, and cultures. And I think we can all agree that all of this is currently under under threat. We are one world and we should behave accordingly. So our thoughts and support are we those who are suffering directly, and their families who have been affected by by the overwhelmingly shocking events that unfolded over the last couple of weeks. I think these are times filled with uncertainty. But I'm sure that we all agree myself, panelists, and all other lock from home contributors and audience that is watching and listening to us. In wishing an immediate then peaceful resolution for for the conflict that has been upon all of us. Before we start our panel discussion and introductions, let me congratulate Cates and the lock from home team. Once again, I know how hard you guys work to put this event together. But also, I would like to praise the way you stood up for what you believe, and by ultimately making the right choice for the event. And it's a broad community and legion of followers. I think we also believe that depriving our community of knowledge that ultimately facilitates global communication means also allowing that this conflict is taking more than what it should already. Let me also share words thanking today's panelists and their presence. For them. Having accepted the challenge. They're not just great localization panelists, highly knowledgeable, but also good people, the kind that our industry definitely needs. Now after this brief intro, and without further ado, let me quickly introduce myself, the panelists and the topic. My name is Rodrigo, Rodrigo, Christina. I'm based out of the UK and I work and represents two works, a, an exciting global multilingual vendor with a very ambitious growth projects, a unique value proposition and a clear focus on localization outcome not just on on delivery. Let me now introduce our incredible panelists. I will start you know, in no particular order by introducing fee, which is a fantastic addition to the pan. They is RS Components localization manager. She's also based out of the UK. She's passionate about languages and localization technology and brings a wealth of experience in maximizing the return of applying that particular language technology to different business scenarios. She has mastered the art of transforming those technological benefits into tangible operational and commercial improvements. She enjoys working with people across cultures. Well, there's no there's no surprise there and has an interest in in language in languages and how they affect the way people think and behave. Out of curiosity note FaZe hidden superpower is that she can measure people's high within two centimeters deviation just by looking at them. Unfortunately, this incredible skill is still not available within the imperial system, but Fei Fei is definitely working on it, so expect some updates soon. So we're super excited to have you here today and know more about your journey. Now time for me house introduction. He's based out of Paris is based in Paris, the city of lights and, and the beautiful Eiffel Tower. And he has PayPals localization technology in its world ready team. In his previous roles on the client side me how I managed to VMware as localization operations for Asia Pacific and Japan, based on Singapore, and led enterprise product localization, engineering, quality assurance, and so development teams at Symantec in Poland. He also held various positions with major language service providers. Welcome me how great to have you here, looking forward to hearing your thoughts and expertise about machine translation. And its impact. Now, let me try to briefly introduce a session topic, I guess that all of us can agree with a claim that now more than ever, content is being created at an unbelievable pace. There are simply not enough linguists to process all the data that is being produced. This is a concept this is a consequence of, I believe, many, many factors, but you know, irreversible changes of our of our, the way we live, you know, lifestyle, the way we interact with each other. And also, very importantly, the way we buy. Probably fueled by the pandemic as well, and those irreversible and profound lifestyle changes. We are shopping online more than ever, and there's a huge need to create the developed content to support this particular process, this new way of, of buying eviction content needs to be made available globally. And fast, sometimes even instantly. And empty plays a pivotal role in achieving the subject. This overall explosion has led to an increasing number of different use cases, for machine translation deployment, from user generated content, like, you know, things that we see almost everyday consumer reviews, to real time multilingual customer support, like multilingual chat bots, that's, you know, another incredible useful use case for, for machine translation. For these reasons, and for being such an important component of many global localization programs, it's important to make sure that we can estimate beforehand the impact of rolling out machine translation to this effect, I would like just to show you, very briefly a model, but I would like to introduce that model as well. And this models objective is to estimate return on investment up front of the point machine translation. I feel that unlike in other industries, in our industry, particularly localization, were traditionally not very good at estimating or measuring the impact of what we do with some exceptions, of course, but if we think about it, it's it's really never been in a localization manager's job description, it's just not there. And if you read through a lot of them, this is a component that is many times not mentioned or forgotten, either by by the hiring parties. And this needs to change, I think, for the sake of who we are as an industry, and to be able to market ourselves in the same language as every other industry, the language that really matters, the language of value. And a value driven approach is focused on the outcome on the positive impact that localization has in our customers, and communities, not just on the delivery of the services. So with this in mind, we need to find ways of estimating before you know the projects start the program start and making upon completion of these projects or programs, programs, the positive impact and the return of investment, the return on investment of deliver to our customers. So it was pretty much with this in mind that a couple of years ago, I started looking at creating a model that estimates and quantify the return on investment of a global enterprise machine translation program. I took the grid tested on machine translation deployment. Because I feel that it's relatively easier to quantify than other localization activities. I decided to approach it from an investment valuation perspective, and applied a financial model called, which is very known and very used, called the net present value. So, for example, every time we think or we take a look, or we're confronted with the possibility of deploying machine translation, at program level, we tend to perform initial study of the overall impact the financial impact that this particular technology deployment will have. So let me share a couple of slides on on you know, an alternative, or a way to estimate machine translation return on investment, before we actually deploy machine translation, and without taking too much of a deep dive into the technical aspects of, of the model. So I'll start by briefly sharing my screen if I can. Now pay me how can you see what I'm projecting? No, not yet. Our about now? Nope, hold on. Michal Antczak 11:58 Since it's coming up now, yes, we can see ya can see that. Rodrigo Cristina 12:05 Yeah, this is going to be a very simple just needs to this definitely went a lot smoother than the last time. So this is definitely an improvement. And so this was, as I mentioned, a consequence of of looking at, at least upfront, looking at a way of showing both our customers that our customers would show their own line managers or budget holders, what would be the positive impact of of deploying machine translation or post editing machine translation into their localization workflows. So the the model has three components. And the first one is, which is depicted there as I zero tax initial investment, or the cost of deploying machine translation into a particular localization workflow, or program, I would say not, you know, deploying to a particular position workflow depends on the actual use case. But the idea is that we, as enterprise localization program holders, we need to spend money in a number of things in order to make our localization program ready to receive the benefits of or harvest the benefits of machine translation. So that's the initial cost. And that's something that we need to incur to deploy machine translation. The second component is the cash flow. And the cash flow is composed of two sub components, the revenues and savings that this particular machine translation program will generate. And this can be anything from savings in great to posterity due to the increase the productivity of the translators, and also the fact that as a company, the products and services of that particular company that's using machine translation are faster in the marketplace. So you basically, you earn weeks, days, months of sales, by by having your products and services of fostering the marketplace. And then the last, which is sort of the most financial component or some components, or components of the of the, of the model has to do with the fact that we've introduced rather than doing an arithmetic operation between revenues and costs, we've introduced also a time factor. So basically what this model does, and you know, don't get too alarmed with the with the, with the formula. It's a relatively simple formula when you when you start adding the numbers. The third factor, which is the discount factor is basically what allows how To introduce time, and the appreciation or depreciation of money, that happens across time, the decision criteria is mega simple. If the sum between revenues generated across time, and discounted to more than zero is higher than the initial investment that you have to make, then it's pretty simple. You decide to invest in machine translation. So this is what the impact in time looks like. You can see that there in this particular case, because this analyzes four distinct periods in time where cash flows occur. So the moment zero, which is today, imagine today that you would have to invest, where you would have to invest your your, your, your art, current money, especially when it comes to Pay Pal and RS components. Hard, hard, hard money in an empty, you'd have to make a decision based on the money that you will be sending today, and how much you would say, and how much would machine translation contribute over the next three years to increase your top line and ultimately your bottom line. And one thing that's very, very important, that's the actual time component of the model. But it's also something very important that we all have, almost subjectively this notion about how different money is today. And tomorrow, one, one pound one euro today is different from one pound. And when you tomorrow. So what this part of component of the model introduces is the actual appreciation or depreciation of money throughout time because we are comparing cash flows generated in different moments in time without getting too complicated, but it's very, very easy. So how does this work? And this is like flow. I haven't had haven't taken too much time in the, like, technical part of it. So how does this work, apply to machine translation, you can see the costs the eye zero that represent the cost that you would typically incur to deploy machine translation. So imagine that you are that you decide to go for a mainstream machine translation solution that's out there, names, but you know, there is an annual license associated to it, it's an annual license fee associated, there might be hosting costs, there might be additional software development, deployment costs specific to your own localization operation, there might be the need to clean optimize your linguistic assets that will serve as basis for customizing machine translation engines. This includes also the development of the customized machine translation engines testing, I mean, they're there, this is just a list that is definitely not pretending or definitely not aiming to get, you know, a full expensive, because there are plenty of other components that you out there. But so this is the cost side of the moment. The good part, the meaty part of the model is the cash flow part. And the cash flow includes, as I mentioned earlier, two components, the pulsating machine physician savings effect, and that's boom, almost instant because you negotiate the rates or the rate discounts based on the expected productivity of the linguists or the increase of that particular linguist, but also that very tangible effect of having products and services in the marketplace a lot sooner than they would be. But this cash flow component also has what we call running costs. So that license, the annual license fee, that is a recurring annual license fee that occurs every years that you You're, you're so happy because it's been so successful, that you want to decide that you decide to include the retraining of existing languages or specific languages. You know, the hosting is also recurring. So, you have basically savings that are generated by applying machine translation to your localizable content, but you also have running so all of this represents the cash flow. And then again, you have the discount factor, which also represents the return rates of the best available option. So this introduces time, but this particular discount factor has, let's say, different features introduced this time oops, This introduces time. But it also introduces the concept of what would you do with that money. Alternatively, if you didn't apply it in machine translation, so this is very simple. Let's say that it comes arithmetic, it will be the returns generated by posters, the machine translation program itself, minus the costs. So I will now stop sharing my my screen. This concludes the prosthetic machine relation model evaluation. There will be no room for questions. I'm sure that SmartCAT promote. But now we are back into the panel. And now that we have sort of a way of estimating the impact of empty deployment, let's hear our panelists the true stars of the show, fail and be how talking about their own individual journeys towards return on investment on their own machine transition programs. So without any particular order, again, I will start by by fee. Okay, the floor is totally yours, Fei Liu 21:13 without any particular orders. So yeah, glad to be joining this panel. And thank you, Kate Locke from home and Roger call for inviting me. And also just want to reiterate what Roger was shared in the beginning, just you know, and, you know, fully support the decision from law from home, and to show that we're standing with the people who are in this turbulent time, and our thoughts are with them, and their families, and hope everybody's staying safe and well, and during this period of time. So, back on the machine translation topic. So at RS components, we certainly are still in the journey of exploring about machine translation, the best use cases and understanding the impact that machine translation has brought to us and what we bring into us. So, just kind of zoom out a little bit in terms of the timeline, we started you know, as other you know, companies would have any way with human translations, that was before the age of machine or any machine translation engine was developed to peer to it to a point that can be a boost up for human proficiency. So, I started on the human translation, and we got good quality translate a good translate good quality translated content, you know, to be presented to our local customers, which has been great. However, on the other hand, the course has becoming has been becoming a problem, and all financial guys, as any organization would have any way during this time, and then it will be coming into this age of machine translation. And we were very excited about what machine translation can bring in terms of the, you know, the cost efficiency, and, you know, we we spend a lot of time understanding the packages or subscriptions, and how the machine translation will be able to benefit us as a company and our customers as well. And then when to kind of the other end of this journey in terms of machine translation of our product information. And then gradually we have realized that it is not all on the positive side, it is not just about you know, money saving is not just about, you know, the time to market as well. It is also about customer experience, it is also about the accuracy of the product information is about, you know, how our customers are able to understand us, particularly as a very customer strong customer focused business. So we have found some, you know, some issues in the machine translation in terms of quality. I mean, I'm not a French speaker, but if I see something like say custard s, mercury, I would understand with my broken French, something is mercury. But when we looked back into the source content, we realize is actually saying this frame is mercury free. So apparently there are limitations for the machine translation at this stage without the full understanding of the context or without taking the context into full consideration in their output without being edited by human and they're also the you know, the training material that we have fed into the machine translation especially the source that might be more apt and the algorithm as well might be you know, American base English as compared to the British base English. So in some languages, because ours components were so safety trainers, trainers was misunderstood, not the pair of shoes that you would want but someone that is in the gym that is telling you what to do as a coach. So are, you know, if we're kind of in this industry, you can understand these kind of foreseeable mistakes accuracies that machine has brought to us. So this is where we find this is a good opportunity to reevaluate the balance between the cost and a cost saving, time saving, and the actual quality and the actual customer experience that we're providing to our local customers. And so this is where we kind of start to understand, you know, what is machine translation post editing can bring to us? What is the level of quality that we're requiring for our customers and for certain products, certain categories? And, and also, you know, with all these, you know, good quality linguists data, linguistic database that we have, what is the boost to the customization of the machine engine, whether we'll be able to find a partner that can, you know, support us in this journey as well. So yeah, so really, a lot of things has happened and will be happening, and, you know, a lot of things that is going on, we are, you know, really proud to be in this journey of fully embracing machine translation and finding, where is that sweet spot? Where is the actual balance? That is, you know, having that cost in mind, time in mind, but also the, you know, the customer experience as a, you know, one of the most important commercial features that we're looking at as a business. And in terms of, you know, return of investment. And I guess, it's really important for any business to define what is the scope of work, what areas that you're looking at what the type of content that you're translating, when we're talking about content is a very, very, very broad concept. You know, it could be user generated content, which I believe me how it will be, you know, talking and sharing in a minute. And it could be, you know, e commerce coming from transactional content, it should be marks, it could be content and marketing content. It could be UX UI content. So what are the content that we're looking at? And also, what is the return that we're looking at? So are you developing an app that you want to reach out to a global audience, or a platform that you want more users in a specific country or, you know, multiple countries to be able to use, or, you know, or we just wanted to, you know, share different opinions from different geographies between our users. So there really are a lot of varieties and variables that need to be defined in the first place, before we actually take action into any machine translation strategy, or, you know, calculating any, you know, return of investment. So yeah, just kind of really broadly sharing kind of the journey with Inaros, about machine translation and some lessons that we have learned, and lessons to be learned as well. That's quickly from me. So I guess I'll hand over to me how sorry, I have been keeping you silent for a long while. Michal Antczak 27:59 It was super interesting to hear actually fade. So. So I enjoyed sitting quietly, just listening. So thank you for having me in this meeting. I will not reply, what we're going to say have said, Thank you love from home and everyone for like, you know, pulling together and supporting all the people who are now impacted by those terrible events. I'll just go straight into into the PayPal side of the empty story, right? So it is important for me to stress one thing, right. So PayPal might be a sort of a technology platform, a digital payments, company, whatever you call us, there are like multiple ways of looking at it. Right. And we might be present in 200 markets and supporting merchants, consumers alike, etc. Right. However, there is one important thing there. So we Yes, ROIs are important from the sort of NMT localization perspective, right? However, a really important part of what we do is actually about democratizing financial services, bringing them to people who might be underprivileged, otherwise, right without PayPal. So this is also part of our quote unquote, ROI. Right. And we're very serious about it. It is like put into papers mission some companies live by their mission, some don't we definitely do. So whenever we look at ROI, we look at the money side. Yes, we look at the markets we expand to Yes, but we also look at this inclusivity factor, and I'm serious about it. So Rodrigo Cristina 29:45 yeah. So I'll give you some Michal Antczak 29:49 little history right so our adventure with empty started around I think 2010 I say I think because I was not with the company at that stage. I worked somewhere else. And of course, back then you had limited options, right? So we started with rules based tensions. And as, as most of us know, they were far from perfect in terms of the outputs they gave. And they were quite a lot of customization and the, the maintenance effort was big, it cost us a lot, right to configure to test it to make it work. And, of course, we weren't super satisfied, but we knew from the very beginning, that there is like, this stuff will grow this stuff will evolve will get better and better, we should continue on that path. Right? So around 2015. So five years later, we started embracing statistical engines. And these we found gave us significantly better quality, and significantly lower customization costs, right, because the whole idea of how you train them, how you set them up to work was very different. Right. And, you know, both the quality side of it, and the customization costs, you know, bringing it down to three important for us, because we really needed at that stage to expand our coverage from that, like, very core set of markets to go beyond to to to like start living that that, you know, expansion inclusivity mission. And, and, you know, we kept on using these engines as they, they morphed over time, into neural, right. So now, essentially, we've been using them for last couple of years, we're on neural only. And the fact is, and I'll get to this in a second, that we basically use neural MT for virtually everything that we don't get good leverage from our translation memories from and I'll like, describe this now, what you need to understand, right that our team provides localization for like the vast majority of both customer facing content. We don't do marketing, for example. But most of the other stuff that you see is actually provided by us by localization within the world ready team. And it covers not only paper, but also our other brands, but I will not be mentioning them here. So that's the external like customer facing content, and you have also have internal facing stuff, right. And the ask there from our stakeholders, internal ones, is to provide just something that is quickly available, understandable, it doesn't have to be top quality, right, they just need to get a quick grasp of what's in the content that they are processing. So if you can imagine it could be for example, ad hoc, you know, technical localization, about certain articles that our customer support agents are interested in definitely not something that you need to be like to have picture perfect output for just, you know, make it understandable. And because we deal with this internal and external facing one, you have two different types of, broadly speaking two different types of engines and related processes, right. So for, for the customer facing content, which I like to call manage content, right? We translate everything that doesn't get a good translation memory leverage, right, using customized and MTS, that's important to understand, right, and then post editing. And I said everything, but it's virtually my favorite word, which doesn't tell you anything, but as of conveys the notion, virtually everything. Because there is some stuff we obviously cannot push through and empty, like non sensitive stuff, any any personal stuff, information, any any, like, highly classified legal stuff, either, right. But for everything else, we can apply it. Now, the other process is unmanaged content. It's the self serve kind of raw and empty. We use generic engines. And these we use to support our internal stakeholders, like I've mentioned, and there's a custom build portal that they can go into. We make sure that it works, we make sure that the engines are at their optimal performance, but we don't really get and manage that that whole localization process there. Yeah, and maybe I'll move on to the strategic impacts now that you understand what it is that we're dealing with, right. So first of all, and this kind of goes to what I've said about inclusivity and democratization, right? We really see Mt as our key enabler. for expanding our services, right, either in the currently serve markets, going deeper, imagine, you know, you have some markets where you have English speakers great, you can reach out to them. But in the same markets, you have people who don't necessarily speak English, right? Even in very friendly, like English friendly markets, right, we want to go, we want to make sure that they feel comfortable using our products, right. So that's, like current markets, and also, of course, new markets, right, we want to go as broadly as possible, make sure we, you know, everyone is, understands what they're dealing with when they're using our products. And like I said, this is where you where we use custom and t, right? And because we use it, right, we are much quicker to the market. And that's a key sort of strategic consideration for us, right, how quickly we are able to get into the market. And if we can trust what we're putting out in front of the customers, right? So I Are we on brand, are we aligned with the local market requirements or some legal conditions that we have to meet credit or some other specific requirements, we wish we can actually have empty, reflect in the custom engines that we use, and sort of apply to our target contents to our localized content. Now, I've mentioned quality, right, so that we have more trust in it, there is an important note on this. So what we found over the years of using NMT now, right, is that typically, our customer engines perform at least 10 to 20% better than the custom ones, sorry, than the generic ones, right. But we have also seen cases where that difference is much bigger. So one such example is Spanish for Spain, where we really see great stuff coming out of out of NMT. And in the case of some languages are bliss course, which for those of you who don't know a lot about Mt. But yes, most of us do, in this panel, right? Or in this on this forum. Blood score is a quality measure of sorts, right? That compares empty output to potential human translated output. And, and these scores are well over 70%, in a lot of cases, in some cases, almost reaching 80%. So this already gives us this confidence, right? And because we have our team set up the way it is, and because we have, like world class, really world, class MC experts on board, right, who focus on both optimizing the engines, and actually I'm having close collaboration with our human linguists who No Pay Pal content and who have worked on it for for a long time, right, we are able to achieve the sort of NMT excellence that I'm talking about. And then from a more strategic perspective, because Rodrigo Cristina 38:17 like, we are Michal Antczak 38:18 happy with the quality, because we can do that stuff quicker. And actually, at a lower cost and of course, lower costs. And then with human translators, right? Just human translators without NMT what we can do is reinvest, right, so we save the money, we don't just like, give it back to the big, old bad Corporation rights to spend on something, right, we essentially can go and cover additional areas, additional languages, or more content, like I've said, so this is again, from from our ROI perspective, this is as crucial as getting money from the markets. And then generic empty, this is actually the overlooked area. Right? But for us, it's huge, right? Just a reminder, I said this serves our internal customers, internal stakeholders, right, they can do their work with more confidence they can serve the customers better. And the interesting thing, this this area is where we see the greatest growth in our translation volumes over the last years. Right. And we started small allows you to be confident, you know, with what you're dealing with the customer feedback or, or or like, you know, technical content or whatever that gets produced all over the place, right. We now process three times as much volumes with generic engines and with our custom engines, and essentially with generic we're now called Getting well over 100 million words a year. And serving like over 100, the last time I checked well over 100 internal stakeholder teams. And the additional thing that is super important from the strategic perspective that kind of has a little bit more to do with, with this narrow definition of ROI, that is that is money related, right? These engines also allow us to do things that would have been very difficult. Sometimes we go, because we can essentially plug them into our analytical engines, right, and they help us find patterns in Multilib. Thank you all feedback from our customers about the real quality of our comp. I know that some supplies or some translators will not like what I'm going to say, right by the session, when I say we have quality, I mean that the part of quality that really bugs the customer, right, and not necessarily a comment that's missing somewhere else, though, that it's and could be important for some customers, right? So we're able to find that feedback, we're able to find patterns. And what that allows us, right, is that it helps us focus, we have these resources, we can focus, we can prioritize, we can fix things that really bother the customers, instead of just applying our resources like like, you know, peanut butter in spreading too thin, over over the sort of all the other area of operations. And it's in fact, this last area analytics is where I see the broadest field, I guess for the most creative collaboration between human linguists and MT in the coming years, right, and MT will be getting better at translating standard content, and there will be less and less room there. For the translators. It's just a fact of life, there is nothing we can do to change that. But it gives back in return, right? It gives back because whatever patterns it finds, whatever suggestions it makes, these need to be verified by humans. And empty any AI at this stage is sorry, I'll be very blunt, too stupid or too limited currently, actually understand what's behind the feedback, right to actually understand the gist of what is communicated. That's where we need humans, that's where we need their suggestions. That's where we need that's where we need their work and expert knowledge. So that would be mostly it from my side, I guess back to yourself. Rodrigo Cristina 42:36 Fascinating. Thank you. Thank you, we have Thank you, thank you to fascinating journeys. As as expected, when I when I invited you, I knew that you that you would have a really cool story to tell. Gonna throw you under the bus. But I wanted to ask you one thing, we how I took a note on something that you said about TM matching. So I'm sure that both of you are aware of this, there has been a lot of research back and forth about when to apply and T to reach thresholds within within TM matching, you know, some would say we should apply below 74% of the say say we should apply under 70%. is, you know, you mentioned the concept which is not good enough, here matching, which is you know, I'm sure that notifiable for both of you, but is it are you in liberty of or disclosing which threshold FTM matching, you apply empty? Or I totally understand I don't Michal Antczak 43:55 think it's any, it's any big secret, right? Anything below 80% we apply them to that, and again, anything I say apart from all this secret personal classified information that we should not really be making available outside of the company or anything like that. Rodrigo Cristina 44:17 How about yourself, Fei Liu 44:18 we don't have a like a fixed baseline for applying the MT and it is basically on a when the linguistic linguists are working on the content, and it would be kind of a lessons learned in progress. You know, where is the best threshold where to apply ante based on the linguist and of the day, it's all linguists that are doing the heavy lifting in our constant translation. And and I guess it can be depending on you know, language, different languages as well. Some languages and might not be, you know, so relevant with, you know, a fixed reference, but some languages might be more flexible and higher or lower than the rest mandate thresholds. So, yeah, we don't really have like a fixed baseline for that. Michal Antczak 45:05 And that's actually a good margin, because we sometimes do have to make an exception, right from that. 80%. But the general rule is 80%. Fei Liu 45:15 Yeah, yeah. I mean, the higher kind of range is very much similar, but there is no kind of fixed rule to say, okay, only 80% to everyone, every language, or 75 to every one every language. So yeah, and I guess we need to kind of respect the practice practice side as well. Because end of the day, we and all the editors, so linguists are helping us doing the heavy lifting with the content translation. Rodrigo Cristina 45:44 That is fascinating. I mean, you mentioned one thing, language dependency. How about content dependency, different content types? Would you consider applying different content to different content types, different threshold levels of when to apply NC. Fei Liu 46:05 Currently, our constant type it is quite kind of consistent on transactional content. So it is not really a big concern for us. But I don't know what about me how, what's your, what's your thoughts on this? Michal Antczak 46:17 I would say it's consistent as well. But of course, we use different engines for the different content types. But I guess that goes without saying great. Rodrigo Cristina 46:28 I have some questions for the two of you regarding you know, your own journey and how things have evolved. Having present that transition qualities is sort of a dynamic concept. It's a moving target that requires a lot of adjusting. How do you measure me? Which, you know, which framework would you use? Is it a standard once a non standard one? Did you adjust the way you measure linguistic quality to post editing machine translated content or raw or raw machine physician? applied? Fei Liu 47:12 I think for us, because we're kind of in this ecommerce, ecommerce world, and, you know, purely linguistic matrix would make a lot sense in terms of our kind of see as a threat holders in the business. So we interpret some linguistic metrics into more commercial contexts. So for example, you know, we talk about turnaround time in the linguistic world, but how does that translate into time to market for example, or part being part of the lead time for your product to be on the market and customer facing? And, you know, we talk about quality, you know, talk about error rates? And how do we interpret that into a call or commercial backgrounds into a commercial context? I think this would be, you know, making the communication to the internal business help stakeholders easier. And, you know, within with, you know, partner earning with their partners, you know, we can still use those linguistic metrics, because that would mean, we're using the same language in the same world. So I guess it's just kind of slight change in the mind sight mindset, how do we interpret some linguist seven metrics, that doesn't resonate so much into the business world into our business language? Michal Antczak 48:30 So, yeah, so for me, I'm sorry, I will not be able to go into a lot of details on this, however, right, like I've already said, or maybe I'll start with this, we used to apply, of course, in the past, as probably most of the localization themes out there, right, surely linguistic criteria to what we delivered, and the terms of, you know, consistency, grammar, and so on and so forth. Right. However, for some time, now, we have tried applying the these analytics that I've mentioned, right, that kind of trying to find what it is that our customers actually are telling us about our content, and what it is that bothers them and trying to basically focus our impact on that. Now, the way we mitigate that purely linguistic quality issues is by having a very specific vendor model, right, in which we give our vendors a relatively relatively a lot of time and, and relatively good rates compared to the other companies on the market, right? So that they have both the time and the motivation to actually figure sorry, focus on delivering the right stuff from the get go, rather than having to scramble and deliver something Very quickly, crazy schedules. And as you know, very, very low cost, which is not very quality, quality centric, I would say. Rodrigo Cristina 50:11 Yeah. Thanks. Thanks for sharing that me how I had a question. And I think both of you have have replied to brilliantly, which would be and you've shared a lot of slides about different dimensions, which the, your your own machine translation program has affected positively your overall localization program, I think both of you, in different in different parts of your explanation have shown that. So I'd like to finalize with a question or panel today by saying that it's been an absolute pleasure having you having you here exchanging this, these ideas about machine translation. But I would like to ask you both a question around company, corporate culture. So how did your company's corporate culture influence deploying machine translation across the organization with, you know, so many different use cases? Was the organization suspicious? Was it sort of an agnostic? Or was it Pro or completely surrender than pro machine translation? I don't know. Perhaps now. We can start by? Michal Antczak 51:35 Yeah, I mean, I mean, I've probably talked about the biggest aspect of the corporate culture already, right, which is, it's ingrained in our mission to basically go as wide as possible, with the languages with the, with the customers that we can serve. Right. So that has definitely impacted us because probably, some of the localization decisions would have been made differently, had they been purely money driven, right. Now, as for, for example, whether there were stakeholders who are less or more suspicious, of course, you always have that, right. Some people just embrace it wholly, some of them have good knowledge about localization, that which is, you know, you can get, you can be really surprised, right. And some of them stress over and over again, that they don't want empty, they don't want to risk quality and similar comments. And I guess in these cases, it's, you know, ultimately, the vast majority of our stakeholders, once they understood how the processes applied, once they understood that there is post editing that is happening, once they understood that a human and a very well qualified human that looks at every string, right? That NMT processes, vast majority of the stakeholders have agreed and basically, I think the trust is there now. Fei Liu 53:03 You asked me how you just capture a really, really important point about understanding, you know, once they understand, but only after they understand, this is kind of the educational piece that we are, you know, kind of responsible for, you know, educating our stakeholders, the right knowledge and the right evaluation, the right assessment of the machine translation and what it can actually bring to be fit for purpose. So, I guess, regardless of the culture of the organization, you know, I've seen personally, you know, attitudes on both ends. Some are super, super suspicious, we don't want machine translation at all. On the other hand, you know, yeah, let's use machine translation fully. So, I've seen both attitudes across different organizations, but end of the day, it is the lack of understanding about machine translation, that is giving too much confidence and too little confidence about machine translation. So end of the day, it is really the education that we need to have with our stakeholders and to show them the real value and you know, demonstrating approach because we are the experts in these areas. So that you know the got the universal buy in from the business that people are comfortable with what we're offering as making sure that is fit for purpose as well. Rodrigo Cristina 54:28 It's this it's empty, and humans not empty versus humans. I think that's I think that the main education points that we even our in our industry, and we're experts as you mentioned trade we still have trouble communicating that across all of the all sides you know, so you hear linguist saying it's empty is not good. It's gonna take our jobs. No, it's not. Anyway guys, it's been an absolute My pleasure. Thank you very, very much for accepting the challenge and the invitation. I think this concludes our session. It's been great having you here and wishing you, the best in the world and the best in the world also happen to us in the next in the next weeks or days or hours. Okay, thank you and have a fantastic rest of Fei Liu 55:29 the day. Thank you, everyone. Thank you for having me. Lovely discussion.