Data-driven decision making for Globalization Strategy and Operations

Max Morkovkin 00:00 Oh yeah. And I can finally introduce our next speaker, Robert O'Keefe, Senior Program Manager globalization at Citrix with his topic about data driven decision making for globalization, strategy and operations, guys, so you know, data is everywhere we need the data, we analyze it, everyday small data, big data. So this is something that becomes more and more accessible. And we have different sources of data. And Robert will tell us how they use the data to maximize the productivity of internal resources and define localization strategy at Citrix. Okay, Robert, I see you already joined us. Hello. Robert O'Keefe 00:49 Good morning. Good afternoon, wherever you are. Yeah. Max Morkovkin 00:54 Good. Good afternoon to those who joined us from Asia Good morning to those who join us from Europe. And I think that attendees from Americans continent will join us soon as well, by the way we have how many 380 80 Plus live attendees right now? Not bad. That's huge. Yeah. Very nice workplace. Robert. Robert O'Keefe 01:18 Thank you very much. Yeah, this is my home office. Max Morkovkin 01:23 Great one. Okay, so are you are you ready to share the screen? Robert O'Keefe 01:29 One second. Click the button. Max Morkovkin 01:33 Okay, so just to remind our attendees that we have this cue and data to leave your questions, and I keep monitoring the Zoom chat. So if you have good feedback, I will, please to share it with Robert. Okay, so I'm stopping the video and muting myself and believe it to you just very Robert O'Keefe 01:54 much. Okay, you see my screen? Max Morkovkin 02:00 Not yet. Please try again. We have from Egypt, Robert O'Keefe 02:16 that button. Sorry, my phone. Okay. Let's do it. So I'm more used to go to meeting on Citrix, you know, so. Okay, here we go. Can you see my screen now? Yes, we can get stuff. Excellent. Okay, let's crack on. So good morning. My name is Raja Keith. I'm the Program Manager here in Citrix. And, you know, Citrix as a company has been going a fair while. And I was actually here in the earlier earlier days around the time of remote access, where we had really just a single product solution metal frame at a time, which provides remote access to, you know, to, for employees to do work on any device really. And then, of course, we moved forward in time and also increased in pace, acquiring a number of technologies in sort of into doesn't tend to 2015 era. And in fact, that pace of change has increased with technology as well over time. And so now the portfolio is much larger in terms of size and what we need to do as a group. So workspace intelligence, where we are today is really made a three areas workspace providing the end user, the end user access, management management functions, you've got networking, which allows us to deliver it securely and an analytics, which kind of takes it to another level with AI and machine learning, provide that extra level Mac level of management. And Citrix workspace as I said, it's all about accessing apps, desktops and fast using any one of a number of devices. And those on the management infrastructure side, there's absent and fast can be provided from really any sort of storage medium, whether it's on premise hybrid cloud, SAS app on premise app, so forth. So it's really a comprehensive solution now, and it's delivered securely over the network. But the company's mission really is to reach a billion users around the globe using our Citrix workspace product, which I'd like to call out here because this is really our main area in terms of globalization services, and user facing software. And as globalization group, we are actually sitting within the engineering arm of the company. And we have resources dotted around the globe, where we have localization and localization Department made of trend Vision Services and engineering. And they work primarily on, you know, the localization product automation in that workflow, a lot of kind of stuff that, you know, Taylor was talking about with API connectivity. And then we have a globalization QA group. And they really work on the functional and localization testing. With, you know, automation and tools we should build in house as well. And with the increase in the number of products and the frequency of releasing products, you know, that team has really had to sort of up their game in the last five to 10 years. And last, but not least, globalization, development, who do the architecture and development, education as well as consultation, and their role really is around making sure that our product is international ready. And then there's myself in Portfolio Management, and I sort of sit in the middle in with these groups and have an interest in in all. So I'd like to go back a little bit in time, before going on the data journey to talk about, you know, where it came from, and how it all began, and so forth. You know, in the early days, we're moving files around manually and sort of quite heavy processes. And, as I said, we bought a lot, we acquired a lot of companies throughout, so the early teens of 2000. And this forced us to really looking at our internal processes. And so we went down the route of Lean Six Sigma, where we looked at our processes, removed all the waste, and over processing, if you like, try to reduce defects or prevent defects as much as possible by you know, left shifting, and this kind of work. And, you know, the interesting thing about this whole, Lean Six Sigma study, and implementation was not only did it save us, you know, well over two percents in terms of operating expense, but it changed our culture and our mindset and allowed conversations to take place, like, you know, this, this may not be the best way to do it the more efficient way because it goes back to this Lean thinking and the way of improving processes and efficiency. And Lean Six Sigma really got us to a healthy place. But what we found as well, a bit later down the road engineering teams, overall, in Citrix, I think we have, you know, a dependency on third party vendors for quality assurance. And there was a company drive to try and you know, reduce the dependency on third party vendors, and to try and get a lot of this in house. And so we came with the sort of the real, you know, the large mood shift forward in automation and automation of testing, and QA and so forth. So we were challenged to bring in, to lessen our dependency on vendors, increase our automation, and just kind of like a big shift in the company. And on top of that, really, we need to decide okay, well, where do we focus? First, we've got to build automation, where do we do it first? And what product line? What's our language focus and so forth? And so we started to ask the big questions like you know, what's the relative value of each product line? Which ones do we do first? What are major languages and regions you know, which language language or languages should we focus on first, you know, sort of tears particular and then where are we spending our time and localization dollars in relation to what's important and so, we actually started done the return on investment routes, you know, measuring our weather making money as certainly the major regions so access to finance at the time, it was back in 2014 and 15. And we started gathering financial data from a broken down by region, Germany, France, Spain, and then also by product line, so you need to have a workspace flirt and analytics split and network and split and so forth. And that allowed us to build in a quite a comprehensive picture of which regions were doing well in what product line and so forth. On top of that, we started measuring our internal costs so the internal resources for the button costs translation spend tooling licenses, you know, the International QA which we would spread across the different languages, so there's like an even spread of internationalization if you like, and then where our time is spent. So you know, allocate innovation to a separate category to product and so forth. But in the past, as I mentioned, it was like a separate blocks you know, midframe single product line used to be a physical blocks, you know, off the shelf product, single binary, one language, one SKU or stock keeping unit, which we ship off to Japan, and we could track the sales of, you know, Japanese product or French products or Spanish products in that way. So it was very easy to get localized value in the party. And then we went down the route of multilingual binary one SKU, all the languages downloadable from the web. Now, I guess a lot harder to track where we're making money, because we don't really know from a SKU perspective where that where that revenue is coming from. And so with the study on the, on the revenue, we and our costs, you know, we had a kind of a good picture on where we're making money around around the world. So region one make that much money in region two, region three, and so forth. We'd have different tiers, so most important region, to region and tier three regions. But it really left us with one big problem, and that we didn't really know, within a given region, which percentage we're using localized product, and which was still using English, because that was still you know, in challenging for us. So then we started down the sort of telemetry routine and gathering data from apps running on a device. So for example, the workspace app client in CFS, first one, now, when it's running on a Windows device, which is French, it uploads the locale, to the tears to the back end system. And so shows it running on French system. And of course, it's showing the operating system language. So it could be installed on any operating system, we get the results of the locales on which the client is running. And so we started small, we started with the Windows client. And we started with this main server component, which I think was studio at the time. And what we realized is, takes a fair bit of effort to build it out across all platforms. But one needs to because what we also found by going down that route, is that you know, that there's variations between regions, you know, America tends to use iOS as very popular. Within like Italy, France, Android is much more popular than iOS. So and in the UK, it's about a 5050 split. So what we found is, you need to take both platforms to really get an idea of the mobile usage. And you'd have to apply that to to all different platforms, and so forth. And then we track the administration consoles, usage, and importantly, in more recent times from started tracking, the Citrix product documentation, which I'll talk about a little bit later. So now we had revenue impact, the usage, and we can say, within a given region, where there's this much usage, and therefore the value of a given language is that much. So we're able to really quantify how much impact we're having with localized product. And this is a really powerful message, because it allows you to say, Okay, well, you know, we're doing really well in this region, let's keep going. But it also allows you to steer conversations and influence in a product management, circle, that region that you want that new language, but let's think about that a bit more, because we need a bit more business case to do that. So it's good information to have. But of course, like anything, you know, two dots in the graph that I was told by someone very wise to listen to graph does not make a trend. So this is a process that one needs to repeat year on year, quarter on quarter, measure the revenue, measure the internal usage of internal spend, sorry, and then really get that trend year over year. But this really helps because especially on a product line, you may want to grow a product line, you look at the regions which are growing and you'll look to serve those regions. And so this is a way to really drive adoption of workspace I was talking about earlier. When I'm speaking with product managers, I try and bring in other other information as well, like our competitor offerings, you know, direct competitor or just in just industry, data in general is all pretty useful to compare. And other elements of information like the English proficiency index by education first, where they measure the special education in the country, particularly in English language. You know, other earnings per capita has kind of inter internet connectivity, and also tests of course, they categorize countries accordingly. This data is kind of like you can use it to steer a conversation is there anything on the right is in need of localization anything unless you get less bang for the buck, but it's still needed. Germans still prefer their products in German, so it got to steer that way as well. And, you know, this is kind of the ROI studies of driving that sort of things, but our development team at the same time started looking at technology. So we had a lot of customer cases which had escalated over time, so we have 38,000 in the database impossible to mine, you know, But by, you know, a human resource, as they started looking at AI or machine learning to actually recognize which of those cases are the 30,000 cases, were related to globalization or some sort of, you know, problem with the international product. Then, once they've refined the algorithm tried out a few machine learning engines figured out which ones worked and which ones didn't, they started to build out, you know, the accuracy. And they've got it from 60% in 2017, to I think, around about 80 plus percent in 2020. So it's pretty accurate. And the really meaningful data is, they've been able to categorize what types of issues are in what product, so the product settings, columns at the bottom, and the types of issues in pie chart. And we recognize it now. But it's, you know, whether a virtualized desktop accessing, you know, some server mice, my desktop is in Amsterdam, and I'm accessing it using, you know, this keyboard, this keyboard specialized. And of course, when you start switching languages, keyboards, it all gets really interesting when it's virtualized. So the generic client input method serves the Far East improve the experience, and the keyboard layout, and language by helps customers to sort of change the layout, it all syncs dynamically. So there are features which have come out of this study, in fact, and there's actually a blog on the web, which talks about analyzing the machine learning data and how they did it should not take your interest. But it's a fascinating story. And it has also made our test cases more robust as well as we recognize which ones slipped through the cracks and can improve that for future. But it would be remiss of me not to talk about how documentation data really transformed our approach to Citrix documentation product documentation. The translation services team and the local engineering team localization incident team, they teamed up. And we recognize that 20% of our customers were using non English content. But only 40% of the content was translated only for just about user stops were translated. And we found that the customer journey was getting broken up reading a piece of translated documents, and they'd link to another document which didn't exist, and it would break. And so not only was it broken, but they weren't being served with the complete documentation. So there was a big need to improve this area. And machine translation, or really the sort of advent of New Zealand, MT and an improvement in quality really allowed us to explore that area. And so in 2018, we started this journey. And in 2020, were delivered 100% of our documentation is now localized. 40% remains on machine machine translation plus both editing. And 60% is machine translation, with some controls, like terminology is aligned and short got branding, right, and a few automated corrections, but barely any editing. And that's for German, Japanese, French, Spanish and simplified Chinese. So the whole the all of our supported languages, in fact, and really is how it looks when you go to the web, Linux. And when you go to Doc's dot citrix.com. The language selector on the right there shows the machine translator language denoted by this little icon. And you know, you click on French, it just appears in French. But in addition to the regular page, you get a banner at the top allowing you to send comments back or feedback to help to help improve the translation. And this is stored on our back end. So we can take remedial action, we can review a page improve it and make sure that that's persisted for the next time. It's not it's not freshly machine translated at all times, machine translated and then persisted. And then we also provide the user with two important pieces of information. So we we advertise that the article is machine translated, providing the cause in the nondisclosure. The legal text at the bottom saying please don't sue us pertaining to encryption, any incorrect text, and we provide them with a button to turn it back to English. So they can go back to the English version. And in addition to that, if they hover over the text, and English, the English text will pop up. So they get this hover over functionalism to allow them to check back to the English if needed. But the data story is really interesting. Now in a translation data is so big, we have a full time resource working on this. And what's you know, we decided to start measuring data has initiated it, but data also measures the success of this particular initiative. So we're seeing that whether the document is this is example in Spanish here. And we're seeing that if the document is human translated, and mtpe, or whether it's empty only. We're seeing good readership throughout. Also, remember, we're capturing the feedback. And we've not seen many negative feedbacks at all, we've seen a lot of comments of some different, but not many, which are bad shows we're on track. Furthermore, we're measuring the actual adoption. Remember, I said to us in the graph, it doesn't really quick trends, we're trying to gather trend information on Mt adoption. So what we tracked it in, I think it was January, February, it was launched. But then we didn't really announce it until around the sort of, you know, later in the year timeframe, when we saw a peak in the actual readership. So sort of public announcement via our blog system went live. And we saw an upturn in the usage quite dramatically. And it's really proving to be quite a successful story, even machine translated content, that are the challenges which still remains. So you know, we are working on improving the workflow, making it faster, removing the errors that I spoke about in those errors that can creep in trying to make the whole process more robust, looking at new technologies, which allow us to, to statistically measure the quality of translation of content to be translated suitability of the image. And so this presentation is about data driven decision making data really drives globalization services, and Citrix and allows us to make decisions, which we believe will add value to the customer. And as I said, in the data driven decisions, where we've kind of used data to support the argument has really been along the lines of Dutch, Italian, Brazilian Portuguese, now available in workspace vs. app with intelligence. The keyboard I, me and language buyers now out there. And there are more enhancements planned for 2020. And we have 100% of our documentation. Now, there's languages but being machine translation, once you make improvements in a natural workflow, there's no stopping. So I believe we can go this is just the beginning of our journey, I believe. And so that's kind of where we are today with machines with data within Citrix and what we've done. So I'd be interested to hear of your data stories. And you know, what you've learned from data in your company. Thank you very much. Max Morkovkin 22:38 Robert, thank you very much for the presentation, you really brought a very important topic. And now we have a lot to think about. Thank you very much. Robert, can you please stop sharing the screen so that attendees can see us and we can start this q&a session? I will help you with the questions. We already have several of them. Let's start. So the first question is from Hannon. Sheriff. Where can I get the data on English proficiency level? By country? Robert O'Keefe 23:11 If you start if you search EFI, English proficiency? Education First, as the groups are doing. It's out there on the way. Yeah, Max Morkovkin 23:23 no. Okay, good. Thank you. So let's take a look on another question. From M psaltis. Did you run market tests between the localized and non localized offerings to help determine the impact was because of localization and not another factor? Robert O'Keefe 23:43 No, we didn't do that from the engineering group. But we worked with the marketing team sales teams to identify whether there was a need for localization in the region. And so we've gained feedback from those teams before we went ahead with a decision. It really is, you know, you can't just you can use data to support a decision like this to add new product or new languages. But you need everyone to work together, their marketing teams, sales team and so forth. They'll need to support the decision. And it needs to be there as well in terms of pipeline revenue and this kind of stuff. So yeah, there's more to it than just data, there is a conversation or three to be happy. Max Morkovkin 24:25 Okay, so the third question is coming from Christina Trevino Castillo, are you including localized documentation in your ROI model? If so, are there any insights you can share on how those performed with regards to ROI? Thanks. Robert O'Keefe 24:46 Really interesting question. At the moment, we do incorporate documentation usage into our ROI model. So I may not have mentioned the politican average of the user interface docs The admin consoles and formerly ROI in that way. So yes, it does. It does, it does take it into account. But indirectly, more importantly, I think is that we have, as a company heard from customers that they need localized content. But it's always been very hard to justify it, because the cut of costs involved to translate such large volumes. So we've been stuck until we came up with this plan to machine translate in a big way. Max Morkovkin 25:36 Got it? Okay, so let's take a look on the question from Malika met arrow. Does the machine translated content website meet the customer information requirements of countries for products served in this specific country to provide information in the native language? Were long question, maybe you want to look at it one more time? Yeah. Yeah. In the the first on top? Robert O'Keefe 26:04 Yeah. So I think. Yeah, it's hard to say whether it meets the absolute requirements of every customer. I think the important point is to say that, you know, they had nothing before in terms of localized content. And now we're providing them with machine translated content, which we are trying, where we are trying to increase the quality bar all the time. As I said, it's just really the beginning of the journey for Citrix with machine transfer, the content is out there, it's sort of a good enough quality at the moment. For for us, we always want to ensure the quality remains, first and foremost, the customer. So we're looking for ways to improve that over time, as we you know, understand as this new technologies evolve, really, because that's where it is, it's all moving so fast that can't stand still, you know, you've got to sort of look at the technology, what's latest on the market? Try it out, improve? You know, it's an iterative process now. Max Morkovkin 27:04 We have three minutes left before our break. So let's answer the rest of the questions. We have actually several of them, so we'll try to do that. The question from Nikolas on Tana, really, Nicholas is trying to do some business development. Do you require Nordic localization at Citrix? If he has to collaborate with freelancers or agencies or with both? Robert O'Keefe 27:27 Who knows the answer to that? Max Morkovkin 27:32 Okay, keep raising the question from annually teen wars? Do you localize the versions of English, French, Spanish, Portuguese, etc. Adopting to each workout? English UK vs. English? Yes. Robert O'Keefe 27:49 No, we do we do. I think it's fairly typical. We do think English US for product product documentation. So for product documentation to English, us, Spanish international, French, France and so forth. Now, with we focus on the largest largest markets really. But of course, with our marketing website, for example, there are flavors, if you like or variants of language. And they are on the webcam. So in the marketing side, we've customized it. Yes. Further. Max Morkovkin 28:27 Okay, and the question from Rebby. Southern, what tips do you have on a good tool to measure statistically? Robert O'Keefe 28:39 I mean, Tableau is seems to be the reference, but it's also really expensive. So, in a we, we started using Tableau, then we move to Power BI, because Power BI essentially is with comes from Microsoft. And it's just as effective really, both tools require you to do some sort of learning, shall we say, you know, so, but it's not impossible. It's, it's achievable. Yeah. And, you know, the back end systems can be anything from a spreadsheet spreadsheets to Google Analytics, you know, the source content is is massive, you know, so you can pick up data from anywhere. Max Morkovkin 29:23 Yeah, many sources. Yeah. The question from Anneli team worth, what's your stand on a global English, for example, as a company, Standard English version, like blogpost, global marketing campaigns, etc. Robert O'Keefe 29:44 I don't think I have a strong opinion. I think there's always the need for like, local content ready to really appeal to the customer, especially in the marketing side, first contact and so forth. But you know, one language for all I'm not quite sure, just yet. Max Morkovkin 30:03 We have several more questions, but we are out of time. Sorry, the attendees. We will answer these questions later with Robert together. Maybe probably one last question occasional one from Michelle, how do you combine your second job as a Red Bull engineer together with your Citrix? Yeah. Robert O'Keefe 30:22 I said tough life. Yeah, I, we sponsor, Red Bull Racing team. And I've had the good fortune of going to the actual factory where they build the cars. And I must say it's absolutely fascinating and highly recommended. Max Morkovkin 30:38 I see. Okay, Robert, thank you very much for joining us today for this video presentation. So, have a good rest of the day and talk to you later.