Data-driven decision making for Globalization Strategy and Operations

that button. Sorry, my phone. Okay. Let's do it. So I'm more used to go to meeting on Citrix, you know, so. Okay, here we go. Can you see my screen now? Yes, we can get stuff. Excellent. Okay, let's crack on. So good morning. My name is Raja Keith. I'm the Program Manager here in Citrix. And, you know, Citrix as a company has been going a fair while. And I was actually here in the earlier earlier days around the time of remote access, where we had really just a single product solution metal frame at a time, which provides remote access to, you know, to, for employees to do work on any device really. And then, of course, we moved forward in time and also increased in pace, acquiring a number of technologies in sort of into doesn't tend to 2015 era. And in fact, that pace of change has increased with technology as well over time. And so now the portfolio is much larger in terms of size and what we need to do as a group. So workspace intelligence, where we are today is really made a three areas workspace providing the end user, the end user access, management management functions, you've got networking, which allows us to deliver it securely and an analytics, which kind of takes it to another level with AI and machine learning, provide that extra level Mac level of management. And Citrix workspace as I said, it's all about accessing apps, desktops and fast using any one of a number of devices. And those on the management infrastructure side, there's absent and fast can be provided from really any sort of storage medium, whether it's on premise hybrid cloud, SAS app on premise app, so forth. So it's really a comprehensive solution now, and it's delivered securely over the network. But the company's mission really is to reach a billion users around the globe using our Citrix workspace product, which I'd like to call out here because this is really our main area in terms of globalization services, and user facing software. And as globalization group, we are actually sitting within the engineering arm of the company. And we have resources dotted around the globe, where we have localization and localization Department made of trend Vision Services and engineering. And they work primarily on, you know, the localization product automation in that workflow, a lot of kind of stuff that, you know, Taylor was talking about with API connectivity. And then we have a globalization QA group. And they really work on the functional and localization testing. With, you know, automation and tools we should build in house as well. And with the increase in the number of products and the frequency of releasing products, you know, that team has really had to sort of up their game in the last five to 10 years. And last, but not least, globalization, development, who do the architecture and development, education as well as consultation, and their role really is around making sure that our product is international ready. And then there's myself in Portfolio Management, and I sort of sit in the middle in with these groups and have an interest in in all. So I'd like to go back a little bit in time, before going on the data journey to talk about, you know, where it came from, and how it all began, and so forth. You know, in the early days, we're moving files around manually and sort of quite heavy processes. And, as I said, we bought a lot, we acquired a lot of companies throughout, so the early teens of 2000. And this forced us to really looking at our internal processes. And so we went down the route of Lean Six Sigma, where we looked at our processes, removed all the waste, and over processing, if you like, try to reduce defects or prevent defects as much as possible by you know, left shifting, and this kind of work. And, you know, the interesting thing about this whole, Lean Six Sigma study, and implementation was not only did it save us, you know, well over two percents in terms of operating expense, but it changed our culture and our mindset and allowed conversations to take place, like, you know, this, this may not be the best way to do it the more efficient way because it goes back to this Lean thinking and the way of improving processes and efficiency. And Lean Six Sigma really got us to a healthy place. But what we found as well, a bit later down the road engineering teams, overall, in Citrix, I think we have, you know, a dependency on third party vendors for quality assurance. And there was a company drive to try and you know, reduce the dependency on third party vendors, and to try and get a lot of this in house. And so we came with the sort of the real, you know, the large mood shift forward in automation and automation of testing, and QA and so forth. So we were challenged to bring in, to lessen our dependency on vendors, increase our automation, and just kind of like a big shift in the company. And on top of that, really, we need to decide okay, well, where do we focus? First, we've got to build automation, where do we do it first? And what product line? What's our language focus and so forth? And so we started to ask the big questions like you know, what's the relative value of each product line? Which ones do we do first? What are major languages and regions you know, which language language or languages should we focus on first, you know, sort of tears particular and then where are we spending our time and localization dollars in relation to what's important and so, we actually started done the return on investment routes, you know, measuring our weather making money as certainly the major regions so access to finance at the time, it was back in 2014 and 15. And we started gathering financial data from a broken down by region, Germany, France, Spain, and then also by product line, so you need to have a workspace flirt and analytics split and network and split and so forth. And that allowed us to build in a quite a comprehensive picture of which regions were doing well in what product line and so forth. On top of that, we started measuring our internal costs so the internal resources for the button costs translation spend tooling licenses, you know, the International QA which we would spread across the different languages, so there's like an even spread of internationalization if you like, and then where our time is spent. So you know, allocate innovation to a separate category to product and so forth. But in the past, as I mentioned, it was like a separate blocks you know, midframe single product line used to be a physical blocks, you know, off the shelf product, single binary, one language, one SKU or stock keeping unit, which we ship off to Japan, and we could track the sales of, you know, Japanese product or French products or Spanish products in that way. So it was very easy to get localized value in the party. And then we went down the route of multilingual binary one SKU, all the languages downloadable from the web. Now, I guess a lot harder to track where we're making money, because we don't really know from a SKU perspective where that where that revenue is coming from. And so with the study on the, on the revenue, we and our costs, you know, we had a kind of a good picture on where we're making money around around the world. So region one make that much money in region two, region three, and so forth. We'd have different tiers, so most important region, to region and tier three regions. But it really left us with one big problem, and that we didn't really know, within a given region, which percentage we're using localized product, and which was still using English, because that was still you know, in challenging for us. So then we started down the sort of telemetry routine and gathering data from apps running on a device. So for example, the workspace app client in CFS, first one, now, when it's running on a Windows device, which is French, it uploads the locale, to the tears to the back end system. And so shows it running on French system. And of course, it's showing the operating system language. So it could be installed on any operating system, we get the results of the locales on which the client is running. And so we started small, we started with the Windows client. And we started with this main server component, which I think was studio at the time. And what we realized is, takes a fair bit of effort to build it out across all platforms. But one needs to because what we also found by going down that route, is that you know, that there's variations between regions, you know, America tends to use iOS as very popular. Within like Italy, France, Android is much more popular than iOS. So and in the UK, it's about a 5050 split. So what we found is, you need to take both platforms to really get an idea of the mobile usage. And you'd have to apply that to to all different platforms, and so forth. And then we track the administration consoles, usage, and importantly, in more recent times from started tracking, the Citrix product documentation, which I'll talk about a little bit later. So now we had revenue impact, the usage, and we can say, within a given region, where there's this much usage, and therefore the value of a given language is that much. So we're able to really quantify how much impact we're having with localized product. And this is a really powerful message, because it allows you to say, Okay, well, you know, we're doing really well in this region, let's keep going. But it also allows you to steer conversations and influence in a product management, circle, that region that you want that new language, but let's think about that a bit more, because we need a bit more business case to do that. So it's good information to have. But of course, like anything, you know, two dots in the graph that I was told by someone very wise to listen to graph does not make a trend. So this is a process that one needs to repeat year on year, quarter on quarter, measure the revenue, measure the internal usage of internal spend, sorry, and then really get that trend year over year. But this really helps because especially on a product line, you may want to grow a product line, you look at the regions which are growing and you'll look to serve those regions. And so this is a way to really drive adoption of workspace I was talking about earlier. When I'm speaking with product managers, I try and bring in other other information as well, like our competitor offerings, you know, direct competitor or just in just industry, data in general is all pretty useful to compare. And other elements of information like the English proficiency index by education first, where they measure the special education in the country, particularly in English language. You know, other earnings per capita has kind of inter internet connectivity, and also tests of course, they categorize countries accordingly. This data is kind of like you can use it to steer a conversation is there anything on the right is in need of localization anything unless you get less bang for the buck, but it's still needed. Germans still prefer their products in German, so it got to steer that way as well. And, you know, this is kind of the ROI studies of driving that sort of things, but our development team at the same time started looking at technology. So we had a lot of customer cases which had escalated over time, so we have 38,000 in the database impossible to mine, you know, But by, you know, a human resource, as they started looking at AI or machine learning to actually recognize which of those cases are the 30,000 cases, were related to globalization or some sort of, you know, problem with the international product. Then, once they've refined the algorithm tried out a few machine learning engines figured out which ones worked and which ones didn't, they started to build out, you know, the accuracy. And they've got it from 60% in 2017, to I think, around about 80 plus percent in 2020. So it's pretty accurate. And the really meaningful data is, they've been able to categorize what types of issues are in what product, so the product settings, columns at the bottom, and the types of issues in pie chart. And we recognize it now. But it's, you know, whether a virtualized desktop accessing, you know, some server mice, my desktop is in Amsterdam, and I'm accessing it using, you know, this keyboard, this keyboard specialized. And of course, when you start switching languages, keyboards, it all gets really interesting when it's virtualized. So the generic client input method serves the Far East improve the experience, and the keyboard layout, and language by helps customers to sort of change the layout, it all syncs dynamically. So there are features which have come out of this study, in fact, and there's actually a blog on the web, which talks about analyzing the machine learning data and how they did it should not take your interest. But it's a fascinating story. And it has also made our test cases more robust as well as we recognize which ones slipped through the cracks and can improve that for future. But it would be remiss of me not to talk about how documentation data really transformed our approach to Citrix documentation product documentation. The translation services team and the local engineering team localization incident team, they teamed up. And we recognize that 20% of our customers were using non English content. But only 40% of the content was translated only for just about user stops were translated. And we found that the customer journey was getting broken up reading a piece of translated documents, and they'd link to another document which didn't exist, and it would break. And so not only was it broken, but they weren't being served with the complete documentation. So there was a big need to improve this area. And machine translation, or really the sort of advent of New Zealand, MT and an improvement in quality really allowed us to explore that area. And so in 2018, we started this journey. And in 2020, were delivered 100% of our documentation is now localized. 40% remains on machine machine translation plus both editing. And 60% is machine translation, with some controls, like terminology is aligned and short got branding, right, and a few automated corrections, but barely any editing. And that's for German, Japanese, French, Spanish and simplified Chinese. So the whole the all of our supported languages, in fact, and really is how it looks when you go to the web, Linux. And when you go to Doc's dot citrix.com. The language selector on the right there shows the machine translator language denoted by this little icon. And you know, you click on French, it just appears in French. But in addition to the regular page, you get a banner at the top allowing you to send comments back or feedback to help to help improve the translation. And this is stored on our back end. So we can take remedial action, we can review a page improve it and make sure that that's persisted for the next time. It's not it's not freshly machine translated at all times, machine translated and then persisted. And then we also provide the user with two important pieces of information. So we we advertise that the article is machine translated, providing the cause in the nondisclosure. The legal text at the bottom saying please don't sue us pertaining to encryption, any incorrect text, and we provide them with a button to turn it back to English. So they can go back to the English version. And in addition to that, if they hover over the text, and English, the English text will pop up. So they get this hover over functionalism to allow them to check back to the English if needed. But the data story is really interesting. Now in a translation data is so big, we have a full time resource working on this. And what's you know, we decided to start measuring data has initiated it, but data also measures the success of this particular initiative. So we're seeing that whether the document is this is example in Spanish here. And we're seeing that if the document is human translated, and mtpe, or whether it's empty only. We're seeing good readership throughout. Also, remember, we're capturing the feedback. And we've not seen many negative feedbacks at all, we've seen a lot of comments of some different, but not many, which are bad shows we're on track. Furthermore, we're measuring the actual adoption. Remember, I said to us in the graph, it doesn't really quick trends, we're trying to gather trend information on Mt adoption. So what we tracked it in, I think it was January, February, it was launched. But then we didn't really announce it until around the sort of, you know, later in the year timeframe, when we saw a peak in the actual readership. So sort of public announcement via our blog system went live. And we saw an upturn in the usage quite dramatically. And it's really proving to be quite a successful story, even machine translated content, that are the challenges which still remains. So you know, we are working on improving the workflow, making it faster, removing the errors that I spoke about in those errors that can creep in trying to make the whole process more robust, looking at new technologies, which allow us to, to statistically measure the quality of translation of content to be translated suitability of the image. And so this presentation is about data driven decision making data really drives globalization services, and Citrix and allows us to make decisions, which we believe will add value to the customer. And as I said, in the data driven decisions, where we've kind of used data to support the argument has really been along the lines of Dutch, Italian, Brazilian Portuguese, now available in workspace vs. app with intelligence. The keyboard I, me and language buyers now out there. And there are more enhancements planned for 2020. And we have 100% of our documentation. Now, there's languages but being machine translation, once you make improvements in a natural workflow, there's no stopping. So I believe we can go this is just the beginning of our journey, I believe. And so that's kind of where we are today with machines with data within Citrix and what we've done. So I'd be interested to hear of your data stories. And you know, what you've learned from data in your company. Thank you very much.