Data Platforms and Data Management: Maximising Data to Deliver Business Insights and Innovation

There is nothing more rewarding for me than to get out among groups of customers at events. There’s a magic that happens; I find myself amidst an eclectic mix of IT and business leaders from just about any industry you can think of, all united in their desire to do better, to go further. At our recent JuiceIT events, we’ve had representatives from schools, miners, banks, supermarkets and not-for-profits, all sharing wisdom gained and challenges overcome.

There’s inevitably a hot topic at any event, and this year, that topic is digital transformation. How do we get from where we are to where we want to be, and what value can we gain from our data in order to extract the business insights and innovation springboard that we need?

We’ve come a long way since my early days within the IT industry. 20 years ago, I felt tremendous satisfaction in a project that involved taking mountains of paper emerging from customer satisfaction survey, creating a database in Microsoft Access, and enabling reports to be printed on stacks of A3 a half inch thick. Even though executives may have had to wait weeks for information back then, at the time it was cutting edge.

Data: The Key to Business Transformation

Although that project now seems archaic, it did demonstrate something vital: access to accurate, timely data is key. Businesses are operating in a drastically different, data-driven world in which survival is not assured, even for the big brands. While the average time on the S&P 500 used to be 67 years in the 1920s, in the next decade, some 75% of today’s businesses will no longer exist. Those using data and machine learning to gain insight, and to innovate, are those building momentum in the fiercest of competition.

One of the best examples of a data-driven business is Netflix. Their powerful machine learning processes keeps you hooked to their service. Every time you kick back to watch a movie, or indulge in your secret passion for classic British comedy, that new information is fed into their systems. Each time you switch on, the offerings presented are better designed to suit your individual tastes; Netflix knows just what you like, and provides a tailored experience. By using machine learning to reduce churn, Netflix has estimated that they have saved $1 billion in revenue from viewers churning from their service.

You don’t have to be Netflix to reap the benefits from data, analytics and AI. It is estimated by Gartner that $3.9 trillion of business value will be derived by AI by 2022 worldwide. With the democratisation of AI, it is possible, with the right alignment of vision and application of technology, for each and every organisation do reap the benefits. Dairy farmers are finding ways to monitor individual cows, retailers are able to better predict stock needs and reduce costs, and we use AI to accelerate digital transformation in our own business so that we can continually improve our own customer service.

What Drives AI?

We’re used to data being seen as the domain of the IT team, but in fact, almost 80% off AI projects are driven by other lines of business, according to Microsoft. This presents an opportunity for IT to be agents of change, to help the business to achieve more. This means that as a profession we face a shift in which our roles are less segregated from the wider organisation, and we must work more closely with diverse teams than ever before.

Much like that early Microsoft Access project, executives are accustomed to using data to see what has happened. Now, the expectation is to go further and use tools that can define why it happened, when future events will happen, and what action should be taken.

The Modern Technology Fuelling AI

A slide at my JuiceIT presentation read, “AI is about amplifying human ingenuity with intelligent technology”. This is Microsoft’s definition of AI, and highlights the challenges needed to be overcome for machines to have ‘human-like’ qualities. Machines need to be able to provide reasoning to make decisions with imperfect or incomplete data, and to interpret that data in many forms, such as text, images and voice. A key thing for the strongest use of AI is the way for machines to interact with many people in natural ways.

All this is made possible by the convergence of three key technologies: cloud, data and intelligence. Most organisations already use cloud to some degree, and its ability to scale, store and compute without a massive on-premises investment. They have never had so much data, and ability to store it is key to training algorithms. Completing the set is the incredible ability of intelligence tools, libraries and services available off the shelf now, without the need for data mining and mathematical experts.

Using the Stanford Question and Answer Dataset (SQuAD) as a benchmark, Microsoft was able to achieve an 85.4% result in reading comprehension, nudging ahead of the human performance of 82.3%. Microsoft currently employs some 8,000 researchers to work on AI and develop the tools for today, and we can expect more breakthroughs as their research gathers momentum.

The Data Journey Map

Embarking on the data journey can be initially daunting, so we break it down for customers into a logical four step journey. From Data, without which an organisation simply won’t exist, through information, knowledge, and onto intelligence.

There is a range of tools, such as Azure data services, to help you modernise applications and shift workloads to cloud. The ability to scale makes it much easier to manage costs without facing capital expenditure.

Information is about bringing diverse datasets together into a coherent view. For one customer, diverse datasets, such as dispatch, finance and client information were available, and we were able to bring them together under a single pane of glass, so that executives could get a clear picture immediately of the customer journey, helping them to transform the service the offer their clients.

Using that information is not the end of the story though. In the knowledge phase, analytics dig deeper to give a better understanding of why something happened, and what is likely to occur in the future. Designed well, it enables all employees to work with diverse datasets in a self-service design that takes them away from labour-intensive reports that took weeks to compile. For one government department we worked with, that meant the ability to quickly and accurately answer questions at an estimates hearing, drilling down to details of expenditure by electorate.

That leads organisations into the intelligence phase, where machine learning can be used to improve customer service, as well as optimise operations, such as the example with Netflix, in improving retention. There are some tremendous applications for machine learning throughout the organisation, making it a playground for innovative thinkers. At Data^#3, we even use AI to replace  paper questionnaires and perform initial workstation assessments where staff work from home. Using off-the-shelf machine vision with a pre-trained object recognition model, we are able to determine if the staff member’s work environment has a suitable desk and chair, for example.

Getting Started

We recognise that digital transformation is hard. One of the key blockers is inertia and getting business funding for these transformative projects. To that end, we have a specialised offering – our Data Innovation Workshop. Through the workshop, we apply our formula for digital transformation with data. The first step is to bring together your business and IT stakeholders in a room, and we then apply design-thinking principles for creating a shared vision and identify measurable business outcomes.

Next, we use our Data Journey Map to perform a current state assessment of your data and technology landscape. This identifies gaps with where you are now and where you need to be – which applies not only to the data and technology, but also to the skills, frameworks and processes that would need to be employed to achieve the shared vision.

Finally, we develop a blueprint, which helps identify the activities that would need to be undertaken to bring the solution together. This workshop provides a path to developing a solid plan, and often used to help support a business case with realistic benefits that could be achieved.

It is a great time to be working in technology for anyone who, like me, is endlessly fascinated by the possibilities surrounding us. For more guidance about how to envision, align and execute in this new digital world, contact me or follow me on LinkedIn.