Advancing Business Knowledge with Project Cortex

I was on the Spaceship Earth ride at Epcot, being taken inside the big dome at Disney World, as a break from Microsoft Ignite 2019. If you have never been on this Disney ride before, you are a time traveler going back in time to learn how our modern way of life emerged. As I listened to Dame Judi Dench’s narration, it became clear that the ride is a love letter to how Man came to possess, retain, distribute, and harness knowledge over time to drive innovation.

I was at Microsoft Ignite to talk about an ambitious new Microsoft project, called Project Cortex, that used many different types of Artificial Intelligence to convert data into curatable knowledge. I was fortunate to be creating the Power BI analytics solution starters as a vendor to the team. I worked with and learned about Cortex at the side of Naomi Moneypenny and Sean Squires. The more I saw and understood what Cortex could do, the more excited I became about the power of the solution.

Knowledge Expands in Four Stages

The Spaceship Earth ride postulated a recurring pattern to expanding knowledge over time.

• Data is gathered and/or generated
• Humans curate the data, generate connections between topics, and derive knowledge
• Knowledge is distributed to others
• Knowledge is used to drive innovations

New innovations then drive the next cycle of data gathering and generation. This pattern repeats over history, from when language was first created to the creation of paper and portable knowledge to the invention of the printing press. Data gathering and sense making of that data were human activities that kept pace with the speed in which the data was generated and gathered for most of history.

When I was a boy, a PBS program called Connections explored these historical innovation chains and how they happened. Serendipity played a huge role in how innovation occurred. Many of the names we know from our education labored to generate knowledge from the data they gathered. Much of the available knowledge that they drew upon came down to who they knew or what books they happened to read. It amazed me that innovations occurred at all.

Computers Changed the Cycle of Knowledge Expansion

In the 20^th century, something unintended changed when computers were invented. For the first time, we began capturing and generating data faster than we could make sense of it and convert it into knowledge. No longer could humans make the necessary connections between data entities fast enough. Existing knowledge making processes were not scalable. Innovations became purpose driven, based on the goals of the people who were driving the effort. These efforts often generated a treasury of data that was usually not fully explored.

For example, President John F. Kennedy declared, “We choose to go to the Moon.” and seven years later, we landed on the Moon. This monumental effort drove an unprecedented flurry of activity that generated mountains of data and a plethora of innovations. Yet, how many more innovations were possible from this effort? We do not really know as there is not an efficient way to mine the knowledge within.

The Role of Artificial Intelligence

Artificial intelligence started in 1956 and has had the promise to help unlock knowledge from our wealth of data for many decades. Simply, we can no longer scale our knowledge without the help of our silicon-based tools. There have been many generations of AI. Finally, AI has had a progression of capabilities related to machine learning and machine teaching, where for the first time, it is possible to use it to create connections between the data entities. These connections expedite the conversion of data into knowledge in a scalable fashion and can be curated by humans.

One Small Step for Microsoft

As the Spaceship Earth ride ended and we returned to Earth, I thought with Project Cortex, this is the next big step on this knowledge journey. A key Cortex capability is its ability to automatically curate data from both purposeful AI enabled data capture activities and from ambient information held within SharePoint files and lists. It will help us discover data we did not know we had.

After the product announcements, I saw several people focused on specific aspects of Cortex’s underlying technologies. There is discussion of enhancement of metadata and of being able to automate manual processes. All those elements are important, but they miss the bigger picture of knowledge transformation.

A Potential Big Leap for Businesses

In business intelligence, we have been attempting to make sense of the data for decades. We are limited however, to structured data and to the report developer’s knowledge of the data. According to some estimates, only 28% of an organization’s data is structured. The rest is locked away in documents, PDFs, and other unstructured repositories. Business intelligence alone cannot unlock organizational knowledge. If businesses can incorporate Cortex data with structured data and create an enterprise semantic model over both, they may discover new learnings and capabilities they weren’t aware they possessed.

The ability for Cortex to help facilitate the transformation of data to knowledge, however rudimentary in this first step, is the innovation we have needed for nearly 70 years. I am excited by what discoveries Cortex will enable. Making organizations aware of the knowledge they possess and helping generate the connections between the knowledge topics will be a new voyage of discovery. Once curated, the new extracted knowledge can then be analyzed using tools like Power BI. I am not certain what we will learn, but I am excited by the possibilities of where it may lead.

Find out more about Project Cortex at aka.ms/projectcortex

UPDATE: The first product of Project Cortex, SharePoint Syntex, has gone live! https://www.microsoft.com/en-us/microsoft-365/enterprise/sharepoint-syntex?activetab=pivot:overviewtab