The CLAIRE-ion Call at Informatica World 2019: AI Needs Managed Data and Data Management Needs AI

From May 20-23, Amalgam Insights attended Informatica World 2019, Informatica’s end user summit dedicated to the world of data management. Over the years, Informatica has transformed from a data integration and data governance vendor to a broad-based enterprise data management vendor offering what it calls the “Intelligent Data Platform,” consisting of data integration, Big Data Management, Integration Platform as a Service, Data Quality and Governance, Master Data Management,  Enterprise Data Catalog, and Data Security & Privacy. Across all of these areas, Informatica has built market-leading products with the goal of providing high-quality point solutions that can work together to solve broad data challenges.

To support this holistic enterprise data management approach, Informatica has developed an artificial intelligence layer, called CLAIRE, to support metadata management, machine learning, and artificial intelligence services that provide context, automation, and anomaly recognition across Informatica’s varied data offerings. And at Informatica World 2019, CLAIRE was everywhere as AI served as a core theme of the event.

CLAIRE was mentioned in Informatica’s Intelligent Cloud Services, automating redundant, manual data processing steps. It was in Informatica Data Quality, cleansing and standardizing incoming data. It was in Data Integration, speeding up the integration process for non-standard data. It was in the Enterprise Data Catalog, helping users to understand where data was going across their organization. And it was in Data Security, identifying patterns and deviations in user activities that could indicate suspicious behavior, while contextually masking sensitive data for secure use.

What’s meant by CLAIRE? It’s the name tying together all of Informatica’s automated smart contextual improvements across its Intelligent Data Platform, surfacing relevant data, information, and recommendations just-in-time throughout various parts of the data pipeline. By bringing “AI” to data management, Informatica hopes to improve efficiency throughout the whole process, helping companies manage the growing pace of data ingestion.

The Data Tsunami

It’s no news to anyone that dealing with data is only getting exponentially harder. Enterprises are collecting more data than ever – every customer interaction is a data point to be analyzed that collectively determines what the right next step is. Website and email interactions, social media posts, image and audio and video data, and service calls combined with broader demographic data mean your data swamp has become a data tsunami. Billions of daily interactions, billions of daily searches, billions of devices and sensors – combined with a shortage of over 800,000 data science professionals – add up to an exponentially-expanding problem. Even the expansion of data science and machine learning training programs isn’t yet sufficient to contain the swell.

We’re well beyond the scope of “throw more people at the problem” to fix it; even including “citizen data scientists” among this number is wielding a bucket against that tsunami. Instead, says Informatica, empowering those people with tools that can amplify their ability to manage the data – by automating the most repetitive tasks to ensure that managed data adheres to standards for more efficient further analysis – is necessary to make any progress.

It is important to note that Informatica’s goal in driving automation, metadata context, and artificial intelligence is not to simply automate people, but to introduce data-driven just-in-time enhancements and recommendations that can be controlled by skilled employees. Fully-automated AI without human supervision in this day and age can result in unfortunate situations, like the recent Boeing 737 crashes where AI ended up hiding key information, or the fatal accident associated with Uber’s autonomous vehicle trial. Currently, the state of AI makes this technology well-suited to automate and analyze data at massive scale and to provide algorithmic and data-based judgments or to conduct deep learning on closed and limited systems with set rules (such as Chess or Go), but AI is still poorly suited to handle the ethical, cultural, and business logic needed to provide judgment at a human level.

Until AI provides human-level judgment and contextualization at any level, it is important for AI to be overseen by people with role-based expertise and that the AI needs to be provided as scalable and interoperable services. Amalgam Insights categorizes this context as a REEP, Role-Based Expert Enhancement Platforms, as an optimal use of AI.

Partnerships

Another thing Informatica is doing to try to smooth the data-driven path is by enhancing and highlighting partnerships along their various digital transformation “journeys.” At Informatica World, partnerships were a key aspect of the event as Informatica described enhancements to partnerships with large enterprise technology providers including Databricks, Tableau, Amazon, Google, and Microsoft.

For those looking to move data to the cloud, Informatica’s new announcement with AWS and supported by Cognizant to conduct a joint “Intelligent Data Migration” assessing what cloud migration of on-premises data to Amazon Redshift would look like for their company – possibly for free! – lowers one barrier to entry. Likewise, adding Microsoft and Databricks to the list of supported data lakes, making Informatica Intelligent Cloud Services available on Google Cloud Platform (GCP), and providing additional support within Informatica for Google BigQuery and Google Cloud Dataproc enhance the understanding that Informatica can manage your data no matter where it is. And bringing Informatica’s Enterprise Data Catalog into Tableau with a simple connector means users whose primary data interface is Tableau can still take advantage of the CLAIRE enhancements to Informatica’s products in a familiar interface, supporting the “journey” towards improved analytics. The support associated with leading data lakes, cloud data warehouses (and for those wondering, Informatica also supports Snowflake and other cloud data sources), and visualization solutions demonstrates Informatica’s continuing commitment to be the “Switzerland of data,” as CEO Anil Chakravarthy stated during his keynote.

So the technical pieces are falling into place for companies on these digital transformation journeys based on continual improvements to the Informatica platform that take advantage of machine learning to make processes more efficient, and partnerships with other best-of-breed data solutions.

However, even long-time Informatica customers are struggling to manage what Informatica is calling “Data 3.0” based on the generational shift of data from a capability that supported applications to data as the core control point for applications and digital businesses. As data has increased in importance, its role in the enterprise has changed from a combination of data stores and archives seen as peripheral to application functions and a potential liability to becoming a core business asset that needs to be managed. As an example, when I asked one customer about her thoughts on the demonstrations of CLAIRE enhancements to existing products, she told me that she called her boss right after the Tuesday keynote and said, “We NEED this.” But her company is still wrestling even with assuring data quality, which is foundational to the capacity for success with these next analytical steps. This isn’t happening from a lack of available technology; Informatica’s Data Quality product has been available for years. The hiccups are with corporate cultures unprepared for change management on an unprecedented scale when it comes to making data-driven business ventures, and this common thread showed up among all of the Informatica customers I spoke with.

Recommendations

Based on our attendance of Informatica World, Amalgam Insights provides the following recommendations:

• For data practitioners, especially those who attended Informatica World, be aware that your world is changing and that the tasks of database management, data integration, data governance, metadata and semantic management, data security, and even machine learning preparation are quickly coming together. This is both intimidating and a massive opportunity for those of you who want to be true data stewards. We already knew that our data (and the third-party data that augments it) were the crown jewels of the company, even in the days when we were reluctantly throwing 90% of it in to tape and archived disks for financial reasons. But in today’s era of data, you are now managing an asset, perhaps the most valuable asset, in your company and the business increasingly has both the financial modeling to account for this and the tools to fully guard, govern, and contextualize data with the aid of appropriate machine learning and automation tools. In this context, provide your managers with guidance on the “Art of the Possible” in managing data with one eye towards both helping your company and with another on your own career.

• For data team managers, prepare for the future of Artificial and Augmented Intelligence by taking the practical steps of organizing and contextualizing data. Amalgam Insights estimates that 90% of current AI projects end up failing and never being put into production, and the key reason is that these projects lack quality data, use too few data sources, or use the wrong data. Before your company starts looking deeply at the power of AI, your organization must effectively organize, catalog, and contextualize its data – remember, AI needs data management!

• Data team managers and C-level officers should note: before your company can effectively take advantage of AI, it must address cultural resistance to AI. Many are concerned that AI is going to take their jobs, or at least change them significantly. Contextually, what Informatica provides right now with CLAIRE are tools to do data management jobs more effectively; the tone struck at Informatica World was very much about empowering humans to make the decisions, while providing them with the right amount of contextually-relevant information. Managers will need to provide a realistic vision of how incorporating AI into existing work changes the nature and responsibilities of their team members’ jobs, and a roadmap for how team members will be trained for new responsibilities.

• For C-level officers seeking to become more digital or to take full advantage of digital transformation efforts, realize both that data is an asset and data is the core of any machine learning or artificial intelligence efforts. This means that your organization should start treating its data as a balance sheet item and not simply as a cost that needs to be minimized by any means necessary. The mindset of treating data as garbage that needs to be kept on tape for seven years from a compliance perspective is being replaced by the mindset of identifying all data that is business-relevant, then supporting the analytics, automation, and machine learning tools that best use that data to augment the business. This approach also has repercussions for being “future-facing” and for dealing with stockholder demands for guidance, as organizations with a proactive approach to data are also better prepared for supporting AI efforts to increase revenue, reduce risk, improve support, optimize logistics, and otherwise improve business efforts algorithmically. Before buying into the buzzwords, buy into the true value of data.

• As a key piece of advice to current and potential Informatica customers, Amalgam Insights provides the reminder that Informatica’s capabilities extend beyond PowerCenter, or even the newer Axon governance or Secure@Source security capabilities. Informatica has built a strategic set of data management capabilities designed both to work with each other and to support the core challenge of supporting enterprise-grade and “Big” data. Because of this, Amalgam Insights advises that current Informatica customers seeking additional data management, governance, or security capabilities should consider looking internally to Informatica as part of their due diligence.

• And for current and potential Informatica customers considering moving on-premises data to the cloud, be aware that there are both tools and assessments available to determine the cost and effort associated with moving data to the cloud. Informatica’s assessment with Amazon Web Services and Cognizant is one example of this that is especially valuable for joint Informatica-Amazon customers. But the bigger picture is that enterprises should not simply treat on-premises hosting or cloud hosting as “experiments” when there are methodological and structured ways of determining what data is better off either on-site or off-campus.

The key takeaway: Dealing with exponentially growing data is hard and getting harder. There are technical solutions that can help. But the bigger hurdle is getting buy-in from a cultural perspective, whether C-level or front-line. In terms of recommendations, businesses must acknowledge that working with data as an asset is a change management problem, not just a problem where throwing bodies or software at it fixes everything. Everyone trying to implement change needs to persuade those further up the chain that these changes needs to be made.