Lynne Baer

June 3, 2019June 3, 2019

The CLAIRE-ion Call at Informatica World 2019: AI Needs Managed Data and Data Management Needs AI

Knowledge of the Past, Uncertainty of the Future

From May 20-23, Amalgam Insights attended Informatica World 2019, Informatica’s end user summit dedicated to the world of data management. Over the years, Informatica has transformed from a data integration and data governance vendor to a broad-based enterprise data management vendor offering what it calls the “Intelligent Data Platform,” consisting of data integration, Big Data Management, Integration Platform as a Service, Data Quality and Governance, Master Data Management, Enterprise Data Catalog, and Data Security & Privacy. Across all of these areas, Informatica has built market-leading products with the goal of providing high-quality point solutions that can work together to solve broad data challenges.

To support this holistic enterprise data management approach, Informatica has developed an artificial intelligence layer, called CLAIRE, to support metadata management, machine learning, and artificial intelligence services that provide context, automation, and anomaly recognition across Informatica’s varied data offerings. And at Informatica World 2019, CLAIRE was everywhere as AI served as a core theme of the event.

CLAIRE was mentioned in Informatica’s Intelligent Cloud Services, automating redundant, manual data processing steps. It was in Informatica Data Quality, cleansing and standardizing incoming data. It was in Data Integration, speeding up the integration process for non-standard data. It was in the Enterprise Data Catalog, helping users to understand where data was going across their organization. And it was in Data Security, identifying patterns and deviations in user activities that could indicate suspicious behavior, while contextually masking sensitive data for secure use.

What’s meant by CLAIRE? It’s the name tying together all of Informatica’s automated smart contextual improvements across its Intelligent Data Platform, surfacing relevant data, information, and recommendations just-in-time throughout various parts of the data pipeline. By bringing “AI” to data management, Informatica hopes to improve efficiency throughout the whole process, helping companies manage the growing pace of data ingestion.

The Data Tsunami

It’s no news to anyone that dealing with data is only getting exponentially harder. Enterprises are collecting more data than ever – every customer interaction is a data point to be analyzed that collectively determines what the right next step is. Website and email interactions, social media posts, image and audio and video data, and service calls combined with broader demographic data mean your data swamp has become a data tsunami. Billions of daily interactions, billions of daily searches, billions of devices and sensors – combined with a shortage of over 800,000 data science professionals – add up to an exponentially-expanding problem. Even the expansion of data science and machine learning training programs isn’t yet sufficient to contain the swell.

We’re well beyond the scope of “throw more people at the problem” to fix it; even including “citizen data scientists” among this number is wielding a bucket against that tsunami. Instead, says Informatica, empowering those people with tools that can amplify their ability to manage the data – by automating the most repetitive tasks to ensure that managed data adheres to standards for more efficient further analysis – is necessary to make any progress.

It is important to note that Informatica’s goal in driving automation, metadata context, and artificial intelligence is not to simply automate people, but to introduce data-driven just-in-time enhancements and recommendations that can be controlled by skilled employees. Fully-automated AI without human supervision in this day and age can result in unfortunate situations, like the recent Boeing 737 crashes where AI ended up hiding key information, or the fatal accident associated with Uber’s autonomous vehicle trial. Currently, the state of AI makes this technology well-suited to automate and analyze data at massive scale and to provide algorithmic and data-based judgments or to conduct deep learning on closed and limited systems with set rules (such as Chess or Go), but AI is still poorly suited to handle the ethical, cultural, and business logic needed to provide judgment at a human level.

Until AI provides human-level judgment and contextualization at any level, it is important for AI to be overseen by people with role-based expertise and that the AI needs to be provided as scalable and interoperable services. Amalgam Insights categorizes this context as a REEP, Role-Based Expert Enhancement Platforms, as an optimal use of AI.

Partnerships

Another thing Informatica is doing to try to smooth the data-driven path is by enhancing and highlighting partnerships along their various digital transformation “journeys.” At Informatica World, partnerships were a key aspect of the event as Informatica described enhancements to partnerships with large enterprise technology providers including Databricks, Tableau, Amazon, Google, and Microsoft.

For those looking to move data to the cloud, Informatica’s new announcement with AWS and supported by Cognizant to conduct a joint “Intelligent Data Migration” assessing what cloud migration of on-premises data to Amazon Redshift would look like for their company – possibly for free! – lowers one barrier to entry. Likewise, adding Microsoft and Databricks to the list of supported data lakes, making Informatica Intelligent Cloud Services available on Google Cloud Platform (GCP), and providing additional support within Informatica for Google BigQuery and Google Cloud Dataproc enhance the understanding that Informatica can manage your data no matter where it is. And bringing Informatica’s Enterprise Data Catalog into Tableau with a simple connector means users whose primary data interface is Tableau can still take advantage of the CLAIRE enhancements to Informatica’s products in a familiar interface, supporting the “journey” towards improved analytics. The support associated with leading data lakes, cloud data warehouses (and for those wondering, Informatica also supports Snowflake and other cloud data sources), and visualization solutions demonstrates Informatica’s continuing commitment to be the “Switzerland of data,” as CEO Anil Chakravarthy stated during his keynote.

So the technical pieces are falling into place for companies on these digital transformation journeys based on continual improvements to the Informatica platform that take advantage of machine learning to make processes more efficient, and partnerships with other best-of-breed data solutions.

However, even long-time Informatica customers are struggling to manage what Informatica is calling “Data 3.0” based on the generational shift of data from a capability that supported applications to data as the core control point for applications and digital businesses. As data has increased in importance, its role in the enterprise has changed from a combination of data stores and archives seen as peripheral to application functions and a potential liability to becoming a core business asset that needs to be managed. As an example, when I asked one customer about her thoughts on the demonstrations of CLAIRE enhancements to existing products, she told me that she called her boss right after the Tuesday keynote and said, “We NEED this.” But her company is still wrestling even with assuring data quality, which is foundational to the capacity for success with these next analytical steps. This isn’t happening from a lack of available technology; Informatica’s Data Quality product has been available for years. The hiccups are with corporate cultures unprepared for change management on an unprecedented scale when it comes to making data-driven business ventures, and this common thread showed up among all of the Informatica customers I spoke with.

Recommendations

Based on our attendance of Informatica World, Amalgam Insights provides the following recommendations:

For data practitioners, especially those who attended Informatica World, be aware that your world is changing and that the tasks of database management, data integration, data governance, metadata and semantic management, data security, and even machine learning preparation are quickly coming together. This is both intimidating and a massive opportunity for those of you who want to be true data stewards. We already knew that our data (and the third-party data that augments it) were the crown jewels of the company, even in the days when we were reluctantly throwing 90% of it in to tape and archived disks for financial reasons. But in today’s era of data, you are now managing an asset, perhaps the most valuable asset, in your company and the business increasingly has both the financial modeling to account for this and the tools to fully guard, govern, and contextualize data with the aid of appropriate machine learning and automation tools. In this context, provide your managers with guidance on the “Art of the Possible” in managing data with one eye towards both helping your company and with another on your own career.

For data team managers, prepare for the future of Artificial and Augmented Intelligence by taking the practical steps of organizing and contextualizing data. Amalgam Insights estimates that 90% of current AI projects end up failing and never being put into production, and the key reason is that these projects lack quality data, use too few data sources, or use the wrong data. Before your company starts looking deeply at the power of AI, your organization must effectively organize, catalog, and contextualize its data – remember, AI needs data management!

Data team managers and C-level officers should note: before your company can effectively take advantage of AI, it must address cultural resistance to AI. Many are concerned that AI is going to take their jobs, or at least change them significantly. Contextually, what Informatica provides right now with CLAIRE are tools to do data management jobs more effectively; the tone struck at Informatica World was very much about empowering humans to make the decisions, while providing them with the right amount of contextually-relevant information. Managers will need to provide a realistic vision of how incorporating AI into existing work changes the nature and responsibilities of their team members’ jobs, and a roadmap for how team members will be trained for new responsibilities.

For C-level officers seeking to become more digital or to take full advantage of digital transformation efforts, realize both that data is an asset and data is the core of any machine learning or artificial intelligence efforts. This means that your organization should start treating its data as a balance sheet item and not simply as a cost that needs to be minimized by any means necessary. The mindset of treating data as garbage that needs to be kept on tape for seven years from a compliance perspective is being replaced by the mindset of identifying all data that is business-relevant, then supporting the analytics, automation, and machine learning tools that best use that data to augment the business. This approach also has repercussions for being “future-facing” and for dealing with stockholder demands for guidance, as organizations with a proactive approach to data are also better prepared for supporting AI efforts to increase revenue, reduce risk, improve support, optimize logistics, and otherwise improve business efforts algorithmically. Before buying into the buzzwords, buy into the true value of data.

As a key piece of advice to current and potential Informatica customers, Amalgam Insights provides the reminder that Informatica’s capabilities extend beyond PowerCenter, or even the newer Axon governance or Secure@Source security capabilities. Informatica has built a strategic set of data management capabilities designed both to work with each other and to support the core challenge of supporting enterprise-grade and “Big” data. Because of this, Amalgam Insights advises that current Informatica customers seeking additional data management, governance, or security capabilities should consider looking internally to Informatica as part of their due diligence.

And for current and potential Informatica customers considering moving on-premises data to the cloud, be aware that there are both tools and assessments available to determine the cost and effort associated with moving data to the cloud. Informatica’s assessment with Amazon Web Services and Cognizant is one example of this that is especially valuable for joint Informatica-Amazon customers. But the bigger picture is that enterprises should not simply treat on-premises hosting or cloud hosting as “experiments” when there are methodological and structured ways of determining what data is better off either on-site or off-campus.

The key takeaway: Dealing with exponentially growing data is hard and getting harder. There are technical solutions that can help. But the bigger hurdle is getting buy-in from a cultural perspective, whether C-level or front-line. In terms of recommendations, businesses must acknowledge that working with data as an asset is a change management problem, not just a problem where throwing bodies or software at it fixes everything. Everyone trying to implement change needs to persuade those further up the chain that these changes needs to be made.

May 6, 2019May 13, 2019

Data Science and Machine Learning News Roundup, April 2019

On a monthly basis, I will be rounding up key news associated with the Data Science Platforms space for Amalgam Insights. Companies covered will include: Alteryx, Amazon, Anaconda, Cambridge Semantics, Cloudera, Databricks, Dataiku, DataRobot, Datawatch, Domino, Elastic, Google, H2O.ai, IBM, Immuta, Informatica, KNIME, MathWorks, Microsoft, Oracle, Paxata, RapidMiner, SAP, SAS, Tableau, Talend, Teradata, TIBCO, Trifacta, TROVE.

Alteryx Acquires ClearStory Data to Accelerate Innovation in Data Science and Analytics

Alteryx acquired ClearStory Data, an analytics solution for complex and unstructured data with a focus on automating Big Data profiling, discovery, and data modeling. This acquisition reflects Alteryx’s interest in expanding its native capabilities to include more in-house data visualization tools. ClearStory Data provides a visual focus on data prep, blending, and dashboarding with their Interactive Storyboards that partners with Alteryx’s ongoing augmentation of internal visualization capabilities throughout the workflow such as Visualytics.

Dataiku Announces the Release of Dataiku Lite Edition

Dataiku released two new versions of its machine learning platform, Dataiku Free and Dataiku Lite, targeted towards small and medium businesses. Dataiku Free will allow teams of up to three users to work together simultaneously; it is available both on-prem and on AWS and Azure. Dataiku Lite will provide support for Hadoop and job scheduling beyond the capabilities of Dataiku Free. Since Dataiku already partners with over 1000 small and medium businesses, creating versions of its existing platform more financially accessible to such organizations lowers a significant barrier to entry, and grooms smaller companies to grow their nascent data science practices within the Dataiku family.

DataRobot Celebrates One Billion Models Built on Its Cloud Platform

DataRobot announced that as of mid-April, its customers had built one billion models on its automatic machine learning program. Vice President of Product Management Phil Gurbacki noted that DataRobot customers build more than 2.5 million models per day. Given that the majority of models created are never successfully deployed – a common theme cited this month at both Enterprise Data World and at last week’s Open Data Science Conference – it seems likely that DataRobot customers don’t currently have one billion models operationalized. If the percentage of deployed models is significantly higher than the norm, though, this would certainly boost DataRobot in potential customers’ eyes, and serve to further legitimize AutoML software solutions as plausible options.

Microsoft, SAS, TIBCO Continue Investments in AI and Data Skills Training

Microsoft announced a new partnership with OpenClassrooms to train students for the AI job marketplace via online coursework and projects. Given an estimate that projects 30% of AI and data jobs will go unfilled by 2022, OpenClassrooms’ recruiting 1000 promising candidates seems like just the beginning of a much-needed effort to address the skills gap.

SAS provided more details on the AI education initiatives they announced last month. First, they launched SAS Viya for Learners, which will allow academic institutions to access SAS AI and machine learning tools for free. A new SAS machine learning course and two new Coursera courses will provide access to SAS Viya for Learners to those wanting to learn AI skills without being affiliated with a traditional academic institution. SAS also expanded on the new certifications they plan to offer: three SAS specialist certifications in machine learning, natural language and computer vision, and forecasting and optimization. Classroom and online options for pursuing both of these certifications will be available.

Meanwhile, TIBCO continued expanding its partnerships with educational institutions in Asia to broaden analytics knowledge in the region. Most recently, it has augmented its existing partnership with Singapore Polytechnic to train 1000 students in analytics and IoT skillsets by 2020. Other analytics education partnerships TIBCO has announced in the last year include Yuan Ze University in Taiwan, Asia Pacific University of Technology and Innovation in Malaysia, and BINUS University in Indonesia.

The big picture: existing data science degree programs and machine learning and AI bootcamps are not providing a large enough volume of highly-skilled job candidates quickly enough to fill many of these data-centric positions. Expect to hear more about additional educational efforts forthcoming from data science, machine learning, and AI vendors.

April 16, 2019

Enterprise Data World 2019: Data Science Will Take Over The World! … Eventually.

Amalgam Insights attended Enterprise Data World, a conference focused on data management, in late March. Though the conference tracks covered a wide variety of data practices, our primary interest was in the sessions on the AI and Machine Learning track. We came away with the impression that the data management world is starting to understand and support some of the challenges that organizations face when trying to get complex data initiatives off the ground, but that the learning process will continue to have growing pains.

Data Strategy Bootcamp

I began my time at Enterprise Data World with the Data Strategy Bootcamp on Monday. Often, organizations focus on getting smaller data projects done quickly in a tactical fashion at the expense of consciously developing their broader data strategy. The bootcamp addressed how to incorporate these “quick wins” into the bigger picture, and delved into the details of what a data strategy should include, and what does the process of building one look like. For people in data analytics and data scientist roles, understanding and contributing to your organization’s data strategy is important because well-documented and properly-managed data means data analysts and data scientists can spend more of their time doing analytics and building machine learning models. The “data scientists spend 80% of their time cleaning and preparing data” number continues to circulate without measurable improvement. To build a successful data strategy, organizations will need to identify business goals that are data-centric to align the organization’s data strategy with its business strategy, assess the organization’s maturity and capabilities across its data ecosystem, and determine long-term goals and “quick wins” that will provide measurable progress towards those goals.

Getting Started with Data Science, Machine Learning, and Artificial Intelligence Initiatives

Actually getting started on data science, machine learning, and artificial intelligence initiatives remains a point of confusion for many organizations looking to expand beyond the basic data analytics they’re currently doing. Both Kristin Serafin and Lizzie Westin of FINRA and Vinay Seth Mohta of Manifold led sessions discussing how to turn talk about machine learning and artificial intelligence into action in your organizations, and how to do so in a way that can scale up quickly. Key takeaways: your organization needs to understand its data to understand what questions it wants answered that require a machine learning approach; it needs to understand what tools are necessary to move forward; it needs to understand who already has pertinent data capabilities within the organization, and who is best positioned to improve their skills in the necessary manner; and you need to obtain buy-in from relevant stakeholders.

Data Job Roles

Data job roles were discussed in multiple sessions; I attended one from the perspective of how analytical jobs themselves are evolving, and one from the perspective of analytical career development. Despite the hype, not everyone is a data scientist, even if they may perform some tasks that are part of a data science pipeline! Data engineers are the difference between data scientists’ experiments sitting in silos and getting them into production where they can affect your company. Data analysts aren’t going anywhere – yet. (Though Michael Stonebraker, in his keynote Tuesday morning, stated that he believed data science would eventually replace BI, pending upskilling a sufficient number of data workers.) And data scientists spend 80% of their time doing data prep instead of building machine learning models; they’d like to do more of the latter, and because they’re an expensive asset, the business needs them to be doing less prep and more building as well.

By the same token, there are so many different specialties across the data environment, and the tool landscape is incredibly large. No one will know everything; even relatively low-level people will need to provide leadership in their particular roles to bridge the much-bemoaned gap between IT and Business. So how can data people do that? They’ll need to learn to talk about their initiatives and accomplishments in business terms – increasing revenue, decreasing cost, managing risk. By doing this, data strategy can be tied to business strategy, and this barrier to success can be surmounted.

Data Integration at Scale

Michael Stonebraker’s keynote highlighted the growing need for people with data science capabilities, but the real meat of his talk centered around how to support complex data science initiatives: doing data integration at scale. One example: General Electric’s procurement system problem. Clearly, the ideal number of procurement systems in any company is “one.” Given mergers and acquisitions, over time, GE had accumulated *75* procurement systems. They could save $100M if they could bring together all of these systems, with all of the information on the terms and conditions negotiated with each vendor via each of these systems. But this required a rather complex data integration process. Once that was done, the same process remained for dealing with their supplier databases, and their customer databases, and a whole host of other data. Machine learning can help with this – once there are sufficient people with machine learning skills to address these large problems. But doing data integration at scale will remain a significant challenge for enterprises for now, with machine learning skills being relatively costly and rare, data accumulation continuing to grow exponentially, and bringing in third-party data to supplement existing analyses..

Knowledge Graphs and Semantic AI

A number of sessions discussed knowledge graphs and their importance for supporting both data management and data science tasks. Knowledge graphs provide a “semantic” layer over standard relational databases – they prioritize documenting the relationships between entities, making it easier to understand how different parts of your organization’s data are interrelated. Because having a knowledge graph about your organization’s data provides natural-language context around data relationships, it can make machine learning models based on that data more “explainable” due to the additional human-legible information available for interpretation and understanding. Another example: if you’re trying to perform a search, most results rely on exact matches. Having a knowledge graph makes it simple to pull up “related” results based on the relationships documented in that knowledge graph.

Data Access, Control, and Usage

My big takeaway from Scott Taylor’s Data Architecture session: data should be a shared, centralized asset for your entire organization; it must be 1) accessible by its consumers 2) in the format they require 3) via the method they require 4) if they have permission to access it (security) 5) and they will use it in a way that abides by governance standards and laws. Data scientists care about this because they need data to do their job, and any hurdle in accessing usable data makes it more likely they’ll avoid using official methods to access the data. Nobody has three months to wait for a data requisition from IT’s data warehouses to be turned around anymore; instead, “I’ll just use this data copy on my desktop” – or more likely these days, in a cloud-hosted data silo. Making centralized access easy to use makes data users much more likely to comply with data usage and access policies, which helps secure data properly, govern its use appropriately, and prevent data silos from forming.

Digging a bit more into the security and governance aspects mentioned above, it’s surprisingly easy to identify individuals in a set of anonymized data. In separate presentations, Matt Vogt of Immuta demonstrated this with a dataset consisting of anonymized NYC taxi data, even as more and more information was redacted from it. Jeff Jonas of Senzing’s keynote took this further – as context accumulates around data, it gets easier to make inferences, even when your data is far from clean. With GDPR on the table, and CCPA coming into effect in nine months, how data workers can use data, ethically and legally, will shift, significantly affecting data workflows. Both the use of data and the results provided by black-box machine learning models will be challenged.

Recommendations

Data scientists and machine learning practitioners should familiarize themselves with the broader data management ecosystem. Said practitioners understand why dirty data is problematic, given that they spend most of their work hours cleaning that data so they can do the actual machine learning model-building, but there are numerous tools available to help with this process, and possibly obviate the need for a particular cleaning job that’s already been done once. As enterprise data catalogs become more common, this will prevent data scientists from spending hours on duplicative work when someone else has already cleaned the set they were planning to use and made it available for the organization’s use.

Data scientists and data science managers should also learn how to communicate the business value of their data initiatives when speaking to business stakeholders. From a technical point of view, making a model more accurate is an achievement in and of itself. But knowing what it means from a business standpoint builds understanding of what that improved accuracy or speed means for the business as a whole. Maybe your 1% improvement in model accuracy means you save your company tens of thousands of dollars by more accurately targeting potential customers who are ready to buy your product – that’s what will get the attention of your line-of-business partners.

Data science directors and Chief Data or Chief Analytics Officers should approach building their organization’s data strategy and culture with the long-term view in mind. Aligning your data strategy with the organization’s business strategy is crucial to your organization’s success. Rather than having both departments tugging on opposite ends of the rope going in different directions, develop an understanding of each others’ needs and capabilities and apply that knowledge to keep everyone focused on the same goal.

Chief Data Officers and Chief Analytics Officers should understand their organization’s capabilities by conducting an assessment both of their data capabilities and capacity available by individual, and to assess the general maturity in each data practice area (such as Master Data Management, Data Integration, Data Architecture, etc.). Knowing the availability of both technical and people-based resources is necessary to develop a scalable set of data processes for your organization with consistent results no matter who the data scientist or analyst is in charge of executing on the process for any given project.

As part of developing their organization’s data strategy, Chief Data Officers and Chief Analytics Officers must work with their legal department to develop rules and processes for accumulating, storing, accessing, and using data appropriately. As laws like GDPR and the California Privacy Act start being enforced, data access and usage will be much more scrutinized; companies not adhering to the letters of those laws will find themselves fined heavily. Data scientists and data science managers who are working on projects that involve sensitive or personal data should talk to their general counsel to ensure they remain on the right side of the law.

April 2, 2019May 24, 2019

Data Science and Machine Learning News Roundup, March 2019

Please register or log into your Amalgam Insights Community account to read more.
Log In Register

March 13, 2019May 4, 2019

At IBM Think, Watson Expands “Anywhere”

At IBM Think in February, IBM made several announcements around the expansion of Watson’s availability and capabilities, framing these announcements as the launch of “Watson Anywhere.” This piece is intended to provide guidance to data analysts, data scientists, and analytic professionals seeking to implement machine learning and artificial intelligence capabilities and evaluating the capabilities of…

Please register or log into your Amalgam Insights Community account to read more.
Log In Register

March 4, 2019May 4, 2019

Data Science and Machine Learning News Roundup, February 2019

Please register or log into your Amalgam Insights Community account to read more.
Log In Register

February 15, 2019March 7, 2019

Four Key Announcements from H2O World San Francisco

Last week at H2O World San Francisco, H2O.ai announced a number of improvements to Driverless AI, H2O, Sparkling Water, and AutoML, as well as several new partnerships for Driverless AI. The improvements provide incremental improvements across the platform, while the partnerships reflect H2O.ai expanding their audience and capabilities. This piece is intended to provide guidance…

Please register or log into your Amalgam Insights Community account to read more.
Log In Register

February 1, 2019

Data Science and Machine Learning News Roundup, January 2019

Cloudera and Hortonworks Complete Planned Merger

In early January, Cloudera and Hortonworks completed their planned merger. With this, Cloudera becomes the default machine learning ecosystem for Hadoop-based data, while providing an easy pathway for expanding into machine learning and analytics capabilities for Hortonworks customers.

Study: 89 Percent of Finance Teams Yet to Embrace Artificial Intelligence

A study conducted by the Association of International Certified Professional Accountants (AICPA) and Oracle revealed that 89% of organizations have not deployed AI to their finance groups. Although a correlation exists between companies with revenue growth and companies that are using AI, the key takeaway is that artificial intelligence is still in the early adopter phase for most organizations.

Gartner Magic Quadrant for Data Science and Machine Learning Platforms

In late January, Gartner released its Magic Quadrant for Data Science and Machine Learning Platforms. New to the Data Science and Machine Learning MQ this year are both DataRobot and Google – two machine learning offerings with completely different audiences and scope. DataRobot offers an automated machine learning service targeted towards “citizen data scientists,” while Google’s machine learning tools, though part of Google Cloud Platform, are more of a DIY data pipeline targeted towards developers. By contrast, I find it curious that Amazon’s SageMaker machine learning platform – and its own collection of task-specific machine learning tools, despite their similarity to Google’s – failed to make the quadrant, given this quadrant’s large umbrella.

While data science and machine learning are still emerging markets, the contrasting demands of these technologies made by citizen data scientists and by cutting-edge developers warrants splitting the next Data Science and Machine Learning Magic Quadrant into separate reports targeted to the considerations of each of these audiences. In particular, the continued growth of automated machine learning technologies will likely drive such a split, as citizen data scientists pursue a “good enough” solution that provides quick results.

December 17, 2018December 19, 2018

Amazon Expands Toolkit of Machine Learning Services at AWS re:Invent

At AWS re:Invent, Amazon Web Services expanded its toolkit of machine learning application services with the announcements of Amazon Comprehend Medical, Amazon Forecast, Amazon Personalize, and Amazon Textract. These new services augment the capabilities Amazon provides to end users when it comes to text analysis, personalized recommendations, and time series forecasts. The continued growth of these individual services removes obstacles for companies looking to get started with common machine learning tasks on a smaller scale; rather than building a wholesale data science pipeline in-house, these services allow companies to quickly get one task done, and this permits an incremental introduction to machine learning for a given organization. Forecast, Personalize, and Textract are in preview, while Comprehend Medical is available now.

Amazon Comprehend Medical, Forecast, Personalize, and Textract join a collection of machine learning services that include speech recognition (Transcribe) and translation (Translate), speech-to-text and text-to-speech (Lex and Polly) to power machine conversation such as chatbots and Alexa, general text analytics (Comprehend), and image and video analysis (Rekognition).

New Capabilities

Amazon Personalize lets developers add personalized recommendations into their apps, based on a given activity stream from that app and a corpus of what’s available to be recommended, whether that’s products, articles, or other things. In addition to recommendations, Personalize can also be used to customize search results and notifications. By combining a given search string or location with contextual behavior data, Amazon looks to provide customers with the ability to build trust.

Amazon Forecast builds private, custom time-series forecast models that predict future trends based on that data. Customers provide both histoical data and related causal data, and Forecast analyzes the data to determine the relevant factors in building its models and providing forecasts.

Amazon Textract extracts text and data from scanned documents, without requiring manual data entry or custom code. In particular, using machine learning to recognize when data is in a table or form field and treat it appropriately will save a significant amount of time over the current OCR standard.

Finally, Amazon Comprehend Medical, an extension of last year’s Amazon Comprehend, uses natural language processing to analyze unstructured medical text such as doctor’s notes or clinical trial records, and extract relevant information from this text.

Recommendations

Organizations doing resource planning, financial planning, or other similar forecasting that currently lack the capability to do time series forecasting in-house should consider using Amazon Forecast to predict product demand, staffing levels, inventory levels, material availability, and to perform financial forecasting. Outsourcing the need to build complex forecasting models in-house lets departments focus on the predictions.

Consumer-oriented organizations looking to build higher levels of engagement with their customers who provide generic, uncontextualized recommendations right now (based on popularity or other simple measures) should consider using Amazon Personalize to provide personalized recommendations, search results, and notifications via their apps and website. Providing high-quality relevant recommendations a la minute builds customer trust in the quality of a given organization’s engagement efforts, particularly compared to the average spray-and-pray marketing communication.

Organizations that still depend on physical documents, or who have an archive of physical documents to scan and analyze, should consider using Amazon Textract. OCR’s limits are well-known, especially when it comes to accurately interpreting and formatting semi-structured blocks of text data such as form fields and tables, resulting in significant time devoted to post-processing manual correction. Textract handles complex documents without the need for custom code or maintaining templates; being able to automate text interpretation and analysis further accelerates document processing workflows, and better permits organizations to maintain compliance.

Medical organizations using software that depends on manually-implemented rules to process their medical text should consider using Amazon Comprehend Medical. By removing the need to maintain a list of rules in-house, Comprehend Medical accelerates the ability to extract and analyze medical information from unstructured text fields like doctor’s notes and health records, improving processes such as medical coding, cohort analysis to recruit patients for clinical trials, and health monitoring of patients.

All organizations looking to use machine learning services from external providers need to consider whether outsourcing will work for their circumstances. Data privacy is a key concern, and even more so in regulated verticals with industry-specific rules such as HIPAA. Does the service you want to use respect those rules? From a compliance perspective, why a model gives the results it does needs to be explained as well; merely accepting results from the black box at face value is insufficient. Machine learning products that automatically provide such an explanation in plain English do exist, but this feature is still uncommon and in its infancy.

Conclusion

With its latest announcements, Amazon continues to broaden the scope of customer issues it addresses with machine learning services. Medical companies need better text analytics yesterday, but struggle to comply with HIPAA while assessing the data they have. Customer-facing organizations face stiff competition when their competitor is only a click away. And any company trying to plan for the future based on past data grapples with understanding what factors affect future results. Amazon’s machine learning application services address common tactical business issues by simplifying the process for customers of implementing task-specific machine learning models to pure inputs and outputs. These services present outsourcing opportunities for overworked departments struggling to keep up.