# Infosys Whitepaper 
Title: The right testing strategy for AI systems - An Infosys viewpoint 
Author: Infosys Limited 
Format: PDF 1.6 

---

  Page: 1 / 8  

---

 PERSPECTIVE THE RIGHT TESTING STRATEGY FOR AI  SYSTEMS AN INFOSYS VIEWPOINT VENKATESH IYENGAR, AVP - Group Practice Engagement Manager, Infosys SUNDARESA SUBRAMANIAN G, Practice Engagement Manager, Infosys 

---

  Page: 2 / 8  

---

 External Document © 2018 Infosys Limited Abstract Over the years, organizations have invested significantly  in optimizing their testing processes to ensure continuous  releases of high-quality software. When it comes to artificial  intelligence, however, testing is more challenging owing to  the complexity of AI. Thus, organizations need a different  approach to test their AI frameworks and systems to ensure  that these meet the desired goals. This paper examines  some key failure points in AI frameworks. It also outlines  how these failures can be avoided using four main use cases  that are critical to ensuring a well-functioning AI system.<|endoftext|>The hierarchy of evolution of AI Introduction Experts in nearly every field are in a race to discover how to replicate brain functions – wholly or partially. In fact, by 2025, the value of the  artificial intelligence (AI) market will surpass US $100 billion1. For corporate organizations, investments in AI are made with the goal of  amplifying the human potential, improving efficiency and optimizing processes. However, it is important to be aware that AI too is prone to error  owing to its complexity. Let us first understand what makes AI systems different from traditional software systems: S.No Software systems AI systems 1 Features – Software is deterministic, i.e., it is pre-pro- grammed to provide a specific output based on a given set  of inputs Features – Artificial intelligence/machine learning (AI/ML) is  non-deterministic, i.e., the algorithm can behave differently for  different runs 2 Accuracy – Accurate software depends on the skill of the  programmer and is deemed successful if it produces an  output in accordance with its design Accuracy – Accuracy of AI algorithms depends on the training  set and data inputs 3 Programming – All software functions are designed based  on if-then and for loops to convert input data to output  data Programming – Different input and output combinations are fed  to the machine based on which it learns and defines the function 4 Errors – When software encounters an error, remediation  depends on human intelligence or a coded exit function Errors – AI systems have self-healing capabilities whereby they  resume operations after handling exceptions/errors  Fig 1: Hierarchy of AI’s evolution and techniques Fig 1: Hierarchy of AI’s evolution and techniques                                              Visualization  Machine learning and  analytics  Input data conditioning  Learning process – data sources  Feedback  Feedback – From sensors, devices, apps, and systems  Visualization – Custom apps, connected devices, web, and bots  Machine learning and analytics – Cognitive learning/algorithms  Input data conditioning – Big data stores and data lakes  Data sources – Dynamic or static sources like text, image,  speech, sensor, video, and touch 

---

  Page: 3 / 8  

---

 External Document © 2018 Infosys Limited External Document © 2018 Infosys Limited The above figure shows the sequential stages of AI algorithms. While each stage is necessary for successful AI programs, there are some typical  failure points that exist within each stage. These must be carefully identified using the right testing technique as shown in the table below: S.No Evolution stage in AI Typical failure points How can they be detected in testing 1 Data sources –  Dynamic or static  sources • Issues of correctness, completeness and appropriateness  of source data quality and formatting • Variety and velocity of dynamic data resulting in errors • Heterogeneous data sources  • Automated data quality checks  • Ability to handle heterogeneous data during  comparison  • Data transformation testing  • Sampling and aggregate strategies 2 Input data condition- ing – Big data stores  and data lakes • Incorrect data load rules and data duplicates • Data nodes partition failure • Truncated data and data drops  • Data ingestion testing  • Knowledge of development model and  codes  • Understanding data needed for testing  • Ability to subset and create test data sets 3 ML and analytics –  Cognitive learning/ algorithms • Determining how data is split for training and testing • Out-of-sample errors like new behavior in previously  unseen data sets • Failure to understand data relationships between enti- ties and tables • Algorithm testing  • System testing  • Regression testing  4 Visualization – Cus- tom apps, connected  devices, web, and  bots • Incorrectly coded rules in custom applications resulting  in data issues • 
Formatting and data reconciliation issues between  reports and the back-end • Communication failure in middleware systems/APIs  resulting in disconnected data communication and  visualization • API testing  • End-to-end functional testing and automa- tion  • Testing of analytical models  • Reconciliation with development models 5 Feedback – From sen- sors, devices, apps,  and systems • Incorrectly coded rules in custom applications resulting  in data issues • Propagation of false positives at the feedback stage  resulting in incorrect predictions • Optical character recognition (OCR) testing  • Speech, image and natural language pro- cessing (NLP) testing  • RPA testing  • Chatbot testing frameworks The right testing strategy for  AI systems Given the fact that there are several failure  points, the test strategy for any AI system  must be carefully structured to mitigate  risk of failure. To begin with, organizations  must first understand the various stages  in an AI framework as shown in Fig 1. With  this understanding, they will be able to  define a comprehensive test strategy with  specific testing techniques across the entire  framework. Here are four key AI use cases  that must be tested to ensure proper AI  system functioning: • Testing standalone cognitive features  such as natural language processing  (NLP), speech recognition, image  recognition, and optical character  recognition (OCR) • Testing AI platforms such as IBM  Watson, Infosys NIA, Azure Machine  Learning Studio, Microsoft Oxford,  and Google DeepMind • Testing ML-based analytical models • Testing AI-powered solutions such as  virtual assistants and robotic process  automation (RPA) 

---

  Page: 4 / 8  

---

 External Document © 2018 Infosys Limited Use case 1: Testing standalone cognitive features Natural language processing  (NLP) • Test for ‘precision’ return of the keyword, i.e., a fraction of relevant instances among the total retrieved  instances of NLP • Test for ‘recall’, i.e., a fraction of retrieved instances over total number of retrieved instances available • Test for true positives (TPs), true negatives (TNs), false positives (FPs), and false negatives (FNs). Ensure  that FPs and FNs are within the defined error/fallout range Speech recognition inputs • Conduct basic testing of the speech recognition software to see if the system recognizes speech inputs  • Test for pattern recognition to determine if the system can identify when a unique phrase is repeated  several times in a known accent and whether it can identify the same phrase when it is repeated in a  different accent • Test deep learning, the ability to differentiate between ‘New York’ and ‘Newark’ • Test how speech translates to response. For example, a query of “Find me a place I can drink coffee”  should not generate a response with coffee shops and driving directions. Instead, it should point to a  public place or park where one can enjoy his/her coffee Image recognition • Test the image recognition algorithm through basic forms and features • Test supervised learning by distorting or blurring the image to determine the extent of recognition by  the algorithm • Test pattern recognition by replacing cartoons with the real image like showing a real dog instead of a  cartoon dog • Test deep learning using scenarios to see if the system can find a portion of an object in a larger image  canvas and complete a specific action Optical character recognition • Test OCR and optical word recognition (OWR) basics by using character or word inputs for the system to  recognize • Test supervised learning to see if the system can recognize characters or words from printed, written or  cursive scripts • Test deep learning, i.e., whether the system can recognize characters or words from skewed, speckled  or binarized (when color is converted to grayscale) documents • Test constrained outputs by introducing a new word in a document that already has a defined lexicon  with permitted words  Use case 2: Testing AI platforms Testing any platform that hosts an AI