In April 2024, the National Institute of Standards and Technology published a Draft publication The goal is to provide guidance on safe software development practices for generative AI systems. Given these requirements, software development teams should start implementing a robust testing strategy to ensure they adhere to these new guidelines.
Testing is a cornerstone of AI-driven development as it validates the integrity, reliability and soundness of AI-based tools. It also protects against security risks and ensures high quality and optimal performance.
Testing is especially important in AI because the system under test is far less transparent than a coded or engineered algorithm. AI exhibits new failure modes and types of errors, such as tone of voice, implicit bias, inaccurate or misleading responses, regulatory errors, and more. Even after development is complete, development teams may not be able to reliably assess the reliability of the system under different conditions. Because of this uncertainty, quality assurance (QA) professionals must step in and become true quality champions. This designation means not only adhering to a strict set of requirements, but also investigating to identify edge cases, engaging in red teaming to try to force the app to provide inappropriate responses, and uncovering undiscovered biases and failure modes in the system. Thorough and curious testing is the guarantor of well-implemented AI initiatives.
Some AI providers, such as Microsoftrequire test reports to provide legal protection against copyright infringement. The regulation of safe and reliable AI uses these reports as core components, and they often appear in both the Executive Order of October 2023 by US President Joe Biden on safe and trustworthy AI and the EU AI lawThoroughly testing AI systems is no longer just a recommendation to ensure a smooth and consistent user experience, but a responsibility.
What makes a good testing strategy?
There are several key elements that should be included in any testing strategy:
Risk assessment – Software development teams must first assess all potential risks associated with their AI system, taking into account how users will interact with a system’s functionality, as well as the severity and likelihood of errors. AI brings a new set of risks that must be addressed. These risks include legal risks (agents making incorrect recommendations on behalf of the business), complex quality risks (dealing with nondeterministic systems, implicit bias, pseudorandom outcomes, etc.), performance risks (AI is computationally intensive and cloud AI endpoints have limitations), operational and cost risks (measuring the cost of running your AI system), novel security risks (prompt hijacking, context extraction, prompt injection, adversarial data attacks), and reputational risks.
An understanding of the limits – AI is only as good as the information it receives. Software development teams need to be aware of the limits of its ability to learn and novel failure modes unique to their AI, such as lack of logical reasoning, hallucinations, and problems with information synthesis.
education and training – As AI usage increases, it is important to ensure teams are educated on its intricacies – including training methods, data science fundamentals, generative AI, and classical AI. This is critical to identifying potential problems, understanding the system, and getting the most out of AI.
Red team testing – Red team AI testing (Red Teaming) provides a structured approach to identifying vulnerabilities and deficiencies in an AI system. This type of testing often simulates real-world attacks and tests techniques that persistent threat actors might use to uncover specific vulnerabilities and identify priorities for risk mitigation. This targeted testing of an AI model is critical to test the limits of its capabilities and ensure that an AI system is secure and ready to anticipate real-world scenarios. Red teaming reports are also becoming a mandatory standard for customers, similar to SOC 2 for AI.
Continuous reviews – AI systems are evolving and so should testing strategies. Organizations need to regularly review and update their testing approaches to adapt to new developments and requirements in AI technology as well as emerging threats.
Documentation and Compliance – Software development teams must ensure that all testing procedures and results are well documented for compliance and audit purposes, for example to adapt to the requirements of the new Executive Order.
Transparency and communication – It is important to provide transparency to stakeholders and users about the capabilities, reliability and limitations of AI.
While these considerations are critical to developing robust AI testing strategies that meet evolving regulatory standards, it is important to remember that as AI technology evolves, our testing and quality assurance approaches must also evolve.
Improved testing, improved AI
AI will get bigger, better, and more widely used in software development in the coming years, so more rigorous testing will be needed to address the changing risks and challenges that come with more advanced systems and data sets. Testing will continue to serve as an important safety measure to ensure that AI tools are reliable, accurate, and responsible for public use.
Software development teams must develop robust testing strategies that not only meet regulatory standards but also ensure that AI technologies are responsible, trustworthy and accessible.
With AI’s increasing use across industries and technologies and its leading role in relevant federal standards and policies in the U.S. and globally, now is the time to develop transformative software solutions. The developer community should see itself as a key player in this effort by developing efficient testing strategies and providing a secure user experience based on trust and reliability.
You might also like…
The impact of AI regulation on research and development
EU adopts AI law, a comprehensive risk-based approach to AI regulation
