Entering into the world of L&D AI solutions can be a complex and overwhelming task. To help ease the pain and provide some structure, the following questions guide you through an in-depth exploration of the five key dimensions for evaluating AI solutions: relevance, usability, scalability, sustainability, and ethics. By discussing and investigating the questions with your stakeholders, you can facilitate a better selection process and make informed decisions about which AI product to implement.
Relevance
The extent to which the solution meets your needs and requirements and has functionalities that are optimally suited to your organization.
- Does the AI solution align with the specific learning goals and objectives of your organization?
- Does the AI solution support the overarching business goals and objectives of your organization?
- Does the AI solution address the specific skills and knowledge gaps that employees need to bridge?
- Does the AI solution provide real-time feedback and analytics that enable learners and managers to track progress and identify areas for improvement?
- Is there evidence that the AI solution is grounded on evidence-based research and/or a sound pedagogical model? While it is still early for AI solutions to be shown to be effective in the long term, ask the vendor for any case studies, white papers, impact studies, or customer use cases that you can review.
Usability
The extent to which the solution supports the desired goals and needs of the users, is accessible, and provides a positive user experience.
- Is the AI solution user-friendly and intuitive to use? (e.g., you know where to go when you are on the platform, the navigation buttons are clearly labelled, the instructions are clear.)
- Does the AI solution enable quick adoption by employees without extensive training?
- If training is needed, does the vendor provide a variety of training formats (e.g., webinars, job aids, training manuals, FAQs, knowledge database, customized training, etc.)?
- Is the AI solution consistently reliable (e.g., for generative AI products – minimum hallucination, accurate and trustworthy outcomes, short wait time)?
- Is the AI solution accessible on multiple devices, does it have the capability to support multiple languages, and can it accommodate users with disabilities such as visual impairment and dyslexia?
Scalability
The extent to which the solution meets the technical requirements, has the capacity to grow, adapt, and can evolve over time to changing organizational needs.
- Can the AI solution accommodate increasing user demand and expanding learning content?
- Can the AI solution learn over time by incorporating more data sources, both internally and externally?
- Do you have any control over the sourcing, cleansing, and training of data?
- Does the AI solution have the capability to support multiple languages, ensuring inclusivity for a diverse workforce?
- Can the AI solution cater to changing business needs, and the specific needs of different learner groups, such as new hires, remote employees, or teams with varying skill levels?
Sustainability
Can the AI solution cater to changing business needs, and the specific needs of different learner groups, such as new hires, remote employees, or teams with varying skill levels?
- Does the vendor’s leadership team have subject matter domain expertise? How diverse is the team (e.g., age, gender, skillsets, knowledge, etc.)? Do they have AI/data science knowledge as well as learning and development background?
- What measures does the AI solution provider have in place to ensure data security, protect against cyber threats, and comply with relevant data privacy regulations?
- Is the AI solution vendor committed to ongoing technical support, including prompt response times and readily available resources for troubleshooting?
- Is the AI solution vendor committed to ongoing technical support, including prompt response times and readily available resources for troubleshooting?
- Does the vendor have sustainability targets, metrics, and practices that they can share with you? (e.g., what does the vendor do to offset the electricity consumption, carbon footprint and greenhouse gas emissions as a result of training large sets of data and running algorithm?) If not, is the vendor aware of the environmental impact of their solutions, and what have they done to mitigate such impact?
Ethics and Responsible Use of AI
The extent to which the vendor ensures the responsible, inclusive, and equitable development of AI technologies.
- Does the AI solution align with your organization’s ethical guidelines (or external standardized ethical guidelines) and do they comply with relevant laws and regulations such as GDPR in Europe?
- Does the vendor incorporate ethical and responsible design practices? Does the vendor prioritize data privacy, transparency, fairness, and accountability in their product and business? If yes, how do they go about doing that? (e.g., do they have an in-house data ethics policy? Do they communicate this policy to all their prospective clients? Do they adopt privacy-preserving techniques such as differential privacy (it is a way to share useful information from a set of data, without giving away details that could identify any person in that data) or federated learning (a machine learning approach where an algorithm is trained across multiple decentralized devices or servers holding local data samples, without exchanging them, thus preserving data privacy) to safeguard sensitive information? Does the solution seek user consents when collecting, using, and interpreting user data?)
- Does the AI solution use unbiased algorithms and employ techniques that minimize the risk of reinforcing bias or discrimination in the learning process? (e.g., do they conduct thorough testing to identify and address potential biases in training data to avoid discriminatory outcomes?)
- How does the AI solution ensure explainability* of its decision-making processes, providing learners with insights into how recommendations or assessments are generated? (e.g., are users able to review, interpret, provide feedback to the outcomes generated? Are the decisions the AI solution produces explained to users?)
- Does the vendor/solution actively seek feedback from users to improve system performance and improve their ethical practices?
* In AI, “explainability” refers to our ability to understand and explain how an AI system makes its decisions.