AI and assessment: The product perspective

Neil McGough, Chief Product Officer, offers his thoughts on the importance of user feedback in creating an AI-enabled product experience that offers real value.

To start off, can you tell me a little about how you initially appraised AI, and what was it that made you think “This changes everything—this is now a priority area for us?”

I think the key thing to differentiate here is the hype around AI as a whole, and the hype around commoditized AI. 

AI and machine learning in general is a fascinating field with wide-ranging impacts and applications, which until late last year required vast levels of funding and seriously specialist knowledge to properly avail of. This pushed the development path towards massive corporations, or key VC-funded smaller businesses racing to catch a unicorn they could monetize. 

The “Singularity” point to me is that transition from that state to commoditized AI—the point where you can rely on specialist vendors to provide the knowledge and functionality to make AI accessible, and we get to focus on the experience of AI throughout our products. This is the “this changes everything” point for me, and it’s a path that we’re well aware of at Learnosity.

A large part of our value proposition for customers is that we are the specialists in the technology and user experience aspects of assessment creation, delivery, and everything that goes along with that, and we allow customers to focus on what’s truly important for them: the overall holistic experience for their users.

How long have we been seriously exploring the technology? What are the areas that are most uncertain and most exciting?

Realistically, AI in the edtech space has been simmering around the fringes for a long time, and we’ve investigated a number of routes with partnerships with specialist companies in the past to find opportunities to bring value to our customer base. That said, there’s a real element of the “before” and “now” thanks to the commoditization and APIfication of AI. We’ve gone from an era when Machine Learning and AI was a skillset and tech stack requiring huge expenditure to a world where the technology matters less, and the integrated experience matters more.

A lot of the most uncertain areas are still fully emergent—the role AI will play going forward in actual test taking, and how to sustainably and safely bring value via AI into the whole test-taking experience, including for the learner, is really still a massive opportunity to be explored—with substantial risk if not done carefully.

It’s a relatively short time to learn about tech with such broad scope for potential use. How did you decide what to focus on initially, and where do you see the greatest value-add over both the short- and longer terms?

Our immediate focus is in the area of authoring—primarily author assistive AI. There are lots of reasons for us to focus on this, but the most compelling for us was that:

1. we have vast experience, expertise, and high-quality tooling in the area and we know that AI can add a lot more value to it;
2. it’s a clearly defined area of functionality, which makes it easy for us to get feedback and make improvements;
3. we want to focus on areas where the value: cost ratio is high.

Right now, the cost basis for auto-grading every essay is huge, whereas focusing on changes to how people create content can have a big impact, with lower cost.

A lot of AI tech is open source, making it available at scale. How do we differentiate our work with it from what others are doing in the market? How have you found a balance between speed to market and product quality?

I touched on the “before” and “now” aspect of commoditized AI previously, but it’s worth noting that what we’re talking about now in terms of the rapid expansion of AI isn’t necessarily the AI and Machine Learning technology itself, but more so the integrated experience of AI naturally permeating your product.

In some ways, the same rules apply: you want to bring value to your end-users, your authors, your test takers, your stakeholders, throughout your application. The difference is you now have an additional “actor” you can rely on throughout this—outside of those people and your staff—one whose generative abilities, when specifically guided and shaped through subject matter expertise, can be far faster and more accurate than any other actor in your system. That subject matter expertise, both in terms of building good product experiences and in building great assessment experiences and content, is what differentiates us.

The goalposts are always moving in tech, but AI marks a whole new era of accelerated innovation. How do we manage uncertainty or risk in the face of such rapid change?

Trust in the quality and efficacy of AI is still in its infancy, and that’s something that needs to be built up with end-users before allowing AI to do larger tasks—in our case, that’s anything directly test-taker facing. Placing AI in an assistive role helps us ensure efficacy and build up that trust.

Speaking more generally, AI and the use cases that genuinely change how we act, think and do our day-to-day are still emerging. It’s a massively changing landscape every day, and having the ability to judge the opportunity and risk, and to adapt to the uncertain will be key indicators to success. There are studies that prove this in relation to businesses effectively capitalizing on the expansion of the EU in 2004—those who were unsure as to whether this would be positive or negative for their business were the ones who subsequently adapted most effectively and capitalized on change. With that in mind, we’re positioning ourselves to accept that nothing is certain in the future of AI, and adaptation is key.

When do you know the product is ready to put in front of customers?

Realistically, I think the question is when do you know a product is fit for the market? I believe it’s when you’re clear on the value it brings to end-users and stakeholders, and you can see that what you’ve brought to market is fulfilling that promise. 

I think the really exciting thing for us, from a product point of view, is that this is a real opportunity to bring our customers into the product experience and journey, earlier than ever before, and utilize direct feedback to shape our AI product vision. We have some clear directions we want to target that we believe will add value, but the emergent nature of AI use cases in the education space means there’s huge potential, but we need to be agile and responsive to realize it.

To receive regular updates on Learnosity’s work in AI, sign up here.

Or to learn more about AI’s impact on assessment, download your copy of our AI guidebook. 👇