18F – When should an advising AI say “I don’t know”? Lessons From 13,000 exchange reports
Abstract:
Every international office sits on years of student exchange reports that are rarely read again. At Yonsei University, we
turned 13,323 exchange program experience reports, accumulated over 28 years, into an AI assistant that helps
outbound students compare destinations and make better-informed decisions. The central challenge was not building the
system but trusting it: an advising tool that invents plausible answers can quietly mislead students. We therefore
designed the assistant to recognize the limits of its evidence and to respond "no relevant information was found" rather
than fabricate guidance. Drawing on a 130-question evaluation and a live student beta, this session shares three findings
that matter for practice. First, an AI that openly admits uncertainty earns more student trust than one that always
answers. Second, higher technical search accuracy did not translate into students finding the answers more useful - a
caution for anyone procuring AI tools on benchmark scores alone. Third, such systems support rather than replace
human advising. The session offers international educators a grounded, replicable view of where AI advising can be
trusted, where it cannot, and what "humanity in the AI age" means for student-facing services.
Learning objectives:
- Why "abstention" - an AI's ability to say it does not know - is a core trustworthiness requirement for student-facing
advising tools, not an optional extra.
- How to critically assess vendor claims about AI advising systems, and why strong retrieval or accuracy
benchmarks do not guarantee real usefulness to students.
- The practical steps and safeguards for turning an institution's dormant student reports into a searchable,
trustworthy resource, including privacy handling and evidence-based answering.
- How to articulate a balanced position on AI in student advising that keeps human judgement central.
Target audience:
International office staff and leaders in outbound mobility and student advising; professionals evaluating or procuring AI
tools for student services; and international education researchers interested in trustworthy AI. No technical background
is required.