NSIA Scholarship in Action: Faculty–Alumni Research Supported by ODNI

Dr. John Borek is a faculty lecturer in the National Security Intelligence Analysis (NSIA) master’s program, and Seth Marcotte is a recent NSIA graduate. Together, they were published in the International Journal of Intelligence and CounterIntelligence. 

Supported by the Office of the Director of National Intelligence (ODNI), “An Evaluation of Large Language Models Using AnalyticTradecraft Standards,” by John Borek and Seth Marcotte reflects the strength of the National Security Intelligence Analysis program at the University of New Hampshire. It also demonstrates the sustained connection between faculty, students, and graduates. 

In the journal article, the authors evaluate the output of Google’s Gemini and Microsoft’s Copilot against ODNI analytic tradecraft standards the ODNI Rating Scale for Evaluating Analytic Tradecraft Standards, and compare those results to human analysts. 

Their evaluation found large language models struggled with tasks requiring disaggregation of information, particularly describing source quality and credibility, expressing uncertainties, and distinguishing between facts, assumptions, and judgments. The models achieved highest ratings on communication-related tasks including analysis of alternatives, relevance and implications, and clear and logical argumentation. 

Learn more about the NSIA program, its faculty, and research: