In this paper, we describe the integration and evaluation of an existing generic Bayesian student model
(GBSM) into an existing computerized testing system within the Mathematics Education Project (PmatE -
Projecto Matemática Ensino) of the University of Aveiro. This generic Bayesian student model had been
previously evaluated with simulated students, but a real application was still missing. In the work
presented here, we have used the GBSM to define Bayesian Student Models (BSMs) for a concrete
domain: first degree equations. In order to test the diagnosis capabilities of such BSMs, an evaluation
with 152 students has been performed. Each of the 152 students took both a computerized test within
PMatE and a written exam, both of them designed to measure students’ knowledge in 12 concepts
related to first degree equations. The written exam was graded by three experts. Then two BSMs were
developed, one for the computer test and another one for the written exam. These BSMs were used to
obtain estimations of student’s knowledge on the same 12 concepts, and the inter-rater agreement
among the different measures was computed. Results show a high degree of agreement among the scores
given by the experts and also among the diagnosis provided by the BSM in the written exam and expert’s
average, but a low degree of agreement among the diagnosis provided by the BSM in the computer test
and expert’s average