Layered evaluation of adaptive learning systems

This paper suggests an alternative to the traditional "as a whole" approach of evaluating adaptive learning systems (ALS), and adaptive systems, in general. We argue that the commonly recognised models of adaptive systems can be used as a basis for a layered evaluation that offers certain benefits to the developers of ALS. Therefore, we propose the layered evaluation framework, where the success of adaptation is addressed at two distinct layers: user modelling and adaptation decision making. We outline how layered evaluation can improve the current evaluation practice of ALS. To build a stronger case for a layered evaluation, we re-visit the evaluation of the InterBook where the layered approach can provide a difference and provide an example of its use in KOD learning system.