Reviewing in Agile Learning Design

The last concept of PODSIR (Plan, Orientate, Design, Select, Iterate, Review) is for the team to review the learning project (prototype or initial iteration) to see if it performs as intended (shown in the blue cycle below).

The first step is to trial it by using representative samples of the target population (the learners). Next, it is evaluated to ensure the goal and objectives that were made in the Planning stage have been met. And lastly, determine what the goals and objectives of the next iteration cycle will consist of (the orange cycle shown below).

The Flow of Agile Learning Design

Trialing

The first part of the Review is to test the training or learning process by performing either a Design Iteration or a Release Iteration (Moggridge, 2007).

Testing a small population of the learners (target population) to perform the learning process or training course is called a Design Iteration— the iteration is performed to test a learning method, function, feature, etc. of the learning platform to a small set of learners to see if it valid.

If you believe the prototype is fully ready or is useful to the learners, but perhaps not fully complete, then you can perform a Release Iteration — the iteration is released as a product to the business unit or customer. Although it may not be fully completed or functional, the designers and customer believe that it is good enough to help the learners to perform better.

For more information, see Validating Instructional Design.

Evaluation

This part of the Review evaluates the results to ensure the goal and objectives that were made in the Planning stage have been met. Evaluation may be defined as the process of gathering information in order to make good decisions. It is broader than testing, and includes both subjective input (opinion) and objective input (fact).

There are at least six major reasons for evaluation, each requiring a different type of evaluation:

In addition, there are five main purposes of evaluation (Bramley, Newby, 1984):

The evaluation determines if your project is complete and may be ended (except for normal maintenance and upgrades), or if the project needs further iterations. If you determine the project is complete, then it normally requires that your customer signs off on it. If it needs another prototype, then perform an iteration.

For information of the various levels of evaluation, see Kirkpatrick's Four Level Evaluation Model.

Plan the Next Iteration

The results of your evaluation determine how much planning is needed in this step. If your evaluation completely failed, then you might need to perform a full Planning and/or Orientation phase again. If everything is going as planned, then you can move on to the Design stage in the next iteration as shown in the flow chart at the top of the page (the orange cycle). If it is somewhere in the middle of the two, then you need to make some adjustments or changes. The chart below can be used as a troubling-shooting heuristic to help pinpoint your problem(s):

Agile trouble-shooting heuristic

Determining if a level of evaluation (right column) was met will tell you the next level to check:

Next Steps

References

Bramley, P., Newby, A.C. (1984). The Evaluation Of Training Part I: Clarifying The Concept. Journal of European & Industrial Training, 8, 6, 10-16.

Moggridge, B. (2007). Designing Interactions. Cambridge, Massachusetts: The MIT Press.