Today ACARA released its National Assessment and Surveys Online Program: tailored test design 2013 study. The study explores how online testing can be structured in ways to make it even more accurate than the existing pencil and paper tests.
ACARA has been exploring how NAPLAN can be moved online since 2012. The initial research focused on ways to ensure that students performed just as well on online testing as they did with traditional paper tests. This new part of the research focused on making NAPLAN more reliable through tailored (branched) testing. It found that NAPLAN online will be even more accurate.
ACARA is planning to move NAPLAN online from 2017.
NAPLAN Is Already Accurate
Despite popular claims to the contrary, NAPLAN is a very reliable and valid form of assessment. It is far more reliable and valid than most teacher-designed assessments and commercial programs such as the DRA2.
There have been some disparaging figures shared by some anti-NAPLAN campaigners, but after personally requesting and obtaining a full set of data on NAPLAN’s reliability, I have more trust in the accuracy of a single NAPLAN test than I do in any other single assessment piece I have come across. Unfortunately, I had to agree not to share this data in order to obtain it. However, if you interested it has been made available under FOI – simply request it from ACARA yourself.
One of the reasons why anti-NAPLAN activists can get away with publishing misleading figures relates to the very structure of a developmental test.
The Structure of Developmental Tests
Unlike traditional tests, developmental tests show where a student is at beyond the content of their current year level. To this end, a Yr 3 test may include questions from Yr 1, Yr 2, Yr 3, Yr 4 and Yr 5.
However, there will be more questions at a Yr 3 level than their will be at other year levels. For example, a Yr 3 test may contain:
- 2 question at a Yr 1 level
- 4 questions at a Yr 2 level
- 8 questions at a Yr 3 level
- 4 questions at a Yr 4 level
- 2 question at a Yr 5 level
This is a simplified example. In the real NAPLAN test, there are more levels and more questions. However, the example should help you to understand how NAPLAN tests are structured.
There is no traditional pass or fail in this test. Rather you are assigned a level based on the questions you answered correctly.
Middle range levels are more reliable than levels at the extreme bottom or top, simply because the student has had to answer more questions at that level.
In the above example, a child who scored 1 would be placed in level 1, while a child who scored 20 would be placed in level 5. However, the child placed in level 5 has only answered 2 questions at that level.
It is feasible that the child answered 2 questions at level 5 by luck. It is also feasible the child could have answered level 6 or level 7 questions if they were given the chance. Therefore, it is quite accurate to say they are well above year level, but not so accurate to say they are at level 5.
The same holds true for students achieving levels at the bottom of the scale.
The results of students towards the middle of the scale are more accurate because they are based on a larger pool of questions.
NAPLAN Online Will Be Even More Accurate
When NAPLAN online is introduced, students scoring at either extreme will be given further questions centred on their predicted score. At the same time, questions are cut from the other extreme so that the test doesn’t become too long.
This is known as a tailored test (or branching). This allows those students to receive a score that is just as reliable as those students scoring closer to the middle range.
Some have argued that the idea is to make the test easier for struggling students, so that more kids pass. This is simply untrue. The tests just don’t work that way.
NAPLAN online tests will be even more accurate and they will reduce the length of time schools wait to get the results back.