High School Students: the New Document Review Professionals?

On his blog, IREvalEtAl, information scientist William Webber reported in a post about a recent experiment he conducted. His results were surprising.
Webber had two teams of document reviewers. One was team from a vendor of professional legal review services. The vendor they worked for had engaged in Topic 204 of the Interactive tasks in the 2009 TREC Legal Track experiments.
The other team was a pair of young high school kids named Marjorie and Bryan.
Webber describes the pair of youths as having, “no legal training, and no prior e-discovery experience, aside from assessing a few dozen documents for a different TREC topic as part of a trial experiment. They performed assessments on .tiff files, displayed on rather underpowered laptops that couldn’t fir the full .tiff onto the screen and tended to crash every now and then. They worked independently and without supervision or correction, though one would be correct to describe them as careful and motivated.”
The Document Review work done by the students was then compared to the work done by the professionals.
Webber reported, “The conclusion that can be reached, though, is that our assessors were able to achieve reliability (with or without detailed assessment guidelines) that is competitive with that of the professional reviewers—and also competitive with that of a commercial e-discovery vendor.”
Well, how do you like that? High school kids are doing a better job at Document Review than a team of people who were professionally trained to do it. That’s ironic, isn’t it? But is it true?
Webber continued his post by referencing a handful of other studies that were in lock step with his own.
“All of this raises the questions that is posed in the subject of this post: if (some) high school students are as reliable as (some) legally-trained, professional e-discovery reviewers, then is legal training a practical (as opposed to legal) requirement for reliable first-pass review for responsiveness? Or are care and general reading skill the more important factors?”
Mr. Webber brings up a good point. While it is probably a good thing for people doing Document Review to have an education in the law, is it really necessary? Particularly when another person, who may not meet the same requirements, is capable of performing the job? Perhaps, even doing a better job than the “professional”?
Webber finished by saying, “Our finding might suggest that first-passed document review is not practice of the law (since it apparently does not employ legal skills), and therefore that using non-lawyers to perform it should not be considered unauthorized practice of the law, but I’ll off a bottle of wine to the lawyer brave enough to argue that in court on the basis of this blog post.”
While arguing cases is certainly best left to the professionals, it does seem a little senseless that first-pass legal reviewers have to hold higher degrees. However, with all of these studies surfacing suggesting that kids can do their jobs better than they can, perhaps all of the professionally trained reviewers will be motivated enough to step up their performance. Or, maybe in time the paralegal will make reappearance, and people with no college degrees will manage to have equal or higher paying jobs than someone with a college degree.