Use of Test Results by Teachers, Parents and Students
Dangers in Test Score Standards
What Should be Iowa City’s Concerns?
What Results Can be Expected from Test-Based Academic Standards?
What Would be Reasonable Academic Ends for the ICCSD?
What is the Most Important Thing to Measure?
How Can Test Results Contribute to Board Governance?
What Other “Ends” or Standards Should the Board Consider?
Dr. H. D. Hoover is a statistician who is Director of the Iowa Statewide Basic Skills Testing Program, and heads the ITBS program at the University of Iowa where he has been a professor for 33 years. He is widely regarded as one of the top experts on measurement in the United States, if not the world. (In this connection it should be noted that, whether or not reflected here, he was always careful to distinguish those matters as to which he had data and those which represented his opinion.)
He was invited to speak to the Iowa City Community School District’s televised Board meeting Tuesday evening, March 21, 2000. He spoke for approximately 30 minutes, following which he answered Board members’ questions for an hour.
The Board’s primary reason for consulting with Dr. Hoover is a consequence of its pursuit of the approach to governance advocated by John Carver in his books Boards That Make a Difference and Reinventing Your Board. To oversimplify, the Carver approach requires that a board establish “ends,” or goals, for the organization along with the standards for their measurement. The CEO -- for a school district that’s the superintendent -- is then given the task of reaching those “ends” without violating any of the “executive limitations” the board puts on the CEO’s means.
Central to any school district’s “ends,” presumably, would be something having to do with its students’ academic achievement. Determining what it is about academics that should be a part of “ends,” let alone what standards should be used for their measurement, is not an easy undertaking. As he puts it, “Measurement problems are tougher than the dickens.”
Students’ standardized test scores are used to some extent by virtually everyone for this task. But there are other available measures, and risks to test scores.
It was thought that Dr. Hoover would provide the Board some essential insights into its use of test results in the fashioning of academic ends for the District. He did.
Although he has devoted his professional lifetime to one of the most widely used tests throughout the world, he was very candid in noting:
The primary focus of the Board is on the use of test results as a part of its establishment, and measurement, of District goals or ends. But much of Dr. Hoover’s presentation emphasized the importance of test results for individual teachers, parents and students.
This is another, and equally important, subject in its own right, and one to which the Board may return.
He said, for example, “Iowa City does the worst job of any district I nearly know of reporting ITBS and ITED information to parents. I’ll just bluntly say that, OK?”
“Whoever said earlier that the biggest issue here, one of these huge issues, is parental involvement, I totally believe that this is the very reason obviously that the biggest plea I make is that Iowa City do a better job of giving not just ITBS information, but all kinds of information, to parents about their kids strengths and weaknesses. Those are the people who are most apt to raise the average.”
But parental involvement, and parents’ access to ITBS scores, is not the primary subject of this summary.
Indeed, on a number of occasions various Board members asked if he was making the point that test results are of use only in this way, and that they are not an appropriate measure for goals or standards. In each instance he insisted that was not his intended message. (“I don’t want to discourage people from setting up goals.”) He said, “I believe schools should have accountability programs.” Moreover, ITBS scores are an appropriate measure to include. (“I have no problem with our tests being one accountability measure, in fact I think they should be part of an accountability program.”)
(1) His primary concern about the use of test scores in setting district academic standards and goals is what he calls “high stakes tests” for which the purposes are more political than pedagogical. Although he did not refer to the following specific examples, the high stakes in some cities and states have been everything from bonuses for teachers or schools to the firing of superintendents or city councils taking over from a school board.
This can lead to undesirable “teaching to the test” (some forms of teaching to the test he thinks quite useful), “cooking the data,” and in some reported instances (for example, Texas and New York City) teachers actually changing students’ test answers.
(2) He is concerned about unrealistically high goals “we’re nearly sure to fail at,” favoring rather “reasonable ones in which we can turn these kinds of trends [Iowa City’s declining scores] around.”
(3) He acknowledges, “Tests like ours are far from perfect measures. But I think what you’ll find out is occasionally it is a bad measure, but most of the time it isn’t. Most of the time it is, in fact, giving you information that, compared to other kids, this kid's reading way below average.” “What we do is give you information, and obviously I think it is decent.”
(4) There is a “problem with the focus on averages. Nearly every teacher in the Iowa City schools has the same basic task in front of them. Nearly all classes in Iowa City have some kids who on the ITBS are in the 90s, and some kids are in single digits. While we might see quite large differences in averages, if you go in and look at the classrooms it’s just sort of the way the kids are distributed in there. The [classrooms] that are scoring high have more kids in the 90s, and fewer kids down here in these lower areas. And the ones that have the lower average percentile ranks, they have kids in the 90s. But they’ll have more kids in the lower.”
He showed some charts demonstrating “declining achievement” in Iowa generally and noted that “the same thing is happening in Iowa City.” It is occurring “for lots of reasons.”
“Test scores in Iowa City are going down. We have many more kids in our District now that are harder to educate than they used to be. The percentages of kids that were, quote, ‘proficient’ have been going down slightly. Now what that means is you’re making below average growth.”
“Grade by grade you’re having declining scores.’
“Iowa City shows more than average growth from year to year actually, slightly. But part of the reason it does is that Iowa City’s scores in grade three are really very low. And one of the ways you can show more than average growth is to start low. Whether the Iowa City scores at grade three should be below the state average – this is a matter of record, I’m not saying something that wasn’t in the paper – whether they should be or not is I think a serious question I’d ask, OK?”
“I truly don’t believe that this is just due to a change in student population. I think to some extent it’s instruction, less instruction in the early grades. Less attention to the specific teaching of reading, language, math skills. This is an opinion. But I talk to lots of people and I visit lots of schools. Some of that is an effect that is fundamentally an instruction effect. But some of it is the fact that these populations of students we are getting year to year are changing and are less able than the ones we’ve had in the past. Less well prepared to learn. But all I am saying is I don’t believe that these declines – that those [changes in demographics] explain all of this. I think there are some real effects [from instruction].”
In this connection he noted, “You can measure kids in grade one.” To which a board member asked, “Why do we start in the third grade in this District?” He responded simply, “Because you choose to.”
There are areas of academic performance in which a district “can change some things fairly dramatically in short periods of time,” such as multiplication tables, punctuation, and other “highly specific very simple skills.”
More complex skills, such as “reading comprehension” will not show dramatic, district-wide change.
However, “If good instruction is going on it will be reflected in the test scores. If good instruction in reading is going on reading comprehension scores will go up. And schools that have programs that don’t do a good job of teaching reading comprehension will tend to score lower than ones that do.”
And even with reading comprehension, he noted in answer to a question, “I think you can have pretty dramatic changes at the building level if you throw enough resources in there, and have enough individualized instruction. Oh, I think you can have a big impact. So I’m not saying that’s not the case.”
“We might want Iowa City to have a higher proportion of students make what we would view as normal, or average, or a year’s growth than kids in the rest of the country. These are perfectly good goals. In fact, I think that in a place like Iowa City, with the kinds of kids here, I think expecting that the average growth in Iowa City would be more than a year is fine. I think that is a reasonable goal.”
One board member asked, “What would be a good measure of increasing achievement?” Dr. Hoover answered, “ITBS is a pretty good measure of ‘increasing achievement.’ And I think this is a very good goal. I think Iowa City should have the goal to increase, on the average, achievement at every grade level in this district -- to have it where growth of kids in the district from grade to grade is above average, because there are things about Iowa City as a community that I think everyone here would say, ‘Gee, we should expect this in Iowa City.’”
He recognizes the limitations of ITBS, the availability of other measures, and never advocated reliance solely on ITBS results, but noted that, “Most of the others have much bigger problems than we do.”
“ITBS is one of the things that you really do want to look at.”
“I don’t think these changes [in demographics, the increase in numbers of low income children] are changing dramatically enough that a community like Iowa City still should not be making more than average growth every year. I think that is a reasonable goal. I don’t know any other way to get at it. That’s a very reasonable goal here I think.”
“There are reasonable quantifiable results. You can set expectations. It’s perfectly reasonable to expect, as I said, more than average growth -- which in fact tends to occur in Iowa City. It’s also reasonable to say, and it’s not been occurring in Iowa City, and I recognize Iowa City’s student population is changing. Though these changes are very minor from year to year in terms of the composition of the student body. I still don’t think there’s any reason why Iowa City’s average performance over the last five or six years, that this year’s third grade, on the average, is tending to go down. In a community like Iowa City it is a perfectly reasonable goal to say that each year our third graders will do a little bit better than they did the year before. And that’s a goal that I absolutely know you’re not meeting right now.”
“Iowa City does have a fairly large proportion of incredibly bright kids. It’s the most highly educated place like this in the country. It’s only normal that you show above average growth. I’d be shocked if that wasn’t the case. But also I don’t see any reason for there not to be a goal that would say that every cohort of kids who comes through here will be as good or better than the cohort that they follow. That seems reasonable to me. That’s not going to be trivial. But I think it’s reasonable and reachable.”
There should also be goals for the students who do not perform well. “The goal would be that these kids, from these kinds of [low income] backgrounds, that the achievement for them will keep going up. I think that is a wonderful goal.”
In this connection, he was asked if he agreed that a significant proportion of our “low income” students have parents who are graduate students, or otherwise do not suffer from the disadvantages (and therefore do not offer the District the “excuse”) of children with parents who are not only poor but also lack college education. He responded, “Sure. I agree with that.”
“When it comes to measures of achievement of kids that are of the kind, ‘How well does the kid read?’ you ultimately fall back on something like what we do, or someone else does.”
“Reading comprehension I believe is the single most important thing we measure.”
“On the whole, if Iowa City – I can take the profile for Iowa City and look at relative strengths and weaknesses just like I can for individual kids. And so if I see that in Iowa City there’s an area that’s down there, I can see relative strengths and weaknesses pretty clearly and they’re not random. These are real effects.”
“If you are going to have a low area then I think you have to make a conscious decision to say, ‘Here’s why we’re willing to have this area this low because we have so much time, and we’re allocating our time this way, and we think it’s better spent over here.’ Because all the things we measure on the ITBS aren’t of equal importance.”
There are “others I don’t feel this strongly about, but they give feedback about specific areas. I think the District looks at those and says, ‘Gee, you know maybe we ought to, for an area like, for example, capitalization.’ To be perfectly frank you could throw in an extra 30 minutes a week, if that, maybe two or three weeks, and probably have some pretty dramatic effects on an area like that. Because it is something that is quite easy to teach.”
“School districts everywhere have had trouble coming to grips – and I’m not an expert, I build tests – with various ways of reporting to the public about all of these kind of things.”
He was asked, “Is attendance in high school a number that we ought to be looking at?”
He responded, “Sure. In fact, I do think that those are very measurable things. And they are things that we can very simply say, ‘Gee, we’d like to have the percentage of kids who are attending school every day go up.’ And I think you should say that. And the only way you’re going to know if that’s happening is to keep track of it.”
“Attendance, graduation rates, kids in debate and music and all those kind of things should be more systematically reported.”
Also numbers in advance placement courses and “how well kids do on college entrance exams, all the kinds of accomplishments of schools at the elementary and secondary level that I think we all know about, but I just don’t think it’s done very systematically.”
“I thought it was imperative
that these measures of the sort of socio-economic level of the community,
that Iowa City needed to start gathering all kinds of information like
that a long time ago. So it’s not just reporting on achievement.”