"So," I say to myself, "you think you know everything, donít you? What is it that you donít know but would really like to?" I ask myself this important question every year in designing the program evaluation for Harlem RBIís REAL Kids program. REAL (Reading and Enrichment Academy for Learning) Kids is a six-week summer literacy and baseball camp for over 200 youth ages 9 to 12. The multifaceted program is designed to improve youthís literacy, social, and emotional skills, while teaching baseball as well.
Since Harlem RBI has been evaluating this program for several years, we have built up significant evaluation expertise, using a variety of tools to determine the programís impact. Each year, we both tweak the evaluation to learn new things and repeat some measures to test that the program remains effective.
Unfortunately, our level of experience makes deciding what to evaluate harder rather than easier! Because we are heavily invested in the program and find evaluation so valuable, we tend to want to evaluate every aspect of the program in order to learn as much as we can.
Thus, narrowing the evaluation to key questions is the most crucial aspect of the undertaking. I start by brainstorming all the possible evaluation questions with the REAL Kids program director. Together we choose the questions that appear to be the most burning, the ones that will:
Sharing this process with the program director ensures that she has a stake and interest in the evaluation.
Last summer, we decided through this process to focus on how the REAL Kids program affected childrenís skills in responding to literature in collaborative discussion groups (or "book clubs"). While we had been using collaborative discussion groups for years, we had never specifically evaluated their value to the program. We knew they were popular with staff and engaging to students, but we did not know how much children were learning.
Next, we had to decide what evaluation tools to use. Since dialogue and engagement are the chief "products" of book clubs, we decided that the most effective evaluation method would be for teachers to assess whether student conversations showed evidence of growth.
At this point, the buy-in of the teaching staff was crucial. In the first staff development meeting, the program director asked teachers to brainstorm ways in which students could grow in their ability to respond to literature. Next, teachers were asked how they could tell if a student was improving. What would students be doing if they were getting better at specific skills? How often would they do it? What wouldnít they do? How much growth could we expect over the course of the six-week program? Teachers reached consensus on which elements would be the strongest evidence of growth. As part of this process, the program director ensured that all teachers had the same interpretation of each element.
Using this information, the program director designed an observation rubric for teachers to use in the classroom. The rubric measured ten possible elements of studentsí response to literature, such as response to characters, ability to focus on text, and use of reading strategies. For each element, teachers were to choose one of four levels of proficiency for each student. For example, in gauging studentsí use of reading strategies, level 4, the most proficient, indicates that a student rereads what does not make sense and uses context clues to figure out unfamiliar words. Level 1, the least proficient, means that a student gives up reading or plows through without trying to understand.
The program director emphasized to teachers that they themselves had designed the rubricís elements as a genuine assessment of what they felt was important. Thus, teachers understood what they were being asked to measure and felt that what they were measuring was important.
Teachers filled out rubric sheets on each student at the beginning and end of the program. The data were collected, and numbers corresponding to rubric responses were entered into spreadsheets and analyzed. Converting responses into numbers enabled us to quantify the results; we could find the areas in which students grew the most by finding the largest difference in the numbers from the start to the end of the program.
Usually, initial data analysis raises questions program staff ask teachers in order to interpret results. For example, if the data showed particularly strong or weak results in one area, I would ask teachers why they thought that was so. In this instance, the data was pretty straightforward, but we still made sure to share the data with teachers so that they would value their contribution and know that we valued it as well.
The results showed that, by participating in book clubs, youth improved their ability to support their opinions with evidence from the text, draw inferences from texts, and ask questions that evidenced higher-order thinking. In short, the evaluation results showed that book clubs were an important way to build literacy skills. This result not only confirmed our instincts, but also provided us with hard data to build further buy-in for book clubs from all stakeholders, including teachers and parents.