The Transparency Files: CAT4 Results

As apparently is my new annual tradition, I again in the lull between parent-teacher conferences reviewed and analyzed our CAT4 results.  [I strongly encourage you to reread (or read for the first time) our philosophy on test-taking and how we both share the tests with parents and utilize the data in our decision-making.]  We provided our teachers with the data they need to better understand their students and to identify which test results fully resemble their children well enough to simply pass on and which results require contextualization in private conversation.  Those contextualizing conversations took place during conferences and, thus, we should be able to return all results to parents next week.

Before we get to the results, there are a few things worth pointing out:

  • This is now our second year taking this assessment at this time of year. However, we expanded our testing from last year’s Grades 3, 6 & 8 to this year’s Grades 3 – 8.  This means that although we now have “apples to apples” data, we can only track two of our grades (current Grades 4 & 7) from last year to this one.  Next year, we will have such tracking data across most grades which will allow us to see if…
    • The same grade scores as well or better each year.
    • The same class grows at least a year’s worth of growth.
  • The other issue is in the proper understanding of what a “grade equivalent score” really is.

Grade-equivalent scores attempt to show at what grade level and month your child is functioning.  However, grade-equivalent scores are not able to show this.  Let me use an example to illustrate this.  In reading comprehension, your son in Grade 5 scored a 7.3 grade equivalent on his Grade 5 test. The 7 represents the grade level while the 3 represents the month. 7.3 would represent the seventh grade, third month, which is December.  The reason it is the third month is because September is zero, October is one, etc.  It is not true though that your son is functioning at the seventh grade level since he was never tested on seventh grade material.  He was only tested on fifth grade material.  He performed like a seventh grader on fifth grade material.  That’s why the grade-equivalent scores should not be used to decide at what grade level a student is functioning.

We do not believe that standardized test scores represent the only, nor surely the best, evidence for academic success.  Our goal continues to be providing each student with a “floor, but no ceiling” representing each student’s maximum success.  Our best outcome is still producing students who become lifelong learners.

But I also don’t want to undersell the objective evidence that shows that the work we are doing here does in fact lead to tangible success!

That’s the headline…let’s look more closely at the story.  (You may wish to zoom in a bit on whatever device you are reading this on…)

A few tips on how to read this:

  • We took this exam in the “.2” of each grade-level year.  That means that “at grade level” [again, please refer above to a more precise definition of “grade equivalent scores”] for any grade we are looking at would be 3.2, 4.2, 5.2, etc.  For example, if you are looking at Grade 6, anything below 6.2 would constitute “below grade level” and anything above 6.2 would constitute “above grade level.”
  • The maximum score for any grade is “.9” of the next year’s grade.  If, for example, you are looking at Grade 8 and see a score of 9.9, on our forms it actually reads “9.9+” – the maximum score that can be recorded.
  • Because of when we take this test – approximately two months into the school year – it is reasonable to assume a significant responsibility for results is attributable to the prior year’s teachers and experiences.  It is very hard to tease it out exactly, of course.

What are the key takeaways from this snapshot of the entire school?

  • Looking at six different grades through six different dimensions there are only two instances of scoring below grade-level: Grade 3 in Spelling (2.9) and Grade 5 in Computation & Estimation (4.1).
  • Relatedly, those two dimensions  – Spelling and Computation & Estimation – are where we score the lowest as a school (even if every other grade is at or above grade level) relative to the other dimensions.
  • What stands out the most is how exceedingly well each and every grade has done in just about each and every section.  In almost all cases, each and every grade is performing significantly above grade-level.

In addition to the overall snapshot, we are now able to begin sharing comparative data.  It will take one more year before we can accurately compare the same grade and the same class year after year.  But we can get a taste of it with Grades 3 & 6.  What you have below is a snapshot of the same class (the same group of children) from last year to this:

What are the key takeaways from this comparison?

For both classes in all categories save one (Grade 3 to 4 “Computation & Estimation”) you see at least a full year’s growth and in many cases you see more than a full year’s growth.  (The one that fell short only showed 8 months of growth.  And it comes in the category we have already recognized as being a weak spot.)

Let’s look at one more data point.  We can also get a taste of how the same grade performs from one year to the next as well.  Again, we only have Grades 3 & 6 to examine:

Now, remember that this represents a completely different group of children, so it is not unusual or surprising to see variances.  Teachers can only grow students from the place they received them and it is that annual growth that we are concerned with.  But over time you are looking for patterns.  If we believe that Spelling is a weakness, we will want to know whether it is a weakness in every grade or does it dip in certain grades.  We have no way to know that or much else new from the above graph. It simply confirms what we presently know.  But in another year or so, we will be able to plot the trajectory of both classes (the same students) and grades over time to see what additional stories they tell.

To sum up, we have a lot to be proud of in our standardized test scores. We have two areas to investigate: Spelling and Computation.  With regard to Spelling, since we noted this as a weakness last year we had already scheduled PD for our faculty.  It just so happens that we are holding a session on “Structured Word Inquiry” for our Language Arts Teachers on Monday!  With that and other efforts we would expect to see those numbers tick up next year.  With regard to Computation, we will – like with Spelling – have an internal conversation which may lead to PD for Math Teachers.  These are examples of how we use data to increase performance.

The bottom line is that our graduates successfully place into the high school programs of their choice.  Each one had a different ceiling – they are all different – but working with them, their families and their teachers, we successfully transitioned them all to the schools (private and public) and programs (IB, Gifted, French Immersion, Arts, etc.) that they qualified for.

And now each year, despite all the qualifications and caveats, our CAT4 scores continue to demonstrate excellence.  Excellence within the grades and between them. And let’s be clear, this academic excellence comes with an inclusive admissions process.

Despite our focus on individual growth, our average growth continues to significantly outpace national percentiles and grade equivalency scores.  Does investing in reflective practices (like blogging) lead to achievement ?  Does being an innovative learning pioneer translate into high academic success?

Two years in a row may not be conclusive, but it may be heading towards it!

NOT Preparing for the CAT4 – How OJCS Thinks About Standardized Testing

From November 5th – 7th, students at the Ottawa Jewish Community School in Grades 3 – 8 will be writing the Fourth Edition of the Canadian Achievement Tests (CAT4).  The purpose of this test is to inform instruction and programming for the 2019-202o school year, and to measure our students’ growth over time.  

  • If this is the first time you are visiting this topic on my blog, I encourage you to read my post on our philosophy of standardized test-taking.
  • If you are curious about how we share the results of our standardized test-taking (and what those results have been), I encourage you to read that post as well.

What’s new for 2019-2020?

We have gone from offering the exam in Grades 3, 6, and 8 to Grades 3 – 8 in order to ensure that the data is actionable on all four levels – that of the individual student (is there something to note about how Jonny did in Mathematics from last year to this year?), individual classes (is there something to note about how Grade 5 scored in Spelling compared to when they were in Grade 4?), grades (is there something to note about how Grade 3 performed in Reading  this year when compared to how Grade 3 did last year?), and the school as a whole (how does OJCS do in Vocabulary across the board?).  Without testing the same students in the same subjects at the same time of year on an annual basis, we would not be able to notice, track or respond to meaningful patterns.

Reminder:

Standardized tests in schools that do not explicitly teach to the test nor use curriculum specifically created to succeed on the tests – like ours – are very valuable snapshots.  Allow me to be overly didactic and emphasize each word: They are valuable – they are; they really do mean something.  And they are snapshots – they are not the entire picture, not by a long shot, of either the child or the school.  Only when contextualized in this way can we avoid the unnecessary anxiety that often bubbles up when results roll in.

Last year it took about six weeks to get results back, analyzed and shared out – to parents with individual results and to community with school metrics.  We hope to be in that window of time again and look forward to making full use of them to help each student and teacher continue to grow and improve.  We look forward to fruitful conversations.  And we welcome questions and feedback through whatever channels they come…

The Transparency Files: CAT*4 Results

In the lull between parent-teacher conferences, I spent my time reading and analyzing the results of this year’s CAT*4 testing.  [I strongly encourage you to reread (or read for the first time) my philosophy on test-taking and how we planned on both sharing the tests with parents and utilizing the data in our decision-making.]  We are in the process of providing our teachers with the data they need to better understand their students and to identify which test results fully resemble their children well enough to simply pass on and which results require contextualization in private conversation.

In terms of sharing out the results publicly, which I will happily do, there are a few things worth pointing out:

  • Although we do have prior years, they are not “apples to apples” enough to plot as comparison data.  This is mostly because of our decision to change our testing window and partially because we don’t have enough grades taking the test often enough.  (I have data on spring tests from two and three years ago for grades 3 & 6.)  If that changes, part of this annual analysis will consist of tracking the grades over time to see if…
    • The same grade scores as well or better each year.
    • The same class grows at least a year’s worth of growth.
  • The other issue is in the proper understanding of what a “grade equivalent score” really is.

Grade-equivalent scores attempt to show at what grade level and month your child is functioning.  However, grade-equivalent scores are not able to show this.  Let me use an example to illustrate this.  In reading comprehension, your son in Grade 5 scored a 7.3 grade equivalent on his Grade 5 test. The 7 represents the grade level while the 3 represents the month.  7.3 would represent the seventh grade, third month, which is December.  The reason it is the third month is because September is zero, October is one, etc.  It is not true though that your son is functioning at the seventh grade level since he was never tested on seventh grade material.  He was only tested on fifth grade material.  He performed like a seventh grader on fifth grade material.  That’s why the grade-equivalent scores should not be used to decide at what grade level a student is functioning.

One final caveat about why share out grade and class averages at all when so much of our focus is on personalized learning and individual growth…

Here, my thinking has been influenced by the work I was doing prior to coming to Ottawa, in my role as Executive Director of the Schechter Day School Network and then part of the transition team which helped create Prizmah.  I cannot tell you how many conversations I have had with colleagues about the different challenges Jewish day schools often have from their secular private school and high-achieving public (and/or gifted programs and in the States and/or magnet and/or charter) school neighbors.  The biggest difference comes down to a philosophy of admissions.  [Please note that although a primary audience for my blog are OJCS parents, other folk read as well, so I am including references to forms of public education that are commonly found in the States.]

Most Jewish day schools attempt to cast the widest net possible, believing it is our mission to provide a Jewish day school education to all who may wish one.  We do not, often, restrict admission to a subset of the population who score X on an admissions test and we do not, often, adjust birthday cutoffs or recommend grade repeating to maximize academic achievement. However, schools who we are most often compared to in terms of academic achievement often do one or both.  If you then factor in whether or not you exempt special needs students from the testing and whether or not you explicitly teach to the test, you may have quite an uneven playing field to say the least.

To reframe and reset the discussion:

Jewish day schools have an inclusive admissions policy, but are expected to compete equally with elite private and high-achieving public (and gifted and, in the States, magnet and charter and suburban public) schools who have exclusive admissions policies or homogeneous populations.

So, in light of all of that – if a Jewish day school with an inclusive admissions policy, a non-exempted special needs population, and a commitment to “not teach to the test” – if that kind of school could demonstrate that it was achieving secular academic excellence on par with elite schools; well to me that would be news worth sharing.

So with all those caveats in mind, in the spirit of full transparency, and with the attitude that all data is valuable data, allow me to present this year’s results:

The bottom line of this graphic is that each grade in the Ottawa Jewish Community School scored, with a few exceptions, at a mean grade equivalent a full year higher than their current grade.  There are a few (Grade 3 Writing, Grade 3 Spelling, Grade 6 Writing, Grade 6 Spelling and Grade 6 Computation) that are closer to their current grade.  [Part of our ongoing analysis and annual comparison would be to learn more about our current spelling and writing outcomes.  Part of our deeper investigation is whether there is a way to layer on standardized French and possibly Hebrew tests to learn more about those important outcomes.]  There are a lot of grades/topics whose averages are significantly higher than that, but let the boldface sink in for a bit.

Too much time dedicated to Jewish Studies?  Nope – a high-quality Jewish Studies program enhances secular academics.  Too much time dedicated to Skyping or blogging?  Nope – an innovative learning paradigm not only positively impacts student motivation, but leads to higher student achievement.

I can sense the tone of triumphalism in my writing and, although I am extremely proud of our students and teachers for their achievements, I do not wish to sound boastful.  But with the state of Jewish day school education being what it is, when there is good news to share…share it one must!  I firmly believe that Jewish day schools with dual-curricula (and in our case tri-curricula!) and innovative pedagogy and philosophy produce unmatched excellence in secular academics.  Here in our school, we will have to prove it year after year, subject after subject, and student after student in order to live up to our mutually high expectations, but what an exciting challenge it shall be coming to school each day to tackle!