Running Towards, or Away, from a Higher Standard for All

“Testing meats at the Department of Agriculture. Beltsville, Maryland.” by United States. Farm Security Administration is licensed under CC0 1.0

Chalkbeat interviews NY Chancellor Betty Rosa. She has few answers about what a “standard” means

Chalkbeat’s Monica Disare interviews Chancellor Rosa and presses her to define academic achievement in the absence of tests or graduation requirements–both of which Rosa’s Board of Regents has worked to walk back.

Rosa reacts strongly to the suggestion that she is lowering standards, but offers little that is concrete that the state will do to ensure that “academic achievement” stands for anything beyond what one teacher or school thinks:

“When you went to school, you obviously took classes. You got grades in your classes, right? If you didn’t live in New York, maybe you didn’t take Regents exams, but the fact of the matter is, there were pop quizzes, there were monthly tests, there were end-of-the-year tests, there were projects that you worked on. There were multiple ways that your teachers knew that you had been successful in acquiring the materials. So all I’m saying to you is that, I want the system to think about [that].”

That said, I agree with Rosa that talking about providing options and opportunities for students to meet standards should not be “two separate conversations.” Which is why I don’t think changing the expectations for graduation for some kids, as I’ve said before, is the right way to go. Instead, let’s work on defining those opportunities.

Disare presses Rosa for details on segregation, and while Rosa suggests that she “can’t even imagine anybody not working on this,” she offers no examples of what NYSED is doing to work on it.

Ideals are nice to espouse, but at some point a policy that upsets the apple cart will need be made.

http://www.chalkbeat.org/posts/ny/2017/03/28/we-need-an-opposite-narrative-chancellor-betty-rosa-on-her-year-of-trying-to-reshape-new-yorks-education-debate/

UPDATE: I had missed this report before posting this. The Board of Regents has discussed how to incorporate reporting school diversity numbers into ESSA and is creating a plan to explicitly address segregation. This is a promising development.

http://www.chalkbeat.org/posts/ny/2017/04/05/new-york-state-plans-to-use-new-federal-education-law-to-integrate-schools/

Test-based Accountability is proven to work

“Despite this track record of modest success, many parties seem poised to throw the policy overboard and use the guise of “parental choice” or “local control” to return us to a time when we had little idea which schools were educating children well and which were not.”

http://educationnext.org/why-accountability-matters-and-why-it-must-evolve-forum-polikoff/

Cami Anderson: School discipline takes more than talking points

“Improving school culture is much more demanding as well as inseparable from the rest of school life. We need to systematically rethink everything we do”

https://www.the74million.org/article/cami-anderson-the-left-right-school-discipline-debate-misses-the-point-we-need-a-third-way?utm_source=The+74+Million+Newsletter&utm_campaign=7e629255d8-EMAIL_CAMPAIGN_2017_03_24&utm_medium=email&utm_term=0_077b986842-7e629255d8-176115853

More than de minimis

“When all is said and done, a student offered an educational program providing ‘merely more than de minimis’ progress from year to year can hardly be said to have been offered an education at all,” Roberts wrote. “For children with disabilities, receiving instruction that aims so low would be tantamount to ‘sitting idly . . . awaiting the time when they were old enough to “drop out.” ’ ”

This SCOTUS ruling will inevitably result in more parent lawsuits against districts. But if that’s what it takes to improve instruction for students with disabilities, than so be it.

https://www.theatlantic.com/education/archive/2017/03/how-a-new-supreme-court-ruling-could-affect-special-education/520662/

Eating healthy food increases test scores

And it’s cost effective, guys

https://www.theatlantic.com/education/archive/2017/03/do-healthy-lunches-improve-student-test-scores/520272/

Conceptual vs. Procedural Math at Mastery charters

13946605527_82e5a3a848_z

“Maths” by Chris de Kok is licensed under CC BY 2.0

There was an interesting recent the74 piece on a Philadelphia charter organization, Mastery, which takes low performing schools and works to “turn them around.”

 

Embedded within this article is the implication that a shift to a focus on the teaching of conceptual math, rather than “rote” procedural teaching, led to a swift downturn in math scores.

“So this year, the network began reintroducing teaching techniques that had been a staple at Mastery schools for years, while seeking a middle ground between no excuses and restorative practices. It’s a ‘journey of trying to find out what’s the right mix,’ Gordon said.

Specifically, the network is reintroducing procedural math instruction, which focuses on rote instruction like memorization and repetition.”

It seems worth digging into this supposition a bit more.

Is Mastery’s downturn in math scores due to the failure of conceptual math in general as a pedagogical approach? Or is it a failure of the network to attract and train teachers who can teach this type of math more effectively?

Or is it a failure in the assessments that were used as a reference? Or was it that conceptual math takes longer to “stick” and pay dividends? Or was it a failure of the curriculum they used to move in a more conceptual direction? . . .

http://the74million.org/article/at-philadelphias-mastery-charter-network-culture-is-key-to-turning-around-failing-schools

Give teachers some privacy

By PICOL- PIctorial COmmunication Language (http://picol.org/) [CC BY 3.0 (http://creativecommons.org/licenses/by/3.0)], via Wikimedia Commons
By PICOL- PIctorial COmmunication Language (http://picol.org/), via Wikimedia Commons

“…many trucking companies use cameras that automatically record a driver whenever there’s sudden braking, swerving or speeding up. But in one company Bernstein studied, the videos never go to management and are not used in performance reviews (unless the driver is texting-at-the-wheel dangerous). Instead, a team of coaches, whose only job is help drivers improve, receives the videos. Drivers, he says, like and trust that the system is there to help them, because it keeps their mistakes within a trusted circle of people who are not wielding power over their lives.

…In instituting these four forms of privacy—privacy within team boundaries, privacy limits on employee data, privacy in decision-making, and privacy about time—the organizations Bernstein studied refused the temptation to observe (or try to observe) everything. That refusal did not cost them profits or effectiveness. Instead, respect for privacy enhanced their success.”

–David Berreby, “Want People to Behave Better? Give Them More Privacy” on PsychologyToday

Observations of teacher practice have become a rote chore of paperwork to try and please accountability mavens. Those same mavens then get in a huff when they don’t find a whole bunch of teachers rated ineffective.

How about you just let teachers observe one another and give each other feedback, and administrators just sit in on that process every now and then? You might find that to be much more effective than the rigmarole of compliance that teacher evaluations have become.

Success can’t be measured by one or two numbers

“Whenever you make huge decisions about complex situations based on one or two numbers, you’re headed for disaster — especially when those numbers can be gamed.”

—Mark Palko and Andrew Gelman, “How schools that obsess about standardized tests ruin them as measures of success” on Vox

We’ve questioned Success Academy’s “success” on this blog before. These statisticians bring a new lens to that question.

I don’t want to denigrate the good work that Success Academy teachers and students are doing. There are practices and systems well worth replicating and investigating in these schools. But Eva Moskowitz’s political framing and marketing of her schools as the solution to poverty is problematic.

Incremental Change

“Silvia Abbato, the district’s superintendent, said she could not pinpoint any one action that had led to the better scores. . . . 

“It’s not something you can do overnight,” Ms. Abbato said. “We have been taking incremental steps everywhere.”

—Motoko Rich, Amanda Cox, and Matthew Bloch, “Money, Race, and Success: How Your District Compares” in The NY Times

School Climate Matters

A classroom in Guipuscoa

Chalk up more research confirming what-we’ve-been-saying-all-along here at Schools & Ecosystems: a school’s learning environment impacts student learning.

In case you don’t know, NYC has been collecting what folks call “school climate” data via surveys administered to teachers, parents, and students since 2007. It’s important information to have about a school–arguably more important, to my mind, than test scores (I believe both should be considered).

Last July, I had quoted Match Education’s Mike Goldstein asking an important question about all this data:

Is anyone aware of scholars and reporters digging deep into this data set?  Is there any other data set in the USA just as good?

I think it’d be hugely productive to identify NYC schools which have made progress in “Total Climate” — and then study why.

Well, Mike, you’ve got your answer.

NYU’s Research Alliance for New York City Schools published a study using NYC’s school climate information that demonstrates that a school’s learning environment not only impacts student learning, but furthermore teacher retention. As Chalkbeat NY’s Alex Zimmerman reports:

Each measure, the report found, is independently linked to decreases in teacher turnover. And gains on two of those measures, high academic expectations and school safety, were directly connected to better scores on state math exams.

The study found that if a school improved from the 50th percentile across the study’s four measures of school climate (leadership, expectations, relationships, and safety) to the 84th percentile, teacher turnover would decline by 25 percent, or 3.8 percentage points.

A similar percentile increase in measures of school safety and high academic expectations alone boosted math scores enough to account for an extra month and a half of instruction. (Improvements in school climate also boosted language arts scores on state tests, but those gains weren’t statistically significant.)

It’s important to note that this study confined its focus to the following aspects of school climate:

  • safety and order
  • leadership and professional development
  • high academic expectations
  • teacher relationships and collaboration

Missing in such an examination (and mostly from these surveys themselves) is a focus on the physical environment of a school. There are questions pertaining to cleanliness and conditions of a school, but as we’ve also been arguing on this blog, the actual design, and the incorporation (or absence) of access to natural light and greenery, colors, furniture, etcetera (all largely subconscious factors), all have an impact on learning and relationships in a school.

If your school is interested in collecting school climate data, the US Department of Education is sharing free surveys and information for collection of data similar to NYC’s. Check it out and share.

My current views on testing, in answer to my past views on testing

While up in Albany a few weeks ago, I was interviewed by someone from NYSED about what I might say to parents who are considering “opting out” their child from state testing. You can view the video here*.

Someone on Twitter, “WiffleCardenal,” voiced a critique to me regarding the video, in contrast to things I’ve said in the past on testing. In fact, they even tweeted quotes of my own words! I deeply appreciate that someone out there is actually listening, and willing to take the time and effort to hold me accountable to them. I have elected to respond here, since Twitter isn’t the greatest venue for nuanced discussion, especially at the end of a long day, and I also hate typing things on my phone.

This is in reference to a live chat I did back in 2012 on The Nation‘s website with journalist Dana Goldstein and educator Tara Brancato. Have my views shifted since then? I would say they have in some ways.

You know, honestly, they’re not as terrible as I thought back then. I proctor these tests each year and go through the experience of answering the questions along with my students. The questions are often cognitively demanding and require multiple reappraisals of the text in question. A few of them are duds, certainly, but having tried to write many of my own text-dependent questions since then, I’ve come to appreciate a well-written multiple choice question. Check out this post from Joe Kirby (UK educator) on the rationale for using multiple choice questions for assessment.

Unfortunately, this continues to hold true. In reaction to this, the Center for American Progress recently created a “testing bill of rights” to advocate for better aligning tests with a more meaningful purpose.

This doesn’t mean, however, that I’m opposed to having test scores factor into my own evaluation or my school’s evaluation. When scores are considered over multiple years, I think they can be an important and useful measure of teacher effectiveness. But they are extremely variable, so I would only want them to be considered alongside other data that can provide adequate context.

One of the things I’ve become more aware of over time is that while our testing and evaluation schemes are extremely problematic, if we look at the big picture, accountability and testing do bring transparency to serving populations of students that were traditionally ignored. No Child Left Behind was certainly faulty and overzealous policy — but it also brought attention to holding school districts accountable to serving students with disabilities and other underserved populations based on data. This was entirely new, and it has raised awareness.

This is why the NAACP, the National Disability Rights Network, and other national civil rights groups oppose anti-testing movements.

Yes, I continue to believe this. Test measures are only one source of data that need to be coupled with qualitative observational data and other forms of understanding. Fortunately, I do feel like our focus, at least in NYC, has shifted to better match this understanding.

To give further context on my statements on the NYSED video, I was speaking about how I use testing data, which I do every week when developing IEPs for my students with disabilities. I compile all information I have on a student, including multiple years of state test data, in-house assessment data, such as reading, writing, and math scores, GPA, attendance, psychoeducational evaluations, social histories, etc. When viewed all together, in tandem with teacher observations and student and parent interviews, I find aggregate state testing data useful!

So it’s important to understand I’m not advocating now and never have advocated for a state test score as a singular reference point to judge myself or a student. But when viewed with appropriate context, I do find state testing data to be useful. (More on how I use that to develop IEPs here.)

No, unfortunately. While I do think that test scores should factor into an account of an individual teacher’s effectiveness (only in aggregate and when considered in terms of growth, not proficiency), we’re creating incentives for competition, rather than collaboration.

If I could set the rules for how we use test scores for accountability, I would do something kind of radical: I would hold all grade-level teachers accountable for student scores on literacy tests. And I’d stop labeling them “ELA” tests and call them “literacy” tests. Why? Because if we are honest about what we’re really testing, we’d acknowledge that the knowledge required to understand complex texts comes not solely from ELA, but furthermore from science, social studies, music, art, and so forth. (More on my argument on this here).

Furthermore, I’d try to better level the playing field for all students by requiring test makers to broadcast one year in advance which texts would be tested (not specific passages, just the general title/author). I would allow parents and educators an opportunity to vote on which texts they wanted tested that year as well to make it more reflective of current interests. The reason I would do this is that this would provide an opportunity for all students to build up the requisite vocabulary and background knowledge to access a text. Right now we just give them random texts, as if every child will be bringing equivalent knowledge and vocabulary to them, which is false.

Yes, unfortunately this continues to hold true in too many schools. But this is also why I have been a consistent supporter of Common Core standards, which have become synonymous with testing in some people’s minds. Yet the Common Core standards provided us an opportunity to move away from test prep, because they are fundamentally about building student knowledge and academic vocabulary through engagement with rich and complex texts — this is the exact opposite of test prep!

This speaks to the problem of making state tests so high stakes, and why we need multiple measures, such as direct observation, to hold schools accountable. It also is the reason for why I would advocate for the seemingly radical measure, as per above, of communicating which texts would be assessed that year so that “test prep” instead would simply be about reading and studying and discussing the rich texts that were selected for that year’s assessment.

Yes, it can be inhumane when a student is several years behind in reading ability or struggles in coping with anxiety and stress.

While computerized testing brings a whole new set of problems, I do believe we should move in this direction, because with computerized testing, we can use adaptive testing that can better scale to meet a student where they are. Otherwise we end up punishing students who are struggling, for whatever reason. Unfortunately, the needs of students with disabilities never seem to be factored into test design except as a final consideration, rather than from the ground up.

But there’s another side to this, too. I think we have to ask ourselves, as a teacher, a school, and a system, how do we prepare all of our students to be able to engage with a challenging text independently? And in what ways are we sequentially building their knowledge and skills and vocabulary in order to prepare them for doing so? It is the failure to do so systematically and adequately that we are failing students who most need those skills and knowledge.

Pearson is out of the picture, in case you didn’t know. I have no idea what Questar tests will be like, though I imagine they will be comparable.

From what I’ve heard, PARCC assessments are far superior to the cheaper assessments NY decided to get from Pearson. I think we get what we pay for, and if we want better test design, we have to be willing to fund them.

Personally, I think if we’re going to just use tests for accountability purposes, then we could make them every 2 or 3 years instead of every year to save money, and they could still continue to be used for that purpose.

What would be awesome is if we could move more towards performance based assessment. There’s a great article on them in the most recent American Educator. This seems like the right direction to go in if we truly interested in assessing the “whole child.”

Well, don’t know if all of this fully says everything I would like to say about testing, but I’m seriously tired after a long week, so this will have to do.

WiffleCardenal, whoever you are, thank you holding me accountable and I welcome continued critical dialogue on these issues.

* This was after a long day of a train ride from NYC and meetings with legislators, so I apologize for my shiny face. Won’t apologize for the winter beard, however. And no, I was not paid for that interview nor given a script. As ever, I speak my own mind (or so I like to think. Certainly let me know if it ever seems like I don’t).

Accountability for the Long-Term

time_distance

I receive a monthly newsletter from bcg.perspectives that I scan for any relevant connections to school systems. Their work often centers on business policy, but sometimes their work has either a direct or indirect connection to the education sector.

A recent post,”Gauging Long-Term Impact in the Social Sector” on developing a system of long-term evaluation for a large international nongovernmental organization (INGO), there are lessons well worth considering in developing systems of long-term accountability for schools.

The INGO discussed, named SOS Children’s Villages, works towards “improving the situation of children who are at risk of losing, or who have already lost, parental care” across 134 countries.

The assessment methodology that SOS Children’s Villages developed jointly with BCG evaluates two elements of the programs’ long-term impact: the nonfinancial (or all-around) impact on the individual program participants and the community and the financial impact on society. The determination of the long-term impact on individual participants is based largely on information gleaned from interviews of former program participants by external researchers. This is supplemented by qualitative research conducted through focus group discussions with former child participants and their caregivers. . . 

The programs’ long-term financial impact on society is gauged by the programs’social return on investment (SROI), a comparison of the programs’ total costs and benefits to society. . .  The calculation of societal benefits is based on easily quantifiable elements.

This combination of intensive qualitative and correlated quantitative data gathering seems to make great sense when considering systems for school accountability. Many school systems have been relying primarily on isolated testing data — but why not go straight to the source, and interview the ones we most seek to impact? The students and families and community. And then correlate that with longer-term impacts via “social return on investment”? What are the long term outcomes of students after they graduate?

Raising test scores is wonderful. But enriching one’s community and society over the long-haul is the true goal of education. Developing better combinations of quantitative and qualitative evaluation of our school systems that can help us determine long-term impact is key to not losing sight of that higher purpose.

Responsible Regulation, Not Ideology

“it’s time for school choice advocates to dispense with ideology, engage regulators, and get serious about a policy environment that promotes measurable quality, scale, and access.”

—John White, “America’s most disadvantaged students need real accountability, not ideology” on Flypaper

Friedrichs v CTA, and Thinking Probabilistically

By Matěj Baťha (Own work) [CC BY-SA 2.5 (http://creativecommons.org/licenses/by-sa/2.5)%5D, via Wikimedia Commons

Yeah, that headline was a mouthful.

But here’s the thing. You’re going to hear a lot of ed folks declaiming on the potential outcome of the Friedrichs v California Teachers Association SCOTUS case over the next few days. For good reason, as this is a case that may well prove to be more determinative of the future of public education in this country than ESSA.*

I’ve been reading Daniel Kahneman’s excellent Thinking, Fast and Slow lately**. Kahneman’s book is all about ideas we’ve touched on before here, such as cognitive bias and uncertainty. We’ve also looked at how “probabilistic thinking” could be used to overcome bias. So when I fortuitously came across this article on how “superforecasters” use probabilistic thinking, as well as a “base rate,” or “reference class” in order to make more accurate predictions, it jibed well with my understanding, and I think there’s useful lessons to heed as Friedrichs case is heard over the course of this week.

Rather than ideologically proclaiming sweeping predictions, as the experts are wont to do, “superforecasters” are less certain about their predictions, which ironically makes them better predictors. Professor Philip Tetlock delineates between “hedgehogs” and “foxes,” and notes that superforecasters are more akin to foxes:***

According to Tetlock, foxes are more pragmatic and open-minded, aggregating information from a wide variety of sources. They talk in terms of probability and possibility, rather than certainty, and they tend to use words like “however,” “but,” “although” and “on the other hand” when speaking. . . 

Unfortunately, most of the predictions you see in the media lack the specificity necessary to test them, like a specific time frame or probability, Tetlock says. . . 

Instead, Tetlock advocates for something he calls “adverserial collaboration” — getting people with opposing opinions in an argument to make very specific predictions about the future in a public setting, so onlookers can measure which side was more correct.

What does this have to do with Friedrichs? Well, I would suggest asking education “experts,” who will write about their ideas on the case, to assign a probability to their predicted outcome.

Based on my own, extremely limited understanding of the case, I think there’s a 65% chance that Friedrichs will win. I could well be completely wrong. But you’ve got my prediction here, in writing, with a timestamp on it, so you can hold me accountable to this.

I’ll write more on my thoughts on the case soon, but in the meantime, my thinking on Friedrichs v. CTA in a nutshell:

I think public sector unions need to change and adapt much more rapidly to a changing workforce and economy, but I believe strongly in the necessity for unions to present a necessary counterbalance to government and private financial interests. If Friedrichs wins, as I’m afraid she might, then we will witness a drastic further decline in the power of unions in our country. I believe this will be to the detriment of the long-term interests of our nation.

The only commentator I’ve seen thus far who’s beginning to think ahead to this outcome is Dan Weisberg of TNTP. He doesn’t assign a probability to the outcome, but implies it when he says the following:

Unfortunately for the unions, at least five Supreme Court justices appear to be more sympathetic to the teachers’ arguments than I am. The Court practically invited this challenge when it stopped just short of striking down agency fees in a similar case a few years ago.

I’m hoping our unions are already preparing for the worst, because no amount of impassioned op-eds can influence the outcome at this point.

*my apologies to all non-US residents for the US-specific jargon in this post.

**thanks to Deputy Chancellor Josh Wallack, who recently bestowed the book on educators at a dinner hosted by the NYC DOE Office of Leadership.

***We’ve looked at hedgehogs and foxes here before:

 

UPDATE 2/13/16:

Justice Scalia has just died, so that completely changes the odds. While I had first assigned a 65% probability to Fredrichs winning this case, my forecast has shifted closed to 40%. Read more on SCOTUSBlog: “The most immediate and important implications involve that union case.  A conservative ruling in that case is now unlikely to issue.”