The Test That Showed Nothing, Then Showed Everything

Here’s a finding that reframed how we think about measuring our own program’s effectiveness.

Early on, we assessed our financial literacy curriculum using standard, generic financial literacy assessments, the kind most programs use. The results showed minimal pre-post change. Almost flat. Meanwhile, residents were telling us directly that the education was highly valuable to them.

Those two facts shouldn’t coexist. Either the assessment was wrong, or the residents were wrong. We didn’t think the residents were wrong.

So we built our own assessments, ones that measured knowledge of emergency assistance programs, eviction defense procedures, tenant rights, and rental application strategies, the actual competencies our curriculum was teaching. Once we measured what we were actually teaching, we documented substantial learning gains.

The standard assessment wasn’t measuring failure. It was measuring the wrong thing entirely, the same disconnect that runs through so much of this field. Generic financial literacy metrics assume long-term financial behaviors, retirement contributions, investment diversification, are the relevant outcomes. For someone navigating active housing instability, those outcomes are premature at best and irrelevant at worst.

This is part of why we built our own evaluation approach into the program rather than relying on generic assessment tools. If you cannot accurately measure what a person actually learned, you cannot know whether your program is working, no matter how confident you are in the curriculum itself.

I’d be curious whether other practitioners have run into this same disconnect, where a standard evaluation tool didn’t reflect what people were actually gaining.

Author: Sandra Roussel
Copyright © 2009-2026 Sandra Roussel. Reproduction and distribution of all materials without written permission is strictly prohibited. All rights reserved.