Dear Survey Girl,
I purchased the Developmental Assets Profile to use with my after-school program. I gave them a pre-test on the first day of the program and am getting ready to give them a post-test when the program ends in two weeks. If I see any change between their scale scores, how will I know that it is significant? For example, if the score for Social Competency was 21.1 at the pre-test and 21.9 at the post-test, does that count as “significant” change?
Wondering,
Data-Abled Program
Hi DAP,
First, you didn’t say what the duration of your program is. If it’s at least 3 months, then you’re okay giving the DAP as a post-test. If your program isn’t at least 3 months in duration, the DAP isn’t designed to pick up change in that brief a program.
Second, there is a section in the DAP user manual that discusses the RCI (Reliable Change Index) guideline for the DAP. The RCI is a guideline for determining how much change in an individual score is required to represent true change, as opposed to measurement error. The average RCI for the DAP is 2.14, which implies that for almost all DAP scales, a change of 1 or 2 points could easily be attributed to measurement error, whereas a change of 3 or more points is likely to represent true change.
Search Institute Senior Research Fellow Peter Scales adds:
What a significant change is, depends on several factors, including sample size. All other things being equal, a small difference in a large sample can be significant, whereas it would take a much larger difference with a small sample to be just as significant. Then there is the matter of effect size. It might be statistically significant in terms of having the right p value, such as </= .05, but if the standard deviation of the change scores is large (that is, a lot of variation in how big or small the scores are among your youth), the meaningfulness of the effect would be small.
So, there are few things to keep in mind while looking at your post-test DAP scores. The natural question to ask after you’ve determined if your scores are statistically significant or not, is “how do I know the change is due to my program?” This is a little trickier to answer. We have not established normative data for the DAP, so we do not offer any comparison to your scores. Unless you have established a control group comparable to the students in the program and measure DAP scores for each, the DAP cannot reveal that change has occurred due to your program. If your program is implementing the Developmental Assets framework or your program goals align closely with the Asset Categories, you can infer that there is a reasonable chance that any change over time is due to your intervention, but without a standard of comparison of youth not going through the program, the program’s possible influence on the scores can only be surmised.
‘til next month,
Got a burning question for Survey Girl? Email her or follow her on 
[back]
