edventuresome

Who Else Shouldn’t Apply for an i3 Grant? People Who Can’t Follow Directions

by John on 25/03/10 at 1:52 pm

I learned a great deal this week by being in Denver to see the slideument presented by staff from the U.S. Department of Education at the Investing in Innovations Fund (i3) Pre-Application Workshop.

First off, Michele McNeil was right on Monday. If you don’t know what internal validity and external validity are, then you shouldn’t be applying for an Investing in Innovation Fund (i3) grant.  Or at least you should go find help.  “If these (terms) are too technical, I recommend you find outside partners to help you,” said Jon Jacobson, IES Senior Research Scientist in the Evaluation Division.

In other words, it’s not our problem to figure this out for you. Bring quality or go home.

And that’s just the tip of the iceberg.

There are three things I’d highlight in addition to the issue raised by Michele:

1. Requirements for Eligibility and Requirements for Selection—the Same but Different
2. Evidence – Get Thee to the WWC
3. Forget your Ps and Qs. Watch your “Ands” and “Ors”

Requirements for Entry and Selection Criteria—the Same but Different

Whether it was Jim Shelton, Erin McHugh, Ron Petracca or Jon Jacobson, the refrain was the same from all throughout the presentations: you must understand there is a barrier to entry, and once you’re in you better kick ass.

Essentially, there are numerous instances throughout the RFP where applicants will need to demonstrate their project, initiative, plan is first eligible, then whether that which makes them eligible is of high quality.

“Everybody is bringing their A-game,” said Petracca, an attorney in ED’s office of general counsel, in reference to the slough of applicants that are expected to send in proposals this spring.

So to get into the dance (be reviewed) you must meet the requirements for the type of grant you are going for:

• Scale-up applications must be supported by strong evidence
• Validation applications must be supported by moderate evidence
• Development applications must be supported by a reasonable hypothesis

and show evidence that an adequate outside match will be obtained.

Applications that do not meet the evidence requirement will not be eligible for a grant award, regardless of scores on the selection criteria.

That means if an application is judged by reviewers not to meet the “standards of evidence” of the grant type applied for, it will not be considered for a different type of i3 grant. So there is a cost to aiming too high (applying for a scale-up award and only bringing validation-award level evidence, for instance).

Then your application will be vetted on the quality you bring to the table with regard to the selection criteria.

For instance, to be eligible for review, proposing applicants who are non-profits must

Demonstrate that the non-profit organization has a record of significantly improving student achievement, attainment, or retention through its record of work with an LEA or schools.

Fine, except that “achievement,” is not specifically defined. As Ron Petracca put it, “we are dealing with 15,000 LEAs and variety of circumstances, so it’s up to the applicant to come up with the evidence, analysis and description [of what achievement means]. It’s a requirement for entry AND a selection criteria.  How well you make that case is important.”

This was reiterated throughout the day.

Evidence: Your Plan Needs to Match Past Research. Oh, And Get Thee to the WWC

Jon Jacobson discussed the importance of demonstrating your application has strong internal and external validity.

Strong internal validity means, according to Jacobsen, that there is strong evidence that the practice you are proposing has, as demonstrated in the literature, the effect you claim.

Strong external validity means that the range of participants and settings in your project will apply.

Because the evidence has to be strong, the practices and strategies have to be the same (at the highest evidence standard) as those in the prior research you cite.

That’s what he said. The same.

How does this play out in the different awards?

For Scale Up Grants

Participants in your project have to have the same kind of participants and settings that are discussed in the research you cite.

Furthermore, the research you cite has to demonstrate that the effects of the practice, strategy or program you plan to unveil is statistically significant.  This means that the research you cite must indicate that the programs you are interested in developing have, in the past, produced statistically significant effects and that statistical significance is likely again in your project.

I’ll note that the absence of the term “effect size” was deafening.

If you intend to cite experimental or quasi-experimental evidence for your practice, strategy or program, the evidence should meet the standards of the What Works Clearinghouse (WWC).  This may be the sleeper criteria that sends many inexperienced applicants packing.

As for magnitude, Jacobson defined that as the breadth of the impact. For scale-up awards the magnitude of the effect has to be “substantial and important” (both explicitly left undefined but couched with the following advice from the ED: because these will no doubt be context-specific issues for every applicant, you must make the case for how and why your magnitude will be substantial and important).

For Validation Grants

Herein the bar is lowered a smidge. The level of evidence you must bring to the table, based on prior research, must be moderate. That means you can make a kind of mix and match validity salad:

• High internal validity and moderate external validity, or
• Moderate internal validity and high external validity

But not

• Moderate internal validity and Moderate e

Your program or practice has to be the same or VERY similar to what you cited has occurred in prior research. You are allowed to have participants in more limited settings, and the statistical significance bar is lowered. You must demonstrate that it is likely that the sample in your project will attain differences or gains that are statistically significant.

On the magnitude front, you must demonstrate that there the magnitude is substantial and there is potential for it.

For Development Grants

This is the lowest bar. You must show that the theories and reported past practices have shown promise, and that you will engage is the same or similar practices. The participants may be a limited subset of what you have cited in the prior research, and the significance bar is set at “this warrants further study.” Likewise, the magnitude of your effect must show “promise.”

For instance, in a development grant you must have evidence that your proposed practice, strategy or program, or one similar to it, has been done in the past and is promising. You must have a rational for the practice that is based on research findings.

Bottom Line

In your application, the more evidence you have on prior research, the more competitive you will be. There is no penalty for stronger evidence. You have to meet the bar to get in. Once you do, you’ll be judged against everyone else who met the bar as to how strong you are once you’re in.

Forget your Ps and Qs. Watch your Ands and Ors

I think my favorite quote of the day came from Jim Shelton, when he said “the government doesn’t care what you think it means.”

Boy is that true. As Ron Petracca underscored in his talks, the federal government goes to great lengths to ensure it says only what it means to say. In other words, follow directions.

For instance, if an LEA wants to apply, it must show its eligibility by

(a) significantly closed achievement gaps between groups of students or demonstrated success in significantly  increasing academic achievement for all groups of students, and

(b) made significant improvement in other areas

So you can show one of the things in (a) and must show (b).

Look how this differs from the eligibility requirement for a nonprofit as applicant, wherein it must show its eligibility by

[Demonstrating] that the non-profit organization has a record of significantly improving student achievement, attainment, or retention through its record of work with an LEA or schools.

That’s an “or” statement, my friend. You get to choose.

I’m willing to bet that many applications will get tossed because applicants treat “and” requirements as “or” requirements.

In Sum

With the myriad of eligibility and selection criteria, it’s clear that there will be many, many opportunities for reviewers to reject your proposal for not following directions.

So I take back what I said. Mind your Ps and Qs and your Ands and Ors. Or you will be out.

Here are some images from the workshop.