On June 28, 2010 the application process for Promise Neighborhoods came to an end, at least for local planning groups. As we write this, the Department of Education (ED) is assembling peer reviewers to score the 339 applications that were submitted. The results will be revealed in September.
As the Department and its reviewers prepare to evaluate our planners, we thought it only fair to get a jump on the process and ask our planners what they think of the Department and the planning process.
This article and those that will follow in the days ahead are based on interviews with 47 local Promise Neighborhoods planning groups, including 9 that planned to apply but did not. The interviews were supplemented by a follow-up survey. (A complete review of methodology can be found here.)
In this, the first (and longest) of our articles, we will review the experiences and thoughts of our local planners about the process created by the U.S. Department of Education. Subsequent articles, to be published over the next several days, will review local politics, issues with local schools, experiences with private funders, the experiences of those who did not apply, thoughts on technical assistance, and the way ahead.
The Pre-application Period: “Waiting and waiting and waiting …”
Excitement over what would eventually become Promise Neighborhoods predated its creation, and even the presidential campaign of then Senator Barack Obama, by several years. In many cases, local leaders had been following the Harlem Children’s Zone from afar in the media, including The New York Times and 60 Minutes. In some cases, excitement grew when Geoffrey Canada visited their communities. In others, local leaders visited HCZ in New York to learn more.
In November 2009, PolicyLink and the Harlem Children’s Zone sponsored a conference that was widely attended, including by many of the planning groups that we interviewed. Of the 42 groups who returned surveys, 26 indicated that someone from their group had either attended the conference or one of the related Practitioner Institute sessions hosted by HCZ. All who attended found them to be highly informative and energizing. Several commented that they thought the conference organizers were smart to require groups to attend as teams.
Still, there were 16 groups that we interviewed (just under half) that did not attend and were disappointed that they could not. In some cases, it was because the conference and related Institute sessions were sold out and unavailable. In other cases it was a lack of local funding to pay for the trip.
While HCZ loomed large during the pre-planning period, our planners also mentioned several other sources of help. One major source was pre-existing work that had been ongoing in some communities for years. The Annie E. Casey Making Connections sites were among these, as was work done by LISC in Chicago and work that had been done as part of the community schools movement. In each of these cases, local planners in these communities felt they had a jump start that had helped them.
Other materials and organizations that were mentioned as helpful during the pre-application period were the book “Whatever It Takes” by Paul Tough, materials from PolicyLink and HCZ, coaching and guidance from the Center for the Study of Social Policy (CSSP), Child Trends, United Way Worldwide, and United Neighborhoods Centers of America (specifically the Tipping Neighborhoods to Success report and Planning How To Guide).
Despite the amount of help that was being made available, however, some of the planners we interviewed felt that it was of limited usefulness. “UNCA and others were prepping the landscape,” said one, “but it was all just noise until the RFP actually came out.”
And even though HCZ’s materials were widely seen as helpful, another planner noted: ““The Department was not actually funding the replication of Harlem Children’s Zone. It is helpful to know what HCZ is doing, but it may be different from what the Department was looking for.”
“The time was just so difficult,” said another, speaking of the pre-application period more generally. “We just couldn’t anticipate what was going to be required.” One problematic point for some planners during this period was the choice of neighborhood or its boundaries, since many were not certain what the requirements would be or how the required indicators would be integrated with schools.
The pressure to find out details was significant. Some called the Department of Education. “The staff was always courteous when I spoke with them,” said one. “But their answer was always the same, can’t share specific information and the RFP will be released.” We at UNCA felt the same pressure to find out more, and the ebb and flow of rumors eventually caused us to report on them. Those rumors ended up being untrue, and we largely ignored the rumors after that.
And They’re Off!
On April 30, the Promise Neighborhoods grant notice was released, setting off a firestorm of activity around the country. The initial deadline was June 25, which provided local applicants less than two months to prepare. “It was a mad dash,” said one planner. “Frenzied.”
The short application period was a major challenge for many local planning groups. Of the 42 local planning groups that answered our follow up survey, 26 said the time period was a significant challenge, while only 11 said it was not.
“I was on lockdown. My boss had her cell phone strapped to her side,” said one planner. “You had to push things so hard you were straining relationships, especially with funders,” said another. “That was awkward.”
Several echoed that, saying foundations typically meet quarterly, so getting prompt action was difficult. The strain on relationships more generally was a common theme. Said one planner:
The tight time constraints were a challenge in a lot of ways. We invited everyone that we thought should be at the table, but sometimes people got overlooked. That is when you start to see issues in the community that you weren’t aware of. It was kind of like stepping on landmines. It brought issues to the table sooner rather than later, so that was good. But there were hurt feelings and peace keeping was required.
Another pointed out: “It would have been fine if the application had not asked for so much (too much, frankly).”
Items where local planners were most pressed for time included data collection and reaching agreement on MOUs. Said one: “We were unable to include a few partners due to not having the time to get their individual information required in the application.”
The timing of the application’s release at the end of the school year also caused problems for roughly half of the groups we interviewed. Some pointed out that, as with the foundations mentioned earlier, school boards were being forced to call special sessions to approve local plans on time. Said one:
Once the school year ended, the schools had more freedom to help pull data. However, having the request published during the last month of school made it difficult for the schools to participate as much as they would have liked to, especially with the activities that are associated with the end of school.
Still, despite these problems, many local planners managed to apply and of course several hundred got their applications in on time.
In some cases, local planners actually thought the time crunch helped. It “drove action” said one. Another suggested it helped their group focus on nuts and bolts. “I don’t care about big thinkers, I want small thinkers. I prefer worker bees.”
“Planning to Plan Is A Dumb Idea”
Many of our local groups thought the applications asked for too much planning during the application period. In the follow up survey, 23 local groups agreed with this, while only 9 disagreed. “It felt like we were asked to create the plan before we were given funds to create the plan,” said one.
Some of the problem may have been confusion about what the Department wanted, but others felt the Department was intentionally pushing the envelope. “LOL. I guess they wanted to narrow it down from 900+ applicants to only 20, so they set the bar high,” wrote one.
“We had heard that the Department was looking for “shovel-ready” projects as the most likely to be funded, so we did not want to appear as if we were too early in the planning process,” said another.
Others who did not see the pre-planning requirements as overly burdensome thought it was a good thing to require local planners to show they could “hit the ground running.” One said “strong projects should have done a lot of this planning, already.” Indeed, this aspect of the program may have benefited local projects that had been developing for several years. Some in this group indicated that they were only changing their plans marginally to apply for Promise Neighborhoods. The money involved in Promise Neighborhoods was not enough to change anyone’s existing plans, one said.
Still, these views were not widely shared. “This concern would have been less of an issue if there was more time between the release of the RFP and the application deadline,” one pointed out. Others went further, arguing that if the Department was deliberately trying to push local groups into planning it was counterproductive. “Planning to plan is a dumb idea,” one said bluntly.
Some offered reasons. Some said that by pushing local groups to set down so much on paper as part of the application, it gave them less flexibility to backtrack and revise and improve their plans during the planning year after they had conducted adequate local community engagement and given sufficient time and thought to the process.
“The idea is we don’t know the answers yet. We won’t know this stuff until the planning year is over. If it is a planning grant, you need to do this during the planning year. If we knew what needed to be done, we wouldn’t be applying for a planning grant.”
A Not-So-Simple Cost-Benefit Decision
One point that a few planners made was that the odds were so small of being chosen to receive a grant that it almost made no sense to apply, given the significant staff costs. “For many of us who participated we almost had to forget our day-job to participate in writing the grant,” said one. Others said it made no sense to invest “thousands of hours” of staff time in an application, only to be told no.
There appeared to be two main overriding factors contributing to the relatively large number of applications that were submitted anyway, despite the odds. One was passion for the idea. The excitement around the nation was significant. This came up over and over again in our interviews. Just as important was another factor, which we will cover in more depth later in our series and that has become more important in recent days as Congress has reduced the president’s recommended funding. That factor was a widespread determination to press forward, regardless of what happened with their Promise Neighborhood application.
Despite their seeming obliviousness to the cost and odds of being chosen, some planners nevertheless suggested ways to make the application less burdensome. Some thought that breaking the application process into steps, as appears to be the plan for Choice Neighborhoods, might make more sense. Others argued that some of the paperwork could be saved for later, if a project had been preliminarily approved.
“All those federal forms they require you to fill out – they are not routine. Have the forms available for review, but don’t worry about filing until you get the gig,” said one. “Whether you have a lobbying firm or not doesn’t make any difference at this stage of the game,” said another.
A Confusing Application: Business As Usual?
The Promise Neighborhoods application package received mixed reviews from our planning groups. In our follow up survey, more groups felt it was confusing (15) than not (11).
For those who experienced problems, some of it may have been that as nonprofit social service providers, many were unfamiliar with the Department of Education’s “arcane” lingo, forms, and certifications. Many had never written a Department of Education grant. “What’s a GEPA certification?” asked one. Some found explaining “Theory of Change” and “Theory of Action” to others to be a challenge and explaining the differences between them even more so.
Still others felt that the confusion stemmed more from the organization of the application itself. “The RFP was a mess. The way it was put together was very confusing. Whoever wrote it seems to have never stood on the other side of this process,” said one. “It was clear that the application was not written by one writer,” said another. According to another it was clearly “written by committee” and “thus was not always as coherent as one might expect.”
In concrete terms, some felt this resulted in unnecessary redundancy and confusion.
The goals did not directly match instructions, which did not directly match scoring criteria. Each had something that wasn’t in the other sections. Even the layout was poor… I had to flip through that RFP 100 times to find the section I needed. Ultimately, I was often only using the electronic version so that I could search for a term and more readily find it.
According to another:
Mixing the numbers under different sections made us a little crazy. We had to do a lot of reorganizing to satisfy both the narrative outline and the evaluation criteria. Where does data go? It required a significant use of Tums.
In some cases, local consultants helped, but not always. “The grant writers were arguing with the lawyers,” said one. “These folks really know their field of knowledge, some of the most vocal just did not know what they really did not know and needed to listen to those who did know.”
At least two objected to the requirement to double space everything, including footnotes. One said requiring charts to be double spaced was “goofy.”
Others felt the application showed a lack of overall vision.
Where is the big vision about how these things fit together? Is there someone in Washington sitting and thinking about how they fit together? It would be great if they shared that. What was the thinking of all this? Where do they see it going?
Another talked specifically about “a vision of cross-departmental jurisdiction: HUD, HHS. I did not see much of that come through the RFP. Arne Duncan talked about it so I hope they find a concrete way to do that.”
Despite these criticisms, however, there were many planning groups that did not find the application that confusing. Some seemed to think the application was typical. It was “way more complicated than it needed to be but you could say the same about nearly every government program in existence,” said one. “I did not find it any more confusing than other Department applications,” said another.
Others seemed to think the confusion may have been avoided by investing enough money in experienced grant writers. “For experienced federal grant writers, the application and the application process was clear and showed exactly what the department wanted,” one said. “I would think for inexperienced federal grant writers, the process was overwhelming and confusing.”
Local groups “needed to get the money to get a consultant to do the writing. None of them had the capacity in one month to get this done,” said another.
Indicators and Evidence: Generally Good, But Some Politics?
In our follow up survey, local planners rated the Department’s core academic, family, and community indicators as adequate (13) to good (15) with a few labeling them excellent (5). None rated them as poor. One planner seemed typical when she said ”I thought they were mostly pretty good, although some of the community and family ones seemed narrow. Of course, they told applicants they could use their own for those, so I guess that helped.”
Some indicators were questioned, however. According to one planner:
The focus on computers and broadband internet in the indicators was odd. There actually isn’t much evidence to support this. In fact, there is information that says putting kids in front of computers has detrimental effects.
For all the stress on evidence, another argued, where was the evidence for choosing these indicators? How would the Department’s own peer reviewers rate their indicators if they were held up to the same evidence standards?
According to another:
The health indicators were skewed because they focused only on nutrition and exercise, not overall health. I’m guessing that might have been a reflection of the First Lady’s priorities, but it caused them to miss other common healthy-equity problems like asthma.
One pointed out there was some tension between the need for innovation and evidence-based practices. “Sometimes innovation flies in the face of evidence-based practices. Innovative ideas need to be developed, evaluated, and refined to generate new evidence-based practices.”
Many local groups used the option to develop their own local indicators, but one pointed out a potential problem with this approach. “Our local public agencies have no interest in developing different data interfaces for several differently designed projects. They want to present a single interface and let the local projects figure out how to connect to them.”
The Notice of Intent to Apply
The Department asked local applicants to file Notices of Intent to Apply by May 21 and most did. The Department said it was doing so to “facilitate potential partnerships” (it also helped them get a jump start on preparing for an appropriate number of peer reviewers). The Department publicly posted the list of 941 potential applicants on May 27.
In our follow up survey, we asked our planners how helpful publishing the list of applicants was. The largest number (16) said it did not make any difference. A few said it was helpful (7) and some said it was hurtful (5). Two others said it made no difference but that they supported the release on transparency grounds. Two more said it encouraged applicants to drop out, without specifying whether this was a good or bad thing.
Among those who said it was helpful, publishing the notices seemed to work as intended, with local applicants consolidating their plans. This was not always the case, though. “It set off a political fight in our city. It would have been better if it was ‘blind’. I don’t think it changed the number of grants at all.”
Those who said it hurt usually referenced a chilling effect on public agencies and especially private funders, who were afraid of playing favorites.
I know that many of the NIA’s from my region never applied, but it made it look like there were so many planning to do this and that created confusion for our partners and funders. After all, all anyone had to do to file an “NIA” was file a one-page form, so development officers just threw these out. So it was a useless and annoying thing for the Department to do, and it did not encourage partnerships, it discouraged them. The pre-conference from HCZ was more helpful since like-minded folks could find each other well in advance. Seeing the NIA list 1.5 months prior to deadline left no time for anything but angst.
“Dancing” Webinars Score Points With the Judges
One item where the Department seems to have scored a significant hit was with its well-received webinars and associated PowerPoint. Several said it was one of the best things about the application process.
“The webinars were great, so was the basic website,” said one. According to another, “The best thing about the application was that the Department was forward thinking enough to host webinars. We REALLY found them helpful in terms of clarifying some of the instructions and requirements. Our entire internal team watched them on several occasions.”
A third gave a bit of a back-handed compliment, saying “The webinars and the accompanying slide show were excellent (thank heavens for those because it would have been hard to deal with the RFP otherwise).”
The FAQ
The Department’s Frequently Asked Questions document, updated weekly, received mixed reviews. Some felt that the Department had done “a good job.” Others, however, felt the FAQ was being used to make new rules, rather than clarify existing ones.
“I have never been involved in a process where the rules kept changing,” said one planner. “The FAQs would change people’s understanding midway through the game,” said another. Referring to the steady stream of updates, one was resigned. “What new thing are we going to find out?”
A final common point, perhaps expected, was that many planners were frustrated that some of their questions were never answered. “It seemed to be that it had to be a common question to get a response. We had plenty of unique questions. We would hate to get defeated because of a mis-interpretation of something and getting dinged on it.”
At least two local planners suggested that the Department should allow live one-on-one responses to all questions, as seemed to be occurring during the Department’s webinars. One planner said this approach had worked well for another federal agency.
The MOU and Narrative “Kerfuffles”
Two of the most consistently remarked upon decisions made by the Department were its decisions to lengthen the allowed page length for the MOU and narratives, both of which were made late in the grant process. We at UNCA played a minor role in each. In the case of the MOU change, we missed the significance at first, but within a few days we eventually heard a chorus of grumbling and griping, which we duly noted on the Building Neighborhoods blog in a post entitled “The 10-page MOU Kerfuffle” (apparently the word “kerfuffle” stuck, although not everyone had heard of the word before). As we reported at the time, local planners had strong and diverging views of this change.
A few weeks later when the Department decided to do the same thing with the narrative, we were not caught unaware. When the “send” button was pressed on the email alert, it was done with a small smile and advance knowledge of how it would light up the field. We were not wrong. Several of our local planners had strong feelings about the Department’s decision.
Perhaps the funniest reaction was one local planner who claimed that she just stared at her computer for a moment. “Is this someone trying to hack into my computer to make us miss the deadline?” she thought to herself. Others had different reactions. “I was shocked when they changed that. I have never seen that,” said one.
In our follow up survey, we asked if the changes affected local projects. Most (20) said it affected them positively, only 6 said it affected them negatively, and 9 said it made no difference. Most seemed to think it was “the right decision at the wrong time,” citing the need for longer page limits, but frustration at how late the changes were made.
For example, many reported that the initial shortness of the MOU prevented them from working with as many local partners as they wanted because they did not have enough room, but that by the time the change was made the MOU had already been signed. “The original shortness of the MOU and narrative left some ill feelings,” said one. “Some people said, ‘nowhere in this grant application am I mentioned.’ “
For most planning groups, the narrative change merely resulted in narratives that were 5-10 pages longer than they would have been because they no longer needed to go through a final edit to get down to 40 pages. At least two said they were sorry the change had not been made earlier because they wanted to address the invitational priorities but felt they could not afford to give them any valuable space. Others worried that the deadline and page length extensions would disadvantage them compared to others who were able to make changes, resulting in blame if they lost. “When you lose, someone will get blamed,” said one.
In most cases, the change in the deadline did not make much difference, although a few did take advantage of it and one claimed that it “saved our butts.” Perhaps the more significant date was not the new deadline, but the date the notice was received. Many local planners were planning to submit several days early and said that they heard of the change from UNCA just before submitting, well before hearing through official channels.
Looking back, at least one planner was suspicious of the motives behind the MOU and narrative changes. “What happened there? It smells of politics, like someone made a high level call. That was a game changer.”
Another was willing to give the Department the benefit of the doubt, but still believed there needed to be increased transparency. “They shouldn’t change their mind about stuff halfway through the process. But if they were hearing feedback, it reflects on them well that they did. But what were they responding to?”
The Department’s Web Submission Interface
One thing that scored well with our planners was something we never saw because we did not apply: the web interface for submitting the final application. Several said it was easy to use. “I loved that once you had submitted, it actually sent you a link to the whole kit and caboodle – very nice,” said one planner. “Pretty handy and effective,” said another.
Another said; “The time of uploading was stressed as being a potential issue, but the longest any of our uploads took was approximately 10 seconds.” This experience was not universal, however. One of our non-applicants fell into that category because of an upload stall that prevented submission. The applicants in question did not blame the Department, saying they knew they had been warned to submit ahead of the deadline, but they were understandably disappointed nevertheless.
Federal Grant Politics
We at UNCA must confess to perhaps a certain level of naivety about the federal grant writing process, not being experienced grant writers ourselves. In September 2009, we duly noted a story in The New York Times, which reported that the White House had instructed federal agencies to keep politics out of federal grant awards. We have reported on possible problems and biases in the peer review process, but never reported that we thought politics would be a major deciding factor.
Apparently these views are not widely shared. When we asked our planners whether they thought politics would play a role in which projects would be selected, most (23) said yes, while 4 more said it would play some role. Only 6 said it would not, while 3 said they hoped not. The rest were unsure.
“I would like to think that politics will not play a part, but I am not naive either,” said one. “Yes. We’ve been around,” said another. “No question on that really,” said a third.
“Can anyone believe that Chicago (President’s and Secretary’s hometown) will not receive a grant????” wrote another. One had a little fun with the same idea. “I don’t know. Chicago can’t get all the grants, can they?”
Interestingly, the perception of politics has already played a role in one city: New York. Several local planners (or potential planners) expressed a belief that no more than one project would be chosen for funding in New York City and that the project chosen would be the Harlem Children’s Zone. As a result of this belief, they did not apply. This is unfortunate because, in fact, HCZ did not apply.
Conclusions and a Final Recommendation
We based this series of articles on a series of questions that asked what worked, what didn’t, and how Promise Neighborhoods could be improved. We knew the request was being taken seriously when some of the comments were directed toward UNCA itself. These interviews and surveys were anonymous, but they were not anonymous to us. We appreciated both the kind words and the constructive criticisms.
Most of the comments, however, were properly directed toward the U.S. Department of Education. We will summarize those comments here and offer a final point of our own.
Overall, the Department seemed to come in for the most criticism for the lateness of the application release date and subsequent short turnaround time. The grant application was confusing to many, too laden with education lingo given the intended audience, and had some indicators that some considered odd. Whereas the Department asked applicants to state their visions, it needed to be more clear about its own.
The Department also got dinged for the initial shortness of the narrative and MOU page limits and received mixed reviews for mid-process changes late in the game. The FAQs were criticized for not always being responsive and for making new policy, rather than clarifying it.
Still, the good news is that all of these criticisms are easily addressable. Moreover, for the first time out, the Department scored some notable successes. They did very well with the webinars and associated PowerPoint, which were widely believed to be excellent. The FAQs were timely, despite the other criticisms. They also get credit for making a hard decision to increase the page lengths for the MOU and narrative, even though they must have known it would create an outcry. This suggests an admirable willingness to admit mistakes and put the success of the program above self-interested organizational or personal interests. It would have been easy to do nothing. They chose to take the hit.
One local planner said it well:
It was the first time out for releasing an RFP like this, so I understand the missteps and hope they are corrected for the next round. They should learn from mistakes not just for next planning grant round, but for the implementation grant, too.
Still, the widespread belief that politics will play a deciding role in the selection of Promise Neighborhoods grant winners caught us off guard. These suspicions came through in several remarks, including those expressing suspicions about the late changes to the MOU and narrative page lengths. Some also noted that the Department did not leave room for a comment period before publishing the grant application. “Please leave room for public comment before inviting applications for the implementation grants,” asked one.
These remarks lead us to make public a final comment and concern that we have held privately for some time.
We are strong believers in transparency and believe that increased transparency is the answer to these concerns. The Department has, to date, shown some willingness to be transparent when it comes to Promise Neighborhoods applicants, but this has not been evident with respect to the Department itself, at least not so far.
President Obama has committed to open and transparent government. And yet, every significant policy decision made by the Department on Promise Neighborhoods has been made behind closed doors, with little or no public knowledge or input. Promise Neighborhoods is a presidential initiative. It should meet the president’s standard.
Of all the recommendations made in this article, this is the one we most hope they take to heart.