Table of Contents Title Page Juvenile Justice Policy and Problems, Policies, and
Program Performance in the States NEBRASKA-Portraying Juvenile Crime over Time MASSACHUSETTS- Evaluating Cops and Kids NEW MEXICO-Mixing Juveniles with Adults |
MASSACHUSETTS-Evaluating Cops and Kids From
"Commonwealth of Massachusetts 1997 Cops and Kids Grant Program Final Report,"
Massachusetts Executive Office of Public Safety. Problem Statement In 1996, the Executive Office of Public Safety (EOPS) in Massachusetts commissioned a study to evaluate the Massachusetts Drug Abuse Resistance Education (D.A.R.E.) Program, to identify the level of substance abuse awareness among youth, and to determine the nature of the relationship between police and youth. In conducting this evaluation, Northeastern University’s Center for Criminal Justice Policy Research administered 1,617 pre- and postprogram surveys to elementary students in five treatment schools and one control school. Surveys were also administered to educators and D.A.R.E. officers. The Northeastern study found that the local D.A.R.E. programs provide age-appropriate information about drugs, alcohol, and tobacco use to fifth and sixth grade students, who also develop a strong bond with their D.A.R.E. officers. The majority of students who completed the D.A.R.E. elementary school curriculum reported that they now know all that they wanted to know about the dangers of smoking, drinking, and drugs and that D.A.R.E. taught them different ways of avoiding drugs and alcohol. It also appeared that D.A.R.E. slightly increased students’ negative views of drugs, alcohol, and cigarettes. There were no significant increases or decreases in the use of tobacco products, alcohol, or marijuana for either the D.A.R.E. group or the control group. However, with regard to substance abuse and violence, as well as building relationships between youth and police, the study found that these issues were not being reinforced for middle school-aged youth. The study recommended that the Secretary of Public Safety implement new anti-drug and alcohol programs that had a community-wide focus and were integrated with existing substance abuse education efforts being taught in the school. It also recommended that new programs, geared to middle school- and high school-aged youth, should include a violence prevention component. The need for positive programming for middle school youth in Massachusetts was further strengthened following a February 1997 incident in Woburn, Massachusetts, where 14 children overdosed on muscle relaxants at a community dance. Looking for a quick high, the middle school-aged youth swallowed handfuls of the prescription drug Baclofen, unconcerned with the physical dangers that their actions posed. Of the youth who took the pills, all were unable to identify any viable consequences that they contemplated before ingesting between 10-55 pills. All of the children involved spoke of the need to belong to a group as a reason for participating in the event. In 1997, EOPS launched the Cops and Kids Program, a pilot program initiated in 20 state communities. Cops and Kids was designed to provide at-risk middle school youth (12-14 years old) with after-school programs during the hours between 3 p.m. and 6 p.m. Over the 1997-1998 school year, Cops and Kids communities established a variety of after-school programs that provided a safe, structured environment for youth. Building on the core principles of risk-focused delinquency prevention, EOPS sought to develop a program that would help curb youth drug abuse and delinquency by providing a healthy, after-school environment for middle school youth to interact with positive role models, while reaffirming the core principles of the D.A.R.E. program. The primary goal of the Cops and Kids Program was to reduce the incidence of youth substance abuse and criminal and gang involvement. Research shows that youth often spend their after-school hours in idle, unsupervised activities, which can sometimes lead to drugs and crime. The Cops and Kids Program initiated a healthy alternative to this practice by providing youth the opportunity for involvement in positive activities during after-school hours in a constructive, supervised environment. Therefore, youth remain safe during the critical after-school hours, and public safety is preserved due to the anticipated decrease in juvenile delinquency. In addition to this primary goal, in working with researchers, law enforcement officials, and policymakers in creating the Cops and Kids Program, EOPS identified the need for developing strong relationships between youth and law enforcement officers, as well as community partnerships. The Cops and Kids Program attempted to create an atmosphere conducive to favorable interaction between youth and police officers. The social bonding and relationship- building components were key protective factors that the Cops and Kids Program intended to implement. With local agencies working together and creating a uniform effort, youth were better served by newly found interagency communication and understanding of youth needs. EOPS challenged local police departments to develop comprehensive, community-based strategies intended to keep youth off the streets and prevent substance abuse and violence. To promote the success of the program and facilitate community cooperation, police departments were required to work directly with other community stakeholders in developing their community’s Cops and Kids Program. All Massachusetts cities and towns were eligible to apply for program funding if they met specified criteria. Through a competitive grant application process, local police departments were asked to develop a strategy that addressed juvenile delinquency. Although communities were provided discretion to devise their own program components, they were required to inventory their existing resources and programs that addressed the needs of youth and to conduct a needs assessment focusing on the targeted group and hours. Each strategy was to address prevention, intervention, enforcement/suppression, and reintegration. EOPS convened a Cops and Kids Advisory Review Board composed of law enforcement experts and policymakers who reviewed and scored each application. In July 1997, EOPS awarded $1 million of federally appropriated Edward Byrne Memorial Grant funding to 20 police departments to develop Cops and Kids programs. After one year of operation the Programs Division of EOPS performed a process evaluation and an impact evaluation based on data collected in the first year of operation. Purposes of Data The researchers collected two sets of data, one for each type of evaluation. The process evaluation data set was designed to document the actual operation of the Cops and Kids Program in its 20 locations and to identify areas for improvement in management and administration. The impact evaluation data set was created to contrast the actual performance of the program against the goals set for the program. The data would be used to inform decisionmakers about the effectiveness of program operation and its successes in attaining the stated program goals. The data would also use the first year of operation to establish a baseline for later comparisons with those programs that continued or followed in the future. Data Collection In the process evaluation EOPS staff were assigned to individual communities as grant monitors and were responsible for the oversight of their programs. Their goals were to determine the extent to which the programs were reaching their target population, whether delivery of services was consistent with the program’s design specifications, and what resources were being expended over the course of the program. To achieve those goals, the EOPS staff used several methods to monitor the individual programs and to assess service delivery. One method used was direct observation. Over the course of the grant year, the EOPS grant monitors visited their assigned programs at least twice. The monitors observed the youth as they participated in their scheduled activities. By directly observing the program in action, the monitors were able to assess how youth interacted both with each other and with the police officers and other adult staff. The monitors were also provided the opportunity to speak with youth and program staff regarding their experiences with the program. The monitors completed a Cops and Kids Site Visit Worksheet each time a site was visited. This standardized report was designed as a structured interview instrument and consisted of a series of questions that determined if the program was operating effectively. In interviewing program personnel, the grant monitors determined the program’s success in meeting its goals and objectives. If it was determined that the specified program components were not implemented, the monitors emphasized the requirement to adhere to the original program design. Any problems experienced in implementing the program were discussed, and technical assistance was provided, if necessary. On-site visits were also useful in examining the number of participating youth, officers, and volunteers. The EOPS grant monitors also used evaluation reports as monitoring devices. During the grant year, Cops and Kids Program administrators were required to submit three periodic evaluation reports that provided information about program implementation and participants. Specifically, the programs reported the number and characteristics (e.g., gender, race, age) of the youth served. Specific program components implemented during the reporting period were provided, as well as the number of youth participating per month and the average session time in hours for each of the components. All Cops and Kids administrators also submitted the Annual Progress Report required of all Byrne grantees, which detailed the program’s goals, objectives, program activities, performance indicators, and accomplishments. Finally, the monitors held information sessions in August 1997 and January 1998 to inform all grantees of what was expected of them in regard to data collection and submission of required reports. The second session required grantees to present information on their program’s successes and on obstacles encountered. Program administrators were provided the opportunity to interact with each other and to learn from the experiences of other programs. Although both sessions were helpful to program participants, the second session also proved especially useful to the EOPS staff, particularly when doing the process evaluation. To evaluate the impact and the effectiveness of the Cops and Kids Program, EOPS staff created surveys in both English and Spanish for youth participants to complete at the beginning and at the end of the program. To encourage honesty, participants completed the survey anonymously. In addition, persons outside of law enforcement actually administered the surveys. The majority of the questions were adapted from well-documented and tested survey instruments (e.g., the Centers for Disease Control and Prevention’s Youth Risk Behavior Survey). Although the Cops and Kids survey contained no questions regarding sexual behavior, grantee communities were required to obtain parental consent from student respondents. The surveys were designed to determine participants’ attitudes and behaviors related to the following:
The Cops and Kids Program served approximately 3,000 youth across the 20 sites; 1,629 youth completed the preprogram survey, and 1,243 completed the postprogram survey. A control group of 596 youth were surveyed before the program, and 249 were surveyed after the program. The large difference in completion rates for the pre- and postprogram surveys was attributed primarily to youths exiting the program in early spring to pursue alternative after-school activities, preventing them from completing the exit survey at the end of the school year. In addition, a relatively small number of juveniles were removed from the program because of inappropriate behavior, and some youth dropped out because they did not like the program. Furthermore, since the pre- and postprogram surveys were not matched, the same youth may not have completed the survey at both intervals. The postprogram survey included an additional section that measured the participants’ level of satisfaction with the program. Of the 1,243 youth completing the postprogram surveys, 67 youth completed the additional Cops and Kids Program evaluation form. This relatively small number was mostly due to the fact that many of the communities did not necessarily call their program the "Cops and Kids Program." Therefore, many youth participants did not recognize the name and did not complete the evaluation. Data Analysis Based on Purposes of Data As stated, the data collected for this project were of two kinds-process and impact. This section will detail, first, the report findings from the process data and second, the findings from the impact data. Regarding the process data, Massachusetts researchers found the following. Although program participation was not mandatory for the majority of programs, three sites did require certain youth to attend. Demographically, the Cops and Kids Program served equal numbers of girls and boys. The program served its target population of middle school students. White youth comprised the majority of program participants (80%), while the remainder of the population was primarily Hispanic and African-American. The number of officers involved in the programs ranged from 1 to 35; the number of civilians ranged from 0 to 34. The hours of operation, the days and times per week, and the frequency of sessions varied from program to program. All programs adhered to the requirement of providing after-school programming between the hours of 3 p.m. and 6 p.m. The researchers studied three consecutive periods of program operation during the school year. The number of student participants in the Cops and Kids Program in Period 1 was 3,191. The number of participants dropped to 2,099 in Period 2, and then rose to 2,574 in Period 3. As mentioned earlier, this variance may have been due to varying availability of other extracurricular activities and may have skewed the impact findings toward those who did not participate in such activities. Problems encountered in the initial offering of the program were similar across participating communities and included difficulties with transportation of students to and from the program, adequacy of food supplies, compensation and provision of custodial services, and lack of parental involvement in the programs. Interviews with program administrators indicated more specific and qualitative accomplishments that parents and students had identified, including better student motivation and tolerance, better classroom behavior, and better relationships with police. In addition, after starting the program, one middle school indicated no alcohol or drug incidents and fewer incidents of violence inside and outside the school. Regarding the impact data, the researchers found no notable changes in youths’ attitudes and behaviors toward school work, extent of parental supervision, self-esteem, general attitudes and behaviors, and how they spent their time after school. Situations in which the youth made contact with the police remained approximately the same from the pre- to postprogram survey as well. Concerning attitudes toward the police, most of the youths’ attitudes remained the same except for a 10% increase in the number of program youth who described their overall past experiences with police as good. The survey results appeared to indicate that youth drug use slightly increased from pre- to postprogram implementation. There were also slight increases in the number of juveniles who reported having engaged in delinquent behavior within the last year, and mixed changes in the number who reported having engaged in such behavior in the past month. However, due to attrition, it was difficult to conclude the program had a negative impact. For example, higher-risk youth might have stayed in the program throughout its entirety while other youth dropped out. Therefore, the postprogram survey recorded at-risk youths’ drug use and delinquent behavior while not taking into account those who had dropped out and had not used drugs or been delinquent. The secondary objective, providing educational activities in which youth could learn the dangers of drugs, crime, and gangs, was not accomplished as originally intended. It is important to note as well that approximately 20% of the youth reported that they quit when their homework was too difficult. The Cops and Kids Program succeeded in meeting its primary objective of involving middle school-aged youth in quality, supervised recreational activities in a safe environment during after-school hours. Prior to the program, certain youth were accustomed to interacting with school officials and police only when facing punishment for poor behavior. The Cops and Kids Program offered those students the opportunity to interact with these authority figures in a relaxed atmosphere. The Cops and Kids Program provided an opportunity for the students to gain a new perspective on police and educators, while at the same time affording police officers and teachers the opportunity to interact positively with at-risk youth. In addition, youth had the chance to participate in after-school activities that might not have been offered previously in their area. Although certain components of the Cops and Kids Program could have been strengthened, the Massachusetts researchers concluded that the overall implementation of the program had been a success. They noted that police officers, teachers, and youth all reported positive experiences with the program and had advocated for its continuation. Recommendations The Massachusetts researchers recommended specific alterations to the existing Cops and Kids Program based on the results of their process and impact data collection and analysis. While recognizing that some programs did focus on drug, gang, and violence prevention education, either informally or by set curriculum, the researchers felt that a more structured educational program should be required for each program in the future. To address the problem of youths quitting the program due to difficult homework, the researchers believed that the program could be strengthened by focusing on both the difficulties students were having and the unique ways in which students could be supported. Another recommended area for improvement was appropriate targeting and retention of eighth-grade participants. In many programs, fifth-grade participation increased while eighth-grade participation diminished. Reasons for the decrease in eighth-grade participants need to be explored and the proper programming adjustments made in each program. Eighth-grade students needed to learn important life skills and make positive connections to others to help themselves be successful in high school. The Cops and Kids Program offered these youths the chance, the researchers stated, to learn those skills and provided youth with positive role models who could guide positive behavior. The final area for recommended improvement was parental involvement and the expansion of parent education. One reason that there was a great need for after-school programs such as the Cops and Kids Program, the researchers believed, was that many parents were not at home during the hours of 2-6 p.m. However, according to the researchers, parents should be encouraged by program officials to participate in the program in some capacity, whether through planning, volunteering, or meeting with program personnel. Parent education was another aspect needing attention. Educational outreach to parents could be beneficial, in the researchers’ view, to parent/child relations, which in turn could positively affect their child’s behavior in general. By educating parents, more informed decisions regarding children could be made and more positive outcomes expected. Real and Potential Applications of Data The data derived from program monitoring and from participant interviews and surveys for the process and impact evaluations allowed for effective analysis of the Cops and Kids Program. Both the process data and the impact data provided a credible portrait of areas of effectiveness and areas needing improvement. As such, the study reinforced the findings of other studies of similar programs; that is, that there are minimal direct effects on substance abuse but substantial secondary effects of improving attitudes and relationships between youths and authority figures. Using these data, policymakers have baseline information for the evaluation of existing programs, and operational information for designing future extensions of the programs to new and existing areas. Moreover, the data collected for the Cops and Kids Program report highlighted a common problem of multisite programs over time; that is, the attrition of participants and the consequent possibility of skewed evaluation results. This report did an effective job of noting all of the qualifying information that must accompany analyses of such programs to provide the fullest possible picture of operations and outcomes to future policymakers. It also provides future analysts, both of this program and similar programs in and out of Massachusetts, with some guidance about designing evaluations. Combined with other evaluations in which participant attrition is frequent, this report will contribute to the effort to develop effective methodologies with positive and practical applications to field operations.
|