A National Collaborative Evaluation of El Sistema-inspired Music Education
The Longy School of Music of Bard College and WolfBrown have formed a strategic partnership to carry out a two-year study of the benefits of El Sistema-inspired programs for children in grades K-8.
A National Collaborative Evaluation of El Sistema-inspired Music Education
Contents
- Brief Overview of the Study
- Evaluation Tools
- More about the Study
- List of Participating Sites
- More about the Funders
- More about the Research Team
- Archive of Newsletters
- Additional Supporting Documents
A Joint Study by The Longy School of Music of Bard College & WolfBrown
Made possible by funding from The Buck Family Foundation and the Andrew W. Mellon Foundation
The past decade has witnessed a rapid proliferation of El Sistema-inspired music programs for children throughout the United States. This is not surprising, given the strong social mission of El Sistema and the musical and social effects reported from the program in Venezuela, its home country. Based on those reports as well as on preliminary findings from U.S.-based programs, there is much hope that El Sistema-inspired programs can build on the aspirations and assets of children and families, buffer the effects of risks like poverty, unsafe neighborhoods, and unequal opportunities to learn and develop, and build a wider understanding of the role that music can play in individual and community development.
But can this promise be realized? To do so requires that we build a compelling body of evidence about the quality of programs, the range of developmental outcomes they can foster, and the long-term outcomes that can be expected from different El Sistema-inspired models. This work is vital — if the claims made for these programs outpace supporting evidence, the El Sistema movement risks losing credibility and, ultimately, the engagement of families and the support of communities. Recognizing this challenge, the Buck Family Foundation wisely called for a national and systemcatic study of the benefits to children associated with participation in El Sistema-inspired programs.
In response to this request for proposals, Longy School of Music of Bard College and the arts research firm WolfBrown formed a strategic partnership to carry out a two-year study of the benefits of El Sistema-inspired programs for the children they serve. The study was further supported by a grant from the Andrew W. Mellon Foundation, who is especially interested in El Sistema-inspired programs that are associated with professional and youth orchestras. The research team invited proposals from programs across the country and formed a consortium of sites that represent a range of geographical and program structure diversity. Over the first year (2014-15) of intensive planning, a core set of common measures was collaboratively designed and piloted. In 2015-16, we gathered our first set of yearlong data using these measures, and worked with sites to design additional measures tailored to their specific programs. In 2016-17, the team collected a second year of data using the common measures and analyzed the full two-year set. The final report can be downloaded here.
The Evaluation Tools, including the common measures, are now available below to any El Sistema-inspired site who wishes to use them.
Between 2014-2017, the National Evaluation Study developed a range of measures and tools that are now available to the wider El Sistema field. The measures are adapted by the research team to be specific and meaningful for use with 3rd – 5th graders who participate in El Sistema-inspired music programs, and their peers in comparison groups from the same schools or after-school programs that do not participate in El Sistema programs. All tools and measures have been piloted and used successfully by colleagues at 12 different sites across a broad spectrum of U.S. locations and program formats. They are designed to collect data that measure student growth in three areas: Music Performance, Socioemotional Learning and Thriving in School and Program Settings. The Guide below links to the downloadable tools and measures available.
Music Performance
The Music Performance Measure assesses student growth in playing an instrument through an individual performance jury. It is designed to be used in the early fall and late spring of each academic year. Students enrolled in El Sistema music programs would participate in this measure, to be administered by teaching artists or music teachers in the program.
The Music Performance Measure Instructions give an overview of the entire measure and serve as a complete guide to administering each aspect of the performance jury.
The Rubric is used to rate each of two pieces that each student plays. It is adapted from a rubric used by the National Association for Music Education.
The Rating Sheets are used by each rater to record the assessment of each student’s performances on each of the excerpts.
The Music Performance Repertoire List is the guide for teachers to select the appropriate required excerpt that each student would perform, depending upon the instrument played and the number of years participating in the music program. These required excerpts come mainly from the Royal Conservatory of Music Development Program. Students also play a current excerpt from their program’s repertoire.
The Repertoire folder contains the printed musical scores for each of the required excerpts.
Socioemotional Learning
Socioemotional learning is tracked by asking to students to self report their experiences by filling out two separate booklet measures, one about themselves in school, and a second about themselves in their music program. Each booklet needs to be administered by a proctor. The booklets may be downloaded and printed directly.
El Sistema All About You as a Musician is the booklet of measures that looks at student socioemotional growth within the context of their El Sistema-inspired music program. It is filled out only by students enrolled in a music program and should be administered twice a year – once in early fall and once in the late spring.
Proctoring Instructions for the Music Measures Booklet is a document that guides the proctor and gives specific instructions for administering the “All about You as a Musician” booklet.
El Sistema All About You in Your School is the booklet of measures that looks at student socioemotional growth within the context of the academic classroom. It is filled out by students enrolled in a music program and by comparison group students who attend the same schools as the program students, but who do not participate in El Sistema music programming. This booklet should also be administered twice a year – once in early fall and once in the late spring.
Proctoring Instructions for the School Measures Booklet is a document that guides the proctor and gives specific instructions for administering the “All about You in School” booklet.
The Measure Sources chart lists each socioemotional domain that is tracked in the booklets and lists the original source from which the measure is adapted.
Thriving in School and Music Program Settings
The Staff Surveys and Documents file contains several measures that are filled out by classroom teachers, music program staff and music program teaching artists. It organizes the different documents and gives instructions for collecting grades for students. These documents are collected once a year – generally in June. Some are for music program students only and some are also for comparison students.
More about the Study
Rationale. The achievement gap has been an area of recognized policy importance in the United States since President Johnson’s Great Society initiative. More recently scholars and policy-makers have begun to direct their attention to the opportunity gap as a catalyst for the achievement gap. One aspect of the opportunity gap is limited access to a “well-rounded education,” which includes access to an education in the arts (Every Student Succeeds Act, 2015). Less affluent students are more likely to attend schools where the positions of full-time arts educators have been eliminated (Parsad & Spiegelman, 2012) and whose families are least likely to have the resources to pay for out-of-school arts instruction (Duncan & Murnane, 2015).
These facts are particularly troubling in light of correlational evidence that arts education in general (Catterall et al., 2012) and music education in particular (Cabanac et al., 2013; Gouzouasis et al., 2007; Vaughn, 2000) may benefit academic achievement. Among young children (ages 4 to 9 years), small-scale randomized-control studies have demonstrated that brief courses of music education can lead to increases in IQ (Kaviani et al., 2014; Portowitz et al., 2009; Schellenberg, 2004) though as others have noted (Schellenberg & Manakarious, 2012) these studies cannot speak to the effects of musical study that occurs over a period of years.
Moreover, while a number of researchers have argued that music education may catalyze neural plasticity (Schlaug, 2015), relatively few studies have examined the specific mechanisms through which music education might bolster academic achievement. There is some evidence that music education may foster aspects of socioemotional development that predict academic achievement, including school engagement (Eerola & Eerola, 2014) and persistence (Smith, 1992). However, evidence regarding music education’s effects on other socioemotional domains is either mixed, as in the case of self-concept (Dege et al., 2014; Portowitz et al., 2009), or absent, as in the cases of goal orientation and growth mindset.
Our study focuses on how music education may foster change in a set of socioemotional domains – school engagement, persistence, achievement motivation (self-concept and goal orientation) and growth mindset – and how this change may drive improvements in students’ academic achievement. In 2015-16 we collected data on socioemotional development and academic achievement from 280 students attending 10 El Sistema–inspired programs across the country and 225 comparison-group students matched by grade and school.[1] These programs feature high but variables levels of ‘dosage,’ ranging from 1 to 5 days of instruction per week throughout the academic year. Once data analysis is complete we will be able to assess whether rates of change in academic achievement and multiple potential mediators differ as a function of program enrollment. If positive rates of change for enrolled students exceed those for unenrolled students it would constitute promising initial evidence for the capacity of music education to benefit developmental domains that are known to support academic achievement. Because all of the programs included in our study disproportionately serve low-income students of color, the findings from our study will yield results that are generalizable to those students at the heart of the debate around inequality of opportunity, filling a gap in a research literature where most studies have been conducted with samples of majority-Caucasian middle- to high-income children.
Collaboratively with the core sites, we have also created a performance measure that looks at student growth in musical skills such as accurate playing of pitches and rhythms, phrasing and articulation, and tone and intonation. Many sites are using these measures in tandem with jury processes they already employ – or in some cases, this measure is jump-starting a jury system.
While individual El Sistema-inspired sites have embarked upon various of their own research studies – to date – this is the first national cross-section of sample students using common measures.
Research Questions.
The Study addresses the following questions:
- What skills and abilities does participation in El Sistema-inspired programs build in participants that could lead to their thriving as people, learners, members of they community, and possibly as musicians ? (How might the orchestra-driven development of interpersonal or intrapersonal skills (focus, persistence, growth mindset, etc).,, lead to thriving in other domains?)
- How is the acquisition of these skills and abilities affected by program design (dosage, duration, quality)?
- How does this process work for different children (of different ages, with different histories, in different schools and communities?)
Conservatory Lab Charter School
Dorchester, MA
Incredible Children’s Art Network
Santa Barbara, CA
Kalamazoo Kids In Tune
Kalamazoo, MI
Kidznotes
Durham, NC
New Jersey Symphony Orchestra CHAMPS
Newark, NJ
OrchKids
Baltimore, MD
The People’s Music School Youth Orchestras
Chicago, IL
Play On Philly!
Philadelphia, PA
Reno Philharmonic Association Kids
Reno, NV
San Diego Youth Symphony Opus Project
San Diego, CA
Yakima Music en Acción
Yakima, WA
Youth Orchestra of St. Luke’s
New York, NY
The Buck Family Foundation is funding the pilot year and two-years of the study. Their investment grows out of a deep interest in music and music education. They see the focus on El Sistema-inspired programs as a way to make a field-wide contribution to understanding the outcomes of intensive and sustained involvement in music.
The Andrew W. Mellon Foundation contributed additional funds to support the study at sites connected to orchestral programs, and to include graduate students as research fellows and assistants to the project – thus developing a next generation of leaders in the field.
More about the Research Team
The evaluation study is a strategic partnership between the Longy School of Music of Bard College and WolfBrown an international arts consulting firm.
Longy School of Music of Bard College
Longy School of Music of Bard College is a small, world-class conservatory that is undergoing a process of growth and radical change unprecedented in its 98-year history. The merger with Bard, finalized in 2012, has enabled Longy to expand its educational offerings, community engagement and research agenda and to access musicians, scholars, philanthropists and world leaders around the country and around the globe. In 2009, Longy’s Board of Governors made the strategic determination that its Conservatory program would drive the mission and focus of the School going forward. That mission is defined as empowering students to use music to make a difference in the world. Longy teaches its students to pursue a life time of community engagement wherever their musical paths take them. A key component of the curriculum is Longy’s Teaching Artist Program. Through this required program, students lead innovative music projects while gaining valuable entrepreneurial and teaching experience in schools, community centers, prisons, hospitals and elderly care centers, as well as Sistema-inspired programs in the Boston area (See http://longy.edu/) These values have led to a deep interest in and engagement with the leaders of FundaMusical, the umbrella organization led by Maestro Jose Antonio Abreu in Caracas, Venezuela.
Longy has actively supported the growth of Sistema nucleos around the country. In 2012, the school formed a three-way partnership with the Los Angeles Philharmonic and Bard College called Take a Stand. The first outcome of this partnership was the launching, in January, 2013, of a Masters of Arts in Teaching in Music (MAT), the only degree program in the world to offer graduate-level training in the principles of Sistema to musicians with the desire to respond to today’s unique educational needs and participate in the growing Sistema movement in the United States.
WolfBrown
WolfBrown is an international consulting group focused on knowledge building in the fields of arts and culture with a particular emphasis on collaborative projects that build the unique strengths of cultural organizations and help them report on their public value. In the field of music and music education, the firm has worked with major and community-based organizations including symphony orchestras, community schools of music, national service organizations, school systems and city governments. Drawing on the experience of two researchers, Dennie Palmer Wolf and Steven Holochwost, the firm specializes in research and evaluation focused on youth and families, with a particular emphasis on the effects of high quality, long-term participation in arts and cultural education. Many of their evaluations underscore the role that organizations, their staff, as well as youth and family participants, can play in designing, collecting, and reflecting on research findings. In recent years, Wolf and Holochwost have worked with a number of music-focused organizations (Carnegie Hall, From the Top, Community Musicworks, and three nucleos: Play on Philly, YOSAL, and the New Jersey Symphony Orchestra). Throughout this work, they have tailored existing and developed original qualitative and quantitative measures designed to capture the consequences of intensive musical experiences. Throughout this work, there is a strong commitment to seeing youth and families as the seedbed for cultural vibrancy.
The Team
Dennie Wolf holds an Ed.D from Harvard University and has taught at Harvard Graduate School of Education and Brown University’s Annenberg Institute. Dr. Wolf has grants from the Barr, Carnegie, and Spencer Foundations, as well as the National Endowment for the Arts. Wolf has collaborated with arts educators in communities as diverse as Boston, Chicago, Dallas, the state of Maine, the region surrounding Portland, OR on building city-wide and regional systems of arts learning. She has published extensively in the fields of symbolic and artistic development, arts education, and assessment in the arts and humanities. She is the 2013 national service to the field award winner from the National Guild of Community Arts Education.
Steven Holochwost holds Ph.D. degrees in Developmental Psychology from the University of North Carolina Chapel Hill and Music Composition and Theory from Rutgers University, where he studied with Charles Wuorinen. He has received awards and fellowships from the National Science Foundation, the American Psychological Association, the Society for Research in Child Development, the American Society of Composers, Authors, and Publishers (ASCAP) and the Renee B. Fisher Foundation. Currently, the team is working on evaluations of three distinctive Sistema-inspired nucleos: an emerging program at the New Jersey Symphony, Youth Orchestra Salinas, and an intensive program in West Philadelphia, “Play on Philly.”
Judith Bose holds a Ph.D. in Urban Education from the City University of New York Graduate Center and an MM in Vocal Performance from the New England Conservatory of Music. As the Director of Teacher Education and Educational Initiatives at Longy School of Music of Bard College she directed the innovative Teaching Artist Program in Cambridge and was the central architect of the curriculum for the MAT in Music at the Los Angeles campus. Having visited El Sistema sites in Venezuela, she is an active collaborator in the Take a A Stand partnership and professional development efforts, as well as served as a lead consultant with WNET and the Annenberg Foundation for an online professional development workshop series on El Sistema-inspired music education.
Archive of Newsletters
• Newsletter No. 10: March 2016
• Newsletter No. 9: January 2016
• Newsletter No. 8: November 2015
• Newsletter No. 7: October 2015
• Newsletter No. 6: September 2015
• Newsletter No. 5: June 2015
• Newsletter No. 4: April 2015
• Newsletter No. 3: March 2015
• Newsletter No. 2: January 2015
Additional Supporting Documents
• Advocacy Flyer (for sites to use with boards, in grant proposals, etc.) – November 2015 (.pdf)
• Grant Announcement from the Andrew W. Mellon Foundation – May 27, 2014 (.pdf)
• Bard College Institutional Review Board Approval – October 24, 2014 (.pdf)
• Consent/Assent Forms – English (.pdf)
• Consent/Assent Forms – Spanish (.pdf)
• Headline Findings from the U.S. National El Sistema Study (October 17, 2016)
- Overview of the Study
- The Study
- Modes of Participation
- List of Core Sites
- More about the Funders
- More about the Research Team
- Archive of Newsletters
- Other Downloads