NARA invests nearly a third of its total budget to improve the bioenergy literacy of students and professionals with the assumption that an improved level of bioenergy literacy will translate into a knowledgeable workforce and a more engaged and enlightened citizenry who can help shape and contribute to a bio-energy/bio-products economy.
A particular challenge facing NARA, and other institutions, is how to measure progress and success in programs designed to improve energy literacy. In a paper partially funded by NARA and published in the Journal of Professional Issues in Engineering Education, NARA researchers Quinn Langfitt, Liv Haselbach and Justin Hougham present a case study that describes a novel method used to gage bioenergy literacy of high school student s participating at the Imagine Tomorrow competition.
Artifact-Based Energy Literacy Assessment Utilizing Rubric Scoring can be obtained here.
Defining energy literacy
The US Department of Energy defines energy literacy as, “an understanding of the nature and role of energy in the universe and in our lives. Energy literacy is also the ability to apply this understanding to answer questions and solve problems.” The authors of this paper describe bioenergy literacy as “ … an extension of energy literacy that acknowledges biologically based feedstock for energy production and recognizes the role that biofuels stand to play in future markets”.
Developing a tool to measure energy literacy from the Imagine Tomorrow competition
The Imagine Tomorrow competition challenges high school teams throughout the Northwest United States to seek new ways to support the transition to alternative energy sources. The event is partially funded by NARA and is held annually at Washington State University. The authors in this study designed a way to score the artifacts (abstracts and posters) generated by the high school teams that participate in the event. In doing so, they can measure the level of energy literacy change reflected in the artifacts presented during five consecutive years of the Imagine Tomorrow event (2008 through 2013).
Developing a scoring tool, or rubric, to measure energy literacy reflected in deliverables turns out to be a novel approach. When the authors searched for existing rubrics that measured energy literacy or biofuels, they found none available; however, many rubrics were available for topics such as energy, alternative energy and science literacy. So with little precedent available, they built a rubric adapted from an existing scoring tool used to assess senior design projects for civil engineering at Washington State University and from elements previously designed to evaluate science writing. The rubric measures elements in the abstracts or posters as either absent, pre-emerging, emerging, developing, competent, effective or mastering. Two versions were developed, one to measure energy literacy and the other to measure bioenergy literacy.
Evaluation results
To measure energy literacy reflected in the abstracts, over 500 abstracts submitted for five consecutive Imagine Tomorrow competitions (2008-2013) were assessed using the developed rubric for energy literacy. In order to test the reliability of the rubric, two raters were used. One rater was a graduate student in civil and environmental engineering the other a faculty member with a PhD in education. A reliable rubric would produce similar scores from multiple raters.
The results show that energy literacy scores from abstracts improved in the last two years of competition compared to the first three years. The score increase in 2013 relative to 2012 may reflect the coaching and workshops delivered to high school teams by the University of Idaho’s McCall Outdoor Science School (MOSS).
When abstracts were grouped into the individual competition categories (behavior, design, multidisciplinary, technology and biofuels), the biofuels category received the highest score. According to the authors, this result was not unexpected; abstracts for social sciences tend to provide more breath than depth; subjects such as biofuels and technology generally have a stronger focus on scientific energy concepts. Interestingly, there was no difference in scoring between schools that had multiple teams or repeat teams present verses first time participating schools. The authors speculate that experienced presenters may have focused more on appealing to judging criteria, which is not entirely focused on energy literacy.
The scores presented by the two raters exhibited a low level of consensus, even though both sets of scores correlated on showing similar trends over the years. The authors point out that a calibration session, in which the two raters discuss the rubric beforehand and draw a consensus for scoring, was not conducted for this study due to time limitations. Calibration sessions have been shown to increase rubric scoring reliability.
Posters from the 2013 Imagine Tomorrow event were scored. Energy literacy scores for posters were higher than abstract scores. This was not surprising as the abstracts were similar to project proposals and did not contain the more descriptive content present in the posters. The results showed that a high abstract score generally correlated with a high poster score for the same team. A team with a higher average grade level scored higher than younger teams.
Further refinement
Additional rubric refinements, the use of calibration sessions, and a larger sample size (as in the case of scoring the biofuel category) are needed to ensure that this approach to measure energy literacy is successful; however, this introductory look seems promising. When validated, this artifacts scoring tool could be used to test the effectiveness of teaching methods, curriculums and project types used to enhance energy and bioenergy literacy.