May 01, 2015
Student success is goal of ‘Early Warning’ program
CARBONDALE, Ill. – College can be a daunting environment for freshmen, who may struggle with demanding class work, time management issues and the simple trials of being out on their own for the first time.
For university officials, the challenge, then, becomes quickly spotting the students who are having trouble and getting them the help they need to be successful.
Researchers at Southern Illinois University Carbondale are using mathematics and modeling to build a program that can spot trouble a long way off – as early as the third week of classes – and reach those students quickly with assistance. And soon the program, which started as a pilot program in a few key mathematics classes, may spread to many more classes across all colleges on campus.
The Early Warning program uses a student’s scores on a diagnostic test, homework, quizzes and tests – filtered through and weighted by a mathematic equation – to reliably spot problems early. The program, which quietly began three years ago, has become more formalized as researchers have figured out how to use it to make reliable predictions early each semester.
The program has the potential to greatly increase student success in key classes, as well as increase retention rates campus-wide.
“This program allows us to identify students who are at risk of failing a key foundational course very early while there is still time to intervene, to provide assistance as needed and to turn around the student’s fate in the course,” Susan Ford, acting provost and vice chancellor for academic affairs, said. “Passing key foundational courses is critical to going on to success in the first year, success in the major, and ultimately to successful graduation.”
Gregory Budzban, professor and chair of the Department of Mathematics in the College of Science, said early detection can lead to much better outcomes.
“Certain early warning systems wait until the end of the first semester, and then simply take a look at grades and other indicators,” Budzban said. “But we thought this was often too late. Often by that time, the student might be on academic probation, and once they’re on probation there’s a whole wealth of problems that come along with that, financial aid issues and such.
“So the question for us really became how early can we see this starting. And our work is a result of us trying to figure out how we can identify these students who need our support to keep them from getting on probation in the first place,” he said.
Budzban began gathering dating in 2012 from students enrolled in Math 108 (college algebra), Math 107 (intermediate algebra) and Math 101 (contemporary mathematics).
Math 108 and 107 are so-called “gateway” courses for students on the STEM track. STEM stands for science, technology, engineering and mathematics. Math 101 fulfills the mathematics requirement for non-technical field majors, and is therefore critical to those students’ success, as well.
“Math 108 is what we call a gateway course for STEM majors, for instance. Students can’t move up unless they get a C or better,” he said. “These are serious, stressful courses. So we were looking at being able to take limited resources and direct them to where they’re going to have the most success.”
To simplify things further, the researchers also came up with a color-coded status system to assign students a warning level. “Green” students have a C-plus or better while “yellow” students’ grades range from D-minus to C-plus. The color “orange’ means the students’ grades were between F-plus and D-minus while students coded “red” were below that.
At key points during the semester, Budzban creates reports using the parameters and formulas that help predict how the student will finish the class. The reports include raw data such as students’ names, as well as homework, quiz and test scores.
The researchers focused on three factors that appeared to have the most predictive value: preparation, motivation and demonstration.
A student’s preparation score is based on a diagnostic test they are given early in the semester. The test determines the student’s mathematics skill level as they enter the classroom. Researchers assigned this score a 25 percent weight.
A student’s motivation score is based on day-in and day-out work in the classroom, such as homework and quiz scores. The scores give an indication as to the student’s engagement in the classwork, Budzban said. Researchers also assigned this score a 25 percent weight.
Lastly, the demonstration score is based on the student’s test scores, to which researchers assigned a 50 percent weight.
“In other words, they have to show us they’ve learned something,” Budzban said.
In fall 2013, the researchers decided to go a step further and get the information they were seeing into the hands of students, faculty and college deans. They continued gathering information at the same time and began identifying important trends.
For example, the researchers found 87 percent of those students who were “green” by week three ended up succeeding in the class, meaning they receive a least a C for the semester. They also found that 95 percent of those who were “green” by the eighth week of class -- halfway through the semester -- were successful, as well. This compared with the 60 percent overall success rate.
At the same time, just 17 percent of those who were “red’ by week three ended up turning things around enough to earn at least a C for the semester. And by week eight, 91 percent of those “red” students ultimately ended up being unsuccessful.
“So we could see pretty much, at least in those two groups, who was probably going to make it and who was not,” Budzban said. “So the predictive value of week three as to the students’ final grade is really, really strong. And this kind of modeling, looking at students and how they moved in and out of these groups as a dynamical system, this is something no other university in the country is doing.”
The fate of the “yellow” and “orange” students, however, is more difficult to predict. So, the researchers decided in fall 2014 to focus on “yellow” students in an attempt to move them up a notch early in the semester.
“Our data by this time was saying that getting green by week eight is the pathway to success,” Budzban said. “So we have to get them green by midterm and we concentrated on the ‘yellow alert’ students, because their status sort of straddles the boundary between a C and a D and by concentrating there, because we have limited resources, we felt we would get the most bang for the buck.”
With a couple semesters of data under their belts, the researchers felt more comfortable stepping up their interventions with students who were on the bubble. Campus leaders came up with a new set of protocols aimed at communicating with students who need the extra help and attention as identified in the new program.
“We have great buy-in by deans and faculty in the colleges of science and business, as well as academic counselors and retention specialists,” Budzban said. “We’ve involved housing, too. In some cases, we went and actually knocked on their doors at the residence halls to see why they had dropped off the face of the earth. And in some cases we found the student was just having a serious problem in their personal life.”
By week three of that semester, 298 students were classified as “green” with another 132 “yellow.” Given the previous fall semester’s data and the probability of students transitioning from one group to another, officials predicted 299 “green” students and 118 “yellow” students by week eight (the mid-point of the semester).
Instead, after the new intervention protocols were in place the green group grew by more than 40 students, a large proportion of them coming from the yellow group. By comparison, in fall 2013 prior to the new intervention protocols, 32 percent of the students moved from yellow to green by midterm. In fall 2014, 42.5 percent moved from yellow to green by midterm.
Overall, the top two categories – green and yellow – increased from 70 percent of the total to 85 percent, meaning far more students were in position to continue on with their chosen field of study.
The program also managed to retain the 95 percent success rate for green students at midterm. “It wouldn’t have done us any good, otherwise,” Budzban aid. “So in that semester, if you were green at week three the data says we were able to get 91 percent of them through successfully.”
Harvey Henson, retention and outreach assistant dean for recruitment in the College of Science, said faculty interaction is on the front line, and Early Warning gets faculty the information they need to be more effective.
“It’s about identifying the problem in the classroom and then bringing in the advisement to help,” said Henson, who helped devise student contact protocols to go along with Early Warning. “We do what we can when we look at the student holistically. We love what we do and we love seeing students be successful. We want them to know they are not going through this journey alone and that SIU is a really caring institution.”
Budzban said there is still much to do, such as reaching out to students who find themselves struggling mightily in the orange and red categories.
“We haven’t yet been able to do much with students in the red, but we haven’t really tried yet,” he said. “But our overall success rate is way up, up almost 20 percent, over the last few years when we’ve been working on this approach.”
Other colleges on campus have begun discussions on how to translate the Early Warning program approach to their classrooms. In the long term, the goal is adapting Early Warning to other foundational and gateway courses in majors with an eye toward helping at-risk students succeed.