Online Homework … Research

We’ve been talking, as a profession, about online homework systems.  To some, some of these systems are the ‘best thing since sliced bread’.  On the other hand, some basic ideas from learning theories suggest that the benefits may not be uniformly positive.

A general source, summarizing various research, is the MDRC report  Unlocking the Gate: What We Know About Improving Developmental Education  http://www.mdrc.org/publications/601/full.pdf  .  On page ES-7, the report states:

                     Little research has been conducted on these strategies despite their popularity, and the research that exists shows mixed results.

In general, we can trust that a report from the MDRC is valid and reliable.  I thought I would look for some additional research, in case you are interested.

http://editlib.org/p/32318

Laurie Lenz
“This study compares student outcomes in multiple sections of a required mathematics course. The sections differed only in the homework delivery method: WBH, PPH, or a combination of the two. Among other results, it was found that students were more likely to attempt and to receive higher homework grades with WBH than with PPH. However, the study found no significant difference between the exam grades of students in the WBH sections and the students in the PPH sections.”
[WBH means web based homework; PPH means paper & pencil homework]

http://www.editlib.org/p/33222

David Shane Brewer, Kurt Becker

“This research compared the effectiveness, in terms of mathematical achievement, of online homework to textbook homework over an entire semester for 145 students enrolled in multiple sections of college algebra at a large community college. A quasi-experimental, posttest design was used to analyze the effect on mathematical achievement, as measured by a final exam. The control group completed their homework using the textbook and the treatment group completed similar homework using an online homework system developed by the textbook publisher. All class sections followed a common syllabus, schedule, and homework list and completed a common, departmental final exam. The results of the research found that while the treatment group generally scored higher on the final exam, no significant difference existed between the mathematical achievement of the control and treatment groups. When students were divided based on incoming math skill level, analysis showed that low-skilled students who used online homework exhibited significantly higher mathematical achievement than low-skilled students who used textbook homework. Exploratory analysis also showed that more students with low incoming skill levels and more repeating students received a passing grade when using online homework than did their higher-skilled, first-time counterparts, although the differences were not significant.”

http://iisit.org/Vol6/IISITv6p285-298Buzzetto556.pdf

Nicole Buzzetto-More and Ojiabo Ukoha

“The results of this study were mixed and marked by high levels of neutrality; however, the findings did indicate that most students felt that the system was easy to use, a valuable learning tool, successful at having helped them to learn course concepts, and an aide that helped them to perform better on their assignments. At the same time, most student responded that they were not satisfied the system. The analysis of the student performance data noted a significant decrease in student withdrawal rates and a marked increase in pass rates for the course under consideration.”

 As you may have discovered, this research is not easy to find.  If you search online for research for online homework systems, most of what you see will be marketing materials from publishers of such systems with an occasional ad-hoc summary of what happened in a class or course — which is not research.  Research strives to answer a question in a way that can be duplicated and validated, and the marketing & ad-hoc reports do not qualify; they might raise valid points, but do not show what the results are relative to online homework systems.

Here is a list of online articles for this research:

http://jolt.merlot.org/vol7no4/baugher_1211.htm                              

 http://editlib.org/p/32318                                                       

 http://coe.ksu.edu/jecdol/Vol_6/pdf/theeffect.pdf

http://academics.georgiasouthern.edu/ijsotl/v4n1/articles/PDFs/Article_DoornJanssenOBrien.pdf                     

http://jolt.merlot.org/vol5no4/hodge_1209.htm

http://iisit.org/Vol6/IISITv6p285-298Buzzetto556.pdf                     

http://www.editlib.org/p/33222                                        

http://www.ifrnd.org/JEVR/1(3)%20June%202011/Enhancing%20the%20Study_of%20Business%20Statistics.pdf

http://www.math.umt.edu/tmme/vol3no2/TMMEvol3no2_SaudiArabia_pp176_183.pdf

 
Join Dev Math Revival on Facebook:

Evidence Based Decisions

How do we prevent ‘evidence based decisions’ from becoming ‘evidence constrained decisions’?

First, let’s get clear on what ‘evidence based decision making’ is about.  Primarily, the idea is to apply evidence from the scientific method to decision making.  This is the definition given at http://en.wikipedia.org/wiki/Evidence-based_medicine ; much of the current push to use this method in education comes from improvements in medicine due to using scientific evidence as a basic methodology.

The idea is to base decisions on the evidence, when appropriate evidence is available.  Remember that we are talking about scientific evidence — which is a stronger standard than ‘data’.  The scientific evidence provides a connection between a practice or treatment with the outcomes (usually stated as a probability or odds).  Sounds good, doesn’t it?

Well, in education, there are difficulties in getting scientific evidence.  We have tons of data, which are raw measurements organized in some manner; however, this has little to do with scientific evidence.  Most commonly, we have either before and after data relative to some change; sometimes, we have data from two groups under different treatments … data on the outcomes, without data on other variables that we suspect have an impact on the outcomes. 

Scientific evidence does not come from one set of data.  After one set of data suggests, scientifically, that we have reason to believe that this treatment results in a change in the outcomes, this hypotheses gets tested by replication — done by different practitioners.  The idea of scientific evidence is that we achieve something close to an empirical proof that we have a cause and effect relationship — not just a one-time correlation.

I can not resist bringing up one of my favorite oxymorons — “data based decision making”.  Data is simply organized measurements; no decisions can be made based on data, because data is not evidence of anything.  I use brand X gasoline one week, and the next week I use brand Y — and get 10% better mileage … which means nothing; this data just means that I get slightly different outcomes, nothing else.  I normally find the phrase ‘data based decisions’ to be used as a cover for a hidden agenda.

Back to evidence based decisions … as mathematicians, we are all scientists; we understand the power of research — and it’s limitations.  The presence of evidence (in the scientific sense) suggests better courses of action (decisions) to the extent that the probable outcomes are ‘likely’.  The presence of evidence does not determine the best decision … wise people still need to evaluate the current situation and apply their understanding of the evidence.

What do we do when there is no scientific evidence relevent to our decision?  Are we constrained by the evidence available?  Even in medicine, with its superior collection of evidence, decisions are not constrained by evidence.  We should be guided by the evidence we have, and use our wisdom combined with our understanding of the outcomes desired to determine the best available decision.

Relative to mathematics education in colleges, I would present these observations:  We have large bodies of evidence about learning which can (and are) being applied to our courses.  We often mistake data for being evidence, and mistake reporting data for research, and this has led to some dramatic failures (and some less dramatic).   When we do remember the distinction between data and research, we tend to skip the step of ‘replication’ before announcing a conclusion; this has led to cynical colleagues and a skeptical public.

If we do not understand what the word ‘evidence’ means, who will?  Certainly not external forces such as politicians.  We need to be much better at articulating what we are basing a decision on, and clearer at describing results.  We need to focus on our shared values, and use them to describe the desired outcomes.  We need to focus on our wisdom, to provide guidance in the absence of evidence.

Join Dev Math Revival on Facebook:

Reducing the Number of Developmental Enrollments

It seemed like a good idea to share some good news, perhaps news that will inspire all of us.

El Paso Community College (EPCC), an Achieving the Dream (AtD) college, identified a major goal of reducing the number of developmental enrollments — not just in mathematics, though that is generally the largest area.

By implementing a systemic method of building readiness for college, EPCC reduced the percent of incoming students placed in the lowest level of mathematics from 33% to 22% … with the percent placed at the highest developmental math course rising from 30% to 41%.  This 10% shift in placement represents a two-course range (low to high), and potentially saves about 1000 student-semesters (courses needed).

How did they do it?  In typical AtD fashion, EPCC was deliberate about the entire process.  Two primary components of the process were the inclusion of the local school districts (in a ‘blameless environment’) and the nearby university (University of Texas El Paso, or UTEP).  During high school, students completed a joint application to both UTEP and EPCC and learned about the placement tests — before taking the placement tests. 

Based on this ‘early placement testing’ students were then given the opportunity to refresh … including a summer bridge program.  Information is shared between EPCC, UTEP, and the student’s high school.  Very impressive!!  For further details, see http://www.achievingthedream.org/Portal/Modules/e080ac9d-266b-45ad-b887-e1e60950cf59.asset?

Could your college develop a similar initiative?  I know that I want my college to do so!!

Join Dev Math Revival on Facebook:

Standards Based Reforms — What Research Says

The United States has seen a number of ‘standards based reforms’ over the past 20 years or so. Many of these deal with school mathematics, though a few of us in the college environment have worked towards a similar process. The most recent effort is the “Common Core Standards” (http://www.corestandards.org/ ), which is the highest profile effort yet.

The Rand Corporation published a report, in 2008, called Standards-Based Reform in the United States: History, Research, and Future Directions (online at http://www.rand.org/content/dam/rand/pubs/reprints/2009/RAND_RP1384.pdf).  I was impressed by some of their observations.

First, “Standards Based Reform” is usually implemented as “Test Based Reform”.  The point here is that content and pedagogy reflects a testing emphasis such that the actual standards are secondary — the tests (such as those used for No Child Left Behind, NCLB) take on the primary importance.  Behind this is a tension you will understand: Standards, by themselves, produce very little change.  “Aligning” testing to the standards is very common, and very understandable, as a method to create change.  Change is not always progress, however.

Second, high-stakes testing with sanctions ‘distorts’ teaching practice; as you’d expect, teachers focus more on preparing for the test when there are sanctions involved.  In general, most of the current testing involves sanctions of some kind such as NCLB or state-level impacts.  Since tests must, by design, address small subsets of the larger domain of knowledge described by the standards, the result tends to produce students who can perform better on the tests connected to the sanctions compared to other measures of their knowledge.  Specifically, they do not do relatively as well on our college placement exams. 

Third, the report goes back to a critical document that describes 4 categories of standards … and also analyzes the track record of some specific efforts.  A shot blog post is not an appropriate venue to report on these comments (I don’t want to inflict a journal-length article on you 🙂 ).

Although community colleges have not faced the standards based reforms and tests with sanctions directly, we deal with the consequences of these efforts.  Some policy makers assume that the “developmental math problem” will go away once the standards are implemented (like the Common Core).  The Rand analysis provides some insight into why the problem is not that simple; we should assume that our problem might change in the next 10 years … not that it will go away.

Join Dev Math Revival on Facebook: