Math – Applications for Living VI

Statistics in the news!!

The report (from the BBC) cites a study which says “55% of Syrians think President Assad should stay”.  The source is http://www.bbc.co.uk/news/magazine-17155349

This is a case where the journalist actually did a good job with the statistics.  The quote above comes from an online survey done with about 1000 respondents in the middle east and north Africa.  The article describes several statistical problems with the conclusion.  Among them:

• Syrians live (generally) in Syria … the report did not state how many were actual Syrians or lived there.  One reference in the report allows an estimate of about 100.
• One thousand is a modest number for a survey — this one covers an entire region.
• One hundred is statistically insufficient to measure the opinions of a country.
• Few Syrians have internet access; since it was an internet survey, the Syrians who did respond are not likely to be representative of all Syrians.

Curiously, our Math — Applications for Living class (Math119) yesterday covered a ‘rule of thumb’ for the margin of error for polls like this.  For most polls, the quick little formula 1/√n (reciprocal of the square root of the sample size) is surprisingly accurate.   For the 100 Syrians actually included in the results, the margin of error is 10% .. the true population parameter would be between 45% and 65% (most likely!).

The sad part of this story is that the original story on this survey did not provide this more complete context for the results.  Take a look at the BBC report for more information on that.  One can only hope that the bad use of statistics does not contribute to an already bad situation.

Join Dev Math Revival on Facebook:

Placement Tests – Valid?

Almost all community colleges use a placement test to identify students who need a developmental course.  Are these tests sufficiently valid for this high-stakes usage?

A recent publication from the Community College Research Center (CCRC) at Columbia College reports on a research study to examine this validity; the report is”Predicting Success in College: The Importance of Placement Tests and High School Transcripts” (CCRC Working Paper No. 42) and is available at  http://ccrc.tc.columbia.edu/DefaultFiles/SendFileToPublic.asp?ft=pdf&FilePath=c:\Websites\ccrc_tc_columbia_edu_documents\332_1030.pdf&fid=332_1030&aid=47&RID=1030&pf=Publication.asp?UID=1030

I’ve spent a little time looking through this study.  One data bit is creating quite a bit of interest … a statement that the two major tests (Compass, Accuplacer) have ‘severe error rates’ of 15% to 28%.  By severe error, they mean either of these situations:  (1) The placement test directs a student to a developmental course when the prediction is actually that they would pass the college level course or (2) The placement test directs a student to the college level course when the prediction is that they would fail.

The methodology in the study begins with the assumption that the placement test score measures degrees of readiness, not just a ‘yes’ or ‘no’ (binary) result.  Using data from a state-wide community college system, the authors correlate the placement test scores with whether students actually passed the math course (either a developmental course or college level) to create a probability value.  Since the colleges involved did not generally allow students with scores below a cutoff to take the college level course, they extrapolated to estimate the probability below the cutoff; a similar approach was done for a probability of passing the developmental course for scores above the cutoff.  For each placement test, the study includes between 300 and 800 students.

Using these models for the probabilities, the authors then calculate the severe error rate cited above.  The values shown were for mathematics — the ‘english’ rates for severe errors were slightly higher (27% to 33%).

Separate from that severe error rate, the study showed the ‘accuracy rate’ for each test for the courses; these accuracy rates reflect the pass rates for those above the cutoff and the failure rate for those below the cutoff.  These values range from 50% to 60% in math (for receiving a C grade or better).

The research also examined the relationship of high school performance to both this placement question and to general college success, and they conclude that the high school GPA is the single best predictor — even for predicting who needs a developmental course.

Several things occur to me relative to this study.  First of all, any measurement has a standard error; in the case of Accuplacer, this standard error varies with the score — for middle value scores (like 60 to 80), the standard error is about 10.  If a student scores 69 when the cutoff is 76, there is some chance that the score is ‘on the wrong side’ of the cutoff just due to the standard error of the measure.  In my experience, this standard error results in something like 10% of what the authors call ‘severe error’.  The main methodology to minimize this source of error is to have repeated measures — like having students take the placement test twice. 

Another thought … the report does not identify the math courses involved, nor the cutoff used.  Most results are given for “math 1” and “math 2”; the predictability of readiness is not uniformly distributed, and is more difficult when there are different levels of expectation (reasoning, abstraction, problem solving, etc) in two levels of courses.   Since the report does not identify which type of severe error is contributing the most to the rate, it is possible that the cutoff itself is what contributes to the severe error rate that is beyond the standard error

Though I doubt if many of us would, the use of high school GPA as a placement measure seems both awkward and risky.  We would need to replicate this study in other settings — other states and regions — to see if the same pattern exists.   Even if that result is validated, the use of a composite measure of prior learning raises issues of equity and fairness; applying this to individual students may produce varying results by student characteristics (even more than the placement tests).

The other thought is that a hidden benefit in this report is a comparison of the two primary tests (Accuplacer, Compass) for various measures of validity.  For example, the Accuplacer accuracy rates in math were somewhat higher than those for Compass.

Overall, I do not see this study raising basic questions about our use of placement tests. 

 
Join Dev Math Revival on Facebook:

Update about the New Life Project

Quite a bit of activity has taken place, relative to the New Life Project. 

The online community continues to be active.  The wiki site (http://dm-live.wikispaces.com) has a high level of use for this type of thing – consistently 20 to 40 different visitors per day.  This companion blog (https://www.devmathrevival.net) averages about 40 visits per day.  The New Life community has several pilots and implementations underway at this time; if your college is planning or doing one, it would be helpful to let me know (via email, or posting on the wiki site).

Some of us have submitted proposals for the Jacksonville AMATYC conference this November on our New Life work.  The AMATYC program committee is currently reviewing these – whatever the outcome, I appreciate our group’s willingness to share (and the work involved in sharing).

There is now enough activity on implementing New Life courses that textbook companies are developing products to support these courses.  I have had conversations with editors at Pearson, and at Cengage.  They would prefer that I not share details of their plans (since publishing plans are subject to changes for many reasons) … however, the companies are working hard to get materials ready.  Based on what I know, some materials will be ready for class testing this coming year (2012-2013), with regular materials published later (late 2013, early 2014).  I hope this information helps you plan as you consider whether you can implement a New Life model (MLCS and Transitions) at your college.

This activity related to the New Life model has also risen on the radar of companies that provide placement tests (especially Accuplacer and Compass).  Accuplacer now has a process to develop ‘customized placement tests’ in the computer adaptive model, based on a group identifying the content needed and a commitment from a set of colleges to use the customized placement test.  Many of us are using “placing into beginning algebra” as a proxy for “placing into MLCS”, and our pilot projects will provide some information whether this works in practice.  There is less confidence about using a proxy (“placing into intermediate algebra”) for the Transitions course, so this might be an area for us to consider in the near future.  Such a placement test is important because we do not want to assume (or require) that all students need to complete both MLCS and Transitions; we want to provide a mechanism for many students to place directly in to Transitions.  For us to proceed with a customized placement test, we would need to form a work group to coordinate with the company (including the gathering of the commitments from colleges to use the customized test).  We are probably not ready for this step, quite yet … though it might be appropriate to form the work group later this year.

Oh, in case you’ve forgotten the “New Life” course names – the first one (MLCS) is “Mathematical Literacy for College Students”, which provides the mathematics useful to all college students (including some algebra).  The second course is “Transitions”, which provides appropriate mathematics for students continuing in to college algebra, pre-calculus, and other courses at that level; Transitions is intended to be more symbolically-oriented than MLCS (which emphasizes numeric and graphical methods, with some symbolic).  You can go to the wiki site listed above for more information.

 
Join Dev Math Revival on Facebook:

Math — Applications for Living V

The class (Math119, called “Math — Applications for Living”) is now covering quite a bit of statistics, and I thought I would share a problem from yesterday’s class that incorporates ‘measures of typical values’ (aka ‘average’). 

So, here is the situation described:  “A small local company has 8 workers, and here are their hourly rates of pay:   $9, $9, $9, $10, $11, $18, $36. What is the average hourly pay?”

In this case, I had students work on this problem in pairs; they had directions for finding the mean, median and mode.  The big question was “Which average reflects ‘typical’?”

This was a good situation to show the weakness of the mean as an average or typical value; those outliers create false impressions.  The group actually thought that the mode was the best average because 3 workers had this pay … even though it was the lowest.  Numerically, the median was the best and we talked a bit about the pros and cons of each average. 

Essentially, this work on the ‘average’ supports the cynical statistician view of the world — we don’t have the answer, all we have are hints at something that might (or might not) be true.  Fortunately, this same class gave a chance to talk about distributions of data, and begin ‘distributional thinking’.  The students got the idea that we should try to represent a set of data with one number.

Some students in class had already noticed that the median was used for some things (like home prices, and the net wealth discussion — see http://www.pewsocialtrends.org/2011/07/26/wealth-gaps-rise-to-record-highs-between-whites-blacks-hispanics/).  It was also clear that the word ‘average’ used so often does not state which one — too often, it is the mean (as in ‘the average number of televisions per household is 2.4’).

The class is going to move on to other statistical topics, some of which have more exciting uses in life.  The one above might be of interest, or at least be enjoyable to read.

 Join Dev Math Revival on Facebook: