The Placement Test Disaster ?

For an internal project at my institution, I’ve been looking at the relationships between Accuplacer test scores, ACT Math scores, and performance in both developmental and college-level courses.  Most of the results are intended for my colleagues here at LCC.  However, some patterns in those relationships are important for us to explore together.

So, the first pattern that is troubling is this:

Students who place into a pre-calculus course based on their ACT Math score have lower outcomes than those who place based on the Accuplacer “College Level Math” test … and lower than those who needed to take intermediate algebra before pre-calculus.

We use the ‘college readiness’ standard on the ACT Math test of 22 (see https://www.act.org/content/act/en/education-and-career-planning/college-and-career-readiness-standards/benchmarks.html ).  The pattern in our data for the ACT Math is similar to some references found at other institutions … though we tend not to talk about this.

Of course, the use of an admissions test (ACT or SAT) for course placement is “off label” — the admissions tests were not designed for this purpose.  We tend to use the ACT option for placement in response to political pressure from administrators (internally) and from stakeholders (externally), and sometimes under the guise of “multiple measures”.  The patterns in our data suggest that the ACT Math score is only valid for placement when used in a true multiple measures system … where two or more data sources are combined to create a placement.  However, most of us operate under ‘alternative measures’, where there are different options … and a student can select the highest outcome if they wish; alternative measures is guaranteed to maximize the error rate in placement, with a single measure placement test almost always providing better results.

The second pattern reflecting areas of concern:

The correlations are low between (A) the ACT Math and Accuplacer College Level Math test, and (B) the Accuplacer Algebra and Accuplacer Arithmetic tests.

The second combination is understandable, in itself; the content of the Algebra and Arithmetic tests have low levels of overlap.  The problem deals with our mythology around a sequence of math courses … that the prerequisite to algebra is ‘passing’ basic math.  Decades of our research on algebra success provide strong evidence that there is little connection between measures of arithmetic mastery and passing a first algebra course.  In spite of this, we continue to test students on arithmetic when there curricular needs are algebraic:  that is a disaster, and a tragedy.

The first ‘low correlation’ (ACT Math, College Level Math) is not what we would expect.  The content domains for the two tests have considerable overlap, and both tests measure aspects of ‘college readiness’.  As an interesting ‘tidbit’, we find that a higher proportion of minorities (African American in particular) place into pre-calculus based on the more reliable College Level Math test compared to majority (white, who have a higher proportion placed based on the ACT Math) — creating a bit of a role reversal (whites placed at a disadvantage).

Placement testing can add considerable value … and placement testing can create extreme problems.  For example, students with an average high school background will frequently earn a ‘college ready’ ACT Math score when they have too many gaps in their preparation for pre-calculus.  A larger problem (in terms of number of students) comes from the group of students a bit ‘below average’ … who tend to do okay on a basic algebra test but not-so-good on arithmetic, which results in thousands of students taking an arithmetic-based course when they could have succeeded in a first algebra course (or Math Literacy).

Those two problems are symptoms of a non-multiple-measures use of multiple measures, where alternative measures allow students to select the ‘maximum placement’ while other measures (with higher reliability) suggest a placement better matched for a success situation.

As a profession, we are under considerable pressure to avoid the use of placement tests.  Policy makers have been attacking remediation for several years now, and more reasonable advocates suggest using other measures.  The professional response is to insist on the best outcomes for students — which is true multiple measures; if that is not viable, a single-measure placement test is better than either a college-admission test or a global measure of high school (like HS GPA).

And, all of us should deal with this challenge:

Why would we require any college student to take a placement test on Arithmetic, when their college program does not specifically require proficiency in the content of that type of test?

At my institution, I don’t think that there are any programs (degrees or certificates) that require basic arithmetic.  We used to have several … back in 1980!  Technology in the workplace has shifted the quantitative needs, while our curriculum and placement have tended to remain fixated on an obsolete view.

 Join Dev Math Revival on Facebook:

Talking About Equity as an Avoidance

My department has begun a process which will (hopefully) lead to meaningful and sustained improvements in our equity picture.  Current, and historical, data makes it clear that our program is not serving all groups adequately.  Black students (aka “african american”) almost always have a pass rate significantly lower than other groups, after accounting for their level of preparation.

I am very pleased with my colleagues and their willingness to spend time working on a problem which involves some discomfort … it’s not always easy to talk about race and equity.  Much of our initial discussion focused on our point of view and problems that make sense to us … phrases like “student skills”, “role models”, and “tutoring” we very common.  “Compassion” and “empathy” were also used.  These are all good thoughts, but tend to focus on the surface and symptoms.

However, I am sure that our conversation will need to progress to deeper levels of understanding.  One reason to believe this is that this conversation has occurred hundreds of times in other institutions and organizations without producing an accepted basket of ‘best practices’ for eliminating the inequity as we generally would like.

One perspective that might help our profession actually make progress on this comes from Danny Martin (University of Illinois at Chicago).  Dr. Martin delivered a talk entitled “The Collective Black and Principles to Actions” (available at http://ed-osprey.gsu.edu/ojs/index.php/JUME/article/view/270/169) .  The ‘Principles to Actions” part of the title refers to the 2014 publication by NCTM of that name.  The “collective black” in the title refers to a way to understand a social structure in the United States.

A quote from near the end of that article is:

Does this document represent, symbolically and in spirit, the kind of disruptive violence to the
status quo that can move the last to first?  Can it truly help in improving the collective conditions
— not isolated examples of success — of African American, Latin@, Indigenous, and poor
students? By success, I do not mean slow growth and incremental gains.

The “disruptive violence” in this quote might bother some readers.  Remember that Dr. Martin is speaking of social institutions, not a personal philosophy of political change.

I think Dr. Martin’s point, perhaps shared by Dr. Martin Luther King as well, is that incremental change and “stuff around the edges” will not produce meaningful changes at the level necessary.  Our  problems are too well established in the existing structures, and even in the vocabulary we use to describe ‘the problem’.  For example, millions of white people have had “compassion” and “empathy” for a wide variety of students (including the group ‘black students’ my department is focused on).

Here is a point … Perhaps “white people” only support working on “equity” when this work does not involve any change in the white power relationships and social structure.  Are we willing to share power and authority to reach the lofty goals we seek?

Perhaps we will find that reaching equity in our department depends upon fundamental changes in the  local community.  The urban schools have old buildings, few resources, and other significant challenges; this district is heavily ‘minority’ (black students in particular) … because our state allows “school of choice’, where THOSE WITH RESOURCES can take their students to a ‘better’ school in the suburbs.   Can ‘separate and sort of equal’ ever allow us to achieve equity in higher education?  [The local condition amounts to sanctioned segregation of schools, especially at the high school level.]

We are likely to encounter large-size problems in our work to eliminate inequity in our courses.  We have only begun the conversation, and I’m proud that my colleagues are willing to begin this journey.  Our success will likely involve changes that would have been difficult to imagine prior to beginning the process.  So … I appreciate your “moral support”.

Is your department ready to face the challenges of doing effective work to reduce inequity?

 
Join Dev Math Revival on Facebook:

A “Golden Age”: Forty Five Years of Dev Math, Part II

In my continuing account of a history of developmental mathematics, we are moving from the early 1970’s to the late 1970’s.  Although ‘dev math’ existed before the 1970s (the ‘origins’), my experience started then … and this period coincides with other shifts (such as the founding of AMATYC).  This post will look at the patterns of the late 1970s and how some of them impact us in 2017.

Faculty in mathematics, and observers, might assume that developmental mathematics has always been trying to justify its existence.  However, for the early part of this story, policy makers tended to ignore both the need for developmental mathematics and out outcomes.  Budgeting in this period would reward enrollment, and developmental math classes were both easy to populate with students and economical for the institution.

These conditions resulted in larger enrollments in our courses, which contributed to one aspect of a ‘golden age’:

Dozens of publishers actively sought authors and new textbooks.  Derivatives of these textbooks still dominate the book ‘market’ today.

One of these textbooks initially begun in this period is “Keedy/Bittinger”, and the “Lial/Miller” texts also began at this time.  Previous textbooks tended to be knock-offs of high school books, and now the focus was placed directly on the needs of our courses and students.  However, the content was still organized by typical topics in chapters like one would see in high school books, and this generally continues until quite recently.  The content was quite traditional and procedural; the innovations focused on the use in a ‘college’ course by adults.  This is when “workbooks” became popular, providing instructors with homework submission before the internet.

The current environment has focused on the price of textbooks.  I think it is interesting that in the 1970s the price of textbooks was just as high (relative to the “CPI”, for example) … and that the buyer got just the book.  Today, with prices a bit above the adjustment for CPI, the buyer often gets online access.  Clearly, perception is the most important issue in an economic decision like ‘buy a textbook’.  [Students also did not have any purchase options in the 1970s.]

As the enterprise of developmental mathematics expanded, some concerns developed around ‘proper placement’.  Since this preceded most of the technology we presume today, “checking prerequisites” was an enormous undertaking for an institution.  Many colleges  had been letting students enroll for courses based on the student’s perception about what was needed.  This period pre-dated the placement tests we are accustomed to, which led to another aspect of a golden age:

Many institutions invested resources in developing their own placement instruments.

In many institutions, this meant that math departments did some analysis of what students needed to know before a given class.  Likely, a majority of these efforts produced assessments very similar to the items on Accuplacer and Compass, with a focus on one type of error … not letting a student register for a class when the test indicated a high chance of not passing.  Some of these institutions were in New Jersey, where (a few years later) the items from these original institutional placement tests were incorporated into the New Jersey Basic Skills tests, which is where many of the Accuplacer original items came from.

The emphasis on avoiding a single type of error has been at the center of mathematics placement until the present, though forces are pushing us to move beyond this concern.  We have been so focused on avoiding “over placement” that we have a strong tendency to under place students — putting them in courses for which there is little need.  That pattern has left us open to external criticism, and lies at the core of the “Complete College America” attack on remedial mathematics.

Placing students has been more about “avoiding failure” in a higher course than with the question of the “best placement” for students.

The current efforts in true ‘multiple measures’ placement are aimed at answering the better question.

I think it is important to recognize that some of the institutional efforts at placement in this era were more sophisticated in their goals and more creative in the resulting assessments.  Many of this novel approaches were shared at the first few AMATYC conferences I attended a decade later.  However, almost all of these indications of diversity were overwhelmed later during the ‘systemic years’ (another period in our history, in the 1990s).  We have generally lost the institutional placement instruments, with a few surviving as supplemental devices used by individual faculty or specific courses

Obviously, this period I am calling “a golden age” was not such a good thing.  The trends begun here caused us to under-place millions of students, and also to use textbooks which presented high-school mathematics at a low level of learning.  However, this period saw growth and large investments by both institutions and publishers.

As we move from a 1970s ‘golden age’ into the 1980s, we will be describing the impact of “back to basics” in an era prior to any content standards in the profession.

 Join Dev Math Revival on Facebook:

The Origins: Forty Five Years of Dev Math, Part I

I’m getting somewhat close to the end of my career, and I expect that AMATYC 2017 is the last AMATYC conference that  I will attend.  Most likely, it is natural for people to contemplate the arc of history at this point (whether this arc bends towards justice is another question).  I will be writing a series of posts on the history of (my) developmental mathematics, which might be presented at a session in San Diego.

For me, the origins lie in a coincidence:  having ‘trained’ as a high school math teacher, I was unable to find a teaching job that did not involve moving.  I applied for a part-time job at the local community college, and in a fit of inexplicable errors, was hired.  The job involved supporting the operations of the college’s “Math Lab”, where several remedial math courses were offered in a self-paced, mastery format.   The time was the early 1970s.

In general, those remedial courses were intentional copies of K-12 courses from a short period prior to this time.  We had middle-school math (basic math), beginning algebra, geometry, and intermediate algebra; soon after I started, we began offering a metric system course, a desk-top computer course, and a sequence of two statistics courses (which had a beginning algebra prerequisite … quite ahead of its time).  The faculty in charge of the courses for students were, in general, current or former high school math teachers; familiarity with K-12 math was a high priority in hiring, and support for student success was not even considered.

The core of the ‘developmental math’ curriculum was the 3 course sequence aligning with grades 8, 9, and 11 .. basic math, beginning algebra and intermediate algebra.  At this time, the mode for a student’s high school math was ‘algebra I’, with a fourth of recent HS graduates never having had any algebra course.  Those on a ‘college-prep’ track certainly had more, but the community college policies were not targeted towards the college-prep students.

This was the time period when a pattern was started that still holds in many parts of the country:

Since most of the students graduating from high school had not taken ‘algebra II’, intermediate algebra is ‘college credit’ and often meets an associate degree requirement for general education.

The rationale for this policy lost its validity within about decade, as the majority of students began to graduate with algebra II credit on their transcript.  Colleges have been slow to update their general education policies to reflect fundamental shifts in HS course taking behavior.

In terms of “hot topics” in developmental mathematics education, it was all about two systemic features:

  • Curricular materials that required little reading and provided ‘clear’ examples with lots of practice.
  • Alternative delivery methods, including self-paced and programmed learning.

The first element reflected the high-school context for  the period prior to this … school textbooks were intended to be ‘teacher-proof’ (anybody could teach math), and the content was all about procedures to calculate answers in arithmetic and algebra.  That context has changed in a basic way, as the result of the teaching standards over the past 30 years (NCTM, AMATYC, etc).  Like the general education policy, math faculty have not altered the core focus of the curriculum; most current materials still focus on clear examples and lots of practice (though there is often more reading involved).

Our focus in alternative delivery methods, though cast in naive terms, was actually critical to trends that continue through today.  Most of us find it funny that ‘programmed learning’ was a “Thing”; the central idea was to have an assessment ‘every page’ and the student was ‘branched’ to a different next page, depending upon their answer.  In more recent times, this idea has been done in a more mature fashion with adaptive computer tutor designs.

The essential transaction that was being developed in these early days was “student — does math — correct OR recycles to re-learn it, repeat”.  Faculty had a role, but this role was not seen as the most essential role for student learning.  In contrast, much of our current professional development puts great emphasis on faculty interacting with students.  Although there is an obvious and valid basis for this emphasis, I wonder if perhaps we would be better off focusing more on the student interaction with mathematics.

A subsequent post will look at the period of a few years following this ‘origins’ time of the early 1970s.

 

Join Dev Math Revival on Facebook:

 

WordPress Themes