In other words, test scores are misleading regarding competency of content.

http://www.sfgate.com/news/article/Test-Most-students-not-proficient-in-writing-3865546.php

My original thought(s) on the above article was to point out a ,”Houston, we have a problem” moment since so few students across the U.S. seem to NOT have quality writing skills.  Of course, it goes without saying this has been true for years and no one group/organization focused on this issue as test scores have been all the rage.

Students who have access to computers at home and regularly use them for assignments are more likely to be strong writers, a national exam suggests. But it also says just a quarter of America’s eighth- and 12th-grade students have solid writing skills. (sic – this is quoted from article)

And then I took on a very part-time position reviewing test questions for a company which sub-contracts to some other company and I began to realize  (yet again) other reasons why our students have poor writing skills.  Not only is it the dilemma of dumbing things down so students need only answer multiple choice and T/F questions, the questions have become significantly more about ‘test taking strategies’ than the higher levels of knowledge – application, synthesis, etc.  Test questions are not open-ended. There is an answer embedded (1/4 = 25% and 1/5 = 20% guess rate while T/F is 50/50 crap shoot!) and students just need to learn better test taking skills which is the older legacy the SAT, ACT and GRE  provided without a writing component.  These tests demonstrated the ability to think through questions.

I realize the SAT and ACT now have a writing component, which is obviously important if only 25% of students in the U.S. can write a five paragraph essay in Grade 12 and college clearly should expect more of a student.  Unfortunately, the five paragraph essay has become so pro forma just about anyone can learn the routine and write something, whether or not it is quality, the writing can meet the proficiency standard.

Great writers come about through reading and vice versa – they go hand in hand. When the brain has to spend time thinking ‘how’ to answer a question rather than the content of the question, it is already dumbed down.  Reading and interpreting a M/C, T/F test question does not lead to good writing skills, it leads to memorization skills and ‘trick’ techniques for understanding how test questions are written.  Having a computer is great – if it is used to read material – not play games and do other tasks with are multiple choice and T/F, yet easy to grade.

This issue has become most evident to me in working with foreigners who are writing test questions  abroad and want them ‘Americanized’ via grammar, etc. yet refuse to understand the quality of the questions are still poor, even when the grammar is corrected. This is based on the fact the questions do not rely on anything more than parsed out common information  and how well some student was able to memorize bits and pieces and think through testing logic.   The tests have little to do with the skills we would expect in a college classroom, workplace or even of students wishing to learn.

I have done this ‘job’ a few different times for different organizations. Each time the scenario is similar – questions are produced by foreigners and my job is to ‘grammatize’ the commodity so the business (American)  will think there is a new and better set of test questions in the question bank. Each and every time, the problems are the same, when you ask a question, making it ‘tricky’ does not make it better.  It proves the questions in the question bank are not promising.  It is the algorithms of how questions are selected and used which would make a good testing program for PRACTICE.

This is an example of what was returned to me when I could not understand what a particular question was asking, both grammatically and by material as it was asked in a convoluted manner:         “……but it’s a false question”.   My response would be (should have been), why a false question (assuming double negative) when it is testing test taking ability and logic, not content knowledge.   I was ‘dinged’ for my response by the question writers as I corrected the grammar since I made it ‘easy’.  All the process did was make me laugh about who might have the larger ego.

Obviously when we read and interpret test scores (the nefarious spring testing ritual), we are also determining how well our students can think through test logic, as opposed  to when there is/are written components.  Why is it then the spring test scores give a different visualization of what NAEP produces? It is not just the idea there are two different types of tests.

As schools (public, private and charter) have jumped further into the cesspool of test scores based on M/C and T/F, writing has diminished. We do not expect students to reason through and logic out a science experiment, do error analysis on math problems, write a fictional critical analysis or well researched scientific piece (all of which is appropriate writing across multiple genres) – we just need them to pick/choose an answer.

Even worse, there are people who would like to see teachers castigated for not teaching well unless test scores go up. How about we start rewarding teachers where writing improves – in all genres and content areas.   Just imagine if test scores remained the same or a bit higher each year AND students could write  compelling essays, papers and ideas by 8th and 12th Grade.

There are many organizations and businesses which would be better served offering services such that student work could be read and graded on a rubric for teachers (eliminating the favoritism and other issues of teacher grading in the classroom) rather than continuing to jump on the test bank bandwagon.  Until we choose to change how we ‘do business’ in the arena of testing, we are getting just what we pay for. Questions written by students abroad, which are then anglicized and made okay for the U.S. We are not changing the ‘known world’.

Devil’s Advocate for the day…….

http://www.marketwatch.com/story/eighth-graders-show-improvement-in-science-achievement-on-the-nations-report-card-2012-05-10

http://lhsfoss.org/introduction/index.html

Unlike being principal for a day, being the devil’s advocate for a day allows one to really ask some tough questions. I don’t know that I can solve the problems noted from tough questions even though I know some of the places to begin the research.

Based on scores released by NAEP (National Assessment for Educational Progress or ‘The Nation’s Reportcard’) for science results in Grade 8 of school year 2011, there is little to cheer about. In fact, if one were to actually read between the lines of the below analysis by NAEP, people should be frightened.

….. those at the 90th percentile, which showed no significant change.

A five-point gain from 2009 to 2011 by Hispanic students was larger than the one-point gain for White students, an improvement that narrowed the score gap between those two groups.  (this translates to a total 4 point gain net)

Black students scored three points higher in 2011 than in 2009, narrowing the achievement gap with White students.

By contrast, the gender gap reflected in the 2009 science assessment remained essentially unchanged

Average scores for both girls and boys were higher in 2011 than in 2009; male students scored five points higher on average than female students in 2011.

Let me explain what I mean. If the entire test is worth 300 points, as noted below, a one, two or five point gain is almost worthless in merit. Another way to look at this is as follows:

The results are reported as average scores on a scale of 0 to 300 and also by three achievement levels: Basic, which denotes partial mastery of the knowledge and skills needed for proficient work; Proficient, which represents solid academic performance; and Advanced, which represents superior work. The percentages of students scoring at or above the Basic and Proficient levels were higher in 2011 than in 2009. In 2011, 65 percent of students performed at or above Basic; 32 percent performed at or above Proficient; and 2 percent performed at Advanced.

Categories are below basic (not listed here however basic is only a baseline of competency), basic, proficient and advanced. If basic denotes partial mastery, less than basic is not even mastered. All students should be at basic as a baseline with proficient and advanced as the categories which show who the real from fake science students are in Grade 8.

The stats are reported in a most confounding way as they are so depressing. Let me break it down for you.

In 2011, 65% performed at or above basic. This means 35% of students, a full one-third of Grade 8 students in the U.S. could not hit the ‘basic’ mark, in Grade 8…..we are not even talking college. 32% performed at or above proficient. This means that of the 65% at or above basic, 32% of the 65% were proficient. Another 33% of students (65-32) is only basic. 2% performed at advanced levels. So, (65-2) 63% were not advanced, in fact only 2% of all students were advanced.

I don’t know about the rest of you but 33% below basic and 33% at basic means 66% or 2/3 of our Grade 8 students are not learning much in science. This is FRIGHTENING, not something to state ‘encouragement’ over.

“The gains are encouraging, but the racial and gender gaps show a cause for concern,” said David P. Driscoll, chair of the National Assessment Governing Board, which sets policy for NAEP. “In order to compete in globally competitive and expanding fields like technology and medicine, we must make sure we give our students the tools necessary to excel in an important subject area.”

So, for today, some tough questions I wish to ask are the following:

(1) What percentage of schools in the U.S. use FOSS up to and including Gr 8?

(2) What are the other ‘line leaders’ for science education text books/packages/kits?  I believe the others to be Mcdougal Littell, Glencoe/McGraw-hill and Houghton Mifflin Harcout. 

(3) If FOSS states their kit is

FOSS is a research-based science curriculum for grades K–8 developed at the Lawrence Hall of Science, University of California at Berkeley. The FOSS project began over 20 years ago during a time of growing concern that our nation was not providing young students with an adequate science education. The FOSS program materials are designed to meet the challenge of providing meaningful science education for all students in diverse American classrooms and to prepare them for life in the 21st century. Development of the FOSS program was, and continues to be, guided by advances in the understanding of how youngsters think and learn.

Science is an active enterprise, made active by our human capacity to think. Scientific knowledge advances when scientists observe objects and events, think about how they relate to what is known, test their ideas in logical ways, and generate explanations that integrate the new information into the established order. Thus the scientific enterprise is both what we know (content) and how we come to know it (process). The best way for students to appreciate the scientific enterprise, learn important scientific concepts, and develop the ability to think critically is to actively construct ideas through their own inquiries, investigations, and analyses. The FOSS program was created to engage students in these processes as they explore the natural world.

What does this mean in the way of test scores? And to that end, what is FOSS doing to shore up the achievement gap?

(4) What do other competitive programs say about their science text books/package/kits and what are they doing to shore up the achievement gap?

Since FOSS has been around 20 years, I would think the research based part of the program would start paying off some dividends in test scores.

It is impossible to blame science teachers as this issue is based in part or wholly on three factors.  NCLB did not start testing science until five years ago. The first year was to obtain a baseline for states. Teachers prior to NCLB and currently do not need a science background to teach science in middle school unless a particular district adopted the higher standards of credentialing.  Most teachers are terrified of teaching science unless they have a science education background. Last, but by far the least of problems is the shoddy text books and materials teachers are expected to use to provide a full, rich, exciting science curriculum.

I am guessing by now that many of you can follow the line of reasoning I am using. The devil is in the details and the details to our science success as a nation sucks.

Observations – Parity Across the States (NAEP)

The past four months provided a very interesting job opportunity in a way I was not expecting.  I was hired by Westat to help administer the ‘national’ exam.   While this position requires a great deal of organizational capacity, it also taught me so much about those who ‘know’ and don’t really know what is involved in educational reform.   In my mind, NAEP is a giant scientific experiment extremely well thought out as well as planned and executed for optimum good statistical analysis.  NAEP is the real deal of data.

My first observation was how many teachers and administrators in public schools (including charter) did not know what NAEP was about.

Example:  Spoke with a principal who, when asked by another principal why his school was participating in NAEP answered he ‘guessed’ he drew the short stick. (I completely understood the humor, however, I was sad he truly did not know what it was about.) We talked on the phone and I asked him why he thought the school was participating (aha – a bit of Socratic methodology) and he answered the following three items (1) his school improved test scores by 17 points (2) his school has the youngest teachers in the district/area (3) this had something to do with spring testing but he was not sure. 

 I replied that while I like his answers, the reason behind national testing was to have some test scores to provide parity  between the various states which have different state standards and different ways of measuring educational success.

Parity in sports is defined as attempting to make an equal playing field for all participants, specifically with regard to financial issues. When parity in a sports league is achieved, all participating teams enjoy roughly equivalent levels of talent. In such a league, the “best” team is not significantly better than the “worst” team. This leads to more competitive contests where the winner cannot be easily predicted in advance. Such games are more entertaining and captivating for the spectators. The opposite condition, which could be considered “disparity” between teams, is a condition where the elite teams are so much more talented that the lesser teams are hopelessly outmatched. – Wikipedia/ January 2011

 I further explained that schools which received federal funds w  ere required to participate if the necessary sample of student attributes was at their school.  We talked for a few more minutes so I could answer some paperwork questions and we each went off on our own separate journey through the day.

In a different phone call with another principal the day before, I was asked about how the school would access these test scores so they could use them to compare with their API and AYP since other schools in the district were not taking this special test. I had to explain the scores were not disaggregated down to the district level.  This particular issue kept cropping up with teachers as well, especially on assessment day. Teachers asked if they would have the scores to use by Spring……

In  the above scenarios, I was talking to people who could speak clearly about certain aspects of educational reform, albeit only those minimal measures which had been drummed into them through some grad school program/administrative credentialling program/school district.   My shock was that these were ‘good’ schools in so called ‘good’ districts so how could these administrators not have run across NAEP?  I actually asked a couple administrators where they attended graduate school.

Along the way there were also some funny stories – a school which is in a very wealthy area had a substitute teacher’s aide show up with alcohol on their breath and the principal had to deal with that issue; another school had some students order a pizza via their cell phone, except when it was delivered, the office staff realized no ‘individual’ student would order a 2 L bottle of coke.  I had a colleague talk to a custodian in Spanish, only to have the principal state the man understands and speaks English…..

In my mind I was surprised as I have known about NAEP since I was a child – I went through at least one of the testing sessions in Grade 4, possibly Grade 8.  I read about NAEP and went to a lecture regarding The Nation’s Report Card when I was in graduate school.  NAEP was the organization where  the National Science Standards were related so people could discuss trends in science ed ucation.  I was beginning to feel as if I had entered some alternate universe where educational reform happened on a different planet on an alternate flat plane.

On a more personal note, I noticed (part of the script I read requires me to ask a few questions) there is not a category for people(s) of Middle East origin.  I am not sure if this was intentional, as in who really cares what those students do (even though we seem to care about Asians) or some one with far more wisdom then myself decided these people are, well, white.  Since I do not know specifically what NAEP is looking for, I can only speculate on a ‘forced’ selection of race/ethnicity.  One question asks students to delineate Latino/Hispanic and then the next question is everything else.  I feel bad for the Philipino’s who actually know their history as they are Latino (Spain) and Asian, not either or.

I have never looked at the test questions as I continue to have a teaching credential and this, in my mind is inappropriate.  I have looked at the release questions published in booklets for parents and/or teachers and administrators who may have questions.  Not much was gleened from this process as I do not support the efficacy of multiple choice exams since there is always an inherent 25% of accuracy by randon choice on a four answer question.   As is the case with SAT prep, it is not about the right answer so much as the ability to use your mind to reason ‘out’ what are the wrong answers.  The SAT is in no way indicative of much, my favorite examples being people who bucked the system and did not complete college, such as Bill Gates or people who did poorly on the SAT and succeeded far more than anyone would have guessed, Timothy Ferriss. 

NAEP allows educators and statisticians to peer into the minds of students to take a peak at how various curriculums play out across the U.S.  Our ‘Nation’s Report Card’  is just that, a report.  In a broad way we are able to see where education seems to have traction (typically, in places with low socioeconomic distress) and where no amount of money seems to change the consequences of childhood poverty. 

I have been to schools with views which certainly must prevent even a lax daydreamer from focusing and I have been to schools which remind me all too often of things I have seen in Peace Corps and traveling various third world countries.  This previous sentence is a different kind of parity – until we have PARITY, we will not change education in any formidable manner.  It takes an abundant amount of community involvement, parental education and literacy resources (notice I did not say monetary resources) to overcome poverty.  No amount of well constructed testing is needed to prove this out – rather, we just need to travel outside our own familiar community.



2009 Science Data by NAEP… the good, bad, ugly and true.

Many people are amazed when I relate why I left teaching science in the classroom – as if it was the most ridiculous decision I could have made – job security was ‘everything’.   I  try to explain  I was using my feet to cast my vote against what I believe to be vapid.  I left the classroom four years ago after teaching science at a charter school – the final frontier for teaching to the test and only knowing about API and AYP in California.   I knew about and followed NAEP which meant API and AYP were only one piece of a larger and more complex puzzle regarding the process of  education. 

Interestingly, most teachers at any school and a fair amount of  principals are inadequately aware of what NAEP is to even have a quality education reform conversation, at least in California.  Education reform centered around such items as question banks for pre/post assessment and data collection (CST’s).  I can not even remember the last time I heard a teacher state the idea of anecdotal evidence.  Part of me felt overpaid for the job I was doing since I sure was not allowed to teach science in the manner which mattered (NAEP results as of Tuesday are indicative of this feeling http://www.huffingtonpost.com/2011/01/26/nations-report-card-science_n_814112.html ). 

At some point that last year in the classroom, it occurred to me I had not gone to graduate school to teach from a text book or kit -as the kind people of Lawrence Hall of Srecience – UC Berkeley (FOSS Kit) were trying to convince me to teach Gr 7 photosynthesis while writing the chemical equation on  paper with paper atoms!    When I discussed the possibilities of using various other hands on methods of exploring the concept of photosynthesis (Elodea in a test tube with indicator fluid – blow in CO2 and cap, allow to photosynthesize in sunlight outside and see the gas on the pond plant leaves and so forth), it was made clear to me ‘this was not what was on the test’ and therefore my students needed to do/study the lessons Lawrence Hall of Science constructed.  I accidentally on purpose cited the infamous Harvard Study (along with The Smithsonian Institution as part of my point of reference), to no avail.  Apparently what was found out all those years ago regarding science misconceptions never quite translated itself from the right coast to the left coast. 

I was caught between a rock and just a place – it was not a hard/difficult place, it was just a place.  Just teach science as you are told and follow the text book.  The rock was my conscience and my better sense of what a quality education could be.  There was nothing compelling about teaching science from a textbook.

The stage for my decision to leave the classroom was set by President Bush as he pushed  NCLB through Congress on an express plane to hell.  NAEP  (National Assessment of Educational Progress) was just beginning to be read/heard and appreciated by a broader group of educators. Although The  National Assessment of Educational Progress has been around since 1969, it seems only professors in the field of education paid attention.  Although NAEP had great data, it could not get traction with an administration which believed evolution was one of the signs of the second coming (NAEP uses scientific methods to obtain data).   It is difficult to refer to the above scenario as few people even understand NAEP.

 Grad schools these days do not discuss NAEP – very few people know what it is when I reference it as its name, the acronym or The Nation’s Report Card.    Apparently the idea of parity across the states is taboo since each state managed to carve out a special meaning for highly qualified teachers.

The NAEP science assessment is not specifically aligned to California’s science content standards. There is no national science curriculum and each state sets its own standards. California’s own science assessment system, as it has for other subjects, shows students making steady progress. – Mr. Torlakson

  This inability to discuss the larger idea of a national curriculum and parity is also part of why the data released 25 January 2011 was so unsettling.  People don’t really know what the data means, so they belittle it.

“As a science teacher, these results are troubling. Despite the enormous efforts being made by educators, we’re seeing the consequences of lagging behind other states in investing in education,” Torlakson said. “This test is a less-than-precise measure of student performance in California, but it is one more signal about where we stand and where we’re headed.  http://www.cde.ca.gov/nr/ne/yr11/yr11rel12.asp

The good part is it is ‘NOT TOO LATE’ for the U.S. to regroup and actually do something about our shoddy education system as it relates to science. The bad is it will be costly since there are many wonderful undereducated and/or poorly educated students laden with misconceptions which must be dealt with.  The ugly is it will be difficult to recruit the people with both the expertise in science/math and the mastery of educational process without the dollars.  Teach for America may have their grant – they are horrible at retention.  The true (truth) is, I left at just the right time – when nothing was happening.  There is hope things will turn around in the not too distant future.  I believe it would be great to teach science again – in a manner which matters.