The higher level gamble

Posted on September 3, 2013 by knlistman

stephs (7)aThe attempt to create “higher level” multiple choice questions does not always result in valid assessment. In fact, I’m beginning to wonder if this kind of close-ended, limited choice question can even legitimately test anything beyond a simple analysis.

In secondary school, I’ve seen students who could easily answer multiple choice questions aimed at synthesis and evaluation (according to Bloom’s taxonomy). Yet, these same students stumble on lower level questions. They had picked up the pattern of how the distractors were written… supposedly to require higher level thinking. But they struggle with lower level questions because the test makers get desperate for distractors that are plausible for themselves. Test writers sometimes expect students to show evidence of learning by the ability to perceive the difference in two words with one letter variance, or two  identical numbers, except for the placement of a decimal. Students misread and get easy questions wrong. In this case are we testing comprehension or visual acuity?

Even on an objective based test there are students who can perform a particular task, and miss the questions testing it.  For example in math, if questions are vaguely worded and require extensive vocabulary, a student who has mastered the math skill can still miss the question. I have discovered that when this type question is read aloud, students answer correctly more frequently. So are the questions testing higher level thinking skills in math or reading skills? What are we really testing?

Formulas for creating questions based on research are used in education. However, the test questions themselves are not often subjected to any kind of real evaluation. The best way to determine efficacy is to “test the test. ”   When developing training for industry we used an alpha review of courses and tests that required problem-solving and applications of theories. Two or three students – the same average age and background as those to be trained -would provide feedback. Many invalid questions were dealt with during alpha review as students recognized ones that were confusing or contained multiple right answers.  Then, a beta review was conducted with 50 to 100 students to provide statistics on validity and reliability.

However in the educational test industry, “testing the test” is often done as part of the actual administration. Because of the high stakes of the test for the schools, there is a great fear of letting any but a select few see the standardized tests before the release date. Even on the day of the exam students are warned not to discuss the questions with anyone after the test. Of course they still do.  However, does that really affect how well students perform? Or only how well they think they have done.

What is the educational system gambling on when it depends on this kind of multiple choice testing ? It is whether or not the test results adequately mirror the students’ actual skills.

Posted in Education trends, Standardized testing | Tagged , , , | Leave a comment

The time of tests

Posted on August 31, 2013 by knlistman

GoujianThe ancient Chinese refined the process of choosing officials based on testing. Prospective government employees would spend days ensconced in little tents taking the massive exam. If they scored high they could expect a cushy job as part of the bureaucracy. Testing was proposed as a way to determine who got these positions in order to avoid the twin evils of obtaining government jobs through bribery and nepotism. Whoever did the best on the test for a position should supposedly be the most qualified candidate regardless of wealth, status or ancestry.

These tests were still biased towards the wealthy and privileged in ancient China. People who grew up in families that had money often did not have to work and could afford private tutoring. So they were much more likely to make the cut. The same thing holds true today. If you check out research you will find that the best predictor of a students’ success in the public schools is the educational level of their parents, closely followed by stability of family life. Both of these characteristic are impacted positively by living in a household with a higher social economic status. So it makes sense that the average student in a wealthy suburb does better in school than the average student in poor inner city area.

However, that difference is deemed unacceptable. Unlike the Chinese we have forgotten the true purpose of testing. For them it was to judge the abilities of the individual candidates – not their families’ status or their tutors’ capabilities. Testing in the United States educational system is more closely linked to grading the school systems and teachers than determining the individual abilities of the students. We need to realize that authentic testing is to help us determine where students stand.

If a test is a worthwhile measure not everyone can be expected to pass. When students fail a test in a class and can consistently make up the grade with test corrections, they soon think “What is the point in studying?” This kind of practice is class has led to inflated grades that are less than accurate. The resulting standardized testing that has been foisted on the schools has led to instruction that is excessively focus on these tests. Some schools require teachers to provide reviews for state tests even when students do not need it. The accountability to pass a course needs to be back on the shoulders of the students. Then, the educational gurus can concentrate on making sure the tests are valid and reliable.

Drawing of Gou Jian of Yue (public domain)

← Back

Thank you for your response. ✨

Posted in Education trends, Standardized testing | Tagged , , | Leave a comment

Multiple Choice Mess

Posted on August 30, 2013 by knlistman

CaviaHaving worked both in education as a teacher, and in industry as a developer of certifications, I understand the prevalence of multiple-choice tests. They are easy to administer and grade without the specter of subjectivity.  But it is not simple trying to make a question of this type assess anything other than memorized information. So often I have seen standardized tests in which  “higher thinking skills” were supposed to be determined by multiple choice questions.

I recall a standardized grade school reading test with an example of many “higher level” questions that failed to hit the mark. In one case students were suppose to infer that the main character in a writing selection thought the vacant lot without a building was ugly. The rationale was the author had described the street as looking “like a child with a gapped-tooth smile.” However, the students being tested were of the age in which children lose their baby teeth. Were they supposed to assume that everyone their age was ugly? The adults who wrote the test forgot what it was like to be a child and came up with an invalid question in an attempt to assess “higher thinking” skills.

Because test creators know more than the students for whom they are creating the test (at least most of the time) they try to make questions harder using subtle differences between correct and incorrect answers. Vague wording with more than one interpretation, or using unusual phrasing which is harder to read end up turning questions on math or science into reading comprehension questions which are not really valid indicators of either math or science ability. Two errors that occur most frequently in these kind of questions are two right answers or no right answer.

Of course, with all these possible problems, the companies creating these standardized assessments need a way to determine the validity of the questions. How do they do this? By putting field test questions in the standardized tests currently given to students. Some standardized tests contain many questions that don’t even matter as far as the score. They are included as samples to get reliability and validity statistics. Then, that particular type of question can be used in the future. I discovered that removing the field test questions from the TAKS was one of the modifications used for learning disabled students in Texas.

So field test questions do affect student performance.  Think about it. You are a students are already stressed about the all-important standardized multiple-choice test and you run into questions that makes no sense. You have no way of knowing if it is your deficiency or one of those field test questions. Now, this same method of “testing the test” is being included in the Common Core assessments.[1]  Do we really want our students to be guinea pigs?

Photo by Tukka (Public Domain)

[1] Strauss, Valerie Standardized test questions kids have to answer that don’t even count, Accessed on April 23, 2013 at 4:00 am
Posted in Education trends, Standardized testing | Tagged , , , , , | Leave a comment

What exactly are we testing?

Posted on August 29, 2013 by knlistman

ink033The two big criteria for judging tests are reliability and validity. Reliability refers to the consistency of the test results each time it is given. For example, psychometric tests attempt to gain a picture of a person’s more permanent personality traits. However,  if a test question asked me if I’d rather be in a crowd or alone, my answer would vary based on my mood. This is why psychometric test tend to ask the same questions, worded slightly different, over  and over and over again. On my part I get a little peeved at this repetition and have a perverse desire to answer the copycat questions in diametrically opposite directions.

The achievement tests, which have become so much a part of education, base questions on knowledge and skills. These test tend to have less of a challenge with reliability as long as they are composed of the old standby multiple choice.  But questions of validity still arise.

Testing anything other than knowledge of memorized facts and word meanings seems difficult. One tactic is to have students read a selection and base answers on it. However, sometimes actually knowing about the subject in a history or science reading selection causes the students to err because they answer what they know rather than what the reading selection reports. Reading selections and questions do not have to be factual. In this age of constant revision it appears that we can only count on what we have just read to be true. This is not a good belief to reinforce in students through tests.

The next tactic is to test higher level thinking skills by adding writing to achievement tests. But that means human scorers are needed to rate these compositions. Inter-rater reliability now comes into play. It’s not easy to set up scoring criteria so that multiple raters will rate the same compositions in the same manner. In fact, it is impossible. Every composition has to be read and rated by multiple people. So we’re back to the same problem of inconsistent results for the same tests.

Artwork by S.L. Listman

Posted in Education trends, Standardized testing | Tagged , , , , | Leave a comment

The schizophrenic approach to testing

Posted on August 28, 2013 by knlistman

Face_Off copyOne of the major difficulties using standardized testing in school is our “schizophrenic” approach to the subject. Is the purpose of testing to ensure that the students is ready to go to the next level? Is it to see if the teachers are competent? Or is standardized testing the basis for grading the school/district itself?  The test and resulting remedial  instruction need to be dealt with differently for each approach. What works for one will not necessarily yield valid results for all three.

If the purpose of testing is to assure that the student is prepared for advancement, then the student needs to be tested on the actual course that they are taking that year. Periodic tests to benchmark progress are useful to determine which students require additional tutoring. Remediation can be provided in whatever manner works – by a specialist or even through e-learning modules. However, at the end of the semester/year the students must past the standardized  test to go on to the next level of that subject. Moving these tests increasingly earlier in the school year to to allow for second and third attempts by struggling students simply means that most other students no longer get a full school year’s worth of instruction. Often little learning goes on between the standardized test and the last days of school.

If tests are used to judge a teacher’s competency, procedures must be observed so that the resulting metrics are free of the bias caused by the differences that students bring to the table. It is absolutely naive to assume that all students of the same age have the same ability. The best way to approach this usage is to test students in the subject at the beginning and end of school year and look not at the final score, but the amount of improvement. Of course students that move during the year would cause problems – in two ways. Their scores could not legitimately be used in the statistics for either set of teachers. But their improvement is one of the most important characteristics in determining the true meddle of a teacher. These constantly moving students are the greatest challenge, showing the poorest performance on the standardized tests.

When school districts are graded based on student’s test achievement, administrators soon learn that they need to tackle this problem of transient students. Larger districts can offer the families the choice of letting students stay put when they move to the apartments with the free first month rent across town, but transportation cost can prohibit this from happening. School districts need to look at a number of services other than those offered by the teachers to help students learn. And then school districts do not have “equal” student bodies or equal amount of tax dollars per student. The students’ improvement adjusted by the amount spent per student should be a factor equal with actual student achievement for evaluating the schools and school district.

Standardized tests are not going away. We need to make up our minds on how they are going to be used.

← Back

Thank you for your response. ✨

Posted in Education trends, Standardized testing | Tagged , , , | Leave a comment

Testing the teachers

Posted on August 27, 2013 by knlistman

734px-Aspen_Crossing_Learning_Community a copyOne of the major challenges in education is preparing new teachers. New ideas for teacher education are constantly being thrown about. But how do we really know what works?

Check out research and you will find that the best predictor of students’ success in the public schools is the educational level of their parents (especially the mother), closely followed by stability of family life. Students in tense situations often struggle because their efforts are channeled elsewhere. Now, these are not characteristics of the school system but the students’  environment. Students whose have higher cognitive abilities in visual language and visual symbolic thinking (reading and math  as opposed to art, music, sports drama etc.) also do better in school, but that ability is innate.

Frequently I see bumper stickers saying “If you can read, thank a teacher.”  Actually both of my children learned to read at home. The first one was unintentional. Trying to be a good parent, I read a little bit to my oldest every night. By the time he was in kindergarten, he had figured out how to sound out words and read by himself. The second child suffered from dyslexia and needed more instruction than the teachers could give her.  So I did my research, and tried a few methods, but it was mostly persistent practice that eventually led her into reading.

Teachers often would like to take credit for high achieving students that are ‘talented’ in reading and math in the same way that another student can carry an exquisite tune, draw a beautiful sketch or throw a fast pitch. But gifted testing shows that students who find it easier to read and do math at younger ages usually continue to do better in these same subjects as they move on to upper levels in school.

So the concern remains – how to redesign the education of new teachers. How you prepare teachers if no one really knows how to measure the quality of existing teachers? Using students’ test results to find out how well teachers are doing has been considered “taboo” by teachers because of the environmental and innate factors that they cannot control. If teachers fear being evaluated based on student achievement as long as there are students who fail to learn, should they even attempt to take credit for those that succeed?

← Back

Thank you for your response. ✨

Posted in Education trends, Standardized testing | Tagged , , , , | Leave a comment

Promptless writing

Posted on August 23, 2013 by knlistman

600px-Tie-dyeWriting prompts are extremely easy to come by – people are giving them away for free on thousands of different web sites – which should give you a clue to exactly how worthless most of them are. The student who cannot string together two coherent sentences can probably come up with half a dozen in a snap. Next time this student balks at a writing assignment, ask him to create a list of prompts that he would like to write about. It’s much easier than actually writing, and he still will not be able to string two coherent sentences together. Writing is the real difficulty. A prompt does not necessarily make it any easier for the writer; only for the person grading the paper. The capacity to create something in response to a prompt is what separates the good writers from the not so good writers.

There are basically two types of prompts for narrative writing:

1) The kind that should not be included verbatim in the text.

An example of type 1 – “Write about the person who influenced you the most.” I warn students not to start the paper with “The person who influenced me the most is ….” like  90% of students do. Everyone is supposed to have a real life experience based on these generic prompts, so they are frequently used for compulsory compositions. However, you risk boring to death both the student and whoever is grading the paper as this same prompt is provided for the umpteenth time.

2) The kind that must be included verbatim in the text.

An example of type 2  – “I returned to the deep, dark woods where I first saw it.” These prompts are supposed to help creative writers that haven’t heard from their muse recently. While it might inspire someone suffering from a lapse in imagination, many students complain about having no connection to what the prompt describes. After all, they will not all destined to be fiction writers.

One option to avoid these dilemmas is to let the students come up with their own ideas for writing. If students do not read a lot, this will pose some difficulty. Spend a few days, reading a wide selection of excerpts and have students record the types of writing which they like best. This list can be used to start their own list of personal writing ideas. A part of a day each week may be spent in simply brainstorming ideas that they can write about. Let them freely share and copy off of each other with the understanding that they should be willing to write about anything on their list. Most importantly, students should write short compositions (250 -500 words) more than one time a week. Ideas come most easily if students actually spend more time writing.

If you are still stuck with having students write to prompts, let every class member get a chance to create prompts. Encourage student to be imaginative, keep the topics clean, but don’t worry about keeping them positive – after all the literature they will be reading is not the most cheerful stuff. Offer a mixture of different ones for each assignment. Don’t forget that there is an inherent problem in writing for adolescents; it’s called self-esteem. Revealing personal exposés are interesting to read, and embarrassing to write. It is often this self-protection of the ego that prevents students from writing on prompts that are advertised as “letting the writer draw on their own experience.” They really do not want anyone to know about particular incidents.

Finally, avoid the ten worst writing prompts. Don’t ask students to write about:

  1. Your most embarrassing (disappointing) experience
  2. A time that you made an important (difficult, good, bad) choice
  3. Your greatest accomplishment (achievement, victory, moment of pride)
  4. An obstacle (difficulty, barrier, impossibility) you overcame
  5. A time you learned a lesson (suffered a consequence, paid the price for your actions)
  6. The person who has influenced (inspired, helped, rescued) you in a positive manner
  7. A time when you influenced (inspired, helped, rescued) another person in a positive manner
  8. The day (moment, week, month, year, decade) that changed your life
  9. The most inspiring encounter (person, event,  etc.)
  10. What was your most favorite ______? (fill in the blank with anything)

Somebody else has already asked them to do this before. If they didn’t do it right the first time, they need more than a prompt to help. If they did it right the first time, why punish them again?

← Back

Thank you for your response. ✨

Posted in Teaching writing skills, Writer's resource | Tagged , , | Leave a comment

Keep it moving

Posted on August 23, 2013 by knlistman

Image

One of the solutions touted to teach writing to the literary challenged adolescent has been the use of a device called the “freeze frame.” Named after the cinematographic technique that stops the action for added intensity. It was intended to add detail and drama. The students focus on writing a detailed description of a scene at one particular brief period of time.  This device is also supposed to stop them from simply telling the story as if they were talking. Of course, among real writers, crafting a story by scenes has been around a long time. The difference is they do not end up with a lot of beautifully written incoherent, little pieces because they understand the art of the transition.

Recently, I listened as a teacher tried to define the transitional phrase so that students could find them it a short story. She kept saying “They make the story move along.” She was right even though she could not explain clearly why an author must use them. This is where writing fiction actually helps. When I write a story I don’t want to bore the reader with every detail of the main character’s life. I concentrate on describing an important event, and then skip ahead to the next one. But, I must include a transition to show a change in time, place, event or idea in order not to confuse the reader. This keeps the story moving.

Last year I had a student who could write quirky and charming essays but she thought the only transition available was “and then.” She also frequently forgot that periods existed. I would have her replace every other “and then” with a period so she could start a new sentence. Still, the monotony of the repeating the transition was marring her writing. There needed to be some way to get her to vary her words other than handing her a standard list from which she could randomly choose different transitions.

Students first collect their own set of transitional phrases (10-20). Then they look at each sentence of a composition and ask:

  1. Is there a change in time ?
  2. Is there a change in place?
  3. Is there a change in events or actions?

Next, they indicate the kind of change made (if any) by writing down the question number above the sentence. Then, they insert suitable transitions into their composition. Too many transitions can be overwhelming, so students just keep the ones that make the composition flow. This tedious procedure usually doesn’t have to be done more than two or three times before students start including the phrases without prompting.

Another  important point to consider is figuring out when you have said enough and are ready to transition. One or two sentences is usually not enough. But, that is all some students want to do. (Hence the emphasis on ‘freeze frame’ style of writing which is supposed to force them to fill in the details.) However, there are times you want tightly written exposition that covers days, months, even years of events in a paragraph. The use of transitions should probably increase to make this passage of time clear.

Transitions can express the relation of an ideas – in fiction as wells as term papers, reports etc. For example when you are letting the reader know what a character is thinking, a fitting phrase to express change bolsters the internal conflict. Try assigning a composition that requires students to create an internal dialog – a kind of self-argument. Have students mark sentence that need transitions because.

  1. There a change in ideas
  2. The idea is different or unexpected
  3. The idea is a result of the previous thought
  4. The person is internally stating a reason or cause

Remind the students that transitions to cue the reader, which actually makes comprehending what they have written easier.

← Back

Thank you for your response. ✨

Posted in Fiction in education, Teaching writing skills, Writer's resource | Tagged , , , , , , | Leave a comment

The writing game

Posted on August 22, 2013 by knlistman

breaking game (3)I have yet to see an app that adequately teaches and assesses writing skills. However, the realm of computer games can help. The students don’t play a ‘writing’ game, but they write the scenarios of the game itself. The more sophisticated video games are like movies – they require characters, setting and the framework of a plot for the various scenarios. Although, students may not be able to produce the game, they can still design it.

The first thing a student needs to write about is the concept: What is the game about? Is it a treasure hunt, a battle, a race or an alternate world in which one must figure out how to gather the necessities for survival? “First-person” games are easier to create than role play, which require a cast of characters along with their motivations. Still the following is necessary for “first-person” games.

1) Write a brief  historical over view or backstory that sets the stage for the game. All other descriptions need to be consistent with the backstory.

2) Next, illustrate how the player moves through the game and goes from one level to the next. This should be done as a flowchart with alternate branching.

3) Describe the initial setting, including camera angle as seen by the player. This should include both visuals and sound.  Students should describe at least two additional settings that the player can move into.

4) Describe the important objects in each setting that the player can use. It is crucial to detail what the object can do, including limitations.

5) Describe the non-player characters or NPCs that the player encounters during the game. These can be foes, accomplices or neutral characters that change depending on how the player interacts with them. This description lists NPC’s responses to various interactions.

6) Finally the student is ready to write the storyboard script which is similar to writing movie scenes.

This is not a short process; the class may spend a week or two, or three, or more depending on the depth of the backstory, complexity of the flow chart and number of settings that must by described. However, this is an excellent way to get students involved in an extended effort to create a product that involves many of the same skills required to produce term papers.

Of course this brings up another point – plagiarism. There are many existing games, and websites or even books with backstories for these games. Students who do not want to put in all the work, will be tempted to copy these.  Internet searches can bring up some reference, to check. However, first you will want to define the extent of copying that results plagiarism so students will know what is expected. Secondly, you must realize that no one creates something completely original. All of the games scripts will refer to things that students have taken from their own experience with games.

For more information and examples read Writing a video game script/storyline by Will Kaliff.
http://www.gameinformer.com/membergroups/game_informer/writers_guild/the_indie_game_developers_group/b/weblog/archive/2010/05/06/writing-a-video-game-script-storyline.aspx

← Back

Thank you for your response. ✨

Posted in Fiction in education, Gaming in education, Writer's resource | Tagged , , , , , | 1 Comment

Looking for an antihero

Posted on August 21, 2013 by knlistman

Picture 036 antihero 2Most of the students are already dreading their “special” assignment. As they walk dawdling into the room and see the prompt “Write about someone who was a good influence on you” the moans begin to crescendo. Many students quickly run out of different ways to say their selected individual is a really great friend or always there for me. Then, they turn in a repetitive fifty word essay. Others blatantly admit, “There is no one I admire.”

It doesn’t matter how this prompt is worded:  Describe a person you admire, Write about someone you look up to. Students still struggle to write cohesive and interesting essays about positive role models. So I try to help, “Tell me about the person you hate the least.” Then, I tried turning this around. “Tell me about the person you hate the most.”  Interesting responses. Evidently this was much easier to write. However, I must admit that I have to lay down some ground rules for this assignment to prevent any possible fallout.

  1. Any references to real people and places need to be altered for the protection of the students themselves.
  2. The composition is considered fiction even though it is based on real events. (Students write the standard disclaimer at the beginning.)
  3. The composition needs to describe events so the reader can understand the student’s attitude toward the character (not just a torrent of name calling).
  4. Use of profanity is limited to what is acceptable for a school paper (which also makes it difficult to fill the paper with name calling).
  5. The composition must include an incident or description that also shows something good about the character. (Think about something the character does that you wish you could do, or something that makes you sympathetic to the character.)

Students struggle the most with the third and fifth requirements. I’m not surprised. The challenges of relating events in way than elicits feeling and creating characters with some amount of depth exist for writers at almost all levels. Students will need to return to their initial rough draft to expand and refine in order to meet this challenge. After they become at least somewhat comfortable with this kind of writing they should be able to transfer lessons learned to the ubiquitous “Write about the person you admire the most” prompt. Of course, that would be so much easier if there were more admirable people in this world.

← Back

Thank you for your response. ✨

Posted in Fiction in education, Teaching writing skills, Writer's resource | Tagged , , , , | 1 Comment