It’s the same with instructors interested in SAGrader. The first thing they want to know is: Who else is using it? Do they like it?

To give you a taste of our user experience, we’ve been highlighting some user experiences on this blog. Today you get a bonus treat.

Pam Thomas, a Biology instructor at the University of Central Florida, recently contributed an article to the UCF Faculty Focus.  The Faculty Focus aims “to provide an exchange of ideas on teaching and learning for the university’s community of teachers and scholars.”

In the October issue, many of the articles dealt with strategies for teaching writing. Pam writes candidly about her SAGrader experience, providing insight into the nuts-and-bolts of using the tool in class.

SAGrader provides Pam with a way to realistically employ writing assignments in the large classes she loves to teach. The result, according to Pam, is not only increased academic performance, but also enhanced engagement with students.

The students react differently when confronted with a blank space to write into. That blank space provides me the window into their minds and a manageable way to grade their responses.

Pam mentions the advantages of assessing student progress before a test or quiz, and the flexibility to design high-level writing assignments just for her class. Pam has found success with scenario questions that require synthesis and application, since it is crucial to help students “develop analytical skills rather than rote responses from open books or notes.”

In one freshman class, Pam saw her in-class test grades increase 12.5 percent when students used SAGrader. Pam attributes this to both the learning and teaching benefits SAGrader provides:

My class was on the way to learning to express ideas in written form, clearly and concisely, while also learning vocabulary, technical scientific information, and critical thinking skills.  You can see exactly why and when students don’t understand, and you can use it to help them learn during the next class or right away if you desire.

If you’re curious about SAGrader, I encourage you to read the entire article (Pam’s article begins on page six). It’s a fantastic look at SAGrader from the teacher’s perspective.

But Willis reminds us that certain tasks — like writing — strengthen parts of the brain that can pay dividends in any domain.

Through writing, students can increase their comfort with and success in understanding complex material, unfamiliar concepts, and subject-specific vocabulary. When writing is embedded throughout the curriculum, it promotes the brain’s attentive focus to classwork and homework, boosts long-term memory, illuminates patterns, gives the brain time for reflection, and when well-guided, is a source of conceptual development and stimulus of the brain’s highest cognition.

We’ve known this for a while. It’s one of the reasons writing across the curriculum (WAC) programs have gained popularity since the 1980s.

At the most basic level, writing requires students to recall knowledge rather than just recognize it (e.g., a multiple-choice question). With more complex writing activities, students must retrieve information, link it with related concepts, then organize and express those ideas in their own words. There’s evidence that this retrieval process produces more learning than even the most thorough study session.

Writing can also have social and emotional benefits. Willis points out that many students are afraid of making mistakes in front of their peers or the teacher. With cloud-based collaborative tools, students can write anonymously and fearlessly express themselves.

Especially with peer anonymity, there is accountability and peer interaction, without the concern about mistakes that is so paralyzing to many students during class time.

Writing activities can empower students to connect with the course material in meaningful ways, confidently explore new ideas, benefit from feedback and reflection, and reflect on their progress over time.

Be sure to check out the full article for more about the benefits of writing for math and science.

Photo via

Joe is passionate about finding effective ways to teach that go beyond lecturing and reading a textbook. He recently finished a fantasy novel called “Need For Magic” that integrates principles of social psychology into the story line. Teachers around the country have been using it to engage their students in discussions on topics like the bystander effect, obedience and the fundamental attribution error.

Joe’s next project is a curated collection of psychology-related videos that can be used to help illustrate concepts in Psych 101, such as classical conditioning and child development.

Based in the Washington D.C. area, Joe teaches almost 200 total students in five sections of AP psychology. He also teaches at the local community college. He’s been teaching for about 10 years.

This is Joe’s second year using SAGrader in his classes. He’s been kind enough to share some of his SAGrader experiences in this informal Q & A.

Why did you decide to use SAGrader in your class?

“I decided to use SAGrader because I know that writing is critical to the study of psychology, but I was getting discouraged that I could not give my students the attention their writing needed.  In addition to improving their writing skills for academic’s sake, I wanted to prepare them better for the writing portion of the AP exam.”

How has your class improved since you started using SAGrader?

“After my students became accustomed to using SAGrader, they have become more engaged in writing.  Despite the fact that SAGrader allows me to assign them more work, they seem to be more proactive in completing the increased work load.  I think it is because the can work on an essay for a few minutes, get good feedback, take a break and retry at another time.  It puts the control in their hands. With their busy schedule of practice, part time work and other classes, SAGrader allows them to work on the essays when it is convenient for them.  I think that allows them to take ownership.”

“For the students who complete all of the assignments in my AP class, I saw a huge connection in a passing AP score. The rubrics of the essays are strikingly similar to the rubrics that are used at the annual AP reading.”

What are your favorite features or benefits of the program?

“By far the greatest benefit is the increased communication I have with my students regarding their writing. I didn’t expect that.  On some level I thought SAGrader would allow me to step back, save some time and let the software grade several hundred of the same essays.”

“What I have found, through answering challenges and watching their scores rise, is that I am more engaged without spending more time.  I can reply back to specific questions and misconceptions.  Communicating through the challenge system allows me to focus on student responses and have an electronic discussion with them.”

What would you say to other instructors thinking about using SAGrader in their class?

“I would say that SAGrader actually increases student and teacher involvement in the writing process.  While the students might spend a bit more time, the teachers will spend less time but get better results.  SAGrader allows teachers to focus their efforts on students that need help and concepts that are challenging rather than spreading their attention too thinly on everything and everyone.”

Thanks Joe! If you are interested in learning more about SAGrader, send me an email at colin@ideaworks.com.

Introductory Psychology courses are filled with interesting concepts that can be demonstrated to students using methods other than lecture and reading. Here are a few activities I’ve put together over the last few years. They work great for high school and college classes.

(By guest blogger Joe Swope).

#1. Understanding Depth Perception using Echo Location

Have a student stand about 8 feet away from a brick wall or other wall with a hard flat surface (no posters, pictures, windows).  Then have the student hum or hold a note out loud as he or she walks toward the wall with EYES CLOSED.  Instruct the student to listen to the tone as he approaches the wall.  Without peeking, he will be able to get within an inch of the wall by sound alone.

#2. Where Rods and Cones Are and Aren’t

Have a student sit in a chair facing the class.  Have another student stand behind him with a handful of different colored pens or markers.  It doesn’t matter what the ink is, the outside of the pens need to have different colors.  Have the student who is standing behind the seated student slowly bring one of the colored pens around to the seated student’s peripheral vision.  Try to keep the colored pen about two feet away from the seated student’s head.  The seated student will be able to see movement but will not be able to determine the color of the pen.

#3. Classical conditioning

This one requires a teacher who paces during lectures and is not afraid of annoying the students.  For a few weeks before the unit on classical conditioning, the teacher should push the top of the pen of a student in the front row.  The instructor should do this for every student in the front row.  It is terribly annoying, but the instructor should do this a few times ever class before the unit on conditioning.  After only a few trials, the students will associate the instructors presence near their desks with pens being jostled.  As a result, after only a few classes of this the instructor can simply walk by each of the student’s desk and watch them automatically pick up their pen until the instructor has passed by.

Once the instructor reveals that students have been trained, a discussion should ensue as to whether it was classical conditioning or operant conditioning.  Note: Even though the students cognitively understand why their pens were annoyingly jostled, the effect is long lasting and students will pull their pen from the paper for weeks afterward.

#4. Conformity

Either wait for or construct a situation where a student leaves the room for a few minutes.  While he is outside of the classroom, instruct the other students to stand up when the student returns and sits in his seat.  As soon as the returning student sits down in his seat, he will see his classmates immediately rise.  For added effect, have the classmates rise with their notebooks and continue with the lecture.  Chances are the student will stand.

A less dramatic but a more sure effect is to have the students already standing when the student returns.  If class continues and the lecture proceeds with the rest of the class ignoring him, the returning student will blend in and remain standing.  Note: If the class is aware that the lesson is on conformity, the student might be wise to the gag.

#5. Serial Position Curve

Have the student put away pens and paper.  Ask them to remember as many words from the following list: giraffe, lion, tiger, rat, alligator, buffalo, eagle, ox, bear, beaver, mouse, cat, deer, fox, raccoon, opossum, rabbit, squirrel , elephant, cheetah. Take a moment to distract the student by reminding them of upcoming homework or by telling them a knock-knock joke.

Then, ask students to write down as many words as possible.  Give them a few minutes to wrack their brains.  Ask, by show of hands how many students remembered which words.  Tally student responses on the board.  Have students notice that the animals in the middle of the list were not remembered as much as the animals at the beginning and at the end of the list.

#6. Remembering by Schema

Have the student put away pens and paper.  Ask them to remember as many words from the following list: refrigerator, oven, cutting board, dishwasher, knife, pot holder, blender, spatula, toaster fork microwave, spoon, toaster, fork, stove, frying pan, dish towel, freezer.

Take a moment to distract the student by reminding them of upcoming homework or by telling them a knock-knock joke.  Then, ask students to write down as many words as possible.  Give them a few minutes to wrack their brains.  Ask, by show of hands how many students remembered refrigerator, how many remembered oven.  Hands will raise as students eagerly display their brilliance.  Then ask how many remembered sink.  A few students will raise their hands despite the fact that sink was never mentioned.

#7. Priming and “Reading Students’ Minds”

This one takes a little planning and works well for a lecture on consciousness.  On the day before the demonstration takes place, create baseline data by having student write down the first animal that pops in their head.  Collect the papers.  Then on the day in question and throughout the lecture, the instructor will lace his words with the theme of jungle cats.  It is a common school mascot and the teacher can remark ask about how the “Lions” did.  Or the instructor can ask about the Cincinnati football team and what the heck is a Bengal anyway.  Throughout the lecture the teacher might refer to Simba or Mufasa.  The teacher might remark about how he played golf on the “linx”.

After weaving such instances into his remarks, lectures and conversation, the instructor should then ask students to write down the first animal that springs to mind.  Collect the responses and explain how instances of big cats have been ‘subtly’ presented to the students.  Record the baseline data on the board from the previous day.  Record the current collection of student response on the board as well.  Check to see if there is a noticeable difference in responses that include big cats.

#8. Operant Conditioning Using Token Economy

The instructor should ask for two student volunteers.  One will be the trainee the other will be the trainer.  The trainee will need to step outside of the room for a few minute.  During that time, the class will determine what behavior the trainee will perform (standing on a desk, walking near a window, erasing the chalkboard, etc.).

When the trainee returns to class the only feedback he or she is given is the words hotter or colder.   Using these as makeshift punishments and rewards in a token economy the trainee will attempt behaviors in order to receive a “warmer” reward and avoid the punishment of “colder.”  This works remarkable well and it is surprising how easy it is to get an average student to do an abnormal behavior with the right system of rewards and punishments.

 #9. Speed of Neural Impulse

Generally, the neural unit comes after the unit on research methods, so this activity is a nice one to tie the two units together.   You will need a stop watch and a calculator. Have students stand up in a circle around the room.  Have each student use their right hand to grab the left wrist of a classmate.

Assign one person the job of starting the impulse.  Start the stopwatch at the same time you say “go.” He will then squeeze the wrist of the person to his right. As soon as that person feels the squeeze, he will then squeeze the left wrist of the other class mate and so on.  Have the last person in the circle say ‘stop’ when they feel the squeeze.  Stop the stop watch as soon has he says “stop”  record time on the board. Do this a few times and record the time for each trial.

For tie-ins to the neural unit, estimate or measure the arm length + shoulders + neck of each student.  Add that combined distance and divide by the time it took the signal to get around the circle.  That is an approximate speed for the neural transmission.

For the research tie-in, have your students predict what would happen if student grabbed the other wrist and reversed the direction. Test it. Have you students predict what would happen with eyes closed. Test it.  What are some confounding variables?  Are their biases?  The activity is flexible enough to be the start of quite a few subsequent activities, discussions and assignments.

#10. Taste Transduction and Spatial Coding

This one is the only tip that requires special material, a bag of Hersey Kisses.  Distribute a Kiss to each student.  Have them “dry” their tongue as much as possible.  One method, however awkward they might be, is to have students blot their tongues with a paper towel or napkin.

Once the tongue is free of saliva, have the students place the Kiss on the back of the tongue.  It is helpful to have students look up while doing this so saliva has a more difficult time reaching the tip of the tongue.  Most students will report that they feel the texture and weight of the candy, but they do not taste sweetness.  Some will report that they can smell the aroma of cocoa but cannot taste chocolate.  Eventually the saliva will carry chocolate molecules to the front of the tongue and the students will taste sweetness.

Today’s guest post is contributed by SAGrader user Joe Swope, a psychology teacher in the Washington D.C. metro area who is always exploring ways to teach beyond the lecture and textbook. Joe has a strong track record developing creative ways to engage students with psychology, including a novel that meshes fantasy and social psychology, and a curated collection of psychology-related videos.

Photo via

As mentioned in recent articles by the Chronicle of Higher Education and New Scientist, one of our newest SAGrader users is Pam Thomas at the University of Central Florida. UCF is the second largest brick-and-mortar university in the country and it’s getting larger. U.S. News & World Report lists their student-teacher ratio at 31:1 while over 7% of their classes have 100 or more students, according to the State University System 2012 annual report.

In October 2009, Pam Thomas was one of two University faculty members selected as Large Class Faculty Fellows by the Faculty Center for Teaching and Learning. The Fellows were tasked with providing support to teachers of classes with 100 or more students through workshops, consultations and the discovery of new tools.  As part of her research, Pam came across SAGrader in the summer of 2010.

“We had looked into multiple other systems,” Pam said about discovering SAGrader, “but this we determined was our best opportunity to assist students to learn and simulate activities that can go on in smaller class sizes. It was also easy to use and cost effective.”  We helped Pam set up a small pilot study with her introductory biology class to give her some hands-on experience with the system.

For Pam, SAGrader addresses many of the challenges associated with teaching and learning in large classes. Students need more opportunities to write and develop their ability to express themselves, reinforce course material and improve higher thinking skills. But in large classes like Pam’s — her biology class can enroll up to 1,200 students — grading all those papers is simply not feasible. Neither is providing each student with personalized feedback, despite it’s learning benefits.

Hoping to address these challenges and encouraged by the results of the pilot study, Pam fully integrated SAGrader with her Introductory Biology class last spring. After the first assignment, the results were intriguing.  Asked to construct a response rather than choose from a list of choices, students couldn’t hide what they did and did not understand. “When confronted with a blank space,” says Pam, “you can actually see the process that the student goes though and identify exactly where the help is needed.”

For the first time, Pam could see exactly where each student misunderstood: “I have been able to rapidly identify areas in which my students need help prior to testing or the next lecture.”

Pam went on to assign six more SAGrader assignments, covering topics like hypotheses, math, genetics and chemistry. By doing some manual grading in conjunction with the program, Pam was able to show that SAGrader was over 99% accurate on most of the assignments, and more consistent than TAs grading the same assignments.

Convinced of its grading fidelity, Pam relished the opportunity to offer a better learning experience for each of her students.  “I really like being allowed to provide feedback to my students,” Pam tells us. “Without this type of automation I could never provide feedback to 1,200 students in a timely manner.”

Perhaps most exciting, students in Pam’s class increased their final test scores by 12.5% compared with previous semesters. Quite an impressive improvement!

Moving forward, Pam is excited to expand her use of SAGrader into additional classes, like the genetics class she is teaching this fall. Despite growing classes, Pam is able to give her students plenty of writing practice and personal feedback, without generating mountains of papers to grade.

Pam’s advice to instructors considering SAGrade is simple: “You will be amazed at the ways in which the system will allow you to assist your students.  My colleagues formerly told me my quest to find a system like this was impossible. It was not. I only wish I had found SAGrader 5 years sooner!”

Photo via

The primary task of an instructor using SAGrader for the first time is to outline some assignment ideas. Instructors often want to know: “Will my assignments work in SAGrader?“

When it comes to assignment design, there are some things SAGrader excels at and some things it can’t handle very well. SAGrader can be used in almost any domain, at almost any level, for any length of student writing — from single word responses to 10-page term papers. As I’ve mentioned before, SAGrader assignments can promote learning on many levels, from simple knowledge recall to analysis and evaluation.

But here’s the tricky part: Assignment compatibility with SAGrader depends on your assessment criteria. What is it that you’re expecting from students and how will you evaluate their efforts?

Domain-specific assessment criteria

SAGrader assesses the expression of domain-specific ideas. Sometimes we explain this by saying SAGrader focuses on grading the content of students’ writing. This is true, but it’s important to note that there are (at least) two different methods of grading content.

The first is a global assessment of content development and support in a general sense. This is often found in rubrics like the 6-Traits, in a category labeled “Content” or “Ideas”. The criteria is something like: Main idea is clear, supported, and enriched by relevant anecdotes and details.

In other words, the grading key doesn’t specify particular concepts that must be explained. Instead it scores students more holistically, based on their focus, development and support of whatever they’re writing about. This general rubric can be applied to an assignment on any topic and work the same way. SAGrader cannot handle this type of assessment.

The second type of content grading is to check for the adequate expression of content knowledge. This is a point-by-point process that checks for specific ideas or explanations.

For example, a very simple assignment about Earth may require students to mention Venus and Mars (nearby planets), Terra (its Latin name), and 23.4° (the tilt of its axis of rotation).

The rubric for one assignment can’t be applied to a different assignment because the relevant concepts vary. SAGrader excels at this type of assessment.

You can think of it this way: SAGrader needs to know, in advance, what you expect students to write about.

Not just the types of information they should include (e.g., a hypothesis, 3 points of support, 1 personal example), but rather the ideas themselves (e.g., Phonology is a subdiscipline of linguistics concerned with the function, behavior and organization of sounds).

Intermediate constraint questions

This means the range of possible responses must be constrained. If students can write about any topic of their choosing, we would have no way of building a comprehensive rubric. And we want students to construct their own answers, rather than selecting from predefined options. So SAGrader aims for the middle ground.

Some educators have referred to SAGrader’s sweet spot as intermediate constraint tasks. These questions fall between fully selected responses (i.e., multiple choice) and fully constructed responses (e.g., an essay about your summer vacation).

For example:

• Fully selected: “Which of these factors encouraged the rise of suburbs because of increased ability to purchase houses? (A) the automobile; (B) movement of jobs to the suburbs; or (C) the GI Bill”
• Intermediate constrained: “From this article, identify three factors that affected the rise of the suburbs and provide evidence for and against the importance of each.”
• Fully constructed: “Analyze historic events during the 20th century that most affected the rise of the suburbs.”

By constraining the range of possible answers to the topics discussed in a particular article, the second option works great with SAGrader.

Assignments that don’t work well

As you can see, there are some assignments that won’t work well in SAGrader. If you want to grade broadly defined essays with no specific content focus, you may need to look elsewhere.

Here some examples of assignments that don’t work well in SAGrader:

• Loosely Restricted Essays: “What do you think is the most important challenge facing teens today?  Why?”
• Expository Short Essays: “If you could change places with another person for a whole day, who would you change places with and why would you choose that person?”
• Creative Writing Essays: “Free write for 5 minutes using this opening line: ‘Behind her, the noise escalated…’”

In addition, SAGrader probably isn’t your best bet if you’re more interested in writing ability (grammar, spelling, style, etc.) than the demonstration of knowledge through writing.

In summary

Your assignment will work in SAGrader if you are interested in assessing content knowledge. You should be able to list out — in outline or concept map form — the full range of answers you expect students to submit. If not, talk to us about modifying your assessment criteria or further constraining your answers.

Most of this is probably new to you, so please don’t hesitate to get in touch (sales@ideaworks.com) with your questions. We’re always happy to take a look at your assignment ideas and offer our expert advice.

If you’d like more examples, download the “Good and Bad SAGrader assignments [PDF]” handout.

Photo via

With this in mind, I’ve compiled responses to the most oft-occurring questions and assumptions about SAGrader, as prompted by reader comments on the Chronicle article and Slashdot.com.

Outsourcing assessment
The Chronicle article presented SAGrader as one method of outsourcing assessment, but SAGrader is really more of an instructor-driven learning tool. SAGrader provides feedback and assessment to students based on an assignment-specific topic outline provided by the instructor. We do not rely on generic algorithms or statistics modeling. In this sense, the instructor has a great deal of direct control over how SAGrader scores each submission. Our instructors don’t see themselves as “outsourcing” their grading work; rather, they are using SAGrader to provide personal feedback to their students more efficiently 24/7.

Where’s the feedback?
A popular concern that readers expressed about automated grading revolved around the role of feedback in improving the student learning experience. This wasn’t covered in the Chronicle article, but it’s probably the most important benefit SAGrader provides. Along with an overall score, SAGrader provides students with detailed, actionable feedback about their submission. Students are encouraged to examine their feedback, make revisions to their answer and submit again.

Immediate feedback, paired with the opportunity to make assignment revisions, supports an iterative writing cycle that promotes learning. Many of our instructors allow unlimited student submissions, so that students can practice as much as they need to before the due date.  We have evidence that this process increases student comprehension and can be especially empowering for students with learning challenges.

Promoting student/instructor interactions
Another concern expressed by readers of the article is that using such an automated tool would widen the communication divide between instructors and students. The thought is that introducing another layer between students and instructors would limit interaction, and thus hinder an instructor’s ability to nurture the student/instructor relationship.

However, this is a misunderstanding of SAGrader’s intended use. SAGrader in its primary function is intended for large classes (100-1000+ students), where personal contact is already constrained by the sheer number of students and student work generated. The program is designed to give these instructors an alternative tool to combat the limits often imposed by classes of this size. Instead of being limited to multiple choice, true/false questions, and a participation grade, instructors can now test and reinforce more substantive knowledge in their students through the use of short answer and essay questions, thereby shifting student learning to higher levels of learning. As one of our instructors told us, SAGrader offers “the ability to have large classes do the same types of work as small classes.”

In addition, SAGrader includes a number of features that allow students and instructors to connect. Students can challenge any feedback they feel is unfair, giving instructors a chance to offer additional explanation or improve the grading rubric. The student performance reports allow instructors to find areas where students may be having trouble, or identify students who may be at risk of failing.

Lloyd, an SAGrader instructor, has this to say about his experience:

“I also appreciate being able to track which students are not doing well, or who are consistently late submitting assignments. It has given me the opportunity to contact those students, express concern about their performance, and in a few cases to eventually help them get back on track with their grades.”

Above all, SAGrader makes it possible to offer more writing assignments in class, giving instructors a better view into their students’ heads. Instructors tell us that allowing their students to write more has given them much more insight into what their students know and don’t know, and identify points where they may be struggling.

Can computers really do that?
As expected, the most common questions readers asked revolved around whether computer software could actually grade student responses in any sort of meaningful manner. It’s a question we at Idea Works answer every day. Advancements in artificial intelligence, natural language processing, and expert systems have come a long way, and today’s computers are capable of things that just ten years ago would have been considered in the realm of science fiction.  As Chronicle commenter becauseisaidso expressed:

“Why not prepare an article on the (obviously not the proprietary, secret) ways an algorithm can grade an essay test?  Clearly, those of us not in the AI biz can’t judge whether or not this makes sense unless we have some clue how it is done…”

Ok, here it goes: SAGrader account representatives work with course instructors to create SAGrader assignments by first developing a conceptual model of the knowledge the instructor wants the student to be able to convey. This information is then modeled in SAGrader and tested to ensure reliability and general cohesiveness of the knowledge domain. We ensure that the program accounts for the hundreds or even thousands of ways a student could express each concept.

When a student submits an answer, SAGrader takes the student response and compares it against the instructor’s conceptual model to identify which concepts in the response were expressed correctly and which were not. If the student fully expresses the concepts and accurately relates them to other ideas the instructor was looking for, they get the points. SAGrader then gives the student feedback on which concepts they got right and which ones need further revision.

So there you go; automated grading of student submissions with assistive feedback in less time than it takes to brush your teeth.  We’d love to hear what you have to say, so if you’re excited, apprehensive, or just plain curious about the technology, sound off in the comments below, or send me an email at Luis@ideaworks.com.

A link to the original Chronicle article can be found here.

As a teacher, it’s essential to effectively assess your students’ knowledge. What do they know? Where do they need to improve?

In Designing Effective Instruction, Gary Morrison, Steven Ross, and Jerrold Kemp discuss the various “tests” available to evaluate different types of knowledge, each with its own strengths and weaknesses. The key is to match each test with your instructional objectives along with any practical limitations, like time or manpower.

Here’s a quick look at how to use each test: (a) Multiple Choice, (b) True/False, (c) Matching, (d) Short-Answer, and (e) Essay.

Multiple Choice

When to use it: Multiple choice can be a very versatile testing method, used to evaluate all but the highest levels of Bloom’s taxonomy. Multiple choice questions can be used to assess knowledge, comprehension, application, and analysis. The practical advantage of multiple choice questions is that they are easy and quick to grade.

Disadvantages: It’s very difficult to assess synthesis and evaluation using multiple choice questions. Additionally, these questions test recognition (choosing an answer) rather than recall (constructing an answer), so it’s possible for students to guess correctly without actually knowing the information.

Tips:

1. Don’t test trivial facts, or try to trick students with cleverly written questions. Focus on testing basic, fundamental knowledge.
2. It’s best to use 3-5 options. Try not to go over 5, and avoid making one option an absurd throwaway.
3. Avoid overusing “all of the above” or “none of the above”. This often allows students to eliminate options without really knowing the information.

True/False

When to use it: True/false questions are fairly easy to write and simple to grade. They can be used to test knowledge or comprehension.

Disadvantages: Like multiple choice questions, true/false questions questions can only test recognition rather than recall. Students have a 50% chance of guessing the correct answer, and it’s difficult to test higher level learning objectives.

Tips:

1. Avoid ambiguous true/false statements.  The statement should be entirely true or entirely false.
2. Don’t mix multiple ideas into one true/false statement. Students may be able to identify the statement as false while only understanding a portion of the concepts involved.
3. Use true/false questions sparingly. It’s a low-level testing method prone to student guessing.

Matching

When to use it: Matching is a great alternative to multiple choice that allows you to fit a lot of material into a small space. It’s used most effectively for testing knowledge of definitions, events, dates, people, principles/descriptions, and function/parts. It’s more difficult to guess on matching questions than it is on multiple choice or true/false.

Disadvantages: Matching assesses recognition rather than recall, and can only be used for low levels of learning (knowledge).

Tips:

1. It’s difficult for students to match more than 7-8 items together.
2. Keep the description of each item short and to the point. Don’t expect students to match paragraphs of text together…it’s just too much information to process.
3. Students hate this, but it’s a good idea to add one or two extra items in the match list. Or allow items to be used more than once.  This can limit the chances of guessing and encourage more thoughtful answers.

Short-Answer

When to use it: Short answer tests are the most “objective” of the constructed-response tests, because scoring can be fairly objective. They can also be used to test a large amount of content fairly quickly, and test recall rather than recognition. As an added bonus, it’s difficult for students to guess on short answer questions.

Disadvantages: Short-answer questions can only be used to assess fairly low level learning like knowledge, comprehension, or application. Also, certain types of short-answer questions may be difficult to score quickly or objectively.

Tips:

1. Short answer questions can range from a single word, to a few words, to a short paragraph. The longer the expected response, the harder it is to maintain objective grading.
2. Be sure to word your questions so that only one answer is correct. This can help the scoring remain more objective and avoid student complaints.
3. Programs like SAGrader can be used to ensure objective and fast scoring.

Essays

When to use it: Essay questions are your best option to evaluate higher levels of learning. It’s a great method to use when you’re interested to see how your students can analyze, synthesize or evaluate information.  Essay questions can range from a few sentences to multiple pages. Essays require students to express themselves in writing, which is a valuable life skill, and provide insight into your students’ understanding of the material.

Disadvantages: Though it is a valuable evaluation method, students often don’t like writing, and it can require more time for them to complete essay questions. As a grader, there is always a danger of being swayed by “good writing” and giving higher grades to well-written papers that don’t cover the course material adequately. Essays take longer to score, and scoring can often be unreliable or biased, especially if multiple people (like TAs) are sharing the grading responsibilities.

Tips:

1. Provide students with your grading criteria ahead of time. How will they be graded? Does spelling count? Do you require a certain format?
2. Grade essays without knowing the writers’ identities. This can help reduce grading bias, and decrease student complaints.
3. Keep your writing prompts as focused as possible to avoid off-topic responses.
4. Programs like SAGrader can be used to ensure objective and fast scoring.  By providing feedback and opportunities for revision, SAGrader can also be used to promote learning through writing, rather than just assessment.

As the authors remind us, whatever test you use needs to be a valid measure of performance. Before choosing your testing method, make sure you understand your learning objectives and rank the level of learning (on Bloom’s taxonomy) you want to assess. Then, select your testing method and write clear questions linked to each learning objective.

 Photo via

Instructors who stumble upon SAGrader for the first time are understandably skeptical that it can evaluate anything beyond keyword mentions.

Usually, I hear something like: “Sure, I believe that SAGrader can score specific content…but I don’t just want my students to parrot the textbook back to me. I don’t see how a machine can grade student understanding, like application of the material.”

In terms of good teaching practices, these instructors are on the right track. They know learning isn’t just about memorization. It’s about internalizing information, combining it with other ideas and applying it to new circumstances. Asking students to apply what they learn also helps them see course concepts as relevant, and can go a long way to help student engagement.

And once they see it in action, these instructors realize that SAGrader can support this learning process. The key is to put some thought into how you construct your writing assignments. With a little creativity, you can build SAGrader assignments that move beyond factual recall and challenge students to pose an argument, make a comparison, synthesize material, and apply ideas to new situations.

You can use a number of different strategies to try to accomplish this.  Here are some of the ones that work well in different circumstances.

(1) Ask students for examples from their own lives.

Frame your prompt in terms of a personal experience, then ask students to reference the appropriate concepts as they discuss their personal example. SAGrader won’t be able to directly assess the relevance of the personal example (because the number of correct responses is infinite) but it can evaluate how well students frame their response. This gets students thinking in terms of application, even if you don’t assess the application directly.

Example:

For this assignment, you must violate a norm in a public setting. Make sure you violate a folkway, NOT a more. Describe the norm you violated. Explain what a norm is, and what type of norm you violated (should be a folkway). What do these terms mean? How did people respond. Were there positive or negative sanctions, or some of each? Describe how it made you feel. Would you say you have internalized this norm?

(2) Ask students for examples from a particular example text.

This might be a contrived example, a newspaper article, something downloaded from the Internet, or anything you want.  The key is that it be a specific text with examples students can use.  For example, you can give a vignette describing a hypothetical student and their daily life then ask students to use examples from that to illustrate their points in the essay.  This can work very well, particularly if you choose an engaging example. One sociology instructor asked students to read some personal ads on the internet, then discuss examples of identity, social statuses, roles, self-concept, etc.

Example:

The following is an excerpt from a life-history interview with a man named George. For this assignment, analyze this transcript, looking for portions of the life history that illustrate important concepts, issues, and/or theories related to socialization. For each issue, concept, or theory you detect in the life-history, make sure to include the specific quote that illustrates the socialization issue. [Include a vignette for students to read and discuss.]

(3) Ask students for examples from a well-known, commonly shared aspect of culture.

Have students respond to various cultural concepts at your school (sports, mascot, frats/sororities), a popular TV show, celebrities or a current news item.  Or link to a YouTube video and ask students to write about what they see. These resources help narrow the range of possible responses (which helps SAGrader grade), while still encouraging application of course material.

Example:

In this assignment, use the University as a case study for culture. For each cultural issue to be discussed, provide examples from your experiences in the University culture. Discuss at least three of the types or varieties of culture (for example sub-, high, and dominant cultures); differentiate between material and nonmaterial culture (also discuss norms and other elements of nonmaterial culture); describe culture and social change (such as cultural diffusion, integration, lag, and leveling, as well as global culture); and finally, discuss one of the common responses to cultural diversity.”

Hopefully, you’re beginning to get some ideas for assignments in your own class. The key is asking students to apply their knowledge to new situations — such as their personal lives or a hypothetical example — while ensuring that the range of possible answers is constrained enough for SAGrader.

When in doubt, ask the SAGrader team. We have a boatload of experience building SAGrader assignments that challenge students and maximize learning.

We love brainstorming assignment ideas with customers and try to take on as much of the workload as possible, leaving you free to, well…teach!

Photo via

Why? For two reasons:

1) Our own research has shown that it’s a great way for students to learn the material — often resulting in a score one or two letter grades higher than they would have achieved without the opportunity.

2) It makes it much easier for the instructor to identify concepts their students are having trouble grasping, since they can track a student’s progress throughout the learning process. This not only helps in identifying students who may be struggling, but can also help instructors target and improve parts of their own instruction that may need further clarification.

We also embrace iterative learning in our own product design. We’re always in a continuous loop of testing, design, and validation; then retesting, redesign, more validation, and so forth. Why? Because it allows us to stay closer to our users! By directly asking our instructors and students for feedback at multiple points throughout the process, we make sure we’re building solutions that our customers really want and appreciate. It also helps us be more efficient by eliminating time and energy wasted on features no one really cares about.

So it’s a win-win for everyone: the instructors get what they want, students get the information they need in a manner that appeals to them, and we save development time and costs by not building unneeded features.

If you’d like to know more about how SAGrader can bring you closer to understanding your students’ learning experience contact Luis at Luis@ideaworks.com.