[comp.edu] Automatic Grading of Student Programs

kent@xanth.UUCP (04/11/87)

I won't begin to claim that this is the last word on the subject (as long as
grades are competitive/pro-survival, there will be students who cheat).

An instructor here, Mr. John Caywood (caywood@xanth.UUCP, xanth.cs.odu.edu, if
anyone wants to approach him directly), has a very nice system for grading
programming assignments.  He made himself an expert in UNIX(tm) csh or sh
scripts to make this work.

First, his assignments are very explicit.  The required results are an exact
string of bytes (including newlines as appropriate) which the student program
must send to stdout.

Second, the student creates a directory, say CS471, under the student's home
directory, in which the software source, executable, and documentation must
reside, with appropriate permissions, and assigned names.

Third, there are three versions of the assignment, corresponding, usually,
to three modules which the student must write.  For the C-grade version, he
must complete the one which demonstrates a basic understanding of the
assignment; for the B-grade version, he must do another module demonstrating
a deeper understanding; for the A-grade version, he must write a third
module demonstrating complete competence in the subject matter.

For the easier versions, the instructor supplies the missing modules in object
form.  The student is given an example data set to work with, not the same as
the one the instructor's grading program uses.

Now the instructor provides a path to his grading script.  The student
executes, say, grade_C, which compiles (or assembles) and links and executes
the student's code.  If there are missing modules, the grading routine
complains.  If the output is wrong, a rather terse diagnostic is given.  If
all goes well, the grading program prints an even terser congratulations.

The student is given some limited number of tries, say three, at each grade
level, and the script records tries and the grades achieved in a protected
file, and refuses tries after the third (say).

The brilliant, brave, or foolhardy start with grade_A.  Mere mortals start
with grade_C and work their way up.

Hardcopies of the assignments are also submitted, because part of the grade
depends on proper internal comments and structure.

John also gives very comprehensive tests, covering the same subject matter
as the programming assignments, and no-one who hasn't made the assignment
work for him/herself has a prayer of passing the exams.

During office hours, students are free to consult the instructor between tries
of each grading level, if they can't understand why their program didn't pass.
Of course, there is also a rather draconian penalty for late assignments, so
there isn't too much slack.

Not only does this work quite well as a grading mechanism, but it is
excellent as a teaching mechanism, because of the nearly immediate feedback
of the results of programming work.  Because students saw this as a fair and
rational approach, there seemed to be little if any sharing of code, but,
(and I think this is good) the more able students seemed to take their less
able students under tutorship, and helped them work their way through the
ideas they needed to solve their programming tasks, showed them debugging
methods, and so on.  Everyone succeeded to the best of their ability because
of this fellowship feeling, promoted by the puzzle aspect of the assignment.

I hope some of these ideas have been of use to you.  When I finally become a
professor, I hope to use them in my own teaching work.  Enjoy!

--
The Contradictor	Member HUP (Happily Unemployed Programmers)    // Also
								      // A
Back at ODU to learn how to program better (after 25 years!)	  \\ // Happy
								   \// Amigan!
UUCP  :  kent@xanth.UUCP   or    ...{sun,cbosgd,harvard}!xanth!kent
CSNET :  kent@odu.csnet    ARPA  :  kent@xanth.cs.odu.edu
Voice :  (804) 587-7760    USnail:  P.O. Box 1559, Norfolk, Va 23501-1559

Copyright 1987 Kent Paul Dolan.			How about if we keep the human
All Rights Reserved.  Author grants		race around long enough to see
retransmission rights recursively only.		a bit more of the universe?

mcintyre@rpics.UUCP (04/13/87)

In article <818@xanth.UUCP>, kent@xanth.UUCP (Kent Paul Dolan) writes:
> An instructor here, Mr. John Caywood (caywood@xanth.UUCP, xanth.cs.odu.edu, if
> anyone wants to approach him directly), has a very nice system for grading
> programming assignments.  He made himself an expert in UNIX(tm) csh or sh
> scripts to make this work.
> 
> First, his assignments are very explicit.  The required results are an exact
> string of bytes (including newlines as appropriate) which the student program
> must send to stdout.

	While forcing the students to have exactly the same output might
make the grading much easier, I think that such policies prevent the
students from having any good ideas as far as input and output.
	The programming student should be encouraged to not only choose
the best algorithms to solve a problem, but also should be encouraged to
find the best input and output formats for a problem.  Let's not stifle
a student's creative impulses.


-- 
Dave "mr question " McIntyre
	
seismo!rpics!mcintyre
mcintyre@csv.rpi.edu

kent@xanth.UUCP (Kent Paul Dolan) (04/16/87)

> = Dave McIntyre
>> = me

In article <1104@rpics.RPI.EDU> mcintyre@rpics.RPI.EDU (David McIntyre) writes:
>In article <818@xanth.UUCP>, kent@xanth.UUCP (Kent Paul Dolan) writes:
>> First, his assignments are very explicit.  The required results are an exact
>> string of bytes (including newlines as appropriate) which the student
>> program must send to stdout.
>
>	While forcing the students to have exactly the same output might
>make the grading much easier, I think that such policies prevent the
>students from having any good ideas as far as input and output.
>	The programming student should be encouraged to not only choose
>the best algorithms to solve a problem, but also should be encouraged to
>find the best input and output formats for a problem.  Let's not stifle
>a student's creative impulses.
>
>
>-- 
>Dave "mr question " McIntyre
>	
>seismo!rpics!mcintyre
>mcintyre@csv.rpi.edu

While output for some few printed reports might be acceptable as free form, I
have found  over 26 years of programming that most of my program output was
required for creating data base records, graphic data representation,
communication packets, and so forth, all requiring bit perfect output.  I have
no quarrel with allowing creativity, but exact programming is also a necessary
skill.  Requiring students, for example, to recognize and locate white
space inadvertantly included at the ends of lines lets them learn such skills
as looking at their data in more than one way (hex or octal dumps are a real
revelation to many beginning CS students), and walking through their code with
an interactive debugger.  Once learned, these skills give the new user a
justifyable feeling of having mastered the machine.  It is a shame if a CS
major graduates without ever acquiring this level of self confidence, or
without ever having met a requirement that the output of a program be perfect.

We have enough buggy programs from poor programmers in circulation.  Please
don't be so liberal that you won't consider means of reversing this trend.
< 0.5 * ;-) >  Thanks again for your attention.

Kent.

--
The Contradictor	Member HUP (Happily Unemployed Programmers)    // Also
								      // A
Back at ODU to learn how to program better (after 25 years!)	  \\ // Happy
								   \// Amigan!
UUCP  :  kent@xanth.UUCP   or    ...{sun,cbosgd,harvard}!xanth!kent
CSNET :  kent@odu.csnet    ARPA  :  kent@xanth.cs.odu.edu
Voice :  (804) 587-7760    USnail:  P.O. Box 1559, Norfolk, Va 23501-1559

Copyright 1987 Kent Paul Dolan.			How about if we keep the human
All Rights Reserved.  Author grants free	race around long enough to see
retransmission rights, recursively only.	a bit more of the universe?