EECS 381: Autograder Information and Policies
To submit your project, please follow the instructions on the main web
page about the Submission Program.
Overview
Your project will be graded by a computer program, called the "autograder."
Our intent with this system is that if your project fully
meets the project specifications in the assignment handout as updated on
the web site, then you should get full grade credit.
While this system is faster and more consistent than human graders, it is necessarily less flexible and pickier. The grading system is designed to be no pickier than a conscientious software development manager would be - industrial grade software has to meet its specifications, not just be "pretty close." Furthermore, the grading system is actually similar in structure and technology to the automated testing systems that many good software organizations use. So as picky as it can be, it is not unrealistic.
Any announcements about the grading system and due dates will be made by
email to the announcements address. Check there first if you have questions about these matters.
Due Dates and Times, Early Submission
We will announce when the autograder is ready about one week or more before the project due date. The due date is normally given in the course schedule, the Project assignment document, the course web page, or in an announcement broadcast to the announcements list. If the dates are different, announcements on the web page or announcements list take precedence over the project document, which takes precedence over the course schedule.
The project is due before midnight on the due date; more exactly it must be submitted before 11:59:59 PM on the due date.
For purposes of early submission credit, a day is defined as midnight-to-midnight. Thus if the due date is Friday, March 17, a project submitted before midnight Wednesday March 15
qualifies for the two-days-early bonus. A project submitted after Wednesday midnight but before midnight Thursday March 16 qualifies for the one-day-early bonus.
A project submitted after Thursday midnight but before midnight of the due date of Friday March 17 is on-time and earns no bonus.
A project submitted after 11:59:59 Friday March 17 is late, and will not be accepted, and will earn zero credit, even if it was only a fraction of second late.
Important concepts about the autograder
-
The autograder is not a debugger. You must test and debug your program separately before submitting it. There are not enough submissions allowed, and the test feedback is not specific enough, to use it to debug your program.
- We will not reveal any details about the autograder tests. Each test has a descriptive name, such as "garbage input" so that if your program fails it, you have a rough idea where to look, but only a rough idea. A good analogy is that the autograder functions like a cranky dissatisfied customer on the phone who says, "Your program crashes whenever I give it bad input!!!" and then hangs up. It is up to you to figure out what the problem is by testing your program in detail. Ask for help if you don't know how to test your program thoroughly enough. Look on the course web page for an essay on program testing. Also keep in mind that almost all tests have to involve multiple functions of the program - e.g. a test for whether the program outputs certain data correctly means that the program must correctly accept and store some test data first. Thus the test names suggest what is being emphasized, but not exactly what commands or functions are being tested.
-
The first test(s) done by the autograder is whether your program matches the
sample output supplied with the project specifications. Your program should produce output
exactly like that in the supplied sample.
Before submitting your program, you should check that it exactly reproduces the sample; use I/O redirection and compare the files with a utility to be sure the match is exact
(see the main course web page for information about these tools).
If it isn't, don't bother to submit the program - it probably will fail many of the tests. Track down and fix the differences.
- If your comparison shows that your program matched the sample output, but the autograder fails it on these first tests, then either
(1) something is wrong with the autograder (probably not the case);
(2) Your program ran like it should, but returned non-zero from main(), which causes the autograder to believe that some kind of error or failure occurred,
meaning your program failed that test. Check that your program returns only 0 from main (the "normal" or "everything's OK" value) (has happened to a few students).
(3) you have found an incompatibility between your compiler and the autograder compiler (unusual);
or (4) your program relies on undefined behavior in C/C++, and happens to work correctly on your machine/compiler, but not on the autograder's (most likely to be the problem).
See the FAQs about this, and ask for help if you are stumped.
What is Graded?
-
We will grade for both output correctness and specification
correctness, and for selected projects, code and design quality which we will do by by human grading.
-
Output correctness means an exact match to the output of our
version of the program. Extra whitespace differences are
allowed in most (but not necessarily all) cases, but your program must produce the correct spelling,
punctuation, and exact data values.
-
Specification correctness is based on whether your program uses the
structures, functions, and program organization specified in the
project specifications. Exact spelling of specified names is required.
Specification correctness will be determined by whether the autograder can use a component of your project in accordance with the specifications - have you correctly implemented the required interface and its behavior?
-
If we are doing a quality grading on the project, then after all projects are submitted, we begin the code quality grading of the project using the last submitted version of your code.
Working with the Autograder
- The autograder keeps on file the last version it received of your project and the score it received.
This last version and score is your official submission and score that will be used. This means that if you submit a
version that gets a lower score than the previous one, you are stuck with the lower score.
Your only recourse is to resubmit a corrected version. This is one reason why working right up to the deadline is a bad idea - one slip and you can end up with a zero score for the project.
-
You can submit your project to the grading system as often as you want, but you will get feedback for only the first two submissions in a day. The
feedback is an email response from the grading system that shows your
scores. A day is defined as midnight-to-midnight. The scores feedback includes how many bonus points are earned, but only the final submission bonus points count.
-
Additional submissions after the first two will be graded and kept
on file, but you will not get any feedback on them - just an
acknowledgement message. You can use the query feature of the autograder to find out the total score that is on record, but only if you have not submitted that day. You should use these extra submissions with great caution - there is nothing to protect you from ending up with a lower score and not knowing about it until the next day.
-
Wait for the feedback message before submitting again. If you send multiple submissions one right after the other, when the system is heavily loaded, your submissions might be
processed out-of-order, making the feedback confusing, and possibly resulting in a lower score being on record. Do not submit again until you have gotten the feedback for your previous
submission.
This is important because more complex projects can take a several minutes to process, and we might be sharing the grading machine with other courses, and so it can get heavily loaded.
-
The autograder keeps only the last submission that it processed. If you submit again without waiting for the feedback, the autograder might process your submissions out of order, due to how the email system works when the machine is too loaded to keep up. The last
submission that the autograder processed before the final deadline is the one that will be used for your final project score, even if its score is lower than other versions that you submitted. Also,
early submission days will be counted based on the last submission kept by the autograder.
-
The autograder gets a time-stamp on your submission
email - this is when you sent it, not when it was received or processed by
the grading system. It uses this submission time to decide whether you
get feedback on the submission, or whether it was submitted on or before time.
This time-stamp is not affected by the load on the autograder machine. But the autograder does not use the time-stamp to determine which submission to use for your final project score. Your final project score is always based on last submission processed by the autograder, regardless of when you sent it.
-
If your feedback is delayed due to email system or autograder machine
load, and it causes problems for you, then it is an unfortunate consequence of your working up against the deadline. We will not give you extra time or special consideration as a result.
Using the Autograder Feedback
-
The autograder is not a debugger. The autograder provides feedback only to help you deal with the inflexibility of an automated grading system, not to help you debug
your project. With only two feedbacks
per day, you really can't use the grading system as a debugger.
-
The first test(s) done by the autograder is whether your program matches the
sample output supplied with the project specifications. Your program should produce output
exactly like that in the samples. The remaining tests check
the output of your program under other conditions which will be briefly
identified in the feedback message.
- The sample outputs are designed to (1) illustrate the overall program behavior; and (2) contain every unique text string used by the program (these were supplied with the project specs).
By checking against the samples, you can quickly find and fix any simple typos or spelling errors that crept in.
The samples are not a complete test of the program! You must perform additional testing to make sure your program performs correctly before you submit it.
-
Except for the first test(s), we will not tell you exactly what the test inputs are.
Experience shows that revealing the specific tests leads to students' programming to pass the
tests, rather than programming to the specifications.
Thus you can expect the feedback to be incomplete and limited in detail - the
project specifications and your own testing provide the necessary detail
to get your project working correctly.
-
Usually our specification tests are those listed in the feedback under the heading "Check of Project n Components" which usually has two parts, "compilation" and "standard output". In these, the autograder tries to compile and link together some component of yours with code of ours, such as a testing driver for a linked-list component. The testing driver is designed to see if your component conforms to the specifications by calling all of its functions and using the component to generate some output which it then compares to that produced by our solution version of the component.
- If your component fails the "compilation" test, it means that the compile/link step failed. This is often due to your forgetting to define some function - you didn't notice this in the main project because it happened not be called from your main project code; your code for testing the component should be carefully checked to make sure you are calling all of the specified functions. A second possibility for a failure to compile is that your component does not have the specified interface, so it doesn't match up with our testing code. Check your interface declarations carefully against the specifications.
- If your component fails the "standard output" test, it means that your component did not behave in the specified way - when we used it, the produced output did not match ours. For example, suppose we try use your list component and delete something from the end of a list, but it doesn't get deleted and still shows up in the output. Such problems are probably in some function that your main code never called, so you may not have tested it thoroughly. It might also be due to some combination of conditions that you did not test.
- If you are stumped and can't figure out why your program is failing tests, first make sure you haven't missed something in the project specifications, the Corrections and Clarifications, and the FAQ for the project.
Study the essay on program testing on the main page, and review whether your testing has been comprehensive and complete.
Please do these before asking for help - it will actually be faster and will save a lot of time. Working on this before the deadline will give you time to get help if you turn out to need it.