What a test is

An automated test is a small program that runs part of your code and checks the result against what should happen. Run after every change, a set of tests answers one question quickly: does everything that used to work still work?

Tests do not prove that code is perfect. They prove specific things, every time, without relying on memory or patience. That is what makes changing code safe.

Kinds of tests

Unit tests
Check one function or rule on its own, such as a single grade boundary. Fast and precise.
Property tests
Generate many random cases and check a rule that must always hold, such as “the planner agrees with the grading engine”.
Browser tests
Open the real page in a browser controlled by a program, click and type like a visitor, and check what appears.
Content checks
Check data before it is used, such as every project having the fields the page needs.
Regression tests
Written after a bug: they fail on the old code and pass on the fixed code, so the bug cannot quietly return.

How to write a test that can fail

In GPA Tracker

GPA Tracker is checked with 564 automated checks: 193 unit tests and 371 window-level checks written for the original interface and run against a stand-in for its toolkit, so that interface could be tested without opening windows. Mutation testing went one step further: 89 bugs were planted deliberately, and the tests caught every one. That is the strongest evidence that the tests fail when they should.

In this site

This site has fourteen test files with more than two hundred and seventy checks. Browser tests cover the keyboard, text contrast, layouts from 320 to 1920 pixels wide, the contact form, the dialogs, both themes, the security policy and the project and note pages. A content check validates the data file in about a second; it was itself tested with eight deliberate mistakes and caught all eight, each with a message naming the problem.

A GitHub workflow runs everything on every push, so a mistake shows up as a failed check on the change that caused it.

What tests cannot see

A test checks only what it was written to check. On this site the tests passed while a hidden spam-trap checkbox was visible, and while decorative chips covered the portrait on phones (see the debugging note, cases 5 and 8). Tests and looking at the result belong together.