Syllabus
Judgment Lab
An 8-week course that rebuilds engineering judgment through deliberate decision practice.
Course overview
This is an 8-week, cohort-based course that rebuilds engineering judgment through deliberate decision practice instead of hand-coding volume.
The problem. Engineers used to build judgment by struggling through hard problems and designs, then living with the results. With AI writing most of the code, that struggle and its feedback loop have mostly disappeared. Developers ship more but form fewer instincts about what will break, scale, or rot.
The approach. The course works like a flight simulator for engineering decisions. Learners make real calls under realistic constraints, commit to them before seeing the outcome, then get fast, honest feedback.
Audience. Engineers with 1–5 years of experience who work mostly with AI coding tools. Senior engineers join as facilitators and case authors.
Format.
- Cohort of 8–16 learners, 2 facilitators
- 8 weeks, about 4 hours per week: one 90-minute live session plus 2–2.5 hours of async exercises
- One capstone decision on a real system in weeks 7–8
Learning outcomes
By the end of the course, learners can:
- Name the trade-offs in a design before choosing one, and write down what they are giving up.
- Review AI-generated code and catch subtle correctness, security, and maintainability problems.
- Predict how a system fails under load, change, and time, and design for those failure modes.
- Debug an unfamiliar incident with a structured hypothesis-and-test process.
- Judge their own confidence accurately and know when to slow down or ask for help.
- Explain and defend a technical decision to peers and senior engineers in writing.
Core principles
Every exercise is built on the same loop: decide, commit, see the outcome, reflect.
- Predict before reveal. Learners write their answer and a confidence score (0–100%) before seeing the real outcome or asking AI. Committing first is what makes the feedback stick.
- Real cases over toy problems. Cases come from real postmortems, design docs, and PRs, preferably from your own codebase.
- Reasoning is graded, not just answers. A right answer with weak reasoning scores lower than a wrong answer with sound reasoning.
- Seniors think out loud. Recorded walkthroughs show how experienced engineers reason through the same cases. Their process is the content.
- Desirable difficulty. Some exercises ban AI on purpose. Struggle is where mental models form.
- AI generates options; humans decide. Learners may use AI to explore, but they own and defend every decision.
Course at a glance
| Week | Theme | Core exercise | Deliverable |
|---|---|---|---|
| 1 | Calibration baseline | Predict outcomes of 20 short scenarios | Confidence log v1 |
| 2 | Reviewing AI code | PRs with planted flaws | Review scorecard |
| 3 | Mental models by hand | No-AI katas: concurrency, data, performance | Kata solutions + reflection |
| 4 | Design under constraints | One-page design with stated trade-offs | Design doc v1 |
| 5 | How systems fail | Pre-mortems + incident replays | Failure-mode map |
| 6 | Decisions over time | Architecture archaeology | Decision retrospective |
| 7 | Capstone, part 1 | Real decision on a live system | Decision proposal |
| 8 | Capstone, part 2 | Defend the decision to a senior panel | Final write-up + confidence log v2 |
Assessment
Grading rewards quality of reasoning and growth over the course, not speed or right answers alone.
| Component | Weight | What’s assessed |
|---|---|---|
| Weekly deliverables | 40% | Clarity of trade-offs, quality of reasoning |
| Code review scorecards | 15% | Flaws caught, false alarms, severity ranking |
| Calibration improvement | 15% | Gap between confidence and accuracy, week 1 vs week 8 |
| Capstone proposal and defense | 30% | Framing, options, failure modes, response to pushback |
Reasoning rubric (1–4 per criterion):
- Options: Considered more than one real alternative.
- Trade-offs: Named what is being given up, specifically.
- Failure modes: Anticipated how it could go wrong.
- Reversibility: Knows which parts are hard to undo and treated them with more care.
- Confidence: Stated how sure they are and what would change their mind.
See the rubrics for what each score looks like.
AI use policy
AI is a tool for exploring options, never a substitute for the decision.
| Activity | AI allowed? | Rule |
|---|---|---|
| Predictions and confidence scores | No | Commit your own answer first |
| Week 3 katas | No | The struggle is the point |
| Code review exercises | No | Review first; you may use AI afterward to compare |
| Researching options for a design | Yes | Cite what AI suggested and what you rejected |
| Writing design docs and proposals | Limited | AI may edit wording; reasoning must be yours |
| Capstone defense | No | You answer the panel directly |
If learners use AI on a design, they add a short note: what it suggested, what they kept, and why.
Measuring course success
The course works if graduates make better decisions on the job, not just in class.
- In-course: Calibration gap shrinks from week 1 to week 8; review scorecards improve on catch rate and false alarms.
- 3 months after: Managers and tech leads rate graduates’ design docs and reviews against a comparison group.
- 6 months after: Track incidents and rework traced to design decisions by graduates vs. non-graduates.
- Ongoing: Graduates contribute new cases to the library and help facilitate later cohorts.