Numerical Methods
Lesson 3 of 3 6 min +40 XP

Error & Precision

Why computers only approximate.

What you'll learn

  • Understand error & precision
  • Apply it to real examples
  • Check yourself with a quiz
The ruler with limited marks

A ruler marked only to millimeters can't measure a hair's width exactly — there's always rounding. Computers face the same limit, so numerical answers carry a small, trackable error.

Truncation and rounding

Truncation error comes from stopping an infinite process early; rounding error from finite decimal storage. Good methods keep both small.

Knowledge Check

+10 XP / correct

1. Rounding error comes from…

2. Stopping an infinite process early causes…