Benchmarking Methods

Learning Objectives

• Design fair benchmarks for code comparisons.
• Interpret benchmark results correctly.
• Avoid misleading measurements.

Benchmarking Principles

• Keep environment and input constant.
• Run multiple iterations and use median.
• Warm up runtime when needed.
• Measure only the target code path.

Common Pitfalls

• Comparing different workloads.
• Timing with too few iterations.
• Ignoring I/O overhead in compute tests.
• Drawing conclusions from one run.

Practice Tasks

1. Benchmark two implementations of the same logic.
2. Run each test 10 times and record median.
3. Report both timing and memory observations.

Summary

Benchmarking is useful only when experiment design is controlled.