![C++ Error Handling Benchmark - A Hacker Handling Errors.](https://i0.wp.com/johnfarrier.com/wp-content/uploads/2024/07/Blog_ErrorHandlingBenchmark.jpg?resize=640%2C320&quality=89&ssl=1)
Error handling is a critical component of many C++ applications. Even so, there are many strategies we can use to handle errors. I recently wanted to better understand the performance implications of various error-handling techniques in C++.
C++ Error Handling Experiment Setup
I decided to benchmark different techniques to see how they stack up against each other. To accomplish this, I used the Celero C++ Microbenchmarking Library. My aim was to provide clear insights into the performance of each method.
For example, std::expected
provides many benefits in terms of readability and maintainability, it’s also important to consider its performance characteristics. std::expected
is generally more efficient than exceptions because it avoids the overhead of stack unwinding, which can be significant in performance-critical applications.
I constructed a benchmark to compare the following C++ error-handling strategies:
- Use of a Return true/false on success/failure (the Baseline)
- Use of an error code return value
- Use of
std::expected
- Use of
std::optional
- Use of
std::variant
- Use of a Callback
- Use of
std::exception
The C++ error handling benchmark was constructed using the Celero C++ Microbenchmarking Library.
The code for the C++ error handling benchmark can be found here: DigitalInBlue/celeroErrorHandlingBenchmark: A Microbenchmark of various C++ error handling techniques. (github.com)
Error Handling Experiment Results
Here is the raw output:
Group | Experiment | Prob. Space | Samples | Iterations | Baseline | us/Iteration | Iterations/sec | RAM (bytes) |
---|---|---|---|---|---|---|---|---|
ErrorHandling | Baseline | Null | 100 | 10000000 | 1.00000 | 0.00324 | 308480118.46 | 667648 |
ErrorHandling | Expected | Null | 50 | 10000000 | 2.18169 | 0.00707 | 141394717.49 | 667648 |
ErrorHandling | ErrorCode | Null | 50 | 10000000 | 1.01505 | 0.00329 | 303905181.58 | 667648 |
ErrorHandling | Optional | Null | 50 | 10000000 | 1.78416 | 0.00578 | 172899700.88 | 667648 |
ErrorHandling | Variant | Null | 50 | 10000000 | 2.83410 | 0.00919 | 108845906.85 | 667648 |
ErrorHandling | ErrorCallback | Null | 50 | 10000000 | 1.32270 | 0.00429 | 233219833.01 | 667648 |
ErrorHandling | Exception | Null | 10 | 5000 | 49540.11784 | 160.59420 | 6226.87 | 679936 |
To simplify, here’s a condensed version focusing on the relative performance of these techniques:
Experiment | Baseline |
---|---|
Baseline | 1.00000 |
Expected | 2.18169 |
ErrorCode | 1.01505 |
Optional | 1.78416 |
Variant | 2.83410 |
ErrorCallback | 1.32270 |
Exception | 49540.11784 |
Given the baseline-normalized performance, using std::expected is about 2.1 times slower than
Performance Comparison
Excluding std::exception, the results show that the various techniques are generally on par with each other.
![Baseline Values for Different Experiments (Sorted, Excluding Exception)](https://i0.wp.com/johnfarrier.com/wp-content/uploads/2024/07/Baseline-Values-for-Different-Experiments-Sorted-Excluding-Exception.png?resize=535%2C310&quality=80&ssl=1)
The C++ error handling benchmark results indicate significant differences in performance among various error handling techniques:
- Baseline: The baseline method, returning true or false, is the fastest technique, with an average time of 0.00324 microseconds per iteration.
- Expected: Using
std::expected
is about 2.18 times slower than the baseline, with an average time of 0.00707 microseconds per iteration. - Error Code: Using error codes is almost as fast as the baseline, with an average time of 0.00329 microseconds per iteration.
- Optional: Using
std::optional
is slower, taking 0.00578 microseconds per iteration, which is about 1.78 times slower than the baseline. - Variant: Using
std::variant
is the slowest among the non-exception techniques, with an average time of 0.00919 microseconds per iteration, making it about 2.83 times slower than the baseline. - Error Callback: Using error callbacks has a moderate impact on performance, with an average time of 0.00429 microseconds per iteration, which is about 1.32 times slower than the baseline.
- Exception: Using
std::exception
is significantly slower than all other techniques, with an average time of 160.59420 microseconds per iteration, making it about 49,540 times slower than the baseline.
Detailed Analysis and Observations
- Trade-offs Between Readability and Performance:
- While
std::expected
and similar constructs (e.g.,std::optional
,std::variant
) offer more readable and maintainable code, but they do come with a measurable performance cost. These constructs are advantageous for applications where readability and maintainability are critical, and performance is less of a concern. However, in performance-critical applications, simpler error-handling methods might be preferred. - While it is generally advisable to pick one strategy for error handling to use throughout an application, there may be times when there is a “default” error handling strategy (such as using
std::expected
) and a “high-performance” alternative to use in very specific cases.
- While
- Error Handling Strategies in Different Contexts:
- High-Performance Systems: Systems requiring high performance, such as real-time systems or high-frequency trading platforms, should prefer error-handling techniques with minimal overhead, such as return codes or error callbacks.
- Modern C++ Applications: Applications that benefit from modern C++ paradigms and have more lenient performance requirements can take advantage of
std::expected
andstd::optional
for better code expressiveness.
- Impact of Exceptions:
- The significant overhead of exceptions (
49,540
times slower than the baseline) highlights why exceptions are often avoided in performance-critical code. The stack unwinding process during exception handling introduces considerable latency. - However, exceptions can be extremely useful for handling unexpected or rare error conditions in less performance-sensitive parts of the application.
- The significant overhead of exceptions (
- Benchmark Environment and Conditions:
- It’s important to note that benchmarks can be influenced by the environment in which they are run. (e.g., CPU, compiler optimizations, etc.) The results provided here are specific to the benchmark conditions and should be validated in the context of your specific application.
- Memory Usage Considerations:
- While runtime performance is crucial, memory usage is also an important factor. The benchmark results indicate similar memory usage across most techniques (excluding exceptions). This suggests that the choice of error-handling method may have minimal impact on memory footprint.
- Composability and Integration:
- Modern error-handling methods like
std::expected
andstd::optional
integrate well with other modern C++ features, such as coroutines and range-based algorithms, potentially simplifying complex error propagation scenarios.
- Modern error-handling methods like
Recommendations for C++ Error Handling
Based on the benchmarks, the choice of C++ error-handling technique should be guided by the specific requirements of the application. Here are some recommendations:
- Performance-Critical Applications: Use simple error handling methods like return codes or error callbacks to minimize overhead.
- Modern, Maintainable Code: Use
std::expected
orstd::optional
for better code clarity and maintainability, especially if the slight performance overhead is acceptable. - Exception Handling: Reserve exceptions for unexpected or infrequent errors where the overhead is justified by the need for comprehensive error information.
These C++ error handling benchmarks highlight the importance of choosing the right error-handling technique based on your application’s performance requirements. While std::expected
and other modern C++ features offer improved readability and maintainability, their performance implications should not be overlooked, especially in performance-critical applications.
Further Reading
- performance – Are exceptions in C++ really slow? – Stack Overflow
- P2544R0: C++ exceptions are becoming more and more problematic (open-std.org)
- Current hardware trends make C++ exceptions harder to justify | Hacker News (ycombinator.com)
- How Slow Are C++ Exceptions? — Wild Wild Wolf
- P1886R0: Error speed benchmarking (open-std.org)
Discover more from John Farrier
Subscribe to get the latest posts sent to your email.
I’ll suggest my own error speed benchmarking paper (https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p1886r0.html) as additional further reading. The results are similar to yours.
Thanks for this article! I think it would have been good to differentiate the performance of ‘using’ the exception mechanism and ‘throwing’ an actual exception. It seems like your performance stat that is almost 50,000 times slower is for actually ‘thrown’ exceptions and the subsequent stack unwinding. I personally am a very heavy user of the exception handling mechanism because in heavily nested function calls it is often by far the cleanest way of handling errors. However, the way I use exceptions is that they are really just used for exceptional error conditions and not during regular program execution. So, in that sense it doesn’t really matter how slow the exception handling mechanism is when an exception is thrown. What matters is if there is any cost to using exceptions when no exception is thrown in terms of speed and memory. My assumption from my own past reading is that modern exception mechanisms have virtually no cost when no exception is thrown. Would have been nice to include this stat for comparison.