C++ Error Handling Strategies – Benchmarks and Performance

Error handling is a critical component of many C++ applications. Even so, there are many strategies we can use to handle errors. I recently wanted to better understand the performance implications of various error-handling techniques in C++.

C++ Error Handling Experiment Setup

I decided to benchmark different techniques to see how they stack up against each other. To accomplish this, I used the Celero C++ Microbenchmarking Library. My aim was to provide clear insights into the performance of each method.

For example, std::expected provides many benefits in terms of readability and maintainability, it’s also important to consider its performance characteristics. std::expected is generally more efficient than exceptions because it avoids the overhead of stack unwinding, which can be significant in performance-critical applications.

I constructed a benchmark to compare the following C++ error-handling strategies:

• Use of a Return true/false on success/failure (the Baseline)
• Use of an error code return value
• Use of std::expected
• Use of std::optional
• Use of std::variant
• Use of a Callback
• Use of std::exception

The C++ error handling benchmark was constructed using the Celero C++ Microbenchmarking Library.

The code for the C++ error handling benchmark can be found here: DigitalInBlue/celeroErrorHandlingBenchmark: A Microbenchmark of various C++ error handling techniques. (github.com)

Error Handling Experiment Results

Here is the raw output:

Group	Experiment	Prob. Space	Samples	Iterations	Baseline	us/Iteration	Iterations/sec	RAM (bytes)
ErrorHandling	Baseline	Null	100	10000000	1.00000	0.00324	308480118.46	667648
ErrorHandling	Expected	Null	50	10000000	2.18169	0.00707	141394717.49	667648
ErrorHandling	ErrorCode	Null	50	10000000	1.01505	0.00329	303905181.58	667648
ErrorHandling	Optional	Null	50	10000000	1.78416	0.00578	172899700.88	667648
ErrorHandling	Variant	Null	50	10000000	2.83410	0.00919	108845906.85	667648
ErrorHandling	ErrorCallback	Null	50	10000000	1.32270	0.00429	233219833.01	667648
ErrorHandling	Exception	Null	10	5000	49540.11784	160.59420	6226.87	679936

To simplify, here’s a condensed version focusing on the relative performance of these techniques:

Experiment	Baseline
Baseline	1.00000
Expected	2.18169
ErrorCode	1.01505
Optional	1.78416
Variant	2.83410
ErrorCallback	1.32270
Exception	49540.11784

Given the baseline-normalized performance, using std::expected is about 2.1 times slower than

Performance Comparison

Excluding std::exception, the results show that the various techniques are generally on par with each other.

The C++ error handling benchmark results indicate significant differences in performance among various error handling techniques:

1. Baseline: The baseline method, returning true or false, is the fastest technique, with an average time of 0.00324 microseconds per iteration.
2. Expected: Using std::expected is about 2.18 times slower than the baseline, with an average time of 0.00707 microseconds per iteration.
3. Error Code: Using error codes is almost as fast as the baseline, with an average time of 0.00329 microseconds per iteration.
4. Optional: Using std::optional is slower, taking 0.00578 microseconds per iteration, which is about 1.78 times slower than the baseline.
5. Variant: Using std::variant is the slowest among the non-exception techniques, with an average time of 0.00919 microseconds per iteration, making it about 2.83 times slower than the baseline.
6. Error Callback: Using error callbacks has a moderate impact on performance, with an average time of 0.00429 microseconds per iteration, which is about 1.32 times slower than the baseline.
7. Exception: Using std::exception is significantly slower than all other techniques, with an average time of 160.59420 microseconds per iteration, making it about 49,540 times slower than the baseline.

Detailed Analysis and Observations

1. Trade-offs Between Readability and Performance:
   • While std::expected and similar constructs (e.g., std::optional, std::variant) offer more readable and maintainable code, but they do come with a measurable performance cost. These constructs are advantageous for applications where readability and maintainability are critical, and performance is less of a concern. However, in performance-critical applications, simpler error-handling methods might be preferred.
   • While it is generally advisable to pick one strategy for error handling to use throughout an application, there may be times when there is a “default” error handling strategy (such as using std::expected) and a “high-performance” alternative to use in very specific cases.
2. Error Handling Strategies in Different Contexts:
   • High-Performance Systems: Systems requiring high performance, such as real-time systems or high-frequency trading platforms, should prefer error-handling techniques with minimal overhead, such as return codes or error callbacks.
   • Modern C++ Applications: Applications that benefit from modern C++ paradigms and have more lenient performance requirements can take advantage of std::expected and std::optional for better code expressiveness.
3. Impact of Exceptions:
   • The significant overhead of exceptions (49,540 times slower than the baseline) highlights why exceptions are often avoided in performance-critical code. The stack unwinding process during exception handling introduces considerable latency.
   • However, exceptions can be extremely useful for handling unexpected or rare error conditions in less performance-sensitive parts of the application.
4. Benchmark Environment and Conditions:
   • It’s important to note that benchmarks can be influenced by the environment in which they are run. (e.g., CPU, compiler optimizations, etc.) The results provided here are specific to the benchmark conditions and should be validated in the context of your specific application.
5. Memory Usage Considerations:
   • While runtime performance is crucial, memory usage is also an important factor. The benchmark results indicate similar memory usage across most techniques (excluding exceptions). This suggests that the choice of error-handling method may have minimal impact on memory footprint.
6. Composability and Integration:
   • Modern error-handling methods like std::expected and std::optional integrate well with other modern C++ features, such as coroutines and range-based algorithms, potentially simplifying complex error propagation scenarios.

Recommendations for C++ Error Handling

Based on the benchmarks, the choice of C++ error-handling technique should be guided by the specific requirements of the application. Here are some recommendations:

• Performance-Critical Applications: Use simple error handling methods like return codes or error callbacks to minimize overhead.
• Modern, Maintainable Code: Use std::expected or std::optional for better code clarity and maintainability, especially if the slight performance overhead is acceptable.
• Exception Handling: Reserve exceptions for unexpected or infrequent errors where the overhead is justified by the need for comprehensive error information.

These C++ error handling benchmarks highlight the importance of choosing the right error-handling technique based on your application’s performance requirements. While std::expected and other modern C++ features offer improved readability and maintainability, their performance implications should not be overlooked, especially in performance-critical applications.

Further Reading

• performance – Are exceptions in C++ really slow? – Stack Overflow
• P2544R0: C++ exceptions are becoming more and more problematic (open-std.org)
• Current hardware trends make C++ exceptions harder to justify | Hacker News (ycombinator.com)
• How Slow Are C++ Exceptions? — Wild Wild Wolf
• P1886R0: Error speed benchmarking (open-std.org)