Reporting Errors in Production: The Type/Stack Conflict

Oren Rosen
Oren Rosen
Apr 9, 2019 7 min read
thumbnail for this post

Lately, my team and I have started reporting errors to Sentry, and I hate to admit it, but only then did I notice that my errors weren’t telling me much.. I would either get my custom error types, or a relevant stack trace, but not both.

In this post I’ll go over these concerns and suggest simple ways to get both information types. After all, the main reason we have errors is for us to have something to report on when our application fails, isn’t it?

What Reasonable Reporting Should Look Like

First, let’s review some of the main concerns when handling errors in production:

The Message should be composed of all the messages in the error chain.

The Stack Trace should be from where the error was created, not from where it was reported.

The Type should preferably be a name that means something to us. Take for example this mock Sentry interface:

Sentry errors interface mock

In the above example, sql.updateArticleError is the only name which means something to us. All of the other types belong to the framework which created them.

To understand how we can fill these requirements, we first need to talk about a popular player in this field, pkg/errors.

pkg/errors

pkg/errors, created in 2016 by Dave Chaney, is a replacement for the standard error package. It provides great functionality for wrapping and unwrapping an error, which you can achieve by these two main functions:

// Wrap annotates cause with a message.
func Wrap(cause error, message string) error {}

// Cause unwraps an annotated error.
func Cause(err error) error {}
Not only that, it also provides a stack trace for the errors it creates. To get the stack trace, just cast the err to stacker:
type stacker interface {
    StackTrace() pkgErrors.StackTrace
}

if err != nil {
	if tracer, ok := err.(stacker); ok {
		errStack := tracer.StackTrace()
		fmt.Printf("error stack trace:\n%+v\n\n", errStack)
	}

	cause := pkgErrors.Cause(err)
	if tracer, ok := cause.(stacker); ok {
		origStack := tracer.StackTrace()
		fmt.Printf("original stack trace:\n%+v\n\n", origStack)
	}
}
Thanks to this package, the message we see in Sentry will be composed of all the messages during the chain. Unfortunately, if we want the relevant stack trace and our own custom type, we will encounter a small issue.

The StackTrace/Type conflict

Sentry unwraps the error before actually reporting it. You can see in the CaptureError implementation, that it uses Cause on the error reported.

The base of the conflict lies in the fact that the same error is used both for the type and for the stack trace. We can easily have a custom type or a relevant stack trace, but not both.

The next pseudo flow gives us a better idea of what’s happening:

publishing.Service is calling articlesRepository to update an article, which is calling its dependency, an external db handler.

Imagine that extsql.db returns an error. How should we treat it in our articlesRepository?

1) Create a new error

We can create a new error using pkg/errors:

if err := r.db.Exec(...); err != nil {
      return pkgErrors.Errorf("articlesRepository.Update: %s", err.Error())
}
In this case, we will have the stack trace from here, which is where it was created, but the type will be meaningless to us. This is because it’s the type of the inner struct of pkg/errors.

2) Wrap a custom error

Assuming we have this const error in our sql package:

package sql

type articlesRepositoryError string
func (e articlesRepositoryError) Error() string { return string(e) }

const updateArticleError = articlesRepositoryError("update article failed")

We can wrap it like this:

if err != nil {
	return pkgErrors.Wrapf(updateArticleError, "db failed exec: %s", err)
}

As a result, we will see sql.articlesRepositoryError as the type,but unfortunately we will have an irrelevant stack trace. This is because our error doesn’t implement the stacker interface.

3) Wrap the error from the db

What’s even worse, is when we wrap an external error, which may not follow pkg/errors practice:

if err := r.db.Exec(...); err != nil {
	return pkgErrors.Wrap(err, "articlesRepository.Update")
}

If the external sql supports pkg/errors and we aren’t interested in our own type, it might be ok and we’ll want to do just that. But in some other cases, this error might just be errors.String. We won’t have either a stack trace,or a meaningful type. So what’s the solution if we don’t want to just implement this stacker interface in all of our custom types? Compromise…

Compromise 1: Keep the Stack Simple, Complicate the Type

How will you make your custom error implement the stacker interface without actually implementing it? You can use embedded struct.

If our custom error is a struct embedding stacker, we can use it as the origin error without worries.

package sql

// This is our custom type sql.updateArticleError, which embeds stacker
type articlesRepositoryError struct {
	error
	stacker
}

func (r *articleRepo) UpdateArticle(...) error { 
	// do some stuff

	if err := r.db.Exec(...); err != nil {
		// first create a new error using pkg/errors
		pkgErr := pkgErrors.Errorf("articles.Update: %s", err)

		// cast to stacker, and use it to create our type
		tracer, _ := pkgErr.(stacker)
		return &articlesRepositoryError{
			error:   pkgErr,
			stacker: tracer,
		}
	}

	// do other stuff
	return nil
}

This way we will get our own type sql.articlesRepositoryError nd the stack trace. We won’t need to do any extra work on the reporting side, thanks to Sentry’s support of the stacker interface.

Compromise 2: Keep the Type Simple, Put Extra Work For the Stack Trace

Since not everyone likes to embed a struct on a daily basis, let’s stick with the type we defined earlier and just wrap it:

package sql

type articlesRepositoryError string
func (e articlesRepositoryError) Error() string { return string(e) }
const updateArticleErr = articlesRepositoryError("article update failed")

func (r *articleRepo) UpdateArticle(...) error {
	// do some stuff

	if err := r.db.Exec(...); err != nil {
		// wrapping our custom error
		return pkgErrors.Wrapf(updateArticleErr, "db: %s", err)
	}

	// do other stuff
	return nil
}
When this error is reported, we will see our type sql.articlesRepositoryError, but not the relevant stack trace, since updateArticleErr doesn’t implement stacker. The compromise will be to report the stack as an extra parameter. Instead of the main view, we will see it in the ‘additional data’ section in Sentry. We will get the deepest stack trace we can find by casting the errors in the chain to stacker:

type causer interface {
	Cause() error
}

func deepestStackTrace(err error) pkgErrors.StackTrace {
	var stackTrace pkgErrors.StackTrace
	for err != nil {
		if tracer, ok := err.(stacker); ok {
			stackTrace = tracer.StackTrace()
		}

		if cause, ok := err.(causer); ok {
			err = cause.Cause()
			continue
		}

		err = nil
	}

	return stackTrace
}

After you get the stack trace, you can pass it to Sentry as a slice of strings. Sentry can handle it pretty well.

func stackTraceStrings(err error) []string {
	toRet := []string{}
	for _, frame := range deepestStackTrace(err) {
		toRet = append(toRet, fmt.Sprintf("%+v", frame))
	}

	return toRet
}

A small improvement is to add this stack trace only in the cases where the original error isn’t a stacker:

func shouldAddStack(err error) bool {
	cause := pkgErrors.Cause(err)
	_, ok := cause.(stacker)
	return !ok
}

When putting it all together, our Sentry abstraction will look like this:

type extra map[string]interface{}

// implements sentry.Interface
func (e extra) Class() string {
	return "extra"
}

func ReportError(err error) {
	params := map[string]interface{}{}
	if shouldAddStack(err) {
		params["stackTrace"] = stackTraceStrings(err)
	}

	client := raven.New("some@dsn")
	client.CaptureError(err, nil, extra(params))
}
It seems like manually adding the stack trace is a lot of work, but remember, it is only done once so it might be a better solution than doing extra work on every error type.

What’s next:

What we just saw, in my opinion, are very basic concerns when it comes to reporting errors. In the next post, we’ll see how we can wrap an error of a dependency, but still keep our custom error type. Not surprisingly, it can easily be done using a simple custom errors package.

Regarding the “deepest stack trace” it’s not a new idea. If it is implemented in Sentry, it will resolve the whole conflict issue.

Meanwhile, the Go 2.0 errors inspection proposal, which just came out a couple of months ago, shares some pkg/errors ideas. It is very interesting to read and I’m excited to see what will happen with errors in Go in the future.