Rework handling of Stripe webhooks

[begedin]

Problem

From the stripe documentation: https://stripe.com/docs/webhooks#responding-to-a-webhook

To acknowledge receipt of a webhook, your endpoint should return a 2xx HTTP status code. Any other information returned in the request headers or request body is ignored. All response codes outside this range, including 3xx codes, will indicate to Stripe that you did not receive the webhook. This does mean that a URL redirection or a "Not Modified" response will be treated as a failure.

If a webhook is not successfully received for any reason, Stripe will continue trying to send the webhook once an hour for up to 3 days. Webhooks cannot be manually retried after this time, though you can query for the event to reconcile your data with any missed events.

Right now, if everything goes OK, or we outright ignore an event, we send a 200, so the webhook won't be sent anymore.

If something goes wrong, we send a 400, so according to documentation, the webhook will be resent every 3 hours for 3 days.

That sounds like OK behavior for me, but I'm not completely sure.

Subtasks

• Figure out if the current behavior is acceptable. Change if it it is not.
• Possibly start sending a 204 (No Content) instead of 200, since the content is ignored by stripe anyway.

References

• https://stripe.com/docs/webhooks#responding-to-a-webhook

[joshsmith]

After reading Stripe's documentation on idempotency and thinking through the edge cases more, I think we need to do the following.

First, read Stripe:

If your webhook script performs complex logic, or makes network calls, it's possible the script would timeout before Stripe sees its complete execution. For that reason, you may want to have your webhook endpoint immediately acknowledge receipt by returning a 2xx HTTP status code, and then perform the rest of its duties.

Webhook endpoints may occasionally receive the same event more than once. We advise you to guard against duplicated event receipts by making your event processing idempotent. One way of doing this is logging the events you've processed, and then not processing already-logged events.

Given this, we should implement:

• A StripeEvent model that logs the event id and a status which is one of unprocessed (default), processing, processed, errored, or unhandled
• We should always respond with a 200 to webhooks
• If we receive the same event from Stripe again and the StripeEvent is unprocessed or errored or unhandled, then we try to process again
• If it’s processing or processed, we do nothing
• The event handling itself can be completely async as a Task

This gives us the flexibility to be idempotent in handling events from Stripe to our webhooks and it allows us the ability to fix errors that happened during the event processing step.

We should make sure that the internal behavior of the event processing itself is handled within a transaction so that the event is completely idempotent, and think through the consequences of replaying old events, even if they were previously errored or unhandled.