Google Workspace Status Dashboard

INCIDENT REPORT (Combined GCP / WS)

Summary

On 22 September 2021 14:10 US/Pacific, Google Cloud, Workspace, and Geo (Maps) customers in the United States (U.S.) incorrectly received alert notifications on the Google Cloud Console user interface and an email regarding an issue with processing automatic payments. The issue persisted for 11 hours, 20 minutes.

All customers in the United States who use credit card billing were inadvertently notified via email “Your payment information could not be processed. Visit the payment overview page to make sure your payment information is up to date and to pay any outstanding charges.” Customers also received a notification alert in the Google Cloud Console to update their primary payment method. After customers attempted to update their payment information in the admin console, they received a message that the payment could not be processed.

We sincerely apologize for the inconvenience caused by the incorrect notification to our customers. Our engineering team has deployed a fix for this and has ensured that account settings are restored. We are taking additional steps to prevent similar problems in the future, as outlined below.

Root Cause

The trigger for this service disruption was an update to the payment configuration intended to support virtual credit cards (one-time credit card numbers from Visa and MasterCard). This update unintentionally caused a situation in which no credit card payments were able to be processed in the U.S. region and resulted in payment failure notifications being sent out to affected customers. Google's standard change control policy is to roll out all production updates progressively; however, in this specific case, errors in the rollout process meant that the change rolled out much more rapidly. Addressing this is a key part of our corrective actions.

Remediation and Prevention

Google engineers were alerted to the issue on 22 September 2021 15:28 US/Pacific and immediately started an investigation. Engineers identified an unexpected increase of payment request failures and initiated a rollback of the configuration change at 17:42, which resolved the issue causing the incorrect email notifications as well as the Cloud Console alert notifications.

Once it was confirmed that affected customers were no longer prompted with the notification alert and impact had ended, engineers began mitigation work to update account state back to the proper payment settings to prevent a recurrence.

To prevent recurrence and reduce the impact of similar events, we are taking the following actions:

Improving our payments infrastructure monitoring and alerting configuration with “early indicator” signals for faster detection and notification of widespread payment failures. Speeding up payment system mitigation procedures by automating the root causing of problems, and increasing the rate at which recovery steps can be processed. Ensuring that the class of change which resulted in this incident is pushed progressively, with time to detect adverse impact at small scope.

Detailed Description of Impact

Google Cloud Console - Google Cloud Platform and Geo Maps

U.S. customer accounts received an alert notification in their admin account warning that they needed to verify their payment method. When attempting to update the preferred credit card method, the user interface was unresponsive.

Customers received an email warning message informing them they needed to verify their payment method. They also might have received an email warning that their project was going to be suspended. After the issue was resolved, account alert notifications disappeared, and the account state was set back to the proper payments setting. Some customers may have observed restrictions and were unable to create new resources. These restrictions were corrected when the account was returned to good standing.

The period of impact was between 14:10 and 00:30.

Google Admin Console - Google Workspace

U.S. customer accounts received an alert notification in their admin account warning that they needed to verify their payment method. When attempting to update the preferred credit card method, the user interface was unresponsive.

Customers received an email warning they needed to verify their payment method. When they attempted to update the credit card they are currently using, they were unable to do so. After the issue was resolved, account alert notifications disappeared, and the account state was set back to the proper payments setting.

The period of impact was between 14:10 and 01:30.

Additional Information for Customers

The impacted geographical scope used in the preliminary incident statement was incorrect. While impact initially was suspected to be global, our engineers confirmed impact only affected U.S. customers with credit cards stored for automatic payments. The corrected details for impacted regions / zones are contained in the above report.

We apologize for the inconvenience this issue may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support

(All Times US/Pacific)

Incident Start: 22 September 2021 14:10

Incident End: 23 September 2021 01:30

Duration: 11 hours, 20 minutes

Affected Services and Features:

• Google Workspace: Admin console user interface received warning notification for US customers with a self-serve billing account.

Regions/Zones: US

Description:

Google Workspace customers received a warning message on the Admin console user interface and an email regarding an issue with processing their account payment. Impacted customers were redirected to update payment information. Attempts to update the primary payment method caused the admin console to become unresponsive. Initial investigation of this issue revealed the impact was triggered after an update to the payment configuration was released during the impact time frame.

Customer Impact:

Google Workspace customers received a warning message on the Admin console user interface and an email: “Your payment information could not be processed. Visit the payment overview page to make sure your payment information is up to date and to pay any outstanding charges.”

• After customers clicked on the “Fix It” option, a pop-up message was displayed, prompting them to update their primary payment method.
• The console user interface was unresponsive after attempting to modify an existing payment method.

After resolution, the account warning message disappeared, and the account state was set back to the proper Payments setting.

Additional details:

Our engineering team has mitigated the customer impact after complete rollback of the changes that triggered the issue.

[Mitigation Underway] Customers may receive "Payment failed" notifications. We will provide more information by Wednesday, 2021-09-22 23:30 US/Pacific.

The mitigation is expected to complete by Wednesday, 2021-09-22 23:30 US/Pacific.

Error Message: “Your payment information could not be processed. Visit the payment overview page to make sure your payment information is up to date and to pay any outstanding charges.”