tag:status.paywhirl.com,2005:/historyPayWhirl Inc. Status - Incident History2024-03-25T12:05:35-07:00PayWhirl Inc.tag:status.paywhirl.com,2005:Incident/160687552023-02-08T20:15:45-08:002023-02-08T20:15:50-08:00PayWhirl Multi-Platform App<p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>20:15</var> PST</small><br><strong>Resolved</strong> - The issue was fully resolved a couple of hours ago, and we’ve been monitoring, with no incidents since the fix went out. Officially marking this one as resolved. Thanks again for your patience.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>16:46</var> PST</small><br><strong>Update</strong> - We have identified the root cause of the issue and are working on a permanent fix. During this time the service will likely be degraded. Thanks again for your patience.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>15:33</var> PST</small><br><strong>Monitoring</strong> - The system is now stable and we're monitoring for the time being. Some users could see degraded performance during this period, but for most is functioning normally. We'll keep you posted as we gather more information.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>15:03</var> PST</small><br><strong>Investigating</strong> - There is currently an issue with PayWhirl's multi-platform app located at https://app.paywhirl.com and we're investigating. Shopify merchants using our latest app are not affected. We'll provide further updates here as we work on a solution. Thank you for your patience.</p>tag:status.paywhirl.com,2005:Incident/91239622022-01-19T16:11:48-08:002022-01-19T16:11:48-08:00503 errors for some users after todays updates<p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>16:11</var> PST</small><br><strong>Resolved</strong> - The issue has been fully resolved. Thanks for your patience. The application is running smoothly.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>13:51</var> PST</small><br><strong>Monitoring</strong> - We identified an issue with our cache and have released another update. The application should be working again for all users. We'll continue to monitor and update here accordingly. If you are still experiencing issues loading the application please let our team know via live chat.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>13:42</var> PST</small><br><strong>Investigating</strong> - A few users are reporting 503 errors appearing after our upgrades from earlier today. Changes are still propagating and we're investigating further. If you are having issues we recommend clearing your local cache and trying again. We will keep you posted. Thanks again for your patience.</p>tag:status.paywhirl.com,2005:Incident/91150492022-01-19T11:24:02-08:002022-01-19T11:24:02-08:00Slow load times on PayWhirl multi-platform app.<p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>11:24</var> PST</small><br><strong>Resolved</strong> - The fix we employed yesterday seemed to work for the vast majority of users. Unfortunately, it seems like it was temporary as the system went down again. Our developers narrowed things down to a database bottleneck and released an upgrade this morning. <br /><br />The was system down while the upgrade was processed (approx. 25 mins) but the update resolved the issue and has improved performance through the application. We appreciate your patience and will continue to give updates on PayWhirl's status in the future should any issues arise. <br /><br />Please note that all background jobs were unaffected, and your active subscriptions will behave normally.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>15:08</var> PST</small><br><strong>Monitoring</strong> - We have implemented a fix for the slow load times reported earlier today, and are now monitoring. The issue seems to be resolved but we will continue to monitor and update accordingly. Widgets, background process, and the app are running smoothly as of now.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>14:54</var> PST</small><br><strong>Identified</strong> - We've identified the root of the issue and are working on a fix currently. We'll keep you posted as things progress.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>13:37</var> PST</small><br><strong>Investigating</strong> - This morning at approximately 7:30 AM PST, we began investigating an issue with slow loading times on our multi-application app. Please note that all your data is safe, and background processes do not seem to be affected. However, while we investigate the issue, you may experience slower than expected loading times when viewing the app. This issue is NOT affecting businesses using our new Shopify app, only on https://app.paywhirl.com <br /><br />Our developers have been exploring, and we will provide updates here as we have them. Thank you for your patience.</p>tag:status.paywhirl.com,2005:Incident/60793592021-01-21T12:30:23-08:002021-01-21T12:30:33-08:00PayWhirl slow response times<p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>12:30</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>11:49</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we're monitoring the results. All systems are operational as of 11:30 AM PST.</p><p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>10:32</var> PST</small><br><strong>Identified</strong> - We've identified the issue and are working on a fix now. An unusually large number of API requests are being made from several large accounts and it's causing a bottleneck on our system resources. As a result, we'll be lowering the API rate limit to 60 calls per minute until the issue is fully resolved.</p><p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>09:47</var> PST</small><br><strong>Investigating</strong> - We're investigating an issue that is causing slow response times for our multi-platform app at https://app.paywhirl.com</p>tag:status.paywhirl.com,2005:Incident/60726022021-01-20T17:09:00-08:002021-01-20T17:09:00-08:00PayWhirl slow response times<p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>17:09</var> PST</small><br><strong>Resolved</strong> - The issue has been resolved and all systems are back to normal. We're still monitoring but everything should be working as expected. Please let us know if you see otherwise at team@paywhirl.com and thank you again for your patience.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>16:57</var> PST</small><br><strong>Monitoring</strong> - We've identified the root cause of the issue and pushed out an update that seems to have addressed the issue. We're monitoring now while we enable invoice processing. API requests may be subject to additional rate-limiting in the meantime.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>16:29</var> PST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>16:20</var> PST</small><br><strong>Update</strong> - We're still investigating the issue, but are making some progress. We've paused invoice processing while we continue to monitor the application. As a result, some invoices and payments could be delayed but we will resume processing ASAP. Thank you for your patience.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>15:32</var> PST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>15:30</var> PST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>15:18</var> PST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>15:10</var> PST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>14:49</var> PST</small><br><strong>Investigating</strong> - We're investigating an issue on our main platform at https://app.paywhirl.com</p>tag:status.paywhirl.com,2005:Incident/39205372020-04-20T09:26:49-07:002020-04-20T09:26:49-07:00Some invoices were being retried inadvertently this morning.<p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>09:26</var> PDT</small><br><strong>Resolved</strong> - The issue has been resolved. The bug affected 30 invoices in the system, across all PayWhirl accounts. We sincerely apologize if your account was impacted and are happy to assist if you have any questions regarding this issue. Please feel free to reach out through chat if you need anything from the team. We will be debriefing over the next couple of days to see what we can do to try and prevent issues like this from occurring in the future. We appreciate your prompt reporting of the problem, and your patience while we worked to get it resolved.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>08:32</var> PDT</small><br><strong>Monitoring</strong> - This morning we identified an issue where some invoices were being retried inadvertently. The problem may have caused some payments to be reattempted repeatedly. PayWhirl's invoice processor had a bug with locking invoices that surfaced when the queue became larger than the scheduler interval, which spiked at 6 AM PST this morning. This lead to invoices getting queued many times over until the team identified the issue. We deployed a fix for the bug at 8:24 AM PST and are monitoring to make sure everything has returned to normal. <br /><br />We sincerely apologize if your account was impacted and are happy to assist if you have any questions regarding this issue. Please feel free to reach out through chat if you need anything from the team.</p>tag:status.paywhirl.com,2005:Incident/36998002020-03-17T14:12:01-07:002020-03-17T14:12:01-07:00Some widgets are experiencing issues with checkout<p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>14:12</var> PDT</small><br><strong>Resolved</strong> - We're happy to report the issue was resolved at 1:20 PM PST and only seemed to affect a small number of embedded widgets with auto-scroll enabled. We sincerely apologize for the problems that occurred with our release today. Historically our track record on new feature releases is outstanding at over 99.9% uptime during the past 12 months. <br /><br />We will be debriefing over the next couple of days to see what we can do to try and prevent issues like this from occurring in the future. We appreciate your prompt reporting of the problem, and your patience while we worked to get it resolved.</p><p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>12:00</var> PDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>12:00</var> PDT</small><br><strong>Identified</strong> - We were notified of an issue with a new feature release we made this AM around 9:10AM PST and are working to resolve the issue. The issue was identified officially around 10:35AM PST. The issue only affects some embedded widgets so you may not be impacted. Our team is working on resolving the issue as quickly as possible and truly appreciate your patience. We apologize for any inconvenience this causes for your business and will report back asap with more information.</p>tag:status.paywhirl.com,2005:Incident/21922072019-02-06T13:48:33-08:002019-02-06T13:48:38-08:00Important information about an ongoing mandrill (MailChimp) email delivery outage<p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>13:48</var> PST</small><br><strong>Resolved</strong> - We've received an update from Mandrill and services have been fully restored. Here is the note from their team:<br /><br />Our Engineering team has restored Mandrill’s operations. For most customers, features should be working as expected. However, there may be some residual effects in your account. We’ve included information on that below.<br /><br />Due to the nature of this outage, it was difficult to determine which individual accounts were affected so we’ve chosen to notify all Mandrill users of the outage, its effects, and our efforts to resolve it.<br /><br />What happened and how we responded<br />For more background information on what caused the outage, please read our previous email to customers here. The outage began around 10:30pm EST on Sunday, February 3, and since then our Engineering team has worked around the clock to diagnose the issue, then develop a plan to fix it. We made code changes to work around the damaged database to preserve as much functionality for as many users as possible. We added machines, storage, and networking to attempt a variety of efforts at preserving all data and getting back online. Ultimately, we decided that our efforts to preserve all data would take too much time. We changed direction to truncate specific tables in order to get back online faster. These efforts were successful.<br /><br />How we resolved it<br />We determined that some data was preventing necessary automatic processes from running. We deleted that data, which allowed those processes to successfully run and return the database to a usable state.<br /><br />Residual effects<br />You may see some lingering effects in your account as a result of the outage. Stats and metrics that should have been tracked during the outage may be incorrect or missing entirely.<br /><br />The Mandrill Team<br /><br />Please let us know if you have any questions!<br />Team PayWhirl</p><p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>09:10</var> PST</small><br><strong>Identified</strong> - This morning we were notified by one of our partners (Mandrill, which is owned by Mailchimp) that they have been experiencing an outage with their transactional email system. <br /><br />PayWhirl uses this system to send emails to both businesses and their customers so you may experience issues related to email delivery until they are able to resolve the matter. Below is a statement provided to us by Mandrill about the issues.<br /><br />OFFICIAL ANNOUNCEMENT FROM MANDRILL:<br /><br />We’re contacting you about an ongoing outage with the Mandrill app. This email provides background on what happened and how users are affected, what we’re doing to address the issue, and what’s next for our customers.<br /><br />What happened<br />Mandrill uses a sharded Postgres setup as one of our main datastores. On Sunday, February 3, at 10:30pm EST, 1 of our 5 physical Postgres instances saw a significant spike in writes. The spike in writes triggered a Transaction ID Wraparound issue. When this occurs, database activity is completely halted. The database sets itself in read-only mode until offline maintenance (known as vacuuming) can occur.<br /><br />The database is large—running the vacuum process takes a significant amount of time and resources, and there’s no clear way to track progress.<br /><br />Customer impact<br />The impact to users could come in the form of not tracking opens, clicks, bounces, email sends, inbound email, webhook events, and more. Right now, it looks like the database outage is affecting up to 20% of our outbound volume as well as a majority of inbound email and webhooks.<br /><br />What we’re doing to address this<br />We don’t have an estimated time for when the vacuum process and cleanup work will be complete. While we have a parallel set of tasks going to try to get the database back in working order, these efforts are also slow and difficult with a database of this size. We’re trying everything we can to finish this process as quickly as possible, but this could take several days, or longer. We hope to have more information and a timeline for resolution soon.<br /><br />In the meantime, it’s possible that you may see errors related to sending and receiving emails. We’ll continue to update you on our progress by email and let you know as soon as these issues are fully resolved.<br /><br />What’s next<br />We apologize for the disruption to your business. You don’t need to take any action at this time. Again, we’re sorry for the interruption and we hope to have good news to share soon.<br /><br />We will also keep you posted on any updates we receive from the Mandrill team. Please let us know if you have any questions in the meantime.<br /><br />Team PayWhirl</p>tag:status.paywhirl.com,2005:Incident/21409852019-01-10T16:02:36-08:002019-01-10T16:02:36-08:00A security release is causing issues with Braintree for some businesses.<p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>16:02</var> PST</small><br><strong>Resolved</strong> - All set... A fix was deployed shortly after the incident was discovered. Thanks for your patience.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>15:30</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring closely.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>14:54</var> PST</small><br><strong>Identified</strong> - While upgrading our nodes to the latest security release we uncovered an issue with a deprecated function (create function) and our engineers are working to resolve the issue now.</p>tag:status.paywhirl.com,2005:Incident/17261572018-05-11T10:09:16-07:002018-05-11T10:09:16-07:00Production Response Time Increase<p><small>May <var data-var='date'>11</var>, <var data-var='time'>10:09</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved. It was a fairly technical issue but stemmed from a traffic bottleneck. The related database activity spiraled into a failed environment health check, resulting in "hick-ups" from our load balancers, which ultimately slowed our server response times for both businesses and customers. Once we were able to identify the issue, we restarted our load balancers and all systems were back up and running quickly. <br /><br />NO DATA WAS LOST DURING THIS INCIDENT AND WE SINCERELY APOLOGIZE FOR ANY INCONVENIENCE THIS HAS CAUSED YOUR BUSINESS. <br /><br />The good news is are prepared for this sort of problem and already had systems in place to protect your data against outages. When problems occur with PayWhirl, our backup queuing systems ensure that scheduled invoices and payemnts will resume normally when services become operational again.<br /><br />Thanks again for your patience. Please let us know if you have any questions.<br /><br />The PayWhirl Team</p><p><small>May <var data-var='date'>11</var>, <var data-var='time'>09:44</var> PDT</small><br><strong>Monitoring</strong> - A fix was implemented at 9:10AM PST and we are monitoring. We will mark the issue as resolved and provide more information shortly. Thanks again for your patience.</p><p><small>May <var data-var='date'>11</var>, <var data-var='time'>09:28</var> PDT</small><br><strong>Identified</strong> - We've identified the source of the issue and are working on correcting the problem. Dashboard statistics may be effected while we work on the issue. Thank you for your patience!</p><p><small>May <var data-var='date'>11</var>, <var data-var='time'>08:39</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating an increase in database response time that began around 1:00AM PST. We will keep you posted while we work on the issue.</p>