tag:status.snaps.io,2005:/historySnaps Status - Incident History2024-03-28T12:50:59ZSnapstag:status.snaps.io,2005:Incident/186707542023-10-03T13:16:21Z2023-10-03T13:16:21ZGoogle Dialogflow Outage<p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>13:16</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>21:15</var> UTC</small><br><strong>Monitoring</strong> - The issue appears to be resolved. Functionality was degraded from 3:33pm to 4:49pm EST. We are waiting for confirmation from Google before we consider this incident closed.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>21:03</var> UTC</small><br><strong>Investigating</strong> - Google Dialogflow is experiencing failures which are affecting Quiq. Bots that use NLU are slow to respond and may fall back to default responses. We are taking steps to reduce the impact, and monitoring the situation. We will update when more information is availble.</p>tag:status.snaps.io,2005:Incident/175651122023-06-13T21:42:09Z2023-06-13T21:42:30ZAWS US-East Lambda Partial Outage<p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>21:42</var> UTC</small><br><strong>Resolved</strong> - This incident with AWS has been resolved and everything is operational.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>21:21</var> UTC</small><br><strong>Identified</strong> - Amazon Web Services Lambdas are not responding in Virginia regions. Bots are functional. Some Quiq features will not be available. Will impact customers with third-party CRM integrations. More updates soon.</p>tag:status.snaps.io,2005:Incident/113630782022-10-03T15:30:14Z2022-10-03T15:31:29ZIntermittent Bot Responses<p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>15:30</var> UTC</small><br><strong>Resolved</strong> - On October 3rd, 2022 from 10:06am through 11:02am Eastern time, we experienced intermittent increased latency to our Snaps' services. Snaps' bots were slow to respond and for some users became unresponsive. This was due to a background process requiring abnormally large resources. The issue was identified and has been resolved.</p>tag:status.snaps.io,2005:Incident/103731262022-06-21T20:06:51Z2022-06-21T20:07:36ZProactive notifications disruption<p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>20:06</var> UTC</small><br><strong>Resolved</strong> - From 1:16PM until 3:33PM on June 21st 2022, we experienced a connectivity issue with a backend service that caused proactive notifications to fail to be picked up. All non proactive bot interactions continued to operate without disruption. Our alerting failed to recognize this scenario and we are investigating strategies to identify this issue more quickly in the future.</p>tag:status.snaps.io,2005:Incident/97605432022-04-11T18:03:18Z2022-04-11T18:03:18ZAs of 11:52AM ET - bots are responding slowly.<p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>18:03</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>17:03</var> UTC</small><br><strong>Monitoring</strong> - An auto scaling configuration was changed for a new deployment we made and this caused our servers to have more than normal latency. Performance is back to normal, and we will continue to monitor the situation.</p><p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>16:49</var> UTC</small><br><strong>Identified</strong> - We are working to resolve the issue.</p>tag:status.snaps.io,2005:Incident/96050552022-03-22T20:04:24Z2022-03-22T20:04:24ZSlowed Bot Responses<p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>20:04</var> UTC</small><br><strong>Resolved</strong> - On March 22nd, 2022 at 1:58PM Eastern Time, code was deployed to our production servers that caused bots to respond slowly. The issue was identified and resolved at 3:11 PM Eastern Time.</p>tag:status.snaps.io,2005:Incident/88369132021-12-13T23:42:39Z2021-12-13T23:45:14ZBot Deployments Intermittent Loading Issues<p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>23:42</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>23:41</var> UTC</small><br><strong>Update</strong> - At 12:37 PM Mountain time on Dec 13th, 2021: we experienced an issue with a backend component that caused bots to not load or respond slowly. We deployed two remediations to fully resolve the issue. The first remediation at 1:47 PM restored most bot implementations, but with some abnormal latency in bot responses. The second remediation was deployed at 4:08 PM and all bots are now performing as expected.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>21:38</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>21:37</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>20:47</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>20:32</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.snaps.io,2005:Incident/88117872021-12-10T20:57:44Z2021-12-10T20:57:44ZDec. 10, 2021, 8:45 AM MT - Facebook Messenger is reporting issues this morning. Our ops team is monitoring.<p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>20:57</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>15:58</var> UTC</small><br><strong>Monitoring</strong> - Monitoring.</p>tag:status.snaps.io,2005:Incident/87873792021-12-08T12:58:43Z2021-12-08T12:58:43ZMonitoring AWS Issues<p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>12:58</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>23:32</var> UTC</small><br><strong>Monitoring</strong> - AWS has been experiencing issues throughout the day. <br /><br />We are not experiencing issues impacting production bot deployments, but we continue to monitor the situation closely.</p>tag:status.snaps.io,2005:Incident/87828192021-12-07T20:12:02Z2021-12-07T20:12:02ZAnalytics slowdowns<p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>20:12</var> UTC</small><br><strong>Resolved</strong> - While our cloud provider (AWS) continues to report issues, analytics is now running as expected.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>16:59</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>16:59</var> UTC</small><br><strong>Monitoring</strong> - We are monitoring the AWS response to the incident.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>16:58</var> UTC</small><br><strong>Investigating</strong> - Our cloud provider, Amazon Web Services, is experiencing some outages in the us-east-1 region. This is affecting our ability to serve analytics. We will update this page as AWS updates their status (available here: https://status.aws.amazon.com/)</p>tag:status.snaps.io,2005:Incident/85461792021-11-16T01:30:30Z2021-11-16T01:30:30ZIntermittent Loading of Bots<p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>01:30</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>00:08</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>23:56</var> UTC</small><br><strong>Identified</strong> - We had a brief network outage that caused bots to not load from 6:20pm to 6:35 Eastern Time. A fix was deployed and bots are coming back online. We are monitoring the situation.</p>tag:status.snaps.io,2005:Incident/81505142021-10-05T00:20:21Z2021-10-05T00:20:21ZFacebook Messenger and Instagram Issues<p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>00:20</var> UTC</small><br><strong>Resolved</strong> - As of Oct. 4, 2021 - 6:00 PM MT, Facebook and Instagram have been coming back online and bots running on these platforms are responding without issue.</p><p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>16:29</var> UTC</small><br><strong>Monitoring</strong> - As of Oct. 4, 2021 - 9:56 AM MT, Facebook and Instagram are reporting issues. Snaps' Engineering continues to monitor the situation.</p>tag:status.snaps.io,2005:Incident/80934952021-09-28T12:18:41Z2021-09-28T12:18:41ZBot deployments down<p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>12:18</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>27</var>, <var data-var='time'>14:53</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Sep <var data-var='date'>27</var>, <var data-var='time'>14:53</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Sep <var data-var='date'>27</var>, <var data-var='time'>14:22</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:status.snaps.io,2005:Incident/77854972021-08-18T00:39:19Z2021-08-18T00:39:19ZIntermittent Bot Responses<p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>00:39</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>23:54</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>23:52</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>23:39</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>23:24</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>21:00</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.snaps.io,2005:Incident/64100002021-02-26T21:55:44Z2021-02-26T21:55:44ZDelayed bot responses<p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>21:55</var> UTC</small><br><strong>Resolved</strong> - Some of our bot deployments experienced delayed responses from 4:08pm until 4:46pm EST on Feb 26th, 2021. The issue was due to a data caching issue. The issue is now resolved and we are investigating mitigation strategies for the future.</p>tag:status.snaps.io,2005:Incident/58542202020-12-21T21:43:52Z2020-12-21T21:43:52ZFB Messenger Bots<p><small>Dec <var data-var='date'>21</var>, <var data-var='time'>21:43</var> UTC</small><br><strong>Resolved</strong> - We have successfully mitigated this issue and have restored automation for affected users.</p><p><small>Dec <var data-var='date'>21</var>, <var data-var='time'>14:41</var> UTC</small><br><strong>Investigating</strong> - Some of our Facebook Messenger bots are experiencing an issue that is causing users to be incorrectly put into a handoff state. When this happens, the bot becomes unresponsive. We believe the issue is due to some recent Facebook Messenger changes regarding the EEA requirements. We are actively working to resolve the issue.</p>tag:status.snaps.io,2005:Incident/56543772020-11-25T20:56:53Z2020-11-25T20:56:53ZAWS Outage<p><small>Nov <var data-var='date'>25</var>, <var data-var='time'>20:56</var> UTC</small><br><strong>Resolved</strong> - Snaps CMS users are reporting that they are now able to log in successfully.</p><p><small>Nov <var data-var='date'>25</var>, <var data-var='time'>16:12</var> UTC</small><br><strong>Monitoring</strong> - Our cloud service provider, Amazon Web Services, is having issues with multiple services in the us-east-1 region. No bots are affected, but you may be unable to log in to the CMS. We will update this page when the AWS issues are resolved.</p>tag:status.snaps.io,2005:Incident/54906082020-10-15T21:49:00Z2020-11-04T21:50:21ZSystem slowdown<p><small>Oct <var data-var='date'>15</var>, <var data-var='time'>21:49</var> UTC</small><br><strong>Resolved</strong> - The system had a slowdown for a 2 hour period.</p>tag:status.snaps.io,2005:Incident/51723332020-09-25T13:10:23Z2020-09-25T13:10:23ZGoogle NLU Service Request Failures<p><small>Sep <var data-var='date'>25</var>, <var data-var='time'>13:10</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>25</var>, <var data-var='time'>01:39</var> UTC</small><br><strong>Monitoring</strong> - On Sept 24th 2020, from 9:10 through 9:25pm EST, we experienced an abnormally high number of failed requests to Google Natural Language Cloud APIs causing our NLU to fail in some cases. Fallback responses, agent handover, product search and knowledge base responses are working as normal. The problem has been reported as a wide spread issue for many Google APIs. <br /><br />The issue has resolved at this time, but we are monitoring the situation.</p>tag:status.snaps.io,2005:Incident/50694522020-09-11T19:06:56Z2020-09-11T19:06:56ZNetwork instability<p><small>Sep <var data-var='date'>11</var>, <var data-var='time'>19:06</var> UTC</small><br><strong>Resolved</strong> - The incident has been resolved, we are continuing to monitor closely.</p><p><small>Sep <var data-var='date'>11</var>, <var data-var='time'>18:47</var> UTC</small><br><strong>Monitoring</strong> - Some of our services were under stress and some bots users may have received slow or timed out bot responses. We have brought new resources online and the problems are resolving. We are monitoring the situation.</p><p><small>Sep <var data-var='date'>11</var>, <var data-var='time'>18:07</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.snaps.io,2005:Incident/50597162020-09-10T14:55:55Z2020-09-10T14:55:55ZBot response timeouts or delays<p><small>Sep <var data-var='date'>10</var>, <var data-var='time'>14:55</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>10</var>, <var data-var='time'>14:15</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Sep <var data-var='date'>10</var>, <var data-var='time'>14:13</var> UTC</small><br><strong>Monitoring</strong> - From 10:01am through 10:07am EST on Sept 10th 2020, some bots become unresponsive responsive or had delayed responses. The issue was due to a large and sudden traffic increase, we have scaled our infrastructure and are investigating and monitoring the issue.</p>tag:status.snaps.io,2005:Incident/50016582020-09-02T12:57:23Z2020-09-02T12:57:23ZNotifications Delays<p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>12:57</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>12:44</var> UTC</small><br><strong>Monitoring</strong> - Some bot deployments experienced delayed notifications from 7:30am EDT until 8:41am EDT on Sept 2nd 2020.</p>tag:status.snaps.io,2005:Incident/47785292020-08-04T13:34:11Z2020-08-04T13:35:16ZLatency and some dropped requests to our rules service.<p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>13:34</var> UTC</small><br><strong>Resolved</strong> - From 10:44 am through 11:36 am EDT on Aug 3rd, 2020 we experienced degraded performance on our rules web service. This impacted some bot deployments, causing notifications to slow or request timeouts to occur. Additional capacity was added to our rules web service and the issue quickly resolved.</p>tag:status.snaps.io,2005:Incident/47638572020-08-04T13:24:02Z2020-08-04T13:24:02ZSome bots are unresponsive.<p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>13:24</var> UTC</small><br><strong>Resolved</strong> - Our cutover to new DB infrastructure has been monitored for 36 hours and service remains stable. We have configured additional monitoring and alarms to improve our ability to capture and proactively respond to any future infrastructure failures. This issue is now resolved.</p><p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>21:15</var> UTC</small><br><strong>Update</strong> - We have successfully cutover to new DB infrastructure and intermittent issues that arose from around 2:58pm should no longer be expected. We will continue to monitor our instances closely and mark this issue resolved within 24 hours.</p><p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>19:49</var> UTC</small><br><strong>Update</strong> - We are in final stages of DB scaling and experiencing some intermittent issues.</p><p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>17:26</var> UTC</small><br><strong>Monitoring</strong> - We experienced a DB issue. A fix has been implemented and we are monitoring the situation. We will provide further post mortem details within 24 hours.</p><p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>15:56</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.snaps.io,2005:Incident/47419702020-07-29T14:30:00Z2020-08-02T21:26:26ZNotifications Dropped<p><small>Jul <var data-var='date'>29</var>, <var data-var='time'>14:30</var> UTC</small><br><strong>Resolved</strong> - All bots implementations experienced delayed or dropped proactive notifications due to a data store issue. The issue has been identified and repaired as of 1:54 pm on July 29th. We have implemented additional monitoring for this scenario and applied other safegaurds to prevent it in the future.</p>