Incidents | Wattch Incidents reported on status page for Wattch https://status.wattch.io/ https://d1lppblt9t2x15.cloudfront.net/logos/fa6150aa13395851f397e6f289cc8557.png Incidents | Wattch https://status.wattch.io/ en Marketing Website recovered https://status.wattch.io/ Fri, 05 Dec 2025 09:12:19 +0000 https://status.wattch.io/#5fdb9280e8f8703f87bb4083e96ab6e1eddfbbeae0f064d49ba4fd11a82ae156 Marketing Website recovered Marketing Website went down https://status.wattch.io/ Fri, 05 Dec 2025 08:48:45 +0000 https://status.wattch.io/#5fdb9280e8f8703f87bb4083e96ab6e1eddfbbeae0f064d49ba4fd11a82ae156 Marketing Website went down Cellular Connectivity Issues https://status.wattch.io/incident/767636 Wed, 26 Nov 2025 00:03:00 -0000 https://status.wattch.io/incident/767636#ce03f2f10c4f453907f5d8362035a64e9e68b35088e00ca65fdb7ef1a3a93edf Our cellular provider pushed a stability enhancement, and connection stability has been much better over the past few days. Please let us know if you still see WECs still dropping out. Cellular Connectivity Issues https://status.wattch.io/incident/767636 Mon, 24 Nov 2025 20:26:00 -0000 https://status.wattch.io/incident/767636#ac89c9c20bb7c65aee48168a185bef73a9275b994a8e3236b9e5e0026845cdbb Our cellular provider has pushed a fix that seems to have resolved the issue. We will keep the incident open until they confirm full resolution. Cellular Connectivity Issues https://status.wattch.io/incident/767636 Tue, 18 Nov 2025 23:26:00 -0000 https://status.wattch.io/incident/767636#82027c139371340886ff21f8d03808fd2c38f07be2006f2d37cb313d2805892e This incident is still ongoing. Our cellular provider has posted an update that they have identified the issue. This seems to affect only a small percentage of the fleet, but for those that are affected, we appreciate your patience. Cellular Connectivity Issues https://status.wattch.io/incident/767636 Mon, 17 Nov 2025 18:23:00 -0000 https://status.wattch.io/incident/767636#1e43f8edf319af7da11a7d19702ec461794f73f2ea837ff12d4b704690f809c9 This is a continuation of the previous incident. We had stability for a few days but have just seen a small portion of the WEC fleet fall offline again. Our upstream provider has an ongoing incident posted here: https://status.hologram.io/incidents/srzr7txpt2hh. We have reached out to them and are continuing to monitor the situation. Thank you for your patience. Intermittent Cellular Connectivity Issues https://status.wattch.io/incident/761732 Mon, 17 Nov 2025 16:31:00 -0000 https://status.wattch.io/incident/761732#98a3f096688dd1abee77bed745669733867bc68482c01206adee6cedd2f01710 This incident has been resolved. Marketing Website recovered https://status.wattch.io/ Thu, 13 Nov 2025 17:02:30 +0000 https://status.wattch.io/#68eccc452c404db6fcc6be0dbac0d932dc0c15162e35da7ab2a08aedf7b569c9 Marketing Website recovered Marketing Website went down https://status.wattch.io/ Thu, 13 Nov 2025 17:02:23 +0000 https://status.wattch.io/#68eccc452c404db6fcc6be0dbac0d932dc0c15162e35da7ab2a08aedf7b569c9 Marketing Website went down Intermittent Cellular Connectivity Issues https://status.wattch.io/incident/761732 Wed, 12 Nov 2025 16:52:00 -0000 https://status.wattch.io/incident/761732#0e434501d1ca7086b89c8092256d688d010d03264e3d5e4d3a691832e6009863 Our cellular provider has posted a resolution to their incident, but we are still seeing an unexpected number of disconnected WECs. We have reached out to the provider for further clarification. Intermittent Cellular Connectivity Issues https://status.wattch.io/incident/761732 Wed, 12 Nov 2025 00:41:00 -0000 https://status.wattch.io/incident/761732#96ec82c9981ccfd684a34e2805e92dc4106f7fb1573ead7e704e731ff96b1634 We are continuing to see intermittent issues with our cellular provider. They have acknowledged the issue and posted an incident here: https://status.hologram.io/incidents/g8wx16h0fp1q. Most WECs are connected, but we expect to continue to see a higher rate of temporary connectivity issues until this is resolved. Thank you for your patience. Intermittent Cellular Connectivity Issues https://status.wattch.io/incident/761732 Mon, 10 Nov 2025 20:36:00 -0000 https://status.wattch.io/incident/761732#4652d0e2060a955a76bf9086da33590204986c45353d95c6205756c8ed1a9f8c We are seeing a high number of cellular connectivity issues across our fleet. Most modems are falling back to their backup SIMs, so any outages should be intermittent. We will continue monitoring the situation and will provide updates whenever they become available. Partial Cellular Outage https://status.wattch.io/incident/753862 Thu, 30 Oct 2025 14:57:00 -0000 https://status.wattch.io/incident/753862#dfd37d28a726613280e3009b836b572e5685073ce782b1b28e2224e6730ba905 All affected WECs have recovered - this incident is resolved. Partial Cellular Outage https://status.wattch.io/incident/753862 Wed, 29 Oct 2025 21:18:00 -0000 https://status.wattch.io/incident/753862#688a22eb505330a7317320a9cda81511dc936a0c8fcbc6f8f254517354717bb9 Hologram has marked the incident as resolved, but we're still seeing a small number of WECs that haven't recovered (<1%). We have confirmed that the SIMs are still disconnected for all affected WECs. We will keep the incident open for now, but the issue should be resolved for the vast majority of WECs. Partial Cellular Outage https://status.wattch.io/incident/753862 Wed, 29 Oct 2025 17:37:00 -0000 https://status.wattch.io/incident/753862#6c11125bc69aabe0feb3d051a506a9c347c9d9e8649ee24c68dace4fc1c756a3 At this point, most WECs have recovered. 1-2% of WECs are still offline, so we will keep this incident open for now. We will continue monitoring the situation and post any more updates when available. Partial Cellular Outage https://status.wattch.io/incident/753862 Wed, 29 Oct 2025 16:42:00 -0000 https://status.wattch.io/incident/753862#feb67af98dc2d8f852de67de7ee1cd418d1e527b019e31e07898dceeee58719f Hologram, our cellular provider, has posted an incident which can be found here: https://status.hologram.io/incidents/83hv6g2h8prv. We will continue to monitor our systems for recovery. Partial Cellular Outage https://status.wattch.io/incident/753862 Wed, 29 Oct 2025 16:07:00 -0000 https://status.wattch.io/incident/753862#88930d8e02edf92473d0a27689120cb50fa5272d1ab079877e2f2d6f550a546f Wattch has detected a potential cellular outage causing approximately 19% of WECs to lose connection with the cloud. Our team is actively monitoring the situation and ensuring no other possible root causes. We are seeing some recovery beginning already as the modems switch to backup cellular carriers, but we will continue to post updates here until the incident is resolved. Marketing Website recovered https://status.wattch.io/ Tue, 28 Oct 2025 03:54:32 +0000 https://status.wattch.io/#c5fae35bc33cd553b31d886b51f5961ec3f139d837c0e07e83245480be689a1d Marketing Website recovered Marketing Website went down https://status.wattch.io/ Tue, 28 Oct 2025 03:54:03 +0000 https://status.wattch.io/#c5fae35bc33cd553b31d886b51f5961ec3f139d837c0e07e83245480be689a1d Marketing Website went down Authentication System Upgrade https://status.wattch.io/incident/744124 Mon, 20 Oct 2025 20:00:00 -0000 https://status.wattch.io/incident/744124#f206bdafdc93212895cc4774d6fd09631a67e697eaa6a16cc414e04ebe32727c Maintenance completed Authentication System Upgrade https://status.wattch.io/incident/744124 Mon, 20 Oct 2025 18:30:00 -0000 https://status.wattch.io/incident/744124#8fe1ed38c3cc4198f74b79137934e29966dcfc3aa500bd846958abc4e0b9c8d9 Wattch will be deploying an upgrade to its authentication system. During this time, all user sessions will terminate. Users will be required to log back in to Wattch. After logging back in, no further disruptions are expected. Marketing Website recovered https://status.wattch.io/ Wed, 15 Oct 2025 03:18:44 +0000 https://status.wattch.io/#30165e440b37ed0a164f641e52b8b31ae03adf22dfe08cd8dc278eda692b0144 Marketing Website recovered Marketing Website went down https://status.wattch.io/ Wed, 15 Oct 2025 03:13:48 +0000 https://status.wattch.io/#30165e440b37ed0a164f641e52b8b31ae03adf22dfe08cd8dc278eda692b0144 Marketing Website went down Database Maintenance https://status.wattch.io/incident/732353 Wed, 08 Oct 2025 20:30:51 -0000 https://status.wattch.io/incident/732353#76dd3f93517ae2039b10d925aafb25a5d8b7c5d96b3d4851d2c335766ce7b2ef Maintenance completed Database Maintenance https://status.wattch.io/incident/732353 Wed, 08 Oct 2025 20:30:51 -0000 https://status.wattch.io/incident/732353#76dd3f93517ae2039b10d925aafb25a5d8b7c5d96b3d4851d2c335766ce7b2ef Maintenance completed Database Maintenance https://status.wattch.io/incident/732353 Wed, 08 Oct 2025 20:30:51 -0000 https://status.wattch.io/incident/732353#76dd3f93517ae2039b10d925aafb25a5d8b7c5d96b3d4851d2c335766ce7b2ef Maintenance completed Database Maintenance https://status.wattch.io/incident/732353 Wed, 08 Oct 2025 20:30:51 -0000 https://status.wattch.io/incident/732353#76dd3f93517ae2039b10d925aafb25a5d8b7c5d96b3d4851d2c335766ce7b2ef Maintenance completed Database Maintenance https://status.wattch.io/incident/732353 Wed, 08 Oct 2025 20:30:51 -0000 https://status.wattch.io/incident/732353#76dd3f93517ae2039b10d925aafb25a5d8b7c5d96b3d4851d2c335766ce7b2ef Maintenance completed Database Maintenance https://status.wattch.io/incident/732353 Wed, 08 Oct 2025 19:30:51 -0000 https://status.wattch.io/incident/732353#edabd017db67e5744ad2c4938ca357a651117cdad50ee84c174abc2b47936d65 Wattch will be undergoing database maintenance during this time. Users may experience brief service interruption. Database Maintenance https://status.wattch.io/incident/732353 Wed, 08 Oct 2025 19:30:51 -0000 https://status.wattch.io/incident/732353#edabd017db67e5744ad2c4938ca357a651117cdad50ee84c174abc2b47936d65 Wattch will be undergoing database maintenance during this time. Users may experience brief service interruption. Database Maintenance https://status.wattch.io/incident/732353 Wed, 08 Oct 2025 19:30:51 -0000 https://status.wattch.io/incident/732353#edabd017db67e5744ad2c4938ca357a651117cdad50ee84c174abc2b47936d65 Wattch will be undergoing database maintenance during this time. Users may experience brief service interruption. Database Maintenance https://status.wattch.io/incident/732353 Wed, 08 Oct 2025 19:30:51 -0000 https://status.wattch.io/incident/732353#edabd017db67e5744ad2c4938ca357a651117cdad50ee84c174abc2b47936d65 Wattch will be undergoing database maintenance during this time. Users may experience brief service interruption. Database Maintenance https://status.wattch.io/incident/732353 Wed, 08 Oct 2025 19:30:51 -0000 https://status.wattch.io/incident/732353#edabd017db67e5744ad2c4938ca357a651117cdad50ee84c174abc2b47936d65 Wattch will be undergoing database maintenance during this time. Users may experience brief service interruption. Marketing Website recovered https://status.wattch.io/ Wed, 17 Sep 2025 12:10:36 +0000 https://status.wattch.io/#41ad17fdb5b0fea8dc38094f5246caef36e2222e1879edc6a75ea8ef82198997 Marketing Website recovered Marketing Website went down https://status.wattch.io/ Wed, 17 Sep 2025 12:10:04 +0000 https://status.wattch.io/#41ad17fdb5b0fea8dc38094f5246caef36e2222e1879edc6a75ea8ef82198997 Marketing Website went down Database Maintenance https://status.wattch.io/incident/700874 Wed, 06 Aug 2025 18:00:00 -0000 https://status.wattch.io/incident/700874#6d24fbc0634f515782757e7e25934c0757ddf29b657cf806e03c368df5209576 Maintenance completed Database Maintenance https://status.wattch.io/incident/700874 Wed, 06 Aug 2025 18:00:00 -0000 https://status.wattch.io/incident/700874#6d24fbc0634f515782757e7e25934c0757ddf29b657cf806e03c368df5209576 Maintenance completed Database Maintenance https://status.wattch.io/incident/700874 Wed, 06 Aug 2025 18:00:00 -0000 https://status.wattch.io/incident/700874#6d24fbc0634f515782757e7e25934c0757ddf29b657cf806e03c368df5209576 Maintenance completed Database Maintenance https://status.wattch.io/incident/700874 Wed, 06 Aug 2025 18:00:00 -0000 https://status.wattch.io/incident/700874#6d24fbc0634f515782757e7e25934c0757ddf29b657cf806e03c368df5209576 Maintenance completed Database Maintenance https://status.wattch.io/incident/700874 Wed, 06 Aug 2025 18:00:00 -0000 https://status.wattch.io/incident/700874#6d24fbc0634f515782757e7e25934c0757ddf29b657cf806e03c368df5209576 Maintenance completed Database Maintenance https://status.wattch.io/incident/700874 Wed, 06 Aug 2025 17:00:01 -0000 https://status.wattch.io/incident/700874#67d8c4ea4f9d94f16815a66af85228be42b0f62e2f9c8072424ba1a9c5af2570 Wattch will be performing database maintenance during this time. Customers may expect a few minutes of downtime. Database Maintenance https://status.wattch.io/incident/700874 Wed, 06 Aug 2025 17:00:01 -0000 https://status.wattch.io/incident/700874#67d8c4ea4f9d94f16815a66af85228be42b0f62e2f9c8072424ba1a9c5af2570 Wattch will be performing database maintenance during this time. Customers may expect a few minutes of downtime. Database Maintenance https://status.wattch.io/incident/700874 Wed, 06 Aug 2025 17:00:01 -0000 https://status.wattch.io/incident/700874#67d8c4ea4f9d94f16815a66af85228be42b0f62e2f9c8072424ba1a9c5af2570 Wattch will be performing database maintenance during this time. Customers may expect a few minutes of downtime. Database Maintenance https://status.wattch.io/incident/700874 Wed, 06 Aug 2025 17:00:01 -0000 https://status.wattch.io/incident/700874#67d8c4ea4f9d94f16815a66af85228be42b0f62e2f9c8072424ba1a9c5af2570 Wattch will be performing database maintenance during this time. Customers may expect a few minutes of downtime. Database Maintenance https://status.wattch.io/incident/700874 Wed, 06 Aug 2025 17:00:01 -0000 https://status.wattch.io/incident/700874#67d8c4ea4f9d94f16815a66af85228be42b0f62e2f9c8072424ba1a9c5af2570 Wattch will be performing database maintenance during this time. Customers may expect a few minutes of downtime. Partial Cellular Outage https://status.wattch.io/incident/628674 Mon, 28 Jul 2025 21:19:00 -0000 https://status.wattch.io/incident/628674#251af46e8c8a00469485542ac3bed68d3fb1f630ec6c40452ff5ab4e12faa1c8 It looks like everything has fully recovered! Partial Cellular Outage https://status.wattch.io/incident/628674 Mon, 28 Jul 2025 20:03:00 -0000 https://status.wattch.io/incident/628674#f766dde694d7463f509313148d2ecbce5eebc4ff84c04ef71aa5200a921ef56d Most modems have recovered and we are monitoring to see the remaining few come back online. Partial Cellular Outage https://status.wattch.io/incident/628674 Mon, 28 Jul 2025 17:41:00 -0000 https://status.wattch.io/incident/628674#28094c0abfacc8f275b4bc2b8b611fef31b904846bdfea9f1218274b4fa3cf5e There is a partial cellular outage on one of our carriers leading to a subset of modems being offline (~9%). Modems should resolve automatically once the upstream issue is resolved. We will post an update as soon as everything is recovered. We are seeing around half of these devices recover/switch SIMs. No data will be lost. SMA API Outage https://status.wattch.io/incident/623361 Tue, 22 Jul 2025 14:53:00 -0000 https://status.wattch.io/incident/623361#0eec12f5552b75441d01bc50cb41db7097365a312d418c58bd5c90dbcc76bf59 SMA's API outage appears to be resolved. All Wattch systems are not operating normally. SMA API Outage https://status.wattch.io/incident/623361 Mon, 21 Jul 2025 15:28:00 -0000 https://status.wattch.io/incident/623361#7b0de194b4c8471cfa0f6b3d4a38fe02a5d555df9015dfe5b052588a36c044e7 SMA's API is currently experiencing downtime. Wattch sites connected through the SMA API will likely not see new data until the incident is resolved. These sites will likely generate Device Communications alerts. Intermittent Availability due to GCP Networking Issues https://status.wattch.io/incident/621994 Fri, 18 Jul 2025 18:34:00 -0000 https://status.wattch.io/incident/621994#0dabce0bef0e3199d4aed7a2cf101babc62f798d67508d738786a44ef3528a83 This incident has been resolved. Intermittent Availability due to GCP Networking Issues https://status.wattch.io/incident/621994 Fri, 18 Jul 2025 18:34:00 -0000 https://status.wattch.io/incident/621994#0dabce0bef0e3199d4aed7a2cf101babc62f798d67508d738786a44ef3528a83 This incident has been resolved. Intermittent Availability due to GCP Networking Issues https://status.wattch.io/incident/621994 Fri, 18 Jul 2025 18:34:00 -0000 https://status.wattch.io/incident/621994#0dabce0bef0e3199d4aed7a2cf101babc62f798d67508d738786a44ef3528a83 This incident has been resolved. Intermittent Availability due to GCP Networking Issues https://status.wattch.io/incident/621994 Fri, 18 Jul 2025 18:34:00 -0000 https://status.wattch.io/incident/621994#0dabce0bef0e3199d4aed7a2cf101babc62f798d67508d738786a44ef3528a83 This incident has been resolved. Intermittent Availability due to GCP Networking Issues https://status.wattch.io/incident/621994 Fri, 18 Jul 2025 16:08:00 -0000 https://status.wattch.io/incident/621994#6ab237c6e57fdea1cf48ed2f263e72d68e1a76cac5cbae44a93c66fd726ea50c Google has posted an incident here: https://status.cloud.google.com/incidents/8cY8jdUpEGGbsSMSQk7J The impact on the Wattch ingestion pipeline seems to be mitigated. We will keep the incident open and post updates until Google marks their incident as resolved. Intermittent Availability due to GCP Networking Issues https://status.wattch.io/incident/621994 Fri, 18 Jul 2025 16:08:00 -0000 https://status.wattch.io/incident/621994#6ab237c6e57fdea1cf48ed2f263e72d68e1a76cac5cbae44a93c66fd726ea50c Google has posted an incident here: https://status.cloud.google.com/incidents/8cY8jdUpEGGbsSMSQk7J The impact on the Wattch ingestion pipeline seems to be mitigated. We will keep the incident open and post updates until Google marks their incident as resolved. Intermittent Availability due to GCP Networking Issues https://status.wattch.io/incident/621994 Fri, 18 Jul 2025 16:08:00 -0000 https://status.wattch.io/incident/621994#6ab237c6e57fdea1cf48ed2f263e72d68e1a76cac5cbae44a93c66fd726ea50c Google has posted an incident here: https://status.cloud.google.com/incidents/8cY8jdUpEGGbsSMSQk7J The impact on the Wattch ingestion pipeline seems to be mitigated. We will keep the incident open and post updates until Google marks their incident as resolved. Intermittent Availability due to GCP Networking Issues https://status.wattch.io/incident/621994 Fri, 18 Jul 2025 16:08:00 -0000 https://status.wattch.io/incident/621994#6ab237c6e57fdea1cf48ed2f263e72d68e1a76cac5cbae44a93c66fd726ea50c Google has posted an incident here: https://status.cloud.google.com/incidents/8cY8jdUpEGGbsSMSQk7J The impact on the Wattch ingestion pipeline seems to be mitigated. We will keep the incident open and post updates until Google marks their incident as resolved. Intermittent Availability due to GCP Networking Issues https://status.wattch.io/incident/621994 Fri, 18 Jul 2025 15:40:00 -0000 https://status.wattch.io/incident/621994#e4bd4d2383ccab6b2324dce19399b6b666455def1b874904e9cc438b47f62f1a A networking issue at GCP seems to be affecting the Wattch data ingestion pipeline. As of now, this does not seem to affect the availability of the site. Users may see spurious Uplink and/or Device Communications alerts. Intermittent Availability due to GCP Networking Issues https://status.wattch.io/incident/621994 Fri, 18 Jul 2025 15:40:00 -0000 https://status.wattch.io/incident/621994#e4bd4d2383ccab6b2324dce19399b6b666455def1b874904e9cc438b47f62f1a A networking issue at GCP seems to be affecting the Wattch data ingestion pipeline. As of now, this does not seem to affect the availability of the site. Users may see spurious Uplink and/or Device Communications alerts. Intermittent Availability due to GCP Networking Issues https://status.wattch.io/incident/621994 Fri, 18 Jul 2025 15:40:00 -0000 https://status.wattch.io/incident/621994#e4bd4d2383ccab6b2324dce19399b6b666455def1b874904e9cc438b47f62f1a A networking issue at GCP seems to be affecting the Wattch data ingestion pipeline. As of now, this does not seem to affect the availability of the site. Users may see spurious Uplink and/or Device Communications alerts. Intermittent Availability due to GCP Networking Issues https://status.wattch.io/incident/621994 Fri, 18 Jul 2025 15:40:00 -0000 https://status.wattch.io/incident/621994#e4bd4d2383ccab6b2324dce19399b6b666455def1b874904e9cc438b47f62f1a A networking issue at GCP seems to be affecting the Wattch data ingestion pipeline. As of now, this does not seem to affect the availability of the site. Users may see spurious Uplink and/or Device Communications alerts. Service Unavailable due to GCP Outage https://status.wattch.io/incident/601835 Thu, 12 Jun 2025 22:45:00 -0000 https://status.wattch.io/incident/601835#e779e2d3769db68b24cd2960a60d42e454ba93e1e781aa375d37116eca43837f All services that affect Wattch have recovered. Service Unavailable due to GCP Outage https://status.wattch.io/incident/601835 Thu, 12 Jun 2025 22:45:00 -0000 https://status.wattch.io/incident/601835#e779e2d3769db68b24cd2960a60d42e454ba93e1e781aa375d37116eca43837f All services that affect Wattch have recovered. Service Unavailable due to GCP Outage https://status.wattch.io/incident/601835 Thu, 12 Jun 2025 22:45:00 -0000 https://status.wattch.io/incident/601835#e779e2d3769db68b24cd2960a60d42e454ba93e1e781aa375d37116eca43837f All services that affect Wattch have recovered. Service Unavailable due to GCP Outage https://status.wattch.io/incident/601835 Thu, 12 Jun 2025 22:45:00 -0000 https://status.wattch.io/incident/601835#e779e2d3769db68b24cd2960a60d42e454ba93e1e781aa375d37116eca43837f All services that affect Wattch have recovered. Service Unavailable due to GCP Outage https://status.wattch.io/incident/601835 Thu, 12 Jun 2025 22:45:00 -0000 https://status.wattch.io/incident/601835#e779e2d3769db68b24cd2960a60d42e454ba93e1e781aa375d37116eca43837f All services that affect Wattch have recovered. Service Unavailable due to GCP Outage https://status.wattch.io/incident/601835 Thu, 12 Jun 2025 20:50:00 -0000 https://status.wattch.io/incident/601835#449495c337599633e096c0cf491847cf35ea3f2d5c9c986faad386cbb006a3f6 The incident has been mostly resolved, according to Google's latest updates. We will wait for Google's full resolution to mark this incident as resolved. Service Unavailable due to GCP Outage https://status.wattch.io/incident/601835 Thu, 12 Jun 2025 20:50:00 -0000 https://status.wattch.io/incident/601835#449495c337599633e096c0cf491847cf35ea3f2d5c9c986faad386cbb006a3f6 The incident has been mostly resolved, according to Google's latest updates. We will wait for Google's full resolution to mark this incident as resolved. Service Unavailable due to GCP Outage https://status.wattch.io/incident/601835 Thu, 12 Jun 2025 20:50:00 -0000 https://status.wattch.io/incident/601835#449495c337599633e096c0cf491847cf35ea3f2d5c9c986faad386cbb006a3f6 The incident has been mostly resolved, according to Google's latest updates. We will wait for Google's full resolution to mark this incident as resolved. Service Unavailable due to GCP Outage https://status.wattch.io/incident/601835 Thu, 12 Jun 2025 20:50:00 -0000 https://status.wattch.io/incident/601835#449495c337599633e096c0cf491847cf35ea3f2d5c9c986faad386cbb006a3f6 The incident has been mostly resolved, according to Google's latest updates. We will wait for Google's full resolution to mark this incident as resolved. Service Unavailable due to GCP Outage https://status.wattch.io/incident/601835 Thu, 12 Jun 2025 20:50:00 -0000 https://status.wattch.io/incident/601835#449495c337599633e096c0cf491847cf35ea3f2d5c9c986faad386cbb006a3f6 The incident has been mostly resolved, according to Google's latest updates. We will wait for Google's full resolution to mark this incident as resolved. Service Unavailable due to GCP Outage https://status.wattch.io/incident/601835 Thu, 12 Jun 2025 19:04:00 -0000 https://status.wattch.io/incident/601835#96d68a93a650a8bbda192654437d7fda828e613f230d2dc1954612e213fa811e An incident has been posted here: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW#RP1d9aZLNFZEJmTBk8e1 Service Unavailable due to GCP Outage https://status.wattch.io/incident/601835 Thu, 12 Jun 2025 19:04:00 -0000 https://status.wattch.io/incident/601835#96d68a93a650a8bbda192654437d7fda828e613f230d2dc1954612e213fa811e An incident has been posted here: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW#RP1d9aZLNFZEJmTBk8e1 Service Unavailable due to GCP Outage https://status.wattch.io/incident/601835 Thu, 12 Jun 2025 19:04:00 -0000 https://status.wattch.io/incident/601835#96d68a93a650a8bbda192654437d7fda828e613f230d2dc1954612e213fa811e An incident has been posted here: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW#RP1d9aZLNFZEJmTBk8e1 Service Unavailable due to GCP Outage https://status.wattch.io/incident/601835 Thu, 12 Jun 2025 19:04:00 -0000 https://status.wattch.io/incident/601835#96d68a93a650a8bbda192654437d7fda828e613f230d2dc1954612e213fa811e An incident has been posted here: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW#RP1d9aZLNFZEJmTBk8e1 Service Unavailable due to GCP Outage https://status.wattch.io/incident/601835 Thu, 12 Jun 2025 19:04:00 -0000 https://status.wattch.io/incident/601835#96d68a93a650a8bbda192654437d7fda828e613f230d2dc1954612e213fa811e An incident has been posted here: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW#RP1d9aZLNFZEJmTBk8e1 Service Unavailable due to GCP Outage https://status.wattch.io/incident/601835 Thu, 12 Jun 2025 18:53:00 -0000 https://status.wattch.io/incident/601835#6fbbdb5d638d013235a1f6410a5c70a9442c8495faa3919560f1681c53d67eeb Google's status can be tracked at https://status.cloud.google.com/. An official incident has not been posted yet, but the product statuses seem to be up to date. Service Unavailable due to GCP Outage https://status.wattch.io/incident/601835 Thu, 12 Jun 2025 18:53:00 -0000 https://status.wattch.io/incident/601835#6fbbdb5d638d013235a1f6410a5c70a9442c8495faa3919560f1681c53d67eeb Google's status can be tracked at https://status.cloud.google.com/. An official incident has not been posted yet, but the product statuses seem to be up to date. Service Unavailable due to GCP Outage https://status.wattch.io/incident/601835 Thu, 12 Jun 2025 18:53:00 -0000 https://status.wattch.io/incident/601835#6fbbdb5d638d013235a1f6410a5c70a9442c8495faa3919560f1681c53d67eeb Google's status can be tracked at https://status.cloud.google.com/. An official incident has not been posted yet, but the product statuses seem to be up to date. Service Unavailable due to GCP Outage https://status.wattch.io/incident/601835 Thu, 12 Jun 2025 18:53:00 -0000 https://status.wattch.io/incident/601835#6fbbdb5d638d013235a1f6410a5c70a9442c8495faa3919560f1681c53d67eeb Google's status can be tracked at https://status.cloud.google.com/. An official incident has not been posted yet, but the product statuses seem to be up to date. Service Unavailable due to GCP Outage https://status.wattch.io/incident/601835 Thu, 12 Jun 2025 18:53:00 -0000 https://status.wattch.io/incident/601835#6fbbdb5d638d013235a1f6410a5c70a9442c8495faa3919560f1681c53d67eeb Google's status can be tracked at https://status.cloud.google.com/. An official incident has not been posted yet, but the product statuses seem to be up to date. Service Unavailable due to GCP Outage https://status.wattch.io/incident/601835 Thu, 12 Jun 2025 18:30:00 -0000 https://status.wattch.io/incident/601835#bc8633519ff3a787fffd83e82087c17ad3246035c28c9d288a636239bd54c712 Users may have trouble accessing Wattch due to an outage in Google Cloud. Our team is monitoring the situation and will provide updates when available. Service Unavailable due to GCP Outage https://status.wattch.io/incident/601835 Thu, 12 Jun 2025 18:30:00 -0000 https://status.wattch.io/incident/601835#bc8633519ff3a787fffd83e82087c17ad3246035c28c9d288a636239bd54c712 Users may have trouble accessing Wattch due to an outage in Google Cloud. Our team is monitoring the situation and will provide updates when available. Service Unavailable due to GCP Outage https://status.wattch.io/incident/601835 Thu, 12 Jun 2025 18:30:00 -0000 https://status.wattch.io/incident/601835#bc8633519ff3a787fffd83e82087c17ad3246035c28c9d288a636239bd54c712 Users may have trouble accessing Wattch due to an outage in Google Cloud. Our team is monitoring the situation and will provide updates when available. Service Unavailable due to GCP Outage https://status.wattch.io/incident/601835 Thu, 12 Jun 2025 18:30:00 -0000 https://status.wattch.io/incident/601835#bc8633519ff3a787fffd83e82087c17ad3246035c28c9d288a636239bd54c712 Users may have trouble accessing Wattch due to an outage in Google Cloud. Our team is monitoring the situation and will provide updates when available. Service Unavailable due to GCP Outage https://status.wattch.io/incident/601835 Thu, 12 Jun 2025 18:30:00 -0000 https://status.wattch.io/incident/601835#bc8633519ff3a787fffd83e82087c17ad3246035c28c9d288a636239bd54c712 Users may have trouble accessing Wattch due to an outage in Google Cloud. Our team is monitoring the situation and will provide updates when available. Alert Generation Temporarily Paused https://status.wattch.io/incident/553921 Tue, 29 Apr 2025 17:47:00 -0000 https://status.wattch.io/incident/553921#fefe4afafc54e3812c76533db3ed8f26ed058a2edf2c4be3b41645bdb61324b5 Our team has identified an issue which prevented certain types of alerts from being created. This issue has been resolved, but customers may notice missing alerts during the affected time frame: 2025-04-28 1:30 PM EDT - 2025-04-29 1:45 PM EDT Alert Generation Temporarily Paused https://status.wattch.io/incident/553921 Tue, 29 Apr 2025 17:47:00 -0000 https://status.wattch.io/incident/553921#fefe4afafc54e3812c76533db3ed8f26ed058a2edf2c4be3b41645bdb61324b5 Our team has identified an issue which prevented certain types of alerts from being created. This issue has been resolved, but customers may notice missing alerts during the affected time frame: 2025-04-28 1:30 PM EDT - 2025-04-29 1:45 PM EDT Database Maintenance https://status.wattch.io/incident/545782 Wed, 16 Apr 2025 19:00:22 -0000 https://status.wattch.io/incident/545782#c80d278f6e14ed72e5ee9d7518539280f3f3602644e3797f7acc09ab00e9d206 Maintenance completed Database Maintenance https://status.wattch.io/incident/545782 Wed, 16 Apr 2025 19:00:22 -0000 https://status.wattch.io/incident/545782#c80d278f6e14ed72e5ee9d7518539280f3f3602644e3797f7acc09ab00e9d206 Maintenance completed Database Maintenance https://status.wattch.io/incident/545782 Wed, 16 Apr 2025 19:00:22 -0000 https://status.wattch.io/incident/545782#c80d278f6e14ed72e5ee9d7518539280f3f3602644e3797f7acc09ab00e9d206 Maintenance completed Database Maintenance https://status.wattch.io/incident/545782 Wed, 16 Apr 2025 19:00:22 -0000 https://status.wattch.io/incident/545782#c80d278f6e14ed72e5ee9d7518539280f3f3602644e3797f7acc09ab00e9d206 Maintenance completed Database Maintenance https://status.wattch.io/incident/545782 Wed, 16 Apr 2025 19:00:22 -0000 https://status.wattch.io/incident/545782#c80d278f6e14ed72e5ee9d7518539280f3f3602644e3797f7acc09ab00e9d206 Maintenance completed Database Maintenance https://status.wattch.io/incident/545782 Wed, 16 Apr 2025 17:00:22 -0000 https://status.wattch.io/incident/545782#8a5387869ca1065005fe6e6616a4716b624eab3b54412528f6db1e015a436517 Wattch will undergo database maintenance during this time. Availability may be degraded. Database Maintenance https://status.wattch.io/incident/545782 Wed, 16 Apr 2025 17:00:22 -0000 https://status.wattch.io/incident/545782#8a5387869ca1065005fe6e6616a4716b624eab3b54412528f6db1e015a436517 Wattch will undergo database maintenance during this time. Availability may be degraded. Database Maintenance https://status.wattch.io/incident/545782 Wed, 16 Apr 2025 17:00:22 -0000 https://status.wattch.io/incident/545782#8a5387869ca1065005fe6e6616a4716b624eab3b54412528f6db1e015a436517 Wattch will undergo database maintenance during this time. Availability may be degraded. Database Maintenance https://status.wattch.io/incident/545782 Wed, 16 Apr 2025 17:00:22 -0000 https://status.wattch.io/incident/545782#8a5387869ca1065005fe6e6616a4716b624eab3b54412528f6db1e015a436517 Wattch will undergo database maintenance during this time. Availability may be degraded. Database Maintenance https://status.wattch.io/incident/545782 Wed, 16 Apr 2025 17:00:22 -0000 https://status.wattch.io/incident/545782#8a5387869ca1065005fe6e6616a4716b624eab3b54412528f6db1e015a436517 Wattch will undergo database maintenance during this time. Availability may be degraded. Cloud Upgrades https://status.wattch.io/incident/534409 Wed, 26 Mar 2025 17:45:39 -0000 https://status.wattch.io/incident/534409#892209ebae6c743d5305e04eab4ef18185ec4e9c9491358ade5f4df673981f0b Maintenance completed Cloud Upgrades https://status.wattch.io/incident/534409 Wed, 26 Mar 2025 17:45:39 -0000 https://status.wattch.io/incident/534409#892209ebae6c743d5305e04eab4ef18185ec4e9c9491358ade5f4df673981f0b Maintenance completed Cloud Upgrades https://status.wattch.io/incident/534409 Wed, 26 Mar 2025 17:45:39 -0000 https://status.wattch.io/incident/534409#892209ebae6c743d5305e04eab4ef18185ec4e9c9491358ade5f4df673981f0b Maintenance completed Cloud Upgrades https://status.wattch.io/incident/534409 Wed, 26 Mar 2025 17:30:39 -0000 https://status.wattch.io/incident/534409#a9dc148da0f263dbce54f819568b87f4802fe9b942351ff251590b575e6b3340 Wattch may experience temporary service degradation during this time. Any interruption should be short-lived. Cloud Upgrades https://status.wattch.io/incident/534409 Wed, 26 Mar 2025 17:30:39 -0000 https://status.wattch.io/incident/534409#a9dc148da0f263dbce54f819568b87f4802fe9b942351ff251590b575e6b3340 Wattch may experience temporary service degradation during this time. Any interruption should be short-lived. Cloud Upgrades https://status.wattch.io/incident/534409 Wed, 26 Mar 2025 17:30:39 -0000 https://status.wattch.io/incident/534409#a9dc148da0f263dbce54f819568b87f4802fe9b942351ff251590b575e6b3340 Wattch may experience temporary service degradation during this time. Any interruption should be short-lived. Some eGauge Data Temporarily Unavailable https://status.wattch.io/incident/524474 Fri, 07 Mar 2025 21:44:00 -0000 https://status.wattch.io/incident/524474#a16fa0823490a8db49c1c8b437a5ee81b458296a8eda0daba1be64db4723a719 We have applied a fix - all of the affected data should now be visible. Some eGauge Data Temporarily Unavailable https://status.wattch.io/incident/524474 Fri, 07 Mar 2025 21:44:00 -0000 https://status.wattch.io/incident/524474#a16fa0823490a8db49c1c8b437a5ee81b458296a8eda0daba1be64db4723a719 We have applied a fix - all of the affected data should now be visible. Some eGauge Data Temporarily Unavailable https://status.wattch.io/incident/524474 Fri, 07 Mar 2025 21:44:00 -0000 https://status.wattch.io/incident/524474#a16fa0823490a8db49c1c8b437a5ee81b458296a8eda0daba1be64db4723a719 We have applied a fix - all of the affected data should now be visible. Some eGauge Data Temporarily Unavailable https://status.wattch.io/incident/524474 Fri, 07 Mar 2025 21:44:00 -0000 https://status.wattch.io/incident/524474#a16fa0823490a8db49c1c8b437a5ee81b458296a8eda0daba1be64db4723a719 We have applied a fix - all of the affected data should now be visible. Some eGauge Data Temporarily Unavailable https://status.wattch.io/incident/524474 Fri, 07 Mar 2025 19:04:00 -0000 https://status.wattch.io/incident/524474#d56d66c54bebbbddc1974124b8b57d4f784b736e747529ac5fad2281a652d040 Some data reported by eGauges on March 6th, 2025, is not currently available. We're working on a fix as quickly as possible. Some eGauge Data Temporarily Unavailable https://status.wattch.io/incident/524474 Fri, 07 Mar 2025 19:04:00 -0000 https://status.wattch.io/incident/524474#d56d66c54bebbbddc1974124b8b57d4f784b736e747529ac5fad2281a652d040 Some data reported by eGauges on March 6th, 2025, is not currently available. We're working on a fix as quickly as possible. Some eGauge Data Temporarily Unavailable https://status.wattch.io/incident/524474 Fri, 07 Mar 2025 19:04:00 -0000 https://status.wattch.io/incident/524474#d56d66c54bebbbddc1974124b8b57d4f784b736e747529ac5fad2281a652d040 Some data reported by eGauges on March 6th, 2025, is not currently available. We're working on a fix as quickly as possible. Some eGauge Data Temporarily Unavailable https://status.wattch.io/incident/524474 Fri, 07 Mar 2025 19:04:00 -0000 https://status.wattch.io/incident/524474#d56d66c54bebbbddc1974124b8b57d4f784b736e747529ac5fad2281a652d040 Some data reported by eGauges on March 6th, 2025, is not currently available. We're working on a fix as quickly as possible. Cloud Upgrades https://status.wattch.io/incident/492151 Wed, 22 Jan 2025 18:45:43 -0000 https://status.wattch.io/incident/492151#673c17dcf4c3dca1199855cf48ff101e912770d699c4272ae453e60816521a00 Maintenance completed Cloud Upgrades https://status.wattch.io/incident/492151 Wed, 22 Jan 2025 18:45:43 -0000 https://status.wattch.io/incident/492151#673c17dcf4c3dca1199855cf48ff101e912770d699c4272ae453e60816521a00 Maintenance completed Cloud Upgrades https://status.wattch.io/incident/492151 Wed, 22 Jan 2025 18:45:43 -0000 https://status.wattch.io/incident/492151#673c17dcf4c3dca1199855cf48ff101e912770d699c4272ae453e60816521a00 Maintenance completed Cloud Upgrades https://status.wattch.io/incident/492151 Wed, 22 Jan 2025 18:45:43 -0000 https://status.wattch.io/incident/492151#673c17dcf4c3dca1199855cf48ff101e912770d699c4272ae453e60816521a00 Maintenance completed Cloud Upgrades https://status.wattch.io/incident/492151 Wed, 22 Jan 2025 18:45:43 -0000 https://status.wattch.io/incident/492151#673c17dcf4c3dca1199855cf48ff101e912770d699c4272ae453e60816521a00 Maintenance completed Cloud Upgrades https://status.wattch.io/incident/492151 Wed, 22 Jan 2025 18:00:43 -0000 https://status.wattch.io/incident/492151#81e2998611f2f4d26162f73cdd2478b6636d0b1fed7d8560bc8bbbc208e9b567 Upgrading database and cloud VMs. Downtime should only last a few minutes. Cloud Upgrades https://status.wattch.io/incident/492151 Wed, 22 Jan 2025 18:00:43 -0000 https://status.wattch.io/incident/492151#81e2998611f2f4d26162f73cdd2478b6636d0b1fed7d8560bc8bbbc208e9b567 Upgrading database and cloud VMs. Downtime should only last a few minutes. Cloud Upgrades https://status.wattch.io/incident/492151 Wed, 22 Jan 2025 18:00:43 -0000 https://status.wattch.io/incident/492151#81e2998611f2f4d26162f73cdd2478b6636d0b1fed7d8560bc8bbbc208e9b567 Upgrading database and cloud VMs. Downtime should only last a few minutes. Cloud Upgrades https://status.wattch.io/incident/492151 Wed, 22 Jan 2025 18:00:43 -0000 https://status.wattch.io/incident/492151#81e2998611f2f4d26162f73cdd2478b6636d0b1fed7d8560bc8bbbc208e9b567 Upgrading database and cloud VMs. Downtime should only last a few minutes. Cloud Upgrades https://status.wattch.io/incident/492151 Wed, 22 Jan 2025 18:00:43 -0000 https://status.wattch.io/incident/492151#81e2998611f2f4d26162f73cdd2478b6636d0b1fed7d8560bc8bbbc208e9b567 Upgrading database and cloud VMs. Downtime should only last a few minutes. Data processing rates are temporarily degraded - no data is being lost https://status.wattch.io/incident/492684 Wed, 08 Jan 2025 19:26:00 -0000 https://status.wattch.io/incident/492684#3da076a8899ac6681982885598935d37a42b6b07332d678d6b35d14aa9b36ee7 All systems are operational. Data processing rates are temporarily degraded - no data is being lost https://status.wattch.io/incident/492684 Wed, 08 Jan 2025 19:26:00 -0000 https://status.wattch.io/incident/492684#3da076a8899ac6681982885598935d37a42b6b07332d678d6b35d14aa9b36ee7 All systems are operational. Data processing rates are temporarily degraded - no data is being lost https://status.wattch.io/incident/492684 Wed, 08 Jan 2025 19:26:00 -0000 https://status.wattch.io/incident/492684#3da076a8899ac6681982885598935d37a42b6b07332d678d6b35d14aa9b36ee7 All systems are operational. Data processing rates are temporarily degraded - no data is being lost https://status.wattch.io/incident/492684 Wed, 08 Jan 2025 17:02:00 -0000 https://status.wattch.io/incident/492684#00e4fe6913dfeb34440bc2843c6bd28e418debcdabac657994de18d349537950 Systems are returning to normal operation. We will continue to monitor and will update with any changes. Data processing rates are temporarily degraded - no data is being lost https://status.wattch.io/incident/492684 Wed, 08 Jan 2025 17:02:00 -0000 https://status.wattch.io/incident/492684#00e4fe6913dfeb34440bc2843c6bd28e418debcdabac657994de18d349537950 Systems are returning to normal operation. We will continue to monitor and will update with any changes. Data processing rates are temporarily degraded - no data is being lost https://status.wattch.io/incident/492684 Wed, 08 Jan 2025 17:02:00 -0000 https://status.wattch.io/incident/492684#00e4fe6913dfeb34440bc2843c6bd28e418debcdabac657994de18d349537950 Systems are returning to normal operation. We will continue to monitor and will update with any changes. Data processing rates are temporarily degraded - no data is being lost https://status.wattch.io/incident/492684 Wed, 08 Jan 2025 16:25:00 -0000 https://status.wattch.io/incident/492684#9eb482c6757e3666cd8a13a2c6d7a89e427b412f872f39b2091c6948208bc02e Our data processing and ingestion queues are currently experiencing elevated latency, resulting in some portions of the app.wattch.io User Interface not rendering properly. All data is being safely recorded at this time and we are not anticipating any lasting impacts or data loss as part of this incident. Our team is working quickly to resolve the issue and restore full functionality to the product. Data processing rates are temporarily degraded - no data is being lost https://status.wattch.io/incident/492684 Wed, 08 Jan 2025 16:25:00 -0000 https://status.wattch.io/incident/492684#9eb482c6757e3666cd8a13a2c6d7a89e427b412f872f39b2091c6948208bc02e Our data processing and ingestion queues are currently experiencing elevated latency, resulting in some portions of the app.wattch.io User Interface not rendering properly. All data is being safely recorded at this time and we are not anticipating any lasting impacts or data loss as part of this incident. Our team is working quickly to resolve the issue and restore full functionality to the product. Data processing rates are temporarily degraded - no data is being lost https://status.wattch.io/incident/492684 Wed, 08 Jan 2025 16:25:00 -0000 https://status.wattch.io/incident/492684#9eb482c6757e3666cd8a13a2c6d7a89e427b412f872f39b2091c6948208bc02e Our data processing and ingestion queues are currently experiencing elevated latency, resulting in some portions of the app.wattch.io User Interface not rendering properly. All data is being safely recorded at this time and we are not anticipating any lasting impacts or data loss as part of this incident. Our team is working quickly to resolve the issue and restore full functionality to the product. Partial cellular outage https://status.wattch.io/incident/437575 Tue, 01 Oct 2024 19:48:00 -0000 https://status.wattch.io/incident/437575#1da66af36eccd7582606f0dc077f42863ea1db014b788e5f5281ec76f577776e The issue has been resolved. All WECs are back online. Partial cellular outage https://status.wattch.io/incident/437575 Tue, 01 Oct 2024 17:43:00 -0000 https://status.wattch.io/incident/437575#9fac9b07ed681acc8755d82e387c6a83a809d900e2af8fae88cdf6c21ba4c9fe Our carrier partner has posted that the issue has been resolved, and we've seen most WECs/modems come back online. We will continue to monitor stability over the next hour. Partial cellular outage https://status.wattch.io/incident/437575 Tue, 01 Oct 2024 14:21:00 -0000 https://status.wattch.io/incident/437575#8ebccf565264e6c188e0bd88d1eb181051b75ce0ab3850de5c7cd3a2dc03a6d1 It seems like we are seeing connections becoming more stable and are trending towards resolution. 93% of devices are online. Partial cellular outage https://status.wattch.io/incident/437575 Tue, 01 Oct 2024 12:24:00 -0000 https://status.wattch.io/incident/437575#ba6342931718c45f19e37826042912b89f5f9b23771f2d3f4146ac937e25bbf8 Our primary cellular partner has reported a partial data outage. They are working to resolve the issue but some WECs/modems will be offline until this is resolved. We are monitoring the issue as closely as possible Investigating connectivity issues. https://status.wattch.io/incident/400107 Thu, 01 Aug 2024 16:45:00 -0000 https://status.wattch.io/incident/400107#1ae72956bbf08b58cb34556e6d68073d2cf81fabce1f8f03a936475e2a99da61 Please find the post-mortem details for the incident occurring on July 18th below: **What happened?** The Wattch Edge Controller (WEC) uses an operating system (OS), called Ubuntu Core. While Wattch maintains an application on the WEC (the “firmware”) that implements all Wattch user facing functionality, the underlying Ubuntu Core OS is maintained by our partner Canonical. Canonical is a global leader in enterprise computing and cyber security - their engineering efforts help ensure the operating system remains secure by shipping periodic patches and constantly evaluating threat vectors. On July 18th, Canonical deployed a security update to the Ubuntu Core OS. As WECs picked up the software update, they began to fall offline. We were able to identify that only the first ~100 devices sold by Wattch were affected . This was because they had an earlier version of the underlying hardware (Broadcom BCM2710A1 vs. BCM2711 CPU core). We concluded that the earlier version of the hardware was incompatible with the updated OS. Wattch’s engineering team worked from Thursday through Sunday with Canonical’s 24/7 incident response team to identify the root cause of the issue. Canonical froze and eventually globally rolled back their deployment of the security patch during this investigation process. Once the root cause was identified, Wattch created a recovery process for the two subtypes of hardware devices affected and worked with customers to deploy the fixes across any affected sites. **What are we doing to prevent this in the future?** Wattch implements a multi-month, three tier approach to rolling out changes to our application payload (the “firmware”): 1. Extensive in-lab bench testing on known-good hardware systems 2. “Canary” hardware owned by Wattch installed at real PV/BESS sites 3. Gradual rollout to customer sites based on geography and asset-type Historically, security and reliability patches to Canonical’s Ubuntu Core OS have bypassed Wattch’s rollout procedure, instead relying on Canonical’s testing process. Moving forward, Wattch will fold all patches issued by Canonical into our standard testing procedure. This will allow us to maintain control of the rollout of any new software and catch any breaking bugs before they reach customer devices. **What was the impact?** - _Site monitoring was unavailable for affected sites during the incident._ All affected sites required a site visit to recover communications and data flow. Sites with Revolution series hardware (orange faceplate) were power cycled and sites with Strato series devices required the WEC to be physically repaired by Wattch staff or replaced. - _Inverter level production data was NOT recorded in Wattch for the period the WEC was offline._ Wattch cellular modems remained functional during the incident. If your devices are connected to an inverter native platform via the Wattch modem, you should have continued to collect device data in that platform(s). - _Revenue Grade Meter data WAS recorded for the period the WEC was offline._ Cumulative revenue values from the meter backfill in Wattch once the WEC is back online. The recovered delta in data will be mis-attributed to the day the WEC was recovered. Investigating connectivity issues. https://status.wattch.io/incident/400107 Tue, 23 Jul 2024 20:35:00 -0000 https://status.wattch.io/incident/400107#d38d3689dc169749e2d5f60f2fa34f5d42ddc888c9dda6d573fccf6264df7d98 A final fix has been identified and we are working with all affected customers to implement the solution. Investigating connectivity issues. https://status.wattch.io/incident/400107 Tue, 23 Jul 2024 17:19:00 -0000 https://status.wattch.io/incident/400107#4f19283a32cb9faea616abb932b3875a43ba9f2d260548f1c8906cc93c8df043 A final fix has been identified and all affected customers have been contacted with a solution. Investigating connectivity issues. https://status.wattch.io/incident/400107 Mon, 22 Jul 2024 18:54:00 -0000 https://status.wattch.io/incident/400107#88268ab58cda51c7bd4cdb9476829e32b5740184c2f3b789481a132beee135bb Over the weekend, we continued to work with Canonical to fully identify the root cause of the issue preventing a subset of WECs from connecting to Wattch. We have been able to successfully recover a StratoPi device on our test bench. Currently, we are working to develop an easy in-field recovery solution for sites with CMP3 core StratoPi devices. Affected customers will be notified with remediation steps once confirmed. _NOTE: If you have not received an email from Wattch about this issue, your account is unaffected._ Investigating connectivity issues. https://status.wattch.io/incident/400107 Fri, 19 Jul 2024 21:36:00 -0000 https://status.wattch.io/incident/400107#3fbe07329da455ee9c4986234a276589671013382c781bf082b8add010e86386 We have worked with our partner, Canonical, to build a remediation strategy for the two types of affected WEC devices. We have an identified solution for "RevPi" devices and are still working on a solution for "StratoPi" devices. We have tested extensively on our in-office bench devices as well as at several sites in the field Customers with affected devices have been emailed with specific instructions and next steps. Further communication and update will be sent on Monday. Investigating connectivity issues. https://status.wattch.io/incident/400107 Fri, 19 Jul 2024 15:21:00 -0000 https://status.wattch.io/incident/400107#0966799db59c9b28cd1b4c48fbf05b70119ad76f73e1942a10f83eb7dd949652 Yesterday, we identified a bug in a regularly scheduled security patch update by our third party provider, Canonical, that caused older model WECs to fall offline. We’re actively working with our third party provider to establish a recovery process for bringing these WECs back online. Please DO NOT try to physically reboot affected devices at this time. We will be communicating to customers with offline sites about the recovery process as soon as it is established. For affected sites, we expect timeseries inverter/device data to be lost for the period of the outage but cumulative (odometer) revenue grade meter data will NOT be lost. Investigating connectivity issues. https://status.wattch.io/incident/400107 Thu, 18 Jul 2024 20:19:00 -0000 https://status.wattch.io/incident/400107#32b53a2561fe29c0860cb4d6cf7d9448849bf9e1ad26086ee7923d1f736f17b7 We've identified an issue and are working with our third-party vendor to fix impacted WECs. Investigating connectivity issues. https://status.wattch.io/incident/400107 Thu, 18 Jul 2024 16:18:00 -0000 https://status.wattch.io/incident/400107#4e79577219c4e02c08af4706af3230494fa15ead64725b20fbb3ff8d95b028fb Wattch is currently investigating issues involving Edge Controller authentication with the Cloud that are preventing devices from periodically refreshing their secure connection. This is manifesting as a small subset of Wattch Core (Edge Controller-connected) sites appearing to go offline. Wattch Lite (API Connected) sites are not impacted. We do not anticipate any loss of data - missing points will backfill once connection is reestablished. Resolution is currently expected within the next 3-4 hours, but we will share more updates as the investigation continues. Investigating connectivity issues. https://status.wattch.io/incident/400107 Thu, 18 Jul 2024 15:29:00 -0000 https://status.wattch.io/incident/400107#6c56418ebc2f2cf0d23581a219f52789e1d83d753bae44b6ff3ae12c2a5909b9 We are currently investigating issues with connectivity to WECs. We will update as we learn more. Degraded performance https://status.wattch.io/incident/355998 Mon, 15 Apr 2024 23:00:00 -0000 https://status.wattch.io/incident/355998#75caf7666194f0206fdeea5d137c972322051fc0b935ce36aa8e9a19ce3c4e78 The incident has been mitigated and we will continue to monitor for any issues Degraded performance https://status.wattch.io/incident/355998 Mon, 15 Apr 2024 23:00:00 -0000 https://status.wattch.io/incident/355998#75caf7666194f0206fdeea5d137c972322051fc0b935ce36aa8e9a19ce3c4e78 The incident has been mitigated and we will continue to monitor for any issues Degraded performance https://status.wattch.io/incident/355998 Mon, 15 Apr 2024 22:39:00 -0000 https://status.wattch.io/incident/355998#0b449911ee8d4e10341de17a03233a263b42c2e66bccc6cda990826e5f857785 Wattch is currently experiencing degradation in some services like charting and performance models due to a deployment that happened today at 6PM. We are working to revert the changes and working towards full service restoration. Degraded performance https://status.wattch.io/incident/355998 Mon, 15 Apr 2024 22:39:00 -0000 https://status.wattch.io/incident/355998#0b449911ee8d4e10341de17a03233a263b42c2e66bccc6cda990826e5f857785 Wattch is currently experiencing degradation in some services like charting and performance models due to a deployment that happened today at 6PM. We are working to revert the changes and working towards full service restoration.