Webex Teams: Ongoing service restoration activities
Incident Report for Cisco Webex Teams
Resolved
Metrics and monitors continue to be positive, and services are operating as expected.

As reported in the previous update, there are two known issues that remain after the repairs were completed:
1. A small number of spaces show the list of users in the space as the space name, rather than the correct name of the space. If a user experiences this issue with their space, they or one of the participants in the space should rename the space to refresh the name.
2. Some spaces may still be missing some historical content. While the data is available in the Teams data set in the cloud, the client is not always picking up the updated content and refreshing its cache. Engineering will push a client update that will address this issue, and users may use the web client to review that content in the interim.

This incident is now resolved. For issues that are experienced outside of the documented known issues, please contact Cisco Technical Support. We are committed to continuously learning and driving initiatives to harden our cloud, and really appreciate your patience as we worked through this service incident.
Posted 2 months ago. Oct 16, 2018 - 05:05 UTC
Update
The scripts to restore user spaces and Teams have been completing as expected. The targeted scripts have been executed, and users should see their spaces with restored content and posting capabilities.

There are two known issues after the repairs are completed:
1. A small number of spaces show the list of users in the space as the space name, rather than the correct name of the space. This is more commonly experienced on the mobile applications than on the desktop applications due to client caching functionalities. If a user experiences this issue with their space, they or one of the participants in the space will need to rename the space. In most cases, if the affected user can log into the desktop app they will see the proper space name. They can then make a minor change to the space name, and then change it back, and it will reset it for the space name for all users. When the mobile app refreshes, users will see the space name as expected.
2. Some spaces may still be missing some historical content. While the data is available in the Teams data set in the cloud, the client is not always picking up the updated content and refreshing its cache. This is limited to the desktop application; users may log into the web client and view the space history. Engineering will push a client update that will address this issue in the upcoming days.

If a user has a space that is not operational, and is experiencing an issue outside of the two known issues above, please completely exit the application, reopen, and try again.

All other services, features, and functionality are fully operational. If issues are being experienced outside of the noted areas above, and a restart of the application has not resolved the issue, please contact Webex Teams technical support.

Thank you for your patience while we addressed the remaining space issues.
Posted 2 months ago. Oct 15, 2018 - 05:40 UTC
Update
Space fixes are ongoing. The fix to address additional 1:1 spaces is completing deployment. The Teams fix is still under development, and we will provide updates on the engineering progress.

Until these fixes are fully deployed, there are some spaces that users cannot send messages in, spaces that may be missing some historical content, and Teams that are missing or have spaces that are not operational. However, these fixes are being continuously deployed, so users may retry those 1:1 spaces and spaces that are not associated with a Team as they are undergoing repair.

The repairs that are being made either through manual workarounds or by the scripts that are being deployed may require that a user close and reopen their application for the repaired spaces to be available. If you have utilized the workaround, or encounter a space that is not operating, please completely exit and reopen the application and retry your operation.

Thank you for your patience as we deploy these updates.
Posted 2 months ago. Oct 14, 2018 - 05:42 UTC
Update
Space fixes are ongoing. The script that targets historical content for spaces is continuing to run, and is nearing completion. The fix to address additional 1:1 spaces has completed and is beginning deployment. The next fix targets Teams that are not displaying or operating properly in the client. This fix is under deployment, and will move to production once it's completed.

Until these fixes are fully deployed, there are some spaces that users cannot send messages in, spaces that may be missing some historical content, and Teams that are missing or have spaces that are not operational. However, these fixes are being continuously deployed, so 1:1 spaces and spaces that are not associated with a Team may be periodically retested.

The repairs that are being made either through manual workarounds or by the scripts that are being deployed may require that a user close and reopen their application for the repaired spaces to be available. If you have utilized the workaround, or encounter a space that is not operating, please completely exit and reopen the application and retry your operation.

Thank you for your continued patience as we deploy these updates.
Posted 2 months ago. Oct 13, 2018 - 07:25 UTC
Update
Space fixes are ongoing. The script that targets historical content for spaces is continuing to run, and has approximately another 24 hours until completion.

A fix is under development to address additional 1:1 spaces, as well as Teams that are not displaying or operating properly in the client. Both of those are under development and will move to production soon.
Until these fixes are deployed, there are some spaces that users cannot send messages in, spaces that may be missing some historical content, and Teams that are missing or have spaces that are not operational.

The repairs that are being made either through manual workarounds or by the scripts that are being deployed may require that a user close and reopen their application for the repaired spaces to be available. If you have utilized the workaround or encounter a space that is not operating, please completely exit and reopen the application and retry your operation.

Thank you for your ongoing patience as we deploy these updates.
Posted 2 months ago. Oct 12, 2018 - 04:38 UTC
Update
Space fixes are ongoing. Scripts have been deployed that address a majority of 1:1 spaces that could not be accessed or messages could not be sent in. A script that is being deployed now is working to restore historical content for spaces that previously only had the client cached content displaying. This script will take additional time to complete, but is running continuously.

A fix is under development to address additional 1:1 spaces, as well as Teams that are not displaying or operating properly in the client.

While the fixes that we have deployed thus far have addressed a majority of the issues, we do have one script that is still deploying and one script that is under development. Until these fixes are deployed, there are some spaces that users cannot send messages in, spaces that may be missing some historical content, and Teams that are missing or have spaces that are not operational.

If you are seeing any spaces or messages that say "Decrypting", please update the client, if an update is available as indicated by a green circle on the left hand panel, and restart the application. A client fix was published late last week that addressed that issue.

The repairs that are being made either through manual workarounds or by the scripts that are being deployed may require that a user close and reopen their application for the repaired spaces to be available. If you have utilized the workaround or encounter a space that is not operating, please completely exit and reopen the application.

Thank you for your ongoing patience as we deploy these updates.
Posted 2 months ago. Oct 11, 2018 - 06:01 UTC
Update
Space repairs are ongoing, and executing as expected. The two scripts mentioned previously that were designed to target additional space repairs have been coded, and are being deployed. These are targeted to complete within the next 24 hours.

The workaround mentioned previously continues to be an available way to trigger the script to run on a specific 1:1 space.

The repairs that are being made either through manual workarounds or by the scripts that are being deployed may require that a user close and reopen their application for the repaired spaces to be available. If you have utilized the workaround or encounter a space that is not operating, please exit and reopen the application.

Thank you for your ongoing patience as we deploy these updates.
Posted 2 months ago. Oct 10, 2018 - 05:36 UTC
Update
The space repair is continuing to progress. The previously reported sets of scripts were completed, and metrics confirmed that users are back in the targeted user spaces. Two additional scripts are being planned which target additional space repairs. The first of these is set to begin deployments today.

These do take time to run as the process includes multiple validation points as it executes across the user spaces. The workaround mentioned previously continues to be an available way to trigger the script to run on a specific 1:1 space. The repairs may require that a user close and reopen their application for the repaired spaces to be available. If you have utilized the workaround or encounter a space that is not operating, please exit and reopen the application.

There are still more updates coming, so while a large majority of the spaces are already repaired, or in the process of being repaired, there are still some fixes in work to address some remaining use cases. Thank you for your ongoing patience as we deploy these updates and fine tune the repair processes.
Posted 2 months ago. Oct 09, 2018 - 06:15 UTC
Update
The space repair is continuing to process with success. The engineers have deployed an additional script that is designed to target the incorrect data that is associated with some previously created spaces. This update is designed to allow message sends in a space where previously the space was available to the user, but the message sends within that space were failing. This fix has been deployed in addition to the previous scripts, and they are continuing to execute. These do take time to run as the process includes multiple validation points as it executes across the user spaces.

The workaround mentioned previously continues to be an available way to trigger the script to run on a specific 1:1 space.

The repairs may require that a user close and reopen their application for the repaired spaces to be available. If you have utilized the workaround or encounter a space that is not operating, please exit and reopen the application. There are still more updates coming, so while a large majority of the spaces are already repaired, or in the process of being repaired, there are still some fixes in work to address some remaining use cases.

Thank you for your ongoing patience as we deploy these updates and fine tune the repair processes.
Posted 2 months ago. Oct 08, 2018 - 05:58 UTC
Update
The 1:1 space repair is processing with success. The engineers continue to enhance the scripts and have expanded some repairs to other spaces beyond the 1:1 spaces.

The process is ongoing, and is moving forward with success. The workaround mentioned previously continues to be a valid way to trigger the script to run on a specific 1:1 space.

Thank you for your ongoing patience as we deploy these updates and fine tune the repair processes.
Posted 2 months ago. Oct 07, 2018 - 07:01 UTC
Update
The cloud metrics and monitors continue to trend positive and fixes are underway for some of the space issues that users are reporting.

We have deployed a script that is running in the cloud that is completing automated repairs on 1:1 spaces. This script is identifying 1:1 spaces where users are attempting to connect, and then running a repair script that targets certain type of space errors.
If a user has a 1:1 space that can be accessed from the client, but messages cannot be sent, a manual workaround can trigger that repair script to run against that specific space more quickly. Please attempt to send a message in that space, or click the ‘add to favorites’ star in the information panel of the space. This message send failure or space activity produces an error code that the client sends to the cloud, which then triggers the repair script. After taking action in the space, please wait 30-60 seconds, and then if it doesn’t automatically repair, close and reopen the client application. After reloading the client, if the space was able to be repaired with this script iteration, the user will then be able to send messages.
Note that this specific script iteration doesn’t cover every use case, but it does fix some, and we have observed the success in the metrics as this script has begun iterating through the space data. This script release does not target the scenarios where the user cannot click favorites in the 1:1 space, the 1:1 space doesn’t appear, or non-1:1 spaces.

While this workaround is having success, we are moving forward with added scripts that do not require manual intervention. The engineering team is working on the next set of scripts, which we will start to deploy shortly. This next fix is designed to expand the 1:1 space repair and move into addressing non-1:1 spaces.

In addition to the space repairs, Engineering released desktop client version 3.0.9693.0, which fixes a user experience where certain spaces had messages stuck in a decrypting state.

As we continue to deploy these solutions, we will provide updates. Thank you for your continued patience.
Posted 2 months ago. Oct 06, 2018 - 04:25 UTC
Update
Engineering continues to work on the tasks to address the remaining open items. The progress in testing over the last 24 hours has been positive, and we're looking forward to deploying the workaround for some of the affected 1:1 spaces soon. As soon as it is available, details will be provided.
The broader code solution is undergoing testing in parallel, and the results are positive. There is still more testing and some additional work to be done, and we will provide updates as we continue to progress.

Thank you for your continued patience.
Posted 2 months ago. Oct 05, 2018 - 05:54 UTC
Update
Engineering continues to work on the engineering tasks to address the remaining open items.
Some Webex Teams users are still experiencing inconsistent results with some user spaces. While much of the space data has been restored and is fully operational, there remain some user spaces that either do not appear, or appear but messages and calls are not working.

These user experiences are caused by a condition where the roster or space data is incorrect. Engineering continues to work on code solutions. The engineering team is looking to provide a workaround for some of the affected 1:1 spaces first, followed by a broader code solution. The workaround is undergoing testing, and the broader code solution is being developed. It's taking longer than we'd like, but we're diligently working to address the issues.

The mobile app updates for the previously referenced OBTP capabilities in certain Space Meetings released on time and is available from the app stores.

Thank you for your continued patience. We will provide ongoing updates on our progress.
Posted 2 months ago. Oct 04, 2018 - 05:00 UTC
Update
Metrics and monitors confirm that the network connectivity across the Teams infrastructure is operating as expected. The latency issue experienced with the Webex Teams client has also been addressed for our consumer customers.

Engineering continues to work on the remaining items to restore the Cisco Webex Teams services. The ongoing data restoration tasks did complete on time, and all data restoration activities have completed.

The remaining issues affect user access to spaces and One Button to Push capabilities in some Space Meetings.

Some users continue to experience issues accessing and sending messages to specific spaces. The engineering teams are continuing to work on a code solution to address those impaired spaces, and an ETA will be provided as soon as one is available.

The mobile app updates for the OBTP capabilities in certain Space Meetings are on track and expected to be deployed to the App Stores soon.

Thank you for your continued patience. We will provide ongoing updates on our progress.
Posted 3 months ago. Oct 03, 2018 - 02:57 UTC
Update
The latency is subsiding for many of our users. Configuration changes and service redeploys have allowed the services to restore and the network connectivity is operating at expected levels. Devices are online and able to make and receive calls. Calendar Connect services are also operational; the @meet and @webex functionality has also restored. Context and Care is also operational.

Most Webex Teams users are no longer experiencing service latency. Users that are connected to our consumer org environment, which include users logged in with a user account that is not connected to a paid Webex organization, may still be experiencing connectivity issues. We continue to work to restore those users.

Thanks for your continued patience as we addressed the latency.
Posted 3 months ago. Oct 02, 2018 - 22:34 UTC
Update
The Webex Teams service continues to experience high service latency. The engineers continue to complete the tasks on our service remediation list, and these efforts are taking additional time to complete than we would like. The latency is due to the overloading of one service as part of the recovery process from last week’s incident. The team is working on some additional service deployments, in conjunction with adding some additional network controls that will allow the connections to stay connected for a longer period of time, which should address some of the connection retries. Some Webex devices are now online and able to make calls, however, incoming calls and other pairing functions will not be available. The Webex Teams and Calendar services continue to experience latency and errors when connecting to the service.

We understand that the service latency and errors are frustrating, and we are doing everything we can to restore the services as quickly as we can. Thank you for your continued patience.
Posted 3 months ago. Oct 02, 2018 - 20:56 UTC
Update
Engineering is continuing to execute the previous mentioned plan of action to stabilize the network connectivity for the clients and endpoints into the data center. Some configurations have been deployed, which have allowed devices to connect. Mobile and Mac clients are connecting much more quickly, as are our web client users. Windows clients are starting to improve, but will still need some additional time to synchronize. The complete restoration is still in flight, and some latency is still expected as we complete our plan of action.

We will provide updates as they become available. Thank you for your continued patience as we address this service issue.
Posted 3 months ago. Oct 02, 2018 - 19:33 UTC
Update
Service restoration activities are ongoing, and users continue to experience latency or connection failures when using the Webex Teams clients, devices, and Hybrid Calendar services.

We've identified an issue within one of the Teams micro-services that is causing client connection requests to fail. This is leading to the latency that is being experienced and the intermittency in the endpoints connecting into the infrastructure. When the client can successfully get the connection, it will update the client data until the session refreshes and potentially loses that connection.

We have an execution path to address the retries that are causing additional connection sessions, and bring stability to the network connections. We sincerely apologize for the delays in getting the devices and clients back online and available. While the restoration will take a bit more time, we are working to restore services as quickly as possible.
Posted 3 months ago. Oct 02, 2018 - 18:21 UTC
Update
Service update: Care and Context Service are experiencing issues with their services.
Posted 3 months ago. Oct 02, 2018 - 17:36 UTC
Update
Restoration efforts are still ongoing.

For Calendar services: @meet and @webex functionality is latent or unable to generate the meeting details. Some meetings may not have One Button to Push displaying.
Posted 3 months ago. Oct 02, 2018 - 17:19 UTC
Update
Update: The Care Assistant bot is also affected by this connectivity issue. Adding this service impact.
Posted 3 months ago. Oct 02, 2018 - 16:59 UTC
Update
Restoration activities are still in flight. Access to the service is still intermittent for both Teams and devices while we work to restore the connectivity.

We appreciate the continued patience during this service incident. We are working diligently to address this incident.
Posted 3 months ago. Oct 02, 2018 - 16:57 UTC
Update
Engineering is continuing to work on the current connectivity issues. Messaging client and endpoint connectivity is still latent or seeing errors. In addition, Calling users may have limited access to leave or retrieve voicemail.

We are working to address these services as quickly as possible. We will continue to provide status on our progress.
Posted 3 months ago. Oct 02, 2018 - 16:24 UTC
Update
We are still working to address the current issue. Teams client users may see intermittent message delays, error creating new spaces, network connectivity messages, or authentication errors.
Cloud registered devices, including Webex Boards, are also seeing connectivity issues or errors.

We are working diligently to restore these services, and will continue to provide updates.
Posted 3 months ago. Oct 02, 2018 - 16:09 UTC
Update
Deployment is complete and while services are starting to stabilize, we continue to work on the issue. In addition to the previous impacts, some message sends were experienced and some cloud registered Webex devices may have had issues connecting into the service, or displayed a "Can't connect to Cisco Webex services" message. There are still some intermittent connectivity issues expected as we continue to address the current issue.

Updates will be provided as they become available.
Posted 3 months ago. Oct 02, 2018 - 15:50 UTC
Update
Users are currently experiencing issues creating spaces and teams. This issue is also affecting some APIs and bots. Some users may not be able to log into the Teams web client and open their spaces.

Engineering is working to deploy an update to address the issue.
Posted 3 months ago. Oct 02, 2018 - 15:33 UTC
Update
Engineering is continuing to work on the remaining user experiences, and we do have some updated ETAs that we'd like to share.

Messaging service status: The Webex Teams Messaging services continue to be operational. New messages, spaces and teams are available and fully functional. Users may still have issues using a small percentage of spaces, or joining a meeting or pairing a device with one of these impacted spaces. The engineering team is working to complete the the ongoing data restoration activities, and they are expected to complete within the next 24 hours. However, some space issues will remain after the restoration is complete. There is an issue where the space rosters are incorrect, and that will cause the experience where users receive an error stating that they are no longer part of a space or have errors accessing/using spaces, including 1:1 spaces. This is being repaired, and a code update is in work to address this condition.

Hybrid Services: Call and Calendar hybrid services are operational. A majority of customers are back online and operational. If administrators have any issues with connectors showing stopped/offline or any other data inconsistencies, please review the recovery documentation at https://collaborationhelp.cisco.com/article/WBX9000012687. If Call Connect is in a pending activation state, please restart the Call Connector to restore service.

Webex Teams Meetings and Devices: We also identified two error conditions affecting Webex Teams calls and Space Meetings. A limited number of Webex Teams calls within specific spaces are hitting an error condition and failing. There are also some Space Meetings where One Button to Push isn't operational. A server side fix is in deployment and a client update was released to address these error conditions. The fix is included in the Desktop Client version 3.0.9615.0 for Mac and Windows. A mobile app update is expected to be deployed to the App Stores in the next day. One Button to Push is also reliant on Hybrid Calendar. If there are any issues with the Hybrid Calendar, that will be reflected in the OBTP availability. If OBTP or a Space Meeting is missing or inoperable, please check for the updated client version and verify the availability of the Hybrid Calendar service.

We appreciate your ongoing patience while we address these remaining issues. Updates will be provided as they become available.
Posted 3 months ago. Oct 02, 2018 - 05:35 UTC
Update
Metrics and monitors confirm that the Calendar Service continues to be healthy, and the latency issue has been addressed.
Posted 3 months ago. Oct 01, 2018 - 18:57 UTC
Update
The delays that were impacting the Google and Office 365 users processing @meet and @webex meetings has subsided. Engineering is monitoring the Hybrid Calendar Services, and will provide another update once the monitoring period has completed.
Posted 3 months ago. Oct 01, 2018 - 18:20 UTC
Update
Engineering identified an issues affecting Hybrid Calendar Services. Some Google and Office 365 users may experience delays in processing @meet and @webex meetings.
Exchange users are not affected by this issue.
We are working to address this issue, and will provide updates as they become available.

Engineering is also continuing to work to address the spaces that are inoperable. While a majority of the spaces have been repaired and are fully operational, there are some spaces where users still cannot access or message within the space. There are deployments in flight to address this issue and we are restoring them as we process the deployments. ETAs will be provided as we have them.

Thank you for your continued patience.
Posted 3 months ago. Oct 01, 2018 - 15:54 UTC
Update
Engineering has completed the restoration of the webhook functionality. All Developer APIs and Integrations are operational. The Developer API team will be documenting some recommended checks for developers that utilize our webhooks. Watch https://developer.webex.com/blog-home.html for updates.

The data deploys are continuing as expected, and more and more historical data is available to our users. The deploys are continuous, so users will see the benefit of these ongoing repairs, and data will become available to them as we work through the remaining tasks. We appreciate your patience as we complete these deployments.
Posted 3 months ago. Oct 01, 2018 - 06:01 UTC
Update
The Care Assistant bot repairs have been completed and the Care application is now fully operational.

The remaining webhook repairs are still underway, but webhoooks created prior to the incident should be operational soon.
Posted 3 months ago. Sep 30, 2018 - 04:07 UTC
Update
The Engineering teams continue to complete the tasks to provide access to the remaining spaces and data that have not been restored. These same tasks will also address the webhooks availability. Fixes are moving on a continuous deployment process, so users will see the benefit of these ongoing repairs, and data will become available to them as we work through the remaining tasks. We are being very methodical as we go through the remaining areas of our restorative actions, so while this is taking longer than we would like, we are working diligently to complete each task.

We appreciate your continued patience throughout this process. While these remaining few tasks are going to take a bit more time, be assured that our Engineers are working around the clock to bring this functionality back to the platform.
Posted 3 months ago. Sep 29, 2018 - 16:08 UTC
Update
While the Webex Hybrid Services infrastructure services are operational, a small number of customers are experiencing issues or errors with their Calendar or Call Connectors. The Engineering team has compiled instructions on how to repair the system if an administrator finds themselves in this state.

If at any point, there are questions on the steps, or an administrator would like assistance, please reach out to Cisco Technical Support.

Recovery Procedure for Organizations Experiencing Issues with Hybrid Services after the September Webex Teams Incident: https://collaborationhelp.cisco.com/article/WBX9000012687
Posted 3 months ago. Sep 29, 2018 - 00:29 UTC
Update
Thank you for your continued patience while we've worked to restore the Webex Teams services.

Service restoration efforts have been ongoing, and many of our service deployments have completed. Webex Calling, Webex Control Hub, Webex Hybrid Services, Webex Teams Meetings and Devices, and Context Service have been fully restored. These are continually monitored and are operating as expected.

Webex Teams:
The Webex Teams Messaging services continue to be operational. New messages, spaces and teams are available and fully functional. Users may still have issues using older 1:1 spaces, accessing Teams or spaces with older content, or joining a meeting or pairing a device with one of these spaces. The fix for this is being deployed, and we expect this to be resolved within the next two days.

Webex Developer API:
Developer APIs are operational, with the exception of webhooks. New webhooks are operational; webhooks created prior to the incident are in a non-working state. We have a fix undergoing pre-production validations and will be deploying soon.

Webex Hybrid Services:
Call and Calendar hybrid services are operational. At this time, a majority of customers are back online and operational. If administrators have any issues with connectors showing stopped or offline, or any other data inconsistencies, please reach out to Cisco Technical Support. If Call Connect is in a pending activation state, please restart the Call Connector to restore service.

Webex Teams Meetings and Devices:
Meetings and Devices services are operational. A fix is in flight for a small set of Teams client meeting join use cases that is being planned for deployment. Of note: One Button to Push is reliant on Hybrid Calendar. If there are any issues with the Hybrid Calendar, that will be reflected in the OBTP availability. If OBTP is missing or inoperable, please check the Hybrid Calendar service.

Context Services and Care:
The Care Assistant Bot will be operational once the webhook fix referenced above is addressed. All other areas are operational.

If you or your users are experiencing issues with the service that is outside of those known items listed above, please contact Cisco Technical Support. Options for contacting our support teams can be found at https://collaborationhelp.cisco.com/article/en-us/WBX88620.
Posted 3 months ago. Sep 28, 2018 - 17:38 UTC
Update
Code deploys have completed for the Webex Teams client calls and meetings components. There are a few edge case conditions that may cause call join failures. Impacted users may retry the call. A code fix is being prepped for deployment to address that use case.
Posted 3 months ago. Sep 28, 2018 - 17:31 UTC
Update
Calendar Service deploys have been completed. At this time, a majority of customers are back online and operational. If administrators have any issues with connectors showing stopped or offline, or any other data inconsistencies, please reach out to Cisco Technical Support.

Call Connector Services are still operational. If Call Connect is in a pending activation state, please restart the Call Connector to restore service.
Posted 3 months ago. Sep 28, 2018 - 15:21 UTC
Update
Webex Room Systems are now also operational.

Users that see any connectivity or issues with the Webex Board or the whiteboarding capabilities in any of the Room Systems should complete a device reboot . One Button to Push capabilities are also in an active restoration state and remaining connection will become available as the Calendar Connect service completes its final restoration activities.
Posted 3 months ago. Sep 28, 2018 - 12:01 UTC
Update
Engineering continues to execute on the webhook fix testing and deployment plan. This fix is on schedule, and once deployed, this fix will complete the remaining Dev API and Integration restoration process.

Webhooks created after the initial service interruption are fully operational. Webhooks created prior to that time may experience latency or issues. If Administrators wish to restore services to those impacted webhooks prior to the code deployment, they can redeploy their webooks manually to resolve the issue.
Posted 3 months ago. Sep 28, 2018 - 05:58 UTC
Update
Monitoring for Call Service Connect and Call Service Aware has completed and services are fully operational.
Posted 3 months ago. Sep 28, 2018 - 03:06 UTC
Update
Download, install, authentication and access to the mobile, web, and desktop app is working, as is the search functionality within the client.
Posted 3 months ago. Sep 28, 2018 - 02:30 UTC
Update
Engineering is testing a fix for the webhook issue reported previously. Testing and pre-production deployment and validation is planned over the next few hours, with the goal of deploying to production within the next 24 hours.
Posted 3 months ago. Sep 28, 2018 - 02:10 UTC
Update
Updates will be provided as they become available.
Posted 3 months ago. Sep 28, 2018 - 00:32 UTC
Update
Call Service Connect and Call Service Aware services are restored.
Posted 3 months ago. Sep 28, 2018 - 00:31 UTC
Update
The Care Assistant Bot is operational. In some corner cases, some space creations are delayed. A fix is being created to address this corner case.
Posted 3 months ago. Sep 27, 2018 - 23:49 UTC
Update
Hybrid Call Services continue to be deployed and should be done in the next few hours. Engineering is also working on the completion of the Hybrid Calendar Services to complete the remaining tasks.

Work is also ongoing in the video infrastructure, with additional nodes being added to complete the video calling capacity.
Posted 3 months ago. Sep 27, 2018 - 22:48 UTC
Update
Engineering is continuing to deploy fixes to address the remaining issues with Messaging. Users may experience issues with creating/continuing 1:1 spaces, missing spaces, message flags, and accessing historical data in spaces and teams. Also, we are continuing code and infrastructure deployments as part of full restoration activities so users may see intermittent delays throughout the day.

Complete service restoration for Messaging is an ongoing effort due to the scale of the task and our need to ensure data integrity. We are actively working on the supporting deployments and will share an ETA for those efforts to be completed once one is available.
Posted 3 months ago. Sep 27, 2018 - 20:53 UTC
Update
Calendar Services have been partially restored. Users will start to see the calendaring capabilities available to them.

Please note that there are still additional efforts underway to complete the service availability. Until those are completed users may experience the following when using Calendar Services:
- Delayed or failed meeting processing for @webex / @meet / @spark meetings
- Missing or incorrect One Button To Push and meeting list
- Duplicate space creation for @meet / @spark meetings

The team will continue to work on the Calendar Services to improve the user experience and address the remaining service items. Thanks for your continued patience!
Posted 3 months ago. Sep 27, 2018 - 19:36 UTC
Update
Engineering has identified an issue with the webhooks. New webhooks are functional, but existing webhooks require some additional code deploys. This is in work by the Developer API team.
Posted 3 months ago. Sep 27, 2018 - 19:01 UTC
Update
Calendar Service and Call Service deployments are continuing.

Developer APIs and Webex Teams Integrations are still in a degraded state, as in some cases, the core platform services which power them are being restored. However, most of the functionality of the actual APIs and integrations are operational. We have received reports of some webhooks not firing as expected. If developers are experiencing problems with webhooks, please contact the Webex API support team and we will investigate.
Posted 3 months ago. Sep 27, 2018 - 17:30 UTC
Update
Calendar Service deployments are continuing, and Call Service updates are beginning to be deployed. We know users are very eager for these services to be deployed, and we are working diligently to get them out to all the organizations utilizing these services. These services will continue to be deployed to the orgs over the next few hours, and we will continue to post updates on our progress as they become available.
Posted 3 months ago. Sep 27, 2018 - 16:15 UTC
Update
A fix was deployed for Context Service registration features. This restores all service functionality to Context Services. Thank you for your patience as we restored the functionality.

Calendar Connect services are still being actively deployed and restored to customer organizations.
Posted 3 months ago. Sep 27, 2018 - 14:28 UTC
Update
All Control Hub services have been restored.
Posted 3 months ago. Sep 27, 2018 - 13:46 UTC
Update
Calendar Connect is now being enabled for sites. This will be a multi-step process to enable for all sites, and we will confirm once it's completed for all customers.
Posted 3 months ago. Sep 27, 2018 - 12:36 UTC
Update
The Engineering teams are working on deployment validation and health checks on the Calendar Connect services and data stores. We expect to be able to start enabling connections in the next few hours.

Updates will continue to be provided as we complete our tasks and enable Calendar Connect services.
Posted 3 months ago. Sep 27, 2018 - 11:09 UTC
Update
Data merge processing has completed, but there is still ongoing work to complete the Calendar Connect service deploys. The teams are working to complete some additional engineering tasks, and we expect to be able to deploy in the next few hours.

Updates will continue to be provided as we complete our tasks and enable Calendar Connect services.
Posted 3 months ago. Sep 27, 2018 - 09:59 UTC
Update
Context Service read, write, and search data capabilities and Agent Desktop and Context Gadget is now operational.

Engineering is working on a fix to restore Context Service Registration capabilities.
Posted 3 months ago. Sep 27, 2018 - 08:56 UTC
Update
A recent deploy has restored most Care services. The Care Assistant bot is having intermittent issues with new spaces. A fix has been identified and is pending deployment.
Posted 3 months ago. Sep 27, 2018 - 08:49 UTC
Update
A data merge process is taking additional time to complete, and this data merge is required to complete Calendar Service restoration. We have revisited our ETA and are now targeting 1130 UTC. Updates will be provided as the process continues.
Posted 3 months ago. Sep 27, 2018 - 07:43 UTC
Update
Deployments continue, and teams are still on track to complete the Hybrid Services deployment. Updates will be provided as progress continues.
Posted 3 months ago. Sep 27, 2018 - 06:44 UTC
Update
Database updates are in flight in support of the upcoming Hybrid Service deployment. ETA for Hybrid Services is approximately 0830 UTC.

Updates will be provided as progress continues.
Posted 3 months ago. Sep 27, 2018 - 05:31 UTC
Update
Deployments continue to be executed according to plan. Additional service nodes are being deployed in support of the upcoming Webex Hybrid Services deployments.

Updates and ETAs will be provided as they become available.
Posted 3 months ago. Sep 27, 2018 - 04:25 UTC
Update
Engineering is working on the hybrid services infrastructure and additional node deployments in our primary and secondary infrastructures.

Messaging and Calling services continue to be operational while these deployments are ongoing.
Posted 3 months ago. Sep 27, 2018 - 03:11 UTC
Update
Thanks for checking on the status of Cisco Webex.

We’re so sorry we’re not giving you the experience you deserve but please know we’re doing everything we can to fully restore your service – and your trust.

We’d like to provide some updates in the meantime. The good news? A large part of our Webex Services are restored and we’re making great headway on the rest of it.

Why the delay? Great question because we’re pretty frustrated too.

This process is taking longer than we’d like. But please know we’re 100% committed to restoring the platform’s full functionality in a way that ensures data integrity for you and our hundreds of millions of users around the world. We’re taking a methodical, programmatic approach due to the scale of the task and this is going to take some time to do it right.

Our teams are working around the clock to bring full functionality back to the entire platform. We’ll keep you updated every step of the way via our status page below.

Our goal is to only give you the best experience with our technology. We’re committed to learning from our mistakes and really appreciate your patience as we make sure this never happens again.

Thanks for sticking with us!

Cisco
Posted 3 months ago. Sep 27, 2018 - 01:56 UTC
Update
Directory Connector and User Management from Control Hub is fully operational.
Posted 3 months ago. Sep 27, 2018 - 01:53 UTC
Update
Updating Webex Teams account status, as those functions are fully operational.
Posted 3 months ago. Sep 27, 2018 - 01:47 UTC
Update
Engineering is working on backend support services for Webex Hybrid Services. Deployments and configurations are ongoing as we work to restore services.

Updates on progress will be posted as they become available.
Posted 3 months ago. Sep 27, 2018 - 00:00 UTC
Update
Updates have been deployed and Webex Calling Voicemail services, custom Music on Hold, and Auto Attendant functionalities have been restored.

Deskphone control services will be restored alongside the Webex Hybrid Services.

Service restoration for remaining services is on going. Updates will be provided as they become available.
Posted 3 months ago. Sep 26, 2018 - 22:31 UTC
Update
Updates have been deployed and Developer API services are restoring.

Additional efforts on service restoration for remaining services are on going. Updates will be provided as they become available.
Posted 3 months ago. Sep 26, 2018 - 21:45 UTC
Update
Deployments have completed and Messaging services are stabilizing. Some spaces or files posted during the period of latency may show decryption messages such as 'Decrypting title' or 'Decrypting content'. Users that see this message may restart their Teams client or leave and rejoin the space to allow the space or content to refresh. File downloads that stalled or failed during that time should be restarted.

Additional efforts on service restoration for remaining services are on going. Updates will be provided as they become available.
Posted 3 months ago. Sep 26, 2018 - 19:43 UTC
Update
The first set of deploys have completed and a second set is in flight. User latency with Teams and Devices is still expected until the additional fixes are deployed. Thank you for your patience.
Posted 3 months ago. Sep 26, 2018 - 18:26 UTC
Update
Users are continuing to experience intermittent issues connecting to Teams. Device and whiteboard functionality is also experiencing intermittent drops or errors.

Deployments are still in flight. We will provide an update as those deployments are completed.
Posted 3 months ago. Sep 26, 2018 - 17:22 UTC
Update
Engineering is deploying additional services to address the ongoing Teams connectivity issues. Users are experiencing intermittent access or latency when connecting to Messaging using any of the Teams apps.

Thank you for your patience and we will provide an update as those fixes are deployed.
Posted 3 months ago. Sep 26, 2018 - 16:50 UTC
Update
Service restoration activities are ongoing. Currently, Webex Teams connectivity and room creation is experiencing latency or errors due to additional service onboarding. We appreciate your continued patience as we continue to work on the service recovery efforts.
Posted 3 months ago. Sep 26, 2018 - 15:33 UTC
Update
Webex Calling device activation is now operational.
Posted 3 months ago. Sep 26, 2018 - 14:17 UTC
Update
Updating Webex Desktop Calling to reflect that it is unaffected by this incident. Apologies for any confusion this may have caused.
Posted 3 months ago. Sep 26, 2018 - 13:46 UTC
Update
Engineering continues to restore access to the remaining services.

Core Messaging services are now available with full message and space history being progressively restored. Along with Messaging and space management capabilities, files are available, and avatar updates are functioning.
Message history is available if previously stored on the user’s device, but they are unable to fetch additional message history from our cloud services . The user will see a loading indicator when unavailable message history is requested.
Other known limitations with Messaging includes the ability to flag messages, see user presence, and contact resolution. Some 1:1 spaces are still intermittently experiencing issues.

Device usability and administration is still restoring. Whiteboarding capabilities are available, as is pairing. Most device administration through Control Hub is not yet available, and other services including one button to push and the ability to bind to a space is still being restored.

Thank you for your continued patience.
Posted 3 months ago. Sep 26, 2018 - 12:44 UTC
Update
Engineering has restored service to many of the Cisco Webex Teams Messaging services. At this time, the services are available for users to log in and use the services. As we are continuing restoration activities, some intermittent latency or failures may be experienced when using the platform. Users that experience a failure should retry their operations.

Thank you for your continued patience. As additional services are restored, updates will be published.
Posted 3 months ago. Sep 26, 2018 - 07:50 UTC
Update
While progress has been made on both the Video Platform 2.0 and Webex device services, there are still some intermittent issues being experienced by users connecting on those services. Engineering identified a service fix that can be implemented which will address the latency that is causing the intermittent impact. Engineering is working to deploy that fix, and the teams will continue to monitor those services as the fix is deployed.

The work to restore the broader Webex Teams services is ongoing. A targeted service restoration plan continues to be executed by the team, and Engineering is continuing to work diligently to get the remaining services available.

As updates or estimates for service restoration are available, we will provide them. Thank you for your continued patience while we work to restore the Webex Teams services.
Posted 3 months ago. Sep 26, 2018 - 03:40 UTC
Update
Engineering is continuing to complete service restoration activities. Webex Device service restoration continues to be underway. While services are still stabilizing, many Webex cloud registered devices are connecting to the service and available for use. However, users may still experience intermittent impact when using the devices as we continue to execute the restoration deploys.

As updates or estimates for service restoration are available, we will provide them. Thank you for your continued patience while we restore the Webex Teams services.
Posted 3 months ago. Sep 25, 2018 - 23:37 UTC
Update
Engineering is continuing to complete service restoration activities. Webex Device service restoration is completing. While services are still stabilizing, Webex cloud registered devices are connecting to the service and available for use.

Continued updates will be provided as they become available. Thank you for your continued patience while we restore the Webex Teams services.
Posted 3 months ago. Sep 25, 2018 - 18:05 UTC
Update
Service restoration activities continue to be ongoing. All engineering teams are laser focused to complete their remediation tasks and restore access to services. Meeting services are being restored, and Video Platform 2.0 users may join Webex Meetings from some video devices while we continue to restore the Webex devices. We are working as quickly as possible and apologize for the delays. As updates are available, or any estimates for service restoration are available, we will provide them.
Posted 3 months ago. Sep 25, 2018 - 15:32 UTC
Update
Service restoration activities continue to be ongoing. All engineering teams are laser focused to complete their remediation tasks and restore access to services. We are working as quickly as possible and apologize for the delays. As updates are available, or any estimates for service restoration are available, we will provide them.
Posted 3 months ago. Sep 25, 2018 - 13:18 UTC
Update
Service restoration activities are ongoing. All engineering teams are engaged and continuing to execute remediation tasks to restore access to all services. We are working as quickly as possible to restore services to all the effected segments. As updates are available, or any estimates for service restoration are available, we will provide them.
Posted 3 months ago. Sep 25, 2018 - 10:42 UTC
Update
Service restoration activities are still in flight. Due to the restoration tasks required to address this service incident, there are multiple component updates required, and this is taking additional time to complete. All engineering teams are engaged and continuing to execute remediation tasks to restore access to all services.

We understand our customer's expectations for a high available service, and we apologize for the impact to the Webex services. We are working diligently to address this issue and ensure all services are restored as quickly as possible. Updates will be provided as they become available.
Posted 3 months ago. Sep 25, 2018 - 07:53 UTC
Identified
Engineering is continuing to work to restore services. We apologize for the continued service impacts experienced on the Webex platform. We are working as quickly as possible to restore services to all the effected segments. As updates are available, or any estimates for service restoration are available, we will provide them.

Thank you for your continued patience.
Posted 3 months ago. Sep 25, 2018 - 04:38 UTC
Update
Engineering is continuing to work to restore services to the Webex Teams platform. We know how critical these services are to our users, and we have all teams actively engaged to restore the service.

Users wishing to join a Webex Meeting may join the meeting through the Webex Meetings applications instead of using the Webex Teams or Webex Room Systems.

We appreciate your continued patience. Updates will be provided as they become available.
Posted 3 months ago. Sep 25, 2018 - 02:24 UTC
Investigating
Webex Teams services are currently impacted by an ongoing service outage. Engineering resources are online and working to restore services. We apologize for the impact and all hands are on deck to restore Teams, Meetings, Calling, Care and Context services.

Updates will be provided as they become available.
Posted 3 months ago. Sep 25, 2018 - 01:22 UTC
This incident affected: Webex Teams account (My Webex Teams, Support, Sign in and sign up), Webex Teams Meetings and Devices (Webex Teams client calls and meetings, Webex Room Systems), Webex Teams (Send and read messages, File sharing and viewing, Room creation and management, Search, Web and desktop apps, Integrations), Context Service (Registration, Read, write, and search data, Agent Desktop and Context Gadget), Webex Control Hub (User management, Hybrid service management, Reports, Directory Connector, Device management), Webex Hybrid Services (Calendar Service, Call Service Connect, Call Service Aware), Care (Care Assistant onboarding and administration, Care Assistant bot, Care Desktop, Chat Service, Callback Service), Webex Calling (Desktop phone calls, Media services, Voicemail services, Device activation), and Developer API.