All Systems Operational

Updated a few seconds ago

Scheduled Maintenance

Schedule

July 22, 2024 13:00 - 16:00 UTC

Components

Web Client, Windows Client, Mobile App – Therefore™ Go, Workflow, Smart Capture, eSignature, Reporting Tools, Email Sending, MFP/MEAP App

Locations

SE Asia

Description

The system will be upgraded to the next Therefore™ Online Bugfix Release (v31.1.1).

Schedule

July 23, 2024 13:00 - 16:00 UTC

Components

Web Client, Windows Client, Mobile App – Therefore™ Go, Workflow, Smart Capture, eSignature, Reporting Tools, Email Sending, MFP/MEAP App

Locations

Australia

Description

The system will be upgraded to the next Therefore™ Online Bugfix Release (v31.1.1).

Schedule

July 24, 2024 15:00 - 18:00 UTC

Components

Web Client, Windows Client, Mobile App – Therefore™ Go, Workflow, Smart Capture, eSignature, Reporting Tools, Email Sending, MFP/MEAP App

Locations

Japan

Description

The system will be upgraded to the next Therefore™ Online Bugfix Release (v31.1.1).

Web Client




Operational

Windows Client




Operational

Mobile App – Therefore™ Go




Operational

Workflow




Operational

Smart Capture




Operational

eSignature




Operational

Reporting Tools




Operational

Email Sending




Operational

MFP/MEAP App




Operational

Locations

0

Active Incidents

0

Active Maintenances

1

Days Since Last Incident

History (Last 7 days)

Incident Status

Service Disruption


Components

Web Client, Windows Client, Mobile App – Therefore™ Go, Workflow, Smart Capture, eSignature, Reporting Tools, Email Sending, MFP/MEAP App


Locations

US




July 19, 2024 14:42 UTC
[Resolved] The issue is confirmed to be fully resolved by Microsoft. A post incident report will be shared here as soon as possible.

July 19, 2024 07:45 UTC
[Monitoring] The issue was caused by a problem with Microsoft. This is the current status from their side: --- Service: Azure Purview, Virtual Machines, Backup, Azure Lab Services, Cognitive Services \ Metrics Advisor, VPN Gateway \ Virtual WAN, Azure Cosmos DB, Azure Data Explorer, Event Hubs, Azure Arc enabled Kubernetes, Azure Frontdoor, Data Lake Store, ExpressRoute \ ExpressRoute Circuits, Stream Analytics, Azure Chaos Studio, Machine Learning Services, Cognitive Services \ Computer Vision API, Azure Private Link, Cloud Services, API Management, Azure Database for MariaDB, Windows Virtual Desktop, Network Watcher, Application Insights, Data Lake Analytics, ExpressRoute \ ExpressRoute Gateways, Azure NetApp Files, Event Grid, Azure HPC Cache, Cognitive Services \ Text Analytics API, Azure Databricks, Cognitive Services \ Translator Text API, SQL Database, IoT Hub \ IoT Hub Device Provisioning Service, Site Recovery, Azure SignalR Service, Azure Database Migration Service, Service Fabric, Azure DevTest Labs, Load Balancer, Time Series Insights, Key Vault, Azure Analysis Services, Azure Firewall, Azure IoT Hub, Azure Data Lake Storage Gen1, Cognitive Services \ Language Understanding (LUIS), Notification Hubs, Azure Red Hat OpenShift, Azure Kubernetes Service (AKS), Logic Apps, Data Factory, Azure VMware Solution, Application Gateway, Azure Managed Applications, Network Infrastructure, Service Bus, Virtual Machine Scale Sets, App Service, Azure Bastion, App Configuration, Automation, Cognitive Services \ Face API, Azure Database for MySQL, Cognitive Services \ Content Moderator API, VPN Gateway, Virtual Network, Azure API for FHIR, Azure Migrate, Media Services, Azure Data Share, Batch, AutoScale, Storage, Container Registry, HDInsight, Redis Cache, Azure Firewall Manager, Azure Active Directory Domain Services, Container Instances, Azure Database for PostgreSQL Region: Central US Previous communications incorrectly stated that the issue is mitigated. However, further investigation determined that the issue is still ongoing. The correct and current communications have been updated as below - Impact Statement: Starting at approximately 21:56 UTC on 18 Jul 2024, a subset of customers may experience issues with multiple Azure services in the Central US region including failures with service management operations and connectivity or availability of services. Current Status: We have determined that a backend cluster management workflow deployed a configuration change causing backend access to be blocked between a subset of Azure Storage clusters and compute resources in the Central US region. This resulted in the compute resources automatically restarting when connectivity was lost to virtual disks. We have confirmed mitigation for Azure Storage and Compute resources. We are continuing to investigate and mitigate a number of downstream impacted services that have not returned to a fully healthy state. Status Timeline update: 21:56 UTC on 18 Jul 2024 – Customer impact began 22:13 UTC on 18 Jul 2024 – Storage team started investigating 22:41 UTC on 18 Jul 2024 – Additional Teams engaged to assist investigations 23:27 UTC on 18 Jul 2024 – All deployments in Central US stopped 23:35 UTC on 18 Jul 2024 – All deployments paused for all regions 00:45 UTC on 18 Jul 2024 – A configuration change as the underlying cause was confirmed 01:10 UTC on 19 Jul 2024 – Mitigation started 01:30 UTC on 19 Jul 2024 – Customers started seeing signs of recovery 02:51 UTC on 19 Jul 2024 – 99% of all impacted compute resources recovered 03:23 UTC on 19 Jul 2024 – all Azure Storage clusters confirmed recovery 03:41 UTC on 19 Jul 2024 – Mitigation confirmed for compute resources 06:30 UTC on 19 Jul 2024 - Investigation and mitigate process continuing for a number of downstream impacted services 08:35 UTC on 19 July 2024 – While the majority of dependant services have now recovered, we are still validating the health of some services before sending a mitigation statement for this incident. 10:35 UTC on 19 July 2024 – The majority of services are fully recovered and customers should no longer experience any impact. However we are aware that a subset of App Service and Azure SQL Database customers may still be experiencing residual impact. Mitigation for those customers is ongoing. --- Once we have all the information from Microsoft about the root cause, a post-incident report will be provided.

July 19, 2024 02:45 UTC
[Monitoring] Operational now - resolved by a failover and restarting the servers.

July 18, 2024 22:36 UTC
[Investigating] Cannot login via Native Client, Web Client and Dynamic Web View

Description

The system will be upgraded to the next Therefore™ Online Bugfix Release (v31.1.1).


Components

Web Client, Windows Client, Mobile App – Therefore™ Go, Workflow, Smart Capture, eSignature, Reporting Tools, Email Sending, MFP/MEAP App


Locations

EU


Schedule

July 16, 2024 19:00 - July 16, 2024 22:00 UTC



July 16, 2024 21:44 UTC
[Update] The upgrade is completed. The system is fully operational again.

July 16, 2024 19:09 UTC
[Update] The upgrade started. The system will be unreachable for short periods during the event. We will notify you as soon as the upgrade is completed.

Incident Status

Service Disruption


Components

Web Client, Windows Client, Mobile App – Therefore™ Go, Workflow, Smart Capture, eSignature, Reporting Tools, Email Sending, MFP/MEAP App


Locations

EU




July 16, 2024 09:47 UTC
[Resolved] Our automatic failover mechanisms successfully redirected operations, restoring full functionality. We are currently investigating the root cause to prevent future occurrences.

July 16, 2024 09:33 UTC
[Investigating] We experienced a temporary service interruption due to an unexpected issue within our application.

Incident Status

Partial Service Disruption


Components

Workflow


Locations

EU




July 15, 2024 19:07 UTC
[Resolved] We are pleased to inform you that the issue has been fully resolved. The workaround has been removed, and we have returned to normal operation. Content Connector remained operational throughout the rest of the day.

July 15, 2024 11:32 UTC
[Monitoring] We have been closely monitoring the system and can confirm that the Content Connector has been operational since 08:05 UTC, with a brief interruption between 09:05 and 09:45 UTC. Our team will continue to actively monitor the system throughout the day to ensure its stability and performance. To fully resolve the root cause of the recent issue, we will be performing maintenance on the system this evening after 18:00 UTC. During this time, there may be temporary disruptions or limited availability.

July 15, 2024 08:12 UTC
[Monitoring] We have identified the source of the problem affecting Content Connector processing. A workaround has been implemented to ensure that processing is running again. We are actively working on getting the source of the issue resolved.

July 14, 2024 22:54 UTC
[Investigating] Content Connector encountered issue and stopped. Processing failed.