All Systems Operational
Simple Docker ? Operational
Windows Docker Host   ? Operational
Linux Docker Host   ? Operational
Central Proxy   ? Operational
Kubernetes ? Operational
Central Kubernetes   ? Operational
Legacy Kubernetes   Operational
Devspaces Kubernetes   Operational
Containeryard Kubernetes   Operational
Central Registry   ? Operational
Central Storage   ? Operational
Databases ? Operational
SqlServerDB   ? Operational
OracleDB   ? Operational
AuroraDB   ? Operational
PostgresDB   ? Operational
DynamoDB   ? Operational
Central VmWare Operational
Network   ? Operational
Hypervisor   ? Operational
Management   Operational
AWS Interconnection   ? Operational
CICD Operational
Jenkins   ? Operational
Jervis   ? Operational
Electric Flow   ? Operational
TeamCity   ? Operational
SonarQube   ? Operational
Nexus   ? Operational
AWS eu-west-1 Operational
AWS EC2 eu-west-1   Operational
AWS RDS eu-west-1   Operational
AWS S3 eu-west-1   Operational
AWS us-east-1 Operational
AWS EC2 us-east-1   Operational
AWS RDS us-east-1   Operational
AWS S3 us-east-1   Operational
Central Generic   ? Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Scheduled Maintenance
Windows Docker Host Maintenance Window - CENTRAL-89276 Jan 22, 03:00-05:00 UTC
On date January 22, 2019 03:00 GMT, Windows Docker Host will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: January 22, 2019 03:00 GMT
Scheduled End Date: January 22, 2019 05:00 GMT

What is being addressed: Every day we have a period when Docker is very slow, and docker run stuck
List of Customers Affected: KnowledgeMarketing
Expected Downtime: 1hr
Posted on Jan 15, 17:03 UTC
Windows Docker Host Maintenance Window - CENTRAL-89274 Jan 22, 03:00-05:00 UTC
On date January 22, 2019 03:00 GMT, Windows Docker Host will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: January 22, 2019 03:00 GMT
Scheduled End Date: January 22, 2019 05:00 GMT

What is being addressed: Change Description:
Carefully stop containers
Stop Docker Daemon
Reboot the host
Start all containers
Type of Change: Planned
SaaS Environment: Development
Service Type: Internal
Change Reason and Expected Benefits: Every day we have a period when Docker is very slow and docker run stuck
List of Customers Affected: No production containers
Implication of not Proceeding: We still will have slow periods every day
Docker Hosts: dw1.aureacentral.com
Expected Downtime: 1hr
Posted on Jan 15, 16:58 UTC
Network Maintenance Window - VMWC-8531 Jan 31, 04:10-10:00 UTC
On date January 31, 2019 04:10 GMT, Network will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: January 31, 2019 04:10 GMT
Scheduled End Date: January 31, 2019 10:00 GMT

What is being addressed: IBM will perform a Firmware update on their storage arrays connected to our VMware infrastructure from 31-Jan-2019 11:30 pm EST until 31-Jan-2019 05:00 am EST

Impact: No impact. IBM VMware already uses NFS v3 as requested by IBM to avoid impact.

============= Original communication follows ===============
Customer Identification: Aurea Software (1625539)
Start Date: Thursday 31-Jan-2019 04:10 UTC
End Date: Thursday 31-Jan-2019 10:00 UTC
Duration: 5 hours
Event Type: Planned Maintenance
Subject: Event 71995987 - Performance/Endurance Block Storage Services (WDC07) 31-Jan-2019 04:10 UTC
Severity: Medium

=================================================================
/ Event Description /
CUSTOMER ACTION REQUIRED TO AVOID DISRUPTION.

This maintenance may be disruptive to client operations. Adherence to the following directions will help to ensure that there will be no disruption to your services.

On 31 January 2019 at 04:10 UTC, engineers will begin working on a firmware upgrade for Performance/Endurance File Storage in WDC07.

==========
NFS Users:
==========
This maintenance should be non-disruptive to any NFS customers. However, Host systems utilizing NFSv4 or NFSv4.1 are strongly recommended to reconfigure for NFSv3 for increased resilience during maintenance operations.

==========
VMware Users:
==========
Host systems utilizing NFSv4.1 are strongly recommended to reconfigure for NFSv3 for increased resilience during maintenance operations.
Host systems running VMware ESXi utilizing NFSv4.1 MUST discontinue the use of session trunking, if configured. Session trunking is not supported and is known to cause disruptions.


If you have any concerns, please open a support ticket for assistance.

/ Items Associated With This Event /
Item ID | Hostname | Public IP | Private IP | Item Type
45205289 | IBM02SEV1625539_1 | | fsf-wdc0701a-fz.adn.networklayer.com | File Storage
45205309 | IBM02SEV1625539_3 | | fsf-wdc0701b-fz.adn.networklayer.com | File Storage
45301489 | IBM02SEV1625539_5 | | fsf-wdc0701a-fz.adn.networklayer.com | File Storage
45301491 | IBM02SEV1625539_6 | | fsf-wdc0701f-fz.adn.networklayer.com | File Storage
45484957 | IBM02SEV1625539_7 | | fsf-wdc0701b-fz.adn.networklayer.com | File Storage
46061105 | IBM02SEV1625539_14 | | fsf-wdc0701b-fz.adn.networklayer.com | File Storage
46061727 | IBM02SEV1625539_15 | | fsf-wdc0701b-fz.adn.networklayer.com | File Storage
47178815 | IBM02SEV1625539_17 | | fsf-wdc0701b-fz.adn.networklayer.com | File Storage
48429303 | IBM02SEV1625539_18 | | fsf-wdc0701c-fz.adn.networklayer.com | File Storage
48429307 | IBM02SEV1625539_19 | | fsf-wdc0701d-fz.adn.networklayer.com | File Storage
48926171 | IBM02SEV1625539_20 | | fsf-wdc0701b-fz.adn.networklayer.com | File Storage
48932273 | IBM02SEV1625539_21 | | fsf-wdc0701b-fz.adn.networklayer.com | File Storage
49340041 | IBM02SEV1625539_22 | | fsf-wdc0701b-fz.adn.networklayer.com | File Storage
49461827 | IBM02SEV1625539_25 | | fsf-wdc0701b-fz.adn.networklayer.com | File Storage
49462155 | IBM02SEV1625539_26 | | fsf-wdc0701a-fz.adn.networklayer.com | File Storage
50403963 | IBM02SEV1625539_27 | | fsf-wdc0701a-fz.adn.networklayer.com | File Storage
50868207 | IBM02SEV1625539_28 | | fsf-wdc0701d-fz.adn.networklayer.com | File Storage
52610925 | IBM02SEV1625539_29 | | fsf-wdc0701f-fz.adn.networklayer.com | File Storage
52616797 | IBM02SEV1625539_30 | | fsf-wdc0701c-fz.adn.networklayer.com | File Storage
53402477 | IBM02SEV1625539_31 | | fsf-wdc0701d-fz.adn.networklayer.com | File Storage
53574691 | IBM02SEV1625539_32 | | fsf-wdc0701e-fz.adn.networklayer.com | File Storage
54034279 | IBM02SEV1625539_33 | | fsf-wdc0701d-fz.adn.networklayer.com | File Storage
54034411 | IBM02SEV1625539_34 | | fsf-wdc0701c-fz.adn.networklayer.com | File Storage
54042723 | IBM02SEV1625539_35 | | fsf-wdc0701f-fz.adn.networklayer.com | File Storage
54125797 | IBM02SEV1625539_36 | | fsf-wdc0701d-fz.adn.networklayer.com | File Storage
54190903 | IBM02SEV1625539_37 | | fsf-wdc0701c-fz.adn.networklayer.com | File Storage
54537167 | IBM02SEV1625539_38 | | fsf-wdc0701e-fz.adn.networklayer.com | File Storage
54809629 | IBM02SEV1625539_39 | | fsf-wdc0701f-fz.adn.networklayer.com | File Storage
55036977 | IBM02SEV1625539_40 | | fsf-wdc0701e-fz.adn.networklayer.com | File Storage
55047085 | IBM02SEV1625539_41 | | fsf-wdc0701a-fz.adn.networklayer.com | File Storage
55125987 | IBM02SEV1625539_42 | | fsf-wdc0701c-fz.adn.networklayer.com | File Storage
55126123 | IBM02SEV1625539_43 | | fsf-wdc0701e-fz.adn.networklayer.com | File Storage
55158913 | IBM02SEV1625539_44 | | fsf-wdc0701f-fz.adn.networklayer.com | File Storage
55184439 | IBM02SEV1625539_45 | | fsf-wdc0701d-fz.adn.networklayer.com | File Storage
55475634 | IBM02SEV1625539_46 | | fsf-wdc0701e-fz.adn.networklayer.com | File Storage
55475638 | IBM02SEV1625539_47 | | fsf-wdc0701c-fz.adn.networklayer.com | File Storage
55475642 | IBM02SEV1625539_48 | | fsf-wdc0701f-fz.adn.networklayer.com | File Storage
55763651 | IBM02SEV1625539_49 | | fsf-wdc0701d-fz.adn.networklayer.com | File Storage
55777275 | IBM02SEV1625539_50 | | fsf-wdc0701d-fz.adn.networklayer.com | File Storage
55777303 | IBM02SEV1625539_51 | | fsf-wdc0701c-fz.adn.networklayer.com | File Storage
55892025 | IBM02SEV1625539_53 | | fsf-wdc0701f-fz.adn.networklayer.com | File Storage
55893975 | IBM02SEV1625539_54 | | fsf-wdc0701b-fz.adn.networklayer.com | File Storage
55942481 | IBM02SEV1625539_55 | | fsf-wdc0701d-fz.adn.networklayer.com | File Storage
55950705 | IBM02SEV1625539_56 | | fsf-wdc0701c-fz.adn.networklayer.com | File Storage
56803565 | IBM02SEV1625539_57 | | fsf-wdc0701c-fz.adn.networklayer.com | File Storage
56804401 | IBM02SEV1625539_58 | | fsf-wdc0701f-fz.adn.networklayer.com | File Storage
56824709 | IBM02SEV1625539_59 | | fsf-wdc0701b-fz.adn.networklayer.com | File Storage
56824819 | IBM02SEV1625539_60 | | fsf-wdc0701f-fz.adn.networklayer.com | File Storage
57652317 | IBM02SEV1625539_61 | | fsf-wdc0701f-fz.adn.networklayer.com | File Storage
57652423 | IBM02SEV1625539_62 | | fsf-wdc0701e-fz.adn.networklayer.com | File Storage
59016177 | IBM02SEV1625539_63 | | fsf-wdc0701d-fz.adn.networklayer.com | File Storage
59726601 | IBM02SEV1625539_64 | | fsf-wdc0701e-fz.adn.networklayer.com | File Storage

/ Event Updates /


=================================================================This email was generated automatically. Do not reply to this email.
Posted on Jan 17, 03:49 UTC
Past Incidents
Jan 21, 2019

No incidents reported today.

Jan 20, 2019

No incidents reported.

Jan 19, 2019
Network is unavailable - VMWC-8706
Resolved - Outage has been resolved, outage duration was 0:10(Hours:Minutes).
Host is up
Jan 19, 21:40 UTC
Identified - Starting January 19, 2019 21:25 GMT we are experiencing an incident: Network is unavailable. This is affecting restricted set of customers.
We are working on identifying the root cause and fixing it.

We will provide an update every 30m or as soon as incident is resolved.
Jan 19, 21:31 UTC
Management Maintenance Window - VMWC-8435
Completed - The scheduled maintenance has been completed.
Jan 19, 17:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 19, 15:00 UTC
Scheduled - On date January 19, 2019 15:00 GMT, Management will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: January 19, 2019 15:00 GMT
Scheduled End Date: January 19, 2019 17:00 GMT

What is being addressed: To solve some of the current HTML 5 bugs we will upgrade vCenter to 6.5 U2 version

Impact: No impact on running workloads. API and GUI will not be available during the maintenance window.
Jan 14, 18:13 UTC
AuroraDB is unavailable - Aurora1 stop responding and was restarted - CENTRAL-90517
Resolved - Outage has been resolved, outage duration was 0:07(Hours:Minutes).
Aurora1 became unresponsive. Was restarted and outage is over.
Jan 19, 00:32 UTC
Identified - Starting January 19, 2019 00:15 GMT we are experiencing an incident: AuroraDB is unavailable. This is affecting all customers.
We are working on the resolution and expect to have it resolved within the next 5 minutes.

We will provide an update every 30m or as soon as incident is resolved.
Jan 19, 00:25 UTC
AuroraDB AuroraDB is unavailable - Aurora1 stop responding and was restarted - CENTRAL-90516
Resolved - Outage has been resolved, outage duration was 0:07(Hours:Minutes).
all services are up
Jan 19, 00:31 UTC
Update - Problems identified:
Aurora1 became unresponsive.
Had to reboot RDS instance
Estimated duration: 20 minutes.
Jan 19, 00:20 UTC
Identified - Starting January 19, 2019 00:14 GMT we are experiencing an incident: AuroraDB is unavailable. This is affecting restricted set of customers.
We are working on the resolution and expect to have it resolved within the next 20 minutes.

We will provide an update every 30m or as soon as incident is resolved.
Jan 19, 00:17 UTC
Jan 17, 2019
Linux Docker Host Linux Docker Host is unavailable - Crossover is failing to send mails to users - CENTRAL-90003
Resolved - Outage has been resolved, outage duration was 2:52(Hours:Minutes).
All services are back to normal
Jan 17, 19:41 UTC
Update - Problems identified:
Crossover is failing to send mails.
Actions taken:
Team is still checking IP issue and investigation carries on.
Jan 17, 18:06 UTC
Update - Problems identified:
Crossover is failing to send mails.
Actions taken:
There is issue of public IP being blacklisted on email-smtp.us-east-1.amazonaws.com. Team is working
Jan 17, 17:36 UTC
Update - Problems identified:
Crossover is failing to send mails.
Actions taken:
Skype group has been created and investigation started. More details will be provided soon.
Jan 17, 17:06 UTC
Identified - Starting January 17, 2019 16:43 GMT we are experiencing an incident: Linux Docker Host is unavailable. This is affecting restricted set of customers.
We are working on identifying the root cause and fixing it.

We will provide an update every 30m or as soon as incident is resolved.
Jan 17, 16:57 UTC
SqlServerDB is unavailable - SQL server is not responding - CENTRAL-89904
Resolved - Outage has been resolved, outage duration was 0:20(Hours:Minutes).
System performance is back to normal
Jan 17, 13:18 UTC
Identified - Starting January 17, 2019 12:58 GMT we are experiencing an incident: SqlServerDB is unavailable. This is affecting restricted set of customers.
We are working on the resolution and expect to have it resolved within the next 60 minutes.

We will provide an update every 30m or as soon as incident is resolved.
Jan 17, 13:00 UTC
Central Kubernetes is experiencing intermittent failures - DL2 rebooted by Cloudwatch alarm - CENTRAL-89811
Resolved - Outage has been resolved, outage duration was 0:09(Hours:Minutes).
system performance is back to normal
Jan 17, 08:12 UTC
Identified - Starting January 17, 2019 08:02 GMT we are experiencing an incident: Central Kubernetes is experiencing intermittent failures. This is affecting all customers.
We are working on the resolution and expect to have it resolved within the next 30 minutes.

We will provide an update every 30m or as soon as incident is resolved.
Jan 17, 08:03 UTC
Jervis Maintenance Window - CICDOPS-3315
Completed - The scheduled maintenance has been completed.
Jan 17, 06:45 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 17, 06:30 UTC
Scheduled - On date January 17, 2019 06:30 GMT, Jervis will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: January 17, 2019 06:30 GMT
Scheduled End Date: January 17, 2019 06:45 GMT

What is being addressed: We are going to update devfactory (aLine) plugin to use bas-V2
If you have any issues with devfactory plugin, use FIRE jira proejct or contact Simone Gianni

+update multiple other plugins
Jan 15, 20:29 UTC
Jenkins Maintenance Window - CICDOPS-3313
Update - On date 2019-01-18T10:30:00.000+0000, the product will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: 2019-01-18T10:30:00.000+0000
Scheduled End Date: 2019-01-18T10:40:00.000+0000

What is being addressed:
We are going to update devfactory (aLine) plugin to use bas-V2
If you have any issues with devfactory plugin, use FIRE jira proejct or contact Simone Gianni

+update multiple other plugins
Jan 18, 10:11 UTC
Completed - The scheduled maintenance has been completed.
Jan 17, 06:30 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 17, 06:15 UTC
Scheduled - On date January 17, 2019 06:15 GMT, Jenkins will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: January 17, 2019 06:15 GMT
Scheduled End Date: January 17, 2019 06:30 GMT

What is being addressed: We are going to update devfactory (aLine) plugin to use bas-V2
If you have any issues with devfactory plugin, use FIRE jira proejct or contact Simone Gianni

+update multiple other plugins
Jan 15, 20:29 UTC
Jenkins Maintenance Window - CICDOPS-3311
Completed - The scheduled maintenance has been completed.
Jan 17, 06:15 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 17, 06:00 UTC
Scheduled - On date January 17, 2019 06:00 GMT, Jenkins will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: January 17, 2019 06:00 GMT
Scheduled End Date: January 17, 2019 06:15 GMT

What is being addressed: We are going to update devfactory (aLine) plugin to use bas-V2
If you have any issues with devfactory plugin, use FIRE jira proejct or contact Simone Gianni

+update multiple other plugins
Jan 15, 20:28 UTC
Jan 16, 2019
Linux Docker Host is experiencing intermittent failures - Twistlock unavailable on DL2 - CENTRAL-89633
Resolved - Outage has been resolved, outage duration was 0:09(Hours:Minutes).
Service is fine now and outage is over. More details will be provided in RCA.
Jan 16, 16:33 UTC
Identified - Starting January 16, 2019 16:13 GMT we are experiencing an incident: Linux Docker Host is experiencing intermittent failures. This is affecting restricted set of customers.
We are working on the resolution and expect to have it resolved within the next 30 minutes.

We will provide an update every 30m or as soon as incident is resolved.
Jan 16, 16:16 UTC
Network is experiencing intermittent failures - VMWC-8511
Resolved - Outage has been resolved, outage duration was 0:01(Hours:Minutes).
Post analysis.
Jan 16, 14:45 UTC
Identified - Starting January 16, 2019 14:09 GMT we are experiencing an incident: Network is experiencing intermittent failures. This is affecting restricted set of customers.
We are working on the resolution and expect to have it resolved within the next 30 minutes.

We will provide an update every 30m or as soon as incident is resolved.
Jan 16, 14:37 UTC
Legacy Kubernetes Legacy Kubernetes is unavailable - Portworx crashed on all storage nodes - CENTRAL-89399
Resolved - Outage has been resolved, outage duration was 3:38(Hours:Minutes).
All services are up
Jan 16, 10:59 UTC
Update - Problems identified:
infrastructure issue is causing the incident
Actions taken:
there are still few containers that are not working. Team is working on a fix.
Estimated duration: 60 minutes.
Jan 16, 09:41 UTC
Update - Problems identified:
infrastructure issue is causing the incident
Actions taken:
PX team is still working on the issue.
Estimated duration: 60 minutes.
Jan 16, 06:03 UTC
Update - Problems identified:
infrastructure issue is causing the incident
Actions taken:
PX team is trying to revive PX on the last node.
Estimated duration: 60 minutes.
Jan 16, 05:16 UTC
Update - Problems identified:
infrastructure issue is causing the incident
Actions taken:
Team has identified it as a known bug so upgrading the portworx with PX team.
Estimated duration: 60 minutes.
Jan 16, 04:18 UTC
Update - Problems identified:
infrastructure issue is causing the incident
Actions taken:
The storage nodes continues to crash and the team is in call with PX team , trying to fix the issue.
Estimated duration: 60 minutes.
Jan 16, 03:48 UTC
Update - Problems identified:
infrastructure issue is causing the incident
Actions taken:
Team has identified error on the log after few services are up , so still troubleshooting the issue.
Estimated duration: 60 minutes.
Jan 16, 03:02 UTC
Update - Problems identified:
infrastructure issue is causing the incident
Actions taken:
Team has identified that daemons are crashing due to some bugs and working to fix the issue.
Estimated duration: 60 minutes.
Jan 16, 02:28 UTC
Identified - Starting January 16, 2019 00:52 GMT we are experiencing an incident: Legacy Kubernetes is unavailable. This is affecting restricted set of customers.
We are working on the resolution and expect to have it resolved within the next 60 minutes.

We will provide an update every 30m or as soon as incident is resolved.
Jan 16, 00:55 UTC
AWS EC2 us-east-1 CentralDB is experiencing intermittent failures - Aurora-1 slow due to the high number of connections - CENTRAL-89514
Resolved - Outage has been resolved, outage duration was 0:03(Hours:Minutes).
Aurora-1 was slow because high number of connections, 12k
Jan 16, 09:18 UTC
Identified - Starting January 16, 2019 09:15 GMT we are experiencing an incident: CentralDB is experiencing intermittent failures. This is affecting all customers.
We are working on the resolution and expect to have it resolved within the next 10 minutes.

We will provide an update every 30m or as soon as incident is resolved.
Jan 16, 09:17 UTC
AuroraDB is experiencing intermittent failures - Aurora-1 Rebooted - CENTRAL-89473
Resolved - Outage has been resolved, outage duration was 0:05(Hours:Minutes).
system performance is back to normal
Jan 16, 07:22 UTC
Identified - Starting January 16, 2019 06:54 GMT we are experiencing an incident: AuroraDB is experiencing intermittent failures. This is affecting all customers.
We are working on the resolution and expect to have it resolved within the next 10 minutes.

We will provide an update every 30m or as soon as incident is resolved.
Jan 16, 07:00 UTC
AWS EC2 us-east-1 CentralDB is experiencing intermittent failures - Aurora-1 Rebooted - CENTRAL-89474
Resolved - Outage has been resolved, outage duration was 0:05(Hours:Minutes).
all services are up
Jan 16, 07:19 UTC
Identified - Starting January 16, 2019 06:54 GMT we are experiencing an incident: CentralDB is experiencing intermittent failures. This is affecting all customers.
We are working on the resolution and expect to have it resolved within the next 10 minutes.

We will provide an update every 30m or as soon as incident is resolved.
Jan 16, 06:56 UTC
TeamCity Maintenance Window - CICDOPS-3305
Completed - The scheduled maintenance has been completed.
Jan 16, 01:30 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 16, 01:00 UTC
Scheduled - On date January 16, 2019 01:00 GMT, TeamCity will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: January 16, 2019 01:00 GMT
Scheduled End Date: January 16, 2019 01:30 GMT

What is being addressed: As requested by [~prkumar], TeamCity is showing again in New Relic and must be removed. A service restart is required to complete this.
Jan 15, 19:42 UTC
Jan 15, 2019
Legacy Kubernetes is unavailable - Appserver for AlertFind is failing and cannot be launched - CENTRAL-89268
Resolved - Outage has been resolved, outage duration was 6:16(Hours:Minutes).
Volume recovered from backup.
All PX volumes are up and operational.
Jan 15, 22:41 UTC
Identified - Starting January 15, 2019 16:14 GMT we are experiencing an incident: Legacy Kubernetes is unavailable. This is affecting customers.
We are working on identifying the root cause and fixing it.

We will provide an update every 30m or as soon as incident is resolved.
Jan 15, 20:14 UTC
AuroraDB performance degraded - DB is not responsive - CENTRAL-89359
Resolved - Outage has been resolved, outage duration was 0:07(Hours:Minutes).
system performance is back to normal
Jan 15, 21:22 UTC
Identified - Starting January 15, 2019 21:14 GMT we are experiencing an incident: AuroraDB performance degraded. This is affecting restricted set of customers.
We are working on the resolution and expect to have it resolved within the next 30 minutes.

We will provide an update every 30m or as soon as incident is resolved.
Jan 15, 21:21 UTC
AuroraDB is unavailable - CENTRAL-89338
Resolved - Outage has been resolved, outage duration was 0:11(Hours:Minutes).
aurora-1 is responding normal
Jan 15, 21:09 UTC
Identified - Starting January 15, 2019 20:09 GMT we are experiencing an incident: AuroraDB is unavailable. This is affecting restricted set of customers.
We are working on the resolution and expect to have it resolved within the next 5 minutes.

We will provide an update every 30m or as soon as incident is resolved.
Jan 15, 20:16 UTC
AWS EC2 us-east-1 CentralDB is unavailable - Aurora-1 is having issues - CENTRAL-89298
Resolved - Outage has been resolved, outage duration was 0:14(Hours:Minutes).
After failover Aurora-1 is working fine now. More details will be provided in RCA
Jan 15, 18:18 UTC
Identified - Starting January 15, 2019 18:00 GMT we are experiencing an incident: CentralDB is unavailable. This is affecting restricted set of customers.
We are working on the resolution and expect to have it resolved within the next 30 minutes.

We will provide an update every 30m or as soon as incident is resolved.
Jan 15, 18:05 UTC
AWS EC2 us-east-1 CentralDB is experiencing intermittent failures - Sudden increase in DB connections - CENTRAL-89216
Resolved - Outage has been resolved, outage duration was 0:16(Hours:Minutes).
all services are up
Jan 15, 13:04 UTC
Update - Problems identified:
Sudden increase in DB connections
Actions taken:
troubleshooting underlying cause of issue
Estimated duration: 30 minutes.
Jan 15, 12:44 UTC
Identified - Starting January 15, 2019 12:32 GMT we are experiencing an incident: CentralDB is experiencing intermittent failures. This is affecting all customers.
We are working on the resolution and expect to have it resolved within the next 30 minutes.

We will provide an update every 30m or as soon as incident is resolved.
Jan 15, 12:38 UTC
Jan 14, 2019

No incidents reported.

Jan 13, 2019

No incidents reported.

Jan 12, 2019
Linux Docker Host Maintenance Window - CENTRAL-87639
Completed - The scheduled maintenance has been completed.
Jan 12, 05:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 12, 03:01 UTC
Scheduled - On date January 12, 2019 03:00 GMT, Linux Docker Host will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: January 12, 2019 03:00 GMT
Scheduled End Date: January 12, 2019 05:00 GMT

What is being addressed: Upgrade monitoring tool for tracking High System CPU and change SG on all DL hosts

*No downtime expected*
Jan 10, 15:25 UTC
Management Maintenance Window - VMWC-8307
Completed - The scheduled maintenance has been completed.
Jan 12, 05:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 12, 03:00 UTC
Scheduled - On date January 12, 2019 03:00 GMT, Management will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: January 12, 2019 03:00 GMT
Scheduled End Date: January 12, 2019 05:00 GMT

What is being addressed: HTML 5 GUI will not be available for a few minutes due to maintenance work needed to fix the VM tagging issues.

Impact: Users cannot manage workloads (start, stop, create, delete VMs), no impact on running applications.
Jan 10, 08:41 UTC
Linux Docker Host Maintenance Window - CENTRAL-85920
Completed - The scheduled maintenance has been completed.
Jan 12, 05:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 12, 03:00 UTC
Scheduled - On date January 12, 2019 03:00 GMT, Linux Docker Host will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: January 12, 2019 03:00 GMT
Scheduled End Date: January 12, 2019 05:00 GMT

What is being addressed: Upgrade ENA driver and Kernel of DL4
Expected downtime duration: 10-15 minutes
Jan 8, 10:40 UTC
Linux Docker Host Maintenance Window - CENTRAL-85918
Completed - The scheduled maintenance has been completed.
Jan 12, 05:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 12, 03:00 UTC
Scheduled - On date January 12, 2019 03:00 GMT, Linux Docker Host will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: January 12, 2019 03:00 GMT
Scheduled End Date: January 12, 2019 05:00 GMT

What is being addressed: Upgrade ENA driver and Kernel of DL5
Expected downtime duration: 10-15 minutes
Jan 8, 10:40 UTC
AuroraDB Maintenance Window - CENTRAL-86021
Completed - The scheduled maintenance has been completed.
Jan 12, 05:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 12, 03:00 UTC
Scheduled - On date January 12, 2019 03:00 GMT, AuroraDB will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: January 12, 2019 03:00 GMT
Scheduled End Date: January 12, 2019 05:00 GMT

What is being addressed: aws will apply a custom patch to collect diagnosis data in case of instance freeze.
About 6 minutes downtime is expected
Jan 7, 21:04 UTC
Linux Docker Host Maintenance Window - CENTRAL-85911
Completed - The scheduled maintenance has been completed.
Jan 12, 05:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 12, 03:00 UTC
Scheduled - On date January 12, 2019 03:00 GMT, Linux Docker Host will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: January 12, 2019 03:00 GMT
Scheduled End Date: January 12, 2019 05:00 GMT

What is being addressed: Upgrade ENA driver and Kernel of DL1.EU
Expected downtime duration: 10-15 minutes
Jan 7, 18:29 UTC
Linux Docker Host Maintenance Window - CENTRAL-85914
Completed - The scheduled maintenance has been completed.
Jan 12, 05:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 12, 03:00 UTC
Scheduled - On date January 12, 2019 03:00 GMT, Linux Docker Host will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: January 12, 2019 03:00 GMT
Scheduled End Date: January 12, 2019 05:00 GMT

What is being addressed: Upgrade ENA driver and Kernel of DL8
Expected downtime duration: 10-15 minutes
Jan 7, 18:29 UTC
Linux Docker Host Maintenance Window - CENTRAL-85916
Completed - The scheduled maintenance has been completed.
Jan 12, 05:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 12, 03:00 UTC
Scheduled - On date January 12, 2019 03:00 GMT, Linux Docker Host will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: January 12, 2019 03:00 GMT
Scheduled End Date: January 12, 2019 05:00 GMT

What is being addressed: Upgrade ENA driver and Kernel of DL7
Expected downtime duration: 10-15 minutes
Jan 7, 18:28 UTC
Linux Docker Host Maintenance Window - CENTRAL-85922
Completed - The scheduled maintenance has been completed.
Jan 12, 05:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 12, 03:00 UTC
Scheduled - On date January 12, 2019 03:00 GMT, Linux Docker Host will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: January 12, 2019 03:00 GMT
Scheduled End Date: January 12, 2019 05:00 GMT

What is being addressed: Upgrade ENA driver and Kernel of DL2
Expected downtime duration: 10-15 minutes
Jan 7, 18:28 UTC
Linux Docker Host Maintenance Window - CENTRAL-85924
Completed - The scheduled maintenance has been completed.
Jan 12, 05:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 12, 03:00 UTC
Scheduled - On date January 12, 2019 03:00 GMT, Linux Docker Host will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: January 12, 2019 03:00 GMT
Scheduled End Date: January 12, 2019 05:00 GMT

What is being addressed: Upgrade ENA driver and Kernel of DL1
Expected downtime duration: 10-15 minutes
Jan 7, 18:27 UTC
AuroraDB Maintenance Window - CENTRAL-85899
Completed - The scheduled maintenance has been completed.
Jan 12, 05:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 12, 03:00 UTC
Scheduled - On date January 12, 2019 03:00 GMT, AuroraDB will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: January 12, 2019 03:00 GMT
Scheduled End Date: January 12, 2019 05:00 GMT

What is being addressed: aws will optimize the internal aurora storage
Downtime will be during failover (about 3 minutes)
Jan 7, 15:13 UTC
Jan 11, 2019
AWS EC2 us-east-1 CentralDB is unavailable - Aurora-1 got rebooted - CENTRAL-88050
Resolved - Outage has been resolved, outage duration was 0:04(Hours:Minutes).
Aurora-1 is up and outage is over. More details will be provided in RCA
Jan 11, 16:25 UTC
Identified - Starting January 11, 2019 16:21 GMT we are experiencing an incident: CentralDB is unavailable. This is affecting restricted set of customers.
We are working on identifying the root cause and fixing it.

We will provide an update every 30m or as soon as incident is resolved.
Jan 11, 16:23 UTC
AWS EC2 us-east-1 CentralDB performance degraded - Products experiencing slowness - CENTRAL-87888
Resolved - Outage has been resolved, outage duration was 0:25(Hours:Minutes).
Because of the high number of connections some of producst feel extremly slowness
Jan 11, 12:49 UTC
Identified - Starting January 11, 2019 08:50 GMT we are experiencing an incident: CentralDB performance degraded. This is affecting all customers.
We are working on the resolution and expect to have it resolved within the next 30 minutes.

We will provide an update every 30m or as soon as incident is resolved.
Jan 11, 12:37 UTC
Jan 10, 2019
Central Registry Central Registry is experiencing intermittent failures - Registry is down - hosted on legacy k8s - CENTRAL-87694
Resolved - Outage has been resolved, outage duration was 0:56(Hours:Minutes).
system performance is back to normal
Jan 10, 18:43 UTC
Update - Problems identified:
Registry is down
Actions taken:
Team is working to find resolution, looks like token expired issue.
Estimated duration: 30 minutes.
Jan 10, 18:08 UTC
Update - Problems identified:
Registry is down
Actions taken:
Skype group has been created and investigation started. More details will be provided soon.
Estimated duration: 30 minutes.
Jan 10, 17:35 UTC
Identified - Starting January 10, 2019 17:28 GMT we are experiencing an incident: Central Registry is experiencing intermittent failures. This is affecting restricted set of customers.
We are working on the resolution and expect to have it resolved within the next 30 minutes.

We will provide an update every 30m or as soon as incident is resolved.
Jan 10, 17:30 UTC
Central Kubernetes is experiencing intermittent failures - Prods are down - CENTRAL-87701
Resolved - Outage has been resolved, outage duration was 0:35(Hours:Minutes).
Registry available again, details on CENTRAL-87694
Jan 10, 18:41 UTC
Identified - Starting January 10, 2019 17:49 GMT we are experiencing an incident: Central Kubernetes is experiencing intermittent failures. This is affecting restricted set of customers.
We are working on the resolution and expect to have it resolved within the next 30 minutes.

We will provide an update every 30m or as soon as incident is resolved.
Jan 10, 17:54 UTC
SqlServerDB SqlServerDB is unavailable - DB server is not accessible - CENTRAL-87601
Resolved - Outage has been resolved, outage duration was 0:51(Hours:Minutes).
DB is up and Outage is over.
Jan 10, 14:27 UTC
Update - Problems identified:
Database server is not responding
Actions taken:
Troubleshooting underlying cause of issue
Estimated duration: 60 minutes.
Jan 10, 13:58 UTC
Identified - Starting January 10, 2019 13:15 GMT we are experiencing an incident: SqlServerDB is unavailable. This is affecting restricted set of customers.
We are working on the resolution and expect to have it resolved within the next 60 minutes.

We will provide an update every 30m or as soon as incident is resolved.
Jan 10, 13:18 UTC
Jan 9, 2019
Linux Docker Host Maintenance Window - CENTRAL-86217
Completed - The scheduled maintenance has been completed.
Jan 9, 04:01 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 9, 03:00 UTC
Scheduled - On date January 09, 2019 03:00 GMT, Linux Docker Host will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: January 09, 2019 03:00 GMT
Scheduled End Date: January 09, 2019 04:00 GMT

What is being addressed: Upgrade ENA driver and Kernel of DL6
Urgent MW to resolve issues with some products
Expected downtime: 10-15 min
Jan 8, 13:12 UTC
Jan 8, 2019
AuroraDB is unavailable - CENTRAL-86408
Resolved - Outage has been resolved, outage duration was 0:09(Hours:Minutes).
Instance failover and restarted. Aurora1 is working normal again.
Jan 8, 20:37 UTC
Identified - Starting January 08, 2019 20:16 GMT we are experiencing an incident: AuroraDB is unavailable. This is affecting all customers.
We are working on identifying the root cause and fixing it.

We will provide an update every 30m or as soon as incident is resolved.
Jan 8, 20:28 UTC
Network is unavailable - VMWC-8247
Resolved - Outage has been resolved, outage duration was 0:21(Hours:Minutes).
all services are up
Jan 8, 19:11 UTC
Identified - Starting January 08, 2019 18:22 GMT we are experiencing an incident: Network is unavailable. This is affecting restricted set of customers.
We are working on identifying the root cause and fixing it.

We will provide an update every 30m or as soon as incident is resolved.
Jan 8, 18:27 UTC
AuroraDB is unavailable - Aurora-5 is down - CENTRAL-86316
Resolved - Outage has been resolved, outage duration was 0:10(Hours:Minutes).
Aurora-5 is back up and outage is over.
Jan 8, 15:19 UTC
Identified - Starting January 08, 2019 15:14 GMT we are experiencing an incident: AuroraDB is unavailable. This is affecting restricted set of customers.
We are working on identifying the root cause and fixing it.

We will provide an update every 30m or as soon as incident is resolved.
Jan 8, 15:17 UTC
Network Maintenance Window - VMWC-8180
Completed - The scheduled maintenance has been completed.
Jan 8, 05:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 8, 03:00 UTC
Scheduled - On date January 08, 2019 03:00 GMT, Network will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: January 08, 2019 03:00 GMT
Scheduled End Date: January 08, 2019 05:00 GMT

What is being addressed: This Mainteance Window is to replace the NIC card on host32.cons01.central.local host (part of the Development cluster)

Impact: NO impact as the VMs running on this host will be migrated out with vMotion
Jan 7, 08:25 UTC
Linux Docker Host Maintenance Window - CENTRAL-85604
Completed - The scheduled maintenance has been completed.
Jan 8, 04:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 8, 03:01 UTC
Scheduled - On date January 08, 2019 03:00 GMT, Linux Docker Host will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: January 08, 2019 03:00 GMT
Scheduled End Date: January 08, 2019 04:00 GMT

What is being addressed: Upgrade ENA driver and Kernel of DL6
Jan 7, 10:59 UTC
Linux Docker Host Maintenance Window - CENTRAL-85474
Completed - The scheduled maintenance has been completed.
Jan 8, 04:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 8, 03:00 UTC
Scheduled - On date January 08, 2019 03:00 GMT, Linux Docker Host will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: January 08, 2019 03:00 GMT
Scheduled End Date: January 08, 2019 04:00 GMT

What is being addressed: Upgrade ENA driver and Kernel of DL3
Jan 7, 10:37 UTC
Jan 7, 2019

No incidents reported.

← Incident History Powered by Statuspage