All Systems Operational
Simple Docker ? Operational
Windows Docker Host ? Operational
Linux Docker Host ? Operational
Central Proxy ? Operational
Kubernetes ? Operational
Central Kubernetes ? Operational
Legacy Kubernetes Operational
Devspaces Kubernetes Operational
Containeryard Kubernetes Operational
Central Registry ? Operational
Central Storage ? Operational
Databases ? Operational
SqlServerDB ? Operational
OracleDB ? Operational
AuroraDB ? Operational
PostgresDB ? Operational
DynamoDB ? Operational
Central VmWare Operational
Network ? Operational
Hypervisor ? Operational
Management Operational
AWS Interconnection ? Operational
CICD Operational
Jenkins ? Operational
Jervis ? Operational
Electric Flow ? Operational
TeamCity ? Operational
SonarQube ? Operational
Nexus ? Operational
AWS eu-west-1 Operational
AWS EC2 eu-west-1 Operational
AWS RDS eu-west-1 Operational
AWS S3 eu-west-1 Operational
AWS us-east-1 Operational
AWS EC2 us-east-1 Operational
AWS RDS us-east-1 Operational
AWS S3 us-east-1 Operational
Central Generic ? Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Scheduled Maintenance
On date March 26, 2019 03:00 GMT, Linux Docker Host will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: March 26, 2019 03:00 GMT
Scheduled End Date: March 26, 2019 05:00 GMT

What is being addressed: We will update the Twistlock Defenders and Console to the latest version 19.03.307 to eliminate the issue with docker-compose and custom networks
It will also improve the stability of Twistlock Defenders for DL3, DL5, and DL1EU

*Affected hosts*: All Central Linux Docker hosts, except DLB1
*Customer impact*: Access to docker commands will be limited during the update
*Duration*: 5-15 minutes per host + 15 minutes per host on rollback in case of issues
*Impact for products*: +No downtime+ for containers/products expected.


Also, we will enable Warn mode for rewritten Docker Enforcer as Authz plugin on DL6 host.
It contains performance and security improvements, and later we will replace the old version by enabling Kill mode for Authz plugin.
*Affected hosts*: dl6.aureacentral.com
*Customer impact*: None
*Duration*: 2 minutes
*Impact for products*: +No downtime+ for containers/products expected.
Posted on Mar 24, 06:11 UTC
Past Incidents
Mar 24, 2019
Resolved - Outage has been resolved, outage duration was 0(Hours:Minutes).
Mar 24, 17:58 UTC
Identified - Starting March 24, 2019 17:11 GMT we are experiencing an incident: CentralDB is unavailable. This is affecting customers.
We are working on identifying the root cause and fixing it.

We will provide an update every 30m or as soon as incident is resolved.
Mar 24, 17:50 UTC
Completed - The scheduled maintenance has been completed.
Mar 24, 08:57 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 24, 08:40 UTC
Update - On date 2019-03-24T08:40:00.000+0000, the product will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: 2019-03-24T08:40:00.000+0000
Scheduled End Date: 2019-03-24T08:57:00.000+0000

What is being addressed:
We need to address couple of issues pending by updating jenkins plugins:

1. CICDOPS-3725 - https://plugins.jenkins.io/pipeline-model-api

2. CICDOPS-3754 - https://plugins.jenkins.io/basic-branch-build-strategies, https://wiki.jenkins.io/display/JENKINS/Basic+Branch+Build+Strategies+Plugin

3. CICDOPS-3757 - https://plugins.jenkins.io/allure-jenkins-plugin
Mar 24, 08:33 UTC
Scheduled - On date March 26, 2019 04:00 GMT, Jenkins will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: March 26, 2019 04:00 GMT
Scheduled End Date: March 26, 2019 05:00 GMT

What is being addressed: We need to address couple of issues pending by updating jenkins plugins:

1. CICDOPS-3725 - https://plugins.jenkins.io/pipeline-model-api

2. CICDOPS-3754 - https://plugins.jenkins.io/basic-branch-build-strategies, https://wiki.jenkins.io/display/JENKINS/Basic+Branch+Build+Strategies+Plugin

3. CICDOPS-3757 - https://plugins.jenkins.io/allure-jenkins-plugin
Mar 24, 08:32 UTC
Completed - The scheduled maintenance has been completed.
Mar 24, 07:30 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 24, 07:20 UTC
Scheduled - On date March 24, 2019 07:20 GMT, Jenkins will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: March 24, 2019 07:20 GMT
Scheduled End Date: March 24, 2019 07:30 GMT

What is being addressed: we are going to upgrade jenkins to latest LTS release along with plugins
+we enable monitoring data export to influxdb
Mar 24, 07:11 UTC
Completed - The scheduled maintenance has been completed.
Mar 24, 06:47 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 24, 06:40 UTC
Scheduled - On date March 24, 2019 06:40 GMT, Jervis will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: March 24, 2019 06:40 GMT
Scheduled End Date: March 24, 2019 06:47 GMT

What is being addressed: we are upgrading jervis to new lts release along with plugins
devfactory plugin to be updated to try fix CICDOPS-3724
Mar 24, 06:39 UTC
Mar 23, 2019

No incidents reported.

Mar 22, 2019
Completed - upgrading docker daemon to 18.09.3 on DL4 and DL8
Downtimes:
DL4 - expected: 45 minutes; maximum (worst case): 2 hours
DL8 - expected: 30 minutes; maximum (worst case): 2 hours
Mar 22, 14:33 UTC
Scheduled - On date March 23, 2019 03:00 GMT, Linux Docker Host will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: March 23, 2019 03:00 GMT
Scheduled End Date: March 23, 2019 05:00 GMT

What is being addressed: upgrading docker daemon to 18.09.3 on DL4 and DL8
Enable Audit mode for Enforcer on DL6
Downtimes:
DL6 - expected: *zero downtime*
DL4 - expected: 45 minutes; maximum: 2 hours
DL8 - expected: 30 minutes; maximum: 2 hours
Mar 19, 16:19 UTC
Mar 21, 2019

No incidents reported.

Mar 20, 2019

No incidents reported.

Mar 19, 2019
Completed - The scheduled maintenance has been completed.
Mar 19, 05:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 19, 03:00 UTC
Scheduled - On date March 19, 2019 03:00 GMT, Linux Docker Host will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: March 19, 2019 03:00 GMT
Scheduled End Date: March 19, 2019 05:00 GMT

What is being addressed: During a problem with M1 EMS containers and they wasn't able to connect to NFS server, the mount point was recovered, but the docker containers are stuck with processes s6-sync on D state, are not recoverable and cannot be killed, then to be able to mount again the mountpoints is required to restart the docker host in order this processes releases that mountpoints and we are able again to mount the NFS folder.

 

The is required to do a restart on dl6 docker host.
Mar 19, 01:52 UTC
Mar 18, 2019

No incidents reported.

Mar 17, 2019

No incidents reported.

Mar 16, 2019
Completed - The scheduled maintenance has been completed.
Mar 16, 04:15 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 16, 03:45 UTC
Scheduled - On date March 16, 2019 03:45 GMT, Linux Docker Host will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: March 16, 2019 03:45 GMT
Scheduled End Date: March 16, 2019 04:15 GMT

What is being addressed: Transfer volume from DLB1 to DL1

Change Request CENTRAL-95994
Mar 16, 03:42 UTC
Mar 15, 2019

No incidents reported.

Mar 14, 2019

No incidents reported.

Mar 13, 2019
Completed - The scheduled maintenance has been completed.
Mar 13, 06:45 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 13, 06:30 UTC
Update - On date 2019-03-13T06:30:00.000+0000, the product will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: 2019-03-13T06:30:00.000+0000
Scheduled End Date: 2019-03-13T06:45:00.000+0000

What is being addressed:
There will be plugins update during MW to fix buggy blueocean plugin
https://jira.devfactory.com/browse/CICDOPS-3688
Mar 13, 05:59 UTC
Scheduled - On date March 13, 2019 06:00 GMT, Jenkins will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: March 13, 2019 06:00 GMT
Scheduled End Date: March 13, 2019 06:15 GMT

What is being addressed: There will be plugins update during MW to fix buggy blueocean plugin
https://jira.devfactory.com/browse/CICDOPS-3688
Mar 13, 05:47 UTC
Mar 12, 2019
Completed - The scheduled maintenance has been completed.
Mar 12, 05:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 12, 03:00 UTC
Scheduled - On date March 12, 2019 03:00 GMT, Linux Docker Host will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: March 12, 2019 03:00 GMT
Scheduled End Date: March 12, 2019 05:00 GMT

What is being addressed: Upgrade docker daemon on DL6 to resolve several issues


By request from Security Team (CENTRAL-95762) we will reconfigure Twistlock for all Docker hosts:
- Enable CNNF on Alert Mode
- On Runtime Enable Twistlock General Protection
- Create Collections for every Security Analyst

 

Expected downtime: Up to 2 hours for containers on DL6 host; no downtime for other Docker hosts
Mar 9, 08:39 UTC
Mar 11, 2019

No incidents reported.

Mar 10, 2019

No incidents reported.