Aurea Central Status

All Systems Operational

Simple Docker Operational

Windows Docker Host Operational

Linux Docker Host Operational

Central Proxy Operational

Kubernetes Operational

Central Kubernetes Operational

Legacy Kubernetes Operational

Devspaces Kubernetes Operational

Containeryard Kubernetes Operational

Central Registry Operational

Central Storage Operational

Databases Operational

SqlServerDB Operational

OracleDB Operational

AuroraDB Operational

PostgresDB Operational

DynamoDB Operational

Central VmWare Operational

Network Operational

Hypervisor Operational

Management Operational

AWS Interconnection Operational

CICD Operational

Jenkins Operational

Jervis Operational

Electric Flow Operational

TeamCity Operational

SonarQube Operational

Nexus Operational

AWS eu-west-1 Operational

AWS EC2 eu-west-1 Operational

AWS RDS eu-west-1 Operational

AWS S3 eu-west-1 Operational

AWS us-east-1 Operational

AWS EC2 us-east-1 Operational

AWS RDS us-east-1 Operational

AWS S3 us-east-1 Operational

Central Generic Operational

Operational

Degraded Performance

Partial Outage

Major Outage

Maintenance

Scheduled Maintenance

Linux Docker Host Maintenance Window - CENTRAL-96295 Mar 26, 03:00-05:00 UTC

On date March 26, 2019 03:00 GMT, Linux Docker Host will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: March 26, 2019 03:00 GMT
Scheduled End Date: March 26, 2019 05:00 GMT

What is being addressed: We will update the Twistlock Defenders and Console to the latest version 19.03.307 to eliminate the issue with docker-compose and custom networks
It will also improve the stability of Twistlock Defenders for DL3, DL5, and DL1EU

*Affected hosts*: All Central Linux Docker hosts, except DLB1
*Customer impact*: Access to docker commands will be limited during the update
*Duration*: 5-15 minutes per host + 15 minutes per host on rollback in case of issues
*Impact for products*: +No downtime+ for containers/products expected.

Also, we will enable Warn mode for rewritten Docker Enforcer as Authz plugin on DL6 host.
It contains performance and security improvements, and later we will replace the old version by enabling Kill mode for Authz plugin.
*Affected hosts*: dl6.aureacentral.com
*Customer impact*: None
*Duration*: 2 minutes
*Impact for products*: +No downtime+ for containers/products expected.
Posted on Mar 24, 06:11 UTC

Past Incidents

Mar 24, 2019

AWS EC2 us-east-1 CentralDB is unavailable - CENTRAL-96345

Resolved - Outage has been resolved, outage duration was 0(Hours:Minutes).
Mar 24, 17:58 UTC

Identified - Starting March 24, 2019 17:11 GMT we are experiencing an incident: CentralDB is unavailable. This is affecting customers.
We are working on identifying the root cause and fixing it.

We will provide an update every 30m or as soon as incident is resolved.
Mar 24, 17:50 UTC

Jenkins Maintenance Window - CICDOPS-3758

Completed - The scheduled maintenance has been completed.
Mar 24, 08:57 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 24, 08:40 UTC

Update - On date 2019-03-24T08:40:00.000+0000, the product will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: 2019-03-24T08:40:00.000+0000
Scheduled End Date: 2019-03-24T08:57:00.000+0000

What is being addressed:
We need to address couple of issues pending by updating jenkins plugins:

1. CICDOPS-3725 - https://plugins.jenkins.io/pipeline-model-api

2. CICDOPS-3754 - https://plugins.jenkins.io/basic-branch-build-strategies, https://wiki.jenkins.io/display/JENKINS/Basic+Branch+Build+Strategies+Plugin

3. CICDOPS-3757 - https://plugins.jenkins.io/allure-jenkins-plugin
Mar 24, 08:33 UTC

Scheduled - On date March 26, 2019 04:00 GMT, Jenkins will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: March 26, 2019 04:00 GMT
Scheduled End Date: March 26, 2019 05:00 GMT

What is being addressed: We need to address couple of issues pending by updating jenkins plugins:

1. CICDOPS-3725 - https://plugins.jenkins.io/pipeline-model-api

2. CICDOPS-3754 - https://plugins.jenkins.io/basic-branch-build-strategies, https://wiki.jenkins.io/display/JENKINS/Basic+Branch+Build+Strategies+Plugin

3. CICDOPS-3757 - https://plugins.jenkins.io/allure-jenkins-plugin
Mar 24, 08:32 UTC

Jenkins Maintenance Window - CICDOPS-3769

Completed - The scheduled maintenance has been completed.
Mar 24, 07:30 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 24, 07:20 UTC

Scheduled - On date March 24, 2019 07:20 GMT, Jenkins will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: March 24, 2019 07:20 GMT
Scheduled End Date: March 24, 2019 07:30 GMT

What is being addressed: we are going to upgrade jenkins to latest LTS release along with plugins
+we enable monitoring data export to influxdb
Mar 24, 07:11 UTC

Jervis Maintenance Window - CICDOPS-3767

Completed - The scheduled maintenance has been completed.
Mar 24, 06:47 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 24, 06:40 UTC

Scheduled - On date March 24, 2019 06:40 GMT, Jervis will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: March 24, 2019 06:40 GMT
Scheduled End Date: March 24, 2019 06:47 GMT

What is being addressed: we are upgrading jervis to new lts release along with plugins
devfactory plugin to be updated to try fix CICDOPS-3724
Mar 24, 06:39 UTC

Mar 23, 2019

No incidents reported.

Mar 22, 2019

[CANCELLED] CENTRAL-96141 - Upgrading docker on DL4 and DL8

Completed - upgrading docker daemon to 18.09.3 on DL4 and DL8
Downtimes:
DL4 - expected: 45 minutes; maximum (worst case): 2 hours
DL8 - expected: 30 minutes; maximum (worst case): 2 hours
Mar 22, 14:33 UTC

Scheduled - On date March 23, 2019 03:00 GMT, Linux Docker Host will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: March 23, 2019 03:00 GMT
Scheduled End Date: March 23, 2019 05:00 GMT

What is being addressed: upgrading docker daemon to 18.09.3 on DL4 and DL8
Enable Audit mode for Enforcer on DL6
Downtimes:
DL6 - expected: *zero downtime*
DL4 - expected: 45 minutes; maximum: 2 hours
DL8 - expected: 30 minutes; maximum: 2 hours
Mar 19, 16:19 UTC

Mar 21, 2019

No incidents reported.

Mar 20, 2019

No incidents reported.

Mar 19, 2019

Linux Docker Host Maintenance Window - CENTRAL-96118

Completed - The scheduled maintenance has been completed.
Mar 19, 05:00 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 19, 03:00 UTC

Scheduled - On date March 19, 2019 03:00 GMT, Linux Docker Host will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: March 19, 2019 03:00 GMT
Scheduled End Date: March 19, 2019 05:00 GMT

What is being addressed: During a problem with M1 EMS containers and they wasn't able to connect to NFS server, the mount point was recovered, but the docker containers are stuck with processes s6-sync on D state, are not recoverable and cannot be killed, then to be able to mount again the mountpoints is required to restart the docker host in order this processes releases that mountpoints and we are able again to mount the NFS folder.

The is required to do a restart on dl6 docker host.
Mar 19, 01:52 UTC

Mar 18, 2019

No incidents reported.

Mar 17, 2019

No incidents reported.

Mar 16, 2019

Linux Docker Host Maintenance Window - CENTRAL-95995

Completed - The scheduled maintenance has been completed.
Mar 16, 04:15 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 16, 03:45 UTC

Scheduled - On date March 16, 2019 03:45 GMT, Linux Docker Host will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: March 16, 2019 03:45 GMT
Scheduled End Date: March 16, 2019 04:15 GMT

What is being addressed: Transfer volume from DLB1 to DL1

Change Request CENTRAL-95994
Mar 16, 03:42 UTC

Mar 15, 2019

No incidents reported.

Mar 14, 2019

No incidents reported.

Mar 13, 2019

Jenkins Maintenance Window - CICDOPS-3719

Completed - The scheduled maintenance has been completed.
Mar 13, 06:45 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 13, 06:30 UTC

Update - On date 2019-03-13T06:30:00.000+0000, the product will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: 2019-03-13T06:30:00.000+0000
Scheduled End Date: 2019-03-13T06:45:00.000+0000

What is being addressed:
There will be plugins update during MW to fix buggy blueocean plugin
https://jira.devfactory.com/browse/CICDOPS-3688
Mar 13, 05:59 UTC

Scheduled - On date March 13, 2019 06:00 GMT, Jenkins will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: March 13, 2019 06:00 GMT
Scheduled End Date: March 13, 2019 06:15 GMT

What is being addressed: There will be plugins update during MW to fix buggy blueocean plugin
https://jira.devfactory.com/browse/CICDOPS-3688
Mar 13, 05:47 UTC

Mar 12, 2019

Linux Docker Host Maintenance Window - CENTRAL-95765

Completed - The scheduled maintenance has been completed.
Mar 12, 05:00 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 12, 03:00 UTC

Scheduled - On date March 12, 2019 03:00 GMT, Linux Docker Host will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: March 12, 2019 03:00 GMT
Scheduled End Date: March 12, 2019 05:00 GMT

What is being addressed: Upgrade docker daemon on DL6 to resolve several issues

By request from Security Team (CENTRAL-95762) we will reconfigure Twistlock for all Docker hosts:
- Enable CNNF on Alert Mode
- On Runtime Enable Twistlock General Protection
- Create Collections for every Security Analyst

Expected downtime: Up to 2 hours for containers on DL6 host; no downtime for other Docker hosts
Mar 9, 08:39 UTC

Mar 11, 2019

No incidents reported.

Mar 10, 2019

No incidents reported.

Subscribe to Incident