Fire at our Strasbourg Site


Fire at our Strasbourg Site

Please note, only OVHcloud US customers with Bare Metal servers in Strasbourg, France have been impacted by this incident.

At this moment, all impacted US customers have been contacted individually and offered solutions to address their situation with Bare Metal server(s) hosted in Strasbourg.

If you have not responded to our email yet, please reply ASAP or contact our customer support team via your OVHcloud Manager.

For more details, please see the communications channels below:

All communication channels, including our incident tracking platform, are regularly updated to keep you informed in real time.

If you are an OVHcloud customer with OVHcloud World, please see details here.

If you are an OVHcloud customer with OVHcloud IE, please see details here.

Our mission remains to provide our customers with the highest quality of service to support their online business. We sincerely apologize for any inconveniences related to this situation.

 

 

Roubaix, France | April 2, 2021 - 15:30 (CET)

NOTE: Only customers with OVHcloud US who have Bare Metal servers in Strasbourg, France are being impacted by this incident.

Customer information:

  • SBG1: Recoverable servers are being inspected, cleaned and possibly repaired in Croix. Some have been returned to the Strasbourg site and are being reconnected.
  • SBG1/2: For all non-recoverable servers, equivalent services in other data centers are offered to all customers. We are also supporting them with alternative solutions.
  • SBG3 is operational.
  • SBG4 is operational.

Below is the status of the services located in these data centers:

Data center

Cloud Universe

Services

Progress in the provisioning of services

SBG-1Bare Metal CloudBare Metal25%* Aim to have 100% of services available by April 4.
SBG-1 & SBG-2Bare Metal CloudVPS34%* of VMs are available to customers. Aim to have 95% of services available by April 4.
  Bare Metal24%* of services have been restored.
 Public CloudPublic Cloud Instance80%* of VMs are available to customers.
SBG-3Public CloudPublic Cloud Instance99%*
  Public Cloud Storage95% of services available to and returned to customers in read and write mode.
 Hosted Private CloudHosted Private CloudServices are being gradually restarted around the clock, depending on their initial configuration.
  Veeam Cloud Connect

Veeam Managed Backup
The services are available to customers.
 Bare Metal CloudVPS99%*
  Bare Metal + vRack99%*
SBG-4Bare Metal CloudBare Metal - vRack99%*

 

*The restoration of services is carried out according to a restart schedule room by room, aisle by aisle and rack by rack. However, server cleanup is required, and this will determine when certain racks are put back into to service. Today, the cleaning time for a rack is 7 hours. This schedule is also dependent on the safety protocol defined for on-site interventions, as the safety of our technical teams must be guaranteed.

In addition, the current health restrictions have also affected logistics at our Strasbourg (SBG) site. We have had to source travel, accommodation and catering for all our employees who have come in to assist on site, while respecting government protocols, which of course made the operations more complex.

Communications

Our mission remains to provide our customers with the highest quality of service to support their online business and we know how important this is to them. We sincerely apologize any inconveniences related this situation. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

The next update is scheduled for April 6, 2021.

 

 

Roubaix, France | April 1, 2021 - 15:30 (CET)

NOTE: Only customers with OVHcloud US who have Bare Metal servers in Strasbourg, France are being impacted by this incident.

Customer information:

  • SBG1: The servers in SBG1 are being inspected, cleaned and possibly repaired in Croix. Some have been returned to the Strasbourg site and are being reconnected. The first servers are in service today. Operations will continue for one week. The SBG1 site is being dismantled.
  • SBG2: Equivalent services in other data centers are offered to all customers. We are also supporting them with alternative solutions.
  • SBG3: is operational: To date, 95% of VPS services and 99% of Bare Metal Cloud, Public Cloud Instance and Public Cloud Storage services have been made available to customers. The vRack service has also been restored.
  • There was a malfunction on SBG3's energy equipment today around 6:00 CETam, with no consequence on the service. Our teams put in place the necessary checks and are continuing operations on-site.
  • SBG4: is operational: 100% of Bare Metal servers are available to customers.

Below is the status of the services located in these data centers:

Data center

Cloud Universe

Services

Progress in the provisioning of services

SBG-3Public CloudPublic Cloud Instance95%*
  Public Cloud Storage99% of services available to and returned to customers in read and write mode.
 Hosted Private CloudHosted Private CloudServices are being gradually restarted around the clock, depending on their initial configuration.
  Veeam Cloud Connect

Veeam Managed Backup
The services are available to customers.
 Bare Metal CloudVPS95%*
  Bare Metal + vRack99%*
SBG-4Bare Metal CloudBare Metal - vRack100%*

 

*The restoration of services is carried out according to a restart schedule room by room, aisle by aisle and rack by rack. However, server cleanup is required, and this will determine when certain racks are put back into to service. Today, the cleaning time for a rack is 7 hours, and the length of time is improving every day. This schedule is also dependent on the safety protocol defined for on-site interventions, as the safety of our technical teams must be guaranteed.

In addition, the current health restrictions have also affected logistics at our Strasbourg (SBG) site. We have had to source travel, accommodation and catering for all our employees who have come in to assist on site, while respecting government protocols, which of course made the operations more complex.

Communications

Our mission remains to provide our customers with the highest quality of service to support their online business and we know how important this is to them. We sincerely apologize any inconveniences related this situation. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

 

 

Roubaix, France | March 31, 2021 - 18:30 (CET)

NOTE: Only customers with OVHcloud US who have Bare Metal servers in Strasbourg, France are being impacted by this incident.

Customer information:

  • SBG1: The servers in SBG1 are being inspected, cleaned and possibly repaired in Croix. Some have been returned to the Strasbourg site and are being reconnected. The first racks will be in service between tonight and tomorrow morning. Operations will continue for one week. The SBG1 site is being dismantled.
  • SBG2: Equivalent services in other data centers are offered to all customers. We are also supporting them with alternative solutions.
  • SBG3: is operational: To date, 95% of VPS services and 99% of Bare Metal Cloud, Public Cloud Instance and Public Cloud Storage services have been made available to customers. The vRack service has also been restored.
  • SBG4: is operational: 100% of Bare Metal servers are available to customers.

Below is the status of the services located in these data centers:

Data center

Cloud Universe

Services

Progress in the provisioning of services

SBG-3Public CloudPublic Cloud Instance95%*
  Public Cloud StorageServices available and delivered to customers in read and write mode.
 Hosted Private CloudHosted Private CloudServices are being gradually restarted, depending on their initial configuration, around the clock.
  Veeam Cloud Connect

Veeam Managed Backup
The services are available to customers.
 Bare Metal CloudVPS95%*
  Bare Metal + vRack99%*
SBG-4Bare Metal CloudBare Metal - vRack100%*

 

*The restoration of services is carried out according to a restart schedule room by room, aisle by aisle and rack by rack. However, server cleanup is required and this will determine when certain racks are put back into to service. Today, the cleaning time for a rack is 7 hours, and the length of time is improving every day. This schedule is also dependent on the safety protocol defined for on-site interventions, as the safety of our technical teams must be guaranteed.

In addition, the current health restrictions have also affected logistics at our Strasbourg (SBG) site. We have had to source travel, accommodation and catering for all our employees who have come in to assist on site, while respecting government protocols, which of course made the operations more complex.

SBG4

SBG3

Communications

Our mission remains to provide our customers with the highest quality of service to support their online business. We sincerely apologize for any inconveniences related to this situation. An investigation to determine the cause of the incident is underway. In parallel, OVHcloud is already evaluating technical and operational measures to bring solutions to all affected customers.

All communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

 

 

Roubaix, France | March 30, 2021 - 17:00 (CET)

NOTE: Only customers with OVHcloud US who have Bare Metal servers in Strasbourg, France are being impacted by this incident.

Customer information:

  • SBG1: The recoverable Bare Metal Cloud servers are being cleaned for reinstallation in Strasbourg (SBG3 and SBG4).
  • SBG3: is operational: 95% of VPS services and Bare Metal Cloud services are available customers. The vRack service has also been restored for a large number of SBG3 customers.
  • SBG4: is operational: 100% of Bare Metal servers are available to customers.

Below is the status of the services located in these data centers:

Data center

Cloud Universe

Services

Progress in the provisioning of services

SBG-3Public CloudPublic Cloud Instance91%*
  Public Cloud StorageServices available and delivered to customers in read and write mode.
 Hosted Private CloudHosted Private CloudServices are being gradually restarted, depending on their initial configuration, around the clock.
  Veeam Cloud Connect

Veeam Managed Backup
The services are available to customers.
 Bare Metal CloudVPS95%*
  Bare Metal95%*
SBG-4Bare Metal CloudBare Metal100%*

 

*The restoration of services is carried out according to a restart schedule room by room, aisle by aisle and rack by rack. However, server cleanup is required, and this will determine when certain racks are put back into to service. Today, the cleaning time for a rack is 7 hours, and the length of time is improving every day. This schedule is also dependent on the safety protocol defined for on-site interventions, as the safety of our technical teams must be guaranteed.

In addition, the current health restrictions have also affected logistics at our Strasbourg (SBG) site. We have had to source travel, accommodation and catering for all our employees who have come to assist on site, while respecting government protocols, which of course made the operations more complex.

**As backups are hosted on Public Cloud Storage services, restarting the PCS services will result in many backups being made available.

SBG4

SBG3

Priority 1: Restoring services to our customers

  • Status summary for each data center.

Data center

Situation

Reboot
Electrical/Network

Server restart

SBG-1Data center not reactivated. After inspection by teams with expertise, technical teams can access the rooms. Moving recoverable servers after inspection and cleaning, to our other datacentres: Gravelines (GRA), Roubaix (RBX), Croix (CRX) and Strasbourg (SBG4) to return to service.

Relaunch servers after cleaning beginning April 1.(1)
SBG-2Data center powered down and secured. Recommendation: replacement of infrastructures in other data centers: Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA).
SBG-3Data center operational since March 18.AccomplishedVPS, Public Cloud, Bare Metal Cloud, Hosted Private Cloud, Veeam Cloud Connect and Veeam Managed Backup services have started to be restored.(2)
SBG-4Data center operational.Accomplished100% of Bare Metal customer servers are accessible.

(1) / (2) Additional information on the progress of operations:

  • SBG1: rooms A to D:
    • Servers to be transferred to racks in SBG4 with a scheduled return to service starting March 31.
  • SBG1: rooms 61E and 62E:
    • Cleaning the servers located in these rooms continues in two eight-hour shifts with teams in rotation at our production site in Croix.
    • A total of 1,212 servers have been received in Croix:
      • 800 servers have been cleaned to date, 310 verified and 90 reconditioned to send to our Strasbourg (SBG) site.
      • The cleaned and tested servers will then be gradually returned, already in their rack.
      • Server transfers at our Strasbourg (SBG3) site: 48 hours between on-site transfer and reboot.
    • The services are scheduled to be gradually restarted beginning Apri 1, with the last servers to be restarted between the April 7 and April 8.
  • SBG3:
    • Cleaning all Bare Metal servers is expected to be completed by the evening of March 31.
    • The cleaned and tested servers will then be returned to their rack and reconnected.
    • 347 servers and 59 switches from 5th floor of SBG3 are returning to our Croix site (CRX) on Wednesday for cleaning.
    • The racks will then be transferred to our Strasbourg (SBG) site.
  • SBG Service Status Pages - OVHcloud Control Panel [DOES NOT APPLY TO OVHCLOUD US CUSTOMERS]

A view per data center is available per service in the Control Panel.

controlpanel

For Public Cloud customers who have services in Strasbourg in racks that are being cleaned, and who have not yet recovered their services, a notification has been added to their Control Panel:

controlpanel

Priority 2: Provisioning of infrastructure for our customers in other data centers

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal) in our other data centers in Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), and Frankfurt (FRA).
  • We are committed to providing nearly 15,000 new servers to our customers in the coming week.
  • Due to the current demand, delivery times on our Bare Metal Cloud services may take longer than usual. Our teams are fully mobilized, and we are working hard to deliver to our customers as quickly as possible, especially all impacted customers. Public Cloud, Web Cloud and VPS services are being delivered as normal in all data centers.

Update on quantities delivered since March 10:

CLOUD UNIVERSE

PRODUCTS

 

QUANTITY

Bare metal

Bare metal

Servers

7,817

 

HA-NAS

TB

259

Public Cloud

VPS

VM

17,075

Hosted Private Cloud

Host

Servers

1,059

 

Datastore

Number of zpool

871

 

 

Volume of TB

4,763

 

Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

  • Our Support teams and Account Managers are in contact with our customers and have a support plan in place.

Communications

Our mission remains to provide our customers with the highest quality of service to support their online business. We sincerely apologize for any inconveniences related to this situation. An investigation to determine the cause of the incident is underway. In parallel, OVHcloud is already evaluating technical and operational measures to bring solutions to all affected customers.

All communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

 

 

Roubaix, France | March 29, 2021 - 19:00 (CET)

NOTE: Only customers with OVHcloud US who have Bare Metal servers in Strasbourg, France are being impacted by this incident.

Customer information:

  • SBG1: The recoverable Bare Metal Cloud servers are being cleaned for reinstallation in Strasbourg (SBG3 and SBG4). They will be gradually restarted beginning this week.
  • SBG3: is operational: 90% of VPS services and 85% of Bare Metal Cloud services have been made available to customers again.
  • SBG4: is operational: 100% of Bare Metal servers are available to customers.

Below is the status of the services located in these data centers:

Data center

Cloud Universe

Services

Progress in the provisioning of services

SBG-3Public CloudPublic Cloud Instance90%*
  Public Cloud StorageServices will be returned to customers in read-only mode on the evening of March 29, with write access by March 30.**
 Hosted Private CloudHosted Private CloudServices are being gradually restarted, depending on their initial configuration, around the clock.
  Veeam Cloud Connect

Veeam Managed Backup
The services are available to customers.
 Bare Metal CloudVPS90%*
  Bare Metal85%*
SBG-4Bare Metal CloudBare Metal100%*

 

*The restoration of services is carried out according to a restart schedule room by room, aisle by aisle and rack by rack. However, server cleanup is required, and this will determine when certain racks are put back into service.

Today, the cleaning time for a rack is 7 hours, and our teams are improving every day.

This schedule is also subject to Covid-19 and security protocols, defined for on-site interventions, with the goal of maintaining the health and ensuring the safety of our technical teams.

In addition, the current health restrictions have also affected logistics at our Strasbourg (SBG) site. We have had to source travel, accommodation and catering for all our employees who have come in to assist on site, while respecting government protocols, which of course made the operations more complex.

**As backups are hosted on Public Cloud Storage services, restarting the PCS services will result in many backups being made available.

SBG4

SBG3

Priority 1: Restoring services to our customers

  • Status summary for each data center.

Data center

Situation

Reboot
Electrical/Network

Server restart

SBG-1Data center not reactivated. After inspection by teams with expertise, technical teams can access the rooms. Moving recoverable servers after inspection and cleaning, to our other datacentres: Gravelines (GRA), Roubaix (RBX), Croix (CRX) and Strasbourg (SBG4) to return to service.

After cleaning, the servers will be restarted by the middle of this week.(1)
SBG-2Data center powered down and secured. Recommendation: replacement of infrastructures in other data centers: Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA).
SBG-3Data center operational since March 18.AccomplishedVPS, Public Cloud, Bare Metal Cloud, Hosted Private Cloud, Veeam Cloud Connect and Veeam Managed Backup services have started to be restored.(2)
SBG-4Data center operational.Accomplished100% of Bare Metal customer servers are accessible.

(1) / (2) Additional information on the progress of operations:

  • SBG1: rooms A to D:
    • Servers to be transferred to racks in SBG4 with a scheduled return to service beginning March 30.
  • SBG1: rooms 61E and 62E:
    • Teams are continuing to work in rotation in two eight-hour shifts to clean all servers located in these rooms at our production site in Croix, and will finish by March 30.
    • A total of 1,212 servers have been received in Croix:
      • As of March 29 at 15:00, 805 had been cleaned.
      • 216 have been retouched (power supply and casing changed): 87 have been tested and are operational, 107 are still being tested (the rest are out of order or waiting to be repaired).
      • The cleaned and tested servers will then be gradually returned, already in their rack.
    • Server transfers at our Strasbourg (SBG3) site: 48 hours between on-site transfer and reboot.The services are scheduled to be gradually restarted beginning April 1, with the last servers to be restarted April 7 or 8.
  • SBG3:
    • The cleaning work on all Bare Metal servers is expected to be completed by the evening of March 31.
    • The cleaned and tested servers will then be returned to their rack and reconnected.
    • 347 servers and 59 switches from 5th floor of SBG3 are returning to our Croix site (CRX) on Wednesday for cleaning.
    • The racks will then be transferred to our Strasbourg (SBG) site.
    • The services are scheduled to be gradually restarted between April 1 and April 2.
  • SBG Service Status Pages - OVHcloud Control Panel [DOES NOT APPLY TO OVHCLOUD US CUSTOMERS]

A view per data center is available per service in the Control Panel.

Customer panel

For Public Cloud customers who have services in Strasbourg in racks that are being cleaned, and who have not yet recovered their services, a notification has been added to their Control Panel:

Instance dashboard

Priority 2: Provisioning of infrastructure for our customers in other data centers

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal) in our other data centers in Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), and Frankfurt (FRA).
  • We are committed to providing nearly 15,000 new servers to our customers in the coming week.
  • Due to the current demand, delivery times on our Bare Metal Cloud services may take longer than usual. Our teams are fully mobilized, and we are working hard to deliver to our customers as quickly as possible, especially all impacted customers. Public Cloud, Web Cloud and VPS services are being delivered as normal in all data centers.

Update on quantities delivered since March 10:

CLOUD UNIVERSE

PRODUCTS

 

QUANTITY

Bare metal

Bare metal

Servers

7,597

 

HA-NAS

TB

259

Public Cloud

VPS

VM

16,719

Hosted Private Cloud

Host

Servers

964

 

Datastore

Number of zpool

823

 

 

Volume of TB

4,718

 

Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

  • Our Support teams and Account Managers are in contact with our customers and have a support plan in place.

Communications

Our mission remains to provide our customers with the highest quality of service to support their online business. We sincerely apologize for any inconveniences related to this situation. An investigation to determine the cause of the incident is underway. In parallel, OVHcloud is already evaluating technical and operational measures to bring solutions to all affected customers.

All communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

 

 

Roubaix, France | March 26, 2021 - 17:00 (CET)

NOTE: Only customers with OVHcloud US who have Bare Metal servers in Strasbourg, France are being impacted by this incident.

Customer information:

  • SBG1: Bare Metal recoverable servers are being cleaned to be put back into service in Strasbourg or other data centers by early next week (following inspection and cleaning). Direct communications to impacted US clients are expected to align.
  • SBG3: is operational.
  • SBG4: is operational: 100% of Bare Metal servers are available to customers.

Below is the status of the services located in these data centers:

Data center

Cloud Universe

Services

Progress

SBG-3Public CloudPublic Cloud Instance

Public Cloud Storage
89%*

There will be a gradual restart for all services beginning March 27**.
 Hosted Private CloudHosted Private CloudServices are being gradually restarted depending on their initial configuration on a 24/7 rotation schedule.
  Veeam Cloud ConnectServices are gradually being restarted as of March 25.
 Bare Metal CloudVPS

Bare Metal
83%*

75%*
SBG-4 Bare Metal100%*

 

*The restoration of services is carried out according to a restart schedule room by room, aisle by aisle and rack by rack. However, server cleanup is required, and this will determine when certain racks are put back into to service.

Today, the cleaning time for a rack is 7 hours, and our teams are improving this every day. This schedule is also dependent on the safety protocol defined for on-site interventions, as the safety of our technical teams must be guaranteed.

**As backups are hosted on Public Cloud Storage services, restarting the PCS services will result in many backups being made available.

SGB3

SGB3

SGB4

Priority 1: Restoring services to our customers

  • Status summary for each data center.

Data center

Situation

Electrical restart

Network restart

Server restart

SBG-1Data center not reactivated. After inspection by teams with expertise, technical teams can access the rooms.  Moving recoverable servers after inspection and cleaning, to our other data centers: Gravelines (GRA), Roubaix (RBX), Croix (CRX) and Strasbourg (SBG4). Restart servers after cleaning the week of March 29.
SBG-2Data center powered down and secured.  Recommendation: replacement of infrastructures in other data centers: Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA).
SBG-3Data center operational since March 18.AccomplishedAccomplishedVPS, Public Cloud, Bare Metal Cloud, Hosted Private Cloud, Veeam Cloud Connect and Veeam Managed Backup services have started to be restored.
SBG-4Data center operational.AccomplishedAccomplished100% of Bare Metal customer servers are accessible.

 

  • SBG Service Status Pages - OVHcloud Control Panel [DOES NOT APPLY TO OVHCLOUD US CUSTOMERS]

A view per data center available per service is in the Control Panel.

Customer panel

For Public Cloud customers based in Strasbourg (SBG) who have not yet recovered their services located in racks that are being cleaned, a notification has been added to their Control Panel:

Instance dashboard

Priority 2: Provisioning of infrastructure for our customers in other data centers

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal) in our other data centers in Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), and Frankfurt (FRA).
  • We are committed to providing nearly 15,000 new servers to our customers in the coming week.

Update on quantities delivered since March 10:

CLOUD UNIVERSE

PRODUCTS

 

QUANTITY

Bare metal

Bare metal

Servers

6,987

 

HA-NAS

TB

259

Public Cloud

VPS

VM

16,087

Hosted Private Cloud

Host

Servers

840

 

Datastore

Number of zpool

764

 

 

Volume of TB

4,158

 

Due to the current demand, delivery times on our Bare Metal services may be longer than usual. Our teams are fully mobilized, and we are working hard to deliver to our customers as quickly as possible, especially all impacted customers. Public Cloud, Web Cloud and VPS services are being delivered as normal in all data centers.

Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

  • Our Support teams and Account Managers are in contact with our customers and have a support plan in place.

Communications

Our mission remains to provide our customers with the highest quality of service to support their online business and we know how important this is to them. We sincerely apologize any inconveniences related this situation. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

 

 

Roubaix, France | March 25, 2021 - 16:30 (CET)

NOTE: Only customers with OVHcloud US who have Bare Metal servers in Strasbourg, France are being impacted by this incident.

Customer information:

  • SBG1: Bare Metal recoverable servers are being cleaned to be put back into service in Strasbourg or other data centers by early next week (following inspection and cleaning). Direct communications to impacted US clients are expected to align.
  • SBG3: is operational. Direct communications to impacted US clients went out yesterday.
  • SBG4: is operational: 95% of Bare Metal servers are available to customers.

Below is the status of the services located in these data centers:

Data center

Cloud Universe

Services

Progress

SBG-3Public CloudPublic Cloud Instance

Public Cloud Storage
86%*

Services restarted on March 27.
 Hosted Private CloudHosted Private CloudServices are being gradually restarted depending on their initial configuration on a 24/7 rotation schedule
  Veeam Cloud ConnectServices will be gradually restarted beginning this evening, March 25.
 Bare Metal CloudVPS

Bare Metal
83%*

71%*
SBG-4 Bare Metal95%*

 

*The restoration of services is carried out according to a restart schedule room by room, aisle by aisle and rack by rack. However, server cleanup is required, and this will determine when certain racks are put back into to service. This schedule is also dependent on the safety protocol defined for on-site interventions, as the safety of our technical teams must be guaranteed.

SGB3

SGB3

Priority 1: Restoring services to our customers

  • Status summary for each data center.

Data center

Situation

Electrical restart

Network restart

Server restart

SBG-1Data center not reactivated. After inspection by teams with expertise, technical teams can access the rooms.  Moving recoverable servers after inspection and cleaning to our other data centers: Gravelines (GRA), Roubaix (RBX), Croix (CRX) and Strasbourg (SBG4). Aim to return service the week of March 29.
SBG-2Data center switched off and secured.  Replacement of infrastructure in other data centers: Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA).
SBG-3Data center operational since March 18.AccomplishedAccomplishedVPS, Bare Metal, Public Cloud, and Veeam Cloud Connect services are currently being restored. Hosted Private Cloud services will be restored gradually.
SBG-4Data center operational.AccomplishedAccomplished95% of Bare Metal customer servers are accessible.

 

  • SBG Service Status Pages - OVHcloud Control Panel [DOES NOT APPLY TO OVHCLOUD US CUSTOMERS]

A view per data center available per service is in the Control Panel.

Customer panel

Custolmer panel - Dashboard

Priority 2: Provisioning of infrastructure for our customers in other data centers

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal) in our other data centers in Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), and Frankfurt (FRA).
  • We are committed to providing nearly 15,000 new servers to our customers in the coming week.

Update on quantities delivered since March 10:

CLOUD UNIVERSE

PRODUCTS

 

QUANTITY

Bare metal

Bare metal

Servers

6,812

 

HA-NAS

TB

259

Public Cloud

VPS

VM

15,732

Hosted Private Cloud

Host

Servers

723

 

Datastore

Number of zpool

706

 

 

Volume of TB

4,079

 

Due to the current demand, delivery times on our Bare Metal services may be longer than usual. Our teams are fully mobilized, and we are working hard to deliver to our customers as quickly as possible, especially all impacted customers. Public Cloud, Web Cloud and VPS services are being delivered as normal in all data centers.

Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

  • Our Support teams and Account Managers are in contact with our customers and have a support plan in place.

Communications

Our mission remains to provide our customers with the highest quality of service to support their online business and we know how important this is to them. We sincerely apologize any inconveniences related this situation. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

 

 

Roubaix, France | March 24, 2021 - 18:00 (CET)

NOTE: Only customers with OVHcloud US who have Bare Metal servers in Strasbourg, France are being impacted by this incident.

Customer information:

  • SBG4 is operational.
  • SBG3: is fully operational, below is the status of the services located in this data center:

Data center SBG3

Cloud Universe

Services

Progress

 Public CloudPublic Cloud Instance

Public Cloud Storage
86%*

Restart on March 27
 Hosted Private CloudHosted Private CloudServices are being gradually restarted, depending on their initial configuration, on around the clock, 24/7, rotation schedule.
 Bare Metal CloudVPS

Bare Metal
83%*

71%*

 

*The restoration of services is carried out according to a restart schedule room by room, aisle by aisle and rack by rack. However, server cleanup is required, and this will determine when racks are put back into to service. This schedule is also dependent on the safety protocol defined for on-site interventions, as the safety of our technical teams must be guaranteed.

VMS 1

VMS SGB3

VMS SGB 3

  • Last week, all customers concerned received a communication on the commercial measures that will be applied to their situation. You can find a dedicated section about this in the FAQ.

Priority 1: Restoring services to our customers

  • Status summary for each data center.

Data center

Situation

Electrical restart

Network restart

Server restart

SBG-1Data center not reactivated. After inspection by teams with expertise, technical teams can access the rooms.  Relocation of SBG1 servers to our other data centers: Gravelines (GRA), Roubaix (RBX), Croix (CRX) and Strasbourg (SBG4) to return to service. ETA: Wednesday, March 24.
SBG-2Data center switched off and secured.  Replacement of infrastructure in other data centers: Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA)
SBG-3Data center operational since March 18.AccomplishedAccomplishedVPS, Bare Metal and Public Cloud services are currently being restored. Hosted Private Cloud services will be restored gradually.
SBG-4Data center operational.AccomplishedAccomplishedWednesday, March 24.

 

  • SBG Service Status Pages - OVHcloud Control Panel [DOES NOT APPLY TO OVHCLOUD US CUSTOMERS]

A view per data center available per service is in the Control Panel.

Customer panel

Custolmer panel - Dashboard

Priority 2: Provisioning of infrastructure for our customers in other data centers

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal) in our other data centers in Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), and Frankfurt (FRA).
  • We are committed to providing nearly 15,000 new servers in the coming weeks.

Update on quantities delivered since March 10:

CLOUD UNIVERSE

PRODUCTS

 

QUANTITY

Bare metal

Bare metal

Servers

6,334

 

HA-NAS

TB

259

Public Cloud

VPS

VM

15,425

Hosted Private Cloud

Host

Servers

676

 

Datastore

Number of zpool

677

 

 

Volume of TB

3,811

 

Due to the current demand, delivery times on our Bare Metal services may be longer than usual. Our teams are fully mobilized, and we are working hard to deliver to customers as quickly as possible, especially all impacted customers. Public Cloud, Web Cloud and VPS services are being delivered as normal in all data centers.

Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

  • Our Support teams and Account Managers are in contact with our customers and have a support plan in place.

Communications

Our mission remains to provide our customers with the highest quality of service to support their online business and we know how important this is to them. We sincerely apologize any inconveniences related this situation. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

 

 

Roubaix, France | March 23, 2021 - 16:00 (CET)

NOTE: Only customers with OVHcloud US who have Bare Metal servers in Strasbourg, France are being impacted by this incident.

Customer information:

  • SBG3: is fully operational, below is the status of the services located in this data center:

Data center SBG3

Cloud Universe

Services

Progress

 Public CloudPublic Cloud Instance86%
 Hosted Private CloudHosted Private CloudServices are being gradually restarted based on their initial configuration, and will continue 24/7.
 Bare Metal CloudVPS

Bare Metal
79%

71%

 

The restoration of services is carried out according to a restart schedule room by room, aisle by aisle and rack by rack. The safety of our technical teams must be guaranteed, therefore this schedule may be affected by the safety protocols that have been defined.

VMS 1

VMS SGB3

VMS SGB 3

  • Last week, all customers concerned received a communication on the commercial measures that will be applied to their situation. You can find a dedicated section about this in the FAQ.

Priority 1: Restoring services to our customers

  • Status summary for each data center.

Data center

Situation

Electrical restart

Network restart

Server restart

SBG-1Data center not reactivated. After inspection by the authorities, technical teams can access the rooms. Switched off as a precautionRelocation of SBG1 servers to our other data centers: Gravelines (GRA), Roubaix (RBX), Croix (CRX) and Strasbourg (SBG4) to return to service. ETA: Wednesday, March 24.
SBG-2Data center switched off and secured.  Replacement of infrastructure in other data centers: Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA)
SBG-3Data center operational since March 18.AccomplishedAccomplishedVPS, Bare Metal and Public Cloud services are currently being restored. Hosted Private Cloud services will be restored gradually.
SBG-4The data center will be operational on Wednesday, March 24.Power restored this evening, March 23Network restart March 23ETA: Wednesday, March 24.

 

  • SBG Service Status Pages - OVHcloud Control Panel [DOES NOT APPLY TO OVHCLOUD US CUSTOMERS]

A view per data center available per service is in the Control Panel.

Customer panel

Custolmer panel - Dashboard

Priority 2: Provisioning of infrastructure for our customers in other data centers

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal) in our other data centers in Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), and Frankfurt (FRA).
  • We are committed to providing nearly 15,000 new servers in the coming weeks.
  • Update on quantities delivered since March 10:

CLOUD UNIVERSE

PRODUCTS

 

QUANTITY

Bare metal

Bare metal

Servers

6,103

 

HA-NAS

TB

259

Public Cloud

VPS

VM

14,973

Hosted Private Cloud

Host

Servers

640

 

Datastore

Number of zpool

662

 

 

Volume of TB

3,764

 

Due to the current demand, delivery times on our Bare Metal services may be longer than usual. Our teams are fully mobilized, and we are working hard to deliver to customers as quickly as possible, especially all impacted customers. Public Cloud, Web Cloud and VPS services are being delivered as normal in all data centers.

Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

  • Our Support teams and Account Managers are in contact with our customers and have a support plan in place.

Communications

Our mission remains to provide our customers with the highest quality of service, to support their online activities and we know how important this is to them. We sincerely apologize any inconveniences related this situation. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

 

 

Roubaix, France | March 22, 2021 - 16:30 (CET)

NOTE: Only customers with OVHcloud US who have Bare Metal servers in Strasbourg, France are being impacted by this incident.

Customer information:

Priority 1: Restoring services to our customers:

  • SBG3: is fully operational, below is the status of the services located in this data center:

Data center SBG3

Cloud Universe

Services

Progress

 Public CloudPublic Cloud InstanceProgressive restart of services beginning March 22 and will continue throughout the week and weekend.
 Hosted Private CloudHosted Private CloudProgressive restart of services beginning March 22 and will continue throughout the week and weekend.
 Bare Metal CloudVPS

Bare Metal
79%

55%

 

  • Public Cloud customers have been informed by email of the gradual restart of their services on Monday, March 22.
  • Real-time monitoring of the restart of SBG by room and rack is available at: status.ovh.com/vms/sbg

VMS 1

VMS SGB3

VMS SGB 3

  • Status summary for each data center.

Data center

Situation

Electrical restart

Network restart

Server restart

SBG-1Data center not reactivated. After inspection by the authorities, technical teams can access the rooms. Switched off as a precautionRelocation of SBG1 servers to our other data centers: Gravelines (GRA), Roubaix (RBX), Croix (CRX) and Strasbourg (SBG4) to return to service. ETA: Wednesday, March 24.
SBG-2Data center switched off and secured.  Replacement of infrastructure in other data centers: Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA)
SBG-3Data center operational.AccomplishedAccomplishedFirst racks in service today, Friday March 19.
SBG-4Data center operational.
All floors and servers undamaged.
AccomplishedAccomplished100% of the racks are operational (80 racks).

 

  • SBG Service Status Pages - OVHcloud Control Panel [DOES NOT APPLY TO OVHCLOUD US CUSTOMERS]

A view per data center available per service is in the Control Panel.

Customer panel

Custolmer panel - Dashboard

Priority 2: Provisioning of infrastructure for our customers in other data centers

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal) in our other data centers in Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), and Frankfurt (FRA).
  • We are committed to providing nearly 15,000 new servers in the coming weeks.
  • Update on quantities delivered since March 10:

CLOUD UNIVERSE

PRODUCTS

 

QUANTITY

Bare metal

Bare metal

Servers

5,767

 

HA-NAS

TB

259

Public Cloud

VPS

VM

14,598

Hosted Private Cloud

Host

Servers

616

 

Datastore

Number of zpool

747

 

 

Volume of TB

3,706

 

Due to the current demand, delivery times on our Bare Metal services may be longer than usual. Our teams are fully mobilized, and we are working hard to deliver to customers as quickly as possible, especially all impacted customers. Public Cloud, Web Cloud and VPS services are being delivered as normal in all data centers.

  • Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers
    • Our Support teams and Account Managers are in contact with our customers and have a support plan in place.

Communications

Our mission remains to provide our customers with the highest quality of service, to support their online activities and we know how important this is to them. We sincerely apologize any inconveniences related this situation. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

 

 

Roubaix, France | March 19 2021 - 22:55 (CET)

This evening at 6:50 pm, our teams on site identified smoke in an unconnected battery room at SBG1.

The fire department was immediately contacted. In regards to the situation, OVHcloud's management decided to apply the precautionary principle: the power supply of SBG1 and SBG4 was immediately cut off. Operations on site were temporarily interrupted, all OVHcloud personnel and all partner teams were asked to return home, only the security, fire safety and maintenance personnel remained on site.

Once on site, firefighters identified the source in the battery room, and the incident was contained within minutes. No injuries were reported among the OVHcloud teams or its partners. As a precautionary measure, two security personnel who were affected by the smoke were examined by medical professionals.

The situation is under control and we are adapting the operations schedule for the coming day.

 

 

Roubaix, France | March 19, 2021 - 15:00 (CET)

NOTE: Only customers with OVHcloud US who have Bare Metal servers in Strasbourg, France are being impacted by this incident.

Customer information:

Priority 1: Restoring services to SBG1, SBG3 and SBG4:

1 - Service Status

  • SBG4: the data center is fully operational
  • SBG1: all racks of servers that have been cleaned are operational (60 racks).
  • SBG3: the first racks in SBG3 have been put in service today.
  • Real-time monitoring of SBG restart by room and planned rack available via: status.ovh.com/vms.
  • Status summary for each data center

Data center

Situation

Electrical restart

Network restart

Server restart

SBG-1Data center operational.AccomplishedAccomplishedAll of the servers in the cleaned racks are operational (60 racks).
SBG-2Building not operational.  Replacement of infrastructure in other data centers: Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA)
SBG-3Data center operational.AccomplishedAccomplishedFirst racks in service today, Friday March 19.
SBG-4Data center operational.
All flooers and servers undamaged.
AccomplishedAccomplished100% of the racks are operational (80 racks).

 

2 - Commercial actions

  • At the beginning of the week, all customers concerned received a communication on the commercial measures that have been applied to their situation.

Priority 2: Provisioning of infrastructure for our customers in other data centers

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal) in our other data centers in Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), and Frankfurt (FRA).
  • We are committed to providing nearly 15,000 new servers in the coming weeks.
  • 430 servers will be produced today.
  • Update on quantities delivered since March 10:

CLOUD UNIVERSE

PRODUCTS

 

QUANTITY

Bare metal

Bare metal

Servers

4,844

 

HA-NAS

TB

259

Public Cloud

VPS

VM

13,519

Hosted Private Cloud

Host

Servers

538

 

Datastore

Number of zpool

583

 

 

Volume of TB

3,429

 

Due to the current demand, delivery times on our Bare Metal services may be longer than usual. Our teams are fully mobilized, and we are working hard to deliver to customers as quickly as possible, especially impacted customers. Public Cloud, Web Cloud and VPS services are being delivered as normal in all data centers.

  • Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers
    • Our Support teams and Account Managers are in contact with impacted customers and have set up a support plan for customers who have not yet activated their DRP.

Communications

Our mission remains to provide our customers with the highest quality of service, to support their online activities and we know how important this is to them. We sincerely apologize any inconveniences related this situation. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

 

 

Roubaix, France | March 18, 2021 - 16:30 (CET)

NOTE: Only customers with OVHcloud US who have Bare Metal servers in Strasbourg, France are being impacted by this incident.

Customer information:

1 - Service Status

  • All OVHcloud US clients impacted by this incident have been informed via email communication of their service status.
  • By following the security protocol defined for on-premises operations, the restart will be carried out room by room, aisle by aisle and rack by rack.
  • Real-time monitoring of SBG restart by room and planned rack available via: status.ovh.com/vms.
  • Procedures for restarting services will be communicated to affected customers as details become available in the coming days.

2 - Commercial actions

  • As of this week, all impacted clients were communicated to regarding the commercial measures that will be applied to their situation.

3 - Plan of action

Priority 1: Restoring services to SBG1, SBG3 and SBG4

  • Status summary for each data center

Data center

Situation

Electrical restart

Network restart

Server restart

SBG-1Data center operational.AccomplishedAccomplishedFirst racks tested with success.
Hand off to the customer today with visibility of the status in their OVHcloud Customer Control Panel.
SBG-2Building non-operational.
Site secured.
  Replacement of infrastructure in other data centers: Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA)
SBG-3Data center operational.
Servers undamaged.
AccomplishedAccomplishedTest today on one rack and if conclusive provisional estimate for Friday, March 19 for progressive restart of the services.
SBG-4Data center operational.
All flooers and servers undamaged.
AccomplishedAccomplishedProvisional ETA, Monday, March 22 for gradual restart of services.

 

  • 136 people are mobilized on site today
    • The team rotation schedule for the coming weekend has been established.
  • Cleaning of the site:
    • Cleaning teams are active in SBG1 and SBG3 in all affected areas.
    • Some of the servers in SBG3 require specific cleaning prior to their return to service. We have started this process.

Priority 2: Provisioning of infrastructure for our customers in other data centers

  • For OVHcloud US customers who have been impacted, we have reached out directly to offer assistance with replacement infrastructure (Bare Metal) in our other data centers in Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), and Frankfurt (FRA). Where client business cases allow, we can also deliver replacement infrastructure in our US data centers.
  • We are committed to providing nearly 15,000 new servers in the coming weeks.
  • 400 servers will be produced today.
  • Update on quantities delivered since March 10:

CLOUD UNIVERSE

PRODUCTS

 

QUANTITY

Bare metal

Bare metal

Servers

4519

 

HA-NAS

TB

246

Public Cloud

VPS

VM

12,353

Hosted Private Cloud

Host

Servers

485

 

Datastore

Number of zpool

522

 

 

Volume of TB

3,067

 

Due to the current demand, delivery times on our Bare Metal services may be longer than usual. Our teams are fully mobilized, and we are working hard to deliver to customers as quickly as possible, especially impacted customers. Public Cloud, Web Cloud and VPS services are being delivered as normal in all data centers

  • Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers
    • Our Support teams and Account Managers are in contact with impacted customers and have set up a support plan for customers who have not yet activated their DRP.

Communications

Our mission remains to provide our customers with the highest quality of service, to support their online activities and we know how important this is to them. We sincerely apologize any inconveniences related this situation. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

 

 

Roubaix, France | March 17, 2021 - 16:30 (CET)

NOTE: Only customers with OVHcloud US who have Bare Metal servers in Strasbourg, France are being impacted by this incident.

Customer information:

1 - Service Status

  • Octave Klaba, founder of OVHcloud, has released a new video on the situation in Strasbourg (SBG).
    By following the security protocol defined for on-premises operations: restarting will be carried out room by room, aisle by aisle and rack by rack.
  • Procedures for restarting services will be communicated to affected customers as details become available in the coming days.
  • Real-time monitoring of SBG restart by room and planned rack available via: status.ovh.com/vms.

2 - Commercial actions

  • As of this week, all impacted clients were communicated to regarding the commercial measures that will be applied to their situation.

3 - Plan of action

Priority 1: Restoring services to SBG1, SBG3 and SBG4

  • Status summary for each data center

Data center

Situation

Electrical restart

Network restart

Server restart

SBG-1Four of 12 rooms were damaged
- Rooms 61AD/62AD: servers not affected
- Rooms 61E/62D: cleaning of servers underway
- Restored power to data center- Backbone reconnected
- Redeployment of the local network (Cabling finalized Wednesday, March 17) and configuration today
A rack was tested today (March 17) and if conclusive a provisional estimate of Friday, March 19 for a gradual restart of services after cleaning and drying servers.
SBG-2Building out of use. Structural review carried out, site secured.  Infrastructure replacement in other data centers Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA)
SBG-3Servers undamaged- Restored power to data center- New network power restored
- Redeployment of the local network this Wednesday March 17 (cabling finalized).
Test tomorrow on a rack and if conclusive a provisional estimate of Friday, March 19 for a gradual restart of services.
SBG-4All floors and servers were spared- Restored power to data centerRedeployment of the local network carried out.Provisional ETA, Monday, March 22 for gradual restart of services.

 

  • 116 people are mobilized on site today
  • Network redeployment:
    • The SBG1, SBG3 and SBG4 data centrer will be connected to the network tonight.
  • Water-cooling:
    • The entire system is operational in the SBG1/SBG3/SBG4 data centers.
  • Cleaning of the site:
    • Intensive mobilization (24/7).
    • SBG1: Cleaning completed Wednesday, March 17.

Priority 2: Providing infrastructures in other data centers for our impacted customers

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal) in our other data centers in Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), and Frankfurt (FRA).
  • We are committed to providing nearly 15,000 new servers in the coming weeks.
  • 400 servers will be produced today.
  • Update on quantities delivered since March 10:

CLOUD UNIVERSE

PRODUCTS

 

QUANTITY

Bare metal

Bare metal

Servers

4200

 

HA-NAS

TB

276

Public Cloud

VPS

VM

11,998

Hosted Private Cloud

Host

Servers

434

 

Datastore

Number of zpool

466

 

 

Volume of TB

2,950

 

Due to the current demand, delivery times on Bare Metal services may be longer than usual. Our teams are fully mobilized, and we are working to deliver to customers as quickly as possible, especially all impacted customers. On the other hand, Public Cloud, Web Cloud and VPS services are being delivered as normal in all data centers.

  • Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers
    • Our Support teams and Account Managers are in contact with our customers and have set up a support plan for customers who have not yet activated their DRP.

Communications

Our mission remains to provide our customers with the highest quality of service, to support their online business and we know how important this is to them. We sincerely apologize any inconveniences related this situation. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

 

 

Roubaix, France | March 16, 2021 - 17:00 (CET)

NOTE: Only customers with OVHcloud US who have Bare Metal servers in Strasbourg, France are being impacted by this incident.

Customer information:

1 - Service Status

  • Finalized procedures for restarting services will soon be communicated to the impacted customers.

2 - Commercial actions

  • Commercial actions have been indicated and the details will be communicated to the impacted customers this week.

3 - Plan of action

Priority 1: Restoring services to SBG1, SBG3 and SBG4

  • Status summary for each data center

Data center

Situation

Electrical restart

Network restart

Server restart

SBG-1Four of 12 rooms were damaged
- Rooms 61AD/62AD: servers not affected
- Rooms 61E/62D: cleaning of servers underway
- Power restored
- Damaged UPS have been checked and are being replaced
- Backbone: temporary connection to SBG on March 15
- LAN redeployment and configuration today
Provisional ETA: Monday March 22 for gradual restart
SBG-2Building out of use. Structural review carried out, site secured.  Infrastructure replacement in other data centers Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA)
SBG-3Servers undamaged- Power restored today
- All inverters are operational
- New network room powered up today
- Redeployment of the local network on March 17
Provisional ETA: Monday March 22 for gradual restart
SBG-4All floors and servers were sparedPower restored todayRedeployment of the local network todayProvisional ETA: Monday March 22 for gradual restart

 

  • 101 people are mobilized and on-site
  • Network redeployment:
    • SBG connection to the backbone today (alternative routes to Paris and Frankfurt underway)
    • Reconnection of buildings and rooms:
      • Fiber optic redeployment in progress
      • Configuration underway
    • The high-voltage power supply for SBG3 is deployed and operational.
    • Inverter recommissioning protocol underway today (verification of all UPS).
  • Water-cooling:
    • The entire system is operational
  • Cleaning of the site:
    • Intensive mobilization (24/7).

Priority 2: Provisioning of infrastructure for our customers in other data centers

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal) in our other data centers: Gravelines (GRA), Roubaix (RBX), Londres (LON), Warsaw (WAW), Frankfurt (FRA).
  • We are committed to providing around 15,000 new servers in the coming weeks.
  • 350 servers will be produced today.
  • Update on quantities delivered since March 10:

CLOUD UNIVERSE

PRODUCTS

 

QUANTITY

Bare metal

Bare metal

Servers

3939

 

HA-NAS

TB

242

Public Cloud

VPS

VM

10731

Hosted Private Cloud

Host

Servers

372

 

Datastore

Number of zpool

393

 

 

Volume of TB

2704

 

  • Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with impacted customers
    • Our Support teams and Account Managers are in contact with our customers and have set up a support plan for customers who have not yet activated their DRP.

Communications

Our mission remains to provide our customers with the highest quality of service to support their online activities, and we know how important this is to them. We sincerely apologize for any problems caused by this incident. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

 

 

Roubaix, France | March 15, 2021 - 17:00 (CET)

NOTE: Only customers with OVHcloud US who have Bare Metal servers in Strasbourg, France are being impacted by this incident.

Customer information:

1 - Server Status

  • Impacted customers have been contacted individually and provided server information details.

2 - Commercial actions

  • We will notify all impacted customers by email regarding eligible compensation credits Monday, March 15, 2021.

3 - Action plan

Priority 1: Restoring services to SBG1, SBG3 and SBG4

  • Status summary for each data center

Data center

Situation

Electrical restart

Network restart

Server restart

SBG-1Four of 12 rooms were damagedTemporarily repowered on March 13 and will be permanently restored on March 16- Backbone: Temporary connection to SBG on March 15
- Internal network to be redeployed on March 16
Provisional ETA: Monday March 22 for gradual restart
SBG-2Building out of use  Replacing infrastructures in other data centers
SBG-3Servers undamagedTemporarily repowered on March 12 and will be restored permanently on March 16- New network room to be deployed and powered on March 16
- Internal network to be redeployed on March 17
Provisional ETA: Monday March 22 for gradual restart
SBG-4Servers undamagedPower will return permanently on March 16Internal network to be redeployed on March 16Provisional ETA: Monday March 22 for gradual restart

 

  • 111 people are on-site and dedicated to connecting network and electricity units (in rotation, 24/7), for:
    • Power redeployment:
      • The electrical installations required to restart SBG1/SBG4 have been verified.
      • The high-voltage power supply for SBG3 is deployed and operational.
      • Inverter recommissioning protocol underway today (verification of all UPS).
    • Network redeployment:
      • The SBG1 network room is being tested.
      • The new network room is being completed and will be supplied with electricity as soon as the inverters are verified.
      • The installation of the network equipment will be finalized this evening.
      • The connection of the backbone is scheduled for tomorrow, March 16.

The local network is being redeployed.

  • The cleaning of any soot in the SBG3 server rooms will begin on Tuesday, March 16.

Priority 2: Providing infrastructures in other data centers for our affected customers

  • For customers who have been impacted, we are offering replacement infrastructures in other data centers.
  • We are committed to providing around 15,000 new servers in the coming weeks.
  • 500 servers will be produced today.
  • Update on quantities delivered since March 10

CLOUD UNIVERSE

PRODUCTS

 

QUANTITY

Bare metal

Bare metal

Servers

3658

 

HA-NAS

TB

220

Public Cloud

VPS

VM

8903

Hosted Private Cloud

Host

Servers

328

 

Datastore

Number of zpool

361

 

 

Volume of TB

2598

 

  • Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with impacted customers
    • Our Support teams and Technical Account Managers are in contact with our customers and have set up a support plan for those who have not yet activated their DRP.

Communications

Our mission remains to provide our customers with the highest quality of service to support their online activities, and we know how important this is to them. We sincerely apologize for the problems caused by this incident. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

 

 

Roubaix, France | March 14, 2021 - 15:00 (CET)

NOTE: Only customers with OVHcloud US who have Bare Metal servers in Strasbourg, France are being impacted by this incident.

Customer information:

1 - Server Status

  • Impacted customers have been contacted individually and provided server information details.

2 - Commercial actions

  • All affected customers will be notified about their updated March invoice in the coming days.

3 - Action plan

Priority 1: Restoring services to SBG1, SBG3 and SBG4

  • Status summary for each data center

Data center

Situation

Electrical restart

Network restart

Server restart

SBG-1Four of 12 rooms were damagedTemporarily repowered on March 13 and will be permanently restored on March 17- Backbone: Temporary connection to SBG on March 15
- Internal network to be redeployed on March 17
Provisional ETA: Monday March 22 for gradual restart
SBG-2Building out of use  Replacing infrastructures in other data centers
SBG-3Servers undamagedTemporarily repowered on March 12 and will be restored permanently on March 16A progressive restart of all services is estimated for Monday, March 22A progressive restart of all services is estimated for Monday, March 22
SBG-4Servers undamagedPower will return permanently on March 17Internal network to be redeployed on March 17Provisional ETA: Monday March 22 for gradual restart

 

  • A team of 60 people is working on site to connect mobile network and electricity units (working in shifts, 24 hours a day):
    • Cabling in progress.
    • Connection planned for tonight.
  • Drying out the SBG3 buildings will be finished tonight.
  • Cleaning the soot in the SBG3 server rooms will begin on Tuesday, March 16.
  • The cooling system is operational:
    • Water cooling is protected from the cold weather (forecast locally on Thursday, March 18 and Friday, March 19).
    • The pumps are operational.
  • Priority 2: Providing infrastructures in other data centers for our affected customers
    • For customers who have been impacted, we are offering replacement infrastructures in other data centers.
    • We are committed to providing around 15,000 new servers in the coming weeks.
    • Update on quantities delivered since March 10

Update on quantities delivered

  • Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with impacted customers
    • Our Support teams and Account Managers are in contact with impacted customers and have set up a support plan for customers who have not yet activated their DRP.

Communications

Our mission remains to provide our customers with the highest quality of service, to support their online activities. And we know how important this is to them. We sincerely apologize for the problems caused by this fire. We will continue to communicate with the utmost transparency about the cause of the fire and its consequences.

An investigation is ongoing. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

 

 

Roubaix, France | March 13, 2021 - 16:00 (CET)

NOTE: Only customers with OVHcloud US who have Bare Metal servers in Strasbourg, France are being impacted by this incident.

Customer information:

1 - Server Status

  • We have sent emails to all impacted customers informing them of the status of their services.

2 - Commercial actions

  • Billing has been suspended for Strasbourg (SBG) data center customers from the date of the incident.
  • Commercial measures are in place for each affected customer.
  • Details on how these measures will be implemented will be sent next week and will apply to March bills for impacted services that were already issued before March 10.

Our three priorities are as follows:

  • Priority 1: Restoring services to SBG1, SBG3, and SBG4
  • Priority 2: Providing infrastructures in other data centers for our affected customers
  • Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

Action plan:

Priority #1:

  • Status summary for each data center

Data center

Situation

Electrical restart

Server restart

SBG-1Four of 12 rooms were damagedScheduled for Monday, March 15Progressive restoration of undamaged servers by Monday, March 22
SBG-2Building out of useAudit and inventory of premisesReplacing infrastructures in other data centers
SBG-3Servers undamagedTests for the high voltage supply will be carried out this weekend. Provisional restart on Monday, March 15A progressive restart of all services is estimated for Monday, March 22
SBG-4Servers undamagedScheduled for Monday, March 15A progressive restart of all services is estimated for Monday, March 22

 

  • Mobile high-voltage network and electricity units arrived at the Strasbourg site today at 12pm.
  • 60 experts are currently working on site (in rotation over the weekend) in four teams, with the following missions:
    1. Team 1 (network) is tasked with connecting the mobile network unit and testing the network room in SBG1,
    2. Team 2 (electricity) is working on high-voltage electricity connections,
    3. Team 3 (electricity) is working on low-voltage electricity connections,
    4. Team 4 is working on protecting and restoring the cooling systems.

The goal is to reconnect, repower and restart the backbone access network from Monday March 15 following the authorization of the ESR (Électricité de Strasbourg Réseaux).

To repower the site:

  • Two generators have been rented beginning March 11 to power the temporary office.
  • One additional generator was rented to recharge the batteries of our generators following their activation during the incident.
  • Several service providers are currently working on site (30 people):
    • The water on the floor of the SBG3 building has been cleaned up,
    • The SBG3 building is currently being dried out, which should be completed by Sunday evening, March 14,
    • The cleaning of any soot in the server rooms of SBG3 will begin on Monday, March 15.
  • Repair of the damaged cooling system:
    • A crane has arrived on site.

Priority #2:

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our Roubaix (RBX) and Gravelines (GRA) data centers.
  • We are committed to providing around 15,000 new servers in the coming weeks.
  • Update on quantities delivered since March 10:

Update on quantities delivered

Priority #3:

  • Implementing all DRP for impacted customers.

Communications

Our mission remains to provide our customers with the highest quality of service, to support their online activities. And we know how important this is to them. We sincerely apologize for the problems caused by this fire. We will continue to communicate with the utmost transparency about the cause of the fire and its consequences.

An investigation is ongoing. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

 

 

Roubaix, France | March 12, 2021 – 14:00 (CET)

Customer Information:

  • Octave Klaba, founder of OVHcloud, has uploaded videos providing information about the situation.
  • Today, we will continue to coordinate with impacted customers by email about the status of their services. We also have a 24/7 support team in place to support them.

As a reminder, our three priorities are as follows:

  • Priority 1: Restoring services to SBG1, SBG3, and SBG4
  • Priority 2: Providing impacted customers with infrastructures in our non-affected data centers
  • Priority 3: Implementating all DRP (Disaster Recovery Plan) mechanisms with impacted customers

Priority 1: Restoring services to SBG1, SBG3 and SBG4

  • Diagnostics and inventory are ongoing.
  • To guarantee the safety of our teams, cleaning and repairs to the damaged buildings are in progress.
  • The tasks of cleaning up the fire extinguishing foam and the removal of water on the public road have been carried out. The supply tank of the hydraulic system has been drained.
  • All long-distance fiber connections are operational.
  • SBG1 was partially damaged (4 out of 12 rooms):
    • The network room was checked yesterday. An isolated electrical test run, without connectivity, is planned for this weekend. We estimate an orderly restart of connectivity early next week.
    • Reactivation of the electrical network is tentatively planned for Monday, March 15.
    • Servers will be audited and reactivated individually.
  • SBG3 was spared from the fire
    • A network room will be set up by the end of this week with the plan to be operational beginning of the week of March 15.
    • Tests for the high voltage power supply will be carried out this weekend. Provisional estimate for the reestablishment of power is Monday, March 15.
    • A progressive restart of all services is estimated to begin Monday, March 22.
  • SBG4 was spared from the fire
    • We will finalize our service reactivation plan by the end of this week.

All affected Email Pro services are being restored beginning on Thursday, March 11, starting at 4:55 p.m. No email was lost.

Priority 2: Providing impacted customers with infrastructures in our non-affected data centers.

  • For customers that have been impacted, we are offering replacement infrastructures (Bare Metal) in our Roubaix (RBX) and Gravelines (GRA) data centers.
  • We have already delivered to our impacted customers:
    • Wednesday 10 March: 2,100 new servers.
    • Thursday, 11 March: 630 new servers.
  • The new organization of our assembly lines has been operational since the morning of Friday, March 12 and has tripled our production capacity (3,000 to 4,000 servers per week).
  • We are committed to producing nearly 15,000 new servers in the coming weeks.
  • Delivery times for the new services will be communicated as soon as possible.

Priority 3: Implementation of all DRP (Disaster Recovery Plan) mechanisms with our customers.

  • By the end of the week, our support teams and Technical Account Managers will define a support plan for our customers which have not yet activated their DRP.

Communications

Our mission is to provide our customers with the highest quality of service to support their online activities, and we know how important this is to them. We sincerely apologize for the issues caused by this fire. We will continue to communicate with the greatest transparency about the cause of the fire and its consequences.

An investigation is ongoing. Regardless of the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to the impacted customers. All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

 

 

Roubaix, France | March 11, 2021 – 15:30 (CET)

Summary:

People:

  • HR teams arrived today to support the on-site teams.
  • Additional industrial and technical teams arrived from other offices and data centers to reinforce the on-site staff and provide specific expertise.
  • Logistic support and food services have been provided to all on-site teams.
  • A temporary office has been established at SBG.
  • We've reinforced the staff at the French production site (Croix) with internal volunteers to address the current production plan and additional workload caused by the Strasbourg incident.
  • The increased production will be carried out with extended working hours and during the weekends.

Production and Data Centers:

  • We’ve started to propose replacement infrastructure (Dedicated Servers, Hosted Private Cloud, and Public Cloud) in Roubaix and Gravelines data centers to our customers.
  • To address the demand, additional assembly lines will be in place in the next 48 hours. This will triple our production capacity.

On the Strasbourg site:

  • Diagnosis and inventory are currently in progress.
  • We are in the process of cleaning up and repairing the damaged buildings to guarantee a safe working environment for our teams.
  • We have finished removing the fire extinguishing foam and water from the street. We will remove the water from the water tank today.
  • SBG2 will need to be almost entirely reconstructed.
  • SBG1 was heavily damaged.
    • Following a preliminary audit, the network room in SBG1 will be restored at the beginning of next week.
    • The provisional date for restoring the power supply is Monday, March 15.
    • In the coming days, servers will be activated room by room after the audit.
  • SBG4: preliminary audits did not reveal any issues. Our ambition now is to restore power during the week of March 22 and then gradually reactivate all services.
  • SBG3 was not impacted by the fire. We aim to restore the power and network during the week of March 22 and then gradually reactivate all services.

Our three priorities remain as follows:

  1. Reserve infrastructures at our other data centers for our affected customers: We have a stock of new servers at the Roubaix and Gravelines sites, which are ready to be delivered to the majority of affected customers. We will further enhance availability in these data centers, with the production of nearly 10,000 new servers in the coming weeks. Affected customers will be notified about this process as soon as possible.
  2. Secure the site: Now that we have regained access, we will clean up the site and reconnect the electricity and the network for the three remaining data centers.
  3. Continue to assess the impact on our customers’ servers at the affected data centers, in order to find the best solutions and communicate to all customers.

Additional action taken by OVHcloud:

Network:

  • All fiber paths that enter the data center site are operational.
  • We have placed orders with different suppliers – due date for delivery on-site is March 11.

Communication:

Our mission is to provide our customers with the highest quality of services to support their online activities, and we know how important this is to them. We sincerely apologize for the issues caused by this fire. We will continue to communicate with the greatest transparency about the cause of the fire and its consequences.

An investigation is ongoing. Regardless of the cause of the incident, OVHcloud is currently evaluating the potential technical and operational measures aimed to provide solutions to the impacted customers.

 

 

2021-03-10 – 21:00 (CET)

Summary:

  • At 00:47 CET on Wednesday 10 March 2021, a fire broke out in a room at one of our four OVHcloud data centers in Strasbourg (SBG2).
  • The fire was contained by the early hours of the morning.
  • There are no injuries.
  • The fire mostly destroyed the SBG2 data center and partially damaged the SBG1 data center (4 of the 12 rooms destroyed). The two other OVHcloud data centers in Strasbourg were not affected by the fire; the SBG3 and SBG4 servers are currently switched off but undamaged.
  • The cause of the fire has yet to be established and an investigation has been launched as mandated by the authorities.

Actions taken by OVHcloud:

  • The technical and commercial teams have been working since this morning to inform our customers and handle the unavailability of our Strasbourg site.
  • The company's founder, Octave Klaba, has been on site since this morning with the industrial and technical teams.
  • To follow updates to this situation in real time:
  • Incident tracking platform: https://status.us.ovhcloud.com
  • Twitter: https://twitter.com/OVHcloud_US
  • A customer announcement, along with an FAQ, will be available tomorrow.
  • To help us handle customer requests we recommend following real-time updates posted on our US Status page or by opening a support request through your OVHcloud Control Panel.

Our three priorities are as follows:

We are doing everything we can to ensure a continuity of service to our customers:

Impact on our operation:

Our mission is to provide our customers with the highest quality of services to support their online activities and we know how important this is to them. We sincerely apologize for the issues caused by this fire. We will continue to communicate with the greatest transparency about the cause of the fire and its consequences.

We are assessing the environmental impact by working with the relevant authorities on a procedure to confirm that no pollution was caused. At this stage, we can confirm that the local residents are not at any risk.

We are continuously assessing the impact of this incident and will communicate as transparently as possible on the progress of our analyses and the solutions to be implemented. All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

  1. Reserve infrastructures at our other data centers for our affected customers: we have a stock of new servers at the Roubaix and Gravelines sites, ready to be delivered to the majority of affected customers. We will further enhance availability in these data centers, with the production of nearly 10,000 new servers in the coming weeks. Affected customers will be notified about this process as soon as possible.
  2. Secure the site now that we have regained access, clean it up, and reconnect the electricity and the network for the three remaining data centers.
  3. Continue to assess the impact on our customers’ servers at the affected data centers, in order to find the best solutions.
    • We are working on a plan to relaunch the two unaffected data centers (SBG3 and SBG4), the partially affected data center (SBG1), as well as our network, as quickly as possible.
    • We ask that our customers exercise caution around the emails they receive: in times of crisis, it is common for malicious activity (phishing, spam, etc.) to increase. It is more important than ever to stay alert.
    • We are continuing to assess the impact of this incident, particularly for the customers whose data was located in the data center destroyed by the fire.
    • All of our services in our other France-based data centers and across the world (including 15 data centers in Europe and two in the US) are fully operational.

About OVHcloud US

OVHcloud US is a subsidiary of OVHcloud, a global cloud provider that specializes in delivering industry-leading performance and cost-effective solutions to better manage, secure, and scale data. OVHcloud US delivers bare metal servers, hosted private cloud, and hybrid and public cloud solutions, and was recognized as a "Strong performer" in Forrester's Hosted Private Cloud Services in North America (2Q2020) and as a "Contender" in the IDC Worldwide Public Cloud as a Service Vendor Assessment (2020). OVHcloud manages 34 data centers across 12 sites on four continents, manufactures its own servers, builds its own data centers, and deploys its own fiber-optic global network to achieve maximum efficiency. Through the OVHcloud spirit of challenging the status quo, the company brings freedom, security, and innovation to solve data challenges – today and tomorrow. With a 22+ year heritage, OVHcloud is committed to developing responsible technology and strives to be the driving force behind the next cloud evolution. us.ovhcloud.com.

Media Contact

Karen Kokiko
OVHcloud US
(703) 582-7504 (m)
karen.kokiko@us.ovhcloud.com
https://us.ovhcloud.com