Prolateral Consulting Ltd
Prolateral Consulting Ltd

Network and Service Status

Below you can see the Prolateral Network and Service Status.  In addition you will see any service posts currently in progress, any future scheduled maintenance, and any recently resolved posts.

System Status

Status

Service

Comments

flag-green-32x32 Portal No reported issues
flag-green-32x32 outMail No reported issues
flag-green-32x32 Backup MX No reported issues
flag-green-32x32 Fake MX No reported issues
flag-green-32x32 proFilter Partial Outage
flag-green-32x32 Hosted Email No reported issues
flag-green-32x32 Master DNS No reported issues
flag-green-32x32 Backup DNS No reported issues
flag-green-32x32 Web Hosting No reported issues

 

Announcements

Outage

Status

Service

Notes

Start Date

End Date

Unplanned

Closed

profilter We are currently experiencing a partial outage in one of the UK data centres.  This is currently affecting the profilter cluster mx442503.

5/3/2024 00:05 Whilst engineers continue to investigate the issue all email traffic to mx442503 will be diverted to another profilter cluster for processing and delivery.

5/3/2024 01:18 Engineers have resolved the issue with mx442503 and emails are being route back through the original cluster.
04/03/2024 22:51  05/03/2024 01:18
Scheduled

Closed

MasterDNS & BackupDNS (ns2.dns-engine.com) There is a scheduled maintenance window for engineers in the Amsterdam data center starting from Wednesday 19th July 2023 at 22:00 (Amsterdam Time) to 01:00 (Amsterdam Time).  This is to allow the data center engineers to do core upgrades on backend equipement.

Note. ns1, ns3, ns4 and ns5 will not be affected.

19/07/2023 21:00 19/07/2023 23:59
Scheduled

Closed

3CX VoIP hosting There is a scheduled maintenance window for engineers in the London data center starting from Monday 17th July 2023 at 22:00 to 01:00 . This is to allow the data center engineers to do core upgrades on backend equipement.

During this window there will be some disruption to 3CX VoIP hosted services.

17/07/2023 22:00 18/07/2023 01:00
Scheduled

Closed

MasterDNS & BackupDNS (ns1.dns-engine.com) There is a scheduled maintenance window for engineers in the Dallas data center starting from Thursday 13th July 2023 at 01:00 (Dallas Time) to 04:00 (Dallas Time).  This is to allow the data center engineers to do core upgrades on backend equipement.

Note. ns2, ns3, ns4 and ns5 will not be affected.

13/07/2023 07:00 13/07/2023 10:00
Unplanned

Closed

MasterDNS & BackupDNS (ns5.dns-engine.com) Engineers are currently investigating an issue with ns5.dns-engine.com.

Note. ns1, ns2, ns3 and ns4 are not affected.

22/2/2023 15:40 Engineers identified a configuration error resulting from a update process.  This has been resolved and this incident closed

22/02/2023 15:25 22/02/2023 15:40
Scheduled

Closed

MasterDNS & BackupDNS (ns2.dns-engine.com)

There is a scheduled maintenance window for engineers in the Amsterdam data center starting from Thursday 24th November 2022 at 00:00 (GMT).

During this maintenance window ns2.dns-engine.com will be off line. Although there is a 6 hour window allocated it is hoped that services will be operational earlier. As always please keep an eye on the twitter feed and this status page.

Note. ns1, ns3, ns4 and ns5 will not be affected by this maintenance window.

24/11/2022 01:35 Maintenance window complete.

24/11/2022 00:00 24/11/2022 06:30
Unplanned Closed Legacy Profilter Service

Some customers still on the legacy Profilter service will not be receiving email. There are issues in the datacentre that our on-site engineers are working on at the moment.

2022-06-17 0930 We are currently investigating issues with the legacy profilter platform and degraded performance. Emails are still flowing but not how we would expect them.

2022-06-17 0945 The legacy profilter platform was due to be retired at the end of 2022, as sush many clients have already been migrated to the current profilter service. A decision has been made this morning to migrate the remaining number of clients from the legacy profilter platform today. All clients affected will be notified this morning. For some domains there is no action required from the client and the migration will be automatic. For domains that do need changes made the client end our support team will contact you directly to discuss options.

2022-06-17 1150 The issues regarding the degraded performance of the profilter legacy platform have been resolved.

2022-06-17 1420 All domains that could be migrated seemlessly have now been completed.

2022-06-24 1730 All domains that were on the legacy profilter platform have been migrated.

17/6/2022 09:28 24/6/2022 17:30
Scheduled

Resolved

WebHosting (hosted-lon-3 & hosted-lon-4)

There is a scheduled maintenance window for engineers in the UK data center starting from Tuesday 31st May 2022 at 16:30 (GMT).

During this maintenance window there will be a slow performance due to high IO and for a short period an outage whilst the clusters are failed over to a new environment.

Note, If your service is hosted on either hosted-lon-3 or hoested-lon-4 then you will see issues with websites hosted and emails hosted on those clusters.

During the maintenance window regular updates will be published on our Twitter feed.

31/5/2022 16:30 31/5/2022 16:46
Scheduled

Postponed (see details)

WebHosting (hosted-lon-3 & hosted-lon-4)

There is a scheduled maintenance window for engineers in the UK data center starting from Thursday 26th May 2022 at 16:30 (GMT).

During this maintenance window there will be a slow performance due to high IO and for a short period an outage whilst the clusters are failed over to a new environment.

Note, If your service is hosted on either hosted-lon-3 or hoested-lon-4 then you see issues with web hosted and emails hosted on those clusters.

Update due to unforeseen circumstances at the data center, engineers have had to postpone the maintaince window till next week.  Tuesday has been identified as the next maintenance window with a backup window of Wednesday all the same time slots (1630 to 1800)

26/5/2022 16:30 26/5/2022 17:12
Scheduled

Resolved

MasterDNS & BackupDNS

On Monday 9th May 2022 at 23:00 (GMT) there is a scheduled maintenance window for engineers in the Amsterdam data center to install a new Automatic Transfer Switch (ATS) in the power delivery system.

During this maintenance window it will only affect the NS2 cluster, whilst each server in the cluster is shutdown and moved to the new power circuits.

Note, DNS services will run normally during this maintenance window as NS1, NS3, NS4 and NS5 clusters will not be affected.

9/5/2022 23:00 10/5/2022 02:00
Unplanned

Resolved

proFilter

2022-04-01 0913 We are currently investigating issues with profilter inbound routing.

2022-04-01 0950 We are working with engineers in the London datacenter to resolve a routing issue.

2022-04-01 1015 We are continuing to work with on-site engineers who have identified an error in a cloud-init configuration which has rolled out across the datacenter.  This is currently preventing profilter clusters from failing over, so we are working on forcing a manual failover to another datacenter.

2022-04-01 1040 Incident resolved.  profilter is processing emails, all queued emails on the failover cluster have been processed as well.

1/4/2022 09:13 1/4/2022 10:40
Unplanned

Resolved

proFilter (Legacy)

2021-10-08 1905 We are currently investigating performance issues with the Legacy profilter cluster. This incident only affects the Legacy profilter cluster and not the mx??25??.smtp-engine.com clusters.

2021-10-08 2330 We are continuing to work on the incident. A disk in the storage area has been replaced and the storage raid is currently being rebuilt. During the rebuild process we will still performance issues.

2021-10-09 0345 Storage area performance is still degraded due to rebuilding.

2021-10-09 0730 Storage area rebuild complete. Engineers will run tests on the cluster.

2021-10-09 0810 Cluster is running normally, will continue to monitor.

2021-10-09 1330 Incident closed.

8/10/2021 19:05 9/10/2021 13:30
Planned

Resolved

DNS-Engine Scheduled maintenance in the datacenter that could affect ns5.dns-engine.com. We are anticipating a short interuption in network connectivity whilst engineers reconfigure the vEdge services. This will not affect ns1, ns2, ns3 or ns4. They will continue to service DNS requests. 24/8/2021 10:00 24/8/2021 16:00
Planned

Resolved

DNS-Engine Scheduled maintenance and upgrade work for ns5.dns-engine.com. 17/8/2021 17:00 17/8/2021 23:00
Unplanned Resolved DNS & proFilter

03:05 Network connectivity lost to one of our datacentres - investigation underway

03:27 We have reports of a fire in the France datacentre.

04:15 We are provisioning failover of affected clusters to a separate datacenter and will re-route traffic through there.
    - Affected cluster nodes are NS5 and profilter (MX332501)

05:30 The datacentre in France is split into multiple units.  One unit has been destroyed due to a fire and the other three buildings are currently unavailable due to fire crews complelety isolating the industrial estate of power.  Usually a cluster node would failover to the sister location but this isn't possible at the moment so engineers are working on a manual failover to the UK datacentre.

06:07 Status on MasterDNS and BackupDNS changed to yellow as global coverage of DNS records isn't affected.  ns5 is currently off-line but ns1 to ns4 are fully operational.  Failover of ns5 to another datacenter will have an impact on IP Address, so engineers are waiting to see if units 1, 3 or 4 in france will be released by the fire crew first in order to preserve IP Addresses. Please continue to watch this status page for updates.

06:45 The fire at the datacenter has destroyed building 2, building 1 has some fire damage.  Units 1, 3 and 4 will remain offline until the fire crews release the scene, which will not be until later on today.  Therefore ns5 will remain offline.  proFilter (MX332501) failover engineers are working on testing the new cluster before switching on live.

08:30 profilter cluster (MX332501) has now been successfully failed over to (MX442503), and emails flowing again.

09:08 profilter status changed to yellow.  No updates as yet to extent of the fire damage. Please keep an eye on this page for status updates

11:30 Incident Update, at 00:47 this morning a fire started in the France datacenter where we (Prolateral) have several clusters of services sited. We weren't aware of the incident until 02:58 this morning when our own network monitoring started to alert us.  The alerts were due to the fire service cutting all power to the site at 02:54.  Sadly by 04:09 this morning the fire had completely destroyed one (Unit2) of the three units.  We are grateful that there has been no loss of life to either datacenter staff or the fire service in attendance.  By 4am this morning we invoked our disaster recovery plan because it was clear that automatic failover of the clusters wasn't going to happen, so engineers worked on adding additional clusters to the UK datacenter and then forced a manual failover.  Between 06:00 and 08:30 clusters were migrated over and back online.

11:35 Incident Update, As previously mentioned unit2 was lost, there looks to be partial fire damage to unit1,  Unit3 and unit4 are ok, just no power at the moment.  As soon as the fire service start releasing parts of the site to the datacenter staff, engineers can further assess and start to power up those buildings.  For your information we have several clusters at that location in various units.

20:00 Incident Update, Datacenter engineers are planning on restoring power to Units1, 3 & 4 by Monday 15th.  Our thanks to all those engineers working day/night to get things back to operational as quick as possible.

23:00 This ticket will remain open throughout the incident (hence the yellow status flags above) of the datacenter, however all affected (Prolateral) services were failed over to the UK Datacenter this morning (see above) and fully operational.

2021-03-15 Update.  It has been confirmed that all clusters hosted in the datacenter have sadly been losted.  Over the next two weeks we look to replace those clusters and add back to the swarm.  This incident will remain open during this time.

2021-04-01 Update on NS5. We are working with the datacentre team in restoring the NS5 cluster. However whilst electrical work is continued in the Units it's not reliable enough to make a production service, as such NS5 we remain failed over to the NS4 cluster. More information to follow on this.

We apologies for any inconvienance caused and we are working hard to resolve the situation as quick as possible.

Progress will be reported here and on our Twitter page.

2021-04-07 Update on NS5. The cluster has been restored and it currently under going testing before going into production ready status.

2021-04-08 Update on NS5. The cluster is live.

2021-04-10 This incident will be closed.

10/03/2021 03:05 10/04/2021 19:44
Unplanned Resolved outMail

11:50 Network connectivity lost to datacentre - investigation underway

13:30 Network connectivity in datacentre is being worked on by provider 

14:30 We are provisioning another cluster in a separate datacenter and will re-route traffic through there.

16:20 Engineers are working in three teams.
    - Team 1 - working with the datacenter engineers that is suffering a complete outage
    - Team 2 - working with the engineers of the sister datacenter to force a failover scenrio
    - Team 3 - working to bring up a new cluster to enable emails to flow from the failed datacenter

17:15 Engineers are still working to resolve the issue.

18:20 Engineers are starting to bring up a new cluster.  Once internal testing has been completed the cluster will go live and be ready to accept email again. Any emails that you are trying to send will be queued on your server ready and waiting to be sent out.  This will happen naturally once the cluster is made live.

We apologies for any inconvienance caused and we are working hard to resolve the situation as quick as possible.

Progress will be reported here and on our Twitter page.

18:55 The issue with the network equipment and configuration at the Datacentre has been resolved and the mx496502 cluster is fully accessible. Queues are clearing. 

 

04/09/2020 11:50 04/09/2020 18:55
Unplanned Resolved outMail

08:30 We are experiencing connection issues to MX496502.  Engineers are working on the problem.

08:35 Engineers have identified the cause and are working on a resolution.

09:15 Engineers are still working on the problem.  Due to the earlier (unrelated DNS issue) the cluster failover hasn't worked.  Engineers are working with the datacenter engineers to resolve as quickly as possible

10:21 The root cause has been identified and engineers are working to resolve the problem as quickly as possible.  

11:18 Service to the cluster MX496502 has been resolved.  Email to that cluster is now flowing. 

30/01/2020 08:30 30/01/2020 11:18
Unplanned Resolved DNS

06:05 We are experiencing MasterDNS and BackupDNS issues.  Engineers are working on the problem.

06:40 Engineers have identified the cause and are working on a resolution.

07:50 All DNS services have been restored.  The fault was due to a cascading configuration error which has been resolved.  Engineers will continue to monitor

30/01/2020 06:05 30/01/2020 08:30
Unplanned Resolved Profilter

11:40 One of our datacentres is currently affected by an ongoing network-outage. More information will be reported here.

12:31 Network routing issue within datacentre now resolved.

14/11/2019 11:40  14/11/2019 12:25
Unplanned Resolved Profilter

A major datacentre outage in our Strasbourg datacentre is currently being investigated.

11:30 update. A total power & backup power outage has affected 3 datacentres and thousands of servers. Work to restore continues.

13:10 update. Power was restored, but we are awaiting datacentre reboot of networks as the systems are brought back online. 

19:20 All IP routing issues resolved and full connectivity to all Profilter servers restored.

09/11/2017 06:30 09/11/2017 19:20
Planned

Scheduled

Resolved

Web Hosting Scheduled maintenance and upgrade work for hosting servers in the London Data Centre. 06/10/2017 18:00 07/10/2017 01:45
Unplanned Resolved Outmail

Microsoft (Outlook, Hotmail, Live.com, etc.) blocked a significant range of IP addresses. We are rerouting email via unaffected IPs while the issue is resolved.

Update: re-route fully in place by 11.15. MS fix ETA unknown.

Update: blocks removed and normal routing resumed.

18/04/2017 9:55  21/4/2017 07:50
Unplanned

Resolved

Web Hosting We are seeing connectivity issues to the London Data Centre. Engineers are working on the problem. 28/07/2016 15:26 28/07/2016
21.18
Planned

Scheduled

Resolved

outMail outMail is currently undergoing upgrades and migration to new fast and exciting cloud clusters. More information can be found here. 01/04/2016 00:00 09/05/2016 23:59
Unplanned

Resolved

profilter

Network routing and DNS issue prevented hostname lookups. DNS server issue resolved. Queues now clear.

28-07-2014 09:15 28-07-2014 10:50
Unplanned

Resolved

profilter

Network outage at data center has caused queues to build. They are clearing and should be clear within the hour.

08-07-2014 07:56 08-07-2014 09:10
Unplanned

Resolved

profilter

Significant Spam attack inbound causing delays on email. Servers rebalanced and queues are clearing.

06-06-2014 11:53 06-06-2014 14:02
Unplanned

Resolved

proFilter

Some Profilter customers are expirencing connection problems to a selection of profilter clusters. This is due to the datacentre provider having issues within that centre. System engineers are currently on site and working on the problem.

Update 17.20: Datacentre outage is more severe than first estimates for resolution.  Outage is affecting a large number of european subscribers.  We (Prolateral) are working with datacentre engineers to find a speedy resolution.  More updates to follow.

Update 20.05: Datacentre network component failure has been resolved, routing rebuilt and is back online. Queues have built up and are clearing steadily, otherwise all services running normally.

17-12-2013 14:00 17-12-2013 20:05
Unplanned

Resolved

outMail

Some customers are seeing bounces due to black-listing, we are migrating those services away from affected IP addresses.

18-09-2013 09:00 18-09-2013 14:00
Planned

Resolved

 

Hosted Email

Scheduled Maintenance work in the LondonCloud

Please be advised there is scheduled maintenance on several servers in the LondonCloud (hosted-lon-1, hosted-lon-2 and hosted-lon3).  The maintenance window is very small to allow for the servers to be rebooted following the maintenance work.  This work will affect web hosting services only for approx 15minutes. The small outage will occur between 2300 and 0400 Thursday 12th Sept 2013.

12-09-2013 23:00 12-09-2013 04:00
Unplanned

Resolved

outMail

Some customers are still experiencing problems sending emails to AOL, Demon Internet and EmailSrvr domains.
It appears these Internet Providers are using a cached database from CloudMark CSI and not a realtime lookup.
Therefore this is a knock on effect from yesterdays problem which was effecting a small number of outMail Clusters
Engineers are working on resolving the issue.

Update: Engineers have resolved the issue

10-09-2013
07:35

10-09-2013 07:45

Unplanned

Resolved

outMail

Some outMail clusters are experiencing email delivery problems. Engineers are working on the issue now.

Update: It appears CloudMark CSI have wrongly blacklisted some outMail Clusters, this has been resolved with CloudMark and all outMail clusters are blacklist free

09-09-2013
10:00

09-09-2013 15:20

Unplanned

Resolved

outMail

Some outMail clusters are experiencing email delivery delays. Engineers are working on the issue now. 
Engineers have resolved the issue and queues are running normally

05-04-2013
11:50

05-04-2013 12:21

Unplanned

Resolved

clients.prolateral.com

Some reports of time-outs when accessing pages. We are investigating possible network routing issues at one of our providers.

14-02-2013 08:00 14-02-2013
08:30
Planned

Resolved

Various

Database maintenance affecting access to some webservers and outmail intermittently.

14-12-2012 03:00 14-12-2012 03:35
Unplanned

Resolved

outMail

Some customers are reporting delays through the outmail cluster. Engineers are investigating.

DDoS SPAM forced re-routing of email queues. All queues now back to normal.

13-12-2012 11:55 13-12-2012 13:30
Unplanned

Resolved

outMail

Some outMail clusters are experiencing high volumes of connection requests which is causing some email delays of upto 5-10minutes.

Actions have already been put in place and we are monitoring the situation.

11-12-2012 12:00 11-12-2012 14:00
Planned

Scheduled

Resolved

Hosted Email

Scheduled Maintenance work in the LondonCloud

Our Tier1 provider is conducting maintenance to their services.  We expect no disruption to our services but this is purely an advisory to our web hosted customers.

02-11-2012 22:00 10-11-2012 01:47
Planned

Scheduled

Resolved

Portal
Webhosting
Hosted Email

Scheduled Maintenance work in the LondonCloud

Please note there will be a short period of downtime which will affect some web hosting, email hosting customers.  This will also affect access to the CI Portal during this time.  This is due to maintenance work being carried out on a core edge firewall.

20-10-2012 21:57 20-10-2012 22:17

 

Service Status Key

Status

Description

green-flag-status-no-issues-yippee

No Current Reported Issues

yollow-flag-status-minor-issues-dooh

Minor Issues

red-flag-status-major-issues-battle-stations-man-the-pumps

Major Issues

blue-flag-status-maintenance-planned-scheduled

Planned Maintenance - see notes