Server move Sunday May 24 - Status: Completed

webwit · 18 May 2015, 23:43

We'll be moving to a new server next Sunday, May 24. Our SSD has been giving errors, causing server hangups and database table crashes (and possibly unknown file corruptions), and its time to move before it fails catastrophically. In the meantime, don't worry, we have backups to two external locations. Our new server will be same server package at hetzner.de, except HDD this time, because we also need the additional space in the not too distant future. This won't be a performance hit as it is not much of a bottleneck.

I don't expect the actual move to last long. There's quite a bit of data to transfer, but on the same network, and if I have some time I might move the bulk before Sunday. DNS propagation might take a while though, depending on your location and ISP. Just before I start moving data, I'll put the site on the old server in read-only mode, otherwise new posts and such during the transfer might get lost.

I'll see you at the other end of the event horizon!

Muirium · 19 May 2015, 01:50

Mal-2 · 19 May 2015, 05:31

All too common a real world scene:

Data center IT Guy: "You have a hard drive about to fail. What time should we do the swap and rebuild?"
Site Owner: I can't afford any down time! Can you do it without taking the server down?
DCITG: I'm afraid you didn't pay for a server with hot-swap ability, so...
SO: But the world will end if my site is down!
DCITG: [deep breath] I can't help that. The drive WILL DIE, so please allow be to rephrase the question. Would you prefer to schedule a graceful swap, or wait until it completely falls over?

Halvar · 19 May 2015, 06:41

SO: I saw a billboard on my commute that said 'Cloud computing!'. So I'm off to Azure, good bye!
DCITG: [even deeper breath] See you around!

andrewjoy · 19 May 2015, 10:12

Mal-2 wrote: All too common a real world scene:

Data center IT Guy: "You have a hard drive about to fail. What time should we do the swap and rebuild?"
Site Owner: I can't afford any down time! Can you do it without taking the server down?
DCITG: I'm afraid you didn't pay for a server with hot-swap ability, so...
SO: But the world will end if my site is down!
DCITG: [deep breath] I can't help that. The drive WILL DIE, so please allow be to rephrase the question. Would you prefer to schedule a graceful swap, or wait until it completely falls over?

Unfortunately this is all too common, getting people to agree to downtime is a PITA, i just turn stuff off now and do it. They have been told when its going down if they don't like it fuck em.

Muirium · 19 May 2015, 13:06

Mal-2 wrote: SO: But the world will end if my site is down!

Mal-2 wrote: DCITG: [deep breath] I can't help that. The drive WILL DIE, so please allow be to rephrase the question. Would you prefer to schedule a graceful swap, or wait until it completely falls over?

webwit · 24 May 2015, 13:06

Welcome to the other end of the event horizon! You made it!

Please report any problems here.

bhtooefr · 24 May 2015, 13:24

That is one hell of a long TTL aiming at the old server. Ended up putting it into my Mikrotik's config manually.

webwit · 24 May 2015, 13:26

Changed it from 1 day to 3 hours yesterday though. Maybe that was too late

chzel · 24 May 2015, 14:26

Smooth ride so far!
It's been just a few minutes since it came back up for me!

seebart · 24 May 2015, 14:47

what`s nice about this administrative technical situation is that the admins here at DT actually take part personally in the community so it`s 1A* updates and communication all the way. That`s not always the case in other online forums. Good job webwit and whoever else worked on this.

webwit · 24 May 2015, 15:03

I was impressed by cPanel's transfer tool here. It was one-click, and it moved everything from files, databases, mail, crontabs etc. Painless really.

Muirium · 24 May 2015, 15:11

My thanks to the server admin team who pulled off this move quite nicely: Webwit and his industrial SSK. Great job guys!

I had nothing to do with it. Other than occasionally nagging that we needed to get off the previous server. Don't know how many of you noticed its occasional outages. Fortunately we've got solid backups so I never saw a single post go missing whenever it did. Good preparation for making this move from a position of strength instead of OMFG WE GOOTTA MOVE!! IT'SA ON FIYAH!! as some other transitions entail…

Halvar · 24 May 2015, 17:31

Thanks webwit!

Mechanical storage for a mechanical keyboard forum. Just how it should be.

bhtooefr · 24 May 2015, 17:36

Ah, but flash uses floating-gate transistors, which are capacitive devices.

matt3o · 24 May 2015, 18:55

cpanel... brrrr... https://youtu.be/xI8G3UefIkU?t=38s

andrewjoy · 24 May 2015, 19:11

An amazing job, as someone who moves website around quite a bit i know how much of a pain it can be , especially when the Muppet who set them up put all the web content in a very strange place. I also dont like how he set up the vhosts, in my view you should have an include for a folder in the config file and then have your vhost's configs in separate files, ideally with symbolic links into the folder thats included so you can quickly enable and disable them without having to move or delete files. And you should also not leave back doors into internal websites from you home IP address you naughty boy, and dont comment it with <insert name here > home IP. But enough of my problems.

Good job!

How are the backups done on the website ?

Just out of interest ? Do you just back up the data or do you do a whole snapshot of the VM ? I am assuming its running on a VM?

As for storage, ZFS! Thais all folks

webwit · 24 May 2015, 20:25

CPanel is a bit of a mixed bag and I had my doubts. We ran hardcore FreeBSD before, but I decided to give it a try the previous move, mostly to reduce the bus factor and make maintenance easier. The interface is still a maze of options. But it has been extremely reliable in keeping the server updated and secure, makes it very easy to setup advanced features and is zero maintenance. It really delivers there.

We don't run a VM, we have the full iron under our control. It uses two backups types: it keeps 5 daily backups to a backup server - complete cPanel backups of deskthority and the system which can be restored easily. And twice a day it rsyncs all deskthority files and a database export to a remote server, where it keeps 5 daily, 4 weekly and 3 monthly backups.

Ace · 26 May 2015, 23:02

I'm a little late, but could this possibly be the reason why I stopped receiving "info" emails upon receiving PMs on the 24th?

XMIT · 26 May 2015, 23:15

This happened to me as well. GMail started flagging them as spam. Check your spam folder. I don't know why this started happening.

Ace · 27 May 2015, 03:15

XMIT wrote: This happened to me as well. GMail started flagging them as spam. Check your spam folder. I don't know why this started happening.

Ah! You're right. I never even bother to check the spam folder. Is there any way to simply tell Gmail to not count them as spam?

webwit · 27 May 2015, 03:16

I guess gmail hated our change of ip address. Although the headers say they think it's all ok.

Ace · 27 May 2015, 04:59

webwit wrote: I guess gmail hated our change of ip address. Although the headers say they think it's all ok.

So it's not fixable?

XMIT · 27 May 2015, 05:14

Two things.

One, in your spam folder - and *before* unflagging a message as Spam - grab the e-mail headers and PM them to Webwit. If Gmail offers an explanation of why the message was flagged it would be good to copy-paste that as well.

Two, Gmail rules will probably stop flagging such messages as Spam when enough users un-flag them, and/or MX records and/or DNS entries propagate for the new IP. My guess anyway.

All hail Google, the new Microsoft.

webwit · 27 May 2015, 23:40

Check this mail delivery error to elecplus:

SMTP error from remote mail server after initial connection: 554 [removed].secureserver.net bizsmtp IB103. Connection refused. 136.243.20.197 has a poor reputation on Cloudmark Sender Intelligence (CSI). Please visit http://

CSI is some kind of spammer monitor, and it doesn't like our IP. Since we are not spammers and this is a new server, I now believe we got a "hot" IP nummer from spammers who then cancelled their server or something, and the IP address was recycled.

I filed a complaint at Hetzner, demanding a new IP!

Also entered a report here: https://csi.cloudmark.com/en/reset

Muirium · 27 May 2015, 23:43

IPs are precious! The world has already run out of the entire IPv4 namespace. Quit banning them folks!

wlhlm · 28 May 2015, 00:07

Why not make Deskthority available via IPv6 as well? (at least I don't see any AAAA record)

andrewjoy · 28 May 2015, 00:12

Muirium wrote: IPs are precious! The world has already run out of the entire IPv4 namespace. Quit banning them folks!

yet in the UK there is 0 development of IPv6 everyone is going to start using CGNAT and that will pretty much break everything thats not a website or email, nat between LAN and WAN is bad enough

wlhlm · 28 May 2015, 00:17

webwit wrote: Check this mail delivery error to elecplus:

SMTP error from remote mail server after initial connection: 554 [removed].secureserver.net bizsmtp IB103. Connection refused. 136.243.20.197 has a poor reputation on Cloudmark Sender Intelligence (CSI). Please visit http://

CSI is some kind of spammer monitor, and it doesn't like our IP. Since we are not spammers and this is a new server, I now believe we got a "hot" IP nummer from spammers who then cancelled their server or something, and the IP address was recycled.

I filed a complaint at Hetzner, demanding a new IP!
Also entered a report here: https://csi.cloudmark.com/en/reset

Good luck. I have read many horror stories of people trying to get their addresses removed from blacklists.

jou · 28 May 2015, 00:19

I receive the notifications just fine, but adding a SPF record for server.deskthority.net might also decrease spam score for some recipients. This is the SPF check result from my email provider:

Code: Select all

Received-SPF: none (server.deskthority.net: No applicable sender policy available) receiver=mx3.messagingengine.com; identity=mailfrom; envelope-from="xxxxx@server.deskthority.net"; helo=server.deskthority.net; client-ip=136.243.20.197

There might be some mail servers that gives a penalty to the absence of SPF.