Updates
May 17 (After Midnight)
Site is migrated. I’m getting some sleep…
May 16th
It’s been a lot of work but the new server is ready to take the forum.
I’m only expecting a short downtime, and probably some instability for the first little while.
There is a rollback plan and a scheduled downtime: 2026-05-17T01:00:00Z→2026-05-17T03:00:00Z
May 11th
I’ve been away from the forum the past few days with minimal oversight.
I’m aware of some issues from users who have DM’d me, I’m sorry I haven’t responded to you yet but I’m very busy with getting this new server up and operational, while also managing my day job.
I have the server, it’s got the hypervisor, firewall, and I’m finishing up on setting up the database.
I am aiming to do the migration This Weekend
There will be an announcement prior to the migration.
May 7th
Due to the delays and issues with the purchase, I am pivoting to a different server.
Location: France
CPU: 16C / 32T AMD Ryzen 9 9950X
MEM: 128GB ECC 3600MHz DDR5
STOR: 2x 960GB SSD NVME (RAID 1) (960GB usable)
NET: 1GBps unmetered
This is still a usable server, and has more memory, the installation fee is a bit more expensive but the recurring cost is a little lower. This server is purchased, and awaiting validation/delivery.
May 5th
Due to some issues with purchasing the server this may be a bit delayed.
Working with the provider to resolve those issues, the server should hopefully be purchased this week
April 22nd
I have spent sometime investigating options and finalizing on a specific one.
Eroscripts will be renting a server of these specs:
Location: Germany
CPU: 16C / 32T AMD EPYC 4584PX
MEM: 64GB ECC 5200MHZ DDR5
STOR: 2x 960GB SSD NVME (RAID 1) (960GB usable)
NET: 3GBps unmetered
A server this size should be able to keep us scaling for long to come.
It also will unlock a number of capabilities for us:
- Firewall
- Increased abilities to defend against API abuse
- Load balancing
- Multiple Nodes
- High availability
- Testing canaries
- Overprovisioning for bursting traffic
- Ownership
- We will control the hardware, so we can provision more servers/software
- Testing playground - we can experiment on a test server to improve things without affecting the main server, this was originally done on my own hardware
- Saved Costs
- The initial investment is much higher, but in the long run will save us thousands in hosting costs since we can scale more for less
- RnD will no longer need to have to pay for additional hosting. I can do it without additional cost
I will be getting it around the turn of the month, I’ll be testing it and playing around with things for a bit.
I plan to do the migration sometime mid May. An announcement will be made in advance
Once the migration is complete and stable, I will be setting up high availability
April 21st
After the meeting I had today, I have landed on the host I plan to use.
I’m speccing out a server for the forum. From the looks of it, we can get a PHAT server.
I’m thinking about our current infrastructure and how I should implement the new infrastructure. I’m putting a lot of thought into this because this migration will not be cheap, and will hopefully be where the forum will live for the rest of it’s days.
Hi all,
I’m more than sure some of you have noticed the performance issues with the forum lately.
I originally thought it might be related to the recent update I did, but that doesn’t seem to be correct.
Why this is happening:
This is the CPU metrics from the other day:
The blue you are seeing is CPU Steal. This is when the server is waiting on the hypervisor (hosting providers server) to provide it CPU time to process tasks.
We are semi-frequently seeing that the server can’t execute tasks because the hosting provider doesn’t have the CPU resources necessary (even though we are paying for it).
This doesn’t necessarily come as a surprise as that is somewhat expected when utilizing an over-provisioned host. This type of hosting is cheaper, and while doesn’t come with guaranteed performance metrics, it’s typically stable, and we’ve been using this a long time, even before the big migration.
This has since changed, I believe the hosting provider has fully saturated the entire DC (cluster of nodes). This is not typical practice even for over-provisioning. I could speculate why they did this, but I presume it’s most likely some AI thing going on.
I have been in a conversation with the hosting provider and they more or less told me that there was nothing they could do. We tried some resolutions that we already knew were not likely to resolve the issue.
So what am I doing?
I’m in talks with another provider. There is a possibility of cheaper and better infrastructure.
I will have a meeting next Tuesday to discuss what best suites the forum.
What I’m aiming to get is to completely replace the original infrastructure, and with the money saved, get a fully dedicated hypervisor myself.
So what will change?
We will likely be doing another migration in the near future. This could mean 1-2 days of downtime, but I will do my best to minimize that downtime.
This also means:
- No more fighting for CPU time with other members of a node
- High availability is on the horizon (finally)
- I will have control of the hypervisor, I can provision as many VMs as needed
- Much more consistent and reliable experience
I am really hoping this doesn’t fall through.
Costs
The migration itself will be costly. For 1 month the costs will be doubled, and I will have a installation fee for the new dedicated server. If you can contribute to alleviate those costs, Myself and the community would be forever grateful.
This was made possible by everyone who contributes to donating to the forum
