Is your server using a wired internet connection or wireless?
Wireless can add ping, likewise a slow or old network adapter, or without the right drivers for it, could slow down your wired connection, I would suggest running a speedtest.net and then checking your speed again on a wired connection.
Further, depending on where your players are in the world, and where you are, will impact ping between server / client. Hosting in the USA with players in Europe for example, would be a higher ping than in the same continent, not really much you can do about that, apart from moving the server
closer to the clients.
Open Resource Monitor on Windows and see if anything is using up a lot of your connection, a background service like an auto-update or constant connection for example could alter your speeds.
If you are connecting to the server from the same machine to play, or an internal network machine, perhaps change your config host to 127.0.0.1 or 192.168.*.* and connect directly to your server, on some routers, the loop created by routing a home connecting through your DNS and back through
your router, can cause some issues.
---
Host of a blacklisted server...