Forum :: Game Development :: Open Source Client (round 2)

Hey everyone,

I wanted to post an update on where I'm at with my open-source client. If you've looked at the GitHub page in the last few months you probably haven't seen much activity - however, that could not be farther from the truth! I've been working hard on a major gut-and-rewrite of the client since the code was quickly becoming more and more difficult to maintain.

For reference, here's the project page: https://github.com/ethanmoffat/EndlessClient

The reason I'm doing a rewrite of the code, other than maintainability, is to focus more closely on having well-designed code. Since starting work at my new(-ish) job like 10 months ago, I've learned a lot about the SOLID design principles and writing unit-testable code as it applies to a C# desktop application we work on. We also loosely follow the Onion Architecture at work. I wanted to apply these architecture and design fundamentals to my client in the hopes of having code that is 1) easy to maintain, 2) well tested, and 3) stable.

There are a few design goals I have personally for this project that I will cover individually in more detail:

1. Use Microsoft's Unity container to provide Inversion of Control / dependency injection for my client
   • This makes code units much easier to test
   • This also reduces how many side-effects your code has on unrelated areas
2. Make data objects and containers immutable where possible and practical
3. Follow a proper design paradigm (SOLID and Onion architecture)
4. Using async/await (part of .Net 4.5 / C# 5)
5. Writing platform-independent code that isn't specific to Windows

Unity Container

Using Unity has been fantastic. The idea behind unity is to build up your application's dependencies in one place, and "inject" them into each other when an object is resolved from the container. What this means is that you "register" a bunch of types that are inter-dependent, and they are each constructed from each other when "resolved".

Here's a sample from the client as it is now: I have an interface IEndlessGame that is implemented by EndlessGame. It has a constructor taking 4 parameters, one of them is of type IClientWindowSizeProvider. The type EndlessGame is registered in the container as IEndlessGame, and the type IClientWindowSizeProvider is also registered with its implementation. When IEndlessGame is resolved, the container automatically finds IClientWindowSizeProvider in the container by using Reflection. Every other dependency resolves in a ripple effect (think of it as a tree data structure).

The game launches by building up all the dependencies, doing some dependency initialization, resolving IEndlessGame, and running IEndlessGame. Many (most) of my classes don't even get initialized with the "new" keyword because of Unity. It makes tracking dependencies a lot simpler and doesn't introduce much overhead when I need to bring a new dependency into an existing class.

Immutability

One of the problems with my initial client implementation is that objects would be modified and cause side-effects. I had to introduce convoluted locking mechanisms and cross-thread signals in order to keep things from being broken, and they were still broken. This is what I'm hoping to solve by making things immutable, but only where practical or applicable. This mostly applies to domain-specific data and not as much to other objects. However, I think it is important to also keep other objects from being too stateful.

Some objects use the Builder pattern, like IPacketBuilder and ICharacterRenderProperties. Each modification returns a new object containing the changes made by the method call (similar to how .Net strings work immutably). A packet can be built for sending like so: 

new PacketBuilder(PacketFamily.Init, PacketAction.Init).AddThree(stupidHash).AddChar(versionMajor).AddChar(versionMinor) etc. etc.

At the end of building the packet, .Build() is called to return an IPacket that can be sent to the server. Received packets are also of type IPacket. IPacket serves more as a stream around a byte array for reading / decoding sets of bytes.

ICharacterRenderProperties (and other objects) uses a .With() method pattern to say "return an object that has all the same properties with some new property value". This is a way of keeping it immutable, and it can be used in a similar way to the packet builder in order to build an object up one step at a time.

Design

Following a good design paradigm is helping me organize my code and make sure that different objects do as little as possible and only know about as much as they need to know. This helps with lowering code dependencies and additional possible side-effects in code. It also follows the Single Responsibility principle ("S" in SOLID).

The design I'm following consists of different basic object "types": Controllers, Actions, Services, and Repositories. 

Controllers bind directly to the UI and represent a workflow. They call actions directly and in some cases access repositories. You should be able to easily tell what a controller method does by reading through the actions it performs.

Actions are commands that "do a thing". They're supposed to do one thing, calling into services and updating repositories (state).

Services are stateless operations that generally create some output based on an input. The word stateless is key here - a service should be able to be repeatedly called with the same result!

Repositories are the application's state. They are what hold on to the data.

Async / Await Pattern

I mentioned earlier that I had a lot of issues a result of multi-threading. So what good does having an asynchronous coding pattern baked in to the core of my code do?

The idea I'm going for is to be able to run operations without locking the game's UI thread (Update() loop in XNA/MonoGame) that operate as tasks. This goes back to my goal with Controllers of creating an easily readable workflow. At the core of my networking functionality, I have the following:

public async Task RunReceiveLoopAsync()
{
    while (!_backgroundReceiveCTS.IsCancellationRequested)
    {
        var lengthData = await _socket.ReceiveAsync(2, _backgroundReceiveCTS.Token);
        if (_backgroundReceiveCTS.IsCancellationRequested || lengthData.Length != 2)
            break;

        var length = _numberEncoderService.DecodeNumber(lengthData);

        var packetData = await _socket.ReceiveAsync(length, _backgroundReceiveCTS.Token);
        if (_backgroundReceiveCTS.IsCancellationRequested || packetData.Length != length)
            break;

        var packet = _packetProcessActions.DecodeData((IEnumerable<byte>) packetData);
        _packetHandlingActions.EnqueuePacketForHandling(packet);
    }
}

This is the core receive loop for the game. What it does is 1) Read 2 bytes; 2) Decode that number into a length; 3) Read <length> bytes of data, 4) Decode the data, and 5) queue it up to be handled by the system. This whole thing runs asynchronously because of the use of async and await. You can see the same sort of pattern used for connecting to the game - in my controller for the 4 main buttons, "Create Account" and "Play Game" both await the result of the INIT_INIT sequence to the server and respond accordingly without locking up the UI.

Platform-Independent Code

I've always wanted to see EO running on OS X or Ubuntu (or whatever your favorite Linux distro is). This has been a secondary goal for development - however, in order to aid the process in case I actually get to the point of being able to do that, I'm trying to write code that relies as little as possible on the use of platform-specific system calls. In Win32, this means avoiding ::LoadLibrary and ::LoadImage. It also means avoiding windows-specific keyboard input. In XNA land, this means using MonoGame.

The way I solved the graphics loading problems was to write a platform-independent PE file reader that can run on Mono. It wasn't working too well on Ubuntu, but I'm hoping that revisiting it in the future I'll be able to figure it out. I'm currently using this PE reader (PELoaderLib) to load the GFX files without relying on Win32 at all. I haven't performance tested it (I assume it is orders of magnitude slower than Win32) but it works well enough.

---
class EOSERV {
Programmer | Oldbie
Open source EO Client: https://github.com/ethanmoffat/EndlessClient
};

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH

Edit for clarity: This is great. I've been popping on to your github quite regularly to look for progress, and I'm quite happy to see that you're working intensively on your client.

A question - I notice that you're making use of asynch and await. Does this mean your client will have .NET 4.5 dependencies? I ask because later versions of .NET don't run on some of the dinosaur machines that some EO players use.

Edit Edit: It occurs to me that this client being a MonoGame build might provide the solution to .NET dependencies from the get-go. Is that how it will work?

---
Want to learn to pixel?
Pixelsource.org

Sounds like you're over-complicating the parts that should be very simple, especially when it comes to involving Unity in the project. Also C# does not work well anywhere outside of desktop Windows, which is circling the "deprecated" trash can.

Here's some very simple code for reading the graphics out of the EGF files in to RBGA arrays while masking out the black colour (both behaviors are in dib_reader::read_line if you're looking to change it), if you're willing to port it:

https://gist.github.com/tehsausage/7102104b977c3c6317963ceeb04e29ea

cio is just a byte interface to files, it can be conceptually replaced with iostreams. int_pack contains the standard ugly byte packing functions.

It's been tested enough to guarantee it'll work with all the standard EO graphics.

Cirras posted: (7th May 2016, 05:15 am)

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH

Edit for clarity: This is great. I've been popping on to your github quite regularly to look for progress, and I'm quite happy to see that you're working intensively on your client.

A question - I notice that you're making use of asynch and await. Does this mean your client will have .NET 4.5 dependencies? I ask because later versions of .NET don't run on some of the dinosaur machines that some EO players use.

Edit Edit: It occurs to me that this client being a MonoGame build might provide the solution to .NET dependencies from the get-go. Is that how it will work?

For development, using Visual Studio will require .Net 4.5 to be installed. I'm interested in using JetBrains Rider IDE, but it is still in EAP and doesn't yet have support for a lot of features (primarily Linux support). I'm excited to see where it goes though.

For running the game, I'll have to double check. I've been trying to get a clean VM set up but its having issues with Windows Update stuck at "Checking for updates". I'll update this post when I know for sure what it depends on.

Update: For running the game, I was able to launch it with an up-to-date install of Windows 7 SP1. The only installed version of the framework I have installed is .Net 4.6, MonoGame binaries are included as part of the release zip. I have a hard time believing there is someone in 2016 with a computer that doesn't meet the minimum specifications for .Net, and Windows 8 and later will automatically support it since it comes bundled with .Net 4.5. There is a lower bound to what I'm willing to support, and this is pretty damn low.

I'm working on updating the README to be more accurate (providing better compiling and installing steps) and capture some of the information posted here.

Sausage posted: (7th May 2016, 10:17 am)

Sounds like you're over-complicating the parts that should be very simple, especially when it comes to involving Unity in the project. Also C# does not work well anywhere outside of desktop Windows, which is circling the "deprecated" trash can.

Here's some very simple code for reading the graphics out of the EGF files in to RBGA arrays while masking out the black colour (both behaviors are in dib_reader::read_line if you're looking to change it), if you're willing to port it:

https://gist.github.com/tehsausage/7102104b977c3c6317963ceeb04e29ea

cio is just a byte interface to files, it can be conceptually replaced with iostreams. int_pack contains the standard ugly byte packing functions.

It's been tested enough to guarantee it'll work with all the standard EO graphics.

Say what you will about the complexity, I don't disagree that things are more complicated this way. The trade-offs I mention above make it worth it.

The whole reason I shared this information is because my development process has benefited so much from it. I found with my older code, more often than not, that I was wanting something like Unity and didn't have it available, which ended up forcing me to make some bad design choices. Unity has made working on my client much much easier than it used to be.

I'll definitely take a look at the dib_reader when I get around to trying this out on Linux again, it looks really promising! I think it's exactly the solution I was looking for a couple months ago.

---
class EOSERV {
Programmer | Oldbie
Open source EO Client: https://github.com/ethanmoffat/EndlessClient
};

Nice! You made a lot of progress with the first implementation of your client and, while I've developed a bit of a dislike for .NET over the past several years, you've still made more progress than most others who have worked on a client replacement and made the source code publicly available so I don't much care that you're using C# despite all that this entails. I used to use C# for rapidly implementing and experimenting with ideas I've had and it worked well enough haha.

I was working on a client of my own last year before I lost it along with some other projects due to a hard drive failure, and recently (this week) decided to try again. I want to properly support multiple platforms with little to no external dependencies and had a bit of a conundrum on Linux when it came to Endless Online's music where I wanted to maintain the original sound, without resorting to ditching the midi format so I ended up deciding that implementing a small software synth into the game itself would probably be the best route to retain the small file sizes, and sound on platforms like Linux where playing midi files is a huge pain haha.

I'm in the process of moving so I doubt work on my client will progress as quickly as yours might, but I'm making backups on a far greater regularity now so hopefully I won't have another incident and can finish it up this time around haha.

Have you considered i18n support in your client? O:

Good luck with your client; I look forward to seeing how this implementation of your client progresses. ^^

---
【ｅｏｓｅｒｖ―ｉｓａｎａ】

Radical Raccoon posted: (11th May 2016, 04:28 am)

I want to properly support multiple platforms with little to no external dependencies and had a bit of a conundrum on Linux when it came to Endless Online's music where I wanted to maintain the original sound, without resorting to ditching the midi format so I ended up deciding that implementing a small software synth into the game itself would probably be the best route to retain the small file sizes, and sound on platforms like Linux where playing midi files is a huge pain haha.

That's awesome - this is one of those things I'm not sure how I would do porting to Linux since the method I'm currently using for the music relies on Media Player.

Radical Raccoon posted: (11th May 2016, 04:28 am)

Have you considered i18n support in your client? O:

If you're talking about internationalization, no, I'm supporting only the localization method in the current .EDF files to keep my client compatible with the official client's localized strings (spelling mistakes and all)

---
class EOSERV {
Programmer | Oldbie
Open source EO Client: https://github.com/ethanmoffat/EndlessClient
};

ethanmoffat posted: (11th May 2016, 06:36 pm)

Radical Raccoon posted: (11th May 2016, 04:28 am)

I want to properly support multiple platforms with little to no external dependencies and had a bit of a conundrum on Linux when it came to Endless Online's music where I wanted to maintain the original sound, without resorting to ditching the midi format so I ended up deciding that implementing a small software synth into the game itself would probably be the best route to retain the small file sizes, and sound on platforms like Linux where playing midi files is a huge pain haha.

That's awesome - this is one of those things I'm not sure how I would do porting to Linux since the method I'm currently using for the music relies on Media Player.

Radical Raccoon posted: (11th May 2016, 04:28 am)

Have you considered i18n support in your client? O:

If you're talking about internationalization, no, I'm supporting only the localization method in the current .EDF files to keep my client compatible with the official client's localized strings (spelling mistakes and all)

As primitive as it sounds, you could be using mmsystem.h as a means to play those midi files in Windows and Linux, but I am really not sure of the library dependencies needed for it to work with Linux. Vult had used that for the official client and I was planning on doing the same in my client. Just a thought if you aren't game for converting to mp3 or wav files.

Apollo posted: (11th May 2016, 07:29 pm)

ethanmoffat posted: (11th May 2016, 06:36 pm)

Radical Raccoon posted: (11th May 2016, 04:28 am)

I want to properly support multiple platforms with little to no external dependencies and had a bit of a conundrum on Linux when it came to Endless Online's music where I wanted to maintain the original sound, without resorting to ditching the midi format so I ended up deciding that implementing a small software synth into the game itself would probably be the best route to retain the small file sizes, and sound on platforms like Linux where playing midi files is a huge pain haha.

That's awesome - this is one of those things I'm not sure how I would do porting to Linux since the method I'm currently using for the music relies on Media Player.

Radical Raccoon posted: (11th May 2016, 04:28 am)

Have you considered i18n support in your client? O:

If you're talking about internationalization, no, I'm supporting only the localization method in the current .EDF files to keep my client compatible with the official client's localized strings (spelling mistakes and all)

As primitive as it sounds, you could be using mmsystem.h as a means to play those midi files in Windows and Linux, but I am really not sure of the library dependencies needed for it to work with Linux. Vult had used that for the official client and I was planning on doing the same in my client. Just a thought if you aren't game for converting to mp3 or wav files.

That's something I hadn't considered. Background music is pretty low on my list of priorities, but if I do get to that point and have a working version of my client running on Linux I'll give it a try. 

---
class EOSERV {
Programmer | Oldbie
Open source EO Client: https://github.com/ethanmoffat/EndlessClient
};

Apollo posted: (11th May 2016, 07:29 pm)

ethanmoffat posted: (11th May 2016, 06:36 pm)

Radical Raccoon posted: (11th May 2016, 04:28 am)

I want to properly support multiple platforms with little to no external dependencies and had a bit of a conundrum on Linux when it came to Endless Online's music where I wanted to maintain the original sound, without resorting to ditching the midi format so I ended up deciding that implementing a small software synth into the game itself would probably be the best route to retain the small file sizes, and sound on platforms like Linux where playing midi files is a huge pain haha.

That's awesome - this is one of those things I'm not sure how I would do porting to Linux since the method I'm currently using for the music relies on Media Player.

Radical Raccoon posted: (11th May 2016, 04:28 am)

Have you considered i18n support in your client? O:

If you're talking about internationalization, no, I'm supporting only the localization method in the current .EDF files to keep my client compatible with the official client's localized strings (spelling mistakes and all)

As primitive as it sounds, you could be using mmsystem.h as a means to play those midi files in Windows and Linux, but I am really not sure of the library dependencies needed for it to work with Linux. Vult had used that for the official client and I was planning on doing the same in my client. Just a thought if you aren't game for converting to mp3 or wav files.

Problems arise when servers create custom music for the EO client, and I believe mmysystem being utilized in the original client may be the cause of that. Glitched or "stuck" notes and other distortions are frequent and frustrating for composers when trying to create working BGMs that weren't packaged in with EO originally.

http://www.motifator.com/index.php/archive/viewthread/243421/                        <--Example?

---
Want to learn to pixel?
Pixelsource.org

Cirras posted: (12th May 2016, 01:17 am)

Apollo posted: (11th May 2016, 07:29 pm)

ethanmoffat posted: (11th May 2016, 06:36 pm)

Radical Raccoon posted: (11th May 2016, 04:28 am)

I want to properly support multiple platforms with little to no external dependencies and had a bit of a conundrum on Linux when it came to Endless Online's music where I wanted to maintain the original sound, without resorting to ditching the midi format so I ended up deciding that implementing a small software synth into the game itself would probably be the best route to retain the small file sizes, and sound on platforms like Linux where playing midi files is a huge pain haha.

That's awesome - this is one of those things I'm not sure how I would do porting to Linux since the method I'm currently using for the music relies on Media Player.

Radical Raccoon posted: (11th May 2016, 04:28 am)

Have you considered i18n support in your client? O:

If you're talking about internationalization, no, I'm supporting only the localization method in the current .EDF files to keep my client compatible with the official client's localized strings (spelling mistakes and all)

As primitive as it sounds, you could be using mmsystem.h as a means to play those midi files in Windows and Linux, but I am really not sure of the library dependencies needed for it to work with Linux. Vult had used that for the official client and I was planning on doing the same in my client. Just a thought if you aren't game for converting to mp3 or wav files.

Problems arise when servers create custom music for the EO client, and I believe mmysystem being utilized in the original client may be the cause of that. Glitched or "stuck" notes and other distortions are frequent and frustrating for composers when trying to create working BGMs that weren't packaged in with EO originally.

http://www.motifator.com/index.php/archive/viewthread/243421/                        <--Example?

There are likely other factors than mmsystem. MIDI has two formats, most generic programs only store in one of the two. Vult could have some hard-coded MIDI parameters for looping, play length, fading, etc. and you are probably doing a good job exposing that. Also, because a stuck note occurs could be a slow PC or a an inferior sound card that fails at MIDI. I could go on and on, but a header file only has instructions included. Considering that EO still plays on pretty much any modern Windows based computer tells me that it works just fine.

Apollo posted: (12th May 2016, 04:21 pm)

Cirras posted: (12th May 2016, 01:17 am)

Apollo posted: (11th May 2016, 07:29 pm)

ethanmoffat posted: (11th May 2016, 06:36 pm)

Radical Raccoon posted: (11th May 2016, 04:28 am)

I want to properly support multiple platforms with little to no external dependencies and had a bit of a conundrum on Linux when it came to Endless Online's music where I wanted to maintain the original sound, without resorting to ditching the midi format so I ended up deciding that implementing a small software synth into the game itself would probably be the best route to retain the small file sizes, and sound on platforms like Linux where playing midi files is a huge pain haha.

That's awesome - this is one of those things I'm not sure how I would do porting to Linux since the method I'm currently using for the music relies on Media Player.

Radical Raccoon posted: (11th May 2016, 04:28 am)

Have you considered i18n support in your client? O:

If you're talking about internationalization, no, I'm supporting only the localization method in the current .EDF files to keep my client compatible with the official client's localized strings (spelling mistakes and all)

As primitive as it sounds, you could be using mmsystem.h as a means to play those midi files in Windows and Linux, but I am really not sure of the library dependencies needed for it to work with Linux. Vult had used that for the official client and I was planning on doing the same in my client. Just a thought if you aren't game for converting to mp3 or wav files.

Problems arise when servers create custom music for the EO client, and I believe mmysystem being utilized in the original client may be the cause of that. Glitched or "stuck" notes and other distortions are frequent and frustrating for composers when trying to create working BGMs that weren't packaged in with EO originally.

http://www.motifator.com/index.php/archive/viewthread/243421/                        <--Example?

There are likely other factors than mmsystem. MIDI has two formats, most generic programs only store in one of the two. Vult could have some hard-coded MIDI parameters for looping, play length, fading, etc. and you are probably doing a good job exposing that. Also, because a stuck note occurs could be a slow PC or a an inferior sound card that fails at MIDI. I could go on and on, but a header file only has instructions included. Considering that EO still plays on pretty much any modern Windows based computer tells me that it works just fine.

Note that I said it was only in cases where new music was being created - It's a very hit or miss cluster of bugs. The original EO music is unaffected by this issue though.

---
Want to learn to pixel?
Pixelsource.org

Any progress? Hope all is going well!

xsepx posted: (8th Nov 2016, 04:23 pm)

Any progress? Hope all is going well!

I haven't made too much progress lately. Work is burning me out...it's much easier to come home and play video games or watch netflix than work on the client :P

I've got a couple local commits related to player movement that I haven't pushed to github yet. I am pretty happy with how the code is now though, its much cleaner than it used to be.

---
class EOSERV {
Programmer | Oldbie
Open source EO Client: https://github.com/ethanmoffat/EndlessClient
};

ethanmoffat posted: (9th Nov 2016, 07:24 pm)

xsepx posted: (8th Nov 2016, 04:23 pm)

Any progress? Hope all is going well!

I haven't made too much progress lately. Work is burning me out...it's much easier to come home and play video games or watch netflix than work on the client :P

I've got a couple local commits related to player movement that I haven't pushed to github yet. I am pretty happy with how the code is now though, its much cleaner than it used to be.

I can see that, it looks great.

Personally, still checking your github pretty frequently. Looking forward to all the big changes that will make the new code base playable.

---
Want to learn to pixel?
Pixelsource.org