Delta Engine Blog

All about multiplatform and game development

How to make server network connections faster than 1gbit

cern data storage

With our company growing and growing we are exploring some new options for faster networking. Sadly 10gbit networking is still way too expensive, network cards, switches and even cables are insanely expensive. It is 20-50 times more expensive than just using cheap 1gbit hardware, especially since every computer has a 1gbit network card on the mainboard anyway.

Since our new server has a pretty big and fast raid we do not want access limited by just 1gbit. Our new raid can read and write data with up to 1 GB/s, which would require a 10gbit network to do this over the network. But since that is too expensive, it is out of the question.

In case you are interested, here are some pics from the old and the new server:

New server

This is the old server (actually one of two), which just had a small Raid-5 plus some extra hard disks. The server was mostly used for backups. It was an older Core2Duo with 2TB storage and we used it not as an file server so much (that's what the other server did), but for database, websites, services and mostly backups.


New server

The new server is not longer just a desktop PC, but a 4HE server (without a server rack) with two Xeon Quad-CPUs (i7 920 equivalents) with 24 GB Ram and a really nice Areca Raid controller for 16+4 hard discs of each 1.5 TB (total=24TB) plus another raid with 3 TB for backups. Not all of that space is available since the 20 hard discs are in a Raid60, but we got plenty of free space. We use this server now for pretty much everything, development, file server, all the websites (including this blog), the build server, the content server, many multiplayer servers, lots of services and much much more.


New server

Another picture of the inside of the new server. Please note that the second raid controller is not connected yet and both the SSD and 4 more Hdds are missing. I build most of the cables myself from stuff lying around from other computers. I also added some more fans because the raid controllers apparently do not like heat so much and the server is pretty loud anyway.

To archive more than 1gbit to and from the server (which is definitely the bottleneck) we are using a different trick, which is called Link Aggregation, also known as trunking, NIC teaming, link bundling, etc. It is usually the easiest way to find support for this technique by searching for the 802.3ad standard, which most switches and multi-network cards support nowadays. The problem is to find a really good switch that can actually handle the workload of many clients plus 4 1gbit cables to the server, which also needs a really good 4x 1gbit network card. One major problem while doing network cards and switch research is the incredible annoying naming and abbreviation all those devices have. Only after a while you find out what LR, SR, LX, TRB, etc. and all the 802 standards mean. After some suggestions from my brother and lots of research (it is not an easy topic if you only play system and network admin once a year ^^) I decided to get a nice Netgear switch with 48 ports (called GS748T) and a powerful Quad Port Server Adapter from Intel. 24 ports plus some other switches we have lying around would also be enough but the only switch I could get easily now was that one and it was only 25% more expensive (it also has a few more cool features). This way I have enough ports to do more trunking in the future (lots of ports will be wasted by connecting switches, servers or even my computer).

Okay, so how does this work now? Let me explain with a little diagram I just drew (the hardware components actually look this way, see above for details):


New server architheture

As you can hopefully see from my amazing drawing skills (now that real graphic artists are blogging on this blog, this is probably not funny anymore) each client is still connected through a 1gbit network card and cable to the new switch. The older switches we had were kinda week and some people could not really enjoy the full speed, either because we cascaded to many 8port switches or because some of them were just overwhelmed by the current load. Now the network is much better and we will do some stress tests this week and since most people are trying to access the server (or one of the internet connections behind the server), much more bandwidth from and to the server can now be provided with help of the Link Aggregation (802.3ad). In fact if 4 people are downloading some huge files from the server simultaneously, each of them can still enjoy up to 120MB/s of data while some other people could write something at the same time. The Raid Controller will probably brake down earlier since it can only provide 1000MB/s when reading or writing sequentially, but this setup should be 10-40 times faster than what we used in the past years. I will blog more next year about the engine, all da cool games and why we need all this server stuff in the near future.


External links:

Comments (39) -

  • buy laptop

    2/15/2010 8:19:58 AM |

    Why didn’t I find this post earlier? Keep up the good work!

  • SEO

    2/16/2010 6:38:06 AM |

    Brilliant post mate, keep up the good work J

  • full tilt poker

    2/17/2010 10:07:08 AM |

    Hi,



    Thanks for the post. Keep the great work.

  • buy laptop

    2/23/2010 12:09:06 PM |

    Top post. I look forward to reading more. Cheers

  • Poker

    2/25/2010 7:49:34 AM |

    Hi,


    Nice job, it’s a great post. The info is good to know!

  • cheap printer ink

    3/3/2010 9:09:41 AM |

    Very useful info. Hope to see more posts soon!

  • salon pos software

    3/4/2010 6:40:30 AM |

    Excellent post.I want to thank you for this informative read, I really appreciate sharing this great post. Keep up your work...

  • Online Poker

    3/4/2010 6:43:06 AM |

    Hi,


    Sensational info. I look forward to seeing more.

  • business consultants

    3/4/2010 8:02:19 AM |

    I’m impressed, you know what you’re talking about...

  • bamboo floors

    3/4/2010 1:22:19 PM |

    I’m impressed, you know what you’re talking about.....

  • blinds melbourne

    3/5/2010 7:11:40 AM |

    Nice post.this is very nice one and gives in depth information. thanks for this nice article

  • redken

    3/5/2010 8:48:28 AM |

    Insightful piece, thanks a lot!

  • Poker

    3/8/2010 7:09:23 AM |

    Hi,



    Why didn’t I find this post earlier? Keep up the good work!

  • compare odds 

    3/8/2010 1:18:13 PM |

    Hi,


    Really appreciate this post. It’s hard to sort the good from the bad sometimes, but I think you’ve nailed it!

  • accommodation in milan

    3/10/2010 11:06:21 AM |

    I will keep visiting this blog very often.It is good to see you verbalise from the heart and your clarity on this important subject can be easily observed. Accommodation available in Milan city centre and surroundings area of Milan. Secure and instant bookings!

  • monitor stands

    3/11/2010 2:40:39 PM |

    Useful info. Hope to see more good posts in the future.

  • dvd packaging

    3/20/2010 8:09:10 AM |

    It’s hard to find knowledgeable people on this topic, but you sound like you know what you’re talking about! Thanks

  • Debt solutions

    3/20/2010 10:45:16 AM |

    This is a really good read for me, Must admit that you are one of the best bloggers I ever saw..People having problems regarding Debts can visit this site..You may get the professional solutions to your financial situations.Have a glance at this and satisfy with the services.

  • new york printing

    3/26/2010 7:01:56 AM |

    Very useful info. Hope to see more posts soon!

  • tennis booking system

    3/29/2010 6:58:22 AM |

    It’s hard to find knowledgeable people on this topic, but you sound like you know what you’re talking about! Thanks

  • carpet cleaning

    4/1/2010 2:03:05 PM |

    It’s hard to find knowledgeable people on this topic, but you sound like you know what you’re talking about! Thanks

  • promotional marketing

    4/6/2010 9:19:30 AM |

    Hi,
      Awesome tips. I’ll be passing this post on for sure

  • iphone applications

    4/6/2010 9:20:09 AM |

    Hi,
      Awesome tips. I’ll be passing this post on for sure

  • Mine Panama

    4/6/2010 11:16:29 PM |

    nice engineering. The content is very well placed as well.

  • laptop sale     

    4/9/2010 7:08:42 AM |

    good work…unique site and interesting too… keep it up…i am looking forward for more updates.

  • buy camera

    4/13/2010 8:47:38 AM |

    Don’t stop writing, you’ve given me lots of good info!

  • camera shop sydney

    4/13/2010 8:48:55 AM |

    Cheers to the author for giving me some solid ideas

  • adelaide web design

    4/16/2010 6:35:02 AM |

    hi........
    I feel a lot more people need to read this, very good info!

  • on site otpimization

    4/25/2010 11:48:02 AM |

    This article gives the light in which we can observe the reality. this is very nice one and gives indepth information. thanks for this nice article

  • learning and assessments

    4/26/2010 9:43:40 AM |

    What software is good to synthesis rock and metal music with real sounding virtual instruments?

  • Prudential West

    4/30/2010 1:22:32 PM |

    Great post. I have learned a lot. I am grateful to my friend who told me to visit your blog. Thanks a lot!

  • 3 ring binders

    5/3/2010 5:20:31 PM |

    but, i think it has some risk factors, so before doing so one needs to consult with a server expert.

  • oil and gas construction

    5/4/2010 2:48:21 PM |

    you are right that we have to find a advanced Rapidhsare search engine to take what we need at this type of websites.but its good to know you are making us aware about it..Thanks

  • Adidas Superstar

    5/15/2010 8:25:34 PM |

    Available at adidas-superstar.com Huge collection of over 200 urban, vintage adidas superstar and rare adidas superstar shoes. One of the worlds largest online retailers with lightning fast worldwide shipping.

Comments are closed