The Minecraft Server protocol has been documented by the community since 2009, there are a wide variety of custom servers in existence today, for example Myne which is open source: https://bitbucket.org/andrewgodwin/myne2/src