HomeForumSourceResearchGuide
Sign in to reply to forum posts.
Network packet formatting

hi, I'm writing a simple custom network protocol between a client and a server using TCP, and I wanted to check on the best way to do this. I want to use a binary protocol for efficiency, and I think I can directly serialise a data instance to use as a packet header.

I basically want to have a header and then a payload, where the header tells me how big the payload is and has some other flags like message type.

What's the easiest way to do this please?

jess

Hi Jess,

You're right that you can directly serialise data instances for use as packet headers. You can do this as long as the data type only contains primitive types, including fixed-length arrays of primitive types (so no reference types).

Let's imagine that we define the data type:

data Header {
	const int2 MAGIC = 1234
	int2 magic
	int4 type
	int4 payload
	}

I've used specific-size integer types here so that the source code has maximum cross-platform compatibility.

I've included a "magic" field which we might use to identify that this is a packet from our protocol; a "type" field which we might use to decide which packet type this is; and a "payload" field which we can use to indicate the size of the data payload which follows the header.

On the client side we might then do this (assuming our server is running on local host):

TCPSocket s = new TCPSocket()
s.connect("127.0.0.1", 8090)

char payload[] = "hello!"

Header header = new Header(Header.MAGIC, 1, payload.arrayLength)

byte stream[] = dana.serial(header)

s.send(stream)
s.send(payload)

...and on the server side we could do:

TCPServerSocket server = new TCPServerSocket()
server.bind("127.0.0.1", 8090)
TCPSocket s
Header header = new Header()
byte stream[] = dana.serial(header)

while (s.accept(server))
   {
   byte buf[] = s.recv(stream.arrayLegth)
   stream =[] buf
	
   //check we actually received enough bytes for a valid header, and check the protocol magic matches
   if (buf.arrayLength == stream.arrayLength && header.magic == Header.MAGIC)
      {
      //now receive the payload, according the number of payload bytes indicated in the header
      buf = s.recv(header.payload)
		
      //check we received the expected number of payload bytes, then dispatch to a handler function
      if (buf.arrayLength == header.payload)
         {
         if (header.type == 1)
            {
            handleHelloMessage(buf)
            }
            else if (header.type == 2)
            {
            handleOtherMessage(buf)
            }
            //etc...
         }
      }
   s.disconnect()
   }

Here we accept new connections from clients, then receive a header and check it's valid, then receive as many payload bytes as the header indicated. We might then write a separate function to handle each message type. This is obviously a very simple example and in practice you would probably want some concurrency approach on the server to handle parallel clients.

The =[] is fairly uncommon syntax, and means "copy the cells of array B into array A". In this case it copies the received header bytes into the header instance directly, because the serial stream array is a shallow reference to the memory of the header instance; we can then use the header's fields directly instead of querying the received bytes.

If you're aiming for interoperability with existing network protocols, note that Dana integers are network big endian across all platforms.

Hope that helps!

Barry

Oh thanks for the super quick reply! That makes sense and is much simpler than I thought, I'll use your example as my starting point :-)

jess