All about programming in GNU/LINUX

Golang

Http/2 and implementation in GoLang-1: Http/2 Frameheaders and parsing HTTP/2 Frameheaders in Golang

Http/2 is around the corner and its specifications look really interesting ! The complete specification of Http/2 can be seen at this following link https://http2.github.io/http2-spec/ .

In this post ill be discussing about Http/2 FrameHeader and lets figure out the binary math done to parse it in GoLang .
The complete Http/2 implementation of Golang is available at https://github.com/bradfitz/http2 .

Now Lets Look at the structure of HTTP Frameheader from its spec

+———————————————–+
| Length (24) |
+—————+—————+—————+
| Type (8) | Flags (8) |
+-+————-+—————+——————————-+
|R| Stream Identifier (31) |
+=+=============================================================+
| Frame Payload (0…) …
+—————————————————————+

The fields of the frame header are defined as:

Length:
The length of the frame payload expressed as an unsigned 24-bit integer. Values greater than 214 (16,384) MUST NOT be sent unless the receiver has set a larger value for SETTINGS_MAX_FRAME_SIZE.

The 9 octets of the frame header are not included in this value.

Type:
The 8-bit type of the frame. The frame type determines the format and semantics of the frame. Implementations MUST ignore and discard any frame that has a type that is unknown.

Flags:
An 8-bit field reserved for frame-type specific boolean flags.

Flags are assigned semantics specific to the indicated frame type. Flags that have no defined semantics for a particular frame type MUST be ignored, and MUST be left unset (0x0) when sending.

R:
A reserved 1-bit field. The semantics of this bit are undefined and the bit MUST remain unset (0x0) when sending and MUST be ignored when receiving.

Stream Identifier:
A stream identifier expressed as an unsigned 31-bit integer. The value 0x0 is reserved for frames that are associated with the connection as a whole as opposed to an individual stream.

Now let us look at the implementation of Parsing Http/2 FrameHeader in GoLang
Let us look at the function readFrameHeader https://github.com/bradfitz/http2/blob/master/frame.go . Here is the code

func readFrameHeader(buf []byte, r io.Reader) (FrameHeader, error) {
	_, err := io.ReadFull(r, buf[:frameHeaderLen])
	if err != nil {
		return FrameHeader{}, err
	}
	return FrameHeader{
		Length:   (uint32(buf[0])<<16 | uint32(buf[1])<<8 | uint32(buf[2])),
		Type:     FrameType(buf[3]),
		Flags:    Flags(buf[4]),
		StreamID: binary.BigEndian.Uint32(buf[5:]) & (1<<31 - 1),
		valid:    true,
	}, nil
}

Looks like Greek isn’t it 😛 Lets do some binary math to understanding the parsing of the buffer containing the HTTP/2 Frame .
From the structure of the HTTP/2 Frameheader the first 24 bytes contains the Length of the payload , but since the Http Frame along with its data is obtained in the form of a byte Array ([]byte) , the header now has to be parsed from the same .Since each element in an byte array is a byte, it can only hold 8 bit of data . So now the 24 bit Payload length is spread across 3 consecutive members of the byte array ranging from buf[0] to buf[2] .

Now the challenge is to parse these 3 elements of the array into one integer number . But first lets see how in the first place a 24 bit integer is made to fit into a byte array

Let do a small exercise to understand this .
Let take an random integer , hmm, lets consider the integer 99999 and convert this to binary and present it in 24 bit

99999 = 00000001 10000110 10011111

Here is its binary representation again and how it is stuffed as bytes inside byte array

00000001 10000110 10011111
| | |
| | |
buf[0] buf[1] buf[2]

Oh now i know how in first place how this 24 bit integer is stuffed inside byte array , but do i now parse it back into an integer

Very Simple , Here is how an 0 valued 32 bit integer looks like

00000000 00000000 00000000 00000000

First take buf[0] and place it at the 3rd byte from left of the 32 bit integer and make it look like this

00000000 00000001 00000000 00000000

How to do that ???

Simple …. First convert buf[0] into a 32 bit int

(uint32(buf[0])

)
After this the 32 bit integer look like this

00000000 00000000 00000000 000000001

Thhhhhh…!! But content of buf[0] is supposed to be in 3rd byte position but its now 1 byte position !

Okay , now just left shift the integer by two bytes (16 bits) , after this the content of buf[0] will be moved to the 3rd byte position .

 (uint32(buf[0])<<16 

)
Now in the same fashion , you gotto generate a 32 bit integer with the content of buf[1] in the 2nd byte position . This is the done by now left shifting the integer by a byte (8 bits) .

uint32(buf[1])<<8

and at the end just convert buf[2] into a 32 bit integer .

Now just perform a logical OR (|) on these 3 integers which will fetch one integer representing the 24 bit Frame Payload Length parsed .

After first 24 bits in the Http Frameheader the next 8 bits correspond to the FrameType . This can be easy retrieved by converting buf[3] into an integer

uint32(buf[3])

)

The next 8 bits correspond to the Flags . This can be easy retrieved by converting buf[3] into an integer

uint32(buf[4])

)

As you can see in the FrameHeader image the next 1 bit is a Reserved bit and 31 bit following it is the Stream ID .

Now the Last 31 bit has to be parsed from the buffer to be able to obtain the stream ID . To be able to do it first fast last 32 bits a 32 bit int

uint32(buf[5])<<24| ]uint32(buf[5]) <<16 | ]uint32(buf[5]) << 8| ]uint32(buf[5])

)

or there is an equivalent function from the package binary, which does the above functionality

 binary.BigEndian.Uint32(buf[5:]) 

)
The following link explains more about binary.BigEndian.Uint32 , http://stackoverflow.com/questions/7380158/how-to-convert-4uint8-into-uint32-in-go

Now the challenge is to turn the 32nd bit to 0 so that the 31 bit Stream ID is retrieved .

But how to achieve it ??
Well , easy . Take a number which in binary is equivalent to 0 followed by 31 1’s and then perform a logical AND (&) operation with the number on which you want to strip out or set the 32nd bit of the number to be zero .

First generate an integer which in binary representation is equivalent to 0 followed by 31 1’s

(1<<31 - 1)

)

This is how at the end last 31 bit representing the Stream ID is obtained

binary.BigEndian.Uint32(buf[5:]) & (1<<31 - 1)

)

Here is the source from https://github.com/bradfitz/http2 , The GoLang Http2 implementation which corresponds to parsing of HTTP2 FrameHeader and i hope after all the explanation the source should make some sense 😛

func readFrameHeader(buf []byte, r io.Reader) (FrameHeader, error) {
	_, err := io.ReadFull(r, buf[:frameHeaderLen])
	if err != nil {
		return FrameHeader{}, err
	}
	return FrameHeader{
		Length:   (uint32(buf[0])<<16 | uint32(buf[1])<<8 | uint32(buf[2])),
		Type:     FrameType(buf[3]),
		Flags:    Flags(buf[4]),
		StreamID: binary.BigEndian.Uint32(buf[5:]) & (1<<31 - 1),
		valid:    true,
	}, nil
}

)a

ssssssssssssssss …..Thats it for now !!! I hope this helped you guys to understand the structure of parsing of Http2 . Will follow up with lots of blogs on Http2 implementation in GoLang..Till then , Happy Coding 😀