HMAC-SHA-256 from Scratch in Go

Karol Moroz on

HMAC, or Keyed-Hash Message Authentication Code algorithm, is a cornerstone of modern cryptography. First standardized in 1997 as RFC 2104, it now secures software that billions of people rely on in everyday life.

To name a few examples: TLS version 1.3 uses HMAC to authenticate handshakes, and, via HKDF, to derive encryption keys. HMAC is also widely used in JSON Web Tokens (JWT). In fact, HMAC with SHA-256 (HS256) is the only signing algorithm that a JWT implementation MUST implement, as per RFC 7519.

Last but not least, TOTP, or the Time-based One-Time Password algorithm, uses HMAC (traditionally with the older SHA-1 hash function) to generate single-use numeric codes for two-factor authentication. You may have used TOTP with Google Authenticator, Microsoft Authenticator, or similar mobile apps. For an in-depth explanation of TOTP, refer to my YouTube video Implementing 2FA Codes (TOTP) in C.

It is with deepest regret that I admit that, until today, I have never truly understood how HMAC worked. I have used it in numerous ways, and I have even tried to read through the RFC one time, but the true meaning of all of these arcane formulæ has eluded me for many years.

There is no better way to understand an algorithm than to implement it yourself, therefore in this post I’m going to implement HMAC-SHA-256 in Go.

You can find the final result of the tutorial on my GitHub account: in the repository moroz/hmac-go.

Formal definition

The general definition of HMAC, as stated in RFC 2104, is as follows:

H(KXORopad,H(KXORipad, 𝑡𝑒𝑥𝑡))

In this formula, H represents the hash function (in our case, SHA-256) and K represents the secret key, which is a binary string of arbitrary length. XOR refers to the bitwise exclusive OR operation, and is explained in depth at a further point in this article. The symbols ipad and opad are constants. In the case of HMAC-SHA-256, ipad is the byte 0x36, repeated 64 times; opad is the byte 0x5C, repeated 64 times. The variable 𝑡𝑒𝑥𝑡 refers to the message being authenticated.

One way to break this cryptic formula into human-digestible morsels of pseudocode would be as follows:

  1. 𝑖𝑛𝑛𝑒𝑟𝐾𝑒𝑦←K0XORipad
    Pad the secret key K with zeroes at the end, so that it is exactly 64 bytes long, and call the result K0. If K is longer than 64 bytes, hash the key with SHA-256 and pad the resulting hash digest rather than the original key. Apply the exclusive OR (XOR) operation on K0 and the constant ipad (the byte 0x36 repeated 64 times), and store the result in the variable innerKey.

  2. 𝑖𝑛𝑛𝑒𝑟𝐻𝑎𝑠ℎ←H(𝑖𝑛𝑛𝑒𝑟𝐾𝑒𝑦‖𝑡𝑒𝑥𝑡)
    Hash innerKey, concatenated with the message, and store the resulting hash in the variable innerHash.

  3. 𝑜𝑢𝑡𝑒𝑟𝐾𝑒𝑦←K0XORopad
    Apply the exclusive OR operation on K0 and the constant opad (the byte 0x5C repeated 64 times), and store the result in the variable outerKey.

  4. ℎ𝑚𝑎𝑐←H(𝑜𝑢𝑡𝑒𝑟𝐾𝑒𝑦‖𝑖𝑛𝑛𝑒𝑟𝐻𝑎𝑠ℎ)
    Hash outerKey, concatenated with innerHash. The resulting binary string is the HMAC digest.

Before converting these steps into Go code, we need to prepare a project.

Project setup

Let’s start by creating a Go module called myhmac:

$ mkdir myhmac
$ cd myhmac
$ go mod init myhmac

It’s a good habit to keep your source files under version control. Initialize a Git repository:

$ git init -b main
$ git add .
$ git commit -m "Initial commit"

In the file myhmac.go, define a package and a single exported function. For now, the implementation will just return nil, which is a valid return value for []byte.

package myhmac

func SHA256(key, message []byte) []byte {
	return nil
}

Set up a test harness

Next, let’s set up a test suite. When implementing HMAC in Go, the natural choice of test data is RFC 4231: Identifiers and Test Vectors for HMAC-SHA-224, HMAC-SHA-256, HMAC-SHA-384, and HMAC-SHA-512. In particular, section 4 lists several test cases for HMAC-SHA-256, HMAC-SHA-512, and variants thereof.

We will use only the test vectors relevant to the standard usage of HMAC-SHA-256, which excludes Test Case 5 (Test with a truncation of output to 128 bits).

In the file myhmac_test.go, define a package and a struct type describing each test example:

package myhmac_test

type example struct {
	// name is a human-readable name of the test example.
	name string

	// key is the HMAC key (K), hex-encoded.
	key string

	// text is the message to be authenticated, hex-encoded.
	text string

	// expected is the correct return value of the
	// HMAC-SHA-256 function, hex-encoded.
	expected string
}

Next, add the examples in a package-level variable:

var examples = []example{
	{
		name: "Test Case 1",
		key:  strings.Repeat("0b", 20),
		text: "4869205468657265", // "Hi There"
		expected: "b0344c61d8db38535ca8afceaf0bf12b" +
			"881dc200c9833da726e9376c2e32cff7",
	},
	{
		name: "Test Case 2 (short key)",
		key:  "4a656665", // "Jefe"
		text: "7768617420646f2079612077616e7420" + // "what do ya want "
			"666f72206e6f7468696e673f", // "for nothing?"
		expected: "5bdcc146bf60754e6a042426089575c7" +
			"5a003f089d2739839dec58b964ec3843",
	},
	{
		name: "Test Case 3 (combined length of key and data > 64 bytes)",
		key:  strings.Repeat("aa", 20),
		text: strings.Repeat("dd", 50),
		expected: "773ea91e36800e46854db8ebd09181a7" +
			"2959098b3ef8c122d9635514ced565fe",
	},
	{
		name: "Test Case 4 (combined length of key and data > 64 bytes)",
		key: "0102030405060708090a0b0c0d0e0f10" +
			"111213141516171819",
		text: strings.Repeat("cd", 50),
		expected: "82558a389a443c0ea4cc819899f2083a" +
			"85f0faa3e578f8077a2e3ff46729665b",
	},
	{
		name: "Test Case 6 (key > 64 bytes)",
		key:  strings.Repeat("aa", 131),
		text: "54657374205573696e67204c61726765" +
			"72205468616e20426c6f636b2d53697a" +
			"65204b6579202d2048617368204b6579" +
			"204669727374",
		expected: "60e431591ee0b67f0d8a26aacbf5b77f" +
			"8e0bc6213728c5140546040f0ee37f54",
	},
	{
		name: "Test Case 7 (key and data both > 128 bytes)",
		key:  strings.Repeat("aa", 131),
		text: "54686973206973206120746573742075" +
			"73696e672061206c6172676572207468" +
			"616e20626c6f636b2d73697a65206b65" +
			"7920616e642061206c61726765722074" +
			"68616e20626c6f636b2d73697a652064" +
			"6174612e20546865206b6579206e6565" +
			"647320746f2062652068617368656420" +
			"6265666f7265206265696e6720757365" +
			"642062792074686520484d414320616c" +
			"676f726974686d2e",
		expected: "9b09ffa71b942fcb27635fbcd5b0e944" +
			"bfdc63644f0713938a7f51535c3a35e2",
	},
}

Add a helper function to decode hex-encoded strings into byte slices. It’s going to come in handy with all the different struct fields:

func mustDecodeHex(data string) []byte {
	bytes, err := hex.DecodeString(data)
	if err != nil {
		panic(err)
	}
	return bytes
}

Finally, we can add the tests themselves. First, run the test vectors against the standard library implementation from crypto/hmac, so we can make sure the tests are correct:

func TestStandardLibraryBaseline(t *testing.T) {
	for _, example := range examples {
		t.Run(example.name, func(t *testing.T) {
			expected := mustDecodeHex(example.expected)

			mac := hmac.New(sha256.New, mustDecodeHex(example.key))
			mac.Write(mustDecodeHex(example.text))
			actual := mac.Sum(nil)

			// hmac.Equal compares the two slices in constant time
			// to prevent timing attacks.
			if !hmac.Equal(actual, expected) {
				t.Errorf(`Expected "%x", got "%x"`, expected, actual)
			}
		})
	}
}

At this point, all tests should pass, as we’re only testing the standard library:

$ go test .
ok      myhmac  0.002s

Next, let’s test our own implementation:

func TestSHA256(t *testing.T) {
	for _, example := range examples {
		t.Run(example.name, func(t *testing.T) {
			key := mustDecodeHex(example.key)
			text := mustDecodeHex(example.text)
			expected := mustDecodeHex(example.expected)

			actual := myhmac.SHA256(key, text)

			if !hmac.Equal(actual, expected) {
				t.Errorf(`Expected "%x", got "%x"`, expected, actual)
			}
		})
	}
}

All the examples for this test should fail, because our implementation currently only returns nil:

$ go test .
--- FAIL: TestSHA256 (0.00s)
    --- FAIL: TestSHA256/Test_Case_1 (0.00s)
        myhmac_test.go:122: Expected "b0344c61d8db38535ca8afceaf0bf12b881dc200c9833da726e9376c2e32cff7", got ""
    --- FAIL: TestSHA256/Test_Case_2_(short_key) (0.00s)
        myhmac_test.go:122: Expected "5bdcc146bf60754e6a042426089575c75a003f089d2739839dec58b964ec3843", got ""
    --- FAIL: TestSHA256/Test_Case_3_(combined_length_of_key_and_data_>_64_bytes) (0.00s)
        myhmac_test.go:122: Expected "773ea91e36800e46854db8ebd09181a72959098b3ef8c122d9635514ced565fe", got ""
    --- FAIL: TestSHA256/Test_Case_4_(combined_length_of_key_and_data_>_64_bytes) (0.00s)
        myhmac_test.go:122: Expected "82558a389a443c0ea4cc819899f2083a85f0faa3e578f8077a2e3ff46729665b", got ""
    --- FAIL: TestSHA256/Test_Case_6_(key_>_64_bytes) (0.00s)
        myhmac_test.go:122: Expected "60e431591ee0b67f0d8a26aacbf5b77f8e0bc6213728c5140546040f0ee37f54", got ""
    --- FAIL: TestSHA256/Test_Case_7_(key_and_data_both_>_128_bytes) (0.00s)
        myhmac_test.go:122: Expected "9b09ffa71b942fcb27635fbcd5b0e944bfdc63644f0713938a7f51535c3a35e2", got ""
FAIL
FAIL    myhmac  0.002s
FAIL

We will fix this soon by implementing the myhmac.SHA256 function. In the meantime, commit the changes to version control:

$ git add .
$ git commit -m "Set up library API and test harness"

Next, let us briefly discuss the exclusive OR (XOR) operation.

How exclusive OR (XOR) works

XOR is the bitwise exclusive OR operation. On the level of single bits, it returns 1 if and only if its two operands are not equal, and 0 otherwise. The following interactive figure displays this principle in practice:

1 XOR 0 = 1

ABA XOR B
000
011
101
110
Exclusive OR of two bits. Click to toggle the operands A and B.

On the level of a byte, the same operation is applied to each pair of bits:

A A = 0x00
B B = 0x5C
A XOR B = 0x5C

0x00 XOR 0x5C = 0x5C, binary 01011100

Bitwise XOR of two bytes. Toggle the bits of A and B, or pick a preset.

If you XOR a value with itself, you always end up with 0, because every bit in both operands is the same on either side:

AXORA=0

Any value XOR-ed with 0, on the other hand, returns the same value unchanged:

AXOR0=A

Like addition and multiplication, XOR is commutative:

AXORB=BXORA

Like addition and multiplication, XOR is associative, meaning that the order in which you chain the operations does not matter:

(AXORB)XORC=AXOR(BXORC)

This set of properties makes it possible to use XOR as a sort of cryptographic binary mask (the one-time pad technique). Assuming the secret message is the binary A, and that B stands for a binary mask that is random, secret, and at least as long as the message, we can compute the ciphertext as AXORB. Applying the same mask again returns the original secret message:

(AXORB)XORB=AXOR(BXORB)=AXOR0=A

This property is widely used in AEAD constructions, such as AES-GCM and ChaCha20-Poly1305.

Side note: XOR is the most concise and fastest way to zero a register in x86 assembly: xor eax, eax takes 2 bytes in Intel machine code, while mov eax, 0 takes a whopping 5 bytes. On ARM processors, the idiom works as well, but since all instructions in ARM machine code have the same length, the XOR idiom has no advantage over a move.

Constant-time comparison

The XOR operation is also used to compare binary strings in constant time: if every bit in two strings is equal, the XOR operation of all of them should return 0. The function hmac.Equal uses this technique under the hood to compare HMAC digests:

// src/crypto/internal/fips140/subtle/constant_time.go
// (standard library, reflowed for æsthetic purposes)
package subtle

// ConstantTimeCompare returns 1 if the two slices, x and y, have equal
// contents and 0 otherwise. The time taken is a function of the length
// of the slices and is independent of the contents. If the lengths of
// x and y do not match it returns 0 immediately.
func ConstantTimeCompare(x, y []byte) int {
	if len(x) != len(y) {
		return 0
	}

	var v byte

	for i := 0; i < len(x); i++ {
		v |= x[i] ^ y[i]
	}

	return constanttime.ByteEq(v, 0)
}

With the XOR primer out of the way, we can proceed to pad the key.

Padding the key

This section explains the steps necessary to calculate the inner and outer padding of the key. For both the inner and the outer padding, the steps are exactly the same, except for the constants ipad and opad that the key is combined with.

  1. If the key is longer than 64 bytes, hash it with SHA-256 and use the resulting digest in place of the key.
  2. Pad the key with zeroes at the end so that it is exactly 64 bytes long.
  3. XOR each byte of the padded key with either the ipad or the opad constant, depending on whether you are calculating the inner or the outer padding.

Expressed in Go, the function could look like this:

func padKey(key []byte, mask byte) []byte {
	buffer := make([]byte, sha256.BlockSize)

	if len(key) > sha256.BlockSize {
		hashed := sha256.Sum256(key)
		key = hashed[:]
	}

	copy(buffer, key)

	for i := range buffer {
		buffer[i] ^= mask
	}

	return buffer
}

Let’s dissect this function.

Function head

padKey takes a key and a padding mask, and returns a byte slice:

func padKey(key []byte, mask byte) []byte

The argument key is a byte slice: it has no fixed length. The argument mask is a single byte. Technically, ipad and opad are defined as 64-byte strings, but since all 64 bytes are identical, we only need a single byte to apply the XOR operation correctly. The returned value is a byte slice: we know it’s going to be exactly 64 bytes long, but we also know it will be concatenated with byte slices; therefore, returning a byte slice is the simpler choice.

Allocate a buffer

Within the function body we first allocate a buffer exactly as long as a SHA-256 block — 64 bytes:

buffer := make([]byte, sha256.BlockSize) // 64 bytes

The RFC specifies that the key should be padded with zeroes at the end. Go takes care of that by automatically filling the buffer with zeroes. In C, you would need to allocate the buffer with calloc(3p) rather than malloc(3p), or call memset(3p) after allocating the buffer.

Maybe hash the key

If the key is longer than the block size, we need to hash it and store the digest in place of the original key:

if len(key) > sha256.BlockSize {
	hashed := sha256.Sum256(key)
	key = hashed[:]
}

sha256.Sum256 returns [32]byte, a fixed-size array rather than a byte slice; therefore, we need to convert it using the slice expression hashed[:].

You might be tempted to try and inline the expression as sha256.Sum256(key)[:]. Alas, this does not compile: in Go, the return value of a function is not addressable, and as such cannot be sliced. To think that I chose Go over Rust because I do not care about memory safety…

Copy and XOR

Copy the key to the buffer. Note that the destination comes first and the source second:

copy(buffer, key)

XOR each byte in the buffer with the mask value. For inner padding, it’s 0x36 (0b00110110). For outer padding, it’s 0x5C (0b01011100).

for i := range buffer {
	buffer[i] ^= mask
}

The ^= operator is a shorthand form, inherited from C. We could also write the block body like so:

buffer[i] = buffer[i] ^ mask

Finally, return the buffer:

return buffer

With the padding algorithm explained, we can attempt to implement the rest of the HMAC-SHA-256 function.

Putting everything together

After all we’ve gone through, the final implementation of SHA256 is anticlimactically simple:

const (
	ipad = 0x36
	opad = 0x5C
)

func SHA256(key, message []byte) []byte {
	innerKey := padKey(key, ipad)
	outerKey := padKey(key, opad)
	innerHash := sha256.Sum256(
		slices.Concat(innerKey, message),
	)
	outerHash := sha256.Sum256(
		slices.Concat(outerKey, innerHash[:]),
	)

	return outerHash[:]
}

Let’s walk through the new code quickly. First, we calculate the inner and outer padding of K0:

innerKey := padKey(key, ipad)
outerKey := padKey(key, opad)

Calculate the hash of the innerKey concatenated with the message:

innerHash := sha256.Sum256(
	slices.Concat(innerKey, message),
)

Calculate the hash of the outerKey concatenated with the innerHash. Due to the fact that sha256.Sum256 returns [32]byte rather than []byte, we need to convert it to a slice:

outerHash := sha256.Sum256(
	slices.Concat(outerKey, innerHash[:]),
)

Finally, we return the outerHash, converted to a slice:

return outerHash[:]

This implementation should pass all tests:

$ go test .
ok      myhmac  0.002s

If you were following along, now is the time for our final commit:

$ git commit -am "Draw the rest of the owl"