Introduction to Binary Exploitation
It is well known that computers run on binary code. All computer programs are eventually “binaries”—a pile of ones and zeros—that a CPU, with the help of its operating system, can understand. Despite this being true of every classical computer, it rarely feels that way when you fire up Clash Royale on your phone. Why is that?
Both software developers and users very rarely interact directly with the binary code through which programs run. They forget that, despite the binary itself being incomprehensible at face value, it still encodes all of the software’s behaviours—including its bugs and features. Reverse engineering is the process by which someone comes to understand what a binary does without having access to the source code from which it was compiled. In doing so, they can find bugs that might help them elicit unusual or incorrect behaviour from the binary, potentially leading to binary exploitation.
To help us understand how binary exploitation commonly manifests, let’s consider one real—but old and well-known—example: the “Heartbleed” vulnerability in OpenSSL.
Key Concepts
Heartbleed: A Case Study
C gives programmers direct access to memory-related functions and system calls
such as malloc, brk, and mmap. This is part of what makes C such a useful
language, but it also makes C code prone to many memory-related bugs.
Understanding program memory is crucial to understanding reverse engineering
and binary exploitation. CS50 is a great way to learn C from scratch, and week
4 specifically covers memory.
The function that concerns Heartbleed is memcpy. It copies bytes from one area
of memory to another. It only knows where to write, where to read from, and how
many bytes to copy. It does not know the programmer’s intent. If the length is
wrong, memcpy may read past the source or write past the destination.
Example vulnerable code:
void handle_message(char *payload, size_t claimed_len) {
char *resp = malloc(claimed_len);
if (!resp) return;
memcpy(resp, payload, claimed_len); // trusts claimed_len
send_response(resp, claimed_len);
free(resp);
}Heartbleed is a vulnerability in the TLS implementation included in older versions of OpenSSL. Part of the TLS specification is a “heartbeat” that serves to keep connections alive, avoiding an expensive handshake when communication resumes. This harmless seeming feature operates as follows: a client sends a small message and asks the server to echo it back.
Client: payload = "HELLO", length = 5
Server: reply = "HELLO"When working as intended, this keeps connections alive and detects unresponsive peers.
But what happens with a one-byte payload, payload = "X", that claims its
length is 65,535? The server trusts that length field and prepares to echo
65,535 bytes back. The bug is that the server believes the client’s claimed
length without checking whether that many payload bytes actually arrived.
size_t claimed = packet->length; // attacker-controlled
char *payload = packet->data; // actual payload length may be smaller
char *reply = malloc(claimed);
memcpy(reply, payload, claimed); // copies claimed bytes
send(reply, claimed);Because memcpy reads the claimed number of bytes from the payload pointer, it
continues into adjacent heap memory when the real payload is shorter. The server
then sends those extra bytes back to the requester. These bytes might contain
private keys, session cookies, tokens, or other sensitive data.
A note on buffer overflows
Heartbleed is a buffer over-read. The opposite mistake is also possible: a program can write beyond the end of a buffer.
Consider a function that reserves eight bytes for a name:
void save_name(char *input, size_t length) {
char name[8];
memcpy(name, input, length);
}If the function is given a length larger than eight, memcpy continues writing
after the name buffer ends. Those other bytes might be significant to the
program. This is called a buffer overflow.
Most buffer overflows simply cause incorrect behaviour or crash the program. Under the right conditions, however, an attacker may be able to choose what is overwritten and influence the program’s execution. Modern systems have protections to make this harder, but the underlying bug is the same: the program writes more data than its destination has been allocated.
Heartbleed and buffer overflows therefore come from a similar failed assumption. In Heartbleed, the program trusts how many bytes it was told to read. In a buffer overflow, the program trusts how many bytes it was told to write. In both cases, the computer obediently performs an unsafe operation because the code never checked whether the request fit within the real boundary.