The primary goal of a web browser is to display the information identified by a URL. To do so, a browser first uses the URL to connect to a server somewhere on the Internet, and then requests information from that server. The web page is the data in the server’s response.
To display a web page, the browser first needs to get a copy of it. To do so, it asks the OS to put it in touch with a server somewhere on the internet; the URL for the web page tells it the server’s host name. The OS then talks to a DNS server which convertsOn some systems, you can run
dig +short example.org to do this conversion yourself. the host name like
example.org into an IP address like
220.127.116.11.Today there are two versions of IP: IPv4 and IPv6. IPv6 addresses are a lot longer and are usually in hex, but otherwise the differences don't matter here. Then the OS decides which hardware is best for communicating with that IP address (say, wireless or wired) using what is called a routing table, and then uses device drivers to sends signals over a wire or over the air.I'm skipping steps here. On wires you first have to wrap communications in ethernet frames, on wireless you have to do even more. I'm trying to be brief. Those signals are picked up and transmitted by a series of routersOr a switch, or an access point, there are a lot of possibilities, but eventually there is a router. which each send your message in the direction they think will take it toward that IP address.They may also record where the message came from so they can forward the reply back, especially in the case of NATs. Eventually this reaches the server, and the connection is created. Anyway, the point of this is that the browser tells the OS, “Hey, put me in touch with
example.org”, and it does.
On many systems, you can set up this kind of connection manually using the
telnet program, like this:The “80” is the port, discussed below.
telnet example.org 80
You might need to install
telnet; it is often disabled by default. On Windows, go to Programs and Features / Turn Windows features on or off in the Control panel. On macOS, you can use the
nc -v command as a replacement:
nc -v example.org 80
The output from
nc is a little different from
telnet but it does basically the same thing. You can install
telnet on most Linux systems; plus, the
nc command is usually available from a package called
You'll get output that looks like this:
Trying 18.104.22.168... Connected to example.org. Escape character is '^]'.
This means that the OS converted
example.org to the IP address of
22.214.171.124 and was able to connect to it.The line about escape characters is just instructions on using obscure
telnet features. You can now type in text and press enter to talk to
Once it’s connected, the browser requests information from the server by name. The name is the part of a URL that comes after the host name, like
/index.html, called the path. The request looks like this:
GET /index.html HTTP/1.0 Host: example.org
Here, the word
GET means that the browser would like to receive information,It could say
POST if it intended to send information, plus there are some other obscure options. then comes the path, and finally there is the word
HTTP/1.0 which tells the host that the browser speaks version 1.0 of HTTP.Why not 1.1? You can use 1.1, but then you need another header (
Connection) to handle a feature called "keep-alive". Using 1.0 avoids this complexity. There are several versions of HTTP (0.9, 1.0, 1.1, and 2.0). The HTTP 1.1 standard adds a variety of useful features, like keep-alive, but in the interest of simplicity our browser won't use them. We're also not implementing HTTP 2.0; HTTP 2.0 is much more complex than the 1.X series, and is intended for large and complex web applications, which our browser can’t run anyway.
After the first line, each line contains a header, which has a name (like
Host) and a value (like
example.org). Different headers mean different things; the
Host header, for example, tells the host who you think it is.This is useful when the same IP address corresponds to multiple host names (for example,
example.org). There are lots of other headers one could send, but let's stick to just
Host for now.Many websites, including
example.org, basically require the
Host header to function properly, since hosting multiple domains on a single computer is very common.
Finally, after the headers comes a single blank line; that tells the host that you are done with headers.
Enter all this into
telnet, remembering to leave add a blank line after the line that begins with
Host. You should get a response.
The server’s response starts with this line:
HTTP/1.0 200 OK
That tells you that the host confirms that it, too, speaks
HTTP/1.0, and that it found your request to be "OK" (which has a numeric code of 200). You may be familiar with
404 Not Found; that’s another numeric code and response, as are
403 Forbidden or
500 Server Error. There are lots of these codes, and they have a pretty neat organization scheme:The status text like
OK can actually be anything and is just there for humans, not for machines.
Note the genius of having two sets of error codes (400s and 500s), which tells you who is at fault, the server or the browser.More precisely, who the server thinks is at fault. You can find a full list of the different codes on Wikipedia, and new ones do get added here and there.
200 OK line, the server sends its own headers. When I did this, I got these headers (but yours will differ):
Cache-Control: max-age=604800 Content-Type: text/html; charset=UTF-8 Date: Mon, 25 Feb 2019 16:49:28 GMT Etag: "1541025663+ident" Expires: Mon, 04 Mar 2019 16:49:28 GMT Last-Modified: Fri, 09 Aug 2013 23:54:35 GMT Server: ECS (sec/96EC) Vary: Accept-Encoding X-Cache: HIT Content-Length: 1270 Connection: close
There is a lot here, about the information you are requesting (
Last-Modified), about the server (
X-Cache), about how long the browser should cache this information (
Etag), about all sorts of other stuff. Let's move on for now.
After the headers there is a blank line followed by a bunch of HTML code. This is called the body of the server’s response, and your browser knows that it is HTML because of the
Content-Type header, which says that it is
text/html. It’s this HTML code that contains the content of the web page itself.
Let’s now switch gears from manual connections to Python.
So far we've communicated with another computer using
telnet. But it turns out that
telnet is quite a simple program, and we can do the same programmatically. It’ll require extracting host name and path from the URL, creating a socket, sending a request, and receiving a response.
A URL like
http://example.org/index.html has several parts:
http, explains how to get the information
example.org, explains where to get it
/index.html, explains what information to get
There are also optional parts to the URL. Sometimes, like in
http://localhost:8080/, there is a port that comes after the host, and there can be something tacked onto the end, a fragment like
#section or a query like
?s=term. We’ll come back to ports later in this chapter, and some other URL components appear in exercises.
In Python, there's a library called
urllib.parse that splits a URL into these pieces, but let’s write our own.There’s nothing wrong with using libraries, but implementing our own is good for learning. Plus, it makes this book easier to follow in a language besides Python. We'll start with the scheme—our browser only supports
http, so we just need to check that the URL starts with
http:// and then strip that off:
Now we must separate the host from the path. The host comes before the first
/, while the path is that slash and everything after it:
split(s, n) function splits a string at the first
n copies of
s. The path is supposed to include the separating slash, so I make sure to add it back after splitting on it.
With the host and path identified, the next step is to connect to the host. The operating system provides a feature called “sockets” for this. When you want to talk to other computers (either to tell them something, or to wait for them to tell you something), you create a socket, and then that socket can be used to send information back and forth. Sockets come in a few different kinds, because there are multiple ways to talk to other computers:
AF. We want
AF_INET, but for example
SOCK. We want
SOCK_STREAM, which means each computer can send arbitrary amounts of data over, but there's also
SOCK_DGRAM, in which case they send each other packets of some fixed size.The
DGRAMstands for "datagram" and think of it like a postcard.
IPPROTO_TCP.Nowadays some browsers support protocols that don't use TCP, like Google Chrome's QUIC protocol.
By picking all of these options, we can create a socket like so:While this code uses the Python
socket library, your favorite language likely contains a very similar library. This API is basically standardized. In Python, the flags we pass are defaults, so you can actually call
socket.socket(); I'm keeping the flags here in case you're following along in another language.
Once you have a socket, you need to tell it to connect to the other computer. For that, you need the host and a port. The port depends on the type of server you’re connecting to, and for now should always be 80.
Note that there are two parentheses in the
connect takes a single argument, and that argument is a pair of a host and a port. This is because different address families have different numbers of arguments.
The syntax of URLs is defined in RFC 3987, which is pretty readable. Try to implement the full URL standard, including encodings for reserved characters.
You can find out more about the "sockets" API on Wikipedia. Python more or less implements that API directly.
Now that we have a connection, we make a request to the other server. To do so, we send it some data using the
There are a few things to be careful of here. First, it’s important to have the letter “b” before the string. Next, it's very important to use
\r\n instead of
\n for newlines. And finally, it’s essential that you put two newlines
\r\n at the end, so that you send that blank line at the end of the request. If you forget that, the other computer will keep waiting on you to send that newline, and you'll keep waiting on its response. Computers are dumb.
Time for a Python quirk. When you send data, it's important to remember that you are sending raw bits and bytes; they could form text or an image or video. That's why here I have a letter
b in front of the string of data: that tells Python that I mean the bits and bytes that represent the text I typed in, not the text itself, which you can tell because it has type
If you forget that letter
b, you will get some error about
bytes. You can turn a
bytes by calling its
encode("utf8") method, and go the other way with
decode("utf8").Well, to be more precise, you need to call
encode and then tell it the character encoding that your string should use. This is a complicated topic. I'm using
utf8 here, which is a common character encoding and will work on many pages, but in the real world you would need to be more careful.
You'll notice that the
send call returns a number, in this case
44. That tells you how many bytes of data you sent to the other computer; if, say, your network connection failed midway through sending the data, you might want to know how much you sent before the connection failed.
To read the response, you'd generally use the
read function on sockets, which gives whatever bits of the response have already arrived. Then you write a loop that collects bits of the response as they arrive. However, in Python you can use the
makefile helper function, which hides the loop:If you're in another language, you might only have
socket.read available. You'll need to write the loop, checking the socket status, yourself.
makefile returns a file-like object containing every byte we receive from the server. I am instructing Python to turn those bytes into a string using the
utf8 encoding, or method of associating bytes to letters.It would be more correct to use
utf8 to decode just the headers and then use the
charset declaration in the
Content-Type header to determine what encoding to use for the body. That's what real browsers do; browsers even guess the encoding if there isn't a
charset declaration, and when they guess wrong you see those ugly � or some strange áççêñ£ß. I am skipping all that complexity and by again hardcoding
utf8. I’m also informing Python of HTTP’s weird line endings.
Let's now split the response into pieces. The first line is the status line:
Note that I do not check that the server's version of HTTP is the same as mine; this might sound like a good idea, but there are a lot of misconfigured servers out there that respond in HTTP 1.1 even when you talk to them in HTTP 1.0. (Luckily the protocols are similar enough as to not cause confusion.)
After the status line come the headers:
For the headers, I split each line at the first colon and fill in a map of header names to header values. Headers are case-insensitive, so I normalize them to lower case. Also, white-space is insignificant in HTTP header values, so I strip off extra whitespace at the beginning and end.
Finally, the body is everything else the server sent us:
It’s that body that we’re going to display. Before we do that, let’s gather up all of the connection, request, and response code into a
Now let’s display the text in the body.
Many common (and uncommon) HTTP headers are described on Wikipedia.
Accept-Encoding header allows a web browser to advertise that it supports receiving compressed documents. Try implementing support for one of the common compression formats (like
The HTML code in the body defines the content you see in your browser window when you go to http://example.org/index.html. I'll be talking much, much more about HTML in future chapters, but for now let me keep it very simple.
In HTML, there are tags and text. Each tag starts with a
< and ends with a
>; generally speaking, tags tell you what kind of thing some content is, while text is the actual content.That said, some tags, like
img, are content, not information about it. Most tags come in pairs of a start and an end tag; for example, the title of the page is enclosed a pair of tags:
</title>. Each tag, inside the angle brackets, has a tag name (like
title here), and then optionally a space followed by attributes, and its pair has a
/ followed by the tag name (and no attributes). Some tags do not have pairs, because they don't surround text, they just carry information. For example, on http://example.org/index.html, there is the tag:
<meta charset="utf-8" />
This tag explains that the character set with which to interpret the page body is
utf-8. Sometimes, tags that don't contain information end in a slash, but not always; it’s a matter of preference.
The most important HTML tag is called
<body> (with its pair,
</body>). Between these tags is the content of the page; outside of these tags is various information about the page, like the aforementioned title, information about how the page should look (
</style>), and metadata (the aforementioned
So, to create our very very simple web browser, let's take the page HTML and print all the text, but not the tags, in it:If this example causes Python to produce a
SyntaxError pointing to the
end on the last line, it is likely because you are running Python 2 instead of Python 3. These chapters assume Python 3.
This code is pretty complex. It goes through the request body character by character, and it has two states:
in_angle, when it is currently between a pair of angle brackets, and
not in_angle. When the current character is an angle bracket, changes between those states; when it is not, and it is not inside a tag, it prints the current character.The
end argument tells Python not to print a newline after the character, which it otherwise would.
Put this code into a new function,
We can now string together
This code uses the
sys library to read the first argument (
sys.argv) from the command line to use as the URL. Try running the code you’ve written, passing the URL
python3 browser.py http://example.org/
You should see some short text welcoming you to the official example web page. You can also try using it on this chapter!
So far, our browser supports the
http scheme. That’s pretty good: it’s the most common scheme on the web today. But more and more, websites are migrating to the
https scheme. I’d like this toy browser to support
https because many websites today require it.
The difference between
https is that
https is more secure—but let’s be a little more specific. The
https scheme, or more formally HTTP over TLS, is identical to the normal
http scheme, except that all communication between the browser and the host is encrypted. There are quite a few details to how this works: which encryption algorithms are used, how a common encryption key is agreed to, and of course how to make sure that the browser is connecting to the correct host.
Luckily, the Python
ssl library implements all of these details for us, so making an encrypted connection is almost as easy as making a regular connection. That ease of use comes with accepting some default settings which could be inappropriate for some situations, but for teaching purposes they are fine.
Making an encrypted connection with
ssl is pretty easy. Suppose you’ve already created a socket,
s, and connected it to
example.org. To encrypt the connection, you use
ssl.create_default_context to create a context
ctx and use that context to wrap the socket
s. That produces a new socket,
When you wrap
s, you pass a
server_hostname argument, and it should match the argument you passed to
s.connect. Note that I save the new socket back into the
s variable. That’s because you don’t want to send over the original socket; it would be unencrypted and also confusing.
Let’s try to take this code and add it to
request. First, we need to detect which scheme is being used:
Encrypted HTTP connections usually use port 443 instead of port 80:
While we’re at it, let’s add support for custom ports, which are specified in a URL by putting a colon after the host name, like in
Custom ports are handy for debugging.
Next, we’ll wrap the socket with the
These two steps should be all you need to connect to HTTPS sites.
TLS is pretty complicated; you can read the details in RFC 8446. Implementing your own is not recommended: writing security-sensitive code is a pretty different and more difficult skill than just writing code, and without a lot of very careful work a custom TLS implementation will be very insecure.
This chapter went from an empty file to a rudimentary web browser that can:
Yes, this is still more of a command-line tool than a web browser, but it already has some of the core capabilities of a browser.
Host, send the
User-Agent header in the
request function. Its value can be whatever you want—it identifies your browser to the host.
Error codes in the 300 range refer to redirects. Change the browser so that, for 300-range statuses, the browser repeats the request with the URL in the
Location header. Note that the
Location header might not include the host and scheme. If it starts with
/, prepend the scheme and host. You can test this with with the URL http://browser.engineering/redirect, which should redirect back to this page.
Add support for Data URLs, which embed the whole resource into the URL. You’ll need to undo the
base64 encoding; use the Python
b64decode function for this.
Only show the text of an HTML document between
</body>. This will avoid printing the title and style information. You will need to add additional variables
tag to that loop, to track whether or not you are between
body tags and to keep around the tag name when inside a tag.
Support multiple file formats in
show: use the
Content-Type header to determine the content type, and if it isn't
text/html, just show the whole document instead of stripping out tags and only showing text in the