Web Client

A client is whatever asks first

Client and server are roles, not machines. Whichever side opens the conversation and asks for something is the client; the side that waits and answers is the server. Your browser is a client, so is curl, so is a phone app, so is another server calling an API.

That is why you can write a web client in one line. It does not need a window, a rendering engine, or a mouse — it needs to connect, send a request, and read what comes back.

Making a web client in the terminal

The quickest way to believe that HTTP is only text is to type a request yourself. Start the shell server from the previous topic, then open a second terminal and talk to it by hand.

Typing a request by hand with nc

netcat connects to a port and passes your keystrokes straight down the wire. Type the two lines, press Enter twice, and the server answers — you have just done a browser's job manually.

# connect, then type the two lines below and press Enter once more
nc localhost 8084

GET / HTTP/1.1
Host: localhost

# or send it all at once (printf's \r\n is HTTP's line break)
printf 'GET / HTTP/1.1\r\nHost: localhost\r\n\r\n' | nc localhost 8084

• The blank line at the end matters. It tells the server the request is finished, which is why you press Enter one extra time.
• HTTP requires carriage-return-and-newline at the end of each line. Typing works because most servers are forgiving, but the printf version sends exactly what the standard asks for.
• The Host line is required by HTTP/1.1. Leave it out against a real site and you will usually get an error back rather than a page.

Wrapping it into a script

Once the request is a single printf, a reusable client is a few lines. This one takes a host, a port and a path, sends the request, and splits the headers from the body so you can see the two parts.

#!/bin/bash
# usage: bash client.sh localhost 8084 /

HOST=${1:-localhost}
PORT=${2:-8084}
PATH_=${3:-/}

printf 'GET %s HTTP/1.1\r\nHost: %s\r\nConnection: close\r\n\r\n' "$PATH_" "$HOST" \
  | nc "$HOST" "$PORT" \
  | tr -d '\r' \
  | awk 'BODY { print; next } /^$/ { BODY = 1; next } { print "[header] " $0 }'

The tr command strips the carriage returns that HTTP leaves at the end of every line, and awk marks each line as a header until it meets the empty line — after that, everything is the body. This is exactly the split a browser does before it starts drawing.

curl: the client you will use every day

curl does properly what our script does roughly: redirects, HTTPS, headers, uploads, timeouts, cookies. It is on almost every machine, and it is the standard way to test a server without a browser in the way.

# body only
curl http://localhost:8084

# body plus the response headers
curl -i http://localhost:8084

# the whole exchange, request headers included
curl -v http://localhost:8084

The options that matter

Calling an API

Most of what a modern app talks to is not a page but an API returning JSON. curl fetches it, and jq makes it readable — together they are how you check an endpoint before writing a single line of application code.

# fetch JSON and pick one field out of it with jq
curl -s https://api.github.com/repos/sveltejs/svelte | jq '.stargazers_count'

# send JSON with POST
curl -X POST https://httpbin.org/post \
  -H 'Content-Type: application/json' \
  -d '{"name":"PES","level":1}'

# download to a file
curl -L -o page.html https://getpes.com

Get into the habit of testing with curl first. If curl gets the right answer, the server is fine and the bug is in your application; if curl gets the wrong answer, you have saved yourself an afternoon of looking in the wrong place.

Chrome is the same client, at a different scale

Chrome sends the request your nc command sent. Then it does the enormous amount of extra work that turns the answer into something a person can look at — and that extra work is the only real difference.

It is worth knowing the steps, because every one of them is a place where a page can go wrong. When something does not appear, the question is always which step failed.

From address bar to picture

1. Resolve the nameThe domain name is turned into an IP address by DNS. If this fails you never reach the server at all, and the error mentions the name rather than the page.
2. Connect and secureIt opens a connection and, for https, checks the certificate before anything else is sent. A warning here means the encryption could not be trusted, not that the page is missing.
3. Send the requestThe same handful of lines you typed by hand, with rather more headers: cookies, accepted formats, preferred languages.
4. Parse and fetch the restIt reads the HTML, builds a tree of elements from your tags, and requests everything the page refers to — stylesheets, images, fonts, scripts — usually several at once.
5. Style, lay out, paintIt applies the CSS, works out where every box goes, draws the result, and runs the JavaScript that can change all of it again afterwards.

Terminal client or browser

DevTools: watching the messages go by

Chrome ships with the best web debugging tool there is, and it is already installed. Everything you sent by hand with nc is listed here, request by request, with the response beside it.

The panels

Three things to try right now

1. Find your own requestOpen the Network panel, reload this page, and click the very first entry. The request and response headers there are the same lines you typed into nc.
2. Copy it as curlRight-click any request and choose Copy as cURL. Paste it into a terminal and you have reproduced exactly what the browser did — headers, cookies and all.
3. Break it on purposeIn the Elements panel, delete a heading and watch it vanish; in Network, throttle the speed to slow and reload. Both are safe, and both teach more than reading about them.