The goal of this article is to provide an overview of HTTP and its principles. If you are new to HTTP, you might find this article to be a helpful primer on the topic. If you are already familiar with HTTP, you can skip to the next section.^WIP This article doesn't discuss much about best practices or implementation details.

Distributed systems

The internet is a vast, interconnected network of computers, servers, and devices spanning the globe. It's a massive distributed system where millions of components must communicate seamlessly across different networks, protocols, and organizational boundaries. Unlike centralized systems where all communication flows through a single point of control, the internet operates as a decentralized mesh where any node can potentially communicate with any other node, often through multiple intermediate systems and across various trust domains. How do we manage to communicate in such a system?¹

HTTP

The Hypertext Transfer Protocol (HTTP) is a stateless communication protocol that allows computers to communicate over a network. It is expressly designed to be a usable as an interface to distributed object systems². It is a longstanding and proven technology that operates as the foundation of the web and is used by almost all modern web applications.

There are a number of RFCs that provide specifications for HTTP. I won't rehash all the specifics here, but I would note that they are worth reading. I know it is tempting to avoid reading documentation. In my experience, the "quick and easy" approach often ends up causing more trouble than if you had just invested a little bit of time learning for yourself. Just read the RFCs.

If after reading this section you are still having trouble getting an intuition for HTTP, I would recommend taking this course. The instructor walks you through implementing the HTTP/1.1 protocol from scratch in go. I enjoyed it thoroughly and learned some new things even with several years of experience building HTTP servers. Although all programming languages have a library for HTTP which will handle many details of the protocol for you, it is worthwhile understanding what is happening under the hood.

Semantics

The semantics of HTTP are messages sent between two computers (a client and a server). A client "connects" to a server and sends a "request" message. The server may accept the request and provide a "response" message. The target of the request is called a "resource". A "representation" is a reflection of a state of a resource.² Messages are stateless, which means we can understand the semantics of a request in isolation. This turns out to be an extremely powerful property. With it, we can reuse proxies and load balance requests across multiple servers.

There are a few key components to a HTTP message:

• Methods indicate the desired action (e.g., GET, POST, PUT, DELETE).
• Status Codes indicate the result of the server's processing of the request (e.g., 200 OK, 404 Not Found).
• Headers provide additional information about the request or response (e.g., Content-Type, Authorization).
• The Body contains the data sent with the request or returned by the server.

Example request message:

GET /users/123 HTTP/1.1
Host: api.example.com

Example response message:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "id": "123",
  "name": "John Doe",
  "email": "john.doe@example.com"
}

REST principles

REST (Representational State Transfer)¹ is an architectural style that builds on top of HTTP semantics to create predictable, discoverable APIs. At its core, REST treats everything as a resource that can be manipulated through a uniform interface. This uniform interface is what makes REST powerful. Instead of having custom actions for each resource, you use the same HTTP methods across all resources in a consistent way. When you design a RESTful API, you're mapping your domain objects to HTTP resources and letting the standard HTTP semantics handle the interaction patterns.

We call an API that adheres to these principles a RESTful API. These APIs model resources (nouns) rather than actions (verbs). Instead of endpoints like /createUser or /deletePost, they design around resources like /users and /posts, using HTTP methods to indicate the action. This creates a consistent, predictable interface where developers can quickly understand how to interact with any resource in your system. For example, if you know how to work with /users, you can immediately understand how to work with /products or /orders because they follow the same patterns.

The uniform interface principle extends beyond just HTTP methods to include consistent response formats, error handling, and pagination across all endpoints. This consistency reduces cognitive load for developers and makes the API feel like a cohesive system rather than a collection of unrelated endpoints.

HTTP methods

HTTP methods (also known as verbs) indicate the desired action to be performed on a given resource.² Each method has specific semantics that inform the server how to process the request.

Method properties

There are a few properties of HTTP methods that are worth understanding.

Side effects and safe methods

If you have read any of the RFCs or have read about APIs, chances are you have come across the word "side effects".

• Side effects: Any observable changes on the server or in the system caused by handling a request, beyond just returning the requested data.
• Safe requests: Requests that do not have any intended effect on the server's state.

In practice, all requests will have some side effect as we will log each request our server receives. From the RFC on HTTP Semantics:

Request methods are considered "safe" if their defined semantics are essentially read-only; i.e., the client does not request, and does not expect, any state change on the origin server as a result of applying a safe method to a target resource. Likewise, reasonable use of a safe method is not expected to cause any harm, loss of property, or unusual burden on the origin server.²

Note that definition of safe requests revolves around the client's expectations. They expect a safe request to neither alter state nor cause harm. Incidental side effects (like logging or analytics) do not violate these expectations and may therefore occur.

The GET, HEAD, OPTIONS, and TRACE methods are defined to be safe.² The purpose of specifying safe methods is to allow for automated retrieval, caching and lets users understand which methods should have some constraints programmed around them.

Idempotent requests

A request method is called "idempotent" if making the same request multiple times has the same intended effect on the server as making it just once.² Put simply, repeating the same request should not cause any additional or unintended changes. PUT, DELETE, and safe request methods are idempotent by definition.² The POST method is not idempotent by default. For example, a POST request to an endpoint /users might create an unintended duplicate user record if the client were to send the POST request twice in a row. We talk more about idempotency later .^WIP

Methods and caching

A cache can store a response if the method explicitly allows it.² Usually, we will only consider caching a response from a GET and HEAD request. We talk more about caching later .^WIP

Methods overview

Here is a quick overview of the methods and what they do.

GET

A GET request is used to retrieve the current selected representation of a resource from a server. This is the primary way we retrieve information from an API. A GET request has no body, meaning all information necessary to specify the request must be included in the request URI, headers, or query parameters.²

Use Cases:

• Retrieving a resource
• Checking the status of a resource
• Retrieving a subset of a resource
• Retrieving many resources

Sample Response Codes:

• 200 OK for successful GET requests
• 400 Bad Request for invalid GET requests
• 404 Not Found for GET requests where the requested resource is not found

Caching: GET responses are cacheable by default. Most caching strategies focus on GET requests since they're the primary method for information retrieval. The Cache-Control header is used to control caching behavior.

Range Requests: The Range header allows clients to request only specific parts of a representation, useful for large resources or resumable downloads.

Security Considerations: When using GET with user-provided data (like form queries), be aware that sensitive information can be exposed in URIs. Consider using a different method instead when transmitting sensitive data that shouldn't appear in URLs.

HEAD

The HEAD method is similar to GET, but it only retrieves the headers of the response. This is useful for checking the status of a resource without downloading the entire representation.²

Caching: HEAD responses are cacheable by default. The Cache-Control header is used to control caching behavior.

POST

The POST method requests that the target resource process the representation enclosed in the request according to the resource's own specific semantics. This is the primary method for creating new resources or performing non-idempotent actions that don't fit neatly into other HTTP methods.²

Common Use Cases:

• Creating new resources (e.g., creating a new user account)
• Submitting form data to a data-handling process
• Performing actions that don't map to standard CRUD operations
• Appending data to existing resources

Response Codes: The server should choose an appropriate status code based on the result:

• 201 Created when a new resource is successfully created (include a Location header)
• 200 OK for successful processing without resource creation (see idempotency)^WIP
• 202 Accepted for asynchronous operations that will complete later
• 400 Bad Request for invalid input data
• 409 Conflict when the request conflicts with current state

Caching: POST responses are only cacheable when they include explicit freshness information and a Content-Location header matching the POST's target URI. A cached POST response can be reused to satisfy later GET or HEAD requests.

Idempotency: POST is not idempotent by default. Use an Idempotency-Key header to make POST requests safely retryable.

PUT

The PUT method requests that the state of the target resource be created or replaced with the state defined by the representation enclosed in the request message content.² It can also be used for performing idempotent actions that don't fit neatly into other HTTP methods.

Key Characteristics:

• Idempotent: Multiple identical PUT requests have the same effect as a single request
• Complete Replacement: PUT replaces the entire resource state, not just parts of it
• Create or Update: PUT can create a new resource if it doesn't exist, or replace an existing one

Response Codes:

• 201 Created when a new resource is successfully created
• 200 OK when an existing resource is successfully updated
• 409 Conflict when the representation conflicts with current resource constraints
• 415 Unsupported Media Type when the content type is not acceptable

Validation: The server should verify that the PUT representation is consistent with configured constraints for the target resource. If inconsistencies exist, the server should either transform the representation to make it consistent or respond with an appropriate error.

Caching: PUT responses are not cacheable. Successful PUT requests invalidate any cached responses for the target URI.

PATCH

The PATCH method applies partial modifications to a resource. Unlike PUT which replaces the entire resource, PATCH allows for targeted updates to specific fields or properties.³

Use Cases:

• Updating specific fields of a resource without affecting others
• Applying incremental changes to complex objects
• Supporting partial updates in a controlled manner
• Modifying resources that may not exist (depending on patch document type)

Key Characteristics:

• Not Safe: PATCH can modify resource state
• Not Idempotent: By default, PATCH is not idempotent, though it can be designed to be
• Conditional: Should use conditional requests (If-Match with ETags) to prevent conflicts

Response Codes:

• 200 OK for successful partial updates
• 400 Bad Request for malformed patch documents
• 409 Conflict when the patch cannot be applied due to resource state
• 412 Precondition Failed when conditional request fails (e.g., ETag mismatch)
• 415 Unsupported Media Type when the patch document format is not supported
• 422 Unprocessable Entity for semantic validation errors

Caching: PATCH responses are only cacheable when they include explicit freshness information and a Content-Location header matching the Request-URI. A cached PATCH response can only be used to respond to subsequent GET and HEAD requests.

Accept-Patch Header: Servers should include an Accept-Patch header in OPTIONS responses to advertise supported patch document formats.

DELETE

The DELETE method requests that the origin server remove the association between the target resource and its current functionality.²

Key Characteristics:

• Idempotent: Multiple DELETE requests to the same resource have the same effect
• Resource Removal: Deletes the resource mapping, not necessarily the underlying data
• Implementation Dependent: The actual effect depends on the server's implementation

Response Codes:

• 200 OK when the deletion succeeds and includes a representation describing the status
• 202 Accepted when the deletion will likely succeed but hasn't been enacted yet
• 204 No Content when the deletion succeeds and no further information is needed

Caching: DELETE responses are not cacheable. Successful DELETE requests invalidate any cached responses for the target URI.

CONNECT

The CONNECT method requests that the recipient establish a tunnel to the destination origin server identified by the request target. This is primarily used for establishing secure connections through proxies.²

Use Cases:

• Creating HTTPS tunnels through HTTP proxies
• Establishing end-to-end encrypted connections
• Supporting secure communication in corporate environments

Request Format: The request target consists of only the host and port number (e.g., server.example.com:443)

Security Considerations:

• Restrict CONNECT to known, safe ports (typically 443 for HTTPS)
• Implement proper authentication for proxy access
• Be cautious of CONNECT requests to non-standard ports

Response Codes:

• 2xx responses indicate successful tunnel establishment
• 4xx responses indicate the tunnel could not be established

Caching: CONNECT responses are not cacheable.

OPTIONS

The OPTIONS method requests information about the communication options available for the target resource or the server in general.²

Use Cases:

• Discovering available HTTP methods for a resource
• Checking server capabilities and supported features
• Implementing CORS preflight requests
• Testing server availability and configuration

Response Headers: The response should include relevant headers such as:

• Allow: Lists the HTTP methods supported by the target resource
• Access-Control-Allow-Methods: For CORS support
• Access-Control-Allow-Headers: For CORS support

Request Target:

• Specific resource: Returns options for that resource
• Asterisk (*): Returns general server capabilities

Caching: OPTIONS responses are not cacheable.

TRACE

The TRACE method requests a remote, application-level loop-back of the request message. This is primarily used for debugging and diagnostic purposes.²

Use Cases:

• Debugging request/response chains through proxies
• Testing proxy configurations
• Diagnosing network issues
• Security testing and validation

Security Considerations:

• Never include sensitive data in TRACE requests
• Be cautious of information disclosure through TRACE responses
• Consider disabling TRACE in production environments

Response Format: The response should reflect the received request message, typically in message/http format

Request Content: TRACE requests should not include a request body

Caching: TRACE responses are not cacheable.

Response codes

Response codes are how the server communicates the result of the request to the client. I won't go into all the details of the response codes here, but they have an interesting history.⁴ There is an RFC that covers them in detail.² Here are some common response codes that you will frequently use.⁵

URI scheme

A Uniform Resource Identifier (URI) is the address that uniquely identifies a resource on the web. The path component indicates the identity and location of the resource within your API's resource hierarchy.⁵

Key principles:

• Consistency: Use a consistent URI scheme across your entire API
• Lowercase: URIs are normalized to lowercase², so always use lowercase to avoid confusion and redirects
• Security: URIs are public and visible in logs, browser history, and network traces. Never include sensitive information like API keys, passwords, or personal data
• Readability: URIs should be human-readable and self-documenting

URI Structure Examples:

http://www.example.com/users/123/posts/123:80

https://www.example.com/users?q=john&age=30

Protocol

The protocol defines the communication method and security level. http sends data in plain text, while https encrypts data using TLS/SSL for secure transmission.

Host

The host identifies the server that will process your request. It can be a domain name (like api.github.com) or an IP address (like 192.168.1.1). The host determines which server receives your API call.

Path

The path specifies the exact location of the resource on the server. It follows a hierarchical structure where each segment represents a level of organization. For example, /users/123/posts means "posts belonging to user with ID 123."

Target resource

The target resource is singular entity you're interacting with. In /users/123, the target resource is a specific user that can be identified by the resource identifier 123.

Collection

A collection represents a group of similar resources. Collections are typically plural nouns that group related items together. Examples include /users, /orders, /products. Use consistent naming conventions across your API.

Resource identifier

The resource identifier uniquely identifies a specific resource within a collection. It's usually a numeric ID, UUID, or slug. For example, /users/123 identifies user #123, while /users/john-doe might identify a user by username.

Port

The port specifies which network port the server listens on for requests. Port 80 is the default for HTTP, and 443 is the default for HTTPS. You typically don't need to specify these in URLs unless using non-standard ports.

Query parameters

Query parameters provide additional information to filter, sort, or modify the request. They come after the ? and are separated by &. Common uses include search terms (?q=john), pagination (?page=2&limit=10), and filtering (?status=active&category=tech).

Headers

Headers communicate metadata about the request or response. There are many headers, but we will only cover the most common ones. You can find more complete information in the RFCs.² You might occasionally see headers that start with X-. This was a historical convention denoting experimental HTTP headers or headers that do not conform to the standard. This practice has now been depreciated and consequently you should not use X- headers.⁷

User Agent

The User-Agent header is a string sent by a client to identify the software making the request (browser, SDK, bot, CLI, etc.).

Example:

GET /users/123 HTTP/1.1
Host: api.example.com
User-Agent: curl/8.1.0

This is useful for debugging and analytics and occasionally for providing a more tailored experience.⁸

Location

A Location ² header is used by the server to indicate the URI of a newly created or redirected resource.

• In 201 Created responses, it tells the client where the newly created resource can be found.
• In 3xx Redirect responses, it specifies the URL the client should request instead.

Example:

POST /users HTTP/1.1
Host: api.example.com
Location: /users/123

Content-Type

The Content-Type header is used to indicate the media type of the resource in the response body.

Example:

POST /users HTTP/1.1
Host: api.example.com
Content-Type: application/json

Idempotency-Key

The Idempotency-Key request header lets clients make non-idempotent methods like POST fault-tolerant by safely retrying the same operation without causing duplicates. See the IETF Internet-Draft for details.⁹ We talk about Idempotency later ^WIP.

Example:

POST /users HTTP/1.1
Host: api.example.com
Idempotency-Key: 8e03978e-40d5-43e8-bc93-6894a57f9324

Range

The Range request header serves two primary purposes: requesting partial content from large resources and implementing pagination for list endpoints. This dual functionality makes it a powerful tool for efficient data retrieval.^{5 10}

Partial content requests

For large files and resources, the Range header allows clients to request only specific portions:²

Units: Most APIs use bytes as the unit for file downloads.

Partial responses: When the server honors a range request, it responds with 206 Partial Content and includes a Content-Range header indicating the range returned.

Example (file download):

GET /files/abc123 HTTP/1.1
Host: api.example.com
Range: bytes=0-999

Response:

HTTP/1.1 206 Partial Content
Content-Range: bytes 0-999/12345

List pagination and sorting

For list endpoints, the Range header may be used to paginate and sort.²

Default behavior: List requests return a Content-Range header indicating the range of values returned. Large lists may require additional requests to retrieve all data.

Pagination: If a list response has been truncated, you'll receive a 206 Partial Content status with a Next-Range header. To retrieve the next range, repeat the request with the Range header set to the value of the previous request's Next-Range header.

Page size control: Use the max parameter to control the number of values returned in a range.

Example (pagination):

GET /users HTTP/1.1
Host: api.example.com
Range: id ..; max=10;

Response (truncated):

HTTP/1.1 206 Partial Content
Content-Range: id 1-10/1000
Next-Range: id 11..; max=10;

Next page request:

GET /users HTTP/1.1
Host: api.example.com
Range: id 11..; max=10;

Cache-Control

The Cache-Control header controls how, if at all, responses are cached by clients, proxies, and CDNs.

Common values:

• no-store - Don't cache at all.
• no-cache - Cache, but always revalidate before using.
• public - Cacheable by any cache.
• private - Cacheable only by the client's browser or device.
• max-age=<seconds> - Cache lifetime before revalidation.

For dynamic or sensitive API responses, Cache-Control: no-store is safest unless caching is intentional.

Example:

GET /users/123 HTTP/1.1
Host: api.example.com
Cache-Control: no-cache

Authorization

The Authorization header carries credentials for authenticating the client with the API.

The most common scheme is Bearer tokens. However, there are other schemes such as Basic auth, HMAC signatures, and custom schemes. Always use HTTPS to protect credentials in transit.

Example (Bearer token):

GET /users/123 HTTP/1.1
Host: api.example.com
Authorization: Bearer sk_live_4eC76HqLyjWTfdrjtT1zds90c

Accept

The Accept request header tells the server which response formats the client can handle.

APIs usually default to application/json but may allow other media types. It is probably a good idea to just use application/json.

Example:

GET /users/123 HTTP/1.1
Host: api.example.com
Accept: application/json

You can also include versioning in the Accept header, but it is also okay to use a custom header for this.⁵

Retry-After

The Retry-After response header tells the client how long to wait before retrying.

Sent with 429 Too Many Requests or 503 Service Unavailable.

Value can be:

• HTTP-date: Exact time to retry (we will discuss why this is a bad idea later).^WIP
• Seconds: Relative wait time.

Example:

HTTP/1.1 429 Too Many Requests
Retry-After: 60

Rate Limit Headers

There are a few standardized headers for communicating API rate limits to clients.

RateLimit-Limit: Max requests allowed in the current window.

RateLimit-Remaining: Requests left before hitting the limit.

RateLimit-Reset: Seconds until the limit resets.

Example:

HTTP/1.1 200 OK
RateLimit-Limit: 100
RateLimit-Remaining: 42
RateLimit-Reset: 120

ETag

The ETag response header contains a unique identifier for a resource version. These can either be "weak" or "strong".

Strong ETags:

Change on any byte difference.

ETag: "abc123"

Weak ETags:

Allow minor changes without invalidating.

ETag: W/"abc123"

It is a good idea to include it in all responses.⁵ The server can determine what exactly the version identifier is and clients should not try to parse it.² However, don't assume that clients will not try to parse it. It might make sense for the identifier to be distinct from typical resource identifiers so that clients won't make any assumptions about its format and utilize it outside of its intended use.

If-None-Match

The If-None-Match request header is sent by the client to the server along with an ETag representing the version of the resource that the client currently has cached. The server can then check to see if that version of the resource matches the current version. If the resource has not changed, the server can return a 304 Not Modified response. This is important because it allows us to avoid sending information that the client already has.⁵

Example (conditional GET):

GET /users/123 HTTP/1.1
Host: api.example.com
If-None-Match: "abc123"

Response (unchanged):

HTTP/1.1 304 Not Modified

If-Match

The If-Match request header is sent by the client to the server along with an ETag representing the version of the resource that the client currently has cached. The server can then determine if the resource has changed since the client last requested it. If the resource has changed, the server can return a 412 Precondition Failed response. This is important because it allows us to avoid race conditions where the client updates a resource that has been updated by another client.⁸

Example (conditional PATCH):

PATCH /users/123 HTTP/1.1
Host: api.example.com
If-Match: "abc123"

Response (changed):

HTTP/1.1 412 Precondition Failed

Response (unchanged):

HTTP/1.1 200 OK

Request-Id

It is a good idea to record every request sent to your API and provide a unique identifier for each request. This can be useful for debugging and tracing.⁵ The Request-Id header is a standardized header for this purpose and should be included in all responses.

Example:

HTTP/1.1 200 OK
Request-Id: req_a1b2c3d4

Body

The body of an HTTP message contains the data being transmitted between client and server. In HTTP requests, the body carries the data that the client wants to send to the server (such as form data, JSON payloads, or file uploads). In HTTP responses, the body contains the data that the server returns to the client (such as HTML pages, JSON responses, or file downloads).

We call the data returned by the server a "representation" of the resource. A resource has underlying state that cannot be seen directly by the client. Hence, we call the data returned by the server a "representation" of that resource state. Clients might want to see that resource in different formats, which we call media types.

The body is optional - many HTTP messages don't have one. GET requests typically don't include a body, while POST, PUT, and PATCH requests commonly do. The presence and format of the body is indicated by headers like Content-Type and Content-Length.