API Schema Showdown! Part 1: REST

code api

Software running on a single device is almost always only the tip of an iceberg: it usually communicates with other software by the use of APIs (Application Programming Interfaces). There are an infinite number of ways to structure data when software communicates, but some conventions have developed over time in order to rein in the chaos of every programming interface implementing its own protocol. This post launches a series examining various conventions to structure APIs.

First up is REST.

REST (Representational State Transfer) #

A cat RESTs on a window sill Photo by Daniela Turcanu on Unsplash

REST was introduced and described in Roy Thomas Fielding's year 2000 doctoral dissertation. This is the most widely used API convention by far, according to the 2022 State of the API Report. Unfortunately, Fielding's dissertation itself alone gives inadequate guidance to construct or even fully identify a REST API, and so a over the subsequent years a mishmash of sometimes contradictory blog posts, discussion forums, militancy, angry flame wars, and books fleshed out the REST.

The 2007 book RESTful Web Services was wildly influential in popularizing the standard by clarifying, elucidating and presenting a set of further guidelines or constraints that it called Resource-Oriented Architecture.

REST is set of design principles to make an API flexible, scalable and comprehensible. While REST can be used to design APIs in any client-server architecture, it was designed in the context of the web.

There are no formal guarantees associated with REST, but adopting RESTful constraints are said to confer quite a few benefits to the API:

  • Scalability: can be easily scaled to handle large numbers of clients and requests.
  • Flexibility: can be used with a wide range of programming languages, frameworks, and tools.
  • Simplicity: has a simple and well-defined interface that makes it easy to understand and use.

Some other benefits in no particular order are modularity, portability, reliability, caching, security, evolvability, interoperability, and maintainability.

REST constraints make it unsuitable for some applications, however. The statelessness constraint can make real-time or streaming applications less efficient, for instance.

According to Fielding, the RESTful API has five required constraints: client-server architecture, statelessness, cacheability, layered system and a uniform interface; this uniform interface itself has four constraints: identification of resources, manipulation of resources through representations, self-descriptive messages, and hypermedia as the engine of application state. There is an optional component called Code-On-Demand.

This post goes through all of the required constraints in turn and discusses later iterations and clarifications afterwards.

Client-Server #

Closeup of an order pad as a waiter takes an order Photo by Jessie McCall on Unsplash

The REST convention is intended to be applied only in the context of client-server architecture. In the context of the web, a client is in practice often understood to be the browser, and the server is understood mostly to be a web server. That is a decent enough short-hand model to understand this architecture, but clients can also be mobile apps, IoT devices, desktop applications, command-line tools such as cURL or wget, servers themselves, among others.

As for alternatives to web servers in this model, there are application servers handling application-specific protocols such as Java EE or .NET, database servers, media servers that provide streaming for audio or video content, DNS servers providing domain-name resolution, among others.

In the context of REST, Fielding describes the client-server style as having a "server component, offering a set of services, listens for requests upon those services" and a "client component, desiring that a service be performed, sends a request to the server via a connector. The server either rejects or performs the request and sends a response back to the client."

He quotes another researcher, "A client is a triggering process; a server is a reactive process. Clients make requests that trigger reactions from servers. Thus, a client initiates activity at times of its choosing; it often then delays until its request has been serviced. On the other hand, a server waits for requests to be made and then reacts to them. A server is usually a non-terminating process and often provides service to more than one client."

Less formally and more generally, a server is "something that provides a service", and a client is "the thing that consumes that service".

Sometimes this architecture is called request-response, because the client sends a request to the server, and the server replies with a response.

Statelessness #

A series of open doors one after another into infinity Photo by Filip Kominik on Unsplash

State refers to the data that describes the current condition of a system or application. Changing the value of a single variable in the system changes the state. Changes in state result in changes in the way that the system or application responds to requests. Handling these changes in state and behavior is called state management, usually with the goal of making the changes and their consequences easier to understand and predict.

REST requires that "no session state is allowed on the server component. Each request from client to server must contain all of the information necessary to understand the request, and cannot take advantage of any stored context on the server. Session state is kept entirely on the client."

In practice this means that all context necessary for the server to understand the request, like user authentication for example, must be sent along with every request from the client.

The intent with this constraint is to improve "visibility, reliability and scalability". Visibility refers to how deep a monitoring system must look to determine "the full nature of the request"; a stateless request has everything necessary. Reliability refers to the system's capacity to "recover from partial failures"; without state, a failure introduces no indeterminate or impossible states. Scalability refers to the expansion of resources as the application receives more requests; freed from the burden of managing state, the server can free resources more quickly.

The trade-off is that state data is pushed across the network with every request reducing efficiency, and all clients across all platforms are responsible for correctly maintaining and implementing states. This concept is revisited later when I discuss Resource-Oriented Architecture.

Cacheability #

A treasure box open slightly, something glows within Photo by Ashin K Suresh on Unsplash

There are only two hard things in Computer Science: cache invalidation and naming things.

— Phil Karlton (probably)

To address the network efficiency issues introduced by the statelessness constraint, REST responses from the server are required to be labeled as cacheable or non-cacheable. However, this label can be implied and not explicit. If for example the application has no cacheable responses, the RESTful API is not required to label every response as non-cacheable. Likewise if they all are.

According to Fielding, "a cache acts as a mediator between client and server in which the responses to prior requests can be reused in response to later requests that are equivalent and likely to result in a response identical to that in the cache if the request were to be forwarded to the server".

Essentially, if the requested data is not likely to change or change is not important, label it as cacheable so that the client can use that same data rather than hitting the server again. A well-designed caching policy can improve network latency and reduce network traffic.

In the context of the web, the response can include these headers to tell the client how to handle cacheing: Cache-Control, Expires, ETag, Last-Modified, and Vary

Layered System #

Abstract colorful layers Photo by Bohdan Orlov on Unsplash

A RESTful API adheres to a layered system design, where each layer is responsible for a specific set of functionality, and components within a layer can only interact with components in adjacent layers. This separation of concerns promotes modularity, making it easier to maintain, update, or extend the API without impacting other layers. The layers are organized hierarchically, where each layer can only communicate with the layers immediately above or below it.

For example, a typical web application might be structured with the following layers:

  • Presentation Layer: Handles the user interface and user experience, responsible for displaying data to the end-users and capturing their inputs.
  • Application Layer: Contains the main business logic, responsible for processing user inputs, managing resources, and coordinating interactions between different system components.
  • Data Access Layer: Manages the persistence and retrieval of data, typically interacting with databases or external APIs to fetch or store information.

This architectural design is intended to make it easier to maintain, update, or replace individual layers without impacting the overall functionality of the system. Additionally, it promotes better organization and encapsulation, resulting in a more robust and resilient system.

Uniform Interface #

Military uniforms with patches, hung up on hangers Photo by Hannah Wernecke on Unsplash

The term interface here refers to the set of rules and conventions that govern how clients interact with a server to access its resources. It is the I in API. In REST, the interface has four constraints. Following these constraints is designed to enforce uniformity across REST implementations.

The tradeoff here is the same tradeoff as all conformity: we gain clarity and met expectations but give up experimentation and optimization. The uniform interface constraint is arguably the most defining characteristic of REST, and that which most developers think of when they think of REST. A RESTful API has an expected look and behavior, and that is by design.

It's worth taking a moment to read the definition of resource as given in Fielding's paper.

Any information that can be named can be a resource: a document or image, a temporal service (e.g. "today's weather in Los Angeles"), a collection of other resources, a non-virtual object (e.g. a person), and so on. In other words, any concept that might be the target of an author's hypertext reference must fit within the definition of a resource.

— 5.2.1.1 Resources and Resource Identifiers

The REST API is primarily concerned with the manipulation of resources.

Identification of Resources #

A closeup of a magnifying glass on a pale blue background Photo by Markus Winkler on Unsplash

A REST resource is identified by a Uniform Resource Identifier (URI), which is a web standard. Fielding's dissertation does not explicitly state that, however, saying only "REST uses a resource identifier to identify the particular resource..." (emphasis mine). That the "resource identifier" is actually the URI remains unfortunately a subtext of the paper. URI was first defined by RFC 2396 in August 1998. It will help you to know that Fielding himself was one of the authors of this standard, and the RFC used the as-yet unpublished REST in the definition of the standard.

Uniform Resource Locator (URL) #

There is serious danger of flagrantly boring pedantry when discussing the differences between URI and URL, so erring on the side of clarity over precision, I'll just note that a) a URL implies an address for a resource where a URI is only expected to be an ID for a resource, and b) all URLs are URIs while not all URIs are URLs. If you find this topic fascinating, I'll direct you to any of the myriad articles discussing it.

Here are two examples of URIs from RFC 3986:

         foo://example.com:8042/over/there?name=ferret#nose
         \_/   \______________/\_________/ \_________/ \__/
          |           |            |            |        |
       scheme     authority       path        query   fragment
          |   _____________________|__
         / \ /                        \
         urn:example:animal:ferret:nose

Manipulation of Resources through Representations #

A closeup of a craftman's hands as he manipulates some leather Photo by Nick Karvounis on Unsplash

This constraint uses common words in uncommon ways. The terms resource, representation, even manipulation do not hold their usual meanings, so "manipulation of resources through representation" will take a bit of explanation. For example, resource refers explicitly to an abstract concept and not a real thing in the real world.

Consider a bin of widgets in a warehouse. Perhaps the company that owns the warehouse exposes a webservice that handles inventory. If you want to know how many widgets the warehouse has, you send a request to the API endpoint

https://example.com/warehouse/inventory/widgets

and the response looks like this:

{
  "name": "widgets",
  "amount": 50
}

This data is a representation. It is data about the current state of the resource identified by the widgets URI. Representation refers also to any metadata, that is, information about the data: e.g. its json format, or its last-modified date, or authentication information, and so forth.

If you do want to buy a widget and have it shipped somewhere, perhaps you would send a request to the API with the delivery address, invoicing details, whatever else required. The web service, among other tasks, subtracts 1 from the amount of widgets in its representation. Here, the resource has been manipulated through a representation.

In this case, the resource is an abstract concept that only happens to correspond to an actual thing, the bin of widgets. Meanwhile in the real world, the API sends a notice to a warehouse worker (or robot), who will still have to locate the bin of widgets, remove one and package it.

Moving from the specific to the general, in REST a resource refers to this abstract concept that represents an entity or an item in the system. Its representation is a machine-readable format that conveys the current state of the resource, typically in JSON, XML, or HTML.

"Manipulation of resources through representations" means that clients interact with these abstract concepts called resources by sending and receiving representations. Clients request the representation of a resource, modify it, and send it back to the server to update the actual resource. The underlying resource remains abstract, while its representations are manipulated to perform operations.

Self-descriptive Messages #

A surreal picture of a mirror in the corner of a room reflecting ocean waves Photo by Nong on Unsplash

In the context of RESTful systems, self-descriptive messages are those that carry enough information for a recipient to understand and process them without requiring any external context or prior knowledge. Fielding's dissertation does not provide a formal definition for self-descriptive messages but highlights some aspects that contribute to achieving self-descriptive quality in RESTful systems:

REST enables intermediate processing by constraining messages to be self-descriptive: interaction is stateless between requests, standard methods and media types are used to indicate semantics and exchange information, and responses explicitly indicate cacheability.
5.3.1 Process View

From this quote, self-descriptive messages in REST can be understood as having the following characteristics:

  1. Stateless interactions: Each message contains all the information needed for a recipient to understand and process it, independent of previous requests or responses.
  2. Standard methods and media types: Messages use standardized methods (e.g., HTTP verbs like GET, POST, PUT, DELETE) and media types (e.g. JSON, XML) to indicate semantics and facilitate data exchange.
  3. Explicit cacheability: Responses clearly indicate whether they can be cached or not, allowing for better control over data consistency and performance.

The 6.3.2 Self-descriptive Messages section highlights the constraint of using standardized methods, largely by cataloguing its lack in earlier standards.

Hypermedia as the engine of application state (HATEOAS) #

An exposed Chevy Impala engine Photo by Tim Mossholder on Unsplash

HATEOAS, or Hypermedia as the Engine of Application State, is one of the key constraints of a RESTful API, and it refers to the inclusion of hypermedia controls (links) within the representations of resources returned by the server. The intent is for the client to use these controls to discover and navigate the API dynamically without the need for hardcoded URIs or out-of-band information.

Ideally, when an API follows HATEOAS, the client only needs to know the initial entry point of the API. The server then provides the client with links to related resources, actions, and states in the response. An example of a HATEOAS-compliant JSON response might look like this:

{
  "label": 1,
  "name": "Sample Widget",
  "type": "Gadget",
  "color": "blue",
  "price": 10.99,
  "amount": 50,
  "_links": {
    "self": {
      "href": "https://example.com/warehouse/inventory/widget/1",
      "method": "GET"
    },
    "update": {
      "href": "https://example.com/warehouse/inventory/widget/1",
      "method": "POST"
    },
    "delete": {
      "href": "https://example.com/warehouse/inventory/widget/1",
      "method": "DELETE"
    },
    "relatedWidgets": {
      "href": "https://example.com/warehouse/inventory/widgets?relatedTo=1"
    }
  }
}

In this example, the server includes links to the current resource itself, as well as links to update, delete, and retrieve related widgets. The client ideally can use these links to perform actions and navigate the API without needing to know the exact structure of the URIs beforehand.

This constraint is not often implemented in practice, which means, in practice, very few APIs are truly RESTful. To build a client for these HATEOAS-less APIs, it's necessary for the developers to hew closely to the documentation. Which means that the builder of the API must be precise with documentation.

If an API conforms to the first 3 constraints but skips HATEOAS, I think it is more clarifying at least to call them REST-like than to insist they are not at all RESTful.

Resource-Oriented Architecture (ROA) #

REST, as formulated in Fielding's dissertation, is a set of design principles that do not apply specifically to the web. RESTful Web Services, published in 2007, filled that gap with objective criteria, solid advice, and made the dissertation's subtext clear to everyone. It's hard to overstate the influence of this book, which turned REST from a niche nerd topic to an overwhelmingly popular web standard. This advice was so influential that in contemporary web practice ROA is tantamount to being RESTful.

There are 4 main areas on which ROA has guidance that REST per se does not:

URIs #

To be in accordance with ROA:

  • A resource is defined as "anything" but it must have "at least one URI".
  • URIs should be descriptive
    • "a resource and its URI ought to have an intuitive correspondence".
    • e.g. /api/widgets as a URI for widgets is intuitive, but for jellyfish it is not.
  • URIs should have naming consistency
    • "you should not go to /search/Jellyfish for jellyfish and /i-want-to-know-about/Mice for mice".
  • A URI must refer to one and only one resource
    • if a URI "designated more than one resource, it wouldn’t be a Universal Resource Identifier."
  • However, one resource can be referred to by many URIs:
    • "Fetching sales/2004/Q4 might get you the same bytestream as fetching sales/Q42004, because they’re different URIs for the same resource “sales for the last quarter of 2004.”"
    • "Fetching releases/1.0.3.tar.gz might give you the exact same bytestream as fetching releases/latest.tar.gz, but they’re different resources because they represent different things: “version 1.0.3” and “the latest version.”"
    • Optionally, there could be a canonical URI for each resource
      • "When a client requests the canonical URI, the server sends the appropriate data along with response code of 200 (“OK”)."
      • "When a client requests one of the other URIs, the server sends a response code 303 (“See Also”) along with the canonical URI."

Addressability #

A white mailbox at the end of a driveway Photo by Brian Patrick Tagalog on Unsplash

Addressability refers to the concept of exposing through unique URIs every piece of information that a web application client needs. ROA values addressability highly.

As an example, consider these two approaches:

  1. Addressable API: https://api.example.com/widgets

This API exposes a unique URI for the "widgets" resource. Clients can interact with this resource by sending HTTP requests to the addressable endpoint, for example:

  • GET https://api.example.com/widgets to retrieve a list of widgets.
  • GET https://api.example.com/widgets/1 to retrieve a specific widget with ID 1.
  1. Non-Addressable API: https://api.example.com

This API does not expose a unique URI for the "widgets" resource. Instead, clients need to send additional information in the HTTP request, such as parameters or data in the request body, to specify which resource they want to interact with:

  • POST https://api.example.com with a request body containing the action {"action": "getWidgets"} to retrieve a list of widgets.
  • POST https://api.example.com with a request body containing the action {"action": "getWidget", "id": 1} to retrieve a specific widget with ID 1.

The addressable API example allows clients to interact with the "widgets" resource using unique URIs and standard HTTP methods, making it easier to understand and use. The non-addressable API, on the other hand, requires clients to send additional information to specify the desired action and resource, making it less intuitive and more challenging to work with.

As an aside, example 2 is how many APIs were designed before REST! That this would be considered bad design today demonstrates how tremendously influential REST has been.

My own advice is to implement addressability only as far as necessary to make the API usable. There are tradeoffs. One could for instance create an endpoint:

https://api.example.com/widgets/1/price

It might be important to do so, for instance, if widget prices change frequently for individual widgets; but the time and overhead of building, testing and maintaining that endpoint might sap strength and energy best applied elsewhere.

Statelessness #

A stateless man waits aimlessly in an airport, staring out the window at departing planes Photo by JESHOOTS on Unsplash

RESTful Web Services defines statelessness in the same terms as does Fielding, but expands on the concept in terms of addressability:

Addressability says that every interesting piece of information the server can provide should be exposed as a resource, and given its own URI. Statelessness says that the possible states of the server are also resources, and should be given their own URIs. The client should not have to coax the server into a certain state to make it receptive to a certain request.

Statelessness means that everything that the application needs to know is sent with each request and nothing is kept on the server. Everything. Whether the client is logged in and authenticated. Where they are in the checkout flow. Where they are in the quest. How long they have been waiting. Their account balance.

A stateless web service ensures that the same request sent to the same URI always handles the request the same way irrespective of earlier requests.

A stateful web service means that the server keeps track of application state. The same request to the same URI may result in the service handling the request differently depending on previous requests.

The benefit is server-side simplicity and scalability. The tradeoff is client-side complexity and less network efficiency.

Representations #

An abstract object that looks like a dodecahedron floating in an abstract space Photo by D koi on Unsplash

If there are multiple representations of a resource ROA recommends distinguishing the specific representation in the URI. For instance, the URI for the English language version might be http://en.example.com/widgets/ and the URI for the Finnish version might be http://fi.example.com/widgets/

ROA acknowledges this might be inadequate for some uses, and so discusses content negotiation, in which there is only one URI for the resource, but the specific representation (e.g. the Finnish language version) is determined by request header (e.g. the Accept-Language header). Similarly, the Accept header could be used to request specific response formats (e.g. XML, JSON, plaintext).

All else being equal, ROA recommends that the URI contain as much of the request as possible to advantage portability and maintainability.

Tools #

A workbench on which sits a set of carpenter's tools Photo by Hunter Haley on Unsplash

Here are tools to help build RESTful or at least REST-like APIs.

OpenAPI #

OpenAPI (formerly Swagger) provides a set of tools for creating API specifications, generating code, and creating interactive API documentation. https://www.openapis.org. I have used this and found the API creation and definition tools to be very helpful. The documentation tools are decent. I did not find the code-generation tools to be useful.

RAML #

RAML is "a simple but powerful syntax for modelling APIs", which can be used to generate API documentation, code samples, and client libraries. https://raml.org/. I have not used this, myself.

Postman #

A popular API client for testing REST-like APIs, which also provides functionality for creating and managing API documentation, generating code samples, and automating API testing. https://www.postman.com/. I have used this and found it very helpful.

Other tools #

In no particular order, here are other REST or REST-like API development tools, none of which I have ever used:

  • Insomnia: An API client similar to Postman, which supports REST and GraphQL APIs, and provides features for designing, debugging, and testing APIs. https://insomnia.rest/
  • Paw: A Mac-based API client for testing RESTful APIs, with features like code generation, environment variables, and a flexible JSON parser. https://paw.cloud/
  • HttpMaster: A testing tool for RESTful APIs, which provides support for dynamic parameters, response data validation, and advanced test scenarios. https://www.httpmaster.net/
  • REST-assured: A Java-based library for testing and validating RESTful APIs, which provides a domain-specific language for writing powerful and readable tests. https://rest-assured.io/
  • Restlet Client: A browser extension and desktop application for testing RESTful APIs, which provides support for various authentication mechanisms, request history, and API documentation generation. https://restlet.com/modules/client/
  • WireMock: A library for stubbing and mocking RESTful APIs, which can be used for creating realistic API simulations for development and testing. http://wiremock.org/
  • NJsonSchema: A .NET library for validating JSON data against a JSON Schema, which can be useful for validating API requests and responses when working with RESTful APIs. https://github.com/RicoSuter/NJsonSchema

API Examples #

There are not many examples of RESTful APIs that conform to the HATEOAS constraint, but there are many, many examples of APIs that are REST-like.

Low-key #

A toy astronaut sitting on a toy moon Photo by Kobby Mendez on Unsplash

Fun examples of REST-like APIs for interesting discovery and to ease the anxious.

  • PokeAPI - information about Pokémon, including characters, moves, and more.
  • Rick and Morty API - information about the Rick and Morty TV show, including characters, locations, and episodes.
  • Open Trivia API - a collection of trivia questions across a variety of categories, including entertainment, history, and science.
  • Studio Ghibli API - information about Studio Ghibli films, including characters, locations, and species.
  • Star Wars API - information about the Star Wars universe, including characters, films, planets, and species.
  • Finna API - access to the Finnish National Bibliography and other Finnish library resources.
  • The Lord of the Rings API - information about The Lord of the Rings books and films, including characters, quotes, and more.
  • Chuck Norris API - Chuck Norris jokes.
  • Dog API - images and information about dogs.
  • Cat API - images and information about cats.

Production #

A toy robot standing on a laptop keyboard Photo by Jem Sahagun on Unsplash

These are a few examples well-known APIs designed to support applications in production.

Lists of lists #

Further Reading #

Coda #

A bricked up window Photo by Kiwihug on Unsplash

As a coda to this post, I'll just add that REST per se is not actually that well-defined.

For but one instance, Fielding asserts in his blog that

API designers, please note the following rules before calling your creation a REST API: A REST API should not be dependent on any single communication protocol... In general, any protocol element that uses a URI for identification must allow any URI scheme to be used for the sake of that identification. [Failure here implies that identification is not separated from interaction.]... (emphasis and formatting in original)

A URI is a resource identifier, and at the same time URI contains a scheme right in the spec. It's contradictory to insist that a resource must be identified uniquely by a URI, and also by multiple URIs that differ only by scheme.

What does "A REST API should not be dependent on any single communication protocol" even mean? An API is not RESTful unless it support two or more communication protocols? What, like ftp? gopher?

Sam Ruby, one of the authors of RESTful Web Services chimed in, in the comments:

I am amused by the people who are valiantly trying to decipher Roy.
... REST isn’t an all or nothing proposition. One can get significant value from partial adoption.

If one or more REST constraints do not fit your use case, toss them without qualm. Usability and performance is more important than unlocking the RESTful API acheivement badge.

Comments:

Leave a comment

There was an error. It's me, not you. You could try again later? Or try another method? Email? Smoke signals? Tell me what went wrong!

please enter a substantive message with no link or website address
please enter your name
please enter an email

Comment successfully sent, and it awaits moderation.