Infront Networked APIs - Resource-Oriented API Design Guidelines

Feel free to use these guidelines as a guidance for your own development. Also consider this to be a living, evolving document. We will revise and update based on our learnings and experiences.

The requirement level keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" used in this document are to be interpreted as described in RFC 2119.

We want to provide RESTful and gRPC APIs, which can be used by internal and external clients, whereas only the gRPC API enables efficient communication between microservices. The RESTful API however enables robust hypermedia REST clients.

This guide borrows from other design guides, which should be introduced briefly in the following. Their updated versions should be considered for possible inclusion in this document.

As discussed in Hypermedia, we figured that providing a complete RESTful API including hypermedia can have beneficial effects for clients, when they make use of it. In order to give us full flexibility in designing the RESTful API we MUST separate it from the gRPC API.

We differ from Zalando’s guidelines, since we SHOULD use hypermedia formats for expressing client side state. Also Zalando’s API is only intended for use by external web clients, not other services unlike most of our APIs.

Every API endpoint MUST be secured using OAuth 2.0. The following examples show how to specify security definitions in you API:

The keycloak server is used for authorization and requesting a token and reachable under the following (actually dev) address:

Authorization endpoints:

Token endpoints:

Hint: %realm% is a variable and depends on the application

The example defines OAuth2 with password flow as security standard used for authentication when accessing endpoints; additionally, there are two API access rights defined via the scopes section for later endpoint authorization usage - please see next section.

It makes little sense specifying the flow to retrieve OAuth tokens in the securityDefinitions section, as API endpoints should not care, how OAuth tokens were created. Unfortunately the flow field is mandatory and cannot be ommited. API endpoints SHOULD always set flow: password and ignore this information.

Every API needs to define access rights, called scopes here, and every endpoint needs to have at least one scope assigned. Scopes are defined by name and description per API specification, as shown in the previous section. Please refer to the following rules when creating scope names:

APIs SHOULD stick to standard scopes by default — for the majority of use cases, restricting access to specific APIs (with read vs. write differentiation) is sufficient for controlling access for client types like merchant or retailer business partners, customers or operational staff. Too many fine grained scopes increasing governance complexity without real value add SHOULD be avoided. In some situations, where the API serves different types of resources for different owners, resource specific scopes MAY make sense.

After scope names are defined and the scope is declared in the security definition at the top of an API specification, it SHOULD be assigned to each API operation by specifiying a security requirement.

In very rare cases a whole API or some selected endpoints may not require specific access control. However, to make this explicit you SHOULD assign the uid pseudo access right scope in this case. It is the user id and always available as OAuth2 default scope.

Hint: you need not explicitly define the "Authorization" header; it is a standard header so to say implicitly defined via the security section.

In cases you want to grant access to certain resources without giving the user access to the keycloak server, API keys SHOULD be used:

The token has to be sent in a metadata field named ACCESS_TOKEN.

The Design Guide suggests taking the following steps when designing resource-oriented APIs (more details are covered in specific sections below):

A resource-oriented API is generally modeled as a resource hierarchy, where each node is either a simple resource or a collection resource. For convenience, they are often called as a resource and a collection, respectively.

The key characteristic of a resource-oriented API is that it emphasizes resources (data model) over the methods performed on the resources (see functionality). A typical resource-oriented API exposes a large number of resources with a small number of methods. The methods can be either the standard methods or custom methods. For this guide, the standard methods are: List, Get, Create, Update, and Delete.

Where API functionality naturally maps to one of the standard methods, that method SHOULD be used in the API design. For functionality that does not naturally map to one of the standard methods, custom methods MAY be used. Custom methods offer the same design freedom as traditional RPC APIs, which can be used to implement common programming patterns, such as database transactions or data analysis.

Whenever it seems tempting to add a method as noun to the API like "subscribe", it is RECOMMENDED to consider defining a new resource, e.g. "subscription".

Field name definitions MUST use lower_case_underscore_separated_names. These names will be mapped to the native naming convention in generated code for each programming language.

Field names SHOULD avoid prepositions (e.g. "for"), for example:

Field names SHOULD also avoid using postpositive adjectives (modifiers placed after the noun), for example:

The naming of query parameters (REST) and method parameters (gRPC) should match.

It is REQUIRED to use snake_case (never camelCase) for query/method parameters.

gRPC specific naming rules regarding packages, interfaces, resources, collections, methods, and messages.

All single responses MUST have the following basic structure:

_meta is an optional property containing additional meta information in a key value map. You MUST use the given definition "EntityMeta" under https://api.development.vwd.com/definitions/collections/basedefinitions.yaml#/EntityMeta

_attributes is an optional property and contains additional information for all attributes. You MUST use the given definition "EntityAttributes" under https://api.development.vwd.com/definitions/collections/basedefinitions.yaml#/EntityAttributes

_links is an optional property and contains links that lead to further information of a requested resource. You MUST use the given definition "EntityLinks" under https://api.development.vwd.com/definitions/collections/basedefinitions.yaml#/EntityLinks

All collection responses MUST have the following basic structure:

_meta is an optional property containing additional meta information in a key value map. You MUST use the given definition "Meta" under https://api.development.vwd.com/definitions/collections/basedefinitions.yaml#/Meta

_pagination is an optional property and contains information about pagination. You MUST use the given definition "Pagination" under https://api.development.vwd.com/definitions/collections/basedefinitions.yaml#/Pagination

_data MUST contain a collection of objects with all information from the requested resource.

Use Problem JSON in case of an error is a MUST.

RFC 7807 defines the media type application/problem+json. Operations should return that (together with a suitable status code) when any problem occurred during processing and you can give more details than the status code itself can supply, whether it be caused by the client or the server (i.e. both for 4xx or 5xx errors).

APIs MAY define custom problems types with extension properties, according to their specific needs.

Example

https://api.development.vwd.com/definitions/collections/basedefinitions.yaml#/ProblemResponse

For many people it doesn’t make sense to include hypermedia links and forms in responses because hypermedia clients won’t know what to do with them anyway. This is true and hypermedia clients need human understanding to understand the goals these links represent and to make a choice. The interaction between hypermedia clients and and the human can be summarized in Bill Verplank’s Do-Feel-Know model [verplank]:

The human (user) is required to feel (sense) the information presented by the hypermedia client (machine) and needs to know what he wants to do next. The machine is then instructed to run the HTTP action on the resource and present the result to the human.

This shows in what circumstances hypermedia is a RECOMMENDED choice. Whenever a client application should be developed, that is human assisted, it makes sense to maintain the client side state by using hypermedia links. When an API is designed without a user facing application in mind, hypermedia is less useful but can still be a valuable choice by eliminating error-prone URI construction.

Therefor we recommend…​

REQUIRED is a good implementation of REST Maturity Level 2 as it enables us to build resource-oriented APIs that make full use of HTTP verbs and status codes. You can see this expressed by many rules throughout these guidelines, e.g.:

Although this is not HATEOAS, it should not prevent you from designing proper link relationships in your APIs as stated in rules below.

We RECOMMEND to implement REST Maturity Level 3 in APIs for user-facing web applications. HATEOAS comes with additional API complexity without real value in our SOA context where client and server interact via REST APIs and provide complex business functions as part of an SaaS platform.

Our major concerns regarding the promised advantages of HATEOAS:

When embedding links to other resources into representations you must use the common hypertext control object. It contains at least one attribute:

In API that contain any hypertext controls, the attribute name href is reserved for usage within hypertext controls.

The schema for hypertext controls can be derived from this model:

The name of an attribute holding such a HttpLink object specifies the relation between the object that contains the link and the linked resource. Implementations should use names from the IANA Link Relation Registry whenever appropriate.

Specific link objects may extend the basic link type with additional attributes, to give additional information related to the linked resource or the relationship between the source resource and the linked one.

E.g. a service providing "Person" resources could model a person who is married with some other person with a hypertext control that contains attributes which describe the other person (id, name) but also the relationship "spouse" between between the two persons (since):

Hypertext controls are allowed anywhere within a JSON model. While this specification would RECOMMEND HAL, we actually don’t enforce the usage of HAL above other hypermedia formats.

Hypertext controls for pagination inside collections and self-references SHOULD use a simple URI value in combination with their corresponding link relations (next, prev, first, last, self) instead of the extensible common hypertext control

See Pagination for information how to best represent pageable collections.

It is REQUIRED to JSON-encode the body payload. The JSON payload must follow RFC-7159 by having (if possible) a serialized object as the top-level structure, since it would allow for future extension. This also applies for collection resources where one naturally would assume an array. See the Pagination section for an example.

If for given use case JSON does not make sense, for instance when providing attachments in form of PDFs, you SHOULD use another, more sufficient media type. But only do this if you can not transfer the information in JSON.

It is REQUIRED to wrap scalar data types (string, int32, float, etc.) to their respective wrapped data types (StringValue, Int32Value, FloatValue) when it is necessary to check fields for absence.

When using the wrapped types, the generated code has a hasXXX() function to check for the presence of a field

Below is a list with all wrapped data types. See also google/protobuf/wrappers.proto for the definition.

Background: gRPC uses Protobuf version 3 as transport protocol. This version of protobuf does not support optional fields anymore - all fields are required. So for scalar data types, a missing field is filled up with the respective default value, e.g. 0 for int32 or an empty string for strings. When it is important to distinguish between the absence of a field or default value of a field, the only technical way to determine is to use this wrapper types.

APIs SHOULD support techniques for reducing bandwidth based on client needs. This holds for APIs that (might) have high payloads and/or are used in high-traffic scenarios like the public Internet and telecommunication networks. Typical examples are APIs used by mobile web app clients with (often) less bandwidth connectivity.

Common techniques include:

Each of these items is described in greater detail below.

It is RECOMMENDED to compress the payload of your APIs responses with gzip, unless there’s a good reason not to - for example, you are serving so many requests that the time to compress becomes a bottleneck. This helps to transport data faster over the network (fewer bytes) and makes frontends respond faster.

Though gzip compression might be the default choice for server payload, the server should also support payload without compression and its client control via Accept-Encoding request header — see also [RFC 7231 Section 5.3.4](http://tools.ietf.org/html/rfc7231#section-5.3.4). The server should indicate used gzip compression via the Content-Encoding header.

Depending on your use case and payload size, you SHOULD significantly reduce network bandwidth need by supporting filtering of returned entity fields. Here, the client can determine the subset of fields he wants to receive via the fields query parameter - example see Google AppEngine APIs partial response:

Access to lists of data items MUST support pagination for best client side batch processing and iteration experience. This holds true for all lists that are (potentially) larger than just a few hundred entries.

There are two page iteration techniques:

There’re some rules and restrictions:

The technical conception of pagination should also consider user experience related issues. As mentioned in this article, jumping to a specific page is far less used than navigation via next/previous page links. This favours cursor-based over offset-based pagination.

The collection should contain additional metadata about the collection or the current page (e.g. index, page_size) when necessary. Links to the single items of you collection should be defined at every item.

You should avoid providing a total count in your API unless there’s a clear need to do so. Very often, there are systems and performance implications to supporting full counts, especially as datasets grow and requests become complex queries or filters that drive full scans (e.g., your database might need to look at all candidate items to count them). While this is an implementation detail relative to the API, it’s important to consider your ability to support serving counts over the life of a service.

If the collection consists of links to other resources, the collection name should use IANA registered link relations as names whenever appropriate, but use plural form.

Optionally you MAY return the number of items which would be returned by the query max. This means that we either get the actual number of the items. This value is an integer.

E.g. a service for articles could represent the collection of hyperlinks to an article’s authors like that:

Access to lists of data items CAN support grouping for best client side drill down into groups. You only have to implement that if requested.

If you want to support grouping you MUST do it that way.

You MUST add the optional query parameter groupings to the api request. You MAY have multible groupings at one request.

The group result items are of the same type as the not grouped list items.

Every single item in the list MUST have _meta (Hashmap of meta data) and _links.

Only the properties, whitch are grouped are filled. All other are empty.

You MUST fill into the _meta the count value, that contains the size of the elements in that group.

You Must fill the _links' with a `self link, that shows how to drill down into that group.

REQUIRED are content or entity headers are headers with a Content- prefix. They describe the content of the body of the message and they can be used in both, HTTP requests and responses. Commonly used content headers include but are not limited to:

The Content-Location header is OPTIONAL and can be used in successful write operations (PUT, POST or PATCH) or read operations (GET, HEAD) to guide caching and signal a receiver the actual location of the resource transmitted in the response body. This allows clients to identify the resource and to update their local copy when receiving a response with this header.

The Content-Location header can be used to support the following use cases:

When creating or updating resources it MAY be necessary to expose conflicts and to prevent the lost update problem. This can be best accomplished by using the ETag header together with the If-Match and If-None-Match. The contents of an ETag: <entity-tag> header is either (a) a hash of the response body, (b) a hash of the last modified field of the entity, or (c) a version number or identifier of the entity version.

To expose conflicts between concurrent update operations via PUT, POST, or PATCH, the If-Match: <entity-tag> header can be used to force the server to check whether the version of the updated entity is conforming to the requested <entity-tag>. If no matching entity is found, the operation is supposed a to respond with status code 412 - precondition failed.

Beside other use cases, the If-None-Match: header with parameter * can be used in a similar way to expose conflicts in resource creation. If any matching entity is found, the operation is supposed a to respond with status code 412 - precondition failed.

The ETag, If-Match, and If-None-Match headers definitions can be used from https://api.development.vwd.com/definitions/headers.openapi-2.0.yaml

Versioning MUST NOT be used.

Sometimes APIs can undergo changes without causing major problems, because Hypermedia connected RESTful APIs use link relations to hide changes in the URL format or because the message format of gRPC APIs allow for optional fields. Breaking changes are changes introduced into the resource, request and response design that cause errors on the consumer side.

So it is RECOMMENDED to rely on backwards-compatible changes in order to avoid unnecessary new endpoints and force users of the API to update. gRPC and RESTful APIs have different capabilities in terms of backwards compatibility so we have to differ between those in the following.

In order to make it transparent for the consumer how long an API will be supported it is REQUIRED to make a statement how long APIs will be maintained, when a newer version under a new endpoint is released. Old endpoints MUST be marked as deprecated.

The detailed list of gRPC backwards-compatible and non backwards-compatible can be obtained from the Google Cloud APIs Design Guide which can be applied to all gRPC APIs.