API - more than it seems

2022-12-20 1764 words 9 minutes

Contents

In this article I’m going to explore what an API is, why we might want to change it, how to do it and how to do it safely.

The context

In business world people are acting confidently only if they have proper, sound legal environment (legal rules are known and well established) and abide to business contracts that they created and/or that they want to stick to in order to do their business.

In programming world people may conduct their programming businness also in the scope or in the frame of some rules and those rules appear on many levels:

funcional requirements define what a system is suppposed to do or how it is supposed to process its inputs
non-functionals say about its behaviour, for example how the system should react in case of high load or what is the expected latency
well-established programming patterns, those coding tricks that got their own name
packages and modules used in a meaningful way, mainly used to promote code isolation
specific programming style (often imposed by a framework, i.e. thinking in terms of beans or services and repositories)
programming paradigm (functional or object-oriented; polimorphic or prototype-based; open or closed world)
libraries (including standard library)
build system (several aspects:managing dependencies; making builds “bit-perfect” repeatable; changes of the build system itself)
language itself (statically vs dynamically typed, low level or general purpose, with huge standard library or with tiny one)

Expectations

The only thing that would make us, programmers, really happy is that the above rulesets don’t change. And if they change - as we all know, there is no place to hide from the inevitable change - they ideally do so in a manner that is somehow gentle, safe, well-thought; such change should really make the world a better place and make prorgamming a better experience.

We want backwards-compatible evolution of libraries and frameworks.
We want a language which feaures makes heavy programming patterns obsolete (not as an idea, but as implementation suggestion, in their literal, oo-based form displayed on diagrams).
We want systems architectures that scale and behave well without the need to re-write huge parts of the codebase.
We want programming environments or runtimes where our old code would run without issues.

Now, having all those “wants” makes us aware of our expectations. At the same time, as adults, we are responsible for some of those things to actually happen. I know, not everyone of us is actively engaged in development of the language or framework. However, we are engaged in writing our own code. This code has a good chance of being called and executed. And there is also a chance that someone else’s code is going to use our code. At this specific moment our code becomes an API.

What is an API

API - application programming interface - is the contract that starts at single function level. A function defines interface to our implementation (in Java we even have @FunctionalInterface annotation which marks single-method interfaces). The contract is defined by function signature (name parameters, result type) and, once established, must stay unchanged.

Of course, there’s more to that than it seems. The signature does not provide enough information (at least in Java language) for the user to be able to conscously use it. Programmers often don’t have other choice but to look under the hood:

we don’t know if the function is a pure function or not (purity keeps us safe from many traps like stale caches or global state modification)
we don’t know if it may block; if it blocks, wether or not it timeouts; or if it is meant for asynchronous uscase and, once called, it will immediatelly return and be potentially executed in the future
we don’t know how heavy it is on our system (how many threads it spawns, for example)
we don’t know the probability or conditions for exceptonal temination (unchecked exceptions, panics, failures)

How to cope with so many unknowns?

some of those issuses may be explained in documentation
some pain can be alleviated by applying specific annotations (@Pure, @NonNull, @Async) None of the above is enforced by compiler, so putting it in the code is the responsibility of the function creator.

As humans, we are not very good at remembering things and we can easily forget to properly document such features. Therefore, as far as possible, we should promote a way of programming where those annotations and explanations are either:

not applicable or
can be explicitly expressed and checked by the compiler.

API evolution

There is one single rule for good API evolution expressed nicely in API Design – API Evolution & API Versioning:

externally observable behavior of an API (from the perspective of the clients) cannot be changed, once the API has been published

Does this rule hold for internal APIs? I’ve been discussing this many times with many colegues. Almost all of them were saying that until it is published, you can do with the API whatever you want. I usually advise them to be careful: internal APIs may have internal clients and breaking changes mean additional expenditure on code updates, test fixes, discussionsa and ocumentation upgrades. No worries, we all work in agile way, such changes are an acceptable part of agile flow.

However, if the API goes public, no incompatible changes are allowed. We need to make the API backwards compatible to sure that old clients can use old versions of it without issues and forwards compatible so that new code can use new features.

Compatible and incompatible changes

Depending on the context (language, environment etc), “(non)compatible change” means different things. One aspect remains the same and the whole discussion bols down to: don’t break existing clients. This rule is called “API Prime Directive” in 3 and is the basis of all considerations in this area.

Web applications

For example, when it comes to REST APIs, there is some form of general agreement regarding compatible and non-compatible changes. The changes that can be safely introduced and won’t break a (well-behaved) client, are:

adding query parameters (which should never be required)
adding headers
adding new (and necessarily optional!) fields in JSON or XML documents
adding endpoints
making existing parameters optional

And incompatible changes are:

removing a data structure (object representation) or changing its shape
removing fields (instead of making it optional) from reqest or response
adding mandatory field to data structure
changing optional request field into mandatory
changing mandatory response field into optional
changing URI (host, port or path)
change the structure of the response, eg. make response structure part of a a wider structure used as a response

The proposition that those actions are (non)compatible are based on conventions (the ability to add additional HTTP header is based on the assumption that clients won’t cry if they see more headers than in previous version of the API) and usually are hard to enforce automatically. Therefore it is crucial to prepare tests that would alert developers if the regression happens: new API cannot be called by old code or new code fails when faced with old datastructures.

Pure code

My experience is manly with Java code and therefore I’m focusing here stronlgy on Java language. In one of my favourite articles on Java, Evolving Java Api, the authors point out that there are two aspects of compatibility:

API contract compatibility: With API contract compatibility, changes to API specification may make existing code behavior incompatible with the specification.
binary compatibility: With binary API compatibility, pre-existing client binary (Java .class files or packages/modules containing .class files) must link and run with new releases without recompilation.

Things are getting complicated because often, instead of component (having an API) and client (using the API) there are actually three actors on the stage: API specifier, API implementor and API user. And this makes the API evolution even harder, for example: strenghtening method preconditions are breaking for callers and good for implementors, and loosening method precoditions is good for callers but breaking for implementors (see examples in Evolving Java API, part 1)

Binary compatibility means the ability to evolve:

The Java Language Specification has a whole chapter about it - this is a “must read” for library authors - with newest addition related to sealed and non-sealed classes and interfaces.

I like to udertand how the changes I introduce in my code would influence the users (if I had those) - although I’m not a library author, I may need this knowledge in the future, for example when conducting a code review.

API versioning

How to propertly create API versions? Again, things are different (but not so much - in essence the same problems apply) - on both fronts: web and pure code. In case of non-compatible change you need to release new API version.

For web APIS, most often used method for making all APIs available for variety of clients is to:

use different URI prefix with API version
```
https://domain.com/v1/API
```
differentiate between API versions used by server based on request headers
```
Accept: application/json; version=1
```
or even having mime-type pointing to propert JSON schema version
```
Accept: com.domain.v3.application/json; version=5
```

For code APIs, you have several options as well:

deprcate the old code (use @Deprecated annotation and point to an alternative or newer API), provide bugfixes for both old and new API and remove depracated API at the end
create a new package; old one may coexist with the new one (and clients should use the newest one)
add “2” to the name of the new class/interface (and maintain both classes/interfaces)

If a breakage is necessary:

try to minimize the scope of incompatibility (ie. narrow down to single package or specific usecase)
make incompatible old code break immediately instead of work in unpredictable way
document well both your intention to break compatibiliyt in future releases
mark the APIs that you’re goint to remove in future releases as @ForRemoval

Good practices

API Evolution Right Way resource lists good practices for API evolution. I especially like those:

avoid bad features
minimize features (keep features narrow)
maintain a changelog

Additionally to good practices which you should be familiar with, you can also use tooling (for example, https://lvc.github.io/japi-compliance-checker/) which allows you to compare two versions of the library and points out possible imcompatibilities.

API design is an art in itself and requires a lot of experience. The above practices focus on essentialism: do only what is necessary, be careful abour what you promise to deliver and be aware of how the system evolves.

Happy hacking!