Is API Planning the same thing as API Design?

As an API Design Management company, we talk a lot about various ways of approaching API Design, and the discussions can sometimes get a little confusing. Some folks consider the “design” phase to encompass any and all forms of planning for an API, while others consider it to be the very technical act of “API modeling”: describing endpoints, requests, responses, property names, status codes, et cetera.

When the planning process begins is dependent on a bunch of factors, from how the business works, to the role you play in that business. When you talk to engineers, the planning process often begins with the technical tools used to plan. Some folks might start writing code prototypes early on, and other folks might start planning with a marker pen and a whiteboard. If you ask somebody in a more business orientated role, there might have been a collection of strategic conversations that the technical people were not part of, or user research conversations only the product team were privy to.

Often the discussions focus on API modeling and how it’s done: API Design-First or API Code-First. This is mostly due to this space having more tooling vendors: ourselves included. We don’t sell whiteboards, so we don’t talk too much about those. We help people describe their APIs, share documentation, and create mock servers for gathering feedback, so those are the parts of the planning process we talk about the most. This article aims to remedy that oversight.

There are three common concepts to talk about: API-first design, API Design-First, and API Code first. All of these names can be misleading, because “API-first design”, ad “API Design-First” are two different things, and “API Code-First” doesn’t mean there is no planning involved.

API-first Design in the Design Process

The term “API-first design” (or simply “API-First”) appeared as a response to organizations flopping out APIs that were rough, unusable, restrictive subsets of functionality copied out of the main application and often just databases-over-JSON. These poorly planned APIs were often not the fault of the engineers, but were rushed for the sake of organizations being able to say “We have an API!” in response to customer demand, even though those customers were rarely satisfied with what they got.

A popular mantra appeared in the API-first movement: “Eat your own Dogfood”, which meant that the company should actually use these APIs. If you’re using an API, you know if it works, if it’s slow, or if it’s difficult to use, so the plan was to identify stakeholders early on and make sure your API would be useful to them.

“API-first design means identifying and/or defining key actors and personas, determining what those actors and personas expect to be able to do with those APIs.”

Source: Kas Thomas, 2009, API-First Design

Many of these stakeholders are API client developers, working on various frontends or other APIs/services. They get together, talk about needs, and the API team would then try to solve those needs. This philosophy only got more popular with the rise of mobile apps, JavaScript frontends, and microservices.

With stakeholder requirements being more commonplace in the world of API planning, making sure the API meets the needs of those stakeholders was the next challenge. Especially important was making sure there was enough time to trial the API out, gather and implement feedback, and integrate the API into the applications without taking forever or rushing.

API Design-First

Design-First doesn’t just mean “we’re gonna plan this API”, it means “we’re going to spend more time in the design phase, with automated tooling, review boards, and stakeholders giving feedback on those designs early and often, so that when we do write code, there’s a really good chance we’re writing the right code.” Designing APIs with the design-first approach is a must. 

The API Code-First workflow can involve a lot of this same stuff, so lets look at how both workflows can go about planning APIs to be the best they can.

Strategy

Seeing as we don’t just build APIs for the sake of it anymore (hopefully) there must be a reason we are building this API.

Any API program has to start with a Strategy phase where you will have to define the goals you want to achieve with your APIs. It also specifies which markets you need to address, which resources you have at your disposal, the time frame you have in which to achieve your goals, and so on. This is a critical step to help you put effort where it matters the most.

Source: Nicolas Massé, Full API Lifecycle Management: A Primer, 2019

Define what the problem is at a very high level. Maybe there is a technical problem with a poorly written old system: maybe it’s become unstable, or the codebase has become an irreparable twisted nightmare of sloppy rewrites. There could be a functional requirement like adding monthly subscriptions to a donations system that previously only handled one-off payments, or something even bigger.

End-User Research

Irakli Nadareishvil points out in Microservices: Up and Running, API teams will deal with two distinct personas in their research activities: end-users that the APIs enable the experiences for, and API client developers that code those user-experiences (i.e. mobile developers). According to Irakli, when his teams design the APIs they conduct research separately for those two teams. With end customers they collect Jobs To Be Done, described in very non-technical terms.

These jobs are used to design API specs. Once the first draft of the specs are ready, Irakli advocates conducting a second series of end-user-research, where API client developers provide feedback regarding the usability of the API design. This way, says Irakli, design research is split in two phases: first one makes sure that the right solution is designed, the second one making sure that the solution is designed the right way addressing both the “what” and the “why” aspects of the design.

This information can be gathered in various ways:

• Asking Customer Support for problems relating to the topic at hand
• Asking Product for insights from tools like Productboard
• Asking API client developers what relevant problems they’ve had
• Directly surveying end-users to ask what use-cases they have

Directly asking questions can be tricky, so taking a lesson from UX Researchers is helpful. Focus on asking non-leading questions to avoid tainting results with biased questions. Don’t ask, “Would you prefer X over Y?” but ask, “Does X satisfy your needs, and if not, what would you change?”

However you go about it, find a way to make sure you’re undertaking will satisfy a good number of users; otherwise, you’re just guessing, which is gambling with company time and money.

Gathering all this feedback can help acquire the Jobs to be Done which will facilitate the rest of the planning and design process.

With a better idea of what your users are going to be doing, it’s time to grab a whiteboard, or a collaborative sketching application, and get doodling this API and how it fits into the API Ecosystem.

When planning an RPC API like gRPC or GraphQL, or an event-driven API like RabbitMQ or Kafka, consider the commands/events that will be sent up and down the wire. These diagrams usually take the form of a Sequence Diagram.

When planning a REST API, consider the resources that will exist, usually in the form of an Entity-Relationship Diagram. That helps with the basic resources, but as REST is a State Machine over HTTP, it can often help to plan those workflows out with a Data-flow Diagram, or maybe another Sequence Diagram to ensure more important workflows look smooth for consumers.

Design

As we mentioned, some folks consider the “Design” phase to be “all planning” but others refer to the Design phase as the part where you start to describe how the actual API implementation is going to look. API Design is the process of creating what many of us these days call a “API Description”, but others will refer to as an “API Contract”, “API Schema”, or “API Specification”, for your API. There are so many terms for basically the same thing in the world of APIs!

For years this phase was done by writing up some sort of document, often a Word document or Markdown file, explaining at the most basic level what endpoints there will be, what requests look like, what responses look like, various validation rules for certain parts of the messages, how pagination is going to be handled, what sort of security credentials are going to be required where, and what types of errors clients could expect to see.

Many developers would get to this point and just start doing that in code, not seeing much point in doing this first only to write it all up again later.

If you’re sticking to the API Design-First workflow, instead of describing the API in words or code, you’re leveraging various API Description Formats like OpenAPI, JSON Schema, AsyncAPI, GraphQL Types, or Protobuf.

Creating these machine-readable descriptions can be done with graphical editors like Stoplight Studio, writing them by hand, leveraging a DSL, or any other approach that helps avoid the temptation of creating a code structure right away. With descriptions, we can focus on the design, and get feedback from stakeholders and consumers.

Check out our API Design Guide for more on this incredibly important step.

Mock

The term “mocking” in the context of planning an API is a little different from the various sorts of mocks that show up in API testing or unit testing, but it’s a similar idea. The idea is to create a fake API that implements the contract you just designed, giving various people the chance to play around with integrating with the real API. At this phase, problems discovered are easier to solve than later, when they’re fully coded, or in production!

If you’ve started writing code already, you can just return a bunch of static JSON from a mostly empty controller, and keep tweaking that based on feedback. You have full control over the mock because it really is just a bunch of code with data structures. You can choose to implement some business logic here if you like, but be careful not to implement too much too soon: it’s just more code you’ll need to throw out and refactor if the feedback is not positive.

If you’re using the API Design-First workflow, there’s no need to start writing code just yet. API descriptions can be used as an input to mock servers. To mock HTTP APIs there are several easy CLI-based tools for different API description formats:

• OpenAPI
  • Stoplight’s Prism
  • Falcon Heavy
  • Bunch of other Mocking Tools
• GraphQL
  • graphql-faker
• gRPC
  • gripmock

Whether you create mocks through a manual process, or a one-line command in the terminal, sharing mocks can be a handy way to solicit quality feedback early on, before investing lots of time in creating prototypes.

Feedback

With an excellent API design, you can gather feedback from a few people:

• API Client Developers that plan to use the application
• API Governance or whoever does API design reviews at your organization

This phase is all about finding out early on if the API is going to be any good. It might have looked good on paper, but no plan survives first contact with the API clients. You’ll know which client developers to ask for feedback because you already identified which teams would be stakeholders in the strategy phase, and spoken to the people on those teams during the user research phase.

Get them playing around trying to integrate with the mocks you’ve created, and tweak those mocks until they’re giving positive feedback.

• Are they unable to figure out how it works?
• Are they missing crucial bits of data?
• Is the API overly-normalized, meaning they have to make 1000 calls?
• Is the API under-normalized, meaning they get way too much information in 1 call?

If you’ve got API descriptions (especially OpenAPI) then you’ve already got API reference documentation, just render it to HTML/PDF for them. If you’ve not been creating API descriptions, now would be a good time to create some documentation. Otherwise, the API consumers might not know how this API works, and that will not lead to very useful feedback.

The harder part of the feedback phase is figuring out how to get the mocks to the API client developers.

API Code-First folks can package the code up in Docker and give the client developers instructions on how to run it locally, or hosted on a semi-realistic hosting environment. Any feedback will likely come in the form of posting issues or writing text documents with problems or sending pull requests if the client developers are familiar enough with the programming language chosen.

API Design-First folks have a similar choice: they could give a docker or something like Prism to client developers to run on their local machines, or use hosted mock solutions like Stoplight’s Mock Servers which update automatically whenever someone pushes to OpenAPI documents to the project. Feedback can be created through posting issues, in text documents, or they could even send their feedback in the form of a modified API description in a pull request! Unlike the code-based approach, there’s no need for those client developers to figure out a programming language to make these tweaks, just edit the YAML, or whatever DSL, or use the Form view in Stoplight Studio.

Prototyping

Prototyping is the planning phase where API Design-First and API Code-First join up, and things become somewhat similar. After going through the mock, feedback, and improvement cycle a few times, it’s time to create a more realistic API. A prototype could be the beginning of the real thing that you iterate until it’s ready to go into production, or it could be a proof of concept in a programming language and framework that leans more toward Rapid-Application Development that is eventually replaced with another piece of code later.

Once the API consumers have successfully integrated the prototype (hopefully rather seamlessly if the mock phase went well), then they could potentially release a pilot project or early “alpha” client to a group of dedicated end-users who have somehow declared interest in “alpha testing” new functionality. If feedback suggests this is a mess, things can be rewritten quickly.

For example, knocking out a prototype API in Ruby on Rails or Laravel (PHP) is considered pretty quick and easy, and if rewrites need to happen that’s going to move much quicker than something like Go or Java, which is slower to write but handles a bajillion users better. Developer speed is a focus for the prototype, whilst performance is a focus for the final codebase, and those things are not always found with the exact same language or framework.

Whatever approach and languages you use for prototyping depends very much on the engineering teams involved, the size of the organization, and the amount of time you have available. The idea of prototyping is to prove what the mock server cannot prove: beyond the design of the contract, does the real business logic check out?

Again, another feedback cycle happens, and eventually, you’re done. At this point, it’s all about Developing an API, and the planning phase is over… that is until you need more functionality added!

That Sounds Like a Lot!

Depending on the sort of experiences you’ve had working at various sized companies, spending 6-12 months planning and build API could sound like a long time or no time at all. I’ve worked at places that spent 6 months failing to deliver a simple API due to barely any planning, and I’ve worked at places which knock out a fantastic API in 3 months due to nailing the planning and design process.

An important thing to realize here is that this is not a waterfall process, with one part being done, then another, then another, with each phase blocking the next. What you really have here are several feedback cycles happening in asynchronous loops.

API client developers are not waiting months for an API that’s already been written then rushing to integrate it, they’re integrating with mocks and prototypes that are edging their way towards completion, which makes the integration efforts asynchronous instead of a all-at-once Herculean push before the deadline.

Keeping client developers involved, included, and heard reduces the chances of them needing to spend time building Backends-for-Frontends (BFFs) to shape useless APIs into something they can work with.

More importantly, the people producing and consuming the APIs working together closely to define what needs to be done, which makes it more likely the API is going to be useful when its done. Plus with end-users involved there’s a better chance that what the API client developers build will be useful for those end-users, reducing the chances of needing a v2 in a few months.

Big-up-front planning and iterative design loops providing constant feedback is the most agile you can be, and overall wastes substantially less time and money, especially if you can automate chunks of your API style guides.

The API lifecycle is a big one, and there’s a lot more to it than just writing code. Code is possibly the least important part of building an API.

Image Source: Nicolas Massé, Full API Lifecycle Management: A Primer

Further Reading

• For a totally different take on this topic, read API Design Methodologies, for great input on using Event Storming, API Modeling, and more examples of Sequence Diagramming.

• Take a spin through our API Design Guide to learn how to design APIs with our tools.

• Peruse our API Development Guide to see how having API Descriptions can speed up development and getting to production after the design phase is complete.

Now you can design an API the right way!