In this 2nd post dedicated to API Architecture re-design, we show how the API Life-cycle and Developer Portal can support your API Management strategy.
The weekly mood
My first quarter as a full-time Architect ends in 3 weeks. I realize that even if I had defined some goals, path and methodology during my first week, I had built no 30-60-90-day plan to build the ground for a successful job or career. I know about that magic number of 100 days when people start to wonder if their new CEO or VP delivered as promised, and I assume this happens also to employees. But the most important to me was to do the right think rather than to secure quick wins.
After ramping-up in the area of DevOps and working a bit on our Data Engineering, I am now committed to our API Architecture re-design. In a previous post, we focused on the following 2 building-blocks:
- API Development: We emphasized on OpenAPI Specification v3 (OAS3) as a Contract-First development approach, implemented a Microservice using Spring Boot and deployed it as a container to Kubernetes.
- API Gateway: We setup Gloo and Kong Gateways, then accessed our API through service discovery and routing. We evaluated the two solutions and their different approaches to fined-grained traffic management, security, observability and administration.
In this article, we set the focus on the 3rd building-block of the API Architecture: API Management (APIM). One of the problems we have is not being enough fast and transparent at publishing and consuming internal APIs. We believe to be able to solve that problem using a highly-configurable Gateway along with an APIM or Developer Portal.
What is APIM
Almost 10 years ago, a new approach emerged for applying SOA Governance to native HTTP APIs. While SOA Governance (mainly) based on a Proxy and a Registry component, APIM would (mainly) base on a Gateway and a Portal component. With these new terms, secure and store functions become just more transparent and collaborative. In the mean time, APIM has gone all the way through the complete Hype cycle and will keep being extremely relevant until the next major technology shift occurs. Since Governance is much easier in theory than in practice, APIM commonly refers to everything around handling APIs, from strategy to platform.
"API management is the set of people, processes and technology that enables an organization to safely and securely publish APIs, either internally or externally. Common components include an API gateway, developer portal, and administrative UI with reporting and analytics capabilities. Some API management solutions include monetization capabilities." [Source: Gartner glossary]
Because of this Fuzzy concept, you may find a different APIM Reference Architecture from each different APIM solution provider. Marketing blogs also often "misuse" the term APIM while referring to the API requirements that cannot be addressed by other known components like the Gateway. In order to avoid this mistake, I suggest to pragmatically recenter the topic of today to API Lifecycle and Developer Portal.
It should be fair to assume that most API platforms try to stay competitive across the whole API Architecture, some being for example better on the Gateway side, and others on the Portal side.
Remind that an API is a living asset that is continuously subject to change. Beside Version control and DevOps automation, you need to define and monitor your API Lifecycle governance otherwise you will soon loose overview, control, and revenue in case of API Monetization. The following goes through the motivation and impact of example API status Created, Published, Deprecated, Retired.
Once the API is created by design, it receives a contract version which the logic in development stage, even going through multiple image versions, will stick to ex. myAPI-conractV1.1-imageV2.15. It is by the way a good practice to add some automatical "diff" between contract and logic as part of the integration tests.
The implementation iteratively gets promoted to QA and Integration environments until it gets positive feedback for release to production. The API is then published to the Gateway. We notice from the last step a status change from "live" to "exposed" and "serving traffic". Even when published in production, some security hot-fixes, runtime maintenance and issue patches will apply.
New-coming API might offer an alternative for the API previously published. It is common practice to keep operating one or multiple versions in parallel. The "old" API is set deprecated until the API provider completed their deployment and API consumers completed their upgrade. Indeed, an upgrade is critical in case breaking-changes are introduced or forward/backward compatibility is excluded.
Because it still generates costs on provider side, and potential constraints on the subscriber side, both the API reference and subscription plans will discourage any further use of the old API. Its end-of-life might be announced publicly or via E-Mail to the subscribers. So that they are aware that support and maintenance will be cut-off soon and the API retired.
Source: The 4 Stages Of API Lifecycle by Teks Mobile
A number of APIM providers like for example Akana have specialized in API Life-cycle management software, which I assume to be useful for given branches like for example Retail that are outsourcing their API development, and therefore need extended governance and collaboration. Since my organisation still develops own APIs in-house, we do see a need for API Life-cycle governance, but not yet as a dedicated software product.
An API-centric Developer Portal is a kind of Web Portal that provides a User Interface (UI) to all API-related information, better to Self-service applications. In a Nutshell, it allows for for API developer and subscriber communities to publish, consume and manage APIs.
According to Capgemini in their article "Separating API management from APIs", APIs are actually managed and governed through 3 key personas:
- Application Developer (API Consumer)
- Goal:
- Easily integrate existing API into their application.
- Need:
- On-boarding and Subscription.
- Expected features:
- Catalog of API References and Use-case scenarios. Not "Swagger-only" but comprehensive documentation in natural language by API, resource and version.
- Code snippets. Not "REST-only" but also SDK or popular programming languages.
- Generation of access keys and billing report.
- Service Provider (API Publisher)
- Goal:
- Fast implement and deliver quality-proof API.
- Need:
- Guidelines and Publication.
- Expected features:
- Standardization for example of how version, pagination, logs and errors are handled.
- Submission of contribution and changes through collaboration workflows.
- Integration with Version control and/or Automation pipelines (CI/CD).
- API Manager
- Goal:
- Comply to SLAs and drive API revenue.
- Need:
- Reporting and Accountability.
- Expected features:
- Dashboard of API stats and reports.
- Collection and analysis of API usage, subscription costs, availability and performance metrics.
- Integrate with API Gateway monitoring and policies.
Obviously like for any other pieces of software, we will also need an owner of the portal layout, a user administrator, an application maintainer and an infrastructure operator. The related portal features can be provided by different portal applications, or a single one via Role Based Access Control (RBAC) once the user is authenticated.
Based on my research, there are different kinds of possible approach, technology basis and maturity levels adopted by API Developer Portals:
- Static HTML portal generator from OpenAPI contract, using standard Redoc or custom Jekyll templates.
- Pro:
- Simple.
- Contra:
- Re-invent the wheel followed by high development and maintenance cost.
- No Integration with API Gateway.
- Not in sync with changes.
- Limited to read-only consumption.
- Content Management System (CMS)
- Pro:
- Content collaboration.
- Layout administration.
- Contra:
- No Integration with API Gateway.
- Not in sync with changes.
- Limited to read-only consumption.
- Off-the-Shelf API Developer Portal
- Pro:
- Content collaboration.
- Layout administration.
- Integrated and in-sync.
- Contra:
- Limited customization.
An API Gateway might ship together with an API Developer. As already mentioned in my previous post on Open-Source API Gateways, Kong- and Gloo-Developer-Portals are not only part of the enterprise subscriptions, as opposed to Ambassador's which can be used for free up to 5 users and 5 APIs per user. For the purpose of learning, here is my non-exhaustive shortlist of open-source API Developer Portals: Gravitee, WSO2, (RedHat) APIMan, (Salesforce) Mulesoft.
Note that major Cloud providers also open-sourced the frontend of their API Developer Portal, however those frontend alone seem to be useless without the corresponding backend availability:
- AWS API Gateway Developer Portal Generator is based on AWS Serverless Application Model (SAM)
- Azure API Management developer portal is based on NodeJS, Azure Blob Storage and Azure API Management API
- Google Apigee Developer Portal and IBM API Connect are based on Drupal CMS
In addition, there seem to be a Niche for API Monetization software and API Marketplace service like RapidAPI. Indeed, some organizations are definitely willing to drive revenue out of their APIs, but not relying on APIM software or services for their API store. We will leave this topic away for now.
What is Gravitee
Gravitee.io (formerly Graviteesource.com) builds a fully Open-Source API Platform which shall be easy to use and cost-effective for organizations who want to secure, publish and analyze APIs. It is still a young project with many breaking changes, e.g. they had to skip v2.x between v1.x and v3.x.
The enterprise product extends to Access Management, Alerting atop of a hosted and supported deployment, or to a fully managed and scalable service (SaaS). The platform is Java-based and extendable via Plugins. It is used by a small number of customers in western Europe.
Setup
We will deploy Gravitee to Kubernetes using Helm charts.
$ kubectl create namespace gravitee namespace/gravitee created $ kubens gravitee Context "microk8s" modified. Active namespace is "gravitee".
A very confusing point is that the installation guide suggests a minimal configuration with Helm default values, which is actually not exactly right. Indeed, you need to configure the APIM integration with a MongoDB replica-set and an Elasticsearch cluster, otherwise the API server will not start properly and the other components will be useless. Gravitee can install both of them as part of its Helm Chart:
- Gravitee Gateway (not part of the evaluation)
- Gravitee Management API
- Gravitee Management UI
- Gravitee Portal UI
- MongoDB replica-set (optional dependency)
- Elasticsearch cluster (optional dependency)
Because of Gravitee default Helm Chart using deprecated ElasticSearch settings, and other pitfalls with persistent volume claims in microk8s, I decided to deploy ElasticSearch separately from Gravitee.
$ helm repo add elastic https://helm.elastic.co $ helm install \ -f https://raw.githubusercontent.com/elastic/helm-charts/master/elasticsearch/examples/microk8s/values.yaml \ --name elasticsearch elastic/elasticsearch \ --set replicas=1,minimumMasterNodes=1 $ curl "http://$(kubectl describe \ svc elasticsearch-master | \ grep Endpoints | grep 9200 | \ awk '{print $2}')/_cat/health?pretty" 1599226906 13:41:46 elasticsearch green 1 1 0 0 0 0 0 0 - 100.0%
Now we may deploy MongoDB and the APIM.
$ helm repo add graviteeio https://helm.gravitee.io $ helm repo update $ helm install --name graviteeio-apim graviteeio/apim3 \ --set mongodb-replicaset.enabled=true \ --set mongodb-replicaset.replicas=1 \ --set es.endpoints="{http://elasticsearch-master.gravitee.svc.cluster.local:9200}"
Note that the APIM services are mapped by some Ingress route resources:
$ kubectl get ing NAME HOSTS ADDRESS PORTS AGE graviteeio-apim-apim3-api-management apim.example.com 80, 443 8m graviteeio-apim-apim3-api-portal apim.example.com 80, 443 8m graviteeio-apim-apim3-gateway apim.example.com 80, 443 8m graviteeio-apim-apim3-portal apim.example.com 80, 443 8m graviteeio-apim-apim3-ui apim.example.com 80, 443 8m
Mapping paths:
$ alling=$(kubectl get ing | grep -v NAME | cut -d' ' -f1 $ for i in ${alling[@]}; do \ echo "$i -> $(kubectl describe \ ing $i | grep ' /' | awk '{print $1}')"; \ done
graviteeio-apim-apim3-api-management -> apim.example.com/management graviteeio-apim-apim3-api-portal -> apim.example.com/portal graviteeio-apim-apim3-gateway -> apim.example.com/gateway graviteeio-apim-apim3-portal -> apim.example.com/ graviteeio-apim-apim3-ui -> apim.example.com/console(/|$)(.*)
We need an Ingress controller for accessing those services from outside the cluster:
$ helm install --name nginx-ingress stable/nginx-ingress \ --set rbac.create=true \ --set controller.publishService.enabled=true
And map the hostname api.example.com with Load balancer IP:
$ echo "$(kubectl get svc | grep nginx-ingress-controller \
| awk '{print $3}') apim.example.com" | sudo tee -a /etc/hosts
Gravitee APIM Console / Management UI
This UI is dedicated to the API provider, i.e. API Developers and Managers.
$ sensible-browser https://apim.example.com/console/#
Once logged-in with admin/admin, we will proceed with some overall settings:
- Adjust the Portal layout (Settings > PORTAL > Theme)
- Create API categories (Settings > PORTAL > Categories)
the first step is to create an API from scratch, or import an existing Swagger specification like this:
- Add a new API from default page (APIs > (Portal) > GENERAL > Details > +).
- Import from link petstore-oas3.yaml and activate the checkbox "Create policies on paths" from "Swagger import configurations".
- Start the API using the button from the "Danger zone".
- Optionally, you can:
- Add a custom picture (APIs > (Portal) > GENERAL > Details).
- Add a Swagger documentation (APIs > (Portal) > DOCUMENTATION > pages).
- Add free labels or select some centrally defined categories.
At this point, the API is more or less a record of the Management database. It has a name, version, context-path, endpoint etc. The top header in yellow reminds you that the API configuration might be out of sync with the Gateway, and therefore needs to re-deploy at any change. This action is automated at Publish time.
API Lifecycle status: created
We need to create a Plan in order to control the subscription of consumer applications:
- Add a "Petstore plan" (APIs > (Portal) > PLANS > Plans > +)
- Activate auto-validation and click NEXT
- Select "Api-key"-based authentication and click NEXT
- Skip the Gateway restrictions by clicking NEXT
- Add "Api key" to the policy chain and click SAVE
- Click on the blue cloud icon to publish the API Plan
- Configure the API Proxy settings (APIs > Proxy)
- Change the context path to /petstore (GENERAL > Entrypoint)
- Enable Cross-origin resource sharing, allow * host and GET method (GENERAL > CORS)
- Change the endpoint to http://petstore.swagger.io/v2/store/inventory (BACKEND SERVICES > Endpoints)
- You can optionally activate Health checks in order to monitor the availability of the API service endpoint
API Health checks
- Click on PUBLISH API within the "Danger zone"
- You can keep the API private if accessing the Portal with the same user, or MAKE PUBLIC otherwise.
Note: The Management UI also allows to deprecate and retire the API.
Gravitee Portal UI
This UI is dedicated to the API consumer, i.e. Application Developers.
$ sensible-browser https://apim.example.com/
At this point our API and its documentation are available for discovery from the Portal Catalog.
API Lifecycle status: Published
Click the SUBSCRIBE button, select the default application and submit request. It is automatically accepted as previously defined as part of our Petstore Plan configuration. You could also setup some approval workflow and notification.
Once completed, the UI says that you are ready to use the API as follow (replace the key accordingly):
$ curl "https://apim.example.com/petstore" \
-H "X-Gravitee-Api-Key:4fafce1d-6080-4654-b6d7-c5aa4ec0611c"
Note: You also need to trust (--cacert) or ignore (--insecure or -k) the certificate.
Automation
Both the Management Console and the Portal are also accessible via a dedicated CLI and a technical REST API.
Take away
Gravitee Platform is very interesting for educational purpose, but probably a bit too complex for our particular purpose. It does not integrate so well in Kubernetes and generally lacks some guidance. The documentation contains broken links and is incomplete. An employee reported that it is maintained by the developers only while the open-source community is still weak or disorganized.
At the same time we were evaluating Kong and Gloo Enterprise. It became clear to us that we will probably need a support SLA that usually comes with additional proprietary features, in this case Kong and Gloo Developer Portals. From what we could see so far, those products are more comprehensive than Gravitee but less complete (no support for CORS, versioning and fine-grained portal authorization etc.).
See you in a future article, where we will discuss about API Operations (APIOps).
Comments
Post a Comment