Application troubleshooting continued

Providing REST API:

StreamProcessing is usually bound to a stream in the stream store which it accesses over the gRPC based connectors. But the usefulness and convenience of making web requests from the application cannot be written off. REST Apis are independently provisioned per resource and the services providing those resources can be hosted anywhere. This flat enumeration of APIs helps with the microservice model and available for mashups within the application. SOAP requires tools to inspect the message. REST can be intercepted by web proxy and displayed with browser and add-on.

SOAP methods require declarative address, binding and contract. REST is URI based and has qualifiers for resources. This is what makes REST popular over gRPC but the latter is more suited for IoT traffic.

There is a technique called HATEOAS where a client can get more information from the web API servers with the help of hypermedia. This technique helps the client to get information outside the documentation by machines instead of humans. Since the storage products are considered commodity, this technique is suitable for plugging in one storage product versus another. The clients that use the web API programmability can then switch services and their resource qualifiers with ease.

Some of the implementors for REST API follow a convention.  This convention makes it easy to construct the APIs for the resources and to bring consistency among those for different resources.

REST Apis also make authentication and authorization simpler to setup and check. There are several solutions to choose from with some involving offloaded functionality or the use of frameworks like Java which can support annotation-based checks for adequate privilege and access control. 

Supported actions are filtered. Whether a user can read or write is determined based on the capability for the corresponding read or write permission. The user is checked for the permission involved. This could be an RBAC access control such that it is just a check against the role for the user. The system user for instance has all the privileges. The capabilities a user has is determined from the license manager for instance or by the explicitly added capabilities. This is also necessary from audit purposes. If the auditing is not necessary, then the check against capabilities could always return true. On the other hand, if all the denies were to be audited, it would leave a trail in the audit log.

One of the best advantages of using REST APIs in application code, is that it can bring in services with little or no dependencies. External services can now be leveraged in the application by simply making a web request. Each invocation of the web request can now generate events and these events along with the data can also be subscribed via a store the persists events or via webhooks. 

In this regard, the application can save both events and data in a stream store. A publisher-subscriber can then be implemented over the stream store.  The persistence of side-outputs and events to a stream enable their querying via stream processing language which would otherwise not be possible.

There are many applications in the K8s marketplace that follow a similar model except that they replace the use of a stream store with a one-off database for their purpose. Some of them also install a separate database such as postgres, badgerdb, and other stores other than the etcd that is available natively on the K8s. These stores come with their own burdens of requiring expertise, troubleshooting and performance tuning which increases the cost for the user. Instead, the use of the stream store scales well to all the runtime events because the events will seldom be as many as the data events.  The use of a stream store with the same model as these products will drive down costs for the end users.