The importer has one challenge different from the exporter. The destination for the exporter can accept multipart upload but it has limitations in sending the same payload back other sending it back as an OutputStream which makes importer to have to wait until the progress completes. The exporter, on the other hand, has the ability to pause and resume independent of the destination. This facilitates the sender to be smart in the way to orchestrate simultaneous transfers to multiple destinations without requiring more replicas.
As with any application, importers and exporters can be a dedicated single thread of activity that can be tested, serviced and monitored with the best practice from testing, dev-ops and call-home functionalities. These functionalities can be independently added via their own stacks or applications that sit well with those deployed by the tool. There is very little need for the application to take on the onus from these perspectives since specialized products continue to serve similar functionalities across applications. For example, reporting stacks can work off the logs and read only queries from the importer and exporter.
API functionality is a separate concern from the above and belongs exclusively to the tool. The tool may take in parameters over the API requiring little or no redeployment for the end user. This kind of functionality alleviates the setup and teardown associated with adhoc and changing requirements.
The importer and exporter have the ability to append or read sections of the stream.
The importer and exporter also make push and pull model easy by acting as a relay in the middle. The role of the importer and exporter then becomes an adapter between heterogeneous systems. The API for example, is a pull model. But most metrics and time series database are a push model relying on the agents like telegraf to transfer data.
The importer and the exporter enables a stream store to participate in a data pipeline. This is a critical business value for the stream store because it adds value as co-inhabitant of a data ecosystem as opposed to competing with time-series database
No comments:
Post a Comment