Data Storage and connectors:
The nature of data storage is that they accumulate data over time for analytics. S3 Apis are a popular example of programmatic access for web access to store data in the cloud. A connector works similarly but they don’t necessarily require the data storage to be remote or code to be written for the data transfer. In fact, one of the most common asks from a storage product is that it facilitates data transfer using standard shell commands.
The purpose of the connector is to move data quickly and steadily between source and destination regardless of their type or the kind of data to be transferred. Therefore, the connector needs to be specific to the destination storage. The connector merely automates the steps to organize and fill the containers in the data storage by sending data in fragments, if necessary. The difference made by a connector is enormous in terms of convenience to stash data and for reusability of the same automation for different sources. The traditional stack of command line storage over programmable interfaces to allow automation beyond the programmatic access is not lost. However requiring customers to write their wrappers for their own command line utility to send data is somewhat tedious and avoidable.
In addition to the programmatic access for receiving data, data stores need to customize the input of data from different contexts such as protocols, bridging technologies such as message queues, and even the eccentricities of the sender. It is in these contexts that a no-code ready-made tool is preferred. Data transfers may also need to be in a chain requiring data to be relayed between systems like piping operations in shell environment. A new connector may wrap another existing connector as well.
One of the most common examples of a connector is a TCP based data connector. The data is simply sent by opening a networking socket to make a connection. This is executed with standard command line tools as follows:
cat logfile | nc ipaddress port
The inclusion of such a data connector for a storage product is probably the most convenient form of data transfer. Even if the storage product requires programmatic access, wrapping the access APIs to facilitate a TCP connector like above will immensely benefit those who do not have to write code to send data to storage.
With the automation for a TCP connector written once, there will not be a need to repeat the effort elsewhere or to reinvent the wheel.
The nature of data storage is that they accumulate data over time for analytics. S3 Apis are a popular example of programmatic access for web access to store data in the cloud. A connector works similarly but they don’t necessarily require the data storage to be remote or code to be written for the data transfer. In fact, one of the most common asks from a storage product is that it facilitates data transfer using standard shell commands.
The purpose of the connector is to move data quickly and steadily between source and destination regardless of their type or the kind of data to be transferred. Therefore, the connector needs to be specific to the destination storage. The connector merely automates the steps to organize and fill the containers in the data storage by sending data in fragments, if necessary. The difference made by a connector is enormous in terms of convenience to stash data and for reusability of the same automation for different sources. The traditional stack of command line storage over programmable interfaces to allow automation beyond the programmatic access is not lost. However requiring customers to write their wrappers for their own command line utility to send data is somewhat tedious and avoidable.
In addition to the programmatic access for receiving data, data stores need to customize the input of data from different contexts such as protocols, bridging technologies such as message queues, and even the eccentricities of the sender. It is in these contexts that a no-code ready-made tool is preferred. Data transfers may also need to be in a chain requiring data to be relayed between systems like piping operations in shell environment. A new connector may wrap another existing connector as well.
One of the most common examples of a connector is a TCP based data connector. The data is simply sent by opening a networking socket to make a connection. This is executed with standard command line tools as follows:
cat logfile | nc ipaddress port
The inclusion of such a data connector for a storage product is probably the most convenient form of data transfer. Even if the storage product requires programmatic access, wrapping the access APIs to facilitate a TCP connector like above will immensely benefit those who do not have to write code to send data to storage.
With the automation for a TCP connector written once, there will not be a need to repeat the effort elsewhere or to reinvent the wheel.
No comments:
Post a Comment