Monday, August 20, 2018

We were discussing gateway and object storage. We  wanted to create content distribution network from the object storage itself using a gateway like functionality to objects as geo-redundant copies. The storage engine layer responsible for the creation of objects would automatically take care of the replication of the objects. Storage engines generally have a notion each for a virtual data center and a replication group. An object created within a virtual data center is owned by that virtual data center. If there are more than one virtual data center within a replication group, the owning virtual data center within a group will be responsible to replicate the object in the other virtual data center and this is usually done after the object has been written. At the end of the replication, each virtual data center has a readable copy of the object. Erasure codes help protect the object without additional software or services because the data and the code of the fragments of the object are so formed that the entire object can be reconstructed. Internal to each virtual center, there may be a pool of cluster nodes and the object may have been written across three chosen nodes. Since each virtual data center needs to know the location of the object, the location information itself may be persisted the same way as an object. The location information is internal to the storage engine unlike the internet accessible address. The address is just another attribute for the object. Since the address has no geography specific information as per our design of the gateway, the rules for the gateway can be used to route a read request to the relevant virtual data center which will use the address to identify the object and use its location to read the object. Copying and caching by a non-owner virtual data center is entirely its discretion because those are enhancements to the existence of the object in each virtual data center within the same replication group. Traditionally replication groups were used for outages but the same may be leveraged for content distribution network and the gateway may decide to route the request to one of the virtual data centers. 
Together the gateway and the storage engine provide address and copies of objects to facilitate access via a geographically close location. However, we are suggesting native gateway functionality to object storage in a way that promotes this Content Distribution Network.  Since we have copies of the object, we don’t need to give an object multiple addresses for access from different geographical regions.

No comments:

Post a Comment