We were discussing the use of Object Storage to stash state for each worker from applications services and clusters and its implementation in the form of distributed services over object storage.
A lease is supposed to help track resource usage. If the callers have to send the requests to the same single point of contention, it is not very useful.  When requests are served from separate leaseing service, the performance generally improves when there ar e load balancers. That is why different proxy servers behind a lease could maintain their own leaseing service.  Shared services may offer at par service level agreement as an individual service for requests. Since a lease will not see performance degradation when sending to a proxy server or a shared dedicated lease, it works in both these cases. Replacing a shared dedicated leaseing service with individual dedicated leaseing service is entirely the call of the consumers.  

The policies for a lease need not be mere regex translation of worker parameters to object hierarchy. We are dealing with objects on a worker by worker basis where we can use all parts of the hierarchical namespace – bucket – object names to differentiate the resource usages for workers. For that matter names may be translated so that the caller may only need a lease id to enumerate all storage resources within the object storage. We are not just putting the lease on steroid, we are also making it smart by allowing more interpretations and actions on the resource usages. These rules can be authored in the form of expressions and statements much like a program with lots of if then conditions ordered by their execution sequence. The leases facilitate a lookup of objects without sacrificing performance and without restrictions to the organization of objects within or distributed stores because it enumerates them within the lease.

We were discussing the use of Object Storage to stash state for each worker from applications services and clusters and its implementation in the form of distributed services over object storage. The nodes in a storage pool assigned to the VDC may have a fully qualified name and public IP address. Although these names and ip address are not shared with anyone, they serve to represent the physical location of the fragments of an object. Generally, an object is written across three such nodes. The storage engine gets a request to write an object. It writes the object to one chunk but the chunk may be physically located on three separate nodes. The writes to these three nodes may even happen in parallel.  The object location index of one chunk and the disk locations corresponding to the chunk are also artifacts that need to be written. For this purpose, also, three separate nodes may be chosen and the location information may be written.   The storage engine records the disk locations of the chunk in a chunk location index and the disk locations corresponding to the chunk to three different disks/nodes. The index locations are chosen independently from the object chunk locations.  Therefore, we already have a mechanism to store locations. When these locations have representations for the node and the site, a copy of an object served over the web has a physical internal location. Even when they are geo-replicated, the object and the location information will be updated together.  The tracking of site-specific locations for an object is a matter of merely maintaining a registry of locations just the same way as we look up the chunks for an object. We just need more information on the location part of the object and the replication group automatically takes care of keeping locations and objects updated as they are copied.    

We were discussing the use of Object Storage to stash state for each worker from applications services and clusters. We referred to the use of leases in the form of tickets. The tickets need not be issued from a singleton service  Tickets can be distributed just as the objects tracked by the tickets can belong to different object storage. We discuss these alternate forms of issuing tickets:
The ticketing service can be chained across object storage. If the current object storage does meet the need for a worker pool, it is possible to pass on the ticketing to another similar stack. The ticketing layer merely needs to forward the requests that it cannot answer to a default pre-registered outbound destination. In a distributed Ticketing service, the ticketing service handlers can make sense out of the requests simply with the help of the object-storage-namespace-bucket-object hierarchy and say if a request can be handled it or forwarded. If it does not, it simply forwards it to another ticketing service cum object storage stack. This is somewhat different from the original notion that the ticketing service is bound to an object storage where the resources for the workers are maintained.  The linked ticketing service does not even need to take time to resolve object location to see if it exists. It can merely translate the hierarchical naming to know if the resources belong to it or not. This shallow lookup means a request can be forwarded faster to another linked object storage and ultimately to where it may be guaranteed to be found. The Linked Storage has no criteria for the object store to be similar and as long as the forwarding logic is enabled, any implementation can exist in each of the storage for translation, lookup and return. another way to distribute tickets is with hashes where the destination is determined based on a hash table.  Whether we use routing tables or a static hash table, the networking over the object storage can be its own layer facilitating request resolution at different object storage and ticketing later stack. 

We were discussing the use of object storage to stash state across workers from applications and cluster nodes on a lease basis in the previous post
This section is not exclusive to Ticketing layer and is mentioned here only for convenience. We suggested earlier that tickets may be issued with any existing planning and tracking software in the ticket layer over object storage. There is no dearth of choices for such software in the market. Almost any tracking software that can issue tickets will suffice. This allows for easy integration with existing software so that applications may leverage their favorite API to create, update and resolve tickets.
In addition, each ticket may have rich metadata that can be annotated by parties other than the workers who create, update and resolve tickets. Such metadata may carry any number of fields to improve the tracking information associated with the objects in the object storage. They could also assist with the update of metadata to objects in the object storage. Such metadata brings many perspectives other than that of the initiator to the users
As long as the tickets are open, they can also maintain links with IDs of resources in systems other than the ticket issuing software. This allows for integration of ticket layer with several other production software that helps navigation from a ticket representing the lease to information in other systems that may have also have additional state for the workers that opened the tickets. This gives a holistic view of all things relevant to the workers merely from their read-write data path
The schema for arranging integrations this way with existing tickets is similar to a snowflake pattern. Each link represents a dimension and therefore the ticket is a representation of all information and not just itself. This pattern also facilitates the independence of the ticket from another system that can go down. The links may only be used if they are available to be reached. Since the ticket itself holds enough information locally about the resource tracking, any external links are nice to have and not mandatory.

This kind of extensibility allows ticket layer to grow without disrupting the path above or below. Since the workers remain unaffected, there is no limit to the extensibility in this layer.

We were discussing the use of object storage to stash state across workers from applications and cluster nodes on a lease basis in the previous post.  The object storage acts as an aggregator across workers in this design. It brings elasticity from the virtualized storage customized to individual workers. There is no need for capacity planning for any organization or the payment for file-system, block or other forms of storage as the object storage not only represents the unification of such storage but also supports billing with the availability of detailed study on worker usages. The performance trade-off for workers is small and they are required to change their usages of conventional storage with preferred S3 access. They become facilitators for moving compute layer beyond virtual machines and disks to separate compute and storage platforms the likes of which can generate a new wave of commodity hardware suitable for both on-premise and datacenters. 

The ticketing and leasing system proposed in this post need not just be applications and clusters. It can also be other local platform providers that want to help ease the migration of workloads from local and unmanaged storage.  

The use of a leasing service facilitates object lifecycle management as much as it can be done individually by workers.  It can be offloaded to workers but the notion is that the maintenance activities move from organization owned to self-managed by this solution. Tickets may be implemented with any software as long as they map workers to resources in the object storage.  
Moreover, not all the requests need to reach the object storage. In some cases, web ticket may use temporary storage from hybrid choices. The benefits of using a web ticket including saving bandwidth, reducing server load, and improving request-response time. If a dedicated content store is required, typically the ticketing and server are encapsulated into a content server. This is quite the opposite paradigm of using object storage and replicated objects to directly serve the content from the store. The distinction here is that there are two layers of functions - The first layer is the Ticket layer that solves the life cycle management of storage resources. The second layer is the storage concerns of the actual storage which we mitigate with the help of object storage.  We will call this the storage engine and will get to it shortly.  

The Ticket would be only the ID that the worker needs to track all of its storage resources.    

We were discussing the use of object storage to stash state across workers from applications and cluster nodes on a lease basis in the previous post.  The object storage acts as an aggregator across workers in this design. It brings elasticity from the virtualized storage customized to individual workers. There is no need for capacity planning for any organization or the payment for file-system, block or other forms of storage as the object storage not only represents the unification of such storage but also supports billing with the availability of detailed study on worker usages. The performance trade-off for workers is small and they are required to change their usages of conventional storage with preferred S3 access. They become facilitators for moving compute layer beyond virtual machines and disks to separate compute and storage platforms the likes of which can generate a new wave of commodity hardware suitable for both on-premise and datacenters. 

Ticket Service works closely with issue management software and traditionally both have been long standing products in the marketplace. Atlassian Jira is a planning and tracking software that generates tickets for managing and tracking work-items. This ticket layer is well-positioned for using such existing software directly. This document does not repeat the internals of tracking management software and instead focuses on its use within object storage so that newer workloads can use a storage platform rather than their disks and local storage. 

The ticketing and leasing system proposed in this document need not just be applications and clusters. It can also be other local platform providers that want to help ease the migration of workloads from local and unmanaged storage.  

Object Storage is perceived as backup and tertiary storage. However, in the next few posts we argue that object storage is a persistent thread local storage for all workers in any system so that there is never any loss of state when a worker disappears. While files have traditionally been the storage of choice for workers, we argue that this is just a matter of access and that file protocols or http access serve just the same to file-system enabled object storage. Moreover, not all data needs to be written deep into the object storage at once. With the help of a cache layer discussed earlier, we can even allow workers higher performance than working with remote storage. The requirements for object storage need not even change while the reads and writes from the workers can be handled 

Many applications maintain concurrent activities. Large scale query processing on big data requires intermediate storage for their computations. Similarly cluster computing also requires storage for the processing of their nodes. Traditionally, these nodes have been sharing volumes or maintaining local file storage. However, most disk operations are already written off as expensive as computing moves more into memory. A large number of workers are generally commodity workers. They don’t need to maintain high performance that object storage cannot step in and fill. Moreover each worker or node gets it own set of objects in the object storage which can be considered as shared-nothing as its own disks.  All they need is their storage to be in the cloud and managed so that they never have to be limited by their disks. 

That is how this Ticket Layer positions itself. It offers a tracking and leasing of universal storage that do away with disks for nodes in a cluster and workers in an application. Several forms of data access can be supported in addition to file-system protocols and S3 access in an object storage. The ticketing system is a planning and tracking software management system that generates tickets and leases for remote storage on a workload by workload basis so that the clouds’ elasticity can be brought to individual workers even within the same node or application.