Storage products are a veritable data destination for workflows originating outside the product. Their operations generate logs which can also be stored in the same product providing portability, availability and participation in the data plane which comes with guarantees that are typical for data. This immensely improves introspection of the product as various logs, metrics and events can be saved directly in the store itself.  Queries can be written against this introspection data. In particular, Flink can be used with any tier 2 such as Aliyun Object Storage Service or Stream storage. These stores can be treated as append-only data storage with skip-level access introduced for efficient querying.

With the use case of storing the cluster logs within the storage product, let us say we write an FlinkApplication to continuously monitor the number of times there are  delays in accessing the tier 2 storage and the distribution of these delays, analyze the data storage and data access patterns with segments, leverage cache where it helps to avoid accessing data segments directly from the store and finally add additional syntax to segments to facilitate improved performance to queries. 

We propose that segments creation strategies do not affect their augmentation with skip level access. Only a handful of skip-levels with orders of magnitude of 2 is required to speed up access of segments. Back references are not required with as many look ahead segments ensured to be available as the highest skip level for each new segment. Groups of segments may be treated as units for higher level strategies which introduces a secondary layer of segment nomenclature. 

 Stream readers read the stream from beginning to end. The number of readers is allowed to scale arbitrarily because the cost for computation is cheap and it can be parallelized. However, data access and storage performance suffer. When the higher-level semantics of segments are maintained, the techniques for accessing and retrieving segments become irrelevant.  

With the number of readers reduced, the performance of data follows the machine efficiency curve

This is a continuation of the earlier posts to enumerate funny software engineering practice:

160) Build a product where there are no applications without involving external software and dependencies and take a hit when any of them in introduce breaking changes
161) Build a product with many types of overrides and the system administrator is left scratching head on how a policy was violated.
162) Build a product where users find ways to share by bypassing the inbuilt mechanisms such as role based access control.
163) Build a product where loopholes are concealed as methods to bypass policy.
164) Build a product that fails to meet compliance rules and have the administrators deploy third party software to fill the shortcomings
165) Build a product that generates so many notifications to clutter up the users’s screens that they can’t get past easily to do their work.
166) Build a product where the ‘select all’ functionality is missing from resource lists.
167) Build a product where the product is overwhelmingly noisy and have administrators grapple with custom rules to modify the execution
168) Build a product without bells and whistles and find that the product goes unnoticed or worse forgotten.
169) Build a product that can generate beautiful reports only to find that product is used more for reports rather than for data accumulation.
170) Build a product that does not allow users to explore broad or deep without administrator privilege and find users dissatisfied.
171) Build a product that does not allow authoring policies and find that the usage of the product is almost a chaos.
172) Build a product that does not allow differentiation to users either with policies authored by the administrator or out of the box and find that 10% of the users may matter more than the remaining 90%.
173) Build a product that does not let frequent users form their own customizations or dashboards and find that the product is pushed behind those that can.
174) Build a product that does not allow hands off operations with alerts and notifications and find that the product is not up for renewal.
175) Build a product that does not show captivating charts and graphs and the users migrate to applications that do.
176) Build a product that does not allow users to track their activities especially on shared resources and have the administrator be flooded with calls for lost or missing resources.
177) Build a product that does not allow operations to be distributed and have the planners complain about capabilities.
178) Build a product that does not protect data at rest or in transit and have all the watchdogs show up at the door.
179) Build a product that fails to meet the industry compliance and government regulations and find that the product cannot be sold in certain regions.
180) Build a product that tries to be ahead of the game in emerging trends and find that a significant legacy from partners still requires older technology integration.

This is a continuation of the earlier posts to enumerate funny software engineering practice:

150) Build a product that requires non commodity parts and hardware and find that customers are trying to pull apart the product to suit their needs.
151) Build a product that throws arcane errors when the users go off the beaten path
152) Build a product that does not guide the user back to the rails when they go off the beaten path
153) Build a product which has all the necessary ingredients for a successful production deployment but find that the secrets used to operate the software are irreplaceable
154) Build a product which has little or no value in terms of data visualization and find the audience looking for alternatives
155) Build a product in parts where one hand makes one part and another makes a different part and both are not coordinated
156) Build a product where the bug count doesn’t seem to come down and find interesting ideas to hide it
157) Build a product with little or no release gates and see the instability reach the customers.
158) Build a product where the fixes go all the way to the customer on the same day but have it repeated until customers make their mind
159) Build a product with breaking changes in published interfaces and the customers complain.
160) Build a product where there are no applications without involving external software and dependencies and take a hit when any of them in introduce breaking changes
161) Build a product with many types of overrides and the system administrator is left scratching head on how a policy was violated.
162) Build a product where users find ways to share by bypassing the inbuilt mechanisms such as role based access control.
163) Build a product where loopholes are concealed as methods to bypass policy.
164) Build a product that fails to meet compliance rules and have the administrators deploy third party software to fill the shortcomings
165) Build a product that generates so many notifications to clutter up the users’s screens that they can’t get past easily to do their work.
166) Build a product where the ‘select all’ functionality is missing from resource lists.
167) Build a product where the product is overwhelmingly noisy and have administrators grapple with custom rules to modify the execution
168) Build a product without bells and whistles and find that the product goes unnoticed or worse forgotten.
169) Build a product that can generate beautiful reports only to find that product is used more for reports rather than for data accumulation.
170) Build a product that does not allow users to explore broad or deep without administrator privilege and find users dissatisfied.

This is a continuation of the earlier posts to enumerate funny software engineering practice:

140) Build a product where the number bug fixes in frequent patches make the customers roll their eyes.
141) Build a product where the product loses data to corruption or loss from disasters bringing on some of the most petulant customers.
142) Build a product where the product does not fully recover from a disaster.
143) Build a product where the data becomes unavailable to users as the system is brought down or performs some task that prevents it from functioning normally.
144) Build a product and find that deployments are increasing data traffic more than they should.
145) Build a product with specific guidance for deployment and see it put to use in manner different from those suggested
146) Build a product with little or no guidance for the choice of deployment model and find the customer’s gravitate towards the easiest one and then try to scale.
147) Build a product with verbose manual and jargon, and find that customers don’t really want to read that.
148) Build a product like a turnkey solution and find that customers who wanted that in the first place now want to customize it.
149) Build a product with automation packaged via scripts and see its adoption grow astronomically.
150) Build a product that requires non commodity parts and hardware and find that customers are trying to pull apart the product to suit their needs.
151) Build a product that throws arcane errors when the users go off the beaten path
152) Build a product that does not guide the user back to the rails when they go off the beaten path
153) Build a product which has all the necessary ingredients for a successful production deployment but find that the secrets used to operate the software are irreplaceable
154) Build a product which has little or no value in terms of data visualization and find the audience looking for alternatives
155) Build a product in parts where one hand makes one part and another makes a different part and both are not coordinated
156) Build a product where the bug count doesn’t seem to come down and find interesting ideas to hide it
157) Build a product with little or no release gates and see the instability reach the customers.
158) Build a product where the fixes go all the way to the customer on the same day but have it repeated until customers make their mind
159) Build a product with breaking changes in published interfaces and the customers complain.
160) Build a product where there are no applications without involving external software and dependencies and take a hit when any of them in introduce breaking changes

This is a continuation of the earlier posts to enumerate funny software engineering practice:



120) Build a product that takes a lot of effort involved in explaining workarounds when a feature could be added.
121) Build a product where the cost of servicing overwhelms the amortized cost of purchase.
122) Build a product where the customers are enticed to subscription model or cloud tenancy only to discover the total cost to be significantly high for developmental activities
123) Build a product where the customers have to deal with two problems simultaneously: on-premise as well as cloud encumbrances
124) Build a product that requires subscription to services and use the legal language in the contract to extort payments
125) Build a product that requires processing power that is only available on desktops.
126) Build a product that is not elastic I’m storage, compute and networking only to be surprised by crowd peak traffic
127) Build a product that has little or no configuration option for sources of input prompting dedicated instances for different use cases
128) Build a product that reveals hidden bugs after warranty period
129) Build a product where the maker funds white hat discovery of vulnerabilities which prompt the customer to upgrade
130) Build a product with changing goals for v1 while the market tide would have lifted any v1 for immediate business need and with a compelling v2 to establish business.
131) Build a product with that has terms of license written in a way that brings more revenue from the same product over time or use.
132) Build a product with so much emphasis on performance that it becomes a nice product in the magic quadrant
133) Build a product with limited or no expansion possibilities for emerging trends and find the technical debt to be rather costly in the next release
134) Build a product with exacting demands for deployment and find the users running away to competitors.
135) Build a product with little or no configuration for toggling features and find that the majority of the functionality is locked down because there is no granularity
136) Build a product with no dedicated management console/portal and find a section of the audience spurned away.
137) Build a product that requires work for integration and find that there are bugs in standalone mode that would have been detected sooner.
138) Build a product which shows errors in a non-deterministic manner and find costs soaring for investigations
139) Build a product where a single bug fix post release as a patch is either too costly or hard to enforce.
140) Build a product where the number bug fixes in frequent patches make the customers roll their eyes.

This is a continuation of the earlier posts to enumerate funny software engineering practice:


110) Build a product that starts to sound discrepant in the ads as releases slip.
111) Build a product that does not uninstall clean
112) Build a product that leaves artifacts even when the customer does not want it
113) Build a product that makes it annoying for the customers to switch out components
114) Build a product that improves security with a trade-off on convenience.
115) Build a product that provides inadequate segregation of user activities or secrets letting others to see the actions taken.
116) Build a product that provides that behaves well in development but fail in production because of deficiencies in system resources.
117) Build a product that is considered golden only to find that the customer has been using an incorrect version
118) Build a product that takes a sweeping approach of solving problems by updating versions while not sufficiently resolving special cases.
119) Build a product that takes does not have sufficient levers to resolve small customer cases.
120) Build a product that takes a lot of effort involved in explaining workarounds when a feature could be added.
121) Build a product where the cost of servicing overwhelms the amortized cost of purchase.
122) Build a product where the customers are enticed to subscription model or cloud tenancy only to discover the total cost to be significantly high for developmental activities
123) Build a product where the customers have to deal with two problems simultaneously: on-premise as well as cloud encumbrances
124) Build a product that requires subscription to services and use the legal language in the contract to extort payments
125) Build a product that requires processing power that is only available on desktops.
126) Build a product that is not elastic I’m storage, compute and networking only to be surprised by crowd peak traffic
127) Build a product that has little or no configuration option for sources of input prompting dedicated instances for different use cases
128) Build a product that reveals hidden bugs after warranty period
129) Build a product where the maker funds white hat discovery of vulnerabilities which prompt the customer to upgrade
130) Build a product with changing goals for v1 while the market tide would have lifted any v1 for immediate business need and with a compelling v2 to establish business.

The benefits of Async stream readers in cache for stream storage:
The cache essentially projects portions of the stream so that the entire stream does not have to scanned from begin to end repeatedly. This mode of operation is different from finding popular stream segments across stream readers.
The cache may use a queue to hold contiguous segments from a stream.
The choice of queue is extremely important for peak throughput. The use of a lock free data structure as opposed to an ArrayBlockingQueue can do away with lock contention. The queues work satisfactorily until they are full. Garbage free Async stream readers have the best response time.
The benefits here are similar to LogAppenders except that they are on write path whole these are in read path.
Asynchronous read together with lock free access and skip level access boost performance. The cache may encounter significant size for each segment and a writer that transfers a segment over the network may take time in the order of hundreds of milliseconds. Instead, having a continuous background import for adjustable window of steam segments tremendously improves the load on the stream store while responding to seek requests on segments

Please refer to the discussion on caching context for stream managers with:

https://1drv.ms/w/s!Ashlm-Nw-wnWvBXkU9jz_Z2EXWLp

and  implementation where it is applicable with: https://github.com/ravibeta/JavaSamples/tree/master/logging