This is a continuation of the earlier posts to enumerate funny aspects of software engineering practice:
240) Build a product where the communication to explain how the product works consumes bulk of the time that could have otherwise been spent on test and development.
241) Build a product where the manual for the product is bulky and can become a headrest
242) Build a product where the user interface real estate is all cluttered up and there is no room to display anything new.
243) Build a product where users register through a third-party identity provider and cannot login to the product if there is an outage
244) Build a product where the home page is a mashup of different websites and the user has to squint at a small portion of the screen
245) Build a product where the forms are so big and large they are never completed in one sitting
246) Build a product that seems to have one form after another to fill on every page.
247) Build a product that shows ads or campaigns that distract the user from their workflows
248) Build a product that makes recommendations even when you want it to stop.
249) Build a product that chronicles every order and makes it creepy to anticipate your next order
250) Build a product that looks simple but when you click the advanced options, it becomes overwhelming.
251) Build a product that makes you think how to query the results
252) Build a product that gives no option to query other than scroll through the results.
253) Build a product that makes you go and forth between pages of search results
254) Build a product that does not let you order the results
255) Build a product that requires you to know the jargon or the domain with little or no warning for consequences.
256) Build a product that has clunky form to create a search criteria
257) Build a product where the forms don’t let you see the entire entry you type
258) Build a product where the forms don’t let you correct your mistakes
259) Build a product that does not let you rename your resources and you have to create more
260) Build a product that does not send out reminders. 

This is a continuation of the earlier posts to enumerate funny aspects of software engineering practice :
230) Build a product where the defects flow to the core of the product from any surface area because it is easier to consult the expert than to do what it takes to resolve the defect upfront
231) Build a product where the core is shielded by layers upon layers and defects in the core lead to workarounds and complexity in outer layers
232) Build a product where the clients can’t keep up with the breaking changes in the server both before and after release
233) Build a product where there are proprietary protocols between components and find that the protocol grows in complexity as opposed to following open, published and popular protocols
234) Build a product where the components fail silently or worse lie when they fail.
235) Build a product where the callers have no way to figure out whether the callee did the task
236) Build a product where the caller does not allow differentiation of callees and find that there is no way to put an expedient fix specific to certain callee.
237) Build a product where there is no protection of boundary such as between layers or external endpoints and find all kinds of security vulnerabilities afterwards
238) Build a product that is well thought through and find the solution to be too elegant and simple to describe.
239) Build a product that provides no mechanism for callers to inject tracers and find that the callers have to rely on communications with the experts
240) Build a product where the communication to explain how the product works consumes bulk of the time that could have otherwise been spent on test and development.
241) Build a product where the manual for the product is bulky and can become a headrest
242) Build a product where the user interface real estate is all cluttered up and there is no room to display anything new.
243) Build a product where users register through a third-party identity provider and cannot login to the product if there is an outage
244) Build a product where the home page is a mashup of different websites and the user has to squint at a small portion of the screen
245) Build a product where the forms are so big and large they are never completed in one sitting
246) Build a product that seems to have one form after another to fill on every page.
247) Build a product that shows ads or campaigns that distract the user from their workflows
248) Build a product that makes recommendations even when you want it to stop.
249) Build a product that chronicles every order and makes it creepy to anticipate your next order
250) Build a product that looks simple but when you click the advanced options, it becomes overwhelming.

This is a continuation of the earlier posts to enumerate funny aspects of software engineering practice :

220) Build a product that specializes monitoring and alerting features and find that customers are not utilizing because they cannot plug it into their existing infrastructure.
221) Build a product where the features are built independently but find that the customer has turned on all of them
222) Build a product as a culmination of development efforts but find a mismatch between customer expectations and delivery
223) Build a product with internal vetting and sometimes friction and find the customer advocacy voice waning.
224) Build a product with the notion of getting a feature into the hands of customer without being explained fully why the customers want it
225) Build a product as a part of a portfolio and find that the entire value proposition of the product subdued by the portfolio perception.
226) Build a product as a part of portfolio that prevents it from changing its price, licensing and distribution and all the innovation in the product averages to that of the portfolio
227) Build a product that cannot be sold separately and find that the product gains attention with the plus and minus of the other.
228) Build a product where the distribution is in flavors that does not work as well as the default and have all the negative feedback be attributed to the product as a whole
229) Build a product that has client-server technology and find that all the client issues are attributed to the server.
230) Build a product where the defects flow to the core of the product from any surface area because it is easier to consult the expert than to do what it takes to resolve the defect upfront
231) Build a product where the core is shielded by layers upon layers and defects in the core lead to workarounds and complexity in outer layers
232) Build a product where the clients can’t keep up with the breaking changes in the server both before and after release
233) Build a product where there are proprietary protocols between components and find that the protocol grows in complexity as opposed to following open, published and popular protocols
234) Build a product where the components fail silently or worse lie when they fail.
235) Build a product where the callers have no way to figure out whether the callee did the task
236) Build a product where the caller does not allow differentiation of callees and find that there is no way to put an expedient fix specific to certain callee.
237) Build a product where there is no protection of boundary such as between layers or external endpoints and find all kinds of security vulnerabilities afterwards
238) Build a product that is well thought through and find the solution to be too elegant and simple to describe.
239) Build a product that provides no mechanism for callers to inject tracers and find that the callers have to rely on communications with the experts
240) Build a product where the communication to explain how the product works consumes bulk of the time that could have otherwise been spent on test and development.

This is a continuation of the earlier posts to enumerate funny aspects of software engineering practice :

205) Build a product with rich mobile enhanced experience across a variety of devices and find that the customers are able to use the product even before going to sleep
206) Build a product with mobile integration of payment methods such as personal wallet and the chore of paying becomes smooth.
207) Build a product that can use digital cards and refill them at will and the product becomes usable at all participating stores increasing adoption via partner networks
208) Build a product with partner sponsored incentives to customers and see the appeal grow as partner network endears itself to the customer more than it would have individually
209) Build a product with little or no ecosystem and the product fails to strike popularity in conferences, exhibitions and symposiums
210) Build a product with little user education and find that the sales booths at an exhibition are not visited.
211) Build a product with very little wording on consequences and see the amazement when users click a button.
212) Build a product that does not warn on consequences and find that the user has just deleted his data.
213) Build a product that does not take any notes the user wants to leave and find that the user does not remember why he took some actions
214) Build a product that does lets users erase their activities and find no record to come to their aid
215) Build a product that works with some users and not with others and find that they have to scratch their heads to find out why.
216) Build a product that has intermittent outages which it hides from the user
217) Build a product that has little or no documentation on escalation policy and find that genuine problems can be described well but not brought up for mitigation.
218) Build a product that tries to tackle the hard 10% scientific problems at the cost of 90% commercial problems.
219) Build a product that mitigates troubleshooting via published documents and find that the support team is building supportability features rather than support case handling
220) Build a product that specializes monitoring and alerting features and find that customers are not utilizing because they cannot plug it into their existing infrastructure.
221) Build a product where the features are built independently but find that the customer has turned on all of them
222) Build a product as a culmination of development efforts but find a mismatch between customer expectations and delivery
223) Build a product with internal vetting and sometimes friction and find the customer advocacy voice waning.
224) Build a product with the notion of getting a feature into the hands of customer without being explained fully why the customers want it
225) Build a product as a part of a portfolio and find that the entire value proposition of the product subdued by the portfolio perception.

The benefits of Async stream readers in cache for stream storage:
The cache essentially projects portions of the stream so that the entire stream does not have to scanned from begin to end repeatedly. This mode of operation is different from finding popular stream segments across stream readers.
The cache may use a queue to hold contiguous segments from a stream.
The choice of queue is extremely important for peak throughput. The use of a lock free data structure as opposed to an ArrayBlockingQueue can do away with lock contention. The queues work satisfactorily until they are full. Garbage free Async stream readers have the best response time.
The benefits here are similar to LogAppenders except that they are on write path whole these are in read path.
Asynchronous read together with lock free access and skip level access boost performance. The cache may encounter significant size for each segment and a writer that transfers a segment over the network may take time in the order of hundreds of milliseconds. Instead, having a continuous background import for adjustable window of steam segments tremendously improves the load on the stream store while responding to seek requests on segments
Another approach is to have a continuous reader read the stream from beginning to end in a loop and have the cache decide which of the segments it needs to read. This approach requires the cache to discard segments that are not necessary for the clients.  The cache can fork as many readers as necessary to replenish the window of segments it needs. The cache can decide which windows if segments it needs based on overlapping intervals of segments requested by clients. These segments are no longer bound to clients and they are managed exclusively by the cache.
The skip level access continues to serve the readers in this case although it doesn’t need to read the streams directly. The readers present the segments and the cache then uses its statistics and algorithm to prefetch the segments that will serve the clients best. These cache entries can be evicted by any algorithm that suits the cache. Storage is not a concern for the cache because we have distributed ring cache with consistency check points
The maximum number of segments in a window is directly dependent on the cache. The segment size may even be a constant in which case the number of segments depends on how many streams and how many windows per stream the cache wants to accommodate. This window size is not the same as a continuously sliding window used to analyze the stream which is usually one per stream.
When there is only one reader to replenish the segments for the cache, the clients are freed from performing the same task over and over again on the actual stream. The segments that the client need will most likely be found in the cache. If the segments are not found, the cache can fetch it from the stream on demand. Otherwise the scheduled reading from the stream by the cache will replenish it. The cache itself alleviates the load from the stream store.
The cache can also fork multiple readers at fixed duration offsets from the start of the first. Multiple readers at fixed periodic intervals from the start of the first will continue to read the steam segments and reduce the latency in replenishing the segments that the cache needs. When the cache advertises it needs segment number n, the readers that have not yet read n will read and make it available. With multiple readers, the chance that one of them is closest to segment n is higher.
There are ways in which the cache can become smart about loading segments using these readers. For example, instead of waiting for the nearest reader to the segment to read it into the cache, the cache can have a dedicated reader fetch the segment or a set of segments via skip-level access.
There are also techniques to make read and writes to go together. Typically reads and writes are not in the same stream and the cache behaves as a write-through, hence the clients can interact directly with the cache rather than the store. When a segment is not found in the cache, it is also efficiently handled with skip level access.

When readers read a stream from beginning to end, they become easy to scale.  This is simple strategy and works well for parallelizing computations. However, it introduces latency as streams can be quite long.

A cache served to provide the stream segments do that clients don’t have to go all the way to the store. It could employ a group of readers that could replenish the segments that are in most demand. There is always the option for clients to reach the store if the cache does not have what it needs. Generally s cache will bring in the segment on behalf of the client if it goes not have it.

The techniques for providing stream segments do not matter to the client and the cache can use any algorithm. The cache also provides the benefits of alleviating load from the stream store without any additional constraints. In fact the cache will also use the same stream reader as the client and with the only difference that there will be fewer stream readers on the stream store than before.
We have not compared this cache layer with a message queue server but there are interesting problems common to both. For example, we have a multiple consumer single producer pattern in the periodic reads from the stream storage. The message queue server or broker enables this kind of publisher-subscriber pattern with retries and dead letter queue. In addition, it journals the messages for review later. Messaging protocols are taken up a notch in performance with the use of a message queue broker and their reliance on sockets with steroids.  This leaves the interaction between the caches and the storage to be handled elegantly with well-known messaging framework. The message broker inherently comes with a scheduler to perform repeated tasks across publishers. Hence it is easy for the message queue server to perform as an orchestrator between the cache and the storage, leaving the cache to focus exclusively on the cache strategy suitable to the workloads.  Journaling of messages also helps with diagnosis and replay and probably there is no better store for these messages than the object storage itself. Since the broker operates in a cluster mode, it can scale to as many caches as available. Moreover, the journaling is not necessarily available with all the messaging protocols which counts as one of the advantages of using a message broker.  Aside from the queues dedicated to handle the backup of objects from cache to storage, the message broker is also uniquely positioned to provide differentiated treatment to the queues.  This introduction of quality of service levels expands the ability of the solution to meet varying and extreme workloads The message queue server is not only a nice to have feature but also a necessity and a convenience when we have a distributed cache to work with the stream storage.
The number of workers for the cache or the store does not matter and they can scale.
Please refer to the discussion on caching context for stream managers
https://1drv.ms/w/s!Ashlm-Nw-wnWvBXkU9jz_Z2EXWLp

and sample implementation with: https://github.com/ravibeta/JavaSamples/tree/master/logging

This is a continuation of the earlier posts to enumerate funny aspects of software engineering practice :

195) Build a product that does not keep history of all actions, assets and activities only to find significant effort in rebuilding it.

196) Build a product that does not maintain registry for all users and find that there is no blacklist or whitelist capability

197) Build a product that assumes secure communication via tunnels and find that the tunnels need not be continuous through a proxy.

198) Build a product that does not gain statistics of its usage and find that the pain grows among users

199) Build a product that has no telemetry and not have any insight into how customers are doing with the product.

200) Build a product that has built in dial home capabilities and find that privacy gurus will find it suspect.

201) Build a product that has little or no test tools shipped with its product and find that customers are knocking

202) Build a product that leverages APIs for all its workflows and find that the product becomes part of more and more workflows.

203) Build a product with enough bread crumbs and sitemap to let the user navigate the pages and see that the usability goes up and so does the customer appeal.

204) Build a product that ships on mobile devices and find the audience skyrocket

205) Build a product with rich mobile enhanced experience across a variety of devices and find that the customers are able to use the product even before going to sleep

206) Build a product with mobile integration of payment methods such as personal wallet and the chore of paying becomes smooth.

207) Build a product that can use digital cards and refill them at will and the product becomes usable at all participating stores increasing adoption via partner networks

208) Build a product with partner sponsored incentives to customers and see the appeal grow as partner network endears itself to the customer more than it would have individually

209) Build a product with little or no ecosystem and the product fails to strike popularity in conferences, exhibitions and symposiums

210) Build a product with little user education and find that the sales booths at an exhibition are not visited.
211) Build a product with very little wording on consequences and see the amazement when users click a button.
212) Build a product that does not warn on consequences and find that the user has just deleted his data.
213) Build a product that does not take any notes the user wants to leave and find that the user does not remember why he took some actions
214) Build a product that does lets users erase their activities and find no record to come to their aid
215) Build a product that works with some users and not with others and find that they have to scratch their heads to find out why.

This is a continuation of the earlier posts to enumerate funny aspects of software engineering practice :

190) Build a product with ambitious charter and find the release timelines reducing the plan.
191) Build a product that does not preserve data between restarts, retries or disaster recovery
192) Build a product that does not retain user settings only to have the user apply them again
193) Build a product by putting together functionality via dependencies and find that the massive surface area attracts all kinds of vulnerabilities.
194) Build a product that creates sdks, command-line clients and other artifacts and find that the versioning for each becomes a maintenance issue.
195) Build a product that does not keep history of all actions, assets and activities only to find significant effort in rebuilding it.
196) Build a product that does not maintain registry for all users and find that there is no blacklist or whitelist capability
197) Build a product that assumes secure communication via tunnels and find that the tunnels need not be continuous through a proxy.
198) Build a product that does not gain statistics of its usage and find that the pain grows among users
199) Build a product that has no telemetry and not have any insight into how customers are doing with the product.
200) Build a product that has built in dial home capabilities and find that privacy gurus will find it suspect.
201) Build a product that has little or no test tools shipped with its product and find that customers are knocking
202) Build a product that leverages APIs for all its workflows and find that the product becomes part of more and more workflows.
203) Build a product with enough bread crumbs and sitemap to let the user navigate the pages and see that the usability goes up and so does the customer appeal.
204) Build a product that ships on mobile devices and find the audience skyrocket
205) Build a product with rich mobile enhanced experience across a variety of devices and find that the customers are able to use the product even before going to sleep
206) Build a product with mobile integration of payment methods such as personal wallet and the chore of paying becomes smooth.
207) Build a product that can use digital cards and refill them at will and the product becomes usable at all participating stores increasing adoption via partner networks
208) Build a product with partner sponsored incentives to customers and see the appeal grow as partner network endears itself to the customer more than it would have individually
209) Build a product with little or no ecosystem and the product fails to strike popularity in conferences, exhibitions and symposiums
210) Build a product with little user education and find that the sales booths at an exhibition are not visited.

Discussion on cache for streams continued:
A cache served to provide the stream segments do that clients don’t have to go all the way to the store. It could employ a group of readers that could replenish the segments that are in most demand. There is always the option for clients to reach the store if the cache does not have what it needs. Generally s cache will bring in the segment on behalf of the client if it goes not have it.

The techniques for providing stream segments do not matter to the client and the cache can use any algorithm. The cache also provides the benefits of alleviating load from the stream store without any additional constraints. In fact the cache will also use the same stream reader as the client and with the only difference that there will be fewer stream readers on the stream store than before.
We have not compared this cache layer with a message queue server but there are interesting problems common to both. For example, we have a multiple consumer single producer pattern in the periodic reads from the stream storage. The message queue server or broker enables this kind of publisher-subscriber pattern with retries and dead letter queue. In addition, it journals the messages for review later. Messaging protocols are taken up a notch in performance with the use of a message queue broker and their reliance on sockets with steroids.  This leaves the interaction between the caches and the storage to be handled elegantly with well-known messaging framework. The message broker inherently comes with a scheduler to perform repeated tasks across publishers. Hence it is easy for the message queue server to perform as an orchestrator between the cache and the storage, leaving the cache to focus exclusively on the cache strategy suitable to the workloads.  Journaling of messages also helps with diagnosis and replay and probably there is no better store for these messages than the object storage itself. Since the broker operates in a cluster mode, it can scale to as many caches as available. Moreover, the journaling is not necessarily available with all the messaging protocols which counts as one of the advantages of using a message broker.  Aside from the queues dedicated to handle the backup of objects from cache to storage, the message broker is also uniquely positioned to provide differentiated treatment to the queues.  This introduction of quality of service levels expands the ability of the solution to meet varying and extreme workloads The message queue server is not only a nice to have feature but also a necessity and a convenience when we have a distributed cache to work with the stream storage.
The number of workers for the cache or the store does not matter and they can scale.
Please refer to the discussion on caching context for stream managers
https://1drv.ms/w/s!Ashlm-Nw-wnWvBXkU9jz_Z2EXWLp

Using caches for stream readers:
When readers read a stream from beginning to end, they become easy to scale.  This is simple strategy and works well for parallelizing computations. However, it introduces latency as streams can be quite long.
A cache served to provide the stream segments do that clients font have to go all the way to the store. It could employ a group of readers that could replenish the segments that are in most demand. There is always the option for clients to reach the store if the cache does not have what it needs. Generally s cache will bring in the segment on behalf of the client if it goes not have it.
The techniques for providing stream segments do not matter to the client and the cache can use any algorithm. The cache also provides the benefits of alleviating load from the stream store without any additional constraints. In fact the cache will also use the same stream reader as the client and with the only difference that there will be fewer stream readers on the stream store than before.
The cache essentially projects portions of the stream so that the entire stream does not have to scanned from begin to end repeatedly. This mode of operation is different from finding popular stream segments across stream readers.
The cache may use a queue to hold contiguous segments from a stream.
The choice of queue is extremely important for peak throughput. The use of a lock free data structure as opposed to an ArrayBlockingQueue can do away with lock contention. The queues work satisfactorily until they are full. Garbage free Async stream readers have the best response time.
The benefits here are similar to LogAppenders except that they are on write path whole these are in read path.
Asynchronous read together with lock free access and skip level access boost performance. The cache may encounter significant size for each segment and a writer that transfers a segment over the network may take time in the order of hundreds of milliseconds. Instead, having a continuous background import for adjustable window of steam segments tremendously improves the load on the stream store while responding to seek requests on segments

This is a continuation of the earlier posts to enumerate funny software engineering practice:

180) Build a product that tries to be ahead of the game in emerging trends and find that a significant legacy from partners still requires older technology integration.
181) Build a product that tries to squeeze different channels into the same stream
182) Build a product that tries to distribute a stream to separate channels when the user is expecting one
183) Build a product that tries to promote a partner application from the same maker to the deteriment of competitors
184) Build a product that tries to emulate open source and invite the world to build with it duing its early stages only to find it shutting doors in its growth stage with bars for acceptance
185) Build a product that does not choose partners to foster growth and find that it cannot sustain innovation by itself
186) Build a product that gives users a tonne of information to go through when they want a summary
187) Build a product to implement a use case and leave it for solution integrators to find bugs
188) Build a product to satisfy some customers only to irk others
189) Build a product with over generalization when some customization would have served neatly
190) Build a product with ambitious charter and find the release timelines reducing the plan.
191) Build a product that does not preserve data between restarts, retries or disaster recovery
192) Build a product that does not retain user settings only to have the user apply them again
193) Build a product by putting together functionality via dependencies and find that the massive surface area attracts all kinds of vulnerabilities.
194) Build a product that creates sdks, command-line clients and other artifacts and find that the versioning for each becomes a maintenance issue.
195) Build a product that does not keep history of all actions, assets and activities only to find significant effort in rebuilding it.
196) Build a product that does not maintain registry for all users and find that there is no blacklist or whitelist capability
197) Build a product that assumes secure comunication via tunnels and find that the tunnels need not be continuous through a proxy.
198) Build a product that does not gain statistics of its usage and find that the pain grows among users
199) Build a product that has no telemetry and not have any insight into how customers are doing with the product.
200) Build a product that has built in dial home capabilities and find that privacy gurus will find it suspect.

This is a continuation of the earlier posts to enumerate funny software engineering practice:

170) Build a product that does not allow users to explore broad or deep without administrator privilege and find users dissatisfied.
171) Build a product that does not allow authoring policies and find that the usage of the product is almost a chaos.
172) Build a product that does not allow differentiation to users either with policies authored by the administrator or out of the box and find that 10% of the users may matter more than the remaining 90%.
173) Build a product that does not let frequent users form their own customizations or dashboards and find that the product is pushed behind those that can.
174) Build a product that does not allow hands off operations with alerts and notifications and find that the product is not up for renewal.
175) Build a product that does not show captivating charts and graphs and the users migrate to applications that do.
176) Build a product that does not allow users to track their activities especially on shared resources and have the administrator be flooded with calls for lost or missing resources.
177) Build a product that does not allow operations to be distributed and have the planners complain about capabilities.
178) Build a product that does not protect data at rest or in transit and have all the watchdogs show up at the door.
179) Build a product that fails to meet the industry compliance and government regulations and find that the product cannot be sold in certain regions.
180) Build a product that tries to be ahead of the game in emerging trends and find that a significant legacy from partners still requires older technology integration.
181) Build a product that tries to squeeze different channels into the same stream
182) Build a product that tries to distribute a stream to separate channels when the user is expecting one
183) Build a product that tries to promote a partner application from the same maker to the deteriment of competitors
184) Build a product that tries to emulate open source and invite the world to build with it duing its early stages only to find it shutting doors in its growth stage with bars for acceptance
185) Build a product that does not choose partners to foster growth and find that it cannot sustain innovation by itself
186) Build a product that gives users a tonne of information to go through when they want a summary
187) Build a product to implement a use case and leave it for solution integrators to find bugs
188) Build a product to satisfy some customers only to irk others
189) Build a product with over generalization when some customization would have served neatly
190) Build a product with ambitious charter and find the release timelines reducing the plan.

Storage products are a veritable data destination for workflows originating outside the product. Their operations generate logs which can also be stored in the same product providing portability, availability and participation in the data plane which comes with guarantees that are typical for data. This immensely improves introspection of the product as various logs, metrics and events can be saved directly in the store itself.  Queries can be written against this introspection data. In particular, Flink can be used with any tier 2 such as Aliyun Object Storage Service or Stream storage. These stores can be treated as append-only data storage with skip-level access introduced for efficient querying.

With the use case of storing the cluster logs within the storage product, let us say we write an FlinkApplication to continuously monitor the number of times there are  delays in accessing the tier 2 storage and the distribution of these delays, analyze the data storage and data access patterns with segments, leverage cache where it helps to avoid accessing data segments directly from the store and finally add additional syntax to segments to facilitate improved performance to queries. 

We propose that segments creation strategies do not affect their augmentation with skip level access. Only a handful of skip-levels with orders of magnitude of 2 is required to speed up access of segments. Back references are not required with as many look ahead segments ensured to be available as the highest skip level for each new segment. Groups of segments may be treated as units for higher level strategies which introduces a secondary layer of segment nomenclature. 

 Stream readers read the stream from beginning to end. The number of readers is allowed to scale arbitrarily because the cost for computation is cheap and it can be parallelized. However, data access and storage performance suffer. When the higher-level semantics of segments are maintained, the techniques for accessing and retrieving segments become irrelevant.  

With the number of readers reduced, the performance of data follows the machine efficiency curve

This is a continuation of the earlier posts to enumerate funny software engineering practice:

160) Build a product where there are no applications without involving external software and dependencies and take a hit when any of them in introduce breaking changes
161) Build a product with many types of overrides and the system administrator is left scratching head on how a policy was violated.
162) Build a product where users find ways to share by bypassing the inbuilt mechanisms such as role based access control.
163) Build a product where loopholes are concealed as methods to bypass policy.
164) Build a product that fails to meet compliance rules and have the administrators deploy third party software to fill the shortcomings
165) Build a product that generates so many notifications to clutter up the users’s screens that they can’t get past easily to do their work.
166) Build a product where the ‘select all’ functionality is missing from resource lists.
167) Build a product where the product is overwhelmingly noisy and have administrators grapple with custom rules to modify the execution
168) Build a product without bells and whistles and find that the product goes unnoticed or worse forgotten.
169) Build a product that can generate beautiful reports only to find that product is used more for reports rather than for data accumulation.
170) Build a product that does not allow users to explore broad or deep without administrator privilege and find users dissatisfied.
171) Build a product that does not allow authoring policies and find that the usage of the product is almost a chaos.
172) Build a product that does not allow differentiation to users either with policies authored by the administrator or out of the box and find that 10% of the users may matter more than the remaining 90%.
173) Build a product that does not let frequent users form their own customizations or dashboards and find that the product is pushed behind those that can.
174) Build a product that does not allow hands off operations with alerts and notifications and find that the product is not up for renewal.
175) Build a product that does not show captivating charts and graphs and the users migrate to applications that do.
176) Build a product that does not allow users to track their activities especially on shared resources and have the administrator be flooded with calls for lost or missing resources.
177) Build a product that does not allow operations to be distributed and have the planners complain about capabilities.
178) Build a product that does not protect data at rest or in transit and have all the watchdogs show up at the door.
179) Build a product that fails to meet the industry compliance and government regulations and find that the product cannot be sold in certain regions.
180) Build a product that tries to be ahead of the game in emerging trends and find that a significant legacy from partners still requires older technology integration.

This is a continuation of the earlier posts to enumerate funny software engineering practice:

150) Build a product that requires non commodity parts and hardware and find that customers are trying to pull apart the product to suit their needs.
151) Build a product that throws arcane errors when the users go off the beaten path
152) Build a product that does not guide the user back to the rails when they go off the beaten path
153) Build a product which has all the necessary ingredients for a successful production deployment but find that the secrets used to operate the software are irreplaceable
154) Build a product which has little or no value in terms of data visualization and find the audience looking for alternatives
155) Build a product in parts where one hand makes one part and another makes a different part and both are not coordinated
156) Build a product where the bug count doesn’t seem to come down and find interesting ideas to hide it
157) Build a product with little or no release gates and see the instability reach the customers.
158) Build a product where the fixes go all the way to the customer on the same day but have it repeated until customers make their mind
159) Build a product with breaking changes in published interfaces and the customers complain.
160) Build a product where there are no applications without involving external software and dependencies and take a hit when any of them in introduce breaking changes
161) Build a product with many types of overrides and the system administrator is left scratching head on how a policy was violated.
162) Build a product where users find ways to share by bypassing the inbuilt mechanisms such as role based access control.
163) Build a product where loopholes are concealed as methods to bypass policy.
164) Build a product that fails to meet compliance rules and have the administrators deploy third party software to fill the shortcomings
165) Build a product that generates so many notifications to clutter up the users’s screens that they can’t get past easily to do their work.
166) Build a product where the ‘select all’ functionality is missing from resource lists.
167) Build a product where the product is overwhelmingly noisy and have administrators grapple with custom rules to modify the execution
168) Build a product without bells and whistles and find that the product goes unnoticed or worse forgotten.
169) Build a product that can generate beautiful reports only to find that product is used more for reports rather than for data accumulation.
170) Build a product that does not allow users to explore broad or deep without administrator privilege and find users dissatisfied.

This is a continuation of the earlier posts to enumerate funny software engineering practice:

140) Build a product where the number bug fixes in frequent patches make the customers roll their eyes.
141) Build a product where the product loses data to corruption or loss from disasters bringing on some of the most petulant customers.
142) Build a product where the product does not fully recover from a disaster.
143) Build a product where the data becomes unavailable to users as the system is brought down or performs some task that prevents it from functioning normally.
144) Build a product and find that deployments are increasing data traffic more than they should.
145) Build a product with specific guidance for deployment and see it put to use in manner different from those suggested
146) Build a product with little or no guidance for the choice of deployment model and find the customer’s gravitate towards the easiest one and then try to scale.
147) Build a product with verbose manual and jargon, and find that customers don’t really want to read that.
148) Build a product like a turnkey solution and find that customers who wanted that in the first place now want to customize it.
149) Build a product with automation packaged via scripts and see its adoption grow astronomically.
150) Build a product that requires non commodity parts and hardware and find that customers are trying to pull apart the product to suit their needs.
151) Build a product that throws arcane errors when the users go off the beaten path
152) Build a product that does not guide the user back to the rails when they go off the beaten path
153) Build a product which has all the necessary ingredients for a successful production deployment but find that the secrets used to operate the software are irreplaceable
154) Build a product which has little or no value in terms of data visualization and find the audience looking for alternatives
155) Build a product in parts where one hand makes one part and another makes a different part and both are not coordinated
156) Build a product where the bug count doesn’t seem to come down and find interesting ideas to hide it
157) Build a product with little or no release gates and see the instability reach the customers.
158) Build a product where the fixes go all the way to the customer on the same day but have it repeated until customers make their mind
159) Build a product with breaking changes in published interfaces and the customers complain.
160) Build a product where there are no applications without involving external software and dependencies and take a hit when any of them in introduce breaking changes

This is a continuation of the earlier posts to enumerate funny software engineering practice:



120) Build a product that takes a lot of effort involved in explaining workarounds when a feature could be added.
121) Build a product where the cost of servicing overwhelms the amortized cost of purchase.
122) Build a product where the customers are enticed to subscription model or cloud tenancy only to discover the total cost to be significantly high for developmental activities
123) Build a product where the customers have to deal with two problems simultaneously: on-premise as well as cloud encumbrances
124) Build a product that requires subscription to services and use the legal language in the contract to extort payments
125) Build a product that requires processing power that is only available on desktops.
126) Build a product that is not elastic I’m storage, compute and networking only to be surprised by crowd peak traffic
127) Build a product that has little or no configuration option for sources of input prompting dedicated instances for different use cases
128) Build a product that reveals hidden bugs after warranty period
129) Build a product where the maker funds white hat discovery of vulnerabilities which prompt the customer to upgrade
130) Build a product with changing goals for v1 while the market tide would have lifted any v1 for immediate business need and with a compelling v2 to establish business.
131) Build a product with that has terms of license written in a way that brings more revenue from the same product over time or use.
132) Build a product with so much emphasis on performance that it becomes a nice product in the magic quadrant
133) Build a product with limited or no expansion possibilities for emerging trends and find the technical debt to be rather costly in the next release
134) Build a product with exacting demands for deployment and find the users running away to competitors.
135) Build a product with little or no configuration for toggling features and find that the majority of the functionality is locked down because there is no granularity
136) Build a product with no dedicated management console/portal and find a section of the audience spurned away.
137) Build a product that requires work for integration and find that there are bugs in standalone mode that would have been detected sooner.
138) Build a product which shows errors in a non-deterministic manner and find costs soaring for investigations
139) Build a product where a single bug fix post release as a patch is either too costly or hard to enforce.
140) Build a product where the number bug fixes in frequent patches make the customers roll their eyes.

This is a continuation of the earlier posts to enumerate funny software engineering practice:


110) Build a product that starts to sound discrepant in the ads as releases slip.
111) Build a product that does not uninstall clean
112) Build a product that leaves artifacts even when the customer does not want it
113) Build a product that makes it annoying for the customers to switch out components
114) Build a product that improves security with a trade-off on convenience.
115) Build a product that provides inadequate segregation of user activities or secrets letting others to see the actions taken.
116) Build a product that provides that behaves well in development but fail in production because of deficiencies in system resources.
117) Build a product that is considered golden only to find that the customer has been using an incorrect version
118) Build a product that takes a sweeping approach of solving problems by updating versions while not sufficiently resolving special cases.
119) Build a product that takes does not have sufficient levers to resolve small customer cases.
120) Build a product that takes a lot of effort involved in explaining workarounds when a feature could be added.
121) Build a product where the cost of servicing overwhelms the amortized cost of purchase.
122) Build a product where the customers are enticed to subscription model or cloud tenancy only to discover the total cost to be significantly high for developmental activities
123) Build a product where the customers have to deal with two problems simultaneously: on-premise as well as cloud encumbrances
124) Build a product that requires subscription to services and use the legal language in the contract to extort payments
125) Build a product that requires processing power that is only available on desktops.
126) Build a product that is not elastic I’m storage, compute and networking only to be surprised by crowd peak traffic
127) Build a product that has little or no configuration option for sources of input prompting dedicated instances for different use cases
128) Build a product that reveals hidden bugs after warranty period
129) Build a product where the maker funds white hat discovery of vulnerabilities which prompt the customer to upgrade
130) Build a product with changing goals for v1 while the market tide would have lifted any v1 for immediate business need and with a compelling v2 to establish business.

The benefits of Async stream readers in cache for stream storage:
The cache essentially projects portions of the stream so that the entire stream does not have to scanned from begin to end repeatedly. This mode of operation is different from finding popular stream segments across stream readers.
The cache may use a queue to hold contiguous segments from a stream.
The choice of queue is extremely important for peak throughput. The use of a lock free data structure as opposed to an ArrayBlockingQueue can do away with lock contention. The queues work satisfactorily until they are full. Garbage free Async stream readers have the best response time.
The benefits here are similar to LogAppenders except that they are on write path whole these are in read path.
Asynchronous read together with lock free access and skip level access boost performance. The cache may encounter significant size for each segment and a writer that transfers a segment over the network may take time in the order of hundreds of milliseconds. Instead, having a continuous background import for adjustable window of steam segments tremendously improves the load on the stream store while responding to seek requests on segments

Please refer to the discussion on caching context for stream managers with:

https://1drv.ms/w/s!Ashlm-Nw-wnWvBXkU9jz_Z2EXWLp

and  implementation where it is applicable with: https://github.com/ravibeta/JavaSamples/tree/master/logging

We were discussing the cache usage for stream access:

When the segments in the stream have skip level access by say 2,4,8 adjacent nodes, the cache can access the segments via skip levels and prefetch those that will be read. Skip level access on streams means that we are able to perform as fast as random access over sequential streams.

The cache may use indexes on locations to augment the deficiency of storing record locations in the stream store. This index is merely a translation of the sequential segment number from the stream store in terms of the leaps of the contiguous segments we need to make. And the best way to do that for that particular segment. Given a segment it’s sequential number from the start may be internal to the stream store. However, if that number is available from the stream store to be  mapped with the segment whenever it is cached, then the translation of the location to the segment in terms of skip-level access is straightforward for example the number 63 from start, will require as many multiples of 8 less than target, same with multiples of 4 starting from the position left with the previous step, then multiples of 2 such that they are maximized in that order so that the overall count is least. This computation benefits in bringing ranges based on numbers alone rather than range indexes based on say BTree

Without the location available from the stream store some persistence is needed for the lookup of the segment number for the corresponding segment and usually involves an iteration of all the segments from the store. A hash of the segment may be sufficient for these lookups.

The hierarchical representation of stream segments may be facilitated with other data structures but they tend to centralize all operations. The purpose of skip level access is faster access on the same sequential access so that no other data structures are necessary

Another approach is to maintain the segment numbers on both sides. For example, the cache may have clients that read from the start of the stream up to a targeted segment. The stream store may perform repeated scans as it serialized the client’s accesses to the stream. The cache has the opportunity to bypass the stream store and alleviate the workload on the stream store by providing the segments that are most popular between accesses. As each client presents a target segment number and the stream store presents the segment numbers from the same or different stream, the cache has the opportunity to come up with segment numbers based on relevance via skip level access and priority via its eviction policy. The cache therefore becomes an intelligent agent that does away with redundant scans of streams by the store.

The overlapping interval range between segment numbers is decided by the cache which it uses with skip level access to fetch the segments. Efficient representation of such a range of segments is easy with the same logic as demonstrated for a targeted segment number. In this case, the same algorithm is repeated for begin and end of targeted range and the interval is represented in terms of the skip level access between the start and the end.

Using cache for stream access:

The cache for streams is a write through cache because the writes are at the end of the stream. These writes are hardly a concern for cache and their performance is mitigated by Bookkeeper. Batched writes from Bookkeeper are not different from a periodic backup schedule.
Read access patterns benefit from some organization of segments
When the segments in the stream have skip level access by say 2,4,8 adjacent nodes, the cache can access the segments via skip levels and prefetch those that will be read. Skip level access on streams means that we are able to perform as fast as random access over sequential streams.
The cache may use indexes on locations to augment the deficiency of storing record locations in the stream store. This index is merely a translation of the sequential segment number from the stream store in terms of the leaps of the contiguous segments we need to make. And the best way to do that for that particular segment. Given a segment it’s sequential number from the start may be internal to the stream store. However, if that number is available from the stream store to be  mapped with the segment whenever it is cached, then the translation of the location to the segment in terms of skip-level access is straightforward for example the number 63 from start, will require as many multiples of 8 less than target, same with multiples of 4 starting from the position left with the previous step, then multiples of 2 such that they are maximized in that order so that the overall count is least. This computation benefits in bringing ranges based on numbers alone rather than range indexes based on say BTree
Without the location available from the stream store some persistence is needed for the lookup of the segment number for the corresponding segment and usually involves an iteration of all the segments from the store. A hash of the segment may be sufficient for these lookups.
The hierarchical representation of stream segments may be facilitated with other data structures but they tend to centralize all operations. The purpose of skip level access is faster access on the same sequential access so that no other data structures are necessary

This is a continuation of the previous post to enumerate funny software engineering practice



90) Build a product that scales out to high workload but with more faults than before.

91) Build a product that is oblivious to market needs as the release cycle grows to years

92) Build a product that trades-off compute in favor of storage but leaves the onus of moving data to users

93) Build a product that trades-off storage for compute but never get exercised in routine activities

94) Build a product that requires proprietary protocol, format or content and have the ecosystem scratch their heads for integration

95) Build a product that does not work well with others because it does not provide a bridge or a connector

96) Build a product that is eager to monopolize the market rather than leaving space for better solutions to thrive without sacrificing the competence of the product.

97) Build a product that is measured by its revenue rather than the mindshare.

98) Build a product without embracing developers with attractive software developer kits or community editions

99) Build a product without embracing enticing developers with easy install lite editions for their developmental work

100) Build a product that does not allow a forum for ideas to be exchanged about the project and find that the home-grown ideas are not appealing enough.

101) Build a product that leaves out the details of the use cases which the customers would find most relevant to their needs and find out afterwards how much cost it would have saved.

102) Build a product that does not explain what it cannot do but keep finding that they try it  anyway. .

103) Build a product with little or no support for administrators to configure against inappropriate usages.

104) Build a product that produces more power brokers than users.

105) Build a product that reimagines functionality tantalizing new users at the cost of habitual old users.

106) Build a product that styles the user interface so drastically it looks designer rather than commercial.

107) Build a product with translated text in local languages that reads funny.

108) Build a product that doesn’t take into consideration of reading right to left in certain regional languages.

109) Build a product that improves the quality but cuts down the features at crunch time

110) Build a product that starts to sound discrepant in the ads as releases slip.

111) Build a product that does not uninstall clean

112) Build a product that leaves artifacts even when the customer does not want it

#codingexercise
Void MinHeapify(List<Node> sorted, int i)
{
Sorted[i-1].left =  ( 2 x i <= sorted.count ) ? sorted[2xi-1] : null;
Sorted[i-1].right = (2 x I + 1 <= sorted.count) ? sorted[2xi+1-1] : null;
}

This is a continuation of the previous post to enumerate funny software engineering practice

90) Build a product that scales out to high workload but with more faults than before.
91) Build a product that is oblivious to market needs as the release cycle grows to years
92) Build a product that trades-off compute in favor of storage but leaves the onus of moving data to users
93) Build a product that trades-off storage for compute but never get exercised in routine activities
94) Build a product that requires proprietary protocol, format or content and have the ecosystem scratch their heads for integration
95) Build a product that does not work well with others because it does not provide a bridge or a connector
96) Build a product that is eager to monopolize the market rather than leaving space for better solutions to thrive without sacrificing the competence of the product.
97) Build a product that is measured by its revenue rather than the mindshare.
98) Build a product without embracing developers with attractive software developer kits or community editions
99) Build a product without embracing enticing developers with easy install lite editions for their developmental work
100) Build a product that does not allow a forum for ideas to be exchanged about the project and find that the home-grown ideas are not appealing enough.
101) Build a product that leaves out the details of the use cases which the customers would find most relevant to their needs and find out afterwards how much cost it would have saved.
102) Build a product that does not explain what it cannot do but keep finding that they try it  anyway. .
103) Build a product with little or no support for administrators to configure against inappropriate usages.
104) Build a product that produces more power brokers than users.
105) Build a product that reimagines functionality tantalizing new users at the cost of habitual old users.
106) Build a product that styles the user interface so drastically it looks designer rather than commercial.
107) Build a product with translated text in local languages that reads funny.
108) Build a product that doesn’t take into consideration of reading right to left in certain regional languages.
109) Build a product that improves the quality but cuts down the features at crunch time
110) Build a product that starts to sound discrepant in the ads as releases slip.

This post tries to discover the optimum access patterns for the stream storage. The equivalent for web accessible storage has proven to be most efficient with a batched consistently uniform periodic writes such as those for object storage. The streams are read from the beginning to the end and this has generally not been a problem because the workers doing the read can scale out with low overhead. The writes are always at the end. Temporary storage with the help of Bookkeeper and coordination with zookeeper helps with the writes to be append only

However statistics collected by Bookkeeper and Zookeeper can immensely improve the discoverability of the access pattern. These statistics are based on the table of stream and their accessors count with attributes. As the workload varies, it may show that some streams are read heavy such as for analysis and others are write heavy such as for IoT traffic. The statistics may also show that even some segments within the stream are more popular than others. While this may have any distribution, the idea that the heavily used segments can benefit with a cache is not disputed

The cache for streams is a write through cache because the writes are at the end of the stream. These writes are hardly a concern for cache and their performance is mitigated by Bookkeeper. Batched writes from Bookkeeper are not different from a periodic backup schedule.

The cache layer for stream store benefits from being close to the stream store. The segments can have a naming convention or a hash.

Reference to a case study for event stream :
https://1drv.ms/w/s!Ashlm-Nw-wnWu2hMRA7zp__mivoo

This is a continuation of the previous post to enumerate funny software engineering practice:


Build a product where users are quick to assume something but the product does another thing

Build a product where the defects are hidden in the form of caveats and workarounds

Build a product that scales out to high workload but with more faults than before.

Build a product that is oblivious to market needs as the release cycle grows to years

Build a product that trades-off compute in favor of storage but leaves the onus of moving data to users

Build a product that trades-off storage for compute but never get exercised in routine activities

Build a product that requires proprietary protocol, format or content and have the ecosystem scratch their heads for integration

Build a product that does not work well with others because it does not provide a bridge or a connector

Build a product that is eager to monopolize the market rather than leaving space for better solutions to thrive without sacrificing the competence of the product.

Build a product that is measured by its revenue rather than the mindshare.

Build a product without embracing developers with attractive software developer kits or community editions

Build a product without embracing enticing developers with easy install lite editions for their developmental work

Build a product that does not allow a forum for ideas to be exchanged about the project and find that the home-grown ideas are not appealing enough.

Build a product that does not allow a forum for ideas to be exchanged about the project and find that the home-grown ideas are not appealing enough.

Build a product that leaves out the details of the use cases which the customers would find most relevant to their needs and find out afterwards how much cost it would have saved.

Build a product that does not explain what it cannot do but keep finding that they try it  anyway. .

Build a product with little or no support for administrators to configure against inappropriate usages.

The Kubernetes Event generation, filtering and propagation considerations for component owners and destination architect:
1) A Kubernetes Label is what makes a regular K8s (short for Kubernetes) event a Selected event.  This label is not something that the Selected team can do without. It is their criteria in their configmap with saying “matchOn” with a name. They cannot create alternate criteria that says matchOn level = critical or events with remedies.  This is a label that they need the event generators to decide and add-on. Therefore, we must work with some of their limitations here. At the same time the event promotion rules is a good idea and having a set for criteria for the rules is helpful to anyone. If the criteria is specific to a component or to Selected, then its best to put it in their corresponding ConfigMap.
2) 1) does not mean we have to generate Selected events only. In fact, we should generate K8S events because with or without the criteria for Selected, this only improves the diagnostics of the component and it is something that only the component owners know best.
3) The advice for the component owner has been generate more K8s events. However, the Kubernetes also raised K8s events for almost all resources. Therefore, there should not be any overlap between the events raised by the Kubernetes by default and those from the component
4) In the past, we have discussed a layered approach to publishing K8s events, promoting K8s events to Selected events which in turn gets notified to Special events . There has been speculation about how Selected picks these events or should be doing so in the future. I suggest that picking and promoting events based on matchOn and rules is something that any layer can do. Therefore, it is better to separate the event generation from the event propagation. These operations are not the dedicated responsibility of any single component or layer but something that can be performed by each if they choose as such.
5) The event generation has also been different from different components but anything to do with the operational aspect is welcome. The difference has been to justify emitting events to authoring the resources with states so that Kubernetes automatically emits the events.
6) In some cases, the native K8s events don’t have enough information or message. They could be enhanced with identifiers and names that may assist SRS and other consumers by finding everything in the event itself.
7) The most important aspect for any specific component has been 5) and 6) above because it handles requests for all service instances and service bindings which are dynamic during the lifetime of the product and the source for most troubleshooting tickets.
8) As the service catalog does not capture the context in which these instances and bindings are created, we have an opportunity to improve these from this component.
9) Lastly, I want to add that the handlers of Special events are concerned about having few events rather than a flood of events. Unlike other products which don’t have infrastructure like Kubernetes, events can be numerous. Therefore, this restriction for number of events must be removed from both special events handlers and user interface otherwise the customers will not benefit as much from these events.

This is a continuation of the previous post to enumerate funny software engineering practice:

Build a product to suit the performance goal for a privileged set at the expense of that for others.



Build a product that scales out to high workload but with more faults than before.





Build a product that users do not know worked in certain ways.







Build a product where users are quick to assume something but the product does another thing







Build a product where the defects are hidden in the form of caveats and workarounds


Build a product that scales out to high workload but with more faults than before.

Build a product that is oblivious to market needs as the release cycle grows to years

Build a product that trades-off compute in favor of storage but leaves the onus of moving data to users

Build a product that trades-off storage for compute but never get exercised in routine activities

Build a product that requires proprietary protocol, format or content and have the ecosystem scratch their heads for integration

Build a product that does not work well with others because it does not provide a bridge or a connector

Build a product that is eager to monopolize the market rather than leaving space for better solutions to thrive without sacrificing the competence of the product.

Build a product that is measured by its revenue rather than the mindshare.

Build a product without embracing developers with attractive software developer kits or community editions

Build a product without embracing enticing developers with easy install lite editions for their developmental work

Build a product that does not allow a forum for ideas to be exchanged about the project and find that the home-grown ideas are not appealing enough.