We discussed techniques for request chaining here:  https://www.blogger.com/blogger.g?blogID=1985795500472842279#editor/target=post;postID=859017183586117344;onPublishedMenu=allposts;onClosedMenu=allposts;postNum=0;src=link

The best way to establish chaining is with a common attribute specific only to the chaining. 

one such attribute is the customer id. For example, this untested but illustration only example for searching indicates the chaining is automatically established as part of selection:

#! /usr/bin/bash
#
if [ $# -ne 1 ]
then
 echo "Usage: chainer $0 customerId"
 exit $E_BADARGS
fi
listfiles=$(ls -d -1 $PWD/manaus*.*)
unset files
while IFS= read -r line
do
    files+=("$line")
done < <(echo "$listfiles")
echo $files
for file in ${files[@]}
do
  echo "file="; echo $file;
  search="zgrep -n \"^customerID=$1\" $file | cut -d\":\" -f1"
  echo $search; echo
  command_test=$(whatis "$search" | grep 'nothing appropriate');
  echo "search=";echo $search;
  unset linesno
  linesno=()
  if [[ -z "$command_test" ]]
  then
    echo "zgrep available"
    while IFS= read -r line
    do
        echo $line; echo
        linesno+=("$line")
    done < <(echo "$search")
  echo "linesno="; echo $linesno;
  for lineno in $linenos
  do
  if [! -z "$lineno" ]
  then
     echo "improper line no: $lineno"; echo
     continue;
  fi
  $text=`zcat $file | head -n $(($lineno-1+2)) | tail -n 3`
  echo $text; echo
  done
  else
  echo "no suitable command found"
  fi
done

: <<'output'
/home/ravi/manaus.tar.gz
file=
/home/ravi/manaus.tar.gz
zgrep -n "^customerID=food" /home/ravi/manaus.tar.gz | cut -d":" -f1
zgrep -n "^customerID=food" /home/ravi/manaus.tar.gz | cut -d":" -f1: nothing appropriate.
search=
zgrep -n "^customerID=food" /home/ravi/manaus.tar.gz | cut -d":" -f1
zgrep available
zgrep -n "^customerID=food" /home/ravi/manaus.tar.gz | cut -d":" -f1
linesno=
zgrep -n "^customerID=food" /home/ravi/manaus.tar.gz | cut -d":" -f1
output

Storage Locker
Data is just as precious as anything else. While storage frameworks in the cloud and on-premise promise perpetual availability and sound security, they do not offer any differentiation to data to treat is as either sensitive or not. Moreover, they may change their policies every few years and do not offer any guarantees that the data will not be handled with minimal intervention.
Companies exist for record management, secure storage and secure destruction but they usually service backup data and often manage the archives. Sensitive data on the other hand may not live in an archive but can remain in a database, unstructured data or even shared among trusted subsidiaries. Locker services does not differentiate between live and aged data.
The vulnerabilities, threats and attacks in the cloud are discussed in survey of cloud security and made publically available. These include:
1) shared technology vulnerabilities - increased leverage of resources gives the attackers a single point of attack.
2) Data breach - with data protection moving from cloud consumer to cloud service provider, the risk for data breach grows
3) Account of service traffic hijacking - Since the data moves over internet, anybody who hijacks the account could mount a loss of service
4) Denial of service - a denial of service attack on the cloud provider affects all
5) malicious insider - a determined insider can find more ways to attack and cover tracks in a cloud scenario 6) Internet protocol : IP connectivity is a requirement for data but comes with its own vulnerabilities
7) injection vulnerabilities - XSS, sql injection and other injection vulnerabilities in the management layer affect even otherwise secure data
8) API & browser vulnerabilities - vulnerability in the cloud provider's API may also affect data security
9) Changes to business models - cloud computing may require consumers to change their business models and this introduces regressions from previous security reviews
10) abusive use - cloud computing is inviting all with zero cost subscription. While it is designed to mitigate denial of service attacks, it does not stop malicious users from trying.
11) malicious insider - even insiders of a cloud provider could become malicious
12) availability - the system has to be available at all times and while cloud providers take extra ordinary efforts, they may suffer from outages such as power

One of the  trends in operational practice is to rely on tools that sets thresholds and raises alerts. This translates to incident response instead of active and strenuously polling. As part of the response, we search the logs. Most of these are interactive command line executions but each step may be time consuming due to the volume of the logs. One way to mitigate this is to run a sequential batch script that repeats the commands on smaller chunks of data. This however means we lose the aggregations unless we store intermediary data. Fortunately this was possible using files. However most log archive systems are read only and the files may not be read from. This also restricts parallelizing tasks using library such as celery because those require network access to message broker and the only access allowed is ssh. One way to overcome this is to scatter and gather data from multiple ssh sessions. This is easier to automate because the controller does not have to be local to the log server.
Another option is to leave the log server as-is and draw all the data into a log index. Then the search and reporting stacks can use the index. Since the index is designed to grow to arbitrary size, we can put all the logs in it. Also, the search stack enables as many search sessions as necessary to perform the task. They may even be made available via API, SDK and UI which enable applications to leverage parallelism as appropriate. For example, the SDK can be used with task parallel libraries such as Celery so that the same processing can be done in batches of partitioned data. The data can be partitioned based on historical timeline or they can be partitioned based on other attributes. The log index server also helps the application to preserve search artifacts so that the same can be used later or in other searches. The reporting stack sits over the search stack because the input to the reporting dashboard is the results of search queries. These search queries may be optimized, parallelized or parameterized so that they have near real-time performance. The presence of search and reporting stacks in new log indexing products indicates that these are separate areas of concerns which cannot be mixed with the conventional log readers into a monolithic console session.

Today we continue our discussion on the AWS papers in software architecture which suggests five pillars:

- Operational Excellence for running and monitoring business critical systems.

- Security to  protect information, systems, and assets with risk assessments and mitigation strategies.

- Reliability to  recover from infrastructure or service disruptions

- Performance Efficiency to ensure efficiency in the usage of resources

- Cost Optimization  to help eliminate unneeded cost and keeps the system trimmed and lean.

The guidelines to achieve the above pillars include:

1. Infrastructure capacity should be estimated not guessed

2. Systems should be tested on production scale to eliminate surprises

3. Architectural experimentation should be made easier with automation

4. There should be flexibility to evolve architectures

5. Changes to the architecture should be driven by data

6. Plan for peak days and test at these loads to observe areas of improvement

We looked at the Operational Excellence, Reliability and security pillar and we reviewed the associated best practices.

One of the  trends in operational practice is to rely on tools that sets thresholds and raises alerts. This translates to incident response instead of active and strenuously polling. As part of the response, we search the logs. Most of these are interactive command line executions but each step may be time consuming due to the volume of the logs. One way to mitigate this is to run a sequential batch script that repeats the commands on smaller chunks of data. This however means we lose the aggregations unless we store intermediary data. Fortunately this was possible using files. However most log archive systems are read only and the files may not be read from. This also restricts parallelizing tasks using library such as celery because those require network access to message broker and the only access allowed is ssh. One way to overcome this is to scatter and gather data from multiple ssh sessions. This is easier to automate because the controller does not have to be local to the log server.

Today we continue our discussion on the AWS papers in software architecture which suggests five pillars:
- Operational Excellence for running and monitoring business critical systems.
- Security to  protect information, systems, and assets with risk assessments and mitigation strategies.
- Reliability to  recover from infrastructure or service disruptions
- Performance Efficiency to ensure efficiency in the usage of resources
- Cost Optimization  to help eliminate unneeded cost and keeps the system trimmed and lean.
The guidelines to achieve the above pillars include:
1. Infrastructure capacity should be estimated not guessed
2. Systems should be tested on production scale to eliminate surprises
3. Architectural experimentation should be made easier with automation
4. There should be flexibility to evolve architectures
5. Changes to the architecture should be driven by data
6. Plan for peak days and test at these loads to observe areas of improvement
We looked at the Operational Excellence, Reliability and security pillar and we reviewed the associated best practices.
#codingexercise
Find the nth multiple of k in Fibonacci Series
solution 1 : iterate through the Fibonacci Series testing and counting success
solution 2: Fibonacci multiples of a number are periodic. depending on k determine the period and hence the position of the result.
int GetNthMultipleFibonacci (int k, int n)
{
int multiple = -1;
for (int I = 0; I  < int_max; i++)
{
if (GetFibonacci (i) % k == 0){
    multiple = i + 1;
    break;
}
}
if (multiple == -1) return -1;
int position = n * multiple;
return GetFibonacci (position);
}

Today we continue our discussion on the AWS papers in software architecture which suggests five pillars:
- Operational Excellence for running and monitoring business critical systems.
- Security to  protect information, systems, and assets with risk assessments and mitigation strategies.
- Reliability to  recover from infrastructure or service disruptions
- Performance Efficiency to ensure efficiency in the usage of resources
- Cost Optimization  to help eliminate unneeded cost and keeps the system trimmed and lean.
The guidelines to achieve the above pillars include:
1. Infrastructure capacity should be estimated not guessed
2. Systems should be tested on production scale to eliminate surprises
3. Architectural experimentation should be made easier with automation
4. There should be flexibility to evolve architectures
5. Changes to the architecture should be driven by data
6. Plan for peak days and test at these loads to observe areas of improvement
We looked at the Operational Excellence, Reliability and security pillar and we reviewed the associated best practices.
Next we review the performance-efficiency pillar which includes the ability to use computing resources efficiently even with the fluctuations in demand and as technology evolves.
It includes five design principles. These are:
Vendor aware deployments - This implies that we don't need to host and run a new technology. Databases, machine learning, encodings are best done at the cloud level by dedicated teams so that our service may simply use it.
global availability - We deploy the system in multiple regions around the world so they provide lower latency and more availability.
serverless architectures - This notion eliminates ownership of servers for the computations and storage services act as static websites. Even the event services can be used to host the code
experiment more often - with virtual and automate-able resources, we can carry out comparative , we can quickly evaluate which T-shirt size works for us
Mechanical sympathy - This calls for using the technology that best helps us to achieve what we want with our service.
 The four best practice areas in this regard are:
Selection - As wirkloads vary, the solution becomes more nuanced about the choice of products and often involves a hybrid approach to overcome trade-offs. If the choices are done on a cyclical basis.the solution improves over time
Review - This is about evaluating newer technologies and retiring older technologies  The cloud services for example become available in new regions and upgrade their services and features.
Monitoring - This gives continuous feedback on the systems as deployed so that alarms can be set in place for actions to be taken
Trade-offs- The initial design may have considered trade-offs such as consistency, durability and space versus time or latency to deliver higher performance but these also need to be done with subsequent change management
#codingexercise
Find the nth multiple of k in Fibonacci Series
solution 1 : iterate through the Fibonacci Series testing and counting success
solution 2: Fibonacci multiples of a number are periodic. depending on k determine the period and hence the position of the result.
int GetNthMultipleFibonacci (int k, int n)
{
int multiple = -1;
for (int I = 0; I  < int_max; i++)
{
if (GetFibonacci (i) % k == 0){
    multiple = i + 1;
    break;
}
}
if (multiple == -1) return -1;
int position = n * multiple;
return GetFibonacci (position);
}

Today we continue our discussion on the AWS papers in software architecture which suggests five pillars:
- Operational Excellence for running and monitoring business critical systems.
- Security to  protect information, systems, and assets with risk assessments and mitigation strategies.
- Reliability to  recover from infrastructure or service disruptions
- Performance Efficiency to ensure efficiency in the usage of resources
- Cost Optimization  to help eliminate unneeded cost and keeps the system trimmed and lean.
The guidelines to achieve the above pillars include:
1. Infrastructure capacity should be estimated not guessed
2. Systems should be tested on production scale to eliminate surprises
3. Architectural experimentation should be made easier with automation
4. There should be flexibility to evolve architectures
5. Changes to the architecture should be driven by data
6. Plan for peak days and test at these loads to observe areas of improvement
We looked at the security pillar and we reviewed its best practices.
They include identity and access management, monitoring controls, infrastructure protection, data protection and incident response.
Next we review the reliability pillar which includes the ability of a system to recover from infrastructure or service disruptions, dynamically acquire computing resources to meet demand, and mitigate disruptions.
It includes five design principles:
Test recovery procedures - Cloud let us simulate failures as well as test recovery procedures because the resources are elastic With simulations we can now test and rectify before a real failure
Automatic recovery from failure : Automation can be event based so that it kicks in only when a threshold is reached.Notification and tracking of failures no longer require polling. Instead recovery and repair can be automated.
Scale horizontally to increase aggregate system reliability. We replace one large resource with multiple small resources to reduce the impact of a single resource
Capacity no longer needs to be guessed. Prior to cloud the resources were saturated frequently leading to more failures. Instead cloud lets us  monitor demand and react accordingly.
Manage change in automation - changes to the infrastructure are done using automation.
The best practice areas for reliability in the cloud include :
Foundations -  Network topology and service limits are two key criteria to establish a foundation. Often the bandwidth compute and storage limits were run-over earlier. Cloud lets us manage foundation better. In AWS, we just have to pay attention to the topology and service limits.
Change Management - Every change may violate the SLA process, therefore change control process are necessary. How a system adapts to changes in demand, how the resources are monitored and how the change is executed determine this best practice.
Failure Management - Becoming aware of failures, responding to them and preventing them from happening again are part of this best practice.
The AWS well architected framework does not provide an example of where such pillars are used in prominence in the industry today. One such example is the management of Governance Risk and Compliance (GRC) The fianancial services industry is highly regulated and has an increasing need to break the tradeoff between compliance and innovation
#codingexercise
Given a number N, find the number of ways you can draw N chords in a circle with 2*N points such that no 2 chords intersect.
If we draw a chord between any two points the set of points get divided into two smaller sets and there can be no chords going from one set to another set. On the other hand the solution for the smaller set in now an optimal sub-problem. Therefore a recurrence involving different configurations of smaller sets is possible. These configurations range in smaller set having variations from 0 to N-1 pairs