Cluster computing

Sunday, December 11, 2016

Today we take a break to talk about LXD containers and how to enable static ip addressing of the containers instead of the conventional DHCP.
Many organizations find it easier to hand out specific ip addresses and ip address ranges rather than have Linux containers obtain ip addresses in a range over DHCP. This lets them tightly control which ip addresses containers may be assigned to and facilitates static ip networking for separation of network from the more heavily used networks.
Static ip networking method for linux containers is described here : http://jason.trickett.us/2016/08/lxd-containers-static-ip-addresses-heres/ That post says that we can specify static ip address with

sudo sh -c "echo 'dhcp-host=containername,ipaddress' >> /etc/default/lxd_dnsmasq.conf"

once we have mentioned LXD_CONFILE="/etc/default/lxd_dnsmasq.conf" in /etc/default/lxd-bridge
Here we briefly mention the containers capabilities and design that facilitate this
First, the container provides the ability to create bridged networks by default. This lets outside in connectivity to the containers over a single interface of the host server
Second, LXD facilitates customizations with the following parameters specified in /var/lib/lxc/ContainerName/config

lxc.network.name this is the name given to the interface and is used to override the dynamically generated name
lxc.network.hwaddr this is the interface MAC address to override the one provided
lxc.network.ipv4 this is the ipv4 address to assign to the virtualized interface and also includes the broadcast address
lxc.network.ipv4.gateway this is the ipv4 address to use as the gateway inside the container.

With these four parameters, we can pretty much define the connectivity for the containers fully.

#codingexercise
Find the minimum number of merge operations to make an array palindrome.
For example,
arr[] = {15, 4, 15 } Answer : 0 // already a palindrome
arr[] = {1, 4, 5, 1} Answer : 1 // merge 4 and 5 to 9
arr[] = {11, 14, 15, 99} Answer : 3 // merge all

int GetMinMerge(List<int> arr)
{
int count = 0;
int start = 0;
int end = arr.Count-1;
while (start <= end)
{
if (arr[i] == arr[j])
{
start++;
end--;
}
else if (arr[i] > arr[j])
{
j--;
arr[j]+=arr[j+1];
count++;
}
else
{
i++;
arr[i] += arr[i-1];
count++;
}
}
return count;
}
#writeup https://1drv.ms/w/s!Ashlm-Nw-wnWlkDRgzKeR5lVB1Ml

Saturday, December 10, 2016

We were comparing Simplex algorithm to Nested Sampling for general Bayesian. We observed that both rely on picking a next estimate and either accepting or rejecting the new candidate. In simplex algorithm given the objective function max 2x1+5x2

1) 2x1-x2 <= 4

2) X1 + x2 <= 9

3) -x1 + x2 <= 3

4) X1 >= 0

5) X2 > = 0

Simplex can start at the origin with inequalities 4) and 5). As x2 is gradually increased, the first constraint is met with equation 3 and therefore it stops at x2 = 3 and the new vertex is given by 3) and 4) We converge towards the solution in the next step

In Nested Sampling if we take an example of j=5 iterations and N=3 samples, initially the three points are taken from the entire parameter space which is equivalent to taking randomly over X from 0 to 1. Their labels are not known. The worst of these is taken and replaced so that all three are then uniform in a reduced range X1. In step 2, the new point is picked as say inner most and is identified as number 5. After five such iterations there are five such discarded points and three survivors. It is over these points that the weighted sum is then evaluated resulting in summation over j+N
The difference between the two procedures is that Simplex is exhaustive in the surface it covers where as Nested Sampling is savvy about what it picks because it seems to sample statistically.
In other words, we could consider transforming the simplex problem into one based on likelihoods and then pick samples from their for improvements.
#codingexercise
Find the minimum number of insertions to make a string palindrome
For example:
String Results #Insertions
ab bab 1
aa aa 0
abcd dcbabcd 3
abcda adcbcda 2
int GetMinInsertions(List<char> A, int start, int end)
{
if (start > end) return INT_MAX;
if (start == end) return 0;
if (start == end - 1) return (A[start] == A[end]) ? 0: 1;
return A[start] == A[end] ?
GetMinInsertions(A, start+1, end-1):
min( GetMinInsertions(A, start, end -1), GetMinInsertions(A, start+1, end)) + 1;
}

Friday, December 9, 2016

Today we compare the techniques between Simplex algorithm and Nested Sampling for general Bayesian.

The Simplex algorithm solves a linear program - set of linear equations. It takes a set of linear inequalities and a linear objective function and finds optimal feasible point by the following strategy.

Find a vertex v in the feasible region

While there’s a neighbor v’ of v with better objective value: set v = v’

A linear equation involving the n dimensions defines a hyperplane when plotted in an n-dimensional space and the corresponding linear inequality defines a half-space, all points that are precisely hyperplane or lie on one particular side of it. The feasible region of a linear program specified by a set of inequalities is the intersection of the corresponding half-spaces, a convex polyhedron.

But what do the concepts of vertex and neighbor mean in this general context. Each vertex is the unique point at which some set of hyperplanes meet.

Each vertex is specified by a set of n inequalities.

Two vertices are neighbours if they have n-1 defining inequalities in common.

On each iteration of the algorithm, the simplex has two tasks

Check whether the current vertex is optimal

Determine where to move next

As we will see, both tasks are easy if the vertex happens to be at the origin. And if the vertex is elsewhere, we will transform the co-ordinate system to move it to the origin.

When the origin is feasible, it is certainly a vertex, since it is the unique point at which the n inequalities are tight.

The origin is optimal if and only if all ci <= 0 where ci is the contraint we try to maximize in each dimension

If all ci <= 0, then considering the constraints x >= 0, we can’t hope for a better objective value. Conversely if the origin is not optimal, we can improve one of the constraints.

Thus for task 2 we can move by increasing some xi for which ci > 0. We can keep on increasing it until we hit another constraint. At that point we again have n tight inequalities and hence we arrive at a new vertex.

When we compare the Simplex algorithm with the Nested Bayesian, we see both rely on picking a next estimate and either accepting or rejecting the new candidate.

#codingexercise
Print a two dimensional matrix in spiral form
static void PrintSpiralRecursive(int[,] A, int left, int right, int top, int bottom)
{
int r = bottom - top + 1;
int c = right - left + 1;
if (r < 0 || c < 0) return;
if (r == 2 && c == 2)
{
Console.Write("{0} {1} {2} {3}", A[top, left], A[top, right], A[bottom, right], A[bottom, left]);
return;
}
if (r == 1 && c == 1)
{
Console.Write("{0} ", A[top, left]);
return;
}
if (r == 1)
{
for (int j = left; j <= right; j++)
Console.Write("{0} ", A[top, j]);
return;
}
if (c == 1)
{
for (int i = top; i <= bottom; i++)
Console.Write("{0} ", A[i, right]);
return;
}

for (int j = left; j < right; j++)
Console.Write("{0} ", A[top, j]);
for (int i = top; i < bottom; i++)
Console.Write("{0} ", A[i, right]);
for (int j = right; j > left; j--)
Console.Write("{0} ", A[bottom, j]);
for (int i = bottom; i > top; i--)
Console.Write("{0} ", A[i, left]);
if (left + 1 < right && top + 1 < bottom)
PrintSpiralRecursive(A, left + 1, right - 1, top + 1, bottom - 1);
}
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16

1 2 3 4 8 12 16 15 14 13 9 5 6 7 11 10

1 2 3
4 5 6
7 8 9

1 2 3 6 9 8 7 4 5

Thursday, December 8, 2016

Wednesday, December 7, 2016

We were discussing System Center Managers such as from AWS. It's a cloud service that helps you automatically collect software inventory, apply OS patches, create system images, and configure windows and Linux operating systems. It enables to seamlessly bridge the existing infrastructure with AWS. In other words, it can help manage both EC2 and on-premise resources. It works with the help of an agent installed on each of the computing resource and an iam role with system manager permissions that give it visibility in the AWS EC2 console. Even on premise instances now appear in the EC2 console with the 'mi-' prefix. This is a huge win for private cloud providers because they no longer need legacy, custom or proprietary server management software.
We were looking at remote administration techniques and saw an example by specifying the RunCommand.
A specific advantage of using AWS System manager is Inventory Management. It collects metadata from all EC2 instances, on-premise servers and virtual machines giving a holistic view of all instances from outside in. And since the metadata includes OS and application data, we can gather which instances are running the software configurations required by the software policy of the organization We can even update the SSM agent using the Amazon EC2 RunCommand.
Thus it provides a consistent and secure way to remotely manage on-premises and cloud workloads
It centralizes access control for actions that can be performed on the servers and VMs using AWS Identity and access management.
It centralizes auditing and viewing because it records events with CloudTrail.
By the same token, it helps with centralized monitoring because we can configure CloudWatch Events with Amazon SNS for notifications.

Find the smallest substring length in a string that has all the characters from another string
Solution:
build a frequency table of the second string.
scan the first string for the first occurrence of a match in the frequency table. this is the start of the window.
increase the window size until all characters in the frequency table are accounted for and note the length.
if the first character of the window is encountered more than its count, move the window start to the next match and update the minimum length of the window found so far. proceed till the end of the string and return the minimum found. This
is based on the idea that the minimum window containing the second substring is contiguous in the original string and each match is the potential start of a new run.

        static int GetMinSubStringLen(String A, String pattern)
        {
            if (String.IsNullOrEmpty(A) || String.IsNullOrEmpty(pattern)) return int.MaxValue;
            var hash = new Dictionary<char, int>();
            var current = new Dictionary<char, int>();
            for (int i = 0; i < pattern.Length; i++)
            {
                if (hash.ContainsKey(pattern[i]))
                    hash[pattern[i]]++;
                else
                    hash.Add(pattern[i], 1);
                if (current.ContainsKey(pattern[i]))
                    current[pattern[i]]++;
                else
                    current.Add(pattern[i], 1);
            }
            int start = 0;
            int end = -1;
            int min = int.MaxValue;
            if (current.Count == 0) return 0;
            for (int i = 0; i < A.Length; i++)
            {
                if (hash.Keys.Contains(A[i]))
                {
                    if (end < start)
                    {
                        start = i;
                        end = start;
                    }
                    if (start < i)
                    {
                        end = i;
                        if (A[start] == A[i])
                        {
                            if (end - start + 1 < min)
                                min = end - start + 1;
                            current[A[start]]++;
                            start++;
                            while (start <= i && hash.Keys.Contains(A[start]) == false)
                                start++;
                        }
                    }
                    current[A[i]]--;
                    if (current.Values.All(x => x == 0))
                    {
                        if (end - start + 1 < min)
                            min = end - start + 1;
                    }
                }
            }
            return min;
        }
            Console.WriteLine("Expected=int.MaxValue, Actual={0}", GetMinSubStringLen("", ""));
            Console.WriteLine("Expected=int.MaxValue, Actual={0}", GetMinSubStringLen("", "A"));
            Console.WriteLine("Expected=int.MaxValue, Actual={0}", GetMinSubStringLen("A", ""));
            Console.WriteLine("Expected=int.MaxValue, Actual={0}", GetMinSubStringLen(null, ""));
            Console.WriteLine("Expected=int.MaxValue, Actual={0}", GetMinSubStringLen("", null));
            Console.WriteLine("Expected=1, Actual={0}", GetMinSubStringLen("AA", "A"));
            Console.WriteLine("Expected=1, Actual={0}", GetMinSubStringLen("AB", "A"));
            Console.WriteLine("Expected=1, Actual={0}", GetMinSubStringLen("BA", "A"));
            Console.WriteLine("Expected=1, Actual={0}", GetMinSubStringLen("BAB", "A"));
            Console.WriteLine("Expected=1, Actual={0}", GetMinSubStringLen("BBA", "A"));
            Console.WriteLine("Expected=1, Actual={0}", GetMinSubStringLen("ABB", "A"));
            Console.WriteLine("Expected=2, Actual={0}", GetMinSubStringLen("AABAA", "AA"));
            Console.WriteLine("Expected=2, Actual={0}", GetMinSubStringLen("ABAA", "AA"));
            Console.WriteLine("Expected=2, Actual={0}", GetMinSubStringLen("AABA", "AA"));
            Console.WriteLine("Expected=2, Actual={0}", GetMinSubStringLen("AABAA", "BA"));
            Console.WriteLine("Expected=4, Actual={0}", GetMinSubStringLen("AABAAD", "BD"));
            Console.WriteLine("Expected=3, Actual={0}", GetMinSubStringLen("AABCAD", "CD"));
            Console.WriteLine("Expected=4, Actual={0}", GetMinSubStringLen("AZBACD", "ZC"));
            //Expected=int.MaxValue, Actual=2147483647
            //Expected=int.MaxValue, Actual=2147483647
            //Expected=int.MaxValue, Actual=2147483647
            //Expected=int.MaxValue, Actual=2147483647
            //Expected=int.MaxValue, Actual=2147483647
            //Expected=1, Actual=1
            //Expected=1, Actual=1
            //Expected=1, Actual=1
            //Expected=1, Actual=1
            //Expected=1, Actual=1
            //Expected=1, Actual=1
            //Expected=2, Actual=2
            //Expected=2, Actual=2
            //Expected=2, Actual=2
            //Expected=2, Actual=2
            //Expected=4, Actual=4
            //Expected=3, Actual=3
            //Expected=4, Actual=4
# the above code is O(N) time complexity

Tuesday, December 6, 2016

We were discussing System Center Managers such as from Azure. We look at AWS System center manager next. It's a service that helps you automatically collect software inventory, apply OS patches, create system images, and configure windows and Linux operating systems. It enables to seamlessly bridge the existing infrastructure with AWS. In other words, it can help manage both EC2 and on-premise resources.
The design is similar to Azure SCOM. EC2 systems manager uses a light-weight agent installed on the EC2 instances and on-premise servers that communicate securely with the Systems Manager service and executes management tasks. This helps you manage resources for Windows and Linux regardless of whether they are running VMWare ESXi, Microsoft Hyper-V and other platforms. The Systems Manager service is a cloud service.
EC2 Systems Manager exposes an interface that lets us define management tasks and then select a set of resources to manage. Tasks can be configured to run automatically based either on the results of the software inventory collection or events registered with Amazon CloudWatch events. Amazon EC2 RunCommand is one such interface. It can be used from the EC2 console, the AWS command line interface, windows Powershell, or the AWS SDKs. As an example of using it from the Windows powershell, we can specify the following:
Set-AWSCredentials –AccessKey key_name –SecretKey key_name
Set-DefaultAWSRegion -Region us-east-1
To List all available documents:
Get-SSMDocumentList
Get-SSMDocumentDescription -Name "AWS-RunPowerShellScript"
$runPSCommand=Send-SSMCommand -InstanceId @('Instance-ID', 'Instance-ID') -DocumentName AWS-RunPowerShellScript -Comment 'Demo AWS-RunPowerShellScript with two instances' -Parameter @{'commands'=@('dir C:\Users', 'dir C:\')}
Most of the commands that you can run with RunCommand are available as documents. For example, if we want to run a command to remote install python, we can use the AWS-InstallApplication document and pipe the output to a local file.
For Linux machines, we can install the SSM agent:
curl https://amazon-ssm-region.s3.amazonaws.com/latest/linux_amd64/amazon-ssm-agent.rpm -o /tmp/ssm/amazon-ssm-agent.rpm
Run the installer and start the service
sudo yum install -y /tmp/ssm/amazon-ssm-agent.rpm
sudo systemctl start amazon-ssm-agent

We can also RunCommand using the AWS console as well as the AWS CLI.
RunCommand is an example of remote administration for on-premise and AWS EC2 instances.
Other tasks facilitated by the System Center Manager include Inventory Management, State Management, Maintenance and deployment automation.

The Inventory Manager automates the collection of software inventory from managed instances.
The State Manager automates the process of keeping managed instances in a predefined state.
Automation automates common maintenance and deployment tasks.

#codingexercise
Given four key strokes : letter 'A', Ctrl-A, Ctrl-C, Ctrl-V, print the maximum number of A in N strokes:

static int getCount(int N, int copy, int current)

{

if(N <=0)

return 0;

if (N <= 6)

return N;

int count0 = current+getCount(N-3, current, current * 2); // ctrl A+C+V

int count1 =  copy * 2 + getCount(N-2, copy, current + copy * 2); // ctrl V + ctrl V

int count2 = copy + getCount(N-1, copy, current+copy); // ctrl V

int count3 = 1 + getCount( N-1, copy, current+1);

var counts = new List<int>(){ count0, count1, count2, count3 };

int max = counts.Max();

return current + max;

}

Monday, December 5, 2016

System Center Manager for Azure and AWS

Introduction: Enterprise on-premise server management was a well-known routine that was arguably facilitated very well by System Center Operations Manager. It involved monitoring an asset using several sources on that computer and providing a central console to view this information across assets. It enabled consolidation of tools while bringing a single console to monitor services and assets from simple to complex. With the move to cloud the assets exploded in size and regions including and not limited to a variety of servers, platforms, hardware and vendors. We discuss how this was overcome technically in brief.

Description: The Systems Center Management Pack for Windows Azure lets us now instrument Azure assets for availability, performance monitoring and reporting using SCOM. The Azure Fabric management pack discovers PaaS and IaaS components from Azure subscriptions and communicates directly with an Azure web service to deliver cloud based management data to SCOM.

At the heart of this technology is the notion of a cloud web service and agents on the inventory. The SCOM agent is already fluent in gathering data from a variety of sources and this was made easier by the operating system features that remained consistent if not better across versions. With the explosion of data sources across assets, a central cloud services scales up to meet the challenge. The SCOM Console continues to provide a holistic picture while providing the detailed drill down that it used to for every single asset.

However, organizations were known to have a heterogeneous environment consisting of a mix of Windows and flavors of Linux. Having multiple monitoring and alerting technologies on different operating system flavors was traditionally a maintenance challenge but limiting the ability to scale to a public cloud made it more so. The SCOM agent from windows server allowed monitoring Unix/Linux flavors in existing infrastructure with just a few more steps:

1) Create accounts to be used for monitoring and agent maintenance.

2) Add the created accounts to the appropriate UNIX/Linux profiles

This was facilitated by Management packs for Unix/Linux from Microsoft as shown here and with guidance as published here

With the move to Cloud, the SCOM agents for these OS Flavors were now deployed to Azure VM instances and with the discovery of these instances, the Azure Fabric management pack would facilitate the inside and outside perspectives of the cloud hosted VMs. If an application went down, the failure could be quickly narrowed down to the VM OS, the Azure Service or the Azure storage.

However the same could not be said for multiple public clouds such as Azure and AWS. With the availability of Amazon EC2 Systems Manager that has changed too and now we can automatically collect software inventory, OS patches, system images, and configure both Windows and Linux Operating systems.

With new era convention of facilitating programmatic access to all the applications and services, there are now APIs to talk to either cloud systems managers. Thus organizations can choose leverage and promote a consistent terminology for both these clouds.

Conclusion : Systems Center Managers are available at cloud level doing away with proprietary and custom software both in house or purchased for managing the IT assets of the organization.