Monday, January 4, 2016

Today we look at a few summation formulas and properties.
Given a sequence a1, a2, ... of numbers, the finite sum a1 + a2 + ... +an where n is a positive number
This sum can be written as n(n+1)/2 when the terms are consecutive n numbers and its terms can be added in any order. This is called the arithmetic series and has an order of n^2. If the series is infinite and a limit exists then it is said to converge otherwise it is said to diverge. We can rearrange the terms of an absolutely convergent series.
For any real number c and any finite sequences a1, a2 ... am and b1, b2 ... bn
Sum  c.ak + bk where k can range from 1 to n is equivalent to writing
c.Sum ak + Sum bk and this is said to be the linearity property.
The linearity property is also obeyed by infinite convergent series.
Summation of squares and cubes also have well defined sums.
For example Sum of squares of consecutive 1 to n numbers is n(n+1)(2n+1)/6
and the sum of cubes of consecutive 1 to n numbers is n^2(n+1)^2 / 4
A series is said to be geometric or exponential when the terms are increasing in power such as 1 + x + x^2 + x^3 ... +x^n and x is not equal to  1.
The sum of this series has the value (x^(n+1)  - 1)/(x-1)
When the summation is infinite and the absolute value of x is less than 1, then we have the sum as 1 / (1-x)
A Harmonic series is one where the nth harmonic number is the sum of the consecutive fractions 1 + 1/2 + 1/3 + ... + 1/n which we calculate as ln n + a constant
Additional formulas can be obtained by integrating and differentiating the series above.
Telescopic series is one where for any sequence a0, a1, ..., an,
Sum for k = 1 to n (ak - ak-1) = an - a0
since each of the terms a1, a2, ..., an-1 is added in  exactly once and subtracted out exactly once. Hence the term telescopes.
#codingexericse 
Given a representation of all distinct digits in an alien system, Generate an increasing sequence from 1 to 9 
For example  given 01 and 0F8 as the alien number system 
0 = 0           0           F, 8, F0, FF, F8, 80, 8F, 88, F00, F0F 
1  = 1          F 
2  = 10        8 
3  = 11        F0 
4 = 100        FF 
5 = 101       F8 
6 =  110      80 
7  = 111      8F 
8 =  1000      88 
9 =  1001      F00 
10 = 1010    F0F 
                     F08 
                     FF0 
                     FFF 
                     FF8 
                     F80 
                     F8F 
                     F88 
                     888 
                   F000 
void GenSequence(String pattern, int number) 
{ 
char[] array = pattern.ToCharArray();  
int len = array.Length; 
String prefix = string.empty; 
while (number) 
{ 
   int rem = number%len; 
   prefix += array[rem]; 
   number = number / len; 
} 
Console.WriteLine(prefix.reverse()); 
}

Sunday, January 3, 2016

Today we review the codejam question to determine the number of intersection point between lines connecting floors of two buildings.
In this case the connectors could connect floors at the same level or those above or those below on the opposite side. There is only one connector on each of the floors. The connectors intersect only when there is a  span across other connectors.
To determine the number of intersection points, we only need to look at the connectors that span others. This happens only in two cases - when the connectors connect floors above or when the connectors connect floors below. In each of those cases, if there are other connectors in their way, their endpoints would be within the range from the start to the end of the current connector. We can choose either the starting endpoints or the finishing endpoints and they could be different for the two cases. In both cases we will have to exclude one of the endpoints to determine the number of intersections. Intuitively, we are articulating that the intersections arise from connectors that lie within the start and the end of the chosen connector. If there are no intersections, the lines are parallel even if they are not horizontal and their start and end will not have the beginning or end of any other connectors within their start and end.

Now back to AWS Billing:
we were discussing how to archive the metrics

CREATE TABLE [dbo].[Metric]
(
[ID] INT NOT NULL PRIMARY KEY IDENTITY,
    [Label] NVARCHAR(50) NOT NULL DEFAULT 'UNKNOWN',
    [Created] DATETIME NOT NULL DEFAULT getdate(),
    [Value] REAL NOT NULL DEFAULT 0.00,
    [Type] NVARCHAR(50) NOT NULL DEFAULT 'Gauge',
    [Units] NVARCHAR(50) NOT NULL DEFAULT 'NA',
    [Service] NVARCHAR(50) NULL,
    [Region] NVARCHAR(50) NULL,
    [InstanceID] INT NULL,
    [ClusterID] INT NULL,
    [ProjectID] INT NULL,
    [Created By] NVARCHAR(50) NULL
)

GO

CREATE INDEX [IX_Metric_Created] ON [dbo].[Metric] ([Created])

GO

DECLARE @ID INT;
while EXISTS(SELECT * FROM Metric WHERE Created < '2014-01-01')
BEGIN
SELECT @ID=ID FROM Metric WHERE Created < '2014-01-01'
INSERT INTO Archive (Label, Created, Value, Type, Units, Service, Region, InstanceID, ClusterID, ProjectID, [Created By])
SELECT Label, Created, Value, Type, Units, Service, Region, InstanceID, ClusterID, ProjectID, [Created By] from Metric WHERE ID = @ID;
DELETE FROM Metric WHERE ID=@ID;
SELECT @ID=ID FROM Metric WHERE Created < '2014-01-01';
END

How do we calculate the CPU instance hours from AWS metrics?
Its true that AWS Metrics use the term CPUCredit which does not immediately tell us how many vcpu we are talking about. But this jargon is much more informational, true and accurate as we will see. In fact, they have a concept of CPUCreditUsage and CPUCreditBalance. So what is a CPU credit ? One CPU Credit is equal to one vCPU running at 100% utilization for one minute. Other combinations of vCPUs, utilization and time are also equal to one CPU credit for example one vCPU running at 50% utilization for two minutes etc. We can translate minutes to hours as well. In this case we can have 1/30 CPU credits for an hour which is equivalent to 1 CPU Credit
We say it is informational because it is not merely based on the static allocation of the number of cpus and instead based on the usage and balance of the utilization. This leads to the question how are CPU Credits earned ? Each instance for example a T2 instance  starts with a healthy initial CPU credit balance and then continuously receives a set rate of CPU credits per hour. This accounting is done at a millisecond level resolution and it ensures a maximum earned CPU credit balance for example 144 CPU credits  for a t2.micro. Thus these metrics are much more true and accurate. There are two metrics associated with CPU Credits - one is CPUCreditUsage and another is CPUCreditBalance. The CPUCreditUsage metric indicates the number of CPU credits used during the measurement period The CPUCreditBalance metric indicates the number of unused CPU Credits a T2 instance has earned. The CPUCreditUsage identifies the amount of time during which the physical CPUs were used for processing instructions by virtual CPUs allocated to the instance. Therefore this metric can directly be used to generate the bill. Moreover the CPU Utilization metric is aggregated for the number of cpus that instance may have. A t2.medium instance has two vCPUs and if one of them is at 100% and the other is rarely used, the CPU Utilization metric will show 50%.


Saturday, January 2, 2016

Today we continue discussing the AWS Billing.
I referred to CloudWatch monitoring and SNS notifications as one way to populate a historical table of usage metrics. But there are some disadvantages to this
1) It requires elaborate setup and incurs further costs
2) As a cloud provider, this requires a lot of overhead for many hundreds of customers
3) Most of the usage is still under the free tier where the benefits are not that much.
On the other hand we can poll for the data that we are most interested in such as the number of instances and their image types. Furthermore, we can switch on CloudWatch for only those accounts where the usage is significantly higher than the free tier.
Therefore we wrote some sample scripts to illustrate :
1) how to get the instance information
2) how to poll for the information using cron jobs
3) how the datapoints from the usages look like
4) how to design a table to store such data
5) how to perform aggregation over this data

These scripts are shared at : https://github.com/ravibeta/PHPSamples

Note that both the poller and the SNS notifications are essentially providing data
1) for the meters that have been configured
2) the data can be accumulated for longer than the fourteen days max worth that the APIs provide
3) the datapoints for advanced metrics although come processed with filtering and aggregation at the source can be easily computed using other datapoints from essential metrics

Hence the use of a table helps with the history.
As with all historical tables, we may need to archive and trim it. This process can be done by moving one record at a time from the source table to the archive table The moving operation essentially identifies one of the records to be moved based on an archival policy (say older than a timeframe) and inserts into the archival table if it doesn't have it already, and then deletes from the source table when the insertion is successful.

#codejam
we now discuss another puzzle from the codejam which involves load testing.
The problem is described this way:
Let us say a site can support L people at a time without errors. but the site may have P visitors as the peak number of all visitors that the site may encounter.  Since the site cannot support that many visitors, the site has to determine within a factor of C, how many people it can support.  This means there is an integer a  such that you know the site can support a people, but you know the site can't support a * C people.
Note that this C is the target that we will use. This is given with each input of L and P. The goal is to determine the number of load tests you need to run in the worst case before knowing within a factor of C how many people the site can support.
Let us say this number is x.
The strategy we use is that if we know a then a*C is the maximum we can support, if P is greater than a*C we need to do more load tests for refining what a and a*C can be. Each refinement or iteration increments x by 1.
Logarithmic or exponential scaling may not always work. a 10x strategy might also not be sufficient. Depending on L and P and how large they can get we need to determine a suitable strategy.
Choosing the strategy depends on the range given by L and P. We want to scale L to P in the minimum number of steps. whichever strategy gives us this minimum number  of steps we take.
We essentially have two categories of problems which determine whether we need a strategy that takes big steps or small steps.
L and P are small
L and P are large.
We observe that since they are known along with C each load test can pick arbitrary number of machines to bound a and a*C, we can use the worst case always as the one where a*C fails the test.
if a*C passes the test then we can alternate with where a*C fails to find x - the number of iterations. The outcome of a load test is binary.

Now we take some examples to see how this works. We will directly use the examples given:
Input
  
Output
  
 total number of cases : 4
 L         P       C
 50    700       2      Case 1
 19      57       3      Case 2
 1    1000       2      Case 3
 24      97       2      Case 4

In case #2, we already know the site can support between 19 and 57 people. Since those are a factor of three apart we don't need to do any testing. Hence x is 0
In case #4, we can test 48 since it is a factor of 2 but in order to test 97 assuming 48 can be tested we will need another test and that will still be less than 97. The worst case is when 49 cannot be supported. 49 then to reliably say 97 can be supported we will need another test.
There is a lower bound and an upper bound of a window where we can say that a certain number (a) can be supported and a certain number (a*C) cannot be supported.

#context switch and revert back to aws billing
The CloudWatch monitoring subscriber can be hosted on an AWS machine because it will automatically provide dog fooding of the subscription. Let us say we host a web server that receives the JSON post callbacks from the subscriber. This subscriber can parse all the incoming messages to fill the metrics table while doing so increases the utilization of the instance which is also being watched, therefore adding validity to the machine. Furthermore, the web server is required to have a Internet addressable endpoint and this is facilitated by AWS instances.


 

Friday, January 1, 2016

yesterday we were discussing a problem where we had to find the number of passwords possible from observed M fingerprints and the password being N letters long.
We stated that when
M == N then the number of permutations is N!
If N > M then there are repetitions of some of the letters and we broke up the case as
               a letter appearing twice
               a letter appearing thrice
               :
               a letter appearing N-M+1 times
              and the above repeated for each of the M letters
The number of permutations in each case can be determined by dividing the permutations by the factorial of  the number of objects that are identical.
Let us take a few examples
For M=3 N=5 we have  M * (N!/(2!2!) + N!/3!)
For M=3 N=6 we have  M * (N!/(2!2!2!) +  N!/(3!2!1!) + N!/(4!1!1!)) where 4 = N-M+1
Note the denominators are such that the repetitions together with single occurrences add up to N
Finding the different denominators is easy because we are trying to find the sum s with at most p components where s = N and p = N-M
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace SumToN
{
    class Program
    {
        static void Main(string[] args)
        {
            var sequence = new List<int>();
            GenSeq(6, 3, ref sequence);
            PrintSequence(sequence);
        }
        static void PrintSequence(List<int> sequence)
        {
            Console.WriteLine();
            for(int i = 0; i < sequence.Count; i++) {Console.Write(sequence[i]);}
            Console.WriteLine();
            return;
        }
        static void GenSeq(int s, int p, ref List<int> sequence)
        {
            if (s == 0) {if(p==0)PrintSequence(sequence); return;}
            if (s != 0 && p == 0) { return; }
            for (int i = 0; i <= s; i++)
            {
                sequence.Add(i);
                GenSeq(s - i, p - 1, ref sequence);
                sequence.RemoveAt(sequence.Count - 1);
            }
        }
    }
}


Thursday, December 31, 2015

An interesting problem solving question.
This questions has been adapted from a codejam session.
Let us take a hacker who wants to identify passwords from fingerprints.
There are M keys with fingerprints and N letters in the password.
The hacker knows both M and N but there are different permutations for the password and the hacker does not know the order in which the fingerprints were made.
Take for instance M = 3 and  N = 4 and the fingerprints on the numbers 3,5 and 7.
This means that the following passwords are possible 3357, 3557 and 3577.
and the corresponding permutations.
The permutations for 3357 are 4! / 2  = 12 in number because two of the letters are identical and so the resulting permutations will be the same.
Since any of the M characters can appear in duplicate to make  a set of 4 in the password, we have 3 * 12 = 36 possible passwords with this configuration.
The permutations can be generated by a sample code below
Void Permute (String a, StringBuilder b, bool[] used) 
{ 
  If ( b.Length == a.Length { print b.ToString(); return;} 
  For (int I = 0; I < a.Length ; i++) 
  { 
    If (used[i]) continue; 
     used[i] = true; 
     B += A[i]; 
     Permute(a, b, used); 
     B [B.Length – 1] = ‘/0’; 
     used[i] = false; 
  } ms
} 
But let us look at how to find out the number of such permutations for different values of M and N
if M = 3 and N = 5, then we know that two letters are repeated or one letter appears thrice.
In general, repetitions are taken care of by dividing the permutation by the factorial of  the number of objects that are identical.
33557  = 5!/(2!2!) = 120/4 = 30
33357  = 5!/3! = 120/3! = 20
In the latter case 5 and 7 can each occupy one of the five positions times the remaining four positions for the other
now the above two cases can be repeated with 5 and 7 as well so we have a total of 3 * (30+20) = 150 cases.
As N increases we have further breakouts of the difference between M and N.
If M = 3 and N = 6, then we can have three letters repeating twice, two letters with one repeating thrice and another repeating twice and one letter or one letter repeating four times.

#billing in AWS
ImageType to vCPU

$imageTypeVCPU = array(
"t2.nano" => 1,
"t2.micro" => 1,
"t2.small" => 1,
"t2.medium" => 2,
"t2.large" => 2,
"m4.large" => 2,
"m4.xlarge" => 4,
"m4.2xlarge" => 8,
"m4.4xlarge" => 16,
"m4.10xlarge" => 40,
"m3.medium" => 1,
"m3.large" => 2,
"m3.xlarge" => 4,
"m3.2xlarge" => 8,
"c4.large" => 2,
"c4.xlarge" => 4,
"c4.2xlarge" => 8,
"c4.4xlarge" => 16,
"c4.8xlarge" => 36,
"c3.large" => 2,
"c3.xlarge" => 4,
"c3.2xlarge" => 8,
"c3.4xlarge" => 16,
"c3.8xlarge" => 32,
"r3.large" => 2,
"r3.xlarge" => 4,
"r3.2xlarge" => 8,
"r3.4xlarge" => 16,
"r3.8xlarge" => 32,
"g2.2xlarge" => 8,
"g2.8xlarge" => 32,
"i2.xlarge" => 4,
"i2.2xlarge" => 8,
"i2.4xlarge" => 16,
"i2.8xlarge" => 32,
"d2.xlarge" => 4,
"d2.2xlarge" => 8,
"d2.4xlarge" => 16,
"d2.8xlarge" => 36,
);

Sample bill generated with : https://github.com/ravibeta/PHPSamples/blob/master/billing.php

Wednesday, December 30, 2015

Today we review another coding question :
We have a grid with R rows and C columns in which every entry is either 0 or 1. We are going to perform N operations on the grid, each of which is one of the following:


  • Operation M: Change a number in one cell of the grid to 0 or 1
  • Operation Q: Determine the number of different connected regions of 1s.
    A connected region of 1s is a subset of cells that are all 1, in which any cell in the region can be reached from any other cell in the region by traveling between cells along edges (not corners).
This is the same as the problem I solved in the earlier post.

we solved it by identifying connected regions where a set of 1s appear together. The only variation here is that if two cells of 1 are adjacent only diagonally and not horizontally or vertically then they are not connected. There we identified the contiguous regions as extending the rightmost corner of the rectangle, here we bound the connected region by exploring top then right then bottom edges. A cell can be said to be in an already identified connected region if we can reach it from another point in the region horizontally or vertically where it shares one of the coordinates.

Now to continue with the AWS billing discussion, we were saying we could even prepare the bill based on allocations not usage. We get information on allocations as follows:
<?php
require 'vendor/autoload.php';
use Aws\CloudWatch\CloudWatchClient;

$key = "";
$secret = "";
$region = "us-west-2";
$version = "latest";
$interval = 15;

// Users/rajamani/Downloads/devserver/bill/vendor/aws/aws-sdk-php/src

// Use the us-west-2 region and latest version of each client.
$sharedConfig = [
    'region'  => $region,
    'version' => $version,
  'credentials' => array(
    'key' => $key,
    'secret'  => $secret,
  ),
    'key'    => $key,
    'secret' => $secret,
];

// Create an SDK class used to share configuration across clients.
$sdk = new Aws\Sdk($sharedConfig);
$client = $sdk->createEc2();
  date_default_timezone_set('America/Los_Angeles');
$result  = $client->describeInstances(array(
));
echo 'RESULT='.serialize($result);

$result = (array)$result;
foreach( $result as $reservations)
{
 echo 'RESER='.serialize($reservations)."\n";
 foreach($reservations['Reservations'] as $reservation)
 {
   echo 'RESERVATION='.serialize($reservation)."\n";
   foreach($reservation["Instances"] as $instance)
   {
        $attributes = $client->describeInstanceAttribute(array(
    'DryRun' => false,
    // InstanceId is required
    'InstanceId' => $instance["InstanceId"],
    // Attribute is required
    'Attribute' => 'instanceType',
));
       echo 'ATTRIBUTES='.serialize($attributes)."\n";
   }
 }

}

?>

Tuesday, December 29, 2015

Openstack billing
Ceilometer measures the usage of resources in terms of disk usage, network usage and CPU usage. For each instance the usage details for following meters will be shown: 
  1. Disk Read Bytes 
  1. Disk Write Bytes 
  1. Disk Write Requests 
  1. Disk Read Requests 
  1. Network outgoing bytes 
  1. Network incoming bytes 
  1. Network outgoing packets 
  1. Network incoming packets 
  1. CPU 
 
Ceilometer arranges all the data in terms of resources and their meters. 
We can get a list of all the meters using  
GET /v2/resources/ and  
GET /v2/resources/(resource_id)  
 Where resource contains links such as a url to identify itself and a url for its meters such as http://localhost:8777/v2/meters/volume?q.field=resource_id&q.value=bd9431c1-8d69-4ad3-803a-8d4a6b89fd36 
  
GET /v2/meters/ that returns all know meters and  
GET /v2/meters/(meter_name) which even come with statistics 
GET /v2/meters/(meter_name)/statistics 
  
Samples and statistics can also be drawn from  
GET /v2/samples 
And GET /v2/            samples/(sample_id) 
 
Tables can be created in the following manner: 
RatePlan
  1. Id 
  1. Name  
  1. CPU cost per second 
  1. Disk cost per MB 
  1. Network cost per MB 
  1. Cost for flavors 

Table for instance-bill will have the following attributes 
  1. Resource_id 
  1. Rateplan_id 
  1. Date and time at which the bill was calculated 
  1. Resource_bill 
Openstack provides three  types of meters:  
Cumulative -  those that increase over time 
Delta -  those that change over time  
and Gauge - those that are discrete items and fluctuating values 
When we pick the metrics for compute and storage, we are typically interested in the first and the last of the above. These will come in helpful to provide information on an instance basis. For example, compute has instance, number of vcpus and memory as gauge and cpu time as cumulative. This theme is common to Openstack as well as Amazon as we have seen.

#codingexercise
Given pairs of numbers which correspond to number of unit-heights on  two vertical lines  and the pair representing starting and ending points for a connecting line, find the number of intersection points in the list of pairs. At every intersection points there are only two connectors. There is only one connector at each starting or ending point

int GetNumberOfIntersections(List<Tuple<int,int>> pairs)
{
int total = 0;
for (int i = 0; i < pairs.Count; i++)
{
if (pairs[i].Item1 > pairs[i].Item2)
{
int start = pairs[i].Item2;
int end = pairs[i].Item1;
int count = 0;
for (int j = 0; j< pairs.Count; j++)
{
if (j == i ) continue;
if (pairs[j].Item2 >= pairs[start] &&
     pairs[j].Item2 < pairs[end])
    count++;
}
total += count;
}
else
{
int start = pairs[i].Item1;
int end = pairs[i].Item2;
int count = 0;
for (int j = 0; j< pairs.Count; j++)
{
if (j == i) continue;
if (pairs[j].Item1 > pairs[start] &&
     pairs[j].Item1 <= pairs[end])
    count++;
}
total += count;
}
return total;
}