Wednesday, July 17, 2013

A quick look at some of the web technologies
web crawling : start with a list of URLs, visit them according to policy, find more and add them to the list.

Tuesday, July 16, 2013

Memory Manager

The memory manager components include a set of executive system services for allocating, deallocating, and managing virtual memory, a translation-not-valid and access fault trapper for hardware exceptions, and a set of kernel mode system thread routines as listed below:
The working set manager to handle global policies such as trimming, aging, and modified page writing.
The process/stack swapper that handles inswapping and outswapping.
The modified page writer that writes dirty pages in mapped files to disk.
The dereference segment thread which is responsible for system cache and page file growth shrinkage.
The zero page thread that zeros out pages on the free list/
The memory manager provides reserving and committing pages, locking memory, allocation granularity, shared memory and mapped files, protecting memory, copy on write, heap functions and address windowing extensions. A look at these in brief:
reserving and committing: reserved pages are not mapped to any storage. This is useful for large contiguous buffer. For example, when a thread is created, a stack is reserved. The committed pages are pages that when accessed, ultimately translate to valid pages in physical memory. Committed pages are either private and not share-able or mapped to a view of the section.
locking memory: Pages can be locked in memory in two ways:
1) Many pages can be requested by say a device driver and locked until it releases.
2) User mode applications can lock pages in their process working set.
Allocation Granularity This defines the system page size so that the risks associated with the assumptions about allocation alignment would be reduced.
Shared memory and mapped files Each process maintains its private memory area in which to store private data, but the program instruction and unmodified data pages could be shared. Shared memory is implemented via section objects
Memory is protected first by accessing all systemwide data structures and pools using kernel mode system components which user threads can't access.
Second, each process has a separate private address space which others can't access.
Third, in addition ti implicit protection, some form of hardware supported protection is also done.
And finally, shared memory section objects are protected via the standard access-control lists.
Copy on write page protection is an optimization the memory manager uses to conserve physical memory. Each new process that writes to a page will also get its own private copy.
A heap is a region of one or more pages of reserved address space that can be subdivided and allocated in smaller chunks by the heap manager.
Address windowing extensions is done by allocating the physical memory to be used, creating a region of virtual address space to act as a window to map views and mapping views of the physical memory into the window.
AWS implements REST API security with apikey and encrypting parameters.

Monday, July 15, 2013

WCF and WPF Fundamentals review

This post mentions the important contracts for each aspect of the communication
1. Service has three main parameters : Address Binding and Contract. Bindings can be of three different types: TCP/IP binding, http binding, net msmq binding.
2. MEX endpoint for metadata exchange. API Versioning using a standard workflow resulting in different actions taken.
3. Hosting can be of three types : IIS, Windows service, and Windows activation service. T
4. Use attributes such as ServiceContractAttribute, DataContractAttribute, and OperationsContractAttribute. Define exception handling via Fault attributes.
5. Use appcmd.exe to determine which instance is hosting your code.
6. Use appcmd.exe to configure site, application, virtual directory or URL.
7. Use TransactionScope for reliability.
8. Use MSMQ for queued calls. Use posion messages and dead letter queues to handle retrying.
9. Security can be none, transport or message.
10. Role based security features of .Net can be reused for service
11. Use PrincipalPermission and UseWindowsGroups for windows role based security.
12. Use certificates for encryption. Certificates have to be in the right store and not have expired.

ASP.Net page life cycle events
The following are the page events during life cycle
1. PreInit : check the IsPostBack
2. Init : raised after all the controls have been initialized.
3. InitComplete : signals end of completion
4. PreLoad : raised after the page loads view state for itself and all controls.
5. Load : The page object calls the OnLoad Method and for each child control
6. Control events: to handle individual controls
7. LoadComplete : Raised at the end of the event handling state.
8. PreRender : Raised after the page object has created all the controls
9. PreRenderComplete : Data binding occurs here
10. SaveStateComplete :  Raised after view state and control state have been saved for the page and for all controls
11. Render: the Page object calls this method on each control.
12. Unload : Raised for each control and then for the page. In controls, use this event to do final cleanup for specific controls.

Reviewed from MSDN

Creating automated load test run report reminders

Load Test Run results can be populated in a database. Scripts to set up this database are available here.
In order to send automated reports on new results, we can register a trigger and an xslt transformation like so:

USE PerfResults;
GO
IF OBJECT_ID ('dbo.LoadTestRunCompleted','TR') IS NOT NULL
    DROP TRIGGER dbo.LoadTestRunCompleted;
GO
CREATE TRIGGER dbo.LoadTestRunCompleted
ON LoadTestRun
AFTER UPDATE
AS
DECLARE @RunID uniqueidentifier;
SELECT TOP 1 @RunID = RunID from LoadTestRun where Outcome = 'Completed' order by EndTime desc;
DECLARE @msg nvarchar(max);
SELECT @msg = dbo.fn_DisplayLoadTestReport(@RunID);
EXEC msdb.dbo.sp_send_dbmail
    @profile_name = 'Performance Reports Administrator',
    @recipients = 'user@xyz.com',
    @body_format='HTML',
    @body = @msg,
    @subject = 'Automated Performance Reports' ;
GO


-- ================================================
-- Template generated from Template Explorer using:
-- Create Scalar Function fn_DisplayLoadReport.SQL
--
-- Use the Specify Values for Template Parameters
-- command (Ctrl-Shift-M) to fill in the parameter
-- values below.
--
-- This block of comments will not be included in
-- the definition of the function.
-- ================================================
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
-- =============================================
-- Author: Ravishankar Rajamani
-- Create date: 15th July, 2013
-- Description: function to render an html for summary display
-- =============================================
CREATE FUNCTION dbo.fn_DisplayLoadTestReport
(
-- Add the parameters for the function here
@RunId uniqueidentifier
)
RETURNS nvarchar(MAX)
AS
BEGIN

    DECLARE @name nvarchar(255)
DECLARE @description nvarchar(255)
DECLARE @starttime datetime
DECLARE @endtime datetime
DECLARE @warmuptime int
DECLARE @runduration int
DECLARE @controller nvarchar(255)
DECLARE @runsettings nvarchar(255)

SELECT @name = LoadTestName, @description = Description, @starttime = StartTime, @endtime = EndTime, @warmuptime = (WarmupTime/60) , @runduration = (RunDuration/60), @controller = ControllerName, @runsettings = RunSettingUsed from LoadTestRun where RunId = @RunId

DECLARE @ret nvarchar(max)
SET @ret = N'<head>
<style type="text/css">
#fullwidth
{
width:100%;
clear:both
}
#leftcol
{
width:49%;
float:left;
clear:all;
position:relative;
}
#rightcol
{
width:49%;
float:right;
position:relative;
}
.pagetitle
{
font-family: Tahoma, Verdana, Arial, Helvetica, sans-serif;
font-size: .8em;
font-weight: bold;
}
.bodytext
{
font-family: Tahoma, Verdana, Arial, Helvetica, sans-serif;
font-size: .7em;
font-weight: normal;
}
.sectiontitle
{
font-family: Tahoma, Verdana, Arial, Helvetica, sans-serif;
font-size: .7em;
font-weight: bold;
text-align: left;
padding-top: 1.5em;
}
.content table
{
width: 100%;
border-top: 1px solid #D4D0C8;
border-right: 1px solid #D4D0C8;
}
.content td
{
font-family: Tahoma, Verdana, Arial, Helvetica, sans-serif;
font-size: .7em;
text-align: left;
border-bottom: 1px solid #D4D0C8;
border-left: 1px solid #D4D0C8;
padding-top: .2em;
padding-left: .3em;
}
.content th
{
font-family: Tahoma, Verdana, Arial, Helvetica, sans-serif;
font-size: .7em;
font-weight: normal;
text-align: left;
border-bottom: 1px solid #D4D0C8;
border-left: 1px solid #D4D0C8;
background-color: #ECE9D8;
padding-top: .2em;
padding-left: .4em;
}
</style>
<script language="JavaScript">
function clickHandler()
{
var targetId, srcElement, targetElement;
srcElement = window.event.srcElement;
if (srcElement.className == "Expandable")
{
targetId = srcElement.id + "div";
targetElement = document.all(targetId);
if (targetElement.style.display == "none")
{
targetElement.style.display = "";
srcElement.innerText = "6";
} else {
targetElement.style.display = "none";
srcElement.innerText = "4";
}}}
document.onclick = clickHandler
</script>
</head>
<body bgcolor="#ffffff">
<div class="pagetitle">Load Test Summary</div>
<div id="leftcol">
<div class="sectiontitle">Test Run Information</div>
<div class="content">
<table cellspacing="0" id="TestRunInformation">
<tr>
<td nowrap width="50%">Load test name</td>
<td nowrap width="50%">' + @name + '</td>
</tr>
<tr>
<td nowrap width="50%">Description</td>
<td nowrap width="50%">' + @description + '&nbsp;</td>
</tr>
<tr>
<td nowrap width="50%">Start time</td>
<td nowrap width="50%">' + CONVERT(nvarchar, @starttime) + '</td>
</tr>
<tr>
<td nowrap width="50%">End time</td>
<td nowrap width="50%">' + CONVERT(nvarchar, @starttime) + '</td>
</tr>
<tr>
<td nowrap width="50%">Warm-up duration</td>
<td nowrap width="50%">' + CONVERT(nvarchar, @warmuptime) + '</td>
</tr>
<tr>
<td nowrap width="50%">Duration</td>
<td nowrap width="50%">' + CONVERT(nvarchar, @runduration) + '</td>
</tr>
<tr>
<td nowrap width="50%">Controller</td>
<td nowrap width="50%">' + @controller + '</td>
</tr>
<tr>
<td nowrap width="50%">Run settings used</td>
<td nowrap width="50%">' + @runsettings + '</td>
</tr>
</table>
</div>
</div>
';

RETURN @ret;


END
GO

Remember to set the content-type of the message as
Content-Type: multipart/mixed
so that it can be displayed  as desired.

Sunday, July 14, 2013

Crash course in file systems

File System Fundamentals :
File attributes include name, type, location, size, protection, and time, date and user identification.
Operations supported are creating a file, writing a file, reading a file, repositioning within a file, deleting a file, and  truncating a file.
Data structures include the following
1) two levels of internal tables : there is a per process table of all the files that each process has opened . This points to the location inside a file where data is to be read or written. This table is arranged by the file handles and has the name, permissions, access dates and pointer to disk block. The other table is a system wide table with open count, file pointer, and disk location of the file.
Sections of the file can be locked for multi-process access and even to map sections of a file on virtual memory systems. The latter is called memory mapping and it enables multiple processes to share the data. Each sharing process' virtual memory map points to the same page of physical memory - the page that holds a copy of the disk block.
File Structure is dependent on the file types.  Internal file structure is operating system dependent. Disk access is done in units of block. Since logical records vary in size, several of them are packed in single physical block as for example at byte size. The logical record size, the physical block size and the packing technique determine how many logical records are in each physical block. There are three major methods of allocation methods : contiguous, linked and indexed. Internal fragmentation is a common occurrence from the wasted bytes in block size.
Access methods are either sequential or direct. The block number is relative to the beginning of the file. The use of relative block number helps the program to determine where the file should be placed  and helps to prevent the users from accessing portions of the file system that may not be part of his file.
File system is broken into partitions. Each disk on the system contains at least one partition. Partitions are like separate devices or virtual disks. Each partition contains information about files within it and is referred to as the directory structure. The directory can be viewed as a symbol table that translates file names into their directory entries. Directories are represented as a tree structure. Each user has a current directory. A tree structure prohibits the sharing of a files or directories. An acyclic graph allows directories to have shared sub-directories and files. Sharing means there's one actual file and changes made by one user are visible to the other. Shared files can be implemented via a symbolic link which is resolved via the path name. Garbage collection may be necessary to avoid cycles.
Protection involves access lists and groups. Consistency is maintained via open and close operation wrapping.
File system is layered in the following manner:
1) application programs, 2) logical file system, 3) file-organization module, 4) basic file system, 4) i/o control  and 5) devices. The last layer is the hardware. The i/o control is the consists of device drivers and interrupt handlers, the basic file system issues generic commands to the appropriate device driver. The file organization module knows about files and their logical blocks. The logical file system uses the directory structure to inform the file organization module. The application program is responsible for creating and deleting files. 
Syllabus :
Algorithms and Data Structures
Programming questions
Design patterns
.Net Technology Stack
ASP.Net
WCF
WPF
SQL Server
Windows operating system