Sunday, June 2, 2013

The following requirements from application are ubiquitous and are being enumerated here.
1) Application data: one or more data stores is typically required for an application unless it is an exclusively in-memory application with no data persistence.
2) Application object model This forms the core of the business logic and enables flexibility to add and remove logic via additional inheritance, encapsulation and composition. This helps to write unit-tests and can interact with a  variety of clients for participating in end to end tests.
3) Application Core components: These are a subset of the overall application object model that is common across different components such as for client and server and are typically scoped to the namespace of a feature or functionality of the overall application.
4) Application  Services : This describes the service host creation and instantiation that is required for housing the services of the application.
5)  Application UI : This organizes the view models and the views so that the business objects can be modified to enable the workflows required from the application. The properties on the UI are defined usually by declarative markup in views. 
6) Application Exception handling : The entire application handles exceptions at each layer and across layers. Usually applications don't allow unhandled exceptions to propagate to the user. Messages are translated into what's meaningful to the user.
7) Application Logging : Application logging via standard application blocks or other utilities is necessary both for development as well as production support.
8) Application user access : controlling who has access to what is another important aspect of design for an application. Applications tend to handle security as close to the entry points as possible. The entry points typically have a login that comprises of authentication, authorization lines that need to be crossed before the user has access.
9) Application performance : Application design requires consideration for performance and often include fast code paths in addition to general more expensive ones as well as external tools or frameworks such as caching.
10) Application messaging : Service bus or some messaging framework is involved to send messages between application and its dependencies. These could be other services or  other data providers. These could be external gateway when there are heterogeneous systems involved.
Overall the application source has different projects with well named and fully qualified namespaces for most of the components mentioned earlier. Each project creation and addition to the application source has to follow a set of rules and conventions that enable good organization of the source. This facilitates changes made to be isolated and following SOLID principles.
 
confirming the previous post with dumpchk output

Saturday, June 1, 2013

Applications are fragile. So we make up with testing and incremental releases and use scaffolding and shared reusable components Today it is easy to write applications in a test driven development method. And there's framework support and libraries to write repository, services and  views. As an example. So to implement the stack trace services in the way we discussed earlier, we will require a local database, EF, sample dump, debugger sdk, and  file watcher. Let's do this now.
We're going to read the dump file ourselves and we wil find the stack trace in the dump file ourselves without copying or moving the file or using a debugger. Here's the header of the dump file as given by dumpchk.exe. This header informs the offsets of each field and their size. We also know the layout of the file to contain the header, followed by the runs list which in turn is followed by the runs.  At offset 0x348 we have the context record and at offset 0xf00 we have the exception with the structure _EXCEPTION_RECORD64 which has the exception address at 0x00C offset. The Exception Record has the exception code at 0x0, exception flags at 0x004, pointer to exception record at 0x008, number of parameters at 0x010 and exception information of 16 pointers at 0x014.
The physical memory block buffer gives the mapping between physical addresses and file offsets. The stuctue for physical memory block  descriptor has a dword for number of runs, a dword for number of pages, and an array of run descriptors for the number of runs. The run descriptors each has a dword for base page and a dword for base page count. Runs have pages and each page has 4096 bytes The context gives the register for stack pointer at 0x0c4 offset and the stack pointer is the start of the stack trace.
As an aside, the PFN database array seen below is the memory manager's array to keep track of each page of physical memory (RAM) with around 28 byte per page data structure.
Once we dump the stack pointer, we could try to find what the module addresses are, the function addresses and check to see that each stack entry makes a call to the next.
Filename . . . . . . .memory.dmp
   Signature. . . . . . .PAGE
   ValidDump. . . . . . .DUMP
   MajorVersion . . . . .free system
   MinorVersion . . . . .1057
   DirectoryTableBase . .0x00030000
   PfnDataBase. . . . . .0xffbae000
   PsLoadedModuleList . .0x801463d0
   PsActiveProcessHead. .0x801462c8
   MachineImageType . . .i386
   NumberProcessors . . .1
   BugCheckCode . . . . .0xc000021a
   BugCheckParameter1 . .0xe131d948
   BugCheckParameter2 . .0x00000000
   BugCheckParameter3 . .0x00000000
   BugCheckParameter4 . .0x00000000
The above is for kernel dump. There are other dump files.
   ExceptionCode. . . . .0x80000003
   ExceptionFlags . . . .0x00000001
   ExceptionAddress . . .0x80146e1c

   NumberOfRuns . . . . .0x3
   NumberOfPages. . . . .0x1f5e
   Run #1
     BasePage . . . . . .0x1
     PageCount. . . . . .0x9e
   Run #2
     BasePage . . . . . .0x100
     PageCount. . . . . .0xec0
   Run #3
     BasePage . . . . . .0x1000
     PageCount. . . . . .0x1000

Friday, May 31, 2013

Pixels in a photograph are translated to color codes and the entire image can be considered a matrix. With this matrix and some matrix manipulation software such as Matlab, these can be toyed with identify edges and regions. There are smoothing techniques and normalization we may need to perform on the images, and use algorithms for segmentation and then we can transform the image into something we can work with. We may not be able to eliminate noise but we can do a lot with the transformed image. 
Some examples of such image processing techniques include image enhancement, image restoration and image compression. Image enhancement is the technique by which different features of the image are accentuated such that they can be prepared for further analysis. Increasing contrast, gray level, contrast manipulation, noise reduction, edge detection and sharpening, filtering, interpolation and magnification are all part of enhancing the image. Image restoration is a technique that works the other way in that it tries to reduce the changes to the image by studying the extent and kinds of changes that have happened. Image  compression is about storing the image with a reduced number of bits. This is very helpful when image size is a concern for example in the storage and retrieval of a large number of images such as for broadcasting, teleconferencing, medical images and other transmissions. If you have taken pictures in different formats, you would have noticed the significant improvement in size with JPEG format. The Joint Photography group came up with this format to reduce the image size among other things.
Among the image enhancement techniques, edge detection is commonly used to quickly identify the objects of interest from a still image. This is helpful in tracking the changes to the edges in a series of frames from a moving camera. Such cameras capture over 30 frames per second and the algorithms used for image processing are costly. So we make adjustments with what we want to detect and make interpretations from that.
Another such application is region growing where we decompose the picture into regions of interest. In the seeded region growing method for example, a set of data points are taken as starting points for the objects to be demarcated along with the input image.  The regions are iteratively grown by comparing all unmarked pixels and including them into the regions. The difference between the pixel intensity and the region's mean is used for the measure of similarity. This way the pixels are included in the regions and the regions grow. 
Another interesting technique is called balanced histogram threshold method in which the image foreground and background are hued differently so that we see the outline of the foreground. The entire image is converted to a histogram of intensities of all the pixels. Then the method tries to find the threshold for which the histogram divides into two groups. This method literally balances the histogram to find the threshold value.  It weights which of the two groups is heavier and adjusts the weights iteratively until the histogram balances. This method is appealing for its simplicity but it does not work well with very noisy images because there are outliers that distort finding the threshold.  This we can workaround by ignoring the outliers.
Thus we have seen some interesting applications of image processing. 

cloud storage

When we deploy applications and database to the cloud, we have to pay attention to the size of the data stored. If the data is in the form of files or in the database, they can arbitrarily large. When the data exceeds several Gigabytes and grows at a considerable rate, the local storage in a VM does not suffice. At that point, a dedicated Storage area network is usually the norm because it can provide a capacity much larger than any disks. Typically for production database servers SAN is preferred. But this is true even for file shares. There are storage appliances that provide large secondary and tertiary storage while making them available as a mounted share visible to all Active Directory users. This is different from the Network Access storage that is spread out in the form of VMs. Data in the case of NAS does not reside on a single virtual host and there is management incurred in finding the VM with the data requested.
That means we need to plan rollout of our software with appropriate configurations. The storage media, the local or remoteness of the storage, the escalation path for incident reports, all need to be decided before deployment. This is important not just for planning but for the way the application software is written. For example, data traffic and network chattiness can be reduced if the round trips and redundancies between the application and input was reduced. Redundancy in operations is often ignored or undetected for want of features. For example, data files are copied from one location to another and then to a third location. If this were to be eliminated by either reducing the data to the metadata that we are interested in and/or copying the file only once to the target destination, we avoid redundancy.

Thursday, May 30, 2013

Unit-testing data provider calls requires that the underlying DataContext is substituted with a dummy and then we call it mocked. The trouble with that is there are no fakes or mocks that can be generated for the DataContext type. This is primarily because the DataContext object derives from a base DbContext object. If it were to implement an interface, then it can be mocked or faked.  However, adding an interface that calls out the salient methods for mocking stubs is not usually a one time solution. The DataContext object is itself auto generated. Each time the DataContext object is regenerated, the interface would likely have to be added again or the tests won't pass. This adds a lot of maintenance for otherwise unchanging code. There is a solution with Text Template Transformation toolkit (T4) templates which comes with Entity Data Modeler in EF 4.0. The T4 template creates the interface we desire.
We generate this the same way that we generate our data objects from EF. We right click on the Entity framework model and set the code generation strategy to none, and then we add a new code generation item and select the ADO.Net mocking context generator.
This way we have enabled testing the data provider classes by mocking the context classes underneath. So our testing can stay limited to this layer without polluting the layers beneath or reaching the database.
Testing the data provider class comes in helpful when we want to build the business logic layer on top of it. The business layer can now assume that layers underneath it know how to send the data across.
In the previous example, we consider a server client for reading stack trace from dumps. If this logic were to be implemented, both server and client side could use EF and data provider classes as described above. The interfaces only help to make it more testable at which point this common code can be factored out and included as a core project with the server and client code. The reason for the server and client code to be written separately is because both of them can be developed separately.
As discussed earlier, the server code handles all the data for the population of the tables. The interface for the server is the same as from a powershell client or UI or a file watcher service. By exposing the object that read the stack trace from a dump, directly in powershell we improve the automation.The file watcher service invokes the same interface for each of the files watched. The code could also keep the data local with a local database file that asp.net and EF can understand. This way we can even do away with the consumer side for the first phase and then add it subsequently in the second phase.  At that point the local database can be promoted to a shared database server.
Finally, the processing of the dump files may involve a debugger process to be launched. This means there has to be diagnosability of the process failures and appropriate messages to the invoker. Since the process invoking the debugger might be handling all exceptions and not passing the exception information across the process boundary, failures to read the dump may be hard to diagnose. If the number of such failures are high such that there is a backlog of unprocessed or partly processed dumps, then the overall success rate of the solution is affected. Some simple ways to handle this would be to stream all exceptions to the error output stream and read it from the process invoker.
Some ways to test a service that relies on a file watcher service and service bus. Service bus is merely a message delivery mechanism that guarantees messages will not be dropped and the sender and receiver (typically the client and the server) process these messages asynchronously.
1) Start copying a large file into the watched folder that takes a considerable time to copy.
2) Remove 'Everyone' or public access from the security settings of the watched folder.
3) Copy a file with extensions inclusive of the pattern being watched
4) Copy a file that triggers an exception in the servie such as a file with junk contents different from what is being expected
5) Send a message that causes an exception in the serve. If the server is expecting the path to a newly added file, send a path which denies access
6) Check the number of retries for the message in step 5)
7) For messages that repeatedly fail, check that the message made it to the poison queue.
8) Turn off the server that the messages are being received by and check the expiration of the messages and their move to dead letter queue.
9) Check the database for duplicate entries with same file name as the one in the watched folder
10) Check that the database entry for the same file input is refreshed on each attempt.
11) Check if this happens in a transaction where all the steps from moving a file, to clearing the message queue, to purging the database entry all occur in a rollback.
12) Check that the server processing the messages does not hang if the input doesn't contain what the server is looking for.
13) Check that the security settings on the file, folder, queue access, database and other dependencies are changed and don't cause service failures.
14) Check that queue cannot be flooded with unprocessed messages or denial of service attacks.
15) Check that watched folder does not have a backlog of files.