confirming the previous post with dumpchk output

Applications are fragile. So we make up with testing and incremental releases and use scaffolding and shared reusable components Today it is easy to write applications in a test driven development method. And there's framework support and libraries to write repository, services and  views. As an example. So to implement the stack trace services in the way we discussed earlier, we will require a local database, EF, sample dump, debugger sdk, and  file watcher. Let's do this now.
We're going to read the dump file ourselves and we wil find the stack trace in the dump file ourselves without copying or moving the file or using a debugger. Here's the header of the dump file as given by dumpchk.exe. This header informs the offsets of each field and their size. We also know the layout of the file to contain the header, followed by the runs list which in turn is followed by the runs.  At offset 0x348 we have the context record and at offset 0xf00 we have the exception with the structure _EXCEPTION_RECORD64 which has the exception address at 0x00C offset. The Exception Record has the exception code at 0x0, exception flags at 0x004, pointer to exception record at 0x008, number of parameters at 0x010 and exception information of 16 pointers at 0x014.
The physical memory block buffer gives the mapping between physical addresses and file offsets. The stuctue for physical memory block  descriptor has a dword for number of runs, a dword for number of pages, and an array of run descriptors for the number of runs. The run descriptors each has a dword for base page and a dword for base page count. Runs have pages and each page has 4096 bytes The context gives the register for stack pointer at 0x0c4 offset and the stack pointer is the start of the stack trace.
As an aside, the PFN database array seen below is the memory manager's array to keep track of each page of physical memory (RAM) with around 28 byte per page data structure.
Once we dump the stack pointer, we could try to find what the module addresses are, the function addresses and check to see that each stack entry makes a call to the next.
Filename . . . . . . .memory.dmp
   Signature. . . . . . .PAGE
   ValidDump. . . . . . .DUMP
   MajorVersion . . . . .free system
   MinorVersion . . . . .1057
   DirectoryTableBase . .0x00030000
   PfnDataBase. . . . . .0xffbae000
   PsLoadedModuleList . .0x801463d0
   PsActiveProcessHead. .0x801462c8
   MachineImageType . . .i386
   NumberProcessors . . .1
   BugCheckCode . . . . .0xc000021a
   BugCheckParameter1 . .0xe131d948
   BugCheckParameter2 . .0x00000000
   BugCheckParameter3 . .0x00000000
   BugCheckParameter4 . .0x00000000
The above is for kernel dump. There are other dump files.
   ExceptionCode. . . . .0x80000003
   ExceptionFlags . . . .0x00000001
   ExceptionAddress . . .0x80146e1c

   NumberOfRuns . . . . .0x3
   NumberOfPages. . . . .0x1f5e
   Run #1
     BasePage . . . . . .0x1
     PageCount. . . . . .0x9e
   Run #2
     BasePage . . . . . .0x100
     PageCount. . . . . .0xec0
   Run #3
     BasePage . . . . . .0x1000
     PageCount. . . . . .0x1000

Pixels in a photograph are translated to color codes and the entire image can be considered a matrix. With this matrix and some matrix manipulation software such as Matlab, these can be toyed with identify edges and regions. There are smoothing techniques and normalization we may need to perform on the images, and use algorithms for segmentation and then we can transform the image into something we can work with. We may not be able to eliminate noise but we can do a lot with the transformed image. 
Some examples of such image processing techniques include image enhancement, image restoration and image compression. Image enhancement is the technique by which different features of the image are accentuated such that they can be prepared for further analysis. Increasing contrast, gray level, contrast manipulation, noise reduction, edge detection and sharpening, filtering, interpolation and magnification are all part of enhancing the image. Image restoration is a technique that works the other way in that it tries to reduce the changes to the image by studying the extent and kinds of changes that have happened. Image  compression is about storing the image with a reduced number of bits. This is very helpful when image size is a concern for example in the storage and retrieval of a large number of images such as for broadcasting, teleconferencing, medical images and other transmissions. If you have taken pictures in different formats, you would have noticed the significant improvement in size with JPEG format. The Joint Photography group came up with this format to reduce the image size among other things.
Among the image enhancement techniques, edge detection is commonly used to quickly identify the objects of interest from a still image. This is helpful in tracking the changes to the edges in a series of frames from a moving camera. Such cameras capture over 30 frames per second and the algorithms used for image processing are costly. So we make adjustments with what we want to detect and make interpretations from that.

Another such application is region growing where we decompose the picture into regions of interest. In the seeded region growing method for example, a set of data points are taken as starting points for the objects to be demarcated along with the input image.  The regions are iteratively grown by comparing all unmarked pixels and including them into the regions. The difference between the pixel intensity and the region's mean is used for the measure of similarity. This way the pixels are included in the regions and the regions grow. 

Another interesting technique is called balanced histogram threshold method in which the image foreground and background are hued differently so that we see the outline of the foreground. The entire image is converted to a histogram of intensities of all the pixels. Then the method tries to find the threshold for which the histogram divides into two groups. This method literally balances the histogram to find the threshold value.  It weights which of the two groups is heavier and adjusts the weights iteratively until the histogram balances. This method is appealing for its simplicity but it does not work well with very noisy images because there are outliers that distort finding the threshold.  This we can workaround by ignoring the outliers.
Thus we have seen some interesting applications of image processing. 

cloud storage

When we deploy applications and database to the cloud, we have to pay attention to the size of the data stored. If the data is in the form of files or in the database, they can arbitrarily large. When the data exceeds several Gigabytes and grows at a considerable rate, the local storage in a VM does not suffice. At that point, a dedicated Storage area network is usually the norm because it can provide a capacity much larger than any disks. Typically for production database servers SAN is preferred. But this is true even for file shares. There are storage appliances that provide large secondary and tertiary storage while making them available as a mounted share visible to all Active Directory users. This is different from the Network Access storage that is spread out in the form of VMs. Data in the case of NAS does not reside on a single virtual host and there is management incurred in finding the VM with the data requested.
That means we need to plan rollout of our software with appropriate configurations. The storage media, the local or remoteness of the storage, the escalation path for incident reports, all need to be decided before deployment. This is important not just for planning but for the way the application software is written. For example, data traffic and network chattiness can be reduced if the round trips and redundancies between the application and input was reduced. Redundancy in operations is often ignored or undetected for want of features. For example, data files are copied from one location to another and then to a third location. If this were to be eliminated by either reducing the data to the metadata that we are interested in and/or copying the file only once to the target destination, we avoid redundancy.

Unit-testing data provider calls requires that the underlying DataContext is substituted with a dummy and then we call it mocked. The trouble with that is there are no fakes or mocks that can be generated for the DataContext type. This is primarily because the DataContext object derives from a base DbContext object. If it were to implement an interface, then it can be mocked or faked.  However, adding an interface that calls out the salient methods for mocking stubs is not usually a one time solution. The DataContext object is itself auto generated. Each time the DataContext object is regenerated, the interface would likely have to be added again or the tests won't pass. This adds a lot of maintenance for otherwise unchanging code. There is a solution with Text Template Transformation toolkit (T4) templates which comes with Entity Data Modeler in EF 4.0. The T4 template creates the interface we desire.
We generate this the same way that we generate our data objects from EF. We right click on the Entity framework model and set the code generation strategy to none, and then we add a new code generation item and select the ADO.Net mocking context generator.
This way we have enabled testing the data provider classes by mocking the context classes underneath. So our testing can stay limited to this layer without polluting the layers beneath or reaching the database.
Testing the data provider class comes in helpful when we want to build the business logic layer on top of it. The business layer can now assume that layers underneath it know how to send the data across.
In the previous example, we consider a server client for reading stack trace from dumps. If this logic were to be implemented, both server and client side could use EF and data provider classes as described above. The interfaces only help to make it more testable at which point this common code can be factored out and included as a core project with the server and client code. The reason for the server and client code to be written separately is because both of them can be developed separately.
As discussed earlier, the server code handles all the data for the population of the tables. The interface for the server is the same as from a powershell client or UI or a file watcher service. By exposing the object that read the stack trace from a dump, directly in powershell we improve the automation.The file watcher service invokes the same interface for each of the files watched. The code could also keep the data local with a local database file that asp.net and EF can understand. This way we can even do away with the consumer side for the first phase and then add it subsequently in the second phase.  At that point the local database can be promoted to a shared database server.
Finally, the processing of the dump files may involve a debugger process to be launched. This means there has to be diagnosability of the process failures and appropriate messages to the invoker. Since the process invoking the debugger might be handling all exceptions and not passing the exception information across the process boundary, failures to read the dump may be hard to diagnose. If the number of such failures are high such that there is a backlog of unprocessed or partly processed dumps, then the overall success rate of the solution is affected. Some simple ways to handle this would be to stream all exceptions to the error output stream and read it from the process invoker.

Some ways to test a service that relies on a file watcher service and service bus. Service bus is merely a message delivery mechanism that guarantees messages will not be dropped and the sender and receiver (typically the client and the server) process these messages asynchronously.
1) Start copying a large file into the watched folder that takes a considerable time to copy.
2) Remove 'Everyone' or public access from the security settings of the watched folder.
3) Copy a file with extensions inclusive of the pattern being watched
4) Copy a file that triggers an exception in the servie such as a file with junk contents different from what is being expected
5) Send a message that causes an exception in the serve. If the server is expecting the path to a newly added file, send a path which denies access
6) Check the number of retries for the message in step 5)
7) For messages that repeatedly fail, check that the message made it to the poison queue.
8) Turn off the server that the messages are being received by and check the expiration of the messages and their move to dead letter queue.
9) Check the database for duplicate entries with same file name as the one in the watched folder
10) Check that the database entry for the same file input is refreshed on each attempt.
11) Check if this happens in a transaction where all the steps from moving a file, to clearing the message queue, to purging the database entry all occur in a rollback.
12) Check that the server processing the messages does not hang if the input doesn't contain what the server is looking for.
13) Check that the security settings on the file, folder, queue access, database and other dependencies are changed and don't cause service failures.
14) Check that queue cannot be flooded with unprocessed messages or denial of service attacks.
15) Check that watched folder does not have a backlog of files.

The web interface for the example mentioned in the previous post could be a simple list view with MVC framework. HTML5 and CSS can be used for the views. The stack trace bucket viewer application could be a visual tool to see and edit individual stack trace records read from dumps as well as a way to force retries by the producer to read the stack trace from the dump. The dump entries could carry an additional flag to denote the state such as new, in progress and completed and processed in that order. If the state is reverted, the processing is required. If there are no intermediary states required  such as for updates then the insertion and deletion of record suffices to trigger reprocessing. The producer service should watch for dump files and keep an association between the dump and the entry in the database. If the dump entry is not in the database, the dump is re-read. The lookup between the database and the dump for processing can be quick since the service could look up the dump based on path and filename.
The file watcher and the service bus are often used together. The service bus helps to queue the dumps for processing. It also helps with error conditions and retries. The queuing goes by other names as well such MSMQ and others. However depending on the workload, this may or may not be required. The benefits of queuing is that it can be processed asynchronously and enable retries. This can be handled by the service itself since it works on one file at a time.
The table for dumps read and processed can grow arbitrarily large as many different dumps are processed. Depending on the number of dumps processed in a day and the size of their metadata that we store, the table can grow large enough to require aging policy and archiving of older records. The archival can be batched to the start of every month and during maintenance window. The archival requires a table similar to the source, possibly in a different database than the live one. The archival stored procedure could read the records a few at a time from the source, insert into the destination and delete the copied from the source. If the source is not a single table but a set of related tables, the archival will do this step for every table in the order that inserts are allowed. The order of deletes will be in the reverse order since the constraints may need to be handled first. The insertion and deletes would not be expected to fail since we will select the records that are in the source but not in the destination. This way we will be in a good state between each incremental move of records. This helps when there is a large number of records that makes the stored procedure run long and become prone to interruptions or failures. The archival can resume from where it left off.
These services work with files and other windows resources so they may require that security is tightened and that dumps are handled only by a service account that has been authorized for read and writes on the folder. This security account may be different for production but may require full access to all folders and sub-folders. File handling exceptions often affect the success rate of such file based services. Internally, the same service account should be enabled access to the database where the parsed dump information is stored. Exceptions handled by the services could be logged or stored in the database. For the consumer side of the store the users will use their own credentials. Their actions can be authenticated and authorized. This way we can tell apart the changes made by either side.
Since the services and dependencies are hosted separately, they may have to tolerate connectivity failures. From an end to end perspective, the file IO operations could all be isolated and made local to the machine with the dumps while all subsequent processing is with the database.