When we deploy applications and database to the cloud, we have to pay attention to the size of the data stored. If the data is in the form of files or in the database, they can arbitrarily large. When the data exceeds several Gigabytes and grows at a considerable rate, the local storage in a VM does not suffice. At that point, a dedicated Storage area network is usually the norm because it can provide a capacity much larger than any disks. Typically for production database servers SAN is preferred. But this is true even for file shares. There are storage appliances that provide large secondary and tertiary storage while making them available as a mounted share visible to all Active Directory users. This is different from the Network Access storage that is spread out in the form of VMs. Data in the case of NAS does not reside on a single virtual host and there is management incurred in finding the VM with the data requested.
That means we need to plan rollout of our software with appropriate configurations. The storage media, the local or remoteness of the storage, the escalation path for incident reports, all need to be decided before deployment. This is important not just for planning but for the way the application software is written. For example, data traffic and network chattiness can be reduced if the round trips and redundancies between the application and input was reduced. Redundancy in operations is often ignored or undetected for want of features. For example, data files are copied from one location to another and then to a third location. If this were to be eliminated by either reducing the data to the metadata that we are interested in and/or copying the file only once to the target destination, we avoid redundancy.
Friday, May 31, 2013
Thursday, May 30, 2013
Unit-testing data provider calls requires that the underlying DataContext is substituted with a dummy and then we call it mocked. The trouble with that is there are no fakes or mocks that can be generated for the DataContext type. This is primarily because the DataContext object derives from a base DbContext object. If it were to implement an interface, then it can be mocked or faked. However, adding an interface that calls out the salient methods for mocking stubs is not usually a one time solution. The DataContext object is itself auto generated. Each time the DataContext object is regenerated, the interface would likely have to be added again or the tests won't pass. This adds a lot of maintenance for otherwise unchanging code. There is a solution with Text Template Transformation toolkit (T4) templates which comes with Entity Data Modeler in EF 4.0. The T4 template creates the interface we desire.
We generate this the same way that we generate our data objects from EF. We right click on the Entity framework model and set the code generation strategy to none, and then we add a new code generation item and select the ADO.Net mocking context generator.
This way we have enabled testing the data provider classes by mocking the context classes underneath. So our testing can stay limited to this layer without polluting the layers beneath or reaching the database.
Testing the data provider class comes in helpful when we want to build the business logic layer on top of it. The business layer can now assume that layers underneath it know how to send the data across.
In the previous example, we consider a server client for reading stack trace from dumps. If this logic were to be implemented, both server and client side could use EF and data provider classes as described above. The interfaces only help to make it more testable at which point this common code can be factored out and included as a core project with the server and client code. The reason for the server and client code to be written separately is because both of them can be developed separately.
As discussed earlier, the server code handles all the data for the population of the tables. The interface for the server is the same as from a powershell client or UI or a file watcher service. By exposing the object that read the stack trace from a dump, directly in powershell we improve the automation.The file watcher service invokes the same interface for each of the files watched. The code could also keep the data local with a local database file that asp.net and EF can understand. This way we can even do away with the consumer side for the first phase and then add it subsequently in the second phase. At that point the local database can be promoted to a shared database server.
Finally, the processing of the dump files may involve a debugger process to be launched. This means there has to be diagnosability of the process failures and appropriate messages to the invoker. Since the process invoking the debugger might be handling all exceptions and not passing the exception information across the process boundary, failures to read the dump may be hard to diagnose. If the number of such failures are high such that there is a backlog of unprocessed or partly processed dumps, then the overall success rate of the solution is affected. Some simple ways to handle this would be to stream all exceptions to the error output stream and read it from the process invoker.
We generate this the same way that we generate our data objects from EF. We right click on the Entity framework model and set the code generation strategy to none, and then we add a new code generation item and select the ADO.Net mocking context generator.
This way we have enabled testing the data provider classes by mocking the context classes underneath. So our testing can stay limited to this layer without polluting the layers beneath or reaching the database.
Testing the data provider class comes in helpful when we want to build the business logic layer on top of it. The business layer can now assume that layers underneath it know how to send the data across.
In the previous example, we consider a server client for reading stack trace from dumps. If this logic were to be implemented, both server and client side could use EF and data provider classes as described above. The interfaces only help to make it more testable at which point this common code can be factored out and included as a core project with the server and client code. The reason for the server and client code to be written separately is because both of them can be developed separately.
As discussed earlier, the server code handles all the data for the population of the tables. The interface for the server is the same as from a powershell client or UI or a file watcher service. By exposing the object that read the stack trace from a dump, directly in powershell we improve the automation.The file watcher service invokes the same interface for each of the files watched. The code could also keep the data local with a local database file that asp.net and EF can understand. This way we can even do away with the consumer side for the first phase and then add it subsequently in the second phase. At that point the local database can be promoted to a shared database server.
Finally, the processing of the dump files may involve a debugger process to be launched. This means there has to be diagnosability of the process failures and appropriate messages to the invoker. Since the process invoking the debugger might be handling all exceptions and not passing the exception information across the process boundary, failures to read the dump may be hard to diagnose. If the number of such failures are high such that there is a backlog of unprocessed or partly processed dumps, then the overall success rate of the solution is affected. Some simple ways to handle this would be to stream all exceptions to the error output stream and read it from the process invoker.
Some ways to test a service that relies on a file watcher service and service bus. Service bus is merely a message delivery mechanism that guarantees messages will not be dropped and the sender and receiver (typically the client and the server) process these messages asynchronously.
1) Start copying a large file into the watched folder that takes a considerable time to copy.
2) Remove 'Everyone' or public access from the security settings of the watched folder.
3) Copy a file with extensions inclusive of the pattern being watched
4) Copy a file that triggers an exception in the servie such as a file with junk contents different from what is being expected
5) Send a message that causes an exception in the serve. If the server is expecting the path to a newly added file, send a path which denies access
6) Check the number of retries for the message in step 5)
7) For messages that repeatedly fail, check that the message made it to the poison queue.
8) Turn off the server that the messages are being received by and check the expiration of the messages and their move to dead letter queue.
9) Check the database for duplicate entries with same file name as the one in the watched folder
10) Check that the database entry for the same file input is refreshed on each attempt.
11) Check if this happens in a transaction where all the steps from moving a file, to clearing the message queue, to purging the database entry all occur in a rollback.
12) Check that the server processing the messages does not hang if the input doesn't contain what the server is looking for.
13) Check that the security settings on the file, folder, queue access, database and other dependencies are changed and don't cause service failures.
14) Check that queue cannot be flooded with unprocessed messages or denial of service attacks.
15) Check that watched folder does not have a backlog of files.
1) Start copying a large file into the watched folder that takes a considerable time to copy.
2) Remove 'Everyone' or public access from the security settings of the watched folder.
3) Copy a file with extensions inclusive of the pattern being watched
4) Copy a file that triggers an exception in the servie such as a file with junk contents different from what is being expected
5) Send a message that causes an exception in the serve. If the server is expecting the path to a newly added file, send a path which denies access
6) Check the number of retries for the message in step 5)
7) For messages that repeatedly fail, check that the message made it to the poison queue.
8) Turn off the server that the messages are being received by and check the expiration of the messages and their move to dead letter queue.
9) Check the database for duplicate entries with same file name as the one in the watched folder
10) Check that the database entry for the same file input is refreshed on each attempt.
11) Check if this happens in a transaction where all the steps from moving a file, to clearing the message queue, to purging the database entry all occur in a rollback.
12) Check that the server processing the messages does not hang if the input doesn't contain what the server is looking for.
13) Check that the security settings on the file, folder, queue access, database and other dependencies are changed and don't cause service failures.
14) Check that queue cannot be flooded with unprocessed messages or denial of service attacks.
15) Check that watched folder does not have a backlog of files.
Wednesday, May 29, 2013
The web interface for the example mentioned in the previous post could be a simple list view with MVC framework. HTML5 and CSS can be used for the views. The stack trace bucket viewer application could be a visual tool to see and edit individual stack trace records read from dumps as well as a way to force retries by the producer to read the stack trace from the dump. The dump entries could carry an additional flag to denote the state such as new, in progress and completed and processed in that order. If the state is reverted, the processing is required. If there are no intermediary states required such as for updates then the insertion and deletion of record suffices to trigger reprocessing. The producer service should watch for dump files and keep an association between the dump and the entry in the database. If the dump entry is not in the database, the dump is re-read. The lookup between the database and the dump for processing can be quick since the service could look up the dump based on path and filename.
The file watcher and the service bus are often used together. The service bus helps to queue the dumps for processing. It also helps with error conditions and retries. The queuing goes by other names as well such MSMQ and others. However depending on the workload, this may or may not be required. The benefits of queuing is that it can be processed asynchronously and enable retries. This can be handled by the service itself since it works on one file at a time.
The table for dumps read and processed can grow arbitrarily large as many different dumps are processed. Depending on the number of dumps processed in a day and the size of their metadata that we store, the table can grow large enough to require aging policy and archiving of older records. The archival can be batched to the start of every month and during maintenance window. The archival requires a table similar to the source, possibly in a different database than the live one. The archival stored procedure could read the records a few at a time from the source, insert into the destination and delete the copied from the source. If the source is not a single table but a set of related tables, the archival will do this step for every table in the order that inserts are allowed. The order of deletes will be in the reverse order since the constraints may need to be handled first. The insertion and deletes would not be expected to fail since we will select the records that are in the source but not in the destination. This way we will be in a good state between each incremental move of records. This helps when there is a large number of records that makes the stored procedure run long and become prone to interruptions or failures. The archival can resume from where it left off.
These services work with files and other windows resources so they may require that security is tightened and that dumps are handled only by a service account that has been authorized for read and writes on the folder. This security account may be different for production but may require full access to all folders and sub-folders. File handling exceptions often affect the success rate of such file based services. Internally, the same service account should be enabled access to the database where the parsed dump information is stored. Exceptions handled by the services could be logged or stored in the database. For the consumer side of the store the users will use their own credentials. Their actions can be authenticated and authorized. This way we can tell apart the changes made by either side.
Since the services and dependencies are hosted separately, they may have to tolerate connectivity failures. From an end to end perspective, the file IO operations could all be isolated and made local to the machine with the dumps while all subsequent processing is with the database.
The file watcher and the service bus are often used together. The service bus helps to queue the dumps for processing. It also helps with error conditions and retries. The queuing goes by other names as well such MSMQ and others. However depending on the workload, this may or may not be required. The benefits of queuing is that it can be processed asynchronously and enable retries. This can be handled by the service itself since it works on one file at a time.
The table for dumps read and processed can grow arbitrarily large as many different dumps are processed. Depending on the number of dumps processed in a day and the size of their metadata that we store, the table can grow large enough to require aging policy and archiving of older records. The archival can be batched to the start of every month and during maintenance window. The archival requires a table similar to the source, possibly in a different database than the live one. The archival stored procedure could read the records a few at a time from the source, insert into the destination and delete the copied from the source. If the source is not a single table but a set of related tables, the archival will do this step for every table in the order that inserts are allowed. The order of deletes will be in the reverse order since the constraints may need to be handled first. The insertion and deletes would not be expected to fail since we will select the records that are in the source but not in the destination. This way we will be in a good state between each incremental move of records. This helps when there is a large number of records that makes the stored procedure run long and become prone to interruptions or failures. The archival can resume from where it left off.
These services work with files and other windows resources so they may require that security is tightened and that dumps are handled only by a service account that has been authorized for read and writes on the folder. This security account may be different for production but may require full access to all folders and sub-folders. File handling exceptions often affect the success rate of such file based services. Internally, the same service account should be enabled access to the database where the parsed dump information is stored. Exceptions handled by the services could be logged or stored in the database. For the consumer side of the store the users will use their own credentials. Their actions can be authenticated and authorized. This way we can tell apart the changes made by either side.
Since the services and dependencies are hosted separately, they may have to tolerate connectivity failures. From an end to end perspective, the file IO operations could all be isolated and made local to the machine with the dumps while all subsequent processing is with the database.
Tuesday, May 28, 2013
Cloud computing gives us the ability to develop applications that are virtualized across hardware and software stacks. Applications are no longer monolithic but sharded into different modules, each of which can reside on a different VM with its own software and hardware stack. Virtual machines, operating systems, server products and hosts, can be different for each module. These modules can still enable the same experience for a user as if the user was interacting with a single application. Sign on for example could be only once while the user visits different modules. Application storage, caching and services are now supported on dedicated resources.
If we want to provide APIs for our services, then they can be scoped to services and different services can meet different needs. APIs can be REST based and these will expand its reachability.
Let us take the example of provisioning a stack trace service that iterates over the dump files in a collection folder and populates a data store with stack traces read from each dump. In this case, we could expect the following APIs from the stackTrace service
IEnumerable<string> GetStackTrace(stream dumpFileStream); // retuns the stack trace associated with a path
IEnumerable<string> ResolveSymbols(IEnumerable<string> stackTrace, IEnumerable<string> symbolPath) to pretty print
IEnumerable<string> GetStackTrace(string pathToDumpAndSymbols); that combines the above operations
Next for the datatable that we populate called StackTraces, we will have attributes such as source information, bucket information and stack trace.
So we can enable all LINQ based operations on this entity.
This entity will be displayed by a service or front end that is independent from the stack trace population service. The front end could be read only that allows users to aggregate, search and sort stack traces from dumps.
In this case we have therefore separated out the producer consumer modules of our system and they are ready to be hosted on different VMs. For example, the producer service could sit on the same server as the collection folder and have a large storage since the dumps can be in the order of Gigabytes and there collections could be arbitrarily large. The consumer is more web appication tier 3 solution and can be hosted on an app server. The data table can be in a cloud datastore on a yet another VM or storage account.
Two services one table can scale to add other functionalities but together they have adequate information shared in the data table for diagnostics, audit and tracking.
If we want to provide APIs for our services, then they can be scoped to services and different services can meet different needs. APIs can be REST based and these will expand its reachability.
Let us take the example of provisioning a stack trace service that iterates over the dump files in a collection folder and populates a data store with stack traces read from each dump. In this case, we could expect the following APIs from the stackTrace service
IEnumerable<string> GetStackTrace(stream dumpFileStream); // retuns the stack trace associated with a path
IEnumerable<string> ResolveSymbols(IEnumerable<string> stackTrace, IEnumerable<string> symbolPath) to pretty print
IEnumerable<string> GetStackTrace(string pathToDumpAndSymbols); that combines the above operations
Next for the datatable that we populate called StackTraces, we will have attributes such as source information, bucket information and stack trace.
So we can enable all LINQ based operations on this entity.
This entity will be displayed by a service or front end that is independent from the stack trace population service. The front end could be read only that allows users to aggregate, search and sort stack traces from dumps.
In this case we have therefore separated out the producer consumer modules of our system and they are ready to be hosted on different VMs. For example, the producer service could sit on the same server as the collection folder and have a large storage since the dumps can be in the order of Gigabytes and there collections could be arbitrarily large. The consumer is more web appication tier 3 solution and can be hosted on an app server. The data table can be in a cloud datastore on a yet another VM or storage account.
Two services one table can scale to add other functionalities but together they have adequate information shared in the data table for diagnostics, audit and tracking.
Monday, May 27, 2013
XPath query language
XPath is a query language for XML. XML is structured data where the document is organized as a tree. The relative position of an element with respect to the root is called a Path and there's only one path from that element to the root.The selection criteria for nodes is called predicate. The different ways of slicing a tree or the line to follow is called Axes and these can be for example parent, child or self with child being the default. Paths can be nested inside predicates and predicates can be nested inside paths. Queries are expressed as full or partial paths with selection. They are also expressed in short forms. The position in the xml tree at which the next processing should take place is tracked with a context node. Nodes can have attributes, namespace and text. Elements positions are 1-based and in document order.
The expressions to denote the path usually describe a starting point, the context node, a line of search if not the child, and other absolute or relative paths. Paths have steps from one level to another and can include a mix or nesting of predicates and grouping via parameters. Query results are returned in document order and XPath does not modify the nodes. Standard operators such as union and standard functions such as count or id, sum, and startswith are available to use with the query. XPath queries can return all elements in a document based on path-steps as given in the Path expressions.Queries can return attributes that begin anywhere as long as the name is matched. Queries can use wild cards say to denote all children of any element matching a given path. Queries can evaluate selection conditions where the text is compared to a constant. Queries can evaluate conditions based on attributes within the elements.
The expressions to denote the path usually describe a starting point, the context node, a line of search if not the child, and other absolute or relative paths. Paths have steps from one level to another and can include a mix or nesting of predicates and grouping via parameters. Query results are returned in document order and XPath does not modify the nodes. Standard operators such as union and standard functions such as count or id, sum, and startswith are available to use with the query. XPath queries can return all elements in a document based on path-steps as given in the Path expressions.Queries can return attributes that begin anywhere as long as the name is matched. Queries can use wild cards say to denote all children of any element matching a given path. Queries can evaluate selection conditions where the text is compared to a constant. Queries can evaluate conditions based on attributes within the elements.
Sunday, May 26, 2013
Various usages of a tree.
1) Use the structure of a tree to find the relative position of a node with the root and to fully qualify that node.
2) Use the structure of a tree to discover siblings that share the same parent.
3) Use the structure of a tree to iterate over the siblings by walking laterally between them.
3) Use the structure of a tree to recursively travese the tree to repeat same operations.
4) Use the structure of a binary tree to traverse the tree in preorder, inorder and post order traversal.
5) Use the structure of a tree to find the common ancestor of two nodes
6) Use the structure of a tree to find the predecessor of a node in a binary search tree
7) Use the structure of a tree to find the successor of a node in a binary search tree
8) Use the structure of a tree to find if a node exists in the binary search tree
9) Use the structure of a tree to identify a dendrogram among a flat list of data points
10) Use the structure of a tree and schema definition to validate the data
11) Use the stucture of a tree to select the elements from a document
12) Use the structure of a tree to slice it in different ways
13) Use the structure of a tree to nest it as an expression in another
14) Use the structure of a tree to make a clone or copy a sub-tree
15) Use the structure of a tree to visit elements to perform operations without affecting the tree
16) Use the structure of a tree to do breadth first search or depth first search
17) Use the structure of a tree to color nodes as red or black in specific ways for specific tasks.
18) Use the structure of a tree to organize conditions where each element of a collection is evaluated against a tree
19) Use the structure of a tree to cluster and organize large sets of data
20) Use the structure of a tree for efficiently retrieving spatial data
21) Use the structure of a tree for scoping operations via inheritance and composition
1) Use the structure of a tree to find the relative position of a node with the root and to fully qualify that node.
2) Use the structure of a tree to discover siblings that share the same parent.
3) Use the structure of a tree to iterate over the siblings by walking laterally between them.
3) Use the structure of a tree to recursively travese the tree to repeat same operations.
4) Use the structure of a binary tree to traverse the tree in preorder, inorder and post order traversal.
5) Use the structure of a tree to find the common ancestor of two nodes
6) Use the structure of a tree to find the predecessor of a node in a binary search tree
7) Use the structure of a tree to find the successor of a node in a binary search tree
8) Use the structure of a tree to find if a node exists in the binary search tree
9) Use the structure of a tree to identify a dendrogram among a flat list of data points
10) Use the structure of a tree and schema definition to validate the data
11) Use the stucture of a tree to select the elements from a document
12) Use the structure of a tree to slice it in different ways
13) Use the structure of a tree to nest it as an expression in another
14) Use the structure of a tree to make a clone or copy a sub-tree
15) Use the structure of a tree to visit elements to perform operations without affecting the tree
16) Use the structure of a tree to do breadth first search or depth first search
17) Use the structure of a tree to color nodes as red or black in specific ways for specific tasks.
18) Use the structure of a tree to organize conditions where each element of a collection is evaluated against a tree
19) Use the structure of a tree to cluster and organize large sets of data
20) Use the structure of a tree for efficiently retrieving spatial data
21) Use the structure of a tree for scoping operations via inheritance and composition
Subscribe to:
Posts (Atom)