Today we continue discussing the best practice from storage engineering:
200) Data modeling and analysis: Data model may be described with entity-relationship diagrams, json documents, objects and graph nodes. However, the models are not final until several trials. Allowing the versions of data models to be kept also helps subsequent analysis.
201) Data Aggregation depends largely on the tools used. A sql query can perform rollups. A map-reduce can perform summation. They are very different usages but the storage tier can improve the performance if it is dedicated to either. In order to separate out the usages on a shared storage tier, we could classify the workloads and present different sites with redundant copies or materialized views.
202) Even if we separate out the query processing use of the storage tier from the raw data transfer into the storage, we need to maintain separate partitions for read-only data from read-write data. This will alleviate performance considerations as well as inconsistent views.
203) Social engineering data has had a phenomenal use case of using unstructured data storage instead of relational databases. This trend only expands and the requirements for the processing of chat messages and group discussions are way different from conventional file or object-based storage.
204) The speed of data transfer and not just the size is also very critical in the case of social engineering applications such as Facebook and Twitter. In such cases, we have to support a large number of concurrent message. A message platform such as one written in Erlang for Whatsapp may be more performant than servers written with extensive inter process communication
205) Social engineering applications also have a lot of load balancing requirements and therefore more number of servers may need to be provisioned to handle their load. Since a storage tier does not necessarily expose load balancing semantics, it could call out when an external load balancer must be used.
200) Data modeling and analysis: Data model may be described with entity-relationship diagrams, json documents, objects and graph nodes. However, the models are not final until several trials. Allowing the versions of data models to be kept also helps subsequent analysis.
201) Data Aggregation depends largely on the tools used. A sql query can perform rollups. A map-reduce can perform summation. They are very different usages but the storage tier can improve the performance if it is dedicated to either. In order to separate out the usages on a shared storage tier, we could classify the workloads and present different sites with redundant copies or materialized views.
202) Even if we separate out the query processing use of the storage tier from the raw data transfer into the storage, we need to maintain separate partitions for read-only data from read-write data. This will alleviate performance considerations as well as inconsistent views.
203) Social engineering data has had a phenomenal use case of using unstructured data storage instead of relational databases. This trend only expands and the requirements for the processing of chat messages and group discussions are way different from conventional file or object-based storage.
204) The speed of data transfer and not just the size is also very critical in the case of social engineering applications such as Facebook and Twitter. In such cases, we have to support a large number of concurrent message. A message platform such as one written in Erlang for Whatsapp may be more performant than servers written with extensive inter process communication
205) Social engineering applications also have a lot of load balancing requirements and therefore more number of servers may need to be provisioned to handle their load. Since a storage tier does not necessarily expose load balancing semantics, it could call out when an external load balancer must be used.
No comments:
Post a Comment