MongoDB provides all aspects of data storage for online retail stores. It can categorize data based on merchandising, content, inventory, customer, channel, sales and fulfilment, insight and social. Out of these merchanding, inventory, customer and insight are the most heavily used for peak holiday sales season. In addition, supply chain management system and data warehouse can also integrate well with this database.
We started looking at storing catalogs in MongoDB. The problem with catalog as with any master data management is that data becomes rigid. If the catalog is accessed more than 100 times per second, it causes issue with performance. When other departments need to access it, ETL, message bus and API services are put in place for this store. This results in fragmented data, redundant processes, heterogeneous data processing and higher costs both in terms of time and money. Even users begin to see degradation in page load times. To solve the local performance, more caches, more message bus and more ETL operations are added which only complicates it. A single view of the product and one central service becomes harder to achieve. The purpose of the catalog is to form a single view of the product with one central service, flexible schema, high read volume, write spike tolerant during catalog update, and to have advanced indexing and querying and geographical distribution for HA and low latency.
MongoDB is not a traditional MDM but it addresses all the above requirements. A traditional Master Data Management has a well known organization of catalog with support for hierarchy and dimensions. MongoDB organizes the catalog in the form of Items, Pricing, Promotions, Variants, and Ratings and Reviews. In Json this appears as nested fields and are pre-joined into objects. The objects live in the cache as well. A search engine provides search over the catalog. Functional data access is provided by the Product API. The API and the engine separately cover all operations on the catalog. The API can then be used by downstream units such as Online Store, Marketing, Inventory, SCMS and other APIs. The Search engine is built on Lucene/Solr Architecture A Lucene index keeps track of terms and their occurrence locations but the index needs to be rebuilt each time the catalog changes. The Product API can retrieve results directly from the catalog or via the search engine. In both cases, the customer only issues a single query.
#codingexercise
Yesterday we were discussing a method to find the smallest number k such that the product of digits of k is equal to n We said we could do this by finding the digits from 1 to 9 that can divide the number n and list them sorted in increasing order with repetitions such that the product can be formed. By trying this with the decreasing order of digits from 9 to 1 we get fewer digit based numbers first.
We extended this to say that if we try the above method repeatedly for consecutive integers or multiples, we do a lot of redundant calculations. Instead we could re-use the results from the factors computed earlier. However, we also said that we have to try the combinations of the factors to see which one gives the smallest numbers. We can improve that by choosing a decreasing order of factors and then working with their results so that we know we will get the smallest number first.
For example 100 = 20 * 5 and 100 = 10 * 10
We get the answer from the decreasing order of factors.
We started looking at storing catalogs in MongoDB. The problem with catalog as with any master data management is that data becomes rigid. If the catalog is accessed more than 100 times per second, it causes issue with performance. When other departments need to access it, ETL, message bus and API services are put in place for this store. This results in fragmented data, redundant processes, heterogeneous data processing and higher costs both in terms of time and money. Even users begin to see degradation in page load times. To solve the local performance, more caches, more message bus and more ETL operations are added which only complicates it. A single view of the product and one central service becomes harder to achieve. The purpose of the catalog is to form a single view of the product with one central service, flexible schema, high read volume, write spike tolerant during catalog update, and to have advanced indexing and querying and geographical distribution for HA and low latency.
MongoDB is not a traditional MDM but it addresses all the above requirements. A traditional Master Data Management has a well known organization of catalog with support for hierarchy and dimensions. MongoDB organizes the catalog in the form of Items, Pricing, Promotions, Variants, and Ratings and Reviews. In Json this appears as nested fields and are pre-joined into objects. The objects live in the cache as well. A search engine provides search over the catalog. Functional data access is provided by the Product API. The API and the engine separately cover all operations on the catalog. The API can then be used by downstream units such as Online Store, Marketing, Inventory, SCMS and other APIs. The Search engine is built on Lucene/Solr Architecture A Lucene index keeps track of terms and their occurrence locations but the index needs to be rebuilt each time the catalog changes. The Product API can retrieve results directly from the catalog or via the search engine. In both cases, the customer only issues a single query.
#codingexercise
Yesterday we were discussing a method to find the smallest number k such that the product of digits of k is equal to n We said we could do this by finding the digits from 1 to 9 that can divide the number n and list them sorted in increasing order with repetitions such that the product can be formed. By trying this with the decreasing order of digits from 9 to 1 we get fewer digit based numbers first.
We extended this to say that if we try the above method repeatedly for consecutive integers or multiples, we do a lot of redundant calculations. Instead we could re-use the results from the factors computed earlier. However, we also said that we have to try the combinations of the factors to see which one gives the smallest numbers. We can improve that by choosing a decreasing order of factors and then working with their results so that we know we will get the smallest number first.
For example 100 = 20 * 5 and 100 = 10 * 10
We get the answer from the decreasing order of factors.
No comments:
Post a Comment