Part 2 of previous post

 

This is a continuation of the article that introduces a crowdsourcing application. The original problem statement is included again for context:

Social engineering applications provide a wealth of information to the end-user but the questions and answers received on it are always limited to just that – social circle. Advice solicited for personal circumstances is never appropriate for forums which can remain in public view. It is also difficult to find the right forums or audience where the responses can be obtained in a short time. When we want more opinions in a discrete manner without the knowledge of those who surround us, the options become fewer and fewer. In addition, crowd-sourcing the opinions for a personal topic is not easily available via applications. This document tries to envision an application to meet this requirement.

The solution proposed in the linked article suggests that crowdsourcing is a collaboration software much like the social engineering applications that are powered by massive compute and storage using services in the backend and the clients as mobile device applications.  But the deployment of the software does not have to be directly to several datacenters. Instead, its roadmap could involve the use of a public cloud where the services are already available to enable fewer overheads and less total cost of ownership.

When a user poses a question, she creates a campaign. This campaign is represented by a document that must support collaborative editing. So, it must be versioned, and merges must be resolved by incrementing the version. Cosmos DB works well for document stores and enables services to implement updates and merge resolutions. It is also highly available and scales large storage for hundreds of users. It comes with read-only and read-write separation of regions so that the traffic can be efficiently separated. Since a single document will not likely be answered by more than a thousand responders for the duration of the question and there are likely to be a large number of documents, this problem is solved sufficiently by the suggested database.

Communication between clients is one of the messaging frameworks. This unlike storage has a sliding scale for performance and suitable technology depending on massively scalable queues and topics. The Service Bus messaging architecture works well in this regard. The emphasis is on scale so we can consider a few technologies for comparison.

The system.messaging library transparently exposes the underlying Message queuing windows APIs . For example, it provides GetPublicQueues method that enumerates the public message queues. It takes the message queue criteria as a parameter. This criterion can be specified with parameters such as category and label. It can also take machine name or cluster name, created and modified times as filter parameters. The GetPublicQueuesEnumerator is available to provide an enumerator to iterate over the results. 

Amazon Simple Queue Services offers an almost unparalleled fully managed message queuing service that makes it easy to integrate with microservices and serverless applications. SQS also makes it simple and cost-effective to intermediate between these producers and consumers.

Then, there is the highly performant Erlang-based messaging framework. It offers free texting in a SMS world. There are no ads, no gimmicks, no games and no feeds. They have hundreds of nodes, thousands of cores, hundreds of terabytes of RAM, serve billions of smartphones. Their server infrastructure is based on Erlang/FreeBSD.  The Erlang is a programming language that is used to build massively scalable soft real-time systems with requirements on high availability.  With this programming language they send out about 70 million messages per second.

The WhatsApp architecture is famous for “it just works” because only a phone number is used as the address to send messages to and messages can be voice, video, images or text.   It also features voice message, group chats and read-receipts.

Together with messaging and storage, we address most of the dependencies for compute.