Thursday, August 29, 2019

Example of streaming queries
    DataStream<StockPrice> socketStockStream = env
            .socketTextStream("localhost", 9999)
            .map(new MapFunction<String, StockPrice>() {
                private String[] tokens;

                @Override
                public StockPrice map(String value) throws Exception {
                    tokens = value.split(",");
                    return new StockPrice(tokens[0],
                        Double.parseDouble(tokens[1]));
                }
            });
    //Merge all stock streams together
    DataStream<StockPrice> stockStream = socketStockStream
        .merge(env.addSource(new StockSource("TCKR", 10)););

WindowedDataStream<StockPrice> windowedStream = stockStream
    .window(Time.of(10, TimeUnit.SECONDS))
    .every(Time.of(5, TimeUnit.SECONDS));

DataStream<StockPrice> maxByStock = windowedStream.groupBy("symbol")
    .maxBy("price").flatten();

 The window method call as a data-driven example could be:
DataStream<String> priceWarnings = stockStream.groupBy("symbol")
    .window(Delta.of(0.05, new DeltaFunction<StockPrice>() {
        @Override
        public double getDelta(StockPrice oldDataPoint, StockPrice newDataPoint) {
            return Math.abs(oldDataPoint.price - newDataPoint.price);
        }
    }, DEFAULT_STOCK_PRICE))
.mapWindow(new SendWarning()).flatten();

Even a stream from social media can be used for correlations:
DataStream<Double> rollingCorrelation = tweetsAndWarning
    .window(Time.of(30, TimeUnit.SECONDS))
    .mapWindow(new WindowCorrelation());

The application stack for stream analysis can independently scale the analysis and storage tiers to their own clusters. Clusters in this case is not just for high availability but a form of distributed processing for scale out purposes. Many traditional desktop centric applications are invested way high on scale up techniques when scale out processing when workloads have become smarter to narrow the gap between peak traffic and regular traffic.

No comments:

Post a Comment