Application troubleshooting continued

Data Ingestion Path techniques:

 

There are multiple paths of ingestion to the data store within the SDP instance. These paths are via the store client, via the store connector, and via connector used in the maven artifact uploaded to the streaming analytics platform. They do not include external access to the Pravega store which could be from other data pipelines.

In this case, we want to look at all the requirements of a lo long-runningg-running application that generates data using the maven artifact. It is assumed that the data is pulled from different sources over the web programmatically by the application and written to a stream in the SDP instance stream store instance stream store.

The set of techniques described in this document will improve all aspects of serviceability for Flink Applications - present and future by improving

• visibility into its execution

• serviceability

• troubleshooting

• backup and restore of application state

• and data handling

All analytical applications require data and often preparing the data using some form of extract-transform-load is the bulk of the activities for its successful execution. Although data migration encompasses transfering data from external systems into the Pravega data store within stream analytics platform with or without the use of FlinkApplications, this part is about the features of stream analytics platform that facilitate these data flows 

Stream analytics platform application developers have the dual concern of code and data maintenance. While the code is exclusively hosted in the FlinkRuntime, the data makes its way through multiple paths. This feature set will improve the streamlining of data transfer to the store inside the stream analytics platform while improving the transparency of the hosted code

Most of these techniques can be implemented by the application developers but stream analytics platform can take ownership of data flows and application serviceability from the application developer and provide a smooth execution environment for the running of analytical applications using enhanced features from its store and runtime. 

This way the stream analytics facilitates and persist runtime information that enables analytical applications to run for a long time smoothly.