A few more considerations for using S3 over document store
for basic storage operations.
The previous article introduced cost as the driving factor
for leveraging simple storage aka S3. The document store has many features but
is priced based on read and write capacity units. All those features may not be
necessary for mere create, update and delete of an object. This results in
significant savings even on low-end applications that typically have a monthly
charge as follows:
API Gateway 0.04 USD
Cognito 10.00 USD
DynamoDB 75.02 USD
S3 2.07 USD
Lambda 0.00 USD
Web Application Firewall 8.00 USD
It is in this context that we strive to use S3 APIs for
ordinary persistence.
The sample code below illustrates the use of Javascript SDK
for making these operations:
const REGION =
"us-west-2";
const s3 = new
S3Client({
region: REGION,
credentials: fromCognitoIdentityPool({
client: new CognitoIdentityClient({ region:
REGION }),
identityPoolId:
"us-west-2:de827e1d-f9b6-4402-bd0e-c7bdce52d8c8",
}),
});
const
docsBucketName = "mybucket";
export const
getAllDocuments = async () => {
if (!client) {
await createAPIClient();
}
try {
const data = await s3.send(
new ListObjectsCommand({ Delimiter:
"/", Bucket: docsBucketName })
);
console.log(JSON.stringify(data, null, 4));
var results = [];
if (typeof data != "undefined"
&& data.hasOwnProperty("Contents")) {
results
= data.Contents.map(function(item,index) {
var identifier = item.Key +
item.LastModified + item.Owner.ID;
return
{
'FileSize' : item.Size,
'Name' : item.Key,
'Owner' : item.Owner.ID,
'DateUploaded' :
item.LastModified,
'FileName' : item.Key,
'SK' : 'Doc#BVNA',
'PK' :
identifier.hashCode().toString(),
'Thumbnail' :
'/images/LoremIpsum.jpg'};
});
}
return results;
} catch (err) {
console.log("Error", err);
return [];
}
};
Unlike the document store that returns a unique identifier
for every item stored, here we must make our own identifier. The file contents
and the file attributes together can help make this identifier if we leverage
basic cryptology functions such as md5.
Also, unlike the document store there is no index. Tags and metadata are
available for querying purposes and it is possible adjust just the tags for
state management but it is even better to populate the operations on an
uploaded object in a dedicated metadata object in the database.
Then, it is possible to query just the contents of that
specific object with:
const S3 = require(‘aws-sdk/clients/s3’);
s3.selectObjectContent(params,
(err, data) => {
if (err) {
// handle error
Return
}
const eventStream = data.Payload;
eventStream.on(‘data’, (event)
=> {
if (event.Records) {
// event.Records.Payload is a
buffer containing
// a single record, partial
records, or multiple records
process.stdout.write(event.Records.Payload.toString());
} else if (event.Stats) {
console.log(`Processed
${event.Stats.Details.BytesProcessed} bytes`);
} else if (event.End) {
console.log('SelectObjectContent
completed');
}
});
// Handle errors encountered
during the API call
eventStream.on('error', (err)
=> {
switch (err.name) {
// Check against specific error
codes that need custom handling
}
});
eventStream.on('end', () =>
{
// Finished receiving events
from S3
});
});
This mechanism is sufficient for low overhead persistence of
objects in the cloud.
