Billing System for IT Cloud Provider
Billing is useful to the consumers to know the usages that
they make of the system resources. Although a private cloud provider may not
require payments from the consumer, a report of various system resource usages
together with some analysis charts and graphs make for a very presentable and
actionable memo. The consumer can then directly participate in capacity
planning and conservation.
This is a where a software billing management system comes
useful. The billing system for a cloud provider is dedicated to the collection
and processing of the usage data as it pertains to the end consumer billing.
Detailed metrics for system resources are drawn from pay per use monitors that
are either available internally or via external providers. By external provider
we mean cloud providing vendor software that may already have its own
telemetry. These gather runtime usage data that is stored in a repository. When
the usage data is coupled with different pricing policies, we can get different
bills generated. Pricing policies can include different schemes such as
resource based or consumer based and models can vary from traditional pay-per
use models to flat rate or pay-per-allocation models or their combinations.
Usage may also be combined with quotas. When quotas are
exceeded, they can appear on the bills with notations for blocking further
usage requests by these cloud consumers.
Different providers provide their own mechanisms for usage
monitoring and billing. Two examples cited here for providers are Openstack and
AWS.
In the case of Openstack, we have Ceilometer. Ceilometer is
OpenStack’s telemetry. It’s source code is available at
https://github.com/openstack/ceilometer. It collects all kinds of monitoring
and metering information such that no two agents need to be written to collect
the same data. While monitoring and metering are different, Ceilometer’s
metering consumer is say the billing system for which it acts as a unique and
central point of contact to acquire all of the information collected across all
OpenStack core components. The project strived to be efficient in terms of CPU
and network costs. It supports both the push notifications from the existing
services and the pull model by polling the infrastructure. Deployers can
configure the type of data collected. Publishers are available as a Python
implementation. A REST API is also available to expose the data collected by
the metering system.
Bills even if written off can be delivered by mail. Their
layout and crispness speak to the metering and usages giving a good indication
to the customer of what is costing how much.
With regard to AWS Billing, we have a console that is built
into the AWS Billing and Cost Management that already has a well known and
documented cost analysis. Bills can be generated and exported as CSVs in S3
containers that can feed into different applications.
As a consolidator over cloud providers, we can maintain a
consistent common user interface that accesses the same information from the
underlying providers when available or adding new.
The design is one of a pull model where each bill is
generated by a pull of the metrics, usage and rates from the underlying
providers and our own database.
Central to the discussion on billing is the notion of a
database for a customer. This is proposed in spite of the direct API access
available to different cloud providers because a database promotes an ecosystem
that leads to different workflows. The APIs are still used to populate the data
except that they are now organized in a time-series manner and analyzed per
customer or resource
UI / Console Provider 1
pull API/Database push
Bills (emails) Provider 2
Openstack Billing provider:
Ceilometer measures the usage of resources in terms of disk
usage, network usage and CPU usage. For each instance the usage details for
following meters will be shown:
1.
Disk Read Bytes
2.
Disk Write Bytes
3.
Disk Write Requests
4.
Disk Read Requests
5.
Network outgoing bytes
6.
Network incoming bytes
7.
Network outgoing packets
8.
Network incoming packets
9.
CPU
Ceilometer arranges all the data in terms of resources and
their meters.
We can get a list of all the meters using
GET /v2/resources/ and
GET /v2/resources/(resource_id)
Where resource
contains links such as a url to identify itself and a url for its meters such
as http://localhost:8777/v2/meters/volume?q.field=resource_id&q.value=bd9431c1-8d69-4ad3-803a-8d4a6b89fd36
GET /v2/meters/ that returns all know meters and
GET /v2/meters/(meter_name) which even come with statistics
GET /v2/meters/(meter_name)/statistics
Samples and statistics can also be drawn from
GET /v2/samples
And GET /v2/ samples/(sample_id)
Tables can be created in the following manner:
1)
RatePlan
a.
Id
b.
Name
c.
CPU cost per second
d.
Disk cost per MB
e.
Network cost per MB
f.
Cost for flavors
2)
Table for instance-bill will have the following
attributes
a.
Resource_id
b.
Rateplan_id
c.
Date and time at which the bill was calculated
d.
Resource_bill
Amazon Billing Provider
AWS Cloud operates on a pay as you go model so the bill
reflects the actual usage. This also means that we need to check the account
activity every so often or get alerts through alarms setup through Amazon
CloudWatch metrics and alarms. The Amazon CloudWatch provides the monitoring of
the estimated charges using billing alerts. Furthermore it delivers via the
Amazon SNS – a well known publisher subscriber service for Mobile and
Enterprise Messaging. The billing alerts sent by Amazon CloudWatch include estimates
stored as CloudWatch metrics and this metric data is stored for 14 days. The data includes variations of the
following:
Estimated charges : Total
Estimated charges : by service
Estimated charges : by Linked Account
Estimated charges : by Linked Account and Service
Note that these are estimates and not predictions. They
approximate the cost and not take the trends or potential changes in the AWS
usage pattern into account
The use of these alerts is unique for the consolidated
billing for the private cloud provider. Essentially the alerts populate a
dynamic view (in-memory) or table (stored) for each customer, account and
service.
The table can have the following attributes
-
Current period cost by service
-
Cumulative cost by service
With this table, we can generate reports with customer
account and service granularity. If we keep time series collection of these
costs, we can pull reports as well. One thing that should stand out is that
CloudWatch already manages most of the cost estimation and can provide
different kinds of analysis builtin into the alerts. Therefore the table we
keep for the data that we need to pull for our customers is merely a staging
store. The bulk of the calculation and processing is already done for us.
The relevant APIs to populate the staging store in this case
would be as follows:
Services/cloudwatch.class.php - __construct ( $options ) :
Constructs a new instance of AmazonCloudWatch:
put_metric_alarm ( $alarm_name, $metric_name, $namespace,
$statistic, $period, $evaluation_periods, $threshold, $comparison_operator,
$opt )
Creates or updates an alarm and associates it with the
specified Amazon CloudWatch metric. Optionally, this operation can associate
one or more Amazon Simple Notification Service resources with the alarm.
delete_alarms ( $alarm_names, $opt )
Deletes all specified alarms
Retrieves history for the specified alarm. Filter alarms by
date range or item type. If an alarm name is not specified, Amazon CloudWatch
returns histories for all of the owner’s alarms.
enable_alarm_actions ( $alarm_names, $opt )
Enables actions for the specified alarms.
disable_alarm_actions ( $alarm_names, $opt )
Disables actions for the specified alarms. When an alarm’s
actions are disabled the alarm’s state may change, but none of the alarm’s
actions will execute.
list_metrics ( $opt )
Returns a list of valid metrics stored for the AWS account
owner. Returned metrics can be used with GetMetricStatistics to obtain
statistical data for a given metric.
There is no need to merge the bills from different providers
since we strive to be more granular in itemizing the costs. They can appear as
different sections one from each provider. This saves the trouble of
artificially merging into a common set of metrics when it may not be available
as such from each provider. And even if they are available, their syntax,
semantics and nuances may vary. Our
customer recognize services and instances of compute and storage. As long as we
maintain those as granular as possible in the bills, the customer has the
luxury of aggregating these himself or seeing the summary wherever possible in one of
the analysis charts.
Note that the providers may also maintain their own billing
database that provides a rich query API. This might even argue for removing our
database and performing the query on demand when the bills are being requested
by the user in an interactive mode or during batch processing. However such
databases may rarely store data more than a few weeks worth leaving our system
to poll and store every so often to build up a time series data for longer
analysis. In addition, the user account or plan may change frequently and may
even to get notifications of when she is exceeding the free tier of usages. Additionally,
one provider may promote a notification model preferred over a polling model
while another may offer the other model to pull the data. A staging store for
the billing data as relevant to the consumer bills enables to talk to different
providers. We can keep the staging store as minimal as necessary, relying more
on getting the information by APIs but it doesn’t seem to be avoidable.
Most of the cloud providers mentioned as examples are
already providing rich set of APIs for their usage as shown. We just have to
focus on the service that extracts, transforms and loads the database for use
by the administrators and the consumers. Reports and publishing as emails can
be done by pulling the data from this database.
This software is easy to build with the existing frameworks
that we already have both in terms of the data providers as well as the
infrastructure for hosting it.
nice post.
ReplyDeleteaws training
r programming training