Billing Estimate

This is the aws lambda pricing calculator if you would like to read more: https://aws.amazon.com/lambda/pricing/

It is difficult for us to track exact costs for all factors but here are some estimates — noting that they can be off by a lot because there are overheads beyond just the lambda calculations including things like SNS messaging, SQS queues, DynamoDB reads/writes, and S3 storage that can all be difficult to figure out.

Focusing on the most expensive costs associated with lambda processing is a good starting point.

I can check the cloudwatch logs for each execution and get the "Billed Duration" rounded to the nearest millisecond.

There are three factors that need to be considered: (1) time, (2) memory & (3) storage.

Below are the current configurations and some real metrics from the logs:

##################################################
[1] For the "cruise-splitter-lambda":

2023-08-07T13:41:47.122-06:00
REPORT RequestId: 370cebdd-fd77-41a7-a8c3-2e267f024303 Duration: 4065.56 ms Billed Duration: 4066 ms Memory Size: 1024 MB Max Memory Used: 183 MB Init Duration: 4049.04 ms

The total billed duration was 4.066 seconds for one lambda execution. Note that we don't pay for time that wasn't used but we do pay for memory & storage that wasn't used.

4.066 seconds with 1024 MB memory & 512 MB ephemeral storage

##################################################
[2] For the "raw-to-zarr-lambda" with an average of the billed durations:

np.mean(np.array([29378, 198084, 204987, 205034, 200731, 199370, 198958, 188257, 171330, 22204, 90900, 94185]))
150284.83 ms  --> 150.285 seconds

150.285 seconds with 10240 MB memory & 10240 MB ephemeral storage

##################################################
[3] For the "zarr-cruise-accumulator-lambda" which runs each time a raw-to-zarr-lambda finishes:

np.sum(np.array([4399, 70, 54, 63, 96, 63, 69, 84, 47, 52, 89, 1971]))
7057 ms --> 7.057 seconds

7.057 seconds with 512 MB memory & 512 MB ephemeral storage

##################################################
[4] The "create-empty-zarr-store-lambda":

127.568 seconds with 10240 MB memory & 10240 MB ephemeral storage

##################################################
[5] The "resample-and-write-to-zarr-store-lambda" which is run for each successfully converted EK60 file:

np.median(np.array([109190, 106547, 105400, 106224, 104337, 103772, 97940, 22080, 50603, 54751, 23739]))
103772.0 --> 103.772 seconds

103.772 seconds with 2048 MB memory & 1024 MB ephemeral storage

###########################################################################################
###########################################################################################
To get an estimate for an example such as the "raw-to-zarr-lambda," we can compute these two values:

AWS_LAMBDA_PRICING = (0.0000001667 price per ms) * (150285 ms) = $0.0250525095
LAMBDA_EPHEMERAL_STORAGE_PRICING = (0.0000000309 for every GB-second) * (10 GB) * (150 seconds) = $0.000046

for a total of $0.025 per EK60 file processed in the "raw-to-zarr-lambda".

###########################################################################################
We can get a cruise-level cost as follows:

cruise-splitter-lambda
1024 MB memory * 4.066 seconds + 512 MB ephemeral storage * 4.066 seconds
0.0000000167 * 4066 + 0.0000000309 * 5 * 4.066 = $0.000069

raw-to-zarr-lambda
10240 MB memory * 150.285 seconds + 10240 MB ephemeral storage * 150.285 seconds
0.0000001667 * 150285 + 0.0000000309 * 10 * 150 = $0.025099

zarr-cruise-accumulator-lambda(s)
512 MB memory * 7.057 seconds total + 512 MB ephemeral storage * 7.057 seconds total
0.0000000833 * 7057 + 0.0000000309 * 5 * 7.057 = $0.0005889

create-empty-zarr-store-lambda
10240 MB memory * 127.568 seconds + 10240 MB ephemeral storage * 127.568 seconds
0.0000001667 * 127568 + 0.0000000309 * 10 * 127.568 = $0.021305

resample-and-write-to-zarr-store-lambda
2048 MB memory * 103.772 seconds + 1024 MB ephemeral storage * 103.772 seconds
0.0000000333 * 103772 + 0.0000000309 * 1 * 103.772 = $0.003459

So the total pipeline processing cost was (very roughly):

(0.000069) + (0.025099 * 12) + (0.0005889) + (0.021305) + (0.003459 * 11) = $0.36

to process the HB0707 cruise with 12 EK60 files (11 of which were successful).

...

Moving forward my main focus will be on optimizing the two longer-lived and parallelized lambdas that will be doing the bulk of
EK60 processing, so reducing the cost for the:
raw-to-zarr-lambda
resample-and-write-to-zarr-store-lambda.
I will start by provisioning less memory and using less ephemeral storage.

...

There were a lot of places for me to screw the math up above so please take all these numbers lightly.