Has anyone benchmarked cold start times for Clojure on AWS Lambda?

Question

I was going over this article: https://mikhail.io/serverless/coldstarts/aws/languages/ and noticed that even for Go, cold starts take in the order of 250ms to 450ms most of the time.

My surprise was how slow even the "fast to start" languages were when hitting an AWS Lambda cold start. So I've been wondering how slow would Clojure actually be? Since Lambda cold start already come with a pretty big overhead, if we go from 300ms average with Go to 600ms average with Clojure for cold start, it might actually not matter that much, since we're already in the range of "slow enough for the user to notice".

But then I tried searching online, and I couldn't find any actual benchmark done with Clojure. I could only find ClojureScript or GraalVM native image Clojure benchmarks.

Anyone has any Clojure benchmarks for AWS Lambda cold start (ideally as recent as possible, since AWS has improved their JVM cold starts quite a lot in recent years).

Answer 1

Hello

You can find some benchmarks which compare GraalVM native image in my blog post at dev.solita.fi. See especially the chapter "Clojure running in JVM AWS Lambdas"

https://dev.solita.fi/2018/12/07/fast-starting-clojure-lambdas-using-graalvm.html

I benchmarked also regular JVM. Unfortunately Clojure is very slow to start and this startup scales badly with larger projects. The slow start-up time is mostly caused by JVM class loading. Unfortunately Clojure generates a lot of classes because every Clojure variable definition and function are compiled to classes.

You are not going to see under 2 seconds start up times using Clojure without GraalVM.

Comments

Interesting results. I wonder why is that program taking so much longer on Lambda than it would on a normal computer. I mean, all it does is return Hello World!. That would definitely start under 1 second on my 10 year old laptop.

Could that deflambda be responsible?

---

You wont see under 1 second start up times when running in Fargate either. I have many years experience building and running Clojure in AWS so these startup times are not unexpected.

Minimal helloworld uberjarred and run with  my MacBook Pro (15-inch, 2017) does run and exit in 1.3 -1.5 seconds.  Lambda enviroments CPU is not known (only memory can be set) but as far as I know it has at most two threads. Clojure startup is CPU intensive operation so this might explain why it is faster in your laptop than in AWS lambda.

Deflambda is only simple macro which generates necessary class for Lambda to run. It doesnt generate any overhead to the test.

---

Right, but still, I'm wondering if something else isn't going on. Like artifact size, having to pull Clojure, Spec, etc.

For example, I'd assume the Python, JS, Ruby, and Java libs are already bundled inside the Lambda runtime. So they don't have to download these and pay the download and decompress cost. Which might not be true of Clojure, and maybe why it adds so much start time?

Native image does tree-shaking for example of the entire byte code, and will also reduce artifact size in that respect most likely.

Not saying the data is wrong, just it doesn't seem we have a true root cause here, because class-loading on old computers still isn't that slow. On my 2013 Dell XPS I hello world clojure app take under 2 second to start for example.

---

The artifact size is not in my experience the cause of start up latency because Python Lambdas does not experience any noticiable slowdown when bundling big libraries in an artifact.

This post is thorough explanation of Clojure startup:
http://clojure-goes-fast.com/blog/clojures-slow-start/

As you can read - Clojure start up takes at least second which cannot be currently avoided. Lambda enviroment adds few hundreds milliseconds to initialize which can be tested by comparing cold and warm starts in different runtimes.

---

I know Clojure has a slow initialization, but even so, a Hello World should not exceed 2 seconds, especially on a Lambda with 3 gig of memory. I'd still be interested to get to the bottom of that. As the link you provided shows, Clojure takes 1 second to initialize. In my old laptop, I have a dual core i7 at 1.7 Ghz and 4 gig Ram: http://www.notebookreview.com/notebookreview/dell-xps-13-review-an-ultrabook-for-business-pros/2/ it still takes under 2 seconds for a Clojure Hello World app.

It could be the JVM Lambda is using, or some other Lambda specific quirk. Knowing that could help in improving the situation. And maybe even the 3 gig Lambda is running some very constrained CPU, or has really slow disk IO, or something like that.

Thanks for all the details and doing that benchmark.