ekg-core/RTS : bogus GC stats when running on Google Cloud Run

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

ekg-core/RTS : bogus GC stats when running on Google Cloud Run

Marco Zocca
Hi all,

I have two services deployed on Google cloud infrastructure; Service 1
runs on Compute Engine and Service 2 on Cloud Run and I'd like to log
their memory usage via the `ekg-core` library
(https://hackage.haskell.org/package/ekg-core-0.1.1.7/docs/System-Metrics.html)
(which is just a thin wrapper around GHC.Stats ).

The logging bracket is basically this :

    mems <- newStore
    registerGcMetrics mems
    void $ concurrently io (loop mems)
    where
      loop ms = do
        m <- sampleAll ms
        ... (lookup the gauges from m and log their values)
        threadDelay dt
      loop ms


I'm very puzzled by this: both rts.gc.current_bytes_used and
rts.gc.max_bytes_used gauges return constant 0 in the case of Service
2 (the Cloud Run one), even though I'm using the same sampling/logging
functionality and build options for both services.
This is about where my knowledge ends; could this behaviour be due to
the Google Cloud Run hypervisor ("gRun") implementing certain syscalls
in a non-standard way (gRun syscall guide :
https://gvisor.dev/docs/user_guide/compatibility/linux/amd64/) ?

Thank you for any pointers to guides/manuals/computer wisdom.

Details :

Both are built with these options :

-optl-pthread -optc-Os -threaded -rtsopts -with-rtsopts=-N -with-rtsopts=-T

the only difference is that Service 2 has an additional flag
-with-rtsopts=-M2G since Cloud Run services must work with 2 GB of
memory at most.
The container OS in both cases is Debian 10.4 ("Buster").
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: ekg-core/RTS : bogus GC stats when running on Google Cloud Run

Marco Zocca
Thinking a bit longer about this, this behaviour is perfectly
reasonable in the "serverless" model; resources
(both CPU and memory) are throttled down to 0 when the service is not
processing requests, which is exactly what
ekg picks up.
Why logs are printed out even outside of requests is still a bit of a
mystery, though ..

> I'm very puzzled by this: both rts.gc.current_bytes_used and
> rts.gc.max_bytes_used gauges return constant 0 in the case of Service
> 2 (the Cloud Run one), even though I'm using the same sampling/logging
> functionality and build options for both services.
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.