Static executables in minimal Docker containers

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

Static executables in minimal Docker containers

Michael Snoyman
I'm trying to put together a minimal Docker container consisting of nothing but GHC-compiled static executables. Below you can see a full interaction I've had with GHC and Docker. The high level summary is that:

* When compiled statically, the executable runs just fine in both my host OS (Ubuntu 14.04) and an Ubuntu 14.04 Docker image
* That same executable run from a busybox (or a "scratch" image, not shown here since it's slightly longer to set up) hangs and then runs out of memory

I've watched the process in top, and it uses up a huge amount of CPU and memory. I get the same behavior whether I compiled with or without optimizations or the multithreaded runtime. I also get identical behavior with both GHC 7.8.4 and 7.10.1.

I'm not sure how best to proceed with trying to debug this, any suggestions?

vagrant@vagrant-ubuntu-trusty-64:~/Desktop$ cat > hello.hs <<EOF
main :: IO ()
main = putStrLn "Static Hello"
EOF
vagrant@vagrant-ubuntu-trusty-64:~/Desktop$ ghc -optl-static -optl-pthread hello.hs -static
[1 of 1] Compiling Main             ( hello.hs, hello.o )
Linking hello ...
/opt/ghc/7.8.4/lib/ghc-7.8.4/rts-1.0/libHSrts.a(Linker.o): In function `internal_dlopen':
(.text+0x5c): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
vagrant@vagrant-ubuntu-trusty-64:~/Desktop$ ./hello
Static Hello
vagrant@vagrant-ubuntu-trusty-64:~/Desktop$ docker run --rm -v $(pwd)/hello:/usr/local/bin/hello:ro ubuntu:14.04 /usr/local/bin/hello
Static Hello
vagrant@vagrant-ubuntu-trusty-64:~/Desktop$ docker run --rm -v $(pwd)/hello:/usr/local/bin/hello:ro busybox /usr/local/bin/hello
hello: out of memory (requested 1048576 bytes)
vagrant@vagrant-ubuntu-trusty-64:~/Desktop$ ghc --version
The Glorious Glasgow Haskell Compilation System, version 7.8.4

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Static executables in minimal Docker containers

Magnus Therning
On 13 April 2015 at 10:54, Michael Snoyman <[hidden email]> wrote:
> I'm trying to put together a minimal Docker container consisting of nothing
> but GHC-compiled static executables. Below you can see a full interaction
> I've had with GHC and Docker. The high level summary is that:
>
> * When compiled statically, the executable runs just fine in both my host OS
> (Ubuntu 14.04) and an Ubuntu 14.04 Docker image
> * That same executable run from a busybox (or a "scratch" image, not shown
> here since it's slightly longer to set up) hangs and then runs out of memory

From what I remember busybox allows for quite a bit of
configurability, so "a busybox" might need a bit more details mabye.

Have you attempted running it under `strace` and/or `ltrace`?

/M

--
Magnus Therning                      OpenPGP: 0xAB4DFBA4
email: [hidden email]   jabber: [hidden email]
twitter: magthe               http://therning.org/magnus
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Static executables in minimal Docker containers

Michael Snoyman


On Mon, Apr 13, 2015 at 12:07 PM Magnus Therning <[hidden email]> wrote:
On 13 April 2015 at 10:54, Michael Snoyman <[hidden email]> wrote:
> I'm trying to put together a minimal Docker container consisting of nothing
> but GHC-compiled static executables. Below you can see a full interaction
> I've had with GHC and Docker. The high level summary is that:
>
> * When compiled statically, the executable runs just fine in both my host OS
> (Ubuntu 14.04) and an Ubuntu 14.04 Docker image
> * That same executable run from a busybox (or a "scratch" image, not shown
> here since it's slightly longer to set up) hangs and then runs out of memory

From what I remember busybox allows for quite a bit of
configurability, so "a busybox" might need a bit more details mabye.

Have you attempted running it under `strace` and/or `ltrace`?



Sorry, I left off a word: I meant "a busybox image." There's a standard busybox Docker image which I'm testing with via the command I pasted below. There are certainly ways to configure busybox itself, but it should be trivial for others to reproduce this error simply by running my series of commands.

Unfortunately, strace and ltrace aren't available in that Docker image, but it's a good idea to see if I can get them running there somehow.

Michael 

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Static executables in minimal Docker containers

Greg Weber
Haskell is not that great at producing statically linked libraries independent of the OS.
The issue you are running into would likely show up in another non-ubuntu image (or even possibly a different version of an ubuntu image), so you could probably use a Fedora image that has tracing.

How are you addressing the linker warning about needing a particular glibc version at runtime?

On Mon, Apr 13, 2015 at 3:28 AM, Sharif Olorin <[hidden email]> wrote:
Unfortunately, strace and ltrace aren't available in that Docker image, but it's a good idea to see if I can get them running there somehow.

Failing that, you might be able to get useful information of the same kind by running docker (the server, not the `docker run` command) under perf[0] and then running your busybox container. It should at least give you an idea of what it's doing when it explodes.

Sharif

--


_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Static executables in minimal Docker containers

Jonathon Delgado
Greg Weber wrote
Haskell is not that great at producing statically linked libraries
independent of the OS.
The issue you are running into would likely show up in another non-ubuntu
image (or even possibly a different version of an ubuntu image), so you
could probably use a Fedora image that has tracing.
You could try compiling on Debian and running it in https://registry.hub.docker.com/u/accursoft/micro-jessie/. Much bigger than busybox, but 33Mb may be small enough for you.

(Remember that the OS layer will be shared between multiple images - I don't know if that helps for your scenario.)
Reply | Threaded
Open this post in threaded view
|

Re: Static executables in minimal Docker containers

Michael Snoyman
In reply to this post by Greg Weber
I'm not sure if this issue would show up, but I can try it in Fedora tomorrow. I didn't address the linker warning at all right now, it seems to not have been triggered, though I suppose it is possible that it's the cause of this issue.

On Mon, Apr 13, 2015 at 7:10 PM Greg Weber <[hidden email]> wrote:
Haskell is not that great at producing statically linked libraries independent of the OS.
The issue you are running into would likely show up in another non-ubuntu image (or even possibly a different version of an ubuntu image), so you could probably use a Fedora image that has tracing.

How are you addressing the linker warning about needing a particular glibc version at runtime?

On Mon, Apr 13, 2015 at 3:28 AM, Sharif Olorin <[hidden email]> wrote:
Unfortunately, strace and ltrace aren't available in that Docker image, but it's a good idea to see if I can get them running there somehow.

Failing that, you might be able to get useful information of the same kind by running docker (the server, not the `docker run` command) under perf[0] and then running your busybox container. It should at least give you an idea of what it's doing when it explodes.

Sharif

--


_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Static executables in minimal Docker containers

Vo Minh Thu
I missed this thread but I guess I tried something similar last month:
https://gist.github.com/noteed/4155ffad2b1d13ab17ee


2015-04-13 20:50 GMT+02:00 Michael Snoyman <[hidden email]>:

> I'm not sure if this issue would show up, but I can try it in Fedora
> tomorrow. I didn't address the linker warning at all right now, it seems to
> not have been triggered, though I suppose it is possible that it's the cause
> of this issue.
>
>
> On Mon, Apr 13, 2015 at 7:10 PM Greg Weber <[hidden email]> wrote:
>>
>> Haskell is not that great at producing statically linked libraries
>> independent of the OS.
>> The issue you are running into would likely show up in another non-ubuntu
>> image (or even possibly a different version of an ubuntu image), so you
>> could probably use a Fedora image that has tracing.
>>
>> How are you addressing the linker warning about needing a particular glibc
>> version at runtime?
>>
>> On Mon, Apr 13, 2015 at 3:28 AM, Sharif Olorin <[hidden email]>
>> wrote:
>>>>
>>>> Unfortunately, strace and ltrace aren't available in that Docker image,
>>>> but it's a good idea to see if I can get them running there somehow.
>>>
>>>
>>> Failing that, you might be able to get useful information of the same
>>> kind by running docker (the server, not the `docker run` command) under
>>> perf[0] and then running your busybox container. It should at least give you
>>> an idea of what it's doing when it explodes.
>>>
>>> Sharif
>>>
>>> [0]: https://perf.wiki.kernel.org/index.php/Tutorial
>>>
>>> --
>>>
>>> "Commercial Haskell" group.
>>>
>>> email to [hidden email].
>>>
>>>
>>> https://groups.google.com/d/msgid/commercialhaskell/86ca2603-37f2-4645-9cd2-f09703f2be67%40googlegroups.com.
>>>
>>>
>>
>>
>
> _______________________________________________
> Haskell-Cafe mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
>
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Static executables in minimal Docker containers

Aaron Levin
FWIW: I experienced something similar to this (hanging) with the standard Debian image.
On Mon, Apr 13, 2015 at 3:16 PM Vo Minh Thu <[hidden email]> wrote:
I missed this thread but I guess I tried something similar last month:
https://gist.github.com/noteed/4155ffad2b1d13ab17ee


2015-04-13 20:50 GMT+02:00 Michael Snoyman <[hidden email]>:
> I'm not sure if this issue would show up, but I can try it in Fedora
> tomorrow. I didn't address the linker warning at all right now, it seems to
> not have been triggered, though I suppose it is possible that it's the cause
> of this issue.
>
>
> On Mon, Apr 13, 2015 at 7:10 PM Greg Weber <[hidden email]> wrote:
>>
>> Haskell is not that great at producing statically linked libraries
>> independent of the OS.
>> The issue you are running into would likely show up in another non-ubuntu
>> image (or even possibly a different version of an ubuntu image), so you
>> could probably use a Fedora image that has tracing.
>>
>> How are you addressing the linker warning about needing a particular glibc
>> version at runtime?
>>
>> On Mon, Apr 13, 2015 at 3:28 AM, Sharif Olorin <[hidden email]>
>> wrote:
>>>>
>>>> Unfortunately, strace and ltrace aren't available in that Docker image,
>>>> but it's a good idea to see if I can get them running there somehow.
>>>
>>>
>>> Failing that, you might be able to get useful information of the same
>>> kind by running docker (the server, not the `docker run` command) under
>>> perf[0] and then running your busybox container. It should at least give you
>>> an idea of what it's doing when it explodes.
>>>
>>> Sharif
>>>
>>> [0]: https://perf.wiki.kernel.org/index.php/Tutorial
>>>
>>> --
>>> >>> "Commercial Haskell" group.
>>> >>> email to [hidden email].
>>> >>> >>> https://groups.google.com/d/msgid/commercialhaskell/86ca2603-37f2-4645-9cd2-f09703f2be67%40googlegroups.com.
>>>
>>> >>
>>
>
> _______________________________________________
> Haskell-Cafe mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
>
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Static executables in minimal Docker containers

Albert Y. C. Lai
I wonder whether you already know the following, and whether it is
relevant to begin with. (Plus, my knowledge is fairly sketchy.)

Even though you statically link glibc, its code will, at run time,
dlopen a certain part of glibc.

Why: To provide a really uniform abstraction layer over user account
queries, e.g., man 3 getpwnam, regardless of whether the accounts are
from /etc/passwd, LDAP, or whatever.

Therefore, during run time, glibc first reads some config files of the
host to see what kind of user account database the host uses. If it's
/etc/passwd, then dlopen the implementation of getpwnam and friends for
/etc/passwd; else, if it's LDAP, then dlopen the implementation of
getpwnam and friends for LDAP; etc etc.

So that later when you call getpwnam, it will happen to "do the right
thing".

This demands the required *.so files to be accessible during run time.
Moreoever, if you statically link glibc, this also demands the required
*.so files to version-match the glibc you statically link.

(It is the main reason why most people give up on statically linking glibc.)
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Static executables in minimal Docker containers

Michael Snoyman
I have a bit more information about this. In particular: I'm able to reproduce this using chroot (no Docker required), and it's reproducing with a dynamically linked executable too. Steps I used to reproduce:

1. Write a minimal "foo.hs" containing `main = putStrLn "Hello World"`
2. Compile that executable and put it in an empty directory
3. Run `ldd` on it and copy all necessary libraries inside that directory
4. Run `sudo strace -o log.txt . /foo`

I've uploaded the logs to:


Note that, due to size of the output, I killed the process just a few seconds after starting it, but when I let the output run much longer, I didn't see any difference in the results. I'll continue poking at this a bit, but most likely I'll open a GHC Trac ticket about it later today.

On Tue, Apr 14, 2015 at 12:39 AM Albert Y. C. Lai <[hidden email]> wrote:
I wonder whether you already know the following, and whether it is
relevant to begin with. (Plus, my knowledge is fairly sketchy.)

Even though you statically link glibc, its code will, at run time,
dlopen a certain part of glibc.

Why: To provide a really uniform abstraction layer over user account
queries, e.g., man 3 getpwnam, regardless of whether the accounts are
from /etc/passwd, LDAP, or whatever.

Therefore, during run time, glibc first reads some config files of the
host to see what kind of user account database the host uses. If it's
/etc/passwd, then dlopen the implementation of getpwnam and friends for
/etc/passwd; else, if it's LDAP, then dlopen the implementation of
getpwnam and friends for LDAP; etc etc.

So that later when you call getpwnam, it will happen to "do the right
thing".

This demands the required *.so files to be accessible during run time.
Moreoever, if you statically link glibc, this also demands the required
*.so files to version-match the glibc you statically link.

(It is the main reason why most people give up on statically linking glibc.)
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Static executables in minimal Docker containers

Michael Snoyman
Actually, I seem to have found the problem:

open("/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/gconv/gconv-modules", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

I found that I needed to copy over the following files to make my program complete:

/usr/lib/x86_64-linux-gnu/gconv/gconv-modules
/usr/lib/x86_64-linux-gnu/gconv/UTF-32.so

Once I did that, I could get the executable to run in the chroot. However, even running the statically linked executable still required most of the shared libraries to be present inside the chroot. So it seems that:

* We can come up with a list of a few files that need to be present inside a Docker image to provide for minimal GHC-compiled executables
* There's a bug in the RTS that results in an infinite loop

I'm going to try to put together a semi-robust solution for the first problem, and I'll report the RTS issue on Trac.

On Tue, Apr 14, 2015 at 9:25 AM Michael Snoyman <[hidden email]> wrote:
I have a bit more information about this. In particular: I'm able to reproduce this using chroot (no Docker required), and it's reproducing with a dynamically linked executable too. Steps I used to reproduce:

1. Write a minimal "foo.hs" containing `main = putStrLn "Hello World"`
2. Compile that executable and put it in an empty directory
3. Run `ldd` on it and copy all necessary libraries inside that directory
4. Run `sudo strace -o log.txt . /foo`

I've uploaded the logs to:


Note that, due to size of the output, I killed the process just a few seconds after starting it, but when I let the output run much longer, I didn't see any difference in the results. I'll continue poking at this a bit, but most likely I'll open a GHC Trac ticket about it later today.

On Tue, Apr 14, 2015 at 12:39 AM Albert Y. C. Lai <[hidden email]> wrote:
I wonder whether you already know the following, and whether it is
relevant to begin with. (Plus, my knowledge is fairly sketchy.)

Even though you statically link glibc, its code will, at run time,
dlopen a certain part of glibc.

Why: To provide a really uniform abstraction layer over user account
queries, e.g., man 3 getpwnam, regardless of whether the accounts are
from /etc/passwd, LDAP, or whatever.

Therefore, during run time, glibc first reads some config files of the
host to see what kind of user account database the host uses. If it's
/etc/passwd, then dlopen the implementation of getpwnam and friends for
/etc/passwd; else, if it's LDAP, then dlopen the implementation of
getpwnam and friends for LDAP; etc etc.

So that later when you call getpwnam, it will happen to "do the right
thing".

This demands the required *.so files to be accessible during run time.
Moreoever, if you statically link glibc, this also demands the required
*.so files to version-match the glibc you statically link.

(It is the main reason why most people give up on statically linking glibc.)
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Static executables in minimal Docker containers

Magnus Therning
On 14 April 2015 at 08:52, Michael Snoyman <[hidden email]> wrote:

> Actually, I seem to have found the problem:
>
> open("/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache", O_RDONLY) = -1
> ENOENT (No such file or directory)
> open("/usr/lib/x86_64-linux-gnu/gconv/gconv-modules", O_RDONLY|O_CLOEXEC) =
> -1 ENOENT (No such file or directory)
>
> I found that I needed to copy over the following files to make my program
> complete:
>
> /usr/lib/x86_64-linux-gnu/gconv/gconv-modules
> /usr/lib/x86_64-linux-gnu/gconv/UTF-32.so
>
> Once I did that, I could get the executable to run in the chroot. However,
> even running the statically linked executable still required most of the
> shared libraries to be present inside the chroot. So it seems that:
>
> * We can come up with a list of a few files that need to be present inside a
> Docker image to provide for minimal GHC-compiled executables
> * There's a bug in the RTS that results in an infinite loop
>
> I'm going to try to put together a semi-robust solution for the first
> problem, and I'll report the RTS issue on Trac.

Excellent that you found the issue.

Is there some way of controlling which libc ghc links with?  There are
quite a few alternatives out there, and maybe it'd be easier to create
a *true* static binary by using another libc?

/M

--
Magnus Therning                      OpenPGP: 0xAB4DFBA4
email: [hidden email]   jabber: [hidden email]
twitter: magthe               http://therning.org/magnus
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Static executables in minimal Docker containers

Michael Snoyman
In reply to this post by Michael Snoyman
Trac ticket created: https://ghc.haskell.org/trac/ghc/ticket/10298#ticket

I've also put together a Docker image called snoyberg/haskell-scratch (source at https://github.com/snoyberg/haskell-scratch), which seems to be working for me. Here's a minimal test I've put together which seems to be succeeding (note that I've also tried some real life programs):

#!/bin/bash

set -e
set -x

cat > tiny.hs <<EOF
main :: IO ()
main = putStrLn "Hello from a tiny Docker image"
EOF

ghc tiny.hs
strip tiny

cat > Dockerfile <<EOF
FROM snoyberg/haskell-scratch
ADD tiny /tiny
CMD ["/tiny"]
EOF

docker build -t tiny .
docker run --rm tiny

On Tue, Apr 14, 2015 at 9:52 AM Michael Snoyman <[hidden email]> wrote:
Actually, I seem to have found the problem:

open("/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/gconv/gconv-modules", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

I found that I needed to copy over the following files to make my program complete:

/usr/lib/x86_64-linux-gnu/gconv/gconv-modules
/usr/lib/x86_64-linux-gnu/gconv/UTF-32.so

Once I did that, I could get the executable to run in the chroot. However, even running the statically linked executable still required most of the shared libraries to be present inside the chroot. So it seems that:

* We can come up with a list of a few files that need to be present inside a Docker image to provide for minimal GHC-compiled executables
* There's a bug in the RTS that results in an infinite loop

I'm going to try to put together a semi-robust solution for the first problem, and I'll report the RTS issue on Trac.

On Tue, Apr 14, 2015 at 9:25 AM Michael Snoyman <[hidden email]> wrote:
I have a bit more information about this. In particular: I'm able to reproduce this using chroot (no Docker required), and it's reproducing with a dynamically linked executable too. Steps I used to reproduce:

1. Write a minimal "foo.hs" containing `main = putStrLn "Hello World"`
2. Compile that executable and put it in an empty directory
3. Run `ldd` on it and copy all necessary libraries inside that directory
4. Run `sudo strace -o log.txt . /foo`

I've uploaded the logs to:


Note that, due to size of the output, I killed the process just a few seconds after starting it, but when I let the output run much longer, I didn't see any difference in the results. I'll continue poking at this a bit, but most likely I'll open a GHC Trac ticket about it later today.

On Tue, Apr 14, 2015 at 12:39 AM Albert Y. C. Lai <[hidden email]> wrote:
I wonder whether you already know the following, and whether it is
relevant to begin with. (Plus, my knowledge is fairly sketchy.)

Even though you statically link glibc, its code will, at run time,
dlopen a certain part of glibc.

Why: To provide a really uniform abstraction layer over user account
queries, e.g., man 3 getpwnam, regardless of whether the accounts are
from /etc/passwd, LDAP, or whatever.

Therefore, during run time, glibc first reads some config files of the
host to see what kind of user account database the host uses. If it's
/etc/passwd, then dlopen the implementation of getpwnam and friends for
/etc/passwd; else, if it's LDAP, then dlopen the implementation of
getpwnam and friends for LDAP; etc etc.

So that later when you call getpwnam, it will happen to "do the right
thing".

This demands the required *.so files to be accessible during run time.
Moreoever, if you statically link glibc, this also demands the required
*.so files to version-match the glibc you statically link.

(It is the main reason why most people give up on statically linking glibc.)
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Static executables in minimal Docker containers

Jon Schneider
In reply to this post by Michael Snoyman
These ring a bell. I had pain with exactly this shared library (or lack
of) when playing with the output of cross-compiling ghc.

> /usr/lib/x86_64-linux-gnu/gconv/UTF-32.so

Jon

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Static executables in minimal Docker containers

Vo Minh Thu
In reply to this post by Michael Snoyman
Michael, thanks for solving this !

I wonder how you came up with the need for /lib/ld-linux-x86-64.so.2.
I have needed only /lib64/ld-linux-x86-64.so.2 (which you have too),
so maybe you added it by mistake.

B.t.w, if you don't want to use a Dockerfile, you can do this:

    > tar -C rootfs -c . | docker import - tiny

where rootfs is your repository.

2015-04-14 10:43 GMT+02:00 Michael Snoyman <[hidden email]>:

> Trac ticket created: https://ghc.haskell.org/trac/ghc/ticket/10298#ticket
>
> I've also put together a Docker image called snoyberg/haskell-scratch
> (source at https://github.com/snoyberg/haskell-scratch), which seems to be
> working for me. Here's a minimal test I've put together which seems to be
> succeeding (note that I've also tried some real life programs):
>
> #!/bin/bash
>
> set -e
> set -x
>
> cat > tiny.hs <<EOF
> main :: IO ()
> main = putStrLn "Hello from a tiny Docker image"
> EOF
>
> ghc tiny.hs
> strip tiny
>
> cat > Dockerfile <<EOF
> FROM snoyberg/haskell-scratch
> ADD tiny /tiny
> CMD ["/tiny"]
> EOF
>
> docker build -t tiny .
> docker run --rm tiny
>
>
> On Tue, Apr 14, 2015 at 9:52 AM Michael Snoyman <[hidden email]> wrote:
>>
>> Actually, I seem to have found the problem:
>>
>> open("/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache", O_RDONLY) = -1
>> ENOENT (No such file or directory)
>> open("/usr/lib/x86_64-linux-gnu/gconv/gconv-modules", O_RDONLY|O_CLOEXEC)
>> = -1 ENOENT (No such file or directory)
>>
>> I found that I needed to copy over the following files to make my program
>> complete:
>>
>> /usr/lib/x86_64-linux-gnu/gconv/gconv-modules
>> /usr/lib/x86_64-linux-gnu/gconv/UTF-32.so
>>
>> Once I did that, I could get the executable to run in the chroot. However,
>> even running the statically linked executable still required most of the
>> shared libraries to be present inside the chroot. So it seems that:
>>
>> * We can come up with a list of a few files that need to be present inside
>> a Docker image to provide for minimal GHC-compiled executables
>> * There's a bug in the RTS that results in an infinite loop
>>
>> I'm going to try to put together a semi-robust solution for the first
>> problem, and I'll report the RTS issue on Trac.
>>
>> On Tue, Apr 14, 2015 at 9:25 AM Michael Snoyman <[hidden email]>
>> wrote:
>>>
>>> I have a bit more information about this. In particular: I'm able to
>>> reproduce this using chroot (no Docker required), and it's reproducing with
>>> a dynamically linked executable too. Steps I used to reproduce:
>>>
>>> 1. Write a minimal "foo.hs" containing `main = putStrLn "Hello World"`
>>> 2. Compile that executable and put it in an empty directory
>>> 3. Run `ldd` on it and copy all necessary libraries inside that directory
>>> 4. Run `sudo strace -o log.txt . /foo`
>>>
>>> I've uploaded the logs to:
>>>
>>> https://gist.github.com/snoyberg/095efb17e36acc1d6360
>>>
>>> Note that, due to size of the output, I killed the process just a few
>>> seconds after starting it, but when I let the output run much longer, I
>>> didn't see any difference in the results. I'll continue poking at this a
>>> bit, but most likely I'll open a GHC Trac ticket about it later today.
>>>
>>> On Tue, Apr 14, 2015 at 12:39 AM Albert Y. C. Lai <[hidden email]> wrote:
>>>>
>>>> I wonder whether you already know the following, and whether it is
>>>> relevant to begin with. (Plus, my knowledge is fairly sketchy.)
>>>>
>>>> Even though you statically link glibc, its code will, at run time,
>>>> dlopen a certain part of glibc.
>>>>
>>>> Why: To provide a really uniform abstraction layer over user account
>>>> queries, e.g., man 3 getpwnam, regardless of whether the accounts are
>>>> from /etc/passwd, LDAP, or whatever.
>>>>
>>>> Therefore, during run time, glibc first reads some config files of the
>>>> host to see what kind of user account database the host uses. If it's
>>>> /etc/passwd, then dlopen the implementation of getpwnam and friends for
>>>> /etc/passwd; else, if it's LDAP, then dlopen the implementation of
>>>> getpwnam and friends for LDAP; etc etc.
>>>>
>>>> So that later when you call getpwnam, it will happen to "do the right
>>>> thing".
>>>>
>>>> This demands the required *.so files to be accessible during run time.
>>>> Moreoever, if you statically link glibc, this also demands the required
>>>> *.so files to version-match the glibc you statically link.
>>>>
>>>> (It is the main reason why most people give up on statically linking
>>>> glibc.)
>>>> _______________________________________________
>>>> Haskell-Cafe mailing list
>>>> [hidden email]
>>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
>
>
> _______________________________________________
> Haskell-Cafe mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
>
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Static executables in minimal Docker containers

Michael Snoyman

The /lib/ version of the file was necessary for statically linked executables I believe. It's arguably better to just remove it and tell people "don't do static executables" I suppose, given that they're clearly not *actually* static in reality ;)


On Sat, Apr 18, 2015, 12:51 PM Vo Minh Thu <[hidden email]> wrote:
Michael, thanks for solving this !

I wonder how you came up with the need for /lib/ld-linux-x86-64.so.2.
I have needed only /lib64/ld-linux-x86-64.so.2 (which you have too),
so maybe you added it by mistake.

B.t.w, if you don't want to use a Dockerfile, you can do this:

    > tar -C rootfs -c . | docker import - tiny

where rootfs is your repository.

2015-04-14 10:43 GMT+02:00 Michael Snoyman <[hidden email]>:
> Trac ticket created: https://ghc.haskell.org/trac/ghc/ticket/10298#ticket
>
> I've also put together a Docker image called snoyberg/haskell-scratch
> (source at https://github.com/snoyberg/haskell-scratch), which seems to be
> working for me. Here's a minimal test I've put together which seems to be
> succeeding (note that I've also tried some real life programs):
>
> #!/bin/bash
>
> set -e
> set -x
>
> cat > tiny.hs <<EOF
> main :: IO ()
> main = putStrLn "Hello from a tiny Docker image"
> EOF
>
> ghc tiny.hs
> strip tiny
>
> cat > Dockerfile <<EOF
> FROM snoyberg/haskell-scratch
> ADD tiny /tiny
> CMD ["/tiny"]
> EOF
>
> docker build -t tiny .
> docker run --rm tiny
>
>
> On Tue, Apr 14, 2015 at 9:52 AM Michael Snoyman <[hidden email]> wrote:
>>
>> Actually, I seem to have found the problem:
>>
>> open("/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache", O_RDONLY) = -1
>> ENOENT (No such file or directory)
>> open("/usr/lib/x86_64-linux-gnu/gconv/gconv-modules", O_RDONLY|O_CLOEXEC)
>> = -1 ENOENT (No such file or directory)
>>
>> I found that I needed to copy over the following files to make my program
>> complete:
>>
>> /usr/lib/x86_64-linux-gnu/gconv/gconv-modules
>> /usr/lib/x86_64-linux-gnu/gconv/UTF-32.so
>>
>> Once I did that, I could get the executable to run in the chroot. However,
>> even running the statically linked executable still required most of the
>> shared libraries to be present inside the chroot. So it seems that:
>>
>> * We can come up with a list of a few files that need to be present inside
>> a Docker image to provide for minimal GHC-compiled executables
>> * There's a bug in the RTS that results in an infinite loop
>>
>> I'm going to try to put together a semi-robust solution for the first
>> problem, and I'll report the RTS issue on Trac.
>>
>> On Tue, Apr 14, 2015 at 9:25 AM Michael Snoyman <[hidden email]>
>> wrote:
>>>
>>> I have a bit more information about this. In particular: I'm able to
>>> reproduce this using chroot (no Docker required), and it's reproducing with
>>> a dynamically linked executable too. Steps I used to reproduce:
>>>
>>> 1. Write a minimal "foo.hs" containing `main = putStrLn "Hello World"`
>>> 2. Compile that executable and put it in an empty directory
>>> 3. Run `ldd` on it and copy all necessary libraries inside that directory
>>> 4. Run `sudo strace -o log.txt . /foo`
>>>
>>> I've uploaded the logs to:
>>>
>>> https://gist.github.com/snoyberg/095efb17e36acc1d6360
>>>
>>> Note that, due to size of the output, I killed the process just a few
>>> seconds after starting it, but when I let the output run much longer, I
>>> didn't see any difference in the results. I'll continue poking at this a
>>> bit, but most likely I'll open a GHC Trac ticket about it later today.
>>>
>>> On Tue, Apr 14, 2015 at 12:39 AM Albert Y. C. Lai <[hidden email]> wrote:
>>>>
>>>> I wonder whether you already know the following, and whether it is
>>>> relevant to begin with. (Plus, my knowledge is fairly sketchy.)
>>>>
>>>> Even though you statically link glibc, its code will, at run time,
>>>> dlopen a certain part of glibc.
>>>>
>>>> Why: To provide a really uniform abstraction layer over user account
>>>> queries, e.g., man 3 getpwnam, regardless of whether the accounts are
>>>> from /etc/passwd, LDAP, or whatever.
>>>>
>>>> Therefore, during run time, glibc first reads some config files of the
>>>> host to see what kind of user account database the host uses. If it's
>>>> /etc/passwd, then dlopen the implementation of getpwnam and friends for
>>>> /etc/passwd; else, if it's LDAP, then dlopen the implementation of
>>>> getpwnam and friends for LDAP; etc etc.
>>>>
>>>> So that later when you call getpwnam, it will happen to "do the right
>>>> thing".
>>>>
>>>> This demands the required *.so files to be accessible during run time.
>>>> Moreoever, if you statically link glibc, this also demands the required
>>>> *.so files to version-match the glibc you statically link.
>>>>
>>>> (It is the main reason why most people give up on statically linking
>>>> glibc.)
>>>> _______________________________________________
>>>> Haskell-Cafe mailing list
>>>> [hidden email]
>>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
>
>
> _______________________________________________
> Haskell-Cafe mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
>

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Static executables in minimal Docker containers

Simon Marlow-7
In reply to this post by Michael Snoyman
Hi Michael,

This rang a bell for me.  It might be the same as these:
https://ghc.haskell.org/trac/ghc/ticket/7695
https://ghc.haskell.org/trac/ghc/ticket/8928

I think the conclusion was that the IO library is failing to start
iconv, and printing the error messages causes it to retry loading iconv,
ad infinitum (or something like that).  There's no fix yet, but it
probably isn't hard to fix, just that nobody got around to it yet.

Cheers,
Simon

On 13/04/2015 11:50, Michael Snoyman wrote:

> I'm not sure if this issue would show up, but I can try it in Fedora
> tomorrow. I didn't address the linker warning at all right now, it seems
> to not have been triggered, though I suppose it is possible that it's
> the cause of this issue.
>
> On Mon, Apr 13, 2015 at 7:10 PM Greg Weber <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Haskell is not that great at producing statically linked libraries
>     independent of the OS.
>     The issue you are running into would likely show up in another
>     non-ubuntu image (or even possibly a different version of an ubuntu
>     image), so you could probably use a Fedora image that has tracing.
>
>     How are you addressing the linker warning about needing a particular
>     glibc version at runtime?
>
>     On Mon, Apr 13, 2015 at 3:28 AM, Sharif Olorin
>     <[hidden email] <mailto:[hidden email]>> wrote:
>
>             Unfortunately, strace and ltrace aren't available in that
>             Docker image, but it's a good idea to see if I can get them
>             running there somehow.
>
>
>         Failing that, you might be able to get useful information of the
>         same kind by running docker (the server, not the `docker run`
>         command) under perf[0] and then running your busybox container.
>         It should at least give you an idea of what it's doing when it
>         explodes.
>
>         Sharif
>
>         [0]: https://perf.wiki.kernel.org/index.php/Tutorial
>
>         --
>         You received this message because you are subscribed to the
>         Google Groups "Commercial Haskell" group.
>        
>         it, send an email to
>         [hidden email]
>         <mailto:[hidden email]>.
>        
>         [hidden email]
>         <mailto:[hidden email]>.
>        
>         https://groups.google.com/d/msgid/commercialhaskell/86ca2603-37f2-4645-9cd2-f09703f2be67%40googlegroups.com
>         <https://groups.google.com/d/msgid/commercialhaskell/86ca2603-37f2-4645-9cd2-f09703f2be67%40googlegroups.com?utm_medium=email&utm_source=footer>.
>
>        
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Commercial Haskell" group.
>
> an email to [hidden email]
> <mailto:[hidden email]>.
>
> <mailto:[hidden email]>.
>
> https://groups.google.com/d/msgid/commercialhaskell/CAKA2Jg%2B%3DzJiXmak2FU_5GWPO1Dcn%2BvwsiB_xWj%2B8GfHvMkoBjw%40mail.gmail.com
> <https://groups.google.com/d/msgid/commercialhaskell/CAKA2Jg%2B%3DzJiXmak2FU_5GWPO1Dcn%2BvwsiB_xWj%2B8GfHvMkoBjw%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Static executables in minimal Docker containers

Michael Snoyman
Thanks for the update Simon.

On Sun, Apr 19, 2015 at 10:54 PM Simon Marlow <[hidden email]> wrote:
Hi Michael,

This rang a bell for me.  It might be the same as these:
https://ghc.haskell.org/trac/ghc/ticket/7695
https://ghc.haskell.org/trac/ghc/ticket/8928

I think the conclusion was that the IO library is failing to start
iconv, and printing the error messages causes it to retry loading iconv,
ad infinitum (or something like that).  There's no fix yet, but it
probably isn't hard to fix, just that nobody got around to it yet.

Cheers,
Simon

On 13/04/2015 11:50, Michael Snoyman wrote:
> I'm not sure if this issue would show up, but I can try it in Fedora
> tomorrow. I didn't address the linker warning at all right now, it seems
> to not have been triggered, though I suppose it is possible that it's
> the cause of this issue.
>
> On Mon, Apr 13, 2015 at 7:10 PM Greg Weber <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Haskell is not that great at producing statically linked libraries
>     independent of the OS.
>     The issue you are running into would likely show up in another
>     non-ubuntu image (or even possibly a different version of an ubuntu
>     image), so you could probably use a Fedora image that has tracing.
>
>     How are you addressing the linker warning about needing a particular
>     glibc version at runtime?
>
>     On Mon, Apr 13, 2015 at 3:28 AM, Sharif Olorin
>     <[hidden email] <mailto:[hidden email]>> wrote:
>
>             Unfortunately, strace and ltrace aren't available in that
>             Docker image, but it's a good idea to see if I can get them
>             running there somehow.
>
>
>         Failing that, you might be able to get useful information of the
>         same kind by running docker (the server, not the `docker run`
>         command) under perf[0] and then running your busybox container.
>         It should at least give you an idea of what it's doing when it
>         explodes.
>
>         Sharif
>
>         [0]: https://perf.wiki.kernel.org/index.php/Tutorial
>
>         --
>         You received this message because you are subscribed to the
>         Google Groups "Commercial Haskell" group.
>         >         it, send an email to
>         [hidden email]
>         <mailto:[hidden email]>.
>         >         [hidden email]
>         <mailto:[hidden email]>.
>         >         https://groups.google.com/d/msgid/commercialhaskell/86ca2603-37f2-4645-9cd2-f09703f2be67%40googlegroups.com
>         <https://groups.google.com/d/msgid/commercialhaskell/86ca2603-37f2-4645-9cd2-f09703f2be67%40googlegroups.com?utm_medium=email&utm_source=footer>.
>
>         >
>
> --
> You received this message because you are subscribed to the Google
> Groups "Commercial Haskell" group.
> > an email to [hidden email]
> <mailto:[hidden email]>.
> > <mailto:[hidden email]>.
> > https://groups.google.com/d/msgid/commercialhaskell/CAKA2Jg%2B%3DzJiXmak2FU_5GWPO1Dcn%2BvwsiB_xWj%2B8GfHvMkoBjw%40mail.gmail.com
> <https://groups.google.com/d/msgid/commercialhaskell/CAKA2Jg%2B%3DzJiXmak2FU_5GWPO1Dcn%2BvwsiB_xWj%2B8GfHvMkoBjw%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>
--

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Static executables in minimal Docker containers

Sylvain HENRY
Hi,

I had the same bug with a single static binary in a initramfs. I fixed GHC.IO.Encoding to avoid using Iconv for the default (ASCII) charmap used with static binaries and now it works. Patch is in my comment on https://ghc.haskell.org/trac/ghc/ticket/10298

If the default LC_CTYPE value is the DEFAULT_CHARMAP from glibc/locale/programs/config.h, it is set to "ANSI_X3.4-1968" since 2000 and was set to "POSIX" before. Maybe we should match this one too?

Sylvain

2015-04-20 6:34 GMT+02:00 Michael Snoyman <[hidden email]>:
Thanks for the update Simon.

On Sun, Apr 19, 2015 at 10:54 PM Simon Marlow <[hidden email]> wrote:
Hi Michael,

This rang a bell for me.  It might be the same as these:
https://ghc.haskell.org/trac/ghc/ticket/7695
https://ghc.haskell.org/trac/ghc/ticket/8928

I think the conclusion was that the IO library is failing to start
iconv, and printing the error messages causes it to retry loading iconv,
ad infinitum (or something like that).  There's no fix yet, but it
probably isn't hard to fix, just that nobody got around to it yet.

Cheers,
Simon

On 13/04/2015 11:50, Michael Snoyman wrote:
> I'm not sure if this issue would show up, but I can try it in Fedora
> tomorrow. I didn't address the linker warning at all right now, it seems
> to not have been triggered, though I suppose it is possible that it's
> the cause of this issue.
>
> On Mon, Apr 13, 2015 at 7:10 PM Greg Weber <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Haskell is not that great at producing statically linked libraries
>     independent of the OS.
>     The issue you are running into would likely show up in another
>     non-ubuntu image (or even possibly a different version of an ubuntu
>     image), so you could probably use a Fedora image that has tracing.
>
>     How are you addressing the linker warning about needing a particular
>     glibc version at runtime?
>
>     On Mon, Apr 13, 2015 at 3:28 AM, Sharif Olorin
>     <[hidden email] <mailto:[hidden email]>> wrote:
>
>             Unfortunately, strace and ltrace aren't available in that
>             Docker image, but it's a good idea to see if I can get them
>             running there somehow.
>
>
>         Failing that, you might be able to get useful information of the
>         same kind by running docker (the server, not the `docker run`
>         command) under perf[0] and then running your busybox container.
>         It should at least give you an idea of what it's doing when it
>         explodes.
>
>         Sharif
>
>         [0]: https://perf.wiki.kernel.org/index.php/Tutorial
>
>         --
>         You received this message because you are subscribed to the
>         Google Groups "Commercial Haskell" group.
>         >         it, send an email to
>         [hidden email]
>         <mailto:[hidden email]>.
>         >         [hidden email]
>         <mailto:[hidden email]>.
>         >         https://groups.google.com/d/msgid/commercialhaskell/86ca2603-37f2-4645-9cd2-f09703f2be67%40googlegroups.com
>         <https://groups.google.com/d/msgid/commercialhaskell/86ca2603-37f2-4645-9cd2-f09703f2be67%40googlegroups.com?utm_medium=email&utm_source=footer>.
>
>         >
>
> --
> You received this message because you are subscribed to the Google
> Groups "Commercial Haskell" group.
> > an email to [hidden email]
> <mailto:[hidden email]>.
> > <mailto:[hidden email]>.
> > https://groups.google.com/d/msgid/commercialhaskell/CAKA2Jg%2B%3DzJiXmak2FU_5GWPO1Dcn%2BvwsiB_xWj%2B8GfHvMkoBjw%40mail.gmail.com
> <https://groups.google.com/d/msgid/commercialhaskell/CAKA2Jg%2B%3DzJiXmak2FU_5GWPO1Dcn%2BvwsiB_xWj%2B8GfHvMkoBjw%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>
--

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe



_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe