cabal keeps relinking

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

cabal keeps relinking

trygub
Dear Haskell-Cafe,

cabal seems to be re-linking every time I run

cabal build

Wait was not an issue for small projects but for big ones
it is starting to get in the way… How could I persuade
cabal to re-link only if there are changes in the source tree
and only these exes that require it?

Many thanks,
Semen

PS I googled around and found this

https://www.haskell.org/pipermail/libraries/2005-August/004306.html

which seems to imply relink is (was?) conditional on multiple exes being
present in the project. However, for me it was always relinking regardless.



--
Семен Тригубенко http://trygub.com

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

attachment0 (188 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: cabal keeps relinking

Niklas Hambüchen
Which cabal version is it?

If older than 1.20, this is the fix that went into 1.20:
  https://github.com/haskell/cabal/issues/1177

On 07/11/14 16:35, Semen Trygubenko / Семен Тригубенко wrote:
> cabal seems to be re-linking every time I run
> cabal build
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: cabal keeps relinking

trygub
On Fri, Nov 07, 2014 at 07:23:16PM +0100, Niklas Hambüchen wrote:
> Which cabal version is it?
>
> If older than 1.20, this is the fix that went into 1.20:
>   https://github.com/haskell/cabal/issues/1177

$ cabal --version
cabal-install version 1.20.0.3
using version 1.20.0.2 of the Cabal library



--
Семен Тригубенко http://trygub.com

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

attachment0 (188 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: cabal keeps relinking

Niklas Hambüchen
Could you detail how cabal relinks for you, e.g. with some cabal -v2 /
-v3 output?

On 07/11/14 21:09, Semen Trygubenko / Семен Тригубенко wrote:

> On Fri, Nov 07, 2014 at 07:23:16PM +0100, Niklas Hambüchen wrote:
>> Which cabal version is it?
>>
>> If older than 1.20, this is the fix that went into 1.20:
>> https://github.com/haskell/cabal/issues/1177
>
> $ cabal --version cabal-install version 1.20.0.3 using version
> 1.20.0.2 of the Cabal library
>
>
>
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: cabal keeps relinking

trygub
On Fri, Nov 07, 2014 at 10:16:31PM +0100, Niklas Hambüchen wrote:
> Could you detail how cabal relinks for you, e.g. with some cabal -v2 /
> -v3 output?

The -vN outputs are verbose, so I ended up creating a minimalistic
cabal package config that illustrates the problem (attached for reference
— see cabalTest.tar.gz).

This is how the non-verbose outputs look like for builds on FreeBSD:

$ cabal build
Package has never been configured. Configuring with default flags. If this
fails, please run configure manually.
Resolving dependencies...
Configuring cabalTest-0.1.0.0...
Building cabalTest-0.1.0.0...
Preprocessing library cabalTest-0.1.0.0...
[1 of 1] Compiling CabalTest        ( src/CabalTest.hs, dist/build/CabalTest.o )
In-place registering cabalTest-0.1.0.0...
Preprocessing executable 'cabalTest' for cabalTest-0.1.0.0...
[1 of 2] Compiling CabalTest        ( src/CabalTest.hs, dist/build/cabalTest/cabalTest-tmp/CabalTest.o )
[2 of 2] Compiling Main             ( src/cabalTest.hs, dist/build/cabalTest/cabalTest-tmp/Main.o )
Linking dist/build/cabalTest/cabalTest ...

$ cabal clean
cleaning...

$ cabal test
Package has never been configured. Configuring with default flags. If this
fails, please run configure manually.
Resolving dependencies...
Configuring cabalTest-0.1.0.0...
Building cabalTest-0.1.0.0...
Preprocessing library cabalTest-0.1.0.0...
[1 of 1] Compiling CabalTest        ( src/CabalTest.hs, dist/build/CabalTest.o )
In-place registering cabalTest-0.1.0.0...
Preprocessing executable 'cabalTest' for cabalTest-0.1.0.0...
[1 of 2] Compiling CabalTest        ( src/CabalTest.hs, dist/build/cabalTest/cabalTest-tmp/CabalTest.o )
[2 of 2] Compiling Main             ( src/cabalTest.hs, dist/build/cabalTest/cabalTest-tmp/Main.o )
Linking dist/build/cabalTest/cabalTest ...
Preprocessing test suite 'CabalTestTest' for cabalTest-0.1.0.0...
[1 of 1] Compiling Main             ( test/cabalTest.hs, dist/build/CabalTestTest/CabalTestTest-tmp/Main.o )
Linking dist/build/CabalTestTest/CabalTestTest ...
Running 1 test suites...
Test suite CabalTestTest: RUNNING...
Test suite CabalTestTest: PASS
Test suite logged to: dist/test/cabalTest-0.1.0.0-CabalTestTest.log
1 of 1 test suites (1 of 1 test cases) passed.


and then for rebuilds:

$ cabal build
Building cabalTest-0.1.0.0...
Preprocessing library cabalTest-0.1.0.0...
In-place registering cabalTest-0.1.0.0...
Preprocessing executable 'cabalTest' for cabalTest-0.1.0.0...
Linking dist/build/cabalTest/cabalTest ...
Preprocessing test suite 'CabalTestTest' for cabalTest-0.1.0.0...
Linking dist/build/CabalTestTest/CabalTestTest ...

$ cabal test
Building cabalTest-0.1.0.0...
Preprocessing library cabalTest-0.1.0.0...
In-place registering cabalTest-0.1.0.0...
Preprocessing executable 'cabalTest' for cabalTest-0.1.0.0...
Linking dist/build/cabalTest/cabalTest ...
Preprocessing test suite 'CabalTestTest' for cabalTest-0.1.0.0...
Linking dist/build/CabalTestTest/CabalTestTest ...
Running 1 test suites...
Test suite CabalTestTest: RUNNING...
Test suite CabalTestTest: PASS
Test suite logged to: dist/test/cabalTest-0.1.0.0-CabalTestTest.log
1 of 1 test suites (1 of 1 test cases) passed.

Then I ran the following five commands:

cabal clean
cabal build
cabal test
cabal build
cabal test

three times --- first without, then with -v2 and -v3, on two platforms.
The outputs are here:

http://trygub.com/download/cabalTest-FreeBSD.out.gz
http://trygub.com/download/cabalTest-Linux.out.gz

The behaviour is different on Linux and FreeBSD. This is the diff of the non-verbose outputs:

10a11
> [1 of 1] Compiling CabalTest        ( src/CabalTest.hs, dist/build/CabalTest.p_o )
14a16,17
> [1 of 2] Compiling CabalTest        ( src/CabalTest.hs, dist/build/cabalTest/cabalTest-tmp/CabalTest.p_o )
> [2 of 2] Compiling Main             ( src/cabalTest.hs, dist/build/cabalTest/cabalTest-tmp/Main.p_o )
25d27
< Linking dist/build/cabalTest/cabalTest ...
27c29
< [1 of 1] Compiling Main             ( test/cabalTest.hs, dist/build/CabalTestTest/CabalTestTest-tmp/Main.o )
---
> [1 of 1] Compiling Main             ( test/cabalTest.hs, dist/build/CabalTestTest/CabalTestTest-tmp/Main.p_o )
39d40
< Linking dist/build/cabalTest/cabalTest ...
41d41
< Linking dist/build/CabalTestTest/CabalTestTest ...
47d46
< Linking dist/build/cabalTest/cabalTest ...
49d47
< Linking dist/build/CabalTestTest/CabalTestTest ...

So it seems that what I've observed is FreeBSD-specific — FreeBSD rebuild outputs have extra "Linking ..." messages in them…

Many thanks for your help,
Semen

PS

Linux $ cabal --version
cabal-install version 1.18.0.2
using version 1.18.1.2 of the Cabal library

FreeBSD $ cabal --version
cabal-install version 1.20.0.3
using version 1.20.0.2 of the Cabal library


--
Семен Тригубенко http://trygub.com

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

cabalTest.tar.gz (1K) Download Attachment
attachment1 (188 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: cabal keeps relinking

Niklas Hambüchen
Ok, so it seems like it works on Linux, but not FreeBSD.

To explain the situation a little bit:

The compiled object files (.o) are linked together in an archive file
(.a) by the `ar` program (similar to `tar`).
The file entries in the .a file by default contain timestamps of the
files, which usually means the mtime (last modification time).
The build only looks at the .a file as a whole, and if its contents
changed from the last build, it has to relink. Of course if a timestamp
inside the .a changed, that's treated as "the .a file is different".

Newer versions of `ar` have a -D option to avoid putting the mtimes of
the .o files into the archive. But not all platforms supported by
GHC/cabal support this flag. That's why we implemented a manual method
to wipe the timestamps (set them to 0), so that they can no longer have
an effect on the build.

Seems like that doesn't work for FreeBSD. That's a shame, because we did
take special care to make sure that it works cross-platform.

You could help debug it:

First, create an issue on cabal's Github so that we can track the bug,
and explain the problem there and paste the output.
Then try to find the craeated .a file (the -v outputs should help you
where it is, something like libHSyourlibrary.a or similar), and observe
how its contents change across different builds (you could use `cmp` to
check if the files are byte-identical or something like `diff -u <(xxd
firstbuild.a) <(xxd secondbuild.a)` to view the contents as a hex diff).

Then try to see if this program I wrote:
  http://hackage.haskell.org/package/ar-timestamp-wiper
makes the two .a files byte-identical.

That will probably help us find the issue and get relinking-avoidance
also on the BSDs.

On 11/11/14 13:01, Semen Trygubenko / Семен Тригубенко wrote:
> So it seems that what I've observed is FreeBSD-specific — FreeBSD rebuild outputs have extra "Linking ..." messages in them…
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: cabal keeps relinking

trygub
On Tue, Nov 11, 2014 at 08:28:26PM +0100, Niklas Hambüchen wrote:

> Ok, so it seems like it works on Linux, but not FreeBSD.
>
> To explain the situation a little bit:
>
> The compiled object files (.o) are linked together in an archive file
> (.a) by the `ar` program (similar to `tar`).
> The file entries in the .a file by default contain timestamps of the
> files, which usually means the mtime (last modification time).
> The build only looks at the .a file as a whole, and if its contents
> changed from the last build, it has to relink. Of course if a timestamp
> inside the .a changed, that's treated as "the .a file is different".
>
> Newer versions of `ar` have a -D option to avoid putting the mtimes of
> the .o files into the archive. But not all platforms supported by
> GHC/cabal support this flag. That's why we implemented a manual method
> to wipe the timestamps (set them to 0), so that they can no longer have
> an effect on the build.
>
> Seems like that doesn't work for FreeBSD. That's a shame, because we did
> take special care to make sure that it works cross-platform.
Thanks for your reply!


> You could help debug it:
>
> First, create an issue on cabal's Github so that we can track the bug,
> and explain the problem there and paste the output.

Done — see https://github.com/haskell/cabal/issues/2384 .

Also, I've re-run it with the latest cabal

cabal-install version 1.22.0.0
using version 1.22.0.0 of the Cabal library

and noticed a bit of a difference in behaviour this time. Namely, for the sequence
I've ran (as before):

cabal clean
cabal build
cabal test
cabal build
cabal test

There's an improvement in both "cabal test" steps when moving to 1.22 in the sense
that the following lines have disappeared from the output:

Building cabalTest-0.1.0.0...
Preprocessing executable 'cabalTest' for cabalTest-0.1.0.0...
Linking dist/build/cabalTest/cabalTest ...

(i.e., when running test target the exe itself (named cabalTest) is no longer relinked unnecessarily,
only the test for it (named cabalTestTest).

However, when compared with old output on Linux there are still some
extra relinks present:

For the second run of "cabal bulid" I still see

Linking dist/build/cabalTest/cabalTest …
Linking dist/build/CabalTestTest/CabalTestTest …

For the second run of "cabal test" I still see

Linking dist/build/CabalTestTest/CabalTestTest ...

So it seems that the problem is still present in the latest version,
it's just that test target no longer relinks the exes, only test exes.


> Then try to find the created .a file (the -v outputs should help you
> where it is, something like libHSyourlibrary.a or similar), and observe
> how its contents change across different builds (you could use `cmp` to
> check if the files are byte-identical or something like `diff -u <(xxd
> firstbuild.a) <(xxd secondbuild.a)` to view the contents as a hex diff).

I ran this:

cabal clean
cabal build
cp ./dist/build/libHScabalTest-0.1.0.0.a build.a
cabal test
cp ./dist/build/libHScabalTest-0.1.0.0.a test.a
cabal build
cp ./dist/build/libHScabalTest-0.1.0.0.a rebuild.a
cabal test
cp ./dist/build/libHScabalTest-0.1.0.0.a retest.a

All .a files are identical.


> Then try to see if this program I wrote:
>   http://hackage.haskell.org/package/ar-timestamp-wiper
> makes the two .a files byte-identical.

Installed ar-timestamp-wiper-0.1.0 .
Running e.g.

ar-timestamp-wiper build.a

does not change its md5.


> That will probably help us find the issue and get relinking-avoidance
> also on the BSDs.
>
> On 11/11/14 13:01, Semen Trygubenko / Семен Тригубенко wrote:
> > So it seems that what I've observed is FreeBSD-specific — FreeBSD rebuild outputs have extra "Linking ..." messages in them…

Let me know if I could be of further help!

Thanks again,
S.


--
Семен Тригубенко http://trygub.com

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

attachment0 (188 bytes) Download Attachment