Unreliability of the build system

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
40 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Unreliability of the build system

Jan Stolarek
Fellow developers,

I need advice on using GHC build system. Recently I've been spending more time on resolving
problems with the build system itself than on developing GHC, and this is not how things are
supposed to be. I have found testsuite and validation to be very unreliable - things seem to
break randomly and results are not repeatable. I suspect that I might be doing something wrong
and I will appreciate any advice or explanations why I'm having these problem.

I have three GHC trees:
  * ghc-working  - source tree I use for development
  * ghc-build    - lndir'ed to ghc-working, used for building
  * ghc-validate - source tree used for validation, ghc-working is set up as one of the remotes

Here's the situation from yesterday evening. I updated ghc-working and ghc-validate trees from the
origin - according to the fingerprint.py script both trees were identical. I ran validation in
ghc-validate and ghc-build trees (on the master branches, no my changes). The only difference
between them should be that ghc-build is symlinked. Validation in ghc-validate passed with two
failures (others on IRC confirmed they are having same failures), while validation in ghc-build
failed with:

Haddock coverage:
 100% (  1 /  1) in 'GHC.IntWord64'
  78% (  7 /  9) in 'GHC.Types'
  17% (  1 /  6) in 'GHC.CString'
   3% (  2 / 63) in 'GHC.Tuple'
   0% (  0 /  3) in 'GHC.Debug'
   0% (  0 /342) in 'GHC.PrimopWrappers'
  30% (160 /531) in 'GHC.Prim'
 100% (  3 /  3) in 'GHC.Magic'
  38% (  6 / 16) in 'GHC.Classes'
haddock: internal error: /dane/uczelnia/projekty/ghc-build/inpl
ace/lib/html/Ocean.std-theme/ocean.css: copyFile: does not exis
t (No such file or directory)
gmake[1]: *** [libraries/ghc-prim/dist-install/doc/html/ghc-pri
m/ghc-prim.haddock] B??d 1
gmake[1]: *** Kasuj? plik `libraries/ghc-prim/dist-install/doc/
html/ghc-prim/ghc-prim.haddock'

Today I ran ./sync-all pull in both trees, but no new commits were added (according to
fingerprint). I run the validation again and the validation in ghc-build fails as previously,
while the validation in ghc-validate reports over 200 unexpected failures. What is more, I found
that rerunning some of the failing tests (make TEST=whatever) often results with these tests
passing. This unreliability is very frustrating.

I've been having these problems since I started working on GHC in December. I'm wondering whether
this is a common thing or am I doing something not-so-obviously wrong? As usual, I will
appreciate any advice from those more experienced than me.

Janek


Reply | Threaded
Open this post in threaded view
|

Unreliability of the build system

Geoffrey Mainland
On 05/15/2013 10:24 AM, Jan Stolarek wrote:

> Fellow developers,
>
> I need advice on using GHC build system. Recently I've been spending
> more time on resolving problems with the build system itself than on
> developing GHC, and this is not how things are supposed to be. I have
> found testsuite and validation to be very unreliable - things seem to
> break randomly and results are not repeatable. I suspect that I might
> be doing something wrong and I will appreciate any advice or
> explanations why I'm having these problem.
>
> I have three GHC trees:
>   * ghc-working  - source tree I use for development
>   * ghc-build    - lndir'ed to ghc-working, used for building
>   * ghc-validate - source tree used for validation, ghc-working is set
>                    up as one of the remotes
>
> Here's the situation from yesterday evening. I updated ghc-working and
> ghc-validate trees from the origin - according to the fingerprint.py
> script both trees were identical. I ran validation in ghc-validate and
> ghc-build trees (on the master branches, no my changes). The only
> difference between them should be that ghc-build is
> symlinked. Validation in ghc-validate passed with two failures (others
> on IRC confirmed they are having same failures), while validation in
> ghc-build failed with:
>
> Haddock coverage:
>  100% (  1 /  1) in 'GHC.IntWord64'
>   78% (  7 /  9) in 'GHC.Types'
>   17% (  1 /  6) in 'GHC.CString'
>    3% (  2 / 63) in 'GHC.Tuple'
>    0% (  0 /  3) in 'GHC.Debug'
>    0% (  0 /342) in 'GHC.PrimopWrappers'
>   30% (160 /531) in 'GHC.Prim'
>  100% (  3 /  3) in 'GHC.Magic'
>   38% (  6 / 16) in 'GHC.Classes'
> haddock: internal error: /dane/uczelnia/projekty/ghc-build/inpl
> ace/lib/html/Ocean.std-theme/ocean.css: copyFile: does not exis
> t (No such file or directory)
> gmake[1]: *** [libraries/ghc-prim/dist-install/doc/html/ghc-pri
> m/ghc-prim.haddock] B??d 1
> gmake[1]: *** Kasuj? plik `libraries/ghc-prim/dist-install/doc/
> html/ghc-prim/ghc-prim.haddock'
>
> Today I ran ./sync-all pull in both trees, but no new commits were
> added (according to fingerprint). I run the validation again and the
> validation in ghc-build fails as previously, while the validation in
> ghc-validate reports over 200 unexpected failures. What is more, I
> found that rerunning some of the failing tests (make TEST=whatever)
> often results with these tests passing. This unreliability is very
> frustrating.
>
> I've been having these problems since I started working on GHC in
> December. I'm wondering whether this is a common thing or am I doing
> something not-so-obviously wrong? As usual, I will appreciate any
> advice from those more experienced than me.
>
> Janek

The haddock failure you're seeing is caused by using relative symlinks
with lndir. See #7879.

I don't know what your other failures are, so I can't speak to
them. Perhaps you have different library versions checked out in
different trees; you could check whether or not this is the case by
looking at the various trees' fingerprints.

Personally, I think the current mix of git submodules and non-git
submodule dependencies is horrible. It makes it impossible to check out
a consistent tree without a full fingerprint while also incurring all
the disadvantages of submodules.

Geoff



Reply | Threaded
Open this post in threaded view
|

Unreliability of the build system

Jan Stolarek
> The haddock failure you're seeing is caused by using relative symlinks
> with lndir. See #7879.
Thanks, that's one problem less :) I applied the patch and I'm running the validation to see if
that works.

> Perhaps you have different library versions checked out in
> different trees; you could check whether or not this is the case by
> looking at the various trees' fingerprints.
According to fingerprints the trees are identical. I diffed both directories and there are some
minor differences, but I think these come from the fact that ghc-working never contained build
artifacts, whereas ghc-validate was cleaned with 'make maintainer-clean', which might have been
inaccurate.

> Personally, I think the current mix of git submodules and non-git
> submodule dependencies is horrible. It makes it impossible to check out
> a consistent tree without a full fingerprint while also incurring all
> the disadvantages of submodules.
I admit I don't understand motivation for using both git submodules and non-git submodules.

What worries me the most is unreliability of the test suite. After getting 200 test failures
mentioned in the previous mail, I ran the validation again and got only 3 failures. I ran one of
the failed tests manually and it passed, leaving me with two known failures.

Janek


Reply | Threaded
Open this post in threaded view
|

Unreliability of the build system

Simon Peyton Jones

| What worries me the most is unreliability of the test suite. After
| getting 200 test failures mentioned in the previous mail, I ran the
| validation again and got only 3 failures. I ran one of the failed tests
| manually and it passed, leaving me with two known failures.

This has never happened to me.  I can't even guess why it might happen to you.

Simon


Reply | Threaded
Open this post in threaded view
|

Unreliability of the build system

Jan Stolarek
> This has never happened to me.  I can't even guess why it might happen to
> you.
Looking through the validation log I see that tests fail with this error message:

Actual stderr output differs from expected:
--- /dev/null 2013-05-14 15:38:10.771000000 +0200
+++ ../../libraries/base/tests/IO/T3307.run.stderr 2013-05-15 09:21:45.695049002 +0200
@@ -0,0 +1,2 @@
+WARNING: cache is out of date: /dane/uczelnia/projekty/ghc-validate/bindisttest/install  
dir/lib/ghc-7.7.20130514/package.conf.d/package.cache
+  use 'ghc-pkg recache' to fix.
*** unexpected failure for T3307(normal)

Any ideas why this happens?

Janek


Reply | Threaded
Open this post in threaded view
|

Unreliability of the build system

David Terei
I no longer use a separate build dir (i.e., lndir). I switched away from it
as it wasn't buying me anything and I would run into issues that only
seemed to manifest with symlinks. Otherwise I've not experienced the issues
you mention. I'd point out though that GHC's 'no validate failures' is very
frequently violated which did confuse me a lot when getting started.


On 15 May 2013 05:10, Jan Stolarek <jan.stolarek at p.lodz.pl> wrote:

> > This has never happened to me.  I can't even guess why it might happen to
> > you.
> Looking through the validation log I see that tests fail with this error
> message:
>
> Actual stderr output differs from expected:
> --- /dev/null   2013-05-14 15:38:10.771000000 +0200
> +++ ../../libraries/base/tests/IO/T3307.run.stderr      2013-05-15
> 09:21:45.695049002 +0200
> @@ -0,0 +1,2 @@
> +WARNING: cache is out of date:
> /dane/uczelnia/projekty/ghc-validate/bindisttest/install
> dir/lib/ghc-7.7.20130514/package.conf.d/package.cache
> +  use 'ghc-pkg recache' to fix.
> *** unexpected failure for T3307(normal)
>
> Any ideas why this happens?
>
> Janek
>
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://www.haskell.org/mailman/listinfo/ghc-devs
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20130515/11b509c9/attachment.htm>

Reply | Threaded
Open this post in threaded view
|

Unreliability of the build system

Geoffrey Mainland
I use separate build dirs because I synchronize my tree across multiple
machines. It also makes it easy to blow away the build.

Jan, you clearly have some stale build artifacts. Don't mix in-tree and
out-of-tree builds. If you want to use out-of-tree builds, you can
eliminate leftover build artifacts from your source tree using 'git
ls-files -o' (you'll have to execute this in libraries/*, etc. also to
make sure you've got everything). Unfortunately make maintainer-clean
doesn't quite get everything.

Geoff

On 05/15/2013 08:25 PM, David Terei wrote:
> I no longer use a separate  build dir (i.e., lndir). I switched away
 > from it as it wasn't buying me anything and I would run into issues
 > that only seemed to manifest with symlinks. Otherwise I've not
 > experienced the issues you mention. I'd point out though that GHC's
 > 'no validate failures' is very frequently violated which did confuse
 > me a lot when getting started.
 >
 > On 15 May 2013 05:10, Jan Stolarek <jan.stolarek at p.lodz.pl> wrote:
 >
 >     > This has never happened to me.  I can't even guess why it might
happen to
 >     > you.
 >     Looking through the validation log I see that tests fail with
this error message:
 >
 >     Actual stderr output differs from expected:
 >     --- /dev/null   2013-05-14 15:38:10.771000000 +0200
 >     +++ ../../libraries/base/tests/IO/T3307.run.stderr 2013-05-15
09:21:45.695049002 +0200
 >     @@ -0,0 +1,2 @@
 >     +WARNING: cache is out of date:
/dane/uczelnia/projekty/ghc-validate/bindisttest/install
 >     dir/lib/ghc-7.7.20130514/package.conf.d/package.cache
 >     +  use 'ghc-pkg recache' to fix.
 >     *** unexpected failure for T3307(normal)
 >
 >     Any ideas why this happens?
 >
 >     Janek



Reply | Threaded
Open this post in threaded view
|

Unreliability of the build system

Jan Stolarek
> Don't mix in-tree and out-of-tree builds.
I *think* I am not mixing them. I'm running builds either in ghc-build (lndired to ghc-working,
this often gets blown away) and ghc-validate (these are in-tree builds, cleaned with
maintainer-clean). There are no builds in ghc-working.

> If you want to use out-of-tree builds, you can
> eliminate leftover build artifacts from your source tree using 'git
> ls-files -o' (you'll have to execute this in libraries/*, etc. also to
> make sure you've got everything).
Thanks for the tip, I'll try that.

I'll also try to debug my problem if I find some time. I think this deserves a bug report.

Janek

> Geoff
>
> On 05/15/2013 08:25 PM, David Terei wrote:
> > I no longer use a separate  build dir (i.e., lndir). I switched away
> >
>  > from it as it wasn't buying me anything and I would run into issues
>  > that only seemed to manifest with symlinks. Otherwise I've not
>  > experienced the issues you mention. I'd point out though that GHC's
>  > 'no validate failures' is very frequently violated which did confuse
>  > me a lot when getting started.
>  >
>  > On 15 May 2013 05:10, Jan Stolarek <jan.stolarek at p.lodz.pl> wrote:
>  >     > This has never happened to me.  I can't even guess why it might
>
> happen to
>
>  >     > you.
>  >
>  >     Looking through the validation log I see that tests fail with
>
> this error message:
>  >     Actual stderr output differs from expected:
>  >     --- /dev/null   2013-05-14 15:38:10.771000000 +0200
>  >     +++ ../../libraries/base/tests/IO/T3307.run.stderr 2013-05-15
>
> 09:21:45.695049002 +0200
>
>  >     @@ -0,0 +1,2 @@
>  >     +WARNING: cache is out of date:
>
> /dane/uczelnia/projekty/ghc-validate/bindisttest/install
>
>  >     dir/lib/ghc-7.7.20130514/package.conf.d/package.cache
>  >     +  use 'ghc-pkg recache' to fix.
>  >     *** unexpected failure for T3307(normal)
>  >
>  >     Any ideas why this happens?
>  >
>  >     Janek




Reply | Threaded
Open this post in threaded view
|

Unreliability of the build system

Ian Lynagh-2
In reply to this post by Jan Stolarek
On Wed, May 15, 2013 at 02:10:29PM +0200, Jan Stolarek wrote:

> > This has never happened to me.  I can't even guess why it might happen to
> > you.
> Looking through the validation log I see that tests fail with this error message:
>
> Actual stderr output differs from expected:
> --- /dev/null 2013-05-14 15:38:10.771000000 +0200
> +++ ../../libraries/base/tests/IO/T3307.run.stderr 2013-05-15 09:21:45.695049002 +0200
> @@ -0,0 +1,2 @@
> +WARNING: cache is out of date: /dane/uczelnia/projekty/ghc-validate/bindisttest/install  
> dir/lib/ghc-7.7.20130514/package.conf.d/package.cache
> +  use 'ghc-pkg recache' to fix.
> *** unexpected failure for T3307(normal)
>
> Any ideas why this happens?

The testsuite does check whether the package.cache timestamp changed
while running a test. If it did then you'll get a
    Package cache timestamps do not match: [...]
framework failure. Do you know whether that happened?

Otherwise, looking at why the package.cache is out of date, along with
what tests were run around the time that it happened, might give a clue
as to what went wrong.


Thanks
Ian



Reply | Threaded
Open this post in threaded view
|

Unreliability of the build system

Ian Lynagh-2
In reply to this post by David Terei
On Wed, May 15, 2013 at 12:25:14PM -0700, David Terei wrote:
>
> I'd point out though that GHC's 'no validate failures' is very
> frequently violated which did confuse me a lot when getting started.

This is at least partly caused by inconsistencies in test results. For
example, a Linux developer might push a change that causes a failure on
OS X, or one person might push a change that means that a perf test
fails on someone else's machine but not his own (or perhaps even fails
randomly, and they got lucky on their validate run).

The first person is not aware that anything is wrong, and the second
person just ignores the failure because their patches didn't cause it.

In an ideal world, the second person would either fix the bug, revert
thc bad change, or file a ticket and mark the test as broken.
Unfortunately, the second person is generally trying to do something
else when they discover the problem, and doesn't want to spend time
fixing something they didn't break.


Thanks
Ian



Reply | Threaded
Open this post in threaded view
|

Unreliability of the build system

Jan Stolarek
In reply to this post by Ian Lynagh-2
> The testsuite does check whether the package.cache timestamp changed
> while running a test. If it did then you'll get a
>     Package cache timestamps do not match: [...]
> framework failure. Do you know whether that happened?
If it checks cache local to the build tree, then I have no idea. I just run the validation and
don't touch the tree until validation ends. If it checks the global cache in ~ then I have a
suspicion of what might have happened - will check that.

> Otherwise, looking at why the package.cache is out of date, along with
> what tests were run around the time that it happened, might give a clue
> as to what went wrong.
Thanks Ian, I will keep that in mind and try to investigate the problem when it happens again.

Janek


Reply | Threaded
Open this post in threaded view
|

Unreliability of the build system

Gabriel Dos Reis
In reply to this post by Ian Lynagh-2
On Fri, May 17, 2013 at 8:52 AM, Ian Lynagh <ian at well-typed.com> wrote:

> On Wed, May 15, 2013 at 12:25:14PM -0700, David Terei wrote:
>>
>> I'd point out though that GHC's 'no validate failures' is very
>> frequently violated which did confuse me a lot when getting started.
>
> This is at least partly caused by inconsistencies in test results. For
> example, a Linux developer might push a change that causes a failure on
> OS X, or one person might push a change that means that a perf test
> fails on someone else's machine but not his own (or perhaps even fails
> randomly, and they got lucky on their validate run).
>
> The first person is not aware that anything is wrong, and the second
> person just ignores the failure because their patches didn't cause it.
>
> In an ideal world, the second person would either fix the bug, revert
> thc bad change, or file a ticket and mark the test as broken.
> Unfortunately, the second person is generally trying to do something
> else when they discover the problem, and doesn't want to spend time
> fixing something they didn't break.

Can't the build bots instructed to systematically fire up on every
commit for testing on the various platforms you currently do?  Maybe
you probably want to optimize for the cases of "whitespaces only"
patches.

Volunteers have set up something similar to this and it has
helped up tremendously in the GCC land.

-- Gaby


Reply | Threaded
Open this post in threaded view
|

Unreliability of the build system

Ian Lynagh-2
On Fri, May 17, 2013 at 11:34:01AM -0500, Gabriel Dos Reis wrote:
>
> Can't the build bots instructed to systematically fire up on every
> commit for testing on the various platforms you currently do?

In principle, yes, although (a) many of them are on machines that are
used to do something else during the day, and (b) the builders may often
still be building the last patches when more patches are pushed.

We already get testsuite results mailed to the list each night, though.
What we need is for people to fix unexpected results in them.
(admittedly we don't currently identify who/which patches caused a test
to start failing; perhaps that would help).


Thanks
Ian
--
Ian Lynagh, Haskell Consultant
Well-Typed LLP, http://www.well-typed.com/


Reply | Threaded
Open this post in threaded view
|

Unreliability of the build system

Gabriel Dos Reis
On Fri, May 17, 2013 at 3:14 PM, Ian Lynagh <ian at well-typed.com> wrote:
> On Fri, May 17, 2013 at 11:34:01AM -0500, Gabriel Dos Reis wrote:
>>
>> Can't the build bots instructed to systematically fire up on every
>> commit for testing on the various platforms you currently do?
>
> In principle, yes, although (a) many of them are on machines that are
> used to do something else during the day, and (b) the builders may often
> still be building the last patches when more patches are pushed.

For (a), we've benefited from the compile farm project made possible by
the Ada-Europe people:

    http://gcc.gnu.org/wiki/CompileFarm

It supports open source projects, and I don't think it is
restricted only to GCC people.   It is available to people
developing open source project (see the webpage for
details.) Maybe the Haskell developer community might
want to give it a try or set up similar projects.
This is a community thing, and it takes a community to grow.

For (b) you may consider job queues and/or concurrent compiles.

> We already get testsuite results mailed to the list each night, though.

Yes, I see those.  But from your previous description it sounds
to me as though they would catch some particular cluster of commits
that may cumulatively break things with no one taking responsibility
or stepping forward to fix (because they didn't break it in the first place.)

> What we need is for people to fix unexpected results in them.

Agreed.  But, from my 15+ years with GCC, it makes a huge
difference when you have an automated assistance that can pinpoint
breakages to specific commits/offenders.

Another thing I've found useful was the "24 hours" rule whereby
when a breakage was identified to be caused by a specific
commits, the author is given 24 hours to fix it.  Beyond that, if
no action is taken by the committer and/or if it is clear that it
will take much longer to fix then a reversal patch is pre-approved.
That helps getting people clean their mess :-)

> (admittedly we don't currently identify who/which patches caused a test
> to start failing; perhaps that would help).

Yes, exactly!

-- Gaby


Reply | Threaded
Open this post in threaded view
|

Unreliability of the build system

Jan Stolarek
In reply to this post by Ian Lynagh-2
> Otherwise, looking at why the package.cache is out of date, along with
> what tests were run around the time that it happened, might give a clue
> as to what went wrong.
I think I got it. After running 'make maintainer-clean' in the validation tree I still have this
files in testsuite left from previous validation:

tests/cabal/local01.package.conf/package.cache
tests/cabal/local03.package.conf/package.cache
tests/cabal/local03.package.conf/testpkg-1.2.3.4-XXX.conf
tests/cabal/local03.package.conf/testpkg-2.0-XXX.conf
tests/cabal/local03.package.conf/testpkg-4.0-XXX.conf
tests/cabal/local04.package.conf/newtestpkg-2.0-XXX.conf
tests/cabal/local04.package.conf/package.cache
tests/cabal/local04.package.conf/testpkg-1.2.3.4-XXX.conf
tests/cabal/local05a.package.conf/package.cache
tests/cabal/local05b.package.conf/package.cache
tests/cabal/local06.package.conf/package.cache
tests/cabal/local06.package.conf/testdup-1.0-XXX.conf
tests/cabal/local06.package.conf/testpkg-1.2.3.4-XXX.conf
tests/cabal/localT1750.package.conf/T1750A-1-XXX.conf
tests/cabal/localT1750.package.conf/T1750B-1-XXX.conf
tests/cabal/localT1750.package.conf/package.cache
tests/cabal/localshadow1.package.conf/package.cache
tests/cabal/localshadow1.package.conf/shadow-1-XXX.conf
tests/cabal/localshadow1.package.conf/shadowdep-1-XXX.conf
tests/cabal/localshadow2.package.conf/package.cache
tests/cabal/localshadow2.package.conf/shadow-1-YYY.conf
tests/cabal/package.conf.ghcpkg02/package.cache

(plus lots of other build artifacts in the testsuite). Deleting files listed by 'git ls-files -o'
solves the problem. I wonder how is it possible that - if this is really the cause - I am the
first person that run into this problem?

Janek


Reply | Threaded
Open this post in threaded view
|

Unreliability of the build system

Geoffrey Mainland
In reply to this post by Ian Lynagh-2
On 05/17/2013 02:43 PM, Ian Lynagh wrote:
> On Wed, May 15, 2013 at 02:10:29PM +0200, Jan Stolarek wrote:
>>> This has never happened to me.  I can't even guess why it might
happen to
>>> you.
>> Looking through the validation log I see that tests fail with this
error message:
>>
>> Actual stderr output differs from expected:
>> --- /dev/null    2013-05-14 15:38:10.771000000 +0200
>> +++ ../../libraries/base/tests/IO/T3307.run.stderr    2013-05-15
09:21:45.695049002 +0200
>> @@ -0,0 +1,2 @@
>> +WARNING: cache is out of date:
/dane/uczelnia/projekty/ghc-validate/bindisttest/install  

>> dir/lib/ghc-7.7.20130514/package.conf.d/package.cache
>> +  use 'ghc-pkg recache' to fix.
>> *** unexpected failure for T3307(normal)
>>
>> Any ideas why this happens?
>
> The testsuite does check whether the package.cache timestamp changed
> while running a test. If it did then you'll get a
>     Package cache timestamps do not match: [...]
> framework failure. Do you know whether that happened?
>
> Otherwise, looking at why the package.cache is out of date, along with
> what tests were run around the time that it happened, might give a clue
> as to what went wrong.

I am now seeing this error, and it's killing my validation runs. I am
quite sure I do not have any stale build artifacts. Relevant bits from
my validate run are below.

It looks like "/home/gmainlan/ghc/ghc-working-build/bindisttest/install
dir/lib/ghc-7.7.20130624/package.conf.d" gets *younger*, leading to the
recache error. Rather odd... Any ideas why this is happening?

Jan, are you still seeing this error?

Thanks,
Geoff

for f in '/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d'/*; do create () { touch "$1" &&
chmod 644 "$1" ; } && create "$f"; done
/usr/bin/install -c -m 755 -d
"/home/gmainlan/ghc/ghc-working-build/bindisttest/install   dir/bin"
for i in utils/hp2ps/dist/build/tmp/hp2ps
utils/hpc/dist-install/build/tmp/hpc; do \
        /usr/bin/install -c -m 755  $i
"/home/gmainlan/ghc/ghc-working-build/bindisttest/install   dir/bin" ;  \
    done
/usr/bin/install -c -m 755 -d
"/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624"
for i in utils/unlit/dist/build/tmp/unlit
libraries/integer-gmp/mkGmpDerivedConstants/dist/build/tmp/mkGmpDerivedConstants;
do \
        /usr/bin/install -c -m 755  $i
"/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624"; \
    done
/usr/bin/install -c -m 755 -d
"/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/share/doc/ghc"
/usr/bin/install -c -m 755 -d
"/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/share/doc/ghc/html"
/usr/bin/install -c -m 644  docs/index.html
"/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/share/doc/ghc/html"
/usr/bin/install -c -m 755 -d
"/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/share/doc/ghc/html/libraries"
for i in libraries/dist-haddock/*; do \
        /usr/bin/install -c -m 644  $i
"/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/share/doc/ghc/html/libraries/"; \
    done
/usr/bin/install -c -m 644  libraries/prologue.txt
"/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/share/doc/ghc/html/libraries/"
/usr/bin/install -c -m 755  libraries/gen_contents_index
"/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/share/doc/ghc/html/libraries/"
for i in utils/haddock/doc/haddock docs/users_guide/users_guide; do \
        /usr/bin/install -c -m 755 -d
"/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/share/doc/ghc/html/`basename $i`"; \
        /usr/bin/install -c -m 644  $i/*
"/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/share/doc/ghc/html/`basename $i`"; \
    done
bindisttest/"install   dir"/bin/runghc bindisttest/HelloWorld >
bindisttest/output
diff -U 1 bindisttest/output bindisttest/expected_output
bindisttest/"install   dir"/bin/ghc --make bindisttest/HelloWorld
[1 of 1] Compiling Main             ( bindisttest/HelloWorld.lhs,
bindisttest/HelloWorld.o )
Linking bindisttest/HelloWorld ...
bindisttest/HelloWorld > bindisttest/output
diff -U 1 bindisttest/output bindisttest/expected_output
bindisttest/"install   dir"/bin/ghc-pkg check --no-user-package-db
== Start post-install package check
Timestamp 2013-06-24 11:02:04.121267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/package.cache
Timestamp 2013-06-24 11:02:04.093781 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d (older than cache)
using cache: /home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/package.cache
== End post-install package check
===--- building phase 0
make -r --no-print-directory -f ghc.mk phase=0 phase_0_builds
make[1]: Nothing to be done for `phase_0_builds'.
===--- building phase 1
make -r --no-print-directory -f ghc.mk phase=1 phase_1_builds
make[1]: Nothing to be done for `phase_1_builds'.
===--- building final phase
make -r --no-print-directory -f ghc.mk phase=final validate_build_xhtml
cd libraries/xhtml &&
"/home/gmainlan/ghc/ghc-working-build/bindisttest/install   dir/bin/ghc"
--make Setup
[1 of 1] Compiling Main             ( Setup.hs, Setup.o )
Linking Setup ...
cd libraries/xhtml && ./Setup configure
--with-ghc="/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/bin/ghc"
--with-haddock="/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/bin/haddock" --enable-shared --disable-library-vanilla
--disable-library-prof --global --builddir=dist-bindist
--prefix="/home/gmainlan/ghc/ghc-working-build/bindisttest/install   dir"
Configuring xhtml-3000.2.1...
cd libraries/xhtml && ./Setup build   --builddir=dist-bindist
Building xhtml-3000.2.1...
Preprocessing library xhtml-3000.2.1...
[ 1 of 15] Compiling Text.XHtml.BlockTable ( Text/XHtml/BlockTable.hs,
dist-bindist/build/Text/XHtml/BlockTable.dyn_o )
[ 2 of 15] Compiling Text.XHtml.Internals ( Text/XHtml/Internals.hs,
dist-bindist/build/Text/XHtml/Internals.dyn_o )
[ 3 of 15] Compiling Text.XHtml.Transitional.Elements (
Text/XHtml/Transitional/Elements.hs,
dist-bindist/build/Text/XHtml/Transitional/Elements.dyn_o )
[ 4 of 15] Compiling Text.XHtml.Transitional.Attributes (
Text/XHtml/Transitional/Attributes.hs,
dist-bindist/build/Text/XHtml/Transitional/Attributes.dyn_o )
[ 5 of 15] Compiling Text.XHtml.Frameset.Attributes (
Text/XHtml/Frameset/Attributes.hs,
dist-bindist/build/Text/XHtml/Frameset/Attributes.dyn_o )
[ 6 of 15] Compiling Text.XHtml.Strict.Attributes (
Text/XHtml/Strict/Attributes.hs,
dist-bindist/build/Text/XHtml/Strict/Attributes.dyn_o )
[ 7 of 15] Compiling Text.XHtml.Frameset.Elements (
Text/XHtml/Frameset/Elements.hs,
dist-bindist/build/Text/XHtml/Frameset/Elements.dyn_o )
[ 8 of 15] Compiling Text.XHtml.Strict.Elements (
Text/XHtml/Strict/Elements.hs,
dist-bindist/build/Text/XHtml/Strict/Elements.dyn_o )
[ 9 of 15] Compiling Text.XHtml.Extras ( Text/XHtml/Extras.hs,
dist-bindist/build/Text/XHtml/Extras.dyn_o )
[10 of 15] Compiling Text.XHtml.Strict ( Text/XHtml/Strict.hs,
dist-bindist/build/Text/XHtml/Strict.dyn_o )
[11 of 15] Compiling Text.XHtml.Frameset ( Text/XHtml/Frameset.hs,
dist-bindist/build/Text/XHtml/Frameset.dyn_o )
[12 of 15] Compiling Text.XHtml.Table ( Text/XHtml/Table.hs,
dist-bindist/build/Text/XHtml/Table.dyn_o )
[13 of 15] Compiling Text.XHtml.Debug ( Text/XHtml/Debug.hs,
dist-bindist/build/Text/XHtml/Debug.dyn_o )
[14 of 15] Compiling Text.XHtml.Transitional (
Text/XHtml/Transitional.hs,
dist-bindist/build/Text/XHtml/Transitional.dyn_o )
[15 of 15] Compiling Text.XHtml       ( Text/XHtml.hs,
dist-bindist/build/Text/XHtml.dyn_o )
In-place registering xhtml-3000.2.1...
cd libraries/xhtml && ./Setup haddock --builddir=dist-bindist
Running Haddock for xhtml-3000.2.1...
Preprocessing library xhtml-3000.2.1...
Haddock coverage:
  73% (  8 / 11) in 'Text.XHtml.BlockTable'
  71% ( 27 / 38) in 'Text.XHtml.Internals'
   8% (  1 / 12) in 'Text.XHtml.Transitional.Elements'
   5% (  2 / 37) in 'Text.XHtml.Transitional.Attributes'
  14% (  1 /  7) in 'Text.XHtml.Frameset.Attributes'
   2% (  1 / 50) in 'Text.XHtml.Strict.Attributes'
  20% (  1 /  5) in 'Text.XHtml.Frameset.Elements'
   1% (  1 / 79) in 'Text.XHtml.Strict.Elements'
  34% ( 11 / 32) in 'Text.XHtml.Extras'
  19% ( 37 /190) in 'Text.XHtml.Strict'
  18% ( 36 /197) in 'Text.XHtml.Frameset'
  30% (  3 / 10) in 'Text.XHtml.Table'
  60% (  3 /  5) in 'Text.XHtml.Debug'
  16% ( 39 /244) in 'Text.XHtml.Transitional'
  71% (  5 /  7) in 'Text.XHtml'
Documentation created: dist-bindist/doc/html/xhtml/index.html
cd libraries/xhtml && ./Setup install --builddir=dist-bindist
Installing library in
/home/gmainlan/ghc/ghc-working-build/bindisttest/install
dir/lib/x86_64-linux-ghc-7.7.20130624/xhtml-3000.2.1
Registering xhtml-3000.2.1...
cd libraries/xhtml && ./Setup clean   --builddir=dist-bindist
cleaning...
cd libraries/xhtml && rm -f Setup Setup.exe Setup.hi Setup.o
== Start post-xhtml package check
Timestamp 2013-06-24 11:02:14.669097 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/package.cache
Timestamp 2013-06-24 11:02:14.669197 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d (NEWER than cache)
WARNING: cache is out of date:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/package.cache
Use 'ghc-pkg recache' to fix.
Timestamp 2013-06-24 11:02:04.109267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/dph-prim-seq-0.8.0.1-210e4f7a8f16529a2ec53e83ab50080a.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/dph-prim-seq-0.8.0.1-210e4f7a8f16529a2ec53e83ab50080a.conf
Timestamp 2013-06-24 11:02:04.125267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/process-1.2.0.0-a57473e1c925594e84382e9056485694.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/process-1.2.0.0-a57473e1c925594e84382e9056485694.conf
Timestamp 2013-06-24 11:02:04.109267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/dph-lifted-vseg-0.8.0.1-d4163d48b63dffeeb08aa85f9f8e0c26.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/dph-lifted-vseg-0.8.0.1-d4163d48b63dffeeb08aa85f9f8e0c26.conf
Timestamp 2013-06-24 11:02:04.121267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/pretty-1.1.1.0-f43642640d939cae65d3c62b2c9f1dfb.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/pretty-1.1.1.0-f43642640d939cae65d3c62b2c9f1dfb.conf
Timestamp 2013-06-24 11:02:04.113267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/ghc-prim-0.3.1.0-cf75c70b85389df439b6e800b148b7c7.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/ghc-prim-0.3.1.0-cf75c70b85389df439b6e800b148b7c7.conf
Timestamp 2013-06-24 11:02:04.105267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/dph-base-0.8.0.1-375baf633d66abdb0a453d26f86c8454.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/dph-base-0.8.0.1-375baf633d66abdb0a453d26f86c8454.conf
Timestamp 2013-06-24 11:02:04.117267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/hoopl-3.10.0.0-1679cf3bd566c16af94d8c17ec6ed56e.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/hoopl-3.10.0.0-1679cf3bd566c16af94d8c17ec6ed56e.conf
Timestamp 2013-06-24 11:02:04.101267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/containers-0.5.0.0-4fdda7b0b1de82f14ede6431f19e166b.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/containers-0.5.0.0-4fdda7b0b1de82f14ede6431f19e166b.conf
Timestamp 2013-06-24 11:02:04.125267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/terminfo-0.3.2.5-1b55547ca99a04baef1d8a155d33656c.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/terminfo-0.3.2.5-1b55547ca99a04baef1d8a155d33656c.conf
Timestamp 2013-06-24 11:02:04.097267023 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/bin-package-db-0.0.0.0-8dd4b537adc30ec2e7dc25c7c5f86325.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/bin-package-db-0.0.0.0-8dd4b537adc30ec2e7dc25c7c5f86325.conf
Timestamp 2013-06-24 11:02:04.097267023 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/Cabal-1.17.0-f5791edb3b79eb7243aca7d14dd4358c.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/Cabal-1.17.0-f5791edb3b79eb7243aca7d14dd4358c.conf
Timestamp 2013-06-24 11:02:04.113267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/haskeline-0.7.0.4-4d3d7a08dcbf998edae9ab6078d72dcf.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/haskeline-0.7.0.4-4d3d7a08dcbf998edae9ab6078d72dcf.conf
Timestamp 2013-06-24 11:02:04.125267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/template-haskell-2.9.0.0-a0186916a6b97accead92ac10deca378.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/template-haskell-2.9.0.0-a0186916a6b97accead92ac10deca378.conf
Timestamp 2013-06-24 11:02:04.101267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/bytestring-0.10.3.0-f3cc36bf9e32e7e8a0b90c75ebbee862.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/bytestring-0.10.3.0-f3cc36bf9e32e7e8a0b90c75ebbee862.conf
Timestamp 2013-06-24 11:02:04.105267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/dph-lifted-boxed-0.8.0.1-85c3c6176452e682a4e5485c58c14e8c.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/dph-lifted-boxed-0.8.0.1-85c3c6176452e682a4e5485c58c14e8c.conf
Timestamp 2013-06-24 11:02:04.117267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/haskell98-2.0.0.3-c5686fa30b4b2f19fe3ad49719a6f74d.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/haskell98-2.0.0.3-c5686fa30b4b2f19fe3ad49719a6f74d.conf
Timestamp 2013-06-24 11:02:04.125267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/random-1.0.1.1-27a60004f9b2ed697f991f21e0caf131.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/random-1.0.1.1-27a60004f9b2ed697f991f21e0caf131.conf
Timestamp 2013-06-24 11:02:04.105267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/dph-lifted-base-0.8.0.1-19a1a371f3392826504a4c386369e828.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/dph-lifted-base-0.8.0.1-19a1a371f3392826504a4c386369e828.conf
Timestamp 2013-06-24 11:02:04.129267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/transformers-0.3.0.0-059787b385844316829d92153e3aae96.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/transformers-0.3.0.0-059787b385844316829d92153e3aae96.conf
Timestamp 2013-06-24 11:02:04.101267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/binary-0.7.0.0-9d193bdf36eb84160dead27cb8fa8cc5.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/binary-0.7.0.0-9d193bdf36eb84160dead27cb8fa8cc5.conf
Timestamp 2013-06-24 11:02:04.101267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/deepseq-1.3.0.2-46fb49e0483a96267b7b96533a2bd05f.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/deepseq-1.3.0.2-46fb49e0483a96267b7b96533a2bd05f.conf
Timestamp 2013-06-24 11:02:04.121267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/old-time-1.1.0.1-3d55dddf6d95f915b092a4754bc649ec.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/old-time-1.1.0.1-3d55dddf6d95f915b092a4754bc649ec.conf
Timestamp 2013-06-24 11:02:04.097267023 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/array-0.4.0.2-1313be63c7703b2d5ff4ab80ba2658a6.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/array-0.4.0.2-1313be63c7703b2d5ff4ab80ba2658a6.conf
Timestamp 2013-06-24 11:02:04.129267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/unix-2.7.0.0-81384b9dccdd72997dd6af9bdd405c5d.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/unix-2.7.0.0-81384b9dccdd72997dd6af9bdd405c5d.conf
Timestamp 2013-06-24 11:02:04.121267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/old-locale-1.0.0.5-d9f80c600a5428ecd373adec12598d4c.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/old-locale-1.0.0.5-d9f80c600a5428ecd373adec12598d4c.conf
Timestamp 2013-06-24 11:02:04.109267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/dph-prim-par-0.8.0.1-e97a3649200e78d827149219a602ca7a.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/dph-prim-par-0.8.0.1-e97a3649200e78d827149219a602ca7a.conf
Timestamp 2013-06-24 11:02:04.129267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/vector-0.9.1-2948d133ec698208cc7e609ab2d2654c.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/vector-0.9.1-2948d133ec698208cc7e609ab2d2654c.conf
Timestamp 2013-06-24 11:02:04.117267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/integer-gmp-0.5.1.0-7d539199b5ff2f1f458e94990ffe7e16.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/integer-gmp-0.5.1.0-7d539199b5ff2f1f458e94990ffe7e16.conf
Timestamp 2013-06-24 11:02:04.117267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/hpc-0.6.0.1-2e7b6723d0c386a07fa4c0a1b1290da1.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/hpc-0.6.0.1-2e7b6723d0c386a07fa4c0a1b1290da1.conf
Timestamp 2013-06-24 11:02:04.101267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/builtin_rts.conf (older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/builtin_rts.conf
Timestamp 2013-06-24 11:02:04.113267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/haskell2010-1.1.1.0-bf951a4e72643b8e4e571e6385964717.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/haskell2010-1.1.1.0-bf951a4e72643b8e4e571e6385964717.conf
Timestamp 2013-06-24 11:02:04.129267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/time-1.4.0.2-7265230817fd209eda7e7a9a202a64ad.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/time-1.4.0.2-7265230817fd209eda7e7a9a202a64ad.conf
Timestamp 2013-06-24 11:02:04.113267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/ghc-7.7.20130624-eeed18236adab46adb4748a37dc20592.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/ghc-7.7.20130624-eeed18236adab46adb4748a37dc20592.conf
Timestamp 2013-06-24 11:02:04.121267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/primitive-0.4.0.1-3fa29b1acb951cdbaf037dd4e246f8c6.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/primitive-0.4.0.1-3fa29b1acb951cdbaf037dd4e246f8c6.conf
Timestamp 2013-06-24 11:02:04.109267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/dph-prim-interface-0.8.0.1-b3148291e0c4b8a80c0cae0a0512a06c.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/dph-prim-interface-0.8.0.1-b3148291e0c4b8a80c0cae0a0512a06c.conf
Timestamp 2013-06-24 11:02:14.649626 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/xhtml-3000.2.1-b455a48b4b5f078bf868322b899efe8e.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/xhtml-3000.2.1-b455a48b4b5f078bf868322b899efe8e.conf
Timestamp 2013-06-24 11:02:04.113267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/filepath-1.3.0.2-6f3e918bcb99996f8081d676fe17bd63.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/filepath-1.3.0.2-6f3e918bcb99996f8081d676fe17bd63.conf
Timestamp 2013-06-24 11:02:04.105267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/directory-1.2.0.1-925245ee360cb91046dfbd0d365d05f3.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/directory-1.2.0.1-925245ee360cb91046dfbd0d365d05f3.conf
Timestamp 2013-06-24 11:02:04.109267024 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/dph-lifted-copy-0.8.0.1-ee48196af7d662d2c19a3c5bd7e94319.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/dph-lifted-copy-0.8.0.1-ee48196af7d662d2c19a3c5bd7e94319.conf
Timestamp 2013-06-24 11:02:04.097267023 UTC for
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/base-4.7.0.0-284daa47db76b97d3aed840842fb656d.conf
(older than cache)
reading package config:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/base-4.7.0.0-284daa47db76b97d3aed840842fb656d.conf
== End post-xhtml package check
make -C testsuite/tests CLEANUP=1
OUTPUT_SUMMARY=../../testsuite_summary.txt fast
WARNING: cache is out of date:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/package.cache
Use 'ghc-pkg recache' to fix.
WARNING: cache is out of date:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/package.cache
Use 'ghc-pkg recache' to fix.
make[1]: Entering directory
`/home/gmainlan/ghc/ghc-working-build/testsuite/tests'
make fast=YES
WARNING: cache is out of date:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/package.cache
Use 'ghc-pkg recache' to fix.
WARNING: cache is out of date:
/home/gmainlan/ghc/ghc-working-build/bindisttest/install  
dir/lib/ghc-7.7.20130624/package.conf.d/package.cache
Use 'ghc-pkg recache' to fix.



Reply | Threaded
Open this post in threaded view
|

Unreliability of the build system

Jan Stolarek
Geoffrey, I think your errors are slightly different than mine. Mine always happend when running
the testsuite, yours seem to be from a different stage of the validation. I will pull the latest
HEAD and validate, but I'm not sure whether my results will mean anything - failures I was
getting seemed quite random (i.e. sometimes they appeared and sometimes not). Do you get these
errors every time you run validation?

Jan


Reply | Threaded
Open this post in threaded view
|

Unreliability of the build system

Geoffrey Mainland
I get the errors when running the testsuite too, but I excerpted the
earliest "recache" warning from my full validation run. The same tests
consistently fail for me.

I only get the errors when running the testsuite with BINDIST=YES, as
expected.

Geoff

On 06/24/2013 12:49 PM, Jan Stolarek wrote:
> Geoffrey, I think your errors are slightly different than mine. Mine always happend when running
> the testsuite, yours seem to be from a different stage of the validation. I will pull the latest
> HEAD and validate, but I'm not sure whether my results will mean anything - failures I was
> getting seemed quite random (i.e. sometimes they appeared and sometimes not). Do you get these
> errors every time you run validation?
>
> Jan



Reply | Threaded
Open this post in threaded view
|

Unreliability of the build system

Jan Stolarek
> The same tests consistently fail for me.
Even if you run them separately with 'make TEST=failingTest'?

> I only get the errors when running the testsuite with BINDIST=YES, as
> expected.
I'm running validation without BINDIST=YES at the moment, will run the second validation later.

Where do you put BINDIST=YES setting?

Jan


Reply | Threaded
Open this post in threaded view
|

Unreliability of the build system

Ian Lynagh-2
In reply to this post by Geoffrey Mainland
On Mon, Jun 24, 2013 at 12:37:17PM +0100, Geoffrey Mainland wrote:
>
> It looks like "/home/gmainlan/ghc/ghc-working-build/bindisttest/install
> dir/lib/ghc-7.7.20130624/package.conf.d" gets *younger*, leading to the
> recache error. Rather odd... Any ideas why this is happening?

That's expected: When we install xhtml, we add it to the DB, and the
cache gets updated.

It looks like the problem is this:

Timestamp 2013-06-24 11:02:14.669097 UTC for [...]/package.conf.d/package.cache
Timestamp 2013-06-24 11:02:14.669197 UTC for [...]/package.conf.d (NEWER

I would guess that we assume that when we write package.cache, the
directory gets updated with the same timestamp; but actually, both use
the timestamp at which they really happen. This is of course much more
likely to be a problem if your filesystem has sub-second precision.

If that's it, then it should be possible to fix it by explicitly setting
the modification time of the directory to match the cache when we write
the cache in ghc-pkg.


Thanks
Ian



12