"resource exhausted" in CI

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

"resource exhausted" in CI

Richard Eisenberg-4
Hi Ben,

The pipeline for a recent patch (https://gitlab.haskell.org/rae/ghc/-/jobs/16793) says:

  HC [stage 1] compiler/stage2/build/TcForeign.o
HC [stage 1] compiler/stage2/build/TcRules.o
/tmp/ghc6477_0/ghc_3.s: hClose: resource exhausted (No space left on device)
compiler/ghc.mk:445: recipe for target 'compiler/stage2/build/TcTyClsDecls.o' failed
make[1]: *** [compiler/stage2/build/TcTyClsDecls.o] Error 1
make[1]: *** Waiting for unfinished jobs....
Is there a way to avoid this? Or should I just restart the CI? And what does Marge think of it all? (As in, if I ask her to take another spin, will she do so on the same commit?)

Thanks,
Richard

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: "resource exhausted" in CI

Ben Gamari-2
Richard Eisenberg <[hidden email]> writes:

> Hi Ben,
>
> The pipeline for a recent patch (https://gitlab.haskell.org/rae/ghc/-/jobs/16793) says:
>
>   HC [stage 1] compiler/stage2/build/TcForeign.o
>   HC [stage 1] compiler/stage2/build/TcRules.o
> /tmp/ghc6477_0/ghc_3.s: hClose: resource exhausted (No space left on device)
> compiler/ghc.mk:445: recipe for target 'compiler/stage2/build/TcTyClsDecls.o' failed
> make[1]: *** [compiler/stage2/build/TcTyClsDecls.o] Error 1
> make[1]: *** Waiting for unfinished jobs....
>
Wow, that was fast; I only configured this runner yesterday. I've
cleaned up the disk in question and will investigate the root cause. I
have also restarted all affected builds.

> Is there a way to avoid this? Or should I just restart the CI? And
> what does Marge think of it all? (As in, if I ask her to take another
> spin, will she do so on the same commit?)
>
I've restarted your job. The new job is [1]. If it passes Marge should
merge if she was so-instructed.

Cheers,

- Ben


[1] https://gitlab.haskell.org/rae/ghc/-/jobs/16913

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

signature.asc (497 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: "resource exhausted" in CI

Richard Eisenberg-4
Something is awry: https://gitlab.haskell.org/rae/ghc/-/jobs/16908 never got off the ground.

> On Jan 24, 2019, at 1:21 PM, Ben Gamari <[hidden email]> wrote:
>
> Richard Eisenberg <[hidden email]> writes:
>
>> Hi Ben,
>>
>> The pipeline for a recent patch (https://gitlab.haskell.org/rae/ghc/-/jobs/16793) says:
>>
>>  HC [stage 1] compiler/stage2/build/TcForeign.o
>>  HC [stage 1] compiler/stage2/build/TcRules.o
>> /tmp/ghc6477_0/ghc_3.s: hClose: resource exhausted (No space left on device)
>> compiler/ghc.mk:445: recipe for target 'compiler/stage2/build/TcTyClsDecls.o' failed
>> make[1]: *** [compiler/stage2/build/TcTyClsDecls.o] Error 1
>> make[1]: *** Waiting for unfinished jobs....
>>
> Wow, that was fast; I only configured this runner yesterday. I've
> cleaned up the disk in question and will investigate the root cause. I
> have also restarted all affected builds.
>
>> Is there a way to avoid this? Or should I just restart the CI? And
>> what does Marge think of it all? (As in, if I ask her to take another
>> spin, will she do so on the same commit?)
>>
> I've restarted your job. The new job is [1]. If it passes Marge should
> merge if she was so-instructed.
>
> Cheers,
>
> - Ben
>
>
> [1] https://gitlab.haskell.org/rae/ghc/-/jobs/16913

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: "resource exhausted" in CI

Ben Gamari-2
Richard Eisenberg <[hidden email]> writes:

> Something is awry: https://gitlab.haskell.org/rae/ghc/-/jobs/16908 never got off the ground.
>
It is possible that you pushed a rebase? This error generally means that
the commit is no longer accessible which may happen when you push a
rebase.

I believe I cited the job for the current version of the patch [1] in my
previous email. Note that the commit SHA is different between [1] and
the job you cited.

Cheers,

- Ben


[1] https://gitlab.haskell.org/rae/ghc/-/jobs/16913

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

signature.asc (497 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: "resource exhausted" in CI

Richard Eisenberg-4
Ah, yes -- I did push a rebase. OK: good to know that this is expected behavior after rebasing (makes sense).

Thanks,
Richard

> On Jan 24, 2019, at 2:01 PM, Ben Gamari <[hidden email]> wrote:
>
> Richard Eisenberg <[hidden email]> writes:
>
>> Something is awry: https://gitlab.haskell.org/rae/ghc/-/jobs/16908 never got off the ground.
>>
> It is possible that you pushed a rebase? This error generally means that
> the commit is no longer accessible which may happen when you push a
> rebase.
>
> I believe I cited the job for the current version of the patch [1] in my
> previous email. Note that the commit SHA is different between [1] and
> the job you cited.
>
> Cheers,
>
> - Ben
>
>
> [1] https://gitlab.haskell.org/rae/ghc/-/jobs/16913

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

RE: "resource exhausted" in CI

GHC - devs mailing list
| Ah, yes -- I did push a rebase. OK: good to know that this is expected
| behavior after rebasing (makes sense).

Does not make sense (yet) to me.

Can someone explain (and perhaps document) the workflow here?

Simon

| -----Original Message-----
| From: ghc-devs <[hidden email]> On Behalf Of Richard
| Eisenberg
| Sent: 24 January 2019 19:22
| To: Ben Gamari <[hidden email]>
| Cc: GHC developers <[hidden email]>
| Subject: Re: "resource exhausted" in CI
|
| Ah, yes -- I did push a rebase. OK: good to know that this is expected
| behavior after rebasing (makes sense).
|
| Thanks,
| Richard
|
| > On Jan 24, 2019, at 2:01 PM, Ben Gamari <[hidden email]> wrote:
| >
| > Richard Eisenberg <[hidden email]> writes:
| >
| >> Something is awry:
| https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.ha
| skell.org%2Frae%2Fghc%2F-
| %2Fjobs%2F16908&amp;data=02%7C01%7Csimonpj%40microsoft.com%7C375c6daea23444
| 7c3d2808d682314861%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63683954552
| 1260172&amp;sdata=8OYI5PAk7B6P%2B%2BoJbqgeqPbTR%2BMH6F8Wm34ThdBhRL0%3D&amp;
| reserved=0 never got off the ground.
| >>
| > It is possible that you pushed a rebase? This error generally means that
| > the commit is no longer accessible which may happen when you push a
| > rebase.
| >
| > I believe I cited the job for the current version of the patch [1] in my
| > previous email. Note that the commit SHA is different between [1] and
| > the job you cited.
| >
| > Cheers,
| >
| > - Ben
| >
| >
| > [1]
| https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.ha
| skell.org%2Frae%2Fghc%2F-
| %2Fjobs%2F16913&amp;data=02%7C01%7Csimonpj%40microsoft.com%7C375c6daea23444
| 7c3d2808d682314861%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63683954552
| 1260172&amp;sdata=wM0hlN60tQmsDtsXS%2BIqLzeAqhgK6RmUoEDzYG2b%2FhI%3D&amp;re
| served=0
|
| _______________________________________________
| ghc-devs mailing list
| [hidden email]
| https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.haske
| ll.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc-
| devs&amp;data=02%7C01%7Csimonpj%40microsoft.com%7C375c6daea234447c3d2808d68
| 2314861%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636839545521260172&amp
| ;sdata=R2PHyXMyWDG4mmusi1KLmklGR0b%2FXAE%2BNp%2BwS4ZOmp8%3D&amp;reserved=0
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: "resource exhausted" in CI

Brandon Allbery
Rebase is more or less stashing and removing all local commits, upgrading the underlying branch to current, then re-applying the local commits. This changes the commit hashes for any re-applied commit that lands on a change to the underlying branch, meaning that old commit hashes can be invalid afterward.

(This is also why force-pushing causes problems, since this is only visible in a local tree *unless* force-pushed upstream, meaning now those changed/deleted commits affect anyone else who has checked out that upstream branch. The normal process of upstreaming commits can't expose local changes like that, since it all goes upstream as a group and becomes a permanent part of the branch's history. Unless someone force-pushes the branch afterward, forcibly overwriting that history with a different one and leaving anyoneelse who'd checked out the branch with dangling commits that no longer exist.)

Here, CI is making a copy of someone's branch and testing each commit in turn to ensure consistency between multiple branches that touch the same code, whose commits may end up interleaved. If the branch is rebased or force-pushed during that testing, intermediate commits may become invalid as above and CI would need to start over with a new list of commits.

On Thu, Jan 24, 2019 at 5:43 PM Simon Peyton Jones via ghc-devs <[hidden email]> wrote:
| Ah, yes -- I did push a rebase. OK: good to know that this is expected
| behavior after rebasing (makes sense).

Does not make sense (yet) to me.

Can someone explain (and perhaps document) the workflow here?

Simon

| -----Original Message-----
| From: ghc-devs <[hidden email]> On Behalf Of Richard
| Eisenberg
| Sent: 24 January 2019 19:22
| To: Ben Gamari <[hidden email]>
| Cc: GHC developers <[hidden email]>
| Subject: Re: "resource exhausted" in CI
|
| Ah, yes -- I did push a rebase. OK: good to know that this is expected
| behavior after rebasing (makes sense).
|
| Thanks,
| Richard
|
| > On Jan 24, 2019, at 2:01 PM, Ben Gamari <[hidden email]> wrote:
| >
| > Richard Eisenberg <[hidden email]> writes:
| >
| >> Something is awry:
| https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.ha
| skell.org%2Frae%2Fghc%2F-
| %2Fjobs%2F16908&amp;data=02%7C01%7Csimonpj%40microsoft.com%7C375c6daea23444
| 7c3d2808d682314861%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63683954552
| 1260172&amp;sdata=8OYI5PAk7B6P%2B%2BoJbqgeqPbTR%2BMH6F8Wm34ThdBhRL0%3D&amp;
| reserved=0 never got off the ground.
| >>
| > It is possible that you pushed a rebase? This error generally means that
| > the commit is no longer accessible which may happen when you push a
| > rebase.
| >
| > I believe I cited the job for the current version of the patch [1] in my
| > previous email. Note that the commit SHA is different between [1] and
| > the job you cited.
| >
| > Cheers,
| >
| > - Ben
| >
| >
| > [1]
| https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.ha
| skell.org%2Frae%2Fghc%2F-
| %2Fjobs%2F16913&amp;data=02%7C01%7Csimonpj%40microsoft.com%7C375c6daea23444
| 7c3d2808d682314861%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63683954552
| 1260172&amp;sdata=wM0hlN60tQmsDtsXS%2BIqLzeAqhgK6RmUoEDzYG2b%2FhI%3D&amp;re
| served=0
|
| _______________________________________________
| ghc-devs mailing list
| [hidden email]
| https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.haske
| ll.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc-
| devs&amp;data=02%7C01%7Csimonpj%40microsoft.com%7C375c6daea234447c3d2808d68
| 2314861%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636839545521260172&amp
| ;sdata=R2PHyXMyWDG4mmusi1KLmklGR0b%2FXAE%2BNp%2BwS4ZOmp8%3D&amp;reserved=0
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs


--
brandon s allbery kf8nh

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: "resource exhausted" in CI

Brandon Allbery
Um, also this seems to have jumped threads; the subject of this message was a different issue, about disk space. Is that part of the confusion?

On Thu, Jan 24, 2019 at 5:53 PM Brandon Allbery <[hidden email]> wrote:
Rebase is more or less stashing and removing all local commits, upgrading the underlying branch to current, then re-applying the local commits. This changes the commit hashes for any re-applied commit that lands on a change to the underlying branch, meaning that old commit hashes can be invalid afterward.

(This is also why force-pushing causes problems, since this is only visible in a local tree *unless* force-pushed upstream, meaning now those changed/deleted commits affect anyone else who has checked out that upstream branch. The normal process of upstreaming commits can't expose local changes like that, since it all goes upstream as a group and becomes a permanent part of the branch's history. Unless someone force-pushes the branch afterward, forcibly overwriting that history with a different one and leaving anyoneelse who'd checked out the branch with dangling commits that no longer exist.)

Here, CI is making a copy of someone's branch and testing each commit in turn to ensure consistency between multiple branches that touch the same code, whose commits may end up interleaved. If the branch is rebased or force-pushed during that testing, intermediate commits may become invalid as above and CI would need to start over with a new list of commits.

On Thu, Jan 24, 2019 at 5:43 PM Simon Peyton Jones via ghc-devs <[hidden email]> wrote:
| Ah, yes -- I did push a rebase. OK: good to know that this is expected
| behavior after rebasing (makes sense).

Does not make sense (yet) to me.

Can someone explain (and perhaps document) the workflow here?

Simon

| -----Original Message-----
| From: ghc-devs <[hidden email]> On Behalf Of Richard
| Eisenberg
| Sent: 24 January 2019 19:22
| To: Ben Gamari <[hidden email]>
| Cc: GHC developers <[hidden email]>
| Subject: Re: "resource exhausted" in CI
|
| Ah, yes -- I did push a rebase. OK: good to know that this is expected
| behavior after rebasing (makes sense).
|
| Thanks,
| Richard
|
| > On Jan 24, 2019, at 2:01 PM, Ben Gamari <[hidden email]> wrote:
| >
| > Richard Eisenberg <[hidden email]> writes:
| >
| >> Something is awry:
| https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.ha
| skell.org%2Frae%2Fghc%2F-
| %2Fjobs%2F16908&amp;data=02%7C01%7Csimonpj%40microsoft.com%7C375c6daea23444
| 7c3d2808d682314861%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63683954552
| 1260172&amp;sdata=8OYI5PAk7B6P%2B%2BoJbqgeqPbTR%2BMH6F8Wm34ThdBhRL0%3D&amp;
| reserved=0 never got off the ground.
| >>
| > It is possible that you pushed a rebase? This error generally means that
| > the commit is no longer accessible which may happen when you push a
| > rebase.
| >
| > I believe I cited the job for the current version of the patch [1] in my
| > previous email. Note that the commit SHA is different between [1] and
| > the job you cited.
| >
| > Cheers,
| >
| > - Ben
| >
| >
| > [1]
| https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.ha
| skell.org%2Frae%2Fghc%2F-
| %2Fjobs%2F16913&amp;data=02%7C01%7Csimonpj%40microsoft.com%7C375c6daea23444
| 7c3d2808d682314861%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63683954552
| 1260172&amp;sdata=wM0hlN60tQmsDtsXS%2BIqLzeAqhgK6RmUoEDzYG2b%2FhI%3D&amp;re
| served=0
|
| _______________________________________________
| ghc-devs mailing list
| [hidden email]
| https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.haske
| ll.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc-
| devs&amp;data=02%7C01%7Csimonpj%40microsoft.com%7C375c6daea234447c3d2808d68
| 2314861%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636839545521260172&amp
| ;sdata=R2PHyXMyWDG4mmusi1KLmklGR0b%2FXAE%2BNp%2BwS4ZOmp8%3D&amp;reserved=0
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs


--
brandon s allbery kf8nh


--
brandon s allbery kf8nh

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs