Workflow question (changing codegen)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Workflow question (changing codegen)

Ömer Sinan Ağacan
Hi all,

I'm currently going through this torturous process and I'm hoping that someone
here will be able to help.

I'm making changes in the codegen. My changes are currently buggy, and I need a
working stage 1 compiler to be able to debug. Basically I need to build
libraries using the branch my changes are based on, then build stage 1 with my
branch, so that I'll be able to build and run programs using stage 1 that uses
my codegen changes. The changes are compatible with the old codegen (i.e. no
changes in calling conventions or anything like that) so this should work.

Normally I do this

    $ git checkout master
    $ git distclean && ./boot && ./configure && make
    $ git checkout my_branch
    $ cd compiler; make 1

This gives me stage 1 compiler that uses my buggy codegen changes, plus
libraries built with the old and correct codegen.

However the problem is I'm also adding a new file in my_branch, and the build
system just doesn't register that fact, even after adding the line I added to
compiler/ghc.cabal.in to compiler/ghc.cabal. So far the only way to fix this
that I could find was to run ./configure again, then run make for a few seconds
at the top level, then do `make 1` in compiler/. Unfortunately even that doesn't
work when the master branch and my_branch have different dates, because `make`
in master branch produces a different version than the `make` in my_branch, so
the interface files become incompatible.

Anyone have any ideas on how to proceed here?

Thanks,

Ömer
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Workflow question (changing codegen)

Ben Gamari-2
On June 28, 2019 5:09:45 AM EDT, "Ömer Sinan Ağacan" <[hidden email]> wrote:

>Hi all,
>
>I'm currently going through this torturous process and I'm hoping that
>someone
>here will be able to help.
>
>I'm making changes in the codegen. My changes are currently buggy, and
>I need a
>working stage 1 compiler to be able to debug. Basically I need to build
>libraries using the branch my changes are based on, then build stage 1
>with my
>branch, so that I'll be able to build and run programs using stage 1
>that uses
>my codegen changes. The changes are compatible with the old codegen
>(i.e. no
>changes in calling conventions or anything like that) so this should
>work.
>
>Normally I do this
>
>    $ git checkout master
>    $ git distclean && ./boot && ./configure && make
>    $ git checkout my_branch
>    $ cd compiler; make 1
>
>This gives me stage 1 compiler that uses my buggy codegen changes, plus
>libraries built with the old and correct codegen.
>
>However the problem is I'm also adding a new file in my_branch, and the
>build
>system just doesn't register that fact, even after adding the line I
>added to
>compiler/ghc.cabal.in to compiler/ghc.cabal. So far the only way to fix
>this
>that I could find was to run ./configure again, then run make for a few
>seconds
>at the top level, then do `make 1` in compiler/. Unfortunately even
>that doesn't
>work when the master branch and my_branch have different dates, because
>`make`
>in master branch produces a different version than the `make` in
>my_branch, so
>the interface files become incompatible.
>
>Anyone have any ideas on how to proceed here?
>
>Thanks,
>
>Ömer
>_______________________________________________
>ghc-devs mailing list
>[hidden email]
>http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

In general I think it is wise to avoid switching branches in a tree you are actively developing in. The cost of switching both in the compilation time that it implies and the uncertain state that it leaves the tree in is in my opinion too high. It you want to compare your change against master I would recommend using two working directories.


Cheers,

- Ben

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Workflow question (changing codegen)

Richard Eisenberg-5
Just to pass on something that looks cool (I haven't tried it myself yet): git worktree. It seems git can hang several different checkouts of a repo in different directories. This seems far superior to my current habit of having many clones of ghc, sometimes going through machinations to get commits from one place to another. The documentation for git worktree seems quite approachable, so you might find it useful. I plan on using it in the future.

Richard

> On Jun 29, 2019, at 8:24 AM, Ben Gamari <[hidden email]> wrote:
>
> On June 28, 2019 5:09:45 AM EDT, "Ömer Sinan Ağacan" <[hidden email]> wrote:
>> Hi all,
>>
>> I'm currently going through this torturous process and I'm hoping that
>> someone
>> here will be able to help.
>>
>> I'm making changes in the codegen. My changes are currently buggy, and
>> I need a
>> working stage 1 compiler to be able to debug. Basically I need to build
>> libraries using the branch my changes are based on, then build stage 1
>> with my
>> branch, so that I'll be able to build and run programs using stage 1
>> that uses
>> my codegen changes. The changes are compatible with the old codegen
>> (i.e. no
>> changes in calling conventions or anything like that) so this should
>> work.
>>
>> Normally I do this
>>
>>   $ git checkout master
>>   $ git distclean && ./boot && ./configure && make
>>   $ git checkout my_branch
>>   $ cd compiler; make 1
>>
>> This gives me stage 1 compiler that uses my buggy codegen changes, plus
>> libraries built with the old and correct codegen.
>>
>> However the problem is I'm also adding a new file in my_branch, and the
>> build
>> system just doesn't register that fact, even after adding the line I
>> added to
>> compiler/ghc.cabal.in to compiler/ghc.cabal. So far the only way to fix
>> this
>> that I could find was to run ./configure again, then run make for a few
>> seconds
>> at the top level, then do `make 1` in compiler/. Unfortunately even
>> that doesn't
>> work when the master branch and my_branch have different dates, because
>> `make`
>> in master branch produces a different version than the `make` in
>> my_branch, so
>> the interface files become incompatible.
>>
>> Anyone have any ideas on how to proceed here?
>>
>> Thanks,
>>
>> Ömer
>> _______________________________________________
>> ghc-devs mailing list
>> [hidden email]
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
> In general I think it is wise to avoid switching branches in a tree you are actively developing in. The cost of switching both in the compilation time that it implies and the uncertain state that it leaves the tree in is in my opinion too high. It you want to compare your change against master I would recommend using two working directories.
>
>
> Cheers,
>
> - Ben
>
> _______________________________________________
> ghc-devs mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Workflow question (changing codegen)

Sebastian Graf
Re: git worktree: That's the workflow I'm currently using. It has its problems with submodules, see https://stackoverflow.com/questions/31871888/what-goes-wrong-when-using-git-worktree-with-git-submodules. But you can make it work with this git alias from the first answer: https://gitlab.com/clacke/gists/blob/0c4a0b6e10f7fbf15127339750a6ff490d9aa3c8/.config/git/config#L12. Just go into your main checkout and do `git wtas ../T9876`. AFAIR it interacts weirdly with MinGW's git or git for Windows, but nothing you can't work around.

Anyway, I was hoping that one day hadrian will be smart enough to have a build directory for each branch or something, so that I would only need one checkout where I can switch between branches as needed. In the meantime, `git wtas` does what I want.

Am Sa., 29. Juni 2019 um 21:53 Uhr schrieb Richard Eisenberg <[hidden email]>:
Just to pass on something that looks cool (I haven't tried it myself yet): git worktree. It seems git can hang several different checkouts of a repo in different directories. This seems far superior to my current habit of having many clones of ghc, sometimes going through machinations to get commits from one place to another. The documentation for git worktree seems quite approachable, so you might find it useful. I plan on using it in the future.

Richard

> On Jun 29, 2019, at 8:24 AM, Ben Gamari <[hidden email]> wrote:
>
> On June 28, 2019 5:09:45 AM EDT, "Ömer Sinan Ağacan" <[hidden email]> wrote:
>> Hi all,
>>
>> I'm currently going through this torturous process and I'm hoping that
>> someone
>> here will be able to help.
>>
>> I'm making changes in the codegen. My changes are currently buggy, and
>> I need a
>> working stage 1 compiler to be able to debug. Basically I need to build
>> libraries using the branch my changes are based on, then build stage 1
>> with my
>> branch, so that I'll be able to build and run programs using stage 1
>> that uses
>> my codegen changes. The changes are compatible with the old codegen
>> (i.e. no
>> changes in calling conventions or anything like that) so this should
>> work.
>>
>> Normally I do this
>>
>>   $ git checkout master
>>   $ git distclean && ./boot && ./configure && make
>>   $ git checkout my_branch
>>   $ cd compiler; make 1
>>
>> This gives me stage 1 compiler that uses my buggy codegen changes, plus
>> libraries built with the old and correct codegen.
>>
>> However the problem is I'm also adding a new file in my_branch, and the
>> build
>> system just doesn't register that fact, even after adding the line I
>> added to
>> compiler/ghc.cabal.in to compiler/ghc.cabal. So far the only way to fix
>> this
>> that I could find was to run ./configure again, then run make for a few
>> seconds
>> at the top level, then do `make 1` in compiler/. Unfortunately even
>> that doesn't
>> work when the master branch and my_branch have different dates, because
>> `make`
>> in master branch produces a different version than the `make` in
>> my_branch, so
>> the interface files become incompatible.
>>
>> Anyone have any ideas on how to proceed here?
>>
>> Thanks,
>>
>> Ömer
>> _______________________________________________
>> ghc-devs mailing list
>> [hidden email]
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
> In general I think it is wise to avoid switching branches in a tree you are actively developing in. The cost of switching both in the compilation time that it implies and the uncertain state that it leaves the tree in is in my opinion too high. It you want to compare your change against master I would recommend using two working directories.
>
>
> Cheers,
>
> - Ben
>
> _______________________________________________
> ghc-devs mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Workflow question (changing codegen)

Ömer Sinan Ağacan
In reply to this post by Ömer Sinan Ağacan
My current workaround is this: I have a branch which is just master + the new
file I've added. I first build it from a clean tree (git clean -xfd; then
build), then switch to my branch, and run `make 1` in `compiler/`. That way I
don't have to run ./configure (because the new file is already built and tracked
by the build system) so the compiler version does not change and my stage 1
compiler can use the libraries I built with master.

I guess the root cause of this is that I have to run ./configure for the build
system to track my new file, but doing that also updates the compiler version.
Avoiding any of these (updating compiler version, or having to run configure
when adding new files) would make this much easier.

Ömer

Ömer Sinan Ağacan <[hidden email]>, 28 Haz 2019 Cum, 12:09
tarihinde şunu yazdı:

>
> Hi all,
>
> I'm currently going through this torturous process and I'm hoping that someone
> here will be able to help.
>
> I'm making changes in the codegen. My changes are currently buggy, and I need a
> working stage 1 compiler to be able to debug. Basically I need to build
> libraries using the branch my changes are based on, then build stage 1 with my
> branch, so that I'll be able to build and run programs using stage 1 that uses
> my codegen changes. The changes are compatible with the old codegen (i.e. no
> changes in calling conventions or anything like that) so this should work.
>
> Normally I do this
>
>     $ git checkout master
>     $ git distclean && ./boot && ./configure && make
>     $ git checkout my_branch
>     $ cd compiler; make 1
>
> This gives me stage 1 compiler that uses my buggy codegen changes, plus
> libraries built with the old and correct codegen.
>
> However the problem is I'm also adding a new file in my_branch, and the build
> system just doesn't register that fact, even after adding the line I added to
> compiler/ghc.cabal.in to compiler/ghc.cabal. So far the only way to fix this
> that I could find was to run ./configure again, then run make for a few seconds
> at the top level, then do `make 1` in compiler/. Unfortunately even that doesn't
> work when the master branch and my_branch have different dates, because `make`
> in master branch produces a different version than the `make` in my_branch, so
> the interface files become incompatible.
>
> Anyone have any ideas on how to proceed here?
>
> Thanks,
>
> Ömer
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Workflow question (changing codegen)

Ben Gamari-2
In reply to this post by Richard Eisenberg-5
Richard Eisenberg <[hidden email]> writes:

> Just to pass on something that looks cool (I haven't tried it myself
> yet): git worktree. It seems git can hang several different checkouts
> of a repo in different directories. This seems far superior to my
> current habit of having many clones of ghc, sometimes going through
> machinations to get commits from one place to another. The
> documentation for git worktree seems quite approachable, so you might
> find it useful. I plan on using it in the future.
>
Indeed I use `git new-workdir`, which is a script similar to `git worktree`
(but I think predates `worktree` by a few years). Like `worktree`, this
allows you to have several working directories sharing a set of refs.

Cheers,

- Ben


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

signature.asc (497 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Workflow question (changing codegen)

Ben Gamari-2
In reply to this post by Sebastian Graf
Sebastian Graf <[hidden email]> writes:

> Re: git worktree: That's the workflow I'm currently using. It has its
> problems with submodules, see
> https://stackoverflow.com/questions/31871888/what-goes-wrong-when-using-git-worktree-with-git-submodules.
> But you can make it work with this git alias from the first answer:
> https://gitlab.com/clacke/gists/blob/0c4a0b6e10f7fbf15127339750a6ff490d9aa3c8/.config/git/config#L12.
> Just go into your main checkout and do `git wtas ../T9876`. AFAIR it
> interacts weirdly with MinGW's git or git for Windows, but nothing you
> can't work around.
>
> Anyway, I was hoping that one day hadrian will be smart enough to have a
> build directory for each branch or something, so that I would only need one
> checkout where I can switch between branches as needed. In the meantime,
> `git wtas` does what I want.
>
For what it's worth, Hadrian can already do this. Just pass the
--build-root=$DIR flag and all build artifacts will end up in $DIR
instead of the usual _build. I'm not sure how robust it is on branch
switches, but it's at least a start.

Cheers,

- Ben


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

signature.asc (497 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Workflow question (changing codegen)

Ben Gamari-2
In reply to this post by Ömer Sinan Ağacan
Ömer Sinan Ağacan <[hidden email]> writes:

> My current workaround is this: I have a branch which is just master + the new
> file I've added. I first build it from a clean tree (git clean -xfd; then
> build), then switch to my branch, and run `make 1` in `compiler/`. That way I
> don't have to run ./configure (because the new file is already built and tracked
> by the build system) so the compiler version does not change and my stage 1
> compiler can use the libraries I built with master.
>
> I guess the root cause of this is that I have to run ./configure for the build
> system to track my new file, but doing that also updates the compiler version.
> Avoiding any of these (updating compiler version, or having to run configure
> when adding new files) would make this much easier.
>
For what it's worth, I sometimes just resort to manually editing the
files generated by ./configure to avoid having to reconfigure and
consequently rebuild.

Cheers,

- Ben


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

signature.asc (497 bytes) Download Attachment