[perl #40817] [TODO] track generated files during the configure/make process

[Jerry Gay]

# New Ticket Created by Jerry Gay
# Please include the string: [perl #40817]
# in the subject line of all future correspondence about this issue.
# <URL: http://rt.perl.org/rt3/Ticket/Display.html?id=40817 >


arguably the best way to track which files are generated during
parrot's configure/build process is to append each file to a list when
it is created. so, we need some makefile hackery in order to do this
during build-time, and some config hackery to do it during
configure-time.

this would replace the MANIFEST.generated file which is created ex
post facto with a file that is created at build-time.
~jerry

[James Keenan via RT]

we need ... some config hackery to do it during
configure-time.
this would replace the MANIFEST.generated file which is created ex
post facto with a file that is created at build-time.

Herewith some configure-time hackery. This patch provides a new
compilation utility subroutine,
Parrot::Configure::Utils::append_configure_log(), which appends the name
of a file about to be created to a file called MANIFEST.configure.generated.

Where files (other than test.c files within auto::* config step classes)
are created by Parrot::Configure::Compiler::cc_gen(),
append_configure_log() is called internally. This is the case for 59 of
the 97 files currently created during compilation. The new function is
called explicitly in the remaining cases.

This cannot yet replace MANIFEST.generated because we still need that
build-time hackery. But it should do no harm, and it gets us part way
there. Please review. I'll apply next week if there are no objections.

Thank you very much.
kid51

[James Keenan via RT]

Here's a corrected version of the patch. It eliminates some
"uninitialized value" warnings in t/configure/034-step.t and prepends
the needed path to the *.pasm entries in MANIFEST.configure.generated.

[Chromatic]

Index: lib/Parrot/Configure/Utils.pm
===================================================================
--- lib/Parrot/Configure/Utils.pm       (revision 26524)
+++ lib/Parrot/Configure/Utils.pm       (working copy)
@@ -311,6 +312,17 @@
 
 =back
 
+=cut
+
+sub append_configure_log {
+    my $target = shift;
+    my $generated_log = q{MANIFEST.configure.generated};
+    open my $GEN, '>>', $generated_log
+        or die "Can't open $generated_log for appending: $!";
+    print $GEN qq{$target\n};
+    close $GEN or die "Can't close $generated_log after appending: $!";
+}
+

I don't understand this mixture of '', q{}, "", and qq{} across four lines. I
recommend sticking with one quoting convention (and in the absence of the
need to escape the quoting character currently in use, my preference is ''
and "").

-- c

[James Keenan via RT]

Hmm, the more I look at this, the more I realize that even the improved
version of my patch is not ready for prime time. While it DWIMs when
you call 'perl Configure.pl', it doesn't when you call 'perl
Configure.pl --test' (as I do frequently). We only want
MANIFEST.configure.generated to be opened or appended to when we are
actually configuring -- not when we're running tests either
pre-configuration or during 'make test'.

Any ideas as to how this might be salvaged?

kid51

[James Keenan via RT]

Alright, take 3: Store the fact that this is real configuration in the
Parrot::Configure object just before calling runsteps().

[James Keenan via RT]

If the patch is applied, can we close this ticket?

No. I only figured out how to keep track of files generated during
configuration, not during build. We need some of what, IIRC, particle
termed "makefile trickery" to keep track of files generated by make.

[Reini Urban]

Post by James Keenan via RT

If the patch is applied, can we close this ticket?

No. I only figured out how to keep track of files generated during
configuration, not during build. We need some of what, IIRC, particle
termed "makefile trickery" to keep track of files generated by make.

My patch 39742-installed-conflict.patch also adds targets and
dependencies for all generated Makefiles.
So 39742 would satisfy this ticket also a bit.

Still no full make depends which adds to the Makefile dynamically
as in perl5 e.g.
By far not all generated files are tracked correctly.
make reconfig still needed.

[Reini Urban]

Post by James Keenan via RT
No. I only figured out how to keep track of files generated during
configuration, not during build. We need some of what, IIRC, particle
termed "makefile trickery" to keep track of files generated by make.

I'm going to relinquish this ticket and give it to Nobody because I
frankly don't have any good ideas on the 'makefile trickery' required to
identify all the files generated during the build process.

"makefile trickery" could be a simple
echo "$(filename) [bla]" >> MANIFEST.generated
which is platform independent. It even works on windows plain.

But most of the examples can be generated more easily during the
build step by better utils.

Examples are arch specific library names and extensions,
now in MANIFEST.generated. But this sucks.
E.g.
blib/lib/libparrot.so.0.8.2 [main]lib
has to be fixed on every VERSION bump.

runtime/parrot/dynext/perl6_group.bundle [library]
runtime/parrot/dynext/perl6_group.dll [library]
runtime/parrot/dynext/perl6_group.dylib [library]
runtime/parrot/dynext/perl6_group.so [library]
can be collapsed on the actual platform.

Also the generated ops for each core can be added during the build step.
runtime/parrot/dynext/wmls_ops.dll [library]
runtime/parrot/dynext/wmls_ops_cg.dll [library]
runtime/parrot/dynext/wmls_ops_cgp.dll [library]
runtime/parrot/dynext/wmls_ops_switch.dll [library]

All the perl6 ops e.g. are currently missing and are therefore not
installed in trunk. (fixed in pdd30install_stage3)

I'll work on that in the pdd30install_stage3 branch since it's only needed
for make install.

[Reini Urban]

Can you assign it to me?

Post by Reini Urban

Post by James Keenan via RT
No. I only figured out how to keep track of files generated during
configuration, not during build. We need some of what, IIRC, particle
termed "makefile trickery" to keep track of files generated by make.

I'm going to relinquish this ticket and give it to Nobody because I
frankly don't have any good ideas on the 'makefile trickery' required to
identify all the files generated during the build process.

"makefile trickery" could be a simple
echo "$(filename) [bla]" >> MANIFEST.generated
which is platform independent. It even works on windows plain.
But most of the examples can be generated more easily during the
build step by better utils.
Examples are arch specific library names and extensions,
now in MANIFEST.generated. But this sucks.
E.g.
blib/lib/libparrot.so.0.8.2 [main]lib
has to be fixed on every VERSION bump.
runtime/parrot/dynext/perl6_group.bundle [library]
runtime/parrot/dynext/perl6_group.dll [library]
runtime/parrot/dynext/perl6_group.dylib [library]
runtime/parrot/dynext/perl6_group.so [library]
can be collapsed on the actual platform.
Also the generated ops for each core can be added during the build step.
runtime/parrot/dynext/wmls_ops.dll [library]
runtime/parrot/dynext/wmls_ops_cg.dll [library]
runtime/parrot/dynext/wmls_ops_cgp.dll [library]
runtime/parrot/dynext/wmls_ops_switch.dll [library]
All the perl6 ops e.g. are currently missing and are therefore not
installed in trunk. (fixed in pdd30install_stage3)
I'll work on that in the pdd30install_stage3 branch since it's only needed
for make install.

I took the task and wrote it.
In the makefiles are now various lines such as
$(PERL) -Ilib -MParrot::BuildUtil -e add_to_generated "$@","[main]","bin"

add_to_generated() also includes the manifest logic for windows.
I'll try now to get all the currently MANIFEST.generated files into this,
so we don't accidently loose some files again (being installed).

patch comes this evening. I will apply to pdd30install_stage3 also.

[Reini Urban]

Post by James Keenan via RT
No. I only figured out how to keep track of files generated during
configuration, not during build. We need some of what, IIRC, particle
termed "makefile trickery" to keep track of files generated by make.

I'm going to relinquish this ticket and give it to Nobody because I
frankly don't have any good ideas on the 'makefile trickery' required to
identify all the files generated during the build process.

Anybody has an idea why compilers/imcc/imclexer.c is in
MANIFEST.generated, and compilers/imcc/imcparser.{c,h} not?
I believe this is a bug, so I added those two also. But what for?

MANIFEST.generated serves two purposes I guess:

1) Track make generated files to be able to clean them, which are not
cleaned by wildcards, or by seperate subdir Makefiles. So only
config/gen/makefiles/root.in generated files.

2) Add all files to be installed, except the languages which have their
own make install. Except perl6, which is also installed by
tools/dev/install_files.pl

I rather want to get rid of point 1, this should be cleaned up in the
Makefile. And adding potentially-to-be-installed installed files
requires some tools/dev/install_files.pl overrides.

Attached is my current state of affair, but there are still some
left-over files. So do not apply, I want to clean up the whole mess
first, so we don't have to deal with a MANIFEST.generated.in for the yet
untracked files.

The headerizer is also not yet finished.

[Jerry Gay]

On Sun, Mar 23, 2008 at 7:49 PM, James Keenan via RT

Post by James Keenan via RT
Hmm, the more I look at this, the more I realize that even the improved
version of my patch is not ready for prime time. While it DWIMs when
you call 'perl Configure.pl', it doesn't when you call 'perl
Configure.pl --test' (as I do frequently). We only want
MANIFEST.configure.generated to be opened or appended to when we are
actually configuring -- not when we're running tests either
pre-configuration or during 'make test'.
Any ideas as to how this might be salvaged?

i suggest have the functions that write to files check as to whether
they should write or not.
of course, DRY suggests that's best handled by calling another function.

for example,

sub append_configure_log {
my $target = shift;
return if $conf->i_should_not_append_the_configure_log($target);
...
}

~jerry

[James Keenan via RT]

I've been staring at this ticket for more than a year. One of my
earlier patches got us part of the way there. Applying Reini's or
something like that would get us closer.

But the more I look at this problem in light of
https://trac.parrot.org/parrot/ticket/586,
https://trac.parrot.org/parrot/ticket/434 and
https://trac.parrot.org/parrot/ticket/426, the more I believe that the
problem is more complex than we at first thought.

That's because, taking MANIFEST and MANIFEST.generated together, we use
these files for *two* distinct purposes:

1. Determining what files a developer or user needs to build Parrot
from source.
2. Determining what files are installed by 'make install' or 'make
install-dev'.

Purpose #1 is fulfilled exclusively by MANIFEST. Or, to be more
precise, by the *left-hand* column in MANIFEST. In a MANIFEST entry
like this:

compilers/imcc/cfg.c [imcc]

... only the left-hand part matters in terms of having the files one
needs for building from source.

Purpose #2, in contrast, requires both files to be fulfilled. And it
requires *both* columns in each file to be fulfilled, because it's the
*right-hand* column that determines where a file is ultimately installed.

The patches we've applied or submitted so far automate, more or less
successfully, the identification of files *generated* by Configure.pl or
make. But, AFAICT, so far it takes a human to determine where a file --
source or generated -- must be installed. In other words, I don't see a
way to automate the assignment of data in the right-hand column.

Some data:

As of last night, a fresh checkout from trunk contains 2142 files.

After Configure.pl on Linux/i386, the total number of files in the build
dir has increased to 2229, an increase of 87.

After make, the total number of files has increased to 2999, an
additional increase of 770.

But if I then call 'make install', I install only 293 files.

What criteria do we have that those 293 files -- and only those 293 --
get installed? We need to have such criteria if we're to have accurate
information in the right-hand columns of MANIFEST and
MANIFEST.generated. Accurate information those files, in turn, is the
only way we can write tests to determine whether 'make install' and
'make install-dev' (and the programs underlying them) are doing the
right thing.

Thoughts?

Thank you very much.
kid51