A User Guide to the Grid Packaging Tools

For the majority of packaging tasks, it is sufficient to install only GPT. If you are building packages then you will need GNU Make and GNU tar in addition to GPT. If you are developing packages or building them out of a source code repository you will need to install a specific set of GNU autotools.

The following GPT scripts are used to install gpt:

GPT uses a special set of GNU autotools for package development. Other versions of these tools can be used but the macros and scripts for GPT may not be compatible. In addition, these versions have been tested on numerous Grid platforms. These tools can be found here in the tarball called gnu_autotools_for_gpt-*.tar.gz

To install the autotools simple untar the the tarball into the GPT source directory. IAW the tar command for the gnu tarball needs to be executed in the same directory as the tar command for the GPT tarball. The autotools source will be located in the support subdirectory. Then run the following in the top of the GPT source directory:

      ./build_gpt --autotools
      

As off version 3.1 GPT installs two versions of the autotools to allow packages to migrate to the latest versions. The original autools are installed in the directory $GPT_LOCATION/oldautotools. The latest version is located in $GPT_LOCATION/newautotools.

GPT relies on several system tools. The script gpt-config is used by gpt_build and the GPT RPM to locate these tools using the $PATH variable. To see what the local settings for a GPT installation is run gpt-config with the -list flag:

bash$ gpt-config -list
GNU tar located at /bin/tar
GNU zip located at /bin/gzip
GNU unzip located at /bin/gunzip
GNU make located at /usr/bin/gmake
GNU perl located at /usr/bin/perl
rpm located at /bin/rpm
rpmbuild located at /usr/bin/rpmbuild
RPM Package License set to GNU
RPM Package Vendor set to NCSA
RPM Package FTP Site set to ftp.ncsa.uiuc.edu
RPM Package URL set to http://www.gridpackaging.org
RPM Packager set to NCSA
RPM Prefix set to /usr/grid
GNU target platform set to i686-pc-linux-gnu

The listing includes a bunch of RPM information because the command was run on a linux machine. Here is what it looks like on a Solaris platform.

bash$ gpt-config -list
GNU tar located at /usr/ncsa/bin/tar
GNU zip located at /usr/ncsa/bin/gzip
GNU unzip located at /usr/ncsa/bin/gunzip
GNU make located at /usr/ncsa/bin/gmake
GNU perl located at /usr/ncsa/bin/perl
rpm located at NOT CONFIGURED
rpmbuild located at NOT CONFIGURED
RPM Package License set to N/A
RPM Package Vendor set to N/A
RPM Package FTP Site set to N/A
RPM Package URL set to N/A
RPM Packager set to N/A
RPM Prefix set to N/A
GNU target platform set to sparc-sun-solaris2.8

You can use gpt-config to change these settings. For example the following sets the location of GNU tar and asks gpt-config to probe for the rest of the settings:

bash$ gpt-config -gtar=/usr/local/bin/tar -probe
bash$ gpt-config       
GNU tar located at /usr/local/bin/tar
GNU zip located at /usr/bin/gzip
GNU unzip located at /usr/bin/gunzip
GNU make located at /usr/local/bin/make
Perl located at /usr/local/bin/perl
rpm located at Not Available
rpmbuild located at Not Available
RPM Package License set to N/A
RPM Package Vendor set to N/A
RPM Package FTP Site set to N/A
RPM Package URL set to N/A
RPM Packager set to N/A
RPM Prefix set to N/A
GNU target platform set to sparc-sun-solaris2.9

gpt-config has a flag for every setting. Run gpt-config -man to get the specifics on each flag or look here.

Since GPT is mainly perl code, there is no real need to do regression testing on every platform which it is installed. However GPT uses a couple of Perl modules that are written in "C". It is a good idea to run the GPT test to try out these modules because there are occasions where these modules have problems. Unfortunately, these problems are not easily detected outside the test and can lead to unexpected behaviour.

To run the test change into the GPT source directory and execute ./build_gpt -test. This will both install and test modules used by GPT. Modules have to be installed so that higher level modules can also be tested. Here is some sample output:

bash$ ./build_gpt -test
build_gpt ====> installing GPT into /home/mbletzin/install/gpt
build_gpt ====> building support/Compress-Zlib-1.16
build_gpt ====> testing support/Compress-Zlib-1.16
make[1]: Entering directory `/home/mbletzin/nmi/gpt/support/Compress-Zlib-1.16/zlib-1.1.4'
make[1]: Leaving directory `/home/mbletzin/nmi/gpt/support/Compress-Zlib-1.16/zlib-1.1.4'
PERL_DL_NONLAZY=1 /usr/bin/perl -Iblib/arch -Iblib/lib -I/usr/lib/perl5/5.6.1/i386-linux -I/usr/lib/perl5/5.6.1 -e 'use Test::Harness qw(&runtests $verbose); $verbose=0; runtests @ARGV;' t/*.tt/examples..........ok                                                       
t/zlib..............ok                                                       
All tests successful.
Files=2, Tests=179,  1 wallclock secs ( 0.64 cusr +  0.09 csys =  0.73 CPU)
make[1]: Entering directory `/home/mbletzin/nmi/gpt/support/Compress-Zlib-1.16/zlib-1.1.4'
make[1]: Nothing to be done for `test'.
make[1]: Leaving directory `/home/mbletzin/nmi/gpt/support/Compress-Zlib-1.16/zlib-1.1.4'
build_gpt ====> building support/Archive-Tar-0.22
build_gpt ====> testing support/Archive-Tar-0.22
PERL_DL_NONLAZY=1 /usr/bin/perl -Iblib/arch -Iblib/lib -I/usr/lib/perl5/5.6.1/i386-linux -I/usr/lib/perl5/5.6.1 test.pl
1..10
ok 1
ok 2
ok 3
ok 4
ok 5
ok 6
ok 7
ok 8
ok 9
ok 10
build_gpt ====> building support/PodParser-1.18
build_gpt ====> building support/Digest-MD5-2.20
build_gpt ====> testing support/Digest-MD5-2.20
PERL_DL_NONLAZY=1 /usr/bin/perl -Iblib/arch -Iblib/lib -I/usr/lib/perl5/5.6.1/i386-linux -I/usr/lib/perl5/5.6.1 -e 'use Test::Harness qw(&runtests $verbose); $verbose=0; runtests @ARGV;' t/*.tt/badfile...........ok                                                       
t/files.............ok                                                       
t/md5-aaa...........ok                                                       
t/utf8..............ok                                                       
All tests successful.
Files=4, Tests=266,  1 wallclock secs ( 0.51 cusr +  0.07 csys =  0.58 CPU)
build_gpt ====> building packaging_tools

This output shows all of the tests passing as seen by the "ok" lines. Errors are seen as "not ok" and will include a lot of other output. If you see errors please rerun the test with the -verbose flag and sent the results to the gpt-help list. Please include the name of the platform.

Solaris machines have a unique set of problems with GPT installations. In general most of the problems with Sun platforms can be solved by installing additional GNU packages from the sun freeware site. Here are some particulars:

GPT provides two solaris packages that contain binaries for GPT and the tools it needs. You need root access to install these packages. Here are the instructions:

The GPT rpm is a convenient way to install GPT without having to deal with the source tarball. However there are some issues that make the rpm less portable than usual. The best solution to these problems is to install GPT from source instead:

There are occasions when globus releases a new globus core package on its advisories page. This package is installed as source into the $GPT_LOCATION. After you download the package, simply copy it into gpt/etc/gpt/globus_core-src.tar.gz. Not that the name of the file needs to be exactly the same.

This chapter discusses the conventions that GPT has adopted. These conventions make it easier to describe and work within the grid deployment environment that GPT is designed to service.

This chapter discusses the purpose of build flavors and how they can be used. An important GPT convention is the format of the flavor label. This label is used in both package and file names. The figure here shows the components of a typical flavor name. The conventions is that any component that has the value none will contribute nothing to the label. The table here shows possible values of the flavor components. This table shows some examples

Figure 2.1. Flavor Labels Disected

GPT has a specific way to refer to binary packages. The format is shown in this figure.

Figure 2.2. Package Name Format

GPT has some wild card conventions that allow groups of packages to be selected. Essentially you can use the asterisk character "*" for any component of the name you don't care about. In addition, If the flavor or package type is left off of the name then wildcards are assumed. See this table for some examples.

This section talks about installing binary bundles and packages with GPT. The mechanics of installing with GPT are very simple. However, GPT gives you the flexibility to decide where to install software which occasionally require some additional thought. All of the examples in the section assume that you have installed GPT

The following GPT scripts are used for installing/updating/uninstalling packages:

The linux binary bundles found in the GPT testing repository are used to demonstrate the installation procedures.

A simple installation is where a package or bundle is installed into an empty installation location. For example, here is how to install a test bundle found here. The steps shown here use the linux GPT bundle installed on a Redhat system.

To remove the bundle from the previous example run gpt-uninstall with the name of the bundle:

bash$ gpt-uninstall -bundles newfoo
Bundle newfoo removed.
bash$ gpt-query
No packages were found that matched your query.

Since bundles are versioned in the same way packages, upgrading a bundle can be done by simply by running gpt-install on the newer bundle. However, there are a lot of things that happen under automatically with this operation. For this reason, the following example shows some switches that can be used to see more details of what GPT is doing.

Usually it is possible to install multiple bundles into the same location if they originate from the same organization. However there are times when certain bundles will conflict. Here are some examples:

This example shows conflicts because of mismatched package versions:

bash$ gpt-install newfoo-2.0-i686-pc-linux-gnu-bin.tar.gz   
ERROR: The following bundles cannot be installed
	newfoo ver: 2.0 is incompatible installed newfoo ver: 2.2.3
bash$ gpt-install newfee-2.0-i686-pc-linux-gnu-bin.tar.gz 
Error: The following package conflicts were found:
	globus_openssl-gcc32-rtl ver: 0.1 in bundle newfee conflicts with globus_openssl-gcc32-rtl ver: 0.10 in bundle newfoo
	globus_openssl-gcc32-pgm ver: 0.1 in bundle newfee conflicts with globus_openssl-gcc32-pgm ver: 0.10 in bundle newfoo
	globus_common_setup-noflavor-pgm ver: 2.0 in bundle newfee conflicts with globus_common_setup-noflavor-pgm ver: 2.1 in bundle newfoo
	globus_common-gcc32-rtl ver: 2.0 in bundle newfee conflicts with globus_common-gcc32-rtl ver: 3.5 in bundle newfoo
	globus_common-gcc32-pgm ver: 2.0 in bundle newfee conflicts with globus_common-gcc32-pgm ver: 3.5 in bundle newfoo

This example shows conflicts because of mismatched package flavors:

bash$ gpt-install newfee_static-2.2.3-i686-pc-linux-gnu-bin.tar.gz 
Error: The following package conflicts were found:
	globus_proxy_utils-gcc32-pgm_static ver: 0.5 in bundle newfee_static conflicts with globus_proxy_utils-gcc32-pgm ver: 0.5 in bundle newfoo
	globus_openssl-gcc32-pgm_static ver: 0.10 in bundle newfee_static conflicts with globus_openssl-gcc32-pgm ver: 0.10 in bundle newfoo
	globus_gsi_cert_utils-gcc32-pgm_static ver: 0.4 in bundle newfee_static conflicts with globus_gsi_cert_utils-gcc32-pgm ver: 0.4 in bundle newfoo
	globus_common-gcc32-pgm_static ver: 3.5 in bundle newfee_static conflicts with globus_common-gcc32-pgm ver: 3.5 in bundle newfoo

A -force flag can be used to install bundles which cause conflicts:

bash$ gpt-install -force newfee_static-2.2.3-i686-pc-linux-gnu-bin.tar.gz 
Bundle newfee_static successfully installed.
bash$ gpt-verify
Verifying Bundles...
Bundle: newfoo
Package globus_trusted_ca_42864e48_setup-noflavor-pgm ver: 0.5 is missing from i
nstallation
Package globus_proxy_utils-gcc32-pgm ver: 0.5 is missing from installation
Package globus_openssl-gcc32-pgm ver: 0.10 is missing from installation
Package globus_gsi_cert_utils-gcc32-pgm ver: 0.4 is missing from installation
Package globus_common_setup-noflavor-pgm ver: 2.1 is missing from installation
Package globus_common-gcc32-pgm ver: 3.5 is missing from installation

Bundle: newfee_static

Verifying run-time dependencies...

Verifying setup dependencies...

Verifying setup packages...
The following setup packages still need to be configured via gpt-postinstall:
	globus_common_setup-noflavor-pgm_static
	globus_trusted_ca_42864e48_setup-noflavor-pgm_static

ERROR: The collection of packages in /home/mbletzin/install/globus is not cohere
nt!

Under normal circumstances, GPT strictly inforces the integrity of installed bundles. However some organizations would rather distribute individual packages as "patches" to bundles. For this reason, gpt-install will allow updated packages to be installed that violate installed bundle integrity. The following shows how package updates work.

The trick to installing update bundles is that they need to be installed all at the same time. If they aren't then GPT may complain about version mis-matches of packages that are shared between the bundles and refuse to do the upgrade. The following exercise illustrates this:

Frequently an installation is used to build other software. The bundles for this are called software development kits (SDK) and are a collection of 'dev' and 'rtl' packages. For this type of installation, an additional step is required to install whatever build flavors are needed. To install a set of flavors do the following:

gpt-build -nosrc LIST OF FLAVORS

If you have no idea what flavors are needed then do the following:

gpt-build -nosrc -std-flavors

GPT also supports bundles that contain rpm's. These bundles are installed and uninstalled in the same way as normal GPT bundles. GPT keeps track of which packages are RPM's and which are GPT.

Bundles created with versions of GPT older than 3.0 are treated differently because they are not versioned. For one thing these older bundles cannot be updated; only installed and uninstalled. Another difference is that many of the package conflicts are ignored by gpt-install to replicate the behaviour of the older GPT's. Here are some examples to illustrate the differences:

This example shows how gpt-install installs two bundles which have package conflicts. When bundle foo_dbg-2 is installed then the packages that are conflicting are ignored. The result is that the integrity of the bundle is lost. Using the -force flag causes the opposite behaviour. The installed bundle foo-2 would loose its integrity.

bash$ gpt-install foo-2.2.3-i686-pc-linux-gnu-bin.tar.gz 
Bundle foo-2 successfully installed.
bash$ gpt-install foo_dbg-2.2.3-i686-pc-linux-gnu-bin.tar.gz 
Bundle foo_dbg-2 successfully installed.
bash$ gpt-verify
Verifying Bundles...
Bundle: foo-2

Bundle: foo_dbg-2
Package globus_openssl-gcc32dbg-pgm ver: 0.10 is missing from installation
Package globus_common-gcc32dbg-pgm ver: 3.5 is missing from installation
Package globus_gsi_cert_utils-gcc32dbg-pgm ver: 0.4 is missing from installation
Package globus_proxy_utils-gcc32dbg-pgm ver: 0.5 is missing from installation

This example shows that gpt-install still pays attention to package versions. Bundles that contain conflicting versions cannot be installed without the -force flag.

bash$ gpt-install foo-2.2.3-i686-pc-linux-gnu-bin.tar.gz 
Bundle foo-2 successfully installed.
bash$ gpt-install fee-2.0-i686-pc-linux-gnu-bin.tar.gz   
Error: The following package conflicts were found:
	globus_common-gcc32-rtl ver: 2.0 in bundle fee-2 conflicts with globus_common-gcc32-rtl ver: 3.5 in bundle foo-2
	globus_openssl-gcc32-pgm ver: 0.1 in bundle fee-2 conflicts with globus_openssl-gcc32-pgm ver: 0.10 in bundle foo-2
	globus_openssl-gcc32-rtl ver: 0.1 in bundle fee-2 conflicts with globus_openssl-gcc32-rtl ver: 0.10 in bundle foo-2
	globus_common-gcc32-pgm ver: 2.0 in bundle fee-2 conflicts with globus_common-gcc32-pgm ver: 3.5 in bundle foo-2
	globus_common_setup-noflavor-pgm ver: 2.0 in bundle fee-2 conflicts with globus_common_setup-noflavor-pgm ver: 2.1 in bundle foo-2

This section discusses building source packages and bundles with GPT. Building packages and bundles is more difficult than installing their binary equivelent. However it does allow the software to be patched and installed on platforms where binaries are not available. GPT has been designed to provide a number of tools that allow for building problems to be debugged and fixed. The focus of this section is to show the various building techniques rather than attempting to solve all building problems. In addition, this section spends some time discussing the difference between an installation that was built and an installation that was installed via binary packages and bundles. All of the examples in the section assume that you have installed GPT

The following GPT scripts are used for building packages:

The source bundles and packages found in the GPT testing repository are used to demonstrate the installation procedures. The source package is globus_common-3.5.tar.gz, and the source bundle is globus-data-management-server-2.2.3-src_bundle.tar.gz

gpt-build accepts flavors, packages, and bundles for input and the builds all of the packages and bundles with all of the flavors. Each flavor is built in the same source directory of a package. To build the globus_common package you star with an empty $GLOBUS_LOCATION and run gpt-build gcc32 gcc32pthr globus_common-3.5.tar.gz. What you see is actually two packages being built with the two flavors. The globus_core package is used to install flavor specific files into $GLOBUS_LOCATION which are then used to help build the other packages. See this section for details. Here are some comments on the build output:

The following is a package removal message:

 gpt-build ====> REMOVING empty package globus_core-gcc32-pgm_static 

Each source package builds several binary packages types. However not all package types are built. In this case the package was built using dynamic linking which means that the static binary package is empty.

The following is noise that should be ignored. The make distclean command is needed to clean out a previous flavor. In this case there was no previous flavor.

 make distclean
make: *** No rule to make target `distclean'.  Stop.

The previous example left behind a BUILD subdirectory. Inside this subdirectory are the unpacked source packages that were built. You can build more flavors of globus_common by running gpt-build -srcdir=globus_common-3.5 gcc32 gcc32dbg gcc32dbgpthr. You can see that gpt-build skips over the gcc32 flavor because it was built previously. Reinstalling the same flavor of a package can be done through the -force flag.

The -verbose flag tells gpt-build to print out all of the details of the build. Run the command gpt-build -force -verbose -srcdir=globus_common-3.5 gcc32 to see this. As you can see gpt-build actually builds the globus_common package using the normal ./configure, make, and make install commands. Other packages may have slightly different commands.

In your original working directory run the command gpt-build gcc32 gcc32dbg -log bundles/globus-data-management-server-2.2.3-src_bundle.tar.gz and then go on a coffee break :). This command creates a ./logs directory which contains the a log file for each package that was built. The file gpt-build.log is the overall log file that shows which package was built when.

After the bundle build is completed you can run the gpt-postinstall and get a working installation. There are some differences though. A gpt-query -what-bundles shows that the globus-data-management-server bundle definition file was not installed. This is a limitation of the current version of gpt-build. In addition a gpt-query globus_common command reveals that there are 4 different flavors of the 'pgm' package installed. These packages in a normal installation would cause file conflicts. However for a built installation gpt-build saves a copy of each file in a flavored location. If you look at for example the $GLOBUS_LOCATION/bin directory, you will see the program globus-hostname in the bin directory and also in the bin/gcc32/shared directory. The flavor of the globus_hostname program that is in the bin directory is the last flavor built which in this case is gcc32dbg. The easiest way to switch the flavor is to run gpt-pkg. For example the command gpt-pkg globus_common-gcc32-pgm will switch the flavor to gcc32 as a side effect of creating a GPT package.

After the bundle building of the previous section, the ./BUILD directory contains a lot more files and subdirectories. In addition to the source packages and source directories, the directory now contains a couple of files related to the bundle itself. It is better to use this set of directories to rebuild packages because using the bundle tarfile will unpack everything again and wipe out any changes made in these directories. The new package directories can be used to rebuild patched packages as shown in the following operation on the package globus_io:

Rebuilding multiple packages in a bundle is trickier because the packages need to be fed into gpt-build in dependency order. The file packaging_list lists the package names in dependency order so this list can be used to order the package source directories. For those of you who are too lazy to write your own, you can use this script.

gpt-build by default will build a package if it is newer than the installed version. However, the script does not use the same algorithm as gpt-install. Because of this it will not do bundle upgrades, nor will it remove packages that are no longer defined in a new bundle.

A build flavor is a label that describes the set of build options used to build a binary package. There are a number of build options implemented by both compilers and linkers that can lead to binary incompatibilities when they are not invoked consistently for an application and all of its supporting libraries. Thread support, number sizes, and the message passing interface (MPI) packages are some examples of these types of build options. An executable that is linked with a library that was build with a different set of build options will not behave as expected and frequently fail. GPT provides a means of describing and managing these options through the use of build flavors. The conventions that GPT uses for build flavors are discussed here

The relationship between the components of the flavor label and the build options are set in the file $GPT_LOCATION/etc/gpt/globus_flavor_labels.conf. The package globus_core translates the definitions in this file into a build environment that can be accessed by the tools gpt_build_config and globus-makefile-header. The tool gpt-build can translate translate build options into options that a packages configure script can understand.

The flavor translation, which happens the globus_core package is configured is a two stage process. First GPT uses the globus_flavor_labels.conf file to translate a flavor label into a set of shell variables and command line switches that globus_core's configure understands. This script then uses autoconf style probing to verify the flavor and then sets up files that define the build environment. These files are then installed with the make install command. All of this happens automatically when gpt-build is invoked with a new flavor.

There are two ways that the build flavors can be customized. The first is to build globus_core manually. The second way involves editing the globus_flavor_labels.conf file.

./configure --help

CC=gcc; export CC; \
LDFLAGS='-L/home/mbletzin/install/globus/lib'; export LDFLAGS; \
./configure  --with-flavor=kumquat --enable-debug --with-threads=pthreads

make

make install

<flavor_choice>
<config std="yes" switch="CC=gcc" label="gcc"/>
<config std="yes" switch="CC=cc" label="vendorcc"/>
</flavor_choice>

<flavor_choice>
<config std="yes" switch="--with-threads=pthreads" label="pthr"/>
<config std="no" switch="--with-threads=solaristhreads" label="solthr"/>
<config std="yes" nolabel="nothreads"/>
</flavor_choice>

Several limitations on build flavors exist because of the way build flavors are currently implemented in GPT.

This section discusses how to create bundles. Bundles can contain either source packages or binary packages. This section discusses the creation of both.

The GPT script is used to bundle packages gpt-bundle is used to create binary and sources packages as well as bundle definition files. The GPT script gpt-pkg can be used to create binary packages that are stand alone.

Binary bundles are created from an installation. Therefore it is simple to build a collection of packages, do some validation testing, and then wrap them up in a bundle. The following exercise shows how this is done although we will cheat by installing binary packages rather than building them from source.

The file newfoo-2.0.gpt-bundle.xml from the previous example is a typical bundle definition file. It is in an XML document that follows this DTD. The bundle definition file is not only a product of gpt-bundle but can also be an input. The file can be used as a template for a group of bundles. With this use, the data in the file is overriden by gpt-bundle command line options.

Lets take a closer look at the definition file.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE GPTBundleData SYSTEM "gpt_bundle.dtd">
<GPTBundleData Name="newfoo" >
<BundleInfo >
<Description >EMPTY</Description>
<ContactInfo ContactEmail="EMPTY" ContactName="EMPTY" />
<BundleDocs BundleDocsDesc="EMPTY" BundleDocsURL="EMPTY" />
</BundleInfo>
<BundleReleaseInfo >
<BundleStability Release="EMPTY" />
<BundleVersion Age="0" Major="1" Minor="0" />
<VersionLabel >2.0</VersionLabel>
<TypeOfBundle ContentsType="gpt" />
</BundleReleaseInfo>

This section is the top of the definition file and contains data that pertains to the bundle as a whole. For this particular file, several of the elements contain "EMPTY" which indicates that they need to be edited. As mentioned previously, several of these elements can also be set from the gpt-bundle command line. The following table describes these elements.

The element TypeOfBundle is generated by GPT.

The next section is a list of packages:

<PackageList >
<IncludedPackages >
<Package PackageFlavor="gcc32" PackageName="globus_common" PackageType="pgm" PackageVersion="3.5" />
<Package PackageFlavor="gcc32" PackageName="globus_common" PackageType="rtl" PackageVersion="3.5" />
<Package PackageFlavor="noflavor" PackageName="globus_common_setup" PackageType="pgm" PackageVersion="2.1" />
<Package PackageFlavor="gcc32" PackageName="globus_gsi_callback" PackageType="rtl" PackageVersion="0.3" />
<Package PackageFlavor="gcc32" PackageName="globus_gsi_cert_utils" PackageType="pgm" PackageVersion="0.4" />

Each package is listed with its build flavor, package name, and GPT version. The version is of the form MAJOR.MINOR. The version listed in the definition is overidden by the version of the package actually found in the installation when the bundle definition file is regenerated by gpt-bundle. Packages can also be added to this section by listing them on the command line.

<ExcludedPackages >
</ExcludedPackages>

This section has the same format as IncludedPackages but lists packages that should be excluded. The default behavior for gpt-bundle is to bundle the entire dependency tree of an included package. Listing packages in the ExcludedPackages section allows the dependency to be pruned. Packages can also be added to this section by using the gpt-bundle command line switch -exclude='PACKAGE_NAME'.

gpt-bundle allows you to specify a template definition that can be used to generate multiple bundles that have elements in common. Any bundle definition (such as the one generated in the previous section) can serve as a template. A completely generic template can be generated with the command gpt-bundle -template. The resulting template is in a file called BundleTemplateXML_FILE.xml. The following exercise shows how to use a template to generate different bundles:

gpt-bundle by default will pull into a bundle the entire dependency tree of a package. This inclusion is keyed off of the package type of the package. The package type specifies the deployment environment which in turn dictates which dependencies are followed. This table shows how this works:

There are three different ways to control the inclusion of dependent packages. These can be specified both from the gpt-bundle command line and the bundle definition file. The following table summarizes these.

The following shows some examples of how dependency inclusion can be controlled from the command line.

gpt-pkg can be used to create stand alone packages. It accepts a subset of the flags for gpt-bundle.

RPM binary bundles and packages can be created using the -native flag. The flags -rpmlicense and -rpmprefix can be used to insert alternate information in the rpm header of each package. The binary bundles are a collection of rpms that are wrapped in the same way that GPT packages are. Therefore they can be installed the same way the other binary bundles are. When the bundles are installed the information in the bundle definition file is not stored in the RPM database. The database will contain the information for each package in the bundle. GPT does not have any way to create source rpm's.

Creating a package from software can be an intimidating experience. Under some circumstances it has to be because the goal of the packaging task is to automate the building, installing, and uninstalling of your software for users who have no interest in becoming familiar with compilers, build tools, or shell debugging; things that you can do in your sleep. The complexity of the task also has a lot to do with how much time has been spent designing the deployment aspect of your software. There are deployment issues such as binary relocatability, runtime configuration, dependencies, and platform portability that can make a packaging task a lot tougher if they are not handled properly in the software. In the next few chapters, we will illustrate packaging tasks from the very easy examples to some more complex ones in the later chapters. This chapter explains some important requirements that you need to consider when packaging software.

One of the main problems with installed binary packages is that there is no guarantee that the programs, libraries, and other package contents will wind up in the same absolute location as the one they were built and installed in. Large packaging organizations such as Redhat and SUSE can offer this guarantee because they ship entire operating systems. However computers involved in grids are independently administered which means that software locations are determined by the local administrator (or user for non-root installations).

A package is considered relocatable if there are no absolute paths that are embedded in the package contents and that are used by the software. Paths that are in text files can be modified during runtime by setup packages. Alternatively packages can use environmental variables to locate files.

All applications have a way to allow users to redefine the application's features. A good way to do this is with a configuration file that the application reads during runtime. Another popular method is to use the tool GNU autoconf to handle user configuration requests during the build process. This particular method however becomes a problem when a user installs binary packages. This type of installation does not have a build step and so the user has to accept whatever configuration the package builder has defined. One could possibly build a set of binary packages that encompass the range of user choices however this can quickly become unmanageble. A better solution is to build in all possible features at runtime and then design a runtime configuration method for the user.

Chances are, your software used libraries or executables from software developed by others. Unfortunately you cannot count on these being installed on end user machines unless they are part of a typical UNIX operating system. If this software hasn't already been packaged, you will need to package it or find some way to do without it. To make the situation worse, you also need to be aware of the version of the dependent software that is installed. There is no guarantee that every user will have the same version of the software installed. Because of this it is better to stick with older versions or bundle your software package with the specific versions that need to be deployed with it.

One of GPT's major strengths is that it allows packages to be built on any UNIX platform that has the appropriate GPT prerequisites installed. Once your software package becomes a part of a distribution of GPT packages, user expectations will include being able to build it on all sorts of UNIX platforms. GPT does not provide an answer for this itself but ti is compatible with tools such as GNU autoconf and GNU libtool which provide some portability support. The last packaging chapter will discuss how to use these tools.

The concepts discussed here are important to GPT. However these deployed requirements are also important in general. The first experience an end users has with software is its deployment. The requirements discussed in this section will make this experience more pleasant.

This chapter starts of with packaging a simple perl script. It then adds a configuration file to illustrate relocation problems. The final example is a version of hello-world written in C which uses a shared library and a Makefile. All of the examples in the section assume that you have installed GPT and added $GPT_LOCATION/sbin and $GPT_INSTALL_LOCATION/bin to your path.

The following GPT scripts are used for creating a simple package:

Here is a simple script that we will package first:

#! /usr/bin/env perl

print "Hello World\n";

There is nothing special about this program other than instead of calling perl directly it calls the program env. The problem with scripts in general is that they need to have the first line hard-coded with the full path of the interpreter which executes the script. This is not a problem with interpreters like the Bourne shell which is always found at /bin/sh. However for a program such as perl there are no such guarantees. The env program is another executable that is always located in the same directory. It can be used to find perl and then execute it with the rest of the file. This is thus a cheap way to achieve relocatability. What is not addressed by this env is finding the correct version of perl. For this you would need a hard-coded path that is set by a setup package.

To turn this into a package we need to do a couple of things. The first is to decide where the script should be located in a user installation. Like most executables we will want that to be $GPT_INSTALL_LOCATION/bin. Next we need to create a couple of files.

The first file is called filelist and it is always located in the top directory of a package. This contains the a listing of the installed files. NOTE that this is not a listing of files in the source directory but rather a listing of files in the installation. Since we only have one file our filelist will only have one entry:

bin/hello-world.pl

Filelists can also be generated and/or modified during the build process. A filelist file only need to be present when gpt-build is installing the packaging data after building the software.

The second file that needs to be created is the source packaging data file called pkg_data_src.gpt. This file also needs to be in the top source directory (it can also be in a subdirectory called pkgdata). This file contains the build instructions for the package as well as package definitions. These definitions are automatically transposed into the binary packages that result from the build.

The first section of the file contains the XML header as well as the name, version, and other identity data of the package. The Aging_Version element has three attributes that are discussed here. One thing further to notice is that we switched the hyphen in the hello-world script to an underline in the pacakge name. Hyphens are special characters with GPT and other package managers. They are often used to delimit fields in a filename. It is legal to use a hyphen in a package name but not wise^[1].

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE gpt_package_metadata SYSTEM "package.dtd">

<gpt_package_metadata Name="hello_world" Format_Version="0.02">

    <Aging_Version Major="0" Minor="0" Age="0"/>
    <Description >Simple Packaging Example</Description>
    <Version_Stability Release="Experimental"/>

The other major section is the Build instructions shown below which instruct GPT how to install the script. As can be seen the build steps are standard bourne shell commands which need to be platform portable.^[2]. The only thing special is INSTALLDIR_GPTMACRO which is a GPT build macro. A list of these macros are found here. This particular macro gets expanded into the user's installation location before the build step is executed.

    <Build_Instructions>

      <Build_Step>mkdir -p INSTALLDIR_GPTMACRO/bin</Build_Step>
      <Build_Step>cp hello-world.pl INSTALLDIR_GPTMACRO/bin</Build_Step>
      <Build_Step>chmod 755 INSTALLDIR_GPTMACRO/bin/hello-world.pl</Build_Step>

    </Build_Instructions>

The complete set of files for this package can be found here.

The directory containing these files can now be recognized as source by GPT. A gpt-build command will install the script and create a binary package entry in the installation:

bash$ cd examples/hello_world
bash$ gpt-build
gpt-build ====> CHECKING BUILD DEPENDENCIES FOR hello_world
gpt-build ====> Changing to /home/mbletzin/nmi/gpt-docs/examples/hello_world
gpt-build ====> BUILDING FLAVOR 
gpt-build ====> Changing to /home/mbletzin/install/globus/etc
gpt-build ====> REMOVING empty package hello_world-noflavor-dev
gpt-build ====> REMOVING empty package hello_world-noflavor-rtl

Running hello-world.pl will now work. In addition, running gpt-query shows that hello_world-noflavor-pgm pkg version: 0.0.0 is now a part of the installation. GPT has figured out for you that no build flavors have been used and that the script belongs in a pgm package.

This next example has hello-world.pl reading a data file. The main problem here is that because of relocatability, the script cannot know where the data file is located. To solve this the script reads the environmental variable $GPT_INSTALL_LOCATION^[3] to find the file. The new script looks like this:

#! /usr/bin/env perl

print "Hello World\n";

# Get the value of the $GPT_INSTALL_LOCATION variable
my $installdir= $ENV{'GPT_INSTALL_LOCATION'};

die "Need to set \$GPT_INSTALL_LOCATION\n" if ! defined $installdir;

open CFG, "$installdir/etc/hello_world.data";

my @contents =  <CFG> ;

print @contents;

The filelist needs to list the new data file which we will install in $GPT_INSTALL_LOCATION/etc. The packaging data file also needs to change. In this file, the first thing to do is to increment the minor version number. We do this because we want to make sure that when the package is built or installed, it will replace the package from the previous example.

Incrementing Major and Minor version numbers is a judgement call. The rule of the thumb is that the Major number is incremented if the runtime interface changes. In this case, the run time interface is the command line invocation of the script. If we add command switches to the script, change its name, or add another script then the Major number should be incremented^[4]. The new header looks like this:

<gpt_package_metadata Name="hello_world" Format_Version="0.02">

    <Aging_Version Major="0" Minor="1" Age="0"/>
    <Description >Simple Packaging Example</Description>
    <Version_Stability Release="Experimental"/>

Running gpt-build on these new changes produces the following:

bash$ gpt-build
gpt-build ====> CHECKING BUILD DEPENDENCIES FOR hello_world
gpt-build ====> Changing to /home/mbletzin/nmi/gpt-docs/examples/hello_world2
gpt-build ====> BUILDING FLAVOR 
gpt-build ====> Changing to /home/mbletzin/install/globus/etc
gpt-build ====> REMOVING empty package hello_world-noflavor-dev
WARNING: "/home/mbletzin/install/globus/etc/hello_world.data" not found
gpt-build ====> REMOVING empty package hello_world-noflavor-rtl

This shows that we forgot to add an installation command for the data file. GPT will produce warnings for any files listed in the filelist that are missing from the installation. When the packaged is archived, these missing files produce errors because native package managers cannot handle missing files^[5].

Here are the fixed build instructions:

    <Build_Instructions>

      <Build_Step>mkdir -p INSTALLDIR_GPTMACRO/bin</Build_Step>
      <Build_Step>cp hello-world.pl INSTALLDIR_GPTMACRO/bin</Build_Step>
      <Build_Step>mkdir -p INSTALLDIR_GPTMACRO/etc</Build_Step>
      <Build_Step>cp hello_world.data INSTALLDIR_GPTMACRO/etc</Build_Step>
      <Build_Step>chmod 755 INSTALLDIR_GPTMACRO/bin/hello-world.pl</Build_Step>

    </Build_Instructions>

The complete set of files for this package can be found here.

This example is a hello package written in C that has a shared library and a makefile. The makefile simplifies the build instructions because it takes care of installing the files. However, the makefile will still need to get location information from GPT in order to install the files correctly. The normal way to do this is by passing in shell variables when make is invoked.

Here is the makefile. It includes the normal makefile targets for building and installing C code:

INSTALL_LOCATION=/usr/local

.c.o :
	cc -I . -c $<

all: hello-world-c

hello-world-c: hello-world.o libhello_world.so
	cc -o $@ hello-world.o -L. -lhello_world

libhello_world.so: libhello_world.c libhello_world.h
	cc -o $@ -shared libhello_world.c

install: hello-world-install

hello-world-install:
	mkdir -p $(INSTALL_LOCATION)/bin
	cp hello-world-c $(INSTALL_LOCATION)/bin
	chmod 755 $(INSTALL_LOCATION)/bin/hello-world-c
	mkdir -p $(INSTALL_LOCATION)/lib
	cp libhello_world.so $(INSTALL_LOCATION)/lib
	mkdir -p $(INSTALL_LOCATION)/include
	cp libhello_world.h $(INSTALL_LOCATION)/include

clean:
	rm -f *.o hello-world-c libhello_world.so

The installation location is stored in the shell variable $INSTALL_LOCATION. The variable is set via a GPT macro with the following build instructions:

    <Build_Instructions>

      <Build_Step>make INSTALL_LOCATION=INSTALLDIR_GPTMACRO</Build_Step>
      <Build_Step>make install INSTALL_LOCATION=INSTALLDIR_GPTMACRO</Build_Step>

    </Build_Instructions>

The complete set of files for this package can be found here.

This example adds dependencies to a package. Dependencies are useful in determining the completeness of installations. Users can be spared obscure errors by doing a dependency check after installing the package^[6]. Dependencies can be determined from the source code and the makefile.

Here is the relevent Makefile section

guten-tag-welt: guten-tag-welt.o
        cc -o $@ guten-tag-welt.o -L$(INSTALL_LOCATION)/lib -lhello_world

This shows that the program needs to be linked with the libhello_world.so library from the previous example package. Since this is a shared library, the dependency exists both during package building and runtime. The dependency is expressed as follows

    <Source_Dependencies Type="pgm_link" >
      <Dependency Name="hello_world_c" Package="pgm">
        <Version>
          <Simple_Version Major="0">
          </Simple_Version>
        </Version>
      </Dependency>
    </Source_Dependencies>

The dependency type is 'pgm_link' because we are linking against a program. If we had a guten_tag library that we were linking against, the dependency type would be 'lib_link'. The reason for the type difference is that lib_link dependencies are used by GPT to assemble dependency trees that are then converted into link lines for building programs. pgm_link dependencies should are not used for this. The hello_world_c package which contains the needed library is a 'pgm' package type when built, which indicates the setting for the Package attribute.

The source code reveals two more dependencies. The first is the include statement shown here:

#include "libhello_world.h"

This is a compile dependency to the hello_world_c package. Although this dependency may seem redundant with the linking dependency discussed earlier, it is still a good idea to express it because GPT will check different types of dependencies at different times