Rebuilding the RPM repository

Sam Isaacson (sbi@nbcs.rutgers.edu)

build-r.pl configuration

build-r.pl generates a script that rebuilds the repository based on information specified in a ``profile.'' The profile is a text file with multiple sections:

Key Use
SPEC_DIRECTORIES This is mandatory. It is a list of directories which contain the specfiles pending rebuilding.
RPM_DIRECTORIES This is a list of directories which contain RPMs produced by the specfiles found in SPEC_DIRECTORIES. The directories listed do not need RPMs for each specfile. RPMs, if present, are used to add to the dependency information in the specfile.
BUILD_FIRST This is a list of specfiles (without the .spec extension). The specfiles listed in BUILD_FIRST will be built in the order listed in the profile before any other specfiles are built.
BUILD_ONLY This is a list of specfiles that specifies the only specfiles that should be built. If it is not present, every specfile in SPEC_DIRECTORIES will be built.
DO_NOT_BUILD This is a list of specfiles that may exist in SPEC_DIRECTORIES that should not be rebuilt.
DO_NOT_INSTALL This is a list of specfiles that should be built but not installed during the rebuilding process.
ADDITIONAL_DEPENDENCY_INFORMATION This is a list of whitespace-delimited pairs of specfiles. It indicates dependency information that should be added to that which is gathered by build-r.pl. Unless circular dependencies arise, the first specfile specified in a pair will be built before the second.
RUN_AFTER_INSTALL(foobar) This contains a shell script that should be run after specfile foobar is installed. Each specfile can have its own RUN_AFTER_INSTALL field.

Unless otherwise indicated, each field is optional. Each list is delimited by newlines. Each field begins with the key of the field, followed by the character '=' and a newline. Each field ends with a blank line. Any line beginning with the character '#' is ignored. See sol6,7.profile for an example of a profile.

Build machine setup

These instructions explain how to rebuild the RPM repository, which is necessary if you are moving to a new architecture or new Solaris version. If you need to restore a build machine, but don't want to rebuild the entire RPM repository, skip step 1, and after step 7, install everything in the stable and testing repositories except for duplicate packages.

  1. Run build-r.pl and save the script it produces. Note that build-r.pl cannot use FTP, so copy build-r.pl to a machine with a local copy of the repository and the specfiles you wish to rebuild. Copy one of the example profiles, edit as necessary, and run perl build-r.pl -v -p your-profile -o script-to-save.sh.

  2. Install Solaris on the build machine. Make sure that you have a huge /usr partition (4G for RPM without licensed software) and fairly large /var partition (1G), and make sure that there is enough room (approximately 1G) for Sun's CC in /opt. If the build machine's drive is too small, mount /usr/local by NFS and link /var/tmp and /opt to subdirectories in /usr/local after the install. Link /usr/local/src/rpm-packages/BUILD to a local directory, though.

    When choosing what packages to install, install everything. Later you can remove the problem packages: gnome, perl (SUNWpl5[pmu]; leave these on the machine if you are trying to build against Sun's perl), bash (SUNWbash), tcsh (SUNWtcsh), zsh (SUNWzsh), and less (SUNWless). If you want to build xboing and Csound, install the SUNWaudmo package. If you are setting up a 64-bit build machine, ensure that the 64-bit libraries are available (most of the packages have names equal to the 32-bit library packages, but with an 'x' at the end).

  3. On the build machine, mount alfred:/sos/oss-dist and alfred:/sos/oss-media to /sos/oss-dist and /sos/oss-media, respectively.

  4. Install all the recommended Solaris patches (use /sos/oss-dist/cfengine/bin/sun_patch.ksh).

  5. Install the latest version of Sun's C compiler and a license file (copy /sos/oss-media/sun-languages/sunpro.lic,node to /usr/local/flexlm/licenses/license.dat). Test the installation.

  6. On the build machine, bootstrap RPM. You do not need to install apt as well; if the rebuild of the repository works, apt will be installed (unconfigured, however) later. Set the permissions of /var/tmp and /var/local/tmp to 1777.

    Install the latest versions of gcc 2, GNU make, gzip, bzip2, slide, patch, and fileutils from the RPM repository. If you are using the repository from the summer of 2000, install gzip-1.2.4a (not gzip-1.3.0). Link the GNU build tools to /usr/local/bin: link /usr/local/gnu/bin/make to /usr/local/bin/make and /usr/local/bin/gmake, link /usr/local/gnu/bin/patch to /usr/local/bin/patch, and link /usr/local/gnu/bin/gzip to /usr/local/bin/gzip (and /bin/gzip on a Solaris 8 machine). Do not edit the PATH in the build script to put /usr/local/gnu/bin first instead; most packages cannot build with GNU binutils, but require GNU patch and GNU make.

  7. On the build machine, add a build user, add slide to /etc/group, and add the build user to slide. Change the owner of /usr/local/src/rpm-packages and everything under it to the build user. Add users and groups for qmail:

    Add to /etc/passwd:

        qmailq:x:30301:199::/:
        qmailr:x:30302:199::/:
        qmails:x:30303:199::/:
        qmaild:x:30297:199::/:
        qmaill:x:30298:199::/:
        alias:x:30296:199::/:
        qmailp:x:30300:199::/:
        

    Add to /etc/shadow:

        qmailq:NP:6445::::::
        qmailr:NP:6445::::::
        qmails:NP:6445::::::
        qmaild:NP:6445::::::
        qmaill:NP:6445::::::
        alias:NP:6445::::::
        qmailp:NP:6445::::::
        

    Add to /etc/group:

        qmail:*:2035:qmailq,qmailr,qmails
        nofiles:*:2036:alias,qmaild,qmaill,qmailp
        

  8. Copy all the specfiles from the repository into rpm-packages/SPECS and copy all the sources into rpm-packages/SOURCES. If you want to build RPMs of licensed software, copy the sources from /sos/oss-dist/sbi/licensed/SOURCES. Edit machine-header.spec if necessary (you might wish to set %{which_gnome} and %{which_perl} to SOLARIS).

  9. As the build user created earlier, edit and run the build script created earlier to build the repository (you need to set RPM_RPM_DIR, RPM_BUILD_CMD, GNOME_PATH, and LONGARCH). Log the output from the build script in case something goes wrong. The script should record lists of packages that failed to build or install in /usr/local/src/rpm-packages/SPECS.

    Don't worry if the list of packages that failed to build is long. It is impossible to maintain the consistency of the RPM repository as packages are added or changed individually. New file locations often break old specfiles in surprising ways. If a lot of packages failed, fix the broken specfiles, remove the packages you built, and run the build script again. Otherwise, build the broken packages by hand.

  10. Sign the output.

Remote build setup

  1. Set up ssh on the remote build machine. This requires installing and configuring task-ssh.

  2. Generate keys so that you can remotely log in as the build user. Use ssh-keygen -t rsa on the control machine to get a keypair; add ~/.ssh/id_rsa.pub to ~/.ssh/authorized_keys2 on the remote build machine.

  3. Create a script on the remote build machine in /usr/local/bin named rbrpm which runs rpmbuild $* with the path that you want rpm to use. For an example script which does this step see setup.sh.

  4. Add your machine to your remote_rpm tab file. Each entry in the tab file is a line which begins with machine-id: followed by a comma-delimited list of settings. machine-id must begin with architecture-manufacturer-os, which can optionally be followed by a '/' character and specific build machine information (which is ignored by remote_rpm). Each setting consists of a key, followed by the '=' character, followed by the value of the setting. remote_rpm recognizes the following keys:
    Key Use
    HOST This is the hostname of the remote build machine.
    USER This is the username of the remote build user. This user's authorized_keys2 (or authorized_keys) file must contain your public key.
    RPM This is RPM's %_topdir (e.g. /usr/local/src/rpm-packages).
    BUILDFLAGS This is a string which will be inserted before the commands you pass to remote_rpm if you are building a spec file. It can include spaces, and is optional.
    EXT This is the file suffix for RPMs produced by the build machine. It includes the preceding period.

    For example, see buildmachinetab.

  5. Run remote_rpm. When building RPMs, remote-rpm expects that the source files referenced in the spec file will be available in $RPM_SOURCE_DIR (i.e. %{_topdir}/SOURCES), and that spec files referenced in the spec file by relative path are located relative to the directory in which remote_rpm is run.

Signing packages

  1. Install perl-module-Digest-MD5, egd, and gpg. Optionally, set the owner of /usr/local/bin/gpg to root and the permissions to 4755 (this way gpg will not use swap). Run egd.pl ~/.gnupg/entropy and gpg --gen-key to generate a key for the repository. Run gpg --gen-revoke name of key and print the output. For security reasons you may wish to do all of this on a machine disconnected from the network.

  2. Add the following entries to ~/.rpmmacros:

    %_signature gpg
    %_gpg_path absolute path to home directory/.gnupg
    %_gpg_name full name of key with which you will sign packages
    %_gpgbin /usr/local/bin/gpg
    

    Note that these additions (save %_gpg_name) are necessary for signature verification as well.

  3. Run rpm --resign /usr/local/src/rpm-packages/RPMS/*/*rpm /usr/local/src/rpm-packages/SRPMS/*rpm.

  4. Publish your public key somewhere (try www.keyserver.net), and publish your fingerprint somewhere else.

Notes

Examples

sol6,7.profile

SPEC_DIRECTORIES=
/sos/oss-dist/solaris/rpm-packages/SPECS

RPM_DIRECTORIES=
/sos/oss-dist/solaris/rpm-packages/RPMS/sparc-sun-solaris2.7
/sos/oss-dist/solaris/rpm-packages/RPMS/sparcv9-sun-solaris2.8
/sos/oss-dist/solaris/rpm-packages/RPMS/sparc-sun-solaris2.6

BUILD_FIRST=
zlib
gzip
tar
bzip2
gnu-m4
autoconf
readline
patch
db
flex
bison
texinfo
gnu-make
gdbm
perl
groff
gnu-fileutils

DO_NOT_BUILD=
ghc
happy
cfengine-1.5.4-2
cfengine-1.6.0.b3-2
perl-header
machine-header
qt
RadiusPerl
msg

DO_NOT_INSTALL=
bonobo
rpm3
acrobat
imsl1
imsl2
lisrel8
lisrel83
matlab
sas6
sas7
sas81
splus
spss5
spss6
X_setup

# Strictly speaking this is unnecessary, since build-r.pl will get the
# dependency information anyway.  However, I'm not sure that all the spec
# files have the correct build requires, and if they are built in the wrong
# order they will not fail but will instead lack features.
ADDITIONAL_DEPENDENCY_INFORMATION=
audiofile      esound
esound         glib
glib           gtk+
freetype       imlib
ImageMagick    imlib
gtk+           imlib
imlib          gtk-engines
gtk-engines    ORBit
ORBit          gnome-libs
gnome-libs     libgtop
libgtop        libxml
libxml         libghttp
libghttp       libglade
libglade       gdk-pixbuf
gdk-pixbuf     control-center
control-center gnome-core
gnome-core     gnome-applets
gnome-applets  mc
DBI            Data-ShowTable
Data-ShowTable perlMysql
perlMysql      MIME-Base64
MIME-Base64    URI
URI            libnet
libnet         HTML-Parser
HTML-Parser    Digest-MD5
Digest-MD5     libwww
libwww         egd

RUN_AFTER_INSTALL(perl)=
for i in /usr/local/perl5/bin*; do
    j=`basename $i`
    slide ln -s $i /usr/local/bin/$j
done

RUN_AFTER_INSTALL(teTeX)=
PATH="$PATH:/usr/local/teTeX/bin/$LONGARCH"
export PATH

RUN_AFTER_INSTALL(emacs)=
PATH="$PATH:/usr/local/emacs20/bin"
export PATH

RUN_AFTER_INSTALL(rpm4)=
slide /usr/bin/chown -R sbi:studsys /usr/local/src/rpm-packages

setup.sh

#!/bin/sh

USERNAME="sbi"
HOME_DIR="/export/home/sbi"
BIN_DIR="/usr/local/bin"
PUBLIC_KEY="ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAteBBJF8Ycc7NFVwnFK3YEO0cWqOXTqN3GPXIy8Hyy/qUGvym4xK5jnfI57yFI3qcjbq0Rjc4TDw5CQsIf8RBdz5Px/SEYUumimemwrHVzZIfYGsafIZnR+UmZrxTX0K9x4XKV6mUR7f51z6tDwUJftDzl0TUsfSfq3kC+1IRiMs= sbi@radiohead.Rutgers.EDU"
echo "Copying $USERNAME's public key..."
mkdir -p $HOME_DIR/.ssh
echo $PUBLIC_KEY >> $HOME_DIR/.ssh/authorized_keys2

echo "Generating rbrpm script..."
cat <<EOF >$HOME_DIR/rbrpm
#!/bin/sh

PATH="/usr/local/bin:/usr/ccs/bin:/usr/bin:/opt/SUNWspro/bin:/usr/ucb:/usr/openwin/bin:/usr/sbin" rpmbuild \$*
EOF
slide mv $HOME_DIR/rbrpm $BIN_DIR/rbrpm
slide chmod 0755 $BIN_DIR/rbrpm

echo "Done."

buildmachinetab

# List of build machines and important data

sparc64-sun-solaris2.8/sun-perl,gnome:HOST=rosewalker,USER=sbi,RPM=/usr/local/src/rpm-packages,EXT=.solaris2.8-sparc64.sun-perl\,gnome.rpm
sparc64-sun-solaris2.8:HOST=mistere,USER=sbi,RPM=/usr/local/src/rpm-packages,EXT=.solaris2.8-sparc64.rpm
sparc64-sun-solaris2.7:HOST=johndee,USER=sbi,RPM=/usr/local/src/rpm-packages,EXT=.solaris2.7-sparc64.rpm
sparc-sun-solaris2.6:HOST=miracleman,USER=sbi,RPM=/usr/local/src/rpm-packages,BUILDFLAGS=--target sparc-sun-solaris2.6,EXT=.solaris2.6-sparc.rpm
sparc-sun-solaris2.7:HOST=pumpkinhead,USER=sbi,RPM=/usr/local/src/rpm-packages,EXT=.solaris2.7-sparc.rpm
sparc-sun-solaris2.8:HOST=omahacatdancer,USER=sbi,RPM=/usr/local/src/rpm-packages,EXT=.solaris2.8-sparc.rpm

$Id: build-repository-instructions.html,v 1.1.1.1 2001/12/14 20:38:46 sbi Exp $