Home

pkg(8) is the new package manager for FreeBSD, it is the only package manager support for FreeBSD 10 and newer, while it is available along with the old package management tools on FreeBSD 8 and 9.

In September 2014 it will be the only package management tool supported on all version of FreeBSD. Unlike the old package management tools, pkg(8) is not integrated directly into the base system, instead of that a bootstrap tool pkg(7).

This article will cover pkg(8) as of version 1.2.6.

history

The old package management tools has been developed in 1993, have been maintained since but received very little improvements. While in 1993 it did fits the requirements (few packages, simple semantics involved in packaging) in the Y2K, they are becoming less and less accurate.

In 2010 Julien Laffaye and I decided to have a look at how complicated it would be to create a package management tool that would be able to properly handle binary upgrades, be fast with lots of packages installed, be safe, and allows to have control over the whole processes of package manipulations. We were also willing to provide a high level library so anyone willing to write tools that deals with packages would be able to just plug on the library and do safe operations.

A few years later, pkg(8) has grown, and regular contributors has join the development including (but not only): Bryan Drewery, Matthew Seaman, Vsevolod Stakhov.

The bootstrap mechanism

preamble

pkg(8) is not available in base and we made the decision to never push it into base for the following reasons:

  1. FreeBSD do maintain multiple releases at the same time, meaning if you introduce a new feature in pkg(8) we have to wait until the End Of Life of the release not supporting that new feature to be able to use it in the ports. In long term this leads to stagnation (no one really improving the package tools) and introducing hacks instead of real solution into the ports tree.
  2. Requirements in packaging evolves quite quickly and the package tools has too follow that, the way we did packaging in 1995 and now is really different, having the ability to improve the package tools very quickly on all supported version is becoming a huge requirement.

the bootstrap

As pkg(8) is not available directly from the base system, a bootstrap mechanism (pkg(7)) as been introduced in the base system, it has been designed to be as seamless.

FreeBSD 8.4 is the oldest version including the bootstrap tool, FreeBSD 10 is the only one to have in its final version. This is focuses on what pkg(7) does on FreeBSD 10.

pkg(7) is in fact /usr/sbin/pkg, when run it will look up for the pkg(8) binary (by default in ${LOCALBASE}/sbin), if the binary has been found then it will directly execute it, if not found it will propose to the user to bootstrap it.

Bootstrapping consist in fetching the latest pkg(8) packages from the official FreeBSD repositories for a given version on a given architecture.

  1. It will also fetch a signature file (pkg.txz.sig), containing a signature and a public key.
  2. Verify the public key against a list of fingerprints of known trusted public key
  3. Validate that the fetched pkg(8) package matches the signature
  4. Extract pkg-static from the package
  5. Install the package with the extracted pkg-static
  6. Execute pkg(8) with the arguments passed to the bootstrap

To determine where to fetch the package from and how to validate it pkg(7) shares the exact same mechanism of configuration files for repositories which will be describe later.

pkg(8) configuration option

In pkg(8) almost all options can be overwritten by an environment variable. the command pkg -vv will show all the support options and the current setup:

Version                 : 1.2.6
PACKAGESITE             : 
PKG_DBDIR               : /var/db/pkg
PKG_CACHEDIR            : /var/cache/pkg
PORTSDIR                : /usr/ports
PUBKEY                  : 
HANDLE_RC_SCRIPTS       : no
ASSUME_ALWAYS_YES       : no
REPOS_DIR               : [
  /etc/pkg/,
  /usr/local/etc/pkg/repos/,
]
PLIST_KEYWORDS_DIR      : 
SYSLOG                  : yes
AUTODEPS                : yes
ABI                     : freebsd:11:x86:64
DEVELOPER_MODE          : no
PORTAUDIT_SITE          : http://portaudit.FreeBSD.org/auditfile.tbz
VULNXML_SITE            : http://www.vuxml.org/freebsd/vuln.xml.bz2
MIRROR_TYPE             : SRV
FETCH_RETRY             : 3
PKG_PLUGINS_DIR         : /usr/local/lib/pkg/
PKG_ENABLE_PLUGINS      : yes
PLUGINS                 : [
]
DEBUG_SCRIPTS           : no
PLUGINS_CONF_DIR        : /usr/local/etc/pkg/
PERMISSIVE              : no
REPO_AUTOUPDATE         : yes
NAMESERVER              : 
EVENT_PIPE              : 
FETCH_TIMEOUT           : 30
UNSET_TIMESTAMP         : no
SSH_RESTRICT_DIR        : 
PKG_SSH_ARGS            : 
PKG_ENV                 : {
}
DISABLE_MTREE           : no
DEBUG_LEVEL             : 0
ALIAS                   : {
}

Repositories:
  FreeBSD: { 
    url             : "pkg+http://pkg.FreeBSD.org/freebsd:11:x86:64/latest",
    enabled         : yes,
    mirror_type     : "SRV",
    signature_type  : "FINGERPRINTS",
    fingerprints    : "/usr/share/keys/pkg"
  }
  • PKG_DBDIR is the directory where the database about local and remote packages is located
  • PKGCACHEDIR is the directory where the packages will be fetched into before being installed
  • HANDLE_RC_SCRIPT will tell pkg(8) to automatically restart the rc script after an upgrade
  • SYSLOG will make pkg(8) log everything it is doing via the syslog mechanism
  • REPOS_DIR is the directories where pkg can find the binary configurations files (when set from environment it should be defined the same way PATH is defined: REPOS_DIR="/path1:/path2:/path3"
  • PLUGINS_CONF_DIR is the directory where the plugins can find there configuration files

pkg(8) cold start

For the rest of this article we will consider a setup where LOCALBASE is defined as /usr/local.

Important files or directory around pkg(8):

/etc/pkg/*.conf
/usr/local/etc/pkg.conf
/usr/local/etc/pkg/repos/*.conf
/usr/local/lib/libpkg.so.1
/usr/local/sbin/pkg
/usr/local/sbin/pkg-static

pkg-static is a statically linked version of pkg(8) it will allow users to do binary package manipulation even when upgrade from a major version to another major version of FreeBSD.

When the pkg(8) binary is invoked, it will first load /usr/local/etc/pkg.conf except if specified another location via the -C option. For all configuration entries if there is already an environment variable defining it, it will have the priority over the configuration file.

Now the configuration file has been read, pkg(8) can lookup for for repository definitions in the directory defined via the REPOS_DIR. It will read all the .conf files in those directories and will look for entries 1 or N entries like this:

<NAME> : {
	url : "<url>",
	enabled: true,
	mirror_type: "SRV",
	signature_type: "FINGERPRINTS",
	fingerprints: "/usr/share/keys/pkg",
	pubkey: "/etc/pkg/pub.key"
}

First time <NAME> is found then the "url" element is mandatory. Each time <NAME> the options it defines will overwrite the previously defined one.

  • url: Represent the url where pkg(8) should fetch the packages from, ssh, http(s), ftp, file are supported protocols, if the MIRROR_TYPE is set to "SRV" then the url should be prepend by "pkg+",
  • enabled: always true by default, one can set it to false to disable a given repository
  • mirror_type: by default it is NONE possible values are:
    • HTTP: The repository URL should download a text document containing a sequence of lines beginning with `URL:' followed by any amount of while space and one URL for a repository mirror. Any lines not matching this pattern are ignored. Mirrors are tried in the order listed until a download succeeds.
    • SRV: For an SRV mirrored repository where the URL is specified as pkg+http://pkgrepo.example.org/ SRV records should provides the list of mirrors where the SRV priority and weight parameters are used to control search order and traffic weighting between sites, and the port number and hostname are used to construct the individual mirror URLs.
  • signature_type: by default it is NONE, the following entries can be set here:
    • FINGERPRINTS: the public key will be fetched along with the catalog and verified against a list of revoked and trusted fingerprint of public keys
    • PUBKEY: use a public key on the system to validate the remote catalog
  • fingerprints: path to a directory will should contain 2 subdirectories: trusted and revoked" each of which should contain file describing the fingerprint of the concerned public key and the unfction used to create that fingerprint (right now only sha256 is supported as a function). This entry is only useful if "signature_type" is set to "FINGERPRINTS"
  • pubkey: path to the public key to be use, only useful if "signature_type" is set to "pubkey"

Once all the above is done pkg(8) is ready to operate. The bootstrap follow the exact same sequence.

Different commands

Unlike the old package management tool, pkg is a single binary with lots of small commands

$ pkg help
Usage: pkg [-v] [-d] [-l] [-N] [-j <jail name or id>|-c <chroot path>] [-C <configuration file>] [-R <repo config dir>] <command> [<args>]

Global options supported:
        -d             Increment debug level
        -j             Execute pkg(8) inside a jail(8)
        -c             Execute pkg(8) inside a chroot(8)
        -C             Use the specified configuration file
        -R             Directory to search for individual repository configurations
        -l             List available commands and exit
        -v             Display pkg(8) version
        -N             Test if pkg(8) is activated and avoid auto-activation


Commands supported:
        add            Registers a package and installs it on the system
        annotate       Add, modify or delete tag-value style annotations on packages
        audit          Reports vulnerable packages
        autoremove     Removes orphan packages
        backup         Backs-up and restores the local package database
        check          Checks for missing dependencies and database consistency
        clean          Cleans old packages from the cache
        config         Display the value of the configuration options
        convert        Convert database from/to pkgng
        create         Creates software package distributions
        delete         Deletes packages from the database and the system
        fetch          Fetches packages from a remote repository
        help           Displays help information
        info           Displays information about installed packages
        install        Installs packages from remote package repositories
        lock           Locks package against modifications or deletion
        plugins        Manages plugins and displays information about plugins
        query          Queries information about installed packages
        register       Registers a package into the local database
        remove         Deletes packages from the database and the system
        repo           Creates a package repository catalogue
        rquery         Queries information in repository catalogues
        search         Performs a search of package repository catalogues
        set            Modifies information about packages in the local database
        ssh            ssh packages to be used via ssh
        shell          Opens a debug shell
        shlib          Displays which packages link against a specific shared library
        stats          Displays package database statistics
        unlock         Unlocks a package, allowing modification or deletion
        update         Updates package repository catalogues
        updating       Displays UPDATING information for a package
        upgrade        Performs upgrades of packaged software distributions
        version        Displays the versions of installed packages
        which          Displays which package installed a specific file

A normal user maintain his system up to date will just have to run a couple of command:

$ pkg upgrade
$ pkg autoremove

The first will upgrade all the installed package to their latest version, the second will remove the now orphans (not depend on anymore, and not installed on purpose by the user).

Creating a package outside of the ports tree

While the ports tree is the natural way of creating packages for FreeBSD, it is rather easy to create your packages outside of the ports tree.

First let's have a look at what pkg create expects:

Usage: pkg create [-On] [-f format] [-o outdir] [-p plist] [-r rootdir] -m manifestdir
       pkg create [-Ognx] [-f format] [-o outdir] [-r rootdir] pkg-name ...
       pkg create [-On] [-f format] [-o outdir] [-r rootdir] -a

The first line it what we are looking for. We are considering here that the sources has been built and installed in a given DESTDIR and we want to package /usr/local/bin/foo and /usr/local/lib/libbar.so.1

$ find ${DESTDIR}
${DESTDIR}/usr/local/bin/foo
${DESTDIR}/usr/local/lib/libbar.so
${DESTDIR}/usr/local/lib/libbar.so.1

First we need to create a directory to push the manifest in there

$ mkdir ${MANIFESTDIR}

now create a ${MANIFESTDIR}/+MANIFEST

name: foo
version: 1.0
origin: mycompany/foo
categories: [ mycompany, devel ]
comment: foo is a nice tool
www: http://mycompany/foo
maintainer: me@mycompany
prefix: /usr/local

The above is the minimum necesarry it is written in libucl format which is very flexible and allow differents syntaxes The description for the packages can be added via a file ${MANIFESTDIR}/+DESC or by adding the following lines to the manifest

desc: <<EOD
foo is a fantastic tool developed by my company
on top of libbar.
EOD

A message can be added to the package by either writing it in a ${MANIFESTDIR}/+DISPLAY or by adding the following line to the manifest message: <<EOD This is a message for our user I hope you do like foo EOD

Create a plist (like the ports do):

bin/foo
lib/libbar.so.1

The package can now be created:

$ pkg create -m ${MANIFESTDIR} -r ${DESTDIR} -o ${OUTDIR}

This will create a ${OUTDIR}/foo-1.0.txz ready to be distributed.

Future challenges of pkg

For next version we aim at bringing lots of new features and improve the user experience. pkg(8) has recently gain a real sat which is the basement to improve the flexibility of the package format and package building tools (the ports tree).

Allowing to bring smart dependencies: instead of depending on the specific version of a package we can depend on a valid range of versions

Allowing to bring provides/requires: do not depend on a package but rather on a feature (Requires: perl, http). Provides/Requires can bevery useful in multiple case for example:

  • program A requires libMagick.so and ImageMagick and ImageMagick-nox11 both provides libMagick.so, that let the user chose the version he wants instead of having to rebuild the A package and chose which ImageMagick package to use at build time.
  • progam B depends on a perl interpreter, it can just requires: perl and not a specific version of perl. right now the version is hardcoded in the dependency.

Work as also begun on having plugable backend for pkg(8) with the current one (the binary repositories) being just one of them, but one could imagine a "ports tree" backend, a CPAN,pear,pypy,gem backend.

For now in the background, pkg(8) uses origin (aka category/portname) to identify a package (determining what is an upgrade of what and so one) this is due to the fact the the same port could have multiple different name in the ports tree depending on options or multiple port could have the same name but different version (and still being different ports, not 2 version of the same port). In the meantime we have been able to fix the ports so that package names are consistent again, meaning we can switch back pkgname as the identifier for packages.

This change while it sounds not very signicant opens a lot of new possibilities for the ports tree:

  • sub packages: from one source in one port build multiple packages (like having one single port for php and all its modules)
  • flavours (provides): having the same package built with different options