Check-in [843c8a89c9]

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Initial import
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | trunk
Files: files | file ages | folders
SHA1: 843c8a89c93273499dec81da28c0900e1dfe1a3b
User & Date: bapt 2014-03-02 01:27:52
Context
2014-03-02
01:27
Initial import Leaf check-in: 843c8a89c9 user: bapt tags: trunk
01:25
initial empty check-in check-in: 738949b4b8 user: bapt tags: trunk
Changes

Added pkg.mkd.

















































































































































































































































































































































































































































































































































































































>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
# Overview 

pkg(8) is the new package manager for FreeBSD, it is the only package manager support for FreeBSD 10 and newer, while it is available along with the old package management tools on FreeBSD 8 and 9.

In September 2014 it will be the only package management tool supported on all version of FreeBSD. Unlike the old package management tools, pkg(8) is not integrated directly into the base system, instead of that a bootstrap tool pkg(7).

This article will cover pkg(8) as of version 1.2.6.

# history

The old package management tools has been developed in 1993, have been maintained since but received very little improvements. While in 1993 it did fits the requirements (few packages, simple semantics involved in packaging) in the Y2K, they are becoming less and less accurate.

In 2010 Julien Laffaye and I decided to have a look at how complicated it would be to create a package management tool that would be able to properly handle binary upgrades, be fast with lots of packages installed, be safe, and allows to have control over the whole processes of package manipulations. We were also willing to provide a high level library so anyone willing to write tools that deals with packages would be able to just plug on the library and do safe operations.

A few years later, pkg(8) has grown, and regular contributors has join the development including (but not only): Bryan Drewery, Matthew Seaman, Vsevolod Stakhov.

# The bootstrap mechanism

## preamble
pkg(8) is not available in base and we made the decision to never push it into base for the following reasons:

1. FreeBSD do maintain multiple releases at the same time, meaning if you introduce a new feature in pkg(8) we have to wait until the End Of Life of the release not supporting that new feature to be able to use it in the ports.
In long term this leads to stagnation (no one really improving the package tools) and introducing hacks instead of real solution into the ports tree.
2. Requirements in packaging evolves quite quickly and the package tools has too follow that, the way we did packaging in 1995 and now is really different, having the ability to improve the package tools very quickly on all supported version is becoming a huge requirement.

## the bootstrap

As pkg(8) is not available directly from the base system, a bootstrap mechanism  (pkg(7)) as been introduced in the base system, it has been designed to be as seamless.

FreeBSD 8.4 is the oldest version including the bootstrap tool, FreeBSD 10 is the only one to have in its final version. This is focuses on what pkg(7) does on FreeBSD 10.

pkg(7) is in fact /usr/sbin/pkg, when run it will look up for the pkg(8) binary (by default in ${LOCALBASE}/sbin), if the binary has been found then it will directly execute it, if not found it will propose to the user to bootstrap it.

Bootstrapping consist in fetching the latest pkg(8) packages from the [official FreeBSD repositories](http://pkg.FreeBSD.org) for a given version on a given architecture.

1. It will also fetch a signature file (pkg.txz.sig), containing a signature and a public key.
2. Verify the public key against a list of fingerprints of known trusted public key
3. Validate that the fetched pkg(8) package matches the signature
4. Extract pkg-static from the package
5. Install the package with the extracted pkg-static
6. Execute pkg(8) with the arguments passed to the bootstrap

To determine where to fetch the package from and how to validate it pkg(7) shares the exact same mechanism of configuration files for repositories which will be describe later.

# pkg(8) configuration option

In pkg(8) almost all options can be overwritten by an environment variable. the command `pkg -vv` will show all the support options and the current setup:

	Version                 : 1.2.6
	PACKAGESITE             : 
	PKG_DBDIR               : /var/db/pkg
	PKG_CACHEDIR            : /var/cache/pkg
	PORTSDIR                : /usr/ports
	PUBKEY                  : 
	HANDLE_RC_SCRIPTS       : no
	ASSUME_ALWAYS_YES       : no
	REPOS_DIR               : [
	  /etc/pkg/,
	  /usr/local/etc/pkg/repos/,
	]
	PLIST_KEYWORDS_DIR      : 
	SYSLOG                  : yes
	AUTODEPS                : yes
	ABI                     : freebsd:11:x86:64
	DEVELOPER_MODE          : no
	PORTAUDIT_SITE          : http://portaudit.FreeBSD.org/auditfile.tbz
	VULNXML_SITE            : http://www.vuxml.org/freebsd/vuln.xml.bz2
	MIRROR_TYPE             : SRV
	FETCH_RETRY             : 3
	PKG_PLUGINS_DIR         : /usr/local/lib/pkg/
	PKG_ENABLE_PLUGINS      : yes
	PLUGINS                 : [
	]
	DEBUG_SCRIPTS           : no
	PLUGINS_CONF_DIR        : /usr/local/etc/pkg/
	PERMISSIVE              : no
	REPO_AUTOUPDATE         : yes
	NAMESERVER              : 
	EVENT_PIPE              : 
	FETCH_TIMEOUT           : 30
	UNSET_TIMESTAMP         : no
	SSH_RESTRICT_DIR        : 
	PKG_SSH_ARGS            : 
	PKG_ENV                 : {
	}
	DISABLE_MTREE           : no
	DEBUG_LEVEL             : 0
	ALIAS                   : {
	}
	
	Repositories:
	  FreeBSD: { 
	    url             : "pkg+http://pkg.FreeBSD.org/freebsd:11:x86:64/latest",
	    enabled         : yes,
	    mirror_type     : "SRV",
	    signature_type  : "FINGERPRINTS",
	    fingerprints    : "/usr/share/keys/pkg"
	  }


- PKG\_DBDIR is the directory where the database about local and remote packages is located
- PKG\CACHEDIR is the directory where the packages will be fetched into before being installed
- HANDLE\_RC\_SCRIPT will tell pkg(8) to automatically restart the rc script after an upgrade
- SYSLOG will make pkg(8) log everything it is doing via the syslog mechanism
- REPOS\_DIR is the directories where pkg can find the binary configurations files (when set from environment it should be defined the same way PATH is defined: REPOS\_DIR="/path1:/path2:/path3"
- PLUGINS\_CONF\_DIR is the directory where the plugins can find there configuration files


# pkg(8) cold start

For the rest of this article we will consider a setup where LOCALBASE is defined as /usr/local.

Important files or directory around pkg(8):

	/etc/pkg/*.conf
	/usr/local/etc/pkg.conf
	/usr/local/etc/pkg/repos/*.conf
	/usr/local/lib/libpkg.so.1
	/usr/local/sbin/pkg
	/usr/local/sbin/pkg-static

pkg-static is a statically linked version of pkg(8) it will allow users to do binary package manipulation even when upgrade from a major version to another major version of FreeBSD.

When the pkg(8) binary is invoked, it will first load /usr/local/etc/pkg.conf except if specified another location via the -C option.
For all configuration entries if there is already an environment variable defining it, it will have the priority over the configuration file.

Now the configuration file has been read, pkg(8) can lookup for for repository definitions in the directory defined via the REPOS\_DIR.
It will read all the .conf files in those directories and will look for entries 1 or N entries like this:

	<NAME> : {
		url : "<url>",
		enabled: true,
		mirror_type: "SRV",
		signature_type: "FINGERPRINTS",
		fingerprints: "/usr/share/keys/pkg",
		pubkey: "/etc/pkg/pub.key"
	}

First time \<NAME\> is found then the "url" element is mandatory.
Each time \<NAME\> the options it defines will overwrite the previously defined one.

- **url**: Represent the url where pkg(8) should fetch the packages from, ssh, http(s), ftp, file are supported protocols, if the MIRROR\_TYPE is set to "SRV" then the url should be prepend by "pkg+",
- **enabled**: always _true_ by default, one can set it to _false_ to disable a given repository
- **mirror\_type**: by default it is _NONE_ possible values are:
  - HTTP: The repository URL should download a text document containing a sequence of lines beginning with `URL:' followed by any amount of while space and one URL for a repository mirror. Any lines not matching this pattern are ignored.  Mirrors are tried in the order listed until a download succeeds.
  - SRV:  For an [SRV](https://en.wikipedia.org/wiki/SRV_record) mirrored repository where the URL is specified as pkg+http://pkgrepo.example.org/ SRV records should provides the list of mirrors where the SRV priority and weight parameters are used to control search order and traffic weighting between sites, and the port number and hostname are used to construct the individual mirror URLs.
- **signature\_type**: by default it is _NONE_, the following entries can be set here:
  - FINGERPRINTS: the public key will be fetched along with the catalog and verified against a list of revoked and trusted fingerprint of public keys
  - PUBKEY: use a public key on the system to validate the remote catalog
- **fingerprints**: path to a directory will should contain 2 subdirectories: _trusted_ and _revoked_" each of which should contain file describing the fingerprint of the concerned public key and the unfction used to create that fingerprint (right now only _sha256_ is supported as a function). This entry is only useful if "signature\_type" is set to "FINGERPRINTS"
- **pubkey**: path to the public key to be use, only useful if "signature\_type" is set to "pubkey"

Once all the above is done pkg(8) is ready to operate.
The bootstrap follow the exact same sequence.

# Different commands

Unlike the old package management tool, pkg is a single binary with lots of small commands

	$ pkg help
	Usage: pkg [-v] [-d] [-l] [-N] [-j <jail name or id>|-c <chroot path>] [-C <configuration file>] [-R <repo config dir>] <command> [<args>]
	
	Global options supported:
	        -d             Increment debug level
	        -j             Execute pkg(8) inside a jail(8)
	        -c             Execute pkg(8) inside a chroot(8)
	        -C             Use the specified configuration file
	        -R             Directory to search for individual repository configurations
	        -l             List available commands and exit
	        -v             Display pkg(8) version
	        -N             Test if pkg(8) is activated and avoid auto-activation
	
	
	Commands supported:
	        add            Registers a package and installs it on the system
	        annotate       Add, modify or delete tag-value style annotations on packages
	        audit          Reports vulnerable packages
	        autoremove     Removes orphan packages
	        backup         Backs-up and restores the local package database
	        check          Checks for missing dependencies and database consistency
	        clean          Cleans old packages from the cache
	        config         Display the value of the configuration options
	        convert        Convert database from/to pkgng
	        create         Creates software package distributions
	        delete         Deletes packages from the database and the system
	        fetch          Fetches packages from a remote repository
	        help           Displays help information
	        info           Displays information about installed packages
	        install        Installs packages from remote package repositories
	        lock           Locks package against modifications or deletion
	        plugins        Manages plugins and displays information about plugins
	        query          Queries information about installed packages
	        register       Registers a package into the local database
	        remove         Deletes packages from the database and the system
	        repo           Creates a package repository catalogue
	        rquery         Queries information in repository catalogues
	        search         Performs a search of package repository catalogues
	        set            Modifies information about packages in the local database
	        ssh            ssh packages to be used via ssh
	        shell          Opens a debug shell
	        shlib          Displays which packages link against a specific shared library
	        stats          Displays package database statistics
	        unlock         Unlocks a package, allowing modification or deletion
	        update         Updates package repository catalogues
	        updating       Displays UPDATING information for a package
	        upgrade        Performs upgrades of packaged software distributions
	        version        Displays the versions of installed packages
	        which          Displays which package installed a specific file

A normal user maintain his system up to date will just have to run a couple of command:

	$ pkg upgrade
	$ pkg autoremove

The first will upgrade all the installed package to their latest version, the second will remove the now orphans (not depend on anymore, and not installed on purpose by the user).

# Creating a package outside of the ports tree

While the ports tree is the natural way of creating packages for FreeBSD, it is rather easy to create your packages outside of the ports tree.

First let's have a look at what pkg create expects:

	Usage: pkg create [-On] [-f format] [-o outdir] [-p plist] [-r rootdir] -m manifestdir
	       pkg create [-Ognx] [-f format] [-o outdir] [-r rootdir] pkg-name ...
	       pkg create [-On] [-f format] [-o outdir] [-r rootdir] -a

The first line it what we are looking for.
We are considering here that the sources has been built and installed in a given _DESTDIR_ and we want to package /usr/local/bin/foo and /usr/local/lib/libbar.so.1

	$ find ${DESTDIR}
	${DESTDIR}/usr/local/bin/foo
	${DESTDIR}/usr/local/lib/libbar.so
	${DESTDIR}/usr/local/lib/libbar.so.1

First we need to create a directory to push the manifest in there
	
	$ mkdir ${MANIFESTDIR}

now create a ${MANIFESTDIR}/+MANIFEST

	name: foo
	version: 1.0
	origin: mycompany/foo
	categories: [ mycompany, devel ]
	comment: foo is a nice tool
	www: http://mycompany/foo
	maintainer: me@mycompany
	prefix: /usr/local

The above is the minimum necesarry it is written in [libucl format](https://github.com/vstakhov/libucl) which is very flexible and allow differents syntaxes
The description for the packages can be added via a file ${MANIFESTDIR}/+DESC or by adding the following lines to the manifest

	desc: <<EOD
	foo is a fantastic tool developed by my company
	on top of libbar.
	EOD

A message can be added to the package by either writing it in a ${MANIFESTDIR}/+DISPLAY or by adding the following line to the manifest
	message: <<EOD
	This is a message for our user
	I hope you do like foo
	EOD

Create a plist (like the ports do):

	bin/foo
	lib/libbar.so.1

The package can now be created:

	$ pkg create -m ${MANIFESTDIR} -r ${DESTDIR} -o ${OUTDIR}

This will create a ${OUTDIR}/foo-1.0.txz ready to be distributed.

# Future challenges of pkg

For next version we aim at bringing lots of new features and improve the user experience. pkg(8) has recently gain a real [sat](http://en.wikipedia.org/wiki/Boolean_satisfiability_problem) which is the basement to improve the flexibility of the package format and package building tools (the ports tree).


Allowing to bring smart dependencies: instead of depending on the specific version of a package we can depend on a valid range of versions


Allowing to bring provides/requires: do not depend on a package but rather on a feature (Requires: perl, http). Provides/Requires can bevery useful in multiple case for example:

- program A requires libMagick.so and ImageMagick and ImageMagick-nox11 both provides libMagick.so, that let the user chose the version he wants instead of having to rebuild the A package and chose which ImageMagick package to use at build time.
- progam B depends on a perl interpreter, it can just requires: perl and not a specific version of perl. right now the version is hardcoded in the dependency.


Work as also begun on having plugable backend for pkg(8) with the current one (the binary repositories) being just one of them, but one could imagine a "ports tree" backend, a CPAN,pear,pypy,gem backend.

For now in the background, pkg(8) uses origin (aka category/portname) to identify a package (determining what is an upgrade of what and so one) this is due to the fact the the same port could have multiple different name in the ports tree depending on options or multiple port could have the same name but different version (and still being different ports, not 2 version of the same port). In the meantime we have been able to fix the ports so that package names are consistent again, meaning we can switch back pkgname as the identifier for packages.

This change while it sounds not very signicant opens a lot of new possibilities for the ports tree:

- sub packages: from one source in one port build multiple packages (like having one single port for php and all its modules)
- flavours (provides): having the same package built with different options