Check-in [469126adaf]

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Merge with trunk at branch for 3.0 Resolved conflicts for: poudriere.8 bulk.sh common.sh jail.sh testport.sh
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | dragonfly
Files: files | file ages | folders
SHA1:469126adaf19e5db7c34c4994784f36a8b61485d
User & Date: marino 2013-05-19 08:16:33
Context
2013-05-19
08:16
Merge with trunk at branch for 3.0 Resolved conflicts for: poudriere.8 bulk.sh common.sh jail.sh testport.sh Leaf check-in: 469126adaf user: marino tags: dragonfly
2013-05-18
19:04
Update to 3.1-pre check-in: cc52157d15 user: bdrewery tags: trunk
2013-04-20
01:15
WIP: Merge DragonFly bits from 2.3 -- untested check-in: ba3ea6d217 user: marino tags: dragonfly
Changes

Changes to conf/poudriere.conf.sample.

67
68
69
70
71
72
73
74









75
76
77
78
79
80
81

# Automatic OPTION change detection
# When bulk building packages, compare the options from kept packages to
# the current options to be built. If they differ, the existing package
# will be deleted and the port will be rebuilt.
# Valid options: yes, no, verbose
# verbose will display the old and new options
#CHECK_CHANGED_OPTIONS=no











# Path to the RSA key to sign the PKGNG repo with. See pkg-repo(8)
#PKG_REPO_SIGNING_KEY=/etc/ssl/keys/repo.key


# ccache support. Supply the path to your ccache cache directory.







|
>
>
>
>
>
>
>
>
>







67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90

# Automatic OPTION change detection
# When bulk building packages, compare the options from kept packages to
# the current options to be built. If they differ, the existing package
# will be deleted and the port will be rebuilt.
# Valid options: yes, no, verbose
# verbose will display the old and new options
#CHECK_CHANGED_OPTIONS=verbose

# Automatic Dependency change detection
# When bulk building packages, compare the dependencies from kept packages to
# the current dependencies for every port. If they differ, the existing package
# will be deleted and the port will be rebuilt. This helps catch changes such
# as DEFAULT_RUBY_VERSION, PERL_VERSION, WITHOUT_X11 that change dependencies
# for many ports.
# Valid options: yes, no
#CHECK_CHANGED_DEPS=yes


# Path to the RSA key to sign the PKGNG repo with. See pkg-repo(8)
#PKG_REPO_SIGNING_KEY=/etc/ssl/keys/repo.key


# ccache support. Supply the path to your ccache cache directory.

Changes to doc/index.wiki.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
..
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
<h1>What is poudriere?</h1>

poudriere is a BSD-2 licensed tool primarily designed to test package production on
FreeBSD. However, most people will find it useful to bulk build ports
for FreeBSD.

Its goals are to use modern facilities present in FreeBSD (as ZFS,
jails), to be easy to use, to depend only on base, and to be parallel.

It is able to build the whole portstree like tinderbox does.

poudriere is able to build packages for versions of FreeBSD that are
different from the box on which it is installed, and also to build
packages for i386 if the host is an amd64 box.

................................................................................
Once the package building is done, the packages will be in a layout
identical to the official mirrors. Therefore they will be useable by any
package management tool (pkg_add, portmaster, and so on).

<h1>What does poudriere require?</h1>

  *  a recent FreeBSD (>= 8.3)
  *  a ZFS pool with at least 7GB of free space
  *  root access

That is all. For more comfort, we recommend to have quite a lot of RAM
in order to make the builds in a VM resident filesystem.

<h1>How to install it?</h1>







|
|







 







<







1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
..
16
17
18
19
20
21
22

23
24
25
26
27
28
29
<h1>What is poudriere?</h1>

poudriere is a BSD-2 licensed tool primarily designed to test package production on
FreeBSD. However, most people will find it useful to bulk build ports
for FreeBSD.

Its goals are to use modern facilities present in FreeBSD, to be easy to use,
to depend only on base, and to be parallel.

It is able to build the whole portstree like tinderbox does.

poudriere is able to build packages for versions of FreeBSD that are
different from the box on which it is installed, and also to build
packages for i386 if the host is an amd64 box.

................................................................................
Once the package building is done, the packages will be in a layout
identical to the official mirrors. Therefore they will be useable by any
package management tool (pkg_add, portmaster, and so on).

<h1>What does poudriere require?</h1>

  *  a recent FreeBSD (>= 8.3)

  *  root access

That is all. For more comfort, we recommend to have quite a lot of RAM
in order to make the builds in a VM resident filesystem.

<h1>How to install it?</h1>

Changes to poudriere.8.

23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
..
77
78
79
80
81
82
83


84
85
86
87
88
89
90
..
96
97
98
99
100
101
102



103
104
105
106
107
108
109
...
415
416
417
418
419
420
421

















422
423
424
425
426
427
428
.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
.\" SUCH DAMAGE.
.\"
.\" $FreeBSD$
.\"
.\" Note: The date here should be updated whenever a non-trivial
.\" change is made to the manual page.
.Dd April 17, 2013
.Dt POUDRIERE 8
.Os FreeBSD
.Sh NAME
.Nm poudriere
.Nd bulk package builder and port tester
.Sh SYNOPSIS
.Nm
................................................................................
This command allows to configure the options for a given port
.It distclean
This command will cleanup old distfiles
.It queue
This command allows a non-root user to queue
.Nm
commands.


.It version
Show version of poudriere.
.Nm.
.El
.Sh SUBCOMMANDS
Here are the list of subcommands and associated options supported by
.Nm ,
................................................................................
These subcommands are mutually exclusive.
.Bl -tag -width "-f conffile"
.It Fl f Ar file
Absolute path to a file which contains the list of ports to build.
Ports must be specified in the form
.Ar category/port
and shell-style comments are allowed.



.It cat/port cat/port2 ...
A list of ports can be specified directly.
.El
.Pp
See
.Sx CUSTOMISATION
to know how to build binary packages with options that differs from
................................................................................
This command takes a
.Nm
command in argument.
.Pp
There are no options associated with the
.Cm queue
command.

















.Sh ENVIRONMENT
The
.Nm
command does not use any
environment variable.
.Sh FILES
.Bl -tag -width ".Pa /usr/local/etc/poudriere.conf" -compact







|







 







>
>







 







>
>
>







 







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
..
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
..
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
...
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
.\" SUCH DAMAGE.
.\"
.\" $FreeBSD$
.\"
.\" Note: The date here should be updated whenever a non-trivial
.\" change is made to the manual page.
.Dd May 16, 2013
.Dt POUDRIERE 8
.Os FreeBSD
.Sh NAME
.Nm poudriere
.Nd bulk package builder and port tester
.Sh SYNOPSIS
.Nm
................................................................................
This command allows to configure the options for a given port
.It distclean
This command will cleanup old distfiles
.It queue
This command allows a non-root user to queue
.Nm
commands.
.It status
This command shows status of current and previous builds
.It version
Show version of poudriere.
.Nm.
.El
.Sh SUBCOMMANDS
Here are the list of subcommands and associated options supported by
.Nm ,
................................................................................
These subcommands are mutually exclusive.
.Bl -tag -width "-f conffile"
.It Fl f Ar file
Absolute path to a file which contains the list of ports to build.
Ports must be specified in the form
.Ar category/port
and shell-style comments are allowed.
Multiple
.Fl f Ar file
arguments may be specified at once.
.It cat/port cat/port2 ...
A list of ports can be specified directly.
.El
.Pp
See
.Sx CUSTOMISATION
to know how to build binary packages with options that differs from
................................................................................
This command takes a
.Nm
command in argument.
.Pp
There are no options associated with the
.Cm queue
command.
.Ss status
This command accepts the following options:
.Bl -tag -width "-f conffile"
.It Fl j Ar name
Specifies the
.Ar name
of the jail to view the status of.
.It Fl p Ar tree
This flag specifies which ports
.Ar tree
the build is running on when using
.Fl j
.It Fl z Ar set
This specifies which SET the build is running on when using
.Fl j
.El
.Pp
.Sh ENVIRONMENT
The
.Nm
command does not use any
environment variable.
.Sh FILES
.Bl -tag -width ".Pa /usr/local/etc/poudriere.conf" -compact

Changes to src/bin/poudriere.sh.

25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

46
47
48
49
50
51
52
..
89
90
91
92
93
94
95



96
97
98
99
100
101
102
# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.

LC_ALL=C
unset SHELL
SAVED_TERM=$TERM
unset TERM
POUDRIERE_VERSION="3.0-pre"

usage() {
	echo "Usage: poudriere command [options]

Commands:
    bulk        -- generate packages for given ports
    distclean   -- clean old distfiles
    daemon      -- launch the poudriere daemon
    help        -- show usage
    jail        -- manage the jails used by poudriere
    ports       -- create, update or delete the portstrees used by poudriere
    options     -- Configure ports options
    queue       -- queue a build request

    testport    -- launch a test on a given port
    version     -- show poudriere version"
	exit 1
}

SETX=""
while getopts "x" FLAG; do
................................................................................
		;;
	queue)
		exec env -i PATH=${PATH} POUDRIERE_VERSION="${POUDRIERE_VERSION}" /bin/sh ${SETX} ${POUDRIEREPREFIX}/queue.sh $@
		;;
	options)
		exec env -i TERM=${SAVED_TERM} PATH=${PATH} POUDRIERE_VERSION="${POUDRIERE_VERSION}" /bin/sh ${SETX} ${POUDRIEREPREFIX}/options.sh $@
		;;



	help)
		usage
		;;
	version)
		echo "${POUDRIERE_VERSION}"
		;;
	daemon)







|











|

>







 







>
>
>







25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
..
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.

LC_ALL=C
unset SHELL
SAVED_TERM=$TERM
unset TERM
POUDRIERE_VERSION="3.1-pre"

usage() {
	echo "Usage: poudriere command [options]

Commands:
    bulk        -- generate packages for given ports
    distclean   -- clean old distfiles
    daemon      -- launch the poudriere daemon
    help        -- show usage
    jail        -- manage the jails used by poudriere
    ports       -- create, update or delete the portstrees used by poudriere
    options     -- configure ports options
    queue       -- queue a build request
    status      -- get status of builds
    testport    -- launch a test on a given port
    version     -- show poudriere version"
	exit 1
}

SETX=""
while getopts "x" FLAG; do
................................................................................
		;;
	queue)
		exec env -i PATH=${PATH} POUDRIERE_VERSION="${POUDRIERE_VERSION}" /bin/sh ${SETX} ${POUDRIEREPREFIX}/queue.sh $@
		;;
	options)
		exec env -i TERM=${SAVED_TERM} PATH=${PATH} POUDRIERE_VERSION="${POUDRIERE_VERSION}" /bin/sh ${SETX} ${POUDRIEREPREFIX}/options.sh $@
		;;
	status)
		exec env -i PATH=${PATH} POUDRIERE_VERSION="${POUDRIERE_VERSION}" /bin/sh ${SETX} ${POUDRIEREPREFIX}/status.sh $@
		;;
	help)
		usage
		;;
	version)
		echo "${POUDRIERE_VERSION}"
		;;
	daemon)

Changes to src/share/poudriere/bulk.sh.

138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
...
203
204
205
206
207
208
209


210


211
212
213
214
215
216
217
...
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
...
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
...
286
287
288
289
290
291
292

293
294
295
296
297
298
		c)
			CLEAN=1
			;;
		C)
			CLEAN_LISTED=1
			;;
		f)
			LISTPKGS=${OPTARG}
			;;
		F)
			export MASTER_SITE_BACKUP=''
			;;
		j)
			jail_exists ${OPTARG} || err 1 "No such jail: ${OPTARG}"
			JAILNAME=${OPTARG}
................................................................................
	rm -rf ${POUDRIERE_DATA}/packages/${MASTERNAME}/*
	rm -rf ${POUDRIERE_DATA}/cache/${JAILNAME}
	echo " done"
fi

if [ $# -eq 0 ]; then
	[ -n "${LISTPKGS}" -o ${ALL} -eq 1 ] || err 1 "No packages specified"


	[ ${ALL} -eq 1 -o -f "${LISTPKGS}" ] || err 1 "No such list of packages: ${LISTPKGS}"


else
	[ ${ALL} -eq 0 ] || err 1 "command line arguments and -a cannot be used at the same time"
	[ -z "${LISTPKGS}" ] || err 1 "command line arguments and list of ports cannot be used at the same time"
	LISTPORTS="$@"
fi

check_jobs
................................................................................

prepare_ports

run_hook bulk_build_start "${JAILNAME}" "${PTNAME}" `bget stats_queued`

bset status "building:"

[ -z "${PORTTESTING}" -a -z "${ALLOW_MAKE_JOBS}" ] &&
	echo "DISABLE_MAKE_JOBS=yes" >> ${MASTERMNT}/etc/make.conf

[ -n "${JOBS_LIMIT}" ] &&
	echo "MAKE_JOBS_NUMBER=${JOBS_LIMIT}" >> ${MASTERMNT}/etc/make.conf

markfs prepkg ${MASTERMNT}

parallel_build ${JAILNAME} ${PTNAME} ${SETNAME} || : # Ignore errors as they are handled below

bset status "done:"

failed=$(bget ports.failed | awk '{print $1 ":" $3 }' | xargs echo)
built=$(bget ports.built | awk '{print $1}' | xargs echo)
ignored=$(bget ports.ignored | awk '{print $1}' | xargs echo)
................................................................................
if [ $nbbuilt -eq 0 ]; then
	if [ $PKGNG -eq 1 ]; then
		msg "No package built, no need to update the repository"
	else
		msg "No package built, no need to update INDEX"
	fi
else
	[ -n "${NO_RESTRICTED}" ] && clean_restricted
	[ ${BUILD_REPO} -eq 1 ] && build_repo
fi

cleanup
if [ $nbbuilt -gt 0 ]; then
	msg_n "Built ports: "
	echo ${built}
................................................................................
fi
if [ $nbskipped -gt 0 ]; then
	msg_n "Skipped ports: "
	echo ${skipped}
	echo ""
fi
msg "[${MASTERNAME}] $nbbuilt packages built, $nbfailed failures, $nbignored ignored, $nbskipped skipped"


set +e
run_hook bulk_build_ended "${JAILNAME}" "${PTNAME}" "${nbbuilt}" \
	"${nbfailed}" "${nbignored}" "${nbskipped}"

exit $((nbfailed + nbskipped))







|







 







>
>
|
>
>







 







<
<
<
<
<
<
<
<







 







|







 







>






138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
...
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
...
232
233
234
235
236
237
238








239
240
241
242
243
244
245
...
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
...
282
283
284
285
286
287
288
289
290
291
292
293
294
295
		c)
			CLEAN=1
			;;
		C)
			CLEAN_LISTED=1
			;;
		f)
			LISTPKGS="${LISTPKGS} ${OPTARG}"
			;;
		F)
			export MASTER_SITE_BACKUP=''
			;;
		j)
			jail_exists ${OPTARG} || err 1 "No such jail: ${OPTARG}"
			JAILNAME=${OPTARG}
................................................................................
	rm -rf ${POUDRIERE_DATA}/packages/${MASTERNAME}/*
	rm -rf ${POUDRIERE_DATA}/cache/${JAILNAME}
	echo " done"
fi

if [ $# -eq 0 ]; then
	[ -n "${LISTPKGS}" -o ${ALL} -eq 1 ] || err 1 "No packages specified"
	if [ ${ALL} -eq 0 ]; then
		for listpkg_name in ${LISTPKGS}; do
			[ -f "${listpkg_name}" ] || err 1 "No such list of packages: ${listpkg_name}"
		done
	fi
else
	[ ${ALL} -eq 0 ] || err 1 "command line arguments and -a cannot be used at the same time"
	[ -z "${LISTPKGS}" ] || err 1 "command line arguments and list of ports cannot be used at the same time"
	LISTPORTS="$@"
fi

check_jobs
................................................................................

prepare_ports

run_hook bulk_build_start "${JAILNAME}" "${PTNAME}" `bget stats_queued`

bset status "building:"









parallel_build ${JAILNAME} ${PTNAME} ${SETNAME} || : # Ignore errors as they are handled below

bset status "done:"

failed=$(bget ports.failed | awk '{print $1 ":" $3 }' | xargs echo)
built=$(bget ports.built | awk '{print $1}' | xargs echo)
ignored=$(bget ports.ignored | awk '{print $1}' | xargs echo)
................................................................................
if [ $nbbuilt -eq 0 ]; then
	if [ $PKGNG -eq 1 ]; then
		msg "No package built, no need to update the repository"
	else
		msg "No package built, no need to update INDEX"
	fi
else
	[ "${NO_RESTRICTED:-no}" != "no" ] && clean_restricted
	[ ${BUILD_REPO} -eq 1 ] && build_repo
fi

cleanup
if [ $nbbuilt -gt 0 ]; then
	msg_n "Built ports: "
	echo ${built}
................................................................................
fi
if [ $nbskipped -gt 0 ]; then
	msg_n "Skipped ports: "
	echo ${skipped}
	echo ""
fi
msg "[${MASTERNAME}] $nbbuilt packages built, $nbfailed failures, $nbignored ignored, $nbskipped skipped"
show_log_info

set +e
run_hook bulk_build_ended "${JAILNAME}" "${PTNAME}" "${nbbuilt}" \
	"${nbfailed}" "${nbignored}" "${nbskipped}"

exit $((nbfailed + nbskipped))

Changes to src/share/poudriere/clean.sh.

53
54
55
56
57
58
59
60
61
62
63
64


65
66
67
68
69
70
71
		# Determine which packages are ready-to-build and
		# handle "impact"/skipping support
		if [ -z "$(find "${JAILMNT}/poudriere/rdeps/${pkgname}" -type d -maxdepth 0 -empty)" ]; then

			# Remove this package from every package depending on this
			# This follows the symlink in rdeps which references
			# deps/<pkgname>/<this pkg>
			find ${JAILMNT}/poudriere/rdeps/${pkgname} -type l | \
				xargs realpath -q | \
				xargs rm -f || :

			for dep_dir in ${JAILMNT}/poudriere/rdeps/${pkgname}/*; do


				dep_pkgname=${dep_dir##*/}

				# Determine everything that depends on the given package
				# Recursively cleanup anything that depends on this port.
				if [ ${clean_rdepends} -eq 1 ]; then
					# clean_pool() in common.sh will pick this up and add to SKIPPED
					echo "${dep_pkgname}"







|




>
>







53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
		# Determine which packages are ready-to-build and
		# handle "impact"/skipping support
		if [ -z "$(find "${JAILMNT}/poudriere/rdeps/${pkgname}" -type d -maxdepth 0 -empty)" ]; then

			# Remove this package from every package depending on this
			# This follows the symlink in rdeps which references
			# deps/<pkgname>/<this pkg>
			find ${JAILMNT}/poudriere/rdeps/${pkgname} -type l 2>/dev/null | \
				xargs realpath -q | \
				xargs rm -f || :

			for dep_dir in ${JAILMNT}/poudriere/rdeps/${pkgname}/*; do
				[ "${dep_dir}" = "${JAILMNT}/poudriere/rdeps/${pkgname}/*" ] &&
					break
				dep_pkgname=${dep_dir##*/}

				# Determine everything that depends on the given package
				# Recursively cleanup anything that depends on this port.
				if [ ${clean_rdepends} -eq 1 ]; then
					# clean_pool() in common.sh will pick this up and add to SKIPPED
					echo "${dep_pkgname}"

Changes to src/share/poudriere/common.sh.

100
101
102
103
104
105
106






107
108
109
110
111
112

113



114
115
116









117
118
119
120
121
122
123
...
163
164
165
166
167
168
169


170






171

172
173
174
175
176
177
178
...
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
...
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
...
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
...
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
...
433
434
435
436
437
438
439
440






441

442
443
444
445
446
447
448
...
451
452
453
454
455
456
457
















458
459
460
461
462

463

464
465
466
467
468

469


470
471
472
473
474
475
476
477
478
479



480
481

482
483
484
485
486

487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
...
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
...
674
675
676
677
678
679
680







681
682
683
684
685
686
687
688
689

















690
691


692
693
694

695
696
697

698
699
700
701
702
703
704
...
707
708
709
710
711
712
713
714
715
716








717
718
719
720
721
722
723
724
725
726
727
...
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
...
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
...
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
...
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
...
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
....
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
....
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
....
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
....
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
....
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
....
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350
....
1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
....
1472
1473
1474
1475
1476
1477
1478



1479

1480
1481
1482
1483
1484
1485

1486
1487
1488
1489
1490

1491
1492
1493
1494
1495
1496
1497
1498
1499
1500
....
1593
1594
1595
1596
1597
1598
1599
1600
1601
1602
1603
1604
1605
1606
1607
1608
1609
....
1611
1612
1613
1614
1615
1616
1617
1618
1619
1620
1621
1622
1623
1624
1625
1626
1627
1628
1629
1630
1631
....
1632
1633
1634
1635
1636
1637
1638
1639
1640
1641
1642
1643
1644
1645
1646

1647
1648
1649

1650
1651
1652
1653
1654
1655
1656
....
1735
1736
1737
1738
1739
1740
1741
1742
1743
1744
1745
1746
1747
1748
1749
....
1758
1759
1760
1761
1762
1763
1764

1765
1766
1767
1768
1769
1770
1771
....
1851
1852
1853
1854
1855
1856
1857








1858
1859
1860
1861
1862
1863
1864
....
1919
1920
1921
1922
1923
1924
1925
1926
1927
1928
1929
1930
1931
1932
1933
}

. $(dirname ${0})/common.sh.${BSDPLATFORM}

run_hook() {
	local hookfile=${HOOKDIR}/${1}.sh
	shift






	[ -f ${hookfile} ] && /bin/sh ${hookfile} $@
	return 0
}

log_start() {
	local logfile=$1





	# Make sure directory exists
	mkdir -p ${logfile%/*}/success










	exec 3>&1 4>&2
	[ ! -e ${logfile}.pipe ] && mkfifo ${logfile}.pipe
	tee ${logfile} < ${logfile}.pipe >&3 &
	export tpid=$!
	exec > ${logfile}.pipe 2>&1

	# Remove fifo pipe file right away to avoid orphaning it.
................................................................................
	echo "---Begin make.conf---"
	cat ${mnt}/etc/make.conf
	echo "---End make.conf---"
}

buildlog_stop() {
	local portdir=$1









	echo "build of ${portdir} ended at $(date)"

}

log_stop() {
	if [ -n "${tpid}" ]; then
		exec 1>&3 3>&- 2>&4 4>&-
		kill $tpid
		wait $tpid 2>/dev/null || :
................................................................................

exit_handler() {
	# Avoid recursively cleaning up here
	trap - EXIT SIGTERM SIGKILL
	# Ignore SIGINT while cleaning up
	trap '' SIGINT

	if [ "${POUDRIERE_BUILD_TYPE}" = "bulk" \
		-o "${POUDRIERE_BUILD_TYPE}" = "testport" ]; then
		log_stop
		stop_html_json
	fi

	# Kill all children - this does NOT recurse, so orphans can still
	# occur. This is just to avoid requiring pid files for parallel_run
................................................................................
		msg "WWW: ${URL_BASE}/${POUDRIERE_BUILD_TYPE}/${MASTERNAME}/${BUILDNAME}"
}

siginfo_handler() {
	[ "${POUDRIERE_BUILD_TYPE}" != "bulk" ] && return 0

	trappedinfo=1
	local status=$(bget status)
	local nbb=$(bget stats_built 2>/dev/null)
	local nbf=$(bget stats_failed 2>/dev/null)
	local nbi=$(bget stats_ignored 2>/dev/null)
	local nbs=$(bget stats_skipped 2>/dev/null)
	local nbq=$(bget stats_queued 2>/dev/null)
	local ndone=$((nbb + nbf + nbi + nbs))
	local queue_width=2
	local now
	local j
	local pkgname origin phase buildtime
	local format_origin_phase format_phase

................................................................................
	elif [ ${nbq} -gt 99 ]; then
		queue_width=3
	fi

	# Skip if stopping or starting jobs
	if [ -n "${JOBS}" -a "${status#starting_jobs:}" = "${status}" -a "${status}" != "stopping_jobs:" ]; then
		now=$(date +%s)
		format_origin_phase="\t[%s]: %-30s %-13s (%s)\n"
		format_phase="\t[%s]: %13s\n"

		# Collect build stats into a string with minimal execs
		pkgname_buildtimes=$(find ${MASTERMNT}/poudriere/building -depth 1 \
			-exec stat -f "%N %m" {} + | \
			awk -v now=${now} -f ${AWKPREFIX}/siginfo_buildtime.awk)

		for j in ${JOBS}; do
................................................................................
}

fetch_file() {
	[ $# -ne 2 ] && eargs destination source
	fetch -p -o $1 $2 || fetch -p -o $1 $2 || err 1 "Failed to fetch from $2"
}



unmarkfs() {
	[ $# -ne 2 ] && eargs name mnt
	local name=$1
	local mnt=$(realpath $2)

	if [ -n "$(zfs_getfs ${mnt})" ]; then
		zfs destroy -f ${fs}@${name} 2>/dev/null || :
................................................................................
	local dozfs=0
	local domtree=0

	case "${name}" in
	clean) [ -n "${fs}" ] && dozfs=1 ;;
	prepkg)
		[ -n "${fs}" ] && dozfs=1
		[ "${dozfs}" -eq 0 -a  "${mnt##*/}" != "ref" ] && domtree=1






		;;

	preinst) domtree=1 ;;
	esac

	if [ $dozfs -eq 1 ]; then
		# remove old snapshot if exists
		zfs destroy -r ${fs}@${name} 2>/dev/null || :
		#create new snapshot
................................................................................

	[ $domtree -eq 0 ] && return 0
	mkdir -p ${mnt}/poudriere/
	if [ "${name}" = "prepkg" ]; then
		cat > ${mnt}/poudriere/mtree.${name}exclude << EOF
./ccache/*
./compat/linux/proc
















./distfiles/*
./new_packages/*
./packages/*
./poudriere/*
./proc/*

.${PORTSRC}/*

./var/db/ports/*
./wrkdirs/*
EOF
	elif [ "${name}" = "preinst" ]; then
		cat >  ${mnt}/poudriere/mtree.${name}exclude << EOF

./compat/linux/proc/*


./etc/group
./etc/make.conf
./etc/make.conf.bak
./etc/master.passwd
./etc/passwd
./etc/pwd.db
./etc/shells
./etc/spwd.db
./poudriere/*
./new_packages/*



.${HOME}/*
./tmp/*

.${LOCALBASE:-/usr/local}/etc/gconf/gconf.xml.defaults
.${LOCALBASE:-/usr/local}/lib/charset.alias
.${LOCALBASE:-/usr/local}/share/applications/mimeinfo.cache
.${LOCALBASE:-/usr/local}/share/nls/POSIX
.${LOCALBASE:-/usr/local}/share/nls/en_US.US-ASCII

./var/db/*
./var/log/*
./var/mail/*
./var/run/*
./wrkdirs/*
EOF
	fi
	mtree -X ${mnt}/poudriere/mtree.${name}exclude \
		-xcn -k uid,gid,mode,size \
		-p ${mnt} > ${mnt}/poudriere/mtree.${name}
}


use_options() {
	[ $# -ne 2 ] && eargs mnt optionsdir
	local mnt=$1
	local optionsdir=$2

	if [ "${optionsdir}" = "-" ]; then
................................................................................
		# Check for non-empty directory with no packages in it
		[ "${pkg}" = "${POUDRIERE_DATA}/packages/${MASTERNAME}/All/*.${PKG_EXT}" ] && break
		depfile=$(deps_file ${pkg})
		while read dep; do
			if [ ! -e "${POUDRIERE_DATA}/packages/${MASTERNAME}/All/${dep}.${PKG_EXT}" ]; then
				ret=1
				msg_debug "${pkg} needs missing ${POUDRIERE_DATA}/packages/${MASTERNAME}/All/${dep}.${PKG_EXT}"
				msg "Deleting ${pkg##*/}: missing dependencies"
				delete_pkg ${pkg}
				break
			fi
		done < "${depfile}"
	done

	return $ret
}

check_leftovers() {
	local mnt=$1
	mtree -X ${mnt}/poudriere/mtree.preinstexclude -x \
		-f ${mnt}/poudriere/mtree.preinst \
		-p ${mnt} | while read l ; do
		case ${l} in
		*extra)
			if [ -d ${mnt}/${l% *} ]; then
				find ${mnt}/${l% *} -exec echo "+ {}" \;
			else
................................................................................
				   extract-depends extract patch-depends patch build-depends \
				   lib-depends configure build install-mtree run-depends \
				   install package ${PORTTESTING:+deinstall}"
	local mnt=$(my_path)
	local log=$(log_path)
	local listfilecmd network
	local hangstatus








	for phase in ${targets}; do
		bset ${MY_JOBID} status "${phase}:${port}"
		job_msg_verbose "Status   ${port}: ${phase}"
		if [ "${phase}" = "fetch" ]; then
			jstop
			jstart 1
		fi
		case ${phase} in

















		install) [ -n "${PORTTESTING}" ] && markfs preinst ${mnt} ;;
		deinstall)


			msg "Checking shared library dependencies"
			listfilecmd="grep -v '^@' /var/db/pkg/${PKGNAME}/+CONTENTS"
			[ ${PKGNG} -eq 1 ] && listfilecmd="pkg query '%Fp' ${PKGNAME}"

			echo "${listfilecmd} | xargs ldd 2>&1 | awk '/=>/ { print $3 }' | sort -u" > ${mnt}/shared.sh
			injail sh /shared.sh
			rm -f ${mnt}/shared.sh

			;;
		esac

		print_phase_header ${phase}

		if [ "${phase}" = "package" ]; then
			echo "PACKAGES=/new_packages" >> ${mnt}/etc/make.conf
................................................................................
			mkdir -p "${POUDRIERE_DATA}/packages/${MASTERNAME}/.new_packages/${PKGNAME}"
			${NULLMOUNT} \
				"${POUDRIERE_DATA}/packages/${MASTERNAME}/.new_packages/${PKGNAME}" \
				${mnt}/new_packages
		fi

		if [ "${phase#*-}" = "depends" ]; then
			# No need for nohang or PKGENV/PORT_FLAGS for *-depends
			injail make -C ${portdir} ${phase}
		else








			# 24 hours for 1 command, or 20 minutes with no log update
			nohang ${MAX_EXECUTION_TIME:-86400} ${NOHANG_TIME:-7200} \
				${log}/logs/${PKGNAME}.log \
				injail env ${PKGENV} ${PORT_FLAGS} \
				make -C ${portdir} ${phase}
			hangstatus=$? # This is done as it may return 1 or 2 or 3
			if [ $hangstatus -ne 0 ]; then
				# 1 = cmd failed, not a timeout
				# 2 = log timed out
				# 3 = cmd timeout
				if [ $hangstatus -eq 2 ]; then
................................................................................
	local port="$2"
	local portdir="$(echo "$3" | sed -e "s|${PORTSRC}/||g")"
	local phase="$4"
	local tardir=${POUDRIERE_DATA}/wrkdirs/${MASTERNAME}/${PTNAME}
	local tarname=${tardir}/${PKGNAME}.${WRKDIR_ARCHIVE_FORMAT}
	local mnted_portdir=${mnt}/wrkdirs/${portdir}

	[ -n "${SAVE_WRKDIR}" ] || return 0
	# Only save if not in fetch/checksum phase
	[ "${failed_phase}" != "fetch" -a "${failed_phase}" != "checksum" -a \
		"${failed_phase}" != "extract" ] || return 0

	mkdir -p ${tardir}

	# Tar up the WRKDIR, and ignore errors
................................................................................

	job_msg "Saved ${port} wrkdir to: ${tarname}"
}


deadlock_detected() {
	local always_fail=${1:-1}
	local mnt=$(my_path)
	local crashed_packages dependency_cycles

	# If there are still packages marked as "building" they have crashed
	# and it's likely some poudriere or system bug
	crashed_packages=$( \
		find ${mnt}/poudriere/building -type d -mindepth 1 -maxdepth 1 | \
		sed -e "s:${mnt}/poudriere/building/::" | tr '\n' ' ' \
	)
	[ -z "${crashed_packages}" ] ||	\
		err 1 "Crashed package builds detected: ${crashed_packages}"

	# Check if there's a cycle in the need-to-build queue
	dependency_cycles=$(\
		find ${mnt}/poudriere/deps -mindepth 2 | \
		sed -e "s:${mnt}/poudriere/deps/::" -e 's:/: :' | \
		# Only cycle errors are wanted
		tsort 2>&1 >/dev/null | \
		sed -e 's/tsort: //' | \
		awk -f ${AWKPREFIX}/dependency_loop.awk \
	)

	if [ -n "${dependency_cycles}" ]; then
................................................................................
		err 1 "Dependency loop detected:
${dependency_cycles}"
	fi

	[ ${always_fail} -eq 1 ] || return 0

	# No cycle, there's some unknown poudriere bug
	err 1 "Unknown stuck queue bug detected. Give this information to poudriere developers:
$(find ${mnt}/poudriere/building ${mnt}/poudriere/pool ${mnt}/poudriere/deps)"
}

queue_empty() {
	local pool_dir
	local mnt=$(my_path)
	dirempty ${mnt}/poudriere/deps || return 1

	for pool_dir in ${POOL_BUCKET_DIRS}; do
		dirempty ${pool_dir} || return 1
	done

	return 0
}
................................................................................

mark_done() {
	[ $# -eq 1 ] || eargs pkgname
	local pkgname="$1"
	local origin=$(cache_get_origin "${pkgname}")
	local cache_dir=$(cache_dir)

	echo -n "${origin} $(date +%s)" >> ${cache_dir}/buildtimes
	stat -f "%m" ${MASTERMNT}/poudriere/building/${pkgname} >> \
		${cache_dir}/buildtimes
	rmdir ${MASTERMNT}/poudriere/building/${pkgname}
}


build_queue() {
	local j name pkgname builders_active queue_empty
	local mnt=$(my_path)

	mkfifo ${MASTERMNT}/poudriere/builders.pipe
	exec 6<> ${MASTERMNT}/poudriere/builders.pipe
	rm -f ${MASTERMNT}/poudriere/builders.pipe
	queue_empty=0

	msg "Hit CTRL+t at any time to see build progress and stats"

	while :; do
		builders_active=0
		for j in ${JOBS}; do
			name="${MASTERNAME}-job-${j}"
			if [ -f  "${mnt}/poudriere/var/run/${j}.pid" ]; then
				if pgrep -qF "${mnt}/poudriere/var/run/${j}.pid" 2>/dev/null; then
					builders_active=1
					continue
				fi
				read pkgname < ${mnt}/poudriere/var/run/${j}.pkgname
				rm -f ${mnt}/poudriere/var/run/${j}.pid \
					${mnt}/poudriere/var/run/${j}.pkgname
				bset ${j} status "idle:"
				mark_done ${pkgname}
			fi

			[ ${queue_empty} -eq 0 ] || continue

			pkgname=$(next_in_queue)
................................................................................
				# Check if the ready-to-build pool and need-to-build pools
				# are empty
				queue_empty && queue_empty=1

				# Pool is waiting on dep, wait until a build
				# is done before checking the queue again
			else
				MY_JOBID="${j}" build_pkg "${pkgname}" > /dev/null 2>&1 &
				echo "$!" > ${mnt}/poudriere/var/run/${j}.pid
				echo "${pkgname}" > ${mnt}/poudriere/var/run/${j}.pkgname

				# A new job is spawned, try to read the queue
				# just to keep things moving
				builders_active=1
			fi
		done

................................................................................
		run_hook pkgbuild skipped "${skipped_origin}" "${skipped_pkgname}" "${port}"
	done

	balance_pool
}

print_phase_header() {
	printf "=======================<phase: %-13s>==========================\n" "$1"
}

print_phase_footer() {
	echo "======================================================================="
}

build_pkg() {
	# If this first check fails, the pool will not be cleaned up,
	# since PKGNAME is not yet set.
	[ $# -ne 1 ] && eargs pkgname
	local pkgname="$1"
................................................................................
	port=$(cache_get_origin ${pkgname})
	portdir="${PORTSRC}/${port}"

	job_msg "Starting build of ${port}"
	bset ${MY_JOBID} status "starting:${port}"
	create_slave ${MASTERMNT} ${MY_JOBID}

	if [ ${TMPFS_LOCALBASE} -eq 1 ]; then
		umount -f ${mnt}/${LOCALBASE:-/usr/local} 2>/dev/null || :
		mount -t tmpfs tmpfs ${mnt}/${LOCALBASE:-/usr/local}
	fi

	# If this port is IGNORED, skip it
	# This is checked here instead of when building the queue
	# as the list may start big but become very small, so here
	# is a less-common check
	ignore="$(injail make -C ${portdir} -VIGNORE)"

	log_start ${log}/logs/${PKGNAME}.log
	msg "Building ${port}"
	buildlog_start ${portdir}

	if [ -n "${ignore}" ]; then
		msg "Ignoring ${port}: ${ignore}"
		badd ports.ignored "${port} ${PKGNAME} ${ignore}"
		job_msg "Finished build of ${port}: Ignored: ${ignore}"
................................................................................
		fi
	fi

	clean_pool ${PKGNAME} ${clean_rdepends}

	bset ${MY_JOBID} status "done:${port}"

	stop_build ${portdir} ${log}/logs/${PKGNAME}.log

	destroy_slave ${MASTERMNT} ${MY_JOBID}
	if [ ${zip_log} -eq 1 ]; then
		bzip2 ${log}/logs/${PKGNAME}.log
		mv ${log}/logs/${PKGNAME}.log.bz2 $(log_path)/success/
	fi

	echo ${MY_JOBID} >&6
}

stop_build() {
	[ $# -eq 2 ] || eargs portdir logfile
	local portdir="$1"
	local mnt=$(my_path)
	local logfile="$2"

	umount -f ${mnt}/new_packages 2>/dev/null || :
	rm -rf "${POUDRIERE_DATA}/packages/${MASTERNAME}/.new_packages/${PKGNAME}"

	# 2 = HEADER+ps itself
	if [ $(injail ps aux | wc -l) -ne 2 ]; then
		msg "Leftover processes:"
		injail ps auxwwd | grep -v 'ps auxwwd'
	fi
	# Always kill to avoid missing anything
	injail kill -9 -1 2>/dev/null || :

	buildlog_stop ${portdir}
	log_stop ${logfile}
}

# Crazy redirection is to add the portname into stderr.
# Idea from http://superuser.com/a/453609/34747
mangle_stderr() {
	local msg_type="$1"
	local extra="$2"
................................................................................
deps_file() {
	[ $# -ne 1 ] && eargs pkg
	local pkg=$1
	local depfile=$(pkg_cache_dir ${pkg})/deps

	if [ ! -f "${depfile}" ]; then
		if [ "${PKG_EXT}" = "tbz" ]; then
			tar -xf "${pkg}" -O +CONTENTS | awk '$1 == "@pkgdep" { print $2 }' > "${depfile}"
		else
			pkg info -qdF "${pkg}" > "${depfile}"
		fi
	fi

	echo ${depfile}
}
................................................................................
	local pkg=$1
	local originfile=$(pkg_cache_dir ${pkg})/origin
	local origin=$2

	if [ ! -f "${originfile}" ]; then
		if [ -z "${origin}" ]; then
			if [ "${PKG_EXT}" = "tbz" ]; then
				origin=$(tar -xf "${pkg}" -O +CONTENTS | \
					awk -F: '$1 == "@comment ORIGIN" { print $2 }')
			else
				origin=$(pkg query -F "${pkg}" "%o")
			fi
		fi
		echo ${origin} > "${originfile}"
	else
................................................................................
	[ $# -ne 1 ] && eargs pkg
	local pkg=$1
	local dep_origin_file=$(pkg_cache_dir ${pkg})/dep_origin
	local compiled_dep_origins

	if [ ! -f "${dep_origin_file}" ]; then
		if [ "${PKG_EXT}" = "tbz" ]; then
			compiled_dep_origins=$(tar -xf "${pkg}" -O +CONTENTS | \
				awk -F: '$1 == "@comment DEPORIGIN" {print $2}' | tr '\n' ' ')
		else
			compiled_dep_origins=$(pkg query -F "${pkg}" '%do' | tr '\n' ' ')
		fi
		echo "${compiled_dep_origins}" > "${dep_origin_file}"
		echo "${compiled_dep_origins}"
		return 0
................................................................................
	[ $# -ne 1 ] && eargs pkg
	local pkg=$1
	local optionsfile=$(pkg_cache_dir ${pkg})/options
	local compiled_options

	if [ ! -f "${optionsfile}" ]; then
		if [ "${PKG_EXT}" = "tbz" ]; then
			compiled_options=$(tar -xf "${pkg}" -O +CONTENTS | \
				awk -F: '$1 == "@comment OPTIONS" {print $2}' | tr ' ' '\n' | \
				sed -n 's/^\+\(.*\)/\1/p' | sort | tr '\n' ' ')
		else
			compiled_options=$(pkg query -F "${pkg}" '%Ov %Ok' | \
				awk '$1 == "on" {print $2}' | sort | tr '\n' ' ')
		fi
		echo "${compiled_options}" > "${optionsfile}"
................................................................................
	v2=${v2##*-}
	if [ "$v" != "$v2" ]; then
		msg "Deleting old version: ${pkg##*/}"
		delete_pkg ${pkg}
		return 0
	fi




	current_deps=$(injail make -C ${PORTSRC}/${o} run-depends-list | sed 's,${PORTSRC}/,,g' | tr '\n' ' ')

	compiled_deps=$(pkg_get_dep_origin ${pkg})
	for d in ${current_deps}; do
		case " $compiled_deps " in
		*\ $d\ *) ;;
		*)
			msg "Direct dependency change, deleting: ${pkg##*/}"

			delete_pkg ${pkg}
			return 0
			;;
		esac
	done


	# Check if the compiled options match the current options from make.conf and /var/db/options
	if [ "${CHECK_CHANGED_OPTIONS:-no}" != "no" ]; then
		current_options=$(injail make -C ${PORTSRC}/${o} pretty-print-config | \
			tr ' ' '\n' | sed -n 's/^\+\(.*\)/\1/p' | sort | tr '\n' ' ')
		compiled_options=$(pkg_get_options ${pkg})

		if [ "${compiled_options}" != "${current_options}" ]; then
			msg "Options changed, deleting: ${pkg##*/}"
			if [ "${CHECK_CHANGED_OPTIONS}" = "verbose" ]; then
................................................................................

# Take optional pkgname to speedup lookup
compute_deps() {
	[ $# -lt 1 ] && eargs port
	[ $# -gt 2 ] && eargs port pkgnme
	local port=$1
	local pkgname="${2:-$(cache_get_pkgname ${port})}"
	local mnt=$(my_path)
	local dep_pkgname dep_port
	local pkg_pooldir="${mnt}/poudriere/deps/${pkgname}"
	mkdir "${pkg_pooldir}" 2>/dev/null || return 0

	msg_verbose "Computing deps for ${port}"

	for dep_port in `list_deps ${port}`; do
		msg_debug "${port} depends on ${dep_port}"
		[ "${port}" != "${dep_port}" ] ||
................................................................................
		# Detect bad cat/origin/ dependency which pkgng will not register properly
		[ "${dep_port}" = "${dep_port%/}" ] ||
			err 1 "${port} depends on bad origin '${dep_port}'; Please contact maintainer of the port to fix this."
		dep_pkgname=$(cache_get_pkgname ${dep_port})

		# Only do this if it's not already done, and not ALL, as everything will
		# be touched anyway
		[ ${ALL:-0} -eq 0 ] && ! [ -d "${mnt}/poudriere/deps/${dep_pkgname}" ] &&
			compute_deps "${dep_port}" "${dep_pkgname}"

		touch "${pkg_pooldir}/${dep_pkgname}"
		mkdir -p "${mnt}/poudriere/rdeps/${dep_pkgname}"
		ln -sf "${pkg_pooldir}/${dep_pkgname}" \
			"${mnt}/poudriere/rdeps/${dep_pkgname}/${pkgname}"
	done
}

listed_ports() {
	if [ ${ALL:-0} -eq 1 ]; then
		PORTSDIR=$(pget ${PTNAME} mnt)
		[ -d "${PORTSDIR}/${PORTLOC}" ] && PORTSDIR="${PORTSDIR}/${PORTLOC}"
................................................................................
		for cat in $(awk '$1 == "SUBDIR" { print $3}' ${PORTSDIR}/Makefile); do
			awk -v cat=${cat} '$1 == "SUBDIR" { print cat"/"$3}' ${PORTSDIR}/${cat}/Makefile
		done
		return 0
	fi
	if [ -z "${LISTPORTS}" ]; then
		[ -n "${LISTPKGS}" ] &&
			grep -v -E '(^[[:space:]]*#|^[[:space:]]*$)' ${LISTPKGS}
	else
		echo ${LISTPORTS} | tr ' ' '\n'
	fi
}

parallel_exec() {
	local cmd="$1"

	shift 1
	${cmd} "$@"
	echo >&6

}

parallel_start() {
	local fifo

	if [ -n "${MASTERMNT}" ]; then
		fifo=${MASTERMNT}/poudriere/parallel.pipe
................................................................................
prepare_ports() {
	local pkg
	local log=$(log_path)
	local n port pn nbq resuming_build

	msg "Calculating ports order and dependencies"
	mkdir -p "${MASTERMNT}/poudriere"
	[ ${TMPFS_DATA} -eq 1 ] && mount -t tmpfs tmpfs "${MASTERMNT}/poudriere"
	rm -rf "${MASTERMNT}/poudriere/var/cache/origin-pkgname" \
		"${MASTERMNT}/poudriere/var/cache/pkgname-origin" 2>/dev/null || :
	mkdir -p "${MASTERMNT}/poudriere/building" \
		"${MASTERMNT}/poudriere/pool" \
		"${MASTERMNT}/poudriere/deps" \
		"${MASTERMNT}/poudriere/rdeps" \
		"${MASTERMNT}/poudriere/var/run" \
................................................................................
			POOL_BUCKET_DIRS="${POOL_BUCKET_DIRS} ${MASTERMNT}/poudriere/pool/${n}"
		done
	fi
	# Add unbalanced at the end
	POOL_BUCKET_DIRS="${POOL_BUCKET_DIRS} ${MASTERMNT}/poudriere/pool/unbalanced"
	mkdir -p ${POOL_BUCKET_DIRS}


	mkdir -p ${log}/logs ${log}/logs/errors
	ln -sfh ${BUILDNAME} ${log%/*}/latest
	cp ${HTMLPREFIX}/* ${log}

	bset status "computingdeps:"
	parallel_start
	for port in $(listed_ports); do
................................................................................
		bset stats_queued ${nbq##* }
	fi

	# Create a pool of ready-to-build from the deps pool
	find "${MASTERMNT}/poudriere/deps" -type d -empty -depth 1 | \
		xargs -J % mv % "${MASTERMNT}/poudriere/pool/unbalanced"
	balance_pool








}

balance_pool() {
	# Don't bother if disabled
	[ ${POOL_BUCKETS} -gt 0 ] || return 0

	local pkgname pkg_dir dep_count rdep lock
................................................................................
# Test if zpool exists
if [ -z "${NO_ZFS}" ]; then
	zpool list ${ZPOOL} >/dev/null 2>&1 || err 1 "No such zpool: ${ZPOOL}"
fi

: ${SVN_HOST="svn.FreeBSD.org"}
: ${GIT_URL="git://github.com/freebsd/freebsd-ports.git"}
: ${FREEBSD_HOST="${FTP_HOST:-ftp.FreeBSD.org}"}
if [ -z "${NO_ZFS}" ]; then
	: ${ZROOTFS="/poudriere"}
	case ${ZROOTFS} in
	[!/]*) err 1 "ZROOTFS shoud start with a /" ;;
	esac
fi








>
>
>
>
>
>
|




|
>

>
>
>

|

>
>
>
>
>
>
>
>
>







 







>
>

>
>
>
>
>
>

>







 







|







 







|
|
|
|
|
|







 







|
|







 







<
<







 







|
>
>
>
>
>
>

>







 







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>





>

>





>

>
>








<

>
>
>


>





>








|


<







 







|











|







 







>
>
>
>
>
>
>









>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>


>
>
|
|
|
>
|
|
|
>







 







|
|

>
>
>
>
>
>
>
>



|







 







|







 







<





|
|






|
|







 







|
|




<
|







 







|








<












|
|



|
|
|







 







|
|
|







 







|



|







 







|










|







 







|











|


<













|







 







|







 







|







 







|







 







|







 







>
>
>
|
>
|
|
|
|
|
<
>
|
|
|
|
|
>


|







 







<

|







 







|



|

|







 







|







>

|
|
>







 







|







 







>







 







>
>
>
>
>
>
>
>







 







|







100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
...
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
...
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
...
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
...
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
...
435
436
437
438
439
440
441


442
443
444
445
446
447
448
...
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
...
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531

532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555

556
557
558
559
560
561
562
...
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
...
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
...
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
...
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
...
983
984
985
986
987
988
989

990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
....
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024

1025
1026
1027
1028
1029
1030
1031
1032
....
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048

1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
....
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
....
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
....
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
....
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328

1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
....
1386
1387
1388
1389
1390
1391
1392
1393
1394
1395
1396
1397
1398
1399
1400
....
1404
1405
1406
1407
1408
1409
1410
1411
1412
1413
1414
1415
1416
1417
1418
....
1425
1426
1427
1428
1429
1430
1431
1432
1433
1434
1435
1436
1437
1438
1439
....
1446
1447
1448
1449
1450
1451
1452
1453
1454
1455
1456
1457
1458
1459
1460
....
1561
1562
1563
1564
1565
1566
1567
1568
1569
1570
1571
1572
1573
1574
1575
1576
1577

1578
1579
1580
1581
1582
1583
1584
1585
1586
1587
1588
1589
1590
1591
1592
1593
1594
....
1687
1688
1689
1690
1691
1692
1693

1694
1695
1696
1697
1698
1699
1700
1701
1702
....
1704
1705
1706
1707
1708
1709
1710
1711
1712
1713
1714
1715
1716
1717
1718
1719
1720
1721
1722
1723
1724
....
1725
1726
1727
1728
1729
1730
1731
1732
1733
1734
1735
1736
1737
1738
1739
1740
1741
1742
1743
1744
1745
1746
1747
1748
1749
1750
1751
....
1830
1831
1832
1833
1834
1835
1836
1837
1838
1839
1840
1841
1842
1843
1844
....
1853
1854
1855
1856
1857
1858
1859
1860
1861
1862
1863
1864
1865
1866
1867
....
1947
1948
1949
1950
1951
1952
1953
1954
1955
1956
1957
1958
1959
1960
1961
1962
1963
1964
1965
1966
1967
1968
....
2023
2024
2025
2026
2027
2028
2029
2030
2031
2032
2033
2034
2035
2036
2037
}

. $(dirname ${0})/common.sh.${BSDPLATFORM}

run_hook() {
	local hookfile=${HOOKDIR}/${1}.sh
	shift

	[ -f ${hookfile} ] &&
		URL_BASE="${URL_BASE}" \
		POUDRIERE_BUILD_TYPE=${POUDRIERE_BUILD_TYPE} \
		MASTERNAME="${MASTERNAME}" \
		BUILDNAME="${BUILDNAME}" \
		/bin/sh ${hookfile} $@
	return 0
}

log_start() {
	local log=$(log_path)
	local latest_log

	logfile="${log}/logs/${PKGNAME}.log"
	latest_log=${POUDRIERE_DATA}/logs/${POUDRIERE_BUILD_TYPE}/latest-per-pkg/${PKGNAME%-*}/${PKGNAME##*-}

	# Make sure directory exists
	mkdir -p ${log}/logs/success ${latest_log}

	# Symlink to /latest-per-pkg/PORTNAME/PKGVERSION/MASTERNAME.log
	ln -fs ${log}/../latest-per-pkg/${PKGNAME}.log \
		${latest_log}/${MASTERNAME}.log

	# Symlink to JAIL/latest-per-pkg/PKGNAME.log
	ln -fs ${log}/logs/${PKGNAME}.log \
		${log}/../latest-per-pkg/${PKGNAME}.log

	# Tee all of the output to the logfile through a pipe
	exec 3>&1 4>&2
	[ ! -e ${logfile}.pipe ] && mkfifo ${logfile}.pipe
	tee ${logfile} < ${logfile}.pipe >&3 &
	export tpid=$!
	exec > ${logfile}.pipe 2>&1

	# Remove fifo pipe file right away to avoid orphaning it.
................................................................................
	echo "---Begin make.conf---"
	cat ${mnt}/etc/make.conf
	echo "---End make.conf---"
}

buildlog_stop() {
	local portdir=$1
	local log=$(log_path)
	local buildtime

	buildtime=$( \
		stat -f '%N %B' ${log}/logs/${PKGNAME}.log  | awk -v now=$(date +%s) \
		-f ${AWKPREFIX}/siginfo_buildtime.awk |
		awk -F'!' '{print $2}' \
	)

	echo "build of ${portdir} ended at $(date)"
	echo "build time: ${buildtime}"
}

log_stop() {
	if [ -n "${tpid}" ]; then
		exec 1>&3 3>&- 2>&4 4>&-
		kill $tpid
		wait $tpid 2>/dev/null || :
................................................................................

exit_handler() {
	# Avoid recursively cleaning up here
	trap - EXIT SIGTERM SIGKILL
	# Ignore SIGINT while cleaning up
	trap '' SIGINT

	if [ ${STATUS} -eq 1 ] && [ "${POUDRIERE_BUILD_TYPE}" = "bulk" \
		-o "${POUDRIERE_BUILD_TYPE}" = "testport" ]; then
		log_stop
		stop_html_json
	fi

	# Kill all children - this does NOT recurse, so orphans can still
	# occur. This is just to avoid requiring pid files for parallel_run
................................................................................
		msg "WWW: ${URL_BASE}/${POUDRIERE_BUILD_TYPE}/${MASTERNAME}/${BUILDNAME}"
}

siginfo_handler() {
	[ "${POUDRIERE_BUILD_TYPE}" != "bulk" ] && return 0

	trappedinfo=1
	local status=$(bget status 2> /dev/null || echo unknown)
	local nbb=$(bget stats_built 2>/dev/null || echo 0)
	local nbf=$(bget stats_failed 2>/dev/null || echo 0)
	local nbi=$(bget stats_ignored 2>/dev/null || echo 0)
	local nbs=$(bget stats_skipped 2>/dev/null || echo 0)
	local nbq=$(bget stats_queued 2>/dev/null || echo 0)
	local ndone=$((nbb + nbf + nbi + nbs))
	local queue_width=2
	local now
	local j
	local pkgname origin phase buildtime
	local format_origin_phase format_phase

................................................................................
	elif [ ${nbq} -gt 99 ]; then
		queue_width=3
	fi

	# Skip if stopping or starting jobs
	if [ -n "${JOBS}" -a "${status#starting_jobs:}" = "${status}" -a "${status}" != "stopping_jobs:" ]; then
		now=$(date +%s)
		format_origin_phase="\t[%s]: %-32s %-15s (%s)\n"
		format_phase="\t[%s]: %15s\n"

		# Collect build stats into a string with minimal execs
		pkgname_buildtimes=$(find ${MASTERMNT}/poudriere/building -depth 1 \
			-exec stat -f "%N %m" {} + | \
			awk -v now=${now} -f ${AWKPREFIX}/siginfo_buildtime.awk)

		for j in ${JOBS}; do
................................................................................
}

fetch_file() {
	[ $# -ne 2 ] && eargs destination source
	fetch -p -o $1 $2 || fetch -p -o $1 $2 || err 1 "Failed to fetch from $2"
}



unmarkfs() {
	[ $# -ne 2 ] && eargs name mnt
	local name=$1
	local mnt=$(realpath $2)

	if [ -n "$(zfs_getfs ${mnt})" ]; then
		zfs destroy -f ${fs}@${name} 2>/dev/null || :
................................................................................
	local dozfs=0
	local domtree=0

	case "${name}" in
	clean) [ -n "${fs}" ] && dozfs=1 ;;
	prepkg)
		[ -n "${fs}" ] && dozfs=1
		# Only create prepkg mtree in ref
		# Everything else may need to snapshot
		if [ "${mnt##*/}" = "ref" ]; then
			domtree=1
		else
			domtree=0
		fi
		;;
	prebuild) domtree=1 ;;
	preinst) domtree=1 ;;
	esac

	if [ $dozfs -eq 1 ]; then
		# remove old snapshot if exists
		zfs destroy -r ${fs}@${name} 2>/dev/null || :
		#create new snapshot
................................................................................

	[ $domtree -eq 0 ] && return 0
	mkdir -p ${mnt}/poudriere/
	if [ "${name}" = "prepkg" ]; then
		cat > ${mnt}/poudriere/mtree.${name}exclude << EOF
./ccache/*
./compat/linux/proc
./dev/*
./distfiles/*
./new_packages/*
./packages/*
./poudriere/*
./proc/*
.${PORTSRC}/*
./usr/src
./var/db/ports/*
./wrkdirs/*
EOF
	elif [ "${name}" = "prebuild" ]; then
		cat >  ${mnt}/poudriere/mtree.${name}exclude << EOF
./ccache/*
./compat/linux/proc
./dev/*
./distfiles/*
./new_packages/*
./packages/*
./poudriere/*
./proc/*
./tmp/*
.${PORTSRC}/*
./usr/src
./var/db/ports/*
./wrkdirs/*
EOF
	elif [ "${name}" = "preinst" ]; then
		cat >  ${mnt}/poudriere/mtree.${name}exclude << EOF
./ccache/*
./compat/linux/proc/*
./dev/*
./distfiles/*
./etc/group
./etc/make.conf
./etc/make.conf.bak
./etc/master.passwd
./etc/passwd
./etc/pwd.db
./etc/shells
./etc/spwd.db

./new_packages/*
./packages/*
./poudriere/*
./proc/*
.${HOME}/*
./tmp/*
.${PORTSRC}/*
.${LOCALBASE:-/usr/local}/etc/gconf/gconf.xml.defaults
.${LOCALBASE:-/usr/local}/lib/charset.alias
.${LOCALBASE:-/usr/local}/share/applications/mimeinfo.cache
.${LOCALBASE:-/usr/local}/share/nls/POSIX
.${LOCALBASE:-/usr/local}/share/nls/en_US.US-ASCII
./usr/src
./var/db/*
./var/log/*
./var/mail/*
./var/run/*
./wrkdirs/*
EOF
	fi
	mtree -X ${mnt}/poudriere/mtree.${name}exclude \
		-cn -k uid,gid,mode,size \
		-p ${mnt} > ${mnt}/poudriere/mtree.${name}
}


use_options() {
	[ $# -ne 2 ] && eargs mnt optionsdir
	local mnt=$1
	local optionsdir=$2

	if [ "${optionsdir}" = "-" ]; then
................................................................................
		# Check for non-empty directory with no packages in it
		[ "${pkg}" = "${POUDRIERE_DATA}/packages/${MASTERNAME}/All/*.${PKG_EXT}" ] && break
		depfile=$(deps_file ${pkg})
		while read dep; do
			if [ ! -e "${POUDRIERE_DATA}/packages/${MASTERNAME}/All/${dep}.${PKG_EXT}" ]; then
				ret=1
				msg_debug "${pkg} needs missing ${POUDRIERE_DATA}/packages/${MASTERNAME}/All/${dep}.${PKG_EXT}"
				msg "Deleting ${pkg##*/}: missing dependency: ${dep}"
				delete_pkg ${pkg}
				break
			fi
		done < "${depfile}"
	done

	return $ret
}

check_leftovers() {
	local mnt=$1
	mtree -X ${mnt}/poudriere/mtree.preinstexclude \
		-f ${mnt}/poudriere/mtree.preinst \
		-p ${mnt} | while read l ; do
		case ${l} in
		*extra)
			if [ -d ${mnt}/${l% *} ]; then
				find ${mnt}/${l% *} -exec echo "+ {}" \;
			else
................................................................................
				   extract-depends extract patch-depends patch build-depends \
				   lib-depends configure build install-mtree run-depends \
				   install package ${PORTTESTING:+deinstall}"
	local mnt=$(my_path)
	local log=$(log_path)
	local listfilecmd network
	local hangstatus
	local pkgenv

	# If not testing, then avoid rechecking deps in build/install;
	# When testing, check depends twice to ensure they depend on
	# proper files, otherwise they'll hit 'package already installed'
	# errors.
	[ -z "${PORTTESTING}" ] && PORT_FLAGS="${PORT_FLAGS} NO_DEPENDS=yes"

	for phase in ${targets}; do
		bset ${MY_JOBID} status "${phase}:${port}"
		job_msg_verbose "Status   ${port}: ${phase}"
		if [ "${phase}" = "fetch" ]; then
			jstop
			jstart 1
		fi
		case ${phase} in
		configure) [ -n "${PORTTESTING}" ] && markfs prebuild ${mnt} ;;
		install-mtree)
			if [ -n "${PORTTESTING}" ]; then
				mtree -X ${mnt}/poudriere/mtree.prebuildexclude \
					-f ${mnt}/poudriere/mtree.prebuild \
					-p ${mnt} > ${mnt}/tmp/preinst
				if [ -s ${mnt}/tmp/preinst ]; then
					msg "Filesystem touched before install:"
					cat ${mnt}/tmp/preinst
					rm -f ${mnt}/tmp/preinst
					bset ${MY_JOBID} status "preinst_fs_violation:${port}"
					job_msg_verbose "Status for build ${port}: preinst_fs_violation"
					return 1
				fi
				rm -f ${mnt}/tmp/preinst
			fi
			;;
		install) [ -n "${PORTTESTING}" ] && markfs preinst ${mnt} ;;
		deinstall)
			# Skip for all linux ports, they are not safe
			if [ "${PKGNAME%%*linux*}" != "" ]; then
				msg "Checking shared library dependencies"
				listfilecmd="grep -v '^@' /var/db/pkg/${PKGNAME}/+CONTENTS"
				[ ${PKGNG} -eq 1 ] && listfilecmd="pkg query '%Fp' ${PKGNAME}"
				echo "${listfilecmd} | xargs ldd 2>&1 |
					awk '/=>/ { print $3 }' | sort -u" > ${mnt}/shared.sh
				injail sh /shared.sh
				rm -f ${mnt}/shared.sh
			fi
			;;
		esac

		print_phase_header ${phase}

		if [ "${phase}" = "package" ]; then
			echo "PACKAGES=/new_packages" >> ${mnt}/etc/make.conf
................................................................................
			mkdir -p "${POUDRIERE_DATA}/packages/${MASTERNAME}/.new_packages/${PKGNAME}"
			${NULLMOUNT} \
				"${POUDRIERE_DATA}/packages/${MASTERNAME}/.new_packages/${PKGNAME}" \
				${mnt}/new_packages
		fi

		if [ "${phase#*-}" = "depends" ]; then
			# No need for nohang or PORT_FLAGS for *-depends
			injail make -C ${portdir} ${phase} || return 1
		else
			# Only set PKGENV during 'package' to prevent testport-built
			# packages from going into the main repo
			if [ "${phase}" = "package" ]; then
				pkgenv="${PKGENV}"
			else
				pkgenv=
			fi

			# 24 hours for 1 command, or 20 minutes with no log update
			nohang ${MAX_EXECUTION_TIME:-86400} ${NOHANG_TIME:-7200} \
				${log}/logs/${PKGNAME}.log \
				injail env ${pkgenv} ${PORT_FLAGS} \
				make -C ${portdir} ${phase}
			hangstatus=$? # This is done as it may return 1 or 2 or 3
			if [ $hangstatus -ne 0 ]; then
				# 1 = cmd failed, not a timeout
				# 2 = log timed out
				# 3 = cmd timeout
				if [ $hangstatus -eq 2 ]; then
................................................................................
	local port="$2"
	local portdir="$(echo "$3" | sed -e "s|${PORTSRC}/||g")"
	local phase="$4"
	local tardir=${POUDRIERE_DATA}/wrkdirs/${MASTERNAME}/${PTNAME}
	local tarname=${tardir}/${PKGNAME}.${WRKDIR_ARCHIVE_FORMAT}
	local mnted_portdir=${mnt}/wrkdirs/${portdir}

	[ "${SAVE_WRKDIR:-no}" != "no" ] || return 0
	# Only save if not in fetch/checksum phase
	[ "${failed_phase}" != "fetch" -a "${failed_phase}" != "checksum" -a \
		"${failed_phase}" != "extract" ] || return 0

	mkdir -p ${tardir}

	# Tar up the WRKDIR, and ignore errors
................................................................................

	job_msg "Saved ${port} wrkdir to: ${tarname}"
}


deadlock_detected() {
	local always_fail=${1:-1}

	local crashed_packages dependency_cycles

	# If there are still packages marked as "building" they have crashed
	# and it's likely some poudriere or system bug
	crashed_packages=$( \
		find ${MASTERMNT}/poudriere/building -type d -mindepth 1 -maxdepth 1 | \
		sed -e "s,${MASTERMNT}/poudriere/building/,," | tr '\n' ' ' \
	)
	[ -z "${crashed_packages}" ] ||	\
		err 1 "Crashed package builds detected: ${crashed_packages}"

	# Check if there's a cycle in the need-to-build queue
	dependency_cycles=$(\
		find ${MASTERMNT}/poudriere/deps -mindepth 2 | \
		sed -e "s,${MASTERMNT}/poudriere/deps/,," -e 's:/: :' | \
		# Only cycle errors are wanted
		tsort 2>&1 >/dev/null | \
		sed -e 's/tsort: //' | \
		awk -f ${AWKPREFIX}/dependency_loop.awk \
	)

	if [ -n "${dependency_cycles}" ]; then
................................................................................
		err 1 "Dependency loop detected:
${dependency_cycles}"
	fi

	[ ${always_fail} -eq 1 ] || return 0

	# No cycle, there's some unknown poudriere bug
	err 1 "Unknown stuck queue bug detected. Please submit the entire build output to poudriere developers.
$(find ${MASTERMNT}/poudriere/building ${MASTERMNT}/poudriere/pool ${MASTERMNT}/poudriere/deps)"
}

queue_empty() {
	local pool_dir

	dirempty ${MASTERMNT}/poudriere/deps || return 1

	for pool_dir in ${POOL_BUCKET_DIRS}; do
		dirempty ${pool_dir} || return 1
	done

	return 0
}
................................................................................

mark_done() {
	[ $# -eq 1 ] || eargs pkgname
	local pkgname="$1"
	local origin=$(cache_get_origin "${pkgname}")
	local cache_dir=$(cache_dir)

	echo -n "${origin} $(date +%s) " >> ${cache_dir}/buildtimes
	stat -f "%m" ${MASTERMNT}/poudriere/building/${pkgname} >> \
		${cache_dir}/buildtimes
	rmdir ${MASTERMNT}/poudriere/building/${pkgname}
}


build_queue() {
	local j name pkgname builders_active queue_empty


	mkfifo ${MASTERMNT}/poudriere/builders.pipe
	exec 6<> ${MASTERMNT}/poudriere/builders.pipe
	rm -f ${MASTERMNT}/poudriere/builders.pipe
	queue_empty=0

	msg "Hit CTRL+t at any time to see build progress and stats"

	while :; do
		builders_active=0
		for j in ${JOBS}; do
			name="${MASTERNAME}-job-${j}"
			if [ -f  "${MASTERMNT}/poudriere/var/run/${j}.pid" ]; then
				if pgrep -qF "${MASTERMNT}/poudriere/var/run/${j}.pid" 2>/dev/null; then
					builders_active=1
					continue
				fi
				read pkgname < ${MASTERMNT}/poudriere/var/run/${j}.pkgname
				rm -f ${MASTERMNT}/poudriere/var/run/${j}.pid \
					${MASTERMNT}/poudriere/var/run/${j}.pkgname
				bset ${j} status "idle:"
				mark_done ${pkgname}
			fi

			[ ${queue_empty} -eq 0 ] || continue

			pkgname=$(next_in_queue)
................................................................................
				# Check if the ready-to-build pool and need-to-build pools
				# are empty
				queue_empty && queue_empty=1

				# Pool is waiting on dep, wait until a build
				# is done before checking the queue again
			else
				MY_JOBID="${j}" build_pkg "${pkgname}" > /dev/null &
				echo "$!" > ${MASTERMNT}/poudriere/var/run/${j}.pid
				echo "${pkgname}" > ${MASTERMNT}/poudriere/var/run/${j}.pkgname

				# A new job is spawned, try to read the queue
				# just to keep things moving
				builders_active=1
			fi
		done

................................................................................
		run_hook pkgbuild skipped "${skipped_origin}" "${skipped_pkgname}" "${port}"
	done

	balance_pool
}

print_phase_header() {
	printf "=======================<phase: %-15s>============================\n" "$1"
}

print_phase_footer() {
	echo "========================================================================="
}

build_pkg() {
	# If this first check fails, the pool will not be cleaned up,
	# since PKGNAME is not yet set.
	[ $# -ne 1 ] && eargs pkgname
	local pkgname="$1"
................................................................................
	port=$(cache_get_origin ${pkgname})
	portdir="${PORTSRC}/${port}"

	job_msg "Starting build of ${port}"
	bset ${MY_JOBID} status "starting:${port}"
	create_slave ${MASTERMNT} ${MY_JOBID}

	if [ ${TMPFS_LOCALBASE} -eq 1 -o ${TMPFS_ALL} -eq 1 ]; then
		umount -f ${mnt}/${LOCALBASE:-/usr/local} 2>/dev/null || :
		mount -t tmpfs tmpfs ${mnt}/${LOCALBASE:-/usr/local}
	fi

	# If this port is IGNORED, skip it
	# This is checked here instead of when building the queue
	# as the list may start big but become very small, so here
	# is a less-common check
	ignore="$(injail make -C ${portdir} -VIGNORE)"

	log_start
	msg "Building ${port}"
	buildlog_start ${portdir}

	if [ -n "${ignore}" ]; then
		msg "Ignoring ${port}: ${ignore}"
		badd ports.ignored "${port} ${PKGNAME} ${ignore}"
		job_msg "Finished build of ${port}: Ignored: ${ignore}"
................................................................................
		fi
	fi

	clean_pool ${PKGNAME} ${clean_rdepends}

	bset ${MY_JOBID} status "done:${port}"

	stop_build ${portdir}

	destroy_slave ${MASTERMNT} ${MY_JOBID}
	if [ ${zip_log} -eq 1 ]; then
		bzip2 ${log}/logs/${PKGNAME}.log
		mv ${log}/logs/${PKGNAME}.log.bz2 $(log_path)/success/
	fi

	echo ${MY_JOBID} >&6
}

stop_build() {
	[ $# -eq 1 ] || eargs portdir
	local portdir="$1"
	local mnt=$(my_path)


	umount -f ${mnt}/new_packages 2>/dev/null || :
	rm -rf "${POUDRIERE_DATA}/packages/${MASTERNAME}/.new_packages/${PKGNAME}"

	# 2 = HEADER+ps itself
	if [ $(injail ps aux | wc -l) -ne 2 ]; then
		msg "Leftover processes:"
		injail ps auxwwd | grep -v 'ps auxwwd'
	fi
	# Always kill to avoid missing anything
	injail kill -9 -1 2>/dev/null || :

	buildlog_stop ${portdir}
	log_stop
}

# Crazy redirection is to add the portname into stderr.
# Idea from http://superuser.com/a/453609/34747
mangle_stderr() {
	local msg_type="$1"
	local extra="$2"
................................................................................
deps_file() {
	[ $# -ne 1 ] && eargs pkg
	local pkg=$1
	local depfile=$(pkg_cache_dir ${pkg})/deps

	if [ ! -f "${depfile}" ]; then
		if [ "${PKG_EXT}" = "tbz" ]; then
			tar -qxf "${pkg}" -O +CONTENTS | awk '$1 == "@pkgdep" { print $2 }' > "${depfile}"
		else
			pkg info -qdF "${pkg}" > "${depfile}"
		fi
	fi

	echo ${depfile}
}
................................................................................
	local pkg=$1
	local originfile=$(pkg_cache_dir ${pkg})/origin
	local origin=$2

	if [ ! -f "${originfile}" ]; then
		if [ -z "${origin}" ]; then
			if [ "${PKG_EXT}" = "tbz" ]; then
				origin=$(tar -qxf "${pkg}" -O +CONTENTS | \
					awk -F: '$1 == "@comment ORIGIN" { print $2 }')
			else
				origin=$(pkg query -F "${pkg}" "%o")
			fi
		fi
		echo ${origin} > "${originfile}"
	else
................................................................................
	[ $# -ne 1 ] && eargs pkg
	local pkg=$1
	local dep_origin_file=$(pkg_cache_dir ${pkg})/dep_origin
	local compiled_dep_origins

	if [ ! -f "${dep_origin_file}" ]; then
		if [ "${PKG_EXT}" = "tbz" ]; then
			compiled_dep_origins=$(tar -qxf "${pkg}" -O +CONTENTS | \
				awk -F: '$1 == "@comment DEPORIGIN" {print $2}' | tr '\n' ' ')
		else
			compiled_dep_origins=$(pkg query -F "${pkg}" '%do' | tr '\n' ' ')
		fi
		echo "${compiled_dep_origins}" > "${dep_origin_file}"
		echo "${compiled_dep_origins}"
		return 0
................................................................................
	[ $# -ne 1 ] && eargs pkg
	local pkg=$1
	local optionsfile=$(pkg_cache_dir ${pkg})/options
	local compiled_options

	if [ ! -f "${optionsfile}" ]; then
		if [ "${PKG_EXT}" = "tbz" ]; then
			compiled_options=$(tar -qxf "${pkg}" -O +CONTENTS | \
				awk -F: '$1 == "@comment OPTIONS" {print $2}' | tr ' ' '\n' | \
				sed -n 's/^\+\(.*\)/\1/p' | sort | tr '\n' ' ')
		else
			compiled_options=$(pkg query -F "${pkg}" '%Ov %Ok' | \
				awk '$1 == "on" {print $2}' | sort | tr '\n' ' ')
		fi
		echo "${compiled_options}" > "${optionsfile}"
................................................................................
	v2=${v2##*-}
	if [ "$v" != "$v2" ]; then
		msg "Deleting old version: ${pkg##*/}"
		delete_pkg ${pkg}
		return 0
	fi

	# Detect ports that have new dependencies that the existing packages
	# do not have and delete them.
	if [ "${CHECK_CHANGED_DEPS:-yes}" != "no" ]; then
		current_deps=$(injail make -C ${PORTSRC}/${o} run-depends-list | \
			sed "s,${PORTSRC}/,,g" | tr '\n' ' ')
		compiled_deps=$(pkg_get_dep_origin ${pkg})
		for d in ${current_deps}; do
			case " $compiled_deps " in
			*\ $d\ *) ;;
			*)

				msg "Deleting ${pkg##*/}: new dependency: ${d}"
				delete_pkg ${pkg}
				return 0
				;;
			esac
		done
	fi

	# Check if the compiled options match the current options from make.conf and /var/db/options
	if [ "${CHECK_CHANGED_OPTIONS:-verbose}" != "no" ]; then
		current_options=$(injail make -C ${PORTSRC}/${o} pretty-print-config | \
			tr ' ' '\n' | sed -n 's/^\+\(.*\)/\1/p' | sort | tr '\n' ' ')
		compiled_options=$(pkg_get_options ${pkg})

		if [ "${compiled_options}" != "${current_options}" ]; then
			msg "Options changed, deleting: ${pkg##*/}"
			if [ "${CHECK_CHANGED_OPTIONS}" = "verbose" ]; then
................................................................................

# Take optional pkgname to speedup lookup
compute_deps() {
	[ $# -lt 1 ] && eargs port
	[ $# -gt 2 ] && eargs port pkgnme
	local port=$1
	local pkgname="${2:-$(cache_get_pkgname ${port})}"

	local dep_pkgname dep_port
	local pkg_pooldir="${MASTERMNT}/poudriere/deps/${pkgname}"
	mkdir "${pkg_pooldir}" 2>/dev/null || return 0

	msg_verbose "Computing deps for ${port}"

	for dep_port in `list_deps ${port}`; do
		msg_debug "${port} depends on ${dep_port}"
		[ "${port}" != "${dep_port}" ] ||
................................................................................
		# Detect bad cat/origin/ dependency which pkgng will not register properly
		[ "${dep_port}" = "${dep_port%/}" ] ||
			err 1 "${port} depends on bad origin '${dep_port}'; Please contact maintainer of the port to fix this."
		dep_pkgname=$(cache_get_pkgname ${dep_port})

		# Only do this if it's not already done, and not ALL, as everything will
		# be touched anyway
		[ ${ALL:-0} -eq 0 ] && ! [ -d "${MASTERMNT}/poudriere/deps/${dep_pkgname}" ] &&
			compute_deps "${dep_port}" "${dep_pkgname}"

		touch "${pkg_pooldir}/${dep_pkgname}"
		mkdir -p "${MASTERMNT}/poudriere/rdeps/${dep_pkgname}"
		ln -sf "${pkg_pooldir}/${dep_pkgname}" \
			"${MASTERMNT}/poudriere/rdeps/${dep_pkgname}/${pkgname}"
	done
}

listed_ports() {
	if [ ${ALL:-0} -eq 1 ]; then
		PORTSDIR=$(pget ${PTNAME} mnt)
		[ -d "${PORTSDIR}/${PORTLOC}" ] && PORTSDIR="${PORTSDIR}/${PORTLOC}"
................................................................................
		for cat in $(awk '$1 == "SUBDIR" { print $3}' ${PORTSDIR}/Makefile); do
			awk -v cat=${cat} '$1 == "SUBDIR" { print cat"/"$3}' ${PORTSDIR}/${cat}/Makefile
		done
		return 0
	fi
	if [ -z "${LISTPORTS}" ]; then
		[ -n "${LISTPKGS}" ] &&
			grep -h -v -E '(^[[:space:]]*#|^[[:space:]]*$)' ${LISTPKGS}
	else
		echo ${LISTPORTS} | tr ' ' '\n'
	fi
}

parallel_exec() {
	local cmd="$1"
	local ret=0
	shift 1
	${cmd} "$@" || ret=1
	echo >&6 || :
	exit ${ret}
}

parallel_start() {
	local fifo

	if [ -n "${MASTERMNT}" ]; then
		fifo=${MASTERMNT}/poudriere/parallel.pipe
................................................................................
prepare_ports() {
	local pkg
	local log=$(log_path)
	local n port pn nbq resuming_build

	msg "Calculating ports order and dependencies"
	mkdir -p "${MASTERMNT}/poudriere"
	[ ${TMPFS_DATA} -eq 1 -o ${TMPFS_ALL} -eq 1 ] && mount -t tmpfs tmpfs "${MASTERMNT}/poudriere"
	rm -rf "${MASTERMNT}/poudriere/var/cache/origin-pkgname" \
		"${MASTERMNT}/poudriere/var/cache/pkgname-origin" 2>/dev/null || :
	mkdir -p "${MASTERMNT}/poudriere/building" \
		"${MASTERMNT}/poudriere/pool" \
		"${MASTERMNT}/poudriere/deps" \
		"${MASTERMNT}/poudriere/rdeps" \
		"${MASTERMNT}/poudriere/var/run" \
................................................................................
			POOL_BUCKET_DIRS="${POOL_BUCKET_DIRS} ${MASTERMNT}/poudriere/pool/${n}"
		done
	fi
	# Add unbalanced at the end
	POOL_BUCKET_DIRS="${POOL_BUCKET_DIRS} ${MASTERMNT}/poudriere/pool/unbalanced"
	mkdir -p ${POOL_BUCKET_DIRS}

	mkdir -p ${log}/../../latest-per-pkg ${log}/../latest-per-pkg
	mkdir -p ${log}/logs ${log}/logs/errors
	ln -sfh ${BUILDNAME} ${log%/*}/latest
	cp ${HTMLPREFIX}/* ${log}

	bset status "computingdeps:"
	parallel_start
	for port in $(listed_ports); do
................................................................................
		bset stats_queued ${nbq##* }
	fi

	# Create a pool of ready-to-build from the deps pool
	find "${MASTERMNT}/poudriere/deps" -type d -empty -depth 1 | \
		xargs -J % mv % "${MASTERMNT}/poudriere/pool/unbalanced"
	balance_pool

	[ -z "${PORTTESTING}" -a -z "${ALLOW_MAKE_JOBS}" ] &&
		echo "DISABLE_MAKE_JOBS=yes" >> ${MASTERMNT}/etc/make.conf

	[ -n "${JOBS_LIMIT}" ] && echo "MAKE_JOBS_NUMBER=${JOBS_LIMIT}" \
		>> ${MASTERMNT}/etc/make.conf

	markfs prepkg ${MASTERMNT}
}

balance_pool() {
	# Don't bother if disabled
	[ ${POOL_BUCKETS} -gt 0 ] || return 0

	local pkgname pkg_dir dep_count rdep lock
................................................................................
# Test if zpool exists
if [ -z "${NO_ZFS}" ]; then
	zpool list ${ZPOOL} >/dev/null 2>&1 || err 1 "No such zpool: ${ZPOOL}"
fi

: ${SVN_HOST="svn.FreeBSD.org"}
: ${GIT_URL="git://github.com/freebsd/freebsd-ports.git"}
: ${FREEBSD_HOST="http://ftp.FreeBSD.org"}
if [ -z "${NO_ZFS}" ]; then
	: ${ZROOTFS="/poudriere"}
	case ${ZROOTFS} in
	[!/]*) err 1 "ZROOTFS shoud start with a /" ;;
	esac
fi

Changes to src/share/poudriere/common.sh.dragonfly.

76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
	      ${BASEIMG}/var \
	      ${BASEIMG}/usr/local

	[ -n "${RESOLV_CONF}" ] && cp -v "${RESOLV_CONF}" "${BASEIMG}/etc2/"

	if [ -d "${CCACHE_DIR:-/nonexistent}" ]; then
		echo "WITH_CCACHE_BUILD=yes" >> ${tomnt}/etc2/make.conf
		echo "MAKE_ENV+= CCACHE_DIR=/ccache" >> ${tomnt}/etc2/make.conf
	fi
	echo "PACKAGES=/packages" >> ${tomnt}/etc2/make.conf
	echo "DISTDIR=/distfiles" >> ${tomnt}/etc2/make.conf

	makeconf="- ${setname} ${ptname} ${name} ${name}-${ptname}"
	[ -n "${setname}" ] && makeconf="${makeconf} ${name}-${setname}"
	makeconf="${makeconf} ${MASTERNAME}"







|







76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
	      ${BASEIMG}/var \
	      ${BASEIMG}/usr/local

	[ -n "${RESOLV_CONF}" ] && cp -v "${RESOLV_CONF}" "${BASEIMG}/etc2/"

	if [ -d "${CCACHE_DIR:-/nonexistent}" ]; then
		echo "WITH_CCACHE_BUILD=yes" >> ${tomnt}/etc2/make.conf
		echo "CCACHE_DIR=/ccache" >> ${tomnt}/etc2/make.conf
	fi
	echo "PACKAGES=/packages" >> ${tomnt}/etc2/make.conf
	echo "DISTDIR=/distfiles" >> ${tomnt}/etc2/make.conf

	makeconf="- ${setname} ${ptname} ${name} ${name}-${ptname}"
	[ -n "${setname}" ] && makeconf="${makeconf} ${name}-${setname}"
	makeconf="${makeconf} ${MASTERNAME}"

Changes to src/share/poudriere/distclean.sh.

96
97
98
99
100
101
102
103
104
105
106
107
108
109
110

	msg_verbose "Gathering distfiles for: ${origin}"

	awk -v distdir="${DISTFILES_CACHE%/}" '/SIZE/ {print distdir "/" substr($2, 2, length($2) - 2)}' \
		"${distinfo_file}" >> ${DISTFILES_LIST}
}

msg "Gathering all expected disfiles"
parallel_start
for origin in $(listed_ports); do
	parallel_run gather_distfiles ${origin}
done
parallel_stop

# Remove duplicates







|







96
97
98
99
100
101
102
103
104
105
106
107
108
109
110

	msg_verbose "Gathering distfiles for: ${origin}"

	awk -v distdir="${DISTFILES_CACHE%/}" '/SIZE/ {print distdir "/" substr($2, 2, length($2) - 2)}' \
		"${distinfo_file}" >> ${DISTFILES_LIST}
}

msg "Gathering all expected distfiles"
parallel_start
for origin in $(listed_ports); do
	parallel_run gather_distfiles ${origin}
done
parallel_stop

# Remove duplicates

Changes to src/share/poudriere/html/poudriere.js.

128
129
130
131
132
133
134

135

136
137
138
139
140
141
142
...
152
153
154
155
156
157
158

159
160
161
162
163
164
165
166
167

168
169
170
171
172
173
174
function format_setname(setname) {
	return setname ? ('-' + setname) : '';
}

function process_data(data) {
	var html, a, n;


	update_canvas(data.stats);


	document.title = 'Poudriere bulk results for ' + data.jail +
		format_setname(data.setname) + '-' + data.ptname + ' ' +
		data.buildname;

	$('#jail').html(data.jail);
	$('#setname').html(data.setname);
................................................................................
		a = builder.status.split(":");
		html += "<td>" + format_origin(a[1]) + "</td>";
		html += "<td>" + a[0] + "</td></tr>";
	}
	$('#builders_body').html(html);

	/* Stats */

	$.each(data.stats, function(status, count) {
		if (status == "queued") {
			html = count;
		} else {
			html = '<a href="#' + status + '">' + count + '</a>';
		}

		$('#stats_' + status).html(html);
	});


	/* For each status, track how many of the existing data has been
	 * added to the table. On each update, only append new data. This
	 * is to lessen the amount of DOM redrawing on -a builds that
	 * may involve looping 24000 times. */

	if (data.ports) {







>
|
>







 







>
|
|
|
|
|
|

|
|
>







128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
...
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
function format_setname(setname) {
	return setname ? ('-' + setname) : '';
}

function process_data(data) {
	var html, a, n;

	if (data.stats) {
		update_canvas(data.stats);
	}

	document.title = 'Poudriere bulk results for ' + data.jail +
		format_setname(data.setname) + '-' + data.ptname + ' ' +
		data.buildname;

	$('#jail').html(data.jail);
	$('#setname').html(data.setname);
................................................................................
		a = builder.status.split(":");
		html += "<td>" + format_origin(a[1]) + "</td>";
		html += "<td>" + a[0] + "</td></tr>";
	}
	$('#builders_body').html(html);

	/* Stats */
	if (data.stats) {
		$.each(data.stats, function(status, count) {
			if (status == "queued") {
				html = count;
			} else {
				html = '<a href="#' + status + '">' + count + '</a>';
			}

			$('#stats_' + status).html(html);
		});
	}

	/* For each status, track how many of the existing data has been
	 * added to the table. On each update, only append new data. This
	 * is to lessen the amount of DOM redrawing on -a builds that
	 * may involve looping 24000 times. */

	if (data.ports) {

Changes to src/share/poudriere/jail.sh.dragonfly.

2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
#
# DragonFly specific

delete_jail() {
	test -z ${JAILNAME} && usage
	jail_exists ${JAILNAME} || err 1 "No such jail: ${JAILNAME}"
	jail_runs ${JAILNAME} &&
		err 1 "Unable to remove jail ${JAILNAME}: it is running"

	msg_n "Removing ${JAILNAME} jail..."
	destroyfs ${JAILMNT} jail
	rm -rf ${POUDRIERED}/jails/${JAILNAME} || :
	echo "done"
}

update_version() {
	local release="$1"
	local login_env osversion

	osversion=`awk '/\#define __DragonFly_version/ { print $3 }' ${JAILMNT}/usr/include/sys/param.h`
	login_env=",UNAME_r=${release},UNAME_v=DragonFly ${release},OSVERSION=${osversion}"

	sed -i "" -e "s/:\(setenv.*\):/:\1${login_env}:/" ${JAILMNT}/etc/login.conf
	cap_mkdb ${JAILMNT}/etc/login.conf
}

establish_jail() {
# REMOVE THIS.
	[ $# -ne 5 ] && eargs name version arch mountpoint filesys
	local name=$1







|
<













|







2
3
4
5
6
7
8
9

10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
#
# DragonFly specific

delete_jail() {
	test -z ${JAILNAME} && usage
	jail_exists ${JAILNAME} || err 1 "No such jail: ${JAILNAME}"
	jail_runs ${JAILNAME} &&
		err 1 "Unable to delete jail ${JAILNAME}: it is running"

	msg_n "Removing ${JAILNAME} jail..."
	destroyfs ${JAILMNT} jail
	rm -rf ${POUDRIERED}/jails/${JAILNAME} || :
	echo "done"
}

update_version() {
	local release="$1"
	local login_env osversion

	osversion=`awk '/\#define __DragonFly_version/ { print $3 }' ${JAILMNT}/usr/include/sys/param.h`
	login_env=",UNAME_r=${release},UNAME_v=DragonFly ${release},OSVERSION=${osversion}"

	sed -i "" -e "s/,UNAME_r.*:/:/ ; s/:\(setenv.*\):/:\1${login_env}:/" ${JAILMNT}/etc/login.conf
	cap_mkdb ${JAILMNT}/etc/login.conf
}

establish_jail() {
# REMOVE THIS.
	[ $# -ne 5 ] && eargs name version arch mountpoint filesys
	local name=$1

Changes to src/share/poudriere/options.sh.

34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
    [ports...]  -- List of ports to set options on

Options:
    -c          -- Use 'make config' target
    -C          -- Use 'make config-conditional' target (default)
    -j name     -- Run on the given jail
    -p tree     -- Specify on which ports tree the configuration will be done
    -n          -- Don't configure/show/remove options of dependicies
    -r          -- Remove port options instead of configuring them
    -s          -- Show port options instead of configuring them
    -z set      -- Specify which SET to use"

	exit 1
}








|







34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
    [ports...]  -- List of ports to set options on

Options:
    -c          -- Use 'make config' target
    -C          -- Use 'make config-conditional' target (default)
    -j name     -- Run on the given jail
    -p tree     -- Specify on which ports tree the configuration will be done
    -n          -- Don't configure/show/remove options of dependencies
    -r          -- Remove port options instead of configuring them
    -s          -- Show port options instead of configuring them
    -z set      -- Specify which SET to use"

	exit 1
}

Changes to src/share/poudriere/queue.sh.

1
2
3
4
5
6
7
8
9
10
#!/bin/sh
# 
# Copyright (c) 2012 jadawin@tuxaco.net
# Copyright (c) 2012-2013 Baptiste Daroussin <bapt@FreeBSD.org>
# All rights reserved.
# 
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright


|







1
2
3
4
5
6
7
8
9
10
#!/bin/sh
# 
# Copyright (c) 2012 Philippe Audéoud <jadawin@FreeBSD.org>
# Copyright (c) 2012-2013 Baptiste Daroussin <bapt@FreeBSD.org>
# All rights reserved.
# 
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright

Added src/share/poudriere/status.sh.









































































































































































































>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
#!/bin/sh
# 
# Copyright (c) 2013 Bryan Drewery <bdrewery@FreeBSD.org>
# All rights reserved.
# 
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
#    notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
# 
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.

usage() {
	echo "poudriere status [options]

Options:
    -j name     -- Run on the given jail
    -p tree     -- Specify on which ports tree the configuration will be done
    -z set      -- Specify which SET to use"

	exit 1
}

SCRIPTPATH=`realpath $0`
SCRIPTPREFIX=`dirname ${SCRIPTPATH}`

PTNAME=default
SETNAME=""

. ${SCRIPTPREFIX}/common.sh

while getopts "j:p:z:" FLAG; do
	case "${FLAG}" in
		j)
			jail_exists ${OPTARG} || err 1 "No such jail"
			JAILNAME=${OPTARG}
			;;
		p)
			PTNAME=${OPTARG}
			;;
		z)
			[ -n "${OPTARG}" ] || err 1 "Empty set name"
			SETNAME="${OPTARG}"
			;;
		*)
			usage
			;;
	esac
done

shift $((OPTIND-1))

if [ ${POUDRIERE_DATA}/build/*/ref = "${POUDRIERE_DATA}/build/*/ref" ]; then
	msg "No running builds"
	exit 0
fi

BUILDNAME=latest
POUDRIERE_BUILD_TYPE=bulk

if [ -n "${JAILNAME}" ]; then
	mastername=${JAILNAME}-${PTNAME}${SETNAME:+-${SETNAME}}
	mastermnt=${POUDRIERE_DATA}/build/${mastername}/ref
	jail_runs ${mastername} || err 1 "No such jail running"
	builders="$(MASTERNAME=$mastername bget builders 2>/dev/null || :)"
	MASTERNAME=$mastername MASTERMNT=$mastermnt \
		JOBS="${builders}" siginfo_handler
else
	format="%-20s %-25s %6s %5s %6s %7s %7s %s\n"
	printf "${format}" "JAIL" "STATUS" "QUEUED" "BUILT" "FAILED" "SKIPPED" \
		"IGNORED"
	for mastermnt in ${POUDRIERE_DATA}/build/*/ref; do
		[ "${mastermnt}" = "${POUDRIERE_DATA}/build/*/ref" ] && break
		mastername=${mastermnt#${POUDRIERE_DATA}/build/}
		mastername=${mastername%/ref}

		status=$(MASTERNAME=$mastername bget status 2>/dev/null || :)
		nbqueued=$(MASTERNAME=$mastername bget stats_queued 2>/dev/null || :)
		nbfailed=$(MASTERNAME=$mastername bget stats_failed 2>/dev/null || :)
		nbignored=$(MASTERNAME=$mastername bget stats_ignored 2>/dev/null || :)
		nbskipped=$(MASTERNAME=$mastername bget stats_skipped 2>/dev/null || :)
		nbbuilt=$(MASTERNAME=$mastername bget stats_built 2>/dev/null || :)
		printf "${format}" "${mastername}" "${status}" "${nbqueued}" \
			"${nbbuilt}" "${nbfailed}" "${nbskipped}" "${nbignored}"
	done
fi

Changes to src/share/poudriere/testport.sh.

39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
...
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
...
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
...
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
...
196
197
198
199
200
201
202




203
204
205
206

207
208
209
210
211
212
213
...
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
    -j name     -- Run inside the given jail
    -i          -- Interactive mode. Enter jail for interactive testing and automatically cleanup when done.
    -I          -- Advanced Interactive mode. Leaves jail running with port installed after test.
    -n          -- No custom prefix
    -p tree     -- Specify the path to the portstree
    -s          -- Skip sanity checks
    -v          -- Be verbose; show more information. Use twice to enable debug output
    -z          -- Define set name"
	exit 1
}

SCRIPTPATH=`realpath $0`
SCRIPTPREFIX=`dirname ${SCRIPTPATH}`
CONFIGSTR=0
. ${SCRIPTPREFIX}/common.sh
................................................................................
jail_start ${JAILNAME} ${PTNAME} ${SETNAME}

[ $CONFIGSTR -eq 1 ] && injail env TERM=${SAVED_TERM} make -C ${PORTSRC}/${ORIGIN} config

LISTPORTS=$(list_deps ${ORIGIN} )
prepare_ports

markfs prepkg ${MASTERMNT}
log=$(log_path)

run_hook test_port_start "${JAILNAME}" "${PTNAME}" "${MASTERMNT}${PORTSRC}/${ORIGIN}" \
	`bget stats_queued`

if ! POUDRIERE_BUILD_TYPE=bulk parallel_build ${JAILNAME} ${PTNAME} ${SETNAME} ; then
	failed=$(bget ports.failed | awk '{print $1 ":" $3 }' | xargs echo)
................................................................................
	msg "Depends failed to build"
	msg "Failed ports: ${failed}"
	[ -n "${skipped}" ] && 	msg "Skipped ports: ${skipped}"

	exit 1
fi

unmarkfs prepkg ${MASTERMNT}

bset status "testing:"

PKGNAME=`injail make -C ${PORTSRC}/${ORIGIN} -VPKGNAME`
LOCALBASE=`injail make -C ${PORTSRC}/${ORIGIN} -VLOCALBASE`
: ${PREFIX:=$(injail make -C ${PORTSRC}/${ORIGIN} -VPREFIX)}
if [ "${USE_PORTLINT}" = "yes" ]; then
................................................................................
	set +e
	cd ${MASTERMNT}${PORTSRC}/${ORIGIN} &&
		PORTSDIR="${MASTERMNT}${PORTSRC}" portlint -C | \
		tee ${log}/logs/${PKGNAME}.portlint.log
	set -e
fi
[ ${NOPREFIX} -ne 1 ] && PREFIX="${BUILDROOT:-/prefix}/`echo ${PKGNAME} | tr '[,+]' _`"
PORT_FLAGS="NO_DEPENDS=yes PREFIX=${PREFIX}"
msg "Building with flags: ${PORT_FLAGS}"

if [ -d ${MASTERMNT}${PREFIX} ]; then
	msg "Removing existing ${PREFIX}"
	[ "${PREFIX}" != "${LOCALBASE}" ] && rm -rf ${MASTERMNT}${PREFIX}
fi

PKGENV="PACKAGES=/tmp/pkgs PKGREPOSITORY=/tmp/pkgs"
mkdir -p ${MASTERMNT}/tmp/pkgs
PORTTESTING=yes
export DEVELOPER_MODE=yes
log_start ${log}/logs/${PKGNAME}.log
buildlog_start ${PORTSRC}/${ORIGIN}
if ! build_port ${PORTSRC}/${ORIGIN}; then
	failed_status=$(bget status)
	failed_phase=${failed_status%:*}

	save_wrkdir ${MASTERMNT} "${PKGNAME}" "${PORTSRC}/${ORIGIN}" "${failed_phase}" || :
	build_result=0
	run_hook port_build_failure "${JAILNAME}" "${PTNAME}" "${MASTERMNT}${PORTSRC}/${ORIGIN}" "${failed_phase}"

	if [ ${INTERACTIVE_MODE} -eq 0 ]; then
		stop_build ${PORTSRC}/${ORIGIN} ${log}/logs/${PKGNAME}.log
		exit 1
	fi
else
	if [ -f ${MASTERMNT}/${PORTSRC}/${ORIGIN}/.keep ]; then
		save_wrkdir ${MASTERMNT} "${PKGNAME}" "${PORTSRC}/${ORIGIN}" "noneed" ||:
	fi
	build_result=1
................................................................................

msg "Installing from package"
injail ${PKG_ADD} /tmp/pkgs/${PKGNAME}.${PKG_EXT}

# Interactive test mode
if [ $INTERACTIVE_MODE -gt 0 ]; then
	print_phase_header "Interactive"





	msg "Installing run-depends"
	# Install run-depends since this is an interactive test
	echo "PACKAGES=/packages" >> ${MASTERMNT}/etc/make.conf

	injail make -C ${PORTSRC}/${ORIGIN} run-depends ||
		msg "Failed to install RUN_DEPENDS"

	if ${BSDPLATFORM} = "freebsd"; then
		# Enable networking
		jstop
		jstart 1
................................................................................
msg "Cleaning up"
injail make -C ${PORTSRC}/${ORIGIN} clean

msg "Deinstalling package"
injail ${PKG_DELETE} ${PKGNAME}

msg "Removing existing ${PREFIX} dir"
stop_build ${PORTSRC}/${ORIGIN} ${log}/logs/${PKGNAME}.log

cleanup
set +e
run_hook test_port_ended "${JAILNAME}" "${PTNAME}" "${MASTERMNT}${PORTSRC}/${ORIGIN}" \
	`bget stats_built` `bget stats_failed` `bget stats_ignored` \
	`bget stats_skipped` "${build_result}"

exit 0







|







 







<







 







<







 







|











|










|







 







>
>
>
>




>







 







|








39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
...
117
118
119
120
121
122
123

124
125
126
127
128
129
130
...
137
138
139
140
141
142
143

144
145
146
147
148
149
150
...
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
...
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
...
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
    -j name     -- Run inside the given jail
    -i          -- Interactive mode. Enter jail for interactive testing and automatically cleanup when done.
    -I          -- Advanced Interactive mode. Leaves jail running with port installed after test.
    -n          -- No custom prefix
    -p tree     -- Specify the path to the portstree
    -s          -- Skip sanity checks
    -v          -- Be verbose; show more information. Use twice to enable debug output
    -z set      -- Specify which SET to use"
	exit 1
}

SCRIPTPATH=`realpath $0`
SCRIPTPREFIX=`dirname ${SCRIPTPATH}`
CONFIGSTR=0
. ${SCRIPTPREFIX}/common.sh
................................................................................
jail_start ${JAILNAME} ${PTNAME} ${SETNAME}

[ $CONFIGSTR -eq 1 ] && injail env TERM=${SAVED_TERM} make -C ${PORTSRC}/${ORIGIN} config

LISTPORTS=$(list_deps ${ORIGIN} )
prepare_ports


log=$(log_path)

run_hook test_port_start "${JAILNAME}" "${PTNAME}" "${MASTERMNT}${PORTSRC}/${ORIGIN}" \
	`bget stats_queued`

if ! POUDRIERE_BUILD_TYPE=bulk parallel_build ${JAILNAME} ${PTNAME} ${SETNAME} ; then
	failed=$(bget ports.failed | awk '{print $1 ":" $3 }' | xargs echo)
................................................................................
	msg "Depends failed to build"
	msg "Failed ports: ${failed}"
	[ -n "${skipped}" ] && 	msg "Skipped ports: ${skipped}"

	exit 1
fi



bset status "testing:"

PKGNAME=`injail make -C ${PORTSRC}/${ORIGIN} -VPKGNAME`
LOCALBASE=`injail make -C ${PORTSRC}/${ORIGIN} -VLOCALBASE`
: ${PREFIX:=$(injail make -C ${PORTSRC}/${ORIGIN} -VPREFIX)}
if [ "${USE_PORTLINT}" = "yes" ]; then
................................................................................
	set +e
	cd ${MASTERMNT}${PORTSRC}/${ORIGIN} &&
		PORTSDIR="${MASTERMNT}${PORTSRC}" portlint -C | \
		tee ${log}/logs/${PKGNAME}.portlint.log
	set -e
fi
[ ${NOPREFIX} -ne 1 ] && PREFIX="${BUILDROOT:-/prefix}/`echo ${PKGNAME} | tr '[,+]' _`"
PORT_FLAGS="PREFIX=${PREFIX}"
msg "Building with flags: ${PORT_FLAGS}"

if [ -d ${MASTERMNT}${PREFIX} ]; then
	msg "Removing existing ${PREFIX}"
	[ "${PREFIX}" != "${LOCALBASE}" ] && rm -rf ${MASTERMNT}${PREFIX}
fi

PKGENV="PACKAGES=/tmp/pkgs PKGREPOSITORY=/tmp/pkgs"
mkdir -p ${MASTERMNT}/tmp/pkgs
PORTTESTING=yes
export DEVELOPER_MODE=yes
log_start
buildlog_start ${PORTSRC}/${ORIGIN}
if ! build_port ${PORTSRC}/${ORIGIN}; then
	failed_status=$(bget status)
	failed_phase=${failed_status%:*}

	save_wrkdir ${MASTERMNT} "${PKGNAME}" "${PORTSRC}/${ORIGIN}" "${failed_phase}" || :
	build_result=0
	run_hook port_build_failure "${JAILNAME}" "${PTNAME}" "${MASTERMNT}${PORTSRC}/${ORIGIN}" "${failed_phase}"

	if [ ${INTERACTIVE_MODE} -eq 0 ]; then
		stop_build ${PORTSRC}/${ORIGIN}
		exit 1
	fi
else
	if [ -f ${MASTERMNT}/${PORTSRC}/${ORIGIN}/.keep ]; then
		save_wrkdir ${MASTERMNT} "${PKGNAME}" "${PORTSRC}/${ORIGIN}" "noneed" ||:
	fi
	build_result=1
................................................................................

msg "Installing from package"
injail ${PKG_ADD} /tmp/pkgs/${PKGNAME}.${PKG_EXT}

# Interactive test mode
if [ $INTERACTIVE_MODE -gt 0 ]; then
	print_phase_header "Interactive"

	# Stop the tee process and stop redirecting stdout so that
	# the terminal can be properly used in the jail
	log_stop

	msg "Installing run-depends"
	# Install run-depends since this is an interactive test
	echo "PACKAGES=/packages" >> ${MASTERMNT}/etc/make.conf
	echo "127.0.0.1 ${MASTERNAME}" >> ${MASTERMNT}/etc/hosts
	injail make -C ${PORTSRC}/${ORIGIN} run-depends ||
		msg "Failed to install RUN_DEPENDS"

	if ${BSDPLATFORM} = "freebsd"; then
		# Enable networking
		jstop
		jstart 1
................................................................................
msg "Cleaning up"
injail make -C ${PORTSRC}/${ORIGIN} clean

msg "Deinstalling package"
injail ${PKG_DELETE} ${PKGNAME}

msg "Removing existing ${PREFIX} dir"
stop_build ${PORTSRC}/${ORIGIN}

cleanup
set +e
run_hook test_port_ended "${JAILNAME}" "${PTNAME}" "${MASTERMNT}${PORTSRC}/${ORIGIN}" \
	`bget stats_built` `bget stats_failed` `bget stats_ignored` \
	`bget stats_skipped` "${build_result}"

exit 0

Changes to zsh-completions.

85
86
87
88
89
90
91






92
93
94
95
96
97
98
...
139
140
141
142
143
144
145



146
147
148
149
150
151
	'-c[run make config for the given port]::'
	'-j[run only inside the given jail]::name:_poudriere_jail'
	'-n[no custom prefix]::'
	'-p[specify on which portstree we work]::tree:_poudriere_pt'
	'-v[Be verbose; show more information. Specify twice to enable debug output.]'
	'-z[Specify on which SET to use]::'
)







_ports=(
	'(-d -u -l)-c[create a portstree]'
	'(-c -u -l)-d[delete a portstree]'
	'(-c -d -l)-u[update a portstree]'
	'(-c -d -u)-l[lists all available portstrees]'
	'-q[quiet (remove the header in list)]'
................................................................................
			;;
			ports)
			_arguments : $_ports "::" && return
			;;
			testport)
			_arguments : $_testport "::" && return
			;;



		esac
	fi

}

_poudriere "$@"







>
>
>
>
>
>







 







>
>
>






85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
...
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
	'-c[run make config for the given port]::'
	'-j[run only inside the given jail]::name:_poudriere_jail'
	'-n[no custom prefix]::'
	'-p[specify on which portstree we work]::tree:_poudriere_pt'
	'-v[Be verbose; show more information. Specify twice to enable debug output.]'
	'-z[Specify on which SET to use]::'
)

_status=(
	'-j[run only inside the given jail]::name:_poudriere_jail'
	'-p[specify on which portstree we work]::tree:_poudriere_pt'
	'-z[Specify on which SET to use]::'
)

_ports=(
	'(-d -u -l)-c[create a portstree]'
	'(-c -u -l)-d[delete a portstree]'
	'(-c -d -l)-u[update a portstree]'
	'(-c -d -u)-l[lists all available portstrees]'
	'-q[quiet (remove the header in list)]'
................................................................................
			;;
			ports)
			_arguments : $_ports "::" && return
			;;
			testport)
			_arguments : $_testport "::" && return
			;;
			status)
			_arguments : $_status "::" && return
			;;
		esac
	fi

}

_poudriere "$@"