Testing FORCE MAKE JOBS on FreeBSD 8.0: Difference between revisions

From TykWiki
Jump to navigationJump to search
No edit summary
No edit summary
 
(7 intermediate revisions by the same user not shown)
Line 1: Line 1:
A whole lot of ports installed:
{{DISPLAYTITLE:Testing FORCE_MAKE_JOBS on FreeBSD 8.0}}
This is a test of the difference in port building speed with and without <code>FORCE_MAKE_JOBS</code> in <code>/etc/make.conf</code>.
 
== Introduction ==
Parallel ports building in FreeBSD was introduced in FreeBSD 8.0, quoting from [[http://lists.freebsd.org/pipermail/freebsd-ports/2009-March/053736.html the initial mailing list post]]:
<pre>
Two days ago, I have checked in probably most requested feature of last
few years. Ports framework now systematically supports building ports on
multiple processing cores. It is achieved by passing -jX flag to make(1)
running on vendor code. Of course not all ports handle this well,
experimental run on pointyhat with this flag globally enabled turned up
shy of 400 failures. Because of that, the feature was designed as a
whitelist. Individual ports need to be enabled, and indeed, fellow
developers took on and already started adding required declarations to
popular ports like Firefox and others.
 
If you are FreeBSD ports user:
 
You don't need to do anything to enable the new feature. Whitelisted
ports will automatically make use of all processors available in your
computer. If you want, for some reasons, to disable this feature, put
DISABLE_MAKE_JOBS=yes to your /etc/make.conf. By default, the level of
parallelization will be equal to a number of processing cores in your
machine. If you want to override this number, use for example
MAKE_JOBS_NUMBER=6, again in /etc/make.conf. And if you are extra brave,
or you want to check out all the yet unmarked ports, if they will build,
you can define FORCE_MAKE_JOBS=yes in /etc/make.conf.
</pre>
 
== Test setup ==
There is a whole lot of ports installed on this machine (X + applications like Firefox and OpenOffice):
<pre>
<pre>
[tykling@tyklappy ~]$ pkg_info | wc -l
[tykling@tyklappy ~]$ pkg_info | wc -l
Line 5: Line 35:
</pre>
</pre>


The machine is a Lenovo X301 with a solid-state disk and the following CPU and RAM:
<pre>
[tykling@tyklappy ~]$ grep "l memory" /var/run/dmesg.boot
real memory  = 6442450944 (6144 MB)
avail memory = 6066290688 (5785 MB)
</pre>
<pre>
[tykling@tyklappy ~]$ grep CPU: /var/run/dmesg.boot
CPU: Intel(R) Core(TM)2 Duo CPU    U9600  @ 1.60GHz (1602.39-MHz K8-class CPU)
</pre>
The machine is configured with ZFS and is running great with X, everything seems to work. I've posted a [[Dmesg_-_Lenovo_x301_FreeBSD_8.0-release_amd64|dmesg here]].
== Testing ==
This is the command I run to rebuild all ports (except for portmaster):
This is the command I run to rebuild all ports (except for portmaster):
<pre>
<pre>
Line 10: Line 55:
</pre>
</pre>


== Test 1 ==
=== Test 1 ===
First test is a build with the default settings.
First test is a build with the default settings.


Line 27: Line 72:
</pre>
</pre>


== Test 2 ==
=== Test 2 ===
Second test is with <code>FORCE_MAKE_JOBS</code> enabled - <code>MAKE_JOBS_NUMBER</code> is set to 2 in this test, which would have been the default if I had not set <code>MAKE_JOBS_NUMBER</code> since there is two cores in the machine I am running this on.
Second test is with <code>FORCE_MAKE_JOBS</code> enabled - <code>MAKE_JOBS_NUMBER</code> is set to 2 in this test, which would have been the default if I had not set <code>MAKE_JOBS_NUMBER</code> since there is two cores in the machine I am running this on.


Line 39: Line 84:
</pre>
</pre>


Resultat:
<pre>
<pre>
real    343m5.886s
real    343m5.886s
Line 46: Line 93:




== Test 3 ==
=== Test 3 ===
Third test is with <code>FORCE_MAKE_JOBS</code> enabled - <code>MAKE_JOBS_NUMBER</code> is set to 4 in this test, double the number of cores in the machine.
Third test is with <code>FORCE_MAKE_JOBS</code> enabled - <code>MAKE_JOBS_NUMBER</code> is set to 4 in this test, double the number of cores in the machine.


Line 58: Line 105:
</pre>
</pre>


Resultat:
<pre>
<pre>
real    339m
real    339m
</pre>
</pre>
(from memory, I don't have the rest of it)
(From memory, I don't have the rest of it)
 
== Conclusion ==
From 378 minutes to 343 minutes, or 35 minutes saved. Not bad.

Latest revision as of 12:59, 17 April 2010

This is a test of the difference in port building speed with and without FORCE_MAKE_JOBS in /etc/make.conf.

Introduction

Parallel ports building in FreeBSD was introduced in FreeBSD 8.0, quoting from [the initial mailing list post]: 

Two days ago, I have checked in probably most requested feature of last
few years. Ports framework now systematically supports building ports on
multiple processing cores. It is achieved by passing -jX flag to make(1)
running on vendor code. Of course not all ports handle this well,
experimental run on pointyhat with this flag globally enabled turned up
shy of 400 failures. Because of that, the feature was designed as a
whitelist. Individual ports need to be enabled, and indeed, fellow
developers took on and already started adding required declarations to
popular ports like Firefox and others.

If you are FreeBSD ports user:

You don't need to do anything to enable the new feature. Whitelisted
ports will automatically make use of all processors available in your
computer. If you want, for some reasons, to disable this feature, put
DISABLE_MAKE_JOBS=yes to your /etc/make.conf. By default, the level of
parallelization will be equal to a number of processing cores in your
machine. If you want to override this number, use for example
MAKE_JOBS_NUMBER=6, again in /etc/make.conf. And if you are extra brave,
or you want to check out all the yet unmarked ports, if they will build,
you can define FORCE_MAKE_JOBS=yes in /etc/make.conf.

Test setup

There is a whole lot of ports installed on this machine (X + applications like Firefox and OpenOffice): 

[tykling@tyklappy ~]$ pkg_info | wc -l
    440

The machine is a Lenovo X301 with a solid-state disk and the following CPU and RAM: 

[tykling@tyklappy ~]$ grep "l memory" /var/run/dmesg.boot 
real memory  = 6442450944 (6144 MB)
avail memory = 6066290688 (5785 MB)

[tykling@tyklappy ~]$ grep CPU: /var/run/dmesg.boot 
CPU: Intel(R) Core(TM)2 Duo CPU     U9600  @ 1.60GHz (1602.39-MHz K8-class CPU)

The machine is configured with ZFS and is running great with X, everything seems to work. I've posted a dmesg here.

Testing

This is the command I run to rebuild all ports (except for portmaster): 

portmaster --no-confirm -afB -x portmaster

Test 1

First test is a build with the default settings.

/etc/make.conf: 

# added by use.perl 2010-04-13 22:07:18
PERL_VERSION=5.8.9


Resultat: 

real    378m53.008s
user    403m40.086s
sys     99m14.101s

Test 2

Second test is with FORCE_MAKE_JOBS enabled - MAKE_JOBS_NUMBER is set to 2 in this test, which would have been the default if I had not set MAKE_JOBS_NUMBER since there is two cores in the machine I am running this on.

/etc/make.conf: 

FORCE_MAKE_JOBS=YES
MAKE_JOBS_NUMBER=2

# added by use.perl 2010-04-17 03:07:25
PERL_VERSION=5.8.9


Resultat: 

real    343m5.886s
user    405m43.557s
sys     98m56.720s


Test 3

Third test is with FORCE_MAKE_JOBS enabled - MAKE_JOBS_NUMBER is set to 4 in this test, double the number of cores in the machine.

/etc/make.conf: 

FORCE_MAKE_JOBS=YES
MAKE_JOBS_NUMBER=4

# added by use.perl 2010-04-17 03:07:25
PERL_VERSION=5.8.9


Resultat: 

real    339m

(From memory, I don't have the rest of it)

Conclusion

From 378 minutes to 343 minutes, or 35 minutes saved. Not bad.

Retrieved from "https://tyk.wiki/index.php?title=Testing_FORCE_MAKE_JOBS_on_FreeBSD_8.0&oldid=273"