linux/Documentation
akpm@osdl.org 198e2f1811 [PATCH] scheduler cache-hot-autodetect
)

From: Ingo Molnar <mingo@elte.hu>

This is the latest version of the scheduler cache-hot-auto-tune patch.

The first problem was that detection time scaled with O(N^2), which is
unacceptable on larger SMP and NUMA systems. To solve this:

- I've added a 'domain distance' function, which is used to cache
  measurement results. Each distance is only measured once. This means
  that e.g. on NUMA distances of 0, 1 and 2 might be measured, on HT
  distances 0 and 1, and on SMP distance 0 is measured. The code walks
  the domain tree to determine the distance, so it automatically follows
  whatever hierarchy an architecture sets up. This cuts down on the boot
  time significantly and removes the O(N^2) limit. The only assumption
  is that migration costs can be expressed as a function of domain
  distance - this covers the overwhelming majority of existing systems,
  and is a good guess even for more assymetric systems.

  [ People hacking systems that have assymetries that break this
    assumption (e.g. different CPU speeds) should experiment a bit with
    the cpu_distance() function. Adding a ->migration_distance factor to
    the domain structure would be one possible solution - but lets first
    see the problem systems, if they exist at all. Lets not overdesign. ]

Another problem was that only a single cache-size was used for measuring
the cost of migration, and most architectures didnt set that variable
up. Furthermore, a single cache-size does not fit NUMA hierarchies with
L3 caches and does not fit HT setups, where different CPUs will often
have different 'effective cache sizes'. To solve this problem:

- Instead of relying on a single cache-size provided by the platform and
  sticking to it, the code now auto-detects the 'effective migration
  cost' between two measured CPUs, via iterating through a wide range of
  cachesizes. The code searches for the maximum migration cost, which
  occurs when the working set of the test-workload falls just below the
  'effective cache size'. I.e. real-life optimized search is done for
  the maximum migration cost, between two real CPUs.

  This, amongst other things, has the positive effect hat if e.g. two
  CPUs share a L2/L3 cache, a different (and accurate) migration cost
  will be found than between two CPUs on the same system that dont share
  any caches.

(The reliable measurement of migration costs is tricky - see the source
for details.)

Furthermore i've added various boot-time options to override/tune
migration behavior.

Firstly, there's a blanket override for autodetection:

	migration_cost=1000,2000,3000

will override the depth 0/1/2 values with 1msec/2msec/3msec values.

Secondly, there's a global factor that can be used to increase (or
decrease) the autodetected values:

	migration_factor=120

will increase the autodetected values by 20%. This option is useful to
tune things in a workload-dependent way - e.g. if a workload is
cache-insensitive then CPU utilization can be maximized by specifying
migration_factor=0.

I've tested the autodetection code quite extensively on x86, on 3
P3/Xeon/2MB, and the autodetected values look pretty good:

Dual Celeron (128K L2 cache):

 ---------------------
 migration cost matrix (max_cache_size: 131072, cpu: 467 MHz):
 ---------------------
           [00]    [01]
 [00]:     -     1.7(1)
 [01]:   1.7(1)    -
 ---------------------
 cacheflush times [2]: 0.0 (0) 1.7 (1784008)
 ---------------------

Here the slow memory subsystem dominates system performance, and even
though caches are small, the migration cost is 1.7 msecs.

Dual HT P4 (512K L2 cache):

 ---------------------
 migration cost matrix (max_cache_size: 524288, cpu: 2379 MHz):
 ---------------------
           [00]    [01]    [02]    [03]
 [00]:     -     0.4(1)  0.0(0)  0.4(1)
 [01]:   0.4(1)    -     0.4(1)  0.0(0)
 [02]:   0.0(0)  0.4(1)    -     0.4(1)
 [03]:   0.4(1)  0.0(0)  0.4(1)    -
 ---------------------
 cacheflush times [2]: 0.0 (33900) 0.4 (448514)
 ---------------------

Here it can be seen that there is no migration cost between two HT
siblings (CPU#0/2 and CPU#1/3 are separate physical CPUs). A fast memory
system makes inter-physical-CPU migration pretty cheap: 0.4 msecs.

8-way P3/Xeon [2MB L2 cache]:

 ---------------------
 migration cost matrix (max_cache_size: 2097152, cpu: 700 MHz):
 ---------------------
           [00]    [01]    [02]    [03]    [04]    [05]    [06]    [07]
 [00]:     -    19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
 [01]:  19.2(1)    -    19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
 [02]:  19.2(1) 19.2(1)    -    19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
 [03]:  19.2(1) 19.2(1) 19.2(1)    -    19.2(1) 19.2(1) 19.2(1) 19.2(1)
 [04]:  19.2(1) 19.2(1) 19.2(1) 19.2(1)    -    19.2(1) 19.2(1) 19.2(1)
 [05]:  19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)    -    19.2(1) 19.2(1)
 [06]:  19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)    -    19.2(1)
 [07]:  19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)    -
 ---------------------
 cacheflush times [2]: 0.0 (0) 19.2 (19281756)
 ---------------------

This one has huge caches and a relatively slow memory subsystem - so the
migration cost is 19 msecs.

Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Ashok Raj <ashok.raj@intel.com>
Signed-off-by: Ken Chen <kenneth.w.chen@intel.com>
Cc: <wilder@us.ibm.com>
Signed-off-by: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-01-12 09:08:50 -08:00
..
DocBook [PATCH] DocBook: add .gitignore file 2006-01-10 08:01:52 -08:00
RCU [PATCH] rcu file: use atomic primitives 2006-01-08 20:13:48 -08:00
aoe [PATCH] aoe [1/2]: support 16 AoE slot addresses per AoE shelf 2005-09-09 14:23:16 -07:00
arm [ARM] Add memory.txt to 00-INDEX 2005-12-04 17:56:46 +00:00
block [PATCH] block/stat.txt 2006-01-08 20:14:01 -08:00
cdrom [PATCH] remove verify_area(): remove or edit references to verify_area in Documentation/ 2005-09-07 16:57:35 -07:00
connector [CONNECTOR]: Fix documentation test module. 2005-11-08 09:38:56 -08:00
cpu-freq [PATCH] cpufreq: documentation for 'ondemand' and 'conservative' 2005-12-01 01:23:23 -08:00
cris
crypto [PATCH] Spelling fixes for Documentation/ 2005-09-10 10:06:28 -07:00
device-mapper [PATCH] Fix dm-snapshot tutorial in Documentation 2005-11-07 07:53:54 -08:00
driver-model [PATCH] Fix documentation of driver suspend/resume callbacks 2005-10-28 09:52:57 -07:00
dvb V4L/DVB (3336): Bluebird firmware hosting moved to common dvb firmware dir on linuxtv.org 2006-01-09 18:21:40 -02:00
early-userspace earlyuserspace/README: fix homonym err 2005-11-08 17:16:50 +01:00
fb [PATCH] Update cyblafb driver 2006-01-09 10:00:36 -08:00
filesystems [PATCH] ext3: fix documentation of online resizing 2006-01-11 18:42:10 -08:00
firmware_class [PATCH] fix missing includes 2005-10-30 17:37:32 -08:00
fujitsu/frv
hwmon [PATCH] hwmon: Clarify the W83627THF VID documentation 2006-01-05 22:16:20 -08:00
i2c [PATCH] i2c: i2c-nforce2 add nforce4 MCP-04 device ID 2006-01-05 22:16:27 -08:00
i2o spelling: s/retreive/retrieve/ 2006-01-10 00:10:13 +01:00
i386 [PATCH] Make the bzImage format self-terminating 2005-09-07 16:57:29 -07:00
ia64 [IA64] Add Documentation/ia64/mca.txt 2005-09-16 10:34:16 -07:00
infiniband [PATCH] IB: Add core locking documentation to Infiniband 2005-07-27 16:26:15 -07:00
input Input: appletouch - add support for Geyser 2 2005-12-21 00:50:23 -05:00
ioctl [PATCH] Spelling fixes for Documentation/ 2005-09-10 10:06:28 -07:00
isdn
kbuild kbuild: remove GCC_VERSION 2006-01-08 19:58:51 +01:00
kdump [PATCH] kdump: add dmesg gdbmacro into document 2006-01-11 18:42:10 -08:00
m68k [PATCH] mm: m68k kill stram swap 2005-10-29 21:40:38 -07:00
mips Cleaned up AMD Au1200 IDE driver: 2005-10-29 19:32:20 +01:00
networking spelling: s/usefull/useful/ 2006-01-10 00:08:17 +01:00
parisc
pcmcia [PATCH] pcmcia: unify attach, EVENT_CARD_INSERTION handlers into one probe callback 2006-01-06 00:03:24 +01:00
power spelling: s/usefull/useful/ 2006-01-10 00:08:17 +01:00
powerpc [PATCH] Documentation/powerpc: index update 2006-01-09 14:53:34 +11:00
s390 [PATCH] s390: "extern inline" -> "static inline" 2005-11-09 07:55:52 -08:00
scsi Merge git://git.kernel.org/pub/scm/linux/kernel/git/bunk/trivial 2006-01-04 16:34:57 -08:00
serial [SERIAL] Update serial_core documentation 2005-10-31 11:53:19 +00:00
sh
sound [ALSA] Add PCXHR driver 2006-01-03 12:30:26 +01:00
sparc
sysctl [PATCH] Make high and batch sizes of per_cpu_pagelists configurable 2006-01-08 20:12:40 -08:00
telephony
uml [PATCH] Spelling fixes for Documentation/ 2005-09-10 10:06:28 -07:00
usb [PATCH] USB: documentation update 2005-11-29 21:39:22 -08:00
video4linux V4L/DVB (3349): Add PCI ID for UltraView DVB-T Plus, rebranded DViCO FusionHDTV DVB-T Plus 2006-01-12 00:10:59 -02:00
vm [PATCH] perform maintenance on Documentation/vm/hugetlbpage.txt 2005-11-07 07:53:39 -08:00
w1
watchdog [PATCH] ppc32: Added support for the Book-E style Watchdog Timer 2005-09-05 00:05:56 -07:00
x86_64 [PATCH] x86_64: Add documentation for CPU hotplug ACPI extension 2006-01-11 19:01:11 -08:00
00-INDEX [PATCH] update Documentation/00-INDEX 2005-11-18 10:51:22 -08:00
BUG-HUNTING
Changes [PATCH] remove gcc-2 checks 2006-01-08 20:14:02 -08:00
CodingStyle [PATCH] CodingStyle correction 2006-01-10 08:01:52 -08:00
DMA-API.txt [PATCH] Spelling fixes for Documentation/ 2005-09-10 10:06:28 -07:00
DMA-ISA-LPC.txt [PATCH] ISA DMA API documentation 2005-09-09 14:03:42 -07:00
DMA-mapping.txt
HOWTO [PATCH] Add HOWTO do kernel development document to the Documentation directory 2005-11-18 10:51:21 -08:00
IO-mapping.txt
IPMI.txt [PATCH] ipmi poweroff: fix chassis control 2005-09-07 16:57:49 -07:00
IRQ-affinity.txt
MSI-HOWTO.txt [PATCH] Doc/MSI-HOWTO: cleanups 2005-11-07 07:53:55 -08:00
ManagementStyle
PCIEBUS-HOWTO.txt
README.DAC960
README.cycladesZ
SAK.txt
SecurityBugs
SubmittingDrivers [PATCH] docs: updated some code docs 2006-01-08 20:13:53 -08:00
SubmittingPatches [PATCH] SubmittingPatches: diffstat options 2006-01-08 20:14:04 -08:00
VGA-softcursor.txt
acpi-hotkey.txt Merge to-linus-stable into to-akpm 2005-08-15 15:52:00 -04:00
applying-patches.txt [PATCH] Docs update: typos, corrections and additions to applying-patches.txt 2006-01-10 08:01:54 -08:00
atomic_ops.txt [PATCH] atomic: inc_not_zero 2005-11-13 18:14:16 -08:00
basic_profiling.txt
binfmt_misc.txt
cachetlb.txt [PATCH] mm: flush_tlb_range outside ptlock 2005-10-29 21:40:40 -07:00
cciss.txt [SCSI] cciss: scsi error handling 2005-11-06 14:12:49 -06:00
cli-sti-removal.txt
computone.txt
cpqarray.txt
cpu-hotplug.txt [PATCH] Updated CPU hotplug documentation 2006-01-08 20:13:55 -08:00
cpusets.txt cpuset two little doc fixes 2006-01-11 02:01:38 +01:00
dcdbas.txt [PATCH] dcdbas: add Dell Systems Management Base Driver with sysfs support 2005-09-07 16:57:27 -07:00
debugging-modules.txt
dell_rbu.txt [PATCH] dell_rbu: changes in packet update mechanism 2005-10-11 09:46:53 -07:00
devices.txt [PATCH] USB: Delete leftovers from bluetty driver 2005-11-17 11:29:53 -08:00
digiepca.txt
dnotify.txt
dontdiff [PATCH] dontdiff: add asm_offsets 2005-09-13 08:22:29 -07:00
eisa.txt
exception.txt [PATCH] remove verify_area(): remove or edit references to verify_area in Documentation/ 2005-09-07 16:57:35 -07:00
feature-removal-schedule.txt Merge ../torvalds-2.6/ 2006-01-06 12:59:59 -08:00
floppy.txt [PATCH] kernel Doc/ URL corrections 2005-11-22 09:14:30 -08:00
ftape.txt
hayes-esp.txt
highuid.txt
hpet.txt [PATCH] Doc/hpet.txt: change to < 80 columns 2005-11-07 07:53:55 -08:00
hrtimers.txt [PATCH] hrtimer: hrtimer documentation 2006-01-10 08:01:37 -08:00
hw_random.txt
ibm-acpi.txt [ACPI] IBM ThinkPad ACPI Extras Driver v0.12 2005-08-25 17:37:51 -04:00
ide.txt
initrd.txt
io_ordering.txt
ioctl-number.txt [PATCH] kernel Doc/ URL corrections 2005-11-22 09:14:30 -08:00
iostats.txt
isapnp.txt
java.txt
kernel-doc-nano-HOWTO.txt
kernel-docs.txt [PATCH] kernel Doc/ URL corrections 2005-11-22 09:14:30 -08:00
kernel-parameters.txt [PATCH] scheduler cache-hot-autodetect 2006-01-12 09:08:50 -08:00
keys-request-key.txt [PATCH] keys: Permit running process to instantiate keys 2006-01-08 20:13:53 -08:00
keys.txt [PATCH] keys: Permit running process to instantiate keys 2006-01-08 20:13:53 -08:00
kobject.txt
kprobes.txt [PATCH] Fix more "if ((err = foo() < 0))" typos 2006-01-09 13:02:51 -08:00
kref.txt
laptop-mode.txt Tiny esthetic changes to Documentation/laptop-mode.txt 2006-01-11 01:48:08 +01:00
ldm.txt
locks.txt [PATCH] Docs update: remove obsolete patch from locks.txt 2006-01-10 08:01:54 -08:00
logo.gif
logo.txt
magic-number.txt [PATCH] mm: rename kmem_cache_s to kmem_cache 2005-11-07 07:53:24 -08:00
mandatory.txt
mca.txt [PATCH] kernel Doc/ URL corrections 2005-11-22 09:14:30 -08:00
md.txt [PATCH] md: allow sync-speed to be controlled per-device 2006-01-06 08:34:10 -08:00
memory.txt
mono.txt [PATCH] merge some from Rusty's trivial patches 2005-09-10 10:06:30 -07:00
moxa-smartio
mtrr.txt
mutex-design.txt [PATCH] mutex subsystem, documentation 2006-01-09 15:59:20 -08:00
nbd.txt
nfsroot.txt
nmi_watchdog.txt
nommu-mmap.txt
numastat.txt
oops-tracing.txt [PATCH] doc: refer to kdump in oops-tracing.txt 2006-01-11 18:42:10 -08:00
paride.txt
parport-lowlevel.txt
parport.txt
pci-error-recovery.txt [PATCH] PCI Error Recovery: documentation 2006-01-09 12:13:21 -08:00
pci.txt [PATCH] merge some from Rusty's trivial patches 2005-09-10 10:06:30 -07:00
pm.txt [ACPI] linux-acpi@vger.kernel.org replaces acpi-devel@lists.sourceforge.net 2006-01-06 16:19:26 -05:00
pnp.txt
preempt-locking.txt
prio_tree.txt
ramdisk.txt
riscom8.txt
rocket.txt
rpc-cache.txt
rtc.txt
sched-arch.txt [PATCH] sched: resched and cpu_idle rework 2005-11-09 07:56:33 -08:00
sched-coding.txt
sched-design.txt
sched-domains.txt
sched-stats.txt
seclvl.txt
serial-console.txt
sgi-ioc4.txt
sgi-visws.txt
sharedsubtree.txt [PATCH] Complete description of shared subtrees. 2005-11-07 18:18:11 -08:00
smart-config.txt
smp.txt
sonypi.txt [PATCH] sonypi SPIC initialisation fix 2005-09-07 16:57:24 -07:00
sparse.txt [PATCH] Documentation/sparse.txt: mention CF=-Wbitwise 2005-11-07 07:53:54 -08:00
specialix.txt
spinlocks.txt
stable_api_nonsense.txt [PATCH] stable_api_nonsense.txt fixes 2005-07-29 13:12:49 -07:00
stable_kernel_rules.txt [PATCH] Docs update: small fixes to stable_kernel_rules.txt 2006-01-10 08:01:55 -08:00
stallion.txt
svga.txt
sx.txt
sysrq.txt update for Documentation/sysrq.txt 2006-01-03 13:31:01 +01:00
time_interpolators.txt
tipar.txt
tty.txt
unicode.txt
voyager.txt
xterm-linux.xpm
zorro.txt

README.cycladesZ

The Cyclades-Z must have firmware loaded onto the card before it will
operate.  This operation should be performed during system startup,

The firmware, loader program and the latest device driver code are
available from Cyclades at
    ftp://ftp.cyclades.com/pub/cyclades/cyclades-z/linux/