sched/numa: Migrate pages to local nodes quicker early in the lifetime of a task
Automatic NUMA Balancing uses a multi-stage pass to decide whether a page should migrate to a local node. This filter avoids excessive ping-ponging if a page is shared or used by threads that migrate cross-node frequently. Threads inherit both page tables and the preferred node ID from the parent. This means that threads can trigger hinting faults earlier than a new task which delays scanning for a number of seconds. As it can be load balanced very early in its lifetime there can be an unnecessary delay before it starts migrating thread-local data. This patch migrates private pages faster early in the lifetime of a thread using the sequence counter as an identifier of new tasks. With this patch applied, STREAM performance is the same as 4.17 even though processes are not spread cross-node prematurely. Other workloads showed a mix of minor gains and losses. This is somewhat expected most workloads are not very sensitive to the starting conditions of a process. 4.19.0-rc5 4.19.0-rc5 4.17.0 numab-v1r1 fastmigrate-v1r1 vanilla MB/sec copy 43298.52 ( 0.00%) 47335.46 ( 9.32%) 47219.24 ( 9.06%) MB/sec scale 30115.06 ( 0.00%) 32568.12 ( 8.15%) 32527.56 ( 8.01%) MB/sec add 32825.12 ( 0.00%) 36078.94 ( 9.91%) 35928.02 ( 9.45%) MB/sec triad 32549.52 ( 0.00%) 35935.94 ( 10.40%) 35969.88 ( 10.51%) Signed-off-by: Mel Gorman <mgorman@techsingularity.net> Reviewed-by: Rik van Riel <riel@surriel.com> Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Jirka Hladky <jhladky@redhat.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Linux-MM <linux-mm@kvack.org> Cc: Srikar Dronamraju <srikar@linux.vnet.ibm.com> Cc: Thomas Gleixner <tglx@linutronix.de> Link: http://lkml.kernel.org/r/20181001100525.29789-3-mgorman@techsingularity.net Signed-off-by: Ingo Molnar <mingo@kernel.org>
This commit is contained in:
parent
efaffc5e40
commit
37355bdc5a
|
@ -1392,6 +1392,17 @@ bool should_numa_migrate_memory(struct task_struct *p, struct page * page,
|
||||||
int last_cpupid, this_cpupid;
|
int last_cpupid, this_cpupid;
|
||||||
|
|
||||||
this_cpupid = cpu_pid_to_cpupid(dst_cpu, current->pid);
|
this_cpupid = cpu_pid_to_cpupid(dst_cpu, current->pid);
|
||||||
|
last_cpupid = page_cpupid_xchg_last(page, this_cpupid);
|
||||||
|
|
||||||
|
/*
|
||||||
|
* Allow first faults or private faults to migrate immediately early in
|
||||||
|
* the lifetime of a task. The magic number 4 is based on waiting for
|
||||||
|
* two full passes of the "multi-stage node selection" test that is
|
||||||
|
* executed below.
|
||||||
|
*/
|
||||||
|
if ((p->numa_preferred_nid == -1 || p->numa_scan_seq <= 4) &&
|
||||||
|
(cpupid_pid_unset(last_cpupid) || cpupid_match_pid(p, last_cpupid)))
|
||||||
|
return true;
|
||||||
|
|
||||||
/*
|
/*
|
||||||
* Multi-stage node selection is used in conjunction with a periodic
|
* Multi-stage node selection is used in conjunction with a periodic
|
||||||
|
@ -1410,7 +1421,6 @@ bool should_numa_migrate_memory(struct task_struct *p, struct page * page,
|
||||||
* This quadric squishes small probabilities, making it less likely we
|
* This quadric squishes small probabilities, making it less likely we
|
||||||
* act on an unlikely task<->page relation.
|
* act on an unlikely task<->page relation.
|
||||||
*/
|
*/
|
||||||
last_cpupid = page_cpupid_xchg_last(page, this_cpupid);
|
|
||||||
if (!cpupid_pid_unset(last_cpupid) &&
|
if (!cpupid_pid_unset(last_cpupid) &&
|
||||||
cpupid_to_nid(last_cpupid) != dst_nid)
|
cpupid_to_nid(last_cpupid) != dst_nid)
|
||||||
return false;
|
return false;
|
||||||
|
|
Loading…
Reference in New Issue