[PATCH] sparsemem memory model
Sparsemem abstracts the use of discontiguous mem_maps[]. This kind of
mem_map[] is needed by discontiguous memory machines (like in the old
CONFIG_DISCONTIGMEM case) as well as memory hotplug systems. Sparsemem
replaces DISCONTIGMEM when enabled, and it is hoped that it can eventually
become a complete replacement.
A significant advantage over DISCONTIGMEM is that it's completely separated
from CONFIG_NUMA. When producing this patch, it became apparent in that NUMA
and DISCONTIG are often confused.
Another advantage is that sparse doesn't require each NUMA node's ranges to be
contiguous. It can handle overlapping ranges between nodes with no problems,
where DISCONTIGMEM currently throws away that memory.
Sparsemem uses an array to provide different pfn_to_page() translations for
each SECTION_SIZE area of physical memory. This is what allows the mem_map[]
to be chopped up.
In order to do quick pfn_to_page() operations, the section number of the page
is encoded in page->flags. Part of the sparsemem infrastructure enables
sharing of these bits more dynamically (at compile-time) between the
page_zone() and sparsemem operations. However, on 32-bit architectures, the
number of bits is quite limited, and may require growing the size of the
page->flags type in certain conditions. Several things might force this to
occur: a decrease in the SECTION_SIZE (if you want to hotplug smaller areas of
memory), an increase in the physical address space, or an increase in the
number of used page->flags.
One thing to note is that, once sparsemem is present, the NUMA node
information no longer needs to be stored in the page->flags. It might provide
speed increases on certain platforms and will be stored there if there is
room. But, if out of room, an alternate (theoretically slower) mechanism is
used.
This patch introduces CONFIG_FLATMEM. It is used in almost all cases where
there used to be an #ifndef DISCONTIG, because SPARSEMEM and DISCONTIGMEM
often have to compile out the same areas of code.
Signed-off-by: Andy Whitcroft <apw@shadowen.org>
Signed-off-by: Dave Hansen <haveblue@us.ibm.com>
Signed-off-by: Martin Bligh <mbligh@aracnet.com>
Signed-off-by: Adrian Bunk <bunk@stusta.de>
Signed-off-by: Yasunori Goto <y-goto@jp.fujitsu.com>
Signed-off-by: Bob Picco <bob.picco@hp.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-06-23 09:07:54 +02:00
|
|
|
/*
|
|
|
|
* sparse memory mappings.
|
|
|
|
*/
|
|
|
|
#include <linux/config.h>
|
|
|
|
#include <linux/mm.h>
|
|
|
|
#include <linux/mmzone.h>
|
|
|
|
#include <linux/bootmem.h>
|
|
|
|
#include <linux/module.h>
|
|
|
|
#include <asm/dma.h>
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Permanent SPARSEMEM data:
|
|
|
|
*
|
|
|
|
* 1) mem_section - memory sections, mem_map's for valid memory
|
|
|
|
*/
|
|
|
|
struct mem_section mem_section[NR_MEM_SECTIONS];
|
|
|
|
EXPORT_SYMBOL(mem_section);
|
|
|
|
|
|
|
|
/* Record a memory area against a node. */
|
|
|
|
void memory_present(int nid, unsigned long start, unsigned long end)
|
|
|
|
{
|
|
|
|
unsigned long pfn;
|
|
|
|
|
|
|
|
start &= PAGE_SECTION_MASK;
|
|
|
|
for (pfn = start; pfn < end; pfn += PAGES_PER_SECTION) {
|
|
|
|
unsigned long section = pfn_to_section_nr(pfn);
|
|
|
|
if (!mem_section[section].section_mem_map)
|
2005-06-23 09:08:00 +02:00
|
|
|
mem_section[section].section_mem_map = SECTION_MARKED_PRESENT;
|
[PATCH] sparsemem memory model
Sparsemem abstracts the use of discontiguous mem_maps[]. This kind of
mem_map[] is needed by discontiguous memory machines (like in the old
CONFIG_DISCONTIGMEM case) as well as memory hotplug systems. Sparsemem
replaces DISCONTIGMEM when enabled, and it is hoped that it can eventually
become a complete replacement.
A significant advantage over DISCONTIGMEM is that it's completely separated
from CONFIG_NUMA. When producing this patch, it became apparent in that NUMA
and DISCONTIG are often confused.
Another advantage is that sparse doesn't require each NUMA node's ranges to be
contiguous. It can handle overlapping ranges between nodes with no problems,
where DISCONTIGMEM currently throws away that memory.
Sparsemem uses an array to provide different pfn_to_page() translations for
each SECTION_SIZE area of physical memory. This is what allows the mem_map[]
to be chopped up.
In order to do quick pfn_to_page() operations, the section number of the page
is encoded in page->flags. Part of the sparsemem infrastructure enables
sharing of these bits more dynamically (at compile-time) between the
page_zone() and sparsemem operations. However, on 32-bit architectures, the
number of bits is quite limited, and may require growing the size of the
page->flags type in certain conditions. Several things might force this to
occur: a decrease in the SECTION_SIZE (if you want to hotplug smaller areas of
memory), an increase in the physical address space, or an increase in the
number of used page->flags.
One thing to note is that, once sparsemem is present, the NUMA node
information no longer needs to be stored in the page->flags. It might provide
speed increases on certain platforms and will be stored there if there is
room. But, if out of room, an alternate (theoretically slower) mechanism is
used.
This patch introduces CONFIG_FLATMEM. It is used in almost all cases where
there used to be an #ifndef DISCONTIG, because SPARSEMEM and DISCONTIGMEM
often have to compile out the same areas of code.
Signed-off-by: Andy Whitcroft <apw@shadowen.org>
Signed-off-by: Dave Hansen <haveblue@us.ibm.com>
Signed-off-by: Martin Bligh <mbligh@aracnet.com>
Signed-off-by: Adrian Bunk <bunk@stusta.de>
Signed-off-by: Yasunori Goto <y-goto@jp.fujitsu.com>
Signed-off-by: Bob Picco <bob.picco@hp.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-06-23 09:07:54 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Only used by the i386 NUMA architecures, but relatively
|
|
|
|
* generic code.
|
|
|
|
*/
|
|
|
|
unsigned long __init node_memmap_size_bytes(int nid, unsigned long start_pfn,
|
|
|
|
unsigned long end_pfn)
|
|
|
|
{
|
|
|
|
unsigned long pfn;
|
|
|
|
unsigned long nr_pages = 0;
|
|
|
|
|
|
|
|
for (pfn = start_pfn; pfn < end_pfn; pfn += PAGES_PER_SECTION) {
|
|
|
|
if (nid != early_pfn_to_nid(pfn))
|
|
|
|
continue;
|
|
|
|
|
|
|
|
if (pfn_valid(pfn))
|
|
|
|
nr_pages += PAGES_PER_SECTION;
|
|
|
|
}
|
|
|
|
|
|
|
|
return nr_pages * sizeof(struct page);
|
|
|
|
}
|
|
|
|
|
2005-06-23 09:08:00 +02:00
|
|
|
/*
|
|
|
|
* Subtle, we encode the real pfn into the mem_map such that
|
|
|
|
* the identity pfn - section_mem_map will return the actual
|
|
|
|
* physical page frame number.
|
|
|
|
*/
|
|
|
|
static unsigned long sparse_encode_mem_map(struct page *mem_map, unsigned long pnum)
|
|
|
|
{
|
|
|
|
return (unsigned long)(mem_map - (section_nr_to_pfn(pnum)));
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We need this if we ever free the mem_maps. While not implemented yet,
|
|
|
|
* this function is included for parity with its sibling.
|
|
|
|
*/
|
|
|
|
static __attribute((unused))
|
|
|
|
struct page *sparse_decode_mem_map(unsigned long coded_mem_map, unsigned long pnum)
|
|
|
|
{
|
|
|
|
return ((struct page *)coded_mem_map) + section_nr_to_pfn(pnum);
|
|
|
|
}
|
|
|
|
|
|
|
|
static int sparse_init_one_section(struct mem_section *ms,
|
|
|
|
unsigned long pnum, struct page *mem_map)
|
|
|
|
{
|
|
|
|
if (!valid_section(ms))
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
ms->section_mem_map |= sparse_encode_mem_map(mem_map, pnum);
|
|
|
|
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct page *sparse_early_mem_map_alloc(unsigned long pnum)
|
|
|
|
{
|
|
|
|
struct page *map;
|
|
|
|
int nid = early_pfn_to_nid(section_nr_to_pfn(pnum));
|
|
|
|
|
|
|
|
map = alloc_remap(nid, sizeof(struct page) * PAGES_PER_SECTION);
|
|
|
|
if (map)
|
|
|
|
return map;
|
|
|
|
|
|
|
|
map = alloc_bootmem_node(NODE_DATA(nid),
|
|
|
|
sizeof(struct page) * PAGES_PER_SECTION);
|
|
|
|
if (map)
|
|
|
|
return map;
|
|
|
|
|
|
|
|
printk(KERN_WARNING "%s: allocation failed\n", __FUNCTION__);
|
|
|
|
mem_section[pnum].section_mem_map = 0;
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
[PATCH] sparsemem memory model
Sparsemem abstracts the use of discontiguous mem_maps[]. This kind of
mem_map[] is needed by discontiguous memory machines (like in the old
CONFIG_DISCONTIGMEM case) as well as memory hotplug systems. Sparsemem
replaces DISCONTIGMEM when enabled, and it is hoped that it can eventually
become a complete replacement.
A significant advantage over DISCONTIGMEM is that it's completely separated
from CONFIG_NUMA. When producing this patch, it became apparent in that NUMA
and DISCONTIG are often confused.
Another advantage is that sparse doesn't require each NUMA node's ranges to be
contiguous. It can handle overlapping ranges between nodes with no problems,
where DISCONTIGMEM currently throws away that memory.
Sparsemem uses an array to provide different pfn_to_page() translations for
each SECTION_SIZE area of physical memory. This is what allows the mem_map[]
to be chopped up.
In order to do quick pfn_to_page() operations, the section number of the page
is encoded in page->flags. Part of the sparsemem infrastructure enables
sharing of these bits more dynamically (at compile-time) between the
page_zone() and sparsemem operations. However, on 32-bit architectures, the
number of bits is quite limited, and may require growing the size of the
page->flags type in certain conditions. Several things might force this to
occur: a decrease in the SECTION_SIZE (if you want to hotplug smaller areas of
memory), an increase in the physical address space, or an increase in the
number of used page->flags.
One thing to note is that, once sparsemem is present, the NUMA node
information no longer needs to be stored in the page->flags. It might provide
speed increases on certain platforms and will be stored there if there is
room. But, if out of room, an alternate (theoretically slower) mechanism is
used.
This patch introduces CONFIG_FLATMEM. It is used in almost all cases where
there used to be an #ifndef DISCONTIG, because SPARSEMEM and DISCONTIGMEM
often have to compile out the same areas of code.
Signed-off-by: Andy Whitcroft <apw@shadowen.org>
Signed-off-by: Dave Hansen <haveblue@us.ibm.com>
Signed-off-by: Martin Bligh <mbligh@aracnet.com>
Signed-off-by: Adrian Bunk <bunk@stusta.de>
Signed-off-by: Yasunori Goto <y-goto@jp.fujitsu.com>
Signed-off-by: Bob Picco <bob.picco@hp.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-06-23 09:07:54 +02:00
|
|
|
/*
|
|
|
|
* Allocate the accumulated non-linear sections, allocate a mem_map
|
|
|
|
* for each and record the physical to section mapping.
|
|
|
|
*/
|
|
|
|
void sparse_init(void)
|
|
|
|
{
|
|
|
|
unsigned long pnum;
|
|
|
|
struct page *map;
|
|
|
|
|
|
|
|
for (pnum = 0; pnum < NR_MEM_SECTIONS; pnum++) {
|
2005-06-23 09:08:00 +02:00
|
|
|
if (!valid_section_nr(pnum))
|
[PATCH] sparsemem memory model
Sparsemem abstracts the use of discontiguous mem_maps[]. This kind of
mem_map[] is needed by discontiguous memory machines (like in the old
CONFIG_DISCONTIGMEM case) as well as memory hotplug systems. Sparsemem
replaces DISCONTIGMEM when enabled, and it is hoped that it can eventually
become a complete replacement.
A significant advantage over DISCONTIGMEM is that it's completely separated
from CONFIG_NUMA. When producing this patch, it became apparent in that NUMA
and DISCONTIG are often confused.
Another advantage is that sparse doesn't require each NUMA node's ranges to be
contiguous. It can handle overlapping ranges between nodes with no problems,
where DISCONTIGMEM currently throws away that memory.
Sparsemem uses an array to provide different pfn_to_page() translations for
each SECTION_SIZE area of physical memory. This is what allows the mem_map[]
to be chopped up.
In order to do quick pfn_to_page() operations, the section number of the page
is encoded in page->flags. Part of the sparsemem infrastructure enables
sharing of these bits more dynamically (at compile-time) between the
page_zone() and sparsemem operations. However, on 32-bit architectures, the
number of bits is quite limited, and may require growing the size of the
page->flags type in certain conditions. Several things might force this to
occur: a decrease in the SECTION_SIZE (if you want to hotplug smaller areas of
memory), an increase in the physical address space, or an increase in the
number of used page->flags.
One thing to note is that, once sparsemem is present, the NUMA node
information no longer needs to be stored in the page->flags. It might provide
speed increases on certain platforms and will be stored there if there is
room. But, if out of room, an alternate (theoretically slower) mechanism is
used.
This patch introduces CONFIG_FLATMEM. It is used in almost all cases where
there used to be an #ifndef DISCONTIG, because SPARSEMEM and DISCONTIGMEM
often have to compile out the same areas of code.
Signed-off-by: Andy Whitcroft <apw@shadowen.org>
Signed-off-by: Dave Hansen <haveblue@us.ibm.com>
Signed-off-by: Martin Bligh <mbligh@aracnet.com>
Signed-off-by: Adrian Bunk <bunk@stusta.de>
Signed-off-by: Yasunori Goto <y-goto@jp.fujitsu.com>
Signed-off-by: Bob Picco <bob.picco@hp.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-06-23 09:07:54 +02:00
|
|
|
continue;
|
|
|
|
|
2005-06-23 09:08:00 +02:00
|
|
|
map = sparse_early_mem_map_alloc(pnum);
|
|
|
|
if (map)
|
|
|
|
sparse_init_one_section(&mem_section[pnum], pnum, map);
|
[PATCH] sparsemem memory model
Sparsemem abstracts the use of discontiguous mem_maps[]. This kind of
mem_map[] is needed by discontiguous memory machines (like in the old
CONFIG_DISCONTIGMEM case) as well as memory hotplug systems. Sparsemem
replaces DISCONTIGMEM when enabled, and it is hoped that it can eventually
become a complete replacement.
A significant advantage over DISCONTIGMEM is that it's completely separated
from CONFIG_NUMA. When producing this patch, it became apparent in that NUMA
and DISCONTIG are often confused.
Another advantage is that sparse doesn't require each NUMA node's ranges to be
contiguous. It can handle overlapping ranges between nodes with no problems,
where DISCONTIGMEM currently throws away that memory.
Sparsemem uses an array to provide different pfn_to_page() translations for
each SECTION_SIZE area of physical memory. This is what allows the mem_map[]
to be chopped up.
In order to do quick pfn_to_page() operations, the section number of the page
is encoded in page->flags. Part of the sparsemem infrastructure enables
sharing of these bits more dynamically (at compile-time) between the
page_zone() and sparsemem operations. However, on 32-bit architectures, the
number of bits is quite limited, and may require growing the size of the
page->flags type in certain conditions. Several things might force this to
occur: a decrease in the SECTION_SIZE (if you want to hotplug smaller areas of
memory), an increase in the physical address space, or an increase in the
number of used page->flags.
One thing to note is that, once sparsemem is present, the NUMA node
information no longer needs to be stored in the page->flags. It might provide
speed increases on certain platforms and will be stored there if there is
room. But, if out of room, an alternate (theoretically slower) mechanism is
used.
This patch introduces CONFIG_FLATMEM. It is used in almost all cases where
there used to be an #ifndef DISCONTIG, because SPARSEMEM and DISCONTIGMEM
often have to compile out the same areas of code.
Signed-off-by: Andy Whitcroft <apw@shadowen.org>
Signed-off-by: Dave Hansen <haveblue@us.ibm.com>
Signed-off-by: Martin Bligh <mbligh@aracnet.com>
Signed-off-by: Adrian Bunk <bunk@stusta.de>
Signed-off-by: Yasunori Goto <y-goto@jp.fujitsu.com>
Signed-off-by: Bob Picco <bob.picco@hp.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-06-23 09:07:54 +02:00
|
|
|
}
|
|
|
|
}
|
2005-06-23 09:08:00 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* returns the number of sections whose mem_maps were properly
|
|
|
|
* set. If this is <=0, then that means that the passed-in
|
|
|
|
* map was not consumed and must be freed.
|
|
|
|
*/
|
|
|
|
int sparse_add_one_section(unsigned long start_pfn, int nr_pages, struct page *map)
|
|
|
|
{
|
|
|
|
struct mem_section *ms = __pfn_to_section(start_pfn);
|
|
|
|
|
|
|
|
if (ms->section_mem_map & SECTION_MARKED_PRESENT)
|
|
|
|
return -EEXIST;
|
|
|
|
|
|
|
|
ms->section_mem_map |= SECTION_MARKED_PRESENT;
|
|
|
|
|
|
|
|
return sparse_init_one_section(ms, pfn_to_section_nr(start_pfn), map);
|
|
|
|
}
|