2005-04-17 00:20:36 +02:00
|
|
|
/*
|
2011-06-28 11:54:48 +02:00
|
|
|
* PPC Huge TLB Page Support for Kernel.
|
2005-04-17 00:20:36 +02:00
|
|
|
*
|
|
|
|
* Copyright (C) 2003 David Gibson, IBM Corporation.
|
2011-06-28 11:54:48 +02:00
|
|
|
* Copyright (C) 2011 Becky Bruce, Freescale Semiconductor
|
2005-04-17 00:20:36 +02:00
|
|
|
*
|
|
|
|
* Based on the IA-32 version:
|
|
|
|
* Copyright (C) 2002, Rohit Seth <rohit.seth@intel.com>
|
|
|
|
*/
|
|
|
|
|
|
|
|
#include <linux/mm.h>
|
2009-10-26 20:24:31 +01:00
|
|
|
#include <linux/io.h>
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 09:04:11 +01:00
|
|
|
#include <linux/slab.h>
|
2005-04-17 00:20:36 +02:00
|
|
|
#include <linux/hugetlb.h>
|
KVM: PPC: Implement MMU notifiers for Book3S HV guests
This adds the infrastructure to enable us to page out pages underneath
a Book3S HV guest, on processors that support virtualized partition
memory, that is, POWER7. Instead of pinning all the guest's pages,
we now look in the host userspace Linux page tables to find the
mapping for a given guest page. Then, if the userspace Linux PTE
gets invalidated, kvm_unmap_hva() gets called for that address, and
we replace all the guest HPTEs that refer to that page with absent
HPTEs, i.e. ones with the valid bit clear and the HPTE_V_ABSENT bit
set, which will cause an HDSI when the guest tries to access them.
Finally, the page fault handler is extended to reinstantiate the
guest HPTE when the guest tries to access a page which has been paged
out.
Since we can't intercept the guest DSI and ISI interrupts on PPC970,
we still have to pin all the guest pages on PPC970. We have a new flag,
kvm->arch.using_mmu_notifiers, that indicates whether we can page
guest pages out. If it is not set, the MMU notifier callbacks do
nothing and everything operates as before.
Signed-off-by: Paul Mackerras <paulus@samba.org>
Signed-off-by: Alexander Graf <agraf@suse.de>
Signed-off-by: Avi Kivity <avi@redhat.com>
2011-12-12 13:38:05 +01:00
|
|
|
#include <linux/export.h>
|
2011-06-28 11:54:48 +02:00
|
|
|
#include <linux/of_fdt.h>
|
|
|
|
#include <linux/memblock.h>
|
|
|
|
#include <linux/bootmem.h>
|
2011-11-24 10:40:07 +01:00
|
|
|
#include <linux/moduleparam.h>
|
2009-10-26 20:24:31 +01:00
|
|
|
#include <asm/pgtable.h>
|
2005-04-17 00:20:36 +02:00
|
|
|
#include <asm/pgalloc.h>
|
|
|
|
#include <asm/tlb.h>
|
2011-06-28 11:54:48 +02:00
|
|
|
#include <asm/setup.h>
|
2013-06-20 11:00:16 +02:00
|
|
|
#include <asm/hugetlb.h>
|
|
|
|
|
|
|
|
#ifdef CONFIG_HUGETLB_PAGE
|
2005-04-17 00:20:36 +02:00
|
|
|
|
2008-07-24 06:27:55 +02:00
|
|
|
#define PAGE_SHIFT_64K 16
|
|
|
|
#define PAGE_SHIFT_16M 24
|
|
|
|
#define PAGE_SHIFT_16G 34
|
2008-01-03 23:59:50 +01:00
|
|
|
|
2011-06-28 11:54:48 +02:00
|
|
|
unsigned int HPAGE_SHIFT;
|
2008-07-24 06:27:53 +02:00
|
|
|
|
2011-06-28 11:54:48 +02:00
|
|
|
/*
|
|
|
|
* Tracks gpages after the device tree is scanned and before the
|
2011-10-10 12:50:43 +02:00
|
|
|
* huge_boot_pages list is ready. On non-Freescale implementations, this is
|
|
|
|
* just used to track 16G pages and so is a single array. FSL-based
|
|
|
|
* implementations may have more than one gpage size, so we need multiple
|
|
|
|
* arrays
|
2011-06-28 11:54:48 +02:00
|
|
|
*/
|
2011-10-10 12:50:40 +02:00
|
|
|
#ifdef CONFIG_PPC_FSL_BOOK3E
|
2011-06-28 11:54:48 +02:00
|
|
|
#define MAX_NUMBER_GPAGES 128
|
|
|
|
struct psize_gpages {
|
|
|
|
u64 gpage_list[MAX_NUMBER_GPAGES];
|
|
|
|
unsigned int nr_gpages;
|
|
|
|
};
|
|
|
|
static struct psize_gpages gpage_freearray[MMU_PAGE_COUNT];
|
2011-10-10 12:50:40 +02:00
|
|
|
#else
|
|
|
|
#define MAX_NUMBER_GPAGES 1024
|
|
|
|
static u64 gpage_freearray[MAX_NUMBER_GPAGES];
|
|
|
|
static unsigned nr_gpages;
|
2011-06-28 11:54:48 +02:00
|
|
|
#endif
|
2006-04-28 07:02:51 +02:00
|
|
|
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
#define hugepd_none(hpd) ((hpd).pd == 0)
|
|
|
|
|
2013-04-28 11:37:30 +02:00
|
|
|
#ifdef CONFIG_PPC_BOOK3S_64
|
|
|
|
/*
|
|
|
|
* At this point we do the placement change only for BOOK3S 64. This would
|
|
|
|
* possibly work on other subarchs.
|
|
|
|
*/
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We have PGD_INDEX_SIZ = 12 and PTE_INDEX_SIZE = 8, so that we can have
|
|
|
|
* 16GB hugepage pte in PGD and 16MB hugepage pte at PMD;
|
2014-11-05 17:27:39 +01:00
|
|
|
*
|
|
|
|
* Defined in such a way that we can optimize away code block at build time
|
|
|
|
* if CONFIG_HUGETLB_PAGE=n.
|
2013-04-28 11:37:30 +02:00
|
|
|
*/
|
|
|
|
int pmd_huge(pmd_t pmd)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* leaf pte for huge page, bottom two bits != 00
|
|
|
|
*/
|
|
|
|
return ((pmd_val(pmd) & 0x3) != 0x0);
|
|
|
|
}
|
|
|
|
|
|
|
|
int pud_huge(pud_t pud)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* leaf pte for huge page, bottom two bits != 00
|
|
|
|
*/
|
|
|
|
return ((pud_val(pud) & 0x3) != 0x0);
|
|
|
|
}
|
|
|
|
|
|
|
|
int pgd_huge(pgd_t pgd)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* leaf pte for huge page, bottom two bits != 00
|
|
|
|
*/
|
|
|
|
return ((pgd_val(pgd) & 0x3) != 0x0);
|
|
|
|
}
|
2015-09-07 09:23:53 +02:00
|
|
|
|
|
|
|
#if defined(CONFIG_PPC_64K_PAGES) && defined(CONFIG_DEBUG_VM)
|
|
|
|
/*
|
|
|
|
* This enables us to catch the wrong page directory format
|
|
|
|
* Moved here so that we can use WARN() in the call.
|
|
|
|
*/
|
|
|
|
int hugepd_ok(hugepd_t hpd)
|
|
|
|
{
|
|
|
|
bool is_hugepd;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We should not find this format in page directory, warn otherwise.
|
|
|
|
*/
|
|
|
|
is_hugepd = (((hpd.pd & 0x3) == 0x0) && ((hpd.pd & HUGEPD_SHIFT_MASK) != 0));
|
|
|
|
WARN(is_hugepd, "Found wrong page directory format\n");
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
2013-04-28 11:37:30 +02:00
|
|
|
#else
|
|
|
|
int pmd_huge(pmd_t pmd)
|
|
|
|
{
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
int pud_huge(pud_t pud)
|
|
|
|
{
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
int pgd_huge(pgd_t pgd)
|
|
|
|
{
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr)
|
|
|
|
{
|
2013-06-20 11:00:18 +02:00
|
|
|
/* Only called for hugetlbfs pages, hence can ignore THP */
|
2015-03-30 07:11:03 +02:00
|
|
|
return __find_linux_pte_or_hugepte(mm->pgd, addr, NULL);
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
}
|
|
|
|
|
2006-04-28 07:02:51 +02:00
|
|
|
static int __hugepte_alloc(struct mm_struct *mm, hugepd_t *hpdp,
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
unsigned long address, unsigned pdshift, unsigned pshift)
|
2006-04-28 07:02:51 +02:00
|
|
|
{
|
2011-06-28 11:54:48 +02:00
|
|
|
struct kmem_cache *cachep;
|
|
|
|
pte_t *new;
|
|
|
|
|
2011-10-10 12:50:40 +02:00
|
|
|
#ifdef CONFIG_PPC_FSL_BOOK3E
|
2011-06-28 11:54:48 +02:00
|
|
|
int i;
|
|
|
|
int num_hugepd = 1 << (pshift - pdshift);
|
|
|
|
cachep = hugepte_cache;
|
2011-10-10 12:50:40 +02:00
|
|
|
#else
|
|
|
|
cachep = PGT_CACHE(pdshift - pshift);
|
2011-06-28 11:54:48 +02:00
|
|
|
#endif
|
|
|
|
|
|
|
|
new = kmem_cache_zalloc(cachep, GFP_KERNEL|__GFP_REPEAT);
|
2006-04-28 07:02:51 +02:00
|
|
|
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
BUG_ON(pshift > HUGEPD_SHIFT_MASK);
|
|
|
|
BUG_ON((unsigned long)new & HUGEPD_SHIFT_MASK);
|
|
|
|
|
2006-04-28 07:02:51 +02:00
|
|
|
if (! new)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
spin_lock(&mm->page_table_lock);
|
2011-10-10 12:50:40 +02:00
|
|
|
#ifdef CONFIG_PPC_FSL_BOOK3E
|
2011-06-28 11:54:48 +02:00
|
|
|
/*
|
|
|
|
* We have multiple higher-level entries that point to the same
|
|
|
|
* actual pte location. Fill in each as we go and backtrack on error.
|
|
|
|
* We need all of these so the DTLB pgtable walk code can find the
|
|
|
|
* right higher-level entry without knowing if it's a hugepage or not.
|
|
|
|
*/
|
|
|
|
for (i = 0; i < num_hugepd; i++, hpdp++) {
|
|
|
|
if (unlikely(!hugepd_none(*hpdp)))
|
|
|
|
break;
|
|
|
|
else
|
2013-04-28 11:37:29 +02:00
|
|
|
/* We use the old format for PPC_FSL_BOOK3E */
|
2011-06-28 11:54:48 +02:00
|
|
|
hpdp->pd = ((unsigned long)new & ~PD_HUGE) | pshift;
|
|
|
|
}
|
|
|
|
/* If we bailed from the for loop early, an error occurred, clean up */
|
|
|
|
if (i < num_hugepd) {
|
|
|
|
for (i = i - 1 ; i >= 0; i--, hpdp--)
|
|
|
|
hpdp->pd = 0;
|
|
|
|
kmem_cache_free(cachep, new);
|
|
|
|
}
|
2011-10-10 12:50:39 +02:00
|
|
|
#else
|
|
|
|
if (!hugepd_none(*hpdp))
|
|
|
|
kmem_cache_free(cachep, new);
|
2013-04-28 11:37:29 +02:00
|
|
|
else {
|
|
|
|
#ifdef CONFIG_PPC_BOOK3S_64
|
|
|
|
hpdp->pd = (unsigned long)new |
|
|
|
|
(shift_to_mmu_psize(pshift) << 2);
|
|
|
|
#else
|
2011-10-10 12:50:39 +02:00
|
|
|
hpdp->pd = ((unsigned long)new & ~PD_HUGE) | pshift;
|
2013-04-28 11:37:29 +02:00
|
|
|
#endif
|
|
|
|
}
|
2011-06-28 11:54:48 +02:00
|
|
|
#endif
|
2006-04-28 07:02:51 +02:00
|
|
|
spin_unlock(&mm->page_table_lock);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2011-10-10 12:50:39 +02:00
|
|
|
/*
|
|
|
|
* These macros define how to determine which level of the page table holds
|
|
|
|
* the hpdp.
|
|
|
|
*/
|
|
|
|
#ifdef CONFIG_PPC_FSL_BOOK3E
|
|
|
|
#define HUGEPD_PGD_SHIFT PGDIR_SHIFT
|
|
|
|
#define HUGEPD_PUD_SHIFT PUD_SHIFT
|
|
|
|
#else
|
|
|
|
#define HUGEPD_PGD_SHIFT PUD_SHIFT
|
|
|
|
#define HUGEPD_PUD_SHIFT PMD_SHIFT
|
|
|
|
#endif
|
|
|
|
|
2013-04-28 11:37:30 +02:00
|
|
|
#ifdef CONFIG_PPC_BOOK3S_64
|
|
|
|
/*
|
|
|
|
* At this point we do the placement change only for BOOK3S 64. This would
|
|
|
|
* possibly work on other subarchs.
|
|
|
|
*/
|
|
|
|
pte_t *huge_pte_alloc(struct mm_struct *mm, unsigned long addr, unsigned long sz)
|
|
|
|
{
|
|
|
|
pgd_t *pg;
|
|
|
|
pud_t *pu;
|
|
|
|
pmd_t *pm;
|
|
|
|
hugepd_t *hpdp = NULL;
|
|
|
|
unsigned pshift = __ffs(sz);
|
|
|
|
unsigned pdshift = PGDIR_SHIFT;
|
|
|
|
|
|
|
|
addr &= ~(sz-1);
|
|
|
|
pg = pgd_offset(mm, addr);
|
|
|
|
|
|
|
|
if (pshift == PGDIR_SHIFT)
|
|
|
|
/* 16GB huge page */
|
|
|
|
return (pte_t *) pg;
|
|
|
|
else if (pshift > PUD_SHIFT)
|
|
|
|
/*
|
|
|
|
* We need to use hugepd table
|
|
|
|
*/
|
|
|
|
hpdp = (hugepd_t *)pg;
|
|
|
|
else {
|
|
|
|
pdshift = PUD_SHIFT;
|
|
|
|
pu = pud_alloc(mm, pg, addr);
|
|
|
|
if (pshift == PUD_SHIFT)
|
|
|
|
return (pte_t *)pu;
|
|
|
|
else if (pshift > PMD_SHIFT)
|
|
|
|
hpdp = (hugepd_t *)pu;
|
|
|
|
else {
|
|
|
|
pdshift = PMD_SHIFT;
|
|
|
|
pm = pmd_alloc(mm, pu, addr);
|
|
|
|
if (pshift == PMD_SHIFT)
|
|
|
|
/* 16MB hugepage */
|
|
|
|
return (pte_t *)pm;
|
|
|
|
else
|
|
|
|
hpdp = (hugepd_t *)pm;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
if (!hpdp)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
BUG_ON(!hugepd_none(*hpdp) && !hugepd_ok(*hpdp));
|
|
|
|
|
|
|
|
if (hugepd_none(*hpdp) && __hugepte_alloc(mm, hpdp, addr, pdshift, pshift))
|
|
|
|
return NULL;
|
|
|
|
|
2014-11-05 17:27:41 +01:00
|
|
|
return hugepte_offset(*hpdp, addr, pdshift);
|
2013-04-28 11:37:30 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
#else
|
|
|
|
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
pte_t *huge_pte_alloc(struct mm_struct *mm, unsigned long addr, unsigned long sz)
|
2008-09-05 03:49:54 +02:00
|
|
|
{
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
pgd_t *pg;
|
|
|
|
pud_t *pu;
|
|
|
|
pmd_t *pm;
|
|
|
|
hugepd_t *hpdp = NULL;
|
|
|
|
unsigned pshift = __ffs(sz);
|
|
|
|
unsigned pdshift = PGDIR_SHIFT;
|
|
|
|
|
|
|
|
addr &= ~(sz-1);
|
|
|
|
|
|
|
|
pg = pgd_offset(mm, addr);
|
2011-10-10 12:50:39 +02:00
|
|
|
|
|
|
|
if (pshift >= HUGEPD_PGD_SHIFT) {
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
hpdp = (hugepd_t *)pg;
|
|
|
|
} else {
|
|
|
|
pdshift = PUD_SHIFT;
|
|
|
|
pu = pud_alloc(mm, pg, addr);
|
2011-10-10 12:50:39 +02:00
|
|
|
if (pshift >= HUGEPD_PUD_SHIFT) {
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
hpdp = (hugepd_t *)pu;
|
|
|
|
} else {
|
|
|
|
pdshift = PMD_SHIFT;
|
|
|
|
pm = pmd_alloc(mm, pu, addr);
|
|
|
|
hpdp = (hugepd_t *)pm;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!hpdp)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
BUG_ON(!hugepd_none(*hpdp) && !hugepd_ok(*hpdp));
|
|
|
|
|
|
|
|
if (hugepd_none(*hpdp) && __hugepte_alloc(mm, hpdp, addr, pdshift, pshift))
|
|
|
|
return NULL;
|
|
|
|
|
2014-11-05 17:27:41 +01:00
|
|
|
return hugepte_offset(*hpdp, addr, pdshift);
|
2008-01-03 23:59:50 +01:00
|
|
|
}
|
2013-04-28 11:37:30 +02:00
|
|
|
#endif
|
2008-01-03 23:59:50 +01:00
|
|
|
|
2011-10-10 12:50:40 +02:00
|
|
|
#ifdef CONFIG_PPC_FSL_BOOK3E
|
2008-07-24 06:27:54 +02:00
|
|
|
/* Build list of addresses of gigantic pages. This function is used in early
|
2014-09-17 14:15:34 +02:00
|
|
|
* boot before the buddy allocator is setup.
|
2008-07-24 06:27:54 +02:00
|
|
|
*/
|
2011-06-28 11:54:48 +02:00
|
|
|
void add_gpage(u64 addr, u64 page_size, unsigned long number_of_pages)
|
|
|
|
{
|
|
|
|
unsigned int idx = shift_to_mmu_psize(__ffs(page_size));
|
|
|
|
int i;
|
|
|
|
|
|
|
|
if (addr == 0)
|
|
|
|
return;
|
|
|
|
|
|
|
|
gpage_freearray[idx].nr_gpages = number_of_pages;
|
|
|
|
|
|
|
|
for (i = 0; i < number_of_pages; i++) {
|
|
|
|
gpage_freearray[idx].gpage_list[i] = addr;
|
|
|
|
addr += page_size;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Moves the gigantic page addresses from the temporary list to the
|
|
|
|
* huge_boot_pages list.
|
|
|
|
*/
|
|
|
|
int alloc_bootmem_huge_page(struct hstate *hstate)
|
|
|
|
{
|
|
|
|
struct huge_bootmem_page *m;
|
2013-07-04 00:02:43 +02:00
|
|
|
int idx = shift_to_mmu_psize(huge_page_shift(hstate));
|
2011-06-28 11:54:48 +02:00
|
|
|
int nr_gpages = gpage_freearray[idx].nr_gpages;
|
|
|
|
|
|
|
|
if (nr_gpages == 0)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
#ifdef CONFIG_HIGHMEM
|
|
|
|
/*
|
|
|
|
* If gpages can be in highmem we can't use the trick of storing the
|
|
|
|
* data structure in the page; allocate space for this
|
|
|
|
*/
|
powerpc: Remove more traces of bootmem
Although we are now selecting NO_BOOTMEM, we still have some traces of
bootmem lying around. That is because even with NO_BOOTMEM there is
still a shim that converts bootmem calls into memblock calls, but
ultimately we want to remove all traces of bootmem.
Most of the patch is conversions from alloc_bootmem() to
memblock_virt_alloc(). In general a call such as:
p = (struct foo *)alloc_bootmem(x);
Becomes:
p = memblock_virt_alloc(x, 0);
We don't need the cast because memblock_virt_alloc() returns a void *.
The alignment value of zero tells memblock to use the default alignment,
which is SMP_CACHE_BYTES, the same value alloc_bootmem() uses.
We remove a number of NULL checks on the result of
memblock_virt_alloc(). That is because memblock_virt_alloc() will panic
if it can't allocate, in exactly the same way as alloc_bootmem(), so the
NULL checks are and always have been redundant.
The memory returned by memblock_virt_alloc() is already zeroed, so we
remove several memsets of the result of memblock_virt_alloc().
Finally we convert a few uses of __alloc_bootmem(x, y, MAX_DMA_ADDRESS)
to just plain memblock_virt_alloc(). We don't use memblock_alloc_base()
because MAX_DMA_ADDRESS is ~0ul on powerpc, so limiting the allocation
to that is pointless, 16XB ought to be enough for anyone.
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2014-11-18 06:47:35 +01:00
|
|
|
m = memblock_virt_alloc(sizeof(struct huge_bootmem_page), 0);
|
2011-06-28 11:54:48 +02:00
|
|
|
m->phys = gpage_freearray[idx].gpage_list[--nr_gpages];
|
|
|
|
#else
|
|
|
|
m = phys_to_virt(gpage_freearray[idx].gpage_list[--nr_gpages]);
|
|
|
|
#endif
|
|
|
|
|
|
|
|
list_add(&m->list, &huge_boot_pages);
|
|
|
|
gpage_freearray[idx].nr_gpages = nr_gpages;
|
|
|
|
gpage_freearray[idx].gpage_list[nr_gpages] = 0;
|
|
|
|
m->hstate = hstate;
|
|
|
|
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
/*
|
|
|
|
* Scan the command line hugepagesz= options for gigantic pages; store those in
|
|
|
|
* a list that we use to allocate the memory once all options are parsed.
|
|
|
|
*/
|
|
|
|
|
|
|
|
unsigned long gpage_npages[MMU_PAGE_COUNT];
|
|
|
|
|
2012-05-07 16:32:22 +02:00
|
|
|
static int __init do_gpage_early_setup(char *param, char *val,
|
module: add extra argument for parse_params() callback
This adds an extra argument onto parse_params() to be used
as a way to make the unused callback a bit more useful and
generic by allowing the caller to pass on a data structure
of its choice. An example use case is to allow us to easily
make module parameters for every module which we will do
next.
@ parse @
identifier name, args, params, num, level_min, level_max;
identifier unknown, param, val, doing;
type s16;
@@
extern char *parse_args(const char *name,
char *args,
const struct kernel_param *params,
unsigned num,
s16 level_min,
s16 level_max,
+ void *arg,
int (*unknown)(char *param, char *val,
const char *doing
+ , void *arg
));
@ parse_mod @
identifier name, args, params, num, level_min, level_max;
identifier unknown, param, val, doing;
type s16;
@@
char *parse_args(const char *name,
char *args,
const struct kernel_param *params,
unsigned num,
s16 level_min,
s16 level_max,
+ void *arg,
int (*unknown)(char *param, char *val,
const char *doing
+ , void *arg
))
{
...
}
@ parse_args_found @
expression R, E1, E2, E3, E4, E5, E6;
identifier func;
@@
(
R =
parse_args(E1, E2, E3, E4, E5, E6,
+ NULL,
func);
|
R =
parse_args(E1, E2, E3, E4, E5, E6,
+ NULL,
&func);
|
R =
parse_args(E1, E2, E3, E4, E5, E6,
+ NULL,
NULL);
|
parse_args(E1, E2, E3, E4, E5, E6,
+ NULL,
func);
|
parse_args(E1, E2, E3, E4, E5, E6,
+ NULL,
&func);
|
parse_args(E1, E2, E3, E4, E5, E6,
+ NULL,
NULL);
)
@ parse_args_unused depends on parse_args_found @
identifier parse_args_found.func;
@@
int func(char *param, char *val, const char *unused
+ , void *arg
)
{
...
}
@ mod_unused depends on parse_args_found @
identifier parse_args_found.func;
expression A1, A2, A3;
@@
- func(A1, A2, A3);
+ func(A1, A2, A3, NULL);
Generated-by: Coccinelle SmPL
Cc: cocci@systeme.lip6.fr
Cc: Tejun Heo <tj@kernel.org>
Cc: Arjan van de Ven <arjan@linux.intel.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Rusty Russell <rusty@rustcorp.com.au>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Felipe Contreras <felipe.contreras@gmail.com>
Cc: Ewan Milne <emilne@redhat.com>
Cc: Jean Delvare <jdelvare@suse.de>
Cc: Hannes Reinecke <hare@suse.de>
Cc: Jani Nikula <jani.nikula@intel.com>
Cc: linux-kernel@vger.kernel.org
Reviewed-by: Tejun Heo <tj@kernel.org>
Acked-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Luis R. Rodriguez <mcgrof@suse.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2015-03-31 01:20:03 +02:00
|
|
|
const char *unused, void *arg)
|
2011-06-28 11:54:48 +02:00
|
|
|
{
|
|
|
|
static phys_addr_t size;
|
|
|
|
unsigned long npages;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The hugepagesz and hugepages cmdline options are interleaved. We
|
|
|
|
* use the size variable to keep track of whether or not this was done
|
|
|
|
* properly and skip over instances where it is incorrect. Other
|
|
|
|
* command-line parsing code will issue warnings, so we don't need to.
|
|
|
|
*
|
|
|
|
*/
|
|
|
|
if ((strcmp(param, "default_hugepagesz") == 0) ||
|
|
|
|
(strcmp(param, "hugepagesz") == 0)) {
|
|
|
|
size = memparse(val, NULL);
|
|
|
|
} else if (strcmp(param, "hugepages") == 0) {
|
|
|
|
if (size != 0) {
|
|
|
|
if (sscanf(val, "%lu", &npages) <= 0)
|
|
|
|
npages = 0;
|
2014-11-14 19:32:24 +01:00
|
|
|
if (npages > MAX_NUMBER_GPAGES) {
|
|
|
|
pr_warn("MMU: %lu pages requested for page "
|
|
|
|
"size %llu KB, limiting to "
|
|
|
|
__stringify(MAX_NUMBER_GPAGES) "\n",
|
|
|
|
npages, size / 1024);
|
|
|
|
npages = MAX_NUMBER_GPAGES;
|
|
|
|
}
|
2011-06-28 11:54:48 +02:00
|
|
|
gpage_npages[shift_to_mmu_psize(__ffs(size))] = npages;
|
|
|
|
size = 0;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This function allocates physical space for pages that are larger than the
|
|
|
|
* buddy allocator can handle. We want to allocate these in highmem because
|
|
|
|
* the amount of lowmem is limited. This means that this function MUST be
|
|
|
|
* called before lowmem_end_addr is set up in MMU_init() in order for the lmb
|
|
|
|
* allocate to grab highmem.
|
|
|
|
*/
|
|
|
|
void __init reserve_hugetlb_gpages(void)
|
|
|
|
{
|
|
|
|
static __initdata char cmdline[COMMAND_LINE_SIZE];
|
|
|
|
phys_addr_t size, base;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
strlcpy(cmdline, boot_command_line, COMMAND_LINE_SIZE);
|
2012-03-26 04:20:51 +02:00
|
|
|
parse_args("hugetlb gpages", cmdline, NULL, 0, 0, 0,
|
module: add extra argument for parse_params() callback
This adds an extra argument onto parse_params() to be used
as a way to make the unused callback a bit more useful and
generic by allowing the caller to pass on a data structure
of its choice. An example use case is to allow us to easily
make module parameters for every module which we will do
next.
@ parse @
identifier name, args, params, num, level_min, level_max;
identifier unknown, param, val, doing;
type s16;
@@
extern char *parse_args(const char *name,
char *args,
const struct kernel_param *params,
unsigned num,
s16 level_min,
s16 level_max,
+ void *arg,
int (*unknown)(char *param, char *val,
const char *doing
+ , void *arg
));
@ parse_mod @
identifier name, args, params, num, level_min, level_max;
identifier unknown, param, val, doing;
type s16;
@@
char *parse_args(const char *name,
char *args,
const struct kernel_param *params,
unsigned num,
s16 level_min,
s16 level_max,
+ void *arg,
int (*unknown)(char *param, char *val,
const char *doing
+ , void *arg
))
{
...
}
@ parse_args_found @
expression R, E1, E2, E3, E4, E5, E6;
identifier func;
@@
(
R =
parse_args(E1, E2, E3, E4, E5, E6,
+ NULL,
func);
|
R =
parse_args(E1, E2, E3, E4, E5, E6,
+ NULL,
&func);
|
R =
parse_args(E1, E2, E3, E4, E5, E6,
+ NULL,
NULL);
|
parse_args(E1, E2, E3, E4, E5, E6,
+ NULL,
func);
|
parse_args(E1, E2, E3, E4, E5, E6,
+ NULL,
&func);
|
parse_args(E1, E2, E3, E4, E5, E6,
+ NULL,
NULL);
)
@ parse_args_unused depends on parse_args_found @
identifier parse_args_found.func;
@@
int func(char *param, char *val, const char *unused
+ , void *arg
)
{
...
}
@ mod_unused depends on parse_args_found @
identifier parse_args_found.func;
expression A1, A2, A3;
@@
- func(A1, A2, A3);
+ func(A1, A2, A3, NULL);
Generated-by: Coccinelle SmPL
Cc: cocci@systeme.lip6.fr
Cc: Tejun Heo <tj@kernel.org>
Cc: Arjan van de Ven <arjan@linux.intel.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Rusty Russell <rusty@rustcorp.com.au>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Felipe Contreras <felipe.contreras@gmail.com>
Cc: Ewan Milne <emilne@redhat.com>
Cc: Jean Delvare <jdelvare@suse.de>
Cc: Hannes Reinecke <hare@suse.de>
Cc: Jani Nikula <jani.nikula@intel.com>
Cc: linux-kernel@vger.kernel.org
Reviewed-by: Tejun Heo <tj@kernel.org>
Acked-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Luis R. Rodriguez <mcgrof@suse.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2015-03-31 01:20:03 +02:00
|
|
|
NULL, &do_gpage_early_setup);
|
2011-06-28 11:54:48 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Walk gpage list in reverse, allocating larger page sizes first.
|
|
|
|
* Skip over unsupported sizes, or sizes that have 0 gpages allocated.
|
|
|
|
* When we reach the point in the list where pages are no longer
|
|
|
|
* considered gpages, we're done.
|
|
|
|
*/
|
|
|
|
for (i = MMU_PAGE_COUNT-1; i >= 0; i--) {
|
|
|
|
if (mmu_psize_defs[i].shift == 0 || gpage_npages[i] == 0)
|
|
|
|
continue;
|
|
|
|
else if (mmu_psize_to_shift(i) < (MAX_ORDER + PAGE_SHIFT))
|
|
|
|
break;
|
|
|
|
|
|
|
|
size = (phys_addr_t)(1ULL << mmu_psize_to_shift(i));
|
|
|
|
base = memblock_alloc_base(size * gpage_npages[i], size,
|
|
|
|
MEMBLOCK_ALLOC_ANYWHERE);
|
|
|
|
add_gpage(base, size, gpage_npages[i]);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2011-10-10 12:50:40 +02:00
|
|
|
#else /* !PPC_FSL_BOOK3E */
|
2011-06-28 11:54:48 +02:00
|
|
|
|
|
|
|
/* Build list of addresses of gigantic pages. This function is used in early
|
2014-09-17 14:15:34 +02:00
|
|
|
* boot before the buddy allocator is setup.
|
2011-06-28 11:54:48 +02:00
|
|
|
*/
|
|
|
|
void add_gpage(u64 addr, u64 page_size, unsigned long number_of_pages)
|
2008-07-24 06:27:54 +02:00
|
|
|
{
|
|
|
|
if (!addr)
|
|
|
|
return;
|
|
|
|
while (number_of_pages > 0) {
|
|
|
|
gpage_freearray[nr_gpages] = addr;
|
|
|
|
nr_gpages++;
|
|
|
|
number_of_pages--;
|
|
|
|
addr += page_size;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2008-07-24 06:27:53 +02:00
|
|
|
/* Moves the gigantic page addresses from the temporary list to the
|
2008-07-24 06:27:56 +02:00
|
|
|
* huge_boot_pages list.
|
|
|
|
*/
|
|
|
|
int alloc_bootmem_huge_page(struct hstate *hstate)
|
2008-07-24 06:27:53 +02:00
|
|
|
{
|
|
|
|
struct huge_bootmem_page *m;
|
|
|
|
if (nr_gpages == 0)
|
|
|
|
return 0;
|
|
|
|
m = phys_to_virt(gpage_freearray[--nr_gpages]);
|
|
|
|
gpage_freearray[nr_gpages] = 0;
|
|
|
|
list_add(&m->list, &huge_boot_pages);
|
2008-07-24 06:27:56 +02:00
|
|
|
m->hstate = hstate;
|
2008-07-24 06:27:53 +02:00
|
|
|
return 1;
|
|
|
|
}
|
2011-06-28 11:54:48 +02:00
|
|
|
#endif
|
2008-07-24 06:27:53 +02:00
|
|
|
|
2011-10-10 12:50:40 +02:00
|
|
|
#ifdef CONFIG_PPC_FSL_BOOK3E
|
2011-06-28 11:54:48 +02:00
|
|
|
#define HUGEPD_FREELIST_SIZE \
|
|
|
|
((PAGE_SIZE - sizeof(struct hugepd_freelist)) / sizeof(pte_t))
|
|
|
|
|
|
|
|
struct hugepd_freelist {
|
|
|
|
struct rcu_head rcu;
|
|
|
|
unsigned int index;
|
|
|
|
void *ptes[0];
|
|
|
|
};
|
|
|
|
|
|
|
|
static DEFINE_PER_CPU(struct hugepd_freelist *, hugepd_freelist_cur);
|
|
|
|
|
|
|
|
static void hugepd_free_rcu_callback(struct rcu_head *head)
|
|
|
|
{
|
|
|
|
struct hugepd_freelist *batch =
|
|
|
|
container_of(head, struct hugepd_freelist, rcu);
|
|
|
|
unsigned int i;
|
|
|
|
|
|
|
|
for (i = 0; i < batch->index; i++)
|
|
|
|
kmem_cache_free(hugepte_cache, batch->ptes[i]);
|
|
|
|
|
|
|
|
free_page((unsigned long)batch);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void hugepd_free(struct mmu_gather *tlb, void *hugepte)
|
|
|
|
{
|
|
|
|
struct hugepd_freelist **batchp;
|
|
|
|
|
powerpc: Replace __get_cpu_var uses
This still has not been merged and now powerpc is the only arch that does
not have this change. Sorry about missing linuxppc-dev before.
V2->V2
- Fix up to work against 3.18-rc1
__get_cpu_var() is used for multiple purposes in the kernel source. One of
them is address calculation via the form &__get_cpu_var(x). This calculates
the address for the instance of the percpu variable of the current processor
based on an offset.
Other use cases are for storing and retrieving data from the current
processors percpu area. __get_cpu_var() can be used as an lvalue when
writing data or on the right side of an assignment.
__get_cpu_var() is defined as :
__get_cpu_var() always only does an address determination. However, store
and retrieve operations could use a segment prefix (or global register on
other platforms) to avoid the address calculation.
this_cpu_write() and this_cpu_read() can directly take an offset into a
percpu area and use optimized assembly code to read and write per cpu
variables.
This patch converts __get_cpu_var into either an explicit address
calculation using this_cpu_ptr() or into a use of this_cpu operations that
use the offset. Thereby address calculations are avoided and less registers
are used when code is generated.
At the end of the patch set all uses of __get_cpu_var have been removed so
the macro is removed too.
The patch set includes passes over all arches as well. Once these operations
are used throughout then specialized macros can be defined in non -x86
arches as well in order to optimize per cpu access by f.e. using a global
register that may be set to the per cpu base.
Transformations done to __get_cpu_var()
1. Determine the address of the percpu instance of the current processor.
DEFINE_PER_CPU(int, y);
int *x = &__get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(&y);
2. Same as #1 but this time an array structure is involved.
DEFINE_PER_CPU(int, y[20]);
int *x = __get_cpu_var(y);
Converts to
int *x = this_cpu_ptr(y);
3. Retrieve the content of the current processors instance of a per cpu
variable.
DEFINE_PER_CPU(int, y);
int x = __get_cpu_var(y)
Converts to
int x = __this_cpu_read(y);
4. Retrieve the content of a percpu struct
DEFINE_PER_CPU(struct mystruct, y);
struct mystruct x = __get_cpu_var(y);
Converts to
memcpy(&x, this_cpu_ptr(&y), sizeof(x));
5. Assignment to a per cpu variable
DEFINE_PER_CPU(int, y)
__get_cpu_var(y) = x;
Converts to
__this_cpu_write(y, x);
6. Increment/Decrement etc of a per cpu variable
DEFINE_PER_CPU(int, y);
__get_cpu_var(y)++
Converts to
__this_cpu_inc(y)
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <paulus@samba.org>
Signed-off-by: Christoph Lameter <cl@linux.com>
[mpe: Fix build errors caused by set/or_softirq_pending(), and rework
assignment in __set_breakpoint() to use memcpy().]
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2014-10-21 22:23:25 +02:00
|
|
|
batchp = this_cpu_ptr(&hugepd_freelist_cur);
|
2011-06-28 11:54:48 +02:00
|
|
|
|
|
|
|
if (atomic_read(&tlb->mm->mm_users) < 2 ||
|
|
|
|
cpumask_equal(mm_cpumask(tlb->mm),
|
|
|
|
cpumask_of(smp_processor_id()))) {
|
|
|
|
kmem_cache_free(hugepte_cache, hugepte);
|
2014-01-20 09:39:34 +01:00
|
|
|
put_cpu_var(hugepd_freelist_cur);
|
2011-06-28 11:54:48 +02:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (*batchp == NULL) {
|
|
|
|
*batchp = (struct hugepd_freelist *)__get_free_page(GFP_ATOMIC);
|
|
|
|
(*batchp)->index = 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
(*batchp)->ptes[(*batchp)->index++] = hugepte;
|
|
|
|
if ((*batchp)->index == HUGEPD_FREELIST_SIZE) {
|
|
|
|
call_rcu_sched(&(*batchp)->rcu, hugepd_free_rcu_callback);
|
|
|
|
*batchp = NULL;
|
|
|
|
}
|
2014-01-20 09:39:34 +01:00
|
|
|
put_cpu_var(hugepd_freelist_cur);
|
2011-06-28 11:54:48 +02:00
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
static void free_hugepd_range(struct mmu_gather *tlb, hugepd_t *hpdp, int pdshift,
|
|
|
|
unsigned long start, unsigned long end,
|
|
|
|
unsigned long floor, unsigned long ceiling)
|
2006-04-28 07:02:51 +02:00
|
|
|
{
|
|
|
|
pte_t *hugepte = hugepd_page(*hpdp);
|
2011-06-28 11:54:48 +02:00
|
|
|
int i;
|
|
|
|
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
unsigned long pdmask = ~((1UL << pdshift) - 1);
|
2011-06-28 11:54:48 +02:00
|
|
|
unsigned int num_hugepd = 1;
|
|
|
|
|
2011-10-10 12:50:40 +02:00
|
|
|
#ifdef CONFIG_PPC_FSL_BOOK3E
|
|
|
|
/* Note: On fsl the hpdp may be the first of several */
|
2011-06-28 11:54:48 +02:00
|
|
|
num_hugepd = (1 << (hugepd_shift(*hpdp) - pdshift));
|
2011-10-10 12:50:40 +02:00
|
|
|
#else
|
|
|
|
unsigned int shift = hugepd_shift(*hpdp);
|
2011-06-28 11:54:48 +02:00
|
|
|
#endif
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
|
|
|
|
start &= pdmask;
|
|
|
|
if (start < floor)
|
|
|
|
return;
|
|
|
|
if (ceiling) {
|
|
|
|
ceiling &= pdmask;
|
|
|
|
if (! ceiling)
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
if (end - 1 > ceiling - 1)
|
|
|
|
return;
|
2006-04-28 07:02:51 +02:00
|
|
|
|
2011-06-28 11:54:48 +02:00
|
|
|
for (i = 0; i < num_hugepd; i++, hpdp++)
|
|
|
|
hpdp->pd = 0;
|
|
|
|
|
2011-10-10 12:50:40 +02:00
|
|
|
#ifdef CONFIG_PPC_FSL_BOOK3E
|
2011-06-28 11:54:48 +02:00
|
|
|
hugepd_free(tlb, hugepte);
|
2011-10-10 12:50:40 +02:00
|
|
|
#else
|
|
|
|
pgtable_free_tlb(tlb, hugepte, pdshift - shift);
|
2011-06-28 11:54:48 +02:00
|
|
|
#endif
|
2006-04-28 07:02:51 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
static void hugetlb_free_pmd_range(struct mmu_gather *tlb, pud_t *pud,
|
|
|
|
unsigned long addr, unsigned long end,
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
unsigned long floor, unsigned long ceiling)
|
2006-04-28 07:02:51 +02:00
|
|
|
{
|
|
|
|
pmd_t *pmd;
|
|
|
|
unsigned long next;
|
|
|
|
unsigned long start;
|
|
|
|
|
|
|
|
start = addr;
|
|
|
|
do {
|
2011-10-10 12:50:39 +02:00
|
|
|
pmd = pmd_offset(pud, addr);
|
2006-04-28 07:02:51 +02:00
|
|
|
next = pmd_addr_end(addr, end);
|
2014-11-05 17:27:41 +01:00
|
|
|
if (!is_hugepd(__hugepd(pmd_val(*pmd)))) {
|
2013-06-19 08:34:26 +02:00
|
|
|
/*
|
|
|
|
* if it is not hugepd pointer, we should already find
|
|
|
|
* it cleared.
|
|
|
|
*/
|
|
|
|
WARN_ON(!pmd_none_or_clear_bad(pmd));
|
2006-04-28 07:02:51 +02:00
|
|
|
continue;
|
2013-06-19 08:34:26 +02:00
|
|
|
}
|
2011-10-10 12:50:39 +02:00
|
|
|
#ifdef CONFIG_PPC_FSL_BOOK3E
|
|
|
|
/*
|
|
|
|
* Increment next by the size of the huge mapping since
|
|
|
|
* there may be more than one entry at this level for a
|
|
|
|
* single hugepage, but all of them point to
|
|
|
|
* the same kmem cache that holds the hugepte.
|
|
|
|
*/
|
|
|
|
next = addr + (1 << hugepd_shift(*(hugepd_t *)pmd));
|
|
|
|
#endif
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
free_hugepd_range(tlb, (hugepd_t *)pmd, PMD_SHIFT,
|
|
|
|
addr, next, floor, ceiling);
|
2011-10-10 12:50:39 +02:00
|
|
|
} while (addr = next, addr != end);
|
2006-04-28 07:02:51 +02:00
|
|
|
|
|
|
|
start &= PUD_MASK;
|
|
|
|
if (start < floor)
|
|
|
|
return;
|
|
|
|
if (ceiling) {
|
|
|
|
ceiling &= PUD_MASK;
|
|
|
|
if (!ceiling)
|
|
|
|
return;
|
2005-04-17 00:20:36 +02:00
|
|
|
}
|
2006-04-28 07:02:51 +02:00
|
|
|
if (end - 1 > ceiling - 1)
|
|
|
|
return;
|
2005-04-17 00:20:36 +02:00
|
|
|
|
2006-04-28 07:02:51 +02:00
|
|
|
pmd = pmd_offset(pud, start);
|
|
|
|
pud_clear(pud);
|
mm: Pass virtual address to [__]p{te,ud,md}_free_tlb()
mm: Pass virtual address to [__]p{te,ud,md}_free_tlb()
Upcoming paches to support the new 64-bit "BookE" powerpc architecture
will need to have the virtual address corresponding to PTE page when
freeing it, due to the way the HW table walker works.
Basically, the TLB can be loaded with "large" pages that cover the whole
virtual space (well, sort-of, half of it actually) represented by a PTE
page, and which contain an "indirect" bit indicating that this TLB entry
RPN points to an array of PTEs from which the TLB can then create direct
entries. Thus, in order to invalidate those when PTE pages are deleted,
we need the virtual address to pass to tlbilx or tlbivax instructions.
The old trick of sticking it somewhere in the PTE page struct page sucks
too much, the address is almost readily available in all call sites and
almost everybody implemets these as macros, so we may as well add the
argument everywhere. I added it to the pmd and pud variants for consistency.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Acked-by: David Howells <dhowells@redhat.com> [MN10300 & FRV]
Acked-by: Nick Piggin <npiggin@suse.de>
Acked-by: Martin Schwidefsky <schwidefsky@de.ibm.com> [s390]
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2009-07-22 07:44:28 +02:00
|
|
|
pmd_free_tlb(tlb, pmd, start);
|
2015-04-11 02:37:34 +02:00
|
|
|
mm_dec_nr_pmds(tlb->mm);
|
2006-04-28 07:02:51 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
static void hugetlb_free_pud_range(struct mmu_gather *tlb, pgd_t *pgd,
|
|
|
|
unsigned long addr, unsigned long end,
|
|
|
|
unsigned long floor, unsigned long ceiling)
|
|
|
|
{
|
|
|
|
pud_t *pud;
|
|
|
|
unsigned long next;
|
|
|
|
unsigned long start;
|
|
|
|
|
|
|
|
start = addr;
|
|
|
|
do {
|
2011-10-10 12:50:39 +02:00
|
|
|
pud = pud_offset(pgd, addr);
|
2006-04-28 07:02:51 +02:00
|
|
|
next = pud_addr_end(addr, end);
|
2014-11-05 17:27:41 +01:00
|
|
|
if (!is_hugepd(__hugepd(pud_val(*pud)))) {
|
2008-01-03 23:59:50 +01:00
|
|
|
if (pud_none_or_clear_bad(pud))
|
|
|
|
continue;
|
2008-07-24 06:27:56 +02:00
|
|
|
hugetlb_free_pmd_range(tlb, pud, addr, next, floor,
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
ceiling);
|
2008-01-03 23:59:50 +01:00
|
|
|
} else {
|
2011-10-10 12:50:39 +02:00
|
|
|
#ifdef CONFIG_PPC_FSL_BOOK3E
|
|
|
|
/*
|
|
|
|
* Increment next by the size of the huge mapping since
|
|
|
|
* there may be more than one entry at this level for a
|
|
|
|
* single hugepage, but all of them point to
|
|
|
|
* the same kmem cache that holds the hugepte.
|
|
|
|
*/
|
|
|
|
next = addr + (1 << hugepd_shift(*(hugepd_t *)pud));
|
|
|
|
#endif
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
free_hugepd_range(tlb, (hugepd_t *)pud, PUD_SHIFT,
|
|
|
|
addr, next, floor, ceiling);
|
2008-01-03 23:59:50 +01:00
|
|
|
}
|
2011-10-10 12:50:39 +02:00
|
|
|
} while (addr = next, addr != end);
|
2006-04-28 07:02:51 +02:00
|
|
|
|
|
|
|
start &= PGDIR_MASK;
|
|
|
|
if (start < floor)
|
|
|
|
return;
|
|
|
|
if (ceiling) {
|
|
|
|
ceiling &= PGDIR_MASK;
|
|
|
|
if (!ceiling)
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
if (end - 1 > ceiling - 1)
|
|
|
|
return;
|
|
|
|
|
|
|
|
pud = pud_offset(pgd, start);
|
|
|
|
pgd_clear(pgd);
|
mm: Pass virtual address to [__]p{te,ud,md}_free_tlb()
mm: Pass virtual address to [__]p{te,ud,md}_free_tlb()
Upcoming paches to support the new 64-bit "BookE" powerpc architecture
will need to have the virtual address corresponding to PTE page when
freeing it, due to the way the HW table walker works.
Basically, the TLB can be loaded with "large" pages that cover the whole
virtual space (well, sort-of, half of it actually) represented by a PTE
page, and which contain an "indirect" bit indicating that this TLB entry
RPN points to an array of PTEs from which the TLB can then create direct
entries. Thus, in order to invalidate those when PTE pages are deleted,
we need the virtual address to pass to tlbilx or tlbivax instructions.
The old trick of sticking it somewhere in the PTE page struct page sucks
too much, the address is almost readily available in all call sites and
almost everybody implemets these as macros, so we may as well add the
argument everywhere. I added it to the pmd and pud variants for consistency.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Acked-by: David Howells <dhowells@redhat.com> [MN10300 & FRV]
Acked-by: Nick Piggin <npiggin@suse.de>
Acked-by: Martin Schwidefsky <schwidefsky@de.ibm.com> [s390]
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2009-07-22 07:44:28 +02:00
|
|
|
pud_free_tlb(tlb, pud, start);
|
2006-04-28 07:02:51 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This function frees user-level page tables of a process.
|
|
|
|
*/
|
2008-07-24 06:27:10 +02:00
|
|
|
void hugetlb_free_pgd_range(struct mmu_gather *tlb,
|
2006-04-28 07:02:51 +02:00
|
|
|
unsigned long addr, unsigned long end,
|
|
|
|
unsigned long floor, unsigned long ceiling)
|
|
|
|
{
|
|
|
|
pgd_t *pgd;
|
|
|
|
unsigned long next;
|
|
|
|
|
|
|
|
/*
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
* Because there are a number of different possible pagetable
|
|
|
|
* layouts for hugepage ranges, we limit knowledge of how
|
|
|
|
* things should be laid out to the allocation path
|
|
|
|
* (huge_pte_alloc(), above). Everything else works out the
|
|
|
|
* structure as it goes from information in the hugepd
|
|
|
|
* pointers. That means that we can't here use the
|
|
|
|
* optimization used in the normal page free_pgd_range(), of
|
|
|
|
* checking whether we're actually covering a large enough
|
|
|
|
* range to have to do anything at the top level of the walk
|
|
|
|
* instead of at the bottom.
|
2006-04-28 07:02:51 +02:00
|
|
|
*
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
* To make sense of this, you should probably go read the big
|
|
|
|
* block comment at the top of the normal free_pgd_range(),
|
|
|
|
* too.
|
2006-04-28 07:02:51 +02:00
|
|
|
*/
|
|
|
|
|
|
|
|
do {
|
|
|
|
next = pgd_addr_end(addr, end);
|
2011-06-28 11:54:48 +02:00
|
|
|
pgd = pgd_offset(tlb->mm, addr);
|
2014-11-05 17:27:41 +01:00
|
|
|
if (!is_hugepd(__hugepd(pgd_val(*pgd)))) {
|
2008-09-05 03:49:54 +02:00
|
|
|
if (pgd_none_or_clear_bad(pgd))
|
|
|
|
continue;
|
|
|
|
hugetlb_free_pud_range(tlb, pgd, addr, next, floor, ceiling);
|
|
|
|
} else {
|
2011-10-10 12:50:40 +02:00
|
|
|
#ifdef CONFIG_PPC_FSL_BOOK3E
|
2011-06-28 11:54:48 +02:00
|
|
|
/*
|
|
|
|
* Increment next by the size of the huge mapping since
|
2011-10-10 12:50:40 +02:00
|
|
|
* there may be more than one entry at the pgd level
|
|
|
|
* for a single hugepage, but all of them point to the
|
|
|
|
* same kmem cache that holds the hugepte.
|
2011-06-28 11:54:48 +02:00
|
|
|
*/
|
|
|
|
next = addr + (1 << hugepd_shift(*(hugepd_t *)pgd));
|
|
|
|
#endif
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
free_hugepd_range(tlb, (hugepd_t *)pgd, PGDIR_SHIFT,
|
|
|
|
addr, next, floor, ceiling);
|
2008-09-05 03:49:54 +02:00
|
|
|
}
|
2011-06-28 11:54:48 +02:00
|
|
|
} while (addr = next, addr != end);
|
2005-04-17 00:20:36 +02:00
|
|
|
}
|
|
|
|
|
2015-03-30 07:11:03 +02:00
|
|
|
/*
|
|
|
|
* We are holding mmap_sem, so a parallel huge page collapse cannot run.
|
|
|
|
* To prevent hugepage split, disable irq.
|
|
|
|
*/
|
2005-04-17 00:20:36 +02:00
|
|
|
struct page *
|
|
|
|
follow_huge_addr(struct mm_struct *mm, unsigned long address, int write)
|
|
|
|
{
|
2015-05-11 08:28:29 +02:00
|
|
|
pte_t *ptep, pte;
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
unsigned shift;
|
2015-03-30 07:11:03 +02:00
|
|
|
unsigned long mask, flags;
|
2015-05-11 08:28:29 +02:00
|
|
|
struct page *page = ERR_PTR(-EINVAL);
|
|
|
|
|
|
|
|
local_irq_save(flags);
|
|
|
|
ptep = find_linux_pte_or_hugepte(mm->pgd, address, &shift);
|
|
|
|
if (!ptep)
|
|
|
|
goto no_page;
|
|
|
|
pte = READ_ONCE(*ptep);
|
2013-06-20 11:00:18 +02:00
|
|
|
/*
|
2015-05-11 08:28:29 +02:00
|
|
|
* Verify it is a huge page else bail.
|
2013-06-20 11:00:18 +02:00
|
|
|
* Transparent hugepages are handled by generic code. We can skip them
|
|
|
|
* here.
|
|
|
|
*/
|
2015-05-11 08:28:29 +02:00
|
|
|
if (!shift || pmd_trans_huge(__pmd(pte_val(pte))))
|
|
|
|
goto no_page;
|
2005-04-17 00:20:36 +02:00
|
|
|
|
2015-05-11 08:28:29 +02:00
|
|
|
if (!pte_present(pte)) {
|
|
|
|
page = NULL;
|
|
|
|
goto no_page;
|
2015-03-30 07:11:03 +02:00
|
|
|
}
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
mask = (1UL << shift) - 1;
|
2015-05-11 08:28:29 +02:00
|
|
|
page = pte_page(pte);
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
if (page)
|
|
|
|
page += (address & mask) / PAGE_SIZE;
|
2005-04-17 00:20:36 +02:00
|
|
|
|
2015-05-11 08:28:29 +02:00
|
|
|
no_page:
|
2015-03-30 07:11:03 +02:00
|
|
|
local_irq_restore(flags);
|
2005-04-17 00:20:36 +02:00
|
|
|
return page;
|
|
|
|
}
|
|
|
|
|
|
|
|
struct page *
|
|
|
|
follow_huge_pmd(struct mm_struct *mm, unsigned long address,
|
|
|
|
pmd_t *pmd, int write)
|
|
|
|
{
|
|
|
|
BUG();
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
mm/hugetlb: reduce arch dependent code around follow_huge_*
Currently we have many duplicates in definitions around
follow_huge_addr(), follow_huge_pmd(), and follow_huge_pud(), so this
patch tries to remove the m. The basic idea is to put the default
implementation for these functions in mm/hugetlb.c as weak symbols
(regardless of CONFIG_ARCH_WANT_GENERAL_HUGETL B), and to implement
arch-specific code only when the arch needs it.
For follow_huge_addr(), only powerpc and ia64 have their own
implementation, and in all other architectures this function just returns
ERR_PTR(-EINVAL). So this patch sets returning ERR_PTR(-EINVAL) as
default.
As for follow_huge_(pmd|pud)(), if (pmd|pud)_huge() is implemented to
always return 0 in your architecture (like in ia64 or sparc,) it's never
called (the callsite is optimized away) no matter how implemented it is.
So in such architectures, we don't need arch-specific implementation.
In some architecture (like mips, s390 and tile,) their current
arch-specific follow_huge_(pmd|pud)() are effectively identical with the
common code, so this patch lets these architecture use the common code.
One exception is metag, where pmd_huge() could return non-zero but it
expects follow_huge_pmd() to always return NULL. This means that we need
arch-specific implementation which returns NULL. This behavior looks
strange to me (because non-zero pmd_huge() implies that the architecture
supports PMD-based hugepage, so follow_huge_pmd() can/should return some
relevant value,) but that's beyond this cleanup patch, so let's keep it.
Justification of non-trivial changes:
- in s390, follow_huge_pmd() checks !MACHINE_HAS_HPAGE at first, and this
patch removes the check. This is OK because we can assume MACHINE_HAS_HPAGE
is true when follow_huge_pmd() can be called (note that pmd_huge() has
the same check and always returns 0 for !MACHINE_HAS_HPAGE.)
- in s390 and mips, we use HPAGE_MASK instead of PMD_MASK as done in common
code. This patch forces these archs use PMD_MASK, but it's OK because
they are identical in both archs.
In s390, both of HPAGE_SHIFT and PMD_SHIFT are 20.
In mips, HPAGE_SHIFT is defined as (PAGE_SHIFT + PAGE_SHIFT - 3) and
PMD_SHIFT is define as (PAGE_SHIFT + PAGE_SHIFT + PTE_ORDER - 3), but
PTE_ORDER is always 0, so these are identical.
Signed-off-by: Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>
Acked-by: Hugh Dickins <hughd@google.com>
Cc: James Hogan <james.hogan@imgtec.com>
Cc: David Rientjes <rientjes@google.com>
Cc: Mel Gorman <mel@csn.ul.ie>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@suse.cz>
Cc: Rik van Riel <riel@redhat.com>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Luiz Capitulino <lcapitulino@redhat.com>
Cc: Nishanth Aravamudan <nacc@linux.vnet.ibm.com>
Cc: Lee Schermerhorn <lee.schermerhorn@hp.com>
Cc: Steve Capper <steve.capper@linaro.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2015-02-12 00:25:15 +01:00
|
|
|
struct page *
|
|
|
|
follow_huge_pud(struct mm_struct *mm, unsigned long address,
|
|
|
|
pud_t *pud, int write)
|
|
|
|
{
|
|
|
|
BUG();
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
2009-11-23 21:03:40 +01:00
|
|
|
static unsigned long hugepte_addr_end(unsigned long addr, unsigned long end,
|
|
|
|
unsigned long sz)
|
|
|
|
{
|
|
|
|
unsigned long __boundary = (addr + sz) & ~(sz-1);
|
|
|
|
return (__boundary - 1 < end - 1) ? __boundary : end;
|
|
|
|
}
|
|
|
|
|
2014-11-05 17:27:41 +01:00
|
|
|
int gup_huge_pd(hugepd_t hugepd, unsigned long addr, unsigned pdshift,
|
|
|
|
unsigned long end, int write, struct page **pages, int *nr)
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
{
|
|
|
|
pte_t *ptep;
|
2014-11-05 17:27:41 +01:00
|
|
|
unsigned long sz = 1UL << hugepd_shift(hugepd);
|
2009-11-23 21:03:40 +01:00
|
|
|
unsigned long next;
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
|
|
|
|
ptep = hugepte_offset(hugepd, addr, pdshift);
|
|
|
|
do {
|
2009-11-23 21:03:40 +01:00
|
|
|
next = hugepte_addr_end(addr, end, sz);
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
if (!gup_hugepte(ptep, sz, addr, end, write, pages, nr))
|
|
|
|
return 0;
|
2009-11-23 21:03:40 +01:00
|
|
|
} while (ptep++, addr = next, addr != end);
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
|
|
|
|
return 1;
|
|
|
|
}
|
2005-04-17 00:20:36 +02:00
|
|
|
|
2011-10-10 12:50:36 +02:00
|
|
|
#ifdef CONFIG_PPC_MM_SLICES
|
2005-04-17 00:20:36 +02:00
|
|
|
unsigned long hugetlb_get_unmapped_area(struct file *file, unsigned long addr,
|
|
|
|
unsigned long len, unsigned long pgoff,
|
|
|
|
unsigned long flags)
|
|
|
|
{
|
2008-07-24 06:27:56 +02:00
|
|
|
struct hstate *hstate = hstate_file(file);
|
|
|
|
int mmu_psize = shift_to_mmu_psize(huge_page_shift(hstate));
|
2008-12-04 05:07:54 +01:00
|
|
|
|
2013-04-29 20:53:52 +02:00
|
|
|
return slice_get_unmapped_area(addr, len, flags, mmu_psize, 1);
|
2005-04-17 00:20:36 +02:00
|
|
|
}
|
2011-10-10 12:50:36 +02:00
|
|
|
#endif
|
2005-04-17 00:20:36 +02:00
|
|
|
|
2009-01-06 23:38:54 +01:00
|
|
|
unsigned long vma_mmu_pagesize(struct vm_area_struct *vma)
|
|
|
|
{
|
2011-09-20 21:58:10 +02:00
|
|
|
#ifdef CONFIG_PPC_MM_SLICES
|
2009-01-06 23:38:54 +01:00
|
|
|
unsigned int psize = get_slice_psize(vma->vm_mm, vma->vm_start);
|
|
|
|
|
|
|
|
return 1UL << mmu_psize_to_shift(psize);
|
2011-06-28 11:54:48 +02:00
|
|
|
#else
|
|
|
|
if (!is_vm_hugetlb_page(vma))
|
|
|
|
return PAGE_SIZE;
|
|
|
|
|
|
|
|
return huge_page_size(hstate_vma(vma));
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline bool is_power_of_4(unsigned long x)
|
|
|
|
{
|
|
|
|
if (is_power_of_2(x))
|
|
|
|
return (__ilog2(x) % 2) ? false : true;
|
|
|
|
return false;
|
2009-01-06 23:38:54 +01:00
|
|
|
}
|
|
|
|
|
2009-10-26 20:24:31 +01:00
|
|
|
static int __init add_huge_page_size(unsigned long long size)
|
2008-01-03 23:59:50 +01:00
|
|
|
{
|
2009-10-26 20:24:31 +01:00
|
|
|
int shift = __ffs(size);
|
|
|
|
int mmu_psize;
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
|
2008-01-03 23:59:50 +01:00
|
|
|
/* Check that it is a page size supported by the hardware and
|
2009-10-26 20:24:31 +01:00
|
|
|
* that it fits within pagetable and slice limits. */
|
2011-06-28 11:54:48 +02:00
|
|
|
#ifdef CONFIG_PPC_FSL_BOOK3E
|
|
|
|
if ((size < PAGE_SIZE) || !is_power_of_4(size))
|
|
|
|
return -EINVAL;
|
|
|
|
#else
|
2009-10-26 20:24:31 +01:00
|
|
|
if (!is_power_of_2(size)
|
|
|
|
|| (shift > SLICE_HIGH_SHIFT) || (shift <= PAGE_SHIFT))
|
|
|
|
return -EINVAL;
|
2011-06-28 11:54:48 +02:00
|
|
|
#endif
|
2008-07-24 06:27:55 +02:00
|
|
|
|
2009-10-26 20:24:31 +01:00
|
|
|
if ((mmu_psize = shift_to_mmu_psize(shift)) < 0)
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
BUG_ON(mmu_psize_defs[mmu_psize].shift != shift);
|
|
|
|
|
|
|
|
/* Return if huge page size has already been setup */
|
|
|
|
if (size_to_hstate(size))
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
hugetlb_add_hstate(shift - PAGE_SHIFT);
|
|
|
|
|
|
|
|
return 0;
|
2008-01-03 23:59:50 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
static int __init hugepage_setup_sz(char *str)
|
|
|
|
{
|
|
|
|
unsigned long long size;
|
|
|
|
|
|
|
|
size = memparse(str, &str);
|
|
|
|
|
2009-10-26 20:24:31 +01:00
|
|
|
if (add_huge_page_size(size) != 0)
|
2008-01-03 23:59:50 +01:00
|
|
|
printk(KERN_WARNING "Invalid huge page size specified(%llu)\n", size);
|
|
|
|
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
__setup("hugepagesz=", hugepage_setup_sz);
|
|
|
|
|
2011-10-10 12:50:40 +02:00
|
|
|
#ifdef CONFIG_PPC_FSL_BOOK3E
|
2011-06-28 11:54:48 +02:00
|
|
|
struct kmem_cache *hugepte_cache;
|
|
|
|
static int __init hugetlbpage_init(void)
|
|
|
|
{
|
|
|
|
int psize;
|
|
|
|
|
|
|
|
for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) {
|
|
|
|
unsigned shift;
|
|
|
|
|
|
|
|
if (!mmu_psize_defs[psize].shift)
|
|
|
|
continue;
|
|
|
|
|
|
|
|
shift = mmu_psize_to_shift(psize);
|
|
|
|
|
|
|
|
/* Don't treat normal page sizes as huge... */
|
|
|
|
if (shift != PAGE_SHIFT)
|
|
|
|
if (add_huge_page_size(1ULL << shift) < 0)
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Create a kmem cache for hugeptes. The bottom bits in the pte have
|
|
|
|
* size information encoded in them, so align them to allow this
|
|
|
|
*/
|
|
|
|
hugepte_cache = kmem_cache_create("hugepte-cache", sizeof(pte_t),
|
|
|
|
HUGEPD_SHIFT_MASK + 1, 0, NULL);
|
|
|
|
if (hugepte_cache == NULL)
|
|
|
|
panic("%s: Unable to create kmem cache for hugeptes\n",
|
|
|
|
__func__);
|
|
|
|
|
|
|
|
/* Default hpage size = 4M */
|
|
|
|
if (mmu_psize_defs[MMU_PAGE_4M].shift)
|
|
|
|
HPAGE_SHIFT = mmu_psize_defs[MMU_PAGE_4M].shift;
|
|
|
|
else
|
|
|
|
panic("%s: Unable to set default huge page size\n", __func__);
|
|
|
|
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
#else
|
2006-04-28 07:02:51 +02:00
|
|
|
static int __init hugetlbpage_init(void)
|
|
|
|
{
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
int psize;
|
2008-07-24 06:27:56 +02:00
|
|
|
|
2011-04-06 21:48:50 +02:00
|
|
|
if (!mmu_has_feature(MMU_FTR_16M_PAGE))
|
2006-04-28 07:02:51 +02:00
|
|
|
return -ENODEV;
|
2008-07-28 08:13:18 +02:00
|
|
|
|
2009-10-26 20:24:31 +01:00
|
|
|
for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) {
|
|
|
|
unsigned shift;
|
|
|
|
unsigned pdshift;
|
2008-07-24 06:27:56 +02:00
|
|
|
|
2009-10-26 20:24:31 +01:00
|
|
|
if (!mmu_psize_defs[psize].shift)
|
|
|
|
continue;
|
2008-07-28 08:13:18 +02:00
|
|
|
|
2009-10-26 20:24:31 +01:00
|
|
|
shift = mmu_psize_to_shift(psize);
|
|
|
|
|
|
|
|
if (add_huge_page_size(1ULL << shift) < 0)
|
|
|
|
continue;
|
|
|
|
|
|
|
|
if (shift < PMD_SHIFT)
|
|
|
|
pdshift = PMD_SHIFT;
|
|
|
|
else if (shift < PUD_SHIFT)
|
|
|
|
pdshift = PUD_SHIFT;
|
|
|
|
else
|
|
|
|
pdshift = PGDIR_SHIFT;
|
2013-04-28 11:37:30 +02:00
|
|
|
/*
|
|
|
|
* if we have pdshift and shift value same, we don't
|
|
|
|
* use pgt cache for hugepd.
|
|
|
|
*/
|
|
|
|
if (pdshift != shift) {
|
|
|
|
pgtable_cache_add(pdshift - shift, NULL);
|
|
|
|
if (!PGT_CACHE(pdshift - shift))
|
|
|
|
panic("hugetlbpage_init(): could not create "
|
|
|
|
"pgtable cache for %d bit pagesize\n", shift);
|
|
|
|
}
|
2008-07-24 06:27:56 +02:00
|
|
|
}
|
2006-04-28 07:02:51 +02:00
|
|
|
|
2009-10-26 20:24:31 +01:00
|
|
|
/* Set default large page size. Currently, we pick 16M or 1M
|
|
|
|
* depending on what is available
|
|
|
|
*/
|
|
|
|
if (mmu_psize_defs[MMU_PAGE_16M].shift)
|
|
|
|
HPAGE_SHIFT = mmu_psize_defs[MMU_PAGE_16M].shift;
|
|
|
|
else if (mmu_psize_defs[MMU_PAGE_1M].shift)
|
|
|
|
HPAGE_SHIFT = mmu_psize_defs[MMU_PAGE_1M].shift;
|
|
|
|
|
2006-04-28 07:02:51 +02:00
|
|
|
return 0;
|
|
|
|
}
|
2011-06-28 11:54:48 +02:00
|
|
|
#endif
|
2015-05-02 02:08:21 +02:00
|
|
|
arch_initcall(hugetlbpage_init);
|
2009-10-26 20:24:31 +01:00
|
|
|
|
|
|
|
void flush_dcache_icache_hugepage(struct page *page)
|
|
|
|
{
|
|
|
|
int i;
|
2011-06-28 11:54:48 +02:00
|
|
|
void *start;
|
2009-10-26 20:24:31 +01:00
|
|
|
|
|
|
|
BUG_ON(!PageCompound(page));
|
|
|
|
|
2011-06-28 11:54:48 +02:00
|
|
|
for (i = 0; i < (1UL << compound_order(page)); i++) {
|
|
|
|
if (!PageHighMem(page)) {
|
|
|
|
__flush_dcache_icache(page_address(page+i));
|
|
|
|
} else {
|
2011-11-25 16:14:16 +01:00
|
|
|
start = kmap_atomic(page+i);
|
2011-06-28 11:54:48 +02:00
|
|
|
__flush_dcache_icache(start);
|
2011-11-25 16:14:16 +01:00
|
|
|
kunmap_atomic(start);
|
2011-06-28 11:54:48 +02:00
|
|
|
}
|
|
|
|
}
|
2009-10-26 20:24:31 +01:00
|
|
|
}
|
2013-06-20 11:00:16 +02:00
|
|
|
|
|
|
|
#endif /* CONFIG_HUGETLB_PAGE */
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We have 4 cases for pgds and pmds:
|
|
|
|
* (1) invalid (all zeroes)
|
|
|
|
* (2) pointer to next table, as normal; bottom 6 bits == 0
|
|
|
|
* (3) leaf pte for huge page, bottom two bits != 00
|
|
|
|
* (4) hugepd pointer, bottom two bits == 00, next 4 bits indicate size of table
|
2013-06-20 11:00:22 +02:00
|
|
|
*
|
|
|
|
* So long as we atomically load page table pointers we are safe against teardown,
|
|
|
|
* we can follow the address down to the the page and take a ref on it.
|
2015-03-30 07:11:03 +02:00
|
|
|
* This function need to be called with interrupts disabled. We use this variant
|
|
|
|
* when we have MSR[EE] = 0 but the paca->soft_enabled = 1
|
2013-06-20 11:00:16 +02:00
|
|
|
*/
|
2013-06-20 11:00:22 +02:00
|
|
|
|
2015-03-30 07:11:03 +02:00
|
|
|
pte_t *__find_linux_pte_or_hugepte(pgd_t *pgdir, unsigned long ea,
|
|
|
|
unsigned *shift)
|
2013-06-20 11:00:16 +02:00
|
|
|
{
|
2013-06-20 11:00:22 +02:00
|
|
|
pgd_t pgd, *pgdp;
|
|
|
|
pud_t pud, *pudp;
|
|
|
|
pmd_t pmd, *pmdp;
|
2013-06-20 11:00:16 +02:00
|
|
|
pte_t *ret_pte;
|
|
|
|
hugepd_t *hpdp = NULL;
|
|
|
|
unsigned pdshift = PGDIR_SHIFT;
|
|
|
|
|
|
|
|
if (shift)
|
|
|
|
*shift = 0;
|
|
|
|
|
2013-06-20 11:00:22 +02:00
|
|
|
pgdp = pgdir + pgd_index(ea);
|
2015-03-25 10:11:57 +01:00
|
|
|
pgd = READ_ONCE(*pgdp);
|
2013-06-20 11:00:17 +02:00
|
|
|
/*
|
2013-06-20 11:00:22 +02:00
|
|
|
* Always operate on the local stack value. This make sure the
|
|
|
|
* value don't get updated by a parallel THP split/collapse,
|
|
|
|
* page fault or a page unmap. The return pte_t * is still not
|
|
|
|
* stable. So should be checked there for above conditions.
|
2013-06-20 11:00:17 +02:00
|
|
|
*/
|
2013-06-20 11:00:22 +02:00
|
|
|
if (pgd_none(pgd))
|
2013-06-20 11:00:17 +02:00
|
|
|
return NULL;
|
2013-06-20 11:00:22 +02:00
|
|
|
else if (pgd_huge(pgd)) {
|
|
|
|
ret_pte = (pte_t *) pgdp;
|
2013-06-20 11:00:16 +02:00
|
|
|
goto out;
|
2014-11-05 17:27:41 +01:00
|
|
|
} else if (is_hugepd(__hugepd(pgd_val(pgd))))
|
2013-06-20 11:00:22 +02:00
|
|
|
hpdp = (hugepd_t *)&pgd;
|
2013-06-20 11:00:17 +02:00
|
|
|
else {
|
2013-06-20 11:00:22 +02:00
|
|
|
/*
|
|
|
|
* Even if we end up with an unmap, the pgtable will not
|
|
|
|
* be freed, because we do an rcu free and here we are
|
|
|
|
* irq disabled
|
|
|
|
*/
|
2013-06-20 11:00:16 +02:00
|
|
|
pdshift = PUD_SHIFT;
|
2013-06-20 11:00:22 +02:00
|
|
|
pudp = pud_offset(&pgd, ea);
|
2015-01-06 22:47:41 +01:00
|
|
|
pud = READ_ONCE(*pudp);
|
2013-06-20 11:00:16 +02:00
|
|
|
|
2013-06-20 11:00:22 +02:00
|
|
|
if (pud_none(pud))
|
2013-06-20 11:00:17 +02:00
|
|
|
return NULL;
|
2013-06-20 11:00:22 +02:00
|
|
|
else if (pud_huge(pud)) {
|
|
|
|
ret_pte = (pte_t *) pudp;
|
2013-06-20 11:00:16 +02:00
|
|
|
goto out;
|
2014-11-05 17:27:41 +01:00
|
|
|
} else if (is_hugepd(__hugepd(pud_val(pud))))
|
2013-06-20 11:00:22 +02:00
|
|
|
hpdp = (hugepd_t *)&pud;
|
2013-06-20 11:00:17 +02:00
|
|
|
else {
|
2013-06-20 11:00:16 +02:00
|
|
|
pdshift = PMD_SHIFT;
|
2013-06-20 11:00:22 +02:00
|
|
|
pmdp = pmd_offset(&pud, ea);
|
2015-01-06 22:47:41 +01:00
|
|
|
pmd = READ_ONCE(*pmdp);
|
2013-06-20 11:00:17 +02:00
|
|
|
/*
|
|
|
|
* A hugepage collapse is captured by pmd_none, because
|
|
|
|
* it mark the pmd none and do a hpte invalidate.
|
|
|
|
*
|
2015-03-30 07:11:04 +02:00
|
|
|
* We don't worry about pmd_trans_splitting here, The
|
|
|
|
* caller if it needs to handle the splitting case
|
|
|
|
* should check for that.
|
2013-06-20 11:00:17 +02:00
|
|
|
*/
|
2015-03-30 07:11:04 +02:00
|
|
|
if (pmd_none(pmd))
|
2013-06-20 11:00:17 +02:00
|
|
|
return NULL;
|
2013-06-20 11:00:16 +02:00
|
|
|
|
2013-06-20 11:00:22 +02:00
|
|
|
if (pmd_huge(pmd) || pmd_large(pmd)) {
|
|
|
|
ret_pte = (pte_t *) pmdp;
|
2013-06-20 11:00:16 +02:00
|
|
|
goto out;
|
2014-11-05 17:27:41 +01:00
|
|
|
} else if (is_hugepd(__hugepd(pmd_val(pmd))))
|
2013-06-20 11:00:22 +02:00
|
|
|
hpdp = (hugepd_t *)&pmd;
|
2013-06-20 11:00:17 +02:00
|
|
|
else
|
2013-06-20 11:00:22 +02:00
|
|
|
return pte_offset_kernel(&pmd, ea);
|
2013-06-20 11:00:16 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
if (!hpdp)
|
|
|
|
return NULL;
|
|
|
|
|
2014-11-05 17:27:41 +01:00
|
|
|
ret_pte = hugepte_offset(*hpdp, ea, pdshift);
|
2013-06-20 11:00:16 +02:00
|
|
|
pdshift = hugepd_shift(*hpdp);
|
|
|
|
out:
|
|
|
|
if (shift)
|
|
|
|
*shift = pdshift;
|
|
|
|
return ret_pte;
|
|
|
|
}
|
2015-03-30 07:11:03 +02:00
|
|
|
EXPORT_SYMBOL_GPL(__find_linux_pte_or_hugepte);
|
2013-06-20 11:00:16 +02:00
|
|
|
|
|
|
|
int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr,
|
|
|
|
unsigned long end, int write, struct page **pages, int *nr)
|
|
|
|
{
|
|
|
|
unsigned long mask;
|
|
|
|
unsigned long pte_end;
|
|
|
|
struct page *head, *page, *tail;
|
|
|
|
pte_t pte;
|
|
|
|
int refs;
|
|
|
|
|
|
|
|
pte_end = (addr + sz) & ~(sz-1);
|
|
|
|
if (pte_end < end)
|
|
|
|
end = pte_end;
|
|
|
|
|
2015-03-25 10:11:57 +01:00
|
|
|
pte = READ_ONCE(*ptep);
|
2013-06-20 11:00:16 +02:00
|
|
|
mask = _PAGE_PRESENT | _PAGE_USER;
|
|
|
|
if (write)
|
|
|
|
mask |= _PAGE_RW;
|
|
|
|
|
|
|
|
if ((pte_val(pte) & mask) != mask)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
/* hugepages are never "special" */
|
|
|
|
VM_BUG_ON(!pfn_valid(pte_pfn(pte)));
|
|
|
|
|
|
|
|
refs = 0;
|
|
|
|
head = pte_page(pte);
|
|
|
|
|
|
|
|
page = head + ((addr & (sz-1)) >> PAGE_SHIFT);
|
|
|
|
tail = page;
|
|
|
|
do {
|
|
|
|
VM_BUG_ON(compound_head(page) != head);
|
|
|
|
pages[*nr] = page;
|
|
|
|
(*nr)++;
|
|
|
|
page++;
|
|
|
|
refs++;
|
|
|
|
} while (addr += PAGE_SIZE, addr != end);
|
|
|
|
|
|
|
|
if (!page_cache_add_speculative(head, refs)) {
|
|
|
|
*nr -= refs;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (unlikely(pte_val(pte) != pte_val(*ptep))) {
|
|
|
|
/* Could be optimized better */
|
|
|
|
*nr -= refs;
|
|
|
|
while (refs--)
|
|
|
|
put_page(head);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Any tail page need their mapcount reference taken before we
|
|
|
|
* return.
|
|
|
|
*/
|
|
|
|
while (refs--) {
|
|
|
|
if (PageTail(tail))
|
|
|
|
get_huge_page_tail(tail);
|
|
|
|
tail++;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 1;
|
|
|
|
}
|