2005-04-17 00:20:36 +02:00
|
|
|
/*
|
2011-06-28 11:54:48 +02:00
|
|
|
* PPC Huge TLB Page Support for Kernel.
|
2005-04-17 00:20:36 +02:00
|
|
|
*
|
|
|
|
* Copyright (C) 2003 David Gibson, IBM Corporation.
|
2011-06-28 11:54:48 +02:00
|
|
|
* Copyright (C) 2011 Becky Bruce, Freescale Semiconductor
|
2005-04-17 00:20:36 +02:00
|
|
|
*
|
|
|
|
* Based on the IA-32 version:
|
|
|
|
* Copyright (C) 2002, Rohit Seth <rohit.seth@intel.com>
|
|
|
|
*/
|
|
|
|
|
|
|
|
#include <linux/mm.h>
|
2009-10-26 20:24:31 +01:00
|
|
|
#include <linux/io.h>
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 09:04:11 +01:00
|
|
|
#include <linux/slab.h>
|
2005-04-17 00:20:36 +02:00
|
|
|
#include <linux/hugetlb.h>
|
KVM: PPC: Implement MMU notifiers for Book3S HV guests
This adds the infrastructure to enable us to page out pages underneath
a Book3S HV guest, on processors that support virtualized partition
memory, that is, POWER7. Instead of pinning all the guest's pages,
we now look in the host userspace Linux page tables to find the
mapping for a given guest page. Then, if the userspace Linux PTE
gets invalidated, kvm_unmap_hva() gets called for that address, and
we replace all the guest HPTEs that refer to that page with absent
HPTEs, i.e. ones with the valid bit clear and the HPTE_V_ABSENT bit
set, which will cause an HDSI when the guest tries to access them.
Finally, the page fault handler is extended to reinstantiate the
guest HPTE when the guest tries to access a page which has been paged
out.
Since we can't intercept the guest DSI and ISI interrupts on PPC970,
we still have to pin all the guest pages on PPC970. We have a new flag,
kvm->arch.using_mmu_notifiers, that indicates whether we can page
guest pages out. If it is not set, the MMU notifier callbacks do
nothing and everything operates as before.
Signed-off-by: Paul Mackerras <paulus@samba.org>
Signed-off-by: Alexander Graf <agraf@suse.de>
Signed-off-by: Avi Kivity <avi@redhat.com>
2011-12-12 13:38:05 +01:00
|
|
|
#include <linux/export.h>
|
2011-06-28 11:54:48 +02:00
|
|
|
#include <linux/of_fdt.h>
|
|
|
|
#include <linux/memblock.h>
|
|
|
|
#include <linux/bootmem.h>
|
2011-11-24 10:40:07 +01:00
|
|
|
#include <linux/moduleparam.h>
|
2009-10-26 20:24:31 +01:00
|
|
|
#include <asm/pgtable.h>
|
2005-04-17 00:20:36 +02:00
|
|
|
#include <asm/pgalloc.h>
|
|
|
|
#include <asm/tlb.h>
|
2011-06-28 11:54:48 +02:00
|
|
|
#include <asm/setup.h>
|
2013-06-20 11:00:16 +02:00
|
|
|
#include <asm/hugetlb.h>
|
|
|
|
|
|
|
|
#ifdef CONFIG_HUGETLB_PAGE
|
2005-04-17 00:20:36 +02:00
|
|
|
|
2008-07-24 06:27:55 +02:00
|
|
|
#define PAGE_SHIFT_64K 16
|
|
|
|
#define PAGE_SHIFT_16M 24
|
|
|
|
#define PAGE_SHIFT_16G 34
|
2008-01-03 23:59:50 +01:00
|
|
|
|
2011-06-28 11:54:48 +02:00
|
|
|
unsigned int HPAGE_SHIFT;
|
2008-07-24 06:27:53 +02:00
|
|
|
|
2011-06-28 11:54:48 +02:00
|
|
|
/*
|
|
|
|
* Tracks gpages after the device tree is scanned and before the
|
2011-10-10 12:50:43 +02:00
|
|
|
* huge_boot_pages list is ready. On non-Freescale implementations, this is
|
|
|
|
* just used to track 16G pages and so is a single array. FSL-based
|
|
|
|
* implementations may have more than one gpage size, so we need multiple
|
|
|
|
* arrays
|
2011-06-28 11:54:48 +02:00
|
|
|
*/
|
2011-10-10 12:50:40 +02:00
|
|
|
#ifdef CONFIG_PPC_FSL_BOOK3E
|
2011-06-28 11:54:48 +02:00
|
|
|
#define MAX_NUMBER_GPAGES 128
|
|
|
|
struct psize_gpages {
|
|
|
|
u64 gpage_list[MAX_NUMBER_GPAGES];
|
|
|
|
unsigned int nr_gpages;
|
|
|
|
};
|
|
|
|
static struct psize_gpages gpage_freearray[MMU_PAGE_COUNT];
|
2011-10-10 12:50:40 +02:00
|
|
|
#else
|
|
|
|
#define MAX_NUMBER_GPAGES 1024
|
|
|
|
static u64 gpage_freearray[MAX_NUMBER_GPAGES];
|
|
|
|
static unsigned nr_gpages;
|
2011-06-28 11:54:48 +02:00
|
|
|
#endif
|
2006-04-28 07:02:51 +02:00
|
|
|
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
#define hugepd_none(hpd) ((hpd).pd == 0)
|
|
|
|
|
2013-04-28 11:37:30 +02:00
|
|
|
#ifdef CONFIG_PPC_BOOK3S_64
|
|
|
|
/*
|
|
|
|
* At this point we do the placement change only for BOOK3S 64. This would
|
|
|
|
* possibly work on other subarchs.
|
|
|
|
*/
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We have PGD_INDEX_SIZ = 12 and PTE_INDEX_SIZE = 8, so that we can have
|
|
|
|
* 16GB hugepage pte in PGD and 16MB hugepage pte at PMD;
|
|
|
|
*/
|
|
|
|
int pmd_huge(pmd_t pmd)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* leaf pte for huge page, bottom two bits != 00
|
|
|
|
*/
|
|
|
|
return ((pmd_val(pmd) & 0x3) != 0x0);
|
|
|
|
}
|
|
|
|
|
|
|
|
int pud_huge(pud_t pud)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* leaf pte for huge page, bottom two bits != 00
|
|
|
|
*/
|
|
|
|
return ((pud_val(pud) & 0x3) != 0x0);
|
|
|
|
}
|
|
|
|
|
|
|
|
int pgd_huge(pgd_t pgd)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* leaf pte for huge page, bottom two bits != 00
|
|
|
|
*/
|
|
|
|
return ((pgd_val(pgd) & 0x3) != 0x0);
|
|
|
|
}
|
2013-09-11 23:22:11 +02:00
|
|
|
|
|
|
|
int pmd_huge_support(void)
|
|
|
|
{
|
|
|
|
return 1;
|
|
|
|
}
|
2013-04-28 11:37:30 +02:00
|
|
|
#else
|
|
|
|
int pmd_huge(pmd_t pmd)
|
|
|
|
{
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
int pud_huge(pud_t pud)
|
|
|
|
{
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
int pgd_huge(pgd_t pgd)
|
|
|
|
{
|
|
|
|
return 0;
|
|
|
|
}
|
2013-09-11 23:22:11 +02:00
|
|
|
|
|
|
|
int pmd_huge_support(void)
|
|
|
|
{
|
|
|
|
return 0;
|
|
|
|
}
|
2013-04-28 11:37:30 +02:00
|
|
|
#endif
|
|
|
|
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr)
|
|
|
|
{
|
2013-06-20 11:00:18 +02:00
|
|
|
/* Only called for hugetlbfs pages, hence can ignore THP */
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
return find_linux_pte_or_hugepte(mm->pgd, addr, NULL);
|
|
|
|
}
|
|
|
|
|
2006-04-28 07:02:51 +02:00
|
|
|
static int __hugepte_alloc(struct mm_struct *mm, hugepd_t *hpdp,
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
unsigned long address, unsigned pdshift, unsigned pshift)
|
2006-04-28 07:02:51 +02:00
|
|
|
{
|
2011-06-28 11:54:48 +02:00
|
|
|
struct kmem_cache *cachep;
|
|
|
|
pte_t *new;
|
|
|
|
|
2011-10-10 12:50:40 +02:00
|
|
|
#ifdef CONFIG_PPC_FSL_BOOK3E
|
2011-06-28 11:54:48 +02:00
|
|
|
int i;
|
|
|
|
int num_hugepd = 1 << (pshift - pdshift);
|
|
|
|
cachep = hugepte_cache;
|
2011-10-10 12:50:40 +02:00
|
|
|
#else
|
|
|
|
cachep = PGT_CACHE(pdshift - pshift);
|
2011-06-28 11:54:48 +02:00
|
|
|
#endif
|
|
|
|
|
|
|
|
new = kmem_cache_zalloc(cachep, GFP_KERNEL|__GFP_REPEAT);
|
2006-04-28 07:02:51 +02:00
|
|
|
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
BUG_ON(pshift > HUGEPD_SHIFT_MASK);
|
|
|
|
BUG_ON((unsigned long)new & HUGEPD_SHIFT_MASK);
|
|
|
|
|
2006-04-28 07:02:51 +02:00
|
|
|
if (! new)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
spin_lock(&mm->page_table_lock);
|
2011-10-10 12:50:40 +02:00
|
|
|
#ifdef CONFIG_PPC_FSL_BOOK3E
|
2011-06-28 11:54:48 +02:00
|
|
|
/*
|
|
|
|
* We have multiple higher-level entries that point to the same
|
|
|
|
* actual pte location. Fill in each as we go and backtrack on error.
|
|
|
|
* We need all of these so the DTLB pgtable walk code can find the
|
|
|
|
* right higher-level entry without knowing if it's a hugepage or not.
|
|
|
|
*/
|
|
|
|
for (i = 0; i < num_hugepd; i++, hpdp++) {
|
|
|
|
if (unlikely(!hugepd_none(*hpdp)))
|
|
|
|
break;
|
|
|
|
else
|
2013-04-28 11:37:29 +02:00
|
|
|
/* We use the old format for PPC_FSL_BOOK3E */
|
2011-06-28 11:54:48 +02:00
|
|
|
hpdp->pd = ((unsigned long)new & ~PD_HUGE) | pshift;
|
|
|
|
}
|
|
|
|
/* If we bailed from the for loop early, an error occurred, clean up */
|
|
|
|
if (i < num_hugepd) {
|
|
|
|
for (i = i - 1 ; i >= 0; i--, hpdp--)
|
|
|
|
hpdp->pd = 0;
|
|
|
|
kmem_cache_free(cachep, new);
|
|
|
|
}
|
2011-10-10 12:50:39 +02:00
|
|
|
#else
|
|
|
|
if (!hugepd_none(*hpdp))
|
|
|
|
kmem_cache_free(cachep, new);
|
2013-04-28 11:37:29 +02:00
|
|
|
else {
|
|
|
|
#ifdef CONFIG_PPC_BOOK3S_64
|
|
|
|
hpdp->pd = (unsigned long)new |
|
|
|
|
(shift_to_mmu_psize(pshift) << 2);
|
|
|
|
#else
|
2011-10-10 12:50:39 +02:00
|
|
|
hpdp->pd = ((unsigned long)new & ~PD_HUGE) | pshift;
|
2013-04-28 11:37:29 +02:00
|
|
|
#endif
|
|
|
|
}
|
2011-06-28 11:54:48 +02:00
|
|
|
#endif
|
2006-04-28 07:02:51 +02:00
|
|
|
spin_unlock(&mm->page_table_lock);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2011-10-10 12:50:39 +02:00
|
|
|
/*
|
|
|
|
* These macros define how to determine which level of the page table holds
|
|
|
|
* the hpdp.
|
|
|
|
*/
|
|
|
|
#ifdef CONFIG_PPC_FSL_BOOK3E
|
|
|
|
#define HUGEPD_PGD_SHIFT PGDIR_SHIFT
|
|
|
|
#define HUGEPD_PUD_SHIFT PUD_SHIFT
|
|
|
|
#else
|
|
|
|
#define HUGEPD_PGD_SHIFT PUD_SHIFT
|
|
|
|
#define HUGEPD_PUD_SHIFT PMD_SHIFT
|
|
|
|
#endif
|
|
|
|
|
2013-04-28 11:37:30 +02:00
|
|
|
#ifdef CONFIG_PPC_BOOK3S_64
|
|
|
|
/*
|
|
|
|
* At this point we do the placement change only for BOOK3S 64. This would
|
|
|
|
* possibly work on other subarchs.
|
|
|
|
*/
|
|
|
|
pte_t *huge_pte_alloc(struct mm_struct *mm, unsigned long addr, unsigned long sz)
|
|
|
|
{
|
|
|
|
pgd_t *pg;
|
|
|
|
pud_t *pu;
|
|
|
|
pmd_t *pm;
|
|
|
|
hugepd_t *hpdp = NULL;
|
|
|
|
unsigned pshift = __ffs(sz);
|
|
|
|
unsigned pdshift = PGDIR_SHIFT;
|
|
|
|
|
|
|
|
addr &= ~(sz-1);
|
|
|
|
pg = pgd_offset(mm, addr);
|
|
|
|
|
|
|
|
if (pshift == PGDIR_SHIFT)
|
|
|
|
/* 16GB huge page */
|
|
|
|
return (pte_t *) pg;
|
|
|
|
else if (pshift > PUD_SHIFT)
|
|
|
|
/*
|
|
|
|
* We need to use hugepd table
|
|
|
|
*/
|
|
|
|
hpdp = (hugepd_t *)pg;
|
|
|
|
else {
|
|
|
|
pdshift = PUD_SHIFT;
|
|
|
|
pu = pud_alloc(mm, pg, addr);
|
|
|
|
if (pshift == PUD_SHIFT)
|
|
|
|
return (pte_t *)pu;
|
|
|
|
else if (pshift > PMD_SHIFT)
|
|
|
|
hpdp = (hugepd_t *)pu;
|
|
|
|
else {
|
|
|
|
pdshift = PMD_SHIFT;
|
|
|
|
pm = pmd_alloc(mm, pu, addr);
|
|
|
|
if (pshift == PMD_SHIFT)
|
|
|
|
/* 16MB hugepage */
|
|
|
|
return (pte_t *)pm;
|
|
|
|
else
|
|
|
|
hpdp = (hugepd_t *)pm;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
if (!hpdp)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
BUG_ON(!hugepd_none(*hpdp) && !hugepd_ok(*hpdp));
|
|
|
|
|
|
|
|
if (hugepd_none(*hpdp) && __hugepte_alloc(mm, hpdp, addr, pdshift, pshift))
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
return hugepte_offset(hpdp, addr, pdshift);
|
|
|
|
}
|
|
|
|
|
|
|
|
#else
|
|
|
|
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
pte_t *huge_pte_alloc(struct mm_struct *mm, unsigned long addr, unsigned long sz)
|
2008-09-05 03:49:54 +02:00
|
|
|
{
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
pgd_t *pg;
|
|
|
|
pud_t *pu;
|
|
|
|
pmd_t *pm;
|
|
|
|
hugepd_t *hpdp = NULL;
|
|
|
|
unsigned pshift = __ffs(sz);
|
|
|
|
unsigned pdshift = PGDIR_SHIFT;
|
|
|
|
|
|
|
|
addr &= ~(sz-1);
|
|
|
|
|
|
|
|
pg = pgd_offset(mm, addr);
|
2011-10-10 12:50:39 +02:00
|
|
|
|
|
|
|
if (pshift >= HUGEPD_PGD_SHIFT) {
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
hpdp = (hugepd_t *)pg;
|
|
|
|
} else {
|
|
|
|
pdshift = PUD_SHIFT;
|
|
|
|
pu = pud_alloc(mm, pg, addr);
|
2011-10-10 12:50:39 +02:00
|
|
|
if (pshift >= HUGEPD_PUD_SHIFT) {
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
hpdp = (hugepd_t *)pu;
|
|
|
|
} else {
|
|
|
|
pdshift = PMD_SHIFT;
|
|
|
|
pm = pmd_alloc(mm, pu, addr);
|
|
|
|
hpdp = (hugepd_t *)pm;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!hpdp)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
BUG_ON(!hugepd_none(*hpdp) && !hugepd_ok(*hpdp));
|
|
|
|
|
|
|
|
if (hugepd_none(*hpdp) && __hugepte_alloc(mm, hpdp, addr, pdshift, pshift))
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
return hugepte_offset(hpdp, addr, pdshift);
|
2008-01-03 23:59:50 +01:00
|
|
|
}
|
2013-04-28 11:37:30 +02:00
|
|
|
#endif
|
2008-01-03 23:59:50 +01:00
|
|
|
|
2011-10-10 12:50:40 +02:00
|
|
|
#ifdef CONFIG_PPC_FSL_BOOK3E
|
2008-07-24 06:27:54 +02:00
|
|
|
/* Build list of addresses of gigantic pages. This function is used in early
|
|
|
|
* boot before the buddy or bootmem allocator is setup.
|
|
|
|
*/
|
2011-06-28 11:54:48 +02:00
|
|
|
void add_gpage(u64 addr, u64 page_size, unsigned long number_of_pages)
|
|
|
|
{
|
|
|
|
unsigned int idx = shift_to_mmu_psize(__ffs(page_size));
|
|
|
|
int i;
|
|
|
|
|
|
|
|
if (addr == 0)
|
|
|
|
return;
|
|
|
|
|
|
|
|
gpage_freearray[idx].nr_gpages = number_of_pages;
|
|
|
|
|
|
|
|
for (i = 0; i < number_of_pages; i++) {
|
|
|
|
gpage_freearray[idx].gpage_list[i] = addr;
|
|
|
|
addr += page_size;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Moves the gigantic page addresses from the temporary list to the
|
|
|
|
* huge_boot_pages list.
|
|
|
|
*/
|
|
|
|
int alloc_bootmem_huge_page(struct hstate *hstate)
|
|
|
|
{
|
|
|
|
struct huge_bootmem_page *m;
|
2013-07-04 00:02:43 +02:00
|
|
|
int idx = shift_to_mmu_psize(huge_page_shift(hstate));
|
2011-06-28 11:54:48 +02:00
|
|
|
int nr_gpages = gpage_freearray[idx].nr_gpages;
|
|
|
|
|
|
|
|
if (nr_gpages == 0)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
#ifdef CONFIG_HIGHMEM
|
|
|
|
/*
|
|
|
|
* If gpages can be in highmem we can't use the trick of storing the
|
|
|
|
* data structure in the page; allocate space for this
|
|
|
|
*/
|
|
|
|
m = alloc_bootmem(sizeof(struct huge_bootmem_page));
|
|
|
|
m->phys = gpage_freearray[idx].gpage_list[--nr_gpages];
|
|
|
|
#else
|
|
|
|
m = phys_to_virt(gpage_freearray[idx].gpage_list[--nr_gpages]);
|
|
|
|
#endif
|
|
|
|
|
|
|
|
list_add(&m->list, &huge_boot_pages);
|
|
|
|
gpage_freearray[idx].nr_gpages = nr_gpages;
|
|
|
|
gpage_freearray[idx].gpage_list[nr_gpages] = 0;
|
|
|
|
m->hstate = hstate;
|
|
|
|
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
/*
|
|
|
|
* Scan the command line hugepagesz= options for gigantic pages; store those in
|
|
|
|
* a list that we use to allocate the memory once all options are parsed.
|
|
|
|
*/
|
|
|
|
|
|
|
|
unsigned long gpage_npages[MMU_PAGE_COUNT];
|
|
|
|
|
2012-05-07 16:32:22 +02:00
|
|
|
static int __init do_gpage_early_setup(char *param, char *val,
|
|
|
|
const char *unused)
|
2011-06-28 11:54:48 +02:00
|
|
|
{
|
|
|
|
static phys_addr_t size;
|
|
|
|
unsigned long npages;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The hugepagesz and hugepages cmdline options are interleaved. We
|
|
|
|
* use the size variable to keep track of whether or not this was done
|
|
|
|
* properly and skip over instances where it is incorrect. Other
|
|
|
|
* command-line parsing code will issue warnings, so we don't need to.
|
|
|
|
*
|
|
|
|
*/
|
|
|
|
if ((strcmp(param, "default_hugepagesz") == 0) ||
|
|
|
|
(strcmp(param, "hugepagesz") == 0)) {
|
|
|
|
size = memparse(val, NULL);
|
|
|
|
} else if (strcmp(param, "hugepages") == 0) {
|
|
|
|
if (size != 0) {
|
|
|
|
if (sscanf(val, "%lu", &npages) <= 0)
|
|
|
|
npages = 0;
|
|
|
|
gpage_npages[shift_to_mmu_psize(__ffs(size))] = npages;
|
|
|
|
size = 0;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This function allocates physical space for pages that are larger than the
|
|
|
|
* buddy allocator can handle. We want to allocate these in highmem because
|
|
|
|
* the amount of lowmem is limited. This means that this function MUST be
|
|
|
|
* called before lowmem_end_addr is set up in MMU_init() in order for the lmb
|
|
|
|
* allocate to grab highmem.
|
|
|
|
*/
|
|
|
|
void __init reserve_hugetlb_gpages(void)
|
|
|
|
{
|
|
|
|
static __initdata char cmdline[COMMAND_LINE_SIZE];
|
|
|
|
phys_addr_t size, base;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
strlcpy(cmdline, boot_command_line, COMMAND_LINE_SIZE);
|
2012-03-26 04:20:51 +02:00
|
|
|
parse_args("hugetlb gpages", cmdline, NULL, 0, 0, 0,
|
|
|
|
&do_gpage_early_setup);
|
2011-06-28 11:54:48 +02:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Walk gpage list in reverse, allocating larger page sizes first.
|
|
|
|
* Skip over unsupported sizes, or sizes that have 0 gpages allocated.
|
|
|
|
* When we reach the point in the list where pages are no longer
|
|
|
|
* considered gpages, we're done.
|
|
|
|
*/
|
|
|
|
for (i = MMU_PAGE_COUNT-1; i >= 0; i--) {
|
|
|
|
if (mmu_psize_defs[i].shift == 0 || gpage_npages[i] == 0)
|
|
|
|
continue;
|
|
|
|
else if (mmu_psize_to_shift(i) < (MAX_ORDER + PAGE_SHIFT))
|
|
|
|
break;
|
|
|
|
|
|
|
|
size = (phys_addr_t)(1ULL << mmu_psize_to_shift(i));
|
|
|
|
base = memblock_alloc_base(size * gpage_npages[i], size,
|
|
|
|
MEMBLOCK_ALLOC_ANYWHERE);
|
|
|
|
add_gpage(base, size, gpage_npages[i]);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2011-10-10 12:50:40 +02:00
|
|
|
#else /* !PPC_FSL_BOOK3E */
|
2011-06-28 11:54:48 +02:00
|
|
|
|
|
|
|
/* Build list of addresses of gigantic pages. This function is used in early
|
|
|
|
* boot before the buddy or bootmem allocator is setup.
|
|
|
|
*/
|
|
|
|
void add_gpage(u64 addr, u64 page_size, unsigned long number_of_pages)
|
2008-07-24 06:27:54 +02:00
|
|
|
{
|
|
|
|
if (!addr)
|
|
|
|
return;
|
|
|
|
while (number_of_pages > 0) {
|
|
|
|
gpage_freearray[nr_gpages] = addr;
|
|
|
|
nr_gpages++;
|
|
|
|
number_of_pages--;
|
|
|
|
addr += page_size;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2008-07-24 06:27:53 +02:00
|
|
|
/* Moves the gigantic page addresses from the temporary list to the
|
2008-07-24 06:27:56 +02:00
|
|
|
* huge_boot_pages list.
|
|
|
|
*/
|
|
|
|
int alloc_bootmem_huge_page(struct hstate *hstate)
|
2008-07-24 06:27:53 +02:00
|
|
|
{
|
|
|
|
struct huge_bootmem_page *m;
|
|
|
|
if (nr_gpages == 0)
|
|
|
|
return 0;
|
|
|
|
m = phys_to_virt(gpage_freearray[--nr_gpages]);
|
|
|
|
gpage_freearray[nr_gpages] = 0;
|
|
|
|
list_add(&m->list, &huge_boot_pages);
|
2008-07-24 06:27:56 +02:00
|
|
|
m->hstate = hstate;
|
2008-07-24 06:27:53 +02:00
|
|
|
return 1;
|
|
|
|
}
|
2011-06-28 11:54:48 +02:00
|
|
|
#endif
|
2008-07-24 06:27:53 +02:00
|
|
|
|
2006-12-07 05:32:03 +01:00
|
|
|
int huge_pmd_unshare(struct mm_struct *mm, unsigned long *addr, pte_t *ptep)
|
|
|
|
{
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2011-10-10 12:50:40 +02:00
|
|
|
#ifdef CONFIG_PPC_FSL_BOOK3E
|
2011-06-28 11:54:48 +02:00
|
|
|
#define HUGEPD_FREELIST_SIZE \
|
|
|
|
((PAGE_SIZE - sizeof(struct hugepd_freelist)) / sizeof(pte_t))
|
|
|
|
|
|
|
|
struct hugepd_freelist {
|
|
|
|
struct rcu_head rcu;
|
|
|
|
unsigned int index;
|
|
|
|
void *ptes[0];
|
|
|
|
};
|
|
|
|
|
|
|
|
static DEFINE_PER_CPU(struct hugepd_freelist *, hugepd_freelist_cur);
|
|
|
|
|
|
|
|
static void hugepd_free_rcu_callback(struct rcu_head *head)
|
|
|
|
{
|
|
|
|
struct hugepd_freelist *batch =
|
|
|
|
container_of(head, struct hugepd_freelist, rcu);
|
|
|
|
unsigned int i;
|
|
|
|
|
|
|
|
for (i = 0; i < batch->index; i++)
|
|
|
|
kmem_cache_free(hugepte_cache, batch->ptes[i]);
|
|
|
|
|
|
|
|
free_page((unsigned long)batch);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void hugepd_free(struct mmu_gather *tlb, void *hugepte)
|
|
|
|
{
|
|
|
|
struct hugepd_freelist **batchp;
|
|
|
|
|
|
|
|
batchp = &__get_cpu_var(hugepd_freelist_cur);
|
|
|
|
|
|
|
|
if (atomic_read(&tlb->mm->mm_users) < 2 ||
|
|
|
|
cpumask_equal(mm_cpumask(tlb->mm),
|
|
|
|
cpumask_of(smp_processor_id()))) {
|
|
|
|
kmem_cache_free(hugepte_cache, hugepte);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (*batchp == NULL) {
|
|
|
|
*batchp = (struct hugepd_freelist *)__get_free_page(GFP_ATOMIC);
|
|
|
|
(*batchp)->index = 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
(*batchp)->ptes[(*batchp)->index++] = hugepte;
|
|
|
|
if ((*batchp)->index == HUGEPD_FREELIST_SIZE) {
|
|
|
|
call_rcu_sched(&(*batchp)->rcu, hugepd_free_rcu_callback);
|
|
|
|
*batchp = NULL;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
static void free_hugepd_range(struct mmu_gather *tlb, hugepd_t *hpdp, int pdshift,
|
|
|
|
unsigned long start, unsigned long end,
|
|
|
|
unsigned long floor, unsigned long ceiling)
|
2006-04-28 07:02:51 +02:00
|
|
|
{
|
|
|
|
pte_t *hugepte = hugepd_page(*hpdp);
|
2011-06-28 11:54:48 +02:00
|
|
|
int i;
|
|
|
|
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
unsigned long pdmask = ~((1UL << pdshift) - 1);
|
2011-06-28 11:54:48 +02:00
|
|
|
unsigned int num_hugepd = 1;
|
|
|
|
|
2011-10-10 12:50:40 +02:00
|
|
|
#ifdef CONFIG_PPC_FSL_BOOK3E
|
|
|
|
/* Note: On fsl the hpdp may be the first of several */
|
2011-06-28 11:54:48 +02:00
|
|
|
num_hugepd = (1 << (hugepd_shift(*hpdp) - pdshift));
|
2011-10-10 12:50:40 +02:00
|
|
|
#else
|
|
|
|
unsigned int shift = hugepd_shift(*hpdp);
|
2011-06-28 11:54:48 +02:00
|
|
|
#endif
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
|
|
|
|
start &= pdmask;
|
|
|
|
if (start < floor)
|
|
|
|
return;
|
|
|
|
if (ceiling) {
|
|
|
|
ceiling &= pdmask;
|
|
|
|
if (! ceiling)
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
if (end - 1 > ceiling - 1)
|
|
|
|
return;
|
2006-04-28 07:02:51 +02:00
|
|
|
|
2011-06-28 11:54:48 +02:00
|
|
|
for (i = 0; i < num_hugepd; i++, hpdp++)
|
|
|
|
hpdp->pd = 0;
|
|
|
|
|
2006-04-28 07:02:51 +02:00
|
|
|
tlb->need_flush = 1;
|
2011-10-10 12:50:40 +02:00
|
|
|
|
|
|
|
#ifdef CONFIG_PPC_FSL_BOOK3E
|
2011-06-28 11:54:48 +02:00
|
|
|
hugepd_free(tlb, hugepte);
|
2011-10-10 12:50:40 +02:00
|
|
|
#else
|
|
|
|
pgtable_free_tlb(tlb, hugepte, pdshift - shift);
|
2011-06-28 11:54:48 +02:00
|
|
|
#endif
|
2006-04-28 07:02:51 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
static void hugetlb_free_pmd_range(struct mmu_gather *tlb, pud_t *pud,
|
|
|
|
unsigned long addr, unsigned long end,
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
unsigned long floor, unsigned long ceiling)
|
2006-04-28 07:02:51 +02:00
|
|
|
{
|
|
|
|
pmd_t *pmd;
|
|
|
|
unsigned long next;
|
|
|
|
unsigned long start;
|
|
|
|
|
|
|
|
start = addr;
|
|
|
|
do {
|
2011-10-10 12:50:39 +02:00
|
|
|
pmd = pmd_offset(pud, addr);
|
2006-04-28 07:02:51 +02:00
|
|
|
next = pmd_addr_end(addr, end);
|
2013-06-19 08:34:26 +02:00
|
|
|
if (!is_hugepd(pmd)) {
|
|
|
|
/*
|
|
|
|
* if it is not hugepd pointer, we should already find
|
|
|
|
* it cleared.
|
|
|
|
*/
|
|
|
|
WARN_ON(!pmd_none_or_clear_bad(pmd));
|
2006-04-28 07:02:51 +02:00
|
|
|
continue;
|
2013-06-19 08:34:26 +02:00
|
|
|
}
|
2011-10-10 12:50:39 +02:00
|
|
|
#ifdef CONFIG_PPC_FSL_BOOK3E
|
|
|
|
/*
|
|
|
|
* Increment next by the size of the huge mapping since
|
|
|
|
* there may be more than one entry at this level for a
|
|
|
|
* single hugepage, but all of them point to
|
|
|
|
* the same kmem cache that holds the hugepte.
|
|
|
|
*/
|
|
|
|
next = addr + (1 << hugepd_shift(*(hugepd_t *)pmd));
|
|
|
|
#endif
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
free_hugepd_range(tlb, (hugepd_t *)pmd, PMD_SHIFT,
|
|
|
|
addr, next, floor, ceiling);
|
2011-10-10 12:50:39 +02:00
|
|
|
} while (addr = next, addr != end);
|
2006-04-28 07:02:51 +02:00
|
|
|
|
|
|
|
start &= PUD_MASK;
|
|
|
|
if (start < floor)
|
|
|
|
return;
|
|
|
|
if (ceiling) {
|
|
|
|
ceiling &= PUD_MASK;
|
|
|
|
if (!ceiling)
|
|
|
|
return;
|
2005-04-17 00:20:36 +02:00
|
|
|
}
|
2006-04-28 07:02:51 +02:00
|
|
|
if (end - 1 > ceiling - 1)
|
|
|
|
return;
|
2005-04-17 00:20:36 +02:00
|
|
|
|
2006-04-28 07:02:51 +02:00
|
|
|
pmd = pmd_offset(pud, start);
|
|
|
|
pud_clear(pud);
|
mm: Pass virtual address to [__]p{te,ud,md}_free_tlb()
mm: Pass virtual address to [__]p{te,ud,md}_free_tlb()
Upcoming paches to support the new 64-bit "BookE" powerpc architecture
will need to have the virtual address corresponding to PTE page when
freeing it, due to the way the HW table walker works.
Basically, the TLB can be loaded with "large" pages that cover the whole
virtual space (well, sort-of, half of it actually) represented by a PTE
page, and which contain an "indirect" bit indicating that this TLB entry
RPN points to an array of PTEs from which the TLB can then create direct
entries. Thus, in order to invalidate those when PTE pages are deleted,
we need the virtual address to pass to tlbilx or tlbivax instructions.
The old trick of sticking it somewhere in the PTE page struct page sucks
too much, the address is almost readily available in all call sites and
almost everybody implemets these as macros, so we may as well add the
argument everywhere. I added it to the pmd and pud variants for consistency.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Acked-by: David Howells <dhowells@redhat.com> [MN10300 & FRV]
Acked-by: Nick Piggin <npiggin@suse.de>
Acked-by: Martin Schwidefsky <schwidefsky@de.ibm.com> [s390]
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2009-07-22 07:44:28 +02:00
|
|
|
pmd_free_tlb(tlb, pmd, start);
|
2006-04-28 07:02:51 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
static void hugetlb_free_pud_range(struct mmu_gather *tlb, pgd_t *pgd,
|
|
|
|
unsigned long addr, unsigned long end,
|
|
|
|
unsigned long floor, unsigned long ceiling)
|
|
|
|
{
|
|
|
|
pud_t *pud;
|
|
|
|
unsigned long next;
|
|
|
|
unsigned long start;
|
|
|
|
|
|
|
|
start = addr;
|
|
|
|
do {
|
2011-10-10 12:50:39 +02:00
|
|
|
pud = pud_offset(pgd, addr);
|
2006-04-28 07:02:51 +02:00
|
|
|
next = pud_addr_end(addr, end);
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
if (!is_hugepd(pud)) {
|
2008-01-03 23:59:50 +01:00
|
|
|
if (pud_none_or_clear_bad(pud))
|
|
|
|
continue;
|
2008-07-24 06:27:56 +02:00
|
|
|
hugetlb_free_pmd_range(tlb, pud, addr, next, floor,
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
ceiling);
|
2008-01-03 23:59:50 +01:00
|
|
|
} else {
|
2011-10-10 12:50:39 +02:00
|
|
|
#ifdef CONFIG_PPC_FSL_BOOK3E
|
|
|
|
/*
|
|
|
|
* Increment next by the size of the huge mapping since
|
|
|
|
* there may be more than one entry at this level for a
|
|
|
|
* single hugepage, but all of them point to
|
|
|
|
* the same kmem cache that holds the hugepte.
|
|
|
|
*/
|
|
|
|
next = addr + (1 << hugepd_shift(*(hugepd_t *)pud));
|
|
|
|
#endif
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
free_hugepd_range(tlb, (hugepd_t *)pud, PUD_SHIFT,
|
|
|
|
addr, next, floor, ceiling);
|
2008-01-03 23:59:50 +01:00
|
|
|
}
|
2011-10-10 12:50:39 +02:00
|
|
|
} while (addr = next, addr != end);
|
2006-04-28 07:02:51 +02:00
|
|
|
|
|
|
|
start &= PGDIR_MASK;
|
|
|
|
if (start < floor)
|
|
|
|
return;
|
|
|
|
if (ceiling) {
|
|
|
|
ceiling &= PGDIR_MASK;
|
|
|
|
if (!ceiling)
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
if (end - 1 > ceiling - 1)
|
|
|
|
return;
|
|
|
|
|
|
|
|
pud = pud_offset(pgd, start);
|
|
|
|
pgd_clear(pgd);
|
mm: Pass virtual address to [__]p{te,ud,md}_free_tlb()
mm: Pass virtual address to [__]p{te,ud,md}_free_tlb()
Upcoming paches to support the new 64-bit "BookE" powerpc architecture
will need to have the virtual address corresponding to PTE page when
freeing it, due to the way the HW table walker works.
Basically, the TLB can be loaded with "large" pages that cover the whole
virtual space (well, sort-of, half of it actually) represented by a PTE
page, and which contain an "indirect" bit indicating that this TLB entry
RPN points to an array of PTEs from which the TLB can then create direct
entries. Thus, in order to invalidate those when PTE pages are deleted,
we need the virtual address to pass to tlbilx or tlbivax instructions.
The old trick of sticking it somewhere in the PTE page struct page sucks
too much, the address is almost readily available in all call sites and
almost everybody implemets these as macros, so we may as well add the
argument everywhere. I added it to the pmd and pud variants for consistency.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Acked-by: David Howells <dhowells@redhat.com> [MN10300 & FRV]
Acked-by: Nick Piggin <npiggin@suse.de>
Acked-by: Martin Schwidefsky <schwidefsky@de.ibm.com> [s390]
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2009-07-22 07:44:28 +02:00
|
|
|
pud_free_tlb(tlb, pud, start);
|
2006-04-28 07:02:51 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This function frees user-level page tables of a process.
|
|
|
|
*/
|
2008-07-24 06:27:10 +02:00
|
|
|
void hugetlb_free_pgd_range(struct mmu_gather *tlb,
|
2006-04-28 07:02:51 +02:00
|
|
|
unsigned long addr, unsigned long end,
|
|
|
|
unsigned long floor, unsigned long ceiling)
|
|
|
|
{
|
|
|
|
pgd_t *pgd;
|
|
|
|
unsigned long next;
|
|
|
|
|
|
|
|
/*
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
* Because there are a number of different possible pagetable
|
|
|
|
* layouts for hugepage ranges, we limit knowledge of how
|
|
|
|
* things should be laid out to the allocation path
|
|
|
|
* (huge_pte_alloc(), above). Everything else works out the
|
|
|
|
* structure as it goes from information in the hugepd
|
|
|
|
* pointers. That means that we can't here use the
|
|
|
|
* optimization used in the normal page free_pgd_range(), of
|
|
|
|
* checking whether we're actually covering a large enough
|
|
|
|
* range to have to do anything at the top level of the walk
|
|
|
|
* instead of at the bottom.
|
2006-04-28 07:02:51 +02:00
|
|
|
*
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
* To make sense of this, you should probably go read the big
|
|
|
|
* block comment at the top of the normal free_pgd_range(),
|
|
|
|
* too.
|
2006-04-28 07:02:51 +02:00
|
|
|
*/
|
|
|
|
|
|
|
|
do {
|
|
|
|
next = pgd_addr_end(addr, end);
|
2011-06-28 11:54:48 +02:00
|
|
|
pgd = pgd_offset(tlb->mm, addr);
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
if (!is_hugepd(pgd)) {
|
2008-09-05 03:49:54 +02:00
|
|
|
if (pgd_none_or_clear_bad(pgd))
|
|
|
|
continue;
|
|
|
|
hugetlb_free_pud_range(tlb, pgd, addr, next, floor, ceiling);
|
|
|
|
} else {
|
2011-10-10 12:50:40 +02:00
|
|
|
#ifdef CONFIG_PPC_FSL_BOOK3E
|
2011-06-28 11:54:48 +02:00
|
|
|
/*
|
|
|
|
* Increment next by the size of the huge mapping since
|
2011-10-10 12:50:40 +02:00
|
|
|
* there may be more than one entry at the pgd level
|
|
|
|
* for a single hugepage, but all of them point to the
|
|
|
|
* same kmem cache that holds the hugepte.
|
2011-06-28 11:54:48 +02:00
|
|
|
*/
|
|
|
|
next = addr + (1 << hugepd_shift(*(hugepd_t *)pgd));
|
|
|
|
#endif
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
free_hugepd_range(tlb, (hugepd_t *)pgd, PGDIR_SHIFT,
|
|
|
|
addr, next, floor, ceiling);
|
2008-09-05 03:49:54 +02:00
|
|
|
}
|
2011-06-28 11:54:48 +02:00
|
|
|
} while (addr = next, addr != end);
|
2005-04-17 00:20:36 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
struct page *
|
|
|
|
follow_huge_addr(struct mm_struct *mm, unsigned long address, int write)
|
|
|
|
{
|
|
|
|
pte_t *ptep;
|
|
|
|
struct page *page;
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
unsigned shift;
|
|
|
|
unsigned long mask;
|
2013-06-20 11:00:18 +02:00
|
|
|
/*
|
|
|
|
* Transparent hugepages are handled by generic code. We can skip them
|
|
|
|
* here.
|
|
|
|
*/
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
ptep = find_linux_pte_or_hugepte(mm->pgd, address, &shift);
|
2005-04-17 00:20:36 +02:00
|
|
|
|
2008-07-24 06:27:56 +02:00
|
|
|
/* Verify it is a huge page else bail. */
|
2013-06-20 11:00:18 +02:00
|
|
|
if (!ptep || !shift || pmd_trans_huge(*(pmd_t *)ptep))
|
2005-04-17 00:20:36 +02:00
|
|
|
return ERR_PTR(-EINVAL);
|
|
|
|
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
mask = (1UL << shift) - 1;
|
2005-04-17 00:20:36 +02:00
|
|
|
page = pte_page(*ptep);
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
if (page)
|
|
|
|
page += (address & mask) / PAGE_SIZE;
|
2005-04-17 00:20:36 +02:00
|
|
|
|
|
|
|
return page;
|
|
|
|
}
|
|
|
|
|
|
|
|
struct page *
|
|
|
|
follow_huge_pmd(struct mm_struct *mm, unsigned long address,
|
|
|
|
pmd_t *pmd, int write)
|
|
|
|
{
|
|
|
|
BUG();
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
2009-11-23 21:03:40 +01:00
|
|
|
static unsigned long hugepte_addr_end(unsigned long addr, unsigned long end,
|
|
|
|
unsigned long sz)
|
|
|
|
{
|
|
|
|
unsigned long __boundary = (addr + sz) & ~(sz-1);
|
|
|
|
return (__boundary - 1 < end - 1) ? __boundary : end;
|
|
|
|
}
|
|
|
|
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
int gup_hugepd(hugepd_t *hugepd, unsigned pdshift,
|
|
|
|
unsigned long addr, unsigned long end,
|
|
|
|
int write, struct page **pages, int *nr)
|
|
|
|
{
|
|
|
|
pte_t *ptep;
|
|
|
|
unsigned long sz = 1UL << hugepd_shift(*hugepd);
|
2009-11-23 21:03:40 +01:00
|
|
|
unsigned long next;
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
|
|
|
|
ptep = hugepte_offset(hugepd, addr, pdshift);
|
|
|
|
do {
|
2009-11-23 21:03:40 +01:00
|
|
|
next = hugepte_addr_end(addr, end, sz);
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
if (!gup_hugepte(ptep, sz, addr, end, write, pages, nr))
|
|
|
|
return 0;
|
2009-11-23 21:03:40 +01:00
|
|
|
} while (ptep++, addr = next, addr != end);
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
|
|
|
|
return 1;
|
|
|
|
}
|
2005-04-17 00:20:36 +02:00
|
|
|
|
2011-10-10 12:50:36 +02:00
|
|
|
#ifdef CONFIG_PPC_MM_SLICES
|
2005-04-17 00:20:36 +02:00
|
|
|
unsigned long hugetlb_get_unmapped_area(struct file *file, unsigned long addr,
|
|
|
|
unsigned long len, unsigned long pgoff,
|
|
|
|
unsigned long flags)
|
|
|
|
{
|
2008-07-24 06:27:56 +02:00
|
|
|
struct hstate *hstate = hstate_file(file);
|
|
|
|
int mmu_psize = shift_to_mmu_psize(huge_page_shift(hstate));
|
2008-12-04 05:07:54 +01:00
|
|
|
|
2013-04-29 20:53:52 +02:00
|
|
|
return slice_get_unmapped_area(addr, len, flags, mmu_psize, 1);
|
2005-04-17 00:20:36 +02:00
|
|
|
}
|
2011-10-10 12:50:36 +02:00
|
|
|
#endif
|
2005-04-17 00:20:36 +02:00
|
|
|
|
2009-01-06 23:38:54 +01:00
|
|
|
unsigned long vma_mmu_pagesize(struct vm_area_struct *vma)
|
|
|
|
{
|
2011-09-20 21:58:10 +02:00
|
|
|
#ifdef CONFIG_PPC_MM_SLICES
|
2009-01-06 23:38:54 +01:00
|
|
|
unsigned int psize = get_slice_psize(vma->vm_mm, vma->vm_start);
|
|
|
|
|
|
|
|
return 1UL << mmu_psize_to_shift(psize);
|
2011-06-28 11:54:48 +02:00
|
|
|
#else
|
|
|
|
if (!is_vm_hugetlb_page(vma))
|
|
|
|
return PAGE_SIZE;
|
|
|
|
|
|
|
|
return huge_page_size(hstate_vma(vma));
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline bool is_power_of_4(unsigned long x)
|
|
|
|
{
|
|
|
|
if (is_power_of_2(x))
|
|
|
|
return (__ilog2(x) % 2) ? false : true;
|
|
|
|
return false;
|
2009-01-06 23:38:54 +01:00
|
|
|
}
|
|
|
|
|
2009-10-26 20:24:31 +01:00
|
|
|
static int __init add_huge_page_size(unsigned long long size)
|
2008-01-03 23:59:50 +01:00
|
|
|
{
|
2009-10-26 20:24:31 +01:00
|
|
|
int shift = __ffs(size);
|
|
|
|
int mmu_psize;
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
|
2008-01-03 23:59:50 +01:00
|
|
|
/* Check that it is a page size supported by the hardware and
|
2009-10-26 20:24:31 +01:00
|
|
|
* that it fits within pagetable and slice limits. */
|
2011-06-28 11:54:48 +02:00
|
|
|
#ifdef CONFIG_PPC_FSL_BOOK3E
|
|
|
|
if ((size < PAGE_SIZE) || !is_power_of_4(size))
|
|
|
|
return -EINVAL;
|
|
|
|
#else
|
2009-10-26 20:24:31 +01:00
|
|
|
if (!is_power_of_2(size)
|
|
|
|
|| (shift > SLICE_HIGH_SHIFT) || (shift <= PAGE_SHIFT))
|
|
|
|
return -EINVAL;
|
2011-06-28 11:54:48 +02:00
|
|
|
#endif
|
2008-07-24 06:27:55 +02:00
|
|
|
|
2009-10-26 20:24:31 +01:00
|
|
|
if ((mmu_psize = shift_to_mmu_psize(shift)) < 0)
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
#ifdef CONFIG_SPU_FS_64K_LS
|
|
|
|
/* Disable support for 64K huge pages when 64K SPU local store
|
|
|
|
* support is enabled as the current implementation conflicts.
|
|
|
|
*/
|
|
|
|
if (shift == PAGE_SHIFT_64K)
|
|
|
|
return -EINVAL;
|
|
|
|
#endif /* CONFIG_SPU_FS_64K_LS */
|
|
|
|
|
|
|
|
BUG_ON(mmu_psize_defs[mmu_psize].shift != shift);
|
|
|
|
|
|
|
|
/* Return if huge page size has already been setup */
|
|
|
|
if (size_to_hstate(size))
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
hugetlb_add_hstate(shift - PAGE_SHIFT);
|
|
|
|
|
|
|
|
return 0;
|
2008-01-03 23:59:50 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
static int __init hugepage_setup_sz(char *str)
|
|
|
|
{
|
|
|
|
unsigned long long size;
|
|
|
|
|
|
|
|
size = memparse(str, &str);
|
|
|
|
|
2009-10-26 20:24:31 +01:00
|
|
|
if (add_huge_page_size(size) != 0)
|
2008-01-03 23:59:50 +01:00
|
|
|
printk(KERN_WARNING "Invalid huge page size specified(%llu)\n", size);
|
|
|
|
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
__setup("hugepagesz=", hugepage_setup_sz);
|
|
|
|
|
2011-10-10 12:50:40 +02:00
|
|
|
#ifdef CONFIG_PPC_FSL_BOOK3E
|
2011-06-28 11:54:48 +02:00
|
|
|
struct kmem_cache *hugepte_cache;
|
|
|
|
static int __init hugetlbpage_init(void)
|
|
|
|
{
|
|
|
|
int psize;
|
|
|
|
|
|
|
|
for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) {
|
|
|
|
unsigned shift;
|
|
|
|
|
|
|
|
if (!mmu_psize_defs[psize].shift)
|
|
|
|
continue;
|
|
|
|
|
|
|
|
shift = mmu_psize_to_shift(psize);
|
|
|
|
|
|
|
|
/* Don't treat normal page sizes as huge... */
|
|
|
|
if (shift != PAGE_SHIFT)
|
|
|
|
if (add_huge_page_size(1ULL << shift) < 0)
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Create a kmem cache for hugeptes. The bottom bits in the pte have
|
|
|
|
* size information encoded in them, so align them to allow this
|
|
|
|
*/
|
|
|
|
hugepte_cache = kmem_cache_create("hugepte-cache", sizeof(pte_t),
|
|
|
|
HUGEPD_SHIFT_MASK + 1, 0, NULL);
|
|
|
|
if (hugepte_cache == NULL)
|
|
|
|
panic("%s: Unable to create kmem cache for hugeptes\n",
|
|
|
|
__func__);
|
|
|
|
|
|
|
|
/* Default hpage size = 4M */
|
|
|
|
if (mmu_psize_defs[MMU_PAGE_4M].shift)
|
|
|
|
HPAGE_SHIFT = mmu_psize_defs[MMU_PAGE_4M].shift;
|
|
|
|
else
|
|
|
|
panic("%s: Unable to set default huge page size\n", __func__);
|
|
|
|
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
#else
|
2006-04-28 07:02:51 +02:00
|
|
|
static int __init hugetlbpage_init(void)
|
|
|
|
{
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 20:24:31 +01:00
|
|
|
int psize;
|
2008-07-24 06:27:56 +02:00
|
|
|
|
2011-04-06 21:48:50 +02:00
|
|
|
if (!mmu_has_feature(MMU_FTR_16M_PAGE))
|
2006-04-28 07:02:51 +02:00
|
|
|
return -ENODEV;
|
2008-07-28 08:13:18 +02:00
|
|
|
|
2009-10-26 20:24:31 +01:00
|
|
|
for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) {
|
|
|
|
unsigned shift;
|
|
|
|
unsigned pdshift;
|
2008-07-24 06:27:56 +02:00
|
|
|
|
2009-10-26 20:24:31 +01:00
|
|
|
if (!mmu_psize_defs[psize].shift)
|
|
|
|
continue;
|
2008-07-28 08:13:18 +02:00
|
|
|
|
2009-10-26 20:24:31 +01:00
|
|
|
shift = mmu_psize_to_shift(psize);
|
|
|
|
|
|
|
|
if (add_huge_page_size(1ULL << shift) < 0)
|
|
|
|
continue;
|
|
|
|
|
|
|
|
if (shift < PMD_SHIFT)
|
|
|
|
pdshift = PMD_SHIFT;
|
|
|
|
else if (shift < PUD_SHIFT)
|
|
|
|
pdshift = PUD_SHIFT;
|
|
|
|
else
|
|
|
|
pdshift = PGDIR_SHIFT;
|
2013-04-28 11:37:30 +02:00
|
|
|
/*
|
|
|
|
* if we have pdshift and shift value same, we don't
|
|
|
|
* use pgt cache for hugepd.
|
|
|
|
*/
|
|
|
|
if (pdshift != shift) {
|
|
|
|
pgtable_cache_add(pdshift - shift, NULL);
|
|
|
|
if (!PGT_CACHE(pdshift - shift))
|
|
|
|
panic("hugetlbpage_init(): could not create "
|
|
|
|
"pgtable cache for %d bit pagesize\n", shift);
|
|
|
|
}
|
2008-07-24 06:27:56 +02:00
|
|
|
}
|
2006-04-28 07:02:51 +02:00
|
|
|
|
2009-10-26 20:24:31 +01:00
|
|
|
/* Set default large page size. Currently, we pick 16M or 1M
|
|
|
|
* depending on what is available
|
|
|
|
*/
|
|
|
|
if (mmu_psize_defs[MMU_PAGE_16M].shift)
|
|
|
|
HPAGE_SHIFT = mmu_psize_defs[MMU_PAGE_16M].shift;
|
|
|
|
else if (mmu_psize_defs[MMU_PAGE_1M].shift)
|
|
|
|
HPAGE_SHIFT = mmu_psize_defs[MMU_PAGE_1M].shift;
|
|
|
|
|
2006-04-28 07:02:51 +02:00
|
|
|
return 0;
|
|
|
|
}
|
2011-06-28 11:54:48 +02:00
|
|
|
#endif
|
2006-04-28 07:02:51 +02:00
|
|
|
module_init(hugetlbpage_init);
|
2009-10-26 20:24:31 +01:00
|
|
|
|
|
|
|
void flush_dcache_icache_hugepage(struct page *page)
|
|
|
|
{
|
|
|
|
int i;
|
2011-06-28 11:54:48 +02:00
|
|
|
void *start;
|
2009-10-26 20:24:31 +01:00
|
|
|
|
|
|
|
BUG_ON(!PageCompound(page));
|
|
|
|
|
2011-06-28 11:54:48 +02:00
|
|
|
for (i = 0; i < (1UL << compound_order(page)); i++) {
|
|
|
|
if (!PageHighMem(page)) {
|
|
|
|
__flush_dcache_icache(page_address(page+i));
|
|
|
|
} else {
|
2011-11-25 16:14:16 +01:00
|
|
|
start = kmap_atomic(page+i);
|
2011-06-28 11:54:48 +02:00
|
|
|
__flush_dcache_icache(start);
|
2011-11-25 16:14:16 +01:00
|
|
|
kunmap_atomic(start);
|
2011-06-28 11:54:48 +02:00
|
|
|
}
|
|
|
|
}
|
2009-10-26 20:24:31 +01:00
|
|
|
}
|
2013-06-20 11:00:16 +02:00
|
|
|
|
|
|
|
#endif /* CONFIG_HUGETLB_PAGE */
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We have 4 cases for pgds and pmds:
|
|
|
|
* (1) invalid (all zeroes)
|
|
|
|
* (2) pointer to next table, as normal; bottom 6 bits == 0
|
|
|
|
* (3) leaf pte for huge page, bottom two bits != 00
|
|
|
|
* (4) hugepd pointer, bottom two bits == 00, next 4 bits indicate size of table
|
2013-06-20 11:00:22 +02:00
|
|
|
*
|
|
|
|
* So long as we atomically load page table pointers we are safe against teardown,
|
|
|
|
* we can follow the address down to the the page and take a ref on it.
|
2013-06-20 11:00:16 +02:00
|
|
|
*/
|
2013-06-20 11:00:22 +02:00
|
|
|
|
2013-06-20 11:00:16 +02:00
|
|
|
pte_t *find_linux_pte_or_hugepte(pgd_t *pgdir, unsigned long ea, unsigned *shift)
|
|
|
|
{
|
2013-06-20 11:00:22 +02:00
|
|
|
pgd_t pgd, *pgdp;
|
|
|
|
pud_t pud, *pudp;
|
|
|
|
pmd_t pmd, *pmdp;
|
2013-06-20 11:00:16 +02:00
|
|
|
pte_t *ret_pte;
|
|
|
|
hugepd_t *hpdp = NULL;
|
|
|
|
unsigned pdshift = PGDIR_SHIFT;
|
|
|
|
|
|
|
|
if (shift)
|
|
|
|
*shift = 0;
|
|
|
|
|
2013-06-20 11:00:22 +02:00
|
|
|
pgdp = pgdir + pgd_index(ea);
|
|
|
|
pgd = ACCESS_ONCE(*pgdp);
|
2013-06-20 11:00:17 +02:00
|
|
|
/*
|
2013-06-20 11:00:22 +02:00
|
|
|
* Always operate on the local stack value. This make sure the
|
|
|
|
* value don't get updated by a parallel THP split/collapse,
|
|
|
|
* page fault or a page unmap. The return pte_t * is still not
|
|
|
|
* stable. So should be checked there for above conditions.
|
2013-06-20 11:00:17 +02:00
|
|
|
*/
|
2013-06-20 11:00:22 +02:00
|
|
|
if (pgd_none(pgd))
|
2013-06-20 11:00:17 +02:00
|
|
|
return NULL;
|
2013-06-20 11:00:22 +02:00
|
|
|
else if (pgd_huge(pgd)) {
|
|
|
|
ret_pte = (pte_t *) pgdp;
|
2013-06-20 11:00:16 +02:00
|
|
|
goto out;
|
2013-06-20 11:00:22 +02:00
|
|
|
} else if (is_hugepd(&pgd))
|
|
|
|
hpdp = (hugepd_t *)&pgd;
|
2013-06-20 11:00:17 +02:00
|
|
|
else {
|
2013-06-20 11:00:22 +02:00
|
|
|
/*
|
|
|
|
* Even if we end up with an unmap, the pgtable will not
|
|
|
|
* be freed, because we do an rcu free and here we are
|
|
|
|
* irq disabled
|
|
|
|
*/
|
2013-06-20 11:00:16 +02:00
|
|
|
pdshift = PUD_SHIFT;
|
2013-06-20 11:00:22 +02:00
|
|
|
pudp = pud_offset(&pgd, ea);
|
|
|
|
pud = ACCESS_ONCE(*pudp);
|
2013-06-20 11:00:16 +02:00
|
|
|
|
2013-06-20 11:00:22 +02:00
|
|
|
if (pud_none(pud))
|
2013-06-20 11:00:17 +02:00
|
|
|
return NULL;
|
2013-06-20 11:00:22 +02:00
|
|
|
else if (pud_huge(pud)) {
|
|
|
|
ret_pte = (pte_t *) pudp;
|
2013-06-20 11:00:16 +02:00
|
|
|
goto out;
|
2013-06-20 11:00:22 +02:00
|
|
|
} else if (is_hugepd(&pud))
|
|
|
|
hpdp = (hugepd_t *)&pud;
|
2013-06-20 11:00:17 +02:00
|
|
|
else {
|
2013-06-20 11:00:16 +02:00
|
|
|
pdshift = PMD_SHIFT;
|
2013-06-20 11:00:22 +02:00
|
|
|
pmdp = pmd_offset(&pud, ea);
|
|
|
|
pmd = ACCESS_ONCE(*pmdp);
|
2013-06-20 11:00:17 +02:00
|
|
|
/*
|
|
|
|
* A hugepage collapse is captured by pmd_none, because
|
|
|
|
* it mark the pmd none and do a hpte invalidate.
|
|
|
|
*
|
|
|
|
* A hugepage split is captured by pmd_trans_splitting
|
|
|
|
* because we mark the pmd trans splitting and do a
|
|
|
|
* hpte invalidate
|
|
|
|
*
|
|
|
|
*/
|
2013-06-20 11:00:22 +02:00
|
|
|
if (pmd_none(pmd) || pmd_trans_splitting(pmd))
|
2013-06-20 11:00:17 +02:00
|
|
|
return NULL;
|
2013-06-20 11:00:16 +02:00
|
|
|
|
2013-06-20 11:00:22 +02:00
|
|
|
if (pmd_huge(pmd) || pmd_large(pmd)) {
|
|
|
|
ret_pte = (pte_t *) pmdp;
|
2013-06-20 11:00:16 +02:00
|
|
|
goto out;
|
2013-06-20 11:00:22 +02:00
|
|
|
} else if (is_hugepd(&pmd))
|
|
|
|
hpdp = (hugepd_t *)&pmd;
|
2013-06-20 11:00:17 +02:00
|
|
|
else
|
2013-06-20 11:00:22 +02:00
|
|
|
return pte_offset_kernel(&pmd, ea);
|
2013-06-20 11:00:16 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
if (!hpdp)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
ret_pte = hugepte_offset(hpdp, ea, pdshift);
|
|
|
|
pdshift = hugepd_shift(*hpdp);
|
|
|
|
out:
|
|
|
|
if (shift)
|
|
|
|
*shift = pdshift;
|
|
|
|
return ret_pte;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(find_linux_pte_or_hugepte);
|
|
|
|
|
|
|
|
int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr,
|
|
|
|
unsigned long end, int write, struct page **pages, int *nr)
|
|
|
|
{
|
|
|
|
unsigned long mask;
|
|
|
|
unsigned long pte_end;
|
|
|
|
struct page *head, *page, *tail;
|
|
|
|
pte_t pte;
|
|
|
|
int refs;
|
|
|
|
|
|
|
|
pte_end = (addr + sz) & ~(sz-1);
|
|
|
|
if (pte_end < end)
|
|
|
|
end = pte_end;
|
|
|
|
|
2013-06-20 11:00:23 +02:00
|
|
|
pte = ACCESS_ONCE(*ptep);
|
2013-06-20 11:00:16 +02:00
|
|
|
mask = _PAGE_PRESENT | _PAGE_USER;
|
|
|
|
if (write)
|
|
|
|
mask |= _PAGE_RW;
|
|
|
|
|
|
|
|
if ((pte_val(pte) & mask) != mask)
|
|
|
|
return 0;
|
|
|
|
|
2013-06-20 11:00:22 +02:00
|
|
|
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
|
|
|
|
/*
|
|
|
|
* check for splitting here
|
|
|
|
*/
|
|
|
|
if (pmd_trans_splitting(pte_pmd(pte)))
|
|
|
|
return 0;
|
|
|
|
#endif
|
|
|
|
|
2013-06-20 11:00:16 +02:00
|
|
|
/* hugepages are never "special" */
|
|
|
|
VM_BUG_ON(!pfn_valid(pte_pfn(pte)));
|
|
|
|
|
|
|
|
refs = 0;
|
|
|
|
head = pte_page(pte);
|
|
|
|
|
|
|
|
page = head + ((addr & (sz-1)) >> PAGE_SHIFT);
|
|
|
|
tail = page;
|
|
|
|
do {
|
|
|
|
VM_BUG_ON(compound_head(page) != head);
|
|
|
|
pages[*nr] = page;
|
|
|
|
(*nr)++;
|
|
|
|
page++;
|
|
|
|
refs++;
|
|
|
|
} while (addr += PAGE_SIZE, addr != end);
|
|
|
|
|
|
|
|
if (!page_cache_add_speculative(head, refs)) {
|
|
|
|
*nr -= refs;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (unlikely(pte_val(pte) != pte_val(*ptep))) {
|
|
|
|
/* Could be optimized better */
|
|
|
|
*nr -= refs;
|
|
|
|
while (refs--)
|
|
|
|
put_page(head);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Any tail page need their mapcount reference taken before we
|
|
|
|
* return.
|
|
|
|
*/
|
|
|
|
while (refs--) {
|
|
|
|
if (PageTail(tail))
|
|
|
|
get_huge_page_tail(tail);
|
|
|
|
tail++;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 1;
|
|
|
|
}
|