As indicated by BZ#23178, concurrent access on some files read by nscd
may result non expected data send through service requisition. This is
due 'sendfile' Linux implementation where for sockets with zero-copy
support, callers must ensure the transferred portions of the the file
reffered by input file descriptor remain unmodified until the reader
on the other end of socket has consumed the transferred data.
I could not find any explicit documentation stating this behaviour on
Linux kernel documentation. However man-pages sendfile entry [1] states
in NOTES the aforementioned remark. It was initially pushed on man-pages
with an explicit testcase [2] that shows changing the file used in
'sendfile' call prior the socket input data consumption results in
previous data being lost.
From commit message it stated on tested Linux version (3.15) only TCP
socket showed this issues, however on recent kernels (4.4) I noticed the
same behaviour for local sockets as well.
Since sendfile on HURD is a read/write operation and the underlying
issue on Linux, the straightforward fix is just remove sendfile use
altogether. I am really skeptical it is hitting some hotstop (there
are indication over internet that sendfile is helpfull only for large
files, more than 10kb) here to justify that extra code complexity or
to pursuit other possible fix (through memory or file locks for
instance, which I am not sure it is doable).
Checked on x86_64-linux-gnu.
[BZ #23178]
* nscd/nscd-client.h (sendfileall): Remove prototype.
* nscd/connections.c [HAVE_SENDFILE] (sendfileall): Remove function.
(handle_request): Use writeall instead of sendfileall.
* nscd/aicache.c (addhstaiX): Likewise.
* nscd/grpcache.c (cache_addgr): Likewise.
* nscd/hstcache.c (cache_addhst): Likewise.
* nscd/initgrcache.c (addinitgroupsX): Likewise.
* nscd/netgroupcache.c (addgetnetgrentX, addinnetgrX): Likewise.
* nscd/pwdcache.c (cache_addpw): Likewise.
* nscd/servicescache.c (cache_addserv): Likewise.
* sysdeps/unix/sysv/linux/Makefile [$(subdir) == nscd]
(sysdep-CFLAGS): Remove -DHAVE_SENDFILE.
* sysdeps/unix/sysv/linux/kernel-features.h (__ASSUME_SENDFILE):
Remove define.
[1] http://man7.org/linux/man-pages/man2/sendfile.2.html
[2] 7b6a329977 (diff-efd6af3a70f0f07c578e85b51e83b3c3)
This patch consolidates the code to initialize the header of a dataset
into a single set of functions (one for positive and another for
negative datasets) primarily to reduce repetition of code. The
secondary reason is to simplify Patch 2/2 which fixes the problem of
an uninitialized byte in the header by initializing an unused field in
the structure and hence preventing a possible data leak into the cache
file.
The nscd/*cache.c files contain assert()s, writeall() and sendfileall() calls
that invalidly use together &dataset->resp and total where either dataset or
dataset->head.recsize should be used instead one of the components. In the
writeall() and sendfileall() cases, it is unlikely to matter in practice, but
the assertions can fail sometimes without a proper reason.
The commit 20e498bd removes the pthread_mutex_rdlock() calls, but not the
corresponding pthread_mutex_unlock() calls. Also, the database lock is never
unlocked in one branch of the mempool_alloc() if.
I think unreproducible random assert(dh->usable) crashes in prune_cache() were
caused by this. But an easy way to make nscd threads hang with the broken
locking was.
2009-02-13 Ulrich Drepper <drepper@redhat.com>
[BZ #5381]
* nscd/nscd.h: Remove definitions and declarations for mem_in_flight.
Change mempool_alloc prototype.
* nscd/mem.c (gc): Don't handle mem_in_flight.
(mempool_alloc): Third parameter now only indicates whether this is the
first call (to allocate data) or not. If it is, get db rdlock.
Release it on error. Don't handle mem_in_flight.
* nscd/aicache.c (addhstaiX): Mark he parameter as const.
Adjust third parameter of mempool_alloc calls.
Nothing to do here in case mempool_alloc fails.
Avoid local variable shadowing parameter. No need to get db rdlock
before calling cache_add.
* nscd/cache.c (cache_add): Adjust call to mempool_alloc. There is
no mem_in_flight array anymore.
* nscd/connections.c: Remove definition and handling of mem_in_flight.
* nscd/grpcache.c (cache_addgr): Adjust third parameter of
mempool_alloc calls. Mark he parameter as const. Nothing to do here
in case mempool_alloc fails. No need to get db rdlock before calling
cache_add.
* nscd/hstcache.c (cache_addhst): Likewise.
* nscd/initgrcache.c (addinitgroupsX): Likewise.
* nscd/servicescache.c (cache_addserv): Likewise.
* nscd/pwdcache.c (cache_addpw): Likewise. Remove some debugging code.
whether this is in response of a cache refill. Check alignment
of package data. Revamp waking of pruning thread.
(prune_cache): Small optimization.
* nscd/nscd.h: Adjust cache_add prototypes.
* nscd/aicache.c: Adjust cache_add calls.
* nscd/grpcache.c: Likewise.
* nscd/hstcache.c: Likewise.
* nscd/initgrcache.c: Likewise.
* nscd/pwdcache.c: Likewise.
* nscd/servicescache.c: Likewise.
* nscd/connections.c (restart): Really disable cache use before
exec attempt. If it fails, reenable cache.
(nscd_run_prune): Initialize wakeup_time. After wakeup, set wakeup
time to max to be able to notice concurrent cache additions. Unlock
prune_lock while performing gc. Afterwards compute wakeup time with
current wakeup_time value in mind.
is the first use of the record, mark it as unusable.
* nscd/aicache.c: Don't touch the dataset after cache_add returns
reporting a failure.
* nscd/grpcache.c: Likewise
* nscd/hstcache.c: Likewise.
* nscd/initgrcache.c: Likewise.
* nscd/pwdcache.c: Likewise.
* nscd/servicecache.c: Likewise.
2008-04-15 Ulrich Drepper <drepper@redhat.com>
[BZ #5209]
* sysdeps/unix/sysv/linux/times.c: New file.
[BZ #5381]
* nscd/nscd.h: Define enum in_flight, mem_in_flight, and
mem_in_flight_list variables. Add new parameter to mempool_alloc
prototype.
* nscd/mem.c (mempool_alloc): Take additional parameter. Initialize
appropriate mem_in_flight element.
(gc): Take allocations which have not yet been committed to the
database into account.
* nscd/cache.c (cache_add): Add new parameter to mempool_alloc call.
Reset mem_in_flight before returning.
* nscd/connections.c (nscd_run_worker): Initialize mem_in_flight and
cue it up in mem_in_flight_list.
* nscd/aicache.c: Adjust mempool_alloc call.
* nscd/grpcache.c: Likewise.
* nscd/hstcache.c: Likewise.
* nscd/initgrcache.c: Likewise.
* nscd/pwdcache.c: Likewise.
* nscd/servicescache.c: Likewise.
* nscd/Makefile (nscd-flags): Until ld is fixed, use -fpic instead
of -fpie.
* nscd/connections.c (handle_request): Provide better error message
in case SELinux forbids the service.
* version.h (VERSION): Bump to 2.8.90.
gethstbynm3_r.
* nscd/gethstbynm2_r.c: Remove.
* nscd/gethstbynm3_r.c: New file.
* nscd/aicache.c (addhstaiX): Use __gethostbyaddr2_r instead of
__gethostbyaddr_r.
* nscd/gethstbyad_r.c: Generate __gethostbyaddr2_r function. Define
__gethostbyaddr_r compatibility wrapper.
* nscd/hstcache.c (cache_addhst): Add ttl parameter. Use it when
determining timeout of entry.
(lookup): Take new parameter and pass it to __gethostbyname3_r and
__gethostbyaddr2_r.
(addhstbyX): Pass reference to variable for TTL to lookup and
cache_addhst.
* nss/Versions [glibc] (GLIBC_PRIVATE): Export __nss_passwd_lookup2,
__nss_group_lookup2, __nss_hosts_lookup2, __nss_services_lookup2,
and __nss_next2. Remove __nss_services_lookup.
* nss/XXX-lookup.c: Name function now *_lookup2. Add new parameter.
Add compat wrapper.
* nss/getXXbyYY_r.c: Changes to call new *_lookup2 functions and
__nss_next2.
* nss/getXXent_r.c: Likewise.
* nss/getnssent_r.c: Likewise.
* nss/nsswitch.c (__nss_lookup): Add new parameter. If first function
does not exist in module, try the optional second name.
(__nss_next2): New function.
(__nss_next): Now wrapper around __nss_next2.
* nss/nsswitch.h: Adjust __nss_lookup prototype.
Declare __nss_next2.
Adjust definition of db_lookup_function type.
* nss/service-lookup.c: Define NO_COMPAT.
* include/netdb.h: Declare __gethostbyaddr2_r and __gethostbyname3_r.
* inet/ether_hton.c: Use __nss_next2 instead of __nss_next.
* inet/ether_ntoh.c: Likewise.
* sunrpc/netname.c: Likewise.
* sunrpc/publickey.c: Likewise.
* inet/getnetgrent.c: Likewise. Adjust calls to __nss_lookup.
* inet/gethstbyad_r.c (DB_LOOKUP_FCT): Change to __nss_hosts_lookup2.
* inet/gethstbynm2_r.c (DB_LOOKUP_FCT): Likewise.
* inet/gethstbynm_r.c (DB_LOOKUP_FCT): Likewise.
* inet/gethstent_r.c (DB_LOOKUP_FCT): Likewise.
* nscd/aicache.c (addhstaiX): Fix default TTL handling.
* inet/getnetgrent.c (setup): Encrypt static pointer.
* ncsd/aicache.c: Don't use sendfile for records on the stack.
* nscd/grpcache.c: Likewise.
* nscd/hstcache.c: Likewise.
* nscd/initgrcache.c: Likewise.
* nscd/pwdcache.c: Likewise.
Patch by dmueller@suse.com.
(struct shmid_ds): Use it for shm_nattch field.
2005-11-18 Jakub Jelinek <jakub@redhat.com>
* sysdeps/unix/sysv/linux/futimesat.c (futimesat): If FILE is NULL,
set access and modification times of the file referenced by FD.
* sysdeps/generic/futimesat.c (futimesat): Don't return EINVAL if
FILE is NULL. Don't check FD if FILE is absolute path.
2005-11-19 Ulrich Drepper <drepper@redhat.com>
* nscd/nscd_gethst_r.c (nscd_gethst_r): Avoid unnecesary read call
if there are no aliases.
* sysdeps/unix/sysv/linux/Makefile (CFLAGS-connections.c,
CFLAGS-pwdcache.c, CFLAGS-grpcache.c, CFLAGS-hstcache.c,
CFLAGS-aicache.c, CFLAGS-initgrcache.c): Add -DHAVE_SENDFILE.
* sysdeps/unix/sysv/linux/kernel-features.h (__ASSUME_SENDFILE):
Define.
* nscd/pwdcache.c [HAVE_SENDFILE]: Include <sys/sendfile.h> and
<kernel-features.h>.
[HAVE_SENDFILE] (cache_addpw): Use sendfile to transmit positive
result.
* nscd/grpcache.c: Likewise.
* nscd/hstcache.c: Likewise.
* nscd/aicache.c: Likewise.
* nscd/initgrcache.c: Likewise.
* nscd/connectionc.c: Likewise.
(dbs): Initialize max_db_size fields.
(nscd_init): When mapping the database, use max_db_size as the
mapping size even if it is bigger than the file size.
* nscd/mem.c (mempool_alloc): When resizing the file make sure the
limit in max_db_size is not exceeded. Don't use mremap, just
posix_fallocate is enough (according to Linus). Use posix_fallocate
correctly.
* nscd/nscd.conf: Add max-db-size parameters.
* nscd/nscd.h (struct database_dyn): Add max_db_size field.
Define DEFAULT_MAX_DB_SIZE and DEFAULT_DATASIZE_PER_BUCKET.
Temporarily define TEMP_FAILURE_RETRY_VAL here.
* nscd/nscd_conf.c (nscd_parse_file): Parse max-db-size parameter
and add sanity checks for it.
* nscd/aicache.c (addhstaiX): Use send with MSG_NOSIGNAL not write to
send reply.
* nscd/connection.c (writeall): Likewise.
(handle_request): Likewise.
* nscd/grpcache.c (cache_addgr): Likewise.
* nscd/hstcache.c (cache_addhst): Likewise.
* nscd/initgrcache.c (addinitgroupsX): Likewise.
* nscd/nscd.c (parse_opt): Likewise.
* nscd/nscd_stat.c (send_stats): Likewise.
(receive_print_stats): Likewise.
* nscd/pwdcache.c (cache_addpw): Likewise.
NETDB_INTERNAL if buffer is too small.
* nscd/hstcache.c (INCR): Remove.
(addhstbyX): Double buflen in each iteration rather than add INCR.
* nscd/grpcache.c: Likewise.
* nscd/pwdcache.c: Likewise.
* nscd/connections.c: Implement r/o sharing of nscd's cache with client
processes via shared memory.
* nscd/nscd-client.h: Likewise.
* nscd/nscd.h: Likewise.
* nscd/nscd_conf.c: Likewise.
* nscd/nscd_getgr_r.c: Likewise.
* nscd/nscd_getpw_r.c: Likewise.
* nscd/nscd_gethst_r.c: Likewise.
* nscd/nscd.conf: Add new config parameters.
* nscd/Makefile (aux): Add nscd_helper.
* nscd/nscd_helper.c: New file.
* nscd/mem.c (gc): Indicate beginning and end of the gc cycle.
* nscd/hstcache.c: Simplify a lot. We cache only the request itself,
no derived information.
* connections.c (nscd_init): Fix bug in testing size of the persistent.
* nis/Makefile (aux): Add nis_hash.
* nis/nis_hash.c: New file. Split out from nis_util.c.
* nis/nis_util.c: Move __nis_hash code in separate file.
* csu/tst-atomic.c: Improve atomic_increment_val test which would
not have found a ppc bug.
2004-08-26 Ulrich Drepper <drepper@redhat.com>
* nscd/cache.c: Major rewrite. The data is now optionally kept in
a mmaped memory region which is automatically mirrored on disk.
This implements persistent data storage. The Memory handled
needed to be completely revamped, it now uses a garbage collection
mechanism instead of malloc.
* nscd/connections.c: Likewise.
* nscd/nscd.c: Likewise.
* nscd/nscd.h: Likewise.
* nscd/nscd_conf.c: Likewise.
* nscd/nscd_stat.c: Likewise.
* nscd/grpcache.c: Likewise.
* nscd/hstcache.c:: Likewise.
* nscd/pwdcache.c:: Likewise.
* nscd/Makefile: Add rules to build mem.c.
* nscd/mem.c: New file.
* nscd/nscd.conf: Describe new configuration options.
Florian Weimer