mirrors
/
kore
mirror of https://git.kore.io/kore.git
1
0
Fork 1
Code Issues Projects Releases Wiki Activity
kore/src/worker.c

498 lines
10 KiB
C
Raw Normal View History

refactor code quite a bit.
2013-06-26 11:18:32 +02:00
/*
bump copyright years
2014-10-14 16:18:23 +02:00
* Copyright (c) 2013-2014 Joris Vink <joris@coders.se>
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
*
* Permission to use, copy, modify, and distribute this software for any
* purpose with or without fee is hereby granted, provided that the above
* copyright notice and this permission notice appear in all copies.
*
* THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
* WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
* ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
* WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
* ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
* OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
#include <sys/types.h>
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
#include <sys/shm.h>
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
#include <sys/wait.h>
Add worker_rlimit_nofiles as a configurable option.
2014-07-31 09:14:03 +02:00
#include <sys/time.h>
#include <sys/resource.h>
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
#include <grp.h>
Clean up header includes, based on a diff from Ewan Higgs via github. And while we're messing in it, make sure bsd.c compiles again.
2013-07-06 20:55:22 +02:00
#include <pwd.h>
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
#include <signal.h>
#include "kore.h"
#include "http.h"
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
#if defined(KORE_USE_PGSQL)
Normalize kore_*.h headers
2014-07-03 22:14:46 +02:00
#include "pgsql.h"
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
#endif
Add initial stab at asynchronous background tasks. More to follow.
2014-06-28 16:17:18 +02:00
#if defined(KORE_USE_TASKS)
Normalize kore_*.h headers
2014-07-03 22:14:46 +02:00
#include "tasks.h"
Add initial stab at asynchronous background tasks. More to follow.
2014-06-28 16:17:18 +02:00
#endif
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
#if defined(WORKER_DEBUG)
#define worker_debug(fmt, ...) printf(fmt, ##__VA_ARGS__)
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
#else
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
#define worker_debug(fmt, ...)
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
#endif
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
#define KORE_SHM_KEY 15000
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
#define WORKER_LOCK_TIMEOUT 500
Properly calculate worker offset, otherwise we'll eventually run into trouble.
2013-07-15 11:24:49 +02:00
#define WORKER(id) \
(struct kore_worker *)((u_int8_t *)kore_workers + \
(sizeof(struct kore_worker) * id))
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
struct wlock {
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
volatile int lock;
pid_t current;
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
};
static int worker_trylock(void);
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
static void worker_unlock(void);
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
static inline int kore_worker_acceptlock_obtain(void);
static inline void kore_worker_acceptlock_release(void);
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
Make worker_clients available everywhere.
2014-12-03 20:40:44 +01:00
static struct connection_list disconnected;
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
static struct kore_worker *kore_workers;
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
static int shm_accept_key;
- Better spread load between all worker processes. - Introduce own memory management system on top of malloc to keep track of all our allocations and free's. Later we should introduce a pooling mechanism for fixed size allocations (http_request comes to mind). - Introduce ssl_cipher in configuration. Memory usage is kind of high right now, but it seems its OpenSSL doing it rather then Kore.
2013-06-27 08:43:07 +02:00
static struct wlock *accept_lock;
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
extern volatile sig_atomic_t sig_recv;
struct kore_worker *worker = NULL;
Make worker_clients available everywhere.
2014-12-03 20:40:44 +01:00
struct connection_list worker_clients;
Add worker_rlimit_nofiles as a configurable option.
2014-07-31 09:14:03 +02:00
u_int32_t worker_rlimit_nofiles = 1024;
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
u_int32_t worker_max_connections = 250;
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
u_int32_t worker_active_connections = 0;
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
void
kore_worker_init(void)
{
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
size_t len;
Long overdue, let Kore pick its own shm key if the default KORE_SHM_KEY is already taken.
2013-10-14 11:26:11 +02:00
key_t key;
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
u_int16_t i, cpu;
if (worker_count == 0)
Make some configuration options fallback to defaults. * If no runas_user is given, use current user. * If no chroot path is given, don't complain if -n * If no workers is set, use the default 1.
2014-07-31 13:59:42 +02:00
worker_count = 1;
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
len = sizeof(*accept_lock) +
(sizeof(struct kore_worker) * worker_count);
Long overdue, let Kore pick its own shm key if the default KORE_SHM_KEY is already taken.
2013-10-14 11:26:11 +02:00
shm_accept_key = -1;
for (key = KORE_SHM_KEY; shm_accept_key == -1; key++) {
shm_accept_key = shmget(key, len, IPC_CREAT | IPC_EXCL | 0700);
if (shm_accept_key == -1 && errno != EEXIST)
fatal("kore_worker_init(): shmget() %s", errno_s);
}
shmat() returns (void *)-1 upon error.
2015-03-25 19:42:24 +01:00
if ((accept_lock = shmat(shm_accept_key, NULL, 0)) == (void *)-1)
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
fatal("kore_worker_init(): shmat() %s", errno_s);
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
accept_lock->lock = 0;
accept_lock->current = 0;
Properly calculate worker offset, otherwise we'll eventually run into trouble.
2013-07-15 11:24:49 +02:00
kore_workers = (struct kore_worker *)((u_int8_t *)accept_lock +
sizeof(*accept_lock));
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
memset(kore_workers, 0, sizeof(struct kore_worker) * worker_count);
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
kore_debug("kore_worker_init(): system has %d cpu's", cpu_count);
kore_debug("kore_worker_init(): starting %d workers", worker_count);
Brackets would be great here.
2015-01-19 15:26:53 +01:00
if (worker_count > cpu_count) {
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
kore_debug("kore_worker_init(): more workers then cpu's");
Brackets would be great here.
2015-01-19 15:26:53 +01:00
}
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
cpu = 0;
for (i = 0; i < worker_count; i++) {
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
kore_worker_spawn(i, cpu++);
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
if (cpu == cpu_count)
cpu = 0;
}
}
void
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
kore_worker_spawn(u_int16_t id, u_int16_t cpu)
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
{
struct kore_worker *kw;
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
kw = WORKER(id);
kw->id = id;
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
kw->cpu = cpu;
Kore will now keep track of page handlers that cause workers to die. This is useful to track down any issues you might have in your module. A log entry with a page handler causing issues looks like: Jul 7 14:44:30 devbook kore[18191]: [parent]: worker 1 (18193)-> status 11 Jul 7 14:44:30 devbook kore[18191]: [parent]: worker 1 (pid: 18193) (hdlr: 0x242d9c0) gone Jul 7 14:44:30 devbook kore[18191]: [parent]: hdlr serve_intro has caused 2 error(s)
2013-07-07 14:48:32 +02:00
kw->has_lock = 0;
kw->active_hdlr = NULL;
- Better spread load between all worker processes. - Introduce own memory management system on top of malloc to keep track of all our allocations and free's. Later we should introduce a pooling mechanism for fixed size allocations (http_request comes to mind). - Introduce ssl_cipher in configuration. Memory usage is kind of high right now, but it seems its OpenSSL doing it rather then Kore.
2013-06-27 08:43:07 +02:00
Kore will now keep track of page handlers that cause workers to die. This is useful to track down any issues you might have in your module. A log entry with a page handler causing issues looks like: Jul 7 14:44:30 devbook kore[18191]: [parent]: worker 1 (18193)-> status 11 Jul 7 14:44:30 devbook kore[18191]: [parent]: worker 1 (pid: 18193) (hdlr: 0x242d9c0) gone Jul 7 14:44:30 devbook kore[18191]: [parent]: hdlr serve_intro has caused 2 error(s)
2013-07-07 14:48:32 +02:00
kw->pid = fork();
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
if (kw->pid == -1)
fatal("could not spawn worker child: %s", errno_s);
if (kw->pid == 0) {
kw->pid = getpid();
kore_worker_entry(kw);
/* NOTREACHED */
}
}
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
void
kore_worker_shutdown(void)
{
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
struct kore_worker *kw;
u_int16_t id, done;
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
kore_log(LOG_NOTICE, "waiting for workers to drain and shutdown");
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
for (;;) {
done = 0;
for (id = 0; id < worker_count; id++) {
kw = WORKER(id);
if (kw->pid != 0)
kore_worker_wait(1);
else
done++;
}
if (done == worker_count)
break;
}
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
if (shmctl(shm_accept_key, IPC_RMID, NULL) == -1) {
kore_log(LOG_NOTICE,
"failed to deleted shm segment: %s", errno_s);
}
}
void
kore_worker_dispatch_signal(int sig)
{
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
u_int16_t id;
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
struct kore_worker *kw;
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
for (id = 0; id < worker_count; id++) {
kw = WORKER(id);
Brackets would be great here.
2015-01-19 15:26:53 +01:00
if (kill(kw->pid, sig) == -1) {
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
kore_debug("kill(%d, %d): %s", kw->pid, sig, errno_s);
Brackets would be great here.
2015-01-19 15:26:53 +01:00
}
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
}
}
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
void
kore_worker_entry(struct kore_worker *kw)
{
Add worker_rlimit_nofiles as a configurable option.
2014-07-31 09:14:03 +02:00
struct rlimit rl;
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
char buf[16];
struct connection *c, *cnext;
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
int quit, had_lock, r;
u_int64_t now, idle_check, next_lock;
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
worker = kw;
Kore no longer requires root to be started.
2013-11-18 00:42:57 +01:00
if (skip_chroot == 0) {
if (chroot(chroot_path) == -1)
fatal("cannot chroot(): %s", errno_s);
if (chdir("/") == -1)
fatal("cannot chdir(): %s", errno_s);
}
if (getuid() != pw->pw_uid) {
if (setgroups(1, &pw->pw_gid) ||
Add OSX support, buildable via "make osx". Make sure you have OpenSSL 1.0.0d+ (available from Macports) installed. Based on diff from Vaibhav Bhembre via github
2013-07-17 20:17:00 +02:00
#ifdef __MACH__
Kore no longer requires root to be started.
2013-11-18 00:42:57 +01:00
setgid(pw->pw_gid) || setegid(pw->pw_gid) ||
setuid(pw->pw_uid) || seteuid(pw->pw_uid))
Add OSX support, buildable via "make osx". Make sure you have OpenSSL 1.0.0d+ (available from Macports) installed. Based on diff from Vaibhav Bhembre via github
2013-07-17 20:17:00 +02:00
#else
Kore no longer requires root to be started.
2013-11-18 00:42:57 +01:00
setresgid(pw->pw_gid, pw->pw_gid, pw->pw_gid) ||
setresuid(pw->pw_uid, pw->pw_uid, pw->pw_uid))
Add OSX support, buildable via "make osx". Make sure you have OpenSSL 1.0.0d+ (available from Macports) installed. Based on diff from Vaibhav Bhembre via github
2013-07-17 20:17:00 +02:00
#endif
Kore no longer requires root to be started.
2013-11-18 00:42:57 +01:00
fatal("unable to drop privileges");
}
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
Add worker_rlimit_nofiles as a configurable option.
2014-07-31 09:14:03 +02:00
rl.rlim_cur = worker_rlimit_nofiles;
rl.rlim_max = worker_rlimit_nofiles;
if (setrlimit(RLIMIT_NOFILE, &rl) == -1) {
kore_log(LOG_ERR, "setrlimit(RLIMIT_NOFILE, %d): %s",
worker_rlimit_nofiles, errno_s);
}
Correct snprintf() usage
2014-04-23 16:29:58 +02:00
(void)snprintf(buf, sizeof(buf), "kore [wrk %d]", kw->id);
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
kore_platform_proctitle(buf);
kore_platform_worker_setcpu(kw);
kore_pid = kw->pid;
sig_recv = 0;
signal(SIGHUP, kore_signal);
signal(SIGQUIT, kore_signal);
signal(SIGPIPE, SIG_IGN);
Allow Kore to be started in foreground mode (-f) Sending SIGINT or Ctrl-c will reap all workers and exit again. Useful for development.
2014-07-31 13:27:04 +02:00
if (foreground)
signal(SIGINT, kore_signal);
else
signal(SIGINT, SIG_IGN);
Add fixed size memory pools and use them throughout Kore.
2013-07-15 10:13:36 +02:00
net_init();
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
http_init();
Add fixed size memory pools and use them throughout Kore.
2013-07-15 10:13:36 +02:00
kore_connection_init();
Add CRL support. Allow Kore to use per domain CRLs when requiring client certificates. The require_client_cert configuration option has been renamed to a more sane client_certificates and can optionally take a second argument which is the CRL in pem format. You'll need a restart in case the CRLs get updated.
2014-10-18 02:32:05 +02:00
kore_domain_load_crl();
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
TAILQ_INIT(&disconnected);
TAILQ_INIT(&worker_clients);
quit = 0;
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
had_lock = 0;
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
next_lock = 0;
No need to set now at the start, thanks clang.
2014-09-18 08:42:41 +02:00
idle_check = 0;
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
kore_platform_event_init();
kore_accesslog_worker_init();
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
#if defined(KORE_USE_PGSQL)
kore_pgsql_init();
#endif
Add initial stab at asynchronous background tasks. More to follow.
2014-06-28 16:17:18 +02:00
#if defined(KORE_USE_TASKS)
kore_task_init();
#endif
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
kore_log(LOG_NOTICE, "worker %d started (cpu#%d)", kw->id, kw->cpu);
Don't call module onloads until after everything is initialized. This means the onload functions for a module are now called after a worker has started and never from the parent ever again.
2014-06-29 21:15:23 +02:00
kore_module_onload();
- Better spread load between all worker processes. - Introduce own memory management system on top of malloc to keep track of all our allocations and free's. Later we should introduce a pooling mechanism for fixed size allocations (http_request comes to mind). - Introduce ssl_cipher in configuration. Memory usage is kind of high right now, but it seems its OpenSSL doing it rather then Kore.
2013-06-27 08:43:07 +02:00
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
if (worker_count > 1)
kw->accept_treshold = worker_max_connections / 16;
else
kw->accept_treshold = worker_max_connections;
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
for (;;) {
if (sig_recv != 0) {
if (sig_recv == SIGHUP)
Don't call module onloads until after everything is initialized. This means the onload functions for a module are now called after a worker has started and never from the parent ever again.
2014-06-29 21:15:23 +02:00
kore_module_reload(1);
Allow Kore to be started in foreground mode (-f) Sending SIGINT or Ctrl-c will reap all workers and exit again. Useful for development.
2014-07-31 13:27:04 +02:00
else if (sig_recv == SIGQUIT || sig_recv == SIGINT)
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
quit = 1;
Add CRL support. Allow Kore to use per domain CRLs when requiring client certificates. The require_client_cert configuration option has been renamed to a more sane client_certificates and can optionally take a second argument which is the CRL in pem format. You'll need a restart in case the CRLs get updated.
2014-10-18 02:32:05 +02:00
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
sig_recv = 0;
}
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
now = kore_time_ms();
if (now > next_lock) {
if (kore_worker_acceptlock_obtain()) {
if (had_lock == 0) {
kore_platform_enable_accept();
had_lock = 1;
}
}
}
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
if (!worker->has_lock) {
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
if (had_lock == 1) {
had_lock = 0;
kore_platform_disable_accept();
}
If we do not own the accept lock, reduce timeout for platform wait. This way multiple workers won't stall when passing the accept lock around for at maximum 100 ms (time spent in epoll/kqueue waits).
2014-07-28 23:27:58 +02:00
}
- Better spread load between all worker processes. - Introduce own memory management system on top of malloc to keep track of all our allocations and free's. Later we should introduce a pooling mechanism for fixed size allocations (http_request comes to mind). - Introduce ssl_cipher in configuration. Memory usage is kind of high right now, but it seems its OpenSSL doing it rather then Kore.
2013-06-27 08:43:07 +02:00
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
r = kore_platform_event_wait(100);
if (worker->has_lock && r > 0) {
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
kore_worker_acceptlock_release();
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
next_lock = now + WORKER_LOCK_TIMEOUT;
}
- Better spread load between all worker processes. - Introduce own memory management system on top of malloc to keep track of all our allocations and free's. Later we should introduce a pooling mechanism for fixed size allocations (http_request comes to mind). - Introduce ssl_cipher in configuration. Memory usage is kind of high right now, but it seems its OpenSSL doing it rather then Kore.
2013-06-27 08:43:07 +02:00
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
http_process();
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
if ((now - idle_check) >= 10000) {
- add idle timer for normal connections (ie: !SPDY), max 20 seconds. - use idle timer to make sure we don't block SPDY connections indefinately when our window size has reached 0.
2013-07-01 12:08:51 +02:00
idle_check = now;
Repopulate now when we are doing idle timer checks. Otherwise in a weird scenario the idle timer could be started with a more recent time then the time we test against causing a connection to disconnect before its idle timer was over.
2014-11-21 16:38:23 +01:00
now = kore_time_ms();
- add idle timer for normal connections (ie: !SPDY), max 20 seconds. - use idle timer to make sure we don't block SPDY connections indefinately when our window size has reached 0.
2013-07-01 12:08:51 +02:00
TAILQ_FOREACH(c, &worker_clients, list) {
if (c->proto == CONN_PROTO_SPDY &&
Kore can now disconnect SPDY session if they've been idle too long. Configurable via spdy_idle_time in your configuration file. Setting this to 0 will keep SPDY sessions open indefinately.
2013-07-13 20:19:01 +02:00
c->idle_timer.length == 0 &&
Add spdy_session_teardown() which can properly teardown a SPDY session. Use this throughout the spdy code to propagate session errors (if any) occur. At the same time fix BSD's missing CONN_WRITE_BLOCK
2013-07-13 19:56:38 +02:00
!(c->flags & CONN_WRITE_BLOCK) &&
!(c->flags & CONN_READ_BLOCK))
- add idle timer for normal connections (ie: !SPDY), max 20 seconds. - use idle timer to make sure we don't block SPDY connections indefinately when our window size has reached 0.
2013-07-01 12:08:51 +02:00
continue;
if (!(c->flags & CONN_IDLE_TIMER_ACT))
continue;
kore_connection_check_idletimer(now, c);
}
}
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
for (c = TAILQ_FIRST(&disconnected); c != NULL; c = cnext) {
cnext = TAILQ_NEXT(c, list);
TAILQ_REMOVE(&disconnected, c, list);
kore_connection_remove(c);
}
if (quit && http_request_count == 0)
break;
}
for (c = TAILQ_FIRST(&worker_clients); c != NULL; c = cnext) {
cnext = TAILQ_NEXT(c, list);
net_send_flush(c);
Call kore_connection_disconnect() when draining connections. This way the disconnect callbacks are called properly when workers are exiting.
2014-09-17 08:46:55 +02:00
kore_connection_disconnect(c);
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
}
for (c = TAILQ_FIRST(&disconnected); c != NULL; c = cnext) {
cnext = TAILQ_NEXT(c, list);
net_send_flush(c);
TAILQ_REMOVE(&disconnected, c, list);
kore_connection_remove(c);
}
kore_debug("worker %d shutting down", kw->id);
exit(0);
}
void
kore_worker_connection_add(struct connection *c)
{
TAILQ_INSERT_TAIL(&worker_clients, c, list);
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
worker_active_connections++;
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
}
void
kore_worker_connection_move(struct connection *c)
{
TAILQ_REMOVE(&worker_clients, c, list);
TAILQ_INSERT_TAIL(&disconnected, c, list);
}
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
void
kore_worker_connection_remove(struct connection *c)
{
worker_active_connections--;
}
Add websocket support to Kore. Introduces a few new api functions: - kore_websocket_handshake(struct http_request *): Performs the handshake on an HTTP request (coming from page handler) - kore_websocket_send(struct connection *, u_int8_t, void *, size_t): Sends data to a websocket connection. - kore_websocket_broadcast(struct connection *, u_int8_t, void *, size_t, int): Broadcast the given websocket op and data to all connected websocket clients on the worker. Note that as of right now the WEBSOCKET_BROADCAST_GLOBAL scope option does not work yet and messages broadcasted will be restricted to workers only. - kore_worker_websocket_broadcast(struct connection *, void *, void *): Backend function used by kore_websocket_broadcast(). Could prove useful for developers to have access to. A simple example is given under examples/websocket. Known issues: Kore does not support PING or CONT frames just yet.
2014-11-24 11:01:12 +01:00
void
kore_worker_websocket_broadcast(struct connection *src,
void (*cb)(struct connection *, void *), void *args)
{
struct connection *c;
TAILQ_FOREACH(c, &worker_clients, list) {
if (c != src && c->proto == CONN_PROTO_WEBSOCKET)
cb(c, args);
}
}
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
void
kore_worker_wait(int final)
{
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
u_int16_t id;
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
pid_t pid;
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
struct kore_worker *kw;
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
int status;
if (final)
pid = waitpid(WAIT_ANY, &status, 0);
else
pid = waitpid(WAIT_ANY, &status, WNOHANG);
if (pid == -1) {
kore_debug("waitpid(): %s", errno_s);
return;
}
if (pid == 0)
return;
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
for (id = 0; id < worker_count; id++) {
kw = WORKER(id);
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
if (kw->pid != pid)
continue;
kore_log(LOG_NOTICE, "worker %d (%d)-> status %d",
kw->id, pid, status);
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
if (final) {
kw->pid = 0;
break;
}
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
if (WEXITSTATUS(status) || WTERMSIG(status) ||
WCOREDUMP(status)) {
kore_log(LOG_NOTICE,
use handler function name if available when logging a worker process that has died.
2013-07-07 14:56:50 +02:00
"worker %d (pid: %d) (hdlr: %s) gone",
kw->id, kw->pid,
(kw->active_hdlr != NULL) ? kw->active_hdlr->func :
"none");
If the worker that owns the accept lock dies, make sure the next one in line can grab it. Otherwise kore comes to a halt (by not accepting new connections).
2013-07-05 20:19:50 +02:00
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
if (kw->pid == accept_lock->current)
worker_unlock();
Kore will now keep track of page handlers that cause workers to die. This is useful to track down any issues you might have in your module. A log entry with a page handler causing issues looks like: Jul 7 14:44:30 devbook kore[18191]: [parent]: worker 1 (18193)-> status 11 Jul 7 14:44:30 devbook kore[18191]: [parent]: worker 1 (pid: 18193) (hdlr: 0x242d9c0) gone Jul 7 14:44:30 devbook kore[18191]: [parent]: hdlr serve_intro has caused 2 error(s)
2013-07-07 14:48:32 +02:00
if (kw->active_hdlr != NULL) {
kw->active_hdlr->errors++;
kore_log(LOG_NOTICE,
"hdlr %s has caused %d error(s)",
kw->active_hdlr->func,
kw->active_hdlr->errors);
If the worker that owns the accept lock dies, make sure the next one in line can grab it. Otherwise kore comes to a halt (by not accepting new connections).
2013-07-05 20:19:50 +02:00
}
Kore will now keep track of page handlers that cause workers to die. This is useful to track down any issues you might have in your module. A log entry with a page handler causing issues looks like: Jul 7 14:44:30 devbook kore[18191]: [parent]: worker 1 (18193)-> status 11 Jul 7 14:44:30 devbook kore[18191]: [parent]: worker 1 (pid: 18193) (hdlr: 0x242d9c0) gone Jul 7 14:44:30 devbook kore[18191]: [parent]: hdlr serve_intro has caused 2 error(s)
2013-07-07 14:48:32 +02:00
kore_log(LOG_NOTICE, "restarting worker %d", kw->id);
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
kore_worker_spawn(kw->id, kw->cpu);
- Better spread load between all worker processes. - Introduce own memory management system on top of malloc to keep track of all our allocations and free's. Later we should introduce a pooling mechanism for fixed size allocations (http_request comes to mind). - Introduce ssl_cipher in configuration. Memory usage is kind of high right now, but it seems its OpenSSL doing it rather then Kore.
2013-06-27 08:43:07 +02:00
} else {
kore_log(LOG_NOTICE,
Kore will now keep track of page handlers that cause workers to die. This is useful to track down any issues you might have in your module. A log entry with a page handler causing issues looks like: Jul 7 14:44:30 devbook kore[18191]: [parent]: worker 1 (18193)-> status 11 Jul 7 14:44:30 devbook kore[18191]: [parent]: worker 1 (pid: 18193) (hdlr: 0x242d9c0) gone Jul 7 14:44:30 devbook kore[18191]: [parent]: hdlr serve_intro has caused 2 error(s)
2013-07-07 14:48:32 +02:00
"worker %d (pid: %d) signaled us (%d)",
kw->id, kw->pid, status);
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
}
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
break;
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
}
}
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
static inline void
Make kore_worker_acceptlock_release() available to loaded modules. This allows us to move the accept lock manually to another Kore worker in case we own it and are about to do some heavy lifting on the current Kore worker.
2013-10-24 09:05:46 +02:00
kore_worker_acceptlock_release(void)
{
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
if (worker_count == 1)
Make kore_worker_acceptlock_release() available to loaded modules. This allows us to move the accept lock manually to another Kore worker in case we own it and are about to do some heavy lifting on the current Kore worker.
2013-10-24 09:05:46 +02:00
return;
if (worker->has_lock != 1)
return;
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
worker_unlock();
worker->has_lock = 0;
Make kore_worker_acceptlock_release() available to loaded modules. This allows us to move the accept lock manually to another Kore worker in case we own it and are about to do some heavy lifting on the current Kore worker.
2013-10-24 09:05:46 +02:00
}
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
static inline int
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
kore_worker_acceptlock_obtain(void)
{
If we do not own the accept lock, reduce timeout for platform wait. This way multiple workers won't stall when passing the accept lock around for at maximum 100 ms (time spent in epoll/kqueue waits).
2014-07-28 23:27:58 +02:00
int r;
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
if (worker->has_lock == 1)
return (1);
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
if (worker_count == 1) {
- Better spread load between all worker processes. - Introduce own memory management system on top of malloc to keep track of all our allocations and free's. Later we should introduce a pooling mechanism for fixed size allocations (http_request comes to mind). - Introduce ssl_cipher in configuration. Memory usage is kind of high right now, but it seems its OpenSSL doing it rather then Kore.
2013-06-27 08:43:07 +02:00
worker->has_lock = 1;
If we do not own the accept lock, reduce timeout for platform wait. This way multiple workers won't stall when passing the accept lock around for at maximum 100 ms (time spent in epoll/kqueue waits).
2014-07-28 23:27:58 +02:00
return (1);
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
}
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
if (worker_active_connections >= worker_max_connections)
return (0);
If we do not own the accept lock, reduce timeout for platform wait. This way multiple workers won't stall when passing the accept lock around for at maximum 100 ms (time spent in epoll/kqueue waits).
2014-07-28 23:27:58 +02:00
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
r = 0;
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
if (worker_trylock()) {
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
r = 1;
worker->has_lock = 1;
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
}
If we do not own the accept lock, reduce timeout for platform wait. This way multiple workers won't stall when passing the accept lock around for at maximum 100 ms (time spent in epoll/kqueue waits).
2014-07-28 23:27:58 +02:00
return (r);
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
}
static int
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
worker_trylock(void)
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
{
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
if (!__sync_bool_compare_and_swap(&(accept_lock->lock), 0, 1))
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
return (0);
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
- Better spread load between all worker processes. - Introduce own memory management system on top of malloc to keep track of all our allocations and free's. Later we should introduce a pooling mechanism for fixed size allocations (http_request comes to mind). - Introduce ssl_cipher in configuration. Memory usage is kind of high right now, but it seems its OpenSSL doing it rather then Kore.
2013-06-27 08:43:07 +02:00
worker_debug("wrk#%d grabbed lock (%d/%d)\n", worker->id,
worker_active_connections, worker_max_connections);
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
accept_lock->current = worker->pid;
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
return (1);
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
}
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
static void
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
worker_unlock(void)
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
{
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
accept_lock->current = 0;
if (!__sync_bool_compare_and_swap(&(accept_lock->lock), 1, 0))
kore_log(LOG_NOTICE, "worker_unlock(): wasnt locked");
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
}