mirrors
/
kore
mirror of https://git.kore.io/kore.git
1
0
Fork 1
Code Issues Projects Releases Wiki Activity
kore/src/worker.c

649 lines
13 KiB
C
Raw Normal View History

refactor code quite a bit.
2013-06-26 11:18:32 +02:00
/*
2018
2018-01-20 22:51:06 +01:00
* Copyright (c) 2013-2018 Joris Vink <joris@coders.se>
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
*
* Permission to use, copy, modify, and distribute this software for any
* purpose with or without fee is hereby granted, provided that the above
* copyright notice and this permission notice appear in all copies.
*
* THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
* WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
* ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
* WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
* ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
* OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
NetBSD fixes, from David Carlier.
2015-06-14 16:44:37 +02:00
#include <sys/param.h>
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
#include <sys/shm.h>
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
#include <sys/wait.h>
Add worker_rlimit_nofiles as a configurable option.
2014-07-31 09:14:03 +02:00
#include <sys/time.h>
#include <sys/resource.h>
Add our messaging framework. With this framework apps can now send messages between worker processes. A new API function exists: int kore_msg_register(u_int8_t id, void (*cb)(const void *, u_int32_t); This API call allows your app to register a new message callback for a given ID. You can then send messages on this ID to other workers using: void kore_msg_send(u_int8_t id, void *data, u_int32_t length); This framework will interally be used for a few things such as allowing websocket data to broadcasted between all workers, adding unified caching and hopefully eventually moving the access log to this as well. Some internals have changed with this commit: * worker_clients has been called connections. * the parent now initializes the net, and event subsystems. * kore_worker_websocket_broadcast() is dead.
2015-06-22 21:13:32 +02:00
#include <sys/socket.h>
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
Let workers fetch entropy from keymgr. At bootup and every 1800 seconds after that the worker processes will ask the keymgr for new entropy that they will seed into their PRNG. Additionally once received the worker calls RAND_poll() to grab more entropy from the system to be mixed in.
2017-02-28 06:28:35 +01:00
#if !defined(KORE_NO_TLS)
#include <openssl/rand.h>
#endif
Properly deal with accept() failing. In cases where accept() failed Kore would not relinquish the lock towards other worker processes. This becomes evident when dealing with a high number of concurrent connections to the point the fd table gets full. In this scenario the worker with the full fd table will spin on attempt to accept newer connections. As a bonus, Kore now has allows exactly up to worker_max_connections of connections per worker before no longer attempting to grab the accept lock.
2015-04-20 15:17:42 +02:00
#include <fcntl.h>
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
#include <grp.h>
Clean up header includes, based on a diff from Ewan Higgs via github. And while we're messing in it, make sure bsd.c compiles again.
2013-07-06 20:55:22 +02:00
#include <pwd.h>
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
#include <signal.h>
#include "kore.h"
Introduce NOHTTP=1 build option. This basically turns off the HTTP layer for Kore. It does not compile in anything for HTTP. This allows Kore to be used as a network application platform as well. Added an example for this called nohttp. Other changes that sneaked in while hacking on this: * Use calloc(), kill pendantic malloc option. * Killed off SPDY/3.1 support completely, will be superseded by http2 Note that comes with massive changes to a lot of the core API functions provided by Kore, these might break your application.
2015-11-27 16:22:50 +01:00
#if !defined(KORE_NO_HTTP)
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
#include "http.h"
Introduce NOHTTP=1 build option. This basically turns off the HTTP layer for Kore. It does not compile in anything for HTTP. This allows Kore to be used as a network application platform as well. Added an example for this called nohttp. Other changes that sneaked in while hacking on this: * Use calloc(), kill pendantic malloc option. * Killed off SPDY/3.1 support completely, will be superseded by http2 Note that comes with massive changes to a lot of the core API functions provided by Kore, these might break your application.
2015-11-27 16:22:50 +01:00
#endif
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
#if defined(KORE_USE_PGSQL)
Normalize kore_*.h headers
2014-07-03 22:14:46 +02:00
#include "pgsql.h"
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
#endif
Add initial stab at asynchronous background tasks. More to follow.
2014-06-28 16:17:18 +02:00
#if defined(KORE_USE_TASKS)
Normalize kore_*.h headers
2014-07-03 22:14:46 +02:00
#include "tasks.h"
Add initial stab at asynchronous background tasks. More to follow.
2014-06-28 16:17:18 +02:00
#endif
Add initial python support. Based on work done by Stanislav Yudin.
2017-01-12 23:38:51 +01:00
#if defined(KORE_USE_PYTHON)
#include "python_api.h"
#endif
Define WAIT_ANY if not defined previously.
2016-01-31 16:47:00 +01:00
#if !defined(WAIT_ANY)
#define WAIT_ANY (-1)
#endif
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
#define WORKER_LOCK_TIMEOUT 500
#if !defined(KORE_NO_TLS)
#define WORKER_SOLO_COUNT 2
#else
#define WORKER_SOLO_COUNT 1
#endif
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
Properly calculate worker offset, otherwise we'll eventually run into trouble.
2013-07-15 11:24:49 +02:00
#define WORKER(id) \
(struct kore_worker *)((u_int8_t *)kore_workers + \
(sizeof(struct kore_worker) * id))
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
struct wlock {
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
volatile int lock;
pid_t current;
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
};
static int worker_trylock(void);
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
static void worker_unlock(void);
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
static inline int kore_worker_acceptlock_obtain(u_int64_t);
static inline int kore_worker_acceptlock_release(u_int64_t);
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
Let workers fetch entropy from keymgr. At bootup and every 1800 seconds after that the worker processes will ask the keymgr for new entropy that they will seed into their PRNG. Additionally once received the worker calls RAND_poll() to grab more entropy from the system to be mixed in.
2017-02-28 06:28:35 +01:00
#if !defined(KORE_NO_TLS)
static void worker_entropy_recv(struct kore_msg *, const void *);
#endif
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
static struct kore_worker *kore_workers;
Allow workers to listen on different ports. This commit allows worker processes to have individual listeners not configured by the kore configuration. If you do the following: - do not configure listeners in your .conf file - call kore_server_bind() in kore_worker_configure() the workers will no longer fight over accept locks as the configured ports no longer conflict with each other. This allows me to create X amount of instances of a worker process that are each individually accessible via unique ports.
2017-03-13 15:32:04 +01:00
static int worker_no_lock;
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
static int shm_accept_key;
- Better spread load between all worker processes. - Introduce own memory management system on top of malloc to keep track of all our allocations and free's. Later we should introduce a pooling mechanism for fixed size allocations (http_request comes to mind). - Introduce ssl_cipher in configuration. Memory usage is kind of high right now, but it seems its OpenSSL doing it rather then Kore.
2013-06-27 08:43:07 +02:00
static struct wlock *accept_lock;
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
extern volatile sig_atomic_t sig_recv;
struct kore_worker *worker = NULL;
Add new configuration option: worker_set_affinity. Controls wether or not worker processes sets its affinity mask to bind themselves to a single CPU. Defaults to on (1).
2015-04-27 10:36:33 +02:00
u_int8_t worker_set_affinity = 1;
Fix typo in configuration option worker_accept_treshold. There is no backwards comptabile option available. Fixes #53
2015-05-18 12:20:28 +02:00
u_int32_t worker_accept_threshold = 0;
Add worker_rlimit_nofiles as a configurable option.
2014-07-31 09:14:03 +02:00
u_int32_t worker_rlimit_nofiles = 1024;
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
u_int32_t worker_max_connections = 250;
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
u_int32_t worker_active_connections = 0;
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
void
kore_worker_init(void)
{
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
size_t len;
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
u_int16_t i, cpu;
Allow workers to listen on different ports. This commit allows worker processes to have individual listeners not configured by the kore configuration. If you do the following: - do not configure listeners in your .conf file - call kore_server_bind() in kore_worker_configure() the workers will no longer fight over accept locks as the configured ports no longer conflict with each other. This allows me to create X amount of instances of a worker process that are each individually accessible via unique ports.
2017-03-13 15:32:04 +01:00
worker_no_lock = 0;
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
if (worker_count == 0)
Make some configuration options fallback to defaults. * If no runas_user is given, use current user. * If no chroot path is given, don't complain if -n * If no workers is set, use the default 1.
2014-07-31 13:59:42 +02:00
worker_count = 1;
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
Separate private keys from worker processes. Kore will now isolate RSA private keys to a separate process (keymgr). Worker processes that require RSA signing for TLS connections will communicate with this keymgr process in order to do so. This behaviour cannot be disabled and is always turned on.
2016-06-08 13:55:14 +02:00
#if !defined(KORE_NO_TLS)
/* account for the key manager. */
worker_count += 1;
#endif
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
len = sizeof(*accept_lock) +
(sizeof(struct kore_worker) * worker_count);
Long overdue, let Kore pick its own shm key if the default KORE_SHM_KEY is already taken.
2013-10-14 11:26:11 +02:00
Use IPC_PRIVATE when creating our shm. Instead of a homegrown mechanism of finding empty SHM keys that may or may not be available. Should fix #44
2015-04-28 10:20:36 +02:00
shm_accept_key = shmget(IPC_PRIVATE, len, IPC_CREAT | IPC_EXCL | 0700);
if (shm_accept_key == -1)
fatal("kore_worker_init(): shmget() %s", errno_s);
shmat() returns (void *)-1 upon error.
2015-03-25 19:42:24 +01:00
if ((accept_lock = shmat(shm_accept_key, NULL, 0)) == (void *)-1)
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
fatal("kore_worker_init(): shmat() %s", errno_s);
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
accept_lock->lock = 0;
accept_lock->current = 0;
Properly calculate worker offset, otherwise we'll eventually run into trouble.
2013-07-15 11:24:49 +02:00
kore_workers = (struct kore_worker *)((u_int8_t *)accept_lock +
sizeof(*accept_lock));
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
memset(kore_workers, 0, sizeof(struct kore_worker) * worker_count);
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
kore_debug("kore_worker_init(): system has %d cpu's", cpu_count);
kore_debug("kore_worker_init(): starting %d workers", worker_count);
Brackets would be great here.
2015-01-19 15:26:53 +01:00
if (worker_count > cpu_count) {
Fix typo in log message
2016-02-12 08:57:58 +01:00
kore_debug("kore_worker_init(): more workers than cpu's");
Brackets would be great here.
2015-01-19 15:26:53 +01:00
}
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
cpu = 0;
for (i = 0; i < worker_count; i++) {
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
kore_worker_spawn(i, cpu++);
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
if (cpu == cpu_count)
cpu = 0;
}
}
void
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
kore_worker_spawn(u_int16_t id, u_int16_t cpu)
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
{
struct kore_worker *kw;
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
kw = WORKER(id);
kw->id = id;
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
kw->cpu = cpu;
Kore will now keep track of page handlers that cause workers to die. This is useful to track down any issues you might have in your module. A log entry with a page handler causing issues looks like: Jul 7 14:44:30 devbook kore[18191]: [parent]: worker 1 (18193)-> status 11 Jul 7 14:44:30 devbook kore[18191]: [parent]: worker 1 (pid: 18193) (hdlr: 0x242d9c0) gone Jul 7 14:44:30 devbook kore[18191]: [parent]: hdlr serve_intro has caused 2 error(s)
2013-07-07 14:48:32 +02:00
kw->has_lock = 0;
kw->active_hdlr = NULL;
- Better spread load between all worker processes. - Introduce own memory management system on top of malloc to keep track of all our allocations and free's. Later we should introduce a pooling mechanism for fixed size allocations (http_request comes to mind). - Introduce ssl_cipher in configuration. Memory usage is kind of high right now, but it seems its OpenSSL doing it rather then Kore.
2013-06-27 08:43:07 +02:00
Add our messaging framework. With this framework apps can now send messages between worker processes. A new API function exists: int kore_msg_register(u_int8_t id, void (*cb)(const void *, u_int32_t); This API call allows your app to register a new message callback for a given ID. You can then send messages on this ID to other workers using: void kore_msg_send(u_int8_t id, void *data, u_int32_t length); This framework will interally be used for a few things such as allowing websocket data to broadcasted between all workers, adding unified caching and hopefully eventually moving the access log to this as well. Some internals have changed with this commit: * worker_clients has been called connections. * the parent now initializes the net, and event subsystems. * kore_worker_websocket_broadcast() is dead.
2015-06-22 21:13:32 +02:00
if (socketpair(AF_UNIX, SOCK_STREAM, 0, kw->pipe) == -1)
fatal("socketpair(): %s", errno_s);
Kill TCP_NODELAY warnings for socketpair() fds.
2015-06-22 22:11:03 +02:00
if (!kore_connection_nonblock(kw->pipe[0], 0) ||
!kore_connection_nonblock(kw->pipe[1], 0))
Add our messaging framework. With this framework apps can now send messages between worker processes. A new API function exists: int kore_msg_register(u_int8_t id, void (*cb)(const void *, u_int32_t); This API call allows your app to register a new message callback for a given ID. You can then send messages on this ID to other workers using: void kore_msg_send(u_int8_t id, void *data, u_int32_t length); This framework will interally be used for a few things such as allowing websocket data to broadcasted between all workers, adding unified caching and hopefully eventually moving the access log to this as well. Some internals have changed with this commit: * worker_clients has been called connections. * the parent now initializes the net, and event subsystems. * kore_worker_websocket_broadcast() is dead.
2015-06-22 21:13:32 +02:00
fatal("could not set pipe fds to nonblocking: %s", errno_s);
Kore will now keep track of page handlers that cause workers to die. This is useful to track down any issues you might have in your module. A log entry with a page handler causing issues looks like: Jul 7 14:44:30 devbook kore[18191]: [parent]: worker 1 (18193)-> status 11 Jul 7 14:44:30 devbook kore[18191]: [parent]: worker 1 (pid: 18193) (hdlr: 0x242d9c0) gone Jul 7 14:44:30 devbook kore[18191]: [parent]: hdlr serve_intro has caused 2 error(s)
2013-07-07 14:48:32 +02:00
kw->pid = fork();
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
if (kw->pid == -1)
fatal("could not spawn worker child: %s", errno_s);
if (kw->pid == 0) {
kw->pid = getpid();
kore_worker_entry(kw);
/* NOTREACHED */
}
}
Add our messaging framework. With this framework apps can now send messages between worker processes. A new API function exists: int kore_msg_register(u_int8_t id, void (*cb)(const void *, u_int32_t); This API call allows your app to register a new message callback for a given ID. You can then send messages on this ID to other workers using: void kore_msg_send(u_int8_t id, void *data, u_int32_t length); This framework will interally be used for a few things such as allowing websocket data to broadcasted between all workers, adding unified caching and hopefully eventually moving the access log to this as well. Some internals have changed with this commit: * worker_clients has been called connections. * the parent now initializes the net, and event subsystems. * kore_worker_websocket_broadcast() is dead.
2015-06-22 21:13:32 +02:00
struct kore_worker *
kore_worker_data(u_int8_t id)
{
if (id >= worker_count)
fatal("id %u too large for worker count", id);
return (WORKER(id));
}
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
void
kore_worker_shutdown(void)
{
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
struct kore_worker *kw;
u_int16_t id, done;
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
kore_log(LOG_NOTICE, "waiting for workers to drain and shutdown");
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
for (;;) {
done = 0;
for (id = 0; id < worker_count; id++) {
kw = WORKER(id);
if (kw->pid != 0)
kore_worker_wait(1);
else
done++;
}
if (done == worker_count)
break;
}
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
if (shmctl(shm_accept_key, IPC_RMID, NULL) == -1) {
kore_log(LOG_NOTICE,
"failed to deleted shm segment: %s", errno_s);
}
}
void
kore_worker_dispatch_signal(int sig)
{
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
u_int16_t id;
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
struct kore_worker *kw;
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
for (id = 0; id < worker_count; id++) {
kw = WORKER(id);
Brackets would be great here.
2015-01-19 15:26:53 +01:00
if (kill(kw->pid, sig) == -1) {
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
kore_debug("kill(%d, %d): %s", kw->pid, sig, errno_s);
Brackets would be great here.
2015-01-19 15:26:53 +01:00
}
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
}
}
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
void
Separate private keys from worker processes. Kore will now isolate RSA private keys to a separate process (keymgr). Worker processes that require RSA signing for TLS connections will communicate with this keymgr process in order to do so. This behaviour cannot be disabled and is always turned on.
2016-06-08 13:55:14 +02:00
kore_worker_privdrop(void)
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
{
If we're iterating over rlimits, we should use rlim_t. > error: comparison of integers of different signs: 'size_t' (aka 'unsigned long') and 'rlim_t' (aka 'long') [-Werror,-Wsign-compare]
2016-04-28 03:18:21 +02:00
rlim_t fd;
Add worker_rlimit_nofiles as a configurable option.
2014-07-31 09:14:03 +02:00
struct rlimit rl;
Make runas behave similarly to chroot. Add new command line knob '-r', that disables runas similar to '-n', it's implied as well for kore command runs. Add default runas (nobody) user and chroot (/var/empty) path, if none are specified, fallback to these.
2015-05-18 21:34:39 +02:00
struct passwd *pw = NULL;
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
Make runas behave similarly to chroot. Add new command line knob '-r', that disables runas similar to '-n', it's implied as well for kore command runs. Add default runas (nobody) user and chroot (/var/empty) path, if none are specified, fallback to these.
2015-05-18 21:34:39 +02:00
/* Must happen before chroot. */
if (skip_runas == 0) {
pw = getpwnam(runas_user);
if (pw == NULL) {
fatal("cannot getpwnam(\"%s\") runas user: %s",
runas_user, errno_s);
}
}
Kore no longer requires root to be started.
2013-11-18 00:42:57 +01:00
if (skip_chroot == 0) {
small style fix.
2015-05-20 11:36:02 +02:00
if (chroot(chroot_path) == -1) {
fatal("cannot chroot(\"%s\"): %s",
chroot_path, errno_s);
}
Kore no longer requires root to be started.
2013-11-18 00:42:57 +01:00
if (chdir("/") == -1)
Make runas behave similarly to chroot. Add new command line knob '-r', that disables runas similar to '-n', it's implied as well for kore command runs. Add default runas (nobody) user and chroot (/var/empty) path, if none are specified, fallback to these.
2015-05-18 21:34:39 +02:00
fatal("cannot chdir(\"/\"): %s", errno_s);
Kore no longer requires root to be started.
2013-11-18 00:42:57 +01:00
}
Properly deal with accept() failing. In cases where accept() failed Kore would not relinquish the lock towards other worker processes. This becomes evident when dealing with a high number of concurrent connections to the point the fd table gets full. In this scenario the worker with the full fd table will spin on attempt to accept newer connections. As a bonus, Kore now has allows exactly up to worker_max_connections of connections per worker before no longer attempting to grab the accept lock.
2015-04-20 15:17:42 +02:00
if (getrlimit(RLIMIT_NOFILE, &rl) == -1) {
kore_log(LOG_WARNING, "getrlimit(RLIMIT_NOFILE): %s", errno_s);
} else {
for (fd = 0; fd < rl.rlim_cur; fd++) {
if (fcntl(fd, F_GETFD, NULL) != -1) {
worker_rlimit_nofiles++;
}
}
}
call setrlimit() before we drop privs.
2015-04-09 10:06:18 +02:00
rl.rlim_cur = worker_rlimit_nofiles;
rl.rlim_max = worker_rlimit_nofiles;
if (setrlimit(RLIMIT_NOFILE, &rl) == -1) {
kore_log(LOG_ERR, "setrlimit(RLIMIT_NOFILE, %d): %s",
worker_rlimit_nofiles, errno_s);
}
Make runas behave similarly to chroot. Add new command line knob '-r', that disables runas similar to '-n', it's implied as well for kore command runs. Add default runas (nobody) user and chroot (/var/empty) path, if none are specified, fallback to these.
2015-05-18 21:34:39 +02:00
if (skip_runas == 0) {
Kore no longer requires root to be started.
2013-11-18 00:42:57 +01:00
if (setgroups(1, &pw->pw_gid) ||
NetBSD fixes, from David Carlier.
2015-06-14 16:44:37 +02:00
#if defined(__MACH__) || defined(NetBSD)
Fix typo.
2015-06-14 16:50:10 +02:00
setgid(pw->pw_gid) || setegid(pw->pw_gid) ||
Kore no longer requires root to be started.
2013-11-18 00:42:57 +01:00
setuid(pw->pw_uid) || seteuid(pw->pw_uid))
Add OSX support, buildable via "make osx". Make sure you have OpenSSL 1.0.0d+ (available from Macports) installed. Based on diff from Vaibhav Bhembre via github
2013-07-17 20:17:00 +02:00
#else
Kore no longer requires root to be started.
2013-11-18 00:42:57 +01:00
setresgid(pw->pw_gid, pw->pw_gid, pw->pw_gid) ||
setresuid(pw->pw_uid, pw->pw_uid, pw->pw_uid))
Add OSX support, buildable via "make osx". Make sure you have OpenSSL 1.0.0d+ (available from Macports) installed. Based on diff from Vaibhav Bhembre via github
2013-07-17 20:17:00 +02:00
#endif
Make runas behave similarly to chroot. Add new command line knob '-r', that disables runas similar to '-n', it's implied as well for kore command runs. Add default runas (nobody) user and chroot (/var/empty) path, if none are specified, fallback to these.
2015-05-18 21:34:39 +02:00
fatal("cannot drop privileges");
Kore no longer requires root to be started.
2013-11-18 00:42:57 +01:00
}
Separate private keys from worker processes. Kore will now isolate RSA private keys to a separate process (keymgr). Worker processes that require RSA signing for TLS connections will communicate with this keymgr process in order to do so. This behaviour cannot be disabled and is always turned on.
2016-06-08 13:55:14 +02:00
}
void
kore_worker_entry(struct kore_worker *kw)
{
Change kore_preload() and kore_onload(). Renamed both of them: kore_preload -> kore_parent_configure kore_onload -> kore_worker_configure These functions will now always be called if they are defined in any module regardless of your application being built as a single binary or not.
2017-02-01 17:12:11 +01:00
struct kore_runtime_call *rcall;
Add initial python support. Based on work done by Stanislav Yudin.
2017-01-12 23:38:51 +01:00
char buf[16];
int quit, had_lock, r;
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
u_int64_t timerwait, netwait;
u_int64_t now, next_lock, next_prune;
Let workers fetch entropy from keymgr. At bootup and every 1800 seconds after that the worker processes will ask the keymgr for new entropy that they will seed into their PRNG. Additionally once received the worker calls RAND_poll() to grab more entropy from the system to be mixed in.
2017-02-28 06:28:35 +01:00
#if !defined(KORE_NO_TLS)
u_int64_t last_seed;
#endif
Separate private keys from worker processes. Kore will now isolate RSA private keys to a separate process (keymgr). Worker processes that require RSA signing for TLS connections will communicate with this keymgr process in order to do so. This behaviour cannot be disabled and is always turned on.
2016-06-08 13:55:14 +02:00
worker = kw;
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
Correct snprintf() usage
2014-04-23 16:29:58 +02:00
(void)snprintf(buf, sizeof(buf), "kore [wrk %d]", kw->id);
Separate private keys from worker processes. Kore will now isolate RSA private keys to a separate process (keymgr). Worker processes that require RSA signing for TLS connections will communicate with this keymgr process in order to do so. This behaviour cannot be disabled and is always turned on.
2016-06-08 13:55:14 +02:00
#if !defined(KORE_NO_TLS)
if (kw->id == KORE_WORKER_KEYMGR)
(void)snprintf(buf, sizeof(buf), "kore [keymgr]");
#endif
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
kore_platform_proctitle(buf);
Add new configuration option: worker_set_affinity. Controls wether or not worker processes sets its affinity mask to bind themselves to a single CPU. Defaults to on (1).
2015-04-27 10:36:33 +02:00
if (worker_set_affinity == 1)
kore_platform_worker_setcpu(kw);
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
kore_pid = kw->pid;
sig_recv = 0;
signal(SIGHUP, kore_signal);
signal(SIGQUIT, kore_signal);
Let SIGTERM properly shutdown Kore as well.
2016-01-31 16:48:17 +01:00
signal(SIGTERM, kore_signal);
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
signal(SIGPIPE, SIG_IGN);
Allow Kore to be started in foreground mode (-f) Sending SIGINT or Ctrl-c will reap all workers and exit again. Useful for development.
2014-07-31 13:27:04 +02:00
if (foreground)
signal(SIGINT, kore_signal);
else
signal(SIGINT, SIG_IGN);
Separate private keys from worker processes. Kore will now isolate RSA private keys to a separate process (keymgr). Worker processes that require RSA signing for TLS connections will communicate with this keymgr process in order to do so. This behaviour cannot be disabled and is always turned on.
2016-06-08 13:55:14 +02:00
#if !defined(KORE_NO_TLS)
if (kw->id == KORE_WORKER_KEYMGR) {
kore_keymgr_run();
exit(0);
}
#endif
kore_worker_privdrop();
Add fixed size memory pools and use them throughout Kore.
2013-07-15 10:13:36 +02:00
net_init();
Introduce NOHTTP=1 build option. This basically turns off the HTTP layer for Kore. It does not compile in anything for HTTP. This allows Kore to be used as a network application platform as well. Added an example for this called nohttp. Other changes that sneaked in while hacking on this: * Use calloc(), kill pendantic malloc option. * Killed off SPDY/3.1 support completely, will be superseded by http2 Note that comes with massive changes to a lot of the core API functions provided by Kore, these might break your application.
2015-11-27 16:22:50 +01:00
#if !defined(KORE_NO_HTTP)
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
http_init();
Introduce NOHTTP=1 build option. This basically turns off the HTTP layer for Kore. It does not compile in anything for HTTP. This allows Kore to be used as a network application platform as well. Added an example for this called nohttp. Other changes that sneaked in while hacking on this: * Use calloc(), kill pendantic malloc option. * Killed off SPDY/3.1 support completely, will be superseded by http2 Note that comes with massive changes to a lot of the core API functions provided by Kore, these might break your application.
2015-11-27 16:22:50 +01:00
kore_accesslog_worker_init();
#endif
Add rudimentary timers to Kore. Timers are in ms resolution and are added using kore_timer_add(cb, interval, flags). Both oneshot timers and continious timers are supported.
2015-04-06 18:54:35 +02:00
kore_timer_init();
Add fixed size memory pools and use them throughout Kore.
2013-07-15 10:13:36 +02:00
kore_connection_init();
Add CRL support. Allow Kore to use per domain CRLs when requiring client certificates. The require_client_cert configuration option has been renamed to a more sane client_certificates and can optionally take a second argument which is the CRL in pem format. You'll need a restart in case the CRLs get updated.
2014-10-18 02:32:05 +02:00
kore_domain_load_crl();
Separate private keys from worker processes. Kore will now isolate RSA private keys to a separate process (keymgr). Worker processes that require RSA signing for TLS connections will communicate with this keymgr process in order to do so. This behaviour cannot be disabled and is always turned on.
2016-06-08 13:55:14 +02:00
kore_domain_keymgr_init();
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
quit = 0;
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
had_lock = 0;
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
next_lock = 0;
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
next_prune = 0;
set worker_active_connections to 0 always. fixes a bug where if a worker died it contained garbled data...
2016-12-04 16:49:42 +01:00
worker_active_connections = 0;
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
kore_platform_event_init();
Add our messaging framework. With this framework apps can now send messages between worker processes. A new API function exists: int kore_msg_register(u_int8_t id, void (*cb)(const void *, u_int32_t); This API call allows your app to register a new message callback for a given ID. You can then send messages on this ID to other workers using: void kore_msg_send(u_int8_t id, void *data, u_int32_t length); This framework will interally be used for a few things such as allowing websocket data to broadcasted between all workers, adding unified caching and hopefully eventually moving the access log to this as well. Some internals have changed with this commit: * worker_clients has been called connections. * the parent now initializes the net, and event subsystems. * kore_worker_websocket_broadcast() is dead.
2015-06-22 21:13:32 +02:00
kore_msg_worker_init();
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
#if defined(KORE_USE_PGSQL)
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
kore_pgsql_sys_init();
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
#endif
Add initial stab at asynchronous background tasks. More to follow.
2014-06-28 16:17:18 +02:00
#if defined(KORE_USE_TASKS)
kore_task_init();
#endif
Let workers fetch entropy from keymgr. At bootup and every 1800 seconds after that the worker processes will ask the keymgr for new entropy that they will seed into their PRNG. Additionally once received the worker calls RAND_poll() to grab more entropy from the system to be mixed in.
2017-02-28 06:28:35 +01:00
#if !defined(KORE_NO_TLS)
last_seed = 0;
kore_msg_register(KORE_MSG_ENTROPY_RESP, worker_entropy_recv);
#endif
Allow workers to listen on different ports. This commit allows worker processes to have individual listeners not configured by the kore configuration. If you do the following: - do not configure listeners in your .conf file - call kore_server_bind() in kore_worker_configure() the workers will no longer fight over accept locks as the configured ports no longer conflict with each other. This allows me to create X amount of instances of a worker process that are each individually accessible via unique ports.
2017-03-13 15:32:04 +01:00
if (nlisteners == 0)
worker_no_lock = 1;
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
kore_log(LOG_NOTICE, "worker %d started (cpu#%d)", kw->id, kw->cpu);
Make kore_onload() to be done per worker. Mimics the behaviour of dso builds.
2016-07-06 21:59:17 +02:00
Change kore_preload() and kore_onload(). Renamed both of them: kore_preload -> kore_parent_configure kore_onload -> kore_worker_configure These functions will now always be called if they are defined in any module regardless of your application being built as a single binary or not.
2017-02-01 17:12:11 +01:00
rcall = kore_runtime_getcall("kore_worker_configure");
Add initial python support. Based on work done by Stanislav Yudin.
2017-01-12 23:38:51 +01:00
if (rcall != NULL) {
Teach single binaries about SIGHUP. Make sure kore_onload() and kore_preload() can be called either in native or python runtime.
2017-01-26 13:26:55 +01:00
kore_runtime_execute(rcall);
Add initial python support. Based on work done by Stanislav Yudin.
2017-01-12 23:38:51 +01:00
kore_free(rcall);
}
Change kore_preload() and kore_onload(). Renamed both of them: kore_preload -> kore_parent_configure kore_onload -> kore_worker_configure These functions will now always be called if they are defined in any module regardless of your application being built as a single binary or not.
2017-02-01 17:12:11 +01:00
Teach single binaries about SIGHUP. Make sure kore_onload() and kore_preload() can be called either in native or python runtime.
2017-01-26 13:26:55 +01:00
kore_module_onload();
- Better spread load between all worker processes. - Introduce own memory management system on top of malloc to keep track of all our allocations and free's. Later we should introduce a pooling mechanism for fixed size allocations (http_request comes to mind). - Introduce ssl_cipher in configuration. Memory usage is kind of high right now, but it seems its OpenSSL doing it rather then Kore.
2013-06-27 08:43:07 +02:00
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
for (;;) {
if (sig_recv != 0) {
Let SIGTERM properly shutdown Kore as well.
2016-01-31 16:48:17 +01:00
switch (sig_recv) {
case SIGHUP:
Don't call module onloads until after everything is initialized. This means the onload functions for a module are now called after a worker has started and never from the parent ever again.
2014-06-29 21:15:23 +02:00
kore_module_reload(1);
Let SIGTERM properly shutdown Kore as well.
2016-01-31 16:48:17 +01:00
break;
case SIGQUIT:
case SIGINT:
case SIGTERM:
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
quit = 1;
Let SIGTERM properly shutdown Kore as well.
2016-01-31 16:48:17 +01:00
break;
default:
break;
}
Add CRL support. Allow Kore to use per domain CRLs when requiring client certificates. The require_client_cert configuration option has been renamed to a more sane client_certificates and can optionally take a second argument which is the CRL in pem format. You'll need a restart in case the CRLs get updated.
2014-10-18 02:32:05 +02:00
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
sig_recv = 0;
}
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
netwait = 100;
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
now = kore_time_ms();
Let workers fetch entropy from keymgr. At bootup and every 1800 seconds after that the worker processes will ask the keymgr for new entropy that they will seed into their PRNG. Additionally once received the worker calls RAND_poll() to grab more entropy from the system to be mixed in.
2017-02-28 06:28:35 +01:00
#if !defined(KORE_NO_TLS)
if ((now - last_seed) > KORE_RESEED_TIME) {
kore_msg_send(KORE_WORKER_KEYMGR,
KORE_MSG_ENTROPY_REQ, NULL, 0);
last_seed = now;
}
#endif
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
if (!worker->has_lock && next_lock <= now) {
if (kore_worker_acceptlock_obtain(now)) {
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
if (had_lock == 0) {
kore_platform_enable_accept();
had_lock = 1;
}
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
} else {
next_lock = now + WORKER_LOCK_TIMEOUT / 2;
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
}
}
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
if (!worker->has_lock) {
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
if (worker_active_connections > 0) {
if (next_lock > now)
netwait = next_lock - now;
} else {
netwait = 10;
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
}
If we do not own the accept lock, reduce timeout for platform wait. This way multiple workers won't stall when passing the accept lock around for at maximum 100 ms (time spent in epoll/kqueue waits).
2014-07-28 23:27:58 +02:00
}
- Better spread load between all worker processes. - Introduce own memory management system on top of malloc to keep track of all our allocations and free's. Later we should introduce a pooling mechanism for fixed size allocations (http_request comes to mind). - Introduce ssl_cipher in configuration. Memory usage is kind of high right now, but it seems its OpenSSL doing it rather then Kore.
2013-06-27 08:43:07 +02:00
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
timerwait = kore_timer_run(now);
if (timerwait < netwait)
netwait = timerwait;
Add rudimentary timers to Kore. Timers are in ms resolution and are added using kore_timer_add(cb, interval, flags). Both oneshot timers and continious timers are supported.
2015-04-06 18:54:35 +02:00
r = kore_platform_event_wait(netwait);
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
if (worker->has_lock && r > 0) {
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
if (netwait > 10)
now = kore_time_ms();
if (kore_worker_acceptlock_release(now))
next_lock = now + WORKER_LOCK_TIMEOUT;
}
if (!worker->has_lock) {
if (had_lock == 1) {
had_lock = 0;
kore_platform_disable_accept();
}
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
}
- Better spread load between all worker processes. - Introduce own memory management system on top of malloc to keep track of all our allocations and free's. Later we should introduce a pooling mechanism for fixed size allocations (http_request comes to mind). - Introduce ssl_cipher in configuration. Memory usage is kind of high right now, but it seems its OpenSSL doing it rather then Kore.
2013-06-27 08:43:07 +02:00
Introduce NOHTTP=1 build option. This basically turns off the HTTP layer for Kore. It does not compile in anything for HTTP. This allows Kore to be used as a network application platform as well. Added an example for this called nohttp. Other changes that sneaked in while hacking on this: * Use calloc(), kill pendantic malloc option. * Killed off SPDY/3.1 support completely, will be superseded by http2 Note that comes with massive changes to a lot of the core API functions provided by Kore, these might break your application.
2015-11-27 16:22:50 +01:00
#if !defined(KORE_NO_HTTP)
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
http_process();
Introduce NOHTTP=1 build option. This basically turns off the HTTP layer for Kore. It does not compile in anything for HTTP. This allows Kore to be used as a network application platform as well. Added an example for this called nohttp. Other changes that sneaked in while hacking on this: * Use calloc(), kill pendantic malloc option. * Killed off SPDY/3.1 support completely, will be superseded by http2 Note that comes with massive changes to a lot of the core API functions provided by Kore, these might break your application.
2015-11-27 16:22:50 +01:00
#endif
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
if (next_prune <= now) {
kore_connection_check_timeout(now);
kore_connection_prune(KORE_CONNECTION_PRUNE_DISCONNECT);
next_prune = now + 500;
}
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
Introduce NOHTTP=1 build option. This basically turns off the HTTP layer for Kore. It does not compile in anything for HTTP. This allows Kore to be used as a network application platform as well. Added an example for this called nohttp. Other changes that sneaked in while hacking on this: * Use calloc(), kill pendantic malloc option. * Killed off SPDY/3.1 support completely, will be superseded by http2 Note that comes with massive changes to a lot of the core API functions provided by Kore, these might break your application.
2015-11-27 16:22:50 +01:00
if (quit)
break;
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
}
rename *_fini to *_cleanup
2016-01-04 22:40:14 +01:00
kore_platform_event_cleanup();
kore_connection_cleanup();
Added cleanup for the module handlers and validators
2016-02-01 12:30:20 +01:00
kore_domain_cleanup();
Separate private keys from worker processes. Kore will now isolate RSA private keys to a separate process (keymgr). Worker processes that require RSA signing for TLS connections will communicate with this keymgr process in order to do so. This behaviour cannot be disabled and is always turned on.
2016-06-08 13:55:14 +02:00
kore_module_cleanup();
Add resource management as part of the kore shutdown process.
2015-12-29 20:39:39 +01:00
#if !defined(KORE_NO_HTTP)
rename *_fini to *_cleanup
2016-01-04 22:40:14 +01:00
http_cleanup();
Add resource management as part of the kore shutdown process.
2015-12-29 20:39:39 +01:00
#endif
rename *_fini to *_cleanup
2016-01-04 22:40:14 +01:00
net_cleanup();
Separate private keys from worker processes. Kore will now isolate RSA private keys to a separate process (keymgr). Worker processes that require RSA signing for TLS connections will communicate with this keymgr process in order to do so. This behaviour cannot be disabled and is always turned on.
2016-06-08 13:55:14 +02:00
Add initial python support. Based on work done by Stanislav Yudin.
2017-01-12 23:38:51 +01:00
#if defined(KORE_USE_PYTHON)
kore_python_cleanup();
#endif
Add asynchronous pgsql query support to python. This commit adds the ability to use python "await" to suspend execution of your page handler until the query sent to postgresql has returned a result. This is built upon the existing asynchrous query framework Kore had. With this you can now write stuff like: async def page(req): result = await req.pgsql("db", "SELECT name FROM table"); req.response(200, json.dumps(result).encode("utf-8")) The above code will fire off a query and suspend itself so Kore can take care of business as usual until the query is successful at which point Kore will jump back into the handler and resume. This does not use threading, it's purely based on Python's excellent coroutines and generators and Kore its built-in pgsql support.
2017-02-06 11:42:53 +01:00
#if defined(KORE_USE_PGSQL)
kore_pgsql_sys_cleanup();
#endif
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
kore_debug("worker %d shutting down", kw->id);
Add asynchronous pgsql query support to python. This commit adds the ability to use python "await" to suspend execution of your page handler until the query sent to postgresql has returned a result. This is built upon the existing asynchrous query framework Kore had. With this you can now write stuff like: async def page(req): result = await req.pgsql("db", "SELECT name FROM table"); req.response(200, json.dumps(result).encode("utf-8")) The above code will fire off a query and suspend itself so Kore can take care of business as usual until the query is successful at which point Kore will jump back into the handler and resume. This does not use threading, it's purely based on Python's excellent coroutines and generators and Kore its built-in pgsql support.
2017-02-06 11:42:53 +01:00
kore_mem_cleanup();
refactor code quite a bit.
2013-06-26 11:18:32 +02:00
exit(0);
}
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
void
kore_worker_wait(int final)
{
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
u_int16_t id;
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
pid_t pid;
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
struct kore_worker *kw;
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
int status;
if (final)
pid = waitpid(WAIT_ANY, &status, 0);
else
pid = waitpid(WAIT_ANY, &status, WNOHANG);
if (pid == -1) {
kore_debug("waitpid(): %s", errno_s);
return;
}
if (pid == 0)
return;
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
for (id = 0; id < worker_count; id++) {
kw = WORKER(id);
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
if (kw->pid != pid)
continue;
kore_log(LOG_NOTICE, "worker %d (%d)-> status %d",
kw->id, pid, status);
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
if (final) {
kw->pid = 0;
break;
}
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
if (WEXITSTATUS(status) || WTERMSIG(status) ||
WCOREDUMP(status)) {
kore_log(LOG_NOTICE,
use handler function name if available when logging a worker process that has died.
2013-07-07 14:56:50 +02:00
"worker %d (pid: %d) (hdlr: %s) gone",
kw->id, kw->pid,
(kw->active_hdlr != NULL) ? kw->active_hdlr->func :
"none");
If the worker that owns the accept lock dies, make sure the next one in line can grab it. Otherwise kore comes to a halt (by not accepting new connections).
2013-07-05 20:19:50 +02:00
Separate private keys from worker processes. Kore will now isolate RSA private keys to a separate process (keymgr). Worker processes that require RSA signing for TLS connections will communicate with this keymgr process in order to do so. This behaviour cannot be disabled and is always turned on.
2016-06-08 13:55:14 +02:00
#if !defined(KORE_NO_TLS)
if (id == KORE_WORKER_KEYMGR) {
kore_log(LOG_CRIT, "keymgr gone, stopping");
kw->pid = 0;
if (raise(SIGTERM) != 0) {
kore_log(LOG_WARNING,
"failed to raise SIGTERM signal");
}
break;
}
#endif
Allow workers to listen on different ports. This commit allows worker processes to have individual listeners not configured by the kore configuration. If you do the following: - do not configure listeners in your .conf file - call kore_server_bind() in kore_worker_configure() the workers will no longer fight over accept locks as the configured ports no longer conflict with each other. This allows me to create X amount of instances of a worker process that are each individually accessible via unique ports.
2017-03-13 15:32:04 +01:00
if (kw->pid == accept_lock->current &&
worker_no_lock == 0)
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
worker_unlock();
Kore will now keep track of page handlers that cause workers to die. This is useful to track down any issues you might have in your module. A log entry with a page handler causing issues looks like: Jul 7 14:44:30 devbook kore[18191]: [parent]: worker 1 (18193)-> status 11 Jul 7 14:44:30 devbook kore[18191]: [parent]: worker 1 (pid: 18193) (hdlr: 0x242d9c0) gone Jul 7 14:44:30 devbook kore[18191]: [parent]: hdlr serve_intro has caused 2 error(s)
2013-07-07 14:48:32 +02:00
if (kw->active_hdlr != NULL) {
kw->active_hdlr->errors++;
kore_log(LOG_NOTICE,
"hdlr %s has caused %d error(s)",
kw->active_hdlr->func,
kw->active_hdlr->errors);
If the worker that owns the accept lock dies, make sure the next one in line can grab it. Otherwise kore comes to a halt (by not accepting new connections).
2013-07-05 20:19:50 +02:00
}
Kore will now keep track of page handlers that cause workers to die. This is useful to track down any issues you might have in your module. A log entry with a page handler causing issues looks like: Jul 7 14:44:30 devbook kore[18191]: [parent]: worker 1 (18193)-> status 11 Jul 7 14:44:30 devbook kore[18191]: [parent]: worker 1 (pid: 18193) (hdlr: 0x242d9c0) gone Jul 7 14:44:30 devbook kore[18191]: [parent]: hdlr serve_intro has caused 2 error(s)
2013-07-07 14:48:32 +02:00
kore_log(LOG_NOTICE, "restarting worker %d", kw->id);
Add our messaging framework. With this framework apps can now send messages between worker processes. A new API function exists: int kore_msg_register(u_int8_t id, void (*cb)(const void *, u_int32_t); This API call allows your app to register a new message callback for a given ID. You can then send messages on this ID to other workers using: void kore_msg_send(u_int8_t id, void *data, u_int32_t length); This framework will interally be used for a few things such as allowing websocket data to broadcasted between all workers, adding unified caching and hopefully eventually moving the access log to this as well. Some internals have changed with this commit: * worker_clients has been called connections. * the parent now initializes the net, and event subsystems. * kore_worker_websocket_broadcast() is dead.
2015-06-22 21:13:32 +02:00
kore_msg_parent_remove(kw);
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
kore_worker_spawn(kw->id, kw->cpu);
Add our messaging framework. With this framework apps can now send messages between worker processes. A new API function exists: int kore_msg_register(u_int8_t id, void (*cb)(const void *, u_int32_t); This API call allows your app to register a new message callback for a given ID. You can then send messages on this ID to other workers using: void kore_msg_send(u_int8_t id, void *data, u_int32_t length); This framework will interally be used for a few things such as allowing websocket data to broadcasted between all workers, adding unified caching and hopefully eventually moving the access log to this as well. Some internals have changed with this commit: * worker_clients has been called connections. * the parent now initializes the net, and event subsystems. * kore_worker_websocket_broadcast() is dead.
2015-06-22 21:13:32 +02:00
kore_msg_parent_add(kw);
- Better spread load between all worker processes. - Introduce own memory management system on top of malloc to keep track of all our allocations and free's. Later we should introduce a pooling mechanism for fixed size allocations (http_request comes to mind). - Introduce ssl_cipher in configuration. Memory usage is kind of high right now, but it seems its OpenSSL doing it rather then Kore.
2013-06-27 08:43:07 +02:00
} else {
kore_log(LOG_NOTICE,
Kore will now keep track of page handlers that cause workers to die. This is useful to track down any issues you might have in your module. A log entry with a page handler causing issues looks like: Jul 7 14:44:30 devbook kore[18191]: [parent]: worker 1 (18193)-> status 11 Jul 7 14:44:30 devbook kore[18191]: [parent]: worker 1 (pid: 18193) (hdlr: 0x242d9c0) gone Jul 7 14:44:30 devbook kore[18191]: [parent]: hdlr serve_intro has caused 2 error(s)
2013-07-07 14:48:32 +02:00
"worker %d (pid: %d) signaled us (%d)",
kw->id, kw->pid, status);
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
}
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
break;
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
}
}
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
static inline int
kore_worker_acceptlock_release(u_int64_t now)
Make kore_worker_acceptlock_release() available to loaded modules. This allows us to move the accept lock manually to another Kore worker in case we own it and are about to do some heavy lifting on the current Kore worker.
2013-10-24 09:05:46 +02:00
{
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
if (worker_count == WORKER_SOLO_COUNT || worker_no_lock == 1)
return (0);
Make kore_worker_acceptlock_release() available to loaded modules. This allows us to move the accept lock manually to another Kore worker in case we own it and are about to do some heavy lifting on the current Kore worker.
2013-10-24 09:05:46 +02:00
if (worker->has_lock != 1)
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
return (0);
if (worker_active_connections < worker_max_connections) {
#if !defined(KORE_NO_HTTP)
if (http_request_count < http_request_limit)
return (0);
#else
return (0);
#endif
}
#if defined(WORKER_DEBUG)
kore_log(LOG_DEBUG, "worker busy, releasing lock");
kore_log(LOG_DEBUG, "had lock for %lu ms", now - worker->time_locked);
#endif
Make kore_worker_acceptlock_release() available to loaded modules. This allows us to move the accept lock manually to another Kore worker in case we own it and are about to do some heavy lifting on the current Kore worker.
2013-10-24 09:05:46 +02:00
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
worker_unlock();
worker->has_lock = 0;
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
return (1);
Make kore_worker_acceptlock_release() available to loaded modules. This allows us to move the accept lock manually to another Kore worker in case we own it and are about to do some heavy lifting on the current Kore worker.
2013-10-24 09:05:46 +02:00
}
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
static inline int
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
kore_worker_acceptlock_obtain(u_int64_t now)
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
{
If we do not own the accept lock, reduce timeout for platform wait. This way multiple workers won't stall when passing the accept lock around for at maximum 100 ms (time spent in epoll/kqueue waits).
2014-07-28 23:27:58 +02:00
int r;
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
if (worker->has_lock == 1)
return (1);
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
if (worker_count == WORKER_SOLO_COUNT || worker_no_lock == 1) {
- Better spread load between all worker processes. - Introduce own memory management system on top of malloc to keep track of all our allocations and free's. Later we should introduce a pooling mechanism for fixed size allocations (http_request comes to mind). - Introduce ssl_cipher in configuration. Memory usage is kind of high right now, but it seems its OpenSSL doing it rather then Kore.
2013-06-27 08:43:07 +02:00
worker->has_lock = 1;
If we do not own the accept lock, reduce timeout for platform wait. This way multiple workers won't stall when passing the accept lock around for at maximum 100 ms (time spent in epoll/kqueue waits).
2014-07-28 23:27:58 +02:00
return (1);
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
}
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
if (worker_active_connections >= worker_max_connections)
return (0);
If we do not own the accept lock, reduce timeout for platform wait. This way multiple workers won't stall when passing the accept lock around for at maximum 100 ms (time spent in epoll/kqueue waits).
2014-07-28 23:27:58 +02:00
Let http_request_limit matter. Before http_request_limit just constrained the number of HTTP requests we'd deal with in a single http_process_requests() call. But it should really mean how many maximum HTTP requests are allowed to be alive in the worker process before we start sending 503s back. While here, drop the lock timeout for a worker to 100ms down from 500ms and do not allow a worker to grab the accept lock if their HTTP request queue is full. This makes things much more pleasant memory wise as the http_request_pool won't just grow over time.
2018-02-13 11:56:51 +01:00
#if !defined(KORE_NO_HTTP)
if (http_request_count >= http_request_limit)
return (0);
#endif
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
r = 0;
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
if (worker_trylock()) {
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
r = 1;
worker->has_lock = 1;
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
worker->time_locked = now;
#if defined(WORKER_DEBUG)
kore_log(LOG_DEBUG, "got lock");
#endif
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
}
If we do not own the accept lock, reduce timeout for platform wait. This way multiple workers won't stall when passing the accept lock around for at maximum 100 ms (time spent in epoll/kqueue waits).
2014-07-28 23:27:58 +02:00
return (r);
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
}
static int
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
worker_trylock(void)
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
{
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
if (!__sync_bool_compare_and_swap(&(accept_lock->lock), 0, 1))
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
return (0);
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
accept_lock->current = worker->pid;
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
return (1);
Add a form of synchronization between what worker will be accepting new connections and which ones will not be notified for it. Fixes the thundering herd problem, and nicely spreads out load between all the workers equally. A configuration option (workers_max_connections) is available to tweak how many connections a worker will have before giving up the accept lock. Two ways are added to this commit for access locking: - Locking via semaphores. - Locking via GCC's builtin atomic methods. The default is running with semaphores disabled (OpenBSD cannot do sem_init() with pshared set to 1, which is required). If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS, and -lpthread to LDFLAGS in the Makefile. Other fixes: - BSD: add a timeout to kevent(). - Merge kore_worker_wait together, linux knows waitpid() as well. - Send the correct SIGQUIT signal to workers instead of SIGINT. - Fix kore_time_ms(). - Log fatal worker messages in syslog. - Refactor code even more. - Do not free our own kore_worker structure.
2013-06-26 16:37:22 +02:00
}
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
static void
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
worker_unlock(void)
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
{
Rework the accept lock. Kore no longer passes the accept lock to the "next in line" worker but instead all workers will attempt to grab the lock if they can. Also remember if we had the lock previous iteration of the event loop and don't constantly disable/enable the accepting sockets. Makes Kore scale even better across multiple cpu's.
2014-07-30 15:20:09 +02:00
accept_lock->current = 0;
if (!__sync_bool_compare_and_swap(&(accept_lock->lock), 1, 0))
kore_log(LOG_NOTICE, "worker_unlock(): wasnt locked");
Rework the way worker processes give each other the accept lock. Instead of waiting until one worker is filled up on connections the workers find the next lowest loaded worker and will hand over the lock to them instead. This will cause a nicer spread of load. Instead of running one accept per event loop, we attempt to accept as many as worker_max_connections allows. Refactor net sending/recv code a bit.
2013-06-27 00:22:48 +02:00
}
Let workers fetch entropy from keymgr. At bootup and every 1800 seconds after that the worker processes will ask the keymgr for new entropy that they will seed into their PRNG. Additionally once received the worker calls RAND_poll() to grab more entropy from the system to be mixed in.
2017-02-28 06:28:35 +01:00
#if !defined(KORE_NO_TLS)
static void
worker_entropy_recv(struct kore_msg *msg, const void *data)
{
add warning + RAND_poll() to callback.
2017-02-28 06:31:09 +01:00
if (msg->length != 1024) {
kore_log(LOG_WARNING,
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
"invalid entropy response (got:%u - wanted:1024)",
add warning + RAND_poll() to callback.
2017-02-28 06:31:09 +01:00
msg->length);
}
RAND_poll();
Let workers fetch entropy from keymgr. At bootup and every 1800 seconds after that the worker processes will ask the keymgr for new entropy that they will seed into their PRNG. Additionally once received the worker calls RAND_poll() to grab more entropy from the system to be mixed in.
2017-02-28 06:28:35 +01:00
RAND_seed(data, msg->length);
}
#endif