By default kore will restart worker processes if they terminate
unexpected. However in certain scenarios you may want to bring down
an entire kore instance if a worker process fails.
By setting worker_death_policy to "terminate" the Kore server will
completely stop if a worker exits unexpected.
this change also stops python coroutines from waking up very
late after their timeout has expired.
in filerefs, don't prime the timer until we actually have something
to expire, and kill the timer when the last ref drops.
With these changes CRLs can be reloaded like certificates
by sending a SIGUSR1 to the parent process.
Track mtime on both certificate files and CRL files as well
and only submit them to the workers if this has changed.
Move away from the parent constantly hitting the disk for every
accesslog the workers are sending.
The workers will now write their own accesslogs to shared
memory before the parent will pick those up. The parent
will flush them to disk once every second or if they grow
larger then 1MB.
This removes the heavy penalty for having access logs
turned on when you are dealing with a large volume
of requests.
This adds kore.proc to the python runtime allowing async processing
handling:
The kore.proc method takes the command to run and an optional timeout
parameter in milliseconds. If the process did not exit normally after
that amount of time a TimeoutError exception is raised.
For instance:
async def run(cmd):
proc = kore.proc(cmd, 1000)
try:
await proc.send("hello")
proc.close_stdin()
except TimeoutError:
proc.kill()
retcode = await proc.reap()
return retcode
If exists these functions are called when the worker is exiting
and when right before the parent exists.
Allows for cleanup code for applications if need to do cleanup on exit.
This means you can now do things like:
resp = await koresock.recv(1024)
await koresock.send(resp)
directly from page handlers if they are defined as async.
Adds lots more to the python goo such as fatalx(), bind_unix(),
task_create() and socket_wrap().
All worker processes will now call pledge(2) after dropping
privileges (even if -rn was specified).
By default Kore will use the following promises:
"stdio rpath inet error"
If your application requires more privileges, you can add more pledges
by setting them in your configuration using the 'pledge' directive:
pledge dns wpath
Calling this from your page handler will cause your current worker
to give up the acceptlock (if it holds it).
This is particularly useful if you are about to run code that may block
a bit longer then you are comfortable with. Calling this will cause
the acceptlock to shuffle to another free worker which in turn makes
sure your application can keep accepting requests.
This commit introduces the ability for the keymgr process
to reload the certificates/keys for domains when receiving
a SIGUSR1 signal.
The keymgr receives 2 new configuration options:
- keymgr_root_path
The root path where the keymgr will live.
If -n is not specified when the application starts the
keymgr process will chroot into here.
- keymgr_runas_user
The user the keymgr will drop privileges towards if
-r was not specified.
All certfile and certkey configuration options are now relative to the
keymgr_root_path configuration setting.
The keymgr process will now also load the certificate for the domain
(rather then the workers) and submit these to the worker processes so
they can be reloaded when required.
Worker processes will refuse connections until the TLS configuration
for a given domain is completed (aka: the workers receive the certificate
for that domain).
Other changes:
- client_certificates renamed to client_verify.
- the chroot configuration option is now called root.
- kore is a little more verbose if privsep options are missing.
- filemaps are now relative to the root configuration option.
A filemap is a way of telling Kore to serve files from a directory
much like a traditional webserver can do.
Kore filemaps only handles files. Kore does not generate directory
indexes or deal with non-regular files.
The way files are sent to a client differs a bit per platform and
build options:
default:
- mmap() backed file transfer due to TLS.
NOTLS=1
- sendfile() under FreeBSD, macOS and Linux.
- mmap() backed file for OpenBSD.
The opened file descriptors/mmap'd regions are cached and reused when
appropriate. If a file is no longer in use it will be closed and evicted
from the cache after 30 seconds.
New API's are available allowing developers to use these facilities via:
void net_send_fileref(struct connection *, struct kore_fileref *);
void http_response_fileref(struct http_request *, struct kore_fileref *);
Kore will attempt to match media types based on file extensions. A few
default types are built-in. Others can be added via the new "http_media_type"
configuration directive.
The HTTP layer used to make a copy of each incoming header and its
value for a request. Stop doing that and make HTTP headers zero-copy
all across the board.
This change comes with some api function changes, notably the
http_request_header() function which now takes a const char ** rather
than a char ** out pointer.
This commit also constifies several members of http_request, beware.
Additional rework how the worker processes deal with the accept lock.
Before:
if a worker held the accept lock and it accepted a new connection
it would release the lock for others and back off for 500ms before
attempting to grab the lock again.
This approach worked but under high load this starts becoming obvious.
Now:
- workers not holding the accept lock and not having any connections
will wait less long before returning from kore_platform_event_wait().
- workers not holding the accept lock will no longer blindly wait
an arbitrary amount in kore_platform_event_wait() but will look
at how long until the next lock grab is and base their timeout
on that.
- if a worker its next_lock timeout is up and failed to grab the
lock it will try again in half the time again.
- the worker process holding the lock will when releasing the lock
double check if it still has space for newer connections, if it does
it will keep the lock until it is full. This prevents the lock from
bouncing between several non busy worker processes all the time.
Additional fixes:
- Reduce the number of times we check the timeout list, only do it twice
per second rather then every event tick.
- Fix solo worker count for TLS (we actually hold two processes, not one).
- Make sure we don't accidentally miscalculate the idle time causing new
connections under heavy load to instantly drop.
- Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
Before http_request_limit just constrained the number of HTTP
requests we'd deal with in a single http_process_requests() call.
But it should really mean how many maximum HTTP requests are allowed
to be alive in the worker process before we start sending 503s back.
While here, drop the lock timeout for a worker to 100ms down from 500ms
and do not allow a worker to grab the accept lock if their HTTP request
queue is full.
This makes things much more pleasant memory wise as the http_request_pool
won't just grow over time.
doing this allows us to get rid of the validator reload
and handler reload as well as fixing websocket runtime
callbacks which were never being resolved upon module reloads.
When the pgsql layer was introduced it was tightly coupled with the
http layer in order to make async work fluently.
The time has come to split these up and follow the same method we
used for tasks, allowing either http requests to be tied to a pgsql
data structure or a simple callback function.
This also reworks the internal queueing of pgsql requests until
connections to the db are available again.
The following API functions were changes:
- kore_pgsql_query_init() -> kore_pgsql_setup()
no longer takes an http_request parameter.
- NEW kore_pgsql_init()
must be called before operating on an kore_pgsql structure.
- NEW kore_pgsql_bind_request()
binds an http_request to a kore_pgsql data structure.
- NEW kore_pgsql_bind_callback()
binds a callback to a kore_pgsql data structure.
With all of this you can now build kore with PGSQL=1 NOHTTP=1.
The pgsql/ example has been updated to reflect these changes and
new features.
This commit allows worker processes to have individual listeners
not configured by the kore configuration.
If you do the following:
- do not configure listeners in your .conf file
- call kore_server_bind() in kore_worker_configure()
the workers will no longer fight over accept locks as the configured
ports no longer conflict with each other.
This allows me to create X amount of instances of a worker process that
are each individually accessible via unique ports.
At bootup and every 1800 seconds after that the worker processes will
ask the keymgr for new entropy that they will seed into their PRNG.
Additionally once received the worker calls RAND_poll() to grab
more entropy from the system to be mixed in.
This commit adds the ability to use python "await" to suspend
execution of your page handler until the query sent to postgresql
has returned a result.
This is built upon the existing asynchrous query framework Kore had.
With this you can now write stuff like:
async def page(req):
result = await req.pgsql("db", "SELECT name FROM table");
req.response(200, json.dumps(result).encode("utf-8"))
The above code will fire off a query and suspend itself so Kore can
take care of business as usual until the query is successful at which
point Kore will jump back into the handler and resume.
This does not use threading, it's purely based on Python's excellent
coroutines and generators and Kore its built-in pgsql support.
Renamed both of them:
kore_preload -> kore_parent_configure
kore_onload -> kore_worker_configure
These functions will now always be called if they are defined in any module
regardless of your application being built as a single binary or not.
Producing single binaries can now be done with building with
"kore build". To get started edit your build.conf and add the
following directives:
single_binary = yes
kore_source = /path/to/kore
optionally you can add kore_flavor to instruct how kore should
be built:
kore_flavor = NOTLS=1
When doing this your build.conf must also include the correct
linking options as the linking is now done fully by kore build.
The binary produced will include your configuration and takes
over a few of kore its command line flags (such as -f, -n or -r).
Kore will now isolate RSA private keys to a separate process (keymgr).
Worker processes that require RSA signing for TLS connections will
communicate with this keymgr process in order to do so.
This behaviour cannot be disabled and is always turned on.
This basically turns off the HTTP layer for Kore. It does not
compile in anything for HTTP.
This allows Kore to be used as a network application platform as well.
Added an example for this called nohttp.
Other changes that sneaked in while hacking on this:
* Use calloc(), kill pendantic malloc option.
* Killed off SPDY/3.1 support completely, will be superseded by http2
Note that comes with massive changes to a lot of the core API
functions provided by Kore, these might break your application.
With this framework apps can now send messages between worker processes.
A new API function exists:
int kore_msg_register(u_int8_t id, void (*cb)(const void *, u_int32_t);
This API call allows your app to register a new message callback for a given ID.
You can then send messages on this ID to other workers using:
void kore_msg_send(u_int8_t id, void *data, u_int32_t length);
This framework will interally be used for a few things such as allowing
websocket data to broadcasted between all workers, adding unified caching
and hopefully eventually moving the access log to this as well.
Some internals have changed with this commit:
* worker_clients has been called connections.
* the parent now initializes the net, and event subsystems.
* kore_worker_websocket_broadcast() is dead.
Add new command line knob '-r', that disables runas similar to '-n',
it's implied as well for kore command runs.
Add default runas (nobody) user and chroot (/var/empty) path, if none
are specified, fallback to these.
This configuration option limits the maximum number
of connections a worker process can accept() in a single
event loop.
It can be used to more evenly spread out incoming connections
across workers when new connections arrive in a burst.
In cases where accept() failed Kore would not relinquish the
lock towards other worker processes.
This becomes evident when dealing with a high number of concurrent
connections to the point the fd table gets full. In this scenario
the worker with the full fd table will spin on attempt to accept
newer connections.
As a bonus, Kore now has allows exactly up to worker_max_connections
of connections per worker before no longer attempting to grab the
accept lock.
Introduces two new configuration knobs:
* socket_backlog (backlog for listen(2))
* http_request_limit
The second one is the most interesting one.
Before, kore would iterate over all received HTTP requests
in its queue before returning out of http_process().
Under heavy load this queue can cause Kore to spend a considerable
amount of time iterating over said queue. With the http_request_limit,
kore will process at MOST http_request_limit requests before returning
back to the event loop.
This means responses to processed requests are sent out much quicker
and allows kore to handle any other incoming requests more gracefully.
Introduces a few new api functions:
- kore_websocket_handshake(struct http_request *):
Performs the handshake on an HTTP request (coming from page handler)
- kore_websocket_send(struct connection *, u_int8_t, void *, size_t):
Sends data to a websocket connection.
- kore_websocket_broadcast(struct connection *, u_int8_t, void *, size_t, int):
Broadcast the given websocket op and data to all connected
websocket clients on the worker. Note that as of right now
the WEBSOCKET_BROADCAST_GLOBAL scope option does not work
yet and messages broadcasted will be restricted to workers
only.
- kore_worker_websocket_broadcast(struct connection *, void *, void *):
Backend function used by kore_websocket_broadcast().
Could prove useful for developers to have access to.
A simple example is given under examples/websocket.
Known issues:
Kore does not support PING or CONT frames just yet.
Otherwise in a weird scenario the idle timer could
be started with a more recent time then the time we
test against causing a connection to disconnect
before its idle timer was over.
- The net code no longer has a recv_queue, instead reuse same recv buffer.
- Introduce net_recv_reset() to reset the recv buffer when needed.
- Have the workers spread the load better between them by slightly
delaying their next accept lock and giving them an accept treshold
so they don't go ahead and keep accepting connections if they end
up winning the race constantly between the workers.
- The kore_worker_acceptlock_release() is no longer available.
- Prepopulate the HTTP server response header that is added to each
response in both normal HTTP and SPDY modes.
- The path and host members of http_request are now allocated on the heap.
These changes overall result better performance on a multicore machine,
especially the worker load changes shine through.
Allow Kore to use per domain CRLs when requiring client certificates.
The require_client_cert configuration option has been renamed to a more
sane client_certificates and can optionally take a second argument
which is the CRL in pem format.
You'll need a restart in case the CRLs get updated.
After revisiting why this exists in Kore I decided it
does not belong in this platform and instead of letting
it sit there staring at me I rather just kill it.
Kore no longer passes the accept lock to the "next in line"
worker but instead all workers will attempt to grab the lock
if they can.
Also remember if we had the lock previous iteration of the
event loop and don't constantly disable/enable the accepting sockets.
Makes Kore scale even better across multiple cpu's.
Joris Vink
Elliot Schlegelmilch
Daniel Fahlgren
Stig Telfer
Thordur Bjornsson