Producing single binaries can now be done with building with
"kore build". To get started edit your build.conf and add the
following directives:
single_binary = yes
kore_source = /path/to/kore
optionally you can add kore_flavor to instruct how kore should
be built:
kore_flavor = NOTLS=1
When doing this your build.conf must also include the correct
linking options as the linking is now done fully by kore build.
The binary produced will include your configuration and takes
over a few of kore its command line flags (such as -f, -n or -r).
Kore will now isolate RSA private keys to a separate process (keymgr).
Worker processes that require RSA signing for TLS connections will
communicate with this keymgr process in order to do so.
This behaviour cannot be disabled and is always turned on.
This basically turns off the HTTP layer for Kore. It does not
compile in anything for HTTP.
This allows Kore to be used as a network application platform as well.
Added an example for this called nohttp.
Other changes that sneaked in while hacking on this:
* Use calloc(), kill pendantic malloc option.
* Killed off SPDY/3.1 support completely, will be superseded by http2
Note that comes with massive changes to a lot of the core API
functions provided by Kore, these might break your application.
With this framework apps can now send messages between worker processes.
A new API function exists:
int kore_msg_register(u_int8_t id, void (*cb)(const void *, u_int32_t);
This API call allows your app to register a new message callback for a given ID.
You can then send messages on this ID to other workers using:
void kore_msg_send(u_int8_t id, void *data, u_int32_t length);
This framework will interally be used for a few things such as allowing
websocket data to broadcasted between all workers, adding unified caching
and hopefully eventually moving the access log to this as well.
Some internals have changed with this commit:
* worker_clients has been called connections.
* the parent now initializes the net, and event subsystems.
* kore_worker_websocket_broadcast() is dead.
Add new command line knob '-r', that disables runas similar to '-n',
it's implied as well for kore command runs.
Add default runas (nobody) user and chroot (/var/empty) path, if none
are specified, fallback to these.
This configuration option limits the maximum number
of connections a worker process can accept() in a single
event loop.
It can be used to more evenly spread out incoming connections
across workers when new connections arrive in a burst.
In cases where accept() failed Kore would not relinquish the
lock towards other worker processes.
This becomes evident when dealing with a high number of concurrent
connections to the point the fd table gets full. In this scenario
the worker with the full fd table will spin on attempt to accept
newer connections.
As a bonus, Kore now has allows exactly up to worker_max_connections
of connections per worker before no longer attempting to grab the
accept lock.
Introduces two new configuration knobs:
* socket_backlog (backlog for listen(2))
* http_request_limit
The second one is the most interesting one.
Before, kore would iterate over all received HTTP requests
in its queue before returning out of http_process().
Under heavy load this queue can cause Kore to spend a considerable
amount of time iterating over said queue. With the http_request_limit,
kore will process at MOST http_request_limit requests before returning
back to the event loop.
This means responses to processed requests are sent out much quicker
and allows kore to handle any other incoming requests more gracefully.
Introduces a few new api functions:
- kore_websocket_handshake(struct http_request *):
Performs the handshake on an HTTP request (coming from page handler)
- kore_websocket_send(struct connection *, u_int8_t, void *, size_t):
Sends data to a websocket connection.
- kore_websocket_broadcast(struct connection *, u_int8_t, void *, size_t, int):
Broadcast the given websocket op and data to all connected
websocket clients on the worker. Note that as of right now
the WEBSOCKET_BROADCAST_GLOBAL scope option does not work
yet and messages broadcasted will be restricted to workers
only.
- kore_worker_websocket_broadcast(struct connection *, void *, void *):
Backend function used by kore_websocket_broadcast().
Could prove useful for developers to have access to.
A simple example is given under examples/websocket.
Known issues:
Kore does not support PING or CONT frames just yet.
Otherwise in a weird scenario the idle timer could
be started with a more recent time then the time we
test against causing a connection to disconnect
before its idle timer was over.
- The net code no longer has a recv_queue, instead reuse same recv buffer.
- Introduce net_recv_reset() to reset the recv buffer when needed.
- Have the workers spread the load better between them by slightly
delaying their next accept lock and giving them an accept treshold
so they don't go ahead and keep accepting connections if they end
up winning the race constantly between the workers.
- The kore_worker_acceptlock_release() is no longer available.
- Prepopulate the HTTP server response header that is added to each
response in both normal HTTP and SPDY modes.
- The path and host members of http_request are now allocated on the heap.
These changes overall result better performance on a multicore machine,
especially the worker load changes shine through.
Allow Kore to use per domain CRLs when requiring client certificates.
The require_client_cert configuration option has been renamed to a more
sane client_certificates and can optionally take a second argument
which is the CRL in pem format.
You'll need a restart in case the CRLs get updated.
After revisiting why this exists in Kore I decided it
does not belong in this platform and instead of letting
it sit there staring at me I rather just kill it.
Kore no longer passes the accept lock to the "next in line"
worker but instead all workers will attempt to grab the lock
if they can.
Also remember if we had the lock previous iteration of the
event loop and don't constantly disable/enable the accepting sockets.
Makes Kore scale even better across multiple cpu's.
Has support for full async pgsql queries. Most of the logic
is hidden behind a KORE_PGSQL() macro allowing you to insert
these pgsql calls in your page handlers without blocking the
kore worker while the query is going off.
There is place for improvement here, and perhaps KORE_PGSQL won't
stay as I feel this might overcomplicate things instead of making
them simpler as I thought it would.
If a worker reached worker_max_connections and it was its turn to
grab the accept lock it would've gotten stuck and no new connections
would be handled even if other workers would be less busy.
Instead, we now skip the lock if we're too busy and pass it along
in the hopes other workers are less busy.
This allows us to move the accept lock manually to another Kore worker
in case we own it and are about to do some heavy lifting on the current
Kore worker.
This is useful to track down any issues you might have in your module.
A log entry with a page handler causing issues looks like:
Jul 7 14:44:30 devbook kore[18191]: [parent]: worker 1 (18193)-> status 11
Jul 7 14:44:30 devbook kore[18191]: [parent]: worker 1 (pid: 18193) (hdlr: 0x242d9c0) gone
Jul 7 14:44:30 devbook kore[18191]: [parent]: hdlr serve_intro has caused 2 error(s)
- Introduce own memory management system on top of malloc to keep track
of all our allocations and free's. Later we should introduce a pooling
mechanism for fixed size allocations (http_request comes to mind).
- Introduce ssl_cipher in configuration.
Memory usage is kind of high right now, but it seems its OpenSSL
doing it rather then Kore.
- Introduce own memory management system on top of malloc to keep track
of all our allocations and free's. Later we should introduce a pooling
mechanism for fixed size allocations (http_request comes to mind).
- Introduce ssl_cipher in configuration.
Memory usage is kind of high right now, but it seems its OpenSSL
doing it rather then Kore.
Instead of waiting until one worker is filled up on connections
the workers find the next lowest loaded worker and will hand
over the lock to them instead. This will cause a nicer spread of load.
Instead of running one accept per event loop, we attempt to accept
as many as worker_max_connections allows.
Refactor net sending/recv code a bit.
new connections and which ones will not be notified for it.
Fixes the thundering herd problem, and nicely spreads out load between
all the workers equally. A configuration option (workers_max_connections)
is available to tweak how many connections a worker will have before
giving up the accept lock.
Two ways are added to this commit for access locking:
- Locking via semaphores.
- Locking via GCC's builtin atomic methods.
The default is running with semaphores disabled (OpenBSD cannot do
sem_init() with pshared set to 1, which is required).
If you want to use semaphores add KORE_USE_SEMAPHORES to CFLAGS,
and -lpthread to LDFLAGS in the Makefile.
Other fixes:
- BSD: add a timeout to kevent().
- Merge kore_worker_wait together, linux knows waitpid() as well.
- Send the correct SIGQUIT signal to workers instead of SIGINT.
- Fix kore_time_ms().
- Log fatal worker messages in syslog.
- Refactor code even more.
- Do not free our own kore_worker structure.
Joris Vink
Elliot Schlegelmilch
Daniel Fahlgren
Stig Telfer
Thordur Bjornsson