Mostly compliant, ignores \uXXXX in strings for now.
New API functions:
void kore_json_init(struct kore_json *json, const u_int8_t *data, size_t len);
- Prepares JSON data for parsing.
int kore_json_parse(struct kore_json *json)
- Parses the JSON data prepared via kore_json_init. Returns KORE_RESULT_ERROR
if parsing failed or KORE_RESULT_OK if it succeeded.
struct kore_json_item *kore_json_get(struct kore_json *json, const char *path,
int type);
- Try to find the object matching a given search patch and type.
eg, given a JSON structure of:
{
"reasons": {
"strings": [
"first reason",
"second"
]
}
}
one can obtain the second element in the reasons.strings array via:
item = kore_json_get(json, "reasons/strings[0]", KORE_JSON_TYPE_STRING);
Returns NULL if the item was not found or a type mismatch was hit,
otherwise will return the item of that type.
The kore_json_item data structure has a data member that contains the
relevant bits depending on the type:
KORE_JSON_TYPE_ARRAY, KORE_JSON_TYPE_OBJECT:
the data.items member is valid.
KORE_JSON_TYPE_STRING:
the data.string member is valid.
KORE_JSON_TYPE_NUMBER:
the data.number member is valid.
KORE_JSON_TYPE_LITERAL:
the data.literal member is valid.
void kore_json_cleanup(struct kore_json *json);
- Cleanup any resources
const char *kore_json_strerror(struct kore_json *json);
- Return pointer to human readable error string.
This allows you to send Python objects that can be run through pickle
to other worker processes.
If your application implements koreapp.onmsg() you will be able to receive
these objects.
Before kore needed to be built with NOTLS=1 to be able to do non TLS
connections. This has been like this for years.
It is time to allow non TLS listeners without having to rebuild Kore.
This commit changes your configuration format and will break existing
applications their config.
Configurations now get listener {} contexts:
listen default {
bind 127.0.0.1 8888
}
The above will create a listener on 127.0.0.1, port 8888 that will serve
TLS (still the default).
If you want to turn off TLS on that listener, specify "tls no" in that
context.
Domains now need to be attached to a listener:
Eg:
domain * {
attach default
}
For the Python API this kills kore.bind(), and kore.bind_unix(). They are
replaced with:
kore.listen("name", ip=None, port=None, path=None, tls=True).
- Kore can now fully be configured via Python code if one wants nothing to
do with configuration files.
- Kore can now start single python files and no longer requires them to be
inside a module directory.
- Pass all regex capture groups to the handler methods, allowing you to
get access to them immediately.
- Change python websocket_handshake to take callable objects directly.
- Added a new deployment configuration option. If set to "dev" or
"development" Kore will automatically foreground, no chroot / etc.
If set to "production" Kore *will* chroot, drop privs, etc.
- Many more..
These are all backported from a project that I was working on a while
ago. I decided these should go back into mainline Kore.
With this commit all Kore processes (minus the parent) are running
under seccomp.
The worker processes get the bare minimum allowed syscalls while each module
like curl, pgsql, etc will add their own filters to allow what they require.
New API functions:
int kore_seccomp_filter(const char *name, void *filter, size_t len);
Adds a filter into the seccomp system (must be called before
seccomp is enabled).
New helpful macro:
define KORE_SYSCALL_ALLOW(name)
Allow the syscall with a given name, should be used in
a sock_filter data structure.
New hooks:
void kore_seccomp_hook(void);
Called before seccomp is enabled, allows developers to add their
own BPF filters into seccomp.
if an iterator is passed kore will send the response with
transfer-encoding: chunked and call the iterator for every
chunk that was sent.
The iterator must return a utf-8 string.
Works wonderful with TemplateStream from jinja2.
This commit adds the CURL=1 build option. When enabled allows
you to schedule CURL easy handles onto the Kore event loop.
It also adds an easy to use HTTP client API that abstracts away the
settings required from libcurl to make HTTP requests.
Tied together with HTTP request state machines this means you can
write fully asynchronous HTTP client requests in an easy way.
Additionally this exposes that API to the Python code as well
allowing you do to things like:
client = kore.httpclient("https://kore.io")
status, body = await client.get()
Introduces 2 configuration options:
- curl_recv_max
Max incoming bytes for a response.
- curl_timeout
Timeout in seconds before a transfer is cancelled.
This API also allows you to take the CURL easy handle and send emails
with it, run FTP, etc. All asynchronously.
By default kore will restart worker processes if they terminate
unexpected. However in certain scenarios you may want to bring down
an entire kore instance if a worker process fails.
By setting worker_death_policy to "terminate" the Kore server will
completely stop if a worker exits unexpected.
this change also stops python coroutines from waking up very
late after their timeout has expired.
in filerefs, don't prime the timer until we actually have something
to expire, and kill the timer when the last ref drops.
With these changes CRLs can be reloaded like certificates
by sending a SIGUSR1 to the parent process.
Track mtime on both certificate files and CRL files as well
and only submit them to the workers if this has changed.
Move away from the parent constantly hitting the disk for every
accesslog the workers are sending.
The workers will now write their own accesslogs to shared
memory before the parent will pick those up. The parent
will flush them to disk once every second or if they grow
larger then 1MB.
This removes the heavy penalty for having access logs
turned on when you are dealing with a large volume
of requests.
Before we just doubled in size the second we exhausted a pool instead
of doing a more controlled expansion.
Now we will expand at 25% of the initial elm count whenever we need to.
Will help with memory pressure in certain scenarios.
This commit removes TLS 1.0 support no matter what OpenSSL
you are linking against.
Changes the value of tls_version from 1.2 to both. Meaning if
you link with OpenSSL 1.1.1 you will get 1.2 + 1.3.
If exists these functions are called when the worker is exiting
and when right before the parent exists.
Allows for cleanup code for applications if need to do cleanup on exit.
This means you can now do things like:
resp = await koresock.recv(1024)
await koresock.send(resp)
directly from page handlers if they are defined as async.
Adds lots more to the python goo such as fatalx(), bind_unix(),
task_create() and socket_wrap().
Now anyone can schedule events and get a callback to work as long
as the user data structure that is added for the event begins
with a kore_event data structure.
All event state is now kept in that kore_event structure and renamed
CONN_[READ|WRITE]_POSSIBLE to KORE_EVENT_[READ|WRITE].
If a worker process dies it automatically gets respawned by the
parent process, but sometimes you want the entire server to go down
if a worker encounters an error. This is what fatalx() does.
Calling fatalx() from a worker process will initiate a full shutdown
of the kore server you are running under.
All worker processes will now call pledge(2) after dropping
privileges (even if -rn was specified).
By default Kore will use the following promises:
"stdio rpath inet error"
If your application requires more privileges, you can add more pledges
by setting them in your configuration using the 'pledge' directive:
pledge dns wpath
Calling this from your page handler will cause your current worker
to give up the acceptlock (if it holds it).
This is particularly useful if you are about to run code that may block
a bit longer then you are comfortable with. Calling this will cause
the acceptlock to shuffle to another free worker which in turn makes
sure your application can keep accepting requests.
This commit introduces the ability for the keymgr process
to reload the certificates/keys for domains when receiving
a SIGUSR1 signal.
The keymgr receives 2 new configuration options:
- keymgr_root_path
The root path where the keymgr will live.
If -n is not specified when the application starts the
keymgr process will chroot into here.
- keymgr_runas_user
The user the keymgr will drop privileges towards if
-r was not specified.
All certfile and certkey configuration options are now relative to the
keymgr_root_path configuration setting.
The keymgr process will now also load the certificate for the domain
(rather then the workers) and submit these to the worker processes so
they can be reloaded when required.
Worker processes will refuse connections until the TLS configuration
for a given domain is completed (aka: the workers receive the certificate
for that domain).
Other changes:
- client_certificates renamed to client_verify.
- the chroot configuration option is now called root.
- kore is a little more verbose if privsep options are missing.
- filemaps are now relative to the root configuration option.
- make sure we can serve updated files even if we have an old
fileref around.
- add filemap_index as a configuration option: allows one to specify
what file to serve if a directory was requested (eg: index.html)
A filemap is a way of telling Kore to serve files from a directory
much like a traditional webserver can do.
Kore filemaps only handles files. Kore does not generate directory
indexes or deal with non-regular files.
The way files are sent to a client differs a bit per platform and
build options:
default:
- mmap() backed file transfer due to TLS.
NOTLS=1
- sendfile() under FreeBSD, macOS and Linux.
- mmap() backed file for OpenBSD.
The opened file descriptors/mmap'd regions are cached and reused when
appropriate. If a file is no longer in use it will be closed and evicted
from the cache after 30 seconds.
New API's are available allowing developers to use these facilities via:
void net_send_fileref(struct connection *, struct kore_fileref *);
void http_response_fileref(struct http_request *, struct kore_fileref *);
Kore will attempt to match media types based on file extensions. A few
default types are built-in. Others can be added via the new "http_media_type"
configuration directive.
You can now per domain configure the depth for x509 chain validation:
client_verify_depth 1
By default this is 1.
While here change around some log messages and properly set
the callback for x509 verification rather then via hoops and loops.
This function now takes any remaining arguments passed on the command line
after kore parsed its own.
For C the new prototype looks like this:
void kore_parent_configure(int argc, char **argv);
For python code, kore will pass each argument to the function so you
can do things like:
def kore_parent_configure(arg1, arg2):
Mimics how the header files are installed on a system
as PREFIX/include/kore.
This is required for getting kodev to use the headers from the
kore_source option instead of requiring the kore headers to be
installed on the system even when building as a single_binary.
Joris Vink
Sebastiaan