- Make sure tls-alpn01 works even if the underlying SSL library ends up
calling the ALPN callback *before* the SNI extension was parsed and
the correct domain was selected.
LibreSSL still does this, and older OpenSSL did too I believe, however
OpenSSL grew a clue and always makes sure SNI is called first.
Yes, TLS extensions have no fixed order but it still makes sense to
notify applications using your library of the SNI extension first
before anything else almost.
Oh well.
Add 2 new types:
KORE_JSON_TYPE_INTEGER
signed integer type, internally stored as s64.
KORE_JSON_TYPE_INTEGER_U64
unsigned integer type, internally stored as u64.
Kore JSON parser will prefer marking integers as INTEGER_U64 if it
was unsigned and did not have fractions.
This handles the default option parsing in Kore and should be called
by single_binary=yes builds in kore_parent_configure() unless they
want to handle their own argument parsing.
- Remove the edge trigger io hacks we had in place.
- Use level triggered io for the libcurl fds instead.
- Batch all curl events together and process them at the end
of our worker event loop.
If a coroutine is killed from another coroutine and the killed coroutine
was waiting on a kore.lock() object, it would have been incorrectly
woken up again once said lock was released.
This would cause a Python exception that a generator was already
running and a crash due to the pool element already being freed.
Track the active locking operation per coroutine so we can remove
the coroutine if it is killed, fixing the problem.
This method allows you to set a Python object and obtain it
by calling the method again without any arguments.
eg:
foo = SomeClass()
kore.app(foo)
foo = kore.app()
- Fix the curl-extract-opt.sh generation script to work on newer
curl releases as the header changed slightly.
- Use the correct handles when calling curl_easy_setopt() inside
of our setopt functions exported via Python.
- Add a curl.setbody() method, allowing a body to be sent to be set.
(eg when sending mail via SMTP).
- Regen of our python_curlopt.h from 7.71.1
Inside the domain contexts a 'redirect' rule will allow you to redirect
a request to another URI.
Ex:
Redirect all requests with a 301 to example.com
redirect ^/.*$ 301 https://example.com
Using capture groups
redirect ^/account/(.*)$ 301 https://example.com/account/$1
Adding the query string in the mix
redirect ^/(.*)$ 301 https://example.com/$1?$qs
Kore already exposed parts of this via the kore.httpclient() method but
this commit takes it a bit further and exposes the libcurl interface
completely (including the setopt options).
tldr:
handle = kore.curl("ftp://ftp.eu.openbsd.org/pub/OpenBSD/README")
handle.setopt(kore.CURLOPT_TIMEOUT, 5)
data = await handle.run()
print("%s" % data.decode())
These changes improve the constraint kore had with client authentication and
multiple domains.
- Add kore_x509_subject_name() which will return a C string containing
the x509 subject name in full (in utf8).
- Log TLS errors if client authentication was turned on, will help debug
issues with client authentication in the future.
- If SNI was present in the TLS handshake, check it against the host specified
in the HTTP request and send a 421 in case they mismatch.
- Throw a 403 if client authentication was enabled but no client certificate
was specified.
A new acme process is created that communicates with the acme servers.
This process does not hold any of your private keys (no account keys,
no domain keys etc).
Whenever the acme process requires a signed payload it will ask the keymgr
process to do the signing with the relevant keys.
This process is also sandboxed with pledge+unveil on OpenBSD and seccomp
syscall filtering on Linux.
The implementation only supports the tls-alpn-01 challenge. This means that
you do not need to open additional ports on your machine.
http-01 and dns-01 are currently not supported (no wildcard support).
A new configuration option "acme_provider" is available and can be set
to the acme server its directory. By default this will point to the
live letsencrypt environment:
https://acme-v02.api.letsencrypt.org/directory
The acme process can be controlled via the following config options:
- acme_root (where the acme process will chroot/chdir into).
- acme_runas (the user the acme process will run as).
If none are set, the values from 'root' and 'runas' are taken.
If you want to turn on acme for domains you do it as follows:
domain kore.io {
acme yes
}
You do not need to specify certkey/certfile anymore, if they are present
still
they will be overwritten by the acme system.
The keymgr will store all certificates and keys under its root
(keymgr_root), the account key is stored as "/account-key.pem" and all
obtained certificates go under "certificates/<domain>/fullchain.pem" while
keys go under "certificates/<domain>/key.pem".
Kore will automatically renew certificates if they will expire in 7 days
or less.
If set to "yes" then Kore will trace its child processes and properly
notify you of seccomp violations while still allowing the syscalls.
This can be very useful when running Kore on new platforms that have
not been properly tested with seccomp, allowing me to adjust the default
policies as we move further.
We actually woke up the coroutine that originally spawned the process
when we reap it, but another coroutine may have taken over the object.
This mimics how we do things for the pysock_op things.
Now everything that has the "newer" OpenSSL API (1.1.x) is hidden
behind a KORE_OPENSSL_NEWER_API define. Tone down minimum libressl
version again to 2.7.5.
Allow JSON to be constructed via kore_json_create_item and its
handy macro family:
- kore_json_create_object()
- kore_json_create_array()
- kore_json_create_string()
- kore_json_create_number()
- kore_json_create_literal().
Adds kore_json_item_tobuf() to convert a JSON item into a string
representation in a kore_buf data structure.
Renames the kore_json_get* family to kore_json_find* instead.
Allows for quite clean code:
struct kore_buf buf;
struct kore_json_item *root;
root = kore_json_create_object(NULL, NULL);
kore_json_create_string(root, "hello", "world");
kore_json_create_number(root, "value", 2.241);
kore_buf_init(&buf, 128);
kore_json_item_tobuf(root, &buf);
kore_json_item_free(root);
kore_buf_cleanup(&buf);
In cases where a request is immediately completed in libcurl its multi
handle and no additional i/o is happening a coro can get stuck waiting
to be run.
Prevent this by lowering netwait from KORE_WAIT_INFINITE if there
are pending python coroutines.
Changes kore_curl_init() to take a flag parameter, much like pgsql api
in which you specify KORE_CURL_ASYNC or KORE_CURL_SYNC.
If KORE_CURL_ASYNC is specified, Kore will behave as before.
If Kore_CURL_SYNC is specified, Kore will execute the libcurl immediately
and return once it has been completed.
Mostly compliant, ignores \uXXXX in strings for now.
New API functions:
void kore_json_init(struct kore_json *json, const u_int8_t *data, size_t len);
- Prepares JSON data for parsing.
int kore_json_parse(struct kore_json *json)
- Parses the JSON data prepared via kore_json_init. Returns KORE_RESULT_ERROR
if parsing failed or KORE_RESULT_OK if it succeeded.
struct kore_json_item *kore_json_get(struct kore_json *json, const char *path,
int type);
- Try to find the object matching a given search patch and type.
eg, given a JSON structure of:
{
"reasons": {
"strings": [
"first reason",
"second"
]
}
}
one can obtain the second element in the reasons.strings array via:
item = kore_json_get(json, "reasons/strings[0]", KORE_JSON_TYPE_STRING);
Returns NULL if the item was not found or a type mismatch was hit,
otherwise will return the item of that type.
The kore_json_item data structure has a data member that contains the
relevant bits depending on the type:
KORE_JSON_TYPE_ARRAY, KORE_JSON_TYPE_OBJECT:
the data.items member is valid.
KORE_JSON_TYPE_STRING:
the data.string member is valid.
KORE_JSON_TYPE_NUMBER:
the data.number member is valid.
KORE_JSON_TYPE_LITERAL:
the data.literal member is valid.
void kore_json_cleanup(struct kore_json *json);
- Cleanup any resources
const char *kore_json_strerror(struct kore_json *json);
- Return pointer to human readable error string.
This allows you to send Python objects that can be run through pickle
to other worker processes.
If your application implements koreapp.onmsg() you will be able to receive
these objects.
A new hook in the koreapp class is called right before seccomp
is enabled. This hook receives a Kore seccomp object which has
the following methods:
seccomp.allow("syscall")
seccomp.allow_arg("syscall", arg, value)
seccomp.allow_flag("syscall", arg, flag)
seccomp.allow_mask("syscall", arg, mask)
seccomp.deny("syscall")
seccomp.deny_arg("syscall", arg, value, errno=EACCES)
seccomp.deny_flag("syscall", arg, flag, errno=EACCES)
seccomp.deny_mask("syscall", arg, mask, errno=EACCES)
This allows you to finetune the seccomp filters for your application
from inside your koreapp.
Before kore needed to be built with NOTLS=1 to be able to do non TLS
connections. This has been like this for years.
It is time to allow non TLS listeners without having to rebuild Kore.
This commit changes your configuration format and will break existing
applications their config.
Configurations now get listener {} contexts:
listen default {
bind 127.0.0.1 8888
}
The above will create a listener on 127.0.0.1, port 8888 that will serve
TLS (still the default).
If you want to turn off TLS on that listener, specify "tls no" in that
context.
Domains now need to be attached to a listener:
Eg:
domain * {
attach default
}
For the Python API this kills kore.bind(), and kore.bind_unix(). They are
replaced with:
kore.listen("name", ip=None, port=None, path=None, tls=True).
- Kore can now fully be configured via Python code if one wants nothing to
do with configuration files.
- Kore can now start single python files and no longer requires them to be
inside a module directory.
- Pass all regex capture groups to the handler methods, allowing you to
get access to them immediately.
- Change python websocket_handshake to take callable objects directly.
- Added a new deployment configuration option. If set to "dev" or
"development" Kore will automatically foreground, no chroot / etc.
If set to "production" Kore *will* chroot, drop privs, etc.
- Many more..
These are all backported from a project that I was working on a while
ago. I decided these should go back into mainline Kore.
More BPF helper macros, more helper for granular syscall checking.
Use these throughout kore where it makes sense.
The new helpers:
- KORE_SYSCALL_DENY_ARG(name, arg, value, errno):
Deny the system call with errno if the argument matches value.
- KORE_SYSCALL_DENY_MASK(name, arg, mask, errno):
Deny the system call with errno if the mask argument does not match
the exact mask given.
- KORE_SYSCALL_DENY_WITH_FLAG(name, arg, flag, errno):
Deny the system call with errno if the argument contains the
given flag.
The reverse also exists:
- KORE_SYSCALL_ALLOW_ARG()
- KORE_SYSCALL_ALLOW_MASK()
- KORE_SYSCALL_ALLOW_WITH_FLAG()
- Add KORE_SECCOMP_FILTER() as a helpful shortcut to create your
application seccomp filter. You can still roll your own and hook
into kore_seccomp_hook() yourself to load your filters.
- Add KORE_SYSCALL_ALLOW_LOG(_name)
Allows a system call but will log it.
With this commit all Kore processes (minus the parent) are running
under seccomp.
The worker processes get the bare minimum allowed syscalls while each module
like curl, pgsql, etc will add their own filters to allow what they require.
New API functions:
int kore_seccomp_filter(const char *name, void *filter, size_t len);
Adds a filter into the seccomp system (must be called before
seccomp is enabled).
New helpful macro:
define KORE_SYSCALL_ALLOW(name)
Allow the syscall with a given name, should be used in
a sock_filter data structure.
New hooks:
void kore_seccomp_hook(void);
Called before seccomp is enabled, allows developers to add their
own BPF filters into seccomp.
Joris Vink
Frederic Cambus