It was hardcoded that if KORE_USE_PYTHON was defined we would
look at the passed argument on the command-line as the python
script or module to be run.
This won't work when adding more runtimes.
So instead call a kore_runtime_resolve() function that in
turn calls each available runtime its resolve function.
That resolve function will check if its a script / module
that it can load, and if so will load it.
This way we can remove all those KORE_USE_PYTHON blocks in the
Kore startup path and we pave the way for lua.
When KORE_MEM_GUARD is set in the environment when Kore is started
it will enable a few memory protection techniques for all kore pools:
1) The metadata is placed away from the actual user pointer returned.
2) Each entry in a pool is placed in such a way that it is followed
immediately by a guard page which has PROT_NONE. Accessing a guard
page will cause an immediate crash.
3) Each entry is marked with PROT_NONE until it is allocated. Once it
is returned to a pool it becomes PROT_NONE again, protecting against
use after frees.
This commit also removes the magic goo from the mem facitilies such
as kore_malloc and friends and moves these as canaries into the kore
pool facilities instead.
Note that using this will increase memory pressure and decrease performance.
It is recommended to enable this during development to catch bugs.
The parent process never differentiated between a worker process
asking for a shutdown or a worker process calling fatalx() when
it came to its exit code.
I made some changes here so the parent process will exit with
an exit code 1 if anything worker related went wrong (fatalx/death policy).
This work moves all TLS / crypto related code into a tls_openssl.c
file and adds a tls_none.c which contains just stubs.
Allows compilation of Kore with TLS_BACKEND=none to remove building
against OpenSSL.
Also adds code for SHA1/SHA2 taken from openssh-portable so we don't
depend on those being present anymore in libcrypto.
If no accesslogs are enabled, the parent has no need for
the vacuum timer to be activated.
This way the parent blocks in epoll_wait() instead of waking up
for no reason when there are no accesslogs enabled.
This commit adds improved hooks for Python and a new signal delivery hook.
For the Python API kore_worker_configure() and kore_worker_teardown() had
to be implemented before this commit. Now one can create a workerstart
and workerend method in their koreapp as those will be called when
they exist.
The new signal hook is either kore_worker_signal() or koreapp.signal.
This new hook is called after the worker event code handles the received
signal itself first.
With this commit there is also a new kore_signal_trap() API call allowing
you to more easily trap new signals. This API also also exported to the
Python part of the code under kore.sigtrap()
- Make sure we drain the worker log channel if it dies
so we can flush out any lingering log messages.
- Get rid of the raise() in the parent to signal ourselves
we should terminate. Instead depend on the new kore_quit.
- Always attempt to reap children one way or the other.
Starting with the privsep config, this commit changes the following:
- Removes the root, runas, keymgr_root, keymgr_runas, acme_root and
acme_runas configuration options.
Instead these are now configured via a privsep configuration context:
privsep worker {
root /tmp
runas nobody
}
This is also configurable via Python using the new kore.privsep() method:
kore.privsep("worker", root="/tmp", runas="nobody", skip=["chroot"])
Tied into this we also better handle worker startup:
- Per worker process, wait until it signalled it is ready.
- If a worker fails at startup, display its last log lines more clearly.
- Don't start acme process if no domain requires acme.
- Remove each process its individual startup log message in favour
of a generalized one that displays its PID, root and user.
- At startup, log the kore version and built-ins in a nicer way.
- The worker processes now check things they need to start running
before signaling they are ready (such as access to CA certs for
TLS client authentication).
Before each worker process would either directly print to stdout if
Kore was running in foreground mode, or syslog otherwise.
With this commit the workers will submit their log messages to the
parent process who will either put it onto stdout or syslog.
This change in completely under the hood and users shouldn't care about it.
1) Add @kore.route as a decorator for Python.
This decorator can be used on non-class methods to automatically
declare their route and parameters.
Takes the same arguments as the kore.domain.route function that
exists today.
Provides a nice clean way of setting up Kore if you dont want
a whole class based approach.
2) Remove the requirement for the name for kore.server() and the
kore.domain(attach=) keywords.
Instead of no name was given, the name "default" is used in both
places resulting in less boilerplating.
3) Allow multiple routes to be defined for the same URI as long
as the methods are different. So you can have one method for GET /
and another for POST /.
All changes combined condense the initial experience of getting
a Kore Python app up and running:
eg:
import kore
kore.server(ip="127.0.0.1", port="8888", tls=False)
kore.domain("*")
@kore.route("/", methods=["get"])
async def index(req):
req.response(200, b'get method')
@kore.route("/", methods=["post"])
async def index_post(req)
req.response(200, b'post method')
- Unlink the socket path if possible before attempting to bind to it.
- Unlink the socket path if possible when Kore is shutting down.
inspired by a diff from Joel Arbring via patches@
- Make sure tls-alpn01 works even if the underlying SSL library ends up
calling the ALPN callback *before* the SNI extension was parsed and
the correct domain was selected.
LibreSSL still does this, and older OpenSSL did too I believe, however
OpenSSL grew a clue and always makes sure SNI is called first.
Yes, TLS extensions have no fixed order but it still makes sense to
notify applications using your library of the SNI extension first
before anything else almost.
Oh well.
This handles the default option parsing in Kore and should be called
by single_binary=yes builds in kore_parent_configure() unless they
want to handle their own argument parsing.
If KORE_SINGLE_BINARY is enabled, remove the getopt() call that
Kore does itself. This way all arguments are passed to the
kore_parent_configure() hook as-is allowing developers to
more easily implement their own option handling.
These changes improve the constraint kore had with client authentication and
multiple domains.
- Add kore_x509_subject_name() which will return a C string containing
the x509 subject name in full (in utf8).
- Log TLS errors if client authentication was turned on, will help debug
issues with client authentication in the future.
- If SNI was present in the TLS handshake, check it against the host specified
in the HTTP request and send a 421 in case they mismatch.
- Throw a 403 if client authentication was enabled but no client certificate
was specified.
A new acme process is created that communicates with the acme servers.
This process does not hold any of your private keys (no account keys,
no domain keys etc).
Whenever the acme process requires a signed payload it will ask the keymgr
process to do the signing with the relevant keys.
This process is also sandboxed with pledge+unveil on OpenBSD and seccomp
syscall filtering on Linux.
The implementation only supports the tls-alpn-01 challenge. This means that
you do not need to open additional ports on your machine.
http-01 and dns-01 are currently not supported (no wildcard support).
A new configuration option "acme_provider" is available and can be set
to the acme server its directory. By default this will point to the
live letsencrypt environment:
https://acme-v02.api.letsencrypt.org/directory
The acme process can be controlled via the following config options:
- acme_root (where the acme process will chroot/chdir into).
- acme_runas (the user the acme process will run as).
If none are set, the values from 'root' and 'runas' are taken.
If you want to turn on acme for domains you do it as follows:
domain kore.io {
acme yes
}
You do not need to specify certkey/certfile anymore, if they are present
still
they will be overwritten by the acme system.
The keymgr will store all certificates and keys under its root
(keymgr_root), the account key is stored as "/account-key.pem" and all
obtained certificates go under "certificates/<domain>/fullchain.pem" while
keys go under "certificates/<domain>/key.pem".
Kore will automatically renew certificates if they will expire in 7 days
or less.
In cases where a request is immediately completed in libcurl its multi
handle and no additional i/o is happening a coro can get stuck waiting
to be run.
Prevent this by lowering netwait from KORE_WAIT_INFINITE if there
are pending python coroutines.
This allows you to send Python objects that can be run through pickle
to other worker processes.
If your application implements koreapp.onmsg() you will be able to receive
these objects.
Before kore needed to be built with NOTLS=1 to be able to do non TLS
connections. This has been like this for years.
It is time to allow non TLS listeners without having to rebuild Kore.
This commit changes your configuration format and will break existing
applications their config.
Configurations now get listener {} contexts:
listen default {
bind 127.0.0.1 8888
}
The above will create a listener on 127.0.0.1, port 8888 that will serve
TLS (still the default).
If you want to turn off TLS on that listener, specify "tls no" in that
context.
Domains now need to be attached to a listener:
Eg:
domain * {
attach default
}
For the Python API this kills kore.bind(), and kore.bind_unix(). They are
replaced with:
kore.listen("name", ip=None, port=None, path=None, tls=True).
- Kore can now fully be configured via Python code if one wants nothing to
do with configuration files.
- Kore can now start single python files and no longer requires them to be
inside a module directory.
- Pass all regex capture groups to the handler methods, allowing you to
get access to them immediately.
- Change python websocket_handshake to take callable objects directly.
- Added a new deployment configuration option. If set to "dev" or
"development" Kore will automatically foreground, no chroot / etc.
If set to "production" Kore *will* chroot, drop privs, etc.
- Many more..
These are all backported from a project that I was working on a while
ago. I decided these should go back into mainline Kore.
With this commit all Kore processes (minus the parent) are running
under seccomp.
The worker processes get the bare minimum allowed syscalls while each module
like curl, pgsql, etc will add their own filters to allow what they require.
New API functions:
int kore_seccomp_filter(const char *name, void *filter, size_t len);
Adds a filter into the seccomp system (must be called before
seccomp is enabled).
New helpful macro:
define KORE_SYSCALL_ALLOW(name)
Allow the syscall with a given name, should be used in
a sock_filter data structure.
New hooks:
void kore_seccomp_hook(void);
Called before seccomp is enabled, allows developers to add their
own BPF filters into seccomp.
- decouple pgsql from the HTTP request allowing it to be used in other
contexts as well (such as a task, etc).
- change names to dbsetup() and dbquery().
eg:
result = kore.dbquery("db", "select foo from bar")
- If Kore is built with PYTHON=1 you can now specify the module that
should be loaded on the command-line.
eg: $ kore -frn myapp
- Add skeleton generation for python applications to kodev.
eg: $ kodev create -p myapp
This should make it a whole lot easier to get started with kore python.
Joris Vink