Commit Graph

69 Commits

Author SHA1 Message Date
Joris Vink 9caa45a050 Allow python validator methods to be async. 2019-03-18 09:34:31 +01:00
Joris Vink 31b6365da3 include sys/types.h explictly. 2019-03-14 19:20:46 +01:00
Joris Vink 3b4574d791 Rework pysocket async/await.
Attach the events directly to the pysocket data structure instead of
one event per pysocket_op.

Makes the code easier, gives us a good performance boost and reduces
the number of system calls required when doing an await on a socket.
2019-03-13 11:07:15 +01:00
Joris Vink 31a9a70d5a cleanup. 2019-03-01 20:56:03 +01:00
Erik Karlsson 01c1a8f8f8 support AF_UNIX in recvfrom()/sendto() python sockets 2019-03-01 20:13:28 +01:00
Joris Vink f4cd70956b Add an optional timeout to socketop.recv(). 2019-02-25 10:35:00 +01:00
Joris Vink bf1e8e5ffb bump copyright to 2019 2019-02-22 16:57:28 +01:00
Joris Vink d6b05bcff7 always force reload cert so we get a new x509 store.
Otherwise older OpenSSL or current LibreSSL will fail to add the new
CRL as they still match on subject name rather then hash of the CRL data.
2019-01-14 20:57:40 +01:00
Joris Vink 73cdbd1a01 Let CRLs be reloadable via keymgr.
With these changes CRLs can be reloaded like certificates
by sending a SIGUSR1 to the parent process.

Track mtime on both certificate files and CRL files as well
and only submit them to the workers if this has changed.
2019-01-14 11:41:50 +01:00
Joris Vink c070e77ea5 include signal.h from kore.h 2019-01-11 11:32:33 +01:00
Joris Vink b07cc0237c Support recvfrom()/sendto() on kore python sockets. 2019-01-08 17:49:00 +01:00
Joris Vink 7aa17df4a1 make ACCESSLOG_* unsigned 2018-12-22 10:52:57 +01:00
Joris Vink 4d184e00de correct define number 2018-12-22 10:22:59 +01:00
Joris Vink 9aa0e95643 Rework accesslog handling.
Move away from the parent constantly hitting the disk for every
accesslog the workers are sending.

The workers will now write their own accesslogs to shared
memory before the parent will pick those up. The parent
will flush them to disk once every second or if they grow
larger then 1MB.

This removes the heavy penalty for having access logs
turned on when you are dealing with a large volume
of requests.
2018-12-22 09:25:00 +01:00
Joris Vink 272d659f1c we use certain things from sys/param.h 2018-11-29 10:31:14 +01:00
Joris Vink d9f543ef5b Allow user-supplied tracer callback. 2018-11-29 09:51:24 +01:00
Joris Vink c431c2bf72 Add support to obtain peer certificate from Python
This will return the DER encoded bytes representing the peer certificate.
2018-11-28 11:28:07 +01:00
Joris Vink baafa4897e Add -q flag.
If specified Kore will run quietly and only log important messages.
2018-11-15 16:01:37 +01:00
Joris Vink 966eaf8f7a Add a kore_python_preinit() hook.
This is called before the python initialization is completed
and allows developers to inject their own built-in methods.
2018-11-12 09:01:05 +01:00
Joris Vink 2dd66586ff several python improvements.
- add kore.time() as equivalent for kore_time_ms().
- call waitpid() until no more children are available for reaping otherwise
  we risk missing a process if several die at the same time and only one
  SIGCHLD is delivered to us.
- drain a RECV socket operation if eof is set but no exception was given.
2018-10-30 20:28:27 +01:00
Joris Vink 71c145932c grow kore_pools at a slower rate.
Before we just doubled in size the second we exhausted a pool instead
of doing a more controlled expansion.

Now we will expand at 25% of the initial elm count whenever we need to.

Will help with memory pressure in certain scenarios.
2018-10-30 10:36:18 +01:00
Joris Vink 1c30da855c Add kore.gather() to the python api.
Allows one to run coroutines concurrently and gather all their
results in a single returned list.

If any of the coroutines throw an exception the exception is
returned as the value of that coroutine in the returned list.
2018-10-29 21:16:08 +01:00
Joris Vink 339df66fd5 Add support for TLS 1.3 via OpenSSL 1.1.1.
This commit removes TLS 1.0 support no matter what OpenSSL
you are linking against.

Changes the value of tls_version from 1.2 to both. Meaning if
you link with OpenSSL 1.1.1 you will get 1.2 + 1.3.
2018-10-29 20:38:58 +01:00
Joris Vink dda2e1fb2c Some things still talk http/1.0. 2018-10-26 21:24:51 +02:00
Joris Vink e2651889e0 Add asynchronous subprocess support.
This adds kore.proc to the python runtime allowing async processing
handling:

The kore.proc method takes the command to run and an optional timeout
parameter in milliseconds. If the process did not exit normally after
that amount of time a TimeoutError exception is raised.

For instance:

async def run(cmd):
	proc = kore.proc(cmd, 1000)

	try:
		await proc.send("hello")
		proc.close_stdin()
	except TimeoutError:
		proc.kill()

	retcode = await proc.reap()

	return retcode
2018-10-26 19:19:47 +02:00
Joris Vink 892814e353 Add kore_[parent|worker]_teardown().
If exists these functions are called when the worker is exiting
and when right before the parent exists.

Allows for cleanup code for applications if need to do cleanup on exit.
2018-10-23 21:46:34 +02:00
Joris Vink 8ea32983ae Add kore.suspend(delay) to python.
Will suspend the coroutine for a number of milliseconds.

Example:

async def page(req):
	await kore.suspend(1000)
	req.response(200, b'')
2018-10-23 21:32:08 +02:00
Joris Vink 47776a9fbb Hook kore timers into python. 2018-10-23 20:44:43 +02:00
Joris Vink c41c1db303 Add kore_shutdown().
Allows workers to cleanly initiate a shutdown of the
entire server process.
2018-10-23 19:49:42 +02:00
Joris Vink b70d1ee80f Add a locking mechanism in pykore.
Support the async with syntax:

	lock = kore.lock()

	async with lock:
		# your block

Fix some small issues with other parts of the python system.
2018-10-22 20:09:23 +02:00
Joris Vink fad5c6ea6f Give pyqueues "popnow". 2018-10-22 08:28:03 +02:00
Joris Vink c8795b7d7f pyqueue improvements.
- cleanup queue if it gets deallocated.
- make sure waitables on a queue get removed if their pyqueue_op dies.
2018-10-21 21:58:34 +02:00
Joris Vink 0cda9ecfb0 Add an asynchronous queue mechanism.
This allows coroutines to submit messages to and pop
messages from a queue in an asynchronous way.
2018-10-18 22:15:21 +02:00
Joris Vink b5958f7d7d Add kore_parent_daemonized().
This is called for single binaries after the parent
process has called daemon().

Also fix kore_parent_configure() for !single binaries.
2018-10-18 17:18:41 +02:00
Joris Vink 29202d7330 Make kore_python_log_error() public.
While here also make kore_module_load() return the
kore_module data structure pointer it just added
to the modules list.
2018-10-16 13:16:36 +02:00
Joris Vink 20a0103f1e Add async/await support for socket i/o in python.
This means you can now do things like:

	resp = await koresock.recv(1024)
	await koresock.send(resp)

directly from page handlers if they are defined as async.

Adds lots more to the python goo such as fatalx(), bind_unix(),
task_create() and socket_wrap().
2018-10-15 20:18:54 +02:00
Joris Vink c463ecb3cb Changes to the event loop inside of Kore.
Now anyone can schedule events and get a callback to work as long
as the user data structure that is added for the event begins
with a kore_event data structure.

All event state is now kept in that kore_event structure and renamed
CONN_[READ|WRITE]_POSSIBLE to KORE_EVENT_[READ|WRITE].
2018-10-09 19:34:40 +02:00
Joris Vink 9427ed8a2e rename unix to sun 2018-10-07 21:03:12 +02:00
Joris Vink 442bdef79b allow kore to bind to unix sockets via bind_unix. 2018-10-07 20:49:16 +02:00
Joris Vink b0074ba45e Add fatalx().
If a worker process dies it automatically gets respawned by the
parent process, but sometimes you want the entire server to go down
if a worker encounters an error. This is what fatalx() does.

Calling fatalx() from a worker process will initiate a full shutdown
of the kore server you are running under.
2018-08-13 09:53:49 +02:00
Joris Vink a927acb7ee Add pledge support under OpenBSD.
All worker processes will now call pledge(2) after dropping
privileges (even if -rn was specified).

By default Kore will use the following promises:
	"stdio rpath inet error"

If your application requires more privileges, you can add more pledges
by setting them in your configuration using the 'pledge' directive:
	pledge dns wpath
2018-07-31 06:51:34 +02:00
Joris Vink f126ba5a86 sprinkle more const around 2018-07-25 09:54:34 +02:00
Joris Vink cf1f624367 let filerefs to operate on ms precision for mtime. 2018-07-24 19:56:36 +02:00
Joris Vink a6e662a805 Add KORE_HTTP_STATE() helper macro.
defines as KORE_HTTP_STATE(f) (#f, f) so we can get a literal representing
the name of the state function which makes debugging a little easier.
2018-07-23 06:14:17 +02:00
Joris Vink 7f820c96e8 Make kore_auth_* public. 2018-07-19 10:28:38 +02:00
Joris Vink 2e321f14de Add KORE_PGSQL_STATE_NOTIFY.
Issue a LISTEN channel on a kore_pgsql, bind a callback to it and you
will get called with pgsql->state being KORE_PGSQL_STATE_NOTIFY.
2018-07-18 11:38:17 +02:00
Joris Vink ac5a89fbeb pull in openssl/sha.h for body digest 2018-07-17 15:07:26 +02:00
Joris Vink 616af063e3 Calculate an md over the incoming HTTP body.
This is calculated while the HTTP body is incoming over the wire, once
the body is fully received the digest will be available for the page
handlers to obtain.

You can obtain a hex string for this md via http_body_digest() or
dereferences the http_request and look at http_body_digest manually
for the bytes.
2018-07-17 14:53:55 +02:00
Joris Vink 40a81a17ba remove kore_module_handle for NOHTTP=1 builds. 2018-07-17 14:28:43 +02:00
Joris Vink 0726a26c0c Allow restriction of methods for paths.
Now Kore will automatically send a 400 bad request in case the
method was not allowed on the path.
2018-07-17 14:23:57 +02:00