mirrors
/
kore
mirror of https://git.kore.io/kore.git
1
0
Fork 1
Code Issues Projects Releases Wiki Activity
1,651 Commits 5 Branches 45 Tags 4.4 MiB
Commit Graph

1182 Commits

Author SHA1 Message Date
Joris Vink 07fc7a9097 Improve HTTP processing. 
If netwait is INFINITE but there are requests pending reduce the
netwait back down to 100ms so we keep processing them.
 2019-05-29 15:27:44 +02:00
Joris Vink c2d9f1413c Fix two concurrency issues with kore.lock(). 
1) If a coroutine was waken up by another releasing a lock it was waiting
on we would incorrectly remove the pylock_op when the newly awoken coroutine
hits the iternext for pylock.

2) If a coroutine attempts to grab a lock it was woken up for only to fail
we did not properly rearm the coroutine to be woken up later to try again.
 2019-05-29 15:25:31 +02:00
Joris Vink 5f0153ba0e Fix unix binds on BSD families. 2019-05-28 21:44:46 +02:00
Joris Vink b4aba48fad small python improvements. 
- Check for an exception when in a timer context and log it if uncaught.
- Make sure tlsverify is settable regardless of client certs being specified.
 2019-05-16 21:07:42 +02:00
Joris Vink a10dfe03fe make sure user-defined headers are set for > 500. 
a commit done in 2018 prevented http responses with error codes
> 500 to include any user-set headers, preventing a developer
to include things like content-type etc.

reported by Arun Babu via users@
 2019-05-12 20:53:27 +02:00
Joris Vink 53f042a5b4 add stdint.h header to keymgr.c 2019-05-09 20:55:49 +02:00
Joris Vink 3299f60df4 tiny curl fixes. 2019-05-05 21:16:42 +02:00
Joris Vink 503f42074c http_timeout must be 0 when upgrading to websockets 
otherwise kore will timeout an established websocket connection
after http_timeout seconds.
 2019-05-05 14:47:04 +02:00
Joris Vink fe84997ce9 add option to use unix paths via kore.httpclient 2019-05-03 15:31:16 +02:00
Joris Vink 3c9a141cd0 allow an iterator to be passed to req.response(). 
if an iterator is passed kore will send the response with
transfer-encoding: chunked and call the iterator for every
chunk that was sent.

The iterator must return a utf-8 string.

Works wonderful with TemplateStream from jinja2.
 2019-05-03 13:42:34 +02:00
Joris Vink 12fc1396c1 use / to split up name/version. 2019-04-30 20:45:56 +02:00
Joris Vink 502368a27e set user-agent in the curl component. 2019-04-30 20:39:46 +02:00
Joris Vink 403938d8e9 make native proctitle better. 
count how much space is available for a mangled process title
only once, and use that as reference later.
 2019-04-29 21:08:58 +02:00
Joris Vink d0a6958747 Let http_state_create() take an "onfree" callback. 
This function is called when an HTTP request is being free'd,
allowing you to perform any sort of state cleanup attached
to the HTTP request.
 2019-04-28 21:48:16 +02:00
Joris Vink 9ac77d0c9a make sure CURL_CSELECT_ERR is OR'd into flags. 2019-04-28 21:44:59 +02:00
Joris Vink e54886960e objs is a pointer, and use Py_XDECREF on it. 2019-04-25 23:25:19 +02:00
Joris Vink d0d0bdeb4f Improve pgsql support. 
- Add kore_pgsql_query_param_fields() which allows you to pass in the
  arrays for values, lengths and formats yourself.

- Add kore_pgsql_column_binary() which will return 1 if the given column
  index contains a binary result or 0 if it contains a text result.

- Change the query call in req.pgsql() for Python to always use the
  parameterized queries.

This adds the 'params' and 'binary' keywords to the req.pgsql method.

Eg:
	result = await req.pgsql("db", "INSERT INTO foo (field) VALUES($1"),
	    params=["this is my value"])
 2019-04-25 23:13:13 +02:00
Joris Vink 994f428a8d add cabundle keyword to kore.httpclient() 2019-04-24 21:09:24 +02:00
Joris Vink 6bd3d92fe2 unbreak. 2019-04-24 18:30:07 +02:00
Joris Vink a6af458cd2 add tlsverify keyword to kore.httpclient(). 2019-04-24 18:28:23 +02:00
Joris Vink 1b9ba84ab0 missing return value check. 2019-04-24 09:33:56 +02:00
Joris Vink a9683caed5 missing header 2019-04-24 00:16:51 +02:00
Joris Vink 2c88bc6120 Add asynchronous libcurl support. 
This commit adds the CURL=1 build option. When enabled allows
you to schedule CURL easy handles onto the Kore event loop.

It also adds an easy to use HTTP client API that abstracts away the
settings required from libcurl to make HTTP requests.

Tied together with HTTP request state machines this means you can
write fully asynchronous HTTP client requests in an easy way.

Additionally this exposes that API to the Python code as well
allowing you do to things like:

	client = kore.httpclient("https://kore.io")
	status, body = await client.get()

Introduces 2 configuration options:
	- curl_recv_max
		Max incoming bytes for a response.

	- curl_timeout
		Timeout in seconds before a transfer is cancelled.

This API also allows you to take the CURL easy handle and send emails
with it, run FTP, etc. All asynchronously.
 2019-04-24 00:15:17 +02:00
Joris Vink c89ba3daa3 check http timeouts better 2019-04-12 14:26:47 +02:00
Joris Vink 612fc034fa remove debug 2019-04-11 20:54:29 +02:00
Joris Vink aa49e181b6 Add http_[header|body]_timeout. 
If the HTTP request headers or the HTTP body have not arrived before
these timeouts expire, Kore will send a 408 back to the client.
 2019-04-11 20:51:49 +02:00
Joris Vink a191445f76 set body length+offset to 0 when populating data. 
otherwise this isn't properly picked up by http_body_read() later
if dealing with in-memory HTTP bodies and you get inconsistent behaviour.
 2019-04-02 22:26:44 +02:00
Joris Vink 5d16a7a123 make sure stdint.h is always included. 
unbreaks NOTLS=1 and NOSENDFILE=1 builds on several platforms.

reported by Sam on irc, thanks.
 2019-03-29 19:25:27 +01:00
Joris Vink a66d259271 kill debug line 2019-03-29 19:15:10 +01:00
Joris Vink d2aa64df5c add kore_proctitle(). 
manipulates the argv+environ pointers to get a sensible process title
under linux / darwin.
 2019-03-29 16:24:14 +01:00
Joris Vink 92fb4974b1 only deschedule if not -1. 2019-03-27 22:15:35 +01:00
Joris Vink 39ffa047ca remove dh parameter check at configure time. 
this is done when the workers startup if a domain with tls was used.
 2019-03-26 21:44:44 +01:00
Joris Vink e4caac9e0c Be more clear when an invalid TLS setup is found 2019-03-26 21:36:45 +01:00
Joris Vink bf42e56c7d kill whitespace 2019-03-26 20:24:46 +01:00
Joris Vink 4ca7f29649 Add a concurrency parameter to kore.gather() 2019-03-25 10:13:52 +01:00
Joris Vink e1766e74ba always capture worker processes exiting. 
even if they terminated normally.
 2019-03-22 10:29:14 +01:00
Joris Vink 4238431b9e Add worker_death_policy setting. 
By default kore will restart worker processes if they terminate
unexpected. However in certain scenarios you may want to bring down
an entire kore instance if a worker process fails.

By setting worker_death_policy to "terminate" the Kore server will
completely stop if a worker exits unexpected.
 2019-03-22 09:49:50 +01:00
Joris Vink cd80685d9d get rid of pyko 2019-03-21 21:37:16 +01:00
Joris Vink 58c2739dee If fd is -1 in pysock_op_iternext just return None. 
None in pysock signals EOF and if it was closed under our feet we should
relay that properly.
 2019-03-21 14:44:41 +01:00
Joris Vink 370041656e Get rid of WORKER_LOCK_TIMEOUT. 
Instead let the workers send a message on the msg channel to each
other when they have given up the accept lock and it is now available
to be grabbed.
 2019-03-21 14:03:11 +01:00
Joris Vink 8b0279879a rework timers so they fire more predictably. 
this change also stops python coroutines from waking up very
late after their timeout has expired.

in filerefs, don't prime the timer until we actually have something
to expire, and kill the timer when the last ref drops.
 2019-03-21 10:17:08 +01:00
Joris Vink ce012e7bd5 set fd to -1 on close. 2019-03-21 10:09:24 +01:00
Joris Vink d371454dad do not grab the result if it was in a gatherop 2019-03-19 13:58:00 +01:00
Joris Vink 8afaac9fde log pid number of worker upon start. 2019-03-19 13:42:22 +01:00
Joris Vink 681a88d535 simplify af_unix recvfrom/sendto support. 2019-03-19 13:27:50 +01:00
Erik Karlsson d60994abb5 add python sendto recvfrom abstract socket support 2019-03-19 11:11:44 +01:00
Joris Vink 9caa45a050 Allow python validator methods to be async. 2019-03-18 09:34:31 +01:00
Joris Vink c3ab570f56 Append full module path if it is a directory. 2019-03-16 16:13:52 +01:00
Joris Vink ec7c8b8e1d fix a few minor problems with pylock. 
- when pylocks are deallocated we ended up deleting the wrong object.
- do not call Py_DECREF on the op when inside the op its iternext call.
 2019-03-14 17:31:57 +01:00
Joris Vink 2217c7a2c8 pysock: call disable read/write only when needed 2019-03-14 09:27:13 +01:00
Joris Vink d41054bd26 remove the socket from the event queue on dealloc. 2019-03-13 16:01:42 +01:00
Joris Vink b06160c768 Make sure coros are woken up on eof. 2019-03-13 15:45:55 +01:00
Joris Vink 3b4574d791 Rework pysocket async/await. 
Attach the events directly to the pysocket data structure instead of
one event per pysocket_op.

Makes the code easier, gives us a good performance boost and reduces
the number of system calls required when doing an await on a socket.
 2019-03-13 11:07:15 +01:00
Joris Vink 01f9b4fcde Check for EAGAIN when calling connect(2). 
Linux does this under certain conditions.
 2019-03-12 20:04:08 +01:00
Joris Vink eb9b7f7b14 explicitly include sys/types.h 
some smaller libc variants do not include this from sys/param.h.
 2019-03-06 09:29:46 +01:00
Joris Vink 61863bfd3a kill recv() timer if we have data. 
also reset any exception that is set when we have successfully
read data from a socket.
 2019-03-04 16:37:25 +01:00
Joris Vink a0c203f507 remove norwegian debug 2019-03-01 20:57:31 +01:00
Joris Vink 31a9a70d5a cleanup. 2019-03-01 20:56:03 +01:00
Erik Karlsson 01c1a8f8f8 support AF_UNIX in recvfrom()/sendto() python sockets 2019-03-01 20:13:28 +01:00
Joris Vink 2057f4db58 sprinkle more kore_quiet around 2019-02-27 19:59:31 +01:00
Joris Vink 1ebd82969c Add timeout support to proc.recv() 2019-02-26 15:22:55 +01:00
Joris Vink f4cd70956b Add an optional timeout to socketop.recv(). 2019-02-25 10:35:00 +01:00
Joris Vink bf1e8e5ffb bump copyright to 2019 2019-02-22 16:57:28 +01:00
Joris Vink 3dbb80463a allow double quotes in kore.proc argument lists. 2019-02-22 16:56:34 +01:00
Joris Vink 66305e551a sig_recv is extern'd in kore.h 2019-02-11 14:01:25 +01:00
Joris Vink 8a8b08f1e6 use %u when logging worker_rlimit_nofiles 2019-02-11 14:00:28 +01:00
Joris Vink 429159ee3c reprime idle timer if SSL_accept() fails. 
reported by Daniel Chavez (@dacechavez)
 2019-02-08 15:40:00 +01:00
Joris Vink 8040103458 fix for multiple consumers for the async queues. 2019-01-22 11:23:59 +01:00
Joris Vink 39467847fb remove SSL_get_verify_result() check. 
If peer verification is turned on this becomes part of the handshake
process anyway and SSL_accept() will fail when appropriate.
 2019-01-21 10:36:50 +01:00
Joris Vink 3f083d6126 deal with crls being expired / not-yet-valid. 
if a crl is expired or not-yet-valid SSL_get_verify_result()
will return these errors too so check for them explicitly
instead of depending on X509_V_OK.

found by @dacechavez
 2019-01-19 11:49:54 +01:00
Joris Vink d1e87c1a54 deal with PyObject_CallObject() returning NULL. 2019-01-15 10:20:13 +01:00
Joris Vink d6b05bcff7 always force reload cert so we get a new x509 store. 
Otherwise older OpenSSL or current LibreSSL will fail to add the new
CRL as they still match on subject name rather then hash of the CRL data.
 2019-01-14 20:57:40 +01:00
Joris Vink 73cdbd1a01 Let CRLs be reloadable via keymgr. 
With these changes CRLs can be reloaded like certificates
by sending a SIGUSR1 to the parent process.

Track mtime on both certificate files and CRL files as well
and only submit them to the workers if this has changed.
 2019-01-14 11:41:50 +01:00
Joris Vink b07cc0237c Support recvfrom()/sendto() on kore python sockets. 2019-01-08 17:49:00 +01:00
Joris Vink a868ff2b25 wrap accesslog calls in !KORE_NO_HTTP. 2018-12-22 09:41:55 +01:00
Joris Vink 9aa0e95643 Rework accesslog handling. 
Move away from the parent constantly hitting the disk for every
accesslog the workers are sending.

The workers will now write their own accesslogs to shared
memory before the parent will pick those up. The parent
will flush them to disk once every second or if they grow
larger then 1MB.

This removes the heavy penalty for having access logs
turned on when you are dealing with a large volume
of requests.
 2018-12-22 09:25:00 +01:00
Joris Vink 0d6a188b01 close fd if we're a directory in filemap. 2018-12-17 16:22:36 +01:00
Joris Vink 61b385ae11 do not set CONN_CLOSE_EMPTY for 1.0 until we reply. 2018-11-30 22:12:43 +01:00
Joris Vink ce3b60b8c8 keep a reference around for tracer. 2018-11-29 09:53:26 +01:00
Joris Vink d9f543ef5b Allow user-supplied tracer callback. 2018-11-29 09:51:24 +01:00
Joris Vink b400fdcd9f include sys/param.h in python.c 2018-11-29 09:11:31 +01:00
Joris Vink fd1ab5879d KODEV_OUTPUT tells kodev the outdir of the binary. 
eg:

$ env KODEV_OUTPUT=/tmp kodev build

will place the resulting binary under /tmp/<binary>
 2018-11-28 14:24:42 +01:00
Joris Vink 8dd075e71c style 2018-11-28 14:22:27 +01:00
Joris Vink e3efeb9465 Move assets.h into object_dir. 2018-11-28 14:19:32 +01:00
Joris Vink c431c2bf72 Add support to obtain peer certificate from Python 
This will return the DER encoded bytes representing the peer certificate.
 2018-11-28 11:28:07 +01:00
Joris Vink 2d8874dd2a change accept threshold default to 16. 2018-11-16 11:37:09 +01:00
Joris Vink 4e70636269 Reduce accept lock accept lockout time to 100ms. 2018-11-16 11:35:53 +01:00
Joris Vink 5c8efde841 allow foo.method symbols in python. 
this way you can create page handlers that reside inside
of other objects.

eg:

static / restapi.index
 2018-11-16 11:07:21 +01:00
Joris Vink baafa4897e Add -q flag. 
If specified Kore will run quietly and only log important messages.
 2018-11-15 16:01:37 +01:00
Joris Vink cb482d8e8f Always use http_response_stream() in req.response. 
Keep around the python bytes object while we stream the contents
of it out over the wire. This avoids an entire copy of the data.
 2018-11-14 10:03:32 +01:00
Joris Vink 966eaf8f7a Add a kore_python_preinit() hook. 
This is called before the python initialization is completed
and allows developers to inject their own built-in methods.
 2018-11-12 09:01:05 +01:00
Joris Vink 3925ba60ad don't call close() no underlying socket. 2018-11-12 09:00:36 +01:00
Joris Vink 4cc9e216bb Don't call close() on the python socket on dealloc 2018-11-06 16:54:26 +01:00
Joris Vink 2dd66586ff several python improvements. 
- add kore.time() as equivalent for kore_time_ms().
- call waitpid() until no more children are available for reaping otherwise
  we risk missing a process if several die at the same time and only one
  SIGCHLD is delivered to us.
- drain a RECV socket operation if eof is set but no exception was given.
 2018-10-30 20:28:27 +01:00
Joris Vink 5456f2e1d5 save/restore coro_running in pygather_op_dealloc(). 2018-10-30 12:37:30 +01:00
Joris Vink 062c80c1ac start enforcing a mininum number of elms in pools. 2018-10-30 10:41:49 +01:00
Joris Vink 71c145932c grow kore_pools at a slower rate. 
Before we just doubled in size the second we exhausted a pool instead
of doing a more controlled expansion.

Now we will expand at 25% of the initial elm count whenever we need to.

Will help with memory pressure in certain scenarios.
 2018-10-30 10:36:18 +01:00
Joris Vink bb00deca2c don't leak op in error path 2018-10-29 21:18:38 +01:00
Joris Vink 1c30da855c Add kore.gather() to the python api. 
Allows one to run coroutines concurrently and gather all their
results in a single returned list.

If any of the coroutines throw an exception the exception is
returned as the value of that coroutine in the returned list.
 2018-10-29 21:16:08 +01:00
Joris Vink 740acb4760 make sure we can still build vs 1.1.0 2018-10-29 21:11:29 +01:00
Joris Vink 818c7847cd simplify even more 2018-10-29 20:43:40 +01:00
Joris Vink ae72b21d35 make sure older openssl/libressl builds work. 2018-10-29 20:42:28 +01:00
Joris Vink 339df66fd5 Add support for TLS 1.3 via OpenSSL 1.1.1. 
This commit removes TLS 1.0 support no matter what OpenSSL
you are linking against.

Changes the value of tls_version from 1.2 to both. Meaning if
you link with OpenSSL 1.1.1 you will get 1.2 + 1.3.
 2018-10-29 20:38:58 +01:00
Joris Vink dda2e1fb2c Some things still talk http/1.0. 2018-10-26 21:24:51 +02:00
Joris Vink e2651889e0 Add asynchronous subprocess support. 
This adds kore.proc to the python runtime allowing async processing
handling:

The kore.proc method takes the command to run and an optional timeout
parameter in milliseconds. If the process did not exit normally after
that amount of time a TimeoutError exception is raised.

For instance:

async def run(cmd):
	proc = kore.proc(cmd, 1000)

	try:
		await proc.send("hello")
		proc.close_stdin()
	except TimeoutError:
		proc.kill()

	retcode = await proc.reap()

	return retcode
 2018-10-26 19:19:47 +02:00
Joris Vink ea7ea48840 actually just query for HW_NCPU on all BSDs. 2018-10-25 19:31:48 +02:00
Joris Vink 9d9e9dcec2 set cpu_count to 1 for unknown BSDs 2018-10-25 19:30:10 +02:00
Joris Vink 892814e353 Add kore_[parent|worker]_teardown(). 
If exists these functions are called when the worker is exiting
and when right before the parent exists.

Allows for cleanup code for applications if need to do cleanup on exit.
 2018-10-23 21:46:34 +02:00
Joris Vink fc5fc4f4ab make sure timers don't go out of scope. 2018-10-23 21:36:19 +02:00
Joris Vink 8ea32983ae Add kore.suspend(delay) to python. 
Will suspend the coroutine for a number of milliseconds.

Example:

async def page(req):
	await kore.suspend(1000)
	req.response(200, b'')
 2018-10-23 21:32:08 +02:00
Joris Vink 47776a9fbb Hook kore timers into python. 2018-10-23 20:44:43 +02:00
Joris Vink c41c1db303 Add kore_shutdown(). 
Allows workers to cleanly initiate a shutdown of the
entire server process.
 2018-10-23 19:49:42 +02:00
Joris Vink 4c35073232 Use number of cpu cores if no worker count is set. 2018-10-22 21:27:17 +02:00
Joris Vink b70d1ee80f Add a locking mechanism in pykore. 
Support the async with syntax:

	lock = kore.lock()

	async with lock:
		# your block

Fix some small issues with other parts of the python system.
 2018-10-22 20:09:23 +02:00
Joris Vink 1ac131c48f If we hit the accept threshold, unlock worker. 2018-10-22 09:01:05 +02:00
Joris Vink 8be316ac0c Let kore_worker_make_busy() set next_lock. 2018-10-22 09:00:55 +02:00
Joris Vink fad5c6ea6f Give pyqueues "popnow". 2018-10-22 08:28:03 +02:00
Joris Vink c8795b7d7f pyqueue improvements. 
- cleanup queue if it gets deallocated.
- make sure waitables on a queue get removed if their pyqueue_op dies.
 2018-10-21 21:58:34 +02:00
Joris Vink 4ae3d23c7e s/INCREF/DECREF 2018-10-18 22:24:58 +02:00
Joris Vink 0cda9ecfb0 Add an asynchronous queue mechanism. 
This allows coroutines to submit messages to and pop
messages from a queue in an asynchronous way.
 2018-10-18 22:15:21 +02:00
Joris Vink b5958f7d7d Add kore_parent_daemonized(). 
This is called for single binaries after the parent
process has called daemon().

Also fix kore_parent_configure() for !single binaries.
 2018-10-18 17:18:41 +02:00
Joris Vink c172d49567 kodev improvements. 
- Take CFLAGS, CXXFLAGS and LDFLAGS from environment
  if present and append them to the build.conf cflags

- Allow overriding of the OBJDIR via KORE_OBJDIR.

- Allow overriding of build flavor via KORE_BUILD_FLAVOR.
 2018-10-17 11:30:14 +02:00
Joris Vink 300daeadef fix comment 2018-10-16 13:17:44 +02:00
Joris Vink 29202d7330 Make kore_python_log_error() public. 
While here also make kore_module_load() return the
kore_module data structure pointer it just added
to the modules list.
 2018-10-16 13:16:36 +02:00
Joris Vink 20a0103f1e Add async/await support for socket i/o in python. 
This means you can now do things like:

	resp = await koresock.recv(1024)
	await koresock.send(resp)

directly from page handlers if they are defined as async.

Adds lots more to the python goo such as fatalx(), bind_unix(),
task_create() and socket_wrap().
 2018-10-15 20:18:54 +02:00
Joris Vink c12f296743 missing EPOLLRDHUP check since event loop rework 2018-10-10 14:33:26 +02:00
Joris Vink c463ecb3cb Changes to the event loop inside of Kore. 
Now anyone can schedule events and get a callback to work as long
as the user data structure that is added for the event begins
with a kore_event data structure.

All event state is now kept in that kore_event structure and renamed
CONN_[READ|WRITE]_POSSIBLE to KORE_EVENT_[READ|WRITE].
 2018-10-09 19:34:40 +02:00
Joris Vink 2449a86085 missing addrtype -> family renames 2018-10-08 20:12:25 +02:00
Joris Vink 884d6d722e add listener to the list earlier. 2018-10-07 21:25:00 +02:00
Joris Vink 19044919b2 don't set nodelay on unix listener sockets 2018-10-07 21:21:37 +02:00
Joris Vink 9427ed8a2e rename unix to sun 2018-10-07 21:03:12 +02:00
Joris Vink 442bdef79b allow kore to bind to unix sockets via bind_unix. 2018-10-07 20:49:16 +02:00
Joris Vink f87624a459 Add missing KORE_NO_TLS guards. 
Fixes NOTLS=1 builds on openssl 1.1.1
 2018-10-01 10:36:33 +02:00
Joris Vink 7db3e4d946 fix compare match on db name. 
from Kevin Lam via patches@
 2018-09-19 07:18:46 +02:00
Joris Vink ba8d6301b3 style 2018-08-30 09:13:11 +02:00
Joris Vink 566fefd031 do not http_argument_urldecode for multipart data. 2018-08-16 14:11:28 +02:00
Joris Vink 754ba47cc4 shuffle stuff around so we're not hitting bad juju. 2018-08-13 13:07:32 +02:00
Joris Vink 8d3f73bc71 call fatalx() instead of fatal() in certain cases. 2018-08-13 13:01:27 +02:00
Joris Vink b0074ba45e Add fatalx(). 
If a worker process dies it automatically gets respawned by the
parent process, but sometimes you want the entire server to go down
if a worker encounters an error. This is what fatalx() does.

Calling fatalx() from a worker process will initiate a full shutdown
of the kore server you are running under.
 2018-08-13 09:53:49 +02:00
Joris Vink cf92097bc2 log what dir we can't chdir() into. 2018-08-13 09:53:38 +02:00
Joris Vink 6ef8d59f7d more directly include assets.h 2018-08-13 09:16:28 +02:00
Joris Vink d1c8f95590 Show the reason why regcomp() failed if it does. 2018-08-10 08:06:09 +02:00
Joris Vink 03b927dd64 missing EPOLLET when we reschedule read/write 2018-08-01 12:17:16 +02:00
Joris Vink a927acb7ee Add pledge support under OpenBSD. 
All worker processes will now call pledge(2) after dropping
privileges (even if -rn was specified).

By default Kore will use the following promises:
	"stdio rpath inet error"

If your application requires more privileges, you can add more pledges
by setting them in your configuration using the 'pledge' directive:
	pledge dns wpath
 2018-07-31 06:51:34 +02:00
Joris Vink 69922598e7 Missing EV_CLEAR for schedule read/write on BSD. 2018-07-30 15:58:47 +02:00
Joris Vink 5b3cee3428 Use a synchronous query to issue a ROLLBACK. 2018-07-28 22:28:19 +02:00
Joris Vink 43234cd36e be less spammy if stat fails with ENOENT. 2018-07-28 22:27:38 +02:00
Joris Vink f126ba5a86 sprinkle more const around 2018-07-25 09:54:34 +02:00
Joris Vink cf1f624367 let filerefs to operate on ms precision for mtime. 2018-07-24 19:56:36 +02:00
Joris Vink 7f820c96e8 Make kore_auth_* public. 2018-07-19 10:28:38 +02:00
Joris Vink 821c1df8ec use method not allowed when required 2018-07-18 16:24:28 +02:00
Joris Vink 916ce222b4 better fix for 5a5d9fd0. 
Don't let net_recv_flush() do things as long as the HTTP layer
owns the buffer. When we have sent a response kick the read end
back into gear ourselves by calling net_recv_flush().
 2018-07-18 16:10:41 +02:00
Joris Vink 086e2e3d5e len is a size_t, not ssize_t 2018-07-18 16:09:05 +02:00
Joris Vink 25c8dfbdaa missing rpath pledge for openbsd 2018-07-18 15:14:07 +02:00
Joris Vink 5a5d9fd0c2 alloc space for nb->buf after taking ownership. 2018-07-18 14:36:13 +02:00
Joris Vink 27d1746940 Consume all notifications on a connection. 2018-07-18 11:40:59 +02:00
Joris Vink 2e321f14de Add KORE_PGSQL_STATE_NOTIFY. 
Issue a LISTEN channel on a kore_pgsql, bind a callback to it and you
will get called with pgsql->state being KORE_PGSQL_STATE_NOTIFY.
 2018-07-18 11:38:17 +02:00
Joris Vink 1447f6573f better http header validation. 2018-07-17 20:17:05 +02:00
Joris Vink 3312a2882f Let KORE_PREFIX override builtin prefix for kodev. 2018-07-17 15:59:59 +02:00
Joris Vink c2289471b1 forcefully limit filemaps to GET | HEAD. 2018-07-17 15:05:20 +02:00
Joris Vink 616af063e3 Calculate an md over the incoming HTTP body. 
This is calculated while the HTTP body is incoming over the wire, once
the body is fully received the digest will be available for the page
handlers to obtain.

You can obtain a hex string for this md via http_body_digest() or
dereferences the http_request and look at http_body_digest manually
for the bytes.
 2018-07-17 14:53:55 +02:00
Joris Vink 9f2759e7b2 missing newline in error 2018-07-17 14:36:21 +02:00
Joris Vink 40a81a17ba remove kore_module_handle for NOHTTP=1 builds. 2018-07-17 14:28:43 +02:00
Joris Vink 0726a26c0c Allow restriction of methods for paths. 
Now Kore will automatically send a 400 bad request in case the
method was not allowed on the path.
 2018-07-17 14:23:57 +02:00
Joris Vink 993c5d2ac2 pledge keymgr process on openbsd 2018-07-14 21:14:02 +02:00
Joris Vink b957d26335 Let kore_worker_make_busy() not unlock solo workers 2018-07-13 21:04:59 +02:00
Joris Vink 02e7359970 Add kore_worker_make_busy(). 
Calling this from your page handler will cause your current worker
to give up the acceptlock (if it holds it).

This is particularly useful if you are about to run code that may block
a bit longer then you are comfortable with. Calling this will cause
the acceptlock to shuffle to another free worker which in turn makes
sure your application can keep accepting requests.
 2018-07-11 18:00:16 +02:00
Joris Vink 52d14a3fbf Unbreak building with LibreSSL. 2018-07-11 12:50:50 +02:00
Joris Vink 2b3119d0e2 add overview of how the keymgr works 2018-07-11 11:53:56 +02:00
Joris Vink 6cf3b3c0dc Only use kore_root_path if its explicitly set. 
Otherwise a relative path works well enough.
 2018-07-11 11:08:44 +02:00
Joris Vink 77e0439225 better log messages for TLS configuration missing. 2018-07-11 09:52:05 +02:00
Joris Vink cffb7ec379 Allow on-the-fly reloading of certificates/keys. 
This commit introduces the ability for the keymgr process
to reload the certificates/keys for domains when receiving
a SIGUSR1 signal.

The keymgr receives 2 new configuration options:
	- keymgr_root_path
		The root path where the keymgr will live.
		If -n is not specified when the application starts the
		keymgr process will chroot into here.

	- keymgr_runas_user
		The user the keymgr will drop privileges towards if
		-r was not specified.

All certfile and certkey configuration options are now relative to the
keymgr_root_path configuration setting.

The keymgr process will now also load the certificate for the domain
(rather then the workers) and submit these to the worker processes so
they can be reloaded when required.

Worker processes will refuse connections until the TLS configuration
for a given domain is completed (aka: the workers receive the certificate
for that domain).

Other changes:
	- client_certificates renamed to client_verify.
	- the chroot configuration option is now called root.
	- kore is a little more verbose if privsep options are missing.
	- filemaps are now relative to the root configuration option.
 2018-07-11 09:44:29 +02:00
Joris Vink bf6c0e150f Let kore_worker_privdrop() take user and chroot. 
This will make it easier when the keymgr gets its own user/chroot settings.
 2018-07-11 06:53:51 +02:00
Joris Vink ac6fcf8981 deal with sendfile() in newer linux kernels. 2018-07-09 10:00:38 +00:00
Joris Vink e665cc900d redirect to /uri/ if need be. 2018-07-09 06:28:28 +02:00
Joris Vink 04f4306cf2 simplify logic 2018-07-08 15:58:56 +00:00
Joris Vink 04deeeafb9 make sure we can still resolve new paths 2018-07-08 15:57:00 +00:00
Joris Vink 5eb2160269 resolve filemap paths after workers chrooted. 
otherwise the paths inside chrooted workers are incorrect.
 2018-07-08 17:51:35 +02:00
Joris Vink 10cf14f756 use realpath() to resolve ondisk paths. 2018-07-08 17:40:16 +02:00
Joris Vink 3ddcaf4661 typo 2018-07-07 13:34:58 +02:00
Joris Vink 71659ab197 correct includes 2018-07-07 13:23:43 +02:00
Joris Vink f02f88295c revert b5e122 for now. 2018-07-06 11:21:46 +02:00
Joris Vink 4f16a5d272 make net_read() and net_write() more sane. 2018-07-05 12:36:47 +00:00
Joris Vink 47c1a1d195 set referer to NULL in http_request_new(). 2018-07-05 05:02:49 +00:00
Joris Vink 3e4b9f4fe8 let filemaps be matched to the domains. 2018-07-04 14:26:38 +02:00
Joris Vink 04077c66b6 Add filemap_ext configuration option. 
Allows you to specify the default extensions used for a file served
via a filemap, eg:
	filemap_ext	.html

Gives us ability to provide clean urls.
 2018-07-03 19:58:43 +02:00
Joris Vink b5e122419b Let http_populate_post() listen to content-type 2018-07-03 08:25:06 +02:00
Joris Vink 4a8d8ab7f8 log referer in accesslog if present. 2018-06-29 22:37:48 +02:00
Joris Vink 72073701b0 Add last-modified and if-modified-since for filemaps. 2018-06-29 09:56:04 +02:00
Joris Vink cca269ff5d make sure we use fd_off in linux sendfile properly 2018-06-29 03:10:28 +00:00
Joris Vink 3e5939a8e3 make sure fileref is dropped if softremoved 2018-06-29 05:03:50 +02:00
Joris Vink 09b362ced4 remove norwegian debugging 2018-06-28 23:02:46 +02:00
Joris Vink 202234cf97 filemap and fileref improvements. 
- make sure we can serve updated files even if we have an old
  fileref around.

- add filemap_index as a configuration option: allows one to specify
  what file to serve if a directory was requested (eg: index.html)
 2018-06-28 23:00:42 +02:00
Joris Vink 521ff6a11d catch more bad ranges in http_argument_urldecode() 2018-06-28 15:39:03 +02:00
Joris Vink 2139527df7 run path via http_argument_urldecode(). 2018-06-28 15:28:25 +02:00
Joris Vink 70e945afb7 limit http_argument_urldecode() to sane characters 2018-06-28 15:27:55 +02:00
Joris Vink c2f66af937 better error log message 2018-06-28 14:53:43 +02:00
Joris Vink 3faf89d83d use server time. 2018-06-28 14:52:49 +02:00
Joris Vink afd76ff55d Change accesslog format to Combined Log Format. 2018-06-28 14:25:32 +02:00
Joris Vink f2c87fd130 limit filemaps to HEAD/GET requests. 2018-06-28 14:24:02 +02:00