Commit Graph

171 Commits

Author SHA1 Message Date
Joris Vink 55aaef875d Add support for setting curlopts in kore.httpclient.
Much of the work done by Matthew Norström with minor cleanup by me.
2021-08-27 10:05:30 +02:00
Joris Vink 90056dbdcb make python_module_init() non static 2021-07-10 10:02:46 +02:00
Joris Vink 95139925ec Add query string support to the Python validator API.
Now you can specify the qs keyword in a route which can contain
validators for the query string.

Eg:

@kore.route("/", methods=["post"], qs={"id": "^[0-9]+$"})
def index:
    ...
2021-06-18 13:00:57 +02:00
Joris Vink f1a65ef236 Small improvement to the Python kore.timer() api.
Do not allow kore.timer() to be called from the parent process
as it shouldn't be run there.

This makes Kore fail more gracefully.
2021-06-03 14:03:45 +02:00
Joris Vink fb335e1e0c Major Python API improvements.
1) Add @kore.route as a decorator for Python.

This decorator can be used on non-class methods to automatically
declare their route and parameters.

Takes the same arguments as the kore.domain.route function that
exists today.

Provides a nice clean way of setting up Kore if you dont want
a whole class based approach.

2) Remove the requirement for the name for kore.server() and the
kore.domain(attach=) keywords.

Instead of no name was given, the name "default" is used in both
places resulting in less boilerplating.

3) Allow multiple routes to be defined for the same URI as long
as the methods are different. So you can have one method for GET /
and another for POST /.

All changes combined condense the initial experience of getting
a Kore Python app up and running:

eg:

import kore

kore.server(ip="127.0.0.1", port="8888", tls=False)
kore.domain("*")

@kore.route("/", methods=["get"])
async def index(req):
    req.response(200, b'get method')

@kore.route("/", methods=["post"])
async def index_post(req)
    req.response(200, b'post method')
2021-05-02 00:32:47 +02:00
Joris Vink 0549295f30 Cleanup integer types for # Python arguments.
Define PY_SSIZE_T_CLEAN before pulling in Python.h
2021-01-28 13:34:43 +01:00
Joris Vink c36a7759f1 Get Kore to build with Python 3.10.0a3. 2021-01-28 10:17:02 +01:00
Joris Vink cef5ac4003 bump copyright year. 2021-01-11 23:46:08 +01:00
Joris Vink cb0f5a4137 Make sure koreapp.onmsg can log errors from Python. 2020-12-18 19:48:17 +01:00
Joris Vink 4313c0eab5 Python: Add kore.socket.recvmsg()
Returns the ancillary data to the caller as a list.
2020-10-02 11:27:52 +02:00
Joris Vink eaef4b654a Only call PyMem_SetupDebugHooks() if DEBUG is 1.
We do not need the memory debug hooks in production environments.
2020-08-13 09:10:19 +02:00
Joris Vink 6b3347ae3a coro_running could be NULL, check it. 2020-08-12 13:43:38 +02:00
Joris Vink 7613a4a135 Python: Improve the kore.socket interface.
- Make sure PyBuffer_Release() is called for y* arguments.
- Make sure buffer is cleaned up for SENDTO socket ops as well.
2020-08-11 15:24:59 +02:00
Joris Vink 68d078766e Clear lockop before removing a reference from the coroutine. 2020-08-10 12:19:42 +02:00
Joris Vink 0268a0ae0a Wrap certain syscalls inside of an ifdef.
Makes kore with python and acme work on my pinebook pro.
2020-07-14 15:38:54 +02:00
Joris Vink bb2f0d8b52 Python: improve kore.lock when handling cancelled coroutines.
If a coroutine is killed from another coroutine and the killed coroutine
was waiting on a kore.lock() object, it would have been incorrectly
woken up again once said lock was released.

This would cause a Python exception that a generator was already
running and a crash due to the pool element already being freed.

Track the active locking operation per coroutine so we can remove
the coroutine if it is killed, fixing the problem.
2020-07-09 20:22:18 +02:00
Joris Vink 8235759bca Python: Add kore.app().
This method allows you to set a Python object and obtain it
by calling the method again without any arguments.

eg:

foo = SomeClass()

kore.app(foo)

foo = kore.app()
2020-07-05 21:47:22 +02:00
Joris Vink e38c6e5d30 Python: Several fixes for our async curl support.
- Fix the curl-extract-opt.sh generation script to work on newer
  curl releases as the header changed slightly.
- Use the correct handles when calling curl_easy_setopt() inside
  of our setopt functions exported via Python.
- Add a curl.setbody() method, allowing a body to be sent to be set.
  (eg when sending mail via SMTP).
- Regen of our python_curlopt.h from 7.71.1
2020-07-02 08:41:17 +02:00
Joris Vink 122a86013b Python: respond with 500 in case of a coroutine error.
If a coroutine throws an exception, respond with a 500
after logging the exception itself.
2020-06-30 09:57:48 +02:00
Joris Vink 8a39d18196 work around different dirname()/basename() implementations. 2020-06-09 12:22:22 +02:00
Joris Vink 9d0aef0079 bump copyright 2020-02-10 14:47:33 +01:00
Joris Vink d86a10afa1 allow use udata in kore.timer() via the data kwarg. 2020-01-22 09:42:41 +01:00
Joris Vink 2d380cac3f Expose our async libcurl support to the Python api.
Kore already exposed parts of this via the kore.httpclient() method but
this commit takes it a bit further and exposes the libcurl interface
completely (including the setopt options).

tldr:

handle = kore.curl("ftp://ftp.eu.openbsd.org/pub/OpenBSD/README")
handle.setopt(kore.CURLOPT_TIMEOUT, 5)

data = await handle.run()
print("%s" % data.decode())
2020-01-18 19:43:38 +01:00
Joris Vink 56c33f85d4 change client_authority to client_verify. 2019-11-18 20:30:52 +01:00
Joris Vink 73757a29d5 Make dumb compilers happy. 2019-11-15 07:49:16 +01:00
Joris Vink b3b5aa37b7 Allow acme config via python api 2019-11-13 23:01:24 +01:00
Joris Vink 7b5046873a Make sure we wakeup the coroutine that called proc.reap().
We actually woke up the coroutine that originally spawned the process
when we reap it, but another coroutine may have taken over the object.

This mimics how we do things for the pysock_op things.
2019-10-29 15:12:20 +01:00
Joris Vink 790d020ce9 Stop a python coro from getting stuck with httpclient.
In cases where a request is immediately completed in libcurl its multi
handle and no additional i/o is happening a coro can get stuck waiting
to be run.

Prevent this by lowering netwait from KORE_WAIT_INFINITE if there
are pending python coroutines.
2019-10-22 17:06:32 +02:00
Joris Vink a119f18a23 Adjust to new kore_curl_init(). 2019-10-21 13:29:26 +02:00
Joris Vink cdc3347120 Add kore.sendmsg(object, worker=None) to the python api.
This allows you to send Python objects that can be run through pickle
to other worker processes.

If your application implements koreapp.onmsg() you will be able to receive
these objects.
2019-10-16 12:05:27 +02:00
Joris Vink e90e3749a7 use Py_InitializeEx and do not install python signals. 2019-10-15 21:18:38 +02:00
Joris Vink bdc75556fe call PyErr_Clear() in python_callable(). 2019-10-15 16:03:39 +02:00
Joris Vink a1c1245ef7 don't call Py_DECREF on list in configure 2019-10-15 15:53:08 +02:00
Joris Vink 0f4d5294ad store active pid into apid so it does not overwritten. 2019-10-15 14:25:57 +02:00
Joris Vink bc33a5def4 Add kore.proc.pid, returns the PID of the proc. 2019-10-15 14:23:49 +02:00
Joris Vink 3dcf94d1ae Add seccomp.bpf_stmt() method to Python api.
Allows a developer to create their own statements, the bpf_jmp equivalent
may follow later if I need it.
2019-10-07 13:44:31 +02:00
Joris Vink 7bc1a92192 remove norwegian debug 2019-10-04 12:44:37 +02:00
Joris Vink 6c8970651c s/hook/method 2019-10-04 11:34:49 +02:00
Joris Vink 11cc5d304f let kore_python_seccomp_hook() take the hook name 2019-10-04 11:28:14 +02:00
Joris Vink 8bbdaedf94 Allow configuring seccomp on Linux via the python api.
A new hook in the koreapp class is called right before seccomp
is enabled. This hook receives a Kore seccomp object which has
the following methods:

	seccomp.allow("syscall")
	seccomp.allow_arg("syscall", arg, value)
	seccomp.allow_flag("syscall", arg, flag)
	seccomp.allow_mask("syscall", arg, mask)

	seccomp.deny("syscall")
	seccomp.deny_arg("syscall", arg, value, errno=EACCES)
	seccomp.deny_flag("syscall", arg, flag, errno=EACCES)
	seccomp.deny_mask("syscall", arg, mask, errno=EACCES)

This allows you to finetune the seccomp filters for your application
from inside your koreapp.
2019-10-04 10:59:48 +02:00
Joris Vink 46375303cb Allow multiple binds on new server directive. 2019-09-27 20:00:35 +02:00
Joris Vink be70118bf2 move l->tls check upwards 2019-09-27 12:33:48 +02:00
Joris Vink 7350131232 Allow listening of tls/notls ports at the same time.
Before kore needed to be built with NOTLS=1 to be able to do non TLS
connections. This has been like this for years.

It is time to allow non TLS listeners without having to rebuild Kore.

This commit changes your configuration format and will break existing
applications their config.

Configurations now get listener {} contexts:

listen default {
	bind 127.0.0.1 8888
}

The above will create a listener on 127.0.0.1, port 8888 that will serve
TLS (still the default).

If you want to turn off TLS on that listener, specify "tls no" in that
context.

Domains now need to be attached to a listener:

Eg:
	domain * {
		attach	default
	}

For the Python API this kills kore.bind(), and kore.bind_unix(). They are
replaced with:

	kore.listen("name", ip=None, port=None, path=None, tls=True).
2019-09-27 12:27:04 +02:00
Joris Vink d4ba6f3485 allow more net related syscalls for python. 2019-09-26 16:13:42 +02:00
Joris Vink 937c39f041 Many many Python improvements.
- Kore can now fully be configured via Python code if one wants nothing to
  do with configuration files.

- Kore can now start single python files and no longer requires them to be
  inside a module directory.

- Pass all regex capture groups to the handler methods, allowing you to
  get access to them immediately.

- Change python websocket_handshake to take callable objects directly.

- Added a new deployment configuration option. If set to "dev" or
  "development" Kore will automatically foreground, no chroot / etc.
  If set to "production" Kore *will* chroot, drop privs, etc.

- Many more..

These are all backported from a project that I was working on a while
ago. I decided these should go back into mainline Kore.
2019-09-26 15:49:00 +02:00
Joris Vink cd9971247c Add seccomp syscall filtering to kore.
With this commit all Kore processes (minus the parent) are running
under seccomp.

The worker processes get the bare minimum allowed syscalls while each module
like curl, pgsql, etc will add their own filters to allow what they require.

New API functions:
    int kore_seccomp_filter(const char *name, void *filter, size_t len);

    Adds a filter into the seccomp system (must be called before
    seccomp is enabled).

New helpful macro:
    define KORE_SYSCALL_ALLOW(name)

    Allow the syscall with a given name, should be used in
    a sock_filter data structure.

New hooks:
    void kore_seccomp_hook(void);

    Called before seccomp is enabled, allows developers to add their
    own BPF filters into seccomp.
2019-09-25 14:31:20 +02:00
Joris Vink d99d0b2d77 add lock.trylock() and make lock.release() sync. 2019-09-20 10:03:43 +02:00
Joris Vink 868b0a71a2 make coro tracing available at runtime, always.
call kore.corotrace(True) to enable it and kore.corotrace(False) to disable.
2019-09-18 10:55:13 +02:00
Joris Vink 95506204e8 Add kore.task_kill() to the python api.
Allows killing of coroutines, given their task id.

The kore.task_create() method now returns the task id for a newly
created task to the caller.

While here, change the coroutine task id to a uint32 from uint64.
There is no need for it to be 64bit. (famous last words)
2019-09-17 20:04:35 +02:00
Joris Vink c1653a5254 hide kore.coroname() behind PYTHON_CORO_TRACE. 2019-09-16 20:33:14 +02:00