mirrors
/
kore
mirror of https://git.kore.io/kore.git
1
0
Fork 1
Code Issues Projects Releases Wiki Activity
kore/src/curl.c

763 lines
17 KiB
C
Raw Permalink Normal View History

Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
/*
i forgot, it's 2022.
2022-01-31 22:02:06 +01:00
* Copyright (c) 2019-2022 Joris Vink <joris@coders.se>
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
*
* Permission to use, copy, modify, and distribute this software for any
* purpose with or without fee is hereby granted, provided that the above
* copyright notice and this permission notice appear in all copies.
*
* THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
* WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
* ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
* WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
* ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
* OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
#include <sys/types.h>
missing header
2019-04-24 00:16:51 +02:00
#include <inttypes.h>
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
#include "kore.h"
#include "http.h"
#include "curl.h"
Add seccomp syscall filtering to kore. With this commit all Kore processes (minus the parent) are running under seccomp. The worker processes get the bare minimum allowed syscalls while each module like curl, pgsql, etc will add their own filters to allow what they require. New API functions: int kore_seccomp_filter(const char *name, void *filter, size_t len); Adds a filter into the seccomp system (must be called before seccomp is enabled). New helpful macro: define KORE_SYSCALL_ALLOW(name) Allow the syscall with a given name, should be used in a sock_filter data structure. New hooks: void kore_seccomp_hook(void); Called before seccomp is enabled, allows developers to add their own BPF filters into seccomp.
2019-09-25 14:25:49 +02:00
#if defined(__linux__)
#include "seccomp.h"
static struct sock_filter filter_curl[] = {
/* Allow sockets and libcurl to call connect. */
KORE_SYSCALL_ALLOW(bind),
Allow listening of tls/notls ports at the same time. Before kore needed to be built with NOTLS=1 to be able to do non TLS connections. This has been like this for years. It is time to allow non TLS listeners without having to rebuild Kore. This commit changes your configuration format and will break existing applications their config. Configurations now get listener {} contexts: listen default { bind 127.0.0.1 8888 } The above will create a listener on 127.0.0.1, port 8888 that will serve TLS (still the default). If you want to turn off TLS on that listener, specify "tls no" in that context. Domains now need to be attached to a listener: Eg: domain * { attach default } For the Python API this kills kore.bind(), and kore.bind_unix(). They are replaced with: kore.listen("name", ip=None, port=None, path=None, tls=True).
2019-09-27 12:22:35 +02:00
KORE_SYSCALL_ALLOW(ioctl),
Add seccomp syscall filtering to kore. With this commit all Kore processes (minus the parent) are running under seccomp. The worker processes get the bare minimum allowed syscalls while each module like curl, pgsql, etc will add their own filters to allow what they require. New API functions: int kore_seccomp_filter(const char *name, void *filter, size_t len); Adds a filter into the seccomp system (must be called before seccomp is enabled). New helpful macro: define KORE_SYSCALL_ALLOW(name) Allow the syscall with a given name, should be used in a sock_filter data structure. New hooks: void kore_seccomp_hook(void); Called before seccomp is enabled, allows developers to add their own BPF filters into seccomp.
2019-09-25 14:25:49 +02:00
KORE_SYSCALL_ALLOW(connect),
More seccomp work.
2023-04-06 10:16:06 +02:00
KORE_SYSCALL_ALLOW(socketpair),
Allow listening of tls/notls ports at the same time. Before kore needed to be built with NOTLS=1 to be able to do non TLS connections. This has been like this for years. It is time to allow non TLS listeners without having to rebuild Kore. This commit changes your configuration format and will break existing applications their config. Configurations now get listener {} contexts: listen default { bind 127.0.0.1 8888 } The above will create a listener on 127.0.0.1, port 8888 that will serve TLS (still the default). If you want to turn off TLS on that listener, specify "tls no" in that context. Domains now need to be attached to a listener: Eg: domain * { attach default } For the Python API this kills kore.bind(), and kore.bind_unix(). They are replaced with: kore.listen("name", ip=None, port=None, path=None, tls=True).
2019-09-27 12:22:35 +02:00
KORE_SYSCALL_ALLOW(getsockopt),
KORE_SYSCALL_ALLOW(getsockname),
seccomp improvements. More BPF helper macros, more helper for granular syscall checking. Use these throughout kore where it makes sense. The new helpers: - KORE_SYSCALL_DENY_ARG(name, arg, value, errno): Deny the system call with errno if the argument matches value. - KORE_SYSCALL_DENY_MASK(name, arg, mask, errno): Deny the system call with errno if the mask argument does not match the exact mask given. - KORE_SYSCALL_DENY_WITH_FLAG(name, arg, flag, errno): Deny the system call with errno if the argument contains the given flag. The reverse also exists: - KORE_SYSCALL_ALLOW_ARG() - KORE_SYSCALL_ALLOW_MASK() - KORE_SYSCALL_ALLOW_WITH_FLAG()
2019-09-26 13:51:53 +02:00
KORE_SYSCALL_ALLOW_ARG(socket, 0, AF_INET),
KORE_SYSCALL_ALLOW_ARG(socket, 0, AF_INET6),
KORE_SYSCALL_ALLOW_ARG(socket, 0, AF_UNIX),
Allow listening of tls/notls ports at the same time. Before kore needed to be built with NOTLS=1 to be able to do non TLS connections. This has been like this for years. It is time to allow non TLS listeners without having to rebuild Kore. This commit changes your configuration format and will break existing applications their config. Configurations now get listener {} contexts: listen default { bind 127.0.0.1 8888 } The above will create a listener on 127.0.0.1, port 8888 that will serve TLS (still the default). If you want to turn off TLS on that listener, specify "tls no" in that context. Domains now need to be attached to a listener: Eg: domain * { attach default } For the Python API this kills kore.bind(), and kore.bind_unix(). They are replaced with: kore.listen("name", ip=None, port=None, path=None, tls=True).
2019-09-27 12:22:35 +02:00
KORE_SYSCALL_ALLOW_ARG(socket, 0, AF_NETLINK),
Add seccomp syscall filtering to kore. With this commit all Kore processes (minus the parent) are running under seccomp. The worker processes get the bare minimum allowed syscalls while each module like curl, pgsql, etc will add their own filters to allow what they require. New API functions: int kore_seccomp_filter(const char *name, void *filter, size_t len); Adds a filter into the seccomp system (must be called before seccomp is enabled). New helpful macro: define KORE_SYSCALL_ALLOW(name) Allow the syscall with a given name, should be used in a sock_filter data structure. New hooks: void kore_seccomp_hook(void); Called before seccomp is enabled, allows developers to add their own BPF filters into seccomp.
2019-09-25 14:25:49 +02:00
/* Threading related. */
add more syscalls to seccomp whitelists.
2023-04-06 09:33:48 +02:00
#if defined(SYS_clone3)
KORE_SYSCALL_ALLOW(clone3),
#endif
#if defined(SYS_rseq)
KORE_SYSCALL_ALLOW(rseq),
#endif
More seccomp work.
2023-04-06 10:16:06 +02:00
KORE_SYSCALL_ALLOW(clone),
KORE_SYSCALL_ALLOW(set_robust_list),
Add seccomp syscall filtering to kore. With this commit all Kore processes (minus the parent) are running under seccomp. The worker processes get the bare minimum allowed syscalls while each module like curl, pgsql, etc will add their own filters to allow what they require. New API functions: int kore_seccomp_filter(const char *name, void *filter, size_t len); Adds a filter into the seccomp system (must be called before seccomp is enabled). New helpful macro: define KORE_SYSCALL_ALLOW(name) Allow the syscall with a given name, should be used in a sock_filter data structure. New hooks: void kore_seccomp_hook(void); Called before seccomp is enabled, allows developers to add their own BPF filters into seccomp.
2019-09-25 14:25:49 +02:00
/* Other */
Add acmev2 (RFC8555) support to Kore. A new acme process is created that communicates with the acme servers. This process does not hold any of your private keys (no account keys, no domain keys etc). Whenever the acme process requires a signed payload it will ask the keymgr process to do the signing with the relevant keys. This process is also sandboxed with pledge+unveil on OpenBSD and seccomp syscall filtering on Linux. The implementation only supports the tls-alpn-01 challenge. This means that you do not need to open additional ports on your machine. http-01 and dns-01 are currently not supported (no wildcard support). A new configuration option "acme_provider" is available and can be set to the acme server its directory. By default this will point to the live letsencrypt environment: https://acme-v02.api.letsencrypt.org/directory The acme process can be controlled via the following config options: - acme_root (where the acme process will chroot/chdir into). - acme_runas (the user the acme process will run as). If none are set, the values from 'root' and 'runas' are taken. If you want to turn on acme for domains you do it as follows: domain kore.io { acme yes } You do not need to specify certkey/certfile anymore, if they are present still they will be overwritten by the acme system. The keymgr will store all certificates and keys under its root (keymgr_root), the account key is stored as "/account-key.pem" and all obtained certificates go under "certificates/<domain>/fullchain.pem" while keys go under "certificates/<domain>/key.pem". Kore will automatically renew certificates if they will expire in 7 days or less.
2019-11-06 19:33:53 +01:00
KORE_SYSCALL_ALLOW(uname),
Add seccomp syscall filtering to kore. With this commit all Kore processes (minus the parent) are running under seccomp. The worker processes get the bare minimum allowed syscalls while each module like curl, pgsql, etc will add their own filters to allow what they require. New API functions: int kore_seccomp_filter(const char *name, void *filter, size_t len); Adds a filter into the seccomp system (must be called before seccomp is enabled). New helpful macro: define KORE_SYSCALL_ALLOW(name) Allow the syscall with a given name, should be used in a sock_filter data structure. New hooks: void kore_seccomp_hook(void); Called before seccomp is enabled, allows developers to add their own BPF filters into seccomp.
2019-09-25 14:25:49 +02:00
KORE_SYSCALL_ALLOW(ioctl),
KORE_SYSCALL_ALLOW(madvise),
KORE_SYSCALL_ALLOW(recvmsg),
KORE_SYSCALL_ALLOW(sendmmsg),
Add acmev2 (RFC8555) support to Kore. A new acme process is created that communicates with the acme servers. This process does not hold any of your private keys (no account keys, no domain keys etc). Whenever the acme process requires a signed payload it will ask the keymgr process to do the signing with the relevant keys. This process is also sandboxed with pledge+unveil on OpenBSD and seccomp syscall filtering on Linux. The implementation only supports the tls-alpn-01 challenge. This means that you do not need to open additional ports on your machine. http-01 and dns-01 are currently not supported (no wildcard support). A new configuration option "acme_provider" is available and can be set to the acme server its directory. By default this will point to the live letsencrypt environment: https://acme-v02.api.letsencrypt.org/directory The acme process can be controlled via the following config options: - acme_root (where the acme process will chroot/chdir into). - acme_runas (the user the acme process will run as). If none are set, the values from 'root' and 'runas' are taken. If you want to turn on acme for domains you do it as follows: domain kore.io { acme yes } You do not need to specify certkey/certfile anymore, if they are present still they will be overwritten by the acme system. The keymgr will store all certificates and keys under its root (keymgr_root), the account key is stored as "/account-key.pem" and all obtained certificates go under "certificates/<domain>/fullchain.pem" while keys go under "certificates/<domain>/key.pem". Kore will automatically renew certificates if they will expire in 7 days or less.
2019-11-06 19:33:53 +01:00
KORE_SYSCALL_ALLOW(faccessat),
KORE_SYSCALL_ALLOW(newfstatat),
Add seccomp syscall filtering to kore. With this commit all Kore processes (minus the parent) are running under seccomp. The worker processes get the bare minimum allowed syscalls while each module like curl, pgsql, etc will add their own filters to allow what they require. New API functions: int kore_seccomp_filter(const char *name, void *filter, size_t len); Adds a filter into the seccomp system (must be called before seccomp is enabled). New helpful macro: define KORE_SYSCALL_ALLOW(name) Allow the syscall with a given name, should be used in a sock_filter data structure. New hooks: void kore_seccomp_hook(void); Called before seccomp is enabled, allows developers to add their own BPF filters into seccomp.
2019-09-25 14:25:49 +02:00
KORE_SYSCALL_ALLOW(getpeername),
More seccomp work.
2023-04-06 10:16:06 +02:00
#if defined(SYS_getdents64)
KORE_SYSCALL_ALLOW(getdents64),
#endif
Add seccomp syscall filtering to kore. With this commit all Kore processes (minus the parent) are running under seccomp. The worker processes get the bare minimum allowed syscalls while each module like curl, pgsql, etc will add their own filters to allow what they require. New API functions: int kore_seccomp_filter(const char *name, void *filter, size_t len); Adds a filter into the seccomp system (must be called before seccomp is enabled). New helpful macro: define KORE_SYSCALL_ALLOW(name) Allow the syscall with a given name, should be used in a sock_filter data structure. New hooks: void kore_seccomp_hook(void); Called before seccomp is enabled, allows developers to add their own BPF filters into seccomp.
2019-09-25 14:25:49 +02:00
};
#endif
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
#define FD_CACHE_BUCKETS 2048
struct fd_cache {
struct kore_event evt;
int fd;
int scheduled;
LIST_ENTRY(fd_cache) list;
};
improve our asynchronous curl support. - Remove the edge trigger io hacks we had in place. - Use level triggered io for the libcurl fds instead. - Batch all curl events together and process them at the end of our worker event loop.
2020-08-17 15:15:04 +02:00
struct curl_run {
int eof;
struct fd_cache *fdc;
TAILQ_ENTRY(curl_run) list;
};
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
static void curl_process(void);
static void curl_event_handle(void *, int);
Improve curl timeout handling. In case libcurl instructs us to call the timeout function as soon as possible (timeout == 0 in curl_timeout), don't try to be clever with a timeout value of 10ms. Instead call the timeout function once we get back in the worker event loop. This makes things a lot snappier as we don't depend on epoll/kqueue waiting for io for 10ms (which actually isn't 10ms...).
2019-06-13 12:59:17 +02:00
static void curl_timeout(void *, u_int64_t);
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
static int curl_timer(CURLM *, long, void *);
improve our asynchronous curl support. - Remove the edge trigger io hacks we had in place. - Use level triggered io for the libcurl fds instead. - Batch all curl events together and process them at the end of our worker event loop.
2020-08-17 15:15:04 +02:00
static void curl_run_handle(struct curl_run *);
static void curl_run_schedule(struct fd_cache *, int);
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
static int curl_socket(CURL *, curl_socket_t, int, void *, void *);
static struct fd_cache *fd_cache_get(int);
improve our asynchronous curl support. - Remove the edge trigger io hacks we had in place. - Use level triggered io for the libcurl fds instead. - Batch all curl events together and process them at the end of our worker event loop.
2020-08-17 15:15:04 +02:00
static TAILQ_HEAD(, curl_run) runlist;
static struct kore_pool run_pool;
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
static int running = 0;
static CURLM *multi = NULL;
static struct kore_timer *timer = NULL;
static struct kore_pool fd_cache_pool;
set user-agent in the curl component.
2019-04-30 20:39:46 +02:00
static char user_agent[64];
Improve curl timeout handling. In case libcurl instructs us to call the timeout function as soon as possible (timeout == 0 in curl_timeout), don't try to be clever with a timeout value of 10ms. Instead call the timeout function once we get back in the worker event loop. This makes things a lot snappier as we don't depend on epoll/kqueue waiting for io for 10ms (which actually isn't 10ms...).
2019-06-13 12:59:17 +02:00
static int timeout_immediate = 0;
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
static LIST_HEAD(, fd_cache) cache[FD_CACHE_BUCKETS];
u_int16_t kore_curl_timeout = KORE_CURL_TIMEOUT;
u_int64_t kore_curl_recv_max = KORE_CURL_RECV_MAX;
void
kore_curl_sysinit(void)
{
CURLMcode res;
set user-agent in the curl component.
2019-04-30 20:39:46 +02:00
int i, len;
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
if (curl_global_init(CURL_GLOBAL_ALL))
fatal("failed to initialize libcurl");
if ((multi = curl_multi_init()) == NULL)
fatal("curl_multi_init(): failed");
/* XXX - make configurable? */
curl_multi_setopt(multi, CURLMOPT_MAXCONNECTS, 500);
if ((res = curl_multi_setopt(multi,
CURLMOPT_SOCKETFUNCTION, curl_socket)) != CURLM_OK)
fatal("curl_multi_setopt: %s", curl_multi_strerror(res));
if ((res = curl_multi_setopt(multi,
CURLMOPT_TIMERFUNCTION, curl_timer)) != CURLM_OK)
fatal("curl_multi_setopt: %s", curl_multi_strerror(res));
for (i = 0; i < FD_CACHE_BUCKETS; i++)
LIST_INIT(&cache[i]);
improve our asynchronous curl support. - Remove the edge trigger io hacks we had in place. - Use level triggered io for the libcurl fds instead. - Batch all curl events together and process them at the end of our worker event loop.
2020-08-17 15:15:04 +02:00
TAILQ_INIT(&runlist);
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
kore_pool_init(&fd_cache_pool, "fd_cache_pool", 100,
sizeof(struct fd_cache));
improve our asynchronous curl support. - Remove the edge trigger io hacks we had in place. - Use level triggered io for the libcurl fds instead. - Batch all curl events together and process them at the end of our worker event loop.
2020-08-17 15:15:04 +02:00
kore_pool_init(&run_pool, "run_pool", 100, sizeof(struct curl_run));
set user-agent in the curl component.
2019-04-30 20:39:46 +02:00
use / to split up name/version.
2019-04-30 20:45:56 +02:00
len = snprintf(user_agent, sizeof(user_agent), "kore/%s", kore_version);
set user-agent in the curl component.
2019-04-30 20:39:46 +02:00
if (len == -1 || (size_t)len >= sizeof(user_agent))
fatal("user-agent string too long");
Add seccomp syscall filtering to kore. With this commit all Kore processes (minus the parent) are running under seccomp. The worker processes get the bare minimum allowed syscalls while each module like curl, pgsql, etc will add their own filters to allow what they require. New API functions: int kore_seccomp_filter(const char *name, void *filter, size_t len); Adds a filter into the seccomp system (must be called before seccomp is enabled). New helpful macro: define KORE_SYSCALL_ALLOW(name) Allow the syscall with a given name, should be used in a sock_filter data structure. New hooks: void kore_seccomp_hook(void); Called before seccomp is enabled, allows developers to add their own BPF filters into seccomp.
2019-09-25 14:25:49 +02:00
#if defined(__linux__)
kore_seccomp_filter("curl", filter_curl, KORE_FILTER_LEN(filter_curl));
#endif
Add dns to pledges when initializing curl Without the dns pledge, curl can't resolve names on OpenBSD.
2019-11-03 11:47:49 +01:00
#if defined(KORE_USE_PLATFORM_PLEDGE)
kore_platform_add_pledge("dns");
#endif
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
}
int
Allow kore_curl calls to be synchronous. Changes kore_curl_init() to take a flag parameter, much like pgsql api in which you specify KORE_CURL_ASYNC or KORE_CURL_SYNC. If KORE_CURL_ASYNC is specified, Kore will behave as before. If Kore_CURL_SYNC is specified, Kore will execute the libcurl immediately and return once it has been completed.
2019-10-21 13:26:44 +02:00
kore_curl_init(struct kore_curl *client, const char *url, int flags)
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
{
CURL *handle;
Allow kore_curl calls to be synchronous. Changes kore_curl_init() to take a flag parameter, much like pgsql api in which you specify KORE_CURL_ASYNC or KORE_CURL_SYNC. If KORE_CURL_ASYNC is specified, Kore will behave as before. If Kore_CURL_SYNC is specified, Kore will execute the libcurl immediately and return once it has been completed.
2019-10-21 13:26:44 +02:00
if ((flags & KORE_CURL_ASYNC) && (flags & KORE_CURL_SYNC)) {
(void)kore_strlcpy(client->errbuf, "invalid flags",
sizeof(client->errbuf));
return (KORE_RESULT_ERROR);
}
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
memset(client, 0, sizeof(*client));
TAILQ_INIT(&client->http.resp_hdrs);
Allow kore_curl calls to be synchronous. Changes kore_curl_init() to take a flag parameter, much like pgsql api in which you specify KORE_CURL_ASYNC or KORE_CURL_SYNC. If KORE_CURL_ASYNC is specified, Kore will behave as before. If Kore_CURL_SYNC is specified, Kore will execute the libcurl immediately and return once it has been completed.
2019-10-21 13:26:44 +02:00
if ((handle = curl_easy_init()) == NULL) {
(void)kore_strlcpy(client->errbuf, "failed to setup curl",
sizeof(client->errbuf));
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
return (KORE_RESULT_ERROR);
Allow kore_curl calls to be synchronous. Changes kore_curl_init() to take a flag parameter, much like pgsql api in which you specify KORE_CURL_ASYNC or KORE_CURL_SYNC. If KORE_CURL_ASYNC is specified, Kore will behave as before. If Kore_CURL_SYNC is specified, Kore will execute the libcurl immediately and return once it has been completed.
2019-10-21 13:26:44 +02:00
}
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
curl_easy_setopt(handle, CURLOPT_WRITEDATA, &client->response);
curl_easy_setopt(handle, CURLOPT_WRITEFUNCTION, kore_curl_tobuf);
curl_easy_setopt(handle, CURLOPT_URL, url);
curl_easy_setopt(handle, CURLOPT_NOSIGNAL, 1);
curl_easy_setopt(handle, CURLOPT_PRIVATE, client);
set user-agent in the curl component.
2019-04-30 20:39:46 +02:00
curl_easy_setopt(handle, CURLOPT_USERAGENT, user_agent);
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
curl_easy_setopt(handle, CURLOPT_TIMEOUT, kore_curl_timeout);
curl_easy_setopt(handle, CURLOPT_ERRORBUFFER, client->errbuf);
Allow kore_curl calls to be synchronous. Changes kore_curl_init() to take a flag parameter, much like pgsql api in which you specify KORE_CURL_ASYNC or KORE_CURL_SYNC. If KORE_CURL_ASYNC is specified, Kore will behave as before. If Kore_CURL_SYNC is specified, Kore will execute the libcurl immediately and return once it has been completed.
2019-10-21 13:26:44 +02:00
client->flags = flags;
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
client->handle = handle;
client->url = kore_strdup(url);
client->type = KORE_CURL_TYPE_CUSTOM;
return (KORE_RESULT_OK);
}
void
kore_curl_cleanup(struct kore_curl *client)
{
struct http_header *hdr, *next;
kore_free(client->url);
if (client->flags & KORE_CURL_FLAG_BOUND)
LIST_REMOVE(client, list);
if (client->handle != NULL) {
curl_multi_remove_handle(multi, client->handle);
curl_easy_cleanup(client->handle);
}
if (client->http.hdrlist != NULL)
curl_slist_free_all(client->http.hdrlist);
if (client->response != NULL)
kore_buf_free(client->response);
if (client->http.headers != NULL)
kore_buf_free(client->http.headers);
if (client->http.tosend != NULL)
kore_buf_free(client->http.tosend);
for (hdr = TAILQ_FIRST(&client->http.resp_hdrs);
hdr != NULL; hdr = next) {
next = TAILQ_NEXT(hdr, list);
TAILQ_REMOVE(&client->http.resp_hdrs, hdr, list);
kore_pool_put(&http_header_pool, hdr);
}
}
Improve curl timeout handling. In case libcurl instructs us to call the timeout function as soon as possible (timeout == 0 in curl_timeout), don't try to be clever with a timeout value of 10ms. Instead call the timeout function once we get back in the worker event loop. This makes things a lot snappier as we don't depend on epoll/kqueue waiting for io for 10ms (which actually isn't 10ms...).
2019-06-13 12:59:17 +02:00
void
kore_curl_do_timeout(void)
{
reset curl timer if nothing is running anymore.
2019-10-23 12:38:31 +02:00
while (timeout_immediate) {
Improve curl timeout handling. In case libcurl instructs us to call the timeout function as soon as possible (timeout == 0 in curl_timeout), don't try to be clever with a timeout value of 10ms. Instead call the timeout function once we get back in the worker event loop. This makes things a lot snappier as we don't depend on epoll/kqueue waiting for io for 10ms (which actually isn't 10ms...).
2019-06-13 12:59:17 +02:00
curl_timeout(NULL, kore_time_ms());
reset curl timer if nothing is running anymore.
2019-10-23 12:38:31 +02:00
if (running == 0)
curl_timer(multi, -1, NULL);
}
Improve curl timeout handling. In case libcurl instructs us to call the timeout function as soon as possible (timeout == 0 in curl_timeout), don't try to be clever with a timeout value of 10ms. Instead call the timeout function once we get back in the worker event loop. This makes things a lot snappier as we don't depend on epoll/kqueue waiting for io for 10ms (which actually isn't 10ms...).
2019-06-13 12:59:17 +02:00
}
improve our asynchronous curl support. - Remove the edge trigger io hacks we had in place. - Use level triggered io for the libcurl fds instead. - Batch all curl events together and process them at the end of our worker event loop.
2020-08-17 15:15:04 +02:00
void
kore_curl_run_scheduled(void)
{
struct curl_run *run;
while ((run = TAILQ_FIRST(&runlist))) {
TAILQ_REMOVE(&runlist, run, list);
curl_run_handle(run);
kore_pool_put(&run_pool, run);
}
curl_process();
}
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
size_t
kore_curl_tobuf(char *ptr, size_t size, size_t nmemb, void *udata)
{
size_t len;
struct kore_buf **buf, *b;
if (SIZE_MAX / nmemb < size)
fatal("%s: %zu * %zu overflow", __func__, nmemb, size);
buf = udata;
len = size * nmemb;
if (*buf == NULL)
*buf = kore_buf_alloc(len);
b = *buf;
if (b->offset + len < b->offset)
fatal("%s: %zu+%zu overflows", __func__, b->offset, len);
if ((b->offset + len) > kore_curl_recv_max) {
kore_log(LOG_ERR,
"received too large transfer (%zu > %" PRIu64 ")",
b->offset + len, kore_curl_recv_max);
return (0);
}
kore_buf_append(b, ptr, len);
return (len);
}
size_t
kore_curl_frombuf(char *ptr, size_t size, size_t nmemb, void *udata)
{
size_t len;
struct kore_buf *buf;
if (SIZE_MAX / nmemb < size)
fatal("%s: %zu * %zu overflow", __func__, nmemb, size);
buf = udata;
len = size * nmemb;
if (buf->offset == buf->length)
return (0);
if (buf->offset + len < buf->offset)
fatal("%s: %zu+%zu overflows", __func__, buf->offset, len);
if ((buf->offset + len) < buf->length) {
memcpy(ptr, buf->data + buf->offset, len);
} else {
len = buf->length - buf->offset;
memcpy(ptr, buf->data + buf->offset, len);
}
buf->offset += len;
return (len);
}
void
kore_curl_bind_request(struct kore_curl *client, struct http_request *req)
{
if (client->cb != NULL)
fatal("%s: already bound to callback", __func__);
client->req = req;
http_request_sleep(req);
client->flags |= KORE_CURL_FLAG_BOUND;
LIST_INSERT_HEAD(&req->chandles, client, list);
}
void
kore_curl_bind_callback(struct kore_curl *client,
void (*cb)(struct kore_curl *, void *), void *arg)
{
if (client->req != NULL)
fatal("%s: already bound to request", __func__);
client->cb = cb;
client->arg = arg;
}
void
kore_curl_run(struct kore_curl *client)
{
Allow kore_curl calls to be synchronous. Changes kore_curl_init() to take a flag parameter, much like pgsql api in which you specify KORE_CURL_ASYNC or KORE_CURL_SYNC. If KORE_CURL_ASYNC is specified, Kore will behave as before. If Kore_CURL_SYNC is specified, Kore will execute the libcurl immediately and return once it has been completed.
2019-10-21 13:26:44 +02:00
if (client->flags & KORE_CURL_ASYNC) {
curl_multi_add_handle(multi, client->handle);
return;
}
client->result = curl_easy_perform(client->handle);
curl_easy_getinfo(client->handle,
CURLINFO_RESPONSE_CODE, &client->http.status);
curl_easy_cleanup(client->handle);
client->handle = NULL;
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
}
int
kore_curl_success(struct kore_curl *client)
{
return (client->result == CURLE_OK);
}
Improve curl error string handling. Introduce kore_curl_strerror(), use this in kore_curl_logerror() instead of assuming our errbuf has been populated. Also use it in the python httpclient when throwing an exception rather then looking at the errbuf member which may or may not be empty.
2019-05-30 14:25:04 +02:00
const char *
kore_curl_strerror(struct kore_curl *client)
{
const char *err;
if (client->errbuf[0] != '\0')
err = &client->errbuf[0];
else
err = curl_easy_strerror(client->result);
return (err);
}
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
void
kore_curl_logerror(struct kore_curl *client)
{
Improve curl error string handling. Introduce kore_curl_strerror(), use this in kore_curl_logerror() instead of assuming our errbuf has been populated. Also use it in the python httpclient when throwing an exception rather then looking at the errbuf member which may or may not be empty.
2019-05-30 14:25:04 +02:00
kore_log(LOG_NOTICE, "curl error: %s -> %s", client->url,
kore_curl_strerror(client));
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
}
void
kore_curl_response_as_bytes(struct kore_curl *client, const u_int8_t **body,
size_t *len)
{
if (client->response == NULL) {
*len = 0;
*body = NULL;
} else {
*len = client->response->offset;
*body = client->response->data;
}
}
char *
kore_curl_response_as_string(struct kore_curl *client)
{
kore_buf_stringify(client->response, NULL);
return ((char *)client->response->data);
}
void
kore_curl_http_setup(struct kore_curl *client, int method, const void *data,
size_t len)
{
const char *mname;
int has_body;
if (client->handle == NULL)
fatal("%s: called without setup", __func__);
mname = NULL;
has_body = 1;
client->type = KORE_CURL_TYPE_HTTP_CLIENT;
curl_easy_setopt(client->handle, CURLOPT_HEADERDATA,
&client->http.headers);
curl_easy_setopt(client->handle, CURLOPT_HEADERFUNCTION,
kore_curl_tobuf);
kore_curl_http_set_header(client, "expect", "");
switch (method) {
case HTTP_METHOD_GET:
Handle DELETE maybe having a request body.
2021-12-06 14:44:07 +01:00
case HTTP_METHOD_OPTIONS:
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
break;
case HTTP_METHOD_HEAD:
curl_easy_setopt(client->handle, CURLOPT_NOBODY, 1);
break;
case HTTP_METHOD_PUT:
has_body = 1;
curl_easy_setopt(client->handle, CURLOPT_UPLOAD, 1);
break;
case HTTP_METHOD_PATCH:
Handle DELETE maybe having a request body.
2021-12-06 14:44:07 +01:00
case HTTP_METHOD_DELETE:
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
mname = http_method_text(method);
/* fallthrough */
case HTTP_METHOD_POST:
has_body = 1;
curl_easy_setopt(client->handle, CURLOPT_POST, 1);
break;
default:
fatal("%s: unknown method %d", __func__, method);
}
if (has_body && data != NULL && len > 0) {
client->http.tosend = kore_buf_alloc(len);
kore_buf_append(client->http.tosend, data, len);
kore_buf_reset(client->http.tosend);
curl_easy_setopt(client->handle, CURLOPT_READDATA,
client->http.tosend);
curl_easy_setopt(client->handle, CURLOPT_READFUNCTION,
kore_curl_frombuf);
}
if (has_body) {
if (method == HTTP_METHOD_PUT) {
curl_easy_setopt(client->handle,
tiny curl fixes.
2019-05-05 21:16:42 +02:00
CURLOPT_INFILESIZE_LARGE, (curl_off_t)len);
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
} else {
curl_easy_setopt(client->handle,
tiny curl fixes.
2019-05-05 21:16:42 +02:00
CURLOPT_POSTFIELDSIZE_LARGE, (curl_off_t)len);
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
}
} else {
if (data != NULL || len != 0) {
fatal("%s: %d should not have a body",
__func__, method);
}
}
if (mname != NULL)
curl_easy_setopt(client->handle, CURLOPT_CUSTOMREQUEST, mname);
}
void
kore_curl_http_set_header(struct kore_curl *client, const char *header,
const char *value)
{
struct kore_buf buf;
const char *hdr;
kore_buf_init(&buf, 512);
if (value != NULL || *value != '\0') {
kore_buf_appendf(&buf, "%s: %s", header, value);
} else {
kore_buf_appendf(&buf, "%s:", header);
}
hdr = kore_buf_stringify(&buf, NULL);
client->http.hdrlist = curl_slist_append(client->http.hdrlist, hdr);
kore_buf_cleanup(&buf);
curl_easy_setopt(client->handle,
CURLOPT_HTTPHEADER, client->http.hdrlist);
}
int
kore_curl_http_get_header(struct kore_curl *client, const char *header,
const char **out)
{
struct http_header *hdr;
if (!(client->flags & KORE_CURL_FLAG_HTTP_PARSED_HEADERS))
kore_curl_http_parse_headers(client);
TAILQ_FOREACH(hdr, &(client->http.resp_hdrs), list) {
if (!strcasecmp(hdr->header, header)) {
*out = hdr->value;
return (KORE_RESULT_OK);
}
}
return (KORE_RESULT_ERROR);
}
void
kore_curl_http_parse_headers(struct kore_curl *client)
{
struct http_header *hdr;
int i, cnt;
char *value, *hbuf, *headers[HTTP_REQ_HEADER_MAX];
if (client->flags & KORE_CURL_FLAG_HTTP_PARSED_HEADERS)
fatal("%s: headers already parsed", __func__);
client->flags |= KORE_CURL_FLAG_HTTP_PARSED_HEADERS;
if (client->http.headers == NULL)
return;
hbuf = kore_buf_stringify(client->http.headers, NULL);
cnt = kore_split_string(hbuf, "\r\n", headers, HTTP_REQ_HEADER_MAX);
for (i = 0; i < cnt; i++) {
if ((value = http_validate_header(headers[i])) == NULL)
continue;
if (*value == '\0')
continue;
hdr = kore_pool_get(&http_header_pool);
hdr->header = headers[i];
hdr->value = value;
TAILQ_INSERT_TAIL(&(client->http.resp_hdrs), hdr, list);
}
}
static int
curl_socket(CURL *easy, curl_socket_t fd, int action, void *arg, void *sock)
{
CURLcode res;
struct fd_cache *fdc;
struct kore_curl *client;
client = NULL;
tiny curl fixes.
2019-05-05 21:16:42 +02:00
res = curl_easy_getinfo(easy, CURLINFO_PRIVATE, (char **)&client);
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
if (res != CURLE_OK)
fatal("curl_easy_getinfo: %s", curl_easy_strerror(res));
if (client == NULL)
fatal("%s: failed to get client context", __func__);
fdc = fd_cache_get(fd);
switch (action) {
case CURL_POLL_NONE:
break;
case CURL_POLL_IN:
improve our asynchronous curl support. - Remove the edge trigger io hacks we had in place. - Use level triggered io for the libcurl fds instead. - Batch all curl events together and process them at the end of our worker event loop.
2020-08-17 15:15:04 +02:00
if (fdc->scheduled) {
kore_platform_disable_read(fd);
#if !defined(__linux__)
kore_platform_disable_write(fd);
#endif
}
fdc->scheduled = 1;
kore_platform_event_level_read(fd, fdc);
break;
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
case CURL_POLL_OUT:
case CURL_POLL_INOUT:
Make sure curl state transitions work properly.
2020-08-17 16:45:10 +02:00
if (fdc->scheduled) {
kore_platform_disable_read(fd);
#if !defined(__linux__)
kore_platform_disable_write(fd);
#endif
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
}
Make sure curl state transitions work properly.
2020-08-17 16:45:10 +02:00
fdc->scheduled = 1;
kore_platform_event_level_all(fd, fdc);
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
break;
case CURL_POLL_REMOVE:
if (fdc->scheduled) {
fdc->evt.flags = 0;
fdc->scheduled = 0;
kore_platform_disable_read(fd);
#if !defined(__linux__)
kore_platform_disable_write(fd);
#endif
}
break;
default:
fatal("unknown action value: %d", action);
}
improve our asynchronous curl support. - Remove the edge trigger io hacks we had in place. - Use level triggered io for the libcurl fds instead. - Batch all curl events together and process them at the end of our worker event loop.
2020-08-17 15:15:04 +02:00
if (action != CURL_POLL_NONE && action != CURL_POLL_REMOVE)
curl_run_schedule(fdc, 0);
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
return (CURLM_OK);
}
static void
curl_process(void)
{
CURLcode res;
CURLMsg *msg;
CURL *handle;
struct kore_curl *client;
int pending;
pending = 0;
while ((msg = curl_multi_info_read(multi, &pending)) != NULL) {
if (msg->msg != CURLMSG_DONE)
continue;
handle = msg->easy_handle;
tiny curl fixes.
2019-05-05 21:16:42 +02:00
res = curl_easy_getinfo(handle, CURLINFO_PRIVATE,
(char **)&client);
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
if (res != CURLE_OK)
fatal("curl_easy_getinfo: %s", curl_easy_strerror(res));
if (client == NULL)
fatal("%s: failed to get client context", __func__);
client->result = msg->data.result;
if (client->type == KORE_CURL_TYPE_HTTP_CLIENT) {
curl_easy_getinfo(client->handle,
CURLINFO_RESPONSE_CODE, &client->http.status);
}
curl_multi_remove_handle(multi, client->handle);
curl_easy_cleanup(client->handle);
client->handle = NULL;
if (client->req != NULL)
http_request_wakeup(client->req);
else if (client->cb != NULL)
client->cb(client, client->arg);
}
}
static void
curl_timeout(void *uarg, u_int64_t now)
{
CURLMcode res;
timer = NULL;
res = curl_multi_socket_action(multi, CURL_SOCKET_TIMEOUT, 0, &running);
if (res != CURLM_OK)
fatal("curl_multi_socket_action: %s", curl_multi_strerror(res));
curl_process();
}
static int
curl_timer(CURLM *mctx, long timeout, void *arg)
{
Improve curl timeout handling. In case libcurl instructs us to call the timeout function as soon as possible (timeout == 0 in curl_timeout), don't try to be clever with a timeout value of 10ms. Instead call the timeout function once we get back in the worker event loop. This makes things a lot snappier as we don't depend on epoll/kqueue waiting for io for 10ms (which actually isn't 10ms...).
2019-06-13 12:59:17 +02:00
timeout_immediate = 0;
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
if (timeout < 0) {
if (timer != NULL) {
kore_timer_remove(timer);
timer = NULL;
}
return (CURLM_OK);
}
if (timer != NULL) {
kore_timer_remove(timer);
timer = NULL;
}
Improve curl timeout handling. In case libcurl instructs us to call the timeout function as soon as possible (timeout == 0 in curl_timeout), don't try to be clever with a timeout value of 10ms. Instead call the timeout function once we get back in the worker event loop. This makes things a lot snappier as we don't depend on epoll/kqueue waiting for io for 10ms (which actually isn't 10ms...).
2019-06-13 12:59:17 +02:00
if (timeout == 0) {
timeout_immediate = 1;
return (CURLM_OK);
}
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
timer = kore_timer_add(curl_timeout, timeout, mctx, KORE_TIMER_ONESHOT);
return (CURLM_OK);
}
improve our asynchronous curl support. - Remove the edge trigger io hacks we had in place. - Use level triggered io for the libcurl fds instead. - Batch all curl events together and process them at the end of our worker event loop.
2020-08-17 15:15:04 +02:00
static void
curl_run_schedule(struct fd_cache *fdc, int eof)
{
struct curl_run *run;
run = kore_pool_get(&run_pool);
run->fdc = fdc;
run->eof = eof;
TAILQ_INSERT_TAIL(&runlist, run, list);
}
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
static void
curl_event_handle(void *arg, int eof)
improve our asynchronous curl support. - Remove the edge trigger io hacks we had in place. - Use level triggered io for the libcurl fds instead. - Batch all curl events together and process them at the end of our worker event loop.
2020-08-17 15:15:04 +02:00
{
curl_run_schedule(arg, eof);
}
static void
curl_run_handle(struct curl_run *run)
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
{
missing return value check.
2019-04-24 09:33:56 +02:00
CURLMcode res;
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
int flags;
improve our asynchronous curl support. - Remove the edge trigger io hacks we had in place. - Use level triggered io for the libcurl fds instead. - Batch all curl events together and process them at the end of our worker event loop.
2020-08-17 15:15:04 +02:00
struct fd_cache *fdc = run->fdc;
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
flags = 0;
if (fdc->evt.flags & KORE_EVENT_READ)
flags |= CURL_CSELECT_IN;
if (fdc->evt.flags & KORE_EVENT_WRITE)
flags |= CURL_CSELECT_OUT;
improve our asynchronous curl support. - Remove the edge trigger io hacks we had in place. - Use level triggered io for the libcurl fds instead. - Batch all curl events together and process them at the end of our worker event loop.
2020-08-17 15:15:04 +02:00
if (run->eof)
make sure CURL_CSELECT_ERR is OR'd into flags.
2019-04-28 21:44:59 +02:00
flags |= CURL_CSELECT_ERR;
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
missing return value check.
2019-04-24 09:33:56 +02:00
res = curl_multi_socket_action(multi, fdc->fd, flags, &running);
if (res != CURLM_OK)
fatal("curl_multi_socket_action: %s", curl_multi_strerror(res));
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
}
static struct fd_cache *
fd_cache_get(int fd)
{
struct fd_cache *fdc;
int bucket;
bucket = fd % FD_CACHE_BUCKETS;
LIST_FOREACH(fdc, &cache[bucket], list) {
if (fdc->fd == fd)
return (fdc);
}
fdc = kore_pool_get(&fd_cache_pool);
fdc->fd = fd;
fdc->scheduled = 0;
fdc->evt.flags = 0;
fdc->evt.handle = curl_event_handle;
fdc->evt.type = KORE_TYPE_CURL_HANDLE;
LIST_INSERT_HEAD(&cache[bucket], fdc, list);
return (fdc);
}