mirrors
/
kore
mirror of https://git.kore.io/kore.git
1
0
Fork 1
Code Issues Projects Releases Wiki Activity
kore/src/http.c

2611 lines
58 KiB
C
Raw Normal View History

properly deflate/inflate of name/value blocks. and add ability to get headers from them.
2013-05-01 00:35:33 +02:00
/*
i forgot, it's 2022.
2022-01-31 22:02:06 +01:00
* Copyright (c) 2013-2022 Joris Vink <joris@coders.se>
properly deflate/inflate of name/value blocks. and add ability to get headers from them.
2013-05-01 00:35:33 +02:00
*
* Permission to use, copy, modify, and distribute this software for any
* purpose with or without fee is hereby granted, provided that the above
* copyright notice and this permission notice appear in all copies.
*
* THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
* WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
* ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
* WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
* ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
* OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
#include <sys/param.h>
explicitly include sys/types.h some smaller libc variants do not include this from sys/param.h.
2019-03-06 09:29:46 +01:00
#include <sys/types.h>
properly deflate/inflate of name/value blocks. and add ability to get headers from them.
2013-05-01 00:35:33 +02:00
shuffle headers around.
2017-02-07 23:21:18 +01:00
#include <sys/socket.h>
#include <netinet/in.h>
Add http_argument_urldecode() which takes a string and decodes any url encoding done to it. Change kore_strtonum() to pass 0 to strtoll by default so we can use it to convert hex numbers (prepended with 0x) as well.
2013-08-13 14:18:47 +02:00
#include <ctype.h>
shuffle headers around.
2017-02-07 23:21:18 +01:00
#include <fcntl.h>
Add http_keepalive_time configuration parameter. Allows you to configure maximum amount of seconds an HTTP connection can stay open (does not affect SPDY connections). If set to 0 it will disable keep-alive all together. Add some inttypes fluff.
2013-10-15 11:10:45 +02:00
#include <inttypes.h>
Add kore_strtodouble(). Use it for http_argument_get_float() and http_argument_get_double().
2018-05-04 15:55:35 +02:00
#include <float.h>
shuffle headers around.
2017-02-07 23:21:18 +01:00
#include <time.h>
Add missing C library headers
2016-01-22 12:08:13 +01:00
#include <stdio.h>
#include <string.h>
Add http_argument_urldecode() which takes a string and decodes any url encoding done to it. Change kore_strtonum() to pass 0 to strtoll by default so we can use it to convert hex numbers (prepended with 0x) as well.
2013-08-13 14:18:47 +02:00
properly deflate/inflate of name/value blocks. and add ability to get headers from them.
2013-05-01 00:35:33 +02:00
#include "kore.h"
#include "http.h"
Split up kore cli tools into new binary. Having the create, build, run tools baked into the kore binary made things harder then they had to be for multiple projects with each different build flavors. So move away this functionality into a new "kodev" (name may change) binary that is installed next to kore. The new build tools will automatically pick up the correct flavors the kore binary it points to is installed with. Or for single builds what flavors where enabled. The new tool also will honor looking into PREFIX for the kore binary when doing a `kodev run`. Additionally add a new command "info" that shows some basic info about your project and how it will be built. For example it will show you the flavors of the kore binary installed on the system or the flavors you configured for a single binary build. Obligitory, hacking on a plane comment.
2017-02-19 06:52:29 +01:00
#if defined(KORE_USE_PYTHON)
#include "python_api.h"
#endif
Attach tasks/pgsqls to http_requests once more. This way if an http request is removed while tasks or pgsqls are still active they are free'd out and cancelled properly.
2014-08-14 22:05:34 +02:00
#if defined(KORE_USE_PGSQL)
#include "pgsql.h"
#endif
Add initial stab at asynchronous background tasks. More to follow.
2014-06-28 16:17:18 +02:00
#if defined(KORE_USE_TASKS)
Normalize kore_*.h headers
2014-07-03 22:14:46 +02:00
#include "tasks.h"
Add initial stab at asynchronous background tasks. More to follow.
2014-06-28 16:17:18 +02:00
#endif
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
#if defined(KORE_USE_CURL)
#include "curl.h"
#endif
Add filemaps. A filemap is a way of telling Kore to serve files from a directory much like a traditional webserver can do. Kore filemaps only handles files. Kore does not generate directory indexes or deal with non-regular files. The way files are sent to a client differs a bit per platform and build options: default: - mmap() backed file transfer due to TLS. NOTLS=1 - sendfile() under FreeBSD, macOS and Linux. - mmap() backed file for OpenBSD. The opened file descriptors/mmap'd regions are cached and reused when appropriate. If a file is no longer in use it will be closed and evicted from the cache after 30 seconds. New API's are available allowing developers to use these facilities via: void net_send_fileref(struct connection *, struct kore_fileref *); void http_response_fileref(struct http_request *, struct kore_fileref *); Kore will attempt to match media types based on file extensions. A few default types are built-in. Others can be added via the new "http_media_type" configuration directive.
2018-06-28 13:27:44 +02:00
static struct {
const char *ext;
const char *type;
} builtin_media[] = {
{ "gif", "image/gif" },
{ "png", "image/png" },
{ "jpeg", "image/jpeg" },
{ "jpg", "image/jpeg" },
{ "zip", "application/zip" },
{ "pdf", "application/pdf" },
{ "json", "application/json" },
{ "js", "application/javascript" },
{ "htm", "text/html" },
{ "txt", "text/plain" },
{ "css", "text/css" },
{ "html", "text/html" },
{ NULL, NULL },
};
better http header validation.
2018-07-17 20:17:05 +02:00
#define HTTP_MAP_LIMIT 127
/*
* token = 1*<any CHAR except CTLs or separators>
* separators = "(" | ")" | "<" | ">" | "@"
* | "," | ";" | ":" | "\" | <">
* | "/" | "[" | "]" | "?" | "="
* | "{" | "}" | SP | HT
*/
static const char http_token[] = {
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, '!' , 0x00, '#' , '$' , '%' , '&' , '\'',
0x00, 0x00, '*' , '+' , 0x00, '-' , '.' , 0x00,
'0' , '1' , '2' , '3' , '4' , '5' , '6' , '7' ,
'8' , '9' , 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 'A' , 'B' , 'C' , 'D' , 'E' , 'F' , 'G' ,
'H' , 'I' , 'J' , 'K' , 'L' , 'M' , 'N' , 'O' ,
'P' , 'Q' , 'R' , 'S' , 'T' , 'U' , 'V' , 'W' ,
'X' , 'Y' , 'Z' , 0x00, 0x00, 0x00, '^' , '_' ,
'`' , 'a' , 'b' , 'c' , 'd' , 'e' , 'f' , 'g' ,
'h' , 'i' , 'j' , 'k' , 'l' , 'm' , 'n' , 'o' ,
'p' , 'q' , 'r' , 's' , 't' , 'u' , 'v' , 'w' ,
Add trailing byte in http_token and http_field_content.
2020-02-19 10:38:41 +01:00
'x' , 'y' , 'z' , 0x00, '|' , 0x00, '~' , 0x00
better http header validation.
2018-07-17 20:17:05 +02:00
};
/*
* field-content = <the OCTETs making up the field-value
* and consisting of either *TEXT or combinations
* of token, separators, and quoted-string>
*/
static const char http_field_content[] = {
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
' ' , '!' , '"' , '#' , '$' , '%' , '&' , '\'',
'(' , ')' , '*' , '+' , ',' , '-' , '.' , '/' ,
'0' , '1' , '2' , '3' , '4' , '5' , '6' , '7' ,
'8' , '9' , ':' , ';' , '<' , '=' , '>' , '?' ,
'@' , 'A' , 'B' , 'C' , 'D' , 'E' , 'F' , 'G' ,
'H' , 'I' , 'J' , 'K' , 'L' , 'M' , 'N' , 'O' ,
'P' , 'Q' , 'R' , 'S' , 'T' , 'U' , 'V' , 'W' ,
'X' , 'Y' , 'Z' , '[' , '\\', ']' , '^' , '_' ,
'`' , 'a' , 'b' , 'c' , 'd' , 'e' , 'f' , 'g' ,
'h' , 'i' , 'j' , 'k' , 'l' , 'm' , 'n' , 'o' ,
'p' , 'q' , 'r' , 's' , 't' , 'u' , 'v' , 'w' ,
Add trailing byte in http_token and http_field_content.
2020-02-19 10:38:41 +01:00
'x' , 'y' , 'z' , '{' , '|' , '}' , '~' , 0x00
better http header validation.
2018-07-17 20:17:05 +02:00
};
Rework http_pretty_error a tiny bit.
2020-03-04 08:36:10 +01:00
/*
* Fixed "pretty" HTTP error HTML page.
*/
static const char *pretty_error_fmt =
"<html>\n<head>\n\t<title>%d %s</title>"
"</head>\n<body>\n\t"
"<h1>%d %s</h1>\n"
"</body>\n</html>\n";
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
static int http_body_recv(struct netbuf *);
Add http_response_json(). Since its an HTTP response function it functions like http_response() but takes a kore_json_item pointer that it will automatically convert to a kore_buf and send/free using http_response_stream(). While here fix a problem with http_response_stream() which could end up not calling the cb() in case of HTTP_METHOD_HEAD. Since the behaviour is that it should call cb() when done it should do so immediately.
2021-09-12 15:10:06 +02:00
static int http_release_buffer(struct netbuf *);
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
static void http_error_response(struct connection *, int);
HTTP improvements. Introduce an on_headers callback for routes, allowing one to inspect the headers before the request is processed further. Additionall, Add a new way of obtaining HTTP headers. Much like http_argument_get_*() functions, these new APIs allow you to fetch the data of an HTTP header as a specified C type. The new APIs are: * http_request_header_int16() * http_request_header_uint16() * http_request_header_int32() * http_request_header_uint32() * http_request_header_int64() * http_request_header_uint64() * http_request_header_float() * http_request_header_double() Should make it easier to operate in HTTP header data in a safe way. No need to always roll your own string to int conversion functions.
2021-09-15 22:16:22 +02:00
static int http_data_convert(void *, void **, void *, int);
cookie improvements. - split up writing of cookies into its own function. - turn maxage into a signed int and use -1 for it not being set. - lots of style fixes - remove HTTP_COOKIE_DEFAULT, just pass 0 if you don't want flags.
2017-02-07 22:03:06 +01:00
static void http_write_response_cookie(struct http_cookie *);
Add the on_body_chunk handler for routes. If set, will call a given handler with the prototype of `void body_chunk(struct http_request *req, const void *data, size_t len);` for each chunk of the received HTTP body, allowing a developer to handle it in their own way. The incoming body is still being handled and retained in the same way as before (in a kore_buf or temporary file). While here, allow HTTP_STATUS_CONTINUE to work via http_response() and make the handling of incoming HTTP header data a bit better.
2021-09-17 19:28:06 +02:00
static int http_body_update(struct http_request *, const void *, size_t);
do not http_argument_urldecode for multipart data.
2018-08-16 14:11:28 +02:00
static void http_argument_add(struct http_request *, char *, char *,
int, int);
Add support for config based redirection. Inside the domain contexts a 'redirect' rule will allow you to redirect a request to another URI. Ex: Redirect all requests with a 301 to example.com redirect ^/.*$ 301 https://example.com Using capture groups redirect ^/account/(.*)$ 301 https://example.com/account/$1 Adding the query string in the mix redirect ^/(.*)$ 301 https://example.com/$1?$qs
2020-02-07 06:42:33 +01:00
static int http_check_redirect(struct http_request *,
struct kore_domain *);
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
static void http_response_normal(struct http_request *,
Code cleanup, several API breaking changes in here
2016-07-08 10:02:18 +02:00
struct connection *, int, const void *, size_t);
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
static void multipart_add_field(struct http_request *, struct kore_buf *,
actually urldecode parameter names.
2017-02-06 22:47:32 +01:00
char *, const char *, const int);
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
static void multipart_file_add(struct http_request *, struct kore_buf *,
const char *, const char *, const char *, const int);
static int multipart_find_data(struct kore_buf *, struct kore_buf *,
size_t *, struct http_request *, const void *, size_t);
static int multipart_parse_headers(struct http_request *,
struct kore_buf *, struct kore_buf *,
const char *, const int);
lots of new stuff, including processing of http requests and an attempt to build an initial spdy response (SYN frame + header block content).
2013-05-01 08:09:04 +02:00
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
static struct http_request *http_request_new(struct connection *,
const char *, const char *, char *,
const char *);
Use a single header_buf in http_response_normal() Avoids constantly allocating and freeing buffers all the time. This can be done safely due to the fact workers are single threaded.
2014-08-22 14:38:48 +02:00
static struct kore_buf *header_buf;
Cookies and arguments parsing improvements (#166) Add new cookie API for handling of cookies.
2017-02-07 21:49:10 +01:00
static struct kore_buf *ckhdr_buf;
bump size of http_version array.
2018-06-23 17:23:45 +02:00
static char http_version[64];
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
static u_int16_t http_version_len;
move from multithreads to single threaded worker processes.
2013-06-04 11:55:38 +02:00
static TAILQ_HEAD(, http_request) http_requests;
Introduce http_request_sleep() and http_request_wakeup(). These 2 functions can be used to move an HTTP request from/to the active http_requests list. Effectively putting them to "sleep" or "waking them up". Sprinkle this through the pgsql and task code. If used correctly greatly reduces overhead for managing sleeping tasks.
2014-07-04 16:51:19 +02:00
static TAILQ_HEAD(, http_request) http_requests_sleeping;
Add filemaps. A filemap is a way of telling Kore to serve files from a directory much like a traditional webserver can do. Kore filemaps only handles files. Kore does not generate directory indexes or deal with non-regular files. The way files are sent to a client differs a bit per platform and build options: default: - mmap() backed file transfer due to TLS. NOTLS=1 - sendfile() under FreeBSD, macOS and Linux. - mmap() backed file for OpenBSD. The opened file descriptors/mmap'd regions are cached and reused when appropriate. If a file is no longer in use it will be closed and evicted from the cache after 30 seconds. New API's are available allowing developers to use these facilities via: void net_send_fileref(struct connection *, struct kore_fileref *); void http_response_fileref(struct http_request *, struct kore_fileref *); Kore will attempt to match media types based on file extensions. A few default types are built-in. Others can be added via the new "http_media_type" configuration directive.
2018-06-28 13:27:44 +02:00
static LIST_HEAD(, http_media_type) http_media_types;
Add fixed size memory pools and use them throughout Kore.
2013-07-15 10:13:36 +02:00
static struct kore_pool http_request_pool;
Cookies and arguments parsing improvements (#166) Add new cookie API for handling of cookies.
2017-02-07 21:49:10 +01:00
static struct kore_pool http_cookie_pool;
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
static struct kore_pool http_body_path;
Add HTTP runlocks. A way to serialize access to HTTP page handlers in case you are using some asynchronous api such as pgsql or libcurl stuff.
2019-06-02 16:29:54 +02:00
static struct kore_pool http_rlq_pool;
move from multithreads to single threaded worker processes.
2013-06-04 11:55:38 +02:00
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
struct kore_pool http_header_pool;
Rework http_pretty_error a tiny bit.
2020-03-04 08:36:10 +01:00
int http_pretty_error = 0;
Let http_request_limit matter. Before http_request_limit just constrained the number of HTTP requests we'd deal with in a single http_process_requests() call. But it should really mean how many maximum HTTP requests are allowed to be alive in the worker process before we start sending 503s back. While here, drop the lock timeout for a worker to 100ms down from 500ms and do not allow a worker to grab the accept lock if their HTTP request queue is full. This makes things much more pleasant memory wise as the http_request_pool won't just grow over time.
2018-02-13 11:56:51 +01:00
u_int32_t http_request_count = 0;
Add http_request_ms configuration option. This option allows a user to finetune the number of milliseconds a worker process will max spend inside the http_process() loop. By default this is 10ms.
2018-03-14 13:41:17 +01:00
u_int32_t http_request_ms = HTTP_REQUEST_MS;
Add http_[header|body]_timeout. If the HTTP request headers or the HTTP body have not arrived before these timeouts expire, Kore will send a 408 back to the client.
2019-04-11 20:51:49 +02:00
u_int16_t http_body_timeout = HTTP_BODY_TIMEOUT;
Improve very heavy load handling. Introduces two new configuration knobs: * socket_backlog (backlog for listen(2)) * http_request_limit The second one is the most interesting one. Before, kore would iterate over all received HTTP requests in its queue before returning out of http_process(). Under heavy load this queue can cause Kore to spend a considerable amount of time iterating over said queue. With the http_request_limit, kore will process at MOST http_request_limit requests before returning back to the event loop. This means responses to processed requests are sent out much quicker and allows kore to handle any other incoming requests more gracefully.
2015-04-09 15:29:44 +02:00
u_int32_t http_request_limit = HTTP_REQUEST_LIMIT;
Add http_hsts_enable (enabled by default with max-age=31536000) to Kore's configuration file. If enabled Kore adds the HSTS header to every response. - Additionally, fix some typos in the example configuration. - Change default SSL cipher list again, no more RC4 and almost PFS for all browsers.
2013-10-15 10:44:56 +02:00
u_int64_t http_hsts_enable = HTTP_HSTS_ENABLE;
Add http_header_max and http_postbody_max configuration variables. - http_header_max: Maximum size of HTTP headers (in non SPDY connections). - http_postbody_max: Maximum size of an HTTP POST body (both in SPDY and HTTP mode). Right now Kore will simply DC the client, ideally we want to send a 413 (entity too large) to the client however. See modules/examples/module.conf for more.
2013-09-22 20:05:24 +02:00
u_int16_t http_header_max = HTTP_HEADER_MAX_LEN;
Add http_keepalive_time configuration parameter. Allows you to configure maximum amount of seconds an HTTP connection can stay open (does not affect SPDY connections). If set to 0 it will disable keep-alive all together. Add some inttypes fluff.
2013-10-15 11:10:45 +02:00
u_int16_t http_keepalive_time = HTTP_KEEPALIVE_TIME;
Add http_[header|body]_timeout. If the HTTP request headers or the HTTP body have not arrived before these timeouts expire, Kore will send a 408 back to the client.
2019-04-11 20:51:49 +02:00
u_int16_t http_header_timeout = HTTP_HEADER_TIMEOUT;
Add asynchronous pgsql query support to python. This commit adds the ability to use python "await" to suspend execution of your page handler until the query sent to postgresql has returned a result. This is built upon the existing asynchrous query framework Kore had. With this you can now write stuff like: async def page(req): result = await req.pgsql("db", "SELECT name FROM table"); req.response(200, json.dumps(result).encode("utf-8")) The above code will fire off a query and suspend itself so Kore can take care of business as usual until the query is successful at which point Kore will jump back into the handler and resume. This does not use threading, it's purely based on Python's excellent coroutines and generators and Kore its built-in pgsql support.
2017-02-06 11:42:53 +01:00
size_t http_body_max = HTTP_BODY_MAX_LEN;
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
char *http_body_disk_path = HTTP_BODY_DISK_PATH;
Add http_[header|body]_timeout. If the HTTP request headers or the HTTP body have not arrived before these timeouts expire, Kore will send a 408 back to the client.
2019-04-11 20:51:49 +02:00
u_int64_t http_body_disk_offload = HTTP_BODY_DISK_OFFLOAD;
upon quit time, workers will not accept new connections but will handle all outstanding http requests.
2013-06-04 13:43:11 +02:00
Add filemaps. A filemap is a way of telling Kore to serve files from a directory much like a traditional webserver can do. Kore filemaps only handles files. Kore does not generate directory indexes or deal with non-regular files. The way files are sent to a client differs a bit per platform and build options: default: - mmap() backed file transfer due to TLS. NOTLS=1 - sendfile() under FreeBSD, macOS and Linux. - mmap() backed file for OpenBSD. The opened file descriptors/mmap'd regions are cached and reused when appropriate. If a file is no longer in use it will be closed and evicted from the cache after 30 seconds. New API's are available allowing developers to use these facilities via: void net_send_fileref(struct connection *, struct kore_fileref *); void http_response_fileref(struct http_request *, struct kore_fileref *); Kore will attempt to match media types based on file extensions. A few default types are built-in. Others can be added via the new "http_media_type" configuration directive.
2018-06-28 13:27:44 +02:00
void
http_parent_init(void)
{
LIST_INIT(&http_media_types);
}
move from multithreads to single threaded worker processes.
2013-06-04 11:55:38 +02:00
void
http_init(void)
{
Add filemaps. A filemap is a way of telling Kore to serve files from a directory much like a traditional webserver can do. Kore filemaps only handles files. Kore does not generate directory indexes or deal with non-regular files. The way files are sent to a client differs a bit per platform and build options: default: - mmap() backed file transfer due to TLS. NOTLS=1 - sendfile() under FreeBSD, macOS and Linux. - mmap() backed file for OpenBSD. The opened file descriptors/mmap'd regions are cached and reused when appropriate. If a file is no longer in use it will be closed and evicted from the cache after 30 seconds. New API's are available allowing developers to use these facilities via: void net_send_fileref(struct connection *, struct kore_fileref *); void http_response_fileref(struct http_request *, struct kore_fileref *); Kore will attempt to match media types based on file extensions. A few default types are built-in. Others can be added via the new "http_media_type" configuration directive.
2018-06-28 13:27:44 +02:00
int prealloc, l, i;
No longer preallocate worker_max_connections for the http pools. Instead use 10% of worker_max_connections up to a maximum of 1000.
2014-07-30 13:20:27 +02:00
move from multithreads to single threaded worker processes.
2013-06-04 11:55:38 +02:00
TAILQ_INIT(&http_requests);
Introduce http_request_sleep() and http_request_wakeup(). These 2 functions can be used to move an HTTP request from/to the active http_requests list. Effectively putting them to "sleep" or "waking them up". Sprinkle this through the pgsql and task code. If used correctly greatly reduces overhead for managing sleeping tasks.
2014-07-04 16:51:19 +02:00
TAILQ_INIT(&http_requests_sleeping);
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
Cookies and arguments parsing improvements (#166) Add new cookie API for handling of cookies.
2017-02-07 21:49:10 +01:00
header_buf = kore_buf_alloc(HTTP_HEADER_BUFSIZE);
ckhdr_buf = kore_buf_alloc(HTTP_COOKIE_BUFSIZE);
Add fixed size memory pools and use them throughout Kore.
2013-07-15 10:13:36 +02:00
Add the http_server_version configuration option. This allows setting a custom server header from the config file, for example to mask the version number.
2020-02-18 22:29:41 +01:00
if (!http_version_len) {
l = snprintf(http_version, sizeof(http_version),
"server: kore (%s)\r\n", kore_version);
if (l == -1 || (size_t)l >= sizeof(http_version))
fatal("http_init(): http_version buffer too small");
http_version_len = l;
}
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
prealloc = MIN((worker_max_connections / 10), 1000);
Add fixed size memory pools and use them throughout Kore.
2013-07-15 10:13:36 +02:00
kore_pool_init(&http_request_pool, "http_request_pool",
Let http_request_limit matter. Before http_request_limit just constrained the number of HTTP requests we'd deal with in a single http_process_requests() call. But it should really mean how many maximum HTTP requests are allowed to be alive in the worker process before we start sending 503s back. While here, drop the lock timeout for a worker to 100ms down from 500ms and do not allow a worker to grab the accept lock if their HTTP request queue is full. This makes things much more pleasant memory wise as the http_request_pool won't just grow over time.
2018-02-13 11:56:51 +01:00
sizeof(struct http_request), http_request_limit);
Add fixed size memory pools and use them throughout Kore.
2013-07-15 10:13:36 +02:00
kore_pool_init(&http_header_pool, "http_header_pool",
No longer preallocate worker_max_connections for the http pools. Instead use 10% of worker_max_connections up to a maximum of 1000.
2014-07-30 13:20:27 +02:00
sizeof(struct http_header), prealloc * HTTP_REQ_HEADER_MAX);
Cookies and arguments parsing improvements (#166) Add new cookie API for handling of cookies.
2017-02-07 21:49:10 +01:00
kore_pool_init(&http_cookie_pool, "http_cookie_pool",
sizeof(struct http_cookie), prealloc * HTTP_MAX_COOKIES);
Add HTTP runlocks. A way to serialize access to HTTP page handlers in case you are using some asynchronous api such as pgsql or libcurl stuff.
2019-06-02 16:29:54 +02:00
kore_pool_init(&http_rlq_pool, "http_rlq_pool",
sizeof(struct http_runlock_queue), http_request_limit);
Place path and host in http_request into pools. Also change the amount of times we end of doing unneccessairy strlen() calls.
2015-04-15 12:40:15 +02:00
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
kore_pool_init(&http_body_path,
"http_body_path", HTTP_BODY_PATH_MAX, prealloc);
Add filemaps. A filemap is a way of telling Kore to serve files from a directory much like a traditional webserver can do. Kore filemaps only handles files. Kore does not generate directory indexes or deal with non-regular files. The way files are sent to a client differs a bit per platform and build options: default: - mmap() backed file transfer due to TLS. NOTLS=1 - sendfile() under FreeBSD, macOS and Linux. - mmap() backed file for OpenBSD. The opened file descriptors/mmap'd regions are cached and reused when appropriate. If a file is no longer in use it will be closed and evicted from the cache after 30 seconds. New API's are available allowing developers to use these facilities via: void net_send_fileref(struct connection *, struct kore_fileref *); void http_response_fileref(struct http_request *, struct kore_fileref *); Kore will attempt to match media types based on file extensions. A few default types are built-in. Others can be added via the new "http_media_type" configuration directive.
2018-06-28 13:27:44 +02:00
for (i = 0; builtin_media[i].ext != NULL; i++) {
if (!http_media_register(builtin_media[i].ext,
builtin_media[i].type)) {
fatal("duplicate media type for %s",
builtin_media[i].ext);
}
}
move from multithreads to single threaded worker processes.
2013-06-04 11:55:38 +02:00
}
Add resource management as part of the kore shutdown process.
2015-12-29 20:39:39 +01:00
void
rename *_fini to *_cleanup
2016-01-04 22:40:14 +01:00
http_cleanup(void)
Add resource management as part of the kore shutdown process.
2015-12-29 20:39:39 +01:00
{
if (header_buf != NULL) {
kore_buf_free(header_buf);
header_buf = NULL;
}
Cookies and arguments parsing improvements (#166) Add new cookie API for handling of cookies.
2017-02-07 21:49:10 +01:00
if (ckhdr_buf != NULL) {
kore_buf_free(ckhdr_buf);
ckhdr_buf = NULL;
}
rename *_fini to *_cleanup
2016-01-04 22:40:14 +01:00
kore_pool_cleanup(&http_request_pool);
kore_pool_cleanup(&http_header_pool);
Add cleanup of http_body_path pool
2016-02-01 11:10:04 +01:00
kore_pool_cleanup(&http_body_path);
Add resource management as part of the kore shutdown process.
2015-12-29 20:39:39 +01:00
}
Set server version (#157)
2017-01-13 13:03:47 +01:00
void
http_server_version(const char *version)
{
int l;
l = snprintf(http_version, sizeof(http_version),
"server: %s\r\n", version);
if (l == -1 || (size_t)l >= sizeof(http_version))
fatal("http_server_version(): http_version buffer too small");
http_version_len = l;
}
Add http_[header|body]_timeout. If the HTTP request headers or the HTTP body have not arrived before these timeouts expire, Kore will send a 408 back to the client.
2019-04-11 20:51:49 +02:00
int
http_check_timeout(struct connection *c, u_int64_t now)
{
check http timeouts better
2019-04-12 14:26:47 +02:00
u_int64_t d;
Add http_[header|body]_timeout. If the HTTP request headers or the HTTP body have not arrived before these timeouts expire, Kore will send a 408 back to the client.
2019-04-11 20:51:49 +02:00
if (c->http_timeout == 0)
return (KORE_RESULT_OK);
check http timeouts better
2019-04-12 14:26:47 +02:00
if (now > c->http_start)
d = now - c->http_start;
else
d = 0;
if (d >= c->http_timeout) {
Stop hardcoding HTTP error codes in http_error_response() calls. Use predefined HTTP_STATUS_* macros instead.
2021-04-14 15:14:26 +02:00
http_error_response(c, HTTP_STATUS_REQUEST_TIMEOUT);
Add http_[header|body]_timeout. If the HTTP request headers or the HTTP body have not arrived before these timeouts expire, Kore will send a 408 back to the client.
2019-04-11 20:51:49 +02:00
kore_connection_disconnect(c);
return (KORE_RESULT_ERROR);
}
return (KORE_RESULT_OK);
}
Introduce http_request_sleep() and http_request_wakeup(). These 2 functions can be used to move an HTTP request from/to the active http_requests list. Effectively putting them to "sleep" or "waking them up". Sprinkle this through the pgsql and task code. If used correctly greatly reduces overhead for managing sleeping tasks.
2014-07-04 16:51:19 +02:00
void
http_request_sleep(struct http_request *req)
{
if (!(req->flags & HTTP_REQUEST_SLEEPING)) {
req->flags |= HTTP_REQUEST_SLEEPING;
TAILQ_REMOVE(&http_requests, req, list);
TAILQ_INSERT_TAIL(&http_requests_sleeping, req, list);
}
}
void
http_request_wakeup(struct http_request *req)
{
if (req->flags & HTTP_REQUEST_SLEEPING) {
req->flags &= ~HTTP_REQUEST_SLEEPING;
TAILQ_REMOVE(&http_requests_sleeping, req, list);
TAILQ_INSERT_TAIL(&http_requests, req, list);
}
}
move from multithreads to single threaded worker processes.
2013-06-04 11:55:38 +02:00
void
http_process(void)
{
Add http_request_ms configuration option. This option allows a user to finetune the number of milliseconds a worker process will max spend inside the http_process() loop. By default this is 10ms.
2018-03-14 13:41:17 +01:00
u_int64_t total;
Chain our sending netbufs together. Gives us a lot less SSL_write() calls.
2013-10-25 14:22:29 +02:00
struct http_request *req, *next;
move from multithreads to single threaded worker processes.
2013-06-04 11:55:38 +02:00
Add http_request_ms configuration option. This option allows a user to finetune the number of milliseconds a worker process will max spend inside the http_process() loop. By default this is 10ms.
2018-03-14 13:41:17 +01:00
total = 0;
Revert "TAILQ_FOREACH_SAFE() exists so use it." Because some asshole distributions claim to have a sane queue.h implementation while they do not.
2017-02-07 22:44:20 +01:00
for (req = TAILQ_FIRST(&http_requests); req != NULL; req = next) {
Add http_request_ms configuration option. This option allows a user to finetune the number of milliseconds a worker process will max spend inside the http_process() loop. By default this is 10ms.
2018-03-14 13:41:17 +01:00
if (total >= http_request_ms)
Improve very heavy load handling. Introduces two new configuration knobs: * socket_backlog (backlog for listen(2)) * http_request_limit The second one is the most interesting one. Before, kore would iterate over all received HTTP requests in its queue before returning out of http_process(). Under heavy load this queue can cause Kore to spend a considerable amount of time iterating over said queue. With the http_request_limit, kore will process at MOST http_request_limit requests before returning back to the event loop. This means responses to processed requests are sent out much quicker and allows kore to handle any other incoming requests more gracefully.
2015-04-09 15:29:44 +02:00
break;
move from multithreads to single threaded worker processes.
2013-06-04 11:55:38 +02:00
Improve very heavy load handling. Introduces two new configuration knobs: * socket_backlog (backlog for listen(2)) * http_request_limit The second one is the most interesting one. Before, kore would iterate over all received HTTP requests in its queue before returning out of http_process(). Under heavy load this queue can cause Kore to spend a considerable amount of time iterating over said queue. With the http_request_limit, kore will process at MOST http_request_limit requests before returning back to the event loop. This means responses to processed requests are sent out much quicker and allows kore to handle any other incoming requests more gracefully.
2015-04-09 15:29:44 +02:00
next = TAILQ_NEXT(req, list);
move from multithreads to single threaded worker processes.
2013-06-04 11:55:38 +02:00
if (req->flags & HTTP_REQUEST_DELETE) {
http_request_free(req);
continue;
}
Introduce http_request_sleep() and http_request_wakeup(). These 2 functions can be used to move an HTTP request from/to the active http_requests list. Effectively putting them to "sleep" or "waking them up". Sprinkle this through the pgsql and task code. If used correctly greatly reduces overhead for managing sleeping tasks.
2014-07-04 16:51:19 +02:00
/* Sleeping requests should be in http_requests_sleeping. */
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
if (req->flags & HTTP_REQUEST_SLEEPING)
Introduce http_request_sleep() and http_request_wakeup(). These 2 functions can be used to move an HTTP request from/to the active http_requests list. Effectively putting them to "sleep" or "waking them up". Sprinkle this through the pgsql and task code. If used correctly greatly reduces overhead for managing sleeping tasks.
2014-07-04 16:51:19 +02:00
fatal("http_process: sleeping request on list");
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
move from multithreads to single threaded worker processes.
2013-06-04 11:55:38 +02:00
if (!(req->flags & HTTP_REQUEST_COMPLETE))
continue;
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
http_process_request(req);
Add http_request_ms configuration option. This option allows a user to finetune the number of milliseconds a worker process will max spend inside the http_process() loop. By default this is 10ms.
2018-03-14 13:41:17 +01:00
total += req->ms;
Immediately remove completed HTTP requests. No need to wait until the next time http_process() is called, which could result in HTTP requests backing up even though we are processing them at a fast pace.
2019-05-29 20:30:43 +02:00
if (req->flags & HTTP_REQUEST_DELETE)
http_request_free(req);
Kore pgsql improvements. Don't wait for a full event loop until we call the page handler for a received pgsql result. This speeds up page loads using KORE_PGSQL by quite a lot, especially on a non busy server.
2014-04-17 10:49:48 +02:00
}
}
void
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
http_process_request(struct http_request *req)
Kore pgsql improvements. Don't wait for a full event loop until we call the page handler for a received pgsql result. This speeds up page loads using KORE_PGSQL by quite a lot, especially on a non busy server.
2014-04-17 10:49:48 +02:00
{
Add initial python support. Based on work done by Stanislav Yudin.
2017-01-12 23:38:51 +01:00
int r;
Kore pgsql improvements. Don't wait for a full event loop until we call the page handler for a received pgsql result. This speeds up page loads using KORE_PGSQL by quite a lot, especially on a non busy server.
2014-04-17 10:49:48 +02:00
Change how routes are configured in Kore. Routes are now configured in a context per route: route /path { handler handler_name methods get post head validate qs:get id v_id } All route related configurations are per-route, allowing multiple routes for the same path (for different methods). The param context is removed and merged into the route context now so that you use the validate keyword to specify what needs validating.
2021-09-15 11:09:52 +02:00
if (req->flags & HTTP_REQUEST_DELETE || req->rt == NULL)
Kore pgsql improvements. Don't wait for a full event loop until we call the page handler for a received pgsql result. This speeds up page loads using KORE_PGSQL by quite a lot, especially on a non busy server.
2014-04-17 10:49:48 +02:00
return;
req->start = kore_time_ms();
Change how routes are configured in Kore. Routes are now configured in a context per route: route /path { handler handler_name methods get post head validate qs:get id v_id } All route related configurations are per-route, allowing multiple routes for the same path (for different methods). The param context is removed and merged into the route context now so that you use the validate keyword to specify what needs validating.
2021-09-15 11:09:52 +02:00
if (req->rt->auth != NULL && !(req->flags & HTTP_REQUEST_AUTHED))
r = kore_auth_run(req, req->rt->auth);
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
else
r = KORE_RESULT_OK;
Kore pgsql improvements. Don't wait for a full event loop until we call the page handler for a received pgsql result. This speeds up page loads using KORE_PGSQL by quite a lot, especially on a non busy server.
2014-04-17 10:49:48 +02:00
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
switch (r) {
case KORE_RESULT_OK:
Change how routes are configured in Kore. Routes are now configured in a context per route: route /path { handler handler_name methods get post head validate qs:get id v_id } All route related configurations are per-route, allowing multiple routes for the same path (for different methods). The param context is removed and merged into the route context now so that you use the validate keyword to specify what needs validating.
2021-09-15 11:09:52 +02:00
r = kore_runtime_http_request(req->rt->rcall, req);
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
break;
case KORE_RESULT_RETRY:
break;
case KORE_RESULT_ERROR:
/*
* Set r to KORE_RESULT_OK so we can properly
* flush the result from kore_auth_run().
*/
r = KORE_RESULT_OK;
break;
default:
fatal("kore_auth() returned unknown %d", r);
Kore pgsql improvements. Don't wait for a full event loop until we call the page handler for a received pgsql result. This speeds up page loads using KORE_PGSQL by quite a lot, especially on a non busy server.
2014-04-17 10:49:48 +02:00
}
req->end = kore_time_ms();
Add http_request_ms configuration option. This option allows a user to finetune the number of milliseconds a worker process will max spend inside the http_process() loop. By default this is 10ms.
2018-03-14 13:41:17 +01:00
req->ms = req->end - req->start;
req->total += req->ms;
move from multithreads to single threaded worker processes.
2013-06-04 11:55:38 +02:00
Kore pgsql improvements. Don't wait for a full event loop until we call the page handler for a received pgsql result. This speeds up page loads using KORE_PGSQL by quite a lot, especially on a non busy server.
2014-04-17 10:49:48 +02:00
switch (r) {
case KORE_RESULT_OK:
r = net_send_flush(req->owner);
if (r == KORE_RESULT_ERROR)
kore_connection_disconnect(req->owner);
break;
case KORE_RESULT_ERROR:
kore_connection_disconnect(req->owner);
break;
case KORE_RESULT_RETRY:
Fix accesslog under SPDY connections
2014-04-23 13:18:37 +02:00
return;
fatal if the return value from a page handler isn't what we want.
2014-08-05 14:17:23 +02:00
default:
fatal("A page handler returned an unknown result: %d", r);
Kore pgsql improvements. Don't wait for a full event loop until we call the page handler for a received pgsql result. This speeds up page loads using KORE_PGSQL by quite a lot, especially on a non busy server.
2014-04-17 10:49:48 +02:00
}
Change how routes are configured in Kore. Routes are now configured in a context per route: route /path { handler handler_name methods get post head validate qs:get id v_id } All route related configurations are per-route, allowing multiple routes for the same path (for different methods). The param context is removed and merged into the route context now so that you use the validate keyword to specify what needs validating.
2021-09-15 11:09:52 +02:00
if (req->rt->dom->accesslog)
Make accesslogs optional
2014-07-30 09:11:21 +02:00
kore_accesslog(req);
Fix accesslog under SPDY connections
2014-04-23 13:18:37 +02:00
req->flags |= HTTP_REQUEST_DELETE;
move from multithreads to single threaded worker processes.
2013-06-04 11:55:38 +02:00
}
allow modules to add headers to the response before calling http_response() by calling http_response_header_add(). fix wrong overflow check in spdy_stream_get_header(). html_inject now exports last modified timestamp for the files that are compiled into the module in the format static_mtime_<type>_<name>. modules can now look into the request headers using http_request_header_get().
2013-05-01 21:16:09 +02:00
void
Rename the two functions used to read/set headers. Rename http_request_header_get() to http_request_header(). Rename http_response_header_add() to http_response_header().
2014-08-05 14:26:31 +02:00
http_response_header(struct http_request *req,
sprinkle const char where appropriate
2014-08-04 09:48:41 +02:00
const char *header, const char *value)
allow modules to add headers to the response before calling http_response() by calling http_response_header_add(). fix wrong overflow check in spdy_stream_get_header(). html_inject now exports last modified timestamp for the files that are compiled into the module in the format static_mtime_<type>_<name>. modules can now look into the request headers using http_request_header_get().
2013-05-01 21:16:09 +02:00
{
struct http_header *hdr;
Let http_response_header() handle duplicates. If a response header was previously by an application for an HTTP request http_response_header() will now overwrite the previous value.
2021-09-12 14:30:33 +02:00
hdr = NULL;
allow modules to add headers to the response before calling http_response() by calling http_response_header_add(). fix wrong overflow check in spdy_stream_get_header(). html_inject now exports last modified timestamp for the files that are compiled into the module in the format static_mtime_<type>_<name>. modules can now look into the request headers using http_request_header_get().
2013-05-01 21:16:09 +02:00
Let http_response_header() handle duplicates. If a response header was previously by an application for an HTTP request http_response_header() will now overwrite the previous value.
2021-09-12 14:30:33 +02:00
TAILQ_FOREACH(hdr, &req->resp_headers, list) {
if (!strcasecmp(hdr->header, header)) {
TAILQ_REMOVE(&req->resp_headers, hdr, list);
kore_free(hdr->header);
kore_free(hdr->value);
break;
}
}
if (hdr == NULL)
hdr = kore_pool_get(&http_header_pool);
allow modules to add headers to the response before calling http_response() by calling http_response_header_add(). fix wrong overflow check in spdy_stream_get_header(). html_inject now exports last modified timestamp for the files that are compiled into the module in the format static_mtime_<type>_<name>. modules can now look into the request headers using http_request_header_get().
2013-05-01 21:16:09 +02:00
hdr->header = kore_strdup(header);
hdr->value = kore_strdup(value);
Let http_response_header() handle duplicates. If a response header was previously by an application for an HTTP request http_response_header() will now overwrite the previous value.
2021-09-12 14:30:33 +02:00
add very basic support for HTTP/1.1.
2013-05-02 03:51:04 +02:00
TAILQ_INSERT_TAIL(&(req->resp_headers), hdr, list);
allow modules to add headers to the response before calling http_response() by calling http_response_header_add(). fix wrong overflow check in spdy_stream_get_header(). html_inject now exports last modified timestamp for the files that are compiled into the module in the format static_mtime_<type>_<name>. modules can now look into the request headers using http_request_header_get().
2013-05-01 21:16:09 +02:00
}
lots of new stuff, including processing of http requests and an attempt to build an initial spdy response (SYN frame + header block content).
2013-05-01 08:09:04 +02:00
void
http_request_free(struct http_request *req)
{
Attach tasks/pgsqls to http_requests once more. This way if an http request is removed while tasks or pgsqls are still active they are free'd out and cancelled properly.
2014-08-14 22:05:34 +02:00
#if defined(KORE_USE_TASKS)
struct kore_task *t, *nt;
int pending_tasks;
#endif
#if defined(KORE_USE_PGSQL)
struct kore_pgsql *pgsql;
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
#endif
#if defined(KORE_USE_CURL)
struct kore_curl *client;
Attach tasks/pgsqls to http_requests once more. This way if an http request is removed while tasks or pgsqls are still active they are free'd out and cancelled properly.
2014-08-14 22:05:34 +02:00
#endif
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
struct http_file *f, *fnext;
add proper query argument parsing, modules must call http_populate_arguments() before using http_argument_lookup() to lookup a given argument. population uses the appropriate query string (from GET or POST) automatically.
2013-05-03 09:09:27 +02:00
struct http_arg *q, *qnext;
allow modules to add headers to the response before calling http_response() by calling http_response_header_add(). fix wrong overflow check in spdy_stream_get_header(). html_inject now exports last modified timestamp for the files that are compiled into the module in the format static_mtime_<type>_<name>. modules can now look into the request headers using http_request_header_get().
2013-05-01 21:16:09 +02:00
struct http_header *hdr, *next;
Cookies and arguments parsing improvements (#166) Add new cookie API for handling of cookies.
2017-02-07 21:49:10 +01:00
struct http_cookie *ck, *cknext;
allow modules to add headers to the response before calling http_response() by calling http_response_header_add(). fix wrong overflow check in spdy_stream_get_header(). html_inject now exports last modified timestamp for the files that are compiled into the module in the format static_mtime_<type>_<name>. modules can now look into the request headers using http_request_header_get().
2013-05-01 21:16:09 +02:00
make sure we only call rt->on_free if req has route
2021-12-15 12:16:37 +01:00
if (req->rt != NULL && req->rt->on_free != NULL)
Python: allow route hooks via kore.route(). Adding the hooks keyword with a dictionary attached to specify the relevant hooks will hook them for the given route. Eg: domain.route("/", self.index, methods=["get"], hooks={ "on_free": self.request_free } ) These are the same hooks available via a normal Kore route configuration.
2021-12-14 23:15:21 +01:00
kore_runtime_http_request_free(req->rt->on_free, req);
Let http_state_create() take an "onfree" callback. This function is called when an HTTP request is being free'd, allowing you to perform any sort of state cleanup attached to the HTTP request.
2019-04-28 21:48:16 +02:00
Add HTTP runlocks. A way to serialize access to HTTP page handlers in case you are using some asynchronous api such as pgsql or libcurl stuff.
2019-06-02 16:29:54 +02:00
if (req->runlock != NULL) {
LIST_REMOVE(req->runlock, list);
req->runlock = NULL;
}
Task improvements. Synchronize access to state/result properly so one can access these from inside the task as well. Introduce KORE_TASK_STATE_ABORT which will be set when a task needs to be abort. You can use this to create tasks that run in a loop until aborted.
2014-07-04 11:28:17 +02:00
#if defined(KORE_USE_TASKS)
Attach tasks/pgsqls to http_requests once more. This way if an http request is removed while tasks or pgsqls are still active they are free'd out and cancelled properly.
2014-08-14 22:05:34 +02:00
pending_tasks = 0;
for (t = LIST_FIRST(&(req->tasks)); t != NULL; t = nt) {
nt = LIST_NEXT(t, rlist);
if (!kore_task_finished(t)) {
pending_tasks++;
} else {
kore_task_destroy(t);
}
}
kore_debug() has been unused for years. Kill all useless messages, convert useful ones into kore_log() instead.
2022-08-18 15:20:55 +02:00
if (pending_tasks)
Task improvements. Synchronize access to state/result properly so one can access these from inside the task as well. Introduce KORE_TASK_STATE_ABORT which will be set when a task needs to be abort. You can use this to create tasks that run in a loop until aborted.
2014-07-04 11:28:17 +02:00
return;
#endif
Add asynchronous pgsql query support to python. This commit adds the ability to use python "await" to suspend execution of your page handler until the query sent to postgresql has returned a result. This is built upon the existing asynchrous query framework Kore had. With this you can now write stuff like: async def page(req): result = await req.pgsql("db", "SELECT name FROM table"); req.response(200, json.dumps(result).encode("utf-8")) The above code will fire off a query and suspend itself so Kore can take care of business as usual until the query is successful at which point Kore will jump back into the handler and resume. This does not use threading, it's purely based on Python's excellent coroutines and generators and Kore its built-in pgsql support.
2017-02-06 11:42:53 +01:00
#if defined(KORE_USE_PYTHON)
Add async/await support for socket i/o in python. This means you can now do things like: resp = await koresock.recv(1024) await koresock.send(resp) directly from page handlers if they are defined as async. Adds lots more to the python goo such as fatalx(), bind_unix(), task_create() and socket_wrap().
2018-10-15 20:18:54 +02:00
if (req->py_coro != NULL) {
kore_python_coro_delete(req->py_coro);
req->py_coro = NULL;
}
Many many Python improvements. - Kore can now fully be configured via Python code if one wants nothing to do with configuration files. - Kore can now start single python files and no longer requires them to be inside a module directory. - Pass all regex capture groups to the handler methods, allowing you to get access to them immediately. - Change python websocket_handshake to take callable objects directly. - Added a new deployment configuration option. If set to "dev" or "development" Kore will automatically foreground, no chroot / etc. If set to "production" Kore *will* chroot, drop privs, etc. - Many more.. These are all backported from a project that I was working on a while ago. I decided these should go back into mainline Kore.
2019-09-26 15:49:00 +02:00
if (req->py_validator != NULL) {
kore_python_coro_delete(req->py_validator);
req->py_validator = NULL;
}
Allow kore.prerequests to be async.
2019-06-07 21:06:54 +02:00
Py_XDECREF(req->py_req);
Add asynchronous pgsql query support to python. This commit adds the ability to use python "await" to suspend execution of your page handler until the query sent to postgresql has returned a result. This is built upon the existing asynchrous query framework Kore had. With this you can now write stuff like: async def page(req): result = await req.pgsql("db", "SELECT name FROM table"); req.response(200, json.dumps(result).encode("utf-8")) The above code will fire off a query and suspend itself so Kore can take care of business as usual until the query is successful at which point Kore will jump back into the handler and resume. This does not use threading, it's purely based on Python's excellent coroutines and generators and Kore its built-in pgsql support.
2017-02-06 11:42:53 +01:00
#endif
Attach tasks/pgsqls to http_requests once more. This way if an http request is removed while tasks or pgsqls are still active they are free'd out and cancelled properly.
2014-08-14 22:05:34 +02:00
#if defined(KORE_USE_PGSQL)
while (!LIST_EMPTY(&(req->pgsqls))) {
pgsql = LIST_FIRST(&(req->pgsqls));
kore_pgsql_cleanup(pgsql);
}
#endif
Add asynchronous libcurl support. This commit adds the CURL=1 build option. When enabled allows you to schedule CURL easy handles onto the Kore event loop. It also adds an easy to use HTTP client API that abstracts away the settings required from libcurl to make HTTP requests. Tied together with HTTP request state machines this means you can write fully asynchronous HTTP client requests in an easy way. Additionally this exposes that API to the Python code as well allowing you do to things like: client = kore.httpclient("https://kore.io") status, body = await client.get() Introduces 2 configuration options: - curl_recv_max Max incoming bytes for a response. - curl_timeout Timeout in seconds before a transfer is cancelled. This API also allows you to take the CURL easy handle and send emails with it, run FTP, etc. All asynchronously.
2019-04-24 00:10:47 +02:00
#if defined(KORE_USE_CURL)
while (!LIST_EMPTY(&req->chandles)) {
client = LIST_FIRST(&req->chandles);
kore_curl_cleanup(client);
}
#endif
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
kore_free(req->headers);
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
req->host = NULL;
req->path = NULL;
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
req->headers = NULL;
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
Task improvements. Synchronize access to state/result properly so one can access these from inside the task as well. Introduce KORE_TASK_STATE_ABORT which will be set when a task needs to be abort. You can use this to create tasks that run in a loop until aborted.
2014-07-04 11:28:17 +02:00
TAILQ_REMOVE(&http_requests, req, list);
set req->owner to NULL when the connection removes it.
2016-12-26 20:08:53 +01:00
if (req->owner != NULL)
TAILQ_REMOVE(&(req->owner->http_requests), req, olist);
Task improvements. Synchronize access to state/result properly so one can access these from inside the task as well. Introduce KORE_TASK_STATE_ABORT which will be set when a task needs to be abort. You can use this to create tasks that run in a loop until aborted.
2014-07-04 11:28:17 +02:00
Revert "TAILQ_FOREACH_SAFE() exists so use it." Because some asshole distributions claim to have a sane queue.h implementation while they do not.
2017-02-07 22:44:20 +01:00
for (hdr = TAILQ_FIRST(&(req->resp_headers)); hdr != NULL; hdr = next) {
next = TAILQ_NEXT(hdr, list);
add very basic support for HTTP/1.1.
2013-05-02 03:51:04 +02:00
TAILQ_REMOVE(&(req->resp_headers), hdr, list);
Large changes to the memory subsystem in kore. - Change pools to use mmap() for allocating regions. - Change kore_malloc() to use pools for commonly sized objects. (split into multiple of 2 buckets, starting at 8 bytes up to 8192). - Rename kore_mem_free() to kore_free(). The preallocated pools will hold up to 128K of elements per block size. In case a larger object is to be allocated kore_malloc() will use malloc() instead.
2016-07-12 13:54:14 +02:00
kore_free(hdr->header);
kore_free(hdr->value);
Add fixed size memory pools and use them throughout Kore.
2013-07-15 10:13:36 +02:00
kore_pool_put(&http_header_pool, hdr);
add very basic support for HTTP/1.1.
2013-05-02 03:51:04 +02:00
}
Revert "TAILQ_FOREACH_SAFE() exists so use it." Because some asshole distributions claim to have a sane queue.h implementation while they do not.
2017-02-07 22:44:20 +01:00
for (hdr = TAILQ_FIRST(&(req->req_headers)); hdr != NULL; hdr = next) {
next = TAILQ_NEXT(hdr, list);
add very basic support for HTTP/1.1.
2013-05-02 03:51:04 +02:00
TAILQ_REMOVE(&(req->req_headers), hdr, list);
Add fixed size memory pools and use them throughout Kore.
2013-07-15 10:13:36 +02:00
kore_pool_put(&http_header_pool, hdr);
allow modules to add headers to the response before calling http_response() by calling http_response_header_add(). fix wrong overflow check in spdy_stream_get_header(). html_inject now exports last modified timestamp for the files that are compiled into the module in the format static_mtime_<type>_<name>. modules can now look into the request headers using http_request_header_get().
2013-05-01 21:16:09 +02:00
}
Revert "TAILQ_FOREACH_SAFE() exists so use it." Because some asshole distributions claim to have a sane queue.h implementation while they do not.
2017-02-07 22:44:20 +01:00
for (ck = TAILQ_FIRST(&(req->resp_cookies)); ck != NULL; ck = cknext) {
cknext = TAILQ_NEXT(ck, list);
Cookies and arguments parsing improvements (#166) Add new cookie API for handling of cookies.
2017-02-07 21:49:10 +01:00
TAILQ_REMOVE(&(req->resp_cookies), ck, list);
kore_free(ck->name);
kore_free(ck->value);
kore_free(ck->path);
kore_free(ck->domain);
kore_pool_put(&http_cookie_pool, ck);
}
Revert "TAILQ_FOREACH_SAFE() exists so use it." Because some asshole distributions claim to have a sane queue.h implementation while they do not.
2017-02-07 22:44:20 +01:00
for (ck = TAILQ_FIRST(&(req->req_cookies)); ck != NULL; ck = cknext) {
cknext = TAILQ_NEXT(ck, list);
Cookies and arguments parsing improvements (#166) Add new cookie API for handling of cookies.
2017-02-07 21:49:10 +01:00
TAILQ_REMOVE(&(req->req_cookies), ck, list);
kore_free(ck->name);
kore_free(ck->value);
kore_pool_put(&http_cookie_pool, ck);
}
Revert "TAILQ_FOREACH_SAFE() exists so use it." Because some asshole distributions claim to have a sane queue.h implementation while they do not.
2017-02-07 22:44:20 +01:00
for (q = TAILQ_FIRST(&(req->arguments)); q != NULL; q = qnext) {
qnext = TAILQ_NEXT(q, list);
add proper query argument parsing, modules must call http_populate_arguments() before using http_argument_lookup() to lookup a given argument. population uses the appropriate query string (from GET or POST) automatically.
2013-05-03 09:09:27 +02:00
TAILQ_REMOVE(&(req->arguments), q, list);
Large changes to the memory subsystem in kore. - Change pools to use mmap() for allocating regions. - Change kore_malloc() to use pools for commonly sized objects. (split into multiple of 2 buckets, starting at 8 bytes up to 8192). - Rename kore_mem_free() to kore_free(). The preallocated pools will hold up to 128K of elements per block size. In case a larger object is to be allocated kore_malloc() will use malloc() instead.
2016-07-12 13:54:14 +02:00
kore_free(q->name);
remove unneeded NULL check before calling free.
2017-02-07 22:08:04 +01:00
kore_free(q->s_value);
Large changes to the memory subsystem in kore. - Change pools to use mmap() for allocating regions. - Change kore_malloc() to use pools for commonly sized objects. (split into multiple of 2 buckets, starting at 8 bytes up to 8192). - Rename kore_mem_free() to kore_free(). The preallocated pools will hold up to 128K of elements per block size. In case a larger object is to be allocated kore_malloc() will use malloc() instead.
2016-07-12 13:54:14 +02:00
kore_free(q);
add proper query argument parsing, modules must call http_populate_arguments() before using http_argument_lookup() to lookup a given argument. population uses the appropriate query string (from GET or POST) automatically.
2013-05-03 09:09:27 +02:00
}
Revert "TAILQ_FOREACH_SAFE() exists so use it." Because some asshole distributions claim to have a sane queue.h implementation while they do not.
2017-02-07 22:44:20 +01:00
for (f = TAILQ_FIRST(&(req->files)); f != NULL; f = fnext) {
fnext = TAILQ_NEXT(f, list);
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
TAILQ_REMOVE(&(req->files), f, list);
Large changes to the memory subsystem in kore. - Change pools to use mmap() for allocating regions. - Change kore_malloc() to use pools for commonly sized objects. (split into multiple of 2 buckets, starting at 8 bytes up to 8192). - Rename kore_mem_free() to kore_free(). The preallocated pools will hold up to 128K of elements per block size. In case a larger object is to be allocated kore_malloc() will use malloc() instead.
2016-07-12 13:54:14 +02:00
kore_free(f->filename);
kore_free(f->name);
kore_free(f);
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
}
Add PUT/DELETE/HEAD methods (finally). This commit renames certain POST centric variable and configuration naming to the correct HTTP body stuff. API changes include http_postbody_text() and http_postbody_bytes() to have become http_body_text() and http_body_bytes(). The developer is still responsible for validating the method their page handler is called with. Hopefully this becomes a configuration option soon enough.
2014-10-08 11:03:14 +02:00
if (req->http_body != NULL)
kore_buf_free(req->http_body);
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
if (req->http_body_fd != -1)
(void)close(req->http_body_fd);
if (req->http_body_path != NULL) {
don't log failure if unlinking fails with ENOENT
2017-02-22 17:52:38 +01:00
if (unlink(req->http_body_path) == -1 && errno != ENOENT) {
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
kore_log(LOG_NOTICE, "failed to unlink %s: %s",
req->http_body_path, errno_s);
}
kore_pool_put(&http_body_path, req->http_body_path);
}
- Add SPDY RST control frame handler. - Keep HTTP requests in connection, so we can delete them if the connection ends before the requests do (this way we don't leak them). - When spdy_stream_close() is called, delete the attached http request. (This shouldn't hurt to do, so hopefully won't cause major fallout). - When parsing HTTP, find the first occurence of end-of-headers so uploads with multipart/form-data can succeed properly. - Add a test upload page to the example module.
2013-09-09 10:59:56 +02:00
Add HTTP_REQUEST_RETAIN_EXTRA flag to http_request Signals Kore to not free any pointer set in req->hdlr_extra. Useful in certain scenarios where you have data per request bound to something in memory but do not want to lose it when the request is freed by Kore. Set this flag before your handler returns.
2015-03-27 23:23:21 +01:00
if (req->hdlr_extra != NULL &&
!(req->flags & HTTP_REQUEST_RETAIN_EXTRA))
Large changes to the memory subsystem in kore. - Change pools to use mmap() for allocating regions. - Change kore_malloc() to use pools for commonly sized objects. (split into multiple of 2 buckets, starting at 8 bytes up to 8192). - Rename kore_mem_free() to kore_free(). The preallocated pools will hold up to 128K of elements per block size. In case a larger object is to be allocated kore_malloc() will use malloc() instead.
2016-07-12 13:54:14 +02:00
kore_free(req->hdlr_extra);
Add fixed size memory pools and use them throughout Kore.
2013-07-15 10:13:36 +02:00
kore_pool_put(&http_request_pool, req);
Task improvements. Synchronize access to state/result properly so one can access these from inside the task as well. Introduce KORE_TASK_STATE_ABORT which will be set when a task needs to be abort. You can use this to create tasks that run in a loop until aborted.
2014-07-04 11:28:17 +02:00
http_request_count--;
lots of new stuff, including processing of http requests and an attempt to build an initial spdy response (SYN frame + header block content).
2013-05-01 08:09:04 +02:00
}
Add auto generated serving functions for assets. These functions are created by the cli tool when building and follow the naming format: asset_serve_<name>_<ext>(). Those serving functions can be used directly in handlers and callthrough to a http_serveable() function that uses the SHA1 of the asset as its ETag and automatically checks for if-none-match.
2016-12-26 21:15:03 +01:00
void
http_serveable(struct http_request *req, const void *data, size_t len,
add compile-time configurable mime types. these types are used for the new builtin asset_serve_* functions. must be defined in conf/build.conf.
2016-12-26 23:37:05 +01:00
const char *etag, const char *type)
Add auto generated serving functions for assets. These functions are created by the cli tool when building and follow the naming format: asset_serve_<name>_<ext>(). Those serving functions can be used directly in handlers and callthrough to a http_serveable() function that uses the SHA1 of the asset as its ETag and automatically checks for if-none-match.
2016-12-26 21:15:03 +01:00
{
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
const char *match;
Add auto generated serving functions for assets. These functions are created by the cli tool when building and follow the naming format: asset_serve_<name>_<ext>(). Those serving functions can be used directly in handlers and callthrough to a http_serveable() function that uses the SHA1 of the asset as its ETag and automatically checks for if-none-match.
2016-12-26 21:15:03 +01:00
if (req->method != HTTP_METHOD_GET) {
http_response_header(req, "allow", "get");
http_response(req, HTTP_STATUS_BAD_REQUEST, NULL, 0);
return;
}
if (http_request_header(req, "if-none-match", &match)) {
if (!strcmp(match, etag)) {
http_response(req, HTTP_STATUS_NOT_MODIFIED, NULL, 0);
return;
}
}
http_response_header(req, "etag", etag);
add compile-time configurable mime types. these types are used for the new builtin asset_serve_* functions. must be defined in conf/build.conf.
2016-12-26 23:37:05 +01:00
http_response_header(req, "content-type", type);
Add auto generated serving functions for assets. These functions are created by the cli tool when building and follow the naming format: asset_serve_<name>_<ext>(). Those serving functions can be used directly in handlers and callthrough to a http_serveable() function that uses the SHA1 of the asset as its ETag and automatically checks for if-none-match.
2016-12-26 21:15:03 +01:00
http_response(req, HTTP_STATUS_OK, data, len);
}
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
void
Add http_response_close() to the C API. This is the same as http_response() except it will automatically close the connection after the response is sent. This is a bit easier than setting CONN_CLOSE_EMPTY yourself manually.
2021-09-12 14:13:24 +02:00
http_response(struct http_request *req, int code, const void *d, size_t l)
lots of new stuff, including processing of http requests and an attempt to build an initial spdy response (SYN frame + header block content).
2013-05-01 08:09:04 +02:00
{
Add asynchronous pgsql query support to python. This commit adds the ability to use python "await" to suspend execution of your page handler until the query sent to postgresql has returned a result. This is built upon the existing asynchrous query framework Kore had. With this you can now write stuff like: async def page(req): result = await req.pgsql("db", "SELECT name FROM table"); req.response(200, json.dumps(result).encode("utf-8")) The above code will fire off a query and suspend itself so Kore can take care of business as usual until the query is successful at which point Kore will jump back into the handler and resume. This does not use threading, it's purely based on Python's excellent coroutines and generators and Kore its built-in pgsql support.
2017-02-06 11:42:53 +01:00
if (req->owner == NULL)
return;
Add http_response_close() to the C API. This is the same as http_response() except it will automatically close the connection after the response is sent. This is a bit easier than setting CONN_CLOSE_EMPTY yourself manually.
2021-09-12 14:13:24 +02:00
req->status = code;
switch (req->owner->proto) {
case CONN_PROTO_HTTP:
case CONN_PROTO_WEBSOCKET:
http_response_normal(req, req->owner, code, d, l);
break;
default:
fatal("%s: bad proto %d", __func__, req->owner->proto);
/* NOTREACHED. */
}
}
void
http_response_close(struct http_request *req, int code, const void *d, size_t l)
{
if (req->owner == NULL)
return;
req->status = code;
req->owner->flags |= CONN_CLOSE_EMPTY;
Add http_keepalive_time configuration parameter. Allows you to configure maximum amount of seconds an HTTP connection can stay open (does not affect SPDY connections). If set to 0 it will disable keep-alive all together. Add some inttypes fluff.
2013-10-15 11:10:45 +02:00
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
switch (req->owner->proto) {
case CONN_PROTO_HTTP:
Fix websockets. Connection must be Upgrade, not keep-alive, Upgrade
2015-06-26 05:14:01 +02:00
case CONN_PROTO_WEBSOCKET:
Add http_response_close() to the C API. This is the same as http_response() except it will automatically close the connection after the response is sent. This is a bit easier than setting CONN_CLOSE_EMPTY yourself manually.
2021-09-12 14:13:24 +02:00
http_response_normal(req, req->owner, code, d, l);
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
break;
Add websocket support to Kore. Introduces a few new api functions: - kore_websocket_handshake(struct http_request *): Performs the handshake on an HTTP request (coming from page handler) - kore_websocket_send(struct connection *, u_int8_t, void *, size_t): Sends data to a websocket connection. - kore_websocket_broadcast(struct connection *, u_int8_t, void *, size_t, int): Broadcast the given websocket op and data to all connected websocket clients on the worker. Note that as of right now the WEBSOCKET_BROADCAST_GLOBAL scope option does not work yet and messages broadcasted will be restricted to workers only. - kore_worker_websocket_broadcast(struct connection *, void *, void *): Backend function used by kore_websocket_broadcast(). Could prove useful for developers to have access to. A simple example is given under examples/websocket. Known issues: Kore does not support PING or CONT frames just yet.
2014-11-24 11:01:12 +01:00
default:
Add http_response_close() to the C API. This is the same as http_response() except it will automatically close the connection after the response is sent. This is a bit easier than setting CONN_CLOSE_EMPTY yourself manually.
2021-09-12 14:13:24 +02:00
fatal("%s: bad proto %d", __func__, req->owner->proto);
Add websocket support to Kore. Introduces a few new api functions: - kore_websocket_handshake(struct http_request *): Performs the handshake on an HTTP request (coming from page handler) - kore_websocket_send(struct connection *, u_int8_t, void *, size_t): Sends data to a websocket connection. - kore_websocket_broadcast(struct connection *, u_int8_t, void *, size_t, int): Broadcast the given websocket op and data to all connected websocket clients on the worker. Note that as of right now the WEBSOCKET_BROADCAST_GLOBAL scope option does not work yet and messages broadcasted will be restricted to workers only. - kore_worker_websocket_broadcast(struct connection *, void *, void *): Backend function used by kore_websocket_broadcast(). Could prove useful for developers to have access to. A simple example is given under examples/websocket. Known issues: Kore does not support PING or CONT frames just yet.
2014-11-24 11:01:12 +01:00
/* NOTREACHED. */
lots of new stuff, including processing of http requests and an attempt to build an initial spdy response (SYN frame + header block content).
2013-05-01 08:09:04 +02:00
}
properly deflate/inflate of name/value blocks. and add ability to get headers from them.
2013-05-01 00:35:33 +02:00
}
lots of new stuff, including processing of http requests and an attempt to build an initial spdy response (SYN frame + header block content).
2013-05-01 08:09:04 +02:00
Add http_response_json(). Since its an HTTP response function it functions like http_response() but takes a kore_json_item pointer that it will automatically convert to a kore_buf and send/free using http_response_stream(). While here fix a problem with http_response_stream() which could end up not calling the cb() in case of HTTP_METHOD_HEAD. Since the behaviour is that it should call cb() when done it should do so immediately.
2021-09-12 15:10:06 +02:00
void
http_response_json(struct http_request *req, int status,
struct kore_json_item *json)
{
struct kore_buf *buf;
if (req->owner == NULL)
return;
buf = kore_buf_alloc(1024);
kore_json_item_tobuf(json, buf);
kore_json_item_free(json);
req->status = status;
http_response_header(req, "content-type", "application/json");
switch (req->owner->proto) {
case CONN_PROTO_HTTP:
http_response_stream(req, status, buf->data, buf->offset,
http_release_buffer, buf);
break;
default:
fatal("%s: bad proto %d", __func__, req->owner->proto);
/* NOTREACHED. */
}
}
Add http_response_stream() which can stream data from a buffer to the client.
2014-08-07 14:23:26 +02:00
void
http_response_stream(struct http_request *req, int status, void *base,
Code cleanup, several API breaking changes in here
2016-07-08 10:02:18 +02:00
size_t len, int (*cb)(struct netbuf *), void *arg)
Add http_response_stream() which can stream data from a buffer to the client.
2014-08-07 14:23:26 +02:00
{
Add a callback to http_response_stream(). This way we can get our code called whenever a stream is completed. This cb handler does stand alone from an http_request and is passed a netbuf data structure.
2014-08-10 18:46:44 +02:00
struct netbuf *nb;
Add asynchronous pgsql query support to python. This commit adds the ability to use python "await" to suspend execution of your page handler until the query sent to postgresql has returned a result. This is built upon the existing asynchrous query framework Kore had. With this you can now write stuff like: async def page(req): result = await req.pgsql("db", "SELECT name FROM table"); req.response(200, json.dumps(result).encode("utf-8")) The above code will fire off a query and suspend itself so Kore can take care of business as usual until the query is successful at which point Kore will jump back into the handler and resume. This does not use threading, it's purely based on Python's excellent coroutines and generators and Kore its built-in pgsql support.
2017-02-06 11:42:53 +01:00
if (req->owner == NULL)
return;
Add http_response_stream() which can stream data from a buffer to the client.
2014-08-07 14:23:26 +02:00
req->status = status;
switch (req->owner->proto) {
case CONN_PROTO_HTTP:
http_response_normal(req, req->owner, status, NULL, len);
break;
Add websocket support to Kore. Introduces a few new api functions: - kore_websocket_handshake(struct http_request *): Performs the handshake on an HTTP request (coming from page handler) - kore_websocket_send(struct connection *, u_int8_t, void *, size_t): Sends data to a websocket connection. - kore_websocket_broadcast(struct connection *, u_int8_t, void *, size_t, int): Broadcast the given websocket op and data to all connected websocket clients on the worker. Note that as of right now the WEBSOCKET_BROADCAST_GLOBAL scope option does not work yet and messages broadcasted will be restricted to workers only. - kore_worker_websocket_broadcast(struct connection *, void *, void *): Backend function used by kore_websocket_broadcast(). Could prove useful for developers to have access to. A simple example is given under examples/websocket. Known issues: Kore does not support PING or CONT frames just yet.
2014-11-24 11:01:12 +01:00
default:
Add http_response_json(). Since its an HTTP response function it functions like http_response() but takes a kore_json_item pointer that it will automatically convert to a kore_buf and send/free using http_response_stream(). While here fix a problem with http_response_stream() which could end up not calling the cb() in case of HTTP_METHOD_HEAD. Since the behaviour is that it should call cb() when done it should do so immediately.
2021-09-12 15:10:06 +02:00
fatal("%s: bad proto %d", __func__, req->owner->proto);
Add websocket support to Kore. Introduces a few new api functions: - kore_websocket_handshake(struct http_request *): Performs the handshake on an HTTP request (coming from page handler) - kore_websocket_send(struct connection *, u_int8_t, void *, size_t): Sends data to a websocket connection. - kore_websocket_broadcast(struct connection *, u_int8_t, void *, size_t, int): Broadcast the given websocket op and data to all connected websocket clients on the worker. Note that as of right now the WEBSOCKET_BROADCAST_GLOBAL scope option does not work yet and messages broadcasted will be restricted to workers only. - kore_worker_websocket_broadcast(struct connection *, void *, void *): Backend function used by kore_websocket_broadcast(). Could prove useful for developers to have access to. A simple example is given under examples/websocket. Known issues: Kore does not support PING or CONT frames just yet.
2014-11-24 11:01:12 +01:00
/* NOTREACHED. */
Add http_response_stream() which can stream data from a buffer to the client.
2014-08-07 14:23:26 +02:00
}
Add http_response_json(). Since its an HTTP response function it functions like http_response() but takes a kore_json_item pointer that it will automatically convert to a kore_buf and send/free using http_response_stream(). While here fix a problem with http_response_stream() which could end up not calling the cb() in case of HTTP_METHOD_HEAD. Since the behaviour is that it should call cb() when done it should do so immediately.
2021-09-12 15:10:06 +02:00
net_send_stream(req->owner, base, len, cb, &nb);
nb->extra = arg;
if (req->method == HTTP_METHOD_HEAD) {
nb->s_off = nb->b_len;
net_remove_netbuf(req->owner, nb);
Add PUT/DELETE/HEAD methods (finally). This commit renames certain POST centric variable and configuration naming to the correct HTTP body stuff. API changes include http_postbody_text() and http_postbody_bytes() to have become http_body_text() and http_body_bytes(). The developer is still responsible for validating the method their page handler is called with. Hopefully this becomes a configuration option soon enough.
2014-10-08 11:03:14 +02:00
}
Add http_response_stream() which can stream data from a buffer to the client.
2014-08-07 14:23:26 +02:00
}
Add filemaps. A filemap is a way of telling Kore to serve files from a directory much like a traditional webserver can do. Kore filemaps only handles files. Kore does not generate directory indexes or deal with non-regular files. The way files are sent to a client differs a bit per platform and build options: default: - mmap() backed file transfer due to TLS. NOTLS=1 - sendfile() under FreeBSD, macOS and Linux. - mmap() backed file for OpenBSD. The opened file descriptors/mmap'd regions are cached and reused when appropriate. If a file is no longer in use it will be closed and evicted from the cache after 30 seconds. New API's are available allowing developers to use these facilities via: void net_send_fileref(struct connection *, struct kore_fileref *); void http_response_fileref(struct http_request *, struct kore_fileref *); Kore will attempt to match media types based on file extensions. A few default types are built-in. Others can be added via the new "http_media_type" configuration directive.
2018-06-28 13:27:44 +02:00
void
http_response_fileref(struct http_request *req, int status,
struct kore_fileref *ref)
{
Add last-modified and if-modified-since for filemaps.
2018-06-29 09:56:04 +02:00
struct tm *tm;
time_t mtime;
char tbuf[128];
const char *media_type, *modified;
Add filemaps. A filemap is a way of telling Kore to serve files from a directory much like a traditional webserver can do. Kore filemaps only handles files. Kore does not generate directory indexes or deal with non-regular files. The way files are sent to a client differs a bit per platform and build options: default: - mmap() backed file transfer due to TLS. NOTLS=1 - sendfile() under FreeBSD, macOS and Linux. - mmap() backed file for OpenBSD. The opened file descriptors/mmap'd regions are cached and reused when appropriate. If a file is no longer in use it will be closed and evicted from the cache after 30 seconds. New API's are available allowing developers to use these facilities via: void net_send_fileref(struct connection *, struct kore_fileref *); void http_response_fileref(struct http_request *, struct kore_fileref *); Kore will attempt to match media types based on file extensions. A few default types are built-in. Others can be added via the new "http_media_type" configuration directive.
2018-06-28 13:27:44 +02:00
if (req->owner == NULL)
return;
media_type = http_media_type(ref->path);
if (media_type != NULL)
http_response_header(req, "content-type", media_type);
Add last-modified and if-modified-since for filemaps.
2018-06-29 09:56:04 +02:00
if (http_request_header(req, "if-modified-since", &modified)) {
mtime = kore_date_to_time(modified);
let filerefs to operate on ms precision for mtime.
2018-07-24 19:56:36 +02:00
if (mtime == ref->mtime_sec) {
Add last-modified and if-modified-since for filemaps.
2018-06-29 09:56:04 +02:00
kore_fileref_release(ref);
http_response(req, HTTP_STATUS_NOT_MODIFIED, NULL, 0);
return;
}
}
let filerefs to operate on ms precision for mtime.
2018-07-24 19:56:36 +02:00
if ((tm = gmtime(&ref->mtime_sec)) != NULL) {
Add last-modified and if-modified-since for filemaps.
2018-06-29 09:56:04 +02:00
if (strftime(tbuf, sizeof(tbuf),
"%a, %d %b %Y %H:%M:%S GMT", tm) > 0) {
http_response_header(req, "last-modified", tbuf);
}
}
Add filemaps. A filemap is a way of telling Kore to serve files from a directory much like a traditional webserver can do. Kore filemaps only handles files. Kore does not generate directory indexes or deal with non-regular files. The way files are sent to a client differs a bit per platform and build options: default: - mmap() backed file transfer due to TLS. NOTLS=1 - sendfile() under FreeBSD, macOS and Linux. - mmap() backed file for OpenBSD. The opened file descriptors/mmap'd regions are cached and reused when appropriate. If a file is no longer in use it will be closed and evicted from the cache after 30 seconds. New API's are available allowing developers to use these facilities via: void net_send_fileref(struct connection *, struct kore_fileref *); void http_response_fileref(struct http_request *, struct kore_fileref *); Kore will attempt to match media types based on file extensions. A few default types are built-in. Others can be added via the new "http_media_type" configuration directive.
2018-06-28 13:27:44 +02:00
req->status = status;
switch (req->owner->proto) {
case CONN_PROTO_HTTP:
http_response_normal(req, req->owner, status, NULL, ref->size);
break;
default:
fatal("http_response_fd() bad proto %d", req->owner->proto);
/* NOTREACHED. */
}
if (req->method != HTTP_METHOD_HEAD)
net_send_fileref(req->owner, ref);
else
kore_fileref_release(ref);
}
allow modules to add headers to the response before calling http_response() by calling http_response_header_add(). fix wrong overflow check in spdy_stream_get_header(). html_inject now exports last modified timestamp for the files that are compiled into the module in the format static_mtime_<type>_<name>. modules can now look into the request headers using http_request_header_get().
2013-05-01 21:16:09 +02:00
int
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
http_request_header(struct http_request *req, const char *header,
const char **out)
allow modules to add headers to the response before calling http_response() by calling http_response_header_add(). fix wrong overflow check in spdy_stream_get_header(). html_inject now exports last modified timestamp for the files that are compiled into the module in the format static_mtime_<type>_<name>. modules can now look into the request headers using http_request_header_get().
2013-05-01 21:16:09 +02:00
{
add very basic support for HTTP/1.1.
2013-05-02 03:51:04 +02:00
struct http_header *hdr;
allow modules to add headers to the response before calling http_response() by calling http_response_header_add(). fix wrong overflow check in spdy_stream_get_header(). html_inject now exports last modified timestamp for the files that are compiled into the module in the format static_mtime_<type>_<name>. modules can now look into the request headers using http_request_header_get().
2013-05-01 21:16:09 +02:00
Introduce NOHTTP=1 build option. This basically turns off the HTTP layer for Kore. It does not compile in anything for HTTP. This allows Kore to be used as a network application platform as well. Added an example for this called nohttp. Other changes that sneaked in while hacking on this: * Use calloc(), kill pendantic malloc option. * Killed off SPDY/3.1 support completely, will be superseded by http2 Note that comes with massive changes to a lot of the core API functions provided by Kore, these might break your application.
2015-11-27 16:22:50 +01:00
TAILQ_FOREACH(hdr, &(req->req_headers), list) {
if (!strcasecmp(hdr->header, header)) {
Change semantics for http_request_header(). The result returned by this function no longer needs to be freed by the caller.
2015-11-29 14:19:44 +01:00
*out = hdr->value;
Introduce NOHTTP=1 build option. This basically turns off the HTTP layer for Kore. It does not compile in anything for HTTP. This allows Kore to be used as a network application platform as well. Added an example for this called nohttp. Other changes that sneaked in while hacking on this: * Use calloc(), kill pendantic malloc option. * Killed off SPDY/3.1 support completely, will be superseded by http2 Note that comes with massive changes to a lot of the core API functions provided by Kore, these might break your application.
2015-11-27 16:22:50 +01:00
return (KORE_RESULT_OK);
add very basic support for HTTP/1.1.
2013-05-02 03:51:04 +02:00
}
allow modules to add headers to the response before calling http_response() by calling http_response_header_add(). fix wrong overflow check in spdy_stream_get_header(). html_inject now exports last modified timestamp for the files that are compiled into the module in the format static_mtime_<type>_<name>. modules can now look into the request headers using http_request_header_get().
2013-05-01 21:16:09 +02:00
}
Preserve the full host header under req->host. Additionally make this header available via http_request_header(). prompted by #184
2017-03-30 09:38:23 +02:00
if (!strcasecmp(header, "host")) {
*out = req->host;
return (KORE_RESULT_OK);
}
Introduce NOHTTP=1 build option. This basically turns off the HTTP layer for Kore. It does not compile in anything for HTTP. This allows Kore to be used as a network application platform as well. Added an example for this called nohttp. Other changes that sneaked in while hacking on this: * Use calloc(), kill pendantic malloc option. * Killed off SPDY/3.1 support completely, will be superseded by http2 Note that comes with massive changes to a lot of the core API functions provided by Kore, these might break your application.
2015-11-27 16:22:50 +01:00
return (KORE_RESULT_ERROR);
allow modules to add headers to the response before calling http_response() by calling http_response_header_add(). fix wrong overflow check in spdy_stream_get_header(). html_inject now exports last modified timestamp for the files that are compiled into the module in the format static_mtime_<type>_<name>. modules can now look into the request headers using http_request_header_get().
2013-05-01 21:16:09 +02:00
}
HTTP improvements. Introduce an on_headers callback for routes, allowing one to inspect the headers before the request is processed further. Additionall, Add a new way of obtaining HTTP headers. Much like http_argument_get_*() functions, these new APIs allow you to fetch the data of an HTTP header as a specified C type. The new APIs are: * http_request_header_int16() * http_request_header_uint16() * http_request_header_int32() * http_request_header_uint32() * http_request_header_int64() * http_request_header_uint64() * http_request_header_float() * http_request_header_double() Should make it easier to operate in HTTP header data in a safe way. No need to always roll your own string to int conversion functions.
2021-09-15 22:16:22 +02:00
int
http_request_header_get(struct http_request *req, const char *header,
void **out, void *nout, int type)
{
struct http_header *hdr;
if (type == HTTP_ARG_TYPE_STRING)
fatal("%s: cannot be called with type string", __func__);
TAILQ_FOREACH(hdr, &req->req_headers, list) {
if (strcasecmp(hdr->header, header))
continue;
if (http_data_convert(hdr->value, out, nout, type))
return (KORE_RESULT_OK);
return (KORE_RESULT_ERROR);
}
return (KORE_RESULT_ERROR);
}
Cookies and arguments parsing improvements (#166) Add new cookie API for handling of cookies.
2017-02-07 21:49:10 +01:00
int
http_request_cookie(struct http_request *req, const char *cookie, char **out)
{
struct http_cookie *ck;
TAILQ_FOREACH(ck, &(req->req_cookies), list) {
if (!strcasecmp(ck->name, cookie)) {
*out = ck->value;
return (KORE_RESULT_OK);
}
}
cookie improvements. - split up writing of cookies into its own function. - turn maxage into a signed int and use -1 for it not being set. - lots of style fixes - remove HTTP_COOKIE_DEFAULT, just pass 0 if you don't want flags.
2017-02-07 22:03:06 +01:00
return (KORE_RESULT_ERROR);
Cookies and arguments parsing improvements (#166) Add new cookie API for handling of cookies.
2017-02-07 21:49:10 +01:00
}
add very basic support for HTTP/1.1.
2013-05-02 03:51:04 +02:00
int
http_header_recv(struct netbuf *nb)
{
better http header validation.
2018-07-17 20:17:05 +02:00
struct connection *c;
Add http_header_max and http_postbody_max configuration variables. - http_header_max: Maximum size of HTTP headers (in non SPDY connections). - http_postbody_max: Maximum size of an HTTP POST body (both in SPDY and HTTP mode). Right now Kore will simply DC the client, ideally we want to send a 413 (entity too large) to the client however. See modules/examples/module.conf for more.
2013-09-22 20:05:24 +02:00
size_t len;
add very basic support for HTTP/1.1.
2013-05-02 03:51:04 +02:00
struct http_header *hdr;
struct http_request *req;
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
u_int8_t *end_headers;
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
int h, i, v, skip, l;
better http header validation.
2018-07-17 20:17:05 +02:00
char *headers[HTTP_REQ_HEADER_MAX];
char *value, *host, *request[4], *hbuf;
add very basic support for HTTP/1.1.
2013-05-02 03:51:04 +02:00
better http header validation.
2018-07-17 20:17:05 +02:00
c = nb->owner;
add very basic support for HTTP/1.1.
2013-05-02 03:51:04 +02:00
Send a 413 if header_recv buffer is full. Kore used to just stall the connection until the timeout kicked in, but if no proper headers were received by the time the header buffer is full we should just error out. While here, use s_off for the inital length check.
2022-09-07 12:41:38 +02:00
if (nb->s_off < 4)
No need to duplicate the received http buffer.
2013-07-16 09:33:46 +02:00
return (KORE_RESULT_OK);
- Add SPDY RST control frame handler. - Keep HTTP requests in connection, so we can delete them if the connection ends before the requests do (this way we don't leak them). - When spdy_stream_close() is called, delete the attached http request. (This shouldn't hurt to do, so hopefully won't cause major fallout). - When parsing HTTP, find the first occurence of end-of-headers so uploads with multipart/form-data can succeed properly. - Add a test upload page to the example module.
2013-09-09 10:59:56 +02:00
Add the on_body_chunk handler for routes. If set, will call a given handler with the prototype of `void body_chunk(struct http_request *req, const void *data, size_t len);` for each chunk of the received HTTP body, allowing a developer to handle it in their own way. The incoming body is still being handled and retained in the same way as before (in a kore_buf or temporary file). While here, allow HTTP_STATUS_CONTINUE to work via http_response() and make the handling of incoming HTTP header data a bit better.
2021-09-17 19:28:06 +02:00
if (!isalpha(nb->buf[0])) {
http_error_response(c, HTTP_STATUS_BAD_REQUEST);
return (KORE_RESULT_ERROR);
}
Do not miscalculate end_headers.
2014-04-09 09:21:22 +02:00
skip = 4;
Chain our sending netbufs together. Gives us a lot less SSL_write() calls.
2013-10-25 14:22:29 +02:00
end_headers = kore_mem_find(nb->buf, nb->s_off, "\r\n\r\n", 4);
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
if (end_headers == NULL) {
end_headers = kore_mem_find(nb->buf, nb->s_off, "\n\n", 2);
Send a 413 if header_recv buffer is full. Kore used to just stall the connection until the timeout kicked in, but if no proper headers were received by the time the header buffer is full we should just error out. While here, use s_off for the inital length check.
2022-09-07 12:41:38 +02:00
if (end_headers == NULL) {
if (nb->s_off == http_header_max) {
http_error_response(c,
HTTP_STATUS_REQUEST_ENTITY_TOO_LARGE);
return (KORE_RESULT_ERROR);
}
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
return (KORE_RESULT_OK);
Send a 413 if header_recv buffer is full. Kore used to just stall the connection until the timeout kicked in, but if no proper headers were received by the time the header buffer is full we should just error out. While here, use s_off for the inital length check.
2022-09-07 12:41:38 +02:00
}
Do not miscalculate end_headers.
2014-04-09 09:21:22 +02:00
skip = 2;
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
}
add very basic support for HTTP/1.1.
2013-05-02 03:51:04 +02:00
- Add SPDY RST control frame handler. - Keep HTTP requests in connection, so we can delete them if the connection ends before the requests do (this way we don't leak them). - When spdy_stream_close() is called, delete the attached http request. (This shouldn't hurt to do, so hopefully won't cause major fallout). - When parsing HTTP, find the first occurence of end-of-headers so uploads with multipart/form-data can succeed properly. - Add a test upload page to the example module.
2013-09-09 10:59:56 +02:00
*end_headers = '\0';
Do not miscalculate end_headers.
2014-04-09 09:21:22 +02:00
end_headers += skip;
support for POST in both spdy and normal connections
2013-05-03 07:35:11 +02:00
len = end_headers - nb->buf;
No need to duplicate the received http buffer.
2013-07-16 09:33:46 +02:00
hbuf = (char *)nb->buf;
add very basic support for HTTP/1.1.
2013-05-02 03:51:04 +02:00
h = kore_split_string(hbuf, "\r\n", headers, HTTP_REQ_HEADER_MAX);
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
if (h < 2) {
Stop hardcoding HTTP error codes in http_error_response() calls. Use predefined HTTP_STATUS_* macros instead.
2021-04-14 15:14:26 +02:00
http_error_response(c, HTTP_STATUS_BAD_REQUEST);
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
return (KORE_RESULT_OK);
}
add very basic support for HTTP/1.1.
2013-05-02 03:51:04 +02:00
v = kore_split_string(headers[0], " ", request, 4);
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
if (v != 3) {
Stop hardcoding HTTP error codes in http_error_response() calls. Use predefined HTTP_STATUS_* macros instead.
2021-04-14 15:14:26 +02:00
http_error_response(c, HTTP_STATUS_BAD_REQUEST);
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
return (KORE_RESULT_OK);
}
add very basic support for HTTP/1.1.
2013-05-02 03:51:04 +02:00
Do not miscalculate end_headers.
2014-04-09 09:21:22 +02:00
skip = 0;
Deal with the Host header in a way IPv6 hosts work Fixes #164.
2017-01-11 11:01:58 +01:00
host = NULL;
host header might not come in second..
2013-05-02 22:35:29 +02:00
for (i = 0; i < h; i++) {
Place path and host in http_request into pools. Also change the amount of times we end of doing unneccessairy strlen() calls.
2015-04-15 12:40:15 +02:00
if (strncasecmp(headers[i], "host", 4))
host header might not come in second..
2013-05-02 22:35:29 +02:00
continue;
better http header validation.
2018-07-17 20:17:05 +02:00
if ((host = http_validate_header(headers[i])) == NULL) {
Stop hardcoding HTTP error codes in http_error_response() calls. Use predefined HTTP_STATUS_* macros instead.
2021-04-14 15:14:26 +02:00
http_error_response(c, HTTP_STATUS_BAD_REQUEST);
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
return (KORE_RESULT_OK);
}
host header might not come in second..
2013-05-02 22:35:29 +02:00
Deal with the Host header in a way IPv6 hosts work Fixes #164.
2017-01-11 11:01:58 +01:00
if (*host == '\0') {
Stop hardcoding HTTP error codes in http_error_response() calls. Use predefined HTTP_STATUS_* macros instead.
2021-04-14 15:14:26 +02:00
http_error_response(c, HTTP_STATUS_BAD_REQUEST);
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
return (KORE_RESULT_OK);
}
host header might not come in second..
2013-05-02 22:35:29 +02:00
skip = i;
break;
add very basic support for HTTP/1.1.
2013-05-02 03:51:04 +02:00
}
Deal with the Host header in a way IPv6 hosts work Fixes #164.
2017-01-11 11:01:58 +01:00
if (host == NULL) {
Stop hardcoding HTTP error codes in http_error_response() calls. Use predefined HTTP_STATUS_* macros instead.
2021-04-14 15:14:26 +02:00
http_error_response(c, HTTP_STATUS_BAD_REQUEST);
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
return (KORE_RESULT_OK);
}
add very basic support for HTTP/1.1.
2013-05-02 03:51:04 +02:00
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
req = http_request_new(c, host, request[0], request[1], request[2]);
if (req == NULL)
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
return (KORE_RESULT_OK);
add very basic support for HTTP/1.1.
2013-05-02 03:51:04 +02:00
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
/* take full ownership of the buffer. */
req->headers = nb->buf;
nb->buf = NULL;
better fix for 5a5d9fd0. Don't let net_recv_flush() do things as long as the HTTP layer owns the buffer. When we have sent a response kick the read end back into gear ourselves by calling net_recv_flush().
2018-07-18 16:10:41 +02:00
nb->m_len = 0;
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
host header might not come in second..
2013-05-02 22:35:29 +02:00
for (i = 1; i < h; i++) {
if (i == skip)
continue;
better http header validation.
2018-07-17 20:17:05 +02:00
if ((value = http_validate_header(headers[i])) == NULL) {
req->flags |= HTTP_REQUEST_DELETE;
Stop hardcoding HTTP error codes in http_error_response() calls. Use predefined HTTP_STATUS_* macros instead.
2021-04-14 15:14:26 +02:00
http_error_response(c, HTTP_STATUS_BAD_REQUEST);
better http header validation.
2018-07-17 20:17:05 +02:00
return (KORE_RESULT_OK);
}
if (*value == '\0') {
req->flags |= HTTP_REQUEST_DELETE;
Stop hardcoding HTTP error codes in http_error_response() calls. Use predefined HTTP_STATUS_* macros instead.
2021-04-14 15:14:26 +02:00
http_error_response(c, HTTP_STATUS_BAD_REQUEST);
better http header validation.
2018-07-17 20:17:05 +02:00
return (KORE_RESULT_OK);
add very basic support for HTTP/1.1.
2013-05-02 03:51:04 +02:00
}
Add fixed size memory pools and use them throughout Kore.
2013-07-15 10:13:36 +02:00
hdr = kore_pool_get(&http_header_pool);
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
hdr->header = headers[i];
better http header validation.
2018-07-17 20:17:05 +02:00
hdr->value = value;
add very basic support for HTTP/1.1.
2013-05-02 03:51:04 +02:00
TAILQ_INSERT_TAIL(&(req->req_headers), hdr, list);
Add access logging to Kore.
2013-06-24 09:36:40 +02:00
if (req->agent == NULL &&
!strcasecmp(hdr->header, "user-agent"))
Correct usage of http_request_header(). Since latest change we no longer need free its result.
2015-11-29 17:22:30 +01:00
req->agent = hdr->value;
log referer in accesslog if present.
2018-06-29 22:37:48 +02:00
if (req->referer == NULL &&
!strcasecmp(hdr->header, "referer"))
req->referer = hdr->value;
add very basic support for HTTP/1.1.
2013-05-02 03:51:04 +02:00
}
Add PUT/DELETE/HEAD methods (finally). This commit renames certain POST centric variable and configuration naming to the correct HTTP body stuff. API changes include http_postbody_text() and http_postbody_bytes() to have become http_body_text() and http_body_bytes(). The developer is still responsible for validating the method their page handler is called with. Hopefully this becomes a configuration option soon enough.
2014-10-08 11:03:14 +02:00
if (req->flags & HTTP_REQUEST_EXPECT_BODY) {
Improve http_body_max directive a bit. Allow setting it to 0 which will disable HTTP requests that have a body (POST/PUT). Reduce default http_body_max to 1MB by default, 10MB seems large. Revisit to this code inspired by #100.
2016-01-08 17:54:40 +01:00
if (http_body_max == 0) {
req->flags |= HTTP_REQUEST_DELETE;
Stop hardcoding HTTP error codes in http_error_response() calls. Use predefined HTTP_STATUS_* macros instead.
2021-04-14 15:14:26 +02:00
http_error_response(req->owner,
HTTP_STATUS_METHOD_NOT_ALLOWED);
Improve http_body_max directive a bit. Allow setting it to 0 which will disable HTTP requests that have a body (POST/PUT). Reduce default http_body_max to 1MB by default, 10MB seems large. Revisit to this code inspired by #100.
2016-01-08 17:54:40 +01:00
return (KORE_RESULT_OK);
}
HTTP improvements. Introduce an on_headers callback for routes, allowing one to inspect the headers before the request is processed further. Additionall, Add a new way of obtaining HTTP headers. Much like http_argument_get_*() functions, these new APIs allow you to fetch the data of an HTTP header as a specified C type. The new APIs are: * http_request_header_int16() * http_request_header_uint16() * http_request_header_int32() * http_request_header_uint32() * http_request_header_int64() * http_request_header_uint64() * http_request_header_float() * http_request_header_double() Should make it easier to operate in HTTP header data in a safe way. No need to always roll your own string to int conversion functions.
2021-09-15 22:16:22 +02:00
if (!http_request_header_uint64(req, "content-length",
&req->content_length)) {
The DELETE method may have a request body.
2021-12-06 14:43:52 +01:00
if (req->method == HTTP_METHOD_DELETE) {
req->flags |= HTTP_REQUEST_COMPLETE;
return (KORE_RESULT_OK);
}
move to a worker based threading approach where we delegate http requests to workers in a round robin basis (later this should be swapped to find the laziest worker and assign the request to that instead).
2013-05-30 19:36:42 +02:00
req->flags |= HTTP_REQUEST_DELETE;
Stop hardcoding HTTP error codes in http_error_response() calls. Use predefined HTTP_STATUS_* macros instead.
2021-04-14 15:14:26 +02:00
http_error_response(req->owner,
HTTP_STATUS_LENGTH_REQUIRED);
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
return (KORE_RESULT_OK);
support for POST in both spdy and normal connections
2013-05-03 07:35:11 +02:00
}
Cut memory usage for incoming HTTP bodies in half. Kore pre-allocates a kore_buf for the full size of the incoming HTTP body ... but also was passing the full size to the net_recv_reset() function. Instead of this, properly read smaller chunks from the network and append them to the body buffer as they roll in.
2016-01-08 20:56:21 +01:00
if (req->content_length == 0) {
Fix a problem with http_timeout when a body is sent. When receiving an HTTP body, Kore never reset http_timeout once the transfer was done. This can result in a 408 being thrown by Kore while a request is activity running.
2022-09-01 12:09:15 +02:00
c->http_timeout = 0;
HTTP Post improvements for handlers. Don't crash if we get a Content-length:0 on POST and our handlers call http_populate_arguments().
2014-04-15 21:18:23 +02:00
req->flags |= HTTP_REQUEST_COMPLETE;
Add PUT/DELETE/HEAD methods (finally). This commit renames certain POST centric variable and configuration naming to the correct HTTP body stuff. API changes include http_postbody_text() and http_postbody_bytes() to have become http_body_text() and http_body_bytes(). The developer is still responsible for validating the method their page handler is called with. Hopefully this becomes a configuration option soon enough.
2014-10-08 11:03:14 +02:00
req->flags &= ~HTTP_REQUEST_EXPECT_BODY;
HTTP Post improvements for handlers. Don't crash if we get a Content-length:0 on POST and our handlers call http_populate_arguments().
2014-04-15 21:18:23 +02:00
return (KORE_RESULT_OK);
}
Cut memory usage for incoming HTTP bodies in half. Kore pre-allocates a kore_buf for the full size of the incoming HTTP body ... but also was passing the full size to the net_recv_reset() function. Instead of this, properly read smaller chunks from the network and append them to the body buffer as they roll in.
2016-01-08 20:56:21 +01:00
if (req->content_length > http_body_max) {
Add http_header_max and http_postbody_max configuration variables. - http_header_max: Maximum size of HTTP headers (in non SPDY connections). - http_postbody_max: Maximum size of an HTTP POST body (both in SPDY and HTTP mode). Right now Kore will simply DC the client, ideally we want to send a 413 (entity too large) to the client however. See modules/examples/module.conf for more.
2013-09-22 20:05:24 +02:00
req->flags |= HTTP_REQUEST_DELETE;
Stop hardcoding HTTP error codes in http_error_response() calls. Use predefined HTTP_STATUS_* macros instead.
2021-04-14 15:14:26 +02:00
http_error_response(req->owner,
HTTP_STATUS_REQUEST_ENTITY_TOO_LARGE);
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
return (KORE_RESULT_OK);
Add http_header_max and http_postbody_max configuration variables. - http_header_max: Maximum size of HTTP headers (in non SPDY connections). - http_postbody_max: Maximum size of an HTTP POST body (both in SPDY and HTTP mode). Right now Kore will simply DC the client, ideally we want to send a 413 (entity too large) to the client however. See modules/examples/module.conf for more.
2013-09-22 20:05:24 +02:00
}
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
req->http_body_length = req->content_length;
if (http_body_disk_offload > 0 &&
req->content_length > http_body_disk_offload) {
req->http_body_path = kore_pool_get(&http_body_path);
l = snprintf(req->http_body_path, HTTP_BODY_PATH_MAX,
"%s/http_body.XXXXXX", http_body_disk_path);
if (l == -1 || (size_t)l >= HTTP_BODY_PATH_MAX) {
req->flags |= HTTP_REQUEST_DELETE;
Stop hardcoding HTTP error codes in http_error_response() calls. Use predefined HTTP_STATUS_* macros instead.
2021-04-14 15:14:26 +02:00
http_error_response(req->owner,
HTTP_STATUS_INTERNAL_ERROR);
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
return (KORE_RESULT_ERROR);
}
req->http_body = NULL;
req->http_body_fd = mkstemp(req->http_body_path);
if (req->http_body_fd == -1) {
req->flags |= HTTP_REQUEST_DELETE;
Stop hardcoding HTTP error codes in http_error_response() calls. Use predefined HTTP_STATUS_* macros instead.
2021-04-14 15:14:26 +02:00
http_error_response(req->owner,
HTTP_STATUS_INTERNAL_ERROR);
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
return (KORE_RESULT_OK);
}
} else {
req->http_body_fd = -1;
kore_buf_create -> kore_buf_alloc.
2016-07-14 12:34:29 +02:00
req->http_body = kore_buf_alloc(req->content_length);
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
}
support for POST in both spdy and normal connections
2013-05-03 07:35:11 +02:00
Initial work splitting OpenSSL code away. This work moves all TLS / crypto related code into a tls_openssl.c file and adds a tls_none.c which contains just stubs. Allows compilation of Kore with TLS_BACKEND=none to remove building against OpenSSL. Also adds code for SHA1/SHA2 taken from openssh-portable so we don't depend on those being present anymore in libcrypto.
2022-02-17 13:45:28 +01:00
SHA256Init(&req->hashctx);
Add the on_body_chunk handler for routes. If set, will call a given handler with the prototype of `void body_chunk(struct http_request *req, const void *data, size_t len);` for each chunk of the received HTTP body, allowing a developer to handle it in their own way. The incoming body is still being handled and retained in the same way as before (in a kore_buf or temporary file). While here, allow HTTP_STATUS_CONTINUE to work via http_response() and make the handling of incoming HTTP header data a bit better.
2021-09-17 19:28:06 +02:00
c->http_timeout = http_body_timeout * 1000;
if (!http_body_update(req, end_headers, nb->s_off - len)) {
req->flags |= HTTP_REQUEST_DELETE;
http_error_response(req->owner,
HTTP_STATUS_INTERNAL_ERROR);
return (KORE_RESULT_OK);
If we already have all the POST data in the netbuf do not try and read more.
2013-07-10 13:34:42 +02:00
}
Add http_[header|body]_timeout. If the HTTP request headers or the HTTP body have not arrived before these timeouts expire, Kore will send a 408 back to the client.
2019-04-11 20:51:49 +02:00
} else {
c->http_timeout = 0;
support for POST in both spdy and normal connections
2013-05-03 07:35:11 +02:00
}
HTTP improvements. Introduce an on_headers callback for routes, allowing one to inspect the headers before the request is processed further. Additionall, Add a new way of obtaining HTTP headers. Much like http_argument_get_*() functions, these new APIs allow you to fetch the data of an HTTP header as a specified C type. The new APIs are: * http_request_header_int16() * http_request_header_uint16() * http_request_header_int32() * http_request_header_uint32() * http_request_header_int64() * http_request_header_uint64() * http_request_header_float() * http_request_header_double() Should make it easier to operate in HTTP header data in a safe way. No need to always roll your own string to int conversion functions.
2021-09-15 22:16:22 +02:00
if (req->rt->on_headers != NULL) {
if (!kore_runtime_http_request(req->rt->on_headers, req)) {
req->flags |= HTTP_REQUEST_DELETE;
return (KORE_RESULT_OK);
}
}
add very basic support for HTTP/1.1.
2013-05-02 03:51:04 +02:00
return (KORE_RESULT_OK);
}
add proper query argument parsing, modules must call http_populate_arguments() before using http_argument_lookup() to lookup a given argument. population uses the appropriate query string (from GET or POST) automatically.
2013-05-03 09:09:27 +02:00
int
Rework the way validation and param extraction works. - Parameter validation is now done only when http_process_*() is called and upon http_argument_add(). - You MUST have defined your params in a param block or they will be filtered out. - http_argument_lookup() is dead, welcome http_argument_get() and its brothers and sisters: http_argument_get_string() http_argument_get_uint16() http_argument_get_int16() http_argument_get_uint32() http_argument_get_int32() They will automatically do bounds checking on integers for you and return proper integers or a NUL-terminated string. - The http_argument_get* functions no longer create an additional copy of the string which you need to free. Easier going. - http_multiple_args() is dead, byebye - Make some stuff we don't want to share with the modules static.
2013-12-12 00:58:32 +01:00
http_argument_get(struct http_request *req, const char *name,
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
void **out, void *nout, int type)
add proper query argument parsing, modules must call http_populate_arguments() before using http_argument_lookup() to lookup a given argument. population uses the appropriate query string (from GET or POST) automatically.
2013-05-03 09:09:27 +02:00
{
struct http_arg *q;
TAILQ_FOREACH(q, &(req->arguments), list) {
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
if (strcmp(q->name, name))
continue;
HTTP improvements. Introduce an on_headers callback for routes, allowing one to inspect the headers before the request is processed further. Additionall, Add a new way of obtaining HTTP headers. Much like http_argument_get_*() functions, these new APIs allow you to fetch the data of an HTTP header as a specified C type. The new APIs are: * http_request_header_int16() * http_request_header_uint16() * http_request_header_int32() * http_request_header_uint32() * http_request_header_int64() * http_request_header_uint64() * http_request_header_float() * http_request_header_double() Should make it easier to operate in HTTP header data in a safe way. No need to always roll your own string to int conversion functions.
2021-09-15 22:16:22 +02:00
if (http_data_convert(q->s_value, out, nout, type))
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
return (KORE_RESULT_OK);
HTTP improvements. Introduce an on_headers callback for routes, allowing one to inspect the headers before the request is processed further. Additionall, Add a new way of obtaining HTTP headers. Much like http_argument_get_*() functions, these new APIs allow you to fetch the data of an HTTP header as a specified C type. The new APIs are: * http_request_header_int16() * http_request_header_uint16() * http_request_header_int32() * http_request_header_uint32() * http_request_header_int64() * http_request_header_uint64() * http_request_header_float() * http_request_header_double() Should make it easier to operate in HTTP header data in a safe way. No need to always roll your own string to int conversion functions.
2021-09-15 22:16:22 +02:00
break;
add proper query argument parsing, modules must call http_populate_arguments() before using http_argument_lookup() to lookup a given argument. population uses the appropriate query string (from GET or POST) automatically.
2013-05-03 09:09:27 +02:00
}
return (KORE_RESULT_ERROR);
}
Add http_argument_urldecode() which takes a string and decodes any url encoding done to it. Change kore_strtonum() to pass 0 to strtoll by default so we can use it to convert hex numbers (prepended with 0x) as well.
2013-08-13 14:18:47 +02:00
int
Change http_argument_urldecode(). Takes a new parameter now `qs`, if set to 1 it'll urldecode according to how its always done it before. If not set to 0 (for posts for example), the decoder will allow \n and \r in addition to the other ones.
2023-03-12 23:33:47 +01:00
http_argument_urldecode(char *arg, int url)
Add http_argument_urldecode() which takes a string and decodes any url encoding done to it. Change kore_strtonum() to pass 0 to strtoll by default so we can use it to convert hex numbers (prepended with 0x) as well.
2013-08-13 14:18:47 +02:00
{
u_int8_t v;
int err;
size_t len;
char *p, *in, h[5];
p = arg;
in = arg;
len = strlen(arg);
while (*p != '\0' && p < (arg + len)) {
if (*p == '+')
*p = ' ';
if (*p != '%') {
*in++ = *p++;
continue;
}
kore_debug() has been unused for years. Kill all useless messages, convert useful ones into kore_log() instead.
2022-08-18 15:20:55 +02:00
if ((p + 2) >= (arg + len))
Add http_argument_urldecode() which takes a string and decodes any url encoding done to it. Change kore_strtonum() to pass 0 to strtoll by default so we can use it to convert hex numbers (prepended with 0x) as well.
2013-08-13 14:18:47 +02:00
return (KORE_RESULT_ERROR);
Add casts for isxdigit and isspace. The argument must be representable as an unsigned char or EOF.
2020-09-08 19:19:56 +02:00
if (!isxdigit((unsigned char)*(p + 1)) ||
!isxdigit((unsigned char)*(p + 2))) {
Add http_argument_urldecode() which takes a string and decodes any url encoding done to it. Change kore_strtonum() to pass 0 to strtoll by default so we can use it to convert hex numbers (prepended with 0x) as well.
2013-08-13 14:18:47 +02:00
*in++ = *p++;
continue;
}
h[0] = '0';
h[1] = 'x';
h[2] = *(p + 1);
h[3] = *(p + 2);
h[4] = '\0';
catch more bad ranges in http_argument_urldecode()
2018-06-28 15:39:03 +02:00
v = kore_strtonum(h, 16, 0x0, 0xff, &err);
Add http_argument_urldecode() which takes a string and decodes any url encoding done to it. Change kore_strtonum() to pass 0 to strtoll by default so we can use it to convert hex numbers (prepended with 0x) as well.
2013-08-13 14:18:47 +02:00
if (err != KORE_RESULT_OK)
return (err);
Change http_argument_urldecode(). Takes a new parameter now `qs`, if set to 1 it'll urldecode according to how its always done it before. If not set to 0 (for posts for example), the decoder will allow \n and \r in addition to the other ones.
2023-03-12 23:33:47 +01:00
if (url) {
if (v <= 0x1f || v == 0x7f)
return (KORE_RESULT_ERROR);
} else {
if ((v <= 0x1f || v == 0x7f) &&
Tabs are ok when decoding post data.
2024-01-14 13:01:23 +01:00
(v != '\n' && v != '\r' && v != '\t'))
Change http_argument_urldecode(). Takes a new parameter now `qs`, if set to 1 it'll urldecode according to how its always done it before. If not set to 0 (for posts for example), the decoder will allow \n and \r in addition to the other ones.
2023-03-12 23:33:47 +01:00
return (KORE_RESULT_ERROR);
}
catch more bad ranges in http_argument_urldecode()
2018-06-28 15:39:03 +02:00
Add http_argument_urldecode() which takes a string and decodes any url encoding done to it. Change kore_strtonum() to pass 0 to strtoll by default so we can use it to convert hex numbers (prepended with 0x) as well.
2013-08-13 14:18:47 +02:00
*in++ = (char)v;
p += 3;
}
*in = '\0';
return (KORE_RESULT_OK);
}
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
struct http_file *
http_file_lookup(struct http_request *req, const char *name)
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
{
struct http_file *f;
TAILQ_FOREACH(f, &(req->files), list) {
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
if (!strcmp(f->name, name))
return (f);
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
}
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
return (NULL);
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
}
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
ssize_t
http_file_read(struct http_file *file, void *buf, size_t len)
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
{
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
ssize_t ret;
size_t toread, off;
if (file->length < file->offset)
return (-1);
if ((file->offset + len) < file->offset)
return (-1);
if ((file->position + file->offset) < file->position)
return (-1);
off = file->position + file->offset;
toread = MIN(len, (file->length - file->offset));
toread is unsigned, it won't ever fall < 0.
2017-03-13 11:02:46 +01:00
if (toread == 0)
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
return (0);
if (file->req->http_body_fd != -1) {
if (lseek(file->req->http_body_fd, off, SEEK_SET) == -1) {
kore_log(LOG_ERR, "http_file_read: lseek(%s): %s",
file->req->http_body_path, errno_s);
return (-1);
}
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
for (;;) {
ret = read(file->req->http_body_fd, buf, toread);
if (ret == -1) {
if (errno == EINTR)
continue;
kore_log(LOG_ERR, "failed to read %s: %s",
file->req->http_body_path, errno_s);
return (-1);
}
if (ret == 0)
return (0);
break;
}
} else if (file->req->http_body != NULL) {
if (off > file->req->http_body->length)
return (0);
memcpy(buf, file->req->http_body->data + off, toread);
ret = toread;
} else {
kore_log(LOG_ERR, "http_file_read: called without body");
return (-1);
Correct snprintf() usage
2014-04-23 16:29:58 +02:00
}
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
file->offset += (size_t)ret;
return (ret);
}
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
void
http_file_rewind(struct http_file *file)
{
file->offset = 0;
}
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
Change default http_cookie behaviour. We now default to httponly & secure for newly created cookies. This should've been the default all along. The http_response_cookie() no longer returns a pointer to http_cookie but rather takes it as a parameter and will populate the pointer with the newly created http_cookie if not NULL. Additionally http_response_cookie() automatically sets the domain based on the http_request passed into the function.
2017-03-10 14:20:40 +01:00
void
http_response_cookie(struct http_request *req, const char *name,
change type of maxage.
2017-03-10 14:36:51 +01:00
const char *val, const char *path, time_t expires, u_int32_t maxage,
screw it, rework interface for cookies. The only reason you would want to directly modify the cookie after creating it should be to unset the HTTPONLY or SECURE flags if that is what you *really* want to do. Change http_response_cookie() to take all required parameters instead of having to marshall those in yourself after. Now you set a sane default cookie in one shot: http_response_cookie(req, "key", "value", "/", 0, -1, NULL); Which would create a session cookie key=value for / under the current domain.
2017-03-10 14:31:08 +01:00
struct http_cookie **out)
Cookies and arguments parsing improvements (#166) Add new cookie API for handling of cookies.
2017-02-07 21:49:10 +01:00
{
cut off port from the domain when needed.
2018-04-24 20:11:13 +02:00
char *p;
Cookies and arguments parsing improvements (#166) Add new cookie API for handling of cookies.
2017-02-07 21:49:10 +01:00
struct http_cookie *ck;
cookie improvements. - split up writing of cookies into its own function. - turn maxage into a signed int and use -1 for it not being set. - lots of style fixes - remove HTTP_COOKIE_DEFAULT, just pass 0 if you don't want flags.
2017-02-07 22:03:06 +01:00
if (name == NULL || val == NULL)
fatal("http_response_cookie: invalid parameters");
Cookies and arguments parsing improvements (#166) Add new cookie API for handling of cookies.
2017-02-07 21:49:10 +01:00
ck = kore_pool_get(&http_cookie_pool);
screw it, rework interface for cookies. The only reason you would want to directly modify the cookie after creating it should be to unset the HTTPONLY or SECURE flags if that is what you *really* want to do. Change http_response_cookie() to take all required parameters instead of having to marshall those in yourself after. Now you set a sane default cookie in one shot: http_response_cookie(req, "key", "value", "/", 0, -1, NULL); Which would create a session cookie key=value for / under the current domain.
2017-03-10 14:31:08 +01:00
ck->maxage = maxage;
ck->expires = expires;
cookie improvements. - split up writing of cookies into its own function. - turn maxage into a signed int and use -1 for it not being set. - lots of style fixes - remove HTTP_COOKIE_DEFAULT, just pass 0 if you don't want flags.
2017-02-07 22:03:06 +01:00
ck->name = kore_strdup(name);
ck->value = kore_strdup(val);
Change default http_cookie behaviour. We now default to httponly & secure for newly created cookies. This should've been the default all along. The http_response_cookie() no longer returns a pointer to http_cookie but rather takes it as a parameter and will populate the pointer with the newly created http_cookie if not NULL. Additionally http_response_cookie() automatically sets the domain based on the http_request passed into the function.
2017-03-10 14:20:40 +01:00
ck->domain = kore_strdup(req->host);
ck->flags = HTTP_COOKIE_HTTPONLY | HTTP_COOKIE_SECURE;
Cookies and arguments parsing improvements (#166) Add new cookie API for handling of cookies.
2017-02-07 21:49:10 +01:00
cut off port from the domain when needed.
2018-04-24 20:11:13 +02:00
if ((p = strrchr(ck->domain, ':')) != NULL)
*p = '\0';
screw it, rework interface for cookies. The only reason you would want to directly modify the cookie after creating it should be to unset the HTTPONLY or SECURE flags if that is what you *really* want to do. Change http_response_cookie() to take all required parameters instead of having to marshall those in yourself after. Now you set a sane default cookie in one shot: http_response_cookie(req, "key", "value", "/", 0, -1, NULL); Which would create a session cookie key=value for / under the current domain.
2017-03-10 14:31:08 +01:00
if (path != NULL)
ck->path = kore_strdup(path);
else
ck->path = NULL;
Cookies and arguments parsing improvements (#166) Add new cookie API for handling of cookies.
2017-02-07 21:49:10 +01:00
TAILQ_INSERT_TAIL(&(req->resp_cookies), ck, list);
cookie improvements. - split up writing of cookies into its own function. - turn maxage into a signed int and use -1 for it not being set. - lots of style fixes - remove HTTP_COOKIE_DEFAULT, just pass 0 if you don't want flags.
2017-02-07 22:03:06 +01:00
Change default http_cookie behaviour. We now default to httponly & secure for newly created cookies. This should've been the default all along. The http_response_cookie() no longer returns a pointer to http_cookie but rather takes it as a parameter and will populate the pointer with the newly created http_cookie if not NULL. Additionally http_response_cookie() automatically sets the domain based on the http_request passed into the function.
2017-03-10 14:20:40 +01:00
if (out != NULL)
*out = ck;
Cookies and arguments parsing improvements (#166) Add new cookie API for handling of cookies.
2017-02-07 21:49:10 +01:00
}
void
http_populate_cookies(struct http_request *req)
{
cookie improvements. - split up writing of cookies into its own function. - turn maxage into a signed int and use -1 for it not being set. - lots of style fixes - remove HTTP_COOKIE_DEFAULT, just pass 0 if you don't want flags.
2017-02-07 22:03:06 +01:00
struct http_cookie *ck;
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
const char *hdr;
cookie improvements. - split up writing of cookies into its own function. - turn maxage into a signed int and use -1 for it not being set. - lots of style fixes - remove HTTP_COOKIE_DEFAULT, just pass 0 if you don't want flags.
2017-02-07 22:03:06 +01:00
int i, v, n;
char *c, *header, *pair[3];
char *cookies[HTTP_MAX_COOKIES];
Cookies and arguments parsing improvements (#166) Add new cookie API for handling of cookies.
2017-02-07 21:49:10 +01:00
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
if (!http_request_header(req, "cookie", &hdr))
cookie improvements. - split up writing of cookies into its own function. - turn maxage into a signed int and use -1 for it not being set. - lots of style fixes - remove HTTP_COOKIE_DEFAULT, just pass 0 if you don't want flags.
2017-02-07 22:03:06 +01:00
return;
Cookies and arguments parsing improvements (#166) Add new cookie API for handling of cookies.
2017-02-07 21:49:10 +01:00
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
header = kore_strdup(hdr);
Cookies and arguments parsing improvements (#166) Add new cookie API for handling of cookies.
2017-02-07 21:49:10 +01:00
v = kore_split_string(header, ";", cookies, HTTP_MAX_COOKIES);
for (i = 0; i < v; i++) {
cookie improvements. - split up writing of cookies into its own function. - turn maxage into a signed int and use -1 for it not being set. - lots of style fixes - remove HTTP_COOKIE_DEFAULT, just pass 0 if you don't want flags.
2017-02-07 22:03:06 +01:00
for (c = cookies[i]; isspace(*(unsigned char *)c); c++)
Cookies and arguments parsing improvements (#166) Add new cookie API for handling of cookies.
2017-02-07 21:49:10 +01:00
;
n = kore_split_string(c, "=", pair, 3);
cookie improvements. - split up writing of cookies into its own function. - turn maxage into a signed int and use -1 for it not being set. - lots of style fixes - remove HTTP_COOKIE_DEFAULT, just pass 0 if you don't want flags.
2017-02-07 22:03:06 +01:00
if (n != 2)
Cookies and arguments parsing improvements (#166) Add new cookie API for handling of cookies.
2017-02-07 21:49:10 +01:00
continue;
cookie improvements. - split up writing of cookies into its own function. - turn maxage into a signed int and use -1 for it not being set. - lots of style fixes - remove HTTP_COOKIE_DEFAULT, just pass 0 if you don't want flags.
2017-02-07 22:03:06 +01:00
Cookies and arguments parsing improvements (#166) Add new cookie API for handling of cookies.
2017-02-07 21:49:10 +01:00
ck = kore_pool_get(&http_cookie_pool);
ck->name = kore_strdup(pair[0]);
ck->value = kore_strdup(pair[1]);
TAILQ_INSERT_TAIL(&(req->req_cookies), ck, list);
}
cookie improvements. - split up writing of cookies into its own function. - turn maxage into a signed int and use -1 for it not being set. - lots of style fixes - remove HTTP_COOKIE_DEFAULT, just pass 0 if you don't want flags.
2017-02-07 22:03:06 +01:00
Cookies and arguments parsing improvements (#166) Add new cookie API for handling of cookies.
2017-02-07 21:49:10 +01:00
kore_free(header);
}
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
void
http_populate_post(struct http_request *req)
{
ssize_t ret;
Unbreak normal POST parsing. It was broken for > 2 fields due to recent large changes, unfortunately i didn't catch this until now.
2016-02-06 15:32:23 +01:00
int i, v;
struct kore_buf *body;
char data[BUFSIZ];
char *args[HTTP_MAX_QUERY_ARGS], *val[3], *string;
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
if (req->method != HTTP_METHOD_POST)
return;
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
Unbreak normal POST parsing. It was broken for > 2 fields due to recent large changes, unfortunately i didn't catch this until now.
2016-02-06 15:32:23 +01:00
if (req->http_body != NULL) {
body = NULL;
req->http_body->offset = req->content_length;
Change kore_buf_stringify() a bit. Takes a size_t pointer as its second argument now, if not NULL this will be populated with the length of the string that is being returned.
2016-06-02 07:08:19 +02:00
string = kore_buf_stringify(req->http_body, NULL);
set body length+offset to 0 when populating data. otherwise this isn't properly picked up by http_body_read() later if dealing with in-memory HTTP bodies and you get inconsistent behaviour.
2019-04-02 22:26:44 +02:00
req->http_body_length = 0;
req->http_body_offset = 0;
Unbreak normal POST parsing. It was broken for > 2 fields due to recent large changes, unfortunately i didn't catch this until now.
2016-02-06 15:32:23 +01:00
} else {
kore_buf_create -> kore_buf_alloc.
2016-07-14 12:34:29 +02:00
body = kore_buf_alloc(128);
Unbreak normal POST parsing. It was broken for > 2 fields due to recent large changes, unfortunately i didn't catch this until now.
2016-02-06 15:32:23 +01:00
for (;;) {
ret = http_body_read(req, data, sizeof(data));
if (ret == -1)
goto out;
if (ret == 0)
break;
kore_buf_append(body, data, ret);
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
}
Change kore_buf_stringify() a bit. Takes a size_t pointer as its second argument now, if not NULL this will be populated with the length of the string that is being returned.
2016-06-02 07:08:19 +02:00
string = kore_buf_stringify(body, NULL);
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
}
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
Unbreak normal POST parsing. It was broken for > 2 fields due to recent large changes, unfortunately i didn't catch this until now.
2016-02-06 15:32:23 +01:00
v = kore_split_string(string, "&", args, HTTP_MAX_QUERY_ARGS);
for (i = 0; i < v; i++) {
kore_split_string(args[i], "=", val, 3);
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
if (val[0] != NULL && val[1] != NULL)
do not http_argument_urldecode for multipart data.
2018-08-16 14:11:28 +02:00
http_argument_add(req, val[0], val[1], 0, 1);
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
}
Make sure opt always contains what we want
2014-08-13 14:32:49 +02:00
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
out:
Unbreak normal POST parsing. It was broken for > 2 fields due to recent large changes, unfortunately i didn't catch this until now.
2016-02-06 15:32:23 +01:00
if (body != NULL)
kore_buf_free(body);
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
}
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
void
Allow param blocks to be marked as "querystring" Before params get would mean querystring and anything else would just count toward a www-encoded body. Now you can prefix the params block with "qs" indicating that those configured parameters are allowed to occur in the query string regardless of the method used. This means you can do something like: params qs:post /uri { ... } to specify what the allowed parameters are in the querystring for a POST request towards /uri. inspired by and properly fixes #205.
2018-01-16 18:47:50 +01:00
http_populate_qs(struct http_request *req)
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
{
int i, v;
char *query, *args[HTTP_MAX_QUERY_ARGS], *val[3];
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
Allow param blocks to be marked as "querystring" Before params get would mean querystring and anything else would just count toward a www-encoded body. Now you can prefix the params block with "qs" indicating that those configured parameters are allowed to occur in the query string regardless of the method used. This means you can do something like: params qs:post /uri { ... } to specify what the allowed parameters are in the querystring for a POST request towards /uri. inspired by and properly fixes #205.
2018-01-16 18:47:50 +01:00
if (req->query_string == NULL)
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
return;
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
query = kore_strdup(req->query_string);
v = kore_split_string(query, "&", args, HTTP_MAX_QUERY_ARGS);
for (i = 0; i < v; i++) {
kore_split_string(args[i], "=", val, 3);
if (val[0] != NULL && val[1] != NULL)
do not http_argument_urldecode for multipart data.
2018-08-16 14:11:28 +02:00
http_argument_add(req, val[0], val[1], 1, 1);
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
}
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
Large changes to the memory subsystem in kore. - Change pools to use mmap() for allocating regions. - Change kore_malloc() to use pools for commonly sized objects. (split into multiple of 2 buckets, starting at 8 bytes up to 8192). - Rename kore_mem_free() to kore_free(). The preallocated pools will hold up to 128K of elements per block size. In case a larger object is to be allocated kore_malloc() will use malloc() instead.
2016-07-12 13:54:14 +02:00
kore_free(query);
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
}
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
void
http_populate_multipart_form(struct http_request *req)
{
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
const char *hdr;
remove unused variable.
2016-01-18 11:43:44 +01:00
int h, blen;
Use kore_bufs on the stack rather than the pools.
2018-01-15 22:31:54 +01:00
struct kore_buf in, out;
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
char *type, *val, *args[3];
char boundary[HTTP_BOUNDARY_MAX];
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
if (req->method != HTTP_METHOD_POST)
return;
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
if (!http_request_header(req, "content-type", &hdr))
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
return;
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
kore_buf_init(&in, 128);
kore_buf_init(&out, 128);
type = kore_strdup(hdr);
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
h = kore_split_string(type, ";", args, 3);
if (h != 2)
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
goto cleanup;
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
if (strcasecmp(args[0], "multipart/form-data"))
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
goto cleanup;
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
if ((val = strchr(args[1], '=')) == NULL)
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
goto cleanup;
add http_post_data_text() which returns the post data NUL-terminated to the caller.
2013-05-03 07:49:40 +02:00
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
val++;
blen = snprintf(boundary, sizeof(boundary), "--%s", val);
if (blen == -1 || (size_t)blen >= sizeof(boundary))
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
goto cleanup;
add http_post_data_text() which returns the post data NUL-terminated to the caller.
2013-05-03 07:49:40 +02:00
Use kore_bufs on the stack rather than the pools.
2018-01-15 22:31:54 +01:00
if (!multipart_find_data(&in, NULL, NULL, req, boundary, blen))
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
goto cleanup;
Make http_post_data_bytes() and http_post_data_text() available
2014-08-04 21:03:26 +02:00
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
for (;;) {
Use kore_bufs on the stack rather than the pools.
2018-01-15 22:31:54 +01:00
if (!multipart_find_data(&in, NULL, NULL, req, "\r\n", 2))
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
break;
Use kore_bufs on the stack rather than the pools.
2018-01-15 22:31:54 +01:00
if (in.offset < 4 && req->http_body_length == 0)
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
break;
Use kore_bufs on the stack rather than the pools.
2018-01-15 22:31:54 +01:00
if (!multipart_find_data(&in, &out, NULL, req, "\r\n\r\n", 4))
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
break;
Use kore_bufs on the stack rather than the pools.
2018-01-15 22:31:54 +01:00
if (!multipart_parse_headers(req, &in, &out, boundary, blen))
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
break;
Make http_post_data_bytes() and http_post_data_text() available
2014-08-04 21:03:26 +02:00
Use kore_bufs on the stack rather than the pools.
2018-01-15 22:31:54 +01:00
kore_buf_reset(&out);
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
}
Make http_post_data_bytes() and http_post_data_text() available
2014-08-04 21:03:26 +02:00
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
cleanup:
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
kore_free(type);
Use kore_bufs on the stack rather than the pools.
2018-01-15 22:31:54 +01:00
kore_buf_cleanup(&in);
kore_buf_cleanup(&out);
Make http_post_data_bytes() and http_post_data_text() available
2014-08-04 21:03:26 +02:00
}
Make http_body_rewind() public. Also let this function reset offset and lengths for http_body_read(). Make sure of this function in the python code so req.body can be called multiple times in succession.
2017-03-13 11:17:55 +01:00
int
http_body_rewind(struct http_request *req)
{
if (req->http_body_fd != -1) {
if (lseek(req->http_body_fd, 0, SEEK_SET) == -1) {
kore_log(LOG_ERR, "lseek(%s) failed: %s",
req->http_body_path, errno_s);
return (KORE_RESULT_ERROR);
}
Add async/await support for socket i/o in python. This means you can now do things like: resp = await koresock.recv(1024) await koresock.send(resp) directly from page handlers if they are defined as async. Adds lots more to the python goo such as fatalx(), bind_unix(), task_create() and socket_wrap().
2018-10-15 20:18:54 +02:00
} else if (req->http_body != NULL) {
Make http_body_rewind() public. Also let this function reset offset and lengths for http_body_read(). Make sure of this function in the python code so req.body can be called multiple times in succession.
2017-03-13 11:17:55 +01:00
kore_buf_reset(req->http_body);
}
req->http_body_offset = 0;
req->http_body_length = req->content_length;
return (KORE_RESULT_OK);
}
Calculate an md over the incoming HTTP body. This is calculated while the HTTP body is incoming over the wire, once the body is fully received the digest will be available for the page handlers to obtain. You can obtain a hex string for this md via http_body_digest() or dereferences the http_request and look at http_body_digest manually for the bytes.
2018-07-17 14:53:55 +02:00
int
http_body_digest(struct http_request *req, char *out, size_t len)
{
size_t idx;
int slen;
if (len != HTTP_BODY_DIGEST_STRLEN) {
Add printf format attributes and fix fallout
2022-08-17 13:16:18 +02:00
fatal("http_body_digest: bad len:%zu wanted:%u",
Calculate an md over the incoming HTTP body. This is calculated while the HTTP body is incoming over the wire, once the body is fully received the digest will be available for the page handlers to obtain. You can obtain a hex string for this md via http_body_digest() or dereferences the http_request and look at http_body_digest manually for the bytes.
2018-07-17 14:53:55 +02:00
len, HTTP_BODY_DIGEST_STRLEN);
}
if (!(req->flags & HTTP_REQUEST_COMPLETE))
return (KORE_RESULT_ERROR);
for (idx = 0; idx < sizeof(req->http_body_digest); idx++) {
slen = snprintf(out + (idx * 2), len - (idx * 2), "%02x",
req->http_body_digest[idx]);
if (slen == -1 || (size_t)slen >= len)
fatal("failed to create hex string");
}
return (KORE_RESULT_OK);
}
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
ssize_t
http_body_read(struct http_request *req, void *out, size_t len)
Make http_post_data_bytes() and http_post_data_text() available
2014-08-04 21:03:26 +02:00
{
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
ssize_t ret;
size_t toread;
toread = MIN(req->http_body_length, len);
toread is unsigned, it won't ever fall < 0.
2017-03-13 11:02:46 +01:00
if (toread == 0)
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
return (0);
if (req->http_body_fd != -1) {
for (;;) {
ret = read(req->http_body_fd, out, toread);
if (ret == -1) {
if (errno == EINTR)
continue;
kore_log(LOG_ERR, "failed to read %s: %s",
req->http_body_path, errno_s);
return (-1);
}
if (ret == 0)
return (0);
break;
}
} else if (req->http_body != NULL) {
memcpy(out,
(req->http_body->data + req->http_body->offset), toread);
req->http_body->offset += toread;
ret = toread;
} else {
kore_log(LOG_ERR, "http_body_read: called without body");
return (-1);
}
Make http_post_data_bytes() and http_post_data_text() available
2014-08-04 21:03:26 +02:00
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
req->http_body_length -= (size_t)ret;
req->http_body_offset += (size_t)ret;
Make http_post_data_bytes() and http_post_data_text() available
2014-08-04 21:03:26 +02:00
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
return (ret);
Make http_post_data_bytes() and http_post_data_text() available
2014-08-04 21:03:26 +02:00
}
Add a hopefully easy to use and optional fsm for page handlers In order to use this, define states for your page handler: struct http_state mystates[] = { { "PAGE_STATE_INIT", page_init }, { "PAGE_STATE_RESULT", page_result }, }; In your page handler you can then simply call http_state_run() with your states and http_request. This will cause Kore to start calling your state callbacks beginning at index 0. State callbacks have the same prototype as page handlers: int func(struct http_request *); However, unlike page handlers they MUST return one of the following: - HTTP_STATE_OK: All good, just continue the fsm. - HTTP_STATE_ERROR: Abort fsm and return KORE_RESULT_OK to Kore (This will cancel the http request). - HTTP_STATE_RETRY: Return KORE_RESULT_RETRY to Kore. (Kore will retry your page handler next event loop). - HTTP_STATE_COMPLETE: The fsm completed, break out cleanly. Note that using this is completely optional and you can still use the traditional way of writing page handlers. The fsm is designed to get rid of the clutter that exists today in Kore when dealing with non blocking tasks or pgsql calls.
2014-08-13 23:34:30 +02:00
int
http_state_run(struct http_state *states, u_int8_t elm,
struct http_request *req)
{
int r, done;
done = 0;
while (!done) {
if (req->fsm_state >= elm) {
fatal("http_state_run: fsm_state > elm (%d/%d)",
req->fsm_state, elm);
}
r = states[req->fsm_state].cb(req);
switch (r) {
case HTTP_STATE_ERROR:
return (KORE_RESULT_OK);
case HTTP_STATE_RETRY:
return (KORE_RESULT_RETRY);
Rename HTTP_STATE_OK to HTTP_STATE_CONTINUE
2014-08-14 11:20:04 +02:00
case HTTP_STATE_CONTINUE:
Add a hopefully easy to use and optional fsm for page handlers In order to use this, define states for your page handler: struct http_state mystates[] = { { "PAGE_STATE_INIT", page_init }, { "PAGE_STATE_RESULT", page_result }, }; In your page handler you can then simply call http_state_run() with your states and http_request. This will cause Kore to start calling your state callbacks beginning at index 0. State callbacks have the same prototype as page handlers: int func(struct http_request *); However, unlike page handlers they MUST return one of the following: - HTTP_STATE_OK: All good, just continue the fsm. - HTTP_STATE_ERROR: Abort fsm and return KORE_RESULT_OK to Kore (This will cancel the http request). - HTTP_STATE_RETRY: Return KORE_RESULT_RETRY to Kore. (Kore will retry your page handler next event loop). - HTTP_STATE_COMPLETE: The fsm completed, break out cleanly. Note that using this is completely optional and you can still use the traditional way of writing page handlers. The fsm is designed to get rid of the clutter that exists today in Kore when dealing with non blocking tasks or pgsql calls.
2014-08-13 23:34:30 +02:00
break;
case HTTP_STATE_COMPLETE:
done = 1;
break;
default:
fatal("http_state_run: unknown return value %d", r);
}
}
Reset fsm_state back to 0 after we're done with it.
2015-05-08 21:19:20 +02:00
req->fsm_state = 0;
Add a hopefully easy to use and optional fsm for page handlers In order to use this, define states for your page handler: struct http_state mystates[] = { { "PAGE_STATE_INIT", page_init }, { "PAGE_STATE_RESULT", page_result }, }; In your page handler you can then simply call http_state_run() with your states and http_request. This will cause Kore to start calling your state callbacks beginning at index 0. State callbacks have the same prototype as page handlers: int func(struct http_request *); However, unlike page handlers they MUST return one of the following: - HTTP_STATE_OK: All good, just continue the fsm. - HTTP_STATE_ERROR: Abort fsm and return KORE_RESULT_OK to Kore (This will cancel the http request). - HTTP_STATE_RETRY: Return KORE_RESULT_RETRY to Kore. (Kore will retry your page handler next event loop). - HTTP_STATE_COMPLETE: The fsm completed, break out cleanly. Note that using this is completely optional and you can still use the traditional way of writing page handlers. The fsm is designed to get rid of the clutter that exists today in Kore when dealing with non blocking tasks or pgsql calls.
2014-08-13 23:34:30 +02:00
return (KORE_RESULT_OK);
}
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
int
http_state_exists(struct http_request *req)
{
return (req->hdlr_extra != NULL);
}
void *
Python: allow route hooks via kore.route(). Adding the hooks keyword with a dictionary attached to specify the relevant hooks will hook them for the given route. Eg: domain.route("/", self.index, methods=["get"], hooks={ "on_free": self.request_free } ) These are the same hooks available via a normal Kore route configuration.
2021-12-14 23:15:21 +01:00
http_state_create(struct http_request *req, size_t len)
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
{
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
if (req->hdlr_extra != NULL)
fatal("http_state_create: state already exists");
req->state_len = len;
req->hdlr_extra = kore_calloc(1, len);
Decouple pgsql from the http layer. When the pgsql layer was introduced it was tightly coupled with the http layer in order to make async work fluently. The time has come to split these up and follow the same method we used for tasks, allowing either http requests to be tied to a pgsql data structure or a simple callback function. This also reworks the internal queueing of pgsql requests until connections to the db are available again. The following API functions were changes: - kore_pgsql_query_init() -> kore_pgsql_setup() no longer takes an http_request parameter. - NEW kore_pgsql_init() must be called before operating on an kore_pgsql structure. - NEW kore_pgsql_bind_request() binds an http_request to a kore_pgsql data structure. - NEW kore_pgsql_bind_callback() binds a callback to a kore_pgsql data structure. With all of this you can now build kore with PGSQL=1 NOHTTP=1. The pgsql/ example has been updated to reflect these changes and new features.
2017-03-24 12:53:07 +01:00
return (req->hdlr_extra);
}
void *
http_state_get(struct http_request *req)
{
return (req->hdlr_extra);
}
void
http_state_cleanup(struct http_request *req)
{
kore_free(req->hdlr_extra);
req->hdlr_extra = NULL;
}
allow an iterator to be passed to req.response(). if an iterator is passed kore will send the response with transfer-encoding: chunked and call the iterator for every chunk that was sent. The iterator must return a utf-8 string. Works wonderful with TemplateStream from jinja2.
2019-05-03 13:42:34 +02:00
void
http_start_recv(struct connection *c)
{
c->http_start = kore_time_ms();
c->http_timeout = http_header_timeout * 1000;
net_recv_reset(c, http_header_max, http_header_recv);
}
Add HTTP runlocks. A way to serialize access to HTTP page handlers in case you are using some asynchronous api such as pgsql or libcurl stuff.
2019-06-02 16:29:54 +02:00
void
http_runlock_init(struct http_runlock *lock)
{
lock->owner = NULL;
LIST_INIT(&lock->queue);
}
int
http_runlock_acquire(struct http_runlock *lock, struct http_request *req)
{
if (lock->owner != NULL) {
if (req->runlock != NULL)
fatal("%s: request already waiting on lock", __func__);
req->runlock = kore_pool_get(&http_rlq_pool);
req->runlock->req = req;
LIST_INSERT_HEAD(&lock->queue, req->runlock, list);
http_request_sleep(req);
return (KORE_RESULT_ERROR);
}
lock->owner = req;
return (KORE_RESULT_OK);
}
void
http_runlock_release(struct http_runlock *lock, struct http_request *req)
{
struct http_runlock_queue *next;
struct http_request *nextreq;
if (lock->owner != req)
fatal("%s: calling request != owner of runlock", __func__);
lock->owner = NULL;
if ((next = LIST_FIRST(&lock->queue)) != NULL) {
LIST_REMOVE(next, list);
nextreq = next->req;
nextreq->runlock = NULL;
http_request_wakeup(nextreq);
kore_pool_put(&http_rlq_pool, next);
}
}
Add support for config based redirection. Inside the domain contexts a 'redirect' rule will allow you to redirect a request to another URI. Ex: Redirect all requests with a 301 to example.com redirect ^/.*$ 301 https://example.com Using capture groups redirect ^/account/(.*)$ 301 https://example.com/account/$1 Adding the query string in the mix redirect ^/(.*)$ 301 https://example.com/$1?$qs
2020-02-07 06:42:33 +01:00
int
http_redirect_add(struct kore_domain *dom, const char *path, int status,
const char *target)
{
struct http_redirect *rdr;
rdr = kore_calloc(1, sizeof(*rdr));
if (regcomp(&(rdr->rctx), path, REG_EXTENDED)) {
kore_free(rdr);
return (KORE_RESULT_ERROR);
}
rdr->status = status;
Add a "return" configuration option. This hooks into the existing redirection framework but allows you to quickly deny certain paths with a 403 or other status code. The snippet below would for example declare a filemap served from 'www' directory but denying all access to the files under the 'data' directory: filemap / www deny /data 403
2020-03-03 11:28:16 +01:00
if (target != NULL)
rdr->target = kore_strdup(target);
else
rdr->target = NULL;
Add support for config based redirection. Inside the domain contexts a 'redirect' rule will allow you to redirect a request to another URI. Ex: Redirect all requests with a 301 to example.com redirect ^/.*$ 301 https://example.com Using capture groups redirect ^/account/(.*)$ 301 https://example.com/account/$1 Adding the query string in the mix redirect ^/(.*)$ 301 https://example.com/$1?$qs
2020-02-07 06:42:33 +01:00
TAILQ_INSERT_TAIL(&dom->redirects, rdr, list);
return (KORE_RESULT_OK);
}
Add http_response_json(). Since its an HTTP response function it functions like http_response() but takes a kore_json_item pointer that it will automatically convert to a kore_buf and send/free using http_response_stream(). While here fix a problem with http_response_stream() which could end up not calling the cb() in case of HTTP_METHOD_HEAD. Since the behaviour is that it should call cb() when done it should do so immediately.
2021-09-12 15:10:06 +02:00
const char *
http_status_text(int status)
Add support for config based redirection. Inside the domain contexts a 'redirect' rule will allow you to redirect a request to another URI. Ex: Redirect all requests with a 301 to example.com redirect ^/.*$ 301 https://example.com Using capture groups redirect ^/account/(.*)$ 301 https://example.com/account/$1 Adding the query string in the mix redirect ^/(.*)$ 301 https://example.com/$1?$qs
2020-02-07 06:42:33 +01:00
{
Add http_response_json(). Since its an HTTP response function it functions like http_response() but takes a kore_json_item pointer that it will automatically convert to a kore_buf and send/free using http_response_stream(). While here fix a problem with http_response_stream() which could end up not calling the cb() in case of HTTP_METHOD_HEAD. Since the behaviour is that it should call cb() when done it should do so immediately.
2021-09-12 15:10:06 +02:00
const char *r;
Add support for config based redirection. Inside the domain contexts a 'redirect' rule will allow you to redirect a request to another URI. Ex: Redirect all requests with a 301 to example.com redirect ^/.*$ 301 https://example.com Using capture groups redirect ^/account/(.*)$ 301 https://example.com/account/$1 Adding the query string in the mix redirect ^/(.*)$ 301 https://example.com/$1?$qs
2020-02-07 06:42:33 +01:00
Add http_response_json(). Since its an HTTP response function it functions like http_response() but takes a kore_json_item pointer that it will automatically convert to a kore_buf and send/free using http_response_stream(). While here fix a problem with http_response_stream() which could end up not calling the cb() in case of HTTP_METHOD_HEAD. Since the behaviour is that it should call cb() when done it should do so immediately.
2021-09-12 15:10:06 +02:00
switch (status) {
case HTTP_STATUS_CONTINUE:
r = "Continue";
break;
case HTTP_STATUS_SWITCHING_PROTOCOLS:
r = "Switching Protocols";
break;
case HTTP_STATUS_OK:
r = "OK";
break;
case HTTP_STATUS_CREATED:
r = "Created";
break;
case HTTP_STATUS_ACCEPTED:
r = "Accepted";
break;
case HTTP_STATUS_NON_AUTHORITATIVE:
r = "Non-Authoritative Information";
break;
case HTTP_STATUS_NO_CONTENT:
r = "No Content";
break;
case HTTP_STATUS_RESET_CONTENT:
r = "Reset Content";
break;
case HTTP_STATUS_PARTIAL_CONTENT:
r = "Partial Content";
break;
case HTTP_STATUS_MULTIPLE_CHOICES:
r = "Multiple Choices";
break;
case HTTP_STATUS_MOVED_PERMANENTLY:
r = "Moved Permanently";
break;
case HTTP_STATUS_FOUND:
r = "Found";
break;
case HTTP_STATUS_SEE_OTHER:
r = "See Other";
break;
case HTTP_STATUS_NOT_MODIFIED:
r = "Not Modified";
break;
case HTTP_STATUS_USE_PROXY:
r = "Use Proxy";
break;
case HTTP_STATUS_TEMPORARY_REDIRECT:
r = "Temporary Redirect";
break;
case HTTP_STATUS_BAD_REQUEST:
r = "Bad Request";
break;
case HTTP_STATUS_UNAUTHORIZED:
r = "Unauthorized";
break;
case HTTP_STATUS_PAYMENT_REQUIRED:
r = "Payment Required";
break;
case HTTP_STATUS_FORBIDDEN:
r = "Forbidden";
break;
case HTTP_STATUS_NOT_FOUND:
r = "Not Found";
break;
case HTTP_STATUS_METHOD_NOT_ALLOWED:
r = "Method Not Allowed";
break;
case HTTP_STATUS_NOT_ACCEPTABLE:
r = "Not Acceptable";
break;
case HTTP_STATUS_PROXY_AUTH_REQUIRED:
r = "Proxy Authentication Required";
break;
case HTTP_STATUS_REQUEST_TIMEOUT:
r = "Request Time-out";
break;
case HTTP_STATUS_CONFLICT:
r = "Conflict";
break;
case HTTP_STATUS_GONE:
r = "Gone";
break;
case HTTP_STATUS_LENGTH_REQUIRED:
r = "Length Required";
break;
case HTTP_STATUS_PRECONDITION_FAILED:
r = "Precondition Failed";
break;
case HTTP_STATUS_REQUEST_ENTITY_TOO_LARGE:
r = "Request Entity Too Large";
break;
case HTTP_STATUS_REQUEST_URI_TOO_LARGE:
r = "Request-URI Too Large";
break;
case HTTP_STATUS_UNSUPPORTED_MEDIA_TYPE:
r = "Unsupported Media Type";
break;
case HTTP_STATUS_REQUEST_RANGE_INVALID:
r = "Requested range not satisfiable";
break;
case HTTP_STATUS_EXPECTATION_FAILED:
r = "Expectation Failed";
break;
case HTTP_STATUS_MISDIRECTED_REQUEST:
r = "Misdirected Request";
break;
case HTTP_STATUS_INTERNAL_ERROR:
r = "Internal Server Error";
break;
case HTTP_STATUS_NOT_IMPLEMENTED:
r = "Not Implemented";
break;
case HTTP_STATUS_BAD_GATEWAY:
r = "Bad Gateway";
break;
case HTTP_STATUS_SERVICE_UNAVAILABLE:
r = "Service Unavailable";
break;
case HTTP_STATUS_GATEWAY_TIMEOUT:
r = "Gateway Time-out";
break;
case HTTP_STATUS_BAD_VERSION:
r = "HTTP Version not supported";
break;
default:
r = "";
break;
Add support for config based redirection. Inside the domain contexts a 'redirect' rule will allow you to redirect a request to another URI. Ex: Redirect all requests with a 301 to example.com redirect ^/.*$ 301 https://example.com Using capture groups redirect ^/account/(.*)$ 301 https://example.com/account/$1 Adding the query string in the mix redirect ^/(.*)$ 301 https://example.com/$1?$qs
2020-02-07 06:42:33 +01:00
}
Add http_response_json(). Since its an HTTP response function it functions like http_response() but takes a kore_json_item pointer that it will automatically convert to a kore_buf and send/free using http_response_stream(). While here fix a problem with http_response_stream() which could end up not calling the cb() in case of HTTP_METHOD_HEAD. Since the behaviour is that it should call cb() when done it should do so immediately.
2021-09-12 15:10:06 +02:00
return (r);
}
Add support for config based redirection. Inside the domain contexts a 'redirect' rule will allow you to redirect a request to another URI. Ex: Redirect all requests with a 301 to example.com redirect ^/.*$ 301 https://example.com Using capture groups redirect ^/account/(.*)$ 301 https://example.com/account/$1 Adding the query string in the mix redirect ^/(.*)$ 301 https://example.com/$1?$qs
2020-02-07 06:42:33 +01:00
Add http_response_json(). Since its an HTTP response function it functions like http_response() but takes a kore_json_item pointer that it will automatically convert to a kore_buf and send/free using http_response_stream(). While here fix a problem with http_response_stream() which could end up not calling the cb() in case of HTTP_METHOD_HEAD. Since the behaviour is that it should call cb() when done it should do so immediately.
2021-09-12 15:10:06 +02:00
const char *
http_method_text(int method)
{
char *r;
Add support for config based redirection. Inside the domain contexts a 'redirect' rule will allow you to redirect a request to another URI. Ex: Redirect all requests with a 301 to example.com redirect ^/.*$ 301 https://example.com Using capture groups redirect ^/account/(.*)$ 301 https://example.com/account/$1 Adding the query string in the mix redirect ^/(.*)$ 301 https://example.com/$1?$qs
2020-02-07 06:42:33 +01:00
Add http_response_json(). Since its an HTTP response function it functions like http_response() but takes a kore_json_item pointer that it will automatically convert to a kore_buf and send/free using http_response_stream(). While here fix a problem with http_response_stream() which could end up not calling the cb() in case of HTTP_METHOD_HEAD. Since the behaviour is that it should call cb() when done it should do so immediately.
2021-09-12 15:10:06 +02:00
switch(method) {
case HTTP_METHOD_GET:
r = "GET";
break;
case HTTP_METHOD_POST:
r = "POST";
break;
case HTTP_METHOD_PUT:
r = "PUT";
break;
case HTTP_METHOD_DELETE:
r = "DELETE";
break;
case HTTP_METHOD_HEAD:
r = "HEAD";
break;
case HTTP_METHOD_OPTIONS:
r = "OPTIONS";
break;
case HTTP_METHOD_PATCH:
r = "PATCH";
break;
default:
r = "";
break;
}
Add support for config based redirection. Inside the domain contexts a 'redirect' rule will allow you to redirect a request to another URI. Ex: Redirect all requests with a 301 to example.com redirect ^/.*$ 301 https://example.com Using capture groups redirect ^/account/(.*)$ 301 https://example.com/account/$1 Adding the query string in the mix redirect ^/(.*)$ 301 https://example.com/$1?$qs
2020-02-07 06:42:33 +01:00
Add http_response_json(). Since its an HTTP response function it functions like http_response() but takes a kore_json_item pointer that it will automatically convert to a kore_buf and send/free using http_response_stream(). While here fix a problem with http_response_stream() which could end up not calling the cb() in case of HTTP_METHOD_HEAD. Since the behaviour is that it should call cb() when done it should do so immediately.
2021-09-12 15:10:06 +02:00
return (r);
}
Add a "return" configuration option. This hooks into the existing redirection framework but allows you to quickly deny certain paths with a 403 or other status code. The snippet below would for example declare a filemap served from 'www' directory but denying all access to the files under the 'data' directory: filemap / www deny /data 403
2020-03-03 11:28:16 +01:00
Add http_response_json(). Since its an HTTP response function it functions like http_response() but takes a kore_json_item pointer that it will automatically convert to a kore_buf and send/free using http_response_stream(). While here fix a problem with http_response_stream() which could end up not calling the cb() in case of HTTP_METHOD_HEAD. Since the behaviour is that it should call cb() when done it should do so immediately.
2021-09-12 15:10:06 +02:00
int
http_method_value(const char *method)
{
if (!strcasecmp(method, "GET"))
return (HTTP_METHOD_GET);
Add a "return" configuration option. This hooks into the existing redirection framework but allows you to quickly deny certain paths with a 403 or other status code. The snippet below would for example declare a filemap served from 'www' directory but denying all access to the files under the 'data' directory: filemap / www deny /data 403
2020-03-03 11:28:16 +01:00
Add http_response_json(). Since its an HTTP response function it functions like http_response() but takes a kore_json_item pointer that it will automatically convert to a kore_buf and send/free using http_response_stream(). While here fix a problem with http_response_stream() which could end up not calling the cb() in case of HTTP_METHOD_HEAD. Since the behaviour is that it should call cb() when done it should do so immediately.
2021-09-12 15:10:06 +02:00
if (!strcasecmp(method, "POST"))
return (HTTP_METHOD_POST);
Add support for config based redirection. Inside the domain contexts a 'redirect' rule will allow you to redirect a request to another URI. Ex: Redirect all requests with a 301 to example.com redirect ^/.*$ 301 https://example.com Using capture groups redirect ^/account/(.*)$ 301 https://example.com/account/$1 Adding the query string in the mix redirect ^/(.*)$ 301 https://example.com/$1?$qs
2020-02-07 06:42:33 +01:00
Add http_response_json(). Since its an HTTP response function it functions like http_response() but takes a kore_json_item pointer that it will automatically convert to a kore_buf and send/free using http_response_stream(). While here fix a problem with http_response_stream() which could end up not calling the cb() in case of HTTP_METHOD_HEAD. Since the behaviour is that it should call cb() when done it should do so immediately.
2021-09-12 15:10:06 +02:00
if (!strcasecmp(method, "PUT"))
return (HTTP_METHOD_PUT);
Add support for config based redirection. Inside the domain contexts a 'redirect' rule will allow you to redirect a request to another URI. Ex: Redirect all requests with a 301 to example.com redirect ^/.*$ 301 https://example.com Using capture groups redirect ^/account/(.*)$ 301 https://example.com/account/$1 Adding the query string in the mix redirect ^/(.*)$ 301 https://example.com/$1?$qs
2020-02-07 06:42:33 +01:00
Add http_response_json(). Since its an HTTP response function it functions like http_response() but takes a kore_json_item pointer that it will automatically convert to a kore_buf and send/free using http_response_stream(). While here fix a problem with http_response_stream() which could end up not calling the cb() in case of HTTP_METHOD_HEAD. Since the behaviour is that it should call cb() when done it should do so immediately.
2021-09-12 15:10:06 +02:00
if (!strcasecmp(method, "DELETE"))
return (HTTP_METHOD_DELETE);
Add support for config based redirection. Inside the domain contexts a 'redirect' rule will allow you to redirect a request to another URI. Ex: Redirect all requests with a 301 to example.com redirect ^/.*$ 301 https://example.com Using capture groups redirect ^/account/(.*)$ 301 https://example.com/account/$1 Adding the query string in the mix redirect ^/(.*)$ 301 https://example.com/$1?$qs
2020-02-07 06:42:33 +01:00
Add http_response_json(). Since its an HTTP response function it functions like http_response() but takes a kore_json_item pointer that it will automatically convert to a kore_buf and send/free using http_response_stream(). While here fix a problem with http_response_stream() which could end up not calling the cb() in case of HTTP_METHOD_HEAD. Since the behaviour is that it should call cb() when done it should do so immediately.
2021-09-12 15:10:06 +02:00
if (!strcasecmp(method, "HEAD"))
return (HTTP_METHOD_HEAD);
Add support for config based redirection. Inside the domain contexts a 'redirect' rule will allow you to redirect a request to another URI. Ex: Redirect all requests with a 301 to example.com redirect ^/.*$ 301 https://example.com Using capture groups redirect ^/account/(.*)$ 301 https://example.com/account/$1 Adding the query string in the mix redirect ^/(.*)$ 301 https://example.com/$1?$qs
2020-02-07 06:42:33 +01:00
Add http_response_json(). Since its an HTTP response function it functions like http_response() but takes a kore_json_item pointer that it will automatically convert to a kore_buf and send/free using http_response_stream(). While here fix a problem with http_response_stream() which could end up not calling the cb() in case of HTTP_METHOD_HEAD. Since the behaviour is that it should call cb() when done it should do so immediately.
2021-09-12 15:10:06 +02:00
if (!strcasecmp(method, "OPTIONS"))
return (HTTP_METHOD_OPTIONS);
Add support for config based redirection. Inside the domain contexts a 'redirect' rule will allow you to redirect a request to another URI. Ex: Redirect all requests with a 301 to example.com redirect ^/.*$ 301 https://example.com Using capture groups redirect ^/account/(.*)$ 301 https://example.com/account/$1 Adding the query string in the mix redirect ^/(.*)$ 301 https://example.com/$1?$qs
2020-02-07 06:42:33 +01:00
Add http_response_json(). Since its an HTTP response function it functions like http_response() but takes a kore_json_item pointer that it will automatically convert to a kore_buf and send/free using http_response_stream(). While here fix a problem with http_response_stream() which could end up not calling the cb() in case of HTTP_METHOD_HEAD. Since the behaviour is that it should call cb() when done it should do so immediately.
2021-09-12 15:10:06 +02:00
if (!strcasecmp(method, "PATCH"))
return (HTTP_METHOD_PATCH);
Add a "return" configuration option. This hooks into the existing redirection framework but allows you to quickly deny certain paths with a 403 or other status code. The snippet below would for example declare a filemap served from 'www' directory but denying all access to the files under the 'data' directory: filemap / www deny /data 403
2020-03-03 11:28:16 +01:00
Add http_response_json(). Since its an HTTP response function it functions like http_response() but takes a kore_json_item pointer that it will automatically convert to a kore_buf and send/free using http_response_stream(). While here fix a problem with http_response_stream() which could end up not calling the cb() in case of HTTP_METHOD_HEAD. Since the behaviour is that it should call cb() when done it should do so immediately.
2021-09-12 15:10:06 +02:00
return (0);
Add support for config based redirection. Inside the domain contexts a 'redirect' rule will allow you to redirect a request to another URI. Ex: Redirect all requests with a 301 to example.com redirect ^/.*$ 301 https://example.com Using capture groups redirect ^/account/(.*)$ 301 https://example.com/account/$1 Adding the query string in the mix redirect ^/(.*)$ 301 https://example.com/$1?$qs
2020-02-07 06:42:33 +01:00
}
Add http_response_json(). Since its an HTTP response function it functions like http_response() but takes a kore_json_item pointer that it will automatically convert to a kore_buf and send/free using http_response_stream(). While here fix a problem with http_response_stream() which could end up not calling the cb() in case of HTTP_METHOD_HEAD. Since the behaviour is that it should call cb() when done it should do so immediately.
2021-09-12 15:10:06 +02:00
int
http_media_register(const char *ext, const char *type)
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
{
Add http_response_json(). Since its an HTTP response function it functions like http_response() but takes a kore_json_item pointer that it will automatically convert to a kore_buf and send/free using http_response_stream(). While here fix a problem with http_response_stream() which could end up not calling the cb() in case of HTTP_METHOD_HEAD. Since the behaviour is that it should call cb() when done it should do so immediately.
2021-09-12 15:10:06 +02:00
struct http_media_type *media;
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
Add http_response_json(). Since its an HTTP response function it functions like http_response() but takes a kore_json_item pointer that it will automatically convert to a kore_buf and send/free using http_response_stream(). While here fix a problem with http_response_stream() which could end up not calling the cb() in case of HTTP_METHOD_HEAD. Since the behaviour is that it should call cb() when done it should do so immediately.
2021-09-12 15:10:06 +02:00
LIST_FOREACH(media, &http_media_types, list) {
if (!strcasecmp(media->ext, ext))
return (KORE_RESULT_ERROR);
}
media = kore_calloc(1, sizeof(*media));
media->ext = kore_strdup(ext);
media->type = kore_strdup(type);
LIST_INSERT_HEAD(&http_media_types, media, list);
return (KORE_RESULT_OK);
}
const char *
http_media_type(const char *path)
{
const char *p;
struct http_media_type *media;
if ((p = strrchr(path, '.')) == NULL)
return (NULL);
p++;
if (*p == '\0')
return (NULL);
LIST_FOREACH(media, &http_media_types, list) {
if (!strcasecmp(media->ext, p))
return (media->type);
}
return (NULL);
}
char *
http_validate_header(char *header)
{
u_int8_t idx;
char *p, *value;
for (p = header; *p != '\0'; p++) {
idx = *p;
if (idx > HTTP_MAP_LIMIT)
return (NULL);
if (*p == ':') {
*(p)++ = '\0';
break;
}
Force lowercase on internal HTTP header names.
2022-12-29 12:58:21 +01:00
if (*p >= 'A' && *p <= 'Z')
*p += 32;
Add http_response_json(). Since its an HTTP response function it functions like http_response() but takes a kore_json_item pointer that it will automatically convert to a kore_buf and send/free using http_response_stream(). While here fix a problem with http_response_stream() which could end up not calling the cb() in case of HTTP_METHOD_HEAD. Since the behaviour is that it should call cb() when done it should do so immediately.
2021-09-12 15:10:06 +02:00
if (http_token[idx] == 0x00)
return (NULL);
}
while (isspace(*(unsigned char *)p))
p++;
if (*p == '\0')
return (NULL);
value = p;
while (*p != '\0') {
idx = *p;
if (idx > HTTP_MAP_LIMIT)
return (NULL);
if (http_field_content[idx] == 0x00)
return (NULL);
p++;
}
return (value);
}
static int
http_release_buffer(struct netbuf *nb)
{
kore_buf_free(nb->extra);
return (KORE_RESULT_OK);
}
static int
http_check_redirect(struct http_request *req, struct kore_domain *dom)
{
int idx;
struct http_redirect *rdr;
const char *uri;
char key[4];
struct kore_buf location;
TAILQ_FOREACH(rdr, &dom->redirects, list) {
if (!regexec(&(rdr->rctx), req->path,
HTTP_CAPTURE_GROUPS, req->cgroups, 0))
break;
}
if (rdr == NULL)
return (KORE_RESULT_ERROR);
uri = NULL;
kore_buf_init(&location, 128);
if (rdr->target) {
kore_buf_appendf(&location, "%s", rdr->target);
if (req->query_string != NULL) {
kore_buf_replace_string(&location, "$qs",
req->query_string, strlen(req->query_string));
}
/* Starts at 1 to skip the full path. */
for (idx = 1; idx < HTTP_CAPTURE_GROUPS - 1; idx++) {
if (req->cgroups[idx].rm_so == -1 ||
req->cgroups[idx].rm_eo == -1)
break;
(void)snprintf(key, sizeof(key), "$%d", idx);
kore_buf_replace_string(&location, key,
req->path + req->cgroups[idx].rm_so,
req->cgroups[idx].rm_eo - req->cgroups[idx].rm_so);
}
uri = kore_buf_stringify(&location, NULL);
}
if (uri)
http_response_header(req, "location", uri);
http_response(req, rdr->status, NULL, 0);
kore_buf_cleanup(&location);
if (dom->accesslog)
kore_accesslog(req);
return (KORE_RESULT_OK);
}
static struct http_request *
http_request_new(struct connection *c, const char *host,
const char *method, char *path, const char *version)
{
struct kore_domain *dom;
struct http_request *req;
Remove dead assignment and unused vars.
2021-09-17 19:49:32 +02:00
size_t qsoff;
Add http_response_json(). Since its an HTTP response function it functions like http_response() but takes a kore_json_item pointer that it will automatically convert to a kore_buf and send/free using http_response_stream(). While here fix a problem with http_response_stream() which could end up not calling the cb() in case of HTTP_METHOD_HEAD. Since the behaviour is that it should call cb() when done it should do so immediately.
2021-09-12 15:10:06 +02:00
char *p, *hp;
int m, flags, exists;
if (http_request_count >= http_request_limit) {
http_error_response(c, HTTP_STATUS_SERVICE_UNAVAILABLE);
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
return (NULL);
}
Remove dead assignment and unused vars.
2021-09-17 19:49:32 +02:00
if (strlen(host) >= KORE_DOMAINNAME_LEN - 1) {
Stop hardcoding HTTP error codes in http_error_response() calls. Use predefined HTTP_STATUS_* macros instead.
2021-04-14 15:14:26 +02:00
http_error_response(c, HTTP_STATUS_BAD_REQUEST);
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
return (NULL);
}
Remove dead assignment and unused vars.
2021-09-17 19:49:32 +02:00
if (strlen(path) >= HTTP_URI_LEN - 1) {
Stop hardcoding HTTP error codes in http_error_response() calls. Use predefined HTTP_STATUS_* macros instead.
2021-04-14 15:14:26 +02:00
http_error_response(c, HTTP_STATUS_REQUEST_URI_TOO_LARGE);
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
return (NULL);
}
if (strcasecmp(version, "http/1.1")) {
Some things still talk http/1.0.
2018-10-26 21:24:51 +02:00
if (strcasecmp(version, "http/1.0")) {
Stop hardcoding HTTP error codes in http_error_response() calls. Use predefined HTTP_STATUS_* macros instead.
2021-04-14 15:14:26 +02:00
http_error_response(c, HTTP_STATUS_BAD_VERSION);
Some things still talk http/1.0.
2018-10-26 21:24:51 +02:00
return (NULL);
}
flags = HTTP_VERSION_1_0;
} else {
flags = HTTP_VERSION_1_1;
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
}
if ((p = strchr(path, '?')) != NULL) {
qsoff = p - path;
} else {
qsoff = 0;
}
hp = NULL;
allow kore to bind to unix sockets via bind_unix.
2018-10-07 20:49:16 +02:00
switch (c->family) {
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
case AF_INET6:
if (*host == '[') {
if ((hp = strrchr(host, ']')) == NULL) {
Stop hardcoding HTTP error codes in http_error_response() calls. Use predefined HTTP_STATUS_* macros instead.
2021-04-14 15:14:26 +02:00
http_error_response(c, HTTP_STATUS_BAD_REQUEST);
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
return (NULL);
}
hp++;
if (*hp == ':')
*hp = '\0';
else
hp = NULL;
}
break;
default:
if ((hp = strrchr(host, ':')) != NULL)
*hp = '\0';
break;
}
TLS improvements. These changes improve the constraint kore had with client authentication and multiple domains. - Add kore_x509_subject_name() which will return a C string containing the x509 subject name in full (in utf8). - Log TLS errors if client authentication was turned on, will help debug issues with client authentication in the future. - If SNI was present in the TLS handshake, check it against the host specified in the HTTP request and send a 421 in case they mismatch. - Throw a 403 if client authentication was enabled but no client certificate was specified.
2019-11-19 11:09:24 +01:00
if (c->owner->server->tls && c->tls_sni != NULL) {
if (strcasecmp(c->tls_sni, host)) {
http_error_response(c, HTTP_STATUS_MISDIRECTED_REQUEST);
return (NULL);
}
}
if ((dom = kore_domain_lookup(c->owner->server, host)) == NULL) {
http_error_response(c, HTTP_STATUS_NOT_FOUND);
return (NULL);
}
rename ssl prefixed things to tls.
2022-02-18 10:20:28 +01:00
if (dom->cafile != NULL && c->tls_cert == NULL) {
TLS improvements. These changes improve the constraint kore had with client authentication and multiple domains. - Add kore_x509_subject_name() which will return a C string containing the x509 subject name in full (in utf8). - Log TLS errors if client authentication was turned on, will help debug issues with client authentication in the future. - If SNI was present in the TLS handshake, check it against the host specified in the HTTP request and send a 421 in case they mismatch. - Throw a 403 if client authentication was enabled but no client certificate was specified.
2019-11-19 11:09:24 +01:00
http_error_response(c, HTTP_STATUS_FORBIDDEN);
return (NULL);
}
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
if (hp != NULL)
*hp = ':';
if (!strcasecmp(method, "get")) {
m = HTTP_METHOD_GET;
Some things still talk http/1.0.
2018-10-26 21:24:51 +02:00
flags |= HTTP_REQUEST_COMPLETE;
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
} else if (!strcasecmp(method, "delete")) {
m = HTTP_METHOD_DELETE;
The DELETE method may have a request body.
2021-12-06 14:43:52 +01:00
flags |= HTTP_REQUEST_EXPECT_BODY;
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
} else if (!strcasecmp(method, "post")) {
m = HTTP_METHOD_POST;
Some things still talk http/1.0.
2018-10-26 21:24:51 +02:00
flags |= HTTP_REQUEST_EXPECT_BODY;
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
} else if (!strcasecmp(method, "put")) {
m = HTTP_METHOD_PUT;
Some things still talk http/1.0.
2018-10-26 21:24:51 +02:00
flags |= HTTP_REQUEST_EXPECT_BODY;
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
} else if (!strcasecmp(method, "head")) {
m = HTTP_METHOD_HEAD;
Some things still talk http/1.0.
2018-10-26 21:24:51 +02:00
flags |= HTTP_REQUEST_COMPLETE;
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
} else if (!strcasecmp(method, "options")) {
m = HTTP_METHOD_OPTIONS;
Some things still talk http/1.0.
2018-10-26 21:24:51 +02:00
flags |= HTTP_REQUEST_COMPLETE;
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
} else if (!strcasecmp(method, "patch")) {
m = HTTP_METHOD_PATCH;
Some things still talk http/1.0.
2018-10-26 21:24:51 +02:00
flags |= HTTP_REQUEST_EXPECT_BODY;
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
} else {
Stop hardcoding HTTP error codes in http_error_response() calls. Use predefined HTTP_STATUS_* macros instead.
2021-04-14 15:14:26 +02:00
http_error_response(c, HTTP_STATUS_BAD_REQUEST);
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
return (NULL);
}
Some things still talk http/1.0.
2018-10-26 21:24:51 +02:00
if (flags & HTTP_VERSION_1_0) {
if (m != HTTP_METHOD_GET && m != HTTP_METHOD_POST &&
m != HTTP_METHOD_HEAD) {
http_error_response(c, HTTP_STATUS_METHOD_NOT_ALLOWED);
return (NULL);
}
}
Add support for config based redirection. Inside the domain contexts a 'redirect' rule will allow you to redirect a request to another URI. Ex: Redirect all requests with a 301 to example.com redirect ^/.*$ 301 https://example.com Using capture groups redirect ^/account/(.*)$ 301 https://example.com/account/$1 Adding the query string in the mix redirect ^/(.*)$ 301 https://example.com/$1?$qs
2020-02-07 06:42:33 +01:00
req = kore_pool_get(&http_request_pool);
Allow restriction of methods for paths. Now Kore will automatically send a 400 bad request in case the method was not allowed on the path.
2018-07-17 14:23:57 +02:00
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
req->end = 0;
req->total = 0;
req->start = 0;
req->owner = c;
req->status = 0;
req->method = m;
req->agent = NULL;
set referer to NULL in http_request_new().
2018-07-05 07:02:49 +02:00
req->referer = NULL;
Add HTTP runlocks. A way to serialize access to HTTP page handlers in case you are using some asynchronous api such as pgsql or libcurl stuff.
2019-06-02 16:29:54 +02:00
req->runlock = NULL;
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
req->flags = flags;
req->fsm_state = 0;
req->http_body = NULL;
req->http_body_fd = -1;
req->hdlr_extra = NULL;
Be sure content_length is 0, just in case.
2021-09-17 19:56:35 +02:00
req->content_length = 0;
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
req->query_string = NULL;
req->http_body_length = 0;
req->http_body_offset = 0;
req->http_body_path = NULL;
req->host = host;
req->path = path;
several python improvements. - add kore.time() as equivalent for kore_time_ms(). - call waitpid() until no more children are available for reaping otherwise we risk missing a process if several die at the same time and only one SIGCHLD is delivered to us. - drain a RECV socket operation if eof is set but no exception was given.
2018-10-30 20:28:27 +01:00
#if defined(KORE_USE_PYTHON)
Allow kore.prerequests to be async.
2019-06-07 21:06:54 +02:00
req->py_req = NULL;
several python improvements. - add kore.time() as equivalent for kore_time_ms(). - call waitpid() until no more children are available for reaping otherwise we risk missing a process if several die at the same time and only one SIGCHLD is delivered to us. - drain a RECV socket operation if eof is set but no exception was given.
2018-10-30 20:28:27 +01:00
req->py_coro = NULL;
Allow kore.prerequests to be async.
2019-06-07 21:06:54 +02:00
req->py_rqnext = NULL;
Many many Python improvements. - Kore can now fully be configured via Python code if one wants nothing to do with configuration files. - Kore can now start single python files and no longer requires them to be inside a module directory. - Pass all regex capture groups to the handler methods, allowing you to get access to them immediately. - Change python websocket_handshake to take callable objects directly. - Added a new deployment configuration option. If set to "dev" or "development" Kore will automatically foreground, no chroot / etc. If set to "production" Kore *will* chroot, drop privs, etc. - Many more.. These are all backported from a project that I was working on a while ago. I decided these should go back into mainline Kore.
2019-09-26 15:49:00 +02:00
req->py_validator = NULL;
several python improvements. - add kore.time() as equivalent for kore_time_ms(). - call waitpid() until no more children are available for reaping otherwise we risk missing a process if several die at the same time and only one SIGCHLD is delivered to us. - drain a RECV socket operation if eof is set but no exception was given.
2018-10-30 20:28:27 +01:00
#endif
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
if (qsoff > 0) {
req->query_string = path + qsoff;
*(req->query_string)++ = '\0';
} else {
req->query_string = NULL;
}
Add support for config based redirection. Inside the domain contexts a 'redirect' rule will allow you to redirect a request to another URI. Ex: Redirect all requests with a 301 to example.com redirect ^/.*$ 301 https://example.com Using capture groups redirect ^/account/(.*)$ 301 https://example.com/account/$1 Adding the query string in the mix redirect ^/(.*)$ 301 https://example.com/$1?$qs
2020-02-07 06:42:33 +01:00
/* Checked further down below if we need to 404. */
Change how routes are configured in Kore. Routes are now configured in a context per route: route /path { handler handler_name methods get post head validate qs:get id v_id } All route related configurations are per-route, allowing multiple routes for the same path (for different methods). The param context is removed and merged into the route context now so that you use the validate keyword to specify what needs validating.
2021-09-15 11:09:52 +02:00
exists = kore_route_lookup(req, dom, m, &req->rt);
Add support for config based redirection. Inside the domain contexts a 'redirect' rule will allow you to redirect a request to another URI. Ex: Redirect all requests with a 301 to example.com redirect ^/.*$ 301 https://example.com Using capture groups redirect ^/account/(.*)$ 301 https://example.com/account/$1 Adding the query string in the mix redirect ^/(.*)$ 301 https://example.com/$1?$qs
2020-02-07 06:42:33 +01:00
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
TAILQ_INIT(&(req->resp_headers));
TAILQ_INIT(&(req->req_headers));
TAILQ_INIT(&(req->resp_cookies));
TAILQ_INIT(&(req->req_cookies));
TAILQ_INIT(&(req->arguments));
TAILQ_INIT(&(req->files));
#if defined(KORE_USE_TASKS)
LIST_INIT(&(req->tasks));
#endif
#if defined(KORE_USE_PGSQL)
LIST_INIT(&(req->pgsqls));
#endif
http_request_count++;
TAILQ_INSERT_HEAD(&http_requests, req, list);
TAILQ_INSERT_TAIL(&(c->http_requests), req, olist);
Add support for config based redirection. Inside the domain contexts a 'redirect' rule will allow you to redirect a request to another URI. Ex: Redirect all requests with a 301 to example.com redirect ^/.*$ 301 https://example.com Using capture groups redirect ^/account/(.*)$ 301 https://example.com/account/$1 Adding the query string in the mix redirect ^/(.*)$ 301 https://example.com/$1?$qs
2020-02-07 06:42:33 +01:00
if (http_check_redirect(req, dom)) {
http_request_free(req);
return (NULL);
}
Major Python API improvements. 1) Add @kore.route as a decorator for Python. This decorator can be used on non-class methods to automatically declare their route and parameters. Takes the same arguments as the kore.domain.route function that exists today. Provides a nice clean way of setting up Kore if you dont want a whole class based approach. 2) Remove the requirement for the name for kore.server() and the kore.domain(attach=) keywords. Instead of no name was given, the name "default" is used in both places resulting in less boilerplating. 3) Allow multiple routes to be defined for the same URI as long as the methods are different. So you can have one method for GET / and another for POST /. All changes combined condense the initial experience of getting a Kore Python app up and running: eg: import kore kore.server(ip="127.0.0.1", port="8888", tls=False) kore.domain("*") @kore.route("/", methods=["get"]) async def index(req): req.response(200, b'get method') @kore.route("/", methods=["post"]) async def index_post(req) req.response(200, b'post method')
2021-05-02 00:23:11 +02:00
if (exists == 0) {
Add support for config based redirection. Inside the domain contexts a 'redirect' rule will allow you to redirect a request to another URI. Ex: Redirect all requests with a 301 to example.com redirect ^/.*$ 301 https://example.com Using capture groups redirect ^/account/(.*)$ 301 https://example.com/account/$1 Adding the query string in the mix redirect ^/(.*)$ 301 https://example.com/$1?$qs
2020-02-07 06:42:33 +01:00
http_request_free(req);
http_error_response(c, HTTP_STATUS_NOT_FOUND);
return (NULL);
}
Change how routes are configured in Kore. Routes are now configured in a context per route: route /path { handler handler_name methods get post head validate qs:get id v_id } All route related configurations are per-route, allowing multiple routes for the same path (for different methods). The param context is removed and merged into the route context now so that you use the validate keyword to specify what needs validating.
2021-09-15 11:09:52 +02:00
if (req->rt == NULL) {
Add support for config based redirection. Inside the domain contexts a 'redirect' rule will allow you to redirect a request to another URI. Ex: Redirect all requests with a 301 to example.com redirect ^/.*$ 301 https://example.com Using capture groups redirect ^/account/(.*)$ 301 https://example.com/account/$1 Adding the query string in the mix redirect ^/(.*)$ 301 https://example.com/$1?$qs
2020-02-07 06:42:33 +01:00
http_request_free(req);
http_error_response(c, HTTP_STATUS_METHOD_NOT_ALLOWED);
return (NULL);
}
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
return (req);
}
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
static int
multipart_find_data(struct kore_buf *in, struct kore_buf *out,
size_t *olen, struct http_request *req, const void *needle, size_t len)
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
{
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
ssize_t ret;
size_t left;
u_int8_t *p, first, data[4096];
if (olen != NULL)
*olen = 0;
first = *(const u_int8_t *)needle;
for (;;) {
if (in->offset < len) {
ret = http_body_read(req, data, sizeof(data));
if (ret == -1)
return (KORE_RESULT_ERROR);
if (ret == 0)
return (KORE_RESULT_ERROR);
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
kore_buf_append(in, data, ret);
continue;
}
p = kore_mem_find(in->data, in->offset, &first, 1);
if (p == NULL) {
if (out != NULL)
kore_buf_append(out, in->data, in->offset);
if (olen != NULL)
*olen += in->offset;
kore_buf_reset(in);
continue;
}
left = in->offset - (p - in->data);
if (left < len) {
if (out != NULL)
kore_buf_append(out, in->data, (p - in->data));
if (olen != NULL)
*olen += (p - in->data);
memmove(in->data, p, left);
in->offset = left;
continue;
}
if (!memcmp(p, needle, len)) {
if (out != NULL)
kore_buf_append(out, in->data, p - in->data);
if (olen != NULL)
*olen += (p - in->data);
in->offset = left - len;
if (in->offset > 0)
memmove(in->data, p + len, in->offset);
return (KORE_RESULT_OK);
}
if (out != NULL)
kore_buf_append(out, in->data, (p - in->data) + 1);
if (olen != NULL)
*olen += (p - in->data) + 1;
in->offset = left - 1;
if (in->offset > 0)
memmove(in->data, p + 1, in->offset);
Rework the way validation and param extraction works. - Parameter validation is now done only when http_process_*() is called and upon http_argument_add(). - You MUST have defined your params in a param block or they will be filtered out. - http_argument_lookup() is dead, welcome http_argument_get() and its brothers and sisters: http_argument_get_string() http_argument_get_uint16() http_argument_get_int16() http_argument_get_uint32() http_argument_get_int32() They will automatically do bounds checking on integers for you and return proper integers or a NUL-terminated string. - The http_argument_get* functions no longer create an additional copy of the string which you need to free. Easier going. - http_multiple_args() is dead, byebye - Make some stuff we don't want to share with the modules static.
2013-12-12 00:58:32 +01:00
}
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
return (KORE_RESULT_ERROR);
}
static int
multipart_parse_headers(struct http_request *req, struct kore_buf *in,
struct kore_buf *hbuf, const char *boundary, const int blen)
{
int h, c, i;
char *headers[5], *args[5], *opt[5];
char *d, *val, *name, *fname, *string;
Change kore_buf_stringify() a bit. Takes a size_t pointer as its second argument now, if not NULL this will be populated with the length of the string that is being returned.
2016-06-02 07:08:19 +02:00
string = kore_buf_stringify(hbuf, NULL);
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
h = kore_split_string(string, "\r\n", headers, 5);
for (i = 0; i < h; i++) {
c = kore_split_string(headers[i], ":", args, 5);
if (c != 2)
Rework the way validation and param extraction works. - Parameter validation is now done only when http_process_*() is called and upon http_argument_add(). - You MUST have defined your params in a param block or they will be filtered out. - http_argument_lookup() is dead, welcome http_argument_get() and its brothers and sisters: http_argument_get_string() http_argument_get_uint16() http_argument_get_int16() http_argument_get_uint32() http_argument_get_int32() They will automatically do bounds checking on integers for you and return proper integers or a NUL-terminated string. - The http_argument_get* functions no longer create an additional copy of the string which you need to free. Easier going. - http_multiple_args() is dead, byebye - Make some stuff we don't want to share with the modules static.
2013-12-12 00:58:32 +01:00
continue;
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
/* Ignore other headers for now. */
if (strcasecmp(args[0], "content-disposition"))
continue;
fixup isspace arguments.
2017-02-07 22:06:14 +01:00
for (d = args[1]; isspace(*(unsigned char *)d); d++)
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
;
c = kore_split_string(d, ";", opt, 5);
if (c < 2)
continue;
if (strcasecmp(opt[0], "form-data"))
continue;
if ((val = strchr(opt[1], '=')) == NULL)
continue;
if (strlen(val) < 3)
continue;
val++;
kore_strip_chars(val, '"', &name);
if (opt[2] == NULL) {
multipart_add_field(req, in, name, boundary, blen);
Large changes to the memory subsystem in kore. - Change pools to use mmap() for allocating regions. - Change kore_malloc() to use pools for commonly sized objects. (split into multiple of 2 buckets, starting at 8 bytes up to 8192). - Rename kore_mem_free() to kore_free(). The preallocated pools will hold up to 128K of elements per block size. In case a larger object is to be allocated kore_malloc() will use malloc() instead.
2016-07-12 13:54:14 +02:00
kore_free(name);
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
continue;
}
fixup isspace arguments.
2017-02-07 22:06:14 +01:00
for (d = opt[2]; isspace(*(unsigned char *)d); d++)
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
;
Rework the way validation and param extraction works. - Parameter validation is now done only when http_process_*() is called and upon http_argument_add(). - You MUST have defined your params in a param block or they will be filtered out. - http_argument_lookup() is dead, welcome http_argument_get() and its brothers and sisters: http_argument_get_string() http_argument_get_uint16() http_argument_get_int16() http_argument_get_uint32() http_argument_get_int32() They will automatically do bounds checking on integers for you and return proper integers or a NUL-terminated string. - The http_argument_get* functions no longer create an additional copy of the string which you need to free. Easier going. - http_multiple_args() is dead, byebye - Make some stuff we don't want to share with the modules static.
2013-12-12 00:58:32 +01:00
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
if (!strncasecmp(d, "filename=", 9)) {
if ((val = strchr(d, '=')) == NULL) {
Large changes to the memory subsystem in kore. - Change pools to use mmap() for allocating regions. - Change kore_malloc() to use pools for commonly sized objects. (split into multiple of 2 buckets, starting at 8 bytes up to 8192). - Rename kore_mem_free() to kore_free(). The preallocated pools will hold up to 128K of elements per block size. In case a larger object is to be allocated kore_malloc() will use malloc() instead.
2016-07-12 13:54:14 +02:00
kore_free(name);
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
continue;
Rework the way validation and param extraction works. - Parameter validation is now done only when http_process_*() is called and upon http_argument_add(). - You MUST have defined your params in a param block or they will be filtered out. - http_argument_lookup() is dead, welcome http_argument_get() and its brothers and sisters: http_argument_get_string() http_argument_get_uint16() http_argument_get_int16() http_argument_get_uint32() http_argument_get_int32() They will automatically do bounds checking on integers for you and return proper integers or a NUL-terminated string. - The http_argument_get* functions no longer create an additional copy of the string which you need to free. Easier going. - http_multiple_args() is dead, byebye - Make some stuff we don't want to share with the modules static.
2013-12-12 00:58:32 +01:00
}
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
val++;
kore_strip_chars(val, '"', &fname);
if (strlen(fname) > 0) {
multipart_file_add(req,
in, name, fname, boundary, blen);
}
Large changes to the memory subsystem in kore. - Change pools to use mmap() for allocating regions. - Change kore_malloc() to use pools for commonly sized objects. (split into multiple of 2 buckets, starting at 8 bytes up to 8192). - Rename kore_mem_free() to kore_free(). The preallocated pools will hold up to 128K of elements per block size. In case a larger object is to be allocated kore_malloc() will use malloc() instead.
2016-07-12 13:54:14 +02:00
kore_free(fname);
Rework the way validation and param extraction works. - Parameter validation is now done only when http_process_*() is called and upon http_argument_add(). - You MUST have defined your params in a param block or they will be filtered out. - http_argument_lookup() is dead, welcome http_argument_get() and its brothers and sisters: http_argument_get_string() http_argument_get_uint16() http_argument_get_int16() http_argument_get_uint32() http_argument_get_int32() They will automatically do bounds checking on integers for you and return proper integers or a NUL-terminated string. - The http_argument_get* functions no longer create an additional copy of the string which you need to free. Easier going. - http_multiple_args() is dead, byebye - Make some stuff we don't want to share with the modules static.
2013-12-12 00:58:32 +01:00
}
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
Large changes to the memory subsystem in kore. - Change pools to use mmap() for allocating regions. - Change kore_malloc() to use pools for commonly sized objects. (split into multiple of 2 buckets, starting at 8 bytes up to 8192). - Rename kore_mem_free() to kore_free(). The preallocated pools will hold up to 128K of elements per block size. In case a larger object is to be allocated kore_malloc() will use malloc() instead.
2016-07-12 13:54:14 +02:00
kore_free(name);
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
}
return (KORE_RESULT_OK);
}
static void
multipart_add_field(struct http_request *req, struct kore_buf *in,
actually urldecode parameter names.
2017-02-06 22:47:32 +01:00
char *name, const char *boundary, const int blen)
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
{
struct kore_buf *data;
char *string;
kore_buf_create -> kore_buf_alloc.
2016-07-14 12:34:29 +02:00
data = kore_buf_alloc(128);
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
if (!multipart_find_data(in, data, NULL, req, boundary, blen)) {
kore_buf_free(data);
return;
Rework the way validation and param extraction works. - Parameter validation is now done only when http_process_*() is called and upon http_argument_add(). - You MUST have defined your params in a param block or they will be filtered out. - http_argument_lookup() is dead, welcome http_argument_get() and its brothers and sisters: http_argument_get_string() http_argument_get_uint16() http_argument_get_int16() http_argument_get_uint32() http_argument_get_int32() They will automatically do bounds checking on integers for you and return proper integers or a NUL-terminated string. - The http_argument_get* functions no longer create an additional copy of the string which you need to free. Easier going. - http_multiple_args() is dead, byebye - Make some stuff we don't want to share with the modules static.
2013-12-12 00:58:32 +01:00
}
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
if (data->offset < 3) {
kore_buf_free(data);
return;
}
data->offset -= 2;
Change kore_buf_stringify() a bit. Takes a size_t pointer as its second argument now, if not NULL this will be populated with the length of the string that is being returned.
2016-06-02 07:08:19 +02:00
string = kore_buf_stringify(data, NULL);
do not http_argument_urldecode for multipart data.
2018-08-16 14:11:28 +02:00
http_argument_add(req, name, string, 0, 0);
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
kore_buf_free(data);
Add support for multipart forms. New API functions (docs need to be updated): - http_file_lookup() - http_file_add() - http_argument_add() - kore_strip_chars() - kore_mem_find() - Add an example under the example module on how files can be read.
2013-09-10 11:02:59 +02:00
}
Rework the way validation and param extraction works. - Parameter validation is now done only when http_process_*() is called and upon http_argument_add(). - You MUST have defined your params in a param block or they will be filtered out. - http_argument_lookup() is dead, welcome http_argument_get() and its brothers and sisters: http_argument_get_string() http_argument_get_uint16() http_argument_get_int16() http_argument_get_uint32() http_argument_get_int32() They will automatically do bounds checking on integers for you and return proper integers or a NUL-terminated string. - The http_argument_get* functions no longer create an additional copy of the string which you need to free. Easier going. - http_multiple_args() is dead, byebye - Make some stuff we don't want to share with the modules static.
2013-12-12 00:58:32 +01:00
static void
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
multipart_file_add(struct http_request *req, struct kore_buf *in,
const char *name, const char *fname, const char *boundary, const int blen)
SPDY works, we can serve content. Now for dynamic modules so we can get stuff on the road. TODO: add plain HTTP support if SPDY isnt working in the browser.
2013-05-01 12:23:21 +02:00
{
Rework the way validation and param extraction works. - Parameter validation is now done only when http_process_*() is called and upon http_argument_add(). - You MUST have defined your params in a param block or they will be filtered out. - http_argument_lookup() is dead, welcome http_argument_get() and its brothers and sisters: http_argument_get_string() http_argument_get_uint16() http_argument_get_int16() http_argument_get_uint32() http_argument_get_int32() They will automatically do bounds checking on integers for you and return proper integers or a NUL-terminated string. - The http_argument_get* functions no longer create an additional copy of the string which you need to free. Easier going. - http_multiple_args() is dead, byebye - Make some stuff we don't want to share with the modules static.
2013-12-12 00:58:32 +01:00
struct http_file *f;
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
size_t position, len;
style
2017-02-01 21:20:43 +01:00
position = req->http_body_offset - in->offset;
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
if (!multipart_find_data(in, NULL, &len, req, boundary, blen))
return;
if (len < 3)
return;
len -= 2;
SPDY works, we can serve content. Now for dynamic modules so we can get stuff on the road. TODO: add plain HTTP support if SPDY isnt working in the browser.
2013-05-01 12:23:21 +02:00
Rework the way validation and param extraction works. - Parameter validation is now done only when http_process_*() is called and upon http_argument_add(). - You MUST have defined your params in a param block or they will be filtered out. - http_argument_lookup() is dead, welcome http_argument_get() and its brothers and sisters: http_argument_get_string() http_argument_get_uint16() http_argument_get_int16() http_argument_get_uint32() http_argument_get_int32() They will automatically do bounds checking on integers for you and return proper integers or a NUL-terminated string. - The http_argument_get* functions no longer create an additional copy of the string which you need to free. Easier going. - http_multiple_args() is dead, byebye - Make some stuff we don't want to share with the modules static.
2013-12-12 00:58:32 +01:00
f = kore_malloc(sizeof(struct http_file));
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
f->req = req;
Explicitly set offset to 0 for http_file.
2016-07-27 16:40:41 +02:00
f->offset = 0;
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
f->length = len;
f->position = position;
Rework the way validation and param extraction works. - Parameter validation is now done only when http_process_*() is called and upon http_argument_add(). - You MUST have defined your params in a param block or they will be filtered out. - http_argument_lookup() is dead, welcome http_argument_get() and its brothers and sisters: http_argument_get_string() http_argument_get_uint16() http_argument_get_int16() http_argument_get_uint32() http_argument_get_int32() They will automatically do bounds checking on integers for you and return proper integers or a NUL-terminated string. - The http_argument_get* functions no longer create an additional copy of the string which you need to free. Easier going. - http_multiple_args() is dead, byebye - Make some stuff we don't want to share with the modules static.
2013-12-12 00:58:32 +01:00
f->name = kore_strdup(name);
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
f->filename = kore_strdup(fname);
Rework the way validation and param extraction works. - Parameter validation is now done only when http_process_*() is called and upon http_argument_add(). - You MUST have defined your params in a param block or they will be filtered out. - http_argument_lookup() is dead, welcome http_argument_get() and its brothers and sisters: http_argument_get_string() http_argument_get_uint16() http_argument_get_int16() http_argument_get_uint32() http_argument_get_int32() They will automatically do bounds checking on integers for you and return proper integers or a NUL-terminated string. - The http_argument_get* functions no longer create an additional copy of the string which you need to free. Easier going. - http_multiple_args() is dead, byebye - Make some stuff we don't want to share with the modules static.
2013-12-12 00:58:32 +01:00
TAILQ_INSERT_TAIL(&(req->files), f, list);
SPDY works, we can serve content. Now for dynamic modules so we can get stuff on the road. TODO: add plain HTTP support if SPDY isnt working in the browser.
2013-05-01 12:23:21 +02:00
}
support for POST in both spdy and normal connections
2013-05-03 07:35:11 +02:00
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
static void
do not http_argument_urldecode for multipart data.
2018-08-16 14:11:28 +02:00
http_argument_add(struct http_request *req, char *name, char *value, int qs,
int decode)
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
{
struct http_arg *q;
Change how routes are configured in Kore. Routes are now configured in a context per route: route /path { handler handler_name methods get post head validate qs:get id v_id } All route related configurations are per-route, allowing multiple routes for the same path (for different methods). The param context is removed and merged into the route context now so that you use the validate keyword to specify what needs validating.
2021-09-15 11:09:52 +02:00
struct kore_route_params *p;
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
http_argument_urldecode() could fail.
2019-07-20 20:35:58 +02:00
if (decode) {
Change http_argument_urldecode(). Takes a new parameter now `qs`, if set to 1 it'll urldecode according to how its always done it before. If not set to 0 (for posts for example), the decoder will allow \n and \r in addition to the other ones.
2023-03-12 23:33:47 +01:00
if (!http_argument_urldecode(name, qs))
http_argument_urldecode() could fail.
2019-07-20 20:35:58 +02:00
return;
}
actually urldecode parameter names.
2017-02-06 22:47:32 +01:00
Change how routes are configured in Kore. Routes are now configured in a context per route: route /path { handler handler_name methods get post head validate qs:get id v_id } All route related configurations are per-route, allowing multiple routes for the same path (for different methods). The param context is removed and merged into the route context now so that you use the validate keyword to specify what needs validating.
2021-09-15 11:09:52 +02:00
TAILQ_FOREACH(p, &req->rt->params, list) {
Allow param blocks to be marked as "querystring" Before params get would mean querystring and anything else would just count toward a www-encoded body. Now you can prefix the params block with "qs" indicating that those configured parameters are allowed to occur in the query string regardless of the method used. This means you can do something like: params qs:post /uri { ... } to specify what the allowed parameters are in the querystring for a POST request towards /uri. inspired by and properly fixes #205.
2018-01-16 18:47:50 +01:00
if (qs == 1 && !(p->flags & KORE_PARAMS_QUERY_STRING))
continue;
if (qs == 0 && (p->flags & KORE_PARAMS_QUERY_STRING))
continue;
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
if (p->method != req->method)
continue;
actually urldecode parameter names.
2017-02-06 22:47:32 +01:00
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
if (strcmp(p->name, name))
continue;
http_argument_decode() can fail.
2019-08-02 11:34:45 +02:00
if (decode) {
Change http_argument_urldecode(). Takes a new parameter now `qs`, if set to 1 it'll urldecode according to how its always done it before. If not set to 0 (for posts for example), the decoder will allow \n and \r in addition to the other ones.
2023-03-12 23:33:47 +01:00
if (!http_argument_urldecode(value, qs))
http_argument_decode() can fail.
2019-08-02 11:34:45 +02:00
return;
}
do not http_argument_urldecode for multipart data.
2018-08-16 14:11:28 +02:00
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
if (!kore_validator_check(req, p->validator, value))
break;
q = kore_malloc(sizeof(struct http_arg));
q->name = kore_strdup(name);
q->s_value = kore_strdup(value);
TAILQ_INSERT_TAIL(&(req->arguments), q, list);
break;
}
}
support for POST in both spdy and normal connections
2013-05-03 07:35:11 +02:00
static int
Add PUT/DELETE/HEAD methods (finally). This commit renames certain POST centric variable and configuration naming to the correct HTTP body stuff. API changes include http_postbody_text() and http_postbody_bytes() to have become http_body_text() and http_body_bytes(). The developer is still responsible for validating the method their page handler is called with. Hopefully this becomes a configuration option soon enough.
2014-10-08 11:03:14 +02:00
http_body_recv(struct netbuf *nb)
Add the on_body_chunk handler for routes. If set, will call a given handler with the prototype of `void body_chunk(struct http_request *req, const void *data, size_t len);` for each chunk of the received HTTP body, allowing a developer to handle it in their own way. The incoming body is still being handled and retained in the same way as before (in a kore_buf or temporary file). While here, allow HTTP_STATUS_CONTINUE to work via http_response() and make the handling of incoming HTTP header data a bit better.
2021-09-17 19:28:06 +02:00
{
struct http_request *req = (struct http_request *)nb->extra;
return (http_body_update(req, nb->buf, nb->s_off));
}
static int
http_body_update(struct http_request *req, const void *data, size_t len)
support for POST in both spdy and normal connections
2013-05-03 07:35:11 +02:00
{
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
ssize_t ret;
Cut memory usage for incoming HTTP bodies in half. Kore pre-allocates a kore_buf for the full size of the incoming HTTP body ... but also was passing the full size to the net_recv_reset() function. Instead of this, properly read smaller chunks from the network and append them to the body buffer as they roll in.
2016-01-08 20:56:21 +01:00
u_int64_t bytes_left;
support for POST in both spdy and normal connections
2013-05-03 07:35:11 +02:00
Initial work splitting OpenSSL code away. This work moves all TLS / crypto related code into a tls_openssl.c file and adds a tls_none.c which contains just stubs. Allows compilation of Kore with TLS_BACKEND=none to remove building against OpenSSL. Also adds code for SHA1/SHA2 taken from openssh-portable so we don't depend on those being present anymore in libcrypto.
2022-02-17 13:45:28 +01:00
SHA256Update(&req->hashctx, data, len);
Calculate an md over the incoming HTTP body. This is calculated while the HTTP body is incoming over the wire, once the body is fully received the digest will be available for the page handlers to obtain. You can obtain a hex string for this md via http_body_digest() or dereferences the http_request and look at http_body_digest manually for the bytes.
2018-07-17 14:53:55 +02:00
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
if (req->http_body_fd != -1) {
Add the on_body_chunk handler for routes. If set, will call a given handler with the prototype of `void body_chunk(struct http_request *req, const void *data, size_t len);` for each chunk of the received HTTP body, allowing a developer to handle it in their own way. The incoming body is still being handled and retained in the same way as before (in a kore_buf or temporary file). While here, allow HTTP_STATUS_CONTINUE to work via http_response() and make the handling of incoming HTTP header data a bit better.
2021-09-17 19:28:06 +02:00
ret = write(req->http_body_fd, data, len);
if (ret == -1 || (size_t)ret != len) {
Improve http_body_recv(). - If we fail to write to our temporary file error instead of fatal. - Return KORE_RESULT_ERROR on other errors as well.
2016-02-13 15:41:37 +01:00
req->flags |= HTTP_REQUEST_DELETE;
Stop hardcoding HTTP error codes in http_error_response() calls. Use predefined HTTP_STATUS_* macros instead.
2021-04-14 15:14:26 +02:00
http_error_response(req->owner,
HTTP_STATUS_INTERNAL_ERROR);
Improve http_body_recv(). - If we fail to write to our temporary file error instead of fatal. - Return KORE_RESULT_ERROR on other errors as well.
2016-02-13 15:41:37 +01:00
return (KORE_RESULT_ERROR);
}
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
} else if (req->http_body != NULL) {
Add the on_body_chunk handler for routes. If set, will call a given handler with the prototype of `void body_chunk(struct http_request *req, const void *data, size_t len);` for each chunk of the received HTTP body, allowing a developer to handle it in their own way. The incoming body is still being handled and retained in the same way as before (in a kore_buf or temporary file). While here, allow HTTP_STATUS_CONTINUE to work via http_response() and make the handling of incoming HTTP header data a bit better.
2021-09-17 19:28:06 +02:00
kore_buf_append(req->http_body, data, len);
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
} else {
req->flags |= HTTP_REQUEST_DELETE;
Stop hardcoding HTTP error codes in http_error_response() calls. Use predefined HTTP_STATUS_* macros instead.
2021-04-14 15:14:26 +02:00
http_error_response(req->owner,
HTTP_STATUS_INTERNAL_ERROR);
Improve http_body_recv(). - If we fail to write to our temporary file error instead of fatal. - Return KORE_RESULT_ERROR on other errors as well.
2016-02-13 15:41:37 +01:00
return (KORE_RESULT_ERROR);
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
}
Add PUT/DELETE/HEAD methods (finally). This commit renames certain POST centric variable and configuration naming to the correct HTTP body stuff. API changes include http_postbody_text() and http_postbody_bytes() to have become http_body_text() and http_body_bytes(). The developer is still responsible for validating the method their page handler is called with. Hopefully this becomes a configuration option soon enough.
2014-10-08 11:03:14 +02:00
Add the on_body_chunk handler for routes. If set, will call a given handler with the prototype of `void body_chunk(struct http_request *req, const void *data, size_t len);` for each chunk of the received HTTP body, allowing a developer to handle it in their own way. The incoming body is still being handled and retained in the same way as before (in a kore_buf or temporary file). While here, allow HTTP_STATUS_CONTINUE to work via http_response() and make the handling of incoming HTTP header data a bit better.
2021-09-17 19:28:06 +02:00
req->content_length -= len;
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
if (req->content_length == 0) {
Fix a problem with http_timeout when a body is sent. When receiving an HTTP body, Kore never reset http_timeout once the transfer was done. This can result in a 408 being thrown by Kore while a request is activity running.
2022-09-01 12:09:15 +02:00
req->owner->http_timeout = 0;
Add the on_body_chunk handler for routes. If set, will call a given handler with the prototype of `void body_chunk(struct http_request *req, const void *data, size_t len);` for each chunk of the received HTTP body, allowing a developer to handle it in their own way. The incoming body is still being handled and retained in the same way as before (in a kore_buf or temporary file). While here, allow HTTP_STATUS_CONTINUE to work via http_response() and make the handling of incoming HTTP header data a bit better.
2021-09-17 19:28:06 +02:00
req->owner->rnb->extra = NULL;
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
http_request_wakeup(req);
Cut memory usage for incoming HTTP bodies in half. Kore pre-allocates a kore_buf for the full size of the incoming HTTP body ... but also was passing the full size to the net_recv_reset() function. Instead of this, properly read smaller chunks from the network and append them to the body buffer as they roll in.
2016-01-08 20:56:21 +01:00
req->flags |= HTTP_REQUEST_COMPLETE;
req->flags &= ~HTTP_REQUEST_EXPECT_BODY;
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
req->content_length = req->http_body_length;
if (!http_body_rewind(req)) {
req->flags |= HTTP_REQUEST_DELETE;
Stop hardcoding HTTP error codes in http_error_response() calls. Use predefined HTTP_STATUS_* macros instead.
2021-04-14 15:14:26 +02:00
http_error_response(req->owner,
HTTP_STATUS_INTERNAL_ERROR);
Improve http_body_recv(). - If we fail to write to our temporary file error instead of fatal. - Return KORE_RESULT_ERROR on other errors as well.
2016-02-13 15:41:37 +01:00
return (KORE_RESULT_ERROR);
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
}
Initial work splitting OpenSSL code away. This work moves all TLS / crypto related code into a tls_openssl.c file and adds a tls_none.c which contains just stubs. Allows compilation of Kore with TLS_BACKEND=none to remove building against OpenSSL. Also adds code for SHA1/SHA2 taken from openssh-portable so we don't depend on those being present anymore in libcrypto.
2022-02-17 13:45:28 +01:00
SHA256Final(req->http_body_digest, &req->hashctx);
Cut memory usage for incoming HTTP bodies in half. Kore pre-allocates a kore_buf for the full size of the incoming HTTP body ... but also was passing the full size to the net_recv_reset() function. Instead of this, properly read smaller chunks from the network and append them to the body buffer as they roll in.
2016-01-08 20:56:21 +01:00
} else {
Massive rework of HTTP layer. This commit is a flag day, your old modules will almost certainly need to be updated in order to build properly with these changes. Summary of changes: - Offload HTTP bodies to disk if they are large (inspired by #100). (disabled by default) - The http_argument_get* macros now takes an explicit http_request parameter. - Kore will now throw 404 errors almost immediately after an HTTP request has come in instead of waiting until all data has arrived. API changes: - http_argument_get* macros now require an explicit http_request parameter. (no more magic invokations). - http_generic_404() is gone - http_populate_arguments() is gone - http_body_bytes() is gone - http_body_text() is gone - http_body_read() has been added - http_populate_post() has been added - http_populate_get() has been added - http_file_read() has been added - http_file_rewind() has been added - http_file_lookup() no longer takes name, fname, data and len parameters. - http_file_lookup() now returns a struct http_file pointer. - http_populate_multipart_form() no longer takes an secondary parameter. New configuration options: - http_body_disk_offload: Number of bytes after which Kore will offload the HTTP body to disk instead of retaining it in memory. If 0 this feature is disabled. (Default: 0) - http_body_disk_path: The path where Kore will store temporary HTTP body files. (this directory does not get created if http_body_disk_offload is 0). New example: The upload example has been added, demonstrating how to deal with file uploads from a multipart form.
2016-01-18 11:30:22 +01:00
bytes_left = req->content_length;
Add the on_body_chunk handler for routes. If set, will call a given handler with the prototype of `void body_chunk(struct http_request *req, const void *data, size_t len);` for each chunk of the received HTTP body, allowing a developer to handle it in their own way. The incoming body is still being handled and retained in the same way as before (in a kore_buf or temporary file). While here, allow HTTP_STATUS_CONTINUE to work via http_response() and make the handling of incoming HTTP header data a bit better.
2021-09-17 19:28:06 +02:00
net_recv_reset(req->owner,
Cut memory usage for incoming HTTP bodies in half. Kore pre-allocates a kore_buf for the full size of the incoming HTTP body ... but also was passing the full size to the net_recv_reset() function. Instead of this, properly read smaller chunks from the network and append them to the body buffer as they roll in.
2016-01-08 20:56:21 +01:00
MIN(bytes_left, NETBUF_SEND_PAYLOAD_MAX),
http_body_recv);
Add the on_body_chunk handler for routes. If set, will call a given handler with the prototype of `void body_chunk(struct http_request *req, const void *data, size_t len);` for each chunk of the received HTTP body, allowing a developer to handle it in their own way. The incoming body is still being handled and retained in the same way as before (in a kore_buf or temporary file). While here, allow HTTP_STATUS_CONTINUE to work via http_response() and make the handling of incoming HTTP header data a bit better.
2021-09-17 19:28:06 +02:00
req->owner->rnb->extra = req;
}
if (req->rt->on_body_chunk != NULL && len > 0) {
kore_runtime_http_body_chunk(req->rt->on_body_chunk,
req, data, len);
Cut memory usage for incoming HTTP bodies in half. Kore pre-allocates a kore_buf for the full size of the incoming HTTP body ... but also was passing the full size to the net_recv_reset() function. Instead of this, properly read smaller chunks from the network and append them to the body buffer as they roll in.
2016-01-08 20:56:21 +01:00
}
wake up the workers once in a while to process anything that is waiting.
2013-05-30 20:55:50 +02:00
support for POST in both spdy and normal connections
2013-05-03 07:35:11 +02:00
return (KORE_RESULT_OK);
}
ok that was wrong, disconnect http clients whne we're done sending or upon error, not immediately after queueing everything.
2013-05-30 21:57:14 +02:00
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
static void
Introduce NOHTTP=1 build option. This basically turns off the HTTP layer for Kore. It does not compile in anything for HTTP. This allows Kore to be used as a network application platform as well. Added an example for this called nohttp. Other changes that sneaked in while hacking on this: * Use calloc(), kill pendantic malloc option. * Killed off SPDY/3.1 support completely, will be superseded by http2 Note that comes with massive changes to a lot of the core API functions provided by Kore, these might break your application.
2015-11-27 16:22:50 +01:00
http_error_response(struct connection *c, int status)
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
{
set CONN_CLOSE_EMPTY for early HTTP errors. while here fix missing connection response headers for errors.
2017-07-04 10:55:11 +02:00
c->flags |= CONN_CLOSE_EMPTY;
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
switch (c->proto) {
case CONN_PROTO_HTTP:
http_response_normal(NULL, c, status, NULL, 0);
break;
Add websocket support to Kore. Introduces a few new api functions: - kore_websocket_handshake(struct http_request *): Performs the handshake on an HTTP request (coming from page handler) - kore_websocket_send(struct connection *, u_int8_t, void *, size_t): Sends data to a websocket connection. - kore_websocket_broadcast(struct connection *, u_int8_t, void *, size_t, int): Broadcast the given websocket op and data to all connected websocket clients on the worker. Note that as of right now the WEBSOCKET_BROADCAST_GLOBAL scope option does not work yet and messages broadcasted will be restricted to workers only. - kore_worker_websocket_broadcast(struct connection *, void *, void *): Backend function used by kore_websocket_broadcast(). Could prove useful for developers to have access to. A simple example is given under examples/websocket. Known issues: Kore does not support PING or CONT frames just yet.
2014-11-24 11:01:12 +01:00
default:
fatal("http_error_response() bad proto %d", c->proto);
/* NOTREACHED. */
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
}
flush out send buffer in http_error_response().
2017-07-04 10:42:14 +02:00
if (!net_send_flush(c))
kore_connection_disconnect(c);
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
}
static void
http_response_normal(struct http_request *req, struct connection *c,
Code cleanup, several API breaking changes in here
2016-07-08 10:02:18 +02:00
int status, const void *d, size_t len)
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
{
Rework http_pretty_error a tiny bit.
2020-03-04 08:36:10 +01:00
struct kore_buf buf;
Cookies and arguments parsing improvements (#166) Add new cookie API for handling of cookies.
2017-02-07 21:49:10 +01:00
struct http_cookie *ck;
cookie improvements. - split up writing of cookies into its own function. - turn maxage into a signed int and use -1 for it not being set. - lots of style fixes - remove HTTP_COOKIE_DEFAULT, just pass 0 if you don't want flags.
2017-02-07 22:03:06 +01:00
struct http_header *hdr;
Some things still talk http/1.0.
2018-10-26 21:24:51 +02:00
char version;
Rework http_pretty_error a tiny bit.
2020-03-04 08:36:10 +01:00
const char *conn, *text;
int connection_close, send_body;
send_body = 1;
text = http_status_text(status);
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
cookie improvements. - split up writing of cookies into its own function. - turn maxage into a signed int and use -1 for it not being set. - lots of style fixes - remove HTTP_COOKIE_DEFAULT, just pass 0 if you don't want flags.
2017-02-07 22:03:06 +01:00
kore_buf_reset(header_buf);
Use a single header_buf in http_response_normal() Avoids constantly allocating and freeing buffers all the time. This can be done safely due to the fact workers are single threaded.
2014-08-22 14:38:48 +02:00
Some things still talk http/1.0.
2018-10-26 21:24:51 +02:00
if (req != NULL) {
if (req->flags & HTTP_VERSION_1_0)
version = '0';
else
version = '1';
} else {
version = '1';
}
kore_buf_appendf(header_buf, "HTTP/1.%c %d %s\r\n",
Rework http_pretty_error a tiny bit.
2020-03-04 08:36:10 +01:00
version, status, text);
Add the on_body_chunk handler for routes. If set, will call a given handler with the prototype of `void body_chunk(struct http_request *req, const void *data, size_t len);` for each chunk of the received HTTP body, allowing a developer to handle it in their own way. The incoming body is still being handled and retained in the same way as before (in a kore_buf or temporary file). While here, allow HTTP_STATUS_CONTINUE to work via http_response() and make the handling of incoming HTTP header data a bit better.
2021-09-17 19:28:06 +02:00
if (status == 100) {
kore_buf_append(header_buf, "\r\n", 2);
net_send_queue(c, header_buf->data, header_buf->offset);
return;
}
Rework net, worker and some http internals. - The net code no longer has a recv_queue, instead reuse same recv buffer. - Introduce net_recv_reset() to reset the recv buffer when needed. - Have the workers spread the load better between them by slightly delaying their next accept lock and giving them an accept treshold so they don't go ahead and keep accepting connections if they end up winning the race constantly between the workers. - The kore_worker_acceptlock_release() is no longer available. - Prepopulate the HTTP server response header that is added to each response in both normal HTTP and SPDY modes. - The path and host members of http_request are now allocated on the heap. These changes overall result better performance on a multicore machine, especially the worker load changes shine through.
2014-10-22 21:16:49 +02:00
kore_buf_append(header_buf, http_version, http_version_len);
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
do not set CONN_CLOSE_EMPTY for 1.0 until we reply.
2018-11-30 22:12:43 +01:00
if ((c->flags & CONN_CLOSE_EMPTY) ||
fix potential NULL dereferences. found by clang --analyze, reminded by fahlgren@
2019-11-13 11:23:02 +01:00
(req != NULL && (req->flags & HTTP_VERSION_1_0))) {
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
connection_close = 1;
do not set CONN_CLOSE_EMPTY for 1.0 until we reply.
2018-11-30 22:12:43 +01:00
} else {
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
connection_close = 0;
do not set CONN_CLOSE_EMPTY for 1.0 until we reply.
2018-11-30 22:12:43 +01:00
}
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
if (connection_close == 0 && req != NULL) {
Rename the two functions used to read/set headers. Rename http_request_header_get() to http_request_header(). Rename http_response_header_add() to http_response_header().
2014-08-05 14:26:31 +02:00
if (http_request_header(req, "connection", &conn)) {
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
if ((*conn == 'c' || *conn == 'C') &&
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
!strcasecmp(conn, "close")) {
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
connection_close = 1;
Rework HTTP and worker processes. The HTTP layer used to make a copy of each incoming header and its value for a request. Stop doing that and make HTTP headers zero-copy all across the board. This change comes with some api function changes, notably the http_request_header() function which now takes a const char ** rather than a char ** out pointer. This commit also constifies several members of http_request, beware. Additional rework how the worker processes deal with the accept lock. Before: if a worker held the accept lock and it accepted a new connection it would release the lock for others and back off for 500ms before attempting to grab the lock again. This approach worked but under high load this starts becoming obvious. Now: - workers not holding the accept lock and not having any connections will wait less long before returning from kore_platform_event_wait(). - workers not holding the accept lock will no longer blindly wait an arbitrary amount in kore_platform_event_wait() but will look at how long until the next lock grab is and base their timeout on that. - if a worker its next_lock timeout is up and failed to grab the lock it will try again in half the time again. - the worker process holding the lock will when releasing the lock double check if it still has space for newer connections, if it does it will keep the lock until it is full. This prevents the lock from bouncing between several non busy worker processes all the time. Additional fixes: - Reduce the number of times we check the timeout list, only do it twice per second rather then every event tick. - Fix solo worker count for TLS (we actually hold two processes, not one). - Make sure we don't accidentally miscalculate the idle time causing new connections under heavy load to instantly drop. - Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
2018-02-14 13:48:49 +01:00
}
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
}
}
Add the on_body_chunk handler for routes. If set, will call a given handler with the prototype of `void body_chunk(struct http_request *req, const void *data, size_t len);` for each chunk of the received HTTP body, allowing a developer to handle it in their own way. The incoming body is still being handled and retained in the same way as before (in a kore_buf or temporary file). While here, allow HTTP_STATUS_CONTINUE to work via http_response() and make the handling of incoming HTTP header data a bit better.
2021-09-17 19:28:06 +02:00
kore_buf_init(&buf, 1024);
req can be NULL if we are coming from an error path.
2015-08-02 16:53:40 +02:00
/* Note that req CAN be NULL. */
set CONN_CLOSE_EMPTY for early HTTP errors. while here fix missing connection response headers for errors.
2017-07-04 10:55:11 +02:00
if (req == NULL || req->owner->proto != CONN_PROTO_WEBSOCKET) {
Do not send connection:close for websocket upgrades.
2015-06-30 16:43:21 +02:00
if (http_keepalive_time && connection_close == 0) {
kore_buf_appendf(header_buf,
"connection: keep-alive\r\n");
kore_buf_appendf(header_buf,
"keep-alive: timeout=%d\r\n", http_keepalive_time);
} else {
c->flags |= CONN_CLOSE_EMPTY;
kore_buf_appendf(header_buf, "connection: close\r\n");
Fix websockets. Connection must be Upgrade, not keep-alive, Upgrade
2015-06-26 05:14:01 +02:00
}
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
}
rename ssl prefixed things to tls.
2022-02-18 10:20:28 +01:00
if (c->tls && http_hsts_enable) {
Use a single header_buf in http_response_normal() Avoids constantly allocating and freeing buffers all the time. This can be done safely due to the fact workers are single threaded.
2014-08-22 14:38:48 +02:00
kore_buf_appendf(header_buf, "strict-transport-security: ");
Always include includeSubDomains for HSTS.
2015-02-10 17:14:07 +01:00
kore_buf_appendf(header_buf,
"max-age=%" PRIu64 "; includeSubDomains\r\n",
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
http_hsts_enable);
}
Add the http_pretty_error configuration option. When enabled, Kore returns HTML error pages for status codes 4xx and 5xx instead of empty content.
2020-03-03 12:53:18 +01:00
if (http_pretty_error && d == NULL && status >= 400) {
Rework http_pretty_error a tiny bit.
2020-03-04 08:36:10 +01:00
kore_buf_appendf(&buf, pretty_error_fmt,
status, text, status, text);
d = buf.data;
len = buf.offset;
Add the http_pretty_error configuration option. When enabled, Kore returns HTML error pages for status codes 4xx and 5xx instead of empty content.
2020-03-03 12:53:18 +01:00
}
make sure user-defined headers are set for > 500. a commit done in 2018 prevented http responses with error codes > 500 to include any user-set headers, preventing a developer to include things like content-type etc. reported by Arun Babu via users@
2019-05-12 20:53:27 +02:00
if (req != NULL) {
cookie improvements. - split up writing of cookies into its own function. - turn maxage into a signed int and use -1 for it not being set. - lots of style fixes - remove HTTP_COOKIE_DEFAULT, just pass 0 if you don't want flags.
2017-02-07 22:03:06 +01:00
TAILQ_FOREACH(ck, &(req->resp_cookies), list)
http_write_response_cookie(ck);
Cookies and arguments parsing improvements (#166) Add new cookie API for handling of cookies.
2017-02-07 21:49:10 +01:00
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
TAILQ_FOREACH(hdr, &(req->resp_headers), list) {
Use a single header_buf in http_response_normal() Avoids constantly allocating and freeing buffers all the time. This can be done safely due to the fact workers are single threaded.
2014-08-22 14:38:48 +02:00
kore_buf_appendf(header_buf, "%s: %s\r\n",
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
hdr->header, hdr->value);
}
Shuffle some HTTP_REQUEST_NO_CONTENT_LENGTH around
2015-05-19 09:13:29 +02:00
if (status != 204 && status >= 200 &&
!(req->flags & HTTP_REQUEST_NO_CONTENT_LENGTH)) {
kore_buf_appendf(header_buf,
Code cleanup, several API breaking changes in here
2016-07-08 10:02:18 +02:00
"content-length: %zu\r\n", len);
Shuffle some HTTP_REQUEST_NO_CONTENT_LENGTH around
2015-05-19 09:13:29 +02:00
}
} else {
if (status != 204 && status >= 200) {
kore_buf_appendf(header_buf,
Code cleanup, several API breaking changes in here
2016-07-08 10:02:18 +02:00
"content-length: %zu\r\n", len);
Shuffle some HTTP_REQUEST_NO_CONTENT_LENGTH around
2015-05-19 09:13:29 +02:00
}
}
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
Properly end headers when doing status codes only.
2014-08-25 10:58:34 +02:00
kore_buf_append(header_buf, "\r\n", 2);
Introduce NOHTTP=1 build option. This basically turns off the HTTP layer for Kore. It does not compile in anything for HTTP. This allows Kore to be used as a network application platform as well. Added an example for this called nohttp. Other changes that sneaked in while hacking on this: * Use calloc(), kill pendantic malloc option. * Killed off SPDY/3.1 support completely, will be superseded by http2 Note that comes with massive changes to a lot of the core API functions provided by Kore, these might break your application.
2015-11-27 16:22:50 +01:00
net_send_queue(c, header_buf->data, header_buf->offset);
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
Add the http_pretty_error configuration option. When enabled, Kore returns HTML error pages for status codes 4xx and 5xx instead of empty content.
2020-03-03 12:53:18 +01:00
if (req != NULL && req->method == HTTP_METHOD_HEAD)
send_body = 0;
if (d != NULL && send_body)
Introduce NOHTTP=1 build option. This basically turns off the HTTP layer for Kore. It does not compile in anything for HTTP. This allows Kore to be used as a network application platform as well. Added an example for this called nohttp. Other changes that sneaked in while hacking on this: * Use calloc(), kill pendantic malloc option. * Killed off SPDY/3.1 support completely, will be superseded by http2 Note that comes with massive changes to a lot of the core API functions provided by Kore, these might break your application.
2015-11-27 16:22:50 +01:00
net_send_queue(c, d, len);
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
allow an iterator to be passed to req.response(). if an iterator is passed kore will send the response with transfer-encoding: chunked and call the iterator for every chunk that was sent. The iterator must return a utf-8 string. Works wonderful with TemplateStream from jinja2.
2019-05-03 13:42:34 +02:00
if (!(c->flags & CONN_CLOSE_EMPTY) && !(c->flags & CONN_IS_BUSY))
http_start_recv(c);
Change accesslog format to Combined Log Format.
2018-06-28 14:25:32 +02:00
if (req != NULL)
req->content_length = len;
Add the http_pretty_error configuration option. When enabled, Kore returns HTML error pages for status codes 4xx and 5xx instead of empty content.
2020-03-03 12:53:18 +01:00
Rework http_pretty_error a tiny bit.
2020-03-04 08:36:10 +01:00
kore_buf_cleanup(&buf);
Major improvements to HTTP/1.1 behaviour, error reporting and handling. This comes with changes to http_response() (no longer returns a result).
2014-01-29 22:48:51 +01:00
}
cookie improvements. - split up writing of cookies into its own function. - turn maxage into a signed int and use -1 for it not being set. - lots of style fixes - remove HTTP_COOKIE_DEFAULT, just pass 0 if you don't want flags.
2017-02-07 22:03:06 +01:00
static void
http_write_response_cookie(struct http_cookie *ck)
{
struct tm tm;
char expires[HTTP_DATE_MAXSIZE];
kore_buf_reset(ckhdr_buf);
kore_buf_appendf(ckhdr_buf, "%s=%s", ck->name, ck->value);
if (ck->path != NULL)
kore_buf_appendf(ckhdr_buf, "; Path=%s", ck->path);
if (ck->domain != NULL)
kore_buf_appendf(ckhdr_buf, "; Domain=%s", ck->domain);
if (ck->expires > 0) {
if (gmtime_r(&ck->expires, &tm) == NULL) {
kore_log(LOG_ERR, "gmtime_r(): %s", errno_s);
return;
}
if (strftime(expires, sizeof(expires),
"%a, %d %b %y %H:%M:%S GMT", &tm) == 0) {
kore_log(LOG_ERR, "strftime(): %s", errno_s);
return;
}
kore_buf_appendf(ckhdr_buf, "; Expires=%s", expires);
}
change type of maxage.
2017-03-10 14:36:51 +01:00
if (ck->maxage > 0)
kore_buf_appendf(ckhdr_buf, "; Max-Age=%u", ck->maxage);
cookie improvements. - split up writing of cookies into its own function. - turn maxage into a signed int and use -1 for it not being set. - lots of style fixes - remove HTTP_COOKIE_DEFAULT, just pass 0 if you don't want flags.
2017-02-07 22:03:06 +01:00
if (ck->flags & HTTP_COOKIE_HTTPONLY)
kore_buf_appendf(ckhdr_buf, "; HttpOnly");
if (ck->flags & HTTP_COOKIE_SECURE)
kore_buf_appendf(ckhdr_buf, "; Secure");
kore_buf_appendf(header_buf, "set-cookie: %s\r\n",
kore_buf_stringify(ckhdr_buf, NULL));
}
HTTP improvements. Introduce an on_headers callback for routes, allowing one to inspect the headers before the request is processed further. Additionall, Add a new way of obtaining HTTP headers. Much like http_argument_get_*() functions, these new APIs allow you to fetch the data of an HTTP header as a specified C type. The new APIs are: * http_request_header_int16() * http_request_header_uint16() * http_request_header_int32() * http_request_header_uint32() * http_request_header_int64() * http_request_header_uint64() * http_request_header_float() * http_request_header_double() Should make it easier to operate in HTTP header data in a safe way. No need to always roll your own string to int conversion functions.
2021-09-15 22:16:22 +02:00
static int
http_data_convert(void *data, void **out, void *nout, int type)
{
switch (type) {
case HTTP_ARG_TYPE_RAW:
case HTTP_ARG_TYPE_STRING:
*out = data;
return (KORE_RESULT_OK);
case HTTP_ARG_TYPE_BYTE:
COPY_ARG_TYPE(*(u_int8_t *)data, u_int8_t);
return (KORE_RESULT_OK);
case HTTP_ARG_TYPE_INT16:
COPY_AS_INTTYPE(SHRT_MIN, SHRT_MAX, int16_t);
return (KORE_RESULT_OK);
case HTTP_ARG_TYPE_UINT16:
COPY_AS_INTTYPE(0, USHRT_MAX, u_int16_t);
return (KORE_RESULT_OK);
case HTTP_ARG_TYPE_INT32:
COPY_AS_INTTYPE(INT_MIN, INT_MAX, int32_t);
return (KORE_RESULT_OK);
case HTTP_ARG_TYPE_UINT32:
COPY_AS_INTTYPE(0, UINT_MAX, u_int32_t);
return (KORE_RESULT_OK);
case HTTP_ARG_TYPE_INT64:
COPY_AS_INTTYPE_64(int64_t, 1);
return (KORE_RESULT_OK);
case HTTP_ARG_TYPE_UINT64:
COPY_AS_INTTYPE_64(u_int64_t, 0);
return (KORE_RESULT_OK);
case HTTP_ARG_TYPE_FLOAT:
COPY_ARG_DOUBLE(-FLT_MAX, FLT_MAX, float);
return (KORE_RESULT_OK);
case HTTP_ARG_TYPE_DOUBLE:
COPY_ARG_DOUBLE(-DBL_MAX, DBL_MAX, double);
return (KORE_RESULT_OK);
default:
break;
}
return (KORE_RESULT_ERROR);
}