Before kodev always picked up the kore headers installed on the system.
This presented some annoying problems as the system headers may not
match the headers used by the kore_source you're actually building
against.
This option allows a user to finetune the number of milliseconds
a worker process will max spend inside the http_process() loop.
By default this is 10ms.
Do not run the idle timer check for client if it still has a request
queued up. Otherwise if the worker process is very busy you might hit
the timeout even though the client sent us a full request which was queued.
The HTTP layer used to make a copy of each incoming header and its
value for a request. Stop doing that and make HTTP headers zero-copy
all across the board.
This change comes with some api function changes, notably the
http_request_header() function which now takes a const char ** rather
than a char ** out pointer.
This commit also constifies several members of http_request, beware.
Additional rework how the worker processes deal with the accept lock.
Before:
if a worker held the accept lock and it accepted a new connection
it would release the lock for others and back off for 500ms before
attempting to grab the lock again.
This approach worked but under high load this starts becoming obvious.
Now:
- workers not holding the accept lock and not having any connections
will wait less long before returning from kore_platform_event_wait().
- workers not holding the accept lock will no longer blindly wait
an arbitrary amount in kore_platform_event_wait() but will look
at how long until the next lock grab is and base their timeout
on that.
- if a worker its next_lock timeout is up and failed to grab the
lock it will try again in half the time again.
- the worker process holding the lock will when releasing the lock
double check if it still has space for newer connections, if it does
it will keep the lock until it is full. This prevents the lock from
bouncing between several non busy worker processes all the time.
Additional fixes:
- Reduce the number of times we check the timeout list, only do it twice
per second rather then every event tick.
- Fix solo worker count for TLS (we actually hold two processes, not one).
- Make sure we don't accidentally miscalculate the idle time causing new
connections under heavy load to instantly drop.
- Swap from gettimeofday() to clock_gettime() now that MacOS caught up.
Limits the number of queued asynchronous queries kore will allow
before starting to return errors for KORE_PGSQL_ASYNC setups.
By default set to 1000.
Avoids uncontrolled growth of the pgsql_queue_wait pool.
Before http_request_limit just constrained the number of HTTP
requests we'd deal with in a single http_process_requests() call.
But it should really mean how many maximum HTTP requests are allowed
to be alive in the worker process before we start sending 503s back.
While here, drop the lock timeout for a worker to 100ms down from 500ms
and do not allow a worker to grab the accept lock if their HTTP request
queue is full.
This makes things much more pleasant memory wise as the http_request_pool
won't just grow over time.
Before params get would mean querystring and anything else
would just count toward a www-encoded body.
Now you can prefix the params block with "qs" indicating that
those configured parameters are allowed to occur in the query
string regardless of the method used.
This means you can do something like:
params qs:post /uri {
...
}
to specify what the allowed parameters are in the querystring for
a POST request towards /uri.
inspired by and properly fixes#205.
build the frame in a kore_buf on the stack and use net_send_stream()
to push it out rather then making yet another copy in net_send().
saves several cpu cycles before the frame starts going out as
we're not allocating a kore_buf, its contents and then memcpy()ing
that contents to a new netbuf.
doing this allows us to get rid of the validator reload
and handler reload as well as fixing websocket runtime
callbacks which were never being resolved upon module reloads.
This allows modules that have global pointers to upon reload repopulate
those with the addresses of when they were first created.
Meaning it now is easier to write modules that can be reloaded if those
modules kept global state one way or the other.
This should only be used at module init/reload time and is very simple:
if ((ptr = kore_mem_lookup(MY_ID_VALUE)) == NULL) {
ptr = kore_malloc_tagged(length, MY_ID_VALUE);
/* initialize for the first time. */
}
If we were in a reload the kore_mem_lookup() will return the original address
returned by the initial kore_malloc_tagged() call for MY_ID_VALUE.
- make sure conn_count per pgsqldb structure is initialized to 0.
- allow pgsql_conn_max to be 0, meaning just create a new connection
if none was free.
if we fail at rolling back an in-error transaction on a connection
just remove that connection and go back to rescanning rather then
returning an error to the caller, there may be more functional
connections in the pipeline.
- Make pgsql_conn_count count per database rather then globally.
This means you now define the number of clients *per* database registered
rather then the number of clients in total of all databases.
- In case a connection is in failed transaction state Kore will now
automatically rollback the transaction before placing that connection
back in the connection pool.
LibreSSL defines OPENSSL_VERSION_NUMBER as 0x20000000L but does not have
the 1.1.0 API so we have to carefully check for LIBRESSL_VERSION_TEXT as
well before using that new API.
Eventually I will phase out 1.0.2 down the line to get rid of the
nightmare that is the 2 different APIs.
This commit adds full support for building kore with 1.1.0e while
retaining the privsep keymanager support.
based on excellent work done by @hiwk.
When the pgsql layer was introduced it was tightly coupled with the
http layer in order to make async work fluently.
The time has come to split these up and follow the same method we
used for tasks, allowing either http requests to be tied to a pgsql
data structure or a simple callback function.
This also reworks the internal queueing of pgsql requests until
connections to the db are available again.
The following API functions were changes:
- kore_pgsql_query_init() -> kore_pgsql_setup()
no longer takes an http_request parameter.
- NEW kore_pgsql_init()
must be called before operating on an kore_pgsql structure.
- NEW kore_pgsql_bind_request()
binds an http_request to a kore_pgsql data structure.
- NEW kore_pgsql_bind_callback()
binds a callback to a kore_pgsql data structure.
With all of this you can now build kore with PGSQL=1 NOHTTP=1.
The pgsql/ example has been updated to reflect these changes and
new features.
This commit allows worker processes to have individual listeners
not configured by the kore configuration.
If you do the following:
- do not configure listeners in your .conf file
- call kore_server_bind() in kore_worker_configure()
the workers will no longer fight over accept locks as the configured
ports no longer conflict with each other.
This allows me to create X amount of instances of a worker process that
are each individually accessible via unique ports.
Convert the data parameter to a string if the op was WEBSOCKET_OP_TEXT
or convert to bytes if op was WEBSOCKET_OP_BINARY so the callee does
not have to do this anymore.
Also let this function reset offset and lengths for http_body_read().
Make sure of this function in the python code so req.body can be called
multiple times in succession.
The only reason you would want to directly modify the cookie
after creating it should be to unset the HTTPONLY or SECURE flags
if that is what you *really* want to do.
Change http_response_cookie() to take all required parameters instead
of having to marshall those in yourself after.
Now you set a sane default cookie in one shot:
http_response_cookie(req, "key", "value", "/", 0, -1, NULL);
Which would create a session cookie key=value for / under the current domain.
We now default to httponly & secure for newly created cookies.
This should've been the default all along.
The http_response_cookie() no longer returns a pointer to http_cookie
but rather takes it as a parameter and will populate the pointer with
the newly created http_cookie if not NULL.
Additionally http_response_cookie() automatically sets the domain
based on the http_request passed into the function.
At bootup and every 1800 seconds after that the worker processes will
ask the keymgr for new entropy that they will seed into their PRNG.
Additionally once received the worker calls RAND_poll() to grab
more entropy from the system to be mixed in.
This option allows the user to specify a file to be used for
seeding the PRNG initially and to write random bytes at exit.
The option is only available if kore has TLS enabled (by default).
If you enable this option Kore will refuse to start if there is
a problem with the file specified (not found, not a file, invalid size, etc).
While here let the keymgr process call RAND_poll() every half hour
to grab more system entropy and seed it into the PRNG.
We were not returning zeroed out memory from kore_calloc() which goes
against what calloc() does. Skip performance for now and simply just
memset() the returned pointer from kore_malloc().
This should be sufficient enough for now.
This will look at the kore.pid file in the current directory
and send a SIGHUP signal to it. It's mostly a handy shortcut
since you could of course do a kill -HUP `cat kore.pid` easily.
Before this function would block client I/O and existing HTTP requests
until the keymgr process responsed with a result.
This commit changes that behaviour and makes this function call
the http_process() function if we end up waiting for the keymgr.
This means that while waiting for a response we at least start
making headway with existing HTTP requests if the response is
not immediate.
Having the create, build, run tools baked into the kore binary
made things harder then they had to be for multiple projects with
each different build flavors.
So move away this functionality into a new "kodev" (name may change)
binary that is installed next to kore.
The new build tools will automatically pick up the correct flavors
the kore binary it points to is installed with. Or for single builds
what flavors where enabled.
The new tool also will honor looking into PREFIX for the kore binary
when doing a `kodev run`.
Additionally add a new command "info" that shows some basic info
about your project and how it will be built. For example it will
show you the flavors of the kore binary installed on the system
or the flavors you configured for a single binary build.
Obligitory, hacking on a plane comment.
- split up writing of cookies into its own function.
- turn maxage into a signed int and use -1 for it not being set.
- lots of style fixes
- remove HTTP_COOKIE_DEFAULT, just pass 0 if you don't want flags.
This commit adds the ability to use python "await" to suspend
execution of your page handler until the query sent to postgresql
has returned a result.
This is built upon the existing asynchrous query framework Kore had.
With this you can now write stuff like:
async def page(req):
result = await req.pgsql("db", "SELECT name FROM table");
req.response(200, json.dumps(result).encode("utf-8"))
The above code will fire off a query and suspend itself so Kore can
take care of business as usual until the query is successful at which
point Kore will jump back into the handler and resume.
This does not use threading, it's purely based on Python's excellent
coroutines and generators and Kore its built-in pgsql support.
- adds new cleanup function that workers will call.
- adds kore_pgsql_nfields() to return number of fields in result.
- add kore_pgsql_fieldname() to return name of a given field.
This commit also changes the behaviour of pgsql_conn_release() in
that it will now cancel the active query before releasing the connection.
This makes sure that if long running queries are active they are hopefully
cancelled if an http request is removed while such queries are still running.
Renamed both of them:
kore_preload -> kore_parent_configure
kore_onload -> kore_worker_configure
These functions will now always be called if they are defined in any module
regardless of your application being built as a single binary or not.
we just increase the lenght requested to 8 bytes if we get a 0 byte request.
additionally when kore_realloc() is called check if we actually have
to do the work, if not just return the original pointer.
These functions are created by the cli tool when building
and follow the naming format: asset_serve_<name>_<ext>().
Those serving functions can be used directly in handlers and
callthrough to a http_serveable() function that uses the SHA1
of the asset as its ETag and automatically checks for if-none-match.
Joris Vink
Marcin Szczepaniak
rouzier
Stanislav Yudin
Yorick de Wid
Yvan Sraka