This commit renames certain POST centric variable and configuration
naming to the correct HTTP body stuff.
API changes include http_postbody_text() and http_postbody_bytes() to
have become http_body_text() and http_body_bytes().
The developer is still responsible for validating the method their
page handler is called with. Hopefully this becomes a configuration
option soon enough.
This function uses PQsendQueryParams() instead of the normal PQsendQuery()
allowing you to pass binary data in a cleaner fashion.
A basic call would look something like:
char *mydata = "Hello";
size_t mydata_len = strlen(mydata);
kore_pgsql_query_params(&pgsql, req,
"INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1
mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT);
kore_pgsql_query_params() is variadic, allowing you to pass any
count of parameters where each parameter has the following:
data pointer, data length, type of parameter.
I rather keep the old idioms instead of adding more complex things
on top of the async ones. Especially since the simple layer would
interfear with existing http state machines from your handler.
This simple query allows you to ditch rolling your own
state machine for handling async pgsql states and instead
asks you to provide 3 functions:
- init
- results
- done
You can see the different in complexity in the pgsql example,
which now contains a pgsql_simple.c holding the same asynchronous
query as in pgsql.c but using the simple pgsql api.
You can of course still roll your own in case you want more control.
Instead of letting http_requests spin, if we cannot allocate
a connection for the request we will queue them up put them to sleep.
When a connection becomes available, we'll wake up a request that
was waiting for a connection and let it continue.
This completely avoids consuming massive amounts of cpu time
when dealing with thousands of requests waiting for a pgsql
worker to become ready.
This way developers can cast assets to char * without first
having to convert it to a proper C string.
The NUL character is NOT included in the length of the asset.
Gone is the ugly KORE_PGSQL macro that hid an overly complex
state machine for the pgsql api.
Gone is the pgsql array that was attached to http_requests.
Gone are the callback hacks inside the pgsql api.
Instead, I strongly encourage people to use the new state machine
api Kore offers to properly deal with asynchronous queries.
The pgsql example in examples/pgsql has been updated to reflect
these changes.
In order to use this, define states for your page handler:
struct http_state mystates[] = {
{ "PAGE_STATE_INIT", page_init },
{ "PAGE_STATE_RESULT", page_result },
};
In your page handler you can then simply call http_state_run() with
your states and http_request. This will cause Kore to start calling
your state callbacks beginning at index 0.
State callbacks have the same prototype as page handlers:
int func(struct http_request *);
However, unlike page handlers they MUST return one of the following:
- HTTP_STATE_OK: All good, just continue the fsm.
- HTTP_STATE_ERROR: Abort fsm and return KORE_RESULT_OK to Kore
(This will cancel the http request).
- HTTP_STATE_RETRY: Return KORE_RESULT_RETRY to Kore.
(Kore will retry your page handler next event loop).
- HTTP_STATE_COMPLETE: The fsm completed, break out cleanly.
Note that using this is completely optional and you can still
use the traditional way of writing page handlers.
The fsm is designed to get rid of the clutter that exists today
in Kore when dealing with non blocking tasks or pgsql calls.
This way we can get our code called whenever a stream is
completed. This cb handler does stand alone from an http_request
and is passed a netbuf data structure.
* Always make sure we end the stream properly
* Check for SPDY_FLOW_WINDOW_MAX on window frame updates
* Kill SPDY_STREAM_BLOCKING, once flow control kicks in its per session
After revisiting why this exists in Kore I decided it
does not belong in this platform and instead of letting
it sit there staring at me I rather just kill it.
* kore build will now only rebuild what was changed (checking on
last modified timestamp, not 100% accurate but it'll do).
* introduce kore clean which cleans out object files and the
resulting .so file
Makes more sense and reads easier:
kore create myapp
kore build myapp
kore run myapp
Note that kore retains its cli options (if no command was given),
meaning you can still start kore in the traditional way as well.
The command options are simply to make development easier.
Personally use this for testing Kore its performance without
letting the OpenSSL stack get in the way too much.
Note that it leaves data structures as is, and just removes
any calls to OpenSSL (and removes the linking vs OpenSSL).
Orbit is a tool used to create/build/run kore code
in a more straight forward way.
Right now, orbit can do the following:
orbit create myapp
orbit build myapp
orbit run myapp
All in the motivation of making it easier to get
started with building Kore libraries and making
life easier on the people hacking Kore things.
When running in -f (foreground) you can now specify
the library Kore needs to load on the command line:
kore -fnc module.conf myapp.so
This has the benefit that your configuration file no
longer needs the load directive when hacking on your code.
Note that you can still specify load in your config file
regardless, if you so chose.
All of this is being done in order to try and move away
from the backwards way of getting up and running with Kore.
Kore no longer passes the accept lock to the "next in line"
worker but instead all workers will attempt to grab the lock
if they can.
Also remember if we had the lock previous iteration of the
event loop and don't constantly disable/enable the accepting sockets.
Makes Kore scale even better across multiple cpu's.
These 2 functions can be used to move an HTTP request
from/to the active http_requests list. Effectively
putting them to "sleep" or "waking them up".
Sprinkle this through the pgsql and task code.
If used correctly greatly reduces overhead for
managing sleeping tasks.
Synchronize access to state/result properly so one
can access these from inside the task as well.
Introduce KORE_TASK_STATE_ABORT which will be set
when a task needs to be abort. You can use this
to create tasks that run in a loop until aborted.
Instead if a task is used from inside a request
you MUST call kore_task_bind_request() on it.
This way we can move forward for tasks that
don't belong to page handlers.
Also, some bug fixes for removing http_requests
that are indeed linked to a currently running task.
Tasks are now assigned to available threads instead
of a global task list.
You can now pass messages between your page handler
and the created task using the kore_task_channel_*
functions.
Only one task per time can be assigned to a request
but I feel this is probably a bad design choice.
Preferably we'd want to be able to start tasks
regardless of being in a page handler or not,
this not only ads flexibility but seems like
a better choice overall as it opens a lot more
possibilities about how tasks can be used.
Don't wait for a full event loop until we call the page handler
for a received pgsql result. This speeds up page loads using
KORE_PGSQL by quite a lot, especially on a non busy server.
Prevents the weird freelist munging that happens when we're
actually freeing memory (which we should be forcing it to do
by setting freelist_max_len to 0).
Including but not limited to:
- Correctly use PQerrorMessage() in case we cleanup with PQfinish
- If we get a network error, cleanup the connection
- No longer call the page handler from inside kore_pgsql_handle()
but instead just put it to sleep in case we don't need it.
This does grow the http_requests list quite a bit with sleeping
connections and can perhaps be improved later on.
- Allow us to on error return OK from a page handler from inside
the completetion block for KORE_PGSQL().
- Count the cummulative time for a request to finish instead
of the latest run time for the handler.
Has support for full async pgsql queries. Most of the logic
is hidden behind a KORE_PGSQL() macro allowing you to insert
these pgsql calls in your page handlers without blocking the
kore worker while the query is going off.
There is place for improvement here, and perhaps KORE_PGSQL won't
stay as I feel this might overcomplicate things instead of making
them simpler as I thought it would.
Double check we actually get a certificate if we are asking for one.
Even though we set SSL_VERIFY_FAIL_IF_NO_PEER_CERT it's a sane thing to do.
Start logging the CN for the received client certificate in the access logs.
As a bonus re-arrange some accesslog stuff for sanity.
request can be used for when you want to validate something for
authentication bmanually. Youur validator will receive the http_reques
passed down.
A practical use of this is doing IP based ACL's.
Using authentication blocks one can define "authentication" mechanisms
in Kore for page handlers.
This can be used to require a session cookie (validated by your own validator)
for certain page handlers, and hopefully in the future provide a framework
for adding more authentication things (like HTTP Auth).
Right now only cookie checking is available.
- Parameter validation is now done only when http_process_*()
is called and upon http_argument_add().
- You MUST have defined your params in a param block or they will
be filtered out.
- http_argument_lookup() is dead, welcome http_argument_get() and
its brothers and sisters:
http_argument_get_string()
http_argument_get_uint16()
http_argument_get_int16()
http_argument_get_uint32()
http_argument_get_int32()
They will automatically do bounds checking on integers for you
and return proper integers or a NUL-terminated string.
- The http_argument_get* functions no longer create an additional
copy of the string which you need to free. Easier going.
- http_multiple_args() is dead, byebye
- Make some stuff we don't want to share with the modules static.
If a worker reached worker_max_connections and it was its turn to
grab the accept lock it would've gotten stuck and no new connections
would be handled even if other workers would be less busy.
Instead, we now skip the lock if we're too busy and pass it along
in the hopes other workers are less busy.
Example:
validator v_id function v_id_function
validator v_url regex ^/url/path/[a-z]*$
You can then call these using kore_validator_run(char *, char *), example:
if (!kore_validator_run("v_url", req->path))
[req->path is bad];
Joris Vink
James Turner
Thordur Bjornsson