mirrors
/
kore
mirror of https://git.kore.io/kore.git
1
0
Fork 1
Code Issues Projects Releases Wiki Activity
kore/src/pgsql.c

467 lines
10 KiB
C
Raw Normal View History

Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
/*
* Copyright (c) 2014 Joris Vink <joris@coders.se>
*
* Permission to use, copy, modify, and distribute this software for any
* purpose with or without fee is hereby granted, provided that the above
* copyright notice and this permission notice appear in all copies.
*
* THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
* WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
* ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
* WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
* ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
* OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
#include <sys/param.h>
#include <sys/queue.h>
#include <libpq-fe.h>
Pull in pg_config.h correctly, fixes compilation for pgsql under OpenBSD
2014-07-18 15:59:07 +02:00
#include <pg_config.h>
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
#include "kore.h"
#include "http.h"
Normalize kore_*.h headers
2014-07-03 22:14:46 +02:00
#include "pgsql.h"
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
struct pgsql_job {
char *query;
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
struct http_request *req;
struct kore_pgsql *pgsql;
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
TAILQ_ENTRY(pgsql_job) list;
};
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
struct pgsql_wait {
struct http_request *req;
TAILQ_ENTRY(pgsql_wait) list;
};
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
#define PGSQL_IS_BLOCKING 0
#define PGSQL_IS_ASYNC 1
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
#define PGSQL_CONN_MAX 2
#define PGSQL_CONN_FREE 0x01
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
static void pgsql_queue_wakeup(void);
static int pgsql_conn_create(struct kore_pgsql *);
static void pgsql_queue_add(struct http_request *);
Properly release pgsql connection if needed
2014-08-14 14:46:21 +02:00
static void pgsql_conn_release(struct kore_pgsql *);
Layout that stuff like a 80 column terminal nerd. And mark a request as DONE when we get PGRES_COMMAND_OK.
2014-03-31 01:04:22 +02:00
static void pgsql_conn_cleanup(struct pgsql_conn *);
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
static void pgsql_read_result(struct kore_pgsql *, int);
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
static void pgsql_schedule(struct kore_pgsql *, struct http_request *);
static int pgsql_prepare(struct kore_pgsql *, struct http_request *,
const char *);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
static struct kore_pool pgsql_job_pool;
static struct kore_pool pgsql_wait_pool;
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
static TAILQ_HEAD(, pgsql_conn) pgsql_conn_free;
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
static TAILQ_HEAD(, pgsql_wait) pgsql_wait_queue;
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
static u_int16_t pgsql_conn_count;
Make the pgsql connstring programmatically configurable
2014-04-14 08:41:41 +02:00
char *pgsql_conn_string = NULL;
Add pgsql_conn_max configuration parameter. Allows you to tune how many pgsql connections kore will make at one time.
2014-07-04 09:14:05 +02:00
u_int16_t pgsql_conn_max = PGSQL_CONN_MAX;
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
void
kore_pgsql_init(void)
{
pgsql_conn_count = 0;
TAILQ_INIT(&pgsql_conn_free);
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
TAILQ_INIT(&pgsql_wait_queue);
kore_pool_init(&pgsql_job_pool, "pgsql_job_pool",
sizeof(struct pgsql_job), 100);
kore_pool_init(&pgsql_wait_pool, "pgsql_wait_pool",
sizeof(struct pgsql_wait), 100);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
}
int
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
kore_pgsql_query(struct kore_pgsql *pgsql, struct http_request *req,
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
const char *query)
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
{
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
if (!pgsql_prepare(pgsql, req, query))
return (KORE_RESULT_ERROR);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
if (!PQsendQuery(pgsql->conn->db, query)) {
pgsql_conn_cleanup(pgsql->conn);
return (KORE_RESULT_ERROR);
}
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
pgsql_schedule(pgsql, req);
return (KORE_RESULT_OK);
}
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
int
kore_pgsql_query_params(struct kore_pgsql *pgsql, struct http_request *req,
const char *query, int result, u_int8_t count, ...)
{
u_int8_t i;
va_list args;
char **values;
int *lengths, *formats;
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
if (!pgsql_prepare(pgsql, req, query))
return (KORE_RESULT_ERROR);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
va_start(args, count);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
lengths = kore_calloc(count, sizeof(int));
formats = kore_calloc(count, sizeof(int));
values = kore_calloc(count, sizeof(char *));
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
for (i = 0; i < count; i++) {
values[i] = va_arg(args, void *);
lengths[i] = va_arg(args, u_int32_t);
formats[i] = va_arg(args, int);
}
Attach tasks/pgsqls to http_requests once more. This way if an http request is removed while tasks or pgsqls are still active they are free'd out and cancelled properly.
2014-08-14 22:05:34 +02:00
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
if (!PQsendQueryParams(pgsql->conn->db, query, count, NULL,
(const char * const *)values, lengths, formats, result)) {
kore_mem_free(values);
kore_mem_free(lengths);
kore_mem_free(formats);
pgsql_conn_cleanup(pgsql->conn);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
return (KORE_RESULT_ERROR);
}
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
kore_mem_free(values);
kore_mem_free(lengths);
kore_mem_free(formats);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
pgsql_schedule(pgsql, req);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
return (KORE_RESULT_OK);
}
void
kore_pgsql_handle(void *c, int err)
{
struct http_request *req;
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
struct kore_pgsql *pgsql;
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
struct pgsql_conn *conn = (struct pgsql_conn *)c;
Improvements to pgsql contrib code. Including but not limited to: - Correctly use PQerrorMessage() in case we cleanup with PQfinish - If we get a network error, cleanup the connection - No longer call the page handler from inside kore_pgsql_handle() but instead just put it to sleep in case we don't need it. This does grow the http_requests list quite a bit with sleeping connections and can perhaps be improved later on. - Allow us to on error return OK from a page handler from inside the completetion block for KORE_PGSQL(). - Count the cummulative time for a request to finish instead of the latest run time for the handler.
2014-03-31 00:57:00 +02:00
if (err) {
pgsql_conn_cleanup(conn);
return;
}
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
req = conn->job->req;
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
pgsql = conn->job->pgsql;
kore_debug("kore_pgsql_handle: %p (%d)", req, pgsql->state);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
if (!PQconsumeInput(conn->db)) {
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
pgsql->state = KORE_PGSQL_STATE_ERROR;
pgsql->error = kore_strdup(PQerrorMessage(conn->db));
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
} else {
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
pgsql_read_result(pgsql, PGSQL_IS_ASYNC);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
}
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
if (pgsql->state == KORE_PGSQL_STATE_WAIT) {
Introduce http_request_sleep() and http_request_wakeup(). These 2 functions can be used to move an HTTP request from/to the active http_requests list. Effectively putting them to "sleep" or "waking them up". Sprinkle this through the pgsql and task code. If used correctly greatly reduces overhead for managing sleeping tasks.
2014-07-04 16:51:19 +02:00
http_request_sleep(req);
Kore pgsql improvements. Don't wait for a full event loop until we call the page handler for a received pgsql result. This speeds up page loads using KORE_PGSQL by quite a lot, especially on a non busy server.
2014-04-17 10:49:48 +02:00
} else {
Introduce http_request_sleep() and http_request_wakeup(). These 2 functions can be used to move an HTTP request from/to the active http_requests list. Effectively putting them to "sleep" or "waking them up". Sprinkle this through the pgsql and task code. If used correctly greatly reduces overhead for managing sleeping tasks.
2014-07-04 16:51:19 +02:00
http_request_wakeup(req);
Kore pgsql improvements. Don't wait for a full event loop until we call the page handler for a received pgsql result. This speeds up page loads using KORE_PGSQL by quite a lot, especially on a non busy server.
2014-04-17 10:49:48 +02:00
}
Improvements to pgsql contrib code. Including but not limited to: - Correctly use PQerrorMessage() in case we cleanup with PQfinish - If we get a network error, cleanup the connection - No longer call the page handler from inside kore_pgsql_handle() but instead just put it to sleep in case we don't need it. This does grow the http_requests list quite a bit with sleeping connections and can perhaps be improved later on. - Allow us to on error return OK from a page handler from inside the completetion block for KORE_PGSQL(). - Count the cummulative time for a request to finish instead of the latest run time for the handler.
2014-03-31 00:57:00 +02:00
}
void
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
kore_pgsql_continue(struct http_request *req, struct kore_pgsql *pgsql)
Improvements to pgsql contrib code. Including but not limited to: - Correctly use PQerrorMessage() in case we cleanup with PQfinish - If we get a network error, cleanup the connection - No longer call the page handler from inside kore_pgsql_handle() but instead just put it to sleep in case we don't need it. This does grow the http_requests list quite a bit with sleeping connections and can perhaps be improved later on. - Allow us to on error return OK from a page handler from inside the completetion block for KORE_PGSQL(). - Count the cummulative time for a request to finish instead of the latest run time for the handler.
2014-03-31 00:57:00 +02:00
{
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
kore_debug("kore_pgsql_continue: %p->%p (%d)",
req->owner, req, pgsql->state);
Kore pgsql improvements. Don't wait for a full event loop until we call the page handler for a received pgsql result. This speeds up page loads using KORE_PGSQL by quite a lot, especially on a non busy server.
2014-04-17 10:49:48 +02:00
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
if (pgsql->error) {
kore_mem_free(pgsql->error);
pgsql->error = NULL;
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
}
pgsql improvements * set pgsql->result to NULL when we freed it * drain the connection pipe just in case in pgsql_conn_release()
2014-08-14 16:43:40 +02:00
if (pgsql->result) {
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
PQclear(pgsql->result);
pgsql improvements * set pgsql->result to NULL when we freed it * drain the connection pipe just in case in pgsql_conn_release()
2014-08-14 16:43:40 +02:00
pgsql->result = NULL;
}
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
switch (pgsql->state) {
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
case KORE_PGSQL_STATE_INIT:
case KORE_PGSQL_STATE_WAIT:
break;
case KORE_PGSQL_STATE_DONE:
Introduce http_request_sleep() and http_request_wakeup(). These 2 functions can be used to move an HTTP request from/to the active http_requests list. Effectively putting them to "sleep" or "waking them up". Sprinkle this through the pgsql and task code. If used correctly greatly reduces overhead for managing sleeping tasks.
2014-07-04 16:51:19 +02:00
http_request_wakeup(req);
Properly release pgsql connection if needed
2014-08-14 14:46:21 +02:00
pgsql_conn_release(pgsql);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
break;
case KORE_PGSQL_STATE_ERROR:
case KORE_PGSQL_STATE_RESULT:
Properly release pgsql connection if needed
2014-08-14 14:46:21 +02:00
kore_pgsql_handle(pgsql->conn, 0);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
break;
default:
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
fatal("unknown pgsql state %d", pgsql->state);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
}
}
void
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
kore_pgsql_cleanup(struct kore_pgsql *pgsql)
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
{
Attach tasks/pgsqls to http_requests once more. This way if an http request is removed while tasks or pgsqls are still active they are free'd out and cancelled properly.
2014-08-14 22:05:34 +02:00
kore_debug("kore_pgsql_cleanup(%p)", pgsql);
Properly release pgsql connection if needed
2014-08-14 14:46:21 +02:00
if (pgsql->result != NULL)
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
PQclear(pgsql->result);
Improvements to pgsql contrib code. Including but not limited to: - Correctly use PQerrorMessage() in case we cleanup with PQfinish - If we get a network error, cleanup the connection - No longer call the page handler from inside kore_pgsql_handle() but instead just put it to sleep in case we don't need it. This does grow the http_requests list quite a bit with sleeping connections and can perhaps be improved later on. - Allow us to on error return OK from a page handler from inside the completetion block for KORE_PGSQL(). - Count the cummulative time for a request to finish instead of the latest run time for the handler.
2014-03-31 00:57:00 +02:00
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
if (pgsql->error != NULL)
kore_mem_free(pgsql->error);
pgsql_conn improvements. set conn inside kore_pgsql to NULL when we've completed. if at kore_pgsql_cleanup() time the conn is still there we'll drain it as per documentation.
2014-03-31 01:09:08 +02:00
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
if (pgsql->conn != NULL)
Properly release pgsql connection if needed
2014-08-14 14:46:21 +02:00
pgsql_conn_release(pgsql);
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
pgsql->result = NULL;
pgsql->error = NULL;
pgsql->conn = NULL;
Attach tasks/pgsqls to http_requests once more. This way if an http request is removed while tasks or pgsqls are still active they are free'd out and cancelled properly.
2014-08-14 22:05:34 +02:00
LIST_REMOVE(pgsql, rlist);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
}
amend example with additional query + kore_pgsql_logerror()
2014-04-02 23:01:47 +02:00
void
kore_pgsql_logerror(struct kore_pgsql *pgsql)
{
kore_log(LOG_NOTICE, "pgsql error: %s",
(pgsql->error) ? pgsql->error : "unknown");
}
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
int
pgsql fixes + new wrappers
2014-04-01 21:43:32 +02:00
kore_pgsql_ntuples(struct kore_pgsql *pgsql)
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
{
pgsql fixes + new wrappers
2014-04-01 21:43:32 +02:00
return (PQntuples(pgsql->result));
}
char *
kore_pgsql_getvalue(struct kore_pgsql *pgsql, int row, int col)
{
return (PQgetvalue(pgsql->result, row, col));
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
}
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
void
kore_pgsql_queue_remove(struct http_request *req)
{
struct pgsql_wait *pgw, *next;
for (pgw = TAILQ_FIRST(&pgsql_wait_queue); pgw != NULL; pgw = next) {
next = TAILQ_NEXT(pgw, list);
if (pgw->req != req)
continue;
TAILQ_REMOVE(&pgsql_wait_queue, pgw, list);
kore_pool_put(&pgsql_wait_pool, pgw);
return;
}
}
Add kore_pgsql_query_params(). This function uses PQsendQueryParams() instead of the normal PQsendQuery() allowing you to pass binary data in a cleaner fashion. A basic call would look something like: char *mydata = "Hello"; size_t mydata_len = strlen(mydata); kore_pgsql_query_params(&pgsql, req, "INSERT INTO foo VALUES($1::text)", KORE_PGSQL_FORMAT_TEXT, 1 mydata, mydata_len, KORE_PGSQL_FORMAT_TEXT); kore_pgsql_query_params() is variadic, allowing you to pass any count of parameters where each parameter has the following: data pointer, data length, type of parameter.
2014-09-28 21:39:16 +02:00
static int
pgsql_prepare(struct kore_pgsql *pgsql, struct http_request *req,
const char *query)
{
struct pgsql_conn *conn;
pgsql->state = KORE_PGSQL_STATE_INIT;
pgsql->result = NULL;
pgsql->error = NULL;
pgsql->conn = NULL;
if (TAILQ_EMPTY(&pgsql_conn_free)) {
if (pgsql_conn_count >= pgsql_conn_max) {
pgsql_queue_add(req);
return (KORE_RESULT_ERROR);
}
if (!pgsql_conn_create(pgsql))
return (KORE_RESULT_ERROR);
}
http_request_sleep(req);
conn = TAILQ_FIRST(&pgsql_conn_free);
if (!(conn->flags & PGSQL_CONN_FREE))
fatal("received a pgsql conn that was not free?");
conn->flags &= ~PGSQL_CONN_FREE;
TAILQ_REMOVE(&pgsql_conn_free, conn, list);
pgsql->conn = conn;
conn->job = kore_pool_get(&pgsql_job_pool);
conn->job->query = kore_strdup(query);
conn->job->pgsql = pgsql;
conn->job->req = req;
LIST_INSERT_HEAD(&(req->pgsqls), pgsql, rlist);
return (KORE_RESULT_OK);
}
static void
pgsql_schedule(struct kore_pgsql *pgsql, struct http_request *req)
{
int fd;
fd = PQsocket(pgsql->conn->db);
if (fd < 0)
fatal("PQsocket returned < 0 fd on open connection");
kore_platform_schedule_read(fd, pgsql->conn);
pgsql->state = KORE_PGSQL_STATE_WAIT;
kore_debug("query '%s' for %p sent on %p", query, req, pgsql->conn);
}
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
static void
pgsql_queue_add(struct http_request *req)
{
struct pgsql_wait *pgw;
http_request_sleep(req);
pgw = kore_pool_get(&pgsql_wait_pool);
pgw->req = req;
pgw->req->flags |= HTTP_REQUEST_PGSQL_QUEUE;
TAILQ_INSERT_TAIL(&pgsql_wait_queue, pgw, list);
}
static void
pgsql_queue_wakeup(void)
{
struct pgsql_wait *pgw, *next;
for (pgw = TAILQ_FIRST(&pgsql_wait_queue); pgw != NULL; pgw = next) {
next = TAILQ_NEXT(pgw, list);
if (pgw->req->flags & HTTP_REQUEST_DELETE)
continue;
http_request_wakeup(pgw->req);
pgw->req->flags &= ~HTTP_REQUEST_PGSQL_QUEUE;
TAILQ_REMOVE(&pgsql_wait_queue, pgw, list);
kore_pool_put(&pgsql_wait_pool, pgw);
return;
}
}
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
static int
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
pgsql_conn_create(struct kore_pgsql *pgsql)
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
{
struct pgsql_conn *conn;
Check connection string right before we connect
2014-04-14 08:45:31 +02:00
if (pgsql_conn_string == NULL)
fatal("pgsql_conn_create: no connection string");
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
pgsql_conn_count++;
conn = kore_malloc(sizeof(*conn));
kore_debug("pgsql_conn_create(): %p", conn);
memset(conn, 0, sizeof(*conn));
Make the pgsql connstring programmatically configurable
2014-04-14 08:41:41 +02:00
conn->db = PQconnectdb(pgsql_conn_string);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
if (conn->db == NULL || (PQstatus(conn->db) != CONNECTION_OK)) {
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
pgsql->state = KORE_PGSQL_STATE_ERROR;
pgsql->error = kore_strdup(PQerrorMessage(conn->db));
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
pgsql_conn_cleanup(conn);
return (KORE_RESULT_ERROR);
}
conn->job = NULL;
conn->flags = PGSQL_CONN_FREE;
conn->type = KORE_TYPE_PGSQL_CONN;
TAILQ_INSERT_TAIL(&pgsql_conn_free, conn, list);
return (KORE_RESULT_OK);
}
Properly release pgsql connection if needed
2014-08-14 14:46:21 +02:00
static void
pgsql_conn_release(struct kore_pgsql *pgsql)
{
int fd;
if (pgsql->conn == NULL)
return;
kore_mem_free(pgsql->conn->job->query);
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
kore_pool_put(&pgsql_job_pool, pgsql->conn->job);
Properly release pgsql connection if needed
2014-08-14 14:46:21 +02:00
pgsql improvements * set pgsql->result to NULL when we freed it * drain the connection pipe just in case in pgsql_conn_release()
2014-08-14 16:43:40 +02:00
/* Drain just in case. */
while (PQgetResult(pgsql->conn->db) != NULL)
;
Properly release pgsql connection if needed
2014-08-14 14:46:21 +02:00
pgsql->conn->job = NULL;
pgsql->conn->flags |= PGSQL_CONN_FREE;
TAILQ_INSERT_TAIL(&pgsql_conn_free, pgsql->conn, list);
fd = PQsocket(pgsql->conn->db);
kore_platform_disable_read(fd);
pgsql->conn = NULL;
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
pgsql->state = KORE_PGSQL_STATE_COMPLETE;
pgsql_queue_wakeup();
Properly release pgsql connection if needed
2014-08-14 14:46:21 +02:00
}
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
static void
pgsql_conn_cleanup(struct pgsql_conn *conn)
{
struct http_request *req;
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
struct kore_pgsql *pgsql;
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
kore_debug("pgsql_conn_cleanup(): %p", conn);
if (conn->flags & PGSQL_CONN_FREE)
TAILQ_REMOVE(&pgsql_conn_free, conn, list);
if (conn->job) {
req = conn->job->req;
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
pgsql = conn->job->pgsql;
Introduce http_request_sleep() and http_request_wakeup(). These 2 functions can be used to move an HTTP request from/to the active http_requests list. Effectively putting them to "sleep" or "waking them up". Sprinkle this through the pgsql and task code. If used correctly greatly reduces overhead for managing sleeping tasks.
2014-07-04 16:51:19 +02:00
http_request_wakeup(req);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
pgsql->conn = NULL;
pgsql->state = KORE_PGSQL_STATE_ERROR;
pgsql->error = kore_strdup(PQerrorMessage(conn->db));
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
kore_mem_free(conn->job->query);
Introduce a wait queue for our pgsql code. Instead of letting http_requests spin, if we cannot allocate a connection for the request we will queue them up put them to sleep. When a connection becomes available, we'll wake up a request that was waiting for a connection and let it continue. This completely avoids consuming massive amounts of cpu time when dealing with thousands of requests waiting for a pgsql worker to become ready.
2014-08-21 16:36:12 +02:00
kore_pool_put(&pgsql_job_pool, conn->job);
Initial stab at entering postgresql contrib code. Has support for full async pgsql queries. Most of the logic is hidden behind a KORE_PGSQL() macro allowing you to insert these pgsql calls in your page handlers without blocking the kore worker while the query is going off. There is place for improvement here, and perhaps KORE_PGSQL won't stay as I feel this might overcomplicate things instead of making them simpler as I thought it would.
2014-03-30 23:54:35 +02:00
conn->job = NULL;
}
if (conn->db != NULL)
PQfinish(conn->db);
pgsql_conn_count--;
kore_mem_free(conn);
}
Layout that stuff like a 80 column terminal nerd. And mark a request as DONE when we get PGRES_COMMAND_OK.
2014-03-31 01:04:22 +02:00
static void
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
pgsql_read_result(struct kore_pgsql *pgsql, int async)
Layout that stuff like a 80 column terminal nerd. And mark a request as DONE when we get PGRES_COMMAND_OK.
2014-03-31 01:04:22 +02:00
{
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
if (async) {
if (PQisBusy(pgsql->conn->db)) {
pgsql->state = KORE_PGSQL_STATE_WAIT;
return;
}
Layout that stuff like a 80 column terminal nerd. And mark a request as DONE when we get PGRES_COMMAND_OK.
2014-03-31 01:04:22 +02:00
}
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
pgsql->result = PQgetResult(pgsql->conn->db);
if (pgsql->result == NULL) {
pgsql->state = KORE_PGSQL_STATE_DONE;
Layout that stuff like a 80 column terminal nerd. And mark a request as DONE when we get PGRES_COMMAND_OK.
2014-03-31 01:04:22 +02:00
return;
}
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
switch (PQresultStatus(pgsql->result)) {
Layout that stuff like a 80 column terminal nerd. And mark a request as DONE when we get PGRES_COMMAND_OK.
2014-03-31 01:04:22 +02:00
case PGRES_COPY_OUT:
case PGRES_COPY_IN:
case PGRES_NONFATAL_ERROR:
case PGRES_COPY_BOTH:
break;
case PGRES_COMMAND_OK:
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
pgsql->state = KORE_PGSQL_STATE_DONE;
Layout that stuff like a 80 column terminal nerd. And mark a request as DONE when we get PGRES_COMMAND_OK.
2014-03-31 01:04:22 +02:00
break;
case PGRES_TUPLES_OK:
Check for pg version >= 90200 before using PGRES_SINGLE_TUPLE
2014-07-03 22:58:04 +02:00
#if PG_VERSION_NUM >= 90200
Layout that stuff like a 80 column terminal nerd. And mark a request as DONE when we get PGRES_COMMAND_OK.
2014-03-31 01:04:22 +02:00
case PGRES_SINGLE_TUPLE:
Check for pg version >= 90200 before using PGRES_SINGLE_TUPLE
2014-07-03 22:58:04 +02:00
#endif
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
pgsql->state = KORE_PGSQL_STATE_RESULT;
Layout that stuff like a 80 column terminal nerd. And mark a request as DONE when we get PGRES_COMMAND_OK.
2014-03-31 01:04:22 +02:00
break;
case PGRES_EMPTY_QUERY:
case PGRES_BAD_RESPONSE:
case PGRES_FATAL_ERROR:
Major pgsql rework. Gone is the ugly KORE_PGSQL macro that hid an overly complex state machine for the pgsql api. Gone is the pgsql array that was attached to http_requests. Gone are the callback hacks inside the pgsql api. Instead, I strongly encourage people to use the new state machine api Kore offers to properly deal with asynchronous queries. The pgsql example in examples/pgsql has been updated to reflect these changes.
2014-08-14 14:34:23 +02:00
pgsql->state = KORE_PGSQL_STATE_ERROR;
pgsql->error = kore_strdup(PQresultErrorMessage(pgsql->result));
Layout that stuff like a 80 column terminal nerd. And mark a request as DONE when we get PGRES_COMMAND_OK.
2014-03-31 01:04:22 +02:00
break;
}
}