Merge branch 'develop' into 'docs/improve-mailer-settings-desc'

# Conflicts:
#   CHANGELOG.md
This commit is contained in:
feld 2021-02-08 16:43:04 +00:00
commit 85cb122986
11 changed files with 149 additions and 8 deletions

View File

@ -18,7 +18,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
- Deprecated `Pleroma.Uploaders.S3, :public_endpoint`. Now `Pleroma.Upload, :base_url` is the standard configuration key for all uploaders. - Deprecated `Pleroma.Uploaders.S3, :public_endpoint`. Now `Pleroma.Upload, :base_url` is the standard configuration key for all uploaders.
- Improved Apache webserver support: updated sample configuration, MediaProxy cache invalidation verified with the included sample script - Improved Apache webserver support: updated sample configuration, MediaProxy cache invalidation verified with the included sample script
- Improve OAuth 2.0 provider support. A missing `fqn` field was added to the response, but does not expose the user's email address. - Improve OAuth 2.0 provider support. A missing `fqn` field was added to the response, but does not expose the user's email address.
- Provide redirect of external posts from `/notice/:id` to their original URL. - Provide redirect of external posts from `/notice/:id` to their original URL
- Admins no longer receive notifications for reports if they are the actor making the report.
- Improved Mailer configuration setting descriptions for AdminFE. - Improved Mailer configuration setting descriptions for AdminFE.
<details> <details>

View File

@ -141,3 +141,21 @@ but should only be run if necessary. **It is safe to cancel this.**
```sh ```sh
mix pleroma.database ensure_expiration mix pleroma.database ensure_expiration
``` ```
## Change Text Search Configuration
Change `default_text_search_config` for database and (if necessary) text_search_config used in index, then rebuild index (it may take time).
=== "OTP"
```sh
./bin/pleroma_ctl database set_text_search_config english
```
=== "From Source"
```sh
mix pleroma.database set_text_search_config english
```
See [PostgreSQL documentation](https://www.postgresql.org/docs/current/textsearch-configuration.html) and `docs/configuration/howto_search_cjk.md` for more detail.

View File

@ -0,0 +1,42 @@
# How to enable text search for Chinese, Japanese and Korean
Pleroma's full text search feature is powered by PostgreSQL's native [text search](https://www.postgresql.org/docs/current/textsearch.html), it works well out of box for most of languages, but needs extra configurations for some asian languages like Chinese, Japanese and Korean (CJK).
## Setup and test the new search config
In most cases, you would need an extension installed to support parsing CJK text. Here are a few extension you may choose from, or you are more than welcome to share additional ones you found working for you with the rest of Pleroma community.
* [a generic n-gram parser](https://github.com/huangjimmy/pg_cjk_parser) supports Simplifed/Traditional Chinese, Japanese, and Korean
* [a Korean parser](https://github.com/i0seph/textsearch_ko) based on mecab
* [a Japanese parser](https://www.amris.co.jp/tsja/index.html) based on mecab
* [zhparser](https://github.com/amutu/zhparser/) is a PostgreSQL extension base on the Simple Chinese Word Segmentation(SCWS)
* [another Chinese parser](https://github.com/jaiminpan/pg_jieba) based on Jieba Chinese Word Segmentation
Once you have the new search config , make sure you test it with the `pleroma` user in PostgreSQL (change `YOUR.CONFIG` to your real configuration name)
```
SELECT ts_debug('YOUR.CONFIG', '安装和配置Nginx, ElixirとErlangをインストールします');
```
Check output of the query, and see if it matches your expectation.
## Update text search config and index in database
=== "OTP"
```sh
./bin/pleroma_ctl database set_text_search_config YOUR.CONFIG
```
=== "From Source"
```sh
mix pleroma.database set_text_search_config YOUR.CONFIG
```
Note: index update may take a while.
## Restart database connection
Since some changes above will only apply with a new database connection, you will have to restart either Pleroma or PostgreSQL process, or use `pg_terminate_backend` SQL command without restarting either.
Now the search results of statuses should be much more friendly for your language of choice, the results for searching users and tags were not changed, as the default parsing/matching should work for most cases.

View File

@ -59,6 +59,13 @@ sub vcl_backend_response {
set beresp.http.CR = beresp.http.content-range; set beresp.http.CR = beresp.http.content-range;
} }
# Bypass cache for large files
# 50000000 ~ 50MB
if (std.integer(beresp.http.content-length, 0) > 50000000) {
set beresp.uncacheable = true;
return(deliver);
}
# Don't cache objects that require authentication # Don't cache objects that require authentication
if (beresp.http.Authorization && !beresp.http.Cache-Control ~ "public") { if (beresp.http.Authorization && !beresp.http.Cache-Control ~ "public") {
set beresp.uncacheable = true; set beresp.uncacheable = true;

View File

@ -167,4 +167,51 @@ defmodule Mix.Tasks.Pleroma.Database do
end) end)
|> Stream.run() |> Stream.run()
end end
def run(["set_text_search_config", tsconfig]) do
start_pleroma()
%{rows: [[tsc]]} = Ecto.Adapters.SQL.query!(Pleroma.Repo, "SHOW default_text_search_config;")
shell_info("Current default_text_search_config: #{tsc}")
%{rows: [[db]]} = Ecto.Adapters.SQL.query!(Pleroma.Repo, "SELECT current_database();")
shell_info("Update default_text_search_config: #{tsconfig}")
%{messages: msg} =
Ecto.Adapters.SQL.query!(
Pleroma.Repo,
"ALTER DATABASE #{db} SET default_text_search_config = '#{tsconfig}';"
)
# non-exist config will not raise excpetion but only give >0 messages
if length(msg) > 0 do
shell_info("Error: #{inspect(msg, pretty: true)}")
else
rum_enabled = Pleroma.Config.get([:database, :rum_enabled])
shell_info("Recreate index, RUM: #{rum_enabled}")
# Note SQL below needs to be kept up-to-date with latest GIN or RUM index definition in future
if rum_enabled do
Ecto.Adapters.SQL.query!(
Pleroma.Repo,
"CREATE OR REPLACE FUNCTION objects_fts_update() RETURNS trigger AS $$ BEGIN
new.fts_content := to_tsvector(new.data->>'content');
RETURN new;
END
$$ LANGUAGE plpgsql"
)
shell_info("Refresh RUM index")
Ecto.Adapters.SQL.query!(Pleroma.Repo, "UPDATE objects SET updated_at = NOW();")
else
Ecto.Adapters.SQL.query!(Pleroma.Repo, "DROP INDEX IF EXISTS objects_fts;")
Ecto.Adapters.SQL.query!(
Pleroma.Repo,
"CREATE INDEX objects_fts ON objects USING gin(to_tsvector('#{tsconfig}', data->>'content')); "
)
end
shell_info('Done.')
end
end
end end

View File

@ -64,7 +64,7 @@ defmodule Pleroma.Activity.Search do
from([a, o] in q, from([a, o] in q,
where: where:
fragment( fragment(
"to_tsvector('english', ?->>'content') @@ plainto_tsquery('english', ?)", "to_tsvector(?->>'content') @@ plainto_tsquery(?)",
o.data, o.data,
^search_query ^search_query
) )
@ -75,7 +75,7 @@ defmodule Pleroma.Activity.Search do
from([a, o] in q, from([a, o] in q,
where: where:
fragment( fragment(
"to_tsvector('english', ?->>'content') @@ websearch_to_tsquery('english', ?)", "to_tsvector(?->>'content') @@ websearch_to_tsquery(?)",
o.data, o.data,
^search_query ^search_query
) )
@ -86,7 +86,7 @@ defmodule Pleroma.Activity.Search do
from([a, o] in q, from([a, o] in q,
where: where:
fragment( fragment(
"? @@ plainto_tsquery('english', ?)", "? @@ plainto_tsquery(?)",
o.fts_content, o.fts_content,
^search_query ^search_query
), ),
@ -98,7 +98,7 @@ defmodule Pleroma.Activity.Search do
from([a, o] in q, from([a, o] in q,
where: where:
fragment( fragment(
"? @@ websearch_to_tsquery('english', ?)", "? @@ websearch_to_tsquery(?)",
o.fts_content, o.fts_content,
^search_query ^search_query
), ),

View File

@ -507,8 +507,8 @@ defmodule Pleroma.Notification do
[object_id] [object_id]
end end
def get_potential_receiver_ap_ids(%{data: %{"type" => "Flag"}}) do def get_potential_receiver_ap_ids(%{data: %{"type" => "Flag", "actor" => actor}}) do
User.all_superusers() |> Enum.map(fn user -> user.ap_id end) (User.all_superusers() |> Enum.map(fn user -> user.ap_id end)) -- [actor]
end end
def get_potential_receiver_ap_ids(activity) do def get_potential_receiver_ap_ids(activity) do

View File

@ -377,6 +377,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
:ok <- :ok <-
maybe_federate(stripped_activity) do maybe_federate(stripped_activity) do
User.all_superusers() User.all_superusers()
|> Enum.filter(fn user -> user.ap_id != actor end)
|> Enum.filter(fn user -> not is_nil(user.email) end) |> Enum.filter(fn user -> not is_nil(user.email) end)
|> Enum.each(fn superuser -> |> Enum.each(fn superuser ->
superuser superuser

View File

@ -0,0 +1,11 @@
defmodule Pleroma.Repo.Migrations.AddDefaultTextSearchConfig do
use Ecto.Migration
def change do
execute("DO $$
BEGIN
execute 'ALTER DATABASE '||current_database()||' SET default_text_search_config = ''english'' ';
END
$$;")
end
end

View File

@ -17,7 +17,7 @@ defmodule Pleroma.Repo.Migrations.AddFtsIndexToObjectsTwo do
execute("CREATE FUNCTION objects_fts_update() RETURNS trigger AS $$ execute("CREATE FUNCTION objects_fts_update() RETURNS trigger AS $$
begin begin
new.fts_content := to_tsvector('english', new.data->>'content'); new.fts_content := to_tsvector(new.data->>'content');
return new; return new;
end end
$$ LANGUAGE plpgsql") $$ LANGUAGE plpgsql")

View File

@ -45,6 +45,20 @@ defmodule Pleroma.NotificationTest do
assert notification.type == "pleroma:report" assert notification.type == "pleroma:report"
end end
test "suppresses notification to reporter if reporter is an admin" do
reporting_admin = insert(:user, is_admin: true)
reported_user = insert(:user)
other_admin = insert(:user, is_admin: true)
{:ok, activity} = CommonAPI.report(reporting_admin, %{account_id: reported_user.id})
{:ok, [notification]} = Notification.create_notifications(activity)
refute notification.user_id == reporting_admin.id
assert notification.user_id == other_admin.id
assert notification.type == "pleroma:report"
end
test "creates a notification for an emoji reaction" do test "creates a notification for an emoji reaction" do
user = insert(:user) user = insert(:user)
other_user = insert(:user) other_user = insert(:user)