Merge branch 'develop' of git.pleroma.social:pleroma/pleroma into alexgleason/pleroma-chat-moderation

2020-09-17 13:26:04 +02:00 · 2020-09-17 13:26:04 +02:00 · 7a73850928
parent c4061f06ba c5acbf8a1b
commit 7a73850928
49 changed files with 767 additions and 442 deletions
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@ -9,15 +9,22 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).

 - Renamed `:await_up_timeout` in `:connections_pool` namespace to `:connect_timeout`, old name is deprecated.
 - Renamed `:timeout` in `pools` namespace to `:recv_timeout`, old name is deprecated.
+- Minimum lifetime for ephmeral activities changed to 10 minutes and made configurable (`:min_lifetime` option).

 ### Removed

 - **Breaking:** `Pleroma.Workers.Cron.StatsWorker` setting from Oban `:crontab` (moved to a simpler implementation).
 - **Breaking:** `Pleroma.Workers.Cron.ClearOauthTokenWorker` setting from Oban `:crontab` (moved to scheduled jobs).
 - **Breaking:** `Pleroma.Workers.Cron.PurgeExpiredActivitiesWorker` setting from Oban `:crontab` (moved to scheduled jobs).
+- Removed `:managed_config` option. In practice, it was accidentally removed with 2.0.0 release when frontends were
+switched to a new configuration mechanism, however it was not officially removed until now.

-### Changed
- Minimum lifetime for ephmeral activities changed to 10 minutes and made configurable (`:min_lifetime` option).
+## unreleased-patch - ???
+
+### Fixed
+
+- Welcome Chat messages preventing user registration with MRF Simple Policy applied to the local instance
+- Mastodon API: the public timeline returning an error when the `reply_visibility` parameter is set to `self` for an unauthenticated user

 ## [2.1.1] - 2020-09-08

--- a/config/config.exs
+++ b/config/config.exs
@ -216,7 +216,6 @@ config :pleroma, :instance,
  allow_relay: true,
  public: true,
  quarantined_instances: [],
-  managed_config: true,
  static_dir: "instance/static/",
  allowed_post_formats: [
    "text/plain",
--- a/config/description.exs
+++ b/config/description.exs
@ -764,12 +764,6 @@ config :pleroma, :config_description, [
          "*.quarantined.com"
        ]
      },
-      %{
-        key: :managed_config,
-        type: :boolean,
-        description:
-          "Whenether the config for pleroma-fe is configured in this config or in static/config.json"
-      },
      %{
        key: :static_dir,
        type: :string,
--- a/docs/configuration/cheatsheet.md
+++ b/docs/configuration/cheatsheet.md
@ -18,7 +18,7 @@ To add configuration to your config file, you can copy it from the base config.
 * `notify_email`: Email used for notifications.
 * `description`: The instance’s description, can be seen in nodeinfo and ``/api/v1/instance``.
 * `limit`: Posts character limit (CW/Subject included in the counter).
-* `discription_limit`: The character limit for image descriptions.
+* `description_limit`: The character limit for image descriptions.
 * `chat_limit`: Character limit of the instance chat messages.
 * `remote_limit`: Hard character limit beyond which remote posts will be dropped.
 * `upload_limit`: File size limit of uploads (except for avatar, background, banner).
@ -40,7 +40,6 @@ To add configuration to your config file, you can copy it from the base config.
 * `allow_relay`: Enable Pleroma’s Relay, which makes it possible to follow a whole instance.
 * `public`: Makes the client API in authenticated mode-only except for user-profiles. Useful for disabling the Local Timeline and The Whole Known Network. Note that there is a dependent setting restricting or allowing unauthenticated access to specific resources, see `restrict_unauthenticated` for more details.
 * `quarantined_instances`: List of ActivityPub instances where private (DMs, followers-only) activities will not be send.
-* `managed_config`: Whenether the config for pleroma-fe is configured in [:frontend_configurations](#frontend_configurations) or in ``static/config.json``.
 * `allowed_post_formats`: MIME-type list of formats allowed to be posted (transformed into HTML).
 * `extended_nickname_format`: Set to `true` to use extended local nicknames format (allows underscores/dashes). This will break federation with
    older software for theses nicknames.
--- a/lib/mix/tasks/pleroma/database.ex
+++ b/lib/mix/tasks/pleroma/database.ex
@ -99,7 +99,7 @@ defmodule Mix.Tasks.Pleroma.Database do
      where: fragment("(?)->>'likes' is not null", object.data),
      select: %{id: object.id, likes: fragment("(?)->>'likes'", object.data)}
    )
-    |> Pleroma.RepoStreamer.chunk_stream(100)
+    |> Pleroma.Repo.chunk_stream(100, :batches)
    |> Stream.each(fn objects ->
      ids =
        objects
@ -145,7 +145,7 @@ defmodule Mix.Tasks.Pleroma.Database do
    |> where(local: true)
    |> where([a], fragment("(? ->> 'type'::text) = 'Create'", a.data))
    |> where([_a, o], fragment("?->>'type' = 'Note'", o.data))
-    |> Pleroma.RepoStreamer.chunk_stream(100)
+    |> Pleroma.Repo.chunk_stream(100, :batches)
    |> Stream.each(fn activities ->
      Enum.each(activities, fn activity ->
        expires_at =
--- a/lib/mix/tasks/pleroma/user.ex
+++ b/lib/mix/tasks/pleroma/user.ex
@ -179,7 +179,7 @@ defmodule Mix.Tasks.Pleroma.User do
    start_pleroma()

    Pleroma.User.Query.build(%{nickname: "@#{instance}"})
-    |> Pleroma.RepoStreamer.chunk_stream(500)
+    |> Pleroma.Repo.chunk_stream(500, :batches)
    |> Stream.each(fn users ->
      users
      |> Enum.each(fn user ->
@ -370,7 +370,7 @@ defmodule Mix.Tasks.Pleroma.User do
    start_pleroma()

    Pleroma.User.Query.build(%{local: true})
-    |> Pleroma.RepoStreamer.chunk_stream(500)
+    |> Pleroma.Repo.chunk_stream(500, :batches)
    |> Stream.each(fn users ->
      users
      |> Enum.each(fn user ->
--- a/lib/pleroma/gun/connection_pool/worker.ex
+++ b/lib/pleroma/gun/connection_pool/worker.ex
@ -93,25 +93,18 @@ defmodule Pleroma.Gun.ConnectionPool.Worker do
      end)

    {ref, state} = pop_in(state.client_monitors[client_pid])
-    # DOWN message can receive right after `remove_client` call and cause worker to terminate
-    state =
-      if is_nil(ref) do
-        state
+
+    Process.demonitor(ref, [:flush])
+
+    timer =
+      if used_by == [] do
+        max_idle = Pleroma.Config.get([:connections_pool, :max_idle_time], 30_000)
+        Process.send_after(self(), :idle_close, max_idle)
      else
-        Process.demonitor(ref)
-
-        timer =
-          if used_by == [] do
-            max_idle = Pleroma.Config.get([:connections_pool, :max_idle_time], 30_000)
-            Process.send_after(self(), :idle_close, max_idle)
-          else
-            nil
-          end
-
-        %{state | timer: timer}
+        nil
      end

-    {:reply, :ok, state, :hibernate}
+    {:reply, :ok, %{state | timer: timer}, :hibernate}
  end

  @impl true
--- a/lib/pleroma/migration_helper/notification_backfill.ex
+++ b/lib/pleroma/migration_helper/notification_backfill.ex
@ -19,13 +19,13 @@ defmodule Pleroma.MigrationHelper.NotificationBackfill do
    query
    |> Repo.chunk_stream(100)
    |> Enum.each(fn notification ->
-      type =
-        notification.activity
-        |> type_from_activity()
+      if notification.activity do
+        type = type_from_activity(notification.activity)

-      notification
-      |> Ecto.Changeset.change(%{type: type})
-      |> Repo.update()
+        notification
+        |> Ecto.Changeset.change(%{type: type})
+        |> Repo.update()
+      end
    end)
  end

@ -72,8 +72,7 @@ defmodule Pleroma.MigrationHelper.NotificationBackfill do
        "pleroma:emoji_reaction"

      "Create" ->
-        activity
-        |> type_from_activity_object()
+        type_from_activity_object(activity)

      t ->
        raise "No notification type for activity type #{t}"
--- a/lib/pleroma/object/fetcher.ex
+++ b/lib/pleroma/object/fetcher.ex
@ -98,8 +98,8 @@ defmodule Pleroma.Object.Fetcher do
      {:containment, _} ->
        {:error, "Object containment failed."}

-      {:transmogrifier, {:error, {:reject, nil}}} ->
-        {:reject, nil}
+      {:transmogrifier, {:error, {:reject, e}}} ->
+        {:reject, e}

      {:transmogrifier, _} = e ->
        {:error, e}
--- a/lib/pleroma/repo.ex
+++ b/lib/pleroma/repo.ex
@ -49,7 +49,21 @@ defmodule Pleroma.Repo do
    end
  end

-  def chunk_stream(query, chunk_size) do
+  @doc """
+  Returns a lazy enumerable that emits all entries from the data store matching the given query.
+
+  `returns_as` use to group records. use the `batches` option to fetch records in bulk.
+
+  ## Examples
+
+  # fetch records one-by-one
+  iex> Pleroma.Repo.chunk_stream(Pleroma.Activity.Queries.by_actor(ap_id), 500)
+
+  # fetch records in bulk
+  iex> Pleroma.Repo.chunk_stream(Pleroma.Activity.Queries.by_actor(ap_id), 500, :batches)
+  """
+  @spec chunk_stream(Ecto.Query.t(), integer(), atom()) :: Enumerable.t()
+  def chunk_stream(query, chunk_size, returns_as \\ :one) do
    # We don't actually need start and end funcitons of resource streaming,
    # but it seems to be the only way to not fetch records one-by-one and
    # have individual records be the elements of the stream, instead of
@ -69,7 +83,12 @@ defmodule Pleroma.Repo do

            records ->
              last_id = List.last(records).id
-              {records, last_id}
+
+              if returns_as == :one do
+                {records, last_id}
+              else
+                {[records], last_id}
+              end
          end
      end,
      fn _ -> :ok end
--- a/lib/pleroma/repo_streamer.ex
+++ b/lib/pleroma/repo_streamer.ex
@ -1,34 +0,0 @@
-# Pleroma: A lightweight social networking server
-# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
-# SPDX-License-Identifier: AGPL-3.0-only
-
-defmodule Pleroma.RepoStreamer do
-  alias Pleroma.Repo
-  import Ecto.Query
-
-  def chunk_stream(query, chunk_size) do
-    Stream.unfold(0, fn
-      :halt ->
-        {[], :halt}
-
-      last_id ->
-        query
-        |> order_by(asc: :id)
-        |> where([r], r.id > ^last_id)
-        |> limit(^chunk_size)
-        |> Repo.all()
-        |> case do
-          [] ->
-            {[], :halt}
-
-          records ->
-            last_id = List.last(records).id
-            {records, last_id}
-        end
-    end)
-    |> Stream.take_while(fn
-      [] -> false
-      _ -> true
-    end)
-  end
-end
--- a/lib/pleroma/user.ex
+++ b/lib/pleroma/user.ex
@ -25,7 +25,6 @@ defmodule Pleroma.User do
  alias Pleroma.Object
  alias Pleroma.Registration
  alias Pleroma.Repo
-  alias Pleroma.RepoStreamer
  alias Pleroma.User
  alias Pleroma.UserRelationship
  alias Pleroma.Web
@ -276,9 +275,9 @@ defmodule Pleroma.User do
  @spec account_status(User.t()) :: account_status()
  def account_status(%User{deactivated: true}), do: :deactivated
  def account_status(%User{password_reset_pending: true}), do: :password_reset_pending
-  def account_status(%User{approval_pending: true}), do: :approval_pending
+  def account_status(%User{local: true, approval_pending: true}), do: :approval_pending

-  def account_status(%User{confirmation_pending: true}) do
+  def account_status(%User{local: true, confirmation_pending: true}) do
    if Config.get([:instance, :account_activation_required]) do
      :confirmation_pending
    else
@ -1775,7 +1774,7 @@ defmodule Pleroma.User do
  def delete_user_activities(%User{ap_id: ap_id} = user) do
    ap_id
    |> Activity.Queries.by_actor()
-    |> RepoStreamer.chunk_stream(50)
+    |> Repo.chunk_stream(50, :batches)
    |> Stream.each(fn activities ->
      Enum.each(activities, fn activity -> delete_activity(activity, user) end)
    end)
--- a/lib/pleroma/web/activity_pub/activity_pub.ex
+++ b/lib/pleroma/web/activity_pub/activity_pub.ex
@ -84,7 +84,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do

  defp increase_replies_count_if_reply(_create_data), do: :noop

-  @object_types ~w[ChatMessage Question Answer Audio Event]
+  @object_types ~w[ChatMessage Question Answer Audio Video Event Article]
  @spec persist(map(), keyword()) :: {:ok, Activity.t() | Object.t()}
  def persist(%{"type" => type} = object, meta) when type in @object_types do
    with {:ok, object} <- Object.create(object) do
@ -154,8 +154,8 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
      {:remote_limit_pass, _} ->
        {:error, :remote_limit}

-      {:reject, reason} ->
-        {:error, reason}
+      {:reject, _} = e ->
+        {:error, e}
    end
  end

@ -767,7 +767,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
  end

  defp restrict_replies(query, %{
-         reply_filtering_user: user,
+         reply_filtering_user: %User{} = user,
         reply_visibility: "self"
       }) do
    from(
@ -783,7 +783,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
  end

  defp restrict_replies(query, %{
-         reply_filtering_user: user,
+         reply_filtering_user: %User{} = user,
         reply_visibility: "following"
       }) do
    from(
--- a/lib/pleroma/web/activity_pub/mrf/simple_policy.ex
+++ b/lib/pleroma/web/activity_pub/mrf/simple_policy.ex
@ -66,7 +66,8 @@ defmodule Pleroma.Web.ActivityPub.MRF.SimplePolicy do
           "type" => "Create",
           "object" => child_object
         } = object
-       ) do
+       )
+       when is_map(child_object) do
    media_nsfw =
      Config.get([:mrf_simple, :media_nsfw])
      |> MRF.subdomains_regex()
--- a/lib/pleroma/web/activity_pub/object_validator.ex
+++ b/lib/pleroma/web/activity_pub/object_validator.ex
@ -12,11 +12,13 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidator do
  alias Pleroma.Activity
  alias Pleroma.EctoType.ActivityPub.ObjectValidators
  alias Pleroma.Object
+  alias Pleroma.Object.Containment
  alias Pleroma.User
  alias Pleroma.Web.ActivityPub.ObjectValidators.AcceptRejectValidator
  alias Pleroma.Web.ActivityPub.ObjectValidators.AnnounceValidator
  alias Pleroma.Web.ActivityPub.ObjectValidators.AnswerValidator
-  alias Pleroma.Web.ActivityPub.ObjectValidators.AudioValidator
+  alias Pleroma.Web.ActivityPub.ObjectValidators.ArticleNoteValidator
+  alias Pleroma.Web.ActivityPub.ObjectValidators.AudioVideoValidator
  alias Pleroma.Web.ActivityPub.ObjectValidators.BlockValidator
  alias Pleroma.Web.ActivityPub.ObjectValidators.ChatMessageValidator
  alias Pleroma.Web.ActivityPub.ObjectValidators.CreateChatMessageValidator
@ -149,10 +151,20 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidator do
    end
  end

-  def validate(%{"type" => "Audio"} = object, meta) do
+  def validate(%{"type" => type} = object, meta) when type in ~w[Audio Video] do
    with {:ok, object} <-
           object
-           |> AudioValidator.cast_and_validate()
+           |> AudioVideoValidator.cast_and_validate()
+           |> Ecto.Changeset.apply_action(:insert) do
+      object = stringify_keys(object)
+      {:ok, object, meta}
+    end
+  end
+
+  def validate(%{"type" => "Article"} = object, meta) do
+    with {:ok, object} <-
+           object
+           |> ArticleNoteValidator.cast_and_validate()
           |> Ecto.Changeset.apply_action(:insert) do
      object = stringify_keys(object)
      {:ok, object, meta}
@ -198,7 +210,7 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidator do
        %{"type" => "Create", "object" => %{"type" => objtype} = object} = create_activity,
        meta
      )
-      when objtype in ~w[Question Answer Audio Event] do
+      when objtype in ~w[Question Answer Audio Video Event Article] do
    with {:ok, object_data} <- cast_and_apply(object),
         meta = Keyword.put(meta, :object_data, object_data |> stringify_keys),
         {:ok, create_activity} <-
@ -232,14 +244,18 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidator do
    AnswerValidator.cast_and_apply(object)
  end

-  def cast_and_apply(%{"type" => "Audio"} = object) do
-    AudioValidator.cast_and_apply(object)
+  def cast_and_apply(%{"type" => type} = object) when type in ~w[Audio Video] do
+    AudioVideoValidator.cast_and_apply(object)
  end

  def cast_and_apply(%{"type" => "Event"} = object) do
    EventValidator.cast_and_apply(object)
  end

+  def cast_and_apply(%{"type" => "Article"} = object) do
+    ArticleNoteValidator.cast_and_apply(object)
+  end
+
  def cast_and_apply(o), do: {:error, {:validator_not_set, o}}

  # is_struct/1 isn't present in Elixir 1.8.x
@ -262,7 +278,8 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidator do
  def stringify_keys(object), do: object

  def fetch_actor(object) do
-    with {:ok, actor} <- ObjectValidators.ObjectID.cast(object["actor"]) do
+    with actor <- Containment.get_actor(object),
+         {:ok, actor} <- ObjectValidators.ObjectID.cast(actor) do
      User.get_or_fetch_by_ap_id(actor)
    end
  end
--- a/lib/pleroma/web/activity_pub/object_validators/article_note_validator.ex
+++ b/lib/pleroma/web/activity_pub/object_validators/article_note_validator.ex
@ -2,7 +2,7 @@
 # Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only

-defmodule Pleroma.Web.ActivityPub.ObjectValidators.AudioValidator do
+defmodule Pleroma.Web.ActivityPub.ObjectValidators.ArticleNoteValidator do
  use Ecto.Schema

  alias Pleroma.EctoType.ActivityPub.ObjectValidators
@ -25,14 +25,19 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.AudioValidator do
    # TODO: Write type
    field(:tag, {:array, :map}, default: [])
    field(:type, :string)
+
+    field(:name, :string)
+    field(:summary, :string)
    field(:content, :string)
+
    field(:context, :string)
+    # short identifier for PleromaFE to group statuses by context
+    field(:context_id, :integer)

    # TODO: Remove actor on objects
    field(:actor, ObjectValidators.ObjectID)

    field(:attributedTo, ObjectValidators.ObjectID)
-    field(:summary, :string)
    field(:published, ObjectValidators.DateTime)
    field(:emoji, ObjectValidators.Emoji, default: %{})
    field(:sensitive, :boolean, default: false)
@ -40,13 +45,11 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.AudioValidator do
    field(:replies_count, :integer, default: 0)
    field(:like_count, :integer, default: 0)
    field(:announcement_count, :integer, default: 0)
-    field(:inReplyTo, :string)
+    field(:inReplyTo, ObjectValidators.ObjectID)
    field(:url, ObjectValidators.Uri)
-    # short identifier for PleromaFE to group statuses by context
-    field(:context_id, :integer)

-    field(:likes, {:array, :string}, default: [])
-    field(:announcements, {:array, :string}, default: [])
+    field(:likes, {:array, ObjectValidators.ObjectID}, default: [])
+    field(:announcements, {:array, ObjectValidators.ObjectID}, default: [])
  end

  def cast_and_apply(data) do
@ -62,19 +65,14 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.AudioValidator do
  end

  def cast_data(data) do
+    data = fix(data)
+
    %__MODULE__{}
    |> changeset(data)
  end

-  defp fix_url(%{"url" => url} = data) when is_list(url) do
-    attachment =
-      Enum.find(url, fn x -> is_map(x) and String.starts_with?(x["mimeType"], "audio/") end)
-
-    link_element = Enum.find(url, fn x -> is_map(x) and x["mimeType"] == "text/html" end)
-
-    data
-    |> Map.put("attachment", [attachment])
-    |> Map.put("url", link_element["href"])
+  defp fix_url(%{"url" => url} = data) when is_map(url) do
+    Map.put(data, "url", url["href"])
  end

  defp fix_url(data), do: data
@ -83,8 +81,9 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.AudioValidator do
    data
    |> CommonFixes.fix_defaults()
    |> CommonFixes.fix_attribution()
-    |> Transmogrifier.fix_emoji()
+    |> CommonFixes.fix_actor()
    |> fix_url()
+    |> Transmogrifier.fix_emoji()
  end

  def changeset(struct, data) do
@ -97,8 +96,8 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.AudioValidator do

  def validate_data(data_cng) do
    data_cng
-    |> validate_inclusion(:type, ["Audio"])
-    |> validate_required([:id, :actor, :attributedTo, :type, :context, :attachment])
+    |> validate_inclusion(:type, ["Article", "Note"])
+    |> validate_required([:id, :actor, :attributedTo, :type, :context, :context_id])
    |> CommonValidations.validate_any_presence([:cc, :to])
    |> CommonValidations.validate_fields_match([:actor, :attributedTo])
    |> CommonValidations.validate_actor_presence()
--- a/lib/pleroma/web/activity_pub/object_validators/attachment_validator.ex
+++ b/lib/pleroma/web/activity_pub/object_validators/attachment_validator.ex
@ -5,6 +5,7 @@
 defmodule Pleroma.Web.ActivityPub.ObjectValidators.AttachmentValidator do
  use Ecto.Schema

+  alias Pleroma.EctoType.ActivityPub.ObjectValidators
  alias Pleroma.Web.ActivityPub.ObjectValidators.UrlObjectValidator

  import Ecto.Changeset
@ -15,7 +16,11 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.AttachmentValidator do
    field(:mediaType, :string, default: "application/octet-stream")
    field(:name, :string)

-    embeds_many(:url, UrlObjectValidator)
+    embeds_many :url, UrlObjectValidator, primary_key: false do
+      field(:type, :string)
+      field(:href, ObjectValidators.Uri)
+      field(:mediaType, :string, default: "application/octet-stream")
+    end
  end

  def cast_and_validate(data) do
@ -37,7 +42,18 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.AttachmentValidator do

    struct
    |> cast(data, [:type, :mediaType, :name])
-    |> cast_embed(:url, required: true)
+    |> cast_embed(:url, with: &url_changeset/2)
+    |> validate_inclusion(:type, ~w[Link Document Audio Image Video])
+    |> validate_required([:type, :mediaType, :url])
+  end
+
+  def url_changeset(struct, data) do
+    data = fix_media_type(data)
+
+    struct
+    |> cast(data, [:type, :href, :mediaType])
+    |> validate_inclusion(:type, ["Link"])
+    |> validate_required([:type, :href, :mediaType])
  end

  def fix_media_type(data) do
@ -75,6 +91,7 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.AttachmentValidator do

  def validate_data(cng) do
    cng
+    |> validate_inclusion(:type, ~w[Document Audio Image Video])
    |> validate_required([:mediaType, :url, :type])
  end
 end
--- a/lib/pleroma/web/activity_pub/object_validators/audio_video_validator.ex
+++ b/lib/pleroma/web/activity_pub/object_validators/audio_video_validator.ex
@ -0,0 +1,134 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.ActivityPub.ObjectValidators.AudioVideoValidator do
+  use Ecto.Schema
+
+  alias Pleroma.EarmarkRenderer
+  alias Pleroma.EctoType.ActivityPub.ObjectValidators
+  alias Pleroma.Web.ActivityPub.ObjectValidators.AttachmentValidator
+  alias Pleroma.Web.ActivityPub.ObjectValidators.CommonFixes
+  alias Pleroma.Web.ActivityPub.ObjectValidators.CommonValidations
+  alias Pleroma.Web.ActivityPub.Transmogrifier
+
+  import Ecto.Changeset
+
+  @primary_key false
+  @derive Jason.Encoder
+
+  embedded_schema do
+    field(:id, ObjectValidators.ObjectID, primary_key: true)
+    field(:to, ObjectValidators.Recipients, default: [])
+    field(:cc, ObjectValidators.Recipients, default: [])
+    field(:bto, ObjectValidators.Recipients, default: [])
+    field(:bcc, ObjectValidators.Recipients, default: [])
+    # TODO: Write type
+    field(:tag, {:array, :map}, default: [])
+    field(:type, :string)
+
+    field(:name, :string)
+    field(:summary, :string)
+    field(:content, :string)
+
+    field(:context, :string)
+    # short identifier for PleromaFE to group statuses by context
+    field(:context_id, :integer)
+
+    # TODO: Remove actor on objects
+    field(:actor, ObjectValidators.ObjectID)
+
+    field(:attributedTo, ObjectValidators.ObjectID)
+    field(:published, ObjectValidators.DateTime)
+    field(:emoji, ObjectValidators.Emoji, default: %{})
+    field(:sensitive, :boolean, default: false)
+    embeds_many(:attachment, AttachmentValidator)
+    field(:replies_count, :integer, default: 0)
+    field(:like_count, :integer, default: 0)
+    field(:announcement_count, :integer, default: 0)
+    field(:inReplyTo, ObjectValidators.ObjectID)
+    field(:url, ObjectValidators.Uri)
+
+    field(:likes, {:array, ObjectValidators.ObjectID}, default: [])
+    field(:announcements, {:array, ObjectValidators.ObjectID}, default: [])
+  end
+
+  def cast_and_apply(data) do
+    data
+    |> cast_data
+    |> apply_action(:insert)
+  end
+
+  def cast_and_validate(data) do
+    data
+    |> cast_data()
+    |> validate_data()
+  end
+
+  def cast_data(data) do
+    %__MODULE__{}
+    |> changeset(data)
+  end
+
+  defp fix_url(%{"url" => url} = data) when is_list(url) do
+    attachment =
+      Enum.find(url, fn x ->
+        mime_type = x["mimeType"] || x["mediaType"] || ""
+
+        is_map(x) and String.starts_with?(mime_type, ["video/", "audio/"])
+      end)
+
+    link_element =
+      Enum.find(url, fn x ->
+        mime_type = x["mimeType"] || x["mediaType"] || ""
+
+        is_map(x) and mime_type == "text/html"
+      end)
+
+    data
+    |> Map.put("attachment", [attachment])
+    |> Map.put("url", link_element["href"])
+  end
+
+  defp fix_url(data), do: data
+
+  defp fix_content(%{"mediaType" => "text/markdown", "content" => content} = data)
+       when is_binary(content) do
+    content =
+      content
+      |> Earmark.as_html!(%Earmark.Options{renderer: EarmarkRenderer})
+      |> Pleroma.HTML.filter_tags()
+
+    Map.put(data, "content", content)
+  end
+
+  defp fix_content(data), do: data
+
+  defp fix(data) do
+    data
+    |> CommonFixes.fix_defaults()
+    |> CommonFixes.fix_attribution()
+    |> CommonFixes.fix_actor()
+    |> Transmogrifier.fix_emoji()
+    |> fix_url()
+    |> fix_content()
+  end
+
+  def changeset(struct, data) do
+    data = fix(data)
+
+    struct
+    |> cast(data, __schema__(:fields) -- [:attachment])
+    |> cast_embed(:attachment)
+  end
+
+  def validate_data(data_cng) do
+    data_cng
+    |> validate_inclusion(:type, ["Audio", "Video"])
+    |> validate_required([:id, :actor, :attributedTo, :type, :context, :attachment])
+    |> CommonValidations.validate_any_presence([:cc, :to])
+    |> CommonValidations.validate_fields_match([:actor, :attributedTo])
+    |> CommonValidations.validate_actor_presence()
+    |> CommonValidations.validate_host_match()
+  end
+end
--- a/lib/pleroma/web/activity_pub/object_validators/common_fixes.ex
+++ b/lib/pleroma/web/activity_pub/object_validators/common_fixes.ex
@ -3,6 +3,7 @@
 # SPDX-License-Identifier: AGPL-3.0-only

 defmodule Pleroma.Web.ActivityPub.ObjectValidators.CommonFixes do
+  alias Pleroma.Object.Containment
  alias Pleroma.Web.ActivityPub.Utils

  # based on Pleroma.Web.ActivityPub.Utils.lazy_put_objects_defaults
@ -19,4 +20,12 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.CommonFixes do
    data
    |> Map.put_new("actor", data["attributedTo"])
  end
+
+  def fix_actor(data) do
+    actor = Containment.get_actor(data)
+
+    data
+    |> Map.put("actor", actor)
+    |> Map.put("attributedTo", actor)
+  end
 end
--- a/lib/pleroma/web/activity_pub/object_validators/create_generic_validator.ex
+++ b/lib/pleroma/web/activity_pub/object_validators/create_generic_validator.ex
@ -10,9 +10,10 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.CreateGenericValidator do

  alias Pleroma.EctoType.ActivityPub.ObjectValidators
  alias Pleroma.Object
+  alias Pleroma.Web.ActivityPub.ObjectValidators.CommonFixes
+  alias Pleroma.Web.ActivityPub.ObjectValidators.CommonValidations

  import Ecto.Changeset
-  import Pleroma.Web.ActivityPub.ObjectValidators.CommonValidations

  @primary_key false

@ -75,14 +76,15 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.CreateGenericValidator do
    data
    |> fix_context(meta)
    |> fix_addressing(meta)
+    |> CommonFixes.fix_actor()
  end

  def validate_data(cng, meta \\ []) do
    cng
    |> validate_required([:actor, :type, :object])
    |> validate_inclusion(:type, ["Create"])
-    |> validate_actor_presence()
-    |> validate_any_presence([:to, :cc])
+    |> CommonValidations.validate_actor_presence()
+    |> CommonValidations.validate_any_presence([:to, :cc])
    |> validate_actors_match(meta)
    |> validate_context_match(meta)
    |> validate_object_nonexistence()
--- a/lib/pleroma/web/activity_pub/object_validators/note_validator.ex
+++ b/lib/pleroma/web/activity_pub/object_validators/note_validator.ex
@ -1,73 +0,0 @@
-# Pleroma: A lightweight social networking server
-# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
-# SPDX-License-Identifier: AGPL-3.0-only
-
-defmodule Pleroma.Web.ActivityPub.ObjectValidators.NoteValidator do
-  use Ecto.Schema
-
-  alias Pleroma.EctoType.ActivityPub.ObjectValidators
-  alias Pleroma.Web.ActivityPub.Transmogrifier
-
-  import Ecto.Changeset
-
-  @primary_key false
-
-  embedded_schema do
-    field(:id, ObjectValidators.ObjectID, primary_key: true)
-    field(:to, ObjectValidators.Recipients, default: [])
-    field(:cc, ObjectValidators.Recipients, default: [])
-    field(:bto, ObjectValidators.Recipients, default: [])
-    field(:bcc, ObjectValidators.Recipients, default: [])
-    # TODO: Write type
-    field(:tag, {:array, :map}, default: [])
-    field(:type, :string)
-
-    field(:name, :string)
-    field(:summary, :string)
-    field(:content, :string)
-
-    field(:context, :string)
-    # short identifier for PleromaFE to group statuses by context
-    field(:context_id, :integer)
-
-    field(:actor, ObjectValidators.ObjectID)
-    field(:attributedTo, ObjectValidators.ObjectID)
-    field(:published, ObjectValidators.DateTime)
-    field(:emoji, ObjectValidators.Emoji, default: %{})
-    field(:sensitive, :boolean, default: false)
-    # TODO: Write type
-    field(:attachment, {:array, :map}, default: [])
-    field(:replies_count, :integer, default: 0)
-    field(:like_count, :integer, default: 0)
-    field(:announcement_count, :integer, default: 0)
-    field(:inReplyTo, ObjectValidators.ObjectID)
-    field(:url, ObjectValidators.Uri)
-
-    field(:likes, {:array, :string}, default: [])
-    field(:announcements, {:array, :string}, default: [])
-  end
-
-  def cast_and_validate(data) do
-    data
-    |> cast_data()
-    |> validate_data()
-  end
-
-  defp fix(data) do
-    data
-    |> Transmogrifier.fix_emoji()
-  end
-
-  def cast_data(data) do
-    data = fix(data)
-
-    %__MODULE__{}
-    |> cast(data, __schema__(:fields))
-  end
-
-  def validate_data(data_cng) do
-    data_cng
-    |> validate_inclusion(:type, ["Note"])
-    |> validate_required([:id, :actor, :to, :cc, :type, :content, :context])
-  end
-end
--- a/lib/pleroma/web/activity_pub/object_validators/question_validator.ex
+++ b/lib/pleroma/web/activity_pub/object_validators/question_validator.ex
@ -47,8 +47,8 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.QuestionValidator do
    # short identifier for PleromaFE to group statuses by context
    field(:context_id, :integer)

-    field(:likes, {:array, :string}, default: [])
-    field(:announcements, {:array, :string}, default: [])
+    field(:likes, {:array, ObjectValidators.ObjectID}, default: [])
+    field(:announcements, {:array, ObjectValidators.ObjectID}, default: [])

    field(:closed, ObjectValidators.DateTime)
    field(:voters, {:array, ObjectValidators.ObjectID}, default: [])
--- a/lib/pleroma/web/activity_pub/object_validators/url_object_validator.ex
+++ b/lib/pleroma/web/activity_pub/object_validators/url_object_validator.ex
@ -1,24 +0,0 @@
-# Pleroma: A lightweight social networking server
-# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
-# SPDX-License-Identifier: AGPL-3.0-only
-
-defmodule Pleroma.Web.ActivityPub.ObjectValidators.UrlObjectValidator do
-  use Ecto.Schema
-
-  alias Pleroma.EctoType.ActivityPub.ObjectValidators
-
-  import Ecto.Changeset
-  @primary_key false
-
-  embedded_schema do
-    field(:type, :string)
-    field(:href, ObjectValidators.Uri)
-    field(:mediaType, :string, default: "application/octet-stream")
-  end
-
-  def changeset(struct, data) do
-    struct
-    |> cast(data, __schema__(:fields))
-    |> validate_required([:type, :href, :mediaType])
-  end
-end
--- a/lib/pleroma/web/activity_pub/side_effects.ex
+++ b/lib/pleroma/web/activity_pub/side_effects.ex
@ -336,7 +336,7 @@ defmodule Pleroma.Web.ActivityPub.SideEffects do
  end

  def handle_object_creation(%{"type" => objtype} = object, meta)
-      when objtype in ~w[Audio Question Event] do
+      when objtype in ~w[Audio Video Question Event Article] do
    with {:ok, object, meta} <- Pipeline.common_pipeline(object, meta) do
      {:ok, object, meta}
    end
--- a/lib/pleroma/web/activity_pub/transmogrifier.ex
+++ b/lib/pleroma/web/activity_pub/transmogrifier.ex
@ -7,7 +7,6 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
  A module to handle coding from internal to wire ActivityPub and back.
  """
  alias Pleroma.Activity
-  alias Pleroma.EarmarkRenderer
  alias Pleroma.EctoType.ActivityPub.ObjectValidators
  alias Pleroma.Maps
  alias Pleroma.Object
@ -45,7 +44,6 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
    |> fix_addressing
    |> fix_summary
    |> fix_type(options)
-    |> fix_content
  end

  def fix_summary(%{"summary" => nil} = object) do
@ -274,24 +272,7 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
    Map.put(object, "url", url["href"])
  end

-  def fix_url(%{"type" => "Video", "url" => url} = object) when is_list(url) do
-    attachment =
-      Enum.find(url, fn x ->
-        media_type = x["mediaType"] || x["mimeType"] || ""
-
-        is_map(x) and String.starts_with?(media_type, "video/")
-      end)
-
-    link_element =
-      Enum.find(url, fn x -> is_map(x) and (x["mediaType"] || x["mimeType"]) == "text/html" end)
-
-    object
-    |> Map.put("attachment", [attachment])
-    |> Map.put("url", link_element["href"])
-  end
-
-  def fix_url(%{"type" => object_type, "url" => url} = object)
-      when object_type != "Video" and is_list(url) do
+  def fix_url(%{"url" => url} = object) when is_list(url) do
    first_element = Enum.at(url, 0)

    url_string =
@ -309,7 +290,7 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
  def fix_emoji(%{"tag" => tags} = object) when is_list(tags) do
    emoji =
      tags
-      |> Enum.filter(fn data -> data["type"] == "Emoji" and data["icon"] end)
+      |> Enum.filter(fn data -> is_map(data) and data["type"] == "Emoji" and data["icon"] end)
      |> Enum.reduce(%{}, fn data, mapping ->
        name = String.trim(data["name"], ":")

@ -371,18 +352,6 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do

  def fix_type(object, _), do: object

-  defp fix_content(%{"mediaType" => "text/markdown", "content" => content} = object)
-       when is_binary(content) do
-    html_content =
-      content
-      |> Earmark.as_html!(%Earmark.Options{renderer: EarmarkRenderer})
-      |> Pleroma.HTML.filter_tags()
-
-    Map.merge(object, %{"content" => html_content, "mediaType" => "text/html"})
-  end
-
-  defp fix_content(object), do: object
-
  # Reduce the object list to find the reported user.
  defp get_reported(objects) do
    Enum.reduce_while(objects, nil, fn ap_id, _ ->
@ -455,7 +424,7 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
        %{"type" => "Create", "object" => %{"type" => objtype} = object} = data,
        options
      )
-      when objtype in ~w{Article Note Video Page} do
+      when objtype in ~w{Note Page} do
    actor = Containment.get_actor(data)

    with nil <- Activity.get_create_by_object_ap_id(object["id"]),
@ -549,7 +518,9 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
        %{"type" => "Create", "object" => %{"type" => objtype}} = data,
        _options
      )
-      when objtype in ~w{Question Answer ChatMessage Audio Event} do
+      when objtype in ~w{Question Answer ChatMessage Audio Video Event Article} do
+    data = Map.put(data, "object", strip_internal_fields(data["object"]))
+
    with {:ok, %User{}} <- ObjectValidator.fetch_actor(data),
         {:ok, activity, _} <- Pipeline.common_pipeline(data, local: false) do
      {:ok, activity}
--- a/lib/pleroma/web/rich_media/helpers.ex
+++ b/lib/pleroma/web/rich_media/helpers.ex
@ -87,6 +87,50 @@ defmodule Pleroma.Web.RichMedia.Helpers do
  def rich_media_get(url) do
    headers = [{"user-agent", Pleroma.Application.user_agent() <> "; Bot"}]

-    Pleroma.HTTP.get(url, headers, @options)
+    head_check =
+      case Pleroma.HTTP.head(url, headers, @options) do
+        # If the HEAD request didn't reach the server for whatever reason,
+        # we assume the GET that comes right after won't either
+        {:error, _} = e ->
+          e
+
+        {:ok, %Tesla.Env{status: 200, headers: headers}} ->
+          with :ok <- check_content_type(headers),
+               :ok <- check_content_length(headers),
+               do: :ok
+
+        _ ->
+          :ok
+      end
+
+    with :ok <- head_check, do: Pleroma.HTTP.get(url, headers, @options)
+  end
+
+  defp check_content_type(headers) do
+    case List.keyfind(headers, "content-type", 0) do
+      {_, content_type} ->
+        case Plug.Conn.Utils.media_type(content_type) do
+          {:ok, "text", "html", _} -> :ok
+          _ -> {:error, {:content_type, content_type}}
+        end
+
+      _ ->
+        :ok
+    end
+  end
+
+  @max_body @options[:max_body]
+  defp check_content_length(headers) do
+    case List.keyfind(headers, "content-length", 0) do
+      {_, maybe_content_length} ->
+        case Integer.parse(maybe_content_length) do
+          {content_length, ""} when content_length <= @max_body -> :ok
+          {_, ""} -> {:error, :body_too_large}
+          _ -> :ok
+        end
+
+      _ ->
+        :ok
+    end
  end
 end
--- a/lib/pleroma/web/rich_media/parser.ex
+++ b/lib/pleroma/web/rich_media/parser.ex
@ -36,6 +36,14 @@ defmodule Pleroma.Web.RichMedia.Parser do
        {:ok, _data} = res ->
          res

+        {:error, :body_too_large} = e ->
+          e
+
+        {:error, {:content_type, _}} = e ->
+          e
+
+        # The TTL is not set for the errors above, since they are unlikely to change
+        # with time
        {:error, _} = e ->
          ttl = Pleroma.Config.get([:rich_media, :failure_backoff], 60_000)
          Cachex.expire(:rich_media_cache, url, ttl)
--- a/priv/gettext/es/LC_MESSAGES/errors.po
+++ b/priv/gettext/es/LC_MESSAGES/errors.po
@ -3,7 +3,7 @@ msgstr ""
 "Project-Id-Version: PACKAGE VERSION\n"
 "Report-Msgid-Bugs-To: \n"
 "POT-Creation-Date: 2020-09-09 09:49+0000\n"
-"PO-Revision-Date: 2020-09-09 10:52+0000\n"
+"PO-Revision-Date: 2020-09-11 21:26+0000\n"
 "Last-Translator: tarteka <info@tarteka.net>\n"
 "Language-Team: Spanish <https://translate.pleroma.social/projects/pleroma/"
 "pleroma/es/>\n"
@ -94,52 +94,52 @@ msgid "must be less than %{number}"
 msgstr ""

 msgid "must be greater than %{number}"
-msgstr ""
+msgstr "debe ser mayor que %{number}"

 msgid "must be less than or equal to %{number}"
-msgstr ""
+msgstr "debe ser menor o igual que %{number}"

 msgid "must be greater than or equal to %{number}"
-msgstr ""
+msgstr "deber ser mayor o igual que %{number}"

 msgid "must be equal to %{number}"
-msgstr ""
+msgstr "deber ser igual a %{number}"

 #: lib/pleroma/web/common_api/common_api.ex:505
 #, elixir-format
 msgid "Account not found"
-msgstr ""
+msgstr "Cuenta no encontrada"

 #: lib/pleroma/web/common_api/common_api.ex:339
 #, elixir-format
 msgid "Already voted"
-msgstr ""
+msgstr "Ya has votado"

 #: lib/pleroma/web/oauth/oauth_controller.ex:359
 #, elixir-format
 msgid "Bad request"
-msgstr ""
+msgstr "Solicitud incorrecta"

 #: lib/pleroma/web/activity_pub/activity_pub_controller.ex:426
 #, elixir-format
 msgid "Can't delete object"
-msgstr ""
+msgstr "No se puede eliminar el objeto"

 #: lib/pleroma/web/controller_helper.ex:105
 #: lib/pleroma/web/controller_helper.ex:111
 #, elixir-format
 msgid "Can't display this activity"
-msgstr ""
+msgstr "No se puede mostrar esta actividad"

 #: lib/pleroma/web/mastodon_api/controllers/account_controller.ex:285
 #, elixir-format
 msgid "Can't find user"
-msgstr ""
+msgstr "No se puede encontrar al usuario"

 #: lib/pleroma/web/pleroma_api/controllers/account_controller.ex:61
 #, elixir-format
 msgid "Can't get favorites"
-msgstr ""
+msgstr "No se puede obtener los favoritos"

 #: lib/pleroma/web/activity_pub/activity_pub_controller.ex:438
 #, elixir-format
@ -149,7 +149,7 @@ msgstr ""
 #: lib/pleroma/web/common_api/utils.ex:563
 #, elixir-format
 msgid "Cannot post an empty status without attachments"
-msgstr ""
+msgstr "No se puede publicar un estado vacío y sin archivos adjuntos"

 #: lib/pleroma/web/common_api/utils.ex:511
 #, elixir-format
--- a/priv/repo/migrations/20200825061316_move_activity_expirations_to_oban.exs
+++ b/priv/repo/migrations/20200825061316_move_activity_expirations_to_oban.exs
@ -4,6 +4,8 @@ defmodule Pleroma.Repo.Migrations.MoveActivityExpirationsToOban do
  import Ecto.Query, only: [from: 2]

  def change do
+    Pleroma.Config.Oban.warn()
+
    Supervisor.start_link([{Oban, Pleroma.Config.get(Oban)}],
      strategy: :one_for_one,
      name: Pleroma.Supervisor
--- a/priv/repo/migrations/20200907092050_move_tokens_expiration_into_oban.exs
+++ b/priv/repo/migrations/20200907092050_move_tokens_expiration_into_oban.exs
@ -4,6 +4,8 @@ defmodule Pleroma.Repo.Migrations.MoveTokensExpirationIntoOban do
  import Ecto.Query, only: [from: 2]

  def change do
+    Pleroma.Config.Oban.warn()
+
    Supervisor.start_link([{Oban, Pleroma.Config.get(Oban)}],
      strategy: :one_for_one,
      name: Pleroma.Supervisor
--- a/priv/repo/migrations/20200910113106_remove_managed_config_from_db.exs
+++ b/priv/repo/migrations/20200910113106_remove_managed_config_from_db.exs
@ -0,0 +1,27 @@
+defmodule Pleroma.Repo.Migrations.RemoveManagedConfigFromDb do
+  use Ecto.Migration
+  import Ecto.Query
+  alias Pleroma.ConfigDB
+  alias Pleroma.Repo
+
+  def up do
+    config_entry =
+      from(c in ConfigDB,
+        select: [:id, :value],
+        where: c.group == ^:pleroma and c.key == ^:instance
+      )
+      |> Repo.one()
+
+    if config_entry do
+      {_, value} = Keyword.pop(config_entry.value, :managed_config)
+
+      config_entry
+      |> Ecto.Changeset.change(value: value)
+      |> Repo.update()
+    end
+  end
+
+  def down do
+    :ok
+  end
+end
--- a/priv/repo/migrations/20200914105638_delete_notification_without_activity.exs
+++ b/priv/repo/migrations/20200914105638_delete_notification_without_activity.exs
@ -0,0 +1,30 @@
+defmodule Pleroma.Repo.Migrations.DeleteNotificationWithoutActivity do
+  use Ecto.Migration
+
+  import Ecto.Query
+  alias Pleroma.Repo
+
+  def up do
+    from(
+      q in Pleroma.Notification,
+      left_join: c in assoc(q, :activity),
+      select: %{id: type(q.id, :integer)},
+      where: is_nil(c.id)
+    )
+    |> Repo.chunk_stream(1_000, :batches)
+    |> Stream.each(fn records ->
+      notification_ids = Enum.map(records, fn %{id: id} -> id end)
+
+      Repo.delete_all(
+        from(n in "notifications",
+          where: n.id in ^notification_ids
+        )
+      )
+    end)
+    |> Stream.run()
+  end
+
+  def down do
+    :ok
+  end
+end
--- a/priv/repo/migrations/20200914105800_add_notification_constraints.exs
+++ b/priv/repo/migrations/20200914105800_add_notification_constraints.exs
@ -0,0 +1,23 @@
+defmodule Pleroma.Repo.Migrations.AddNotificationConstraints do
+  use Ecto.Migration
+
+  def up do
+    drop(constraint(:notifications, "notifications_activity_id_fkey"))
+
+    alter table(:notifications) do
+      modify(:activity_id, references(:activities, type: :uuid, on_delete: :delete_all),
+        null: false
+      )
+    end
+  end
+
+  def down do
+    drop(constraint(:notifications, "notifications_activity_id_fkey"))
+
+    alter table(:notifications) do
+      modify(:activity_id, references(:activities, type: :uuid, on_delete: :delete_all),
+        null: true
+      )
+    end
+  end
+end
--- a/test/fixtures/tesla_mock/framatube.org-video.json
+++ b/test/fixtures/tesla_mock/framatube.org-video.json
--- a/test/fixtures/tesla_mock/wedistribute-create-article.json
+++ b/test/fixtures/tesla_mock/wedistribute-create-article.json
@ -0,0 +1 @@
+{"@context":["https:\/\/www.w3.org\/ns\/activitystreams"],"type":"Create","actor":"https:\/\/wedistribute.org\/wp-json\/pterotype\/v1\/actor\/-blog","object":{"@context":["https:\/\/www.w3.org\/ns\/activitystreams"],"type":"Article","name":"The end is near: Mastodon plans to drop OStatus support","content":"<!-- wp:paragraph {\"dropCap\":true} -->\n<p class=\"has-drop-cap\">The days of OStatus are numbered. The venerable protocol has served as a glue between many different types of servers since the early days of the Fediverse, connecting StatusNet (now GNU Social) to Friendica, Hubzilla, Mastodon, and Pleroma.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>Now that many fediverse platforms support ActivityPub as a successor protocol, Mastodon appears to be drawing a line in the sand. In <a href=\"https:\/\/www.patreon.com\/posts\/mastodon-2-9-and-28121681\">a Patreon update<\/a>, Eugen Rochko writes:<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:quote -->\n<blockquote class=\"wp-block-quote\"><p>...OStatus...has overstayed its welcome in the code...and now that most of the network uses ActivityPub, it's time for it to go. <\/p><cite>Eugen Rochko, Mastodon creator<\/cite><\/blockquote>\n<!-- \/wp:quote -->\n\n<!-- wp:paragraph -->\n<p>The <a href=\"https:\/\/github.com\/tootsuite\/mastodon\/pull\/11205\">pull request<\/a> to remove Pubsubhubbub and Salmon, two of the main components of OStatus, has already been merged into Mastodon's master branch.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>Some projects will be left in the dark as a side effect of this. GNU Social and PostActiv, for example, both only communicate using OStatus. While <a href=\"https:\/\/mastodon.social\/@dansup\/102076573310057902\">some discussion<\/a> exists regarding adopting ActivityPub for GNU Social, and <a href=\"https:\/\/notabug.org\/diogo\/gnu-social\/src\/activitypub\/plugins\/ActivityPub\">a plugin is in development<\/a>, it hasn't been formally adopted yet. We just hope that the <a href=\"https:\/\/status.fsf.org\/main\/public\">Free Software Foundation's instance<\/a> gets updated in time!<\/p>\n<!-- \/wp:paragraph -->","summary":"One of the largest platforms in the federated social web is dropping the protocol that it started with.","attributedTo":"https:\/\/wedistribute.org\/wp-json\/pterotype\/v1\/actor\/-blog","url":"https:\/\/wedistribute.org\/2019\/07\/mastodon-drops-ostatus\/","to":["https:\/\/www.w3.org\/ns\/activitystreams#Public","https:\/\/wedistribute.org\/wp-json\/pterotype\/v1\/actor\/-blog\/followers"],"id":"https:\/\/wedistribute.org\/wp-json\/pterotype\/v1\/object\/85810","likes":"https:\/\/wedistribute.org\/wp-json\/pterotype\/v1\/object\/85810\/likes","shares":"https:\/\/wedistribute.org\/wp-json\/pterotype\/v1\/object\/85810\/shares"},"to":["https:\/\/www.w3.org\/ns\/activitystreams#Public","https:\/\/wedistribute.org\/wp-json\/pterotype\/v1\/actor\/-blog\/followers"],"id":"https:\/\/wedistribute.org\/wp-json\/pterotype\/v1\/object\/85809"}
--- a/test/marker_test.exs
+++ b/test/marker_test.exs
@ -33,8 +33,8 @@ defmodule Pleroma.MarkerTest do
    test "returns user markers" do
      user = insert(:user)
      marker = insert(:marker, user: user)
-      insert(:notification, user: user)
-      insert(:notification, user: user)
+      insert(:notification, user: user, activity: insert(:note_activity))
+      insert(:notification, user: user, activity: insert(:note_activity))
      insert(:marker, timeline: "home", user: user)

      assert Marker.get_markers(
--- a/test/object/fetcher_test.exs
+++ b/test/object/fetcher_test.exs
@ -6,10 +6,12 @@ defmodule Pleroma.Object.FetcherTest do
  use Pleroma.DataCase

  alias Pleroma.Activity
+  alias Pleroma.Config
  alias Pleroma.Object
  alias Pleroma.Object.Fetcher
-  import Tesla.Mock
+
  import Mock
+  import Tesla.Mock

  setup do
    mock(fn
@ -71,20 +73,20 @@ defmodule Pleroma.Object.FetcherTest do
    setup do: clear_config([:instance, :federation_incoming_replies_max_depth])

    test "it returns thread depth exceeded error if thread depth is exceeded" do
-      Pleroma.Config.put([:instance, :federation_incoming_replies_max_depth], 0)
+      Config.put([:instance, :federation_incoming_replies_max_depth], 0)

      assert {:error, "Max thread distance exceeded."} =
               Fetcher.fetch_object_from_id(@ap_id, depth: 1)
    end

    test "it fetches object if max thread depth is restricted to 0 and depth is not specified" do
-      Pleroma.Config.put([:instance, :federation_incoming_replies_max_depth], 0)
+      Config.put([:instance, :federation_incoming_replies_max_depth], 0)

      assert {:ok, _} = Fetcher.fetch_object_from_id(@ap_id)
    end

    test "it fetches object if requested depth does not exceed max thread depth" do
-      Pleroma.Config.put([:instance, :federation_incoming_replies_max_depth], 10)
+      Config.put([:instance, :federation_incoming_replies_max_depth], 10)

      assert {:ok, _} = Fetcher.fetch_object_from_id(@ap_id, depth: 10)
    end
@ -120,6 +122,16 @@ defmodule Pleroma.Object.FetcherTest do

      assert object == object_again
    end
+
+    test "Return MRF reason when fetched status is rejected by one" do
+      clear_config([:mrf_keyword, :reject], ["yeah"])
+      clear_config([:mrf, :policies], [Pleroma.Web.ActivityPub.MRF.KeywordPolicy])
+
+      assert {:reject, "[KeywordPolicy] Matches with rejected keyword"} ==
+               Fetcher.fetch_object_from_id(
+                 "http://mastodon.example.org/@admin/99541947525187367"
+               )
+    end
  end

  describe "implementation quirks" do
@ -212,7 +224,7 @@ defmodule Pleroma.Object.FetcherTest do
                   Pleroma.Signature,
                   [:passthrough],
                   [] do
-      Pleroma.Config.put([:activitypub, :sign_object_fetches], true)
+      Config.put([:activitypub, :sign_object_fetches], true)

      Fetcher.fetch_object_from_id("http://mastodon.example.org/@admin/99541947525187367")

@ -223,7 +235,7 @@ defmodule Pleroma.Object.FetcherTest do
                   Pleroma.Signature,
                   [:passthrough],
                   [] do
-      Pleroma.Config.put([:activitypub, :sign_object_fetches], false)
+      Config.put([:activitypub, :sign_object_fetches], false)

      Fetcher.fetch_object_from_id("http://mastodon.example.org/@admin/99541947525187367")

--- a/test/repo_test.exs
+++ b/test/repo_test.exs
@ -37,7 +37,9 @@ defmodule Pleroma.RepoTest do

    test "get one-to-many assoc from repo" do
      user = insert(:user)
-      notification = refresh_record(insert(:notification, user: user))
+
+      notification =
+        refresh_record(insert(:notification, user: user, activity: insert(:note_activity)))

      assert Repo.get_assoc(user, :notifications) == {:ok, [notification]}
    end
@ -47,4 +49,32 @@ defmodule Pleroma.RepoTest do
      assert Repo.get_assoc(token, :user) == {:error, :not_found}
    end
  end
+
+  describe "chunk_stream/3" do
+    test "fetch records one-by-one" do
+      users = insert_list(50, :user)
+
+      {fetch_users, 50} =
+        from(t in User)
+        |> Repo.chunk_stream(5)
+        |> Enum.reduce({[], 0}, fn %User{} = user, {acc, count} ->
+          {acc ++ [user], count + 1}
+        end)
+
+      assert users == fetch_users
+    end
+
+    test "fetch records in bulk" do
+      users = insert_list(50, :user)
+
+      {fetch_users, 10} =
+        from(t in User)
+        |> Repo.chunk_stream(5, :batches)
+        |> Enum.reduce({[], 0}, fn users, {acc, count} ->
+          {acc ++ users, count + 1}
+        end)
+
+      assert users == fetch_users
+    end
+  end
 end
--- a/test/support/http_request_mock.ex
+++ b/test/support/http_request_mock.ex
@ -1262,4 +1262,21 @@ defmodule HttpRequestMock do
       inspect(headers)
     }"}
  end
+
+  # Most of the rich media mocks are missing HEAD requests, so we just return 404.
+  @rich_media_mocks [
+    "https://example.com/ogp",
+    "https://example.com/ogp-missing-data",
+    "https://example.com/twitter-card"
+  ]
+  def head(url, _query, _body, _headers) when url in @rich_media_mocks do
+    {:ok, %Tesla.Env{status: 404, body: ""}}
+  end
+
+  def head(url, query, body, headers) do
+    {:error,
+     "Mock response not implemented for HEAD #{inspect(url)}, #{query}, #{inspect(body)}, #{
+       inspect(headers)
+     }"}
+  end
 end
--- a/test/user_test.exs
+++ b/test/user_test.exs
@ -440,6 +440,45 @@ defmodule Pleroma.UserTest do
      assert activity.actor == welcome_user.ap_id
    end

+    setup do:
+            clear_config(:mrf_simple,
+              media_removal: [],
+              media_nsfw: [],
+              federated_timeline_removal: [],
+              report_removal: [],
+              reject: [],
+              followers_only: [],
+              accept: [],
+              avatar_removal: [],
+              banner_removal: [],
+              reject_deletes: []
+            )
+
+    setup do:
+            clear_config(:mrf,
+              policies: [
+                Pleroma.Web.ActivityPub.MRF.SimplePolicy
+              ]
+            )
+
+    test "it sends a welcome chat message when Simple policy applied to local instance" do
+      Pleroma.Config.put([:mrf_simple, :media_nsfw], ["localhost"])
+
+      welcome_user = insert(:user)
+      Pleroma.Config.put([:welcome, :chat_message, :enabled], true)
+      Pleroma.Config.put([:welcome, :chat_message, :sender_nickname], welcome_user.nickname)
+      Pleroma.Config.put([:welcome, :chat_message, :message], "Hello, this is a chat message")
+
+      cng = User.register_changeset(%User{}, @full_user_data)
+      {:ok, registered_user} = User.register(cng)
+      ObanHelpers.perform_all()
+
+      activity = Repo.one(Pleroma.Activity)
+      assert registered_user.ap_id in activity.recipients
+      assert Object.normalize(activity).data["content"] =~ "chat message"
+      assert activity.actor == welcome_user.ap_id
+    end
+
    test "it sends a welcome email message if it is set" do
      welcome_user = insert(:user)
      Pleroma.Config.put([:welcome, :email, :enabled], true)
@ -1637,7 +1676,7 @@ defmodule Pleroma.UserTest do
      assert User.visible_for(user, user) == :visible
    end

-    test "returns false when the account is unauthenticated and auth is required" do
+    test "returns false when the account is unconfirmed and confirmation is required" do
      Pleroma.Config.put([:instance, :account_activation_required], true)

      user = insert(:user, local: true, confirmation_pending: true)
@ -1646,14 +1685,23 @@ defmodule Pleroma.UserTest do
      refute User.visible_for(user, other_user) == :visible
    end

-    test "returns true when the account is unauthenticated and auth is not required" do
+    test "returns true when the account is unconfirmed and confirmation is required but the account is remote" do
+      Pleroma.Config.put([:instance, :account_activation_required], true)
+
+      user = insert(:user, local: false, confirmation_pending: true)
+      other_user = insert(:user, local: true)
+
+      assert User.visible_for(user, other_user) == :visible
+    end
+
+    test "returns true when the account is unconfirmed and confirmation is not required" do
      user = insert(:user, local: true, confirmation_pending: true)
      other_user = insert(:user, local: true)

      assert User.visible_for(user, other_user) == :visible
    end

-    test "returns true when the account is unauthenticated and being viewed by a privileged account (auth required)" do
+    test "returns true when the account is unconfirmed and being viewed by a privileged account (confirmation required)" do
      Pleroma.Config.put([:instance, :account_activation_required], true)

      user = insert(:user, local: true, confirmation_pending: true)
--- a/test/web/activity_pub/activity_pub_test.exs
+++ b/test/web/activity_pub/activity_pub_test.exs
@ -1810,6 +1810,14 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubTest do
        |> Enum.map(& &1.id)

      assert activities_ids == []
+
+      activities_ids =
+        %{}
+        |> Map.put(:reply_visibility, "self")
+        |> Map.put(:reply_filtering_user, nil)
+        |> ActivityPub.fetch_public_activities()
+
+      assert activities_ids == []
    end

    test "home timeline", %{users: %{u1: user}} do
--- a/test/web/activity_pub/object_validators/article_note_validator_test.exs
+++ b/test/web/activity_pub/object_validators/article_note_validator_test.exs
@ -2,10 +2,10 @@
 # Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only

-defmodule Pleroma.Web.ActivityPub.ObjectValidators.NoteValidatorTest do
+defmodule Pleroma.Web.ActivityPub.ObjectValidators.ArticleNoteValidatorTest do
  use Pleroma.DataCase

-  alias Pleroma.Web.ActivityPub.ObjectValidators.NoteValidator
+  alias Pleroma.Web.ActivityPub.ObjectValidators.ArticleNoteValidator
  alias Pleroma.Web.ActivityPub.Utils

  import Pleroma.Factory
@ -29,7 +29,7 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.NoteValidatorTest do
    end

    test "a basic note validates", %{note: note} do
-      %{valid?: true} = NoteValidator.cast_and_validate(note)
+      %{valid?: true} = ArticleNoteValidator.cast_and_validate(note)
    end
  end
 end
--- a/test/web/activity_pub/transmogrifier/article_handling_test.exs
+++ b/test/web/activity_pub/transmogrifier/article_handling_test.exs
@ -0,0 +1,75 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.ActivityPub.Transmogrifier.ArticleHandlingTest do
+  use Oban.Testing, repo: Pleroma.Repo
+  use Pleroma.DataCase
+
+  alias Pleroma.Activity
+  alias Pleroma.Object
+  alias Pleroma.Object.Fetcher
+  alias Pleroma.Web.ActivityPub.Transmogrifier
+
+  test "Pterotype (Wordpress Plugin) Article" do
+    Tesla.Mock.mock(fn %{url: "https://wedistribute.org/wp-json/pterotype/v1/actor/-blog"} ->
+      %Tesla.Env{status: 200, body: File.read!("test/fixtures/tesla_mock/wedistribute-user.json")}
+    end)
+
+    data =
+      File.read!("test/fixtures/tesla_mock/wedistribute-create-article.json") |> Jason.decode!()
+
+    {:ok, %Activity{data: data, local: false}} = Transmogrifier.handle_incoming(data)
+
+    object = Object.normalize(data["object"])
+
+    assert object.data["name"] == "The end is near: Mastodon plans to drop OStatus support"
+
+    assert object.data["summary"] ==
+             "One of the largest platforms in the federated social web is dropping the protocol that it started with."
+
+    assert object.data["url"] == "https://wedistribute.org/2019/07/mastodon-drops-ostatus/"
+  end
+
+  test "Plume Article" do
+    Tesla.Mock.mock(fn
+      %{url: "https://baptiste.gelez.xyz/~/PlumeDevelopment/this-month-in-plume-june-2018/"} ->
+        %Tesla.Env{
+          status: 200,
+          body: File.read!("test/fixtures/tesla_mock/baptiste.gelex.xyz-article.json")
+        }
+
+      %{url: "https://baptiste.gelez.xyz/@/BaptisteGelez"} ->
+        %Tesla.Env{
+          status: 200,
+          body: File.read!("test/fixtures/tesla_mock/baptiste.gelex.xyz-user.json")
+        }
+    end)
+
+    {:ok, object} =
+      Fetcher.fetch_object_from_id(
+        "https://baptiste.gelez.xyz/~/PlumeDevelopment/this-month-in-plume-june-2018/"
+      )
+
+    assert object.data["name"] == "This Month in Plume: June 2018"
+
+    assert object.data["url"] ==
+             "https://baptiste.gelez.xyz/~/PlumeDevelopment/this-month-in-plume-june-2018/"
+  end
+
+  test "Prismo Article" do
+    Tesla.Mock.mock(fn %{url: "https://prismo.news/@mxb"} ->
+      %Tesla.Env{
+        status: 200,
+        body: File.read!("test/fixtures/tesla_mock/https___prismo.news__mxb.json")
+      }
+    end)
+
+    data = File.read!("test/fixtures/prismo-url-map.json") |> Jason.decode!()
+
+    {:ok, %Activity{data: data, local: false}} = Transmogrifier.handle_incoming(data)
+    object = Object.normalize(data["object"])
+
+    assert object.data["url"] == "https://prismo.news/posts/83"
+  end
+end
--- a/test/web/activity_pub/transmogrifier/video_handling_test.exs
+++ b/test/web/activity_pub/transmogrifier/video_handling_test.exs
@ -0,0 +1,93 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.ActivityPub.Transmogrifier.VideoHandlingTest do
+  use Oban.Testing, repo: Pleroma.Repo
+  use Pleroma.DataCase
+
+  alias Pleroma.Activity
+  alias Pleroma.Object
+  alias Pleroma.Object.Fetcher
+  alias Pleroma.Web.ActivityPub.Transmogrifier
+
+  setup_all do
+    Tesla.Mock.mock_global(fn env -> apply(HttpRequestMock, :request, [env]) end)
+    :ok
+  end
+
+  test "skip converting the content when it is nil" do
+    data =
+      File.read!("test/fixtures/tesla_mock/framatube.org-video.json")
+      |> Jason.decode!()
+      |> Kernel.put_in(["object", "content"], nil)
+
+    {:ok, %Activity{local: false} = activity} = Transmogrifier.handle_incoming(data)
+
+    assert object = Object.normalize(activity, false)
+
+    assert object.data["content"] == nil
+  end
+
+  test "it converts content of object to html" do
+    data = File.read!("test/fixtures/tesla_mock/framatube.org-video.json") |> Jason.decode!()
+
+    {:ok, %Activity{local: false} = activity} = Transmogrifier.handle_incoming(data)
+
+    assert object = Object.normalize(activity, false)
+
+    assert object.data["content"] ==
+             "<p>Après avoir mené avec un certain succès la campagne « Dégooglisons Internet » en 2014, l’association Framasoft annonce fin 2019 arrêter progressivement un certain nombre de ses services alternatifs aux GAFAM. Pourquoi ?</p><p>Transcription par @aprilorg ici : <a href=\"https://www.april.org/deframasoftisons-internet-pierre-yves-gosset-framasoft\">https://www.april.org/deframasoftisons-internet-pierre-yves-gosset-framasoft</a></p>"
+  end
+
+  test "it remaps video URLs as attachments if necessary" do
+    {:ok, object} =
+      Fetcher.fetch_object_from_id(
+        "https://peertube.moe/videos/watch/df5f464b-be8d-46fb-ad81-2d4c2d1630e3"
+      )
+
+    assert object.data["url"] ==
+             "https://peertube.moe/videos/watch/df5f464b-be8d-46fb-ad81-2d4c2d1630e3"
+
+    assert object.data["attachment"] == [
+             %{
+               "type" => "Link",
+               "mediaType" => "video/mp4",
+               "name" => nil,
+               "url" => [
+                 %{
+                   "href" =>
+                     "https://peertube.moe/static/webseed/df5f464b-be8d-46fb-ad81-2d4c2d1630e3-480.mp4",
+                   "mediaType" => "video/mp4",
+                   "type" => "Link"
+                 }
+               ]
+             }
+           ]
+
+    data = File.read!("test/fixtures/tesla_mock/framatube.org-video.json") |> Jason.decode!()
+
+    {:ok, %Activity{local: false} = activity} = Transmogrifier.handle_incoming(data)
+
+    assert object = Object.normalize(activity, false)
+
+    assert object.data["attachment"] == [
+             %{
+               "type" => "Link",
+               "mediaType" => "video/mp4",
+               "name" => nil,
+               "url" => [
+                 %{
+                   "href" =>
+                     "https://framatube.org/static/webseed/6050732a-8a7a-43d4-a6cd-809525a1d206-1080.mp4",
+                   "mediaType" => "video/mp4",
+                   "type" => "Link"
+                 }
+               ]
+             }
+           ]
+
+    assert object.data["url"] ==
+             "https://framatube.org/videos/watch/6050732a-8a7a-43d4-a6cd-809525a1d206"
+  end
+end
--- a/test/web/activity_pub/transmogrifier_test.exs
+++ b/test/web/activity_pub/transmogrifier_test.exs
@ -8,7 +8,6 @@ defmodule Pleroma.Web.ActivityPub.TransmogrifierTest do

  alias Pleroma.Activity
  alias Pleroma.Object
-  alias Pleroma.Object.Fetcher
  alias Pleroma.Tests.ObanHelpers
  alias Pleroma.User
  alias Pleroma.Web.ActivityPub.Transmogrifier
@ -45,15 +44,6 @@ defmodule Pleroma.Web.ActivityPub.TransmogrifierTest do
      assert "test" in object.data["tag"]
    end

-    test "it works for incoming notices with url not being a string (prismo)" do
-      data = File.read!("test/fixtures/prismo-url-map.json") |> Poison.decode!()
-
-      {:ok, %Activity{data: data, local: false}} = Transmogrifier.handle_incoming(data)
-      object = Object.normalize(data["object"])
-
-      assert object.data["url"] == "https://prismo.news/posts/83"
-    end
-
    test "it cleans up incoming notices which are not really DMs" do
      user = insert(:user)
      other_user = insert(:user)
@ -355,83 +345,6 @@ defmodule Pleroma.Web.ActivityPub.TransmogrifierTest do
      refute User.following?(User.get_cached_by_ap_id(data["actor"]), user)
    end

-    test "skip converting the content when it is nil" do
-      object_id = "https://peertube.social/videos/watch/278d2b7c-0f38-4aaa-afe6-9ecc0c4a34fe"
-
-      {:ok, object} = Fetcher.fetch_and_contain_remote_object_from_id(object_id)
-
-      result =
-        Pleroma.Web.ActivityPub.Transmogrifier.fix_object(Map.merge(object, %{"content" => nil}))
-
-      assert result["content"] == nil
-    end
-
-    test "it converts content of object to html" do
-      object_id = "https://peertube.social/videos/watch/278d2b7c-0f38-4aaa-afe6-9ecc0c4a34fe"
-
-      {:ok, %{"content" => content_markdown}} =
-        Fetcher.fetch_and_contain_remote_object_from_id(object_id)
-
-      {:ok, %Pleroma.Object{data: %{"content" => content}} = object} =
-        Fetcher.fetch_object_from_id(object_id)
-
-      assert content_markdown ==
-               "Support this and our other Michigan!/usr/group videos and meetings. Learn more at http://mug.org/membership\n\nTwenty Years in Jail: FreeBSD's Jails, Then and Now\n\nJails started as a limited virtualization system, but over the last two years they've..."
-
-      assert content ==
-               "<p>Support this and our other Michigan!/usr/group videos and meetings. Learn more at <a href=\"http://mug.org/membership\">http://mug.org/membership</a></p><p>Twenty Years in Jail: FreeBSD’s Jails, Then and Now</p><p>Jails started as a limited virtualization system, but over the last two years they’ve…</p>"
-
-      assert object.data["mediaType"] == "text/html"
-    end
-
-    test "it remaps video URLs as attachments if necessary" do
-      {:ok, object} =
-        Fetcher.fetch_object_from_id(
-          "https://peertube.moe/videos/watch/df5f464b-be8d-46fb-ad81-2d4c2d1630e3"
-        )
-
-      assert object.data["url"] ==
-               "https://peertube.moe/videos/watch/df5f464b-be8d-46fb-ad81-2d4c2d1630e3"
-
-      assert object.data["attachment"] == [
-               %{
-                 "type" => "Link",
-                 "mediaType" => "video/mp4",
-                 "url" => [
-                   %{
-                     "href" =>
-                       "https://peertube.moe/static/webseed/df5f464b-be8d-46fb-ad81-2d4c2d1630e3-480.mp4",
-                     "mediaType" => "video/mp4",
-                     "type" => "Link"
-                   }
-                 ]
-               }
-             ]
-
-      {:ok, object} =
-        Fetcher.fetch_object_from_id(
-          "https://framatube.org/videos/watch/6050732a-8a7a-43d4-a6cd-809525a1d206"
-        )
-
-      assert object.data["attachment"] == [
-               %{
-                 "type" => "Link",
-                 "mediaType" => "video/mp4",
-                 "url" => [
-                   %{
-                     "href" =>
-                       "https://framatube.org/static/webseed/6050732a-8a7a-43d4-a6cd-809525a1d206-1080.mp4",
-                     "mediaType" => "video/mp4",
-                     "type" => "Link"
-                   }
-                 ]
-               }
-             ]
-
-      assert object.data["url"] ==
-               "https://framatube.org/videos/watch/6050732a-8a7a-43d4-a6cd-809525a1d206"
-    end
-
    test "it accepts Flag activities" do
      user = insert(:user)
      other_user = insert(:user)
@ -1133,75 +1046,7 @@ defmodule Pleroma.Web.ActivityPub.TransmogrifierTest do
             }
    end

-    test "fixes data for video object" do
-      object = %{
-        "type" => "Video",
-        "url" => [
-          %{
-            "type" => "Link",
-            "mimeType" => "video/mp4",
-            "href" => "https://peede8d-46fb-ad81-2d4c2d1630e3-480.mp4"
-          },
-          %{
-            "type" => "Link",
-            "mimeType" => "video/mp4",
-            "href" => "https://peertube46fb-ad81-2d4c2d1630e3-240.mp4"
-          },
-          %{
-            "type" => "Link",
-            "mimeType" => "text/html",
-            "href" => "https://peertube.-2d4c2d1630e3"
-          },
-          %{
-            "type" => "Link",
-            "mimeType" => "text/html",
-            "href" => "https://peertube.-2d4c2d16377-42"
-          }
-        ]
-      }
-
-      assert Transmogrifier.fix_url(object) == %{
-               "attachment" => [
-                 %{
-                   "href" => "https://peede8d-46fb-ad81-2d4c2d1630e3-480.mp4",
-                   "mimeType" => "video/mp4",
-                   "type" => "Link"
-                 }
-               ],
-               "type" => "Video",
-               "url" => "https://peertube.-2d4c2d1630e3"
-             }
-    end
-
-    test "fixes url for not Video object" do
-      object = %{
-        "type" => "Text",
-        "url" => [
-          %{
-            "type" => "Link",
-            "mimeType" => "text/html",
-            "href" => "https://peertube.-2d4c2d1630e3"
-          },
-          %{
-            "type" => "Link",
-            "mimeType" => "text/html",
-            "href" => "https://peertube.-2d4c2d16377-42"
-          }
-        ]
-      }
-
-      assert Transmogrifier.fix_url(object) == %{
-               "type" => "Text",
-               "url" => "https://peertube.-2d4c2d1630e3"
-             }
-
-      assert Transmogrifier.fix_url(%{"type" => "Text", "url" => []}) == %{
-               "type" => "Text",
-               "url" => ""
-             }
-    end
-
-    test "retunrs not modified object" do
+    test "returns non-modified object" do
      assert Transmogrifier.fix_url(%{"type" => "Text"}) == %{"type" => "Text"}
    end
  end
--- a/test/web/mastodon_api/controllers/account_controller_test.exs
+++ b/test/web/mastodon_api/controllers/account_controller_test.exs
@ -1442,7 +1442,10 @@ defmodule Pleroma.Web.MastodonAPI.AccountControllerTest do
  describe "verify_credentials" do
    test "verify_credentials" do
      %{user: user, conn: conn} = oauth_access(["read:accounts"])
-      [notification | _] = insert_list(7, :notification, user: user)
+
+      [notification | _] =
+        insert_list(7, :notification, user: user, activity: insert(:note_activity))
+
      Pleroma.Notification.set_read_up_to(user, notification.id)
      conn = get(conn, "/api/v1/accounts/verify_credentials")

--- a/test/web/mastodon_api/controllers/marker_controller_test.exs
+++ b/test/web/mastodon_api/controllers/marker_controller_test.exs
@ -11,7 +11,7 @@ defmodule Pleroma.Web.MastodonAPI.MarkerControllerTest do
    test "gets markers with correct scopes", %{conn: conn} do
      user = insert(:user)
      token = insert(:oauth_token, user: user, scopes: ["read:statuses"])
-      insert_list(7, :notification, user: user)
+      insert_list(7, :notification, user: user, activity: insert(:note_activity))

      {:ok, %{"notifications" => marker}} =
        Pleroma.Marker.upsert(
--- a/test/web/mastodon_api/views/account_view_test.exs
+++ b/test/web/mastodon_api/views/account_view_test.exs
@ -448,7 +448,7 @@ defmodule Pleroma.Web.MastodonAPI.AccountViewTest do

    test "shows unread_count only to the account owner" do
      user = insert(:user)
-      insert_list(7, :notification, user: user)
+      insert_list(7, :notification, user: user, activity: insert(:note_activity))
      other_user = insert(:user)

      user = User.get_cached_by_ap_id(user.ap_id)
--- a/test/web/rich_media/parser_test.exs
+++ b/test/web/rich_media/parser_test.exs
@ -56,6 +56,27 @@ defmodule Pleroma.Web.RichMedia.ParserTest do

      %{method: :get, url: "http://example.com/error"} ->
        {:error, :overload}
+
+      %{
+        method: :head,
+        url: "http://example.com/huge-page"
+      } ->
+        %Tesla.Env{
+          status: 200,
+          headers: [{"content-length", "2000001"}, {"content-type", "text/html"}]
+        }
+
+      %{
+        method: :head,
+        url: "http://example.com/pdf-file"
+      } ->
+        %Tesla.Env{
+          status: 200,
+          headers: [{"content-length", "1000000"}, {"content-type", "application/pdf"}]
+        }
+
+      %{method: :head} ->
+        %Tesla.Env{status: 404, body: "", headers: []}
    end)

    :ok
@ -144,4 +165,12 @@ defmodule Pleroma.Web.RichMedia.ParserTest do
  test "returns error if getting page was not successful" do
    assert {:error, :overload} = Parser.parse("http://example.com/error")
  end
+
+  test "does a HEAD request to check if the body is too large" do
+    assert {:error, :body_too_large} = Parser.parse("http://example.com/huge-page")
+  end
+
+  test "does a HEAD request to check if the body is html" do
+    assert {:error, {:content_type, _}} = Parser.parse("http://example.com/pdf-file")
+  end
 end
				`@ -0,0 +1 @@`
				{"@context":["https:\/\/www.w3.org\/ns\/activitystreams"],"type":"Create","actor":"https:\/\/wedistribute.org\/wp-json\/pterotype\/v1\/actor\/-blog","object":{"@context":["https:\/\/www.w3.org\/ns\/activitystreams"],"type":"Article","name":"The end is near: Mastodon plans to drop OStatus support","content":"<!-- wp:paragraph {\"dropCap\":true} -->\n<p class=\"has-drop-cap\">The days of OStatus are numbered. The venerable protocol has served as a glue between many different types of servers since the early days of the Fediverse, connecting StatusNet (now GNU Social) to Friendica, Hubzilla, Mastodon, and Pleroma.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>Now that many fediverse platforms support ActivityPub as a successor protocol, Mastodon appears to be drawing a line in the sand. In <a href=\"https:\/\/www.patreon.com\/posts\/mastodon-2-9-and-28121681\">a Patreon update<\/a>, Eugen Rochko writes:<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:quote -->\n<blockquote class=\"wp-block-quote\"><p>...OStatus...has overstayed its welcome in the code...and now that most of the network uses ActivityPub, it's time for it to go. <\/p><cite>Eugen Rochko, Mastodon creator<\/cite><\/blockquote>\n<!-- \/wp:quote -->\n\n<!-- wp:paragraph -->\n<p>The <a href=\"https:\/\/github.com\/tootsuite\/mastodon\/pull\/11205\">pull request<\/a> to remove Pubsubhubbub and Salmon, two of the main components of OStatus, has already been merged into Mastodon's master branch.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>Some projects will be left in the dark as a side effect of this. GNU Social and PostActiv, for example, both only communicate using OStatus. While <a href=\"https:\/\/mastodon.social\/@dansup\/102076573310057902\">some discussion<\/a> exists regarding adopting ActivityPub for GNU Social, and <a href=\"https:\/\/notabug.org\/diogo\/gnu-social\/src\/activitypub\/plugins\/ActivityPub\">a plugin is in development<\/a>, it hasn't been formally adopted yet. We just hope that the <a href=\"https:\/\/status.fsf.org\/main\/public\">Free Software Foundation's instance<\/a> gets updated in time!<\/p>\n<!-- \/wp:paragraph -->","summary":"One of the largest platforms in the federated social web is dropping the protocol that it started with.","attributedTo":"https:\/\/wedistribute.org\/wp-json\/pterotype\/v1\/actor\/-blog","url":"https:\/\/wedistribute.org\/2019\/07\/mastodon-drops-ostatus\/","to":["https:\/\/www.w3.org\/ns\/activitystreams#Public","https:\/\/wedistribute.org\/wp-json\/pterotype\/v1\/actor\/-blog\/followers"],"id":"https:\/\/wedistribute.org\/wp-json\/pterotype\/v1\/object\/85810","likes":"https:\/\/wedistribute.org\/wp-json\/pterotype\/v1\/object\/85810\/likes","shares":"https:\/\/wedistribute.org\/wp-json\/pterotype\/v1\/object\/85810\/shares"},"to":["https:\/\/www.w3.org\/ns\/activitystreams#Public","https:\/\/wedistribute.org\/wp-json\/pterotype\/v1\/actor\/-blog\/followers"],"id":"https:\/\/wedistribute.org\/wp-json\/pterotype\/v1\/object\/85809"}