Compare commits

...

613 Commits

Author SHA1 Message Date
Stypox fbe9e6223a
Merge pull request #1168 from AudricV/yt_upd-cver-rm-keys-and-do-fixes
[YouTube] Update clients versions, restore access to some streams and more
2024-04-20 11:54:24 +02:00
Stypox 4e9e7cb29c
Improve assertTabsContain() to also check size 2024-04-20 11:48:36 +02:00
Stypox 9d0dd36034
[YouTube] Create constants for client names/versions 2024-04-20 11:43:54 +02:00
Stypox d4e6d22e64
[YouTube] Improve meta info code for review 2024-04-20 11:43:08 +02:00
AudricV 74bf000473
[YouTube] Update mocks 2024-04-11 18:59:11 +02:00
AudricV f9792cf3a9
[YouTube] Fix InteractiveTabbedHeader.testTabs test
YouTube now returns a Shorts tab for InteractiveTabbedHeader gaming channels,
which contains Shorts about the game of the topic channel but are not uploaded
on the game topic channel.

As this tab is already supported by the extractor, fetching a gaming topic
channel now returns a tab instead of none.

Channel name and channel URL of these Shorts needs to be set to null in a
separate commit, as Shorts on this tab do not have the topic channel as their
uploader.
2024-04-11 18:59:10 +02:00
AudricV f40fc0aa4f
[YouTube] Add support for new crisis meta info action data
The new action data can return multiple contact actions instead of only one,
which will be concatenated by a new line return.

This commit fixes tests of the CrisisResources test class of
YoutubeSearchExtractorTest.
2024-04-11 18:59:09 +02:00
AudricV 2a3c6f80d2
[YouTube] Fix YoutubeStreamExtractorRelatedMixTest 2024-04-10 21:19:03 +02:00
AudricV 657b4377aa
[YouTube] Fix YoutubeStreamExtractorDefaultTest tests 2024-04-10 21:19:03 +02:00
AudricV 7bf50bf1cb
[YouTube] Update Android client player parameters
YouTube disabled the effectiveness of the parameters which were used (the
player response we get redirects to another video), but new parameters which
work around Android's client integrity checks have been found.
2024-04-10 21:19:03 +02:00
AudricV 27dc1b1f50
[YouTube] Remove usage of API keys for InnerTube requests, bump versions
The API keys are not used anymore by official clients in almost all cases
(still used by the Android app until it gets a configuration) for all requests
we made.

Clients and device OS versions have been bumped to their latest stable version
known.

Methods and fields related to API keys have been renamed or deleted if they're
no longer relevant.
2024-04-10 21:19:02 +02:00
AudricV e380bb4bc3
[YouTube] Add missing prettyPrint query parameter to mixes continuations 2024-04-10 19:06:36 +02:00
Audric V 6c3c2e25d7
Merge pull request #1163 from AudricV/yt-fix_comments_extraction
[YouTube] Support new comments data
2024-04-10 18:19:59 +02:00
Stypox 02274d5395
[YouTube] Avoid XSS attacks in description or comments 2024-04-08 11:21:31 +02:00
Stypox 3f7b2653e3
[YouTube] Add YoutubeDescriptionHelperTest 2024-04-08 11:21:31 +02:00
Stypox a90237816a
[YouTube] Cleanup description helper
Remove unneeded isClose field, and make constants private
2024-04-08 11:21:31 +02:00
Stypox b80c3f5d51
[YouTube] Replace link text with accessibility label 2024-04-08 00:14:28 +02:00
Stypox 09732d6785
[YouTube] Add support for styles in attributed descriptions
Also refactor descriptions parsing.
2024-04-04 21:14:27 +02:00
AudricV 293c3e9e47
[YouTube] Support new A/B tested comments data
Also improve current comments code by removing outdated comment
renderer data.
2024-04-04 21:14:26 +02:00
Stypox e5b30ae8c3
Merge pull request #1151 from Profpatsch/localization-return-optional
LocaleCompat.forLanguageTag: return Optional if parsing fails
2024-03-29 13:50:48 +01:00
Stypox 23fc7aa209
Throw ParsingException instead of IllegalArg 2024-03-29 13:44:42 +01:00
Stypox fb468a23f4
Merge pull request #1142 from TeamNewPipe/peertube-v6
[PeerTube] Add support for PeerTube v6 features
2024-03-29 12:25:38 +01:00
Stypox 6589e2c15d
Merge pull request #1148 from Stypox/mediaccc-channel-tab-handler
[MediaCCC] Allow obtaining channel tab link handler
2024-03-28 13:45:05 +01:00
Tobi ad71864b23
Merge pull request #1162 from Stypox/fix-comment-description-npe
Make getCommentText @Nonnull
2024-03-27 20:04:17 +01:00
Stypox c57016b79b
Make getCommentText @Nonnull 2024-03-27 15:26:06 +01:00
Tobi adcc1f17ee
Merge pull request #1160 from TeamNewPipe/fix/tests
Fix some failing unit tests and detect new account termination messages for YouTube
2024-03-20 15:17:55 +01:00
TobiGr 51ddacc81d [SoundCloud] Fix SoundcloudSearchExtractorTest.NoNextPage
Search did not return no item at all, causing a NothingFoundException. New search query yields three items on first page
2024-03-20 15:10:39 +01:00
TobiGr 8392d50ba6 Update mocks for YoutubeChannelExtractorTest.NotAvailable 2024-03-20 14:59:44 +01:00
TobiGr aaccfecda8 [YouTube] Detect new account termination messages 2024-03-20 14:57:41 +01:00
TobiGr 73f0c63a9d [PeerTube] Fix tests for "What is PeerTube?" video 2024-03-20 14:44:06 +01:00
Tobi 896a55e319
Merge pull request #1139 from TeamNewPipe/dependabot/github_actions/actions/upload-artifact-4
Bump actions/upload-artifact from 3 to 4
2024-03-18 08:59:35 +01:00
dependabot[bot] e58fc652e0
Bump actions/upload-artifact from 3 to 4
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3 to 4.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-03-18 07:53:24 +00:00
Tobi e3f2c9aec7
Merge pull request #1153 from TeamNewPipe/dependabot/github_actions/actions/cache-4
Bump actions/cache from 3 to 4
2024-03-18 08:51:24 +01:00
Tobi 6b0fc14c04
Merge pull request #1156 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.10.2
Bump org.junit:junit-bom from 5.10.0 to 5.10.2
2024-03-18 08:46:13 +01:00
dependabot[bot] d579b608e5
Bump org.junit:junit-bom from 5.10.0 to 5.10.2
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.10.0 to 5.10.2.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.10.0...r5.10.2)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-02-05 09:15:26 +00:00
TobiGr fe47a4311f [PeerTube] Add test for segments and framesets 2024-01-29 10:22:06 +01:00
TobiGr 15e0e74b48 [PeerTube] Add support for stream frames/storyboards extraction
Implement PeerTubeStreamExtractor.getFrames()
2024-01-29 10:22:06 +01:00
dependabot[bot] da04eded5d
Bump actions/cache from 3 to 4
Bumps [actions/cache](https://github.com/actions/cache) from 3 to 4.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-01-18 09:32:42 +00:00
Profpatsch 7408173246 LocaleCompat.forLanguageTag: return Optional if parsing fails
It’s not obvious that the function will fail in some cases and throw
an `IllegalArgumentException`.

So instead of just failing if parsing fails, return an Optional that
all callers have to decide what to do (e.g. the YoutubeExtractor can
just ignore the locale in that case, like it does with most other
fields in the json if they are unexpected).
2024-01-07 14:31:34 +01:00
Stypox aaf3231fc7
[MediaCCC] Fix lambda link handler keeping reference to extractor
This caused problems in NewPipe, because extractors are not serializable, and well, keeping references to them is a bad idea anyway.
2023-12-30 23:23:19 +01:00
Stypox 137e924035
[MediaCCC] Add ChannelTabExtractorTest 2023-12-30 22:53:51 +01:00
Stypox cc9ade962e
[MediaCCC] Allow obtaining channel tab extractor from scratch
i.e. without needing to pass through the conference/channel extractor
This was needed because clients (like NewPipe) might rely on link handlers to hold as little data as possible, since they might be kept around for long or passed around in system transactions, so this commit allows obtaining a standalone link handler that does not hold a JsonObject within itself.
2023-12-30 22:53:27 +01:00
Stypox 3402cdb666
Merge pull request #1147 from petlyh/youtube-albums
[YouTube] Add Albums channel tab
2023-12-30 21:58:19 +01:00
petlyh 6dc25f7b97
[YouTube] Add Albums channel tab mocks 2023-12-30 14:46:39 +01:00
petlyh 4408e2d0ac
[YouTube] Add Albums channel tab 2023-12-30 14:01:30 +01:00
TobiGr 9ab932e394 Rename testDoNotAcceptNonURLs() -> assertDoNotAcceptNonURLs() 2023-12-29 16:38:11 +01:00
Stypox 9d66debf3c
Merge pull request #1132 from TeamNewPipe/dependabot/github_actions/actions/setup-java-4
Bump actions/setup-java from 3 to 4
2023-12-29 16:20:46 +01:00
Tobi 038ebdedc4
Merge pull request #1143 from petlyh/peertube-non-urls
Avoid PeerTube accepting non-URLs
2023-12-29 12:47:30 +01:00
TobiGr 61d237de02 [PeerTube] Test onAccept(String URL) in LinkHandlerFactories for non-URLs 2023-12-29 12:45:02 +01:00
petlyh 2b2c1546d1 Avoid PeerTube accepting non-URLs 2023-12-29 12:27:39 +01:00
Tobi 1e93b1dc20
Merge pull request #1135 from Stypox/yt-emergency-info
[YouTube] Implement emergency meta info
2023-12-29 12:01:40 +01:00
dependabot[bot] 3400af99b3
Bump actions/setup-java from 3 to 4
Bumps [actions/setup-java](https://github.com/actions/setup-java) from 3 to 4.
- [Release notes](https://github.com/actions/setup-java/releases)
- [Commits](https://github.com/actions/setup-java/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/setup-java
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-29 10:57:47 +00:00
Tobi 1f8a044462
Merge pull request #1138 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.8.3
Bump com.github.spotbugs:spotbugs-annotations from 4.8.0 to 4.8.3
2023-12-29 11:56:18 +01:00
dependabot[bot] 1470aa7303
Bump com.github.spotbugs:spotbugs-annotations from 4.8.0 to 4.8.3
Bumps [com.github.spotbugs:spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.8.0 to 4.8.3.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.8.0...4.8.3)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-29 10:53:28 +00:00
TobiGr 8f9ebdcb77 [PeerTube] Fix failing PeertubeTrendingLinkHandlerFactoryTest
The factory was updated in #1144
2023-12-29 11:52:19 +01:00
Stypox 1553931027
Merge pull request #1145 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.17.2
Bump org.jsoup:jsoup from 1.16.2 to 1.17.2
2023-12-29 11:27:01 +01:00
Stypox b2ec1b15fb
Merge pull request #1144 from dragfyre/patch-1
Update PeertubeTrendingLinkHandlerFactory.java
2023-12-29 11:21:08 +01:00
dependabot[bot] 151ee99da3
Bump org.jsoup:jsoup from 1.16.2 to 1.17.2
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup) from 1.16.2 to 1.17.2.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES.md)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.16.2...jsoup-1.17.2)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-29 09:47:00 +00:00
dragfyre 65e7bc5b95
Update PeertubeTrendingLinkHandlerFactory.java
correcting Peertube local trending api URL (per #10685 in main NewPipe repo); see https://docs.joinpeertube.org/api-rest-reference.html#tag/Video/operation/getVideos
2023-12-28 14:50:31 +07:00
Stypox f276caf54a
Release v0.23.1 2023-12-21 21:59:21 +01:00
Stypox fc54fb2fdb
Merge pull request #1140 from Stypox/yt-shorts-no-duration
[YouTube] Always return -1 as duration of Shorts returned inside reel items
2023-12-21 21:52:40 +01:00
Stypox 0518487d26
Fix SearchInfo's non-null MetaInfo being null when initialized or when an extraction error occurs (#1141)
The meta info might have been null either when SearchInfo is first initialized, or when extractor.getMetaInfo() throws an exception in getInfo().
This caused NewPipe to crash instead of showing a nice error in https://www.reddit.com/r/youtube/comments/184ttmw/what_exactly_about_blue_whales_has_youtube_so/.
2023-12-21 21:39:25 +01:00
Stypox 5b59a1a8c5
[YouTube] Move meta info extraction to separate file
YoutubeParsingHelper was longer than 2000 lines which caused checkstyle issues
2023-12-21 21:19:08 +01:00
Stypox b8e12dd76c
[YouTube] Implement emergency meta info
YouTube provides that meta info panel when users search for really sensitive content like suicide (e.g. "blue whale").

It contains:
- an encouragement as title (e.g. "We are with you")
- a phone number as action
- details about how to call the phone number (e.g. availability)
- an url pointing to the website of an association

Also add a test that just checks if a meta info is properly extracted
2023-12-21 21:19:08 +01:00
Stypox 83c1737f70
[YouTube] Update shorts channel tab mocks 2023-12-21 20:54:16 +01:00
Stypox 2938067c2c
[YouTube] Shorts don't provide a duration anymore 2023-12-21 20:41:01 +01:00
Stypox 91419ec6e8
Release v0.23.0 2023-12-10 15:59:32 +01:00
Audric V 678c98f24c
Merge pull request #1127 from AudricV/yt_improvements-and-fixes
[YouTube] Make some improvements and fixes
2023-12-09 14:07:31 +01:00
AudricV ec0194cfbf
[YouTube] Update mocks 2023-12-08 21:46:52 +01:00
AudricV 00a0f1a103
[YouTube] Add a blocking crisis resources bypass in search results test
This test only tests that search results are returned, when no content filters
are provided and crisis resources blocking search results should be returned.

Searches with blocking crisis resources and content filters should work too, as
the bypass has been implemented for them.
2023-12-08 21:46:49 +01:00
AudricV 06838d7245
[YouTube] Fix YoutubeSearchQHTest tests
As search parameters to bypass crisis resources blocking search results have
been implemented, they need to be added to search tests, in order to pass
them.
2023-12-08 21:46:49 +01:00
AudricV 642bb01388
[YouTube] Fix testSubscriberCount method of YoutubeChannelExtractorTest.CarouselHeader
The subscriber count is now lower than the expected count as some people
unsubscribed to the Sports system channel. The expected count has been so
lowered.
2023-12-08 21:46:49 +01:00
AudricV bedc9e5bc0
[YouTube] Remove Channels channel tab in tests 2023-12-08 21:46:48 +01:00
AudricV 5fa22ae25b
[YouTube] Enable and change testRelatedItems method of YoutubePlaylistExtractorTest.LearningPlaylist
This change makes the method test whether there is no more items instead of
being disabled.
2023-12-08 21:46:48 +01:00
AudricV 29dc7625f2
[YouTube] Fix YoutubeSearchExtractorTest.Suggestion
YouTube doesn't return anymore a suggestion for the query "algorythm", but does
for the query "on board ing" ("on boarding"). This search query is now used and
had to be URL-encoded.

URL encoding in the complete YoutubeSearchExtractorTest test class uses now
extractor's Util class instead of Java's URLDecoder class directly.
2023-12-08 21:46:48 +01:00
AudricV 56ab35423e
[YouTube] Fix potential NullPointerException in YoutubeSearchExtractor.getSearchSuggestion 2023-12-08 21:46:48 +01:00
AudricV 9dc1eab28c
[YouTube] Fix expected upload dates of StreamExtractor tests
YouTube is rolling out or A/B testing a new date format returned inside player
responses, which are precise to the second instead of the day.

This commit makes the StreamExtractor tests use these more precise dates.
2023-12-08 21:46:48 +01:00
AudricV ad3d187ac7
[YouTube] Fix testRelatedItems method of YoutubeStreamExtractorRelatedMixTest
This commit fixes the testRelatedItems test method by:
- accepting consent in the test class, in order to extract mixes in
recommendations;
- removing assertion of a music mix inside the recommendations, as YouTube
doesn't seem to return such mixes anymore, at least for the video used in the
test class.
2023-12-08 21:46:47 +01:00
AudricV e111814401
[YouTube] Fix YoutubeStreamExtractorDefaultTest.RatingsDisabledTest
Replace the video used in this test class with another one publicly available
and update the corresponding expected test values.

The test class's mocks will be updated in a different commit.
2023-12-08 21:46:47 +01:00
AudricV fc45941ead
[YouTube] Fix YoutubeChannelExtractorTest tests
- Change CarouselHeader test channel to Sports system one, as the Coachella one
doesn't return this channel header anymore;
- Fix InteractiveTabbedHeader test by checking whether the test's channel
description is not empty instead of containing some words, as it is changing
frequently.
2023-12-08 21:46:47 +01:00
AudricV 0bcb241c38
[YouTube] Fix expected data in YoutubeStreamExtractorRelatedMixTest
Video's title and tags have been changed by its uploader, so they have to be
updated.

Also make some package-private constants private, as they are not used outside
of the class, and remove unneeded test overrides.
2023-12-08 21:46:47 +01:00
AudricV 6ba8251be1
[YouTube] Bypass crisis resources blocking search results
These crisis resources are preventing search results to be returned. See
https://support.google.com/youtube/answer/10726080?hl=en for more info on them.

This commit changes search parameters to include the property allowing to show
search results.
2023-12-08 21:46:47 +01:00
AudricV 7dea2d0d27
[YouTube] Remove Channels channel tab support
This tab has been removed by YouTube.
2023-12-08 21:46:47 +01:00
AudricV 3782d9a02a
[YouTube] Support new A/B tested like data and avoid like count conversion from integer to long
Also make minor improvements to current like data extraction and remove
previous like count data support, as it is not returned anymore.
2023-12-08 21:46:46 +01:00
AudricV b71ce1123f
[YouTube] Extract only search results corresponding to a search type
YouTube returns sometimes videos inside channel search results. As we only want
results corresponding to the type we requested, this commits makes
YoutubeSearchExtractor ignoring non-requested search results we get, using the
extractor LinkHandler's first content filter value.

Also remove an unneeded exception throwing declaration in
YoutubeSearchExtractor.
2023-12-08 21:46:46 +01:00
AudricV ff8ed7247f
[YouTube] Switch to new consent cookie
Also move the documentation of the consent in its setter method in order to be
accessible publicly and improve it.
2023-12-08 21:46:46 +01:00
AudricV ec838d7421
[YouTube] Add missing prettyPrint query parameter to some test InnerTube requests
This query parameter for which its value is set to false was not added to two
requests made in test classes of YoutubeMixPlaylistExtractorTest.

Also remove an unneeded ParsingException exception throwing declaration in a
test method.
2023-12-08 21:46:46 +01:00
AudricV 2c941794c0
[YouTube] Add utcOffsetMinutes to all InnerTube payloads
This should make returned dates consistent between timezones and countries on
which the extractor is ran.

It was previously only set on YouTube Music search continuations.
2023-12-08 21:46:46 +01:00
AudricV d97c9e0db1
[YouTube] Improve payloads and URLs of InnerTube requests
For every InnerTube request:
- Always add a `request` object with the following properties:
  - "internalExperimentFlags" set to an empty array;
  - "useSsl" set to "true";
  - "lockedSafetyMode" set to "false".
- Use proper TODO comment to provide a way to enable restricted mode on every
request and add it on requests on which it wasn't present.

For YouTube Music:
- Remove alt query parameter, as it is not used anymore by the website;
- Add prettyPrint query parameter with false value on YouTube Music search
continuations.
2023-12-08 21:46:45 +01:00
AudricV 8a9ebcc373
[YouTube] Update InnerTube clients' version and devices' OS version and model 2023-12-08 21:46:45 +01:00
Tobi eac850ca10
Merge pull request #1114 from FineFindus/feat/comment-author-is-owner
[YouTube] Add channel owner to comments
2023-10-25 09:51:12 +02:00
Tobi 5ab1f784e8
Merge pull request #1117 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.16.2
Bump org.jsoup:jsoup from 1.16.1 to 1.16.2
2023-10-21 19:26:36 +02:00
dependabot[bot] 9d7bcba050
Bump org.jsoup:jsoup from 1.16.1 to 1.16.2
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup) from 1.16.1 to 1.16.2.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.16.1...jsoup-1.16.2)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-20 09:13:21 +00:00
dependabot[bot] e26065148a Bump com.github.spotbugs:spotbugs-annotations from 4.7.3 to 4.8.0
Bumps [com.github.spotbugs:spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.7.3 to 4.8.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.3...4.8.0)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-12 14:19:10 +02:00
FineFindus 34b05a0dda
feat(youtube/comments): support creator replies 2023-10-09 16:33:43 +02:00
TobiGr 0821f09114
Add missing mocks 2023-10-09 16:33:43 +02:00
FineFindus c1784a4bdb
[YouTube] Add channel owner to comments 2023-10-09 16:33:43 +02:00
TobiGr f9846352ea Fix wrong `@Nullable` annotation 2023-10-09 16:02:57 +02:00
Tobi d6f5cba6e2
Merge pull request #1111 from FineFindus/feat/creator-reply
Add `hasCreatorReply()` to CommentsInfoItem
2023-10-09 12:45:56 +02:00
TobiGr 9d63c75623 Add missing mocks 2023-10-09 11:24:39 +02:00
TobiGr d49f8411d7 [PeerTube] Implement CommentsInfoItemExtractor.hasCreatorReply() 2023-10-09 02:47:12 +02:00
Stypox bb132167d5
Merge pull request #1113 from AudricV/snd_fix-non-jpg-images
[SoundCloud] Fix extraction of non-JPG images
2023-10-02 19:40:57 +02:00
AudricV c98695fcea
[SoundCloud] Fix extraction of non-JPG images
Default image qualities were removed in image URLs with the jpg extension,
causing the addition of the image suffix to full non-JPG images URLs and so to
invalid image URLs.

Only the image quality name with its leading "-" character and the "."
character after the name is now removed and replaced by a string format
replaced itself with the image quality name for each quality.

As the image suffixes do not contain the image extension, the name of image
qualities lists has been adapted with these changes and some related comments
have been also improved.
2023-10-01 20:33:25 +02:00
AudricV ac00459c1a
Change requirement of image extensions in ImageSuffix class' Javadoc to a possibility
Some services may provide different image formats using the same suffix,
without we know what format the service provide. Enforcing an image extension
could so lead to provide invalid image URLs, like for SoundCloud PNG images
currently.

With this documentation change, it is now clear that users of this class decide
of whether they want to include image extensions in the suffix. The previous
behavior described in the Javadoc was not enforced.
2023-09-30 21:11:09 +02:00
FineFindus dd7b2d9798
feat(youtube/comments): support creator replies 2023-09-25 10:40:45 +02:00
Youssif Shaaban Alsager 917554acc4
[YouTube] Add support for ultralow audio formats (#1063) 2023-09-24 19:04:34 +02:00
Tobi 8b0068f8f4
Merge pull request #1110 from christian-2hu/chore/update-copyright
chore: Update copyright notices
2023-09-23 00:29:19 +02:00
Christian fc67d49f59 Update copyright notices
Update copyright notices to comply to GPLv3 and change NewPipe to NewPipe Extractor on some notices that were not updated.
2023-09-22 19:10:15 -03:00
Stypox 289db1178a
Merge pull request #1108 from AudricV/yt_refactor-js-usage
[YouTube] Refactor JavaScript usage and fix extraction of obfuscated signature deobfuscation function
2023-09-22 10:41:57 +02:00
AudricV 6ed22099a2
[YouTube] Update stream mocks 2023-09-21 21:59:34 +02:00
AudricV 714b141ecb
[YouTube] Catch any exception when extracting something from JavaScript's base player 2023-09-21 21:59:33 +02:00
AudricV 588c6a8422
[YouTube] Quote signature deobfuscation function name and add semicolon only where needed 2023-09-21 21:59:33 +02:00
AudricV 1fa85ec6ca
[YouTube] Add tests for signature timestamp extraction and signature deobfuscation function extraction and execution 2023-09-21 21:59:33 +02:00
AudricV a04bc320de
[YouTube] Convert signature timestamp to integer
The signature timestamp is used as a number by HTML5 clients, so it should be
used in the same way by the extractor too instead of being a string.

As the timestamp doesn't seem to exceed 5 digits, an integer is used to store
its value.
2023-09-21 21:59:32 +02:00
AudricV 7de3753a81
[YouTube] Refactor JavaScript player management API
This commit is introducing breaking changes.

For clients, everything is managed in a new class called
YoutubeJavaScriptPlayerManager:
- caching JavaScript base player code and its extracted code (functions and
variables);
- getting player signature timestamp;
- getting deobfuscated signatures of streaming URLs;
- getting streaming URLs with a throttling parameter deobfuscated, if
applicable.

The class delegates the extraction parts to external package-private classes:
- YoutubeJavaScriptExtractor, to extract and download YouTube's JavaScript base
player code: it always already present before and has been edited to mainly
remove the previous caching system and made it package-private;
- YoutubeSignatureUtils, for player signature timestamp and signature
deobfuscation function of streaming URLs, added in a recent commit;
- YoutubeThrottlingParameterUtils, which was originally
YoutubeThrottlingDecrypter, for throttling parameter of streaming URLs
deobfuscation function and checking whether this parameter is in a streaming
URL.

YoutubeJavaScriptPlayerManager caches and then runs the extracted code if it
has been executed successfully. The cache system of throttling parameters
deobfuscated values has been kept, its size can be get using the
getThrottlingParametersCacheSize method and can be cleared independently using
the clearThrottlingParametersCache method.

If an exception occurs during the extraction or the parsing of a function
property which is not related to JavaScript base player code fetching, it is
stored until caches are cleared, making subsequent failing extraction calls of
the requested function or property faster and consuming less resources, as the
result should be the same until the base player code changes.

All caches can be reset using the clearAllCaches method of
YoutubeJavaScriptPlayerManager.

Classes using JavaScript base player code and utilities directly (in the code
and its tests) have been also updated in this commit.
2023-09-21 21:59:32 +02:00
AudricV 6884d191cd
[YouTube] Add utility class around signatures and fix signature deobfuscation function extraction
The goal of this class is to decouple the extraction of signature timestamp and
signature deobfuscation function from YoutubeStreamExtractor.

The extraction of the signature deobfuscation function has been also adapted to
support the latest YouTube player versions.

This new class, YoutubeSignatureUtils, doens't store anything temporary such as
a copy of the player code, which has to be passed where required. It is not
public, as it will be used by a JavaScript player manager class in the future,
in order to handle in a better way fetching, caching and resetting cache of the
player code.
2023-09-21 21:59:26 +02:00
Tobi 3be76a6406
Merge pull request #1107 from Isira-Seneviratne/Locale_forLanguageTag
Use Locale.forLanguageTag() in tests
2023-09-18 16:48:30 +02:00
TobiGr 17790328cd Improve doc 2023-09-18 16:44:51 +02:00
Isira Seneviratne 4bc8ae7812 Use Locale.forLanguageTag() in tests 2023-09-18 08:59:13 +05:30
Tobi 90aed06a63
Merge pull request #1105 from TeamNewPipe/fix/bandcamp-streame-extractor-test
[Badcamp] Fix StreamExtractorTest
2023-09-18 01:49:04 +02:00
TobiGr cf49f4a31c [Badcamp] Fix StreamExtractorTest
The song was renamed and the URL changed
2023-09-17 23:58:07 +02:00
Tobi 7c7ceaceab
Merge pull request #1103 from TeamNewPipe/dependabot/github_actions/actions/checkout-4
Bump actions/checkout from 3 to 4
2023-09-17 22:42:01 +02:00
dependabot[bot] 72c475d944
Bump actions/checkout from 3 to 4
Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-09-05 10:00:13 +00:00
Stypox 1f08d28ae5
Merge pull request #889 from AudricV/multiple-images-support
Multiple images support
2023-08-13 11:35:11 +02:00
AudricV e8bfd20170
[MediaCCC] Apply changes in extractor tests
Also remove some public test methods modifiers.
2023-08-12 22:56:33 +02:00
AudricV 0292c4f3e8
[Bandcamp] Apply changes in extractor tests
Also remove some public test methods modifiers, add missing Test annotations on
old Junit 4 tests (and update them if needed), and use final in some places
where it was possible.

BandcampChannelExtractorTest.testLength has been removed as the test is always
true.
2023-08-12 22:56:32 +02:00
AudricV 2578f22054
[Bandcamp] Add utility test method to test images
This method, testImages(Collection<Image>), will use first the default image
collection test in DefaultTests and then will check that each image URL
contains f4.bcbits.com/img and ends with .jpg or .png.

To do so, a new non-instantiable final class has been added: BandcampTestUtils.
2023-08-12 22:56:32 +02:00
AudricV ba5315c72d
[PeerTube] Apply changes in extractor tests
Also remove some public test methods modifiers, add missing Test annotations on
old Junit 4 tests (and update them if needed), and improve some code.
2023-08-12 22:56:32 +02:00
AudricV 1d72bac53d
[SoundCloud] Apply changes in extractor tests 2023-08-12 22:56:32 +02:00
AudricV 93a210394d
[YouTube] Apply changes in extractor tests
Also remove some public test methods modifiers, add missing Test annotations on
old Junit 4 tests (and update them if needed), and use final in some places
where it was possible.
2023-08-12 22:56:31 +02:00
AudricV 2c436d428c
[YouTube] Add utility test method to test images in YoutubeTestsUtils
This method, testImages(Collection<Image>), will use first the default image
collection test in DefaultTests and then will check that each image URL
contains the string yt.

The JavaDoc of the class has been also updated to reflect the changes made in
it (it is now more general).
2023-08-12 22:56:31 +02:00
AudricV d381f3b70b
Update avatar, banners and thumbnail methods' name and apply changes in DefaultStreamExtractorTest 2023-08-12 22:56:31 +02:00
AudricV 434e885708
Add utility methods in ExtractorAsserts to check whether a collection is empty and to test image collections
Two new methods have been added in ExtractorAsserts to check if a collection is
empty:

- assertNotEmpty(String, Collection<?>), checking:
  - the non nullity of the collection;
  - its non emptiness (if that's not case, an exception will be thrown using
    the provided message).

- assertNotEmpty(Collection<?>), calling assertNotEmpty(String, Collection<?>)
  with null as the value of the string argument.

A new one has been added to this assertion class to check the contrary:
assertEmpty(Collection<?>), checking emptiness of the collection only if it is
not null.

Three new methods have been added in ExtractorAsserts as utility test methods
for image collections:

- assertContainsImageUrlInImageCollection(String, Collection<Image>), checking
that:
  - the provided URL and image collection are not null;
  - the image collection contains at least one image which has the provided
    string value as its URL (which is a string) property.

- assertContainsOnlyEquivalentImages(Collection<Image>, Collection<Image>),
  checking that:
  - both collections are not null;
  - they have the same size;
  - each image of the first collection has its equivalent in the second one.
    This means that the properties of an image in the first collection must be
    equal in an image of the second one.

- assertNotOnlyContainsEquivalentImages(Collection<Image>, Collection<Image>),
  checking that:
  - both collections are not null;
  - one of the following conditions is met:
    - they have different sizes;
    - an image of the first collection has not its equivalent in the second one.
      This means that the properties of an image in the first collection must
      be not equal in an image of the second one.

These methods will be used by services extractors tests (and default ones) to
test image collections.
2023-08-12 22:56:31 +02:00
AudricV 5158472852
Apply changes in DefaultTests and add utility method to test image lists
This new method, defaultTestImageList(List<Image), will check that the image
list is not null.

For each image, it will test that its URL is secure and its height and width
are more than or equal to their relevant unknown constants in the Image class
(HEIGHT_UNKNOWN and WIDTH_UNKNOWN).
2023-08-12 22:56:31 +02:00
AudricV 70fb3aa38e
Update BaseExtractorTests image methods' name
Also suppress unused warnings in BaseStreamExtractorTest, like it is done on
other BaseExtractorTests interfaces.
2023-08-12 22:56:30 +02:00
AudricV e16d521b7b
[MediaCCC] Apply changes in Extractors
Also remove usage of the conference logo as the banner of a conference, as it
is a logo and not a banner.
2023-08-12 22:56:30 +02:00
AudricV 306068a63b
[MediaCCC] Apply changes in InfoItemExtractors 2023-08-12 22:56:30 +02:00
AudricV 2f40861428
[MediaCCC] Add utility methods to get image lists from conference logos and streams
These three new methods, added in MediaCCCParsingHelper,
getImageListFromImageUrl(String), getThumbnailsFromStreamItem(JsonObject) and
getThumbnailsFromLiveStreamItem(JsonObject) (the last two are based on a common
method, getThumbnailsFromObject(JsonObject, String, String)), return an empty
list if the case no image URL could be extracted.

Images returned have their height and width unknown and a resolution level
depending on the image key of the JSON API response.
2023-08-12 22:56:30 +02:00
AudricV 71cda03c4c
[Bandcamp] Apply changes in Extractors 2023-08-12 22:56:29 +02:00
AudricV 7e01eaac33
[Bandcamp] Apply changes in InfoItemExtractors 2023-08-12 22:56:29 +02:00
AudricV 4b80d737a4
[Bandcamp] Add utility methods to get multiple images
Bandcamp images work with image IDs, which provide different resolutions.

Images on Bandcamp are not always squares, and some IDs respect aspect ratios
where some others not.

The extractor will only use the ones which preserve aspect ratio and will not
provide original images, for performance and size purposes.

Because of this aspect ratio preservation constraint, only one dimension will
be known at a time.

The image IDs with their respective dimension used are:

- 10: 1200w;
- 101: 90h;
- 170: 422h;
- 171: 646h;
- 20: 1024w;
- 200: 420h;
- 201: 280h;
- 202: 140h;
- 204: 360h;
- 205: 240h;
- 206: 180h;
- 207: 120h;
- 43: 100h;
- 44: 200h.

(Where w represents the width of the image and h the height of the image)

Note that these dimensions are theoretical because if the image size is less
than the dimensions of the image ID, it will be not upscaled but kept to its
original size.

All these resolutions are stored in a private static list of ThumbnailSuffixes
in BandcampExtractorHelper, in which the methods to get mutliple images have
been added:

- getImagesFromImageUrl(String): public method to get images from an image URL;
- getImagesFromImageId(long, boolean): public method to get images from an
  image ID;
- getImagesFromImageBaseUrl(String): private utility method to get images from
  the static list of ThumbnailSuffixes from a given image base URL, containing
  the path to the image, a "a" letter if it comes from an album, its ID and an
  underscore.

Some existing methods have been also edited:

- the documentation of getImageUrl(long, boolean) has been changed to reflect
  the Bandcamp images findings;
- getThumbnailUrlFromSearchResult has been renamed to
  getImagesFromSearchResult, and a documentation has been added to this method.

The method replaceHttpWithHttps of the Utils class has been also used in
BandcampExtractorHelper instead of doing manually what the method does.
2023-08-12 22:56:29 +02:00
AudricV 4e6fb368bc
[PeerTube] Apply changes in Extractors and remove usages of default avatar picture
The default avatar picture was used when no profile picture was found, but it
was removed and split in multiple images.

Thumbnails' size is not known, as this data is not provided by the API.
2023-08-12 22:56:29 +02:00
AudricV 0a6011a50e
[PeerTube] Apply changes in InfoItemExtractors
Also lower the visibility of attributes of channels and playlists InfoItems to
private.
2023-08-12 22:56:29 +02:00
AudricV 6f8331524b
[PeerTube] Add utility method to get thumbnails of playlists and videos
This method, getThumbnailsFromPlaylistOrVideoItem, has been added in
PeertubeParsingHelper and returns the two image variants for playlists and
videos.
2023-08-12 22:56:28 +02:00
AudricV 81c0d80a54
[PeerTube] Add utility methods to get avatars and banners of accounts and channels
Four new static methods have been added in PeertubeParsingHelper to do so:
- two public methods to get the corresponding image type:
  getAvatarsFromOwnerAccountOrVideoChannelObject(String, JsonObject) and
  getBannersFromAccountOrVideoChannelObject(String, JsonObject);
- two private methods as helper methods: getImagesFromAvatarsOrBanners(String,
  JsonObject, String, String) and getImagesFromAvatarOrBannerArray(String,
  JsonArray).
2023-08-12 22:56:28 +02:00
AudricV 31da5beb51
[SoundCloud] Apply changes in Extractors 2023-08-12 22:56:28 +02:00
AudricV a3a74cd566
[SoundCloud] Apply changes in InfoItemExtractors and return track user avatars as uploader avatars in SoundcloudStreamInfoItemExtractor 2023-08-12 22:56:28 +02:00
AudricV 7f818217d2
[SoundCloud] Add utility methods to get images from track JSON objects and image URLs
These new public and static methods, added in SoundcloudParsingHelper,
getAllImagesFromArtworkOrAvatarUrl(String) and
getAllImagesFromVisualUrl(String) (which call a common private method,
getAllImagesFromImageUrlReturned(String, List<ImageSuffix>, List<Image>)),
return an unmodifiable list of JPEG images containing almost every image
resolution provided by SoundCloud except the original size and the tiny
resolution (for artworks and avatars, as the image size is 20x20 for artworks
and 18x18 for avatars, so very close to or equal to the t20x20 resolution):

- for artworks and avatars:
  - mini: 16x16;
  - t20x20: 20x20;
  - small: 32x32;
  - badge: 47x47;
  - t50x50: 50x50;
  - t60x60: 60x60;
  - t67x67: 67x67;
  - large: 100x100;
  - t120x120: 120x120;
  - t200x200: 200x200;
  - t240x240: 240x240;
  - t250x250: 250x250;
  - t300x300: 300x300;
  - t500x500: 500x500.

- for visuals/user banners:
  - t1240x260: 1240x260;
  - t2480x520: 2480x520.

Duplicated code in two methods of SoundcloudParsingHelper
(getUsersFromApi(ChannelInfoItemsCollector, String) and
getStreamsFromApi(StreamInfoItemsCollector, String, boolean)) has been merged
into one common private method, getNextPageUrlFromResponseObject(JsonObject).
2023-08-12 22:56:28 +02:00
AudricV 266cd1f76b
[YouTube] Apply changes in YoutubeMusicSearchExtractor and split its InfoItemExtractors into separate classes
Splitting YoutubeMusicSearchExtractor's InfoItemExtractors into separate
classes (YoutubeMusicSongOrVideoInfoItemExtractor,
YoutubeMusicAlbumOrPlaylistInfoItemExtractor and
YoutubeMusicArtistInfoItemExtractor) allows to simplify
YoutubeMusicSearchExtractor,improves reading and applying changes to InfoItems
(no more losing at least quarter of a line due to indentations).

These InfoItems, in which the image changes have been applied, don't extend the
YouTube ones anymore, as most methods were overridden and the few ones that are
not don't apply in YouTube Music items responses, so it was useless to extend
them.

The code of YoutubeMusicSearchExtractor have been also improved a bit.
2023-08-12 22:56:27 +02:00
AudricV c1981ed54f
[YouTube] Apply changes in Extractors except YoutubeMusicSearchExtractor
Also improve a bit some code related to the changes.
2023-08-12 22:56:27 +02:00
AudricV 4cc99f9ce1
[YouTube] Apply changes in InfoItemExtractors except YouTube Music ones 2023-08-12 22:56:27 +02:00
AudricV adfad086ac
[YouTube] Add utility methods to get images from InfoItems and thumbnails arrays
Unmodifiable lists of Images are returned, parsed from a given YouTube
"thumbnails" JSON array.

These methods will be used in all YouTube extractors and InfoItems, as the
structures between content types (videos, channels, playlists, ...) are common.
2023-08-12 22:56:27 +02:00
AudricV d56b880cae
Replace avatar and thumbnail URLs attributes and methods to List<Image> in Infos 2023-08-12 22:56:26 +02:00
AudricV 9d8098576e
Replace avatar and thumbnail URLs attributes and methods to List<Image> in Extractors 2023-08-12 22:56:26 +02:00
AudricV 0f4a5a8184
Replace avatar and thumbnail URLs attributes and methods to List<Image> in InfoItemsCollectors 2023-08-12 22:56:26 +02:00
AudricV ca1d4a6fa4
Replace avatar and thumbnail URLs attributes and methods to List<Image> in InfoItemExtractors 2023-08-12 22:56:26 +02:00
AudricV 2f3ee8a3f2
Replace avatar and thumbnail URLs attributes and methods to List<Image> in InfoItems 2023-08-12 22:56:25 +02:00
AudricV 78ce65769f
Add an ImageSuffix class to the extractor
The goal of this utility class is to simply store suffixes which need to be
appended to image URLs, in order to get images at the suffix resolution.

This class contains four properties: the suffix (as a string), the height,
the width (as integers) and the estimated resolution level of the image
corresponding to the one represented by the suffix.
2023-08-12 22:56:25 +02:00
AudricV d85454186a
Add an Image class to the extractor
Objects of this serializable class contains four properties: a URL (as a
string), a width, a height (represented as integers) and an estimated
resolution level, which can be constructed from a given height.

Possible resolution levels are:
- UNKNOWN: for unknown heights or heights <= 0;
- LOW: for heights > 0 & < 175;
- MEDIUM: for heights >= 175 & < 720;
- HIGH: for heights >= 720.

Getters of these properties are available and the constructor needs these four
properties.
2023-08-12 22:56:25 +02:00
Stypox 7294675aea
Merge pull request #1093 from AudricV/yt_support-shorts-ui-playlists
[YouTube] Support Shorts UI in playlists
2023-08-12 11:11:36 +02:00
Stypox 93a90b816d
Merge pull request #1094 from AudricV/yt_support-more-channel-headers
[YouTube] Support more channel headers
2023-08-12 11:08:30 +02:00
Stypox 44b664af15
[YouTube] Simplify Optional chains in channel 2023-08-12 11:02:51 +02:00
AudricV 2f7bfd3e7f
[YouTube] Add mocks of interactiveTabbedHeaderRenderer channel header test 2023-08-08 19:12:29 +02:00
AudricV b147904571
[YouTube] Add test for interactiveTabbedHeaderRenderer channel header
This test uses the Minecraft game topic channel.
2023-08-08 19:12:28 +02:00
AudricV 1852031a0b
[YouTube] Support pageHeaderRenderer and interactiveTabbedHeaderRenderer channel headers
The addition of this support required to turn the isCarouselHeader boolean into
an enum containing all supported channel headers named HeaderType.

Also assert that the page has been fetched where needed to avoid
NullPointerExceptions when the channel page has been not fetched and remove the
getChannelHeaderJson method in YoutubeChannelExtractor, method for which its
code has been moved to its sole usage after the new headers support changes.
2023-08-08 19:12:27 +02:00
AudricV 698c710685
Do not require knowledge of uploader in default StreamInfoItems tests
This change is required as some services can return no uploader info, such as
YouTube for playlists with a Shorts UI.
2023-08-07 19:43:15 +02:00
AudricV 8237052ef5
Fix wrong assertion in assertNotEmpty
The non-null assertion was made on the exception message instead of the string
to check, causing a NullPointerException if the string to check was null.
2023-08-07 19:43:09 +02:00
AudricV 162c261577
[YouTube] Add mocks of the playlist with Shorts UI test 2023-08-07 19:07:53 +02:00
AudricV e2f4ee47b9
[YouTube] Add a playlist with Shorts UI test
The system Shorts videos uploads playlist of the YouTube official channel has
been chosen for this test.
2023-08-07 19:06:09 +02:00
AudricV e6f371fb94
[YouTube] Support Shorts UI in playlists
Also remove an outdated A/B test comment.
2023-08-07 19:01:08 +02:00
Stypox 7936987955
Merge pull request #1092 from Stypox/channel-tabs-improvements
Channel tabs code improvements
2023-08-06 21:44:10 +02:00
Stypox 6d2227111f
[YouTube] Assert that videos tab is ready after channel fetching 2023-08-06 21:14:57 +02:00
Stypox ee625c325c
Inherit from DefaultListExtractorTest in channel tab tests 2023-08-06 21:14:56 +02:00
Stypox 276c293889
Rename assertTabsContain 2023-08-06 21:14:56 +02:00
Stypox 9d3761a371
[YouTube] Directly use playlist collector in channel tabs wrapper
Note that this introduces a "Raw use of parameterized class 'InfoItemsPage'" warning, but it can be ignored since the type missing would be <InfoItem>, and StreamInfoItem extends InfoItem
2023-08-06 21:13:25 +02:00
Stypox 95a3cc0a17
Merge pull request #1083 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.10.0
Bump org.junit:junit-bom from 5.9.3 to 5.10.0
2023-08-06 18:58:27 +02:00
Stypox e34b4f1978
[YouTube] Avoid using Consumer 2023-08-06 13:02:31 +02:00
Stypox ef67c7cd74
[YouTube] Simplify usage of channel header json 2023-08-06 13:02:31 +02:00
Stypox a104cf3227
[YouTube] Fix docs in channel helper 2023-08-06 13:02:31 +02:00
Stypox bb47f05f89
Merge pull request #1091 from Stypox/update-mocks
[YouTube] Update stream mocks after #1087
2023-08-06 12:58:24 +02:00
Stypox 468bcc045d
[YouTube] Update mocks after #1087 2023-08-06 12:33:04 +02:00
Stypox 35f3a4ad01
Merge pull request #1082 from AudricV/channel-tabs-and-tags-support
Add support for channel tabs and channel tags
2023-08-06 12:21:42 +02:00
AudricV e7d64099a7
[YouTube] Update channel mocks and add channel tabs mocks 2023-08-06 12:15:06 +02:00
AudricV 684101c47d
[YouTube] Implement age-restricted channels support, link handlers and channels tabs and tags changes on tests
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV eaf2600ce0
[SoundCloud] Implement link handlers and channels tabs and tags changes on tests
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV 0ee2072de5
[PeerTube] Implement link handlers and channels tabs and tags changes on tests
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV d3801dd0e9
[MediaCCC] Implement link handlers and channels tabs and tags changes on tests
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV 8baec04611
[Bandcamp] Implement link handlers and channels tabs and tags changes on tests
Tests in BandcampChannelExtractorTest and BandcampChannelLinkHandlerFactoryTest
have been also fixed.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV e0ba29cd19
Add utility method to assert that given channel tabs are in the ones returned by a channel extractor
Only the first content filter of the ListLinkHandler instances provided is
used when collecting all channel tabs of the ListLinkHandler list, as channel
tabs implementations only use one content filter per ListLinkHandler instance.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV 18846baba7
Add tabs and tags methods in tests interfaces and annotate all methods with the Test JUnit annotation
These changes should help to detect tests as tests, when running a subset of
tests or all tests.
They should be also implemented in these interfaces' implementations (new and
existing ones).

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
ThetaDev c70a0e3543
Add a test for textual durations parsing using TimeAgoParser's patterns 2023-08-06 12:15:06 +02:00
AudricV 7366eab156
[YouTube] Add support for channel tabs and tags and age-restricted channels
Support of tags and videos, shorts, live, playlists and channels tabs has been
added for non-age restricted channels.

Age-restricted channels are now also supported and always returned the videos,
shorts and live tabs, accessible using system playlists. These tabs are the
only ones which can be accessed using YouTube's desktop website without being
logged-in.

The videos channel tab parameter has been updated to the one used by the
desktop website and when a channel extraction is fetched, this tab is returned
in the list of tabs as a cached one in the corresponding link handler.

Visitor data support per request has been added, as a valid visitor data is
required to fetch continuations with contents on the shorts tab. It is only
used in this case to enhance privacy.

A dedicated shorts UI elements (reelItemRenderers) extractor has been added,
YoutubeReelInfoItemExtractor. These elements do not provide the exact view
count, any uploader info (name, URL, avatar, verified status) and the upload
date.

All service's LinkHandlers are now using the singleton pattern and some code
has been also improved on the files changed.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:15:04 +02:00
AudricV 4586067934
Add utility method to parse textual durations using TimeAgoParser's patterns
This is required to parse duration of YouTube's reelItemRenderers, returned
only inside accessibility data.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:13:33 +02:00
AudricV d4bfe791ee
[SoundCloud] Add tabs support for users
Support of tracks, playlists and albums has been added for users.

Also add the declaration of the UnsupportedOperationException exception to the
service's LinkHandlers.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:13:32 +02:00
AudricV 6f7d1f079f
[Bandcamp] Add tabs support for artists
Support of tracks and albums has been added for artists.

Also use the singleton pattern and add the declaration of the
UnsupportedOperationException exception to the service's LinkHandlers and
improved some code in the files changed.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:12:19 +02:00
AudricV 1e8474b22d
[PeerTube] Add tabs support for accounts and video channels
Support of channels and videos has been added for accounts and support of
videos and playlists has been added for video channels.

The following changes have been also done:
- collectStreamsFrom method in PeertubeParsingHelper has been renamed to
collectItemsFrom;
- PeertubeChannelInfoItemExtractor.getStreamCount method has been fixed due to
ChannelExtractor's new inheritance;
- the declaration of the UnsupportedOperationException exception thrown has
been added to the service's LinkHandlers;
- a channel tab LinkHandlerFactory has been added,
PeertubeChannelTabLinkHandlerFactory;
- all service's LinkHandlers are now using properly the singleton pattern.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:12:15 +02:00
AudricV 652c2c8408
Add a ListLinkHandler which can be used to be returned from ChannelInfo.getTabs() when a specific tab's data has already been fetched
This new ListLinkHandler, ReadyChannelTabListLinkHandler, should help saving
clients data, energy and time by helping to reduce duplicate requests.

Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:11:12 +02:00
AudricV de823a6b68
Add an UnsupportedTabException exception class
This class makes easier for LinkHandlerFactory implementations to declare an
UnsupportedOperationException.
2023-08-06 12:11:12 +02:00
AudricV 76fb9dcdd7
Add UnsupportedOperationException to exceptions which can be thrown by getId and getUrl methods of LinkHandlerFactory and its base implementations
This change advertise to clients that channel tabs' link handler factories can
return an UnsupportedOperationException when a tab provided to them is
unsupported.
2023-08-06 12:11:12 +02:00
AudricV 946eb9bd91
Add structure of channel tags
Tags' getters and/or setters have been added in ChannelExtractor and
ChannelInfo to do so.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:11:12 +02:00
AudricV 356a888d6c
Add structure of channel tabs
This commit introduces the following breaking changes:

- Three new classes have been added:
  - ChannelTabExtractor, class extending ListExtractor<InfoItem>, which
  extracts InfoItems from a channel tab;
  - ChannelTabInfo extending ListInfo<InfoItem>, which extracts InfoItems from
  a ChannelTabExtractor and returns them as a ChannelTabInfo;
  - ChannelTabs, an immutable class containing all supported channel tabs.
- StreamingService implementations must implement new methods returning a
channel tab LinkHandlerFactory (getChannelTabsLHFactory) and a
ChannelTabExtractor (getChannelTabExtractor);
- ChannelExtractor inherits Extractor instead of ListExtractor<StreamInfoItem>
and ChannelInfo inherits Info instead of ListInfo<StreamInfoItem>;
- ChannelExtractor and ChannelInfo have now getters and/or setters of tabs.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:11:11 +02:00
Stypox 3faaf4301c
Merge pull request #1087 from AudricV/yt_js-extractor-improvements-and-fixes
[YouTube] Improve and fix YoutubeJavaScriptExtractor
2023-08-06 12:01:00 +02:00
Stypox 8fb6ba36fa
Merge pull request #1081 from TeamNewPipe/fix/sc/search-next-page
[SoundCloud] Detect whether there are any more search results
2023-08-06 11:49:35 +02:00
Stypox 2947257111
[SoundCloud] Properly calculate if results have finished 2023-08-06 11:38:22 +02:00
Stypox 485bfbca9d
[SoundCloud] Move try-catch inside getOffsetFromUrl 2023-08-06 11:35:37 +02:00
Stypox 7c70fef197
Merge pull request #1089 from TeamNewPipe/ccc
[media.ccc.de] Only extract kiosk live stream rooms if they are streaming
2023-08-06 10:12:04 +02:00
Stypox de0a9bb797
Merge pull request #1088 from FireMasterK/pseudo-rng
Replace cryptographically secure random with regular random
2023-08-05 11:37:49 +02:00
TobiGr 340095515d Make Kiosk IDs accessible if possible 2023-08-05 03:18:40 +02:00
TobiGr fe27d6a0ec [media.ccc.de] Only extract live streams if the conference is streaming 2023-08-05 01:53:43 +02:00
Kavin 25082d78b0
Replace SecureRandom with Random 2023-08-03 23:00:02 +01:00
TobiGr aa6c17dc77 [SoundCloud] Deduplicate some code 2023-08-03 14:41:30 +02:00
TobiGr 2fb9922a15 [SoundCloud] Detect whether there are any more search results
Add test for this edge case.
2023-08-03 14:37:13 +02:00
AudricV a3d160edab
[YouTube] Improve and fix YoutubeJavaScriptExtractor
- Enhance documentation;
- Fix the regular expression fallback on HTML embed watch page;
- Use HTML scripts tag search first instead of the regular expression approach,
now used as a last resort;
- Compile regular expressions only once, in order to improve the performance of
subsequent extraction calls when clearing the cache;
- Provide original exceptions when fetching or parsing pages on which the base
JavaScript's player could be found failed, allowing clients to detect network
errors when they are the cause of the failures for instance;
- Remove delegate method which was not taking a video ID and hardcoding one, as
we can provide the video ID in all cases or do not provide a video ID at worse;
- Rename and make extraction methods package-private, as they are not intended
to be used publicly.

These breaking internal changes have been applied where needed, in
YoutubeJavaScriptExtractorTest and YoutubeStreamExtractor (in which an unneeded
initStsFromPlayerJsIfNeeded call have been removed).
2023-08-02 23:05:08 +02:00
Stypox 5492343b8e
Pre-release v0.22.7 2023-08-02 20:02:29 +02:00
Tobi 9a59afbcf5
Merge pull request #1086 from AudricV/yt_no-exception-channels-without-banner
[YouTube] Don't throw an exception when there is no banner available on a channel
2023-08-02 18:56:26 +02:00
AudricV bb1ab166bf
[YouTube] Test that no banner is returned for carouselHeaderRenders 2023-08-01 22:19:43 +02:00
AudricV f1fa84b4e3
[YouTube] Don't throw an exception when there is no banner available on a channel
Channels may not have a banner, so no exception should be thrown if no banner
is found.
2023-08-01 12:40:20 +02:00
Tobi 39a911db9f
Merge pull request #1084 from AudricV/yt_android-403s-workaround-and-streams-tests-fixes
[YouTube] Workaround again 403 HTTP issues on the ANDROID InnerTube client and fix stream tests
2023-07-31 23:51:10 +02:00
dependabot[bot] bda3a3fc5d
Bump org.junit:junit-bom from 5.9.3 to 5.10.0
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.9.3 to 5.10.0.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.9.3...r5.10.0)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-07-24 09:53:05 +00:00
AudricV 522c78160f
[YouTube] Update stream tests mocks 2023-07-23 19:36:28 +02:00
AudricV 7528eb2bd9
[YouTube] Fix stream tests failures
- Fix testCheckAudioStreams test of
YoutubeStreamExtractorDefaultTest.AudioTrackLanguage test class, by updating
the excepted audio track name test to use the updated English audio track name
(audio track type info has been added on the video tested);
- Fix YoutubeStreamExtractorDefaultTest.PublicBroadcasterTest test class by
using a different video from a French and German public broadcast channel, as
the channel Dinge Erklärt – Kurzgesagt is not affiliated with a public
broadcast channel anymore;
- Fix YoutubeStreamExtractorLivestreamTest test class, by updating the excepted
name of the livestream to the current one.
2023-07-23 19:19:02 +02:00
AudricV 164c8e3abb
[YouTube] Workaround again 403 HTTP issues on the Android client by using new player parameters
These parameters are the only ones currently known to bypass 403 HTTP issues
related to failure of passing Android client integrity checks, as the ones of
stories (and the base of the shorts ones) do not work anymore, which may be
related to end of this format on the service.
2023-07-22 20:22:16 +02:00
Tobi 8ebbe1b7e1
Merge pull request #706 from FireMasterK/av1-itags
Add support for AV1 itags.
2023-07-22 13:31:45 +02:00
FireMasterK 6db0d116fe Add support for AV1 itags. 2023-07-22 13:23:44 +02:00
Tobi 986a3f7747
Merge pull request #1080 from TeamNewPipe/proguard-rules
Add proguard rules to README
2023-07-07 12:52:10 +02:00
Tobi d6132a9ed9
Merge pull request #1076 from AudricV/yt-fix-throttling-parameter-regex
[YouTube] Support multiple declarations for throttling parameter function name array
2023-07-07 12:51:26 +02:00
Tobi 87eb29a25d Add proguard rules to README 2023-07-05 08:25:23 +02:00
AudricV 4e22c5ee87
[YouTube] Support multiple declarations for throttling parameter function name array
Also moved the corresponding regex parts in static constants for easier future
modifications
2023-06-26 15:25:53 +02:00
Kavin d961d349c3
[YouTube] Check whether player responses are valid for all InnerTube clients used (#1070)
Co-authored-by: Audric V <74829229+AudricV@users.noreply.github.com>
2023-06-18 21:54:52 +02:00
ThetaDev ad97f08048
[YouTube] Fix parsing short relative date formats (English only) (#1068) 2023-06-18 21:41:29 +02:00
Tobi d294ccb433
Merge pull request #1071 from TeamNewPipe/feat/ServiceList
Init services at the correct place
2023-06-17 20:50:24 +02:00
Tobi 5d112aa772
Merge pull request #1074 from TeamNewPipe/imp/media-formats
Add more audio media formats and MediaFormat.getAllForMimeType(mimeType)
2023-06-17 20:49:38 +02:00
TobiGr 0e01d90562 Do not init services while creating services array 2023-06-17 18:08:09 +02:00
TobiGr 5809904cf7 Add annotations to MediaFormat 2023-06-17 18:01:54 +02:00
TobiGr 53dfd871e2 Add more audio media formats and MediaFormat.getAllForMimeType(mimeType) 2023-06-17 18:01:29 +02:00
Stypox 5eb886243f
Merge pull request #1072 from TeamNewPipe/fix/bandcamp_test
Fix BandcampRadioStreamExtractorTest.testGetAudioStreams()
2023-06-09 17:35:35 +02:00
TobiGr 19ce06fe00 Fix BandcampRadioStreamExtractorTest.testGetAudioStreams() 2023-06-06 23:07:56 +02:00
Audric V 533121fb81
Merge pull request #1045 from Theta-Dev/fix/trending-video-tab
[YouTube] Extract trends from A/B tested "Videos" tab and fix extraction of trends name from A/B tested new title design
2023-05-19 11:22:49 +02:00
Audric V 92a0024424
Merge pull request #1052 from TeamNewPipe/peertube/fix/nested-comment-replies
[PeerTube] Fix multi level comment replies
2023-05-18 18:49:06 +02:00
Tobi 3c036a9c03
Merge pull request #1061 from ChunkyProgrammer/develop/extract-yt-playlist-descrption
Extract playlists description
2023-05-17 22:43:17 +02:00
TobiGr c70bb83801 [Bandcamp] Implement PlaylistExtractor.getDescsription() 2023-05-15 15:23:03 +02:00
TobiGr ca0ce00753 Add PlaylistInfoItem.getDescription() and PlaylistInfoItemExtractor.getDescription()
[PeerTube] Implement the corresponding extractor method.
TODO: add tests
2023-05-12 01:43:59 +02:00
TobiGr b218bf69bd Implement PlaylistInfo.getDescription()
Implement PlaylistExtractor.getDescription() for PeerTube and SoundCloud.
Anotate   PlaylistExtractor.getDescription() as Nonnull
2023-05-12 00:44:10 +02:00
chunky programmer 81f29116ba switch from string to Description object 2023-05-11 00:36:57 -04:00
chunky programmer e147867d41 Add tests 2023-05-11 00:00:34 -04:00
chunky programmer 5ab6cd7420 Extract YouTube playlist description 2023-05-11 00:00:22 -04:00
TobiGr d358ba1c41 Improve PeertubeCommentsInfoItemExtractor constructor 2023-05-07 22:55:26 +02:00
TobiGr aff3e795f8 [PeerTube] Fix multi level comment replies 2023-05-07 22:49:14 +02:00
ThetaDev 3673d4ae01
fix: YouTube trending name extraction 2023-05-03 21:16:35 +02:00
ThetaDev 0addb98cd7
tests: update mocks 2023-05-03 21:16:35 +02:00
ThetaDev 24eba62305
fix: extract YouTube trends from new "Videos" tab 2023-05-03 21:16:23 +02:00
Kavin a9ca5c49e4
Merge pull request #1056 from AudricV/yt-improve-search-suggestions-extraction
[YouTube] Switch to new search suggestion domain and improve error handling
2023-05-02 20:17:48 +01:00
Stypox 00408db959
Merge pull request #1058 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.16.1
Bump org.jsoup:jsoup from 1.15.4 to 1.16.1
2023-05-01 19:59:02 +02:00
dependabot[bot] 108f8a7a17
Bump org.jsoup:jsoup from 1.15.4 to 1.16.1
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup) from 1.15.4 to 1.16.1.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.4...jsoup-1.16.1)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-05-01 09:58:32 +00:00
Tobi 11565db17f
Merge pull request #1055 from AudricV/peertube-no-description-channels
[PeerTube] Don't return "No description" when there is no description for a channel or an account
2023-04-30 23:38:10 +02:00
AudricV cf6040ddb3
Update YoutubeSuggestionExtractorTest mocks 2023-04-30 19:53:44 +02:00
AudricV 593122342f
[YouTube] Improve YoutubeSuggestionExtractorTest
- Remove useless concatenation on the downloader path;
- Remove unneeded public test modifier;
- Update license header;
- Specify the service class tested instead of the generic class.
2023-04-30 19:53:43 +02:00
AudricV e923fca440
[YouTube] Switch to new search suggestion domain and improve error handling
- Switch to the new domain used by YouTube for search suggestions,
suggestqueries-clients6.youtube.com, and add the xhr query parameter with the
t value, to allow getting responses without requiring trim;
- Use the Java 8 Stream API to collect search suggestions and improve invalid
response detection by checking whether the content type of the response
returned is JSON;
- Move the licence header at the top of the file.
2023-04-30 19:53:42 +02:00
AudricV 945165a3c0
[PeerTube] Don't return "No description" when there is no description for a channel or an account
When a description is missing, no description should be returned, even the ones
indicating there is no description. This behavior is represented by a null
return instead.

Also update PeertubeAccountExtractorTest to reflect these changes.
2023-04-30 18:41:38 +02:00
Stypox d61dc27406
Merge pull request #1054 from TeamNewPipe/fix-ci-variable
Make sure CI is not vulnerable to bash word splitting
2023-04-27 18:52:07 +02:00
Stypox 51cba5a287
Make sure CI is not vulnerable to bash word splitting
See https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-an-intermediate-environment-variable
2023-04-26 18:36:25 +02:00
Stypox f819c2fe34
Merge pull request #1053 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.3
Bump org.junit:junit-bom from 5.9.2 to 5.9.3
2023-04-26 15:42:19 +02:00
dependabot[bot] 4b92c5a34d
Bump org.junit:junit-bom from 5.9.2 to 5.9.3
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.9.2 to 5.9.3.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.9.2...r5.9.3)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-04-26 09:57:57 +00:00
Stypox 2deb023da4
Merge pull request #1050 from Theta-Dev/fix/channel-carousel-header
[YouTube] Add support for CarouselHeaderRenderer
2023-04-25 15:17:31 +02:00
ThetaDev 4aada7f91b refactor: rename carousel header channel test 2023-04-21 22:48:37 +02:00
Tobi b2b426048f
Merge pull request #1051 from Theta-Dev/fix/ytm-client-version-rg
[YouTube] fix: set musicClientVersion regex capture group
2023-04-16 23:48:19 +02:00
ThetaDev 47aa9fed40 fix: set musicClientVersion regex capture group 2023-04-16 19:25:05 +02:00
ThetaDev 20370395c5 fix: add support for CarouselHeaderRenderer 2023-04-16 17:40:13 +02:00
Stypox 7dba6e3891
Merge pull request #1033 from petlyh/bandcamp-paywalled-content
[Bandcamp] Handle paywalled tracks
2023-04-12 13:04:26 +02:00
petlyh e6aad117e7
[Bandcamp] Throw PaidContentException on paywalled albums 2023-04-03 19:27:09 +02:00
TobiGr 8495ad619e Bump version to 0.22.6 2023-04-02 22:35:02 +02:00
fynngodau 69705138e4
[Bandcamp] Fix extraction of related playlist items URL (#1047)
Small change in HTML structure
2023-04-02 22:24:29 +02:00
Björn Sigurbergsson 1b6fe5edd6
[YouTube] Fix ParsingException when comments are unavailable in a video (#1040)
Co-authored-by: bjs <bjs@elect-it.com>
Co-authored-by: Audric V. <74829229+AudricV@users.noreply.github.com>
Co-authored-by: Kavin <20838718+FireMasterK@users.noreply.github.com>
2023-03-30 19:58:06 +02:00
ThetaDev 8d1303e18f
Add track types to audio streams (#1041) 2023-03-28 00:02:20 +02:00
Tobi 44ae139d33
Merge pull request #1034 from AudricV/peertube-fix-commentsinfo-test
[PeerTube] Fix comments info test
2023-03-20 19:30:35 +01:00
Tobi d5e9df5381
Merge pull request #1038 from TeamNewPipe/docs-jdk-11
Update JDK to 11 in JDoc workflow
2023-03-20 19:25:21 +01:00
Tobi 0486ad7a08
Update JDK to 11 in JDoc workflow 2023-03-06 18:45:11 +01:00
AudricV 80a6fc2c63
[PeerTube] Fix testGetCommentsFromCommentsInfo test of PeertubeCommentsExtractorTest.Default
The tested comment has been removed, so it couldn't be found in the comments
list.

This comment has been replaced by a new one from the current comments of the
video.

Also, in the parent class PeertubeCommentsExtractorTest, final has been used as
much as possible and for-each loops of lists have been replaced by their
forEach method or the Stream API, in order to simplify code.
2023-03-04 16:49:10 +01:00
petlyh 5a9b6ed2e3
[Bandcamp] Support loading additional comments (#1030) 2023-03-04 14:01:06 +01:00
Stypox 6bdd698c25
Merge pull request #1026 from AudricV/audio-streams-descriptive-and-locale-properties
Add descriptive and locale properties to audio streams
2023-03-01 11:15:46 +01:00
Stypox 19e4b216c9
Merge pull request #1032 from AudricV/yt_fix-comments-hashtags-links-extraction
[YouTube] Fix hashtags links extraction and escape HTML links
2023-03-01 10:47:37 +01:00
Stypox b1298490c0
Merge pull request #1029 from AudricV/yt_fix-no-views-extraction-playlist-items
[YouTube] Fix partial non-extraction of "No views" string in stream items
2023-03-01 10:46:52 +01:00
petlyh 9dc1832733
[Bandcamp] Handle paywalled tracks 2023-02-28 17:51:30 +01:00
fynngodau 3fdb6ee476
Merge pull request #1031 from petlyh/bandcamp-fix-radio-comments
[Bandcamp] Show comments as disabled on radio streams
2023-02-27 20:50:12 +01:00
AudricV bd79b921e8
[YouTube] Refactor the code to get stream items' view count
This refactoring avoids code duplication as much as possible.
2023-02-27 10:25:46 +01:00
AudricV 51f9b39953
[YouTube] Fix partial non-extraction of no views string in stream items
As the "No views" string is returned in the case there is no view on a video, a
number cannot be parsed in this case, so -1 was returned.

This string is now detected in all methods to get the view count of a stream.
2023-02-27 10:18:45 +01:00
AudricV 95b3f5e391
[MediaCCC] Test audio language property extraction 2023-02-26 19:06:18 +01:00
AudricV 30a0f8c510
[MediaCCC] Extract audio language property for single language audio tracks 2023-02-26 19:06:18 +01:00
AudricV 7f0269c4c7
[YouTube] Edit YoutubeStreamExtractorDefaultTest.AudioTrackLanguage to test audio locale property
The Hindi audio track language presence test has been changed from audio track
label to audio locale.
2023-02-26 19:06:18 +01:00
AudricV 034f82dae7
[YouTube] Test language and descriptive audio in YoutubeDashManifestCreatorsTest 2023-02-26 19:06:17 +01:00
AudricV 05e8cb39f7
[YouTube] Add language and descriptive audio properties to DASH manifests 2023-02-26 19:06:17 +01:00
AudricV bf30d70152
[YouTube] Add descriptive audio test
This test uses video TjxC-evzxdk.

Also improve a bit YoutubeStreamExtractorDefaultTest.AudioTrackLanguage test.
2023-02-26 19:06:17 +01:00
AudricV 76b7c19c5d
[YouTube] Extract whether a track is a descriptive audio and audio locale when available
Also use audio track setters only for audio itags.
2023-02-26 19:06:17 +01:00
AudricV 3bb5eeef30
[YouTube] Add descriptive and locale audio support in ItagItem 2023-02-26 19:06:16 +01:00
AudricV 14bf3fb05b
Add ability to know the locale of an audio stream
Getting audio tracks locales by parsing their ID or their label, should not be
done by clients, but by the extractor.

This commit adds the ability to store the Locale of an AudioStream, which is
used to compare similar AudioStreams (in the equalStats method).
2023-02-26 19:06:16 +01:00
AudricV f92426560c
Add descriptive audio properties
Also improve AudioStream's audio language documentation
2023-02-26 19:06:16 +01:00
AudricV a63f289667
[YouTube] Update mocks of YoutubeCommentsExtractorTest.FormattingTest 2023-02-26 18:50:07 +01:00
AudricV 9483dcd9fa
[YouTube] Update mocks of YoutubeCommentsExtractorTest.RepliesTest 2023-02-26 18:43:36 +01:00
AudricV 1556adbb2d
[YouTube] Fix hashtags links extraction and escape text in attribute descriptions + HTML links
webCommandMetadata object is contained inside a commandMetadata one, so it is
not accessible from the root of the navigationEndpoint object.

The corresponding statement has been moved at the bottom of the specific
endpoints parsing, as the webCommandMetadata object is present almost
everywhere, otherwise URLs of some endpoints would have be changed, such as
uploader URLs (from channel IDs to handles).

As no ParsingException is now thrown by getUrlFromNavigationEndpoint, and so by
getTextFromObject, getUrlFromObject and getTextAtKey, the methods which were
catching ParsingExceptions thrown by these methods had to be updated.

URLs got in the HTML version of getTextFromObject are now escaped properly to
provide valid HTML to clients. This has been also done for attribute
descriptions, with the description text for this type of descriptions.

As YouTube descriptions are in HTML format (except for the fallback on the JSON
player response, which is plain text and only happens when there is no visual
metadata or a breaking change), all URLs returned are escaped, so tests which
are testing presence of URLs with escaped characters had to be updated (it was
only the case for YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing).
2023-02-26 18:43:36 +01:00
Stypox 99ab9777ad
Release 0.22.5 2023-02-26 16:33:25 +01:00
Stypox 39c2e1fda0
Merge pull request #1028 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.4
Bump org.jsoup:jsoup from 1.15.3 to 1.15.4
2023-02-26 16:27:31 +01:00
petlyh f7a7a236fb
[Bandcamp] Show comments as disabled on radio streams 2023-02-23 18:42:43 +01:00
dependabot[bot] f5599ff08d
Bump org.jsoup:jsoup from 1.15.3 to 1.15.4
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup) from 1.15.3 to 1.15.4.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.3...jsoup-1.15.4)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-20 10:05:26 +00:00
Stypox 7e793c11ae
Empty commit to make JitPack happy
The build for e920ab3f5f succeeded, but for some reason it is considered as failed...
2023-02-08 22:35:26 +01:00
Stypox e920ab3f5f
Merge pull request #1021 from TeamNewPipe/fix/yt-comment-text-hashtag
[YouTube] Fix getting the comment text if the comment contains a hashtag
2023-01-29 21:47:59 +01:00
TobiGr 3f7df9536e [YouTube] Fix getting the comment text if the comment contains a hashtag 2023-01-29 20:33:51 +01:00
Stypox 999fb7f812
Merge pull request #1024 from AudricV/snd_fix-tracks-like-count
[SoundCloud] Fix extraction of tracks like count
2023-01-29 10:52:54 +01:00
Stypox 3519d4c367
Merge pull request #1015 from AudricV/yt_fix-channel-id-rss-feeds
[YouTube] Fix channel ID extraction of YouTube channels RSS feeds
2023-01-29 10:41:38 +01:00
Stypox 9aca710e86
Merge pull request #1013 from Stypox/fix-music-mixes
[YouTube] Now music mixes can be treated as normal mixes
2023-01-29 09:48:51 +01:00
Stypox 76eeabac45
Merge pull request #1020 from TeamNewPipe/fix/yt-subscriber-count
[YouTube] Fix NPE in search when getting channel items without subscriber count
2023-01-29 09:44:22 +01:00
AudricV 676622f6df
[SoundCloud] Fix expectedLikeCountAtLeast tests of SoundcloudStreamExtractorTest test classes
As like count is now returned by the extractor, we need to assert a positive
minimum like count, which is close to the actual value, in order to avoid test
failures due to lower like counts than the ones excepted.
2023-01-29 01:08:02 +01:00
AudricV 2a24d407d5
[SoundCloud] Fix extraction of tracks like count
SoundCloud is using likes_count to return the like count of a track, like it
was the case before they switched to favoritings_count.
2023-01-29 01:00:49 +01:00
Tobi 0e4e6a9bac
Merge pull request #1022 from AudricV/yt_support-live-urls
[YouTube] Support live URLs
2023-01-28 21:52:26 +01:00
AudricV ba24976e41
[YouTube] Add live URLs test and do minor improvements to YoutubeStreamLinkHandlerFactoryTest
- Remove unused imports;
- Replace wildcard imports by single class imports;
- Suppress "HTTP links are not secured" warnings from IDEA IDEs;
- Replace removed video jZViOEv90dI by an existing video, 9Dpqou5cI08 (the
corresponding test has been of course renamed).
2023-01-28 19:36:21 +01:00
AudricV 57f850bc2d
[YouTube] Support live URLs and do minor improvements to YoutubeStreamLinkHandlerFactory
- Move license header at the top;
- Use an unmodifiable set for the subpaths instead of a modifiable list;
- Add missing Nonnull and Nullable annotations;
- Improve exception messages.
2023-01-28 19:36:20 +01:00
AudricV 1f4ed9dce9
[YouTube] Fix channel ID extraction of YouTube channel RSS feeds
The yt:channelId element doesn't provide the channel ID anymore and is empty,
like the id element, so we need now to extract it from the channel URL provided
in two elements: author -> uri and feed -> link.

Also avoid a NullPointerException in getUrl and getName methods.
2023-01-28 11:53:33 +01:00
Tobi c589a2c1a2
Merge pull request #1014 from TeamNewPipe/fix/yt-comments
[YouTube] Fix getting next comments pages
2023-01-27 11:14:55 +01:00
Tobi c2b36fd405
Merge pull request #1016 from lonewolf2208/bug-fix
[YouTube] Remove topStandaloneBadge check for view count of stream items
2023-01-25 08:30:36 +01:00
TobiGr 72573932cf [YouTube] Fix NPE in search when getting channel items without subscriber count 2023-01-24 23:03:45 +01:00
TobiGr f50b7275af [YouTube] Fix getting next comments pages 2023-01-24 22:39:08 +01:00
Kunal 9bdad40b06 Removed topStandaloneBadge 2023-01-20 02:41:21 +05:30
Stypox 5945057227
[YouTube] Add music mix test 2023-01-15 23:30:30 +01:00
Stypox 7293991832
[YouTube] Now music mixes can be treated as normal mixes
Using a playlist extractor on them would result in "Unviewable playlist" errors
2023-01-15 23:28:59 +01:00
Stypox ff94e9f30b
Merge pull request #1009 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.10.1
Bump gson from 2.10 to 2.10.1
2023-01-11 15:35:36 +01:00
Stypox c1040bccac
Merge pull request #794 from FireMasterK/comments-count
[YouTube] Add support to extract total comment count
2023-01-11 15:32:19 +01:00
Stypox 6ccc43e57e
Merge pull request #1010 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.2
Bump junit-bom from 5.9.1 to 5.9.2
2023-01-11 13:19:33 +01:00
dependabot[bot] 1bc44cb949
Bump junit-bom from 5.9.1 to 5.9.2
Bumps [junit-bom](https://github.com/junit-team/junit5) from 5.9.1 to 5.9.2.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.9.1...r5.9.2)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-01-11 09:04:12 +00:00
dependabot[bot] f43049985e
Bump gson from 2.10 to 2.10.1
Bumps [gson](https://github.com/google/gson) from 2.10 to 2.10.1.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.10...gson-parent-2.10.1)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-01-09 09:05:55 +00:00
Stypox 33f9f2cf66
Merge pull request #1006 from TeamNewPipe/yt/fix-html-escape
[YouTube] Fix escaping links in YouTubeParsingHelper.getTextFromObject
2023-01-05 19:02:23 +01:00
TobiGr 56aab4d971 [YouTube] Fix escaping links in YouTubeParsingHelper.getTextFromObject 2023-01-05 00:28:12 +01:00
Kavin 22a47da8c7
Fix requested change and remove outdated comment. 2023-01-02 20:42:32 +00:00
Kavin 98a90fd9c8
Don't cache comments count and return early on page fetch if no token. 2023-01-02 20:40:48 +00:00
Kavin 2974dfaa48
Only store ajaxJson for initial page and eager fetch the initial continuation. 2023-01-02 20:40:48 +00:00
Kavin 64d24aa09e
Fix request changes. 2023-01-02 20:40:48 +00:00
Kavin 67ef4f4c30
Cleanup and remove optional. 2023-01-02 20:40:48 +00:00
FireMasterK 22f71b010c
Fix for requested changes. 2023-01-02 20:40:48 +00:00
FireMasterK 656b7c1cd9
Improve method documentation. 2023-01-02 20:40:48 +00:00
FireMasterK 981aee4092
Add support to extract total comment count. 2023-01-02 20:40:48 +00:00
Stypox 45636b0d00
Merge pull request #986 from Isira-Seneviratne/Static_maps
Use immutable Map factory methods.
2023-01-02 18:11:14 +01:00
Stypox 219c5c5be5
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java 2023-01-02 18:11:03 +01:00
Stypox 259de3cba6
Merge pull request #995 from TeamNewPipe/feat/soundcloud-playlistinfoitemextractor
[SoundCloud] Implement getUploaderUrl() and isUploaderVerified() for PlaylistInfoItemExtractor
2023-01-02 15:10:40 +01:00
Stypox 991394b53a
Merge pull request #1005 from FireMasterK/fix-escaping-xss
Fix for potential XSS attacks and formatting issues
2023-01-02 15:06:17 +01:00
Stypox ce15f7cc50
Merge pull request #996 from TeamNewPipe/feat/peeertube-playlists
[PeerTube] Support searching for playlists and channels
2023-01-02 15:04:09 +01:00
Isira Seneviratne d8ce08d969 Use immutable Map factory methods. 2023-01-02 07:50:31 +05:30
Kavin 01acf79436
Fix for potential XSS attacks. 2022-12-31 20:05:32 +00:00
TobiGr 292e0d8ce7 [SoundCloud] Implement getUploaderUrl() and isUploaderVerified() for PlaylistInfoItemExtractor 2022-12-31 18:46:39 +01:00
TobiGr 2a8729aeb2 Apply suggestions
Co-authored-by: Stypox <stypox@pm.me>
2022-12-31 18:24:33 +01:00
TobiGr d75a997611 [PeerTube] Support searching for channels 2022-12-31 18:24:33 +01:00
TobiGr dea6d8ce4c [PeerTube] Support searching for playlists 2022-12-31 18:24:33 +01:00
Stypox 95cc6aefbb
Merge pull request #994 from TeamNewPipe/fix/peertube-subtitles-exception
[PeerTube] Report Exceptions thrown while getting a stream's subtitles
2022-12-31 15:01:39 +01:00
Stypox 7b54457789
Merge pull request #941 from TeamNewPipe/feat/peertube-comment-replies
[PeerTube]  Support comment replies
2022-12-31 14:57:51 +01:00
AudricV f45966d449
Merge pull request #910 from Isira-Seneviratne/Locale_forLanguageTag
Add compat Locale.forLanguageTag() implementation.
2022-12-24 23:53:30 +01:00
AudricV d5437e0bc5
Merge pull request #863 from AudricV/add-content-type-and-content-length-headers-to-post-requests
Add Content-Type header to all POST requests without an empty body
2022-12-16 19:32:56 +01:00
Tobi 88e07e555d
Merge pull request #1000 from AudricV/yt-streaminfoitemextractor-improvements
[YouTube] Improve YoutubeStreamInfoItemExtractor
2022-12-11 17:02:29 +01:00
AudricV 0766b1d211
[YouTube] Improve YoutubeStreamInfoItemExtractor
- Return duration of video premieres;
- Add another non-localized method to determine whether a stream is a running
livestream;
- Return view count and upload date of videos in playlists;
- Store isPremiere result;
- Remove shorts workaround code, as it was only useful on channels and shorts
have been moved into a separated channel tab;
- Improve some other code.
2022-12-08 13:59:12 +01:00
Tobi 896d7e09eb
Merge pull request #978 from Theta-Dev/fix/search-channel-handles
[YouTube] Fix search subscriber count extraction with channel handles
2022-12-05 17:52:05 +01:00
TobiGr cd3262745d [PeerTube] Report Exceptions thrown while getting a stream's subtitles 2022-12-03 16:11:21 +01:00
TobiGr 4e66b2287e [PeerTube] Add support for comment replies 2022-12-01 14:05:18 +01:00
Tobi 41c8dce452
Merge pull request #992 from Isira-Seneviratne/String_isBlank
Use String.isBlank().
2022-11-30 17:48:54 +01:00
Isira Seneviratne 2bca56f0df Use String.isBlank(). 2022-11-30 08:26:21 +05:30
Isira Seneviratne 3b80547976 Add code review suggestions. 2022-11-30 07:57:45 +05:30
Tobi a1128ecbdc
Merge pull request #991 from FireMasterK/fix-compile
Fix complication error in comment test
2022-11-29 19:20:01 +01:00
ThetaDev 016623131e docs: update comment in YoutubeChannelInfoItemExtractor 2022-11-29 19:06:03 +01:00
Kavin 2e08eaad96
Fix complication error in comment test. 2022-11-29 16:07:48 +00:00
Kavin abf08e1496
Merge pull request #990 from FireMasterK/bold-italic-strikethrough
[YouTube] Implement bold/italic/strike-through support
2022-11-29 15:59:38 +00:00
Tobi 72d5ed3318
Merge pull request #987 from FireMasterK/comments-text-description
Use Description object for comments text.
2022-11-29 16:54:56 +01:00
Kavin 57e7a6fb7c
Add mocks test. 2022-11-28 20:27:55 +00:00
Kavin 1d3d7fa5c3
Add test for formatting. 2022-11-28 20:26:37 +00:00
Kavin 52fda37915
Implement bold/italic/strike-through support. 2022-11-28 19:06:18 +00:00
Kavin b566084cac
Use Description object for comments text. 2022-11-28 17:02:19 +00:00
Tobi 40f1ec4a54
Merge pull request #989 from Stypox/forbid-wrong-nullable-notnull-imports
Block wrong nullable/nonnull imports in checkstyle
2022-11-28 14:57:46 +01:00
Stypox 5f6101698b
Block wrong nullable/nonnull imports in checkstyle 2022-11-28 14:42:18 +01:00
Tobi f8162b049d
Merge pull request #984 from FireMasterK/unused-dep
Remove unused autolink dependency
2022-11-28 11:28:42 +01:00
Tobi ad7c3c2374
Merge pull request #985 from FireMasterK/checkstyle-update
Update checkstyle dependency.
2022-11-28 11:22:40 +01:00
Tobi 1da0190056
Merge pull request #980 from TeamNewPipe/fix/yt/unavailable
[YouTube] Fix extracting the detailed error message for unavailable streams
2022-11-28 10:07:34 +01:00
Stypox 60fb30f835
Merge pull request #928 from FireMasterK/comment-urls
Parse YouTube comments as HTML
2022-11-27 19:16:34 +01:00
Kavin 5abea22225
Fix throwing correct reason. 2022-11-26 21:09:08 +00:00
Kavin 7874d3c8aa
Update checkstyle dependency.
Since we now use JDK 11
2022-11-26 20:22:25 +00:00
Kavin faf28f5c11
Remove unused dependency. 2022-11-26 20:17:25 +00:00
Kavin a3ed947b2d
Merge pull request #983 from FireMasterK/update-countries
Update supported countries list.
2022-11-26 20:06:00 +00:00
Kavin c043597255
Update supported countries list. 2022-11-26 19:01:33 +00:00
TobiGr 4680df0bdf Fix throwing correct reason 2022-11-23 17:03:22 +01:00
TobiGr 9de8405c9f [YouTube] Fix extracting the detailed error message of streams which are unavailable 2022-11-23 08:33:06 +01:00
Stypox 34d79bd267
[YouTube] Update mocks 2022-11-22 17:10:04 +01:00
AudricV 2ec296e674
Fix YoutubeSearchExtractorTest.MetaInfoTest
Not all the "learn more" button is uppercase anymore, that's only the case for
the first letter.
2022-11-22 16:34:54 +01:00
AudricV 3891542ca1
Use Downloader's postWithContentType and postWithContentTypeJson methods in services and extractors 2022-11-22 11:37:18 +01:00
AudricV b2862f3cd1
Add postWithContentType and postWithContentTypeJson utility methods in Downloader
Co-authored-by: Stypox <stypox@pm.me>
2022-11-22 11:37:17 +01:00
AudricV e9a0d3bd95
[YouTube] Send Content-Type header in all POST requests
This header was not sent partially before and was added and guessed by OkHttp. This can create issues when using other HTTP clients than OkHttp, such as Cronet.

Some code in the modified classes has been improved and / or deduplicated, and usages of the UTF_8 constant of the Utils class has been replaced by StandardCharsets.UTF_8 where possible.

Note that this header has been not added in except in YoutubeDashManifestCreatorsUtils, as an empty body is sent in the POST requests made by this class.
2022-11-22 11:37:16 +01:00
AudricV b9e463de49
[Bandcamp] Send Content-Type header in POST requests
This header was not sent before and was added and guessed by OkHttp. This can create issues when using other HTTP clients than OkHttp, such as Cronet.

Also make use of StandardCharsets.UTF_8 when getting bytes of bodies instead of the platform default's charset, to make sure to prevent some encoding issues on some JVMs.
2022-11-22 11:35:46 +01:00
AudricV 65d6321e3d
Fix typos in Downloader.post JavaDocs
Post methods in Downloader return the result of a POST request and not the one of a GET request.
2022-11-22 11:35:46 +01:00
ThetaDev 5daabd1793 fix: #976 search subscriber count extraction with channel handles 2022-11-22 02:17:10 +01:00
Kavin c953e23414
Merge pull request #968 from AudricV/yt-support-no-video-info-renderers-for-streams
[YouTube] Support lack of video info renderers for streams
2022-11-16 20:20:01 +00:00
Tobi 2211a24b69
Merge pull request #971 from lrusso96/patch-1
[YouTube] Improve duration parsing
2022-11-16 16:14:54 +01:00
Tobi f6eefdc76c
Merge pull request #975 from FireMasterK/jitpack
Use JDK 11 for Jitpack gradle builds
2022-11-15 20:23:21 +01:00
Kavin a1f22be0dd
Use JDK 11 for Jitpack gradle builds. 2022-11-15 18:43:19 +00:00
Kavin 9bdad55508
Merge pull request #973 from FireMasterK/audio-track
Add support for extracting audio track information
2022-11-15 08:19:47 +00:00
Kavin 86f06b333a
Address review. 2022-11-14 00:05:31 +00:00
Kavin b16e6082e1
Add test for audio stream languages. 2022-11-13 23:10:44 +00:00
Kavin 30909da1df
Fix audio track similar comparison for IDs. 2022-11-13 23:08:54 +00:00
Kavin 6d59cdbe3a
Add support for extracting audio tracks. 2022-11-13 21:39:29 +00:00
Tobi 43d1c1f8b1
Merge pull request #880 from Isira-Seneviratne/Use_StandardCharsets
Use StandardCharsets.UTF_8.
2022-11-12 03:23:44 +01:00
Isira Seneviratne e4d982c7ea Fix license. 2022-11-12 07:29:15 +05:30
Isira Seneviratne 416089146e Fix failing tests. 2022-11-12 07:29:15 +05:30
Isira Seneviratne ff5f223d3f Update README.md 2022-11-12 07:29:15 +05:30
Isira Seneviratne ddbce3b83d Add Utils methods for URL encoding/decoding using UTF-8. 2022-11-12 07:29:15 +05:30
Isira Seneviratne 366f5c1632 Use StandardCharsets.UTF_8. 2022-11-12 07:29:15 +05:30
Tobi 0ea16c0b73
Merge pull request #954 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.10
Bump gson from 2.9.1 to 2.10
2022-11-09 17:28:31 +01:00
dependabot[bot] 0b82fade51
Bump gson from 2.9.1 to 2.10
Bumps [gson](https://github.com/google/gson) from 2.9.1 to 2.10.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.9.1...gson-parent-2.10)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-09 16:21:16 +00:00
Luigi Russo c9635218e2
[YouTube] Improve duration parsing 2022-11-09 09:41:29 +01:00
Isira Seneviratne 1be270768f
Merge pull request #893 from Isira-Seneviratne/Static_sets
Use immutable sets in YoutubeParsingHelper.
2022-11-09 09:07:01 +05:30
Isira Seneviratne 316d8573fa Use immutable sets in YoutubeParsingHelper. 2022-11-07 07:50:26 +05:30
AudricV 6a2c680d8f
[YouTube] Add mocks for YoutubeStreamExtractorDefaultTest.NoVisualMetadataVideoTest 2022-11-04 19:43:06 +01:00
AudricV e66fed41d6
[YouTube] Add a StreamExtractor test for a video without visual metadata
The video "Makani’s first commercial-scale energy kite" (video ID:
An8vtD1FDqs), which has this behavior, is used for the new test,
NoVisualMetadataVideoTest, added in YoutubeStreamExtractorDefaultTest.

Tests of elements who throw an exception in this case (subscriber count, like
count, uploader avatar URL) test if the ParsingException exception is thrown by
YoutubeStreamExtractor.
2022-11-04 19:42:12 +01:00
AudricV aa9a8ca23c
[YouTube] Make non-extraction of videoPrimaryInfoRenderer and/or videoSecondaryInfoRenderer not fatal
Also de-duplicated common code related to the obtain of these video info renderers.

This change allows extraction of videos without visual metadata.
2022-11-04 18:35:53 +01:00
AudricV eb07d70a2c
Merge pull request #964 from AudricV/yt-support-handles-and-all-channel-usernames
[YouTube] Support handles and all channel usernames
2022-11-04 12:09:15 +01:00
AudricV 20cd8e8a4a
[YouTube] Update mocks of YoutubeChannelExtractorTest.Gronkh 2022-11-03 19:46:42 +01:00
AudricV a34f060ba0
[YouTube] Use a handle for YoutubeChannelExtractorTest.Gronkh 2022-11-03 19:46:42 +01:00
AudricV 724f669ff7
[YouTube] Add tests for handles and user IDs with non ASCII characters support
Unneeded public modifiers in test methods of
YoutubeChannelLinkHandlerFactoryTest have been also removed.
2022-11-03 19:46:42 +01:00
AudricV 61ce041bda
[YouTube] Support handles and all custom channel names
More non-channel paths have been also added to the excluded custom name paths,
documentation and exception messages have been improved and fixed in some
places, and the licence header of YoutubeChannelLinkHandlerFactory has been
moved to its beginning and updated.
2022-11-03 19:46:42 +01:00
AudricV ffffb04439
Merge pull request #953 from Theta-Dev/attributed-text-desc
[YouTube] Add support for attributed text description
2022-11-03 18:34:30 +01:00
ThetaDev 592e1d6386 fix: parsing attributed description with no command runs 2022-11-03 12:10:52 +01:00
AudricV 31bf7046b7
Merge pull request #944 from Theta-Dev/fix-rich-grid-renderer
[YouTube] Support richGridRenderer on channel page to fix extraction of videos in channels
2022-11-03 00:10:54 +01:00
ThetaDev 099b53cc4f
[YouTube] Add parser for attributedDescription
Also update the mock of the next InnerTube endpoint response of the
YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing test class with an
attributedDescription instead of a regular description
2022-11-02 23:11:33 +01:00
AudricV e4c24d4c36
[YouTube] Regenerate supported channels mocks 2022-11-02 19:13:59 +01:00
Theta-Dev 20e4a35814
[YouTube] Support richGridRenderer on channel pages
YouTube is deploying a new layout on their channel pages, which uses richGridRenderer JSON objects.
2022-11-02 19:01:29 +01:00
AudricV 4cae66f1f9
Merge pull request #946 from chowder/dev
Add ability to identify short-form `StreamInfoItem`s
2022-11-01 12:19:58 +01:00
Tobi eb40bb8458
Merge pull request #959 from FireMasterK/playlist-info-item-uploader
Add uploaderUrl and uploaderVerified to PlaylistInfoItem.
2022-10-31 13:10:33 +01:00
chowder b1a899fd47 Fix null pointer exception 2022-10-31 11:12:23 +00:00
Stypox a4db106a66
Merge pull request #960 from AudricV/yt-workaround-403-errors-android-client
[YouTube] Workaround 403 HTTP errors of ANDROID client streams
2022-10-30 21:53:26 +01:00
Kavin b441910257
Mark uploaderUrl as nullable. 2022-10-30 13:36:04 +00:00
chowder 3fdc0e72cc Line breaks for long docstrings 2022-10-30 13:28:39 +00:00
Kavin 6a256d0631
Add uploader url and verified to PlaylistInfoItem. 2022-10-30 13:00:19 +00:00
Tobi 430504b4b5
Merge pull request #958 from AudricV/yt-playlists-support-new-metadata-format
[YouTube] Support new metadata format of playlists
2022-10-30 12:31:43 +01:00
Kavin f9bd08c649 Address reviews. 2022-10-30 01:25:30 +00:00
Caleb 9282c3c13b Fix exception message for YoutubeStreamInfoItemExtractor#isShortFormContent
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
2022-10-30 01:23:15 +00:00
Caleb c5216f7c12 Update docstring for StreamExtractor#isShortFormContent
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
2022-10-30 01:23:15 +00:00
Caleb 04795fe5d2 Use Stream API for ShortFormContent#testShortFormContent
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
2022-10-30 01:23:15 +00:00
chowder 975b38a0b9 Regenerate test mocks 2022-10-30 01:23:15 +00:00
chowder 4cccd33f3d Implement isShortFormContent for StreamExtractor and StreamInfo 2022-10-30 01:23:15 +00:00
chowder 09544ceb23 Fix tests 2022-10-30 01:23:15 +00:00
chowder 644cc6cd76 Rename test function name 2022-10-30 01:23:15 +00:00
chowder daf5674951 Add ability to identify short-form StreamInfoItems 2022-10-30 01:23:12 +00:00
Tobi 3d314169b9
Merge pull request #943 from TeamNewPipe/fix/sc/comments
[SoundCloud] Fix getting more comments
2022-10-29 22:19:50 +02:00
AudricV 65125a3eb4
[YouTube] Fix YoutubePlaylistExtractorTest.LearningPlaylist
The video count is now returned for this playlist, so it isn't unknown.

The testStreamCount method of this test class asserts now that the stream count
is greater than 40.
2022-10-29 18:12:10 +02:00
AudricV 7258a53225
[YouTube] Support new playlist layout
This new layout doesn't provide author thumbnails and is completely different
for metadata, so the code to get them has been refactored.

The code of learning playlists video count check has been also removed, as it
seems to be not relevant anymore (the video count seems to be returned for
these playlists with both layouts).

Finally, unneeded overrides of subchannel methods, which don't apply to the
YouTube service, have been removed.
2022-10-29 18:12:10 +02:00
AudricV e37e8b358e
[YouTube] Update mocks that need to be updated 2022-10-29 18:10:18 +02:00
AudricV 60e97cd274
[YouTube] Workaround getting streaming URLs returning 403 HTTP response codes
Using the player parameters used to get stories seems to fix the issue, which
affects currently only certain countries such as UK.

This is a workaround and should be fixed in a better way (by changing the
InnerTube additional client used for videos or finding what is now required in
Android player requests).
2022-10-29 17:58:33 +02:00
AudricV 4bc90cd9d8
Merge pull request #957 from AudricV/fix-checkstyle-error
Fix Checkstyle error in YoutubeCommentsInfoItemExtractor
2022-10-29 17:55:33 +02:00
AudricV c230d84df1
Fix Checkstyle error in YoutubeCommentsInfoItemExtractor 2022-10-29 13:24:19 +02:00
Tobi 9ffdd0948b
Merge pull request #945 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.3
Bump spotbugs-annotations from 4.7.2 to 4.7.3
2022-10-18 22:55:24 +02:00
dependabot[bot] 24137bb15f
Bump spotbugs-annotations from 4.7.2 to 4.7.3
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.7.2 to 4.7.3.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.2...4.7.3)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-10-17 09:20:51 +00:00
xz-dev 0ffcb32d9c
[YouTube] Add comment reply count support (#936)
Add comment reply count support for YouTube and introduce `CommentsInfoItem.UNKNOWN_REPLY_COUNT` constant

Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2022-10-15 12:40:06 +02:00
Isira Seneviratne b90a566dd8 Add backport implementation of Locale.forLanguageTag(). 2022-10-12 09:21:39 +05:30
Isira Seneviratne b232c29d22 Use Locale.forLanguageTag(). 2022-10-12 09:21:38 +05:30
TobiGr 4d136599bd [SoundCloud] Fix getting more comments 2022-10-11 15:44:54 +02:00
Tobi a822e91909
Merge pull request #939 from TurtleArmyMc/fix/SoundcloudPlaylistExtractor_track_order
Fix SoundcloudPlaylistExtractor: tracks are in correct order
2022-10-10 22:28:18 +02:00
TobiGr 02810a7db7 Add a comment 2022-10-10 22:22:12 +02:00
Tobi 2d50369c77
Merge pull request #888 from Isira-Seneviratne/EnumMap
Use EnumMap in PatternsHolder.
2022-10-10 12:00:06 +02:00
Tobi 54092fc3c7
Merge pull request #930 from Isira-Seneviratne/Avoid_Comparator_NPE
Avoid possible NullPointerException in MediaCCCRecentKiosk.
2022-10-10 11:43:34 +02:00
Tobi d7c23ef118
Merge pull request #924 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.2
Bump spotbugs-annotations from 4.7.1 to 4.7.2
2022-10-10 11:35:40 +02:00
dependabot[bot] 08339d6b0c
Bump spotbugs-annotations from 4.7.1 to 4.7.2
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.7.1 to 4.7.2.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.1...4.7.2)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-10-10 09:32:17 +00:00
Tobi 5537e84100
Merge pull request #931 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.1
Bump junit-bom from 5.9.0 to 5.9.1
2022-10-10 11:30:53 +02:00
dependabot[bot] 719664593f
Bump junit-bom from 5.9.0 to 5.9.1
Bumps [junit-bom](https://github.com/junit-team/junit5) from 5.9.0 to 5.9.1.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.9.0...r5.9.1)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-10-10 09:25:42 +00:00
TurtleArmyMc bf70d32eb4 Fix SoundcloudPlaylistExtractor: tracks are in correct order 2022-09-30 15:26:25 -04:00
Stypox 5c710da160
Merge pull request #934 from AudricV/fix-yt-throttling-decryption-function-regex
[YouTube] Fix throttling decryption function regex
2022-09-25 13:26:08 +02:00
AudricV 8067c43837
[YouTube] Don't use a specific letter for the decryption function name pattern
Use the same possible characters for variables everywhere, in order to avoid
potential future throttling parameter decryption function name parsing issues
related to the usage of other letter(s) than b.
2022-09-24 21:49:22 +02:00
AudricV abcee87167
[YouTube] Fix throttling parameter decryption function regex
- Quote the function name, as it may contain special regex symbols, such as
  dollar;
- Support multiple lines;
- Use what looks like the end of the function for the end of the regex (this
  part is inspired from yt-dlp throttling parameter decryption regex);
- Move the throttling function body regex into a private and static constant.
2022-09-24 21:28:09 +02:00
ThetaDev 7244be7627
[YouTube] Add JavaScript lexer to parse completely throttling decryption function (#905) 2022-09-24 19:55:46 +02:00
Isira Seneviratne 0e31c86aee Avoid possible NullPointerException in MediaCCCRecentKiosk. 2022-09-21 05:40:07 +05:30
Stypox a99af9bb6e
Merge pull request #927 from Stypox/fix-feed-thumbnail
[YouTube] Return mqdefault thumbnails in fast feed
2022-09-19 08:52:01 +02:00
Kavin 4e14644c41
Ensure comments are parsed with URLs and timestamps. 2022-09-17 16:03:39 +05:30
Stypox 14ef430546
Merge pull request #920 from AudricV/fix-yt-trending-extraction
[YouTube] Fix extraction of the Trending kiosk with the new data model returned
2022-09-14 11:14:13 +02:00
Stypox 93ef73e2f2
[YouTube] Return mqdefault thumbnails in feed
The hqdefault thumbnails, which are used by default, have black bars at the top and at the bottom, and are thus not optimal.
2022-09-13 16:46:54 +02:00
AudricV d8ddeb549c
[YouTube] Support new trending structure and filter recently trending section
This new structure allow us to filter easily Trending shorts and Recently
trending sections.

On the previous one, this Recently trending section is now filtered, by
checking whether sections have a title, which isn't the case for normal trends
contrary to the other ones.

This makes that the extractor returns now only the real 50 "Now" YouTube
trends.

Elements inside arrays are now extracted dynamically instead of only the ones
of the first index, using Java 8's Stream API.

The getInitialPage() method of YoutubeTrendingExtractor can now throw a
ParsingException if no selected tab (corresponding to the one of the trends
type extracted) has been found.

Finally, the licence header has been moved to the top of the file and updated.
2022-09-11 18:52:12 +02:00
AudricV 884da40f65
Merge pull request #926 from AudricV/fix-yt-like-count-extraction
[YouTube] Fix extraction of like count with the new data model
2022-09-11 12:21:38 +02:00
AudricV 58b09cf78b
Merge pull request #925 from Theta-Dev/fix-td-regex
[YouTube] Fix throttling parameter decryption on Android
2022-09-10 23:30:50 +02:00
AudricV 119b9129e2
[YouTube] Fix extraction of like count with the new data model
In the new data model currently A/B tested or deployed, the like count button
data is the same, but its path has been changed.

The extraction of the like count has been also improved, by using the multiple
accessibility data available instead of the only one which was used before.
This accessibility data has been also deprioritized, because it is language
dependent when there is no view.

Like count is always known, even for age-restricted videos, so returning -1 if
the like count extraction fails on this type of videos is not needed and hides
an extraction error: that's the reason why this return has been removed and the
exception will always be thrown, even if a video is age-restricted.
2022-09-10 23:28:38 +02:00
ThetaDev 4905f74700
[YouTube] Fix throttling parameter decryption on Android
Escape the curly brace in the regular expression used to parse the throttling
parameter decryption function to allow its compatibility on Android.
2022-09-10 17:03:39 +02:00
Isira Seneviratne e101d58ae7 Use EnumMap in PatternsHolder. 2022-08-26 08:15:21 +05:30
Stypox 6a858368c8
Merge pull request #912 from Stypox/downgrade-rhino
Downgrade Rhino since it uses a class not available on Android
2022-08-25 10:10:27 +02:00
Stypox fa25a7280f
Downgrade Rhino since it uses a class not available on Android
Reverts #774, see TeamNewPipe/NewPipe#8876
2022-08-25 09:49:13 +02:00
Stypox 25f0ce7305
Merge pull request #911 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.3
Bump jsoup from 1.15.2 to 1.15.3
2022-08-24 14:23:44 +02:00
Stypox d8e10dd3ad
Merge pull request #907 from Isira-Seneviratne/Remove_EmptyString
Remove EMPTY_STRING.
2022-08-24 14:22:55 +02:00
dependabot[bot] 1d7a3a90a4
Bump jsoup from 1.15.2 to 1.15.3
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.15.2 to 1.15.3.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.2...jsoup-1.15.3)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-08-24 09:14:26 +00:00
Isira Seneviratne 41254ae12a Remove annotation. 2022-08-24 06:59:17 +05:30
Isira Seneviratne 943b7c033b Remove EMPTY_STRING. 2022-08-24 06:59:17 +05:30
litetex 20914a6035
Merge pull request #903 from TacoTheDank/updateGradle
Update gradle wrapper to 7.5.1
2022-08-23 16:21:56 +02:00
litetex d6577e5e0b
Merge pull request #882 from litetex/fix-all-tests
Fix all tests
2022-08-23 16:19:27 +02:00
litetex 0a5ad90724
Merge pull request #901 from AudricV/remove-deprecated-code-yt-throttling-decrypter
[YouTube] Remove deprecated throttling decryption code
2022-08-22 17:13:00 +02:00
AudricV 03d9a4fe9d
Fix typos in YoutubeStreamExtractor.tryDecryptUrl 2022-08-21 21:17:03 +02:00
litetex 0beb55a232 Improve ``YoutubeThrottlingDecrypterTest`` 2022-08-21 20:17:05 +02:00
litetex 9e93d6b193 YoutubeThrottlingDecrypter: Check if returned string is a valid JavaScript function 2022-08-21 20:16:45 +02:00
litetex ed0a07af4e YoutubeThrottlingDecrypter: Patch regex 2022-08-21 20:15:53 +02:00
litetex 641a447d9b Updated consent-related mock-data 2022-08-21 18:43:48 +02:00
litetex 8ff7a90f52 Improved consent cookie related constants and documentation 2022-08-21 18:41:40 +02:00
TacoTheDank d8339d4060 Update gradle wrapper to 7.5.1 2022-08-18 15:09:35 -04:00
AudricV cb64a480cd
[YouTube] Remove deprecated code in YoutubeThrottlingDecrypter
YoutubeThrottlingDecrypter is now a non-instantiable final class and non-static
attributes have been removed.

The static attributes of this class have been renamed, in order to respect the
naming specification used.
2022-08-16 15:01:05 +02:00
litetex 5bbea3a8f2 Fix ``YoutubeMixPlaylistExtractorTest`` again
Doesn't work for kurgesagt videos anymore, used music video (copyright free) instead
2022-08-14 15:12:27 +02:00
litetex da06166065 Updated mock data of ``YoutubeSearchExtractorTest$Suggestion`` 2022-08-14 15:12:26 +02:00
litetex 2e36ab1578 Fixed ``YoutubeSearchExtractorTest$Suggestion`` 2022-08-14 15:12:26 +02:00
litetex 2a8a623643 ``YoutubePlaylistLinkHandlerFactoryTest``: Use parameterized tests 2022-08-14 14:48:31 +02:00
litetex 938e69a16a Fixed ``YoutubePlaylistLinkHandlerFactoryTest`` 2022-08-14 14:48:30 +02:00
litetex 844de3e378 Fix ``YoutubeStreamLinkHandlerFactoryTest `` and parameterized the tests
used code from #833
2022-08-14 14:48:30 +02:00
litetex c1a72b8240 Use Android Studios code style 2022-08-14 14:48:29 +02:00
litetex fc27b8a5b8 Use preexisting ``ContentNotAvailableException`` 2022-08-14 14:48:29 +02:00
Stypox 17bad6cedf Fix test about exception type: NPE, not IllegalArgment
The relevant change was made in #877
2022-08-14 14:48:28 +02:00
litetex ecfc370685 Fixed all YTMixPlaylists
Added option to choose if you want to consent or not - currently this is done by a static variable in ``YoutubeParsingHelper`` - may not be the best long-term solution but for now the tests work again (in EU countries) 🥳
2022-08-14 14:48:27 +02:00
litetex 2b6fe294b2 Fixed ``YoutubeMusicSearchExtractorTest``
The renderers and queries got changed.
2022-08-14 14:48:27 +02:00
litetex d6586da614 Failing test works locally without any problems. Reformatted it a bit. 2022-08-14 14:48:27 +02:00
litetex bf3ae5e679 Fixed ``SoundcloudStreamLinkHandlerFactoryTest``
* Removed the unnecessary ``public``
* Use parameterized tests
* One song got removed (which caused the test failure), replaced it with another song
2022-08-14 14:48:27 +02:00
litetex 504f81036e Fixed ``PeertubePlaylistExtractorTest``
Also removed the unnecessary ``public`` and reformatted the code a bit
2022-08-14 14:48:26 +02:00
litetex e7c12258f4 Fixed ``BandcampSearchExtractorTest``
Also removed the unnecessary ``public``
2022-08-14 14:48:26 +02:00
litetex 9e8724df4d Fixed ``YoutubeStreamExtractorLivestreamTest``
The "Lofi Girl"-stream got interrupted by a copyright strike and had to be restarted. Because of this a new id is now used.
2022-08-14 14:48:26 +02:00
Tobi 76aad92fa5
Merge pull request #890 from AudricV/yt-clients-update-improvements-and-extraction-fixes
[YouTube] Update clients version and improve extraction of API key and client version of WEB client
2022-08-12 23:32:41 +02:00
AudricV 472f5d9e9c
[YouTube] Update mocks 2022-08-12 19:20:32 +02:00
AudricV 7bdca33a87
[YouTube] Ensure that an additional player response is the correct one
If YouTube detect that requests come from a third party client, they may
replace the real player response by another one of a video saying that this
content is not available on this app and to watch it on the latest version of
YouTube. We can detect this by checking whether the video ID of the player
response returned is the same as the one requested by the extractor.
2022-08-12 19:20:31 +02:00
AudricV c82317e318
[YouTube] Spoof more mobile clients
Additional parameters have been added to the player requests of ANDROID and IOS
clients:

- for both clients: osName and osVersion: their respective values are:
  - for the ANDROID one: Android and 12;
  - for the IOS one: iOS and 15.6.0.19G71.
- for the ANDROID client: androidTargetSdkVersion, with the Android SDK version
  corresponding to the Android version used in the player requests of this
  client. This parameter is now required with this client to be sure to get a
  correct player response, otherwise, the one of a video saying that this
  content is not available in this app and to watch it with the latest version
  of YouTube can be returned instead;
- for the IOS client: deviceMake, with Apple as its value.

The iOS version sent in the IOS client player requests has been also updated to
the version 15.6 of the OS.

Finally, a comment about the requirement to use the signature timestamp from
the player JavaScript base file for HTML5 player requests on videos with
obfuscated URLs has been added and replaces a previous one which may be not
true.
2022-08-12 19:20:31 +02:00
AudricV d0549a5a52
[YouTube] Update client versions and use a real version for the iOS client
The iOS version can be got easily in fact, by looking at the What's New section of the App Store' app page.
2022-08-12 19:20:31 +02:00
AudricV d7e678aca2
[YouTube] Improve WEB client version and API key HTML extraction
Common code in WEB client version HTML extraction has been deduplicated, usage of the Java 8 Stream API has been made and initial data fallback has been used as a last resort.
This means that the client version extraction from regexes will be used before this fallback, as it doesn't contain the full client version.
This can be used as a way to fingerprint the extractor, even if it seems to be not the case.
2022-08-12 19:20:30 +02:00
AudricV 6a885ef5ab
Merge pull request #891 from Theta-Dev/fix-throttling-decrypter
[YouTube] Fix extraction of more complex nsig functions
2022-08-12 18:24:21 +02:00
AudricV 5b548340e8
[YouTube] Catch any exception in YoutubeThrottlingDecrypter.apply and improve docs
This will prevent any future extractor break due to decryption failure, like it was excepted to be the case before.

Some documentation about the throttling decryption has been also improved.
2022-08-12 17:49:36 +02:00
ThetaDev 52ded6e3d7
Handle curly braces inside strings in StringUtils.matchToClosingParenthesis
This is required to extract fully more complex YouTube nsig functions.
2022-08-12 16:32:00 +02:00
Stypox d12003651b
Merge pull request #886 from Isira-Seneviratne/toArray_improvements
Make improvements to methods using toArray().
2022-08-06 22:34:23 +02:00
Isira Seneviratne 7daca10a06 Make improvements to methods using toArray(). 2022-08-06 05:21:12 +05:30
Stypox 2906be22af
Merge pull request #881 from Isira-Seneviratne/String_join
Use String.join() and Collectors.joining().
2022-08-04 12:09:33 +02:00
Stypox 4ddb96a86f
Merge pull request #883 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.9.1
Bump gson from 2.9.0 to 2.9.1
2022-08-04 11:20:41 +02:00
Isira Seneviratne 64771c5712 Use String.join() and Collectors.joining(). 2022-08-04 05:18:13 +05:30
Stypox fc8b5ebbc6
Merge pull request #878 from Isira-Seneviratne/Use_Collections
Use Collections methods.
2022-08-03 22:50:41 +02:00
Stypox 4a4939d89c
Merge pull request #877 from Isira-Seneviratne/Use_Objects_requireNonNull
Use Objects.requireNonNull().
2022-08-03 22:45:35 +02:00
Stypox c336bd58a5
Merge pull request #879 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.0
Bump junit-bom from 5.8.2 to 5.9.0
2022-08-03 21:06:20 +02:00
dependabot[bot] 325af31e5f
Bump gson from 2.9.0 to 2.9.1
Bumps [gson](https://github.com/google/gson) from 2.9.0 to 2.9.1.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.9.0...gson-parent-2.9.1)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-08-01 09:26:14 +00:00
dependabot[bot] d905636021
Bump junit-bom from 5.8.2 to 5.9.0
Bumps [junit-bom](https://github.com/junit-team/junit5) from 5.8.2 to 5.9.0.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.8.2...r5.9.0)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-07-27 09:20:12 +00:00
Isira Seneviratne 1af6b8eedb Use Collections.singletonList(). 2022-07-27 07:35:57 +05:30
Isira Seneviratne ff60e05c76 Use Collections.singletonMap(). 2022-07-27 07:35:57 +05:30
Isira Seneviratne 682a4263e5 Use Objects.requireNonNull(). 2022-07-27 06:55:26 +05:30
Stypox 8c5f014a6f
Merge pull request #869 from mhmdanas/add-workflow-permissions
Use minimum required permissions for GitHub workflows
2022-07-13 18:58:32 +02:00
Stypox 954a294e27
Merge pull request #870 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.2
Bump jsoup from 1.15.1 to 1.15.2
2022-07-13 17:48:06 +02:00
Mohammed Anas f57049d2c0
Use temurin instead of adopt (#868)
See
https://blog.adoptopenjdk.net/2021/08/goodbye-adoptopenjdk-hello-adoptium/.
2022-07-10 18:38:08 +03:00
Stypox 5bd7bf20cc
Merge pull request #867 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.1
Bump spotbugs-annotations from 4.7.0 to 4.7.1
2022-07-06 14:23:44 +02:00
Stypox 5ab74b3631
Merge pull request #857 from FireMasterK/video-title
Get original untranslated title for YouTube
2022-07-06 10:26:45 +02:00
dependabot[bot] 122365005a
Bump jsoup from 1.15.1 to 1.15.2
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.15.1 to 1.15.2.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.1...jsoup-1.15.2)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-07-04 09:38:16 +00:00
mhmdanas 414186cff5 Use minimum required permissions for GitHub workflows
This reduces the attack surface if the workflows are ever compromised.
2022-07-03 23:55:28 +03:00
dependabot[bot] e9b4be3e3c
Bump spotbugs-annotations from 4.7.0 to 4.7.1
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.7.0 to 4.7.1.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.0...4.7.1)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-06-28 09:15:07 +00:00
Stypox 5219a705ba
Merge pull request #864 from AudricV/fetch-android-client-for-ended/post-livestreams
[YouTube] Fetch the ANDROID client for ended/post livestreams
2022-06-24 14:55:20 +02:00
AudricV 090debd83b
[YouTube] Fetch the ANDROID client for ended/post livestreams
The ANDROID client was only fetched for video contents, where it can be useful on ended/post livestreams, if the n parameter of the WEB client cannot be decrypted, to avoid throttling issues (because the WEB client was only used before for ended/post livestreams).

It also provides an exclusive 48kbps M4A audio format in the adaptiveFormats array of the JSON player response, like other mobile clients (which can be also extracted from the response of the DASH manifest URL returned into the WEB client player's response, but the DASH manifest is not used by the extractor).

A note about non-fatality of fetching or parsing issues of the ANDROID and IOS clients has been added.
2022-06-21 18:53:49 +02:00
litetex a26bcc55c4
Merge pull request #845 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.0
Bump spotbugs-annotations from 4.6.0 to 4.7.0
2022-06-19 15:36:55 +02:00
dependabot[bot] 424eb1c559
Bump spotbugs-annotations from 4.6.0 to 4.7.0
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.6.0 to 4.7.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.6.0...4.7.0)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-06-19 13:29:18 +00:00
litetex 2712c3d788
Merge pull request #847 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.1
Bump jsoup from 1.14.3 to 1.15.1
2022-06-19 15:28:43 +02:00
dependabot[bot] 281d2b9f81
Bump jsoup from 1.14.3 to 1.15.1
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.14.3 to 1.15.1.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.14.3...jsoup-1.15.1)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-06-19 13:15:40 +00:00
litetex f775155d25
Merge pull request #846 from litetex/remove-unused-methods
Remove unused methods
2022-06-19 15:12:15 +02:00
litetex 9d625dd75f
Merge pull request #860 from gliptak/patch-1
Correct PeertubePlaylistExtractorTest unit tests
2022-06-19 15:09:27 +02:00
Gábor Lipták 28c9340d69
Correct unit tests 2022-06-18 11:18:38 -04:00
Stypox 1b51eab664
Merge pull request #859 from AudricV/delivery-methods-fixes-and-improvements
Fix extraction of some properties in ItagItems for YouTube livestreams and post-live streams and remove completely SoundCloud HLS workaround
2022-06-17 15:46:13 +02:00
AudricV 301a795ed3
[SoundCloud] Remove completely workaround for HLS streams
SoundCloud is currently removing this workaround completely, so there is no need to keep it, because it impacts the loading time (a HLS playlist was downloaded and parsed).
2022-06-16 12:12:54 +02:00
AudricV e960a417ec
[YouTube] Fix extraction of fps, audioSampleRate and audioChannels fields for ItagItems of live streams and post live streams
These values were only set before for video streams.

A fallback for the audio channels count has been added, in order to prevent exceptions when generating DASH manifests of audio streams: the fallback value is 2, because most audio streams on YouTube have 2 audio channels.
2022-06-16 12:12:54 +02:00
Stypox c8a77da2ab
Merge pull request #810 from TiA4f8R/delivery-methods-v2
Support delivery methods other than progressive HTTP
2022-06-02 22:44:24 +02:00
Kavin 7635aeed2c
Get original untranslated title for YouTube. 2022-06-02 09:57:52 +01:00
TiA4f8R 287d1dfd63
[SoundCloud] Use the HLS delivery method for all streams and extract only a single stream URL from HLS manifest for MP3 streams
SoundCloud broke the workaround used to get a single file from HLS manifests for Opus manifests, but it still works for MP3 ones.

The code has been adapted to prevent an unneeded request (the one to the Opus HLS manifest) and the HLS delivery method is now used for SoundCloud MP3 and Opus streams, plus the progressive one (for tracks which have a progressive stream (MP3) and for the ones which doesn't have one, it is still used by trying to get a progressive stream, using the workaround).

Streams extraction has been also moved to Java 8 Stream's API and the relevant test has been also updated.
2022-05-29 19:08:18 +02:00
Stypox b3c620f0d8
Apply code review and Streams rework 2022-05-28 12:00:58 +02:00
Stypox d652e05874
[MediaCCC] Fix comments about containsSimilarStream 2022-05-28 12:00:58 +02:00
Stypox 044639c32b
Solve some review comments 2022-05-28 12:00:57 +02:00
litetex c33d392958
Fixed typo XEE → XXE (Xml eXternal Entity attack)
See also:
https://en.wikipedia.org/wiki/XML_external_entity_attack
https://owasp.org/www-community/vulnerabilities/XML_External_Entity_(XXE)_Processing
2022-05-28 12:00:57 +02:00
TiA4f8R fffbbee7f3
[YouTube] Add missing Nonnull annotations in getCache method of YouTube DASH manifest creators 2022-05-28 12:00:56 +02:00
TiA4f8R f7b1515290
[YouTube] Refactor DASH manifests creation
Move DASH manifests creation into a new subpackage of the YouTube package, dashmanifestcreators.
This subpackage contains:

- CreationException, exception extending Java's RuntimeException, thrown by manifest creators when something goes wrong;
- YoutubeDashManifestCreatorsUtils, class which contains all common methods and constants of all or a part of the manifest creators;
- a manifest creator has been added per delivery type of YouTube streams:
  - YoutubeProgressiveDashManifestCreator, for progressive streams;
  - YoutubeOtfDashManifestCreator, for OTF streams;
  - YoutubePostLiveStreamDvrDashManifestCreator, for post-live DVR streams (which use the live delivery method).

Every DASH manifest creator has a getCache() static method, which returns the ManifestCreatorCache instance used to cache results.

DeliveryType has been also extracted from the YouTube DASH manifest creators part of the extractor and moved to the YouTube package.

YoutubeDashManifestCreatorTest has been updated and renamed to YoutubeDashManifestCreatorsTest, and YoutubeDashManifestCreator has been removed.

Finally, several documentation and exception messages fixes and improvements have been made.
2022-05-28 12:00:56 +02:00
TiA4f8R f17f7b9842
Apply requested changes in YoutubeParsingHelper 2022-05-28 12:00:55 +02:00
TiA4f8R 301b9fa024
Remove hashCode and equals methods overrides of Stream classes 2022-05-28 12:00:55 +02:00
TiA4f8R 2f3920c648
[YouTube] Return approxDurationMs value from YouTube's player response in ItagItems generated
This change allows to build DASH manifests using YoutubeDashManifestCreator with the real duration of streams and prevent potential cuts of the end of progressive streams, because the duration in YouTube's player response is in seconds and not milliseconds.
2022-05-28 12:00:54 +02:00
TiA4f8R 4158fc46a0
[Bandcamp] Fix regression of Opus radio streams extraction
When moving opus-lo into a constant, opus-lo was renamed to opus_lo and was only used if no MP3 stream was available (which was not the case before the changes in BandcampRadioStreamExtractor related to the addition of the support of all delivery methods), so these changes removed the ability to get Opus streams of Bandcamp radios.

This commit reverts this unwanted change.
2022-05-28 12:00:54 +02:00
TiA4f8R 54d323c2ae
Fix Checkstyle issue in YoutubeDashManifestCreator 2022-05-28 12:00:53 +02:00
Stypox 2321822844
Rename Stream's baseUrl to manifestUrl 2022-05-28 12:00:53 +02:00
Stypox cfc13f4a6f
[YouTube] Reduce exception generation code and move several attributes of MPD documents into constants
Rename YoutubeDashManifestCreationException to CreationException.
Also use these constants in YoutubeDashManifestCreatorTest.
2022-05-28 12:00:53 +02:00
Stypox 00bbe5eb4d
[YouTube] Make exception messages smaller 2022-05-28 12:00:52 +02:00
Stypox 4da05afe11
[YouTube] Inline collectSegmentsData in YoutubeDashManifestCreator 2022-05-28 12:00:52 +02:00
Stypox 3708ab9ed5
[YouTube] Refactor YoutubeDashManifestCreator
- Remove all of the methods used to access caches and replace them with three caches getters
- Rename caches to shorter and more meaningful names
- Remove redundant @throws tags that just say "if this method fails to do what it should do", which is obvious
2022-05-28 12:00:51 +02:00
Stypox 5c83409039
[YouTube] Rewrite manifest test and rename long methods 2022-05-28 12:00:51 +02:00
Stypox 8226fd044f
[YouTube] Suppress Sonar security warning for XEE 2022-05-28 12:00:50 +02:00
Stypox ba68b8c014
[YouTube] Secure DashManifestCreator against XEE attack
Also remove duplicate lines from javadoc comments, otherwise checkstyle complaints.
XEE prevention is inspired from https://github.com/ChuckerTeam/chucker/pull/201/files
For an explanation of XEE see https://rules.sonarsource.com/java/RSPEC-2755, though the solution reported there causes crashes on Android
2022-05-28 12:00:50 +02:00
Stypox 159d05c91b
Remove unused DashMpdParser (but kept in git history) 2022-05-28 12:00:49 +02:00
Stypox 50272db946
Apply reviews: improve comments, remove FILE, remove Stream#equals(Stream) 2022-05-28 12:00:49 +02:00
TiA4f8R 07b045f20d
[YouTube] Support the iOS client in YoutubeDashManifestCreator and decrypt again the n parameter, if present, for all clients
This commits reverts a new behavior introduced in this branch, which only applied the decryption if needed on streams from the WEB client.
Also fix rebase issues and documentations style in YoutubeDashManifestCreator.
2022-05-28 12:00:48 +02:00
TiA4f8R 436ddde29f
Use assertThrows in YoutubeDashManifestCreatorTest 2022-05-28 12:00:48 +02:00
TiA4f8R d64d7bbd01
Move ManifestCreatorCache tests to a separate class and remove override of equals and hashCode methods in ManifestCreatorCache
These methods don't need to be overriden, as they are not excepted to be used in collections.
Also improve the toString method of this class, which contains also now clearFactor and maximumSize attributes and for each operations.
2022-05-28 12:00:48 +02:00
TiA4f8R 2fb1a412a6
Fix Checkstyle issues, revert resolution string changes for YouTube video streams and don't return the rn parameter in DASH manifests
This parameter is still used to get the initialization sequence of OTF and POST-live streams, but is not returned anymore in the manifests.
It has been removed in order to avoid fingerprinting based on the number sent (e.g. when starting to play a stream close to the end and using 123 as the request number where it should be 1) and should be added dynamically by clients in their requests.
The relevant test has been also updated.

Checkstyle issues in YoutubeDashManifestCreator have been fixed, and the changes in the resolution string returned for video streams in YoutubeStreamExtractor have been reverted, as they create issues on NewPipe right now.
2022-05-28 12:00:47 +02:00
TiA4f8R f61e2092a1
[YouTube] Return a copy of the hardcoded ItagItem instead of returning the reference to the hardcoded one in ItagItem.getItag
To do so, a copy constructor has been added in the class.

This fixes, for instance, an issue in NewPipe, in which the ItagItem values where not the ones corresponsing to a stream but to another, when generating DASH manifests.
2022-05-28 12:00:47 +02:00
TiA4f8R aa4c10e751
Improve documentation and adress most of the requested changes
Also fix some issues in several places, in the code and the documentation.
2022-05-28 12:00:46 +02:00
TiA4f8R 6985167e63
Add tests for YoutubeDashManifestCreator and ManifestCreatorCache
The test added in YoutubeDashManifestCreator uses a video of the Creative Commons channel, licenced under the Creative Commons Attribution licence (reuse allowed).
Also remove public keywords of tests in UtilsTest, as suggested by SonarLint, because they are not needed with Junit 5.
2022-05-28 12:00:46 +02:00
TiA4f8R f6ec7f9a61
Update DefaultStreamExtractorTest and SoundcloudStreamExtractorTest to support changes made in Stream classes 2022-05-28 12:00:45 +02:00
TiA4f8R 7477ed0f3d
[YouTube] Add ability to generate manifests of progressive, OTF and post live streams
A new class has been added to do so: YoutubeDashManifestCreator.
It relies on a new class: ManifestCreatorCache, to cache the content, which relies on a new pair class named Pair.
Results are cached and there is a cache per delivery type, on which cache limit, clear factor, clearing and resetting can be applied to each cache and to all caches.
Look at code changes for more details.
2022-05-28 12:00:45 +02:00
TiA4f8R a857684442
Apply changes in YoutubeStreamExtractor
Extract post live DVR streams as post live streams instead of live streams.

A new class has been in order to improve code: ItagInfo, which stores an itag, the content (URL) extracted and if its an URL or not.
A functional interface has been added in order to abstract the stream building: StreamBuilderHelper.
Also add the cver parameter added by the desktop web client on the corresponding streams (a new method has been added in YoutubeParsingHelper to check this and another for Android streams).

Some code in these classes has been also refactored/improved/optimized.
2022-05-28 12:00:44 +02:00
TiA4f8R 4330b5f7be
Add POST_LIVE_STREAM and POST_LIVE_AUDIO_STREAM stream types
This allows the extractor to determine if a content is an ended audio or video livestream.
2022-05-28 12:00:43 +02:00
TiA4f8R 881969f1da
Apply changes in all StreamExtractors except YouTube's one and fix extraction of PeerTube audio streams as video streams
Some code in these classes has been also refactored/improved/optimized.
Also fix the extraction of PeerTube audio streams as video streams, which are now returned as audio streams.
2022-05-28 12:00:43 +02:00
TiA4f8R d5f3637fc3
[YouTube] Return more values returned inside the ItagItems of the player response and deprecate use of public audio and video fields
These fields can be now replaced by a getter and a setter.

New fields have been added and will allow the creation of DASH manifests for OTF and ended livestreams. There are:
- contentLength;
- approxDurationMs;
- targetDurationSec;
- sampleRate;
- audioChannels.
2022-05-28 12:00:42 +02:00
TiA4f8R 7c67d46e09
Move DashMpdParser to the YouTube package and fix extraction of streams
DashMpdParser is only working with YouTube streams, as it uses the ItagItem class.
Also update creation of AudioStreams and VideoStreams objects.
2022-05-28 12:00:41 +02:00
TiA4f8R ad993b920f
Remove fetching of the DASH manifest extracted when getting information of a content with StreamInfo
DashMpdParser is only working with YouTube streams, as it uses the ItagItem class.

Also improve code and comments of StreamInfo (especially final use where possible).
2022-05-28 12:00:41 +02:00
TiA4f8R 2f061b8dbd
Add support of other delivery methods than progressive HTTP in Stream classes
Stream constructors are now private and streams can be constructed with new Builder classes per stream class. This change has been made to prevent creating and using several constructors in stream classes.

Some default cases have been also added in these Builder classes, so not everything has to be set, depending of the service and the content.
2022-05-28 12:00:27 +02:00
Stypox 1dc80957d8
Merge pull request #848 from TiA4f8R/fix_yt_music_website_extraction_eu
[YouTube] Fix extracting YouTube Music client version and key in the EU
2022-05-22 22:05:05 +02:00
TiA4f8R c34b5e3a8b
[YouTube] Fix extraction of YouTube Music client version and API key when using YouTube Music's website in EU
Google returns now the consent page of YouTube for YouTube Music in EU, which can be also avoided by adding the ucbcb parameter to the URL with the value 1 ("?ucbcb=1").
2022-05-15 11:20:06 +02:00
litetex 2015eb374a Removed more unused methods 2022-05-09 21:05:03 +02:00
litetex f69b0ff77b Remove unused methods 2022-05-09 20:59:25 +02:00
Stypox 23fa31a1ec
Merge pull request #843 from TiA4f8R/fix-checkstyle-issues
Remove Checkstyle suppressions file and fix Checkstyle issues
2022-05-02 22:07:27 +02:00
TiA4f8R 3c3cd78676
Remove Checkstyle suppressions file and fix Checkstyle issues introduced in 24e8399 and 8c1041d
The Checkstyle suppressions file is now replaced by // CHECKSTYLE:OFF and // CHECKSTYLE:ON comments.
2022-05-02 21:51:25 +02:00
Stypox 2e1c5c119d
Merge pull request #822 from Stypox/more-refactors
More refactors
2022-05-02 19:03:54 +02:00
Stypox 598ebb92ea
Merge pull request #839 from TeamNewPipe/bandcamp/extract-length
Bandcamp: extract stream length
2022-05-02 15:49:41 +02:00
litetex 5db4d1faf3
Merge pull request #782 from litetex/cleanup-yt-stream-extractor
Cleanup of ``YoutubeStreamExtractor`` and some related classes
2022-05-01 16:44:11 +02:00
litetex fe30eb43a9 Cleanup ``YoutubeStreamExtractor`` and some related classes
* Fixed obvious sonar(lint) warnings
* Abstracted some code (get*Streams)
* Used some new lines to make code better readable
* Chopped down brace-jungle in some methods
* Use StandardCharset (Java 8 4tw)
2022-05-01 16:39:07 +02:00
Stypox c2b5370517
Apply suggestions: improve switch and use EMPTY_STRING 2022-04-30 16:39:51 +02:00
Stypox 7c78c39230
Merge pull request #821 from litetex/cleanup-TimeAgoParser-java
Cleanup ``TimeAgoParser``
2022-04-30 16:20:09 +02:00
Fynn Godau c38c016de5 Bandcamp: extract stream length 2022-04-24 21:24:19 +02:00
Stypox 8c1041def6
Add @ null annotations where Android Studio suggested it
That is, basically where the overriding function was missing an annotation from the base method.

Also apply renaming of emptyDescription to EMPTY_DESCRIPTION
2022-03-26 22:07:14 +01:00
Stypox adbbdc7a5b
[YouTube] Fix regex warning: use ' {2}' instead of ' ' 2022-03-26 22:07:14 +01:00
Stypox 24e83997b4
[Bandcamp] Add Java 8 streams 2022-03-26 22:07:12 +01:00
Stypox 349990fd48
Fix redundant escape \\ in regex in Utils 2022-03-26 22:01:30 +01:00
litetex 3bf7aa3762 Cleanup ``TimeAgoParser`` 2022-03-26 21:09:31 +01:00
695 changed files with 100501 additions and 42130 deletions

View File

@ -10,21 +10,23 @@ on:
- master
pull_request:
permissions:
contents: read
jobs:
build-and-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: set up JDK 8
uses: actions/setup-java@v3
- name: set up JDK 11
uses: actions/setup-java@v4
with:
java-version: '8'
distribution: 'adopt'
java-version: '11'
distribution: 'temurin'
- name: Cache Gradle dependencies
uses: actions/cache@v3
uses: actions/cache@v4
with:
path: ~/.gradle/caches
key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle') }}
@ -33,7 +35,7 @@ jobs:
# See gradle file for difference between downloaders
- name: Build and run Tests
run: |
if [[ $GITHUB_EVENT_NAME == 'schedule' ]]; then
if [[ "$GITHUB_EVENT_NAME" == 'schedule' ]]; then
echo running with real downloader
./gradlew check --stacktrace -Ddownloader=REAL
else
@ -42,7 +44,7 @@ jobs:
fi
- name: Upload test reports when failure occurs
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
if: failure()
with:
name: NewPipeExtractor-test-reports

View File

@ -5,20 +5,24 @@ on:
branches:
- master
permissions:
# The generated docs are written to the `gh-pages` branch.
contents: write
jobs:
build-and-deploy-docs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: set up JDK 8
uses: actions/setup-java@v3
- name: set up JDK 11
uses: actions/setup-java@v4
with:
java-version: '8'
distribution: 'adopt'
java-version: '11'
distribution: 'temurin'
- name: Cache Gradle dependencies
uses: actions/cache@v3
uses: actions/cache@v4
with:
path: ~/.gradle/caches
key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle') }}

View File

@ -12,8 +12,16 @@ If you're using Gradle, you could add NewPipe Extractor as a dependency with the
1. Add `maven { url 'https://jitpack.io' }` to the `repositories` in your `build.gradle`.
2. Add `implementation 'com.github.TeamNewPipe:NewPipeExtractor:INSERT_VERSION_HERE'` to the `dependencies` in your `build.gradle`. Replace `INSERT_VERSION_HERE` with the [latest release](https://github.com/TeamNewPipe/NewPipeExtractor/releases/latest).
3. If you are using tools to minimize your project, make sure to keep the files below, by e.g. adding the following lines to your proguard file:
```
## Rules for NewPipeExtractor
-keep class org.schabi.newpipe.extractor.timeago.patterns.** { *; }
-keep class org.mozilla.javascript.** { *; }
-keep class org.mozilla.classfile.ClassFileWriter
-dontwarn org.mozilla.javascript.tools.**
```
**Note:** To use NewPipe Extractor in projects with a `minSdkVersion` below 26, [API desugaring](https://developer.android.com/studio/write/java8-support#library-desugaring) is required.
**Note:** To use NewPipe Extractor in Android projects with a `minSdk` below 26, [API desugaring](https://developer.android.com/studio/write/java8-support#library-desugaring) is required. If the `minSdk` is below 19, the `desugar_jdk_libs_nio` artifact is required, which requires Android Gradle Plugin (AGP) version 7.4.0.
### Testing changes
@ -50,7 +58,7 @@ The following sites are currently supported:
[![GNU GPLv3 Image](https://www.gnu.org/graphics/gplv3-127x51.png)](https://www.gnu.org/licenses/gpl-3.0.en.html)
NewPipe is Free Software: You can use, study share and improve it at your
NewPipe Extractor is Free Software: You can use, study share and improve it at your
will. Specifically you can redistribute and/or modify it under the terms of the
[GNU General Public License](https://www.gnu.org/licenses/gpl.html) as
published by the Free Software Foundation, either version 3 of the License, or

View File

@ -5,10 +5,10 @@ allprojects {
compileJava.options.encoding = 'UTF-8'
compileTestJava.options.encoding = 'UTF-8'
sourceCompatibility = 1.8
targetCompatibility = 1.8
sourceCompatibility = JavaVersion.VERSION_11
targetCompatibility = JavaVersion.VERSION_11
version 'v0.22.1'
version 'v0.23.1'
group 'com.github.TeamNewPipe'
repositories {
@ -28,9 +28,9 @@ allprojects {
ext {
nanojsonVersion = "1d9e1aea9049fc9f85e68b43ba39fe7be1c1f751"
spotbugsVersion = "4.6.0"
junitVersion = "5.8.2"
checkstyleVersion = "9.3" // do not use latest version (10.0) as it requires compile JDK 11
spotbugsVersion = "4.8.3"
junitVersion = "5.10.2"
checkstyleVersion = "10.4"
}
}

View File

@ -20,12 +20,6 @@
<property name="fileNamePattern" value="module\-info\.java$"/>
</module>
<!-- https://checkstyle.org/config_filters.html#SuppressionFilter -->
<module name="SuppressionFilter">
<property name="file" value="${config_loc}/suppressions.xml" />
<property name="optional" value="true"/>
</module>
<!-- Checks that a package-info.java file exists for each package. -->
<!-- See https://checkstyle.org/config_javadoc.html#JavadocPackage -->
<!--<module name="JavadocPackage"/>-->
@ -68,6 +62,8 @@
<module name="SuppressWarningsFilter" />
<module name="SuppressWithPlainTextCommentFilter"/>
<module name="TreeWalker">
<!-- Checks for Javadoc comments. -->
<!-- See https://checkstyle.org/config_javadoc.html -->
@ -100,7 +96,15 @@
<!-- Checks for imports -->
<!-- See https://checkstyle.org/config_import.html -->
<module name="AvoidStarImport"/>
<module name="IllegalImport"/> <!-- defaults to sun.* packages -->
<module name="IllegalImport"> <!-- defaults to sun.* packages -->
<property name="illegalClasses" value="
org.jetbrains.annotations.Nullable,
org.jetbrains.annotations.NotNull,
androidx.annotation.Nullable,
androidx.annotation.NonNull,
io.reactivex.rxjava3.annotations.NonNull,
io.reactivex.rxjava3.annotations.Nullable" />
</module>
<module name="RedundantImport"/>
<module name="UnusedImports"/>

View File

@ -1,15 +0,0 @@
<?xml version="1.0"?>
<!DOCTYPE suppressions PUBLIC
"-//Checkstyle//DTD SuppressionFilter Configuration 1.2//EN"
"https://checkstyle.org/dtds/suppressions_1_2.dtd">
<suppressions>
<!-- Use @SuppressWarnings("...") if it is possible, only use this file if it is not -->
<suppress checks="LineLength"
files="BandcampExtractorHelper.java"
lines="54"/>
<suppress checks="LineLength"
files="ItagItem.java"
lines="19"/>
</suppressions>

View File

@ -26,10 +26,12 @@ dependencies {
implementation project(':timeago-parser')
implementation "com.github.TeamNewPipe:nanojson:$nanojsonVersion"
implementation 'org.jsoup:jsoup:1.14.3'
implementation 'org.mozilla:rhino:1.7.14'
implementation 'org.jsoup:jsoup:1.17.2'
implementation "com.github.spotbugs:spotbugs-annotations:$spotbugsVersion"
implementation 'org.nibor.autolink:autolink:0.10.0'
// do not upgrade to 1.7.14, since in 1.7.14 Rhino uses the `SourceVersion` class, which is not
// available on Android (even when using desugaring), and `NoClassDefFoundError` is thrown
implementation 'org.mozilla:rhino:1.7.13'
checkstyle "com.puppycrawl.tools:checkstyle:$checkstyleVersion"
@ -39,5 +41,5 @@ dependencies {
testImplementation 'org.junit.jupiter:junit-jupiter-params'
testImplementation "com.squareup.okhttp3:okhttp:3.12.13"
testImplementation 'com.google.code.gson:gson:2.9.0'
testImplementation 'com.google.code.gson:gson:2.10.1'
}

View File

@ -0,0 +1,211 @@
package org.schabi.newpipe.extractor;
import javax.annotation.Nonnull;
import java.io.Serializable;
import java.util.Objects;
/**
* Class representing images in the extractor.
*
* <p>
* An image has four properties: its URL, its height, its width and its estimated quality level.
* </p>
*
* <p>
* Depending of the services, the height, the width or both properties may be not known.
* Implementations <b>must use</b> the relevant unknown constants in this case
* ({@link #HEIGHT_UNKNOWN} and {@link #WIDTH_UNKNOWN}), to ensure properly the lack of knowledge
* of one or both of these properties to extractor clients.
* </p>
*
* <p>
* They should also respect the ranges defined in the estimated image resolution levels as much as
* possible, to ensure consistency to extractor clients.
* </p>
*/
public final class Image implements Serializable {
/**
* Constant representing that the height of an {@link Image} is unknown.
*/
public static final int HEIGHT_UNKNOWN = -1;
/**
* Constant representing that the width of an {@link Image} is unknown.
*/
public static final int WIDTH_UNKNOWN = -1;
@Nonnull
private final String url;
private final int height;
private final int width;
@Nonnull
private final ResolutionLevel estimatedResolutionLevel;
/**
* Construct an {@link Image} instance.
*
* @param url the URL to the image, which should be not null or empty
* @param height the image's height
* @param width the image's width
* @param estimatedResolutionLevel the image's estimated resolution level, which must not be
* null
* @throws NullPointerException if {@code estimatedResolutionLevel} is null
*/
public Image(@Nonnull final String url,
final int height,
final int width,
@Nonnull final ResolutionLevel estimatedResolutionLevel)
throws NullPointerException {
this.url = url;
this.height = height;
this.width = width;
this.estimatedResolutionLevel = Objects.requireNonNull(
estimatedResolutionLevel, "estimatedResolutionLevel is null");
}
/**
* Get the URL of this {@link Image}.
*
* @return the {@link Image}'s URL.
*/
@Nonnull
public String getUrl() {
return url;
}
/**
* Get the height of this {@link Image}.
*
* <p>
* If it is unknown, {@link #HEIGHT_UNKNOWN} is returned instead.
* </p>
*
* @return the {@link Image}'s height or {@link #HEIGHT_UNKNOWN}
*/
public int getHeight() {
return height;
}
/**
* Get the width of this {@link Image}.
*
* <p>
* If it is unknown, {@link #WIDTH_UNKNOWN} is returned instead.
* </p>
*
* @return the {@link Image}'s width or {@link #WIDTH_UNKNOWN}
*/
public int getWidth() {
return width;
}
/**
* Get the estimated resolution level of this image.
*
* <p>
* If it is unknown, {@link ResolutionLevel#UNKNOWN} is returned instead.
* </p>
*
* @return the estimated resolution level, which is never {@code null}
* @see ResolutionLevel
*/
@Nonnull
public ResolutionLevel getEstimatedResolutionLevel() {
return estimatedResolutionLevel;
}
/**
* Get a string representation of this {@link Image} instance.
*
* <p>
* The representation will be in the following format, where {@code url}, {@code height},
* {@code width} and {@code estimatedResolutionLevel} represent the corresponding properties:
* <br>
* <br>
* {@code Image {url=url, height='height, width=width,
* estimatedResolutionLevel=estimatedResolutionLevel}'}
* </p>
*
* @return a string representation of this {@link Image} instance
*/
@Nonnull
@Override
public String toString() {
return "Image {" + "url=" + url + ", height=" + height + ", width=" + width
+ ", estimatedResolutionLevel=" + estimatedResolutionLevel + "}";
}
/**
* The estimated resolution level of an {@link Image}.
*
* <p>
* Some services don't return the size of their images, but we may know for a specific image
* type that a service returns, according to real data, an approximation of the resolution
* level.
* </p>
*/
public enum ResolutionLevel {
/**
* The high resolution level.
*
* <p>
* This level applies to images with a height greater than or equal to 720px.
* </p>
*/
HIGH,
/**
* The medium resolution level.
*
* <p>
* This level applies to images with a height between 175px inclusive and 720px exclusive.
* </p>
*/
MEDIUM,
/**
* The low resolution level.
*
* <p>
* This level applies to images with a height between 1px inclusive and 175px exclusive.
* </p>
*/
LOW,
/**
* The unknown resolution level.
*
* <p>
* This value is returned when the extractor doesn't know what resolution level an image
* could have, for example if the extractor loops in an array of images with different
* resolution levels without knowing the height.
* </p>
*/
UNKNOWN;
/**
* Get a {@link ResolutionLevel} based from the given height.
*
* @param heightPx the height from which returning the good {@link ResolutionLevel}
* @return the {@link ResolutionLevel} corresponding to the height provided. See the
* {@link ResolutionLevel} values for details about what value is returned.
*/
public static ResolutionLevel fromHeight(final int heightPx) {
if (heightPx <= 0) {
return UNKNOWN;
}
if (heightPx < 175) {
return LOW;
}
if (heightPx < 720) {
return MEDIUM;
}
return HIGH;
}
}
}

View File

@ -1,33 +1,36 @@
package org.schabi.newpipe.extractor;
/*
* Created by Christian Schabesberger on 11.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* InfoItem.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* InfoItem.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <https://www.gnu.org/licenses/>.
*/
package org.schabi.newpipe.extractor;
import javax.annotation.Nonnull;
import java.io.Serializable;
import java.util.List;
public abstract class InfoItem implements Serializable {
private final InfoType infoType;
private final int serviceId;
private final String url;
private final String name;
private String thumbnailUrl;
@Nonnull
private List<Image> thumbnails = List.of();
public InfoItem(final InfoType infoType,
final int serviceId,
@ -55,12 +58,13 @@ public abstract class InfoItem implements Serializable {
return name;
}
public void setThumbnailUrl(final String thumbnailUrl) {
this.thumbnailUrl = thumbnailUrl;
public void setThumbnails(@Nonnull final List<Image> thumbnails) {
this.thumbnails = thumbnails;
}
public String getThumbnailUrl() {
return thumbnailUrl;
@Nonnull
public List<Image> getThumbnails() {
return thumbnails;
}
@Override

View File

@ -2,8 +2,12 @@ package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import javax.annotation.Nonnull;
import java.util.List;
public interface InfoItemExtractor {
String getName() throws ParsingException;
String getUrl() throws ParsingException;
String getThumbnailUrl() throws ParsingException;
@Nonnull
List<Image> getThumbnails() throws ParsingException;
}

View File

@ -12,21 +12,21 @@ import java.util.List;
/*
* Created by Christian Schabesberger on 12.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* InfoItemsCollector.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* InfoItemsCollector.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public abstract class InfoItemsCollector<I extends InfoItem, E extends InfoItemExtractor>

View File

@ -3,27 +3,31 @@ package org.schabi.newpipe.extractor;
/*
* Created by Adam Howard on 08/11/15.
*
* Copyright (c) Christian Schabesberger <chris.schabesberger@mailbox.org>
* and Adam Howard <achdisposable1@gmail.com> 2015
* Copyright (c) 2015 Christian Schabesberger <chris.schabesberger@mailbox.org>
* and Adam Howard <achdisposable1@gmail.com>
*
* MediaFormat.java is part of NewPipe.
* MediaFormat.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.Arrays;
import java.util.List;
import java.util.function.Function;
import java.util.stream.Collectors;
/**
* Static data about various media formats support by NewPipe, eg mime type, extension
@ -41,9 +45,18 @@ public enum MediaFormat {
M4A (0x100, "m4a", "m4a", "audio/mp4"),
WEBMA (0x200, "WebM", "webm", "audio/webm"),
MP3 (0x300, "MP3", "mp3", "audio/mpeg"),
MP2 (0x310, "MP2", "mp2", "audio/mpeg"),
OPUS (0x400, "opus", "opus", "audio/opus"),
OGG (0x500, "ogg", "ogg", "audio/ogg"),
WEBMA_OPUS(0x200, "WebM Opus", "webm", "audio/webm"),
AIFF (0x600, "AIFF", "aiff", "audio/aiff"),
/**
* Same as {@link MediaFormat.AIFF}, just with the shorter suffix/file extension
*/
AIF (0x600, "AIFF", "aif", "audio/aiff"),
WAV (0x700, "WAV", "wav", "audio/wav"),
FLAC (0x800, "FLAC", "flac", "audio/flac"),
ALAC (0x900, "ALAC", "alac", "audio/alac"),
// subtitles formats
VTT (0x1000, "WebVTT", "vtt", "text/vtt"),
TTML (0x2000, "Timed Text Markup Language", "ttml", "application/ttml+xml"),
@ -54,11 +67,15 @@ public enum MediaFormat {
// @formatter:on
public final int id;
@Nonnull
public final String name;
@Nonnull
public final String suffix;
@Nonnull
public final String mimeType;
MediaFormat(final int id, final String name, final String suffix, final String mimeType) {
MediaFormat(final int id, @Nonnull final String name,
@Nonnull final String suffix, @Nonnull final String mimeType) {
this.id = id;
this.name = name;
this.suffix = suffix;
@ -82,6 +99,7 @@ public enum MediaFormat {
* @return the friendly name of the MediaFormat associated with this ids,
* or an empty String if none match it.
*/
@Nonnull
public static String getNameById(final int id) {
return getById(id, MediaFormat::getName, "");
}
@ -93,6 +111,7 @@ public enum MediaFormat {
* @return the file extension of the MediaFormat associated with this ids,
* or an empty String if none match it.
*/
@Nonnull
public static String getSuffixById(final int id) {
return getById(id, MediaFormat::getSuffix, "");
}
@ -104,16 +123,20 @@ public enum MediaFormat {
* @return the MIME type of the MediaFormat associated with this ids,
* or an empty String if none match it.
*/
@Nullable
public static String getMimeById(final int id) {
return getById(id, MediaFormat::getMimeType, null);
}
/**
* Return the MediaFormat with the supplied mime type
* Return the first {@link MediaFormat} with the supplied mime type.
* There might be more formats which have the same mime type.
* To retrieve those, use {@link #getAllFromMimeType(String)}.
*
* @return MediaFormat associated with this mime type,
* or null if none match it.
*/
@Nullable
public static MediaFormat getFromMimeType(final String mimeType) {
return Arrays.stream(MediaFormat.values())
.filter(mediaFormat -> mediaFormat.mimeType.equals(mimeType))
@ -121,16 +144,35 @@ public enum MediaFormat {
.orElse(null);
}
/**
* Get all media formats which have the given mime type.
* @param mimeType the mime type to search for
* @return a modifiable {@link List} which contains the {@link MediaFormat}s
* that have the given mime type.
*/
@Nonnull
public static List<MediaFormat> getAllFromMimeType(final String mimeType) {
return Arrays.stream(MediaFormat.values())
.filter(mediaFormat -> mediaFormat.mimeType.equals(mimeType))
.collect(Collectors.toList());
}
/**
* Get the media format by its id.
*
* @param id the id
* @return the id of the media format or null.
*/
@Nullable
public static MediaFormat getFormatById(final int id) {
return getById(id, mediaFormat -> mediaFormat, null);
}
/**
* Get the first media format that has the given suffix/file extension.
* @return the matching {@link MediaFormat} or {@code null} if no associated format is found
*/
@Nullable
public static MediaFormat getFromSuffix(final String suffix) {
return Arrays.stream(MediaFormat.values())
.filter(mediaFormat -> mediaFormat.suffix.equals(suffix))
@ -143,6 +185,7 @@ public enum MediaFormat {
*
* @return the name of the format
*/
@Nonnull
public String getName() {
return name;
}
@ -152,6 +195,7 @@ public enum MediaFormat {
*
* @return the filename extension
*/
@Nonnull
public String getSuffix() {
return suffix;
}
@ -161,6 +205,7 @@ public enum MediaFormat {
*
* @return the mime type
*/
@Nonnull
public String getMimeType() {
return mimeType;
}

View File

@ -15,21 +15,21 @@ import java.util.List;
/*
* Created by Christian Schabesberger on 12.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* InfoItemsSearchCollector.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* InfoItemsSearchCollector.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
/**

View File

@ -3,21 +3,21 @@ package org.schabi.newpipe.extractor;
/*
* Created by Christian Schabesberger on 23.08.15.
*
* Copyright (C) Christian Schabesberger 2015 <chris.schabesberger@mailbox.org>
* NewPipe.java is part of NewPipe.
* Copyright (C) 2015 Christian Schabesberger <chris.schabesberger@mailbox.org>
* NewPipe Extractor.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
import org.schabi.newpipe.extractor.downloader.Downloader;
@ -25,9 +25,10 @@ import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.localization.ContentCountry;
import org.schabi.newpipe.extractor.localization.Localization;
import java.util.List;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.List;
/**
* Provides access to streaming services supported by NewPipe.
@ -92,24 +93,6 @@ public final class NewPipe {
throw new ExtractionException("No service can handle the url = \"" + url + "\"");
}
public static int getIdOfService(final String serviceName) {
try {
return getService(serviceName).getServiceId();
} catch (final ExtractionException ignored) {
return -1;
}
}
public static String getNameOfService(final int id) {
try {
return getService(id).getServiceInfo().getName();
} catch (final Exception e) {
System.err.println("Service id not known");
e.printStackTrace();
return "<unknown>";
}
}
/*//////////////////////////////////////////////////////////////////////////
// Localization
//////////////////////////////////////////////////////////////////////////*/

View File

@ -37,6 +37,10 @@ public class Page implements Serializable {
this(url, id, null, null, null);
}
public Page(final String url, final String id, final byte[] body) {
this(url, id, null, null, body);
}
public Page(final String url, final byte[] body) {
this(url, null, null, null, body);
}

View File

@ -6,26 +6,24 @@ import org.schabi.newpipe.extractor.services.peertube.PeertubeService;
import org.schabi.newpipe.extractor.services.soundcloud.SoundcloudService;
import org.schabi.newpipe.extractor.services.youtube.YoutubeService;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
/*
* Copyright (C) Christian Schabesberger 2018 <chris.schabesberger@mailbox.org>
* ServiceList.java is part of NewPipe.
* Copyright (C) 2018 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ServiceList.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
/**
@ -34,27 +32,21 @@ import java.util.List;
@SuppressWarnings({"ConstantName", "InnerAssignment"}) // keep unusual names and inner assignments
public final class ServiceList {
private ServiceList() {
//no instance
// no instance
}
public static final YoutubeService YouTube;
public static final SoundcloudService SoundCloud;
public static final MediaCCCService MediaCCC;
public static final PeertubeService PeerTube;
public static final BandcampService Bandcamp;
public static final YoutubeService YouTube = new YoutubeService(0);
public static final SoundcloudService SoundCloud = new SoundcloudService(1);
public static final MediaCCCService MediaCCC = new MediaCCCService(2);
public static final PeertubeService PeerTube = new PeertubeService(3);
public static final BandcampService Bandcamp = new BandcampService(4);
/**
* When creating a new service, put this service in the end of this list,
* and give it the next free id.
*/
private static final List<StreamingService> SERVICES = Collections.unmodifiableList(
Arrays.asList(
YouTube = new YoutubeService(0),
SoundCloud = new SoundcloudService(1),
MediaCCC = new MediaCCCService(2),
PeerTube = new PeertubeService(3),
Bandcamp = new BandcampService(4)
));
private static final List<StreamingService> SERVICES = List.of(
YouTube, SoundCloud, MediaCCC, PeerTube, Bandcamp);
/**
* Get all the supported services.

View File

@ -1,6 +1,7 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.channel.ChannelExtractor;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import org.schabi.newpipe.extractor.comments.CommentsExtractor;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
@ -28,21 +29,21 @@ import java.util.Collections;
import java.util.List;
/*
* Copyright (C) Christian Schabesberger 2018 <chris.schabesberger@mailbox.org>
* StreamingService.java is part of NewPipe.
* Copyright (C) 2018 Christian Schabesberger <chris.schabesberger@mailbox.org>
* StreamingService.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public abstract class StreamingService {
@ -140,6 +141,14 @@ public abstract class StreamingService {
*/
public abstract ListLinkHandlerFactory getChannelLHFactory();
/**
* Must return a new instance of an implementation of ListLinkHandlerFactory for channel tabs.
* If support for channel tabs is not given null must be returned.
*
* @return an instance of a ListLinkHandlerFactory for channels or null
*/
public abstract ListLinkHandlerFactory getChannelTabLHFactory();
/**
* Must return a new instance of an implementation of ListLinkHandlerFactory for playlists.
* If support for playlists is not given null must be returned.
@ -204,6 +213,15 @@ public abstract class StreamingService {
public abstract ChannelExtractor getChannelExtractor(ListLinkHandler linkHandler)
throws ExtractionException;
/**
* Must create a new instance of a ChannelTabExtractor implementation.
*
* @param linkHandler is pointing to the channel which should be handled by this new instance.
* @return a new ChannelTabExtractor
*/
public abstract ChannelTabExtractor getChannelTabExtractor(ListLinkHandler linkHandler)
throws ExtractionException;
/**
* Must crete a new instance of a PlaylistExtractor implementation.
* @param linkHandler is pointing to the playlist which should be handled by this new instance.
@ -262,6 +280,20 @@ public abstract class StreamingService {
return getChannelExtractor(getChannelLHFactory().fromUrl(url));
}
public ChannelTabExtractor getChannelTabExtractorFromId(final String id, final String tab)
throws ExtractionException {
return getChannelTabExtractor(getChannelTabLHFactory().fromQuery(
id, Collections.singletonList(tab), ""));
}
public ChannelTabExtractor getChannelTabExtractorFromIdAndBaseUrl(final String id,
final String tab,
final String baseUrl)
throws ExtractionException {
return getChannelTabExtractor(getChannelTabLHFactory().fromQuery(
id, Collections.singletonList(tab), "", baseUrl));
}
public PlaylistExtractor getPlaylistExtractor(final String url) throws ExtractionException {
return getPlaylistExtractor(getPlaylistLHFactory().fromUrl(url));
}

View File

@ -1,47 +1,58 @@
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
/*
* Created by Christian Schabesberger on 25.07.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* ChannelExtractor.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ChannelExtractor.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <https://www.gnu.org/licenses/>.
*/
public abstract class ChannelExtractor extends ListExtractor<StreamInfoItem> {
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.Extractor;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import javax.annotation.Nonnull;
import java.util.List;
public abstract class ChannelExtractor extends Extractor {
public static final long UNKNOWN_SUBSCRIBER_COUNT = -1;
public ChannelExtractor(final StreamingService service, final ListLinkHandler linkHandler) {
protected ChannelExtractor(final StreamingService service, final ListLinkHandler linkHandler) {
super(service, linkHandler);
}
public abstract String getAvatarUrl() throws ParsingException;
public abstract String getBannerUrl() throws ParsingException;
@Nonnull
public abstract List<Image> getAvatars() throws ParsingException;
@Nonnull
public abstract List<Image> getBanners() throws ParsingException;
public abstract String getFeedUrl() throws ParsingException;
public abstract long getSubscriberCount() throws ParsingException;
public abstract String getDescription() throws ParsingException;
public abstract String getParentChannelName() throws ParsingException;
public abstract String getParentChannelUrl() throws ParsingException;
public abstract String getParentChannelAvatarUrl() throws ParsingException;
@Nonnull
public abstract List<Image> getParentChannelAvatars() throws ParsingException;
public abstract boolean isVerified() throws ParsingException;
@Nonnull
public abstract List<ListLinkHandler> getTabs() throws ParsingException;
@Nonnull
public List<String> getTags() throws ParsingException {
return List.of();
}
}

View File

@ -1,47 +1,45 @@
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.ListExtractor.InfoItemsPage;
import org.schabi.newpipe.extractor.ListInfo;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import org.schabi.newpipe.extractor.utils.ExtractorHelper;
import java.io.IOException;
/*
* Created by Christian Schabesberger on 31.07.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* ChannelInfo.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ChannelInfo.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <https://www.gnu.org/licenses/>.
*/
public class ChannelInfo extends ListInfo<StreamInfoItem> {
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.Info;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import java.io.IOException;
import java.util.List;
import javax.annotation.Nonnull;
public class ChannelInfo extends Info {
public ChannelInfo(final int serviceId,
final String id,
final String url,
final String originalUrl,
final String name,
final ListLinkHandler listLinkHandler) {
super(serviceId, id, url, originalUrl, name, listLinkHandler.getContentFilters(),
listLinkHandler.getSortFilter());
final String name) {
super(serviceId, id, url, originalUrl, name);
}
public static ChannelInfo getInfo(final String url) throws IOException, ExtractionException {
@ -55,13 +53,6 @@ public class ChannelInfo extends ListInfo<StreamInfoItem> {
return getInfo(extractor);
}
public static InfoItemsPage<StreamInfoItem> getMoreItems(final StreamingService service,
final String url,
final Page page)
throws IOException, ExtractionException {
return service.getChannelExtractor(url).getPage(page);
}
public static ChannelInfo getInfo(final ChannelExtractor extractor)
throws IOException, ExtractionException {
@ -71,35 +62,32 @@ public class ChannelInfo extends ListInfo<StreamInfoItem> {
final String originalUrl = extractor.getOriginalUrl();
final String name = extractor.getName();
final ChannelInfo info =
new ChannelInfo(serviceId, id, url, originalUrl, name, extractor.getLinkHandler());
final ChannelInfo info = new ChannelInfo(serviceId, id, url, originalUrl, name);
try {
info.setAvatarUrl(extractor.getAvatarUrl());
info.setAvatars(extractor.getAvatars());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setBannerUrl(extractor.getBannerUrl());
info.setBanners(extractor.getBanners());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setFeedUrl(extractor.getFeedUrl());
} catch (final Exception e) {
info.addError(e);
}
final InfoItemsPage<StreamInfoItem> itemsPage =
ExtractorHelper.getItemsPageOrLogError(info, extractor);
info.setRelatedItems(itemsPage.getItems());
info.setNextPage(itemsPage.getNextPage());
try {
info.setSubscriberCount(extractor.getSubscriberCount());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setDescription(extractor.getDescription());
} catch (final Exception e) {
@ -119,7 +107,7 @@ public class ChannelInfo extends ListInfo<StreamInfoItem> {
}
try {
info.setParentChannelAvatarUrl(extractor.getParentChannelAvatarUrl());
info.setParentChannelAvatars(extractor.getParentChannelAvatars());
} catch (final Exception e) {
info.addError(e);
}
@ -130,19 +118,36 @@ public class ChannelInfo extends ListInfo<StreamInfoItem> {
info.addError(e);
}
try {
info.setTabs(extractor.getTabs());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setTags(extractor.getTags());
} catch (final Exception e) {
info.addError(e);
}
return info;
}
private String avatarUrl;
private String parentChannelName;
private String parentChannelUrl;
private String parentChannelAvatarUrl;
private String bannerUrl;
private String feedUrl;
private long subscriberCount = -1;
private String description;
private String[] donationLinks;
@Nonnull
private List<Image> avatars = List.of();
@Nonnull
private List<Image> banners = List.of();
@Nonnull
private List<Image> parentChannelAvatars = List.of();
private boolean verified;
private List<ListLinkHandler> tabs = List.of();
private List<String> tags = List.of();
public String getParentChannelName() {
return parentChannelName;
@ -160,28 +165,31 @@ public class ChannelInfo extends ListInfo<StreamInfoItem> {
this.parentChannelUrl = parentChannelUrl;
}
public String getParentChannelAvatarUrl() {
return parentChannelAvatarUrl;
@Nonnull
public List<Image> getParentChannelAvatars() {
return parentChannelAvatars;
}
public void setParentChannelAvatarUrl(final String parentChannelAvatarUrl) {
this.parentChannelAvatarUrl = parentChannelAvatarUrl;
public void setParentChannelAvatars(@Nonnull final List<Image> parentChannelAvatars) {
this.parentChannelAvatars = parentChannelAvatars;
}
public String getAvatarUrl() {
return avatarUrl;
@Nonnull
public List<Image> getAvatars() {
return avatars;
}
public void setAvatarUrl(final String avatarUrl) {
this.avatarUrl = avatarUrl;
public void setAvatars(@Nonnull final List<Image> avatars) {
this.avatars = avatars;
}
public String getBannerUrl() {
return bannerUrl;
@Nonnull
public List<Image> getBanners() {
return banners;
}
public void setBannerUrl(final String bannerUrl) {
this.bannerUrl = bannerUrl;
public void setBanners(@Nonnull final List<Image> banners) {
this.banners = banners;
}
public String getFeedUrl() {
@ -223,4 +231,22 @@ public class ChannelInfo extends ListInfo<StreamInfoItem> {
public void setVerified(final boolean verified) {
this.verified = verified;
}
@Nonnull
public List<ListLinkHandler> getTabs() {
return tabs;
}
public void setTabs(@Nonnull final List<ListLinkHandler> tabs) {
this.tabs = tabs;
}
@Nonnull
public List<String> getTags() {
return tags;
}
public void setTags(@Nonnull final List<String> tags) {
this.tags = tags;
}
}

View File

@ -5,21 +5,21 @@ import org.schabi.newpipe.extractor.InfoItem;
/*
* Created by Christian Schabesberger on 11.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* ChannelInfoItem.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ChannelInfoItem.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public class ChannelInfoItem extends InfoItem {

View File

@ -6,21 +6,21 @@ import org.schabi.newpipe.extractor.exceptions.ParsingException;
/*
* Created by Christian Schabesberger on 12.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* ChannelInfoItemExtractor.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ChannelInfoItemExtractor.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public interface ChannelInfoItemExtractor extends InfoItemExtractor {

View File

@ -1,28 +1,28 @@
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.InfoItemsCollector;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
/*
* Created by Christian Schabesberger on 12.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* ChannelInfoItemsCollector.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ChannelInfoItemsCollector.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <https://www.gnu.org/licenses/>.
*/
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.InfoItemsCollector;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
public final class ChannelInfoItemsCollector
extends InfoItemsCollector<ChannelInfoItem, ChannelInfoItemExtractor> {
public ChannelInfoItemsCollector(final int serviceId) {
@ -47,7 +47,7 @@ public final class ChannelInfoItemsCollector
addError(e);
}
try {
resultItem.setThumbnailUrl(extractor.getThumbnailUrl());
resultItem.setThumbnails(extractor.getThumbnails());
} catch (final Exception e) {
addError(e);
}

View File

@ -0,0 +1,25 @@
package org.schabi.newpipe.extractor.channel.tabs;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import javax.annotation.Nonnull;
/**
* A {@link ListExtractor} of {@link InfoItem}s for tabs of channels.
*/
public abstract class ChannelTabExtractor extends ListExtractor<InfoItem> {
protected ChannelTabExtractor(@Nonnull final StreamingService service,
@Nonnull final ListLinkHandler linkHandler) {
super(service, linkHandler);
}
@Nonnull
@Override
public String getName() {
return getLinkHandler().getContentFilters().get(0);
}
}

View File

@ -0,0 +1,70 @@
package org.schabi.newpipe.extractor.channel.tabs;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.ListInfo;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.ChannelInfo;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.utils.ExtractorHelper;
import javax.annotation.Nonnull;
import java.io.IOException;
public class ChannelTabInfo extends ListInfo<InfoItem> {
public ChannelTabInfo(final int serviceId,
@Nonnull final ListLinkHandler linkHandler) {
super(serviceId, linkHandler, linkHandler.getContentFilters().get(0));
}
/**
* Get a {@link ChannelTabInfo} instance from the given service and tab handler.
*
* @param service streaming service
* @param linkHandler Channel tab handler (from {@link ChannelInfo})
* @return the extracted {@link ChannelTabInfo}
*/
@Nonnull
public static ChannelTabInfo getInfo(@Nonnull final StreamingService service,
@Nonnull final ListLinkHandler linkHandler)
throws ExtractionException, IOException {
final ChannelTabExtractor extractor = service.getChannelTabExtractor(linkHandler);
extractor.fetchPage();
return getInfo(extractor);
}
/**
* Get a {@link ChannelTabInfo} instance from a {@link ChannelTabExtractor}.
*
* @param extractor an extractor where {@code fetchPage()} was already got called on
* @return the extracted {@link ChannelTabInfo}
*/
@Nonnull
public static ChannelTabInfo getInfo(@Nonnull final ChannelTabExtractor extractor) {
final ChannelTabInfo info =
new ChannelTabInfo(extractor.getServiceId(), extractor.getLinkHandler());
try {
info.setOriginalUrl(extractor.getOriginalUrl());
} catch (final Exception e) {
info.addError(e);
}
final ListExtractor.InfoItemsPage<InfoItem> page
= ExtractorHelper.getItemsPageOrLogError(info, extractor);
info.setRelatedItems(page.getItems());
info.setNextPage(page.getNextPage());
return info;
}
public static ListExtractor.InfoItemsPage<InfoItem> getMoreItems(
@Nonnull final StreamingService service,
@Nonnull final ListLinkHandler linkHandler,
@Nonnull final Page page) throws ExtractionException, IOException {
return service.getChannelTabExtractor(linkHandler).getPage(page);
}
}

View File

@ -0,0 +1,17 @@
package org.schabi.newpipe.extractor.channel.tabs;
/**
* Constants of channel tabs supported by the extractor.
*/
public final class ChannelTabs {
public static final String VIDEOS = "videos";
public static final String TRACKS = "tracks";
public static final String SHORTS = "shorts";
public static final String LIVESTREAMS = "livestreams";
public static final String CHANNELS = "channels";
public static final String PLAYLISTS = "playlists";
public static final String ALBUMS = "albums";
private ChannelTabs() {
}
}

View File

@ -22,6 +22,13 @@ public abstract class CommentsExtractor extends ListExtractor<CommentsInfoItem>
return false;
}
/**
* @return the total number of comments
*/
public int getCommentsCount() throws ExtractionException {
return -1;
}
@Nonnull
@Override
public String getName() throws ParsingException {

View File

@ -48,6 +48,11 @@ public final class CommentsInfo extends ListInfo<CommentsInfoItem> {
ExtractorHelper.getItemsPageOrLogError(commentsInfo, commentsExtractor);
commentsInfo.setCommentsDisabled(commentsExtractor.isCommentsDisabled());
commentsInfo.setRelatedItems(initialCommentsPage.getItems());
try {
commentsInfo.setCommentsCount(commentsExtractor.getCommentsCount());
} catch (final Exception e) {
commentsInfo.addError(e);
}
commentsInfo.setNextPage(initialCommentsPage.getNextPage());
return commentsInfo;
@ -76,6 +81,7 @@ public final class CommentsInfo extends ListInfo<CommentsInfoItem> {
private transient CommentsExtractor commentsExtractor;
private boolean commentsDisabled = false;
private int commentsCount;
public CommentsExtractor getCommentsExtractor() {
return commentsExtractor;
@ -86,7 +92,6 @@ public final class CommentsInfo extends ListInfo<CommentsInfoItem> {
}
/**
* @apiNote Warning: This method is experimental and may get removed in a future release.
* @return {@code true} if the comments are disabled otherwise {@code false} (default)
* @see CommentsExtractor#isCommentsDisabled()
*/
@ -95,10 +100,27 @@ public final class CommentsInfo extends ListInfo<CommentsInfoItem> {
}
/**
* @apiNote Warning: This method is experimental and may get removed in a future release.
* @param commentsDisabled {@code true} if the comments are disabled otherwise {@code false}
*/
public void setCommentsDisabled(final boolean commentsDisabled) {
this.commentsDisabled = commentsDisabled;
}
/**
* Returns the total number of comments.
*
* @return the total number of comments
*/
public int getCommentsCount() {
return commentsCount;
}
/**
* Sets the total number of comments.
*
* @param commentsCount the commentsCount to set.
*/
public void setCommentsCount(final int commentsCount) {
this.commentsCount = commentsCount;
}
}

View File

@ -1,17 +1,23 @@
package org.schabi.newpipe.extractor.comments;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.localization.DateWrapper;
import org.schabi.newpipe.extractor.stream.Description;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.List;
public class CommentsInfoItem extends InfoItem {
private String commentId;
private String commentText;
@Nonnull
private Description commentText = Description.EMPTY_DESCRIPTION;
private String uploaderName;
private String uploaderAvatarUrl;
@Nonnull
private List<Image> uploaderAvatars = List.of();
private String uploaderUrl;
private boolean uploaderVerified;
private String textualUploadDate;
@ -22,12 +28,17 @@ public class CommentsInfoItem extends InfoItem {
private boolean heartedByUploader;
private boolean pinned;
private int streamPosition;
private int replyCount;
@Nullable
private Page replies;
private boolean isChannelOwner;
private boolean creatorReply;
public static final int NO_LIKE_COUNT = -1;
public static final int NO_STREAM_POSITION = -1;
public static final int UNKNOWN_REPLY_COUNT = -1;
public CommentsInfoItem(final int serviceId, final String url, final String name) {
super(InfoType.COMMENT, serviceId, url, name);
}
@ -40,11 +51,12 @@ public class CommentsInfoItem extends InfoItem {
this.commentId = commentId;
}
public String getCommentText() {
@Nonnull
public Description getCommentText() {
return commentText;
}
public void setCommentText(final String commentText) {
public void setCommentText(@Nonnull final Description commentText) {
this.commentText = commentText;
}
@ -56,12 +68,13 @@ public class CommentsInfoItem extends InfoItem {
this.uploaderName = uploaderName;
}
public String getUploaderAvatarUrl() {
return uploaderAvatarUrl;
@Nonnull
public List<Image> getUploaderAvatars() {
return uploaderAvatars;
}
public void setUploaderAvatarUrl(final String uploaderAvatarUrl) {
this.uploaderAvatarUrl = uploaderAvatarUrl;
public void setUploaderAvatars(@Nonnull final List<Image> uploaderAvatars) {
this.uploaderAvatars = uploaderAvatars;
}
public String getUploaderUrl() {
@ -90,8 +103,8 @@ public class CommentsInfoItem extends InfoItem {
}
/**
* @return the comment's like count
* or {@link CommentsInfoItem#NO_LIKE_COUNT} if it is unavailable
* @return the comment's like count or {@link CommentsInfoItem#NO_LIKE_COUNT} if it is
* unavailable
*/
public int getLikeCount() {
return likeCount;
@ -140,12 +153,21 @@ public class CommentsInfoItem extends InfoItem {
/**
* Get the playback position of the stream to which this comment belongs.
* This is not supported by all services.
*
* @return the playback position in seconds or {@link #NO_STREAM_POSITION} if not available
*/
public int getStreamPosition() {
return streamPosition;
}
public void setReplyCount(final int replyCount) {
this.replyCount = replyCount;
}
public int getReplyCount() {
return replyCount;
}
public void setReplies(@Nullable final Page replies) {
this.replies = replies;
}
@ -154,4 +176,22 @@ public class CommentsInfoItem extends InfoItem {
public Page getReplies() {
return this.replies;
}
public void setChannelOwner(final boolean channelOwner) {
this.isChannelOwner = channelOwner;
}
public boolean isChannelOwner() {
return isChannelOwner;
}
public void setCreatorReply(final boolean creatorReply) {
this.creatorReply = creatorReply;
}
public boolean hasCreatorReply() {
return creatorReply;
}
}

View File

@ -1,14 +1,17 @@
package org.schabi.newpipe.extractor.comments;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.InfoItemExtractor;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.localization.DateWrapper;
import org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeCommentsInfoItemExtractor;
import org.schabi.newpipe.extractor.stream.Description;
import org.schabi.newpipe.extractor.stream.StreamExtractor;
import org.schabi.newpipe.extractor.utils.Utils;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.List;
public interface CommentsInfoItemExtractor extends InfoItemExtractor {
@ -17,14 +20,14 @@ public interface CommentsInfoItemExtractor extends InfoItemExtractor {
* or {@link CommentsInfoItem#NO_LIKE_COUNT} if it is unavailable.
*
* <br>
*
* <p>
* NOTE: Currently only implemented for YT {@link
* YoutubeCommentsInfoItemExtractor#getLikeCount()}
* with limitations (only approximate like count is returned)
*
* @see StreamExtractor#getLikeCount()
* @return the comment's like count
* or {@link CommentsInfoItem#NO_LIKE_COUNT} if it is unavailable
* @see StreamExtractor#getLikeCount()
*/
default int getLikeCount() throws ParsingException {
return CommentsInfoItem.NO_LIKE_COUNT;
@ -36,14 +39,15 @@ public interface CommentsInfoItemExtractor extends InfoItemExtractor {
* It may be language dependent
*/
default String getTextualLikeCount() throws ParsingException {
return Utils.EMPTY_STRING;
return "";
}
/**
* The text of the comment
*/
default String getCommentText() throws ParsingException {
return Utils.EMPTY_STRING;
@Nonnull
default Description getCommentText() throws ParsingException {
return Description.EMPTY_DESCRIPTION;
}
/**
@ -52,7 +56,7 @@ public interface CommentsInfoItemExtractor extends InfoItemExtractor {
* @see StreamExtractor#getTextualUploadDate()
*/
default String getTextualUploadDate() throws ParsingException {
return Utils.EMPTY_STRING;
return "";
}
/**
@ -66,19 +70,20 @@ public interface CommentsInfoItemExtractor extends InfoItemExtractor {
}
default String getCommentId() throws ParsingException {
return Utils.EMPTY_STRING;
return "";
}
default String getUploaderUrl() throws ParsingException {
return Utils.EMPTY_STRING;
return "";
}
default String getUploaderName() throws ParsingException {
return Utils.EMPTY_STRING;
return "";
}
default String getUploaderAvatarUrl() throws ParsingException {
return Utils.EMPTY_STRING;
@Nonnull
default List<Image> getUploaderAvatars() throws ParsingException {
return List.of();
}
/**
@ -104,18 +109,44 @@ public interface CommentsInfoItemExtractor extends InfoItemExtractor {
/**
* The playback position of the stream to which this comment belongs.
*
* @see CommentsInfoItem#getStreamPosition()
*/
default int getStreamPosition() throws ParsingException {
return CommentsInfoItem.NO_STREAM_POSITION;
}
/**
* The count of comment replies.
*
* @return the count of the replies
* or {@link CommentsInfoItem#UNKNOWN_REPLY_COUNT} if replies are not supported
*/
default int getReplyCount() throws ParsingException {
return CommentsInfoItem.UNKNOWN_REPLY_COUNT;
}
/**
* The continuation page which is used to get comment replies from.
*
* @return the continuation Page for the replies, or null if replies are not supported
*/
@Nullable
default Page getReplies() throws ParsingException {
return null;
}
/**
* Whether the comment was made by the channel owner.
*/
default boolean isChannelOwner() throws ParsingException {
return false;
}
/**
* Whether the comment was replied to by the creator.
*/
default boolean hasCreatorReply() throws ParsingException {
return false;
}
}

View File

@ -36,7 +36,7 @@ public final class CommentsInfoItemsCollector
addError(e);
}
try {
resultItem.setUploaderAvatarUrl(extractor.getUploaderAvatarUrl());
resultItem.setUploaderAvatars(extractor.getUploaderAvatars());
} catch (final Exception e) {
addError(e);
}
@ -66,7 +66,7 @@ public final class CommentsInfoItemsCollector
addError(e);
}
try {
resultItem.setThumbnailUrl(extractor.getThumbnailUrl());
resultItem.setThumbnails(extractor.getThumbnails());
} catch (final Exception e) {
addError(e);
}
@ -89,12 +89,32 @@ public final class CommentsInfoItemsCollector
addError(e);
}
try {
resultItem.setReplyCount(extractor.getReplyCount());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setReplies(extractor.getReplies());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setChannelOwner(extractor.isChannelOwner());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setCreatorReply(extractor.hasCreatorReply());
} catch (final Exception e) {
addError(e);
}
return resultItem;
}

View File

@ -7,6 +7,8 @@ import org.schabi.newpipe.extractor.localization.Localization;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.io.IOException;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
@ -39,7 +41,7 @@ public abstract class Downloader {
* @param localization the source of the value of the {@code Accept-Language} header
* @return the result of the GET request
*/
public Response get(final String url, @Nullable final Localization localization)
public Response get(final String url, final Localization localization)
throws IOException, ReCaptchaException {
return get(url, null, localization);
}
@ -70,7 +72,7 @@ public abstract class Downloader {
*/
public Response get(final String url,
@Nullable final Map<String, List<String>> headers,
@Nullable final Localization localization)
final Localization localization)
throws IOException, ReCaptchaException {
return execute(Request.newBuilder()
.get(url)
@ -112,7 +114,7 @@ public abstract class Downloader {
* @param headers a list of headers that will be used in the request.
* Any default headers <b>should</b> be overridden by these.
* @param dataToSend byte array that will be sent when doing the request.
* @return the result of the GET request
* @return the result of the POST request
*/
public Response post(final String url,
@Nullable final Map<String, List<String>> headers,
@ -131,12 +133,12 @@ public abstract class Downloader {
* Any default headers <b>should</b> be overridden by these.
* @param dataToSend byte array that will be sent when doing the request.
* @param localization the source of the value of the {@code Accept-Language} header
* @return the result of the GET request
* @return the result of the POST request
*/
public Response post(final String url,
@Nullable final Map<String, List<String>> headers,
@Nullable final byte[] dataToSend,
@Nullable final Localization localization)
final Localization localization)
throws IOException, ReCaptchaException {
return execute(Request.newBuilder()
.post(url, dataToSend)
@ -145,6 +147,95 @@ public abstract class Downloader {
.build());
}
/**
* Convenient method to send a POST request using the specified value of the
* {@code Content-Type} header with a given {@link Localization}.
*
* @param url the URL that is pointing to the wanted resource
* @param headers a list of headers that will be used in the request.
* Any default headers <b>should</b> be overridden by these.
* @param dataToSend byte array that will be sent when doing the request.
* @param localization the source of the value of the {@code Accept-Language} header
* @param contentType the mime type of the body sent, which will be set as the value of the
* {@code Content-Type} header
* @return the result of the POST request
* @see #post(String, Map, byte[], Localization)
*/
public Response postWithContentType(final String url,
@Nullable final Map<String, List<String>> headers,
@Nullable final byte[] dataToSend,
final Localization localization,
final String contentType)
throws IOException, ReCaptchaException {
final Map<String, List<String>> actualHeaders = new HashMap<>();
if (headers != null) {
actualHeaders.putAll(headers);
}
actualHeaders.put("Content-Type", Collections.singletonList(contentType));
return post(url, actualHeaders, dataToSend, localization);
}
/**
* Convenient method to send a POST request using the specified value of the
* {@code Content-Type} header.
*
* @param url the URL that is pointing to the wanted resource
* @param headers a list of headers that will be used in the request.
* Any default headers <b>should</b> be overridden by these.
* @param dataToSend byte array that will be sent when doing the request.
* @param contentType the mime type of the body sent, which will be set as the value of the
* {@code Content-Type} header
* @return the result of the POST request
* @see #post(String, Map, byte[], Localization)
*/
public Response postWithContentType(final String url,
@Nullable final Map<String, List<String>> headers,
@Nullable final byte[] dataToSend,
final String contentType)
throws IOException, ReCaptchaException {
return postWithContentType(url, headers, dataToSend, NewPipe.getPreferredLocalization(),
contentType);
}
/**
* Convenient method to send a POST request the JSON mime type as the value of the
* {@code Content-Type} header with a given {@link Localization}.
*
* @param url the URL that is pointing to the wanted resource
* @param headers a list of headers that will be used in the request.
* Any default headers <b>should</b> be overridden by these.
* @param dataToSend byte array that will be sent when doing the request.
* @param localization the source of the value of the {@code Accept-Language} header
* @return the result of the POST request
* @see #post(String, Map, byte[], Localization)
*/
public Response postWithContentTypeJson(final String url,
@Nullable final Map<String, List<String>> headers,
@Nullable final byte[] dataToSend,
final Localization localization)
throws IOException, ReCaptchaException {
return postWithContentType(url, headers, dataToSend, localization, "application/json");
}
/**
* Convenient method to send a POST request the JSON mime type as the value of the
* {@code Content-Type} header.
*
* @param url the URL that is pointing to the wanted resource
* @param headers a list of headers that will be used in the request.
* Any default headers <b>should</b> be overridden by these.
* @param dataToSend byte array that will be sent when doing the request.
* @return the result of the POST request
* @see #post(String, Map, byte[], Localization)
*/
public Response postWithContentTypeJson(final String url,
@Nullable final Map<String, List<String>> headers,
@Nullable final byte[] dataToSend)
throws IOException, ReCaptchaException {
return postWithContentTypeJson(url, headers, dataToSend,
NewPipe.getPreferredLocalization());
}
/**
* Do a request using the specified {@link Request} object.
*

View File

@ -32,15 +32,8 @@ public class Request {
@Nullable final byte[] dataToSend,
@Nullable final Localization localization,
final boolean automaticLocalizationHeader) {
if (httpMethod == null) {
throw new IllegalArgumentException("Request's httpMethod is null");
}
if (url == null) {
throw new IllegalArgumentException("Request's url is null");
}
this.httpMethod = httpMethod;
this.url = url;
this.httpMethod = Objects.requireNonNull(httpMethod, "Request's httpMethod is null");
this.url = Objects.requireNonNull(url, "Request's url is null");
this.dataToSend = dataToSend;
this.localization = localization;
@ -49,7 +42,7 @@ public class Request {
actualHeaders.putAll(headers);
}
if (automaticLocalizationHeader && localization != null) {
actualHeaders.putAll(headersFromLocalization(localization));
actualHeaders.putAll(getHeadersFromLocalization(localization));
}
this.headers = Collections.unmodifiableMap(actualHeaders);
@ -98,7 +91,7 @@ public class Request {
* A localization object that should be used when executing a request.<br>
* <br>
* Usually the {@code Accept-Language} will be set to this value (a helper
* method to do this easily: {@link Request#headersFromLocalization(Localization)}).
* method to do this easily: {@link Request#getHeadersFromLocalization(Localization)}).
*/
@Nullable
public Localization localization() {
@ -165,7 +158,7 @@ public class Request {
* A localization object that should be used when executing a request.<br>
* <br>
* Usually the {@code Accept-Language} will be set to this value (a helper
* method to do this easily: {@link Request#headersFromLocalization(Localization)}).
* method to do this easily: {@link Request#getHeadersFromLocalization(Localization)}).
*/
public Builder localization(final Localization localizationToSet) {
this.localization = localizationToSet;
@ -245,23 +238,17 @@ public class Request {
@SuppressWarnings("WeakerAccess")
@Nonnull
public static Map<String, List<String>> headersFromLocalization(
public static Map<String, List<String>> getHeadersFromLocalization(
@Nullable final Localization localization) {
if (localization == null) {
return Collections.emptyMap();
}
final Map<String, List<String>> headers = new LinkedHashMap<>();
if (!localization.getCountryCode().isEmpty()) {
headers.put("Accept-Language",
Collections.singletonList(localization.getLocalizationCode()
+ ", " + localization.getLanguageCode() + ";q=0.9"));
} else {
headers.put("Accept-Language",
Collections.singletonList(localization.getLanguageCode()));
}
return headers;
final String languageCode = localization.getLanguageCode();
final List<String> languageCodeList = Collections.singletonList(
localization.getCountryCode().isEmpty() ? languageCode
: localization.getLocalizationCode() + ", " + languageCode + ";q=0.9");
return Collections.singletonMap("Accept-Language", languageCodeList);
}
/*//////////////////////////////////////////////////////////////////////////

View File

@ -3,21 +3,21 @@ package org.schabi.newpipe.extractor.exceptions;
/*
* Created by Christian Schabesberger on 30.01.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* ExtractionException.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ExtractionException.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public class ExtractionException extends Exception {

View File

@ -3,21 +3,21 @@ package org.schabi.newpipe.extractor.exceptions;
/*
* Created by Christian Schabesberger on 12.09.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* FoundAdException.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* FoundAdException.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public class FoundAdException extends ParsingException {

View File

@ -3,21 +3,21 @@ package org.schabi.newpipe.extractor.exceptions;
/*
* Created by Christian Schabesberger on 31.01.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* ParsingException.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ParsingException.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/

View File

@ -3,21 +3,21 @@ package org.schabi.newpipe.extractor.exceptions;
/*
* Created by beneth <bmauduit@beneth.fr> on 07.12.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* ReCaptchaException.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ReCaptchaException.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public class ReCaptchaException extends ExtractionException {

View File

@ -0,0 +1,7 @@
package org.schabi.newpipe.extractor.exceptions;
public final class UnsupportedTabException extends UnsupportedOperationException {
public UnsupportedTabException(final String unsupportedTab) {
super("Unsupported tab " + unsupportedTab);
}
}

View File

@ -3,21 +3,21 @@ package org.schabi.newpipe.extractor.kiosk;
/*
* Created by Christian Schabesberger on 12.08.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* KioskExtractor.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* KioskExtractor.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
import org.schabi.newpipe.extractor.InfoItem;

View File

@ -3,21 +3,21 @@ package org.schabi.newpipe.extractor.kiosk;
/*
* Created by Christian Schabesberger on 12.08.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* KioskInfo.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* KioskInfo.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
import org.schabi.newpipe.extractor.ListExtractor;

View File

@ -1,5 +1,7 @@
package org.schabi.newpipe.extractor.kiosk;
import static org.schabi.newpipe.extractor.utils.Utils.isNullOrEmpty;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
@ -8,13 +10,12 @@ import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import org.schabi.newpipe.extractor.localization.ContentCountry;
import org.schabi.newpipe.extractor.localization.Localization;
import javax.annotation.Nullable;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import java.util.Set;
import static org.schabi.newpipe.extractor.utils.Utils.isNullOrEmpty;
import javax.annotation.Nullable;
public class KioskList {
@ -78,10 +79,10 @@ public class KioskList {
if (!isNullOrEmpty(defaultKiosk)) {
return getExtractorById(defaultKiosk, nextPage, localization);
} else {
if (!kioskList.isEmpty()) {
final String first = kioskList.keySet().stream().findAny().orElse(null);
if (first != null) {
// if not set get any entry
final Object[] keySet = kioskList.keySet().toArray();
return getExtractorById(keySet[0].toString(), nextPage, localization);
return getExtractorById(first, nextPage, localization);
} else {
return null;
}

View File

@ -3,24 +3,26 @@ package org.schabi.newpipe.extractor.linkhandler;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.utils.Utils;
import java.util.Objects;
/*
* Created by Christian Schabesberger on 26.07.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* LinkHandlerFactory.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* LinkHandlerFactory.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public abstract class LinkHandlerFactory {
@ -29,13 +31,14 @@ public abstract class LinkHandlerFactory {
// To Override
///////////////////////////////////
public abstract String getId(String url) throws ParsingException;
public abstract String getId(String url) throws ParsingException, UnsupportedOperationException;
public abstract String getUrl(String id) throws ParsingException;
public abstract String getUrl(String id) throws ParsingException, UnsupportedOperationException;
public abstract boolean onAcceptUrl(String url) throws ParsingException;
public String getUrl(final String id, final String baseUrl) throws ParsingException {
public String getUrl(final String id, final String baseUrl)
throws ParsingException, UnsupportedOperationException {
return getUrl(id);
}
@ -72,9 +75,7 @@ public abstract class LinkHandlerFactory {
* @return a {@link LinkHandler} complete with information
*/
public LinkHandler fromUrl(final String url, final String baseUrl) throws ParsingException {
if (url == null) {
throw new IllegalArgumentException("URL cannot be null");
}
Objects.requireNonNull(url, "URL cannot be null");
if (!acceptUrl(url)) {
throw new ParsingException("URL not accepted: " + url);
}
@ -84,17 +85,13 @@ public abstract class LinkHandlerFactory {
}
public LinkHandler fromId(final String id) throws ParsingException {
if (id == null) {
throw new IllegalArgumentException("id can not be null");
}
Objects.requireNonNull(id, "ID cannot be null");
final String url = getUrl(id);
return new LinkHandler(url, url, id);
}
public LinkHandler fromId(final String id, final String baseUrl) throws ParsingException {
if (id == null) {
throw new IllegalArgumentException("id can not be null");
}
Objects.requireNonNull(id, "ID cannot be null");
final String url = getUrl(id, baseUrl);
return new LinkHandler(url, url, id);
}

View File

@ -1,7 +1,5 @@
package org.schabi.newpipe.extractor.linkhandler;
import static org.schabi.newpipe.extractor.utils.Utils.EMPTY_STRING;
import java.util.Collections;
import java.util.List;
@ -32,7 +30,7 @@ public class ListLinkHandler extends LinkHandler {
handler.url,
handler.id,
Collections.emptyList(),
EMPTY_STRING);
"");
}
public List<String> getContentFilters() {

View File

@ -5,6 +5,7 @@ import org.schabi.newpipe.extractor.utils.Utils;
import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
public abstract class ListLinkHandlerFactory extends LinkHandlerFactory {
@ -13,12 +14,13 @@ public abstract class ListLinkHandlerFactory extends LinkHandlerFactory {
///////////////////////////////////
public abstract String getUrl(String id, List<String> contentFilter, String sortFilter)
throws ParsingException;
throws ParsingException, UnsupportedOperationException;
public String getUrl(final String id,
final List<String> contentFilter,
final String sortFilter,
final String baseUrl) throws ParsingException {
final String baseUrl)
throws ParsingException, UnsupportedOperationException {
return getUrl(id, contentFilter, sortFilter);
}
@ -35,10 +37,7 @@ public abstract class ListLinkHandlerFactory extends LinkHandlerFactory {
@Override
public ListLinkHandler fromUrl(final String url, final String baseUrl) throws ParsingException {
if (url == null) {
throw new IllegalArgumentException("url may not be null");
}
Objects.requireNonNull(url, "URL may not be null");
return new ListLinkHandler(super.fromUrl(url, baseUrl));
}
@ -74,7 +73,7 @@ public abstract class ListLinkHandlerFactory extends LinkHandlerFactory {
*
* @return the url corresponding to id without any filters applied
*/
public String getUrl(final String id) throws ParsingException {
public String getUrl(final String id) throws ParsingException, UnsupportedOperationException {
return getUrl(id, new ArrayList<>(0), "");
}

View File

@ -0,0 +1,55 @@
package org.schabi.newpipe.extractor.linkhandler;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import javax.annotation.Nonnull;
import java.io.Serializable;
import java.util.List;
/**
* A {@link ListLinkHandler} which can be used to be returned from {@link
* org.schabi.newpipe.extractor.channel.ChannelInfo#getTabs() ChannelInfo#getTabs()} when a
* specific tab's data has already been fetched.
*
* <p>
* This class allows passing a builder for a {@link ChannelTabExtractor} that can hold references
* to variables.
* </p>
*
* <p>
* Note: a service that wishes to use this class in one of its {@link
* org.schabi.newpipe.extractor.channel.ChannelExtractor ChannelExtractor}s must also add the
* following snippet of code in the service's
* {@link StreamingService#getChannelTabExtractor(ListLinkHandler)}:
* <pre>
* if (linkHandler instanceof ReadyChannelTabListLinkHandler) {
* return ((ReadyChannelTabListLinkHandler) linkHandler).getChannelTabExtractor(this);
* }
* </pre>
* </p>
*/
public class ReadyChannelTabListLinkHandler extends ListLinkHandler {
public interface ChannelTabExtractorBuilder extends Serializable {
@Nonnull
ChannelTabExtractor build(@Nonnull StreamingService service,
@Nonnull ListLinkHandler linkHandler);
}
private final ChannelTabExtractorBuilder extractorBuilder;
public ReadyChannelTabListLinkHandler(
final String url,
final String channelId,
@Nonnull final String channelTab,
@Nonnull final ChannelTabExtractorBuilder extractorBuilder) {
super(url, url, channelId, List.of(channelTab), "");
this.extractorBuilder = extractorBuilder;
}
@Nonnull
public ChannelTabExtractor getChannelTabExtractor(@Nonnull final StreamingService service) {
return extractorBuilder.build(service, new ListLinkHandler(this));
}
}

View File

@ -1,7 +1,5 @@
package org.schabi.newpipe.extractor.linkhandler;
import static org.schabi.newpipe.extractor.utils.Utils.EMPTY_STRING;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import java.util.Collections;
@ -15,7 +13,7 @@ public abstract class SearchQueryHandlerFactory extends ListLinkHandlerFactory {
@Override
public abstract String getUrl(String query, List<String> contentFilter, String sortFilter)
throws ParsingException;
throws ParsingException, UnsupportedOperationException;
@SuppressWarnings("unused")
public String getSearchString(final String url) {
@ -27,7 +25,7 @@ public abstract class SearchQueryHandlerFactory extends ListLinkHandlerFactory {
///////////////////////////////////
@Override
public String getId(final String url) {
public String getId(final String url) throws ParsingException, UnsupportedOperationException {
return getSearchString(url);
}
@ -39,7 +37,7 @@ public abstract class SearchQueryHandlerFactory extends ListLinkHandlerFactory {
}
public SearchQueryHandler fromQuery(final String query) throws ParsingException {
return fromQuery(query, Collections.emptyList(), EMPTY_STRING);
return fromQuery(query, Collections.emptyList(), "");
}
/**

View File

@ -1,9 +1,7 @@
package org.schabi.newpipe.extractor.localization;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import org.schabi.newpipe.extractor.utils.LocaleCompat;
import java.io.Serializable;
import java.util.ArrayList;
@ -13,6 +11,11 @@ import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
public class Localization implements Serializable {
public static final Localization DEFAULT = new Localization("en", "GB");
@ -25,32 +28,28 @@ public class Localization implements Serializable {
/**
* @param localizationCodeList a list of localization code, formatted like {@link
* #getLocalizationCode()}
* @throws IllegalArgumentException If any of the localizationCodeList is formatted incorrectly
* @return list of Localization objects
*/
@Nonnull
public static List<Localization> listFrom(final String... localizationCodeList) {
final List<Localization> toReturn = new ArrayList<>();
for (final String localizationCode : localizationCodeList) {
toReturn.add(fromLocalizationCode(localizationCode));
toReturn.add(fromLocalizationCode(localizationCode)
.orElseThrow(() -> new IllegalArgumentException(
"Not a localization code: " + localizationCode
)));
}
return Collections.unmodifiableList(toReturn);
}
/**
* @param localizationCode a localization code, formatted like {@link #getLocalizationCode()}
* @return A Localization, if the code was valid.
*/
public static Localization fromLocalizationCode(final String localizationCode) {
final int indexSeparator = localizationCode.indexOf("-");
final String languageCode;
final String countryCode;
if (indexSeparator != -1) {
languageCode = localizationCode.substring(0, indexSeparator);
countryCode = localizationCode.substring(indexSeparator + 1);
} else {
languageCode = localizationCode;
countryCode = null;
}
return new Localization(languageCode, countryCode);
@Nonnull
public static Optional<Localization> fromLocalizationCode(final String localizationCode) {
return LocaleCompat.forLanguageTag(localizationCode).map(Localization::fromLocale);
}
public Localization(@Nonnull final String languageCode, @Nullable final String countryCode) {
@ -72,10 +71,6 @@ public class Localization implements Serializable {
return countryCode == null ? "" : countryCode;
}
public Locale asLocale() {
return new Locale(getLanguageCode(), getCountryCode());
}
public static Localization fromLocale(@Nonnull final Locale locale) {
return new Localization(locale.getLanguage(), locale.getCountry());
}
@ -83,6 +78,8 @@ public class Localization implements Serializable {
/**
* Return a formatted string in the form of: {@code language-Country}, or
* just {@code language} if country is {@code null}.
*
* @return A correctly formatted localizationCode for this localization.
*/
public String getLocalizationCode() {
return languageCode + (countryCode == null ? "" : "-" + countryCode);

View File

@ -7,15 +7,21 @@ import org.schabi.newpipe.extractor.utils.Parser;
import java.time.OffsetDateTime;
import java.time.ZoneOffset;
import java.time.temporal.ChronoUnit;
import java.util.Collection;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.regex.Pattern;
import java.util.regex.Matcher;
import java.util.regex.MatchResult;
/**
* A helper class that is meant to be used by services that need to parse upload dates in the
* format '2 days ago' or similar.
* A helper class that is meant to be used by services that need to parse durations such as
* {@code 23 seconds} and/or upload dates in the format {@code 2 days ago} or similar.
*/
public class TimeAgoParser {
private static final Pattern DURATION_PATTERN = Pattern.compile("(?:(\\d+) )?([A-z]+)");
private final PatternsHolder patternsHolder;
private final OffsetDateTime now;
@ -58,36 +64,69 @@ public class TimeAgoParser {
}
}
int timeAgoAmount;
try {
timeAgoAmount = parseTimeAgoAmount(textualDate);
} catch (final NumberFormatException e) {
// If there is no valid number in the textual date,
// assume it is 1 (as in 'a second ago').
timeAgoAmount = 1;
}
final ChronoUnit chronoUnit = parseChronoUnit(textualDate);
return getResultFor(timeAgoAmount, chronoUnit);
return getResultFor(parseTimeAgoAmount(textualDate), parseChronoUnit(textualDate));
}
private int parseTimeAgoAmount(final String textualDate) throws NumberFormatException {
return Integer.parseInt(textualDate.replaceAll("\\D+", ""));
/**
* Parses a textual duration into a duration computer number.
*
* @param textualDuration the textual duration to parse
* @return the textual duration parsed, as a primitive {@code long}
* @throws ParsingException if the textual duration could not be parsed
*/
public long parseDuration(final String textualDuration) throws ParsingException {
// We can't use Matcher.results, as it is only available on Android 14 and above
final Matcher matcher = DURATION_PATTERN.matcher(textualDuration);
final List<MatchResult> results = new ArrayList<>();
while (matcher.find()) {
results.add(matcher.toMatchResult());
}
return results.stream()
.map(match -> {
final String digits = match.group(1);
final String word = match.group(2);
int amount;
try {
amount = Integer.parseInt(digits);
} catch (final NumberFormatException ignored) {
amount = 1;
}
final ChronoUnit unit;
try {
unit = parseChronoUnit(word);
} catch (final ParsingException ignored) {
return 0L;
}
return amount * unit.getDuration().getSeconds();
})
.filter(n -> n > 0)
.reduce(Long::sum)
.orElseThrow(() -> new ParsingException(
"Could not parse duration \"" + textualDuration + "\""));
}
private int parseTimeAgoAmount(final String textualDate) {
try {
return Integer.parseInt(textualDate.replaceAll("\\D+", ""));
} catch (final NumberFormatException ignored) {
// If there is no valid number in the textual date,
// assume it is 1 (as in 'a second ago').
return 1;
}
}
private ChronoUnit parseChronoUnit(final String textualDate) throws ParsingException {
for (final Map.Entry<ChronoUnit, Collection<String>> entry
: patternsHolder.asMap().entrySet()) {
final ChronoUnit chronoUnit = entry.getKey();
for (final String agoPhrase : entry.getValue()) {
if (textualDateMatches(textualDate, agoPhrase)) {
return chronoUnit;
}
}
}
throw new ParsingException("Unable to parse the date: " + textualDate);
return patternsHolder.asMap().entrySet().stream()
.filter(e -> e.getValue().stream()
.anyMatch(agoPhrase -> textualDateMatches(textualDate, agoPhrase)))
.map(Map.Entry::getKey)
.findFirst()
.orElseThrow(() ->
new ParsingException("Unable to parse the date: " + textualDate));
}
private boolean textualDateMatches(final String textualDate, final String agoPhrase) {
@ -97,24 +136,22 @@ public class TimeAgoParser {
if (patternsHolder.wordSeparator().isEmpty()) {
return textualDate.toLowerCase().contains(agoPhrase.toLowerCase());
} else {
final String escapedPhrase = Pattern.quote(agoPhrase.toLowerCase());
final String escapedSeparator;
if (patternsHolder.wordSeparator().equals(" ")) {
}
final String escapedPhrase = Pattern.quote(agoPhrase.toLowerCase());
final String escapedSeparator = patternsHolder.wordSeparator().equals(" ")
// From JDK8 \h - Treat horizontal spaces as a normal one
// (non-breaking space, thin space, etc.)
escapedSeparator = "[ \\t\\xA0\\u1680\\u180e\\u2000-\\u200a\\u202f\\u205f\\u3000]";
} else {
escapedSeparator = Pattern.quote(patternsHolder.wordSeparator());
}
// Also split the string on numbers to be able to parse strings like "2wk"
? "[ \\t\\xA0\\u1680\\u180e\\u2000-\\u200a\\u202f\\u205f\\u3000\\d]"
: Pattern.quote(patternsHolder.wordSeparator());
// (^|separator)pattern($|separator)
// Check if the pattern is surrounded by separators or start/end of the string.
final String pattern =
"(^|" + escapedSeparator + ")" + escapedPhrase + "($|" + escapedSeparator + ")";
// (^|separator)pattern($|separator)
// Check if the pattern is surrounded by separators or start/end of the string.
final String pattern =
"(^|" + escapedSeparator + ")" + escapedPhrase + "($|" + escapedSeparator + ")";
return Parser.isMatch(pattern, textualDate.toLowerCase());
}
return Parser.isMatch(pattern, textualDate.toLowerCase());
}
private DateWrapper getResultFor(final int timeAgoAmount, final ChronoUnit chronoUnit) {

View File

@ -1,14 +1,17 @@
package org.schabi.newpipe.extractor.playlist;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.stream.Description;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import javax.annotation.Nonnull;
import static org.schabi.newpipe.extractor.utils.Utils.EMPTY_STRING;
import java.util.Collections;
import java.util.List;
public abstract class PlaylistExtractor extends ListExtractor<StreamInfoItem> {
@ -18,36 +21,38 @@ public abstract class PlaylistExtractor extends ListExtractor<StreamInfoItem> {
public abstract String getUploaderUrl() throws ParsingException;
public abstract String getUploaderName() throws ParsingException;
public abstract String getUploaderAvatarUrl() throws ParsingException;
@Nonnull
public abstract List<Image> getUploaderAvatars() throws ParsingException;
public abstract boolean isUploaderVerified() throws ParsingException;
public abstract long getStreamCount() throws ParsingException;
@Nonnull
public String getThumbnailUrl() throws ParsingException {
return EMPTY_STRING;
public abstract Description getDescription() throws ParsingException;
@Nonnull
public List<Image> getThumbnails() throws ParsingException {
return Collections.emptyList();
}
@Nonnull
public String getBannerUrl() throws ParsingException {
// Banner can't be handled by frontend right now.
// Whoever is willing to implement this should also implement it in the frontend.
return EMPTY_STRING;
public List<Image> getBanners() throws ParsingException {
return List.of();
}
@Nonnull
public String getSubChannelName() throws ParsingException {
return EMPTY_STRING;
return "";
}
@Nonnull
public String getSubChannelUrl() throws ParsingException {
return EMPTY_STRING;
return "";
}
@Nonnull
public String getSubChannelAvatarUrl() throws ParsingException {
return EMPTY_STRING;
public List<Image> getSubChannelAvatars() throws ParsingException {
return List.of();
}
public PlaylistInfo.PlaylistType getPlaylistType() throws ParsingException {

View File

@ -1,5 +1,6 @@
package org.schabi.newpipe.extractor.playlist;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.ListExtractor.InfoItemsPage;
import org.schabi.newpipe.extractor.ListInfo;
import org.schabi.newpipe.extractor.NewPipe;
@ -8,9 +9,11 @@ import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.stream.Description;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import org.schabi.newpipe.extractor.utils.ExtractorHelper;
import javax.annotation.Nonnull;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
@ -103,26 +106,28 @@ public final class PlaylistInfo extends ListInfo<StreamInfoItem> {
info.addError(e);
}
try {
info.setThumbnailUrl(extractor.getThumbnailUrl());
info.setDescription(extractor.getDescription());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setThumbnails(extractor.getThumbnails());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setUploaderUrl(extractor.getUploaderUrl());
} catch (final Exception e) {
info.setUploaderUrl("");
uploaderParsingErrors.add(e);
}
try {
info.setUploaderName(extractor.getUploaderName());
} catch (final Exception e) {
info.setUploaderName("");
uploaderParsingErrors.add(e);
}
try {
info.setUploaderAvatarUrl(extractor.getUploaderAvatarUrl());
info.setUploaderAvatars(extractor.getUploaderAvatars());
} catch (final Exception e) {
info.setUploaderAvatarUrl("");
uploaderParsingErrors.add(e);
}
try {
@ -136,12 +141,12 @@ public final class PlaylistInfo extends ListInfo<StreamInfoItem> {
uploaderParsingErrors.add(e);
}
try {
info.setSubChannelAvatarUrl(extractor.getSubChannelAvatarUrl());
info.setSubChannelAvatars(extractor.getSubChannelAvatars());
} catch (final Exception e) {
uploaderParsingErrors.add(e);
}
try {
info.setBannerUrl(extractor.getBannerUrl());
info.setBanners(extractor.getBanners());
} catch (final Exception e) {
info.addError(e);
}
@ -165,31 +170,38 @@ public final class PlaylistInfo extends ListInfo<StreamInfoItem> {
return info;
}
private String thumbnailUrl;
private String bannerUrl;
private String uploaderUrl;
private String uploaderName;
private String uploaderAvatarUrl;
private String uploaderUrl = "";
private String uploaderName = "";
private String subChannelUrl;
private String subChannelName;
private String subChannelAvatarUrl;
private long streamCount = 0;
private Description description;
@Nonnull
private List<Image> banners = List.of();
@Nonnull
private List<Image> subChannelAvatars = List.of();
@Nonnull
private List<Image> thumbnails = List.of();
@Nonnull
private List<Image> uploaderAvatars = List.of();
private long streamCount;
private PlaylistType playlistType;
public String getThumbnailUrl() {
return thumbnailUrl;
@Nonnull
public List<Image> getThumbnails() {
return thumbnails;
}
public void setThumbnailUrl(final String thumbnailUrl) {
this.thumbnailUrl = thumbnailUrl;
public void setThumbnails(@Nonnull final List<Image> thumbnails) {
this.thumbnails = thumbnails;
}
public String getBannerUrl() {
return bannerUrl;
@Nonnull
public List<Image> getBanners() {
return banners;
}
public void setBannerUrl(final String bannerUrl) {
this.bannerUrl = bannerUrl;
public void setBanners(@Nonnull final List<Image> banners) {
this.banners = banners;
}
public String getUploaderUrl() {
@ -208,12 +220,13 @@ public final class PlaylistInfo extends ListInfo<StreamInfoItem> {
this.uploaderName = uploaderName;
}
public String getUploaderAvatarUrl() {
return uploaderAvatarUrl;
@Nonnull
public List<Image> getUploaderAvatars() {
return uploaderAvatars;
}
public void setUploaderAvatarUrl(final String uploaderAvatarUrl) {
this.uploaderAvatarUrl = uploaderAvatarUrl;
public void setUploaderAvatars(@Nonnull final List<Image> uploaderAvatars) {
this.uploaderAvatars = uploaderAvatars;
}
public String getSubChannelUrl() {
@ -232,12 +245,13 @@ public final class PlaylistInfo extends ListInfo<StreamInfoItem> {
this.subChannelName = subChannelName;
}
public String getSubChannelAvatarUrl() {
return subChannelAvatarUrl;
@Nonnull
public List<Image> getSubChannelAvatars() {
return subChannelAvatars;
}
public void setSubChannelAvatarUrl(final String subChannelAvatarUrl) {
this.subChannelAvatarUrl = subChannelAvatarUrl;
public void setSubChannelAvatars(@Nonnull final List<Image> subChannelAvatars) {
this.subChannelAvatars = subChannelAvatars;
}
public long getStreamCount() {
@ -248,6 +262,14 @@ public final class PlaylistInfo extends ListInfo<StreamInfoItem> {
this.streamCount = streamCount;
}
public Description getDescription() {
return description;
}
public void setDescription(final Description description) {
this.description = description;
}
public PlaylistType getPlaylistType() {
return playlistType;
}

View File

@ -1,14 +1,20 @@
package org.schabi.newpipe.extractor.playlist;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.stream.Description;
import javax.annotation.Nullable;
public class PlaylistInfoItem extends InfoItem {
private String uploaderName;
private String uploaderUrl;
private boolean uploaderVerified;
/**
* How many streams this playlist have
*/
private long streamCount = 0;
private Description description;
private PlaylistInfo.PlaylistType playlistType;
public PlaylistInfoItem(final int serviceId, final String url, final String name) {
@ -23,6 +29,23 @@ public class PlaylistInfoItem extends InfoItem {
this.uploaderName = uploaderName;
}
@Nullable
public String getUploaderUrl() {
return uploaderUrl;
}
public void setUploaderUrl(@Nullable final String uploaderUrl) {
this.uploaderUrl = uploaderUrl;
}
public boolean isUploaderVerified() {
return uploaderVerified;
}
public void setUploaderVerified(final boolean uploaderVerified) {
this.uploaderVerified = uploaderVerified;
}
public long getStreamCount() {
return streamCount;
}
@ -31,6 +54,14 @@ public class PlaylistInfoItem extends InfoItem {
this.streamCount = streamCount;
}
public Description getDescription() {
return description;
}
public void setDescription(final Description description) {
this.description = description;
}
public PlaylistInfo.PlaylistType getPlaylistType() {
return playlistType;
}

View File

@ -2,6 +2,7 @@ package org.schabi.newpipe.extractor.playlist;
import org.schabi.newpipe.extractor.InfoItemExtractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.stream.Description;
import javax.annotation.Nonnull;
@ -13,12 +14,34 @@ public interface PlaylistInfoItemExtractor extends InfoItemExtractor {
*/
String getUploaderName() throws ParsingException;
/**
* Get the uploader url
* @return the uploader url
*/
String getUploaderUrl() throws ParsingException;
/**
* Get whether the uploader is verified
* @return whether the uploader is verified
*/
boolean isUploaderVerified() throws ParsingException;
/**
* Get the number of streams
* @return the number of streams
*/
long getStreamCount() throws ParsingException;
/**
* Get the description of the playlist if there is any.
* Otherwise, an {@link Description#EMPTY_DESCRIPTION EMPTY_DESCRIPTION} is returned.
* @return the playlist's description
*/
@Nonnull
default Description getDescription() throws ParsingException {
return Description.EMPTY_DESCRIPTION;
}
/**
* @return the type of this playlist, see {@link PlaylistInfo.PlaylistType} for a description
* of types. If not overridden always returns {@link PlaylistInfo.PlaylistType#NORMAL}.

View File

@ -22,7 +22,17 @@ public class PlaylistInfoItemsCollector
addError(e);
}
try {
resultItem.setThumbnailUrl(extractor.getThumbnailUrl());
resultItem.setUploaderUrl(extractor.getUploaderUrl());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setUploaderVerified(extractor.isUploaderVerified());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setThumbnails(extractor.getThumbnails());
} catch (final Exception e) {
addError(e);
}
@ -31,6 +41,11 @@ public class PlaylistInfoItemsCollector
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setDescription(extractor.getDescription());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setPlaylistType(extractor.getPlaylistType());
} catch (final Exception e) {

View File

@ -19,7 +19,7 @@ public class SearchInfo extends ListInfo<InfoItem> {
private final String searchString;
private String searchSuggestion;
private boolean isCorrectedSearch;
private List<MetaInfo> metaInfo;
private List<MetaInfo> metaInfo = List.of();
public SearchInfo(final int serviceId,
final SearchQueryHandler qIHandler,

View File

@ -12,6 +12,7 @@ import static org.schabi.newpipe.extractor.services.bandcamp.extractors.Bandcamp
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.ChannelExtractor;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import org.schabi.newpipe.extractor.comments.CommentsExtractor;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.kiosk.KioskList;
@ -19,11 +20,13 @@ import org.schabi.newpipe.extractor.linkhandler.LinkHandler;
import org.schabi.newpipe.extractor.linkhandler.LinkHandlerFactory;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import org.schabi.newpipe.extractor.linkhandler.ReadyChannelTabListLinkHandler;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandler;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandlerFactory;
import org.schabi.newpipe.extractor.playlist.PlaylistExtractor;
import org.schabi.newpipe.extractor.search.SearchExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampChannelExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampChannelTabExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampCommentsExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampFeaturedExtractor;
@ -34,6 +37,7 @@ import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampSearchE
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampStreamExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampSuggestionExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampChannelLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampChannelTabLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampCommentsLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampFeaturedLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampPlaylistLinkHandlerFactory;
@ -58,27 +62,32 @@ public class BandcampService extends StreamingService {
@Override
public LinkHandlerFactory getStreamLHFactory() {
return new BandcampStreamLinkHandlerFactory();
return BandcampStreamLinkHandlerFactory.getInstance();
}
@Override
public ListLinkHandlerFactory getChannelLHFactory() {
return new BandcampChannelLinkHandlerFactory();
return BandcampChannelLinkHandlerFactory.getInstance();
}
@Override
public ListLinkHandlerFactory getChannelTabLHFactory() {
return BandcampChannelTabLinkHandlerFactory.getInstance();
}
@Override
public ListLinkHandlerFactory getPlaylistLHFactory() {
return new BandcampPlaylistLinkHandlerFactory();
return BandcampPlaylistLinkHandlerFactory.getInstance();
}
@Override
public SearchQueryHandlerFactory getSearchQHFactory() {
return new BandcampSearchQueryHandlerFactory();
return BandcampSearchQueryHandlerFactory.getInstance();
}
@Override
public ListLinkHandlerFactory getCommentsLHFactory() {
return new BandcampCommentsLinkHandlerFactory();
return BandcampCommentsLinkHandlerFactory.getInstance();
}
@Override
@ -98,27 +107,27 @@ public class BandcampService extends StreamingService {
@Override
public KioskList getKioskList() throws ExtractionException {
final KioskList kioskList = new KioskList(this);
final ListLinkHandlerFactory h = BandcampFeaturedLinkHandlerFactory.getInstance();
try {
kioskList.addKioskEntry(
(streamingService, url, kioskId) -> new BandcampFeaturedExtractor(
BandcampService.this,
new BandcampFeaturedLinkHandlerFactory().fromUrl(FEATURED_API_URL),
h.fromUrl(FEATURED_API_URL),
kioskId
),
new BandcampFeaturedLinkHandlerFactory(),
h,
KIOSK_FEATURED
);
kioskList.addKioskEntry(
(streamingService, url, kioskId) -> new BandcampRadioExtractor(
BandcampService.this,
new BandcampFeaturedLinkHandlerFactory().fromUrl(RADIO_API_URL),
h.fromUrl(RADIO_API_URL),
kioskId
),
new BandcampFeaturedLinkHandlerFactory(),
h,
KIOSK_RADIO
);
@ -136,6 +145,15 @@ public class BandcampService extends StreamingService {
return new BandcampChannelExtractor(this, linkHandler);
}
@Override
public ChannelTabExtractor getChannelTabExtractor(final ListLinkHandler linkHandler) {
if (linkHandler instanceof ReadyChannelTabListLinkHandler) {
return ((ReadyChannelTabListLinkHandler) linkHandler).getChannelTabExtractor(this);
} else {
return new BandcampChannelTabExtractor(this, linkHandler);
}
}
@Override
public PlaylistExtractor getPlaylistExtractor(final ListLinkHandler linkHandler) {
return new BandcampPlaylistExtractor(this, linkHandler);

View File

@ -0,0 +1,62 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.playlist.PlaylistInfoItemExtractor;
import java.util.List;
import javax.annotation.Nonnull;
public class BandcampAlbumInfoItemExtractor implements PlaylistInfoItemExtractor {
private final JsonObject albumInfoItem;
private final String uploaderUrl;
public BandcampAlbumInfoItemExtractor(final JsonObject albumInfoItem,
final String uploaderUrl) {
this.albumInfoItem = albumInfoItem;
this.uploaderUrl = uploaderUrl;
}
@Override
public String getName() throws ParsingException {
return albumInfoItem.getString("title");
}
@Override
public String getUrl() throws ParsingException {
return BandcampExtractorHelper.getStreamUrlFromIds(
albumInfoItem.getLong("band_id"),
albumInfoItem.getLong("item_id"),
albumInfoItem.getString("item_type"));
}
@Nonnull
@Override
public List<Image> getThumbnails() throws ParsingException {
return BandcampExtractorHelper.getImagesFromImageId(albumInfoItem.getLong("art_id"), true);
}
@Override
public String getUploaderName() throws ParsingException {
return albumInfoItem.getString("band_name");
}
@Override
public String getUploaderUrl() {
return uploaderUrl;
}
@Override
public boolean isUploaderVerified() {
return false;
}
@Override
public long getStreamCount() {
return ListExtractor.ITEM_COUNT_UNKNOWN;
}
}

View File

@ -2,23 +2,37 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import static org.schabi.newpipe.extractor.Image.HEIGHT_UNKNOWN;
import static org.schabi.newpipe.extractor.Image.WIDTH_UNKNOWN;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getArtistDetails;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
import static org.schabi.newpipe.extractor.utils.Utils.replaceHttpWithHttps;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import org.jsoup.Jsoup;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.Image.ResolutionLevel;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.ChannelExtractor;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.exceptions.ReCaptchaException;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabs;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.streaminfoitem.BandcampDiscographStreamInfoItemExtractor;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import org.schabi.newpipe.extractor.stream.StreamInfoItemsCollector;
import org.schabi.newpipe.extractor.linkhandler.ReadyChannelTabListLinkHandler;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampChannelTabLinkHandlerFactory;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Objects;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import javax.annotation.Nonnull;
@ -31,38 +45,36 @@ public class BandcampChannelExtractor extends ChannelExtractor {
super(service, linkHandler);
}
@Nonnull
@Override
public String getAvatarUrl() {
if (channelInfo.getLong("bio_image_id") == 0) {
return "";
}
return BandcampExtractorHelper.getImageUrl(channelInfo.getLong("bio_image_id"), false);
public List<Image> getAvatars() {
return getImagesFromImageId(channelInfo.getLong("bio_image_id"), false);
}
@Nonnull
@Override
public String getBannerUrl() throws ParsingException {
public List<Image> getBanners() throws ParsingException {
/*
* Mobile API does not return the header or not the correct header.
* Therefore, we need to query the website
*/
try {
final String html = getDownloader()
.get(channelInfo.getString("bandcamp_url")
.replace("http://", "https://"))
.responseBody();
.get(replaceHttpWithHttps(channelInfo.getString("bandcamp_url")))
.responseBody();
return Jsoup.parse(html)
.getElementById("customHeader")
.getElementsByTag("img")
.first()
.attr("src");
return Stream.of(Jsoup.parse(html).getElementById("customHeader"))
.filter(Objects::nonNull)
.flatMap(element -> element.getElementsByTag("img").stream())
.map(element -> element.attr("src"))
.filter(url -> !url.isEmpty())
.map(url -> new Image(
replaceHttpWithHttps(url), HEIGHT_UNKNOWN, WIDTH_UNKNOWN,
ResolutionLevel.UNKNOWN))
.collect(Collectors.toUnmodifiableList());
} catch (final IOException | ReCaptchaException e) {
throw new ParsingException("Could not download artist web site", e);
} catch (final NullPointerException e) {
// No banner available
return "";
}
}
@ -94,9 +106,10 @@ public class BandcampChannelExtractor extends ChannelExtractor {
return null;
}
@Nonnull
@Override
public String getParentChannelAvatarUrl() {
return null;
public List<Image> getParentChannelAvatars() {
return List.of();
}
@Override
@ -106,35 +119,53 @@ public class BandcampChannelExtractor extends ChannelExtractor {
@Nonnull
@Override
public InfoItemsPage<StreamInfoItem> getInitialPage() throws ParsingException {
final StreamInfoItemsCollector collector = new StreamInfoItemsCollector(getServiceId());
public List<ListLinkHandler> getTabs() throws ParsingException {
final JsonArray discography = channelInfo.getArray("discography");
final TabExtractorBuilder builder = new TabExtractorBuilder(discography);
for (int i = 0; i < discography.size(); i++) {
// A discograph is as an item appears in a discography
final JsonObject discograph = discography.getObject(i);
final List<ListLinkHandler> tabs = new ArrayList<>();
if (!discograph.getString("item_type").equals("track")) {
boolean foundTrackItem = false;
boolean foundAlbumItem = false;
for (final Object discographyItem : discography) {
if (foundTrackItem && foundAlbumItem) {
break;
}
if (!(discographyItem instanceof JsonObject)) {
continue;
}
collector.commit(new BandcampDiscographStreamInfoItemExtractor(discograph, getUrl()));
final JsonObject discographyJsonItem = (JsonObject) discographyItem;
final String itemType = discographyJsonItem.getString("item_type");
if (!foundTrackItem && "track".equals(itemType)) {
foundTrackItem = true;
tabs.add(new ReadyChannelTabListLinkHandler(getUrl()
+ BandcampChannelTabLinkHandlerFactory.getUrlSuffix(ChannelTabs.TRACKS),
getId(),
ChannelTabs.TRACKS,
builder));
}
if (!foundAlbumItem && "album".equals(itemType)) {
foundAlbumItem = true;
tabs.add(new ReadyChannelTabListLinkHandler(getUrl()
+ BandcampChannelTabLinkHandlerFactory.getUrlSuffix(ChannelTabs.ALBUMS),
getId(),
ChannelTabs.ALBUMS,
builder));
}
}
return new InfoItemsPage<>(collector, null);
}
@Override
public InfoItemsPage<StreamInfoItem> getPage(final Page page) {
return null;
return Collections.unmodifiableList(tabs);
}
@Override
public void onFetchPage(@Nonnull final Downloader downloader)
throws IOException, ExtractionException {
channelInfo = BandcampExtractorHelper.getArtistDetails(getId());
channelInfo = getArtistDetails(getId());
}
@Nonnull
@ -142,4 +173,20 @@ public class BandcampChannelExtractor extends ChannelExtractor {
public String getName() {
return channelInfo.getString("name");
}
private static final class TabExtractorBuilder
implements ReadyChannelTabListLinkHandler.ChannelTabExtractorBuilder {
private final JsonArray discography;
TabExtractorBuilder(final JsonArray discography) {
this.discography = discography;
}
@Nonnull
@Override
public ChannelTabExtractor build(@Nonnull final StreamingService service,
@Nonnull final ListLinkHandler linkHandler) {
return BandcampChannelTabExtractor.fromDiscography(service, linkHandler, discography);
}
}
}

View File

@ -3,9 +3,15 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import org.jsoup.nodes.Element;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.channel.ChannelInfoItemExtractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import javax.annotation.Nonnull;
import java.util.List;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromSearchResult;
public class BandcampChannelInfoItemExtractor implements ChannelInfoItemExtractor {
private final Element resultInfo;
@ -26,15 +32,10 @@ public class BandcampChannelInfoItemExtractor implements ChannelInfoItemExtracto
return resultInfo.getElementsByClass("itemurl").text();
}
@Nonnull
@Override
public String getThumbnailUrl() throws ParsingException {
final Element img = searchResult.getElementsByClass("art").first()
.getElementsByTag("img").first();
if (img != null) {
return img.attr("src");
} else {
return null;
}
public List<Image> getThumbnails() throws ParsingException {
return getImagesFromSearchResult(searchResult);
}
@Override

View File

@ -0,0 +1,95 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.MultiInfoItemsCollector;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabs;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.streaminfoitem.BandcampDiscographStreamInfoItemExtractor;
import javax.annotation.Nonnull;
import java.io.IOException;
public class BandcampChannelTabExtractor extends ChannelTabExtractor {
private JsonArray discography;
private final String filter;
public BandcampChannelTabExtractor(final StreamingService service,
final ListLinkHandler linkHandler) {
super(service, linkHandler);
final String tab = linkHandler.getContentFilters().get(0);
switch (tab) {
case ChannelTabs.TRACKS:
filter = "track";
break;
case ChannelTabs.ALBUMS:
filter = "album";
break;
default:
throw new IllegalArgumentException("Unsupported channel tab: " + tab);
}
}
public static BandcampChannelTabExtractor fromDiscography(final StreamingService service,
final ListLinkHandler linkHandler,
final JsonArray discography) {
final BandcampChannelTabExtractor tabExtractor =
new BandcampChannelTabExtractor(service, linkHandler);
tabExtractor.discography = discography;
return tabExtractor;
}
@Override
public void onFetchPage(@Nonnull final Downloader downloader) throws ParsingException {
if (discography == null) {
discography = BandcampExtractorHelper.getArtistDetails(getId())
.getArray("discography");
}
}
@Nonnull
@Override
public InfoItemsPage<InfoItem> getInitialPage() throws IOException, ExtractionException {
final MultiInfoItemsCollector collector = new MultiInfoItemsCollector(getServiceId());
for (final Object discograph : discography) {
// A discograph is as an item appears in a discography
if (!(discograph instanceof JsonObject)) {
continue;
}
final JsonObject discographJsonObject = (JsonObject) discograph;
final String itemType = discographJsonObject.getString("item_type", "");
if (!itemType.equals(filter)) {
continue;
}
switch (itemType) {
case "track":
collector.commit(new BandcampDiscographStreamInfoItemExtractor(
discographJsonObject, getUrl()));
break;
case "album":
collector.commit(new BandcampAlbumInfoItemExtractor(
discographJsonObject, getUrl()));
break;
}
}
return new InfoItemsPage<>(collector, null);
}
@Override
public InfoItemsPage<InfoItem> getPage(final Page page) {
return null;
}
}

View File

@ -1,9 +1,13 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_API_URL;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonWriter;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.comments.CommentsExtractor;
@ -11,13 +15,22 @@ import org.schabi.newpipe.extractor.comments.CommentsInfoItem;
import org.schabi.newpipe.extractor.comments.CommentsInfoItemsCollector;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.exceptions.ReCaptchaException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.utils.JsonUtils;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.Collections;
import java.util.List;
import javax.annotation.Nonnull;
import java.io.IOException;
public class BandcampCommentsExtractor extends CommentsExtractor {
private static final String REVIEWS_API_URL = BASE_API_URL + "/tralbumcollectors/2/reviews";
private Document document;
@ -39,18 +52,85 @@ public class BandcampCommentsExtractor extends CommentsExtractor {
final CommentsInfoItemsCollector collector = new CommentsInfoItemsCollector(getServiceId());
final Elements writings = document.getElementsByClass("writing");
final JsonObject collectorsData = JsonUtils.toJsonObject(
document.getElementById("collectors-data").attr("data-blob"));
final JsonArray reviews = collectorsData.getArray("reviews");
for (final Element writing : writings) {
collector.commit(new BandcampCommentsInfoItemExtractor(writing, getUrl()));
for (final Object review : reviews) {
collector.commit(
new BandcampCommentsInfoItemExtractor((JsonObject) review, getUrl()));
}
return new InfoItemsPage<>(collector, null);
if (!collectorsData.getBoolean("more_reviews_available")) {
return new InfoItemsPage<>(collector, null);
}
final String trackId = getTrackId();
final String token = getNextPageToken(reviews);
return new InfoItemsPage<>(collector, new Page(List.of(trackId, token)));
}
@Override
public InfoItemsPage<CommentsInfoItem> getPage(final Page page)
throws IOException, ExtractionException {
return null;
final CommentsInfoItemsCollector collector = new CommentsInfoItemsCollector(getServiceId());
final List<String> pageIds = page.getIds();
final String trackId = pageIds.get(0);
final String token = pageIds.get(1);
final JsonObject reviewsData = fetchReviewsData(trackId, token);
final JsonArray reviews = reviewsData.getArray("results");
for (final Object review : reviews) {
collector.commit(
new BandcampCommentsInfoItemExtractor((JsonObject) review, getUrl()));
}
if (!reviewsData.getBoolean("more_available")) {
return new InfoItemsPage<>(collector, null);
}
return new InfoItemsPage<>(collector,
new Page(List.of(trackId, getNextPageToken(reviews))));
}
private JsonObject fetchReviewsData(final String trackId, final String token)
throws ParsingException {
try {
return JsonUtils.toJsonObject(getDownloader().postWithContentTypeJson(
REVIEWS_API_URL,
Collections.emptyMap(),
JsonWriter.string().object()
.value("tralbum_type", "t")
.value("tralbum_id", trackId)
.value("token", token)
.value("count", 7)
.array("exclude_fan_ids").end()
.end().done().getBytes(StandardCharsets.UTF_8)).responseBody());
} catch (final IOException | ReCaptchaException e) {
throw new ParsingException("Could not fetch reviews", e);
}
}
private String getNextPageToken(final JsonArray reviews) throws ParsingException {
return reviews.stream()
.filter(JsonObject.class::isInstance)
.map(JsonObject.class::cast)
.map(review -> review.getString("token"))
.reduce((a, b) -> b) // keep only the last element
.orElseThrow(() -> new ParsingException("Could not get token"));
}
private String getTrackId() throws ParsingException {
final JsonObject pageProperties = JsonUtils.toJsonObject(
document.selectFirst("meta[name=bc-page-properties]")
.attr("content"));
return Long.toString(pageProperties.getLong("item_id"));
}
@Override
public boolean isCommentsDisabled() throws ExtractionException {
return BandcampExtractorHelper.isRadioUrl(getUrl());
}
}

View File

@ -1,22 +1,30 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import org.jsoup.nodes.Element;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.comments.CommentsInfoItemExtractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.stream.Description;
import javax.annotation.Nonnull;
import java.util.List;
public class BandcampCommentsInfoItemExtractor implements CommentsInfoItemExtractor {
private final Element writing;
private final JsonObject review;
private final String url;
public BandcampCommentsInfoItemExtractor(final Element writing, final String url) {
this.writing = writing;
public BandcampCommentsInfoItemExtractor(final JsonObject review, final String url) {
this.review = review;
this.url = url;
}
@Override
public String getName() throws ParsingException {
return writing.getElementsByClass("text").first().ownText();
return getCommentText().getContent();
}
@Override
@ -24,23 +32,26 @@ public class BandcampCommentsInfoItemExtractor implements CommentsInfoItemExtrac
return url;
}
@Nonnull
@Override
public String getThumbnailUrl() throws ParsingException {
return writing.getElementsByClass("thumb").attr("src");
public List<Image> getThumbnails() throws ParsingException {
return getUploaderAvatars();
}
@Nonnull
@Override
public String getCommentText() {
return writing.getElementsByClass("text").first().ownText();
public Description getCommentText() throws ParsingException {
return new Description(review.getString("why"), Description.PLAIN_TEXT);
}
@Override
public String getUploaderName() throws ParsingException {
return writing.getElementsByClass("name").first().text();
return review.getString("name");
}
@Nonnull
@Override
public String getUploaderAvatarUrl() {
return writing.getElementsByClass("thumb").attr("src");
public List<Image> getUploaderAvatars() {
return getImagesFromImageId(review.getLong("image_id"), false);
}
}

View File

@ -8,19 +8,79 @@ import com.grack.nanojson.JsonParserException;
import com.grack.nanojson.JsonWriter;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Element;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.Image.ResolutionLevel;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.exceptions.ReCaptchaException;
import org.schabi.newpipe.extractor.localization.DateWrapper;
import org.schabi.newpipe.extractor.utils.ImageSuffix;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.time.DateTimeException;
import java.time.ZonedDateTime;
import java.time.format.DateTimeFormatter;
import java.util.Collections;
import java.util.List;
import java.util.Locale;
import java.util.stream.Collectors;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import static org.schabi.newpipe.extractor.Image.HEIGHT_UNKNOWN;
import static org.schabi.newpipe.extractor.Image.WIDTH_UNKNOWN;
import static org.schabi.newpipe.extractor.utils.Utils.isNullOrEmpty;
import static org.schabi.newpipe.extractor.utils.Utils.replaceHttpWithHttps;
public final class BandcampExtractorHelper {
/**
* List of image IDs which preserve aspect ratio with their theoretical dimension known.
*
* <p>
* Bandcamp images are not always squares, so images which preserve aspect ratio are only used.
* </p>
*
* <p>
* One of the direct consequences of this specificity is that only one dimension of images is
* known at time, depending of the image ID.
* </p>
*
* <p>
* Note also that dimensions are only theoretical because if the image size is less than the
* dimensions of the image ID, it will be not upscaled but kept to its original size.
* </p>
*
* <p>
* IDs come from <a href="https://gist.github.com/f2k1de/06f5fd0ae9c919a7c3693a44ee522213">the
* GitHub Gist "Bandcamp File Format Parameters" by f2k1de</a>
* </p>
*/
private static final List<ImageSuffix> IMAGE_URL_SUFFIXES_AND_RESOLUTIONS = List.of(
// ID | HEIGHT | WIDTH
new ImageSuffix("10.jpg", HEIGHT_UNKNOWN, 1200, ResolutionLevel.HIGH),
new ImageSuffix("101.jpg", 90, WIDTH_UNKNOWN, ResolutionLevel.LOW),
new ImageSuffix("170.jpg", 422, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM),
// 180 returns the same image aspect ratio and size as 171
new ImageSuffix("171.jpg", 646, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM),
new ImageSuffix("20.jpg", HEIGHT_UNKNOWN, 1024, ResolutionLevel.HIGH),
// 203 returns the same image aspect ratio and size as 200
new ImageSuffix("200.jpg", 420, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM),
new ImageSuffix("201.jpg", 280, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM),
new ImageSuffix("202.jpg", 140, WIDTH_UNKNOWN, ResolutionLevel.LOW),
new ImageSuffix("204.jpg", 360, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM),
new ImageSuffix("205.jpg", 240, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM),
new ImageSuffix("206.jpg", 180, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM),
new ImageSuffix("207.jpg", 120, WIDTH_UNKNOWN, ResolutionLevel.LOW),
new ImageSuffix("43.jpg", 100, WIDTH_UNKNOWN, ResolutionLevel.LOW),
new ImageSuffix("44.jpg", 200, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM));
private static final String IMAGE_URL_APPENDIX_AND_EXTENSION_REGEX = "_\\d+\\.\\w+";
private static final String IMAGES_DOMAIN_AND_PATH = "https://f4.bcbits.com/img/";
public static final String BASE_URL = "https://bandcamp.com";
public static final String BASE_API_URL = BASE_URL + "/api";
@ -40,8 +100,8 @@ public final class BandcampExtractorHelper {
+ "&tralbum_id=" + itemId + "&tralbum_type=" + itemType.charAt(0))
.responseBody();
return JsonParser.object().from(jsonString)
.getString("bandcamp_url").replace("http://", "https://");
return replaceHttpWithHttps(JsonParser.object().from(jsonString)
.getString("bandcamp_url"));
} catch (final JsonParserException | ReCaptchaException | IOException e) {
throw new ParsingException("Ids could not be translated to URL", e);
@ -51,41 +111,47 @@ public final class BandcampExtractorHelper {
/**
* Fetch artist details from mobile endpoint.
* <a href=https://notabug.org/fynngodau/bandcampDirect/wiki/rewindBandcamp+%E2%80%93+Fetching+artist+details>
* <a href="https://notabug.org/fynngodau/bandcampDirect/wiki/
* rewindBandcamp+%E2%80%93+Fetching+artist+details">
* More technical info.</a>
*/
public static JsonObject getArtistDetails(final String id) throws ParsingException {
try {
return
JsonParser.object().from(
NewPipe.getDownloader().post(
BASE_API_URL + "/mobile/22/band_details",
null,
JsonWriter.string()
.object()
.value("band_id", id)
.end()
.done()
.getBytes()
).responseBody()
);
return JsonParser.object().from(NewPipe.getDownloader().postWithContentTypeJson(
BASE_API_URL + "/mobile/22/band_details",
Collections.emptyMap(),
JsonWriter.string()
.object()
.value("band_id", id)
.end()
.done()
.getBytes(StandardCharsets.UTF_8)).responseBody());
} catch (final IOException | ReCaptchaException | JsonParserException e) {
throw new ParsingException("Could not download band details", e);
}
}
/**
* Generate image url from image ID.
* <p>
* The appendix "_10" was chosen because it provides images sized 1200x1200. Other integer
* values are possible as well (e.g. 0 is a very large resolution, possibly the original).
* Generate an image url from an image ID.
*
* @param id The image ID
* @param album True if this is the cover of an album or track
* @return URL of image with this ID sized 1200x1200
* <p>
* The image ID {@code 10} was chosen because it provides images wide up to 1200px (when
* the original image width is more than or equal this resolution).
* </p>
*
* <p>
* Other integer values are possible as well (e.g. 0 is a very large resolution, possibly the
* original); see {@link #IMAGE_URL_SUFFIXES_AND_RESOLUTIONS} for more details about image
* resolution IDs.
* </p>
*
* @param id the image ID
* @param isAlbum whether the image is the cover of an album or a track
* @return a URL of the image with this ID with a width up to 1200px
*/
public static String getImageUrl(final long id, final boolean album) {
return "https://f4.bcbits.com/img/" + (album ? 'a' : "") + id + "_10.jpg";
@Nonnull
public static String getImageUrl(final long id, final boolean isAlbum) {
return IMAGES_DOMAIN_AND_PATH + (isAlbum ? 'a' : "") + id + "_10.jpg";
}
/**
@ -119,7 +185,7 @@ public final class BandcampExtractorHelper {
/**
* Whether the URL points to a radio kiosk.
* @param url the URL to check
* @return true if the URL matches <code>https://bandcamp.com/?show=SHOW_ID</code>
* @return true if the URL matches {@code https://bandcamp.com/?show=SHOW_ID}
*/
public static boolean isRadioUrl(final String url) {
return url.toLowerCase().matches("https?://bandcamp\\.com/\\?show=\\d+");
@ -134,4 +200,95 @@ public final class BandcampExtractorHelper {
throw new ParsingException("Could not parse date '" + textDate + "'", e);
}
}
/**
* Get a list of images from a search result {@link Element}.
*
* <p>
* This method will call {@link #getImagesFromImageUrl(String)} using the first non null and
* non empty image URL found from the {@code src} attribute of {@code img} HTML elements, or an
* empty string if no valid image URL was found.
* </p>
*
* @param searchResult a search result {@link Element}
* @return an unmodifiable list of {@link Image}s, which is never null but can be empty, in the
* case where no valid image URL was found
*/
@Nonnull
public static List<Image> getImagesFromSearchResult(@Nonnull final Element searchResult) {
return getImagesFromImageUrl(searchResult.getElementsByClass("art")
.stream()
.flatMap(element -> element.getElementsByTag("img").stream())
.map(element -> element.attr("src"))
.filter(imageUrl -> !isNullOrEmpty(imageUrl))
.findFirst()
.orElse(""));
}
/**
* Get all images which have resolutions preserving aspect ratio from an image URL.
*
* <p>
* This method will remove the image ID and its extension from the end of the URL and then call
* {@link #getImagesFromImageBaseUrl(String)}.
* </p>
*
* @param imageUrl the full URL of an image provided by Bandcamp, such as in its HTML code
* @return an unmodifiable list of {@link Image}s, which is never null but can be empty, in the
* case where the image URL has been not extracted (and so is null or empty)
*/
@Nonnull
public static List<Image> getImagesFromImageUrl(@Nullable final String imageUrl) {
if (isNullOrEmpty(imageUrl)) {
return List.of();
}
return getImagesFromImageBaseUrl(
imageUrl.replaceFirst(IMAGE_URL_APPENDIX_AND_EXTENSION_REGEX, "_"));
}
/**
* Get all images which have resolutions preserving aspect ratio from an image ID.
*
* <p>
* This method will call {@link #getImagesFromImageBaseUrl(String)}.
* </p>
*
* @param id the id of an image provided by Bandcamp
* @param isAlbum whether the image is the cover of an album
* @return an unmodifiable list of {@link Image}s, which is never null but can be empty, in the
* case where the image ID has been not extracted (and so equal to 0)
*/
@Nonnull
public static List<Image> getImagesFromImageId(final long id, final boolean isAlbum) {
if (id == 0) {
return List.of();
}
return getImagesFromImageBaseUrl(IMAGES_DOMAIN_AND_PATH + (isAlbum ? 'a' : "") + id + "_");
}
/**
* Get all images resolutions preserving aspect ratio from a base image URL.
*
* <p>
* Base image URLs are images containing the image path, a {@code a} letter if it comes from an
* album, its ID and an underscore.
* </p>
*
* <p>
* Images resolutions returned are the ones of {@link #IMAGE_URL_SUFFIXES_AND_RESOLUTIONS}.
* </p>
*
* @param baseUrl the base URL of the image
* @return an unmodifiable and non-empty list of {@link Image}s
*/
@Nonnull
private static List<Image> getImagesFromImageBaseUrl(@Nonnull final String baseUrl) {
return IMAGE_URL_SUFFIXES_AND_RESOLUTIONS.stream()
.map(imageSuffix -> new Image(baseUrl + imageSuffix.getSuffix(),
imageSuffix.getHeight(), imageSuffix.getWidth(),
imageSuffix.getResolutionLevel()))
.collect(Collectors.toUnmodifiableList());
}
}

View File

@ -18,6 +18,8 @@ import org.schabi.newpipe.extractor.playlist.PlaylistInfoItemsCollector;
import javax.annotation.Nonnull;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.Collections;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_API_URL;
@ -40,11 +42,11 @@ public class BandcampFeaturedExtractor extends KioskExtractor<PlaylistInfoItem>
public void onFetchPage(@Nonnull final Downloader downloader)
throws IOException, ExtractionException {
try {
json = JsonParser.object().from(
getDownloader().post(
FEATURED_API_URL, null, "{\"platform\":\"\",\"version\":0}".getBytes()
).responseBody()
);
json = JsonParser.object().from(getDownloader().postWithContentTypeJson(
FEATURED_API_URL,
Collections.emptyMap(),
"{\"platform\":\"\",\"version\":0}".getBytes(StandardCharsets.UTF_8))
.responseBody());
} catch (final JsonParserException e) {
throw new ParsingException("Could not parse Bandcamp featured API response", e);
}

View File

@ -1,31 +1,38 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageUrl;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampStreamExtractor.getAlbumInfoJson;
import static org.schabi.newpipe.extractor.utils.JsonUtils.getJsonData;
import static org.schabi.newpipe.extractor.utils.Utils.HTTPS;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParserException;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ContentNotAvailableException;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.PaidContentException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.playlist.PlaylistExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.streaminfoitem.BandcampPlaylistStreamInfoItemExtractor;
import org.schabi.newpipe.extractor.stream.Description;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import org.schabi.newpipe.extractor.stream.StreamInfoItemsCollector;
import javax.annotation.Nonnull;
import java.io.IOException;
import java.util.Objects;
import java.util.List;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImageUrl;
import static org.schabi.newpipe.extractor.utils.JsonUtils.getJsonData;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampStreamExtractor.getAlbumInfoJson;
import static org.schabi.newpipe.extractor.utils.Utils.EMPTY_STRING;
import static org.schabi.newpipe.extractor.utils.Utils.HTTPS;
import javax.annotation.Nonnull;
public class BandcampPlaylistExtractor extends PlaylistExtractor {
@ -64,17 +71,17 @@ public class BandcampPlaylistExtractor extends PlaylistExtractor {
if (trackInfo.isEmpty()) {
// Albums without trackInfo need to be purchased before they can be played
throw new ContentNotAvailableException("Album needs to be purchased");
throw new PaidContentException("Album needs to be purchased");
}
}
@Nonnull
@Override
public String getThumbnailUrl() throws ParsingException {
public List<Image> getThumbnails() throws ParsingException {
if (albumJson.isNull("art_id")) {
return EMPTY_STRING;
return List.of();
} else {
return getImageUrl(albumJson.getLong("art_id"), true);
return getImagesFromImageId(albumJson.getLong("art_id"), true);
}
}
@ -90,14 +97,14 @@ public class BandcampPlaylistExtractor extends PlaylistExtractor {
return albumJson.getString("artist");
}
@Nonnull
@Override
public String getUploaderAvatarUrl() {
try {
return Objects.requireNonNull(document.getElementsByClass("band-photo").first())
.attr("src");
} catch (final NullPointerException e) {
return EMPTY_STRING;
}
public List<Image> getUploaderAvatars() {
return getImagesFromImageUrl(document.getElementsByClass("band-photo")
.stream()
.map(element -> element.attr("src"))
.findFirst()
.orElse(""));
}
@Override
@ -110,6 +117,32 @@ public class BandcampPlaylistExtractor extends PlaylistExtractor {
return trackInfo.size();
}
@Nonnull
@Override
public Description getDescription() throws ParsingException {
final Element tInfo = document.getElementById("trackInfo");
if (tInfo == null) {
throw new ParsingException("Could not find trackInfo in document");
}
final Elements about = tInfo.getElementsByClass("tralbum-about");
final Elements credits = tInfo.getElementsByClass("tralbum-credits");
final Element license = document.getElementById("license");
if (about.isEmpty() && credits.isEmpty() && license == null) {
return Description.EMPTY_DESCRIPTION;
}
final StringBuilder sb = new StringBuilder();
if (!about.isEmpty()) {
sb.append(Objects.requireNonNull(about.first()).html());
}
if (!credits.isEmpty()) {
sb.append(Objects.requireNonNull(credits.first()).html());
}
if (license != null) {
sb.append(license.html());
}
return new Description(sb.toString(), Description.HTML);
}
@Nonnull
@Override
public InfoItemsPage<StreamInfoItem> getInitialPage() throws ExtractionException {
@ -126,7 +159,7 @@ public class BandcampPlaylistExtractor extends PlaylistExtractor {
} else {
// Pretend every track has the same cover art as the album
collector.commit(new BandcampPlaylistStreamInfoItemExtractor(
track, getUploaderUrl(), getThumbnailUrl()));
track, getUploaderUrl(), getThumbnails()));
}
}

View File

@ -1,9 +1,13 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import org.jsoup.nodes.Element;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.playlist.PlaylistInfoItemExtractor;
import javax.annotation.Nonnull;
import java.util.List;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromSearchResult;
public class BandcampPlaylistInfoItemExtractor implements PlaylistInfoItemExtractor {
private final Element searchResult;
@ -20,6 +24,16 @@ public class BandcampPlaylistInfoItemExtractor implements PlaylistInfoItemExtrac
.split(" by")[0];
}
@Override
public String getUploaderUrl() {
return null;
}
@Override
public boolean isUploaderVerified() {
return false;
}
@Override
public long getStreamCount() {
final String length = resultInfo.getElementsByClass("length").text();
@ -36,14 +50,9 @@ public class BandcampPlaylistInfoItemExtractor implements PlaylistInfoItemExtrac
return resultInfo.getElementsByClass("itemurl").text();
}
@Nonnull
@Override
public String getThumbnailUrl() {
final Element img = searchResult.getElementsByClass("art").first()
.getElementsByTag("img").first();
if (img != null) {
return img.attr("src");
} else {
return null;
}
public List<Image> getThumbnails() {
return getImagesFromSearchResult(searchResult);
}
}

View File

@ -1,9 +1,14 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.playlist.PlaylistInfoItemExtractor;
import org.schabi.newpipe.extractor.utils.Utils;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImageUrl;
import javax.annotation.Nonnull;
import java.util.List;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
public class BandcampPlaylistInfoItemFeaturedExtractor implements PlaylistInfoItemExtractor {
@ -18,6 +23,16 @@ public class BandcampPlaylistInfoItemFeaturedExtractor implements PlaylistInfoIt
return featuredStory.getString("band_name");
}
@Override
public String getUploaderUrl() {
return null;
}
@Override
public boolean isUploaderVerified() {
return false;
}
@Override
public long getStreamCount() {
return featuredStory.getInt("num_streamable_tracks");
@ -30,12 +45,14 @@ public class BandcampPlaylistInfoItemFeaturedExtractor implements PlaylistInfoIt
@Override
public String getUrl() {
return featuredStory.getString("item_url").replaceAll("http://", "https://");
return Utils.replaceHttpWithHttps(featuredStory.getString("item_url"));
}
@Nonnull
@Override
public String getThumbnailUrl() {
return featuredStory.has("art_id") ? getImageUrl(featuredStory.getLong("art_id"), true)
: getImageUrl(featuredStory.getLong("item_art_id"), true);
public List<Image> getThumbnails() {
return featuredStory.has("art_id")
? getImagesFromImageId(featuredStory.getLong("art_id"), true)
: getImagesFromImageId(featuredStory.getLong("item_art_id"), true);
}
}

View File

@ -3,15 +3,20 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.localization.DateWrapper;
import org.schabi.newpipe.extractor.stream.StreamInfoItemExtractor;
import org.schabi.newpipe.extractor.stream.StreamType;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.List;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_URL;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImageUrl;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.parseDate;
public class BandcampRadioInfoItemExtractor implements StreamInfoItemExtractor {
@ -39,7 +44,7 @@ public class BandcampRadioInfoItemExtractor implements StreamInfoItemExtractor {
@Nullable
@Override
public DateWrapper getUploadDate() throws ParsingException {
return BandcampExtractorHelper.parseDate(getTextualUploadDate());
return parseDate(getTextualUploadDate());
}
@Override
@ -52,9 +57,10 @@ public class BandcampRadioInfoItemExtractor implements StreamInfoItemExtractor {
return BASE_URL + "/?show=" + show.getInt("id");
}
@Nonnull
@Override
public String getThumbnailUrl() {
return getImageUrl(show.getLong("image_id"), false);
public List<Image> getThumbnails() {
return getImagesFromImageId(show.getLong("image_id"), false);
}
@Override
@ -78,12 +84,6 @@ public class BandcampRadioInfoItemExtractor implements StreamInfoItemExtractor {
return "";
}
@Nullable
@Override
public String getUploaderAvatarUrl() {
return null;
}
@Override
public boolean isUploaderVerified() throws ParsingException {
return false;

View File

@ -1,10 +1,19 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_API_URL;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_URL;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImageUrl;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParser;
import com.grack.nanojson.JsonParserException;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Element;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.Image.ResolutionLevel;
import org.schabi.newpipe.extractor.MediaFormat;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.StreamingService;
@ -19,19 +28,18 @@ import org.schabi.newpipe.extractor.stream.AudioStream;
import org.schabi.newpipe.extractor.stream.Description;
import org.schabi.newpipe.extractor.stream.StreamSegment;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_API_URL;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_URL;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImageUrl;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
public class BandcampRadioStreamExtractor extends BandcampStreamExtractor {
private static final String OPUS_LO = "opus-lo";
private static final String MP3_128 = "mp3-128";
private JsonObject showInfo;
public BandcampRadioStreamExtractor(final StreamingService service,
@ -77,9 +85,11 @@ public class BandcampRadioStreamExtractor extends BandcampStreamExtractor {
@Nonnull
@Override
public String getUploaderName() {
return Jsoup.parse(showInfo.getString("image_caption"))
.getElementsByTag("a").first().text();
public String getUploaderName() throws ParsingException {
return Jsoup.parse(showInfo.getString("image_caption")).getElementsByTag("a").stream()
.map(Element::text)
.findFirst()
.orElseThrow(() -> new ParsingException("Could not get uploader name"));
}
@Nullable
@ -90,14 +100,16 @@ public class BandcampRadioStreamExtractor extends BandcampStreamExtractor {
@Nonnull
@Override
public String getThumbnailUrl() throws ParsingException {
return getImageUrl(showInfo.getLong("show_image_id"), false);
public List<Image> getThumbnails() throws ParsingException {
return getImagesFromImageId(showInfo.getLong("show_image_id"), false);
}
@Nonnull
@Override
public String getUploaderAvatarUrl() {
return BASE_URL + "/img/buttons/bandcamp-button-circle-whitecolor-512.png";
public List<Image> getUploaderAvatars() {
return Collections.singletonList(
new Image(BASE_URL + "/img/buttons/bandcamp-button-circle-whitecolor-512.png",
512, 512, ResolutionLevel.MEDIUM));
}
@Nonnull
@ -113,23 +125,27 @@ public class BandcampRadioStreamExtractor extends BandcampStreamExtractor {
@Override
public List<AudioStream> getAudioStreams() {
final ArrayList<AudioStream> list = new ArrayList<>();
final List<AudioStream> audioStreams = new ArrayList<>();
final JsonObject streams = showInfo.getObject("audio_stream");
if (streams.has("opus-lo")) {
list.add(new AudioStream(
streams.getString("opus-lo"),
MediaFormat.OPUS, 100
));
}
if (streams.has("mp3-128")) {
list.add(new AudioStream(
streams.getString("mp3-128"),
MediaFormat.MP3, 128
));
if (streams.has(MP3_128)) {
audioStreams.add(new AudioStream.Builder()
.setId(MP3_128)
.setContent(streams.getString(MP3_128), true)
.setMediaFormat(MediaFormat.MP3)
.setAverageBitrate(128)
.build());
}
return list;
if (streams.has(OPUS_LO)) {
audioStreams.add(new AudioStream.Builder()
.setId(OPUS_LO)
.setContent(streams.getString(OPUS_LO), true)
.setMediaFormat(MediaFormat.OPUS)
.setAverageBitrate(100).build());
}
return audioStreams;
}
@Nonnull

View File

@ -3,10 +3,14 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import org.jsoup.nodes.Element;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.playlist.PlaylistInfoItemExtractor;
import javax.annotation.Nonnull;
import java.util.List;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageUrl;
/**
* Extracts recommended albums from tracks' website
@ -25,12 +29,13 @@ public class BandcampRelatedPlaylistInfoItemExtractor implements PlaylistInfoIte
@Override
public String getUrl() throws ParsingException {
return relatedAlbum.getElementsByClass("title-and-artist").attr("abs:href");
return relatedAlbum.getElementsByClass("album-link").attr("abs:href");
}
@Nonnull
@Override
public String getThumbnailUrl() throws ParsingException {
return relatedAlbum.getElementsByClass("album-art").attr("src");
public List<Image> getThumbnails() throws ParsingException {
return getImagesFromImageUrl(relatedAlbum.getElementsByClass("album-art").attr("src"));
}
@Override
@ -38,6 +43,16 @@ public class BandcampRelatedPlaylistInfoItemExtractor implements PlaylistInfoIte
return relatedAlbum.getElementsByClass("by-artist").text().replace("by ", "");
}
@Override
public String getUploaderUrl() throws ParsingException {
return null;
}
@Override
public boolean isUploaderVerified() throws ParsingException {
return false;
}
@Override
public long getStreamCount() throws ParsingException {
return -1;

View File

@ -2,28 +2,28 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import edu.umd.cs.findbugs.annotations.NonNull;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.MetaInfo;
import org.schabi.newpipe.extractor.MultiInfoItemsCollector;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandler;
import org.schabi.newpipe.extractor.MultiInfoItemsCollector;
import org.schabi.newpipe.extractor.search.SearchExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.streaminfoitem.BandcampSearchStreamInfoItemExtractor;
import javax.annotation.Nonnull;
import java.io.IOException;
import java.util.Collections;
import java.util.List;
import javax.annotation.Nonnull;
public class BandcampSearchExtractor extends SearchExtractor {
public BandcampSearchExtractor(final StreamingService service,
@ -31,7 +31,7 @@ public class BandcampSearchExtractor extends SearchExtractor {
super(service, linkHandler);
}
@NonNull
@Nonnull
@Override
public String getSearchSuggestion() {
return "";
@ -50,40 +50,30 @@ public class BandcampSearchExtractor extends SearchExtractor {
public InfoItemsPage<InfoItem> getPage(final Page page)
throws IOException, ExtractionException {
final String html = getDownloader().get(page.getUrl()).responseBody();
final MultiInfoItemsCollector collector = new MultiInfoItemsCollector(getServiceId());
final Document d = Jsoup.parse(getDownloader().get(page.getUrl()).responseBody());
final Document d = Jsoup.parse(html);
final Elements searchResultsElements = d.getElementsByClass("searchresult");
for (final Element searchResult : searchResultsElements) {
final String type = searchResult.getElementsByClass("result-info").first()
.getElementsByClass("itemtype").first().text();
for (final Element searchResult : d.getElementsByClass("searchresult")) {
final String type = searchResult.getElementsByClass("result-info").stream()
.flatMap(element -> element.getElementsByClass("itemtype").stream())
.map(Element::text)
.findFirst()
.orElse("");
switch (type) {
default:
continue;
case "FAN":
// don't display fan results
break;
case "ARTIST":
collector.commit(new BandcampChannelInfoItemExtractor(searchResult));
break;
case "ALBUM":
collector.commit(new BandcampPlaylistInfoItemExtractor(searchResult));
break;
case "TRACK":
collector.commit(new BandcampSearchStreamInfoItemExtractor(searchResult, null));
break;
default:
// don't display fan results ("FAN") or other things
break;
}
}
// Count pages
@ -92,7 +82,10 @@ public class BandcampSearchExtractor extends SearchExtractor {
return new InfoItemsPage<>(collector, null);
}
final Elements pages = pageLists.first().getElementsByTag("li");
final Elements pages = pageLists.stream()
.map(element -> element.getElementsByTag("li"))
.findFirst()
.orElseGet(Elements::new);
// Find current page
int currentPage = -1;

View File

@ -2,7 +2,11 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImageUrl;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageUrl;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.parseDate;
import static org.schabi.newpipe.extractor.utils.Utils.HTTPS;
import static org.schabi.newpipe.extractor.utils.Utils.replaceHttpWithHttps;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParserException;
@ -10,11 +14,12 @@ import com.grack.nanojson.JsonParserException;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.MediaFormat;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.PaidContentException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.LinkHandler;
import org.schabi.newpipe.extractor.localization.DateWrapper;
@ -28,15 +33,14 @@ import org.schabi.newpipe.extractor.utils.JsonUtils;
import org.schabi.newpipe.extractor.utils.Utils;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.stream.Collectors;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
public class BandcampStreamExtractor extends StreamExtractor {
private JsonObject albumJson;
private JsonObject current;
private Document document;
@ -58,6 +62,10 @@ public class BandcampStreamExtractor extends StreamExtractor {
// In this case, we are actually viewing an album page!
throw new ExtractionException("Page is actually an album, not a track");
}
if (albumJson.getArray("trackinfo").getObject(0).isNull("file")) {
throw new PaidContentException("This track is not available without being purchased");
}
}
/**
@ -88,13 +96,13 @@ public class BandcampStreamExtractor extends StreamExtractor {
public String getUploaderUrl() throws ParsingException {
final String[] parts = getUrl().split("/");
// https: (/) (/) * .bandcamp.com (/) and leave out the rest
return "https://" + parts[2] + "/";
return HTTPS + parts[2] + "/";
}
@Nonnull
@Override
public String getUrl() throws ParsingException {
return albumJson.getString("url").replace("http://", "https://");
return replaceHttpWithHttps(albumJson.getString("url"));
}
@Nonnull
@ -112,53 +120,54 @@ public class BandcampStreamExtractor extends StreamExtractor {
@Nullable
@Override
public DateWrapper getUploadDate() throws ParsingException {
return BandcampExtractorHelper.parseDate(getTextualUploadDate());
return parseDate(getTextualUploadDate());
}
@Nonnull
@Override
public String getThumbnailUrl() throws ParsingException {
public List<Image> getThumbnails() throws ParsingException {
if (albumJson.isNull("art_id")) {
return "";
} else {
return getImageUrl(albumJson.getLong("art_id"), true);
return List.of();
}
return getImagesFromImageId(albumJson.getLong("art_id"), true);
}
@Nonnull
@Override
public String getUploaderAvatarUrl() {
try {
return document.getElementsByClass("band-photo").first().attr("src");
} catch (final NullPointerException e) {
return "";
}
public List<Image> getUploaderAvatars() {
return getImagesFromImageUrl(document.getElementsByClass("band-photo")
.stream()
.map(element -> element.attr("src"))
.findFirst()
.orElse(""));
}
@Nonnull
@Override
public Description getDescription() {
final String s = Utils.nonEmptyAndNullJoin(
"\n\n",
new String[]{
current.getString("about"),
current.getString("lyrics"),
current.getString("credits")
}
);
final String s = Utils.nonEmptyAndNullJoin("\n\n", current.getString("about"),
current.getString("lyrics"), current.getString("credits"));
return new Description(s, Description.PLAIN_TEXT);
}
@Override
public List<AudioStream> getAudioStreams() {
final List<AudioStream> audioStreams = new ArrayList<>();
return Collections.singletonList(new AudioStream.Builder()
.setId("mp3-128")
.setContent(albumJson.getArray("trackinfo")
.getObject(0)
.getObject("file")
.getString("mp3-128"), true)
.setMediaFormat(MediaFormat.MP3)
.setAverageBitrate(128)
.build());
}
audioStreams.add(new AudioStream(
albumJson.getArray("trackinfo").getObject(0)
.getObject("file").getString("mp3-128"),
MediaFormat.MP3, 128
));
return audioStreams;
@Override
public long getLength() throws ParsingException {
return (long) albumJson.getArray("trackinfo").getObject(0)
.getDouble("duration");
}
@Override
@ -179,11 +188,11 @@ public class BandcampStreamExtractor extends StreamExtractor {
@Override
public PlaylistInfoItemsCollector getRelatedItems() {
final PlaylistInfoItemsCollector collector = new PlaylistInfoItemsCollector(getServiceId());
final Elements recommendedAlbums = document.getElementsByClass("recommended-album");
document.getElementsByClass("recommended-album")
.stream()
.map(BandcampRelatedPlaylistInfoItemExtractor::new)
.forEach(collector::commit);
for (final Element album : recommendedAlbums) {
collector.commit(new BandcampRelatedPlaylistInfoItemExtractor(album));
}
return collector;
}
@ -201,9 +210,11 @@ public class BandcampStreamExtractor extends StreamExtractor {
@Nonnull
@Override
public String getLicence() {
/* Tests resulted in this mapping of ints to licence:
/*
Tests resulted in this mapping of ints to licence:
https://cloud.disroot.org/s/ZTWBxbQ9fKRmRWJ/preview (screenshot from a Bandcamp artist's
account) */
account)
*/
switch (current.getInt("license_type")) {
case 1:
@ -228,14 +239,9 @@ public class BandcampStreamExtractor extends StreamExtractor {
@Nonnull
@Override
public List<String> getTags() {
final Elements tagElements = document.getElementsByAttributeValue("itemprop", "keywords");
final List<String> tags = new ArrayList<>();
for (final Element e : tagElements) {
tags.add(e.text());
}
return tags;
return document.getElementsByAttributeValue("itemprop", "keywords")
.stream()
.map(Element::text)
.collect(Collectors.toList());
}
}

View File

@ -4,7 +4,6 @@ package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_API_URL;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParser;
import com.grack.nanojson.JsonParserException;
@ -14,12 +13,12 @@ import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.suggestion.SuggestionExtractor;
import org.schabi.newpipe.extractor.utils.Utils;
import java.io.IOException;
import java.net.URLEncoder;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.stream.Collectors;
public class BandcampSuggestionExtractor extends SuggestionExtractor {
@ -34,25 +33,16 @@ public class BandcampSuggestionExtractor extends SuggestionExtractor {
try {
final JsonObject fuzzyResults = JsonParser.object().from(downloader
.get(AUTOCOMPLETE_URL + URLEncoder.encode(query, "UTF-8")).responseBody());
.get(AUTOCOMPLETE_URL + Utils.encodeUrlUtf8(query)).responseBody());
final JsonArray jsonArray = fuzzyResults.getObject("auto")
.getArray("results");
final List<String> suggestions = new ArrayList<>();
for (final Object fuzzyResult : jsonArray) {
final String res = ((JsonObject) fuzzyResult).getString("name");
if (!suggestions.contains(res)) {
suggestions.add(res);
}
}
return suggestions;
return fuzzyResults.getObject("auto").getArray("results").stream()
.filter(JsonObject.class::isInstance)
.map(JsonObject.class::cast)
.map(jsonObject -> jsonObject.getString("name"))
.distinct()
.collect(Collectors.toList());
} catch (final JsonParserException e) {
return Collections.emptyList();
}
}
}

View File

@ -1,10 +1,14 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors.streaminfoitem;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper;
import javax.annotation.Nullable;
import javax.annotation.Nonnull;
import java.util.List;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
public class BandcampDiscographStreamInfoItemExtractor extends BandcampStreamInfoItemExtractor {
@ -20,12 +24,6 @@ public class BandcampDiscographStreamInfoItemExtractor extends BandcampStreamInf
return discograph.getString("band_name");
}
@Nullable
@Override
public String getUploaderAvatarUrl() {
return null;
}
@Override
public String getName() {
return discograph.getString("title");
@ -40,11 +38,10 @@ public class BandcampDiscographStreamInfoItemExtractor extends BandcampStreamInf
);
}
@Nonnull
@Override
public String getThumbnailUrl() throws ParsingException {
return BandcampExtractorHelper.getImageUrl(
discograph.getLong("art_id"), true
);
public List<Image> getThumbnails() throws ParsingException {
return getImagesFromImageId(discograph.getLong("art_id"), true);
}
@Override

View File

@ -3,19 +3,21 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors.streaminfoitem;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.stream.StreamExtractor;
import javax.annotation.Nullable;
import javax.annotation.Nonnull;
import java.io.IOException;
import java.util.Collections;
import java.util.List;
public class BandcampPlaylistStreamInfoItemExtractor extends BandcampStreamInfoItemExtractor {
private final JsonObject track;
private String substituteCoverUrl;
private List<Image> substituteCovers;
private final StreamingService service;
public BandcampPlaylistStreamInfoItemExtractor(final JsonObject track,
@ -24,13 +26,14 @@ public class BandcampPlaylistStreamInfoItemExtractor extends BandcampStreamInfoI
super(uploaderUrl);
this.track = track;
this.service = service;
substituteCovers = Collections.emptyList();
}
public BandcampPlaylistStreamInfoItemExtractor(final JsonObject track,
final String uploaderUrl,
final String substituteCoverUrl) {
final List<Image> substituteCovers) {
this(track, uploaderUrl, (StreamingService) null);
this.substituteCoverUrl = substituteCoverUrl;
this.substituteCovers = substituteCovers;
}
@Override
@ -56,28 +59,23 @@ public class BandcampPlaylistStreamInfoItemExtractor extends BandcampStreamInfoI
return "";
}
@Nullable
@Override
public String getUploaderAvatarUrl() {
return null;
}
/**
* Each track can have its own cover art. Therefore, unless a substitute is provided,
* the thumbnail is extracted using a stream extractor.
*/
@Nonnull
@Override
public String getThumbnailUrl() throws ParsingException {
if (substituteCoverUrl != null) {
return substituteCoverUrl;
} else {
public List<Image> getThumbnails() throws ParsingException {
if (substituteCovers.isEmpty()) {
try {
final StreamExtractor extractor = service.getStreamExtractor(getUrl());
extractor.fetchPage();
return extractor.getThumbnailUrl();
return extractor.getThumbnails();
} catch (final ExtractionException | IOException e) {
throw new ParsingException("could not download cover art location", e);
throw new ParsingException("Could not download cover art location", e);
}
}
return substituteCovers;
}
}

View File

@ -1,9 +1,13 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors.streaminfoitem;
import org.jsoup.nodes.Element;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import javax.annotation.Nullable;
import javax.annotation.Nonnull;
import java.util.List;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromSearchResult;
public class BandcampSearchStreamInfoItemExtractor extends BandcampStreamInfoItemExtractor {
@ -28,12 +32,6 @@ public class BandcampSearchStreamInfoItemExtractor extends BandcampStreamInfoIte
}
}
@Nullable
@Override
public String getUploaderAvatarUrl() {
return null;
}
@Override
public String getName() throws ParsingException {
return resultInfo.getElementsByClass("heading").text();
@ -44,15 +42,10 @@ public class BandcampSearchStreamInfoItemExtractor extends BandcampStreamInfoIte
return resultInfo.getElementsByClass("itemurl").text();
}
@Nonnull
@Override
public String getThumbnailUrl() throws ParsingException {
final Element img = searchResult.getElementsByClass("art").first()
.getElementsByTag("img").first();
if (img != null) {
return img.attr("src");
} else {
return null;
}
public List<Image> getThumbnails() throws ParsingException {
return getImagesFromSearchResult(searchResult);
}
@Override

View File

@ -10,6 +10,7 @@ import org.schabi.newpipe.extractor.exceptions.ReCaptchaException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper;
import org.schabi.newpipe.extractor.utils.JsonUtils;
import org.schabi.newpipe.extractor.utils.Utils;
import java.io.IOException;
import java.util.List;
@ -17,11 +18,20 @@ import java.util.List;
/**
* Artist do have IDs that are useful
*/
public class BandcampChannelLinkHandlerFactory extends ListLinkHandlerFactory {
public final class BandcampChannelLinkHandlerFactory extends ListLinkHandlerFactory {
private static final BandcampChannelLinkHandlerFactory INSTANCE
= new BandcampChannelLinkHandlerFactory();
private BandcampChannelLinkHandlerFactory() {
}
public static BandcampChannelLinkHandlerFactory getInstance() {
return INSTANCE;
}
@Override
public String getId(final String url) throws ParsingException {
public String getId(final String url) throws ParsingException, UnsupportedOperationException {
try {
final String response = NewPipe.getDownloader().get(url).responseBody();
@ -41,16 +51,13 @@ public class BandcampChannelLinkHandlerFactory extends ListLinkHandlerFactory {
*/
@Override
public String getUrl(final String id, final List<String> contentFilter, final String sortFilter)
throws ParsingException {
try {
return BandcampExtractorHelper.getArtistDetails(id)
.getString("bandcamp_url")
.replace("http://", "https://");
} catch (final NullPointerException e) {
throws ParsingException, UnsupportedOperationException {
final JsonObject artistDetails = BandcampExtractorHelper.getArtistDetails(id);
if (artistDetails.getBoolean("error")) {
throw new ParsingException(
"JSON does not contain URL (invalid id?) or is otherwise invalid", e);
"JSON does not contain a channel URL (invalid id?) or is otherwise invalid");
}
return Utils.replaceHttpWithHttps(artistDetails.getString("bandcamp_url"));
}
/**
@ -61,22 +68,21 @@ public class BandcampChannelLinkHandlerFactory extends ListLinkHandlerFactory {
final String lowercaseUrl = url.toLowerCase();
// https: | | artist.bandcamp.com | releases
// 0 1 2 3
// https: | | artist.bandcamp.com | releases - music - album - track ( | name)
// 0 1 2 3 (4)
final String[] splitUrl = lowercaseUrl.split("/");
// URL is too short
if (splitUrl.length < 3) {
if (splitUrl.length != 3 && splitUrl.length != 4) {
return false;
}
// Must have "releases" or "music" as segment after url or none at all
if (splitUrl.length > 3 && !(
splitUrl[3].equals("releases") || splitUrl[3].equals("music")
)) {
// Must have "releases", "music", "album" or "track" as segment after URL or none at all
if (splitUrl.length == 4 && !(splitUrl[3].equals("releases")
|| splitUrl[3].equals("music")
|| splitUrl[3].equals("album")
|| splitUrl[3].equals("track"))) {
return false;
} else {
if (splitUrl[2].equals("daily.bandcamp.com")) {
// Refuse links to daily.bandcamp.com as that is not an artist

View File

@ -0,0 +1,72 @@
package org.schabi.newpipe.extractor.services.bandcamp.linkHandler;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabs;
import org.schabi.newpipe.extractor.exceptions.UnsupportedTabException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import javax.annotation.Nonnull;
import java.util.List;
public final class BandcampChannelTabLinkHandlerFactory extends ListLinkHandlerFactory {
private static final BandcampChannelTabLinkHandlerFactory INSTANCE
= new BandcampChannelTabLinkHandlerFactory();
private BandcampChannelTabLinkHandlerFactory() {
}
public static BandcampChannelTabLinkHandlerFactory getInstance() {
return INSTANCE;
}
/**
* Get a tab's URL suffix.
*
* <p>
* These URLs don't actually exist on the Bandcamp website, as both albums and tracks are
* listed on the main page, but redirect to the main page, which is perfect for us as we need a
* unique URL for each tab.
* </p>
*
* @param tab the tab value, which must not be null
* @return a URL suffix
* @throws UnsupportedTabException if the tab is not supported
*/
@Nonnull
public static String getUrlSuffix(@Nonnull final String tab) throws UnsupportedTabException {
switch (tab) {
case ChannelTabs.TRACKS:
return "/track";
case ChannelTabs.ALBUMS:
return "/album";
}
throw new UnsupportedTabException(tab);
}
@Override
public String getId(final String url) throws ParsingException, UnsupportedOperationException {
return BandcampChannelLinkHandlerFactory.getInstance().getId(url);
}
@Override
public String getUrl(final String id, final List<String> contentFilter, final String sortFilter)
throws ParsingException, UnsupportedOperationException {
return BandcampChannelLinkHandlerFactory.getInstance().getUrl(id)
+ getUrlSuffix(contentFilter.get(0));
}
@Override
public boolean onAcceptUrl(final String url) throws ParsingException {
return BandcampChannelLinkHandlerFactory.getInstance().onAcceptUrl(url);
}
@Override
public String[] getAvailableContentFilter() {
return new String[]{
ChannelTabs.TRACKS,
ChannelTabs.ALBUMS,
};
}
}

View File

@ -10,15 +10,29 @@ import java.util.List;
* Like in {@link BandcampStreamLinkHandlerFactory}, tracks have no meaningful IDs except for
* their URLs
*/
public class BandcampCommentsLinkHandlerFactory extends ListLinkHandlerFactory {
public final class BandcampCommentsLinkHandlerFactory extends ListLinkHandlerFactory {
private static final BandcampCommentsLinkHandlerFactory INSTANCE
= new BandcampCommentsLinkHandlerFactory();
private BandcampCommentsLinkHandlerFactory() {
}
public static BandcampCommentsLinkHandlerFactory getInstance() {
return INSTANCE;
}
@Override
public String getId(final String url) throws ParsingException {
public String getId(final String url) throws ParsingException, UnsupportedOperationException {
return url;
}
@Override
public boolean onAcceptUrl(final String url) throws ParsingException {
if (BandcampExtractorHelper.isRadioUrl(url)) {
return true;
}
// Don't accept URLs that don't point to a track
if (!url.toLowerCase().matches("https?://.+\\..+/(track|album)/.+")) {
return false;
@ -31,7 +45,8 @@ public class BandcampCommentsLinkHandlerFactory extends ListLinkHandlerFactory {
@Override
public String getUrl(final String id,
final List<String> contentFilter,
final String sortFilter) throws ParsingException {
final String sortFilter)
throws ParsingException, UnsupportedOperationException {
return id;
}
}

View File

@ -2,6 +2,7 @@
package org.schabi.newpipe.extractor.services.bandcamp.linkHandler;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper;
import org.schabi.newpipe.extractor.utils.Utils;
@ -13,12 +14,23 @@ import static org.schabi.newpipe.extractor.services.bandcamp.extractors.Bandcamp
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampRadioExtractor.KIOSK_RADIO;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampRadioExtractor.RADIO_API_URL;
public class BandcampFeaturedLinkHandlerFactory extends ListLinkHandlerFactory {
public final class BandcampFeaturedLinkHandlerFactory extends ListLinkHandlerFactory {
private static final BandcampFeaturedLinkHandlerFactory INSTANCE =
new BandcampFeaturedLinkHandlerFactory();
private BandcampFeaturedLinkHandlerFactory() {
}
public static BandcampFeaturedLinkHandlerFactory getInstance() {
return INSTANCE;
}
@Override
public String getUrl(final String id,
final List<String> contentFilter,
final String sortFilter) {
final String sortFilter)
throws ParsingException, UnsupportedOperationException {
if (id.equals(KIOSK_FEATURED)) {
return FEATURED_API_URL; // doesn't have a website
} else if (id.equals(KIOSK_RADIO)) {
@ -29,7 +41,7 @@ public class BandcampFeaturedLinkHandlerFactory extends ListLinkHandlerFactory {
}
@Override
public String getId(final String url) {
public String getId(final String url) throws ParsingException, UnsupportedOperationException {
final String fixedUrl = Utils.replaceHttpWithHttps(url);
if (BandcampExtractorHelper.isRadioUrl(fixedUrl) || fixedUrl.equals(RADIO_API_URL)) {
return KIOSK_RADIO;

View File

@ -11,16 +11,28 @@ import java.util.List;
/**
* Just as with streams, the album ids are essentially useless for us.
*/
public class BandcampPlaylistLinkHandlerFactory extends ListLinkHandlerFactory {
public final class BandcampPlaylistLinkHandlerFactory extends ListLinkHandlerFactory {
private static final BandcampPlaylistLinkHandlerFactory INSTANCE
= new BandcampPlaylistLinkHandlerFactory();
private BandcampPlaylistLinkHandlerFactory() {
}
public static BandcampPlaylistLinkHandlerFactory getInstance() {
return INSTANCE;
}
@Override
public String getId(final String url) throws ParsingException {
public String getId(final String url) throws ParsingException, UnsupportedOperationException {
return getUrl(url);
}
@Override
public String getUrl(final String url,
final List<String> contentFilter,
final String sortFilter) throws ParsingException {
final String sortFilter)
throws ParsingException, UnsupportedOperationException {
return url;
}

View File

@ -2,24 +2,34 @@
package org.schabi.newpipe.extractor.services.bandcamp.linkHandler;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandlerFactory;
import java.io.UnsupportedEncodingException;
import java.net.URLEncoder;
import java.util.List;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_URL;
public class BandcampSearchQueryHandlerFactory extends SearchQueryHandlerFactory {
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandlerFactory;
import org.schabi.newpipe.extractor.utils.Utils;
import java.io.UnsupportedEncodingException;
import java.util.List;
public final class BandcampSearchQueryHandlerFactory extends SearchQueryHandlerFactory {
private static final BandcampSearchQueryHandlerFactory INSTANCE
= new BandcampSearchQueryHandlerFactory();
private BandcampSearchQueryHandlerFactory() {
}
public static BandcampSearchQueryHandlerFactory getInstance() {
return INSTANCE;
}
@Override
public String getUrl(final String query,
final List<String> contentFilter,
final String sortFilter) throws ParsingException {
final String sortFilter)
throws ParsingException, UnsupportedOperationException {
try {
return BASE_URL + "/search?q=" + URLEncoder.encode(query, "UTF-8") + "&page=1";
return BASE_URL + "/search?q=" + Utils.encodeUrlUtf8(query) + "&page=1";
} catch (final UnsupportedEncodingException e) {
throw new ParsingException("query \"" + query + "\" could not be encoded", e);
}

View File

@ -14,14 +14,24 @@ import static org.schabi.newpipe.extractor.services.bandcamp.extractors.Bandcamp
*
* <p>Radio (bandcamp weekly) shows do have ids.</p>
*/
public class BandcampStreamLinkHandlerFactory extends LinkHandlerFactory {
public final class BandcampStreamLinkHandlerFactory extends LinkHandlerFactory {
private static final BandcampStreamLinkHandlerFactory INSTANCE
= new BandcampStreamLinkHandlerFactory();
private BandcampStreamLinkHandlerFactory() {
}
public static BandcampStreamLinkHandlerFactory getInstance() {
return INSTANCE;
}
/**
* @see BandcampStreamLinkHandlerFactory
*/
@Override
public String getId(final String url) throws ParsingException {
public String getId(final String url) throws ParsingException, UnsupportedOperationException {
if (BandcampExtractorHelper.isRadioUrl(url)) {
return url.split("bandcamp.com/\\?show=")[1];
} else {
@ -34,7 +44,8 @@ public class BandcampStreamLinkHandlerFactory extends LinkHandlerFactory {
* @see BandcampStreamLinkHandlerFactory
*/
@Override
public String getUrl(final String input) {
public String getUrl(final String input)
throws ParsingException, UnsupportedOperationException {
if (input.matches("\\d+")) {
return BASE_URL + "/?show=" + input;
} else {

View File

@ -6,6 +6,7 @@ import static java.util.Arrays.asList;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.ChannelExtractor;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import org.schabi.newpipe.extractor.comments.CommentsExtractor;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.kiosk.KioskList;
@ -13,10 +14,12 @@ import org.schabi.newpipe.extractor.linkhandler.LinkHandler;
import org.schabi.newpipe.extractor.linkhandler.LinkHandlerFactory;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import org.schabi.newpipe.extractor.linkhandler.ReadyChannelTabListLinkHandler;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandler;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandlerFactory;
import org.schabi.newpipe.extractor.playlist.PlaylistExtractor;
import org.schabi.newpipe.extractor.search.SearchExtractor;
import org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCChannelTabExtractor;
import org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCConferenceExtractor;
import org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCConferenceKiosk;
import org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCLiveStreamExtractor;
@ -27,8 +30,6 @@ import org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCSearch
import org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCStreamExtractor;
import org.schabi.newpipe.extractor.services.media_ccc.linkHandler.MediaCCCConferenceLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.media_ccc.linkHandler.MediaCCCConferencesListLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.media_ccc.linkHandler.MediaCCCLiveListLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.media_ccc.linkHandler.MediaCCCRecentListLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.media_ccc.linkHandler.MediaCCCSearchQueryHandlerFactory;
import org.schabi.newpipe.extractor.services.media_ccc.linkHandler.MediaCCCStreamLinkHandlerFactory;
import org.schabi.newpipe.extractor.stream.StreamExtractor;
@ -47,12 +48,19 @@ public class MediaCCCService extends StreamingService {
@Override
public LinkHandlerFactory getStreamLHFactory() {
return new MediaCCCStreamLinkHandlerFactory();
return MediaCCCStreamLinkHandlerFactory.getInstance();
}
@Override
public ListLinkHandlerFactory getChannelLHFactory() {
return new MediaCCCConferenceLinkHandlerFactory();
return MediaCCCConferenceLinkHandlerFactory.getInstance();
}
@Override
public ListLinkHandlerFactory getChannelTabLHFactory() {
// there is just one channel tab in MediaCCC, the one containing conferences, so there is
// no need for a specific channel tab link handler, but we can just use the channel one
return MediaCCCConferenceLinkHandlerFactory.getInstance();
}
@Override
@ -62,7 +70,7 @@ public class MediaCCCService extends StreamingService {
@Override
public SearchQueryHandlerFactory getSearchQHFactory() {
return new MediaCCCSearchQueryHandlerFactory();
return MediaCCCSearchQueryHandlerFactory.getInstance();
}
@Override
@ -78,6 +86,18 @@ public class MediaCCCService extends StreamingService {
return new MediaCCCConferenceExtractor(this, linkHandler);
}
@Override
public ChannelTabExtractor getChannelTabExtractor(final ListLinkHandler linkHandler) {
if (linkHandler instanceof ReadyChannelTabListLinkHandler) {
// conference data has already been fetched, let the ReadyChannelTabListLinkHandler
// create a MediaCCCChannelTabExtractor with that data
return ((ReadyChannelTabListLinkHandler) linkHandler).getChannelTabExtractor(this);
} else {
// conference data has not been fetched yet, so pass null instead
return new MediaCCCChannelTabExtractor(this, linkHandler, null);
}
}
@Override
public PlaylistExtractor getPlaylistExtractor(final ListLinkHandler linkHandler) {
return null;
@ -91,40 +111,41 @@ public class MediaCCCService extends StreamingService {
@Override
public KioskList getKioskList() throws ExtractionException {
final KioskList list = new KioskList(this);
final ListLinkHandlerFactory h = MediaCCCConferencesListLinkHandlerFactory.getInstance();
// add kiosks here e.g.:
try {
list.addKioskEntry(
(streamingService, url, kioskId) -> new MediaCCCConferenceKiosk(
MediaCCCService.this,
new MediaCCCConferencesListLinkHandlerFactory().fromUrl(url),
h.fromUrl(url),
kioskId
),
new MediaCCCConferencesListLinkHandlerFactory(),
"conferences"
h,
MediaCCCConferenceKiosk.KIOSK_ID
);
list.addKioskEntry(
(streamingService, url, kioskId) -> new MediaCCCRecentKiosk(
MediaCCCService.this,
new MediaCCCRecentListLinkHandlerFactory().fromUrl(url),
h.fromUrl(url),
kioskId
),
new MediaCCCRecentListLinkHandlerFactory(),
"recent"
h,
MediaCCCRecentKiosk.KIOSK_ID
);
list.addKioskEntry(
(streamingService, url, kioskId) -> new MediaCCCLiveStreamKiosk(
MediaCCCService.this,
new MediaCCCLiveListLinkHandlerFactory().fromUrl(url),
h.fromUrl(url),
kioskId
),
new MediaCCCLiveListLinkHandlerFactory(),
"live"
h,
MediaCCCLiveStreamKiosk.KIOSK_ID
);
list.setDefaultKiosk("recent");
list.setDefaultKiosk(MediaCCCRecentKiosk.KIOSK_ID);
} catch (final Exception e) {
throw new ExtractionException(e);
}

View File

@ -0,0 +1,69 @@
package org.schabi.newpipe.extractor.services.media_ccc.extractors;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.MultiInfoItemsCollector;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.services.media_ccc.extractors.infoItems.MediaCCCStreamInfoItemExtractor;
import java.io.IOException;
import java.util.Objects;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
/**
* MediaCCC does not really have channel tabs, but rather a list of videos for each conference,
* so this class just acts as a videos channel tab extractor.
*/
public class MediaCCCChannelTabExtractor extends ChannelTabExtractor {
@Nullable
private JsonObject conferenceData;
/**
* @param conferenceData will be not-null if conference data has already been fetched by
* {@link MediaCCCConferenceExtractor}. Otherwise, if this parameter is
* {@code null}, conference data will be fetched anew.
*/
public MediaCCCChannelTabExtractor(final StreamingService service,
final ListLinkHandler linkHandler,
@Nullable final JsonObject conferenceData) {
super(service, linkHandler);
this.conferenceData = conferenceData;
}
@Override
public void onFetchPage(@Nonnull final Downloader downloader)
throws ExtractionException, IOException {
if (conferenceData == null) {
// only fetch conference data if we don't have it already
conferenceData = MediaCCCConferenceExtractor.fetchConferenceData(downloader, getId());
}
}
@Nonnull
@Override
public ListExtractor.InfoItemsPage<InfoItem> getInitialPage() {
final MultiInfoItemsCollector collector =
new MultiInfoItemsCollector(getServiceId());
Objects.requireNonNull(conferenceData) // will surely be != null after onFetchPage
.getArray("events")
.stream()
.filter(JsonObject.class::isInstance)
.map(JsonObject.class::cast)
.forEach(event -> collector.commit(new MediaCCCStreamInfoItemExtractor(event)));
return new ListExtractor.InfoItemsPage<>(collector, null);
}
@Override
public ListExtractor.InfoItemsPage<InfoItem> getPage(final Page page) {
return ListExtractor.InfoItemsPage.emptyPage();
}
}

View File

@ -1,23 +1,27 @@
package org.schabi.newpipe.extractor.services.media_ccc.extractors;
import com.grack.nanojson.JsonArray;
import static org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCParsingHelper.getImageListFromLogoImageUrl;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParser;
import com.grack.nanojson.JsonParserException;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.ChannelExtractor;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabs;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.services.media_ccc.extractors.infoItems.MediaCCCStreamInfoItemExtractor;
import org.schabi.newpipe.extractor.linkhandler.ReadyChannelTabListLinkHandler;
import org.schabi.newpipe.extractor.services.media_ccc.linkHandler.MediaCCCConferenceLinkHandlerFactory;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import org.schabi.newpipe.extractor.stream.StreamInfoItemsCollector;
import java.io.IOException;
import java.util.Collections;
import java.util.List;
import javax.annotation.Nonnull;
import java.io.IOException;
public class MediaCCCConferenceExtractor extends ChannelExtractor {
private JsonObject conferenceData;
@ -27,14 +31,29 @@ public class MediaCCCConferenceExtractor extends ChannelExtractor {
super(service, linkHandler);
}
@Override
public String getAvatarUrl() {
return conferenceData.getString("logo_url");
static JsonObject fetchConferenceData(@Nonnull final Downloader downloader,
@Nonnull final String conferenceId)
throws IOException, ExtractionException {
final String conferenceUrl
= MediaCCCConferenceLinkHandlerFactory.CONFERENCE_API_ENDPOINT + conferenceId;
try {
return JsonParser.object().from(downloader.get(conferenceUrl).responseBody());
} catch (final JsonParserException jpe) {
throw new ExtractionException("Could not parse json returned by URL: " + conferenceUrl);
}
}
@Nonnull
@Override
public String getBannerUrl() {
return conferenceData.getString("logo_url");
public List<Image> getAvatars() {
return getImageListFromLogoImageUrl(conferenceData.getString("logo_url"));
}
@Nonnull
@Override
public List<Image> getBanners() {
return Collections.emptyList();
}
@Override
@ -62,9 +81,10 @@ public class MediaCCCConferenceExtractor extends ChannelExtractor {
return "";
}
@Nonnull
@Override
public String getParentChannelAvatarUrl() {
return "";
public List<Image> getParentChannelAvatars() {
return Collections.emptyList();
}
@Override
@ -74,30 +94,18 @@ public class MediaCCCConferenceExtractor extends ChannelExtractor {
@Nonnull
@Override
public InfoItemsPage<StreamInfoItem> getInitialPage() {
final StreamInfoItemsCollector collector = new StreamInfoItemsCollector(getServiceId());
final JsonArray events = conferenceData.getArray("events");
for (int i = 0; i < events.size(); i++) {
collector.commit(new MediaCCCStreamInfoItemExtractor(events.getObject(i)));
}
return new InfoItemsPage<>(collector, null);
}
@Override
public InfoItemsPage<StreamInfoItem> getPage(final Page page) {
return InfoItemsPage.emptyPage();
public List<ListLinkHandler> getTabs() throws ParsingException {
// avoid keeping a reference to MediaCCCConferenceExtractor inside the lambda
final JsonObject theConferenceData = conferenceData;
return List.of(new ReadyChannelTabListLinkHandler(getUrl(), getId(), ChannelTabs.VIDEOS,
(service, linkHandler) ->
new MediaCCCChannelTabExtractor(service, linkHandler, theConferenceData)));
}
@Override
public void onFetchPage(@Nonnull final Downloader downloader)
throws IOException, ExtractionException {
final String conferenceUrl
= MediaCCCConferenceLinkHandlerFactory.CONFERENCE_API_ENDPOINT + getId();
try {
conferenceData = JsonParser.object().from(downloader.get(conferenceUrl).responseBody());
} catch (final JsonParserException jpe) {
throw new ExtractionException("Could not parse json returnd by url: " + conferenceUrl);
}
conferenceData = fetchConferenceData(downloader, getId());
}
@Nonnull

View File

@ -21,6 +21,8 @@ import java.io.IOException;
import javax.annotation.Nonnull;
public class MediaCCCConferenceKiosk extends KioskExtractor<ChannelInfoItem> {
public static final String KIOSK_ID = "conferences";
private JsonObject doc;
public MediaCCCConferenceKiosk(final StreamingService streamingService,

View File

@ -1,8 +1,13 @@
package org.schabi.newpipe.extractor.services.media_ccc.extractors;
import static org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCParsingHelper.getThumbnailsFromLiveStreamItem;
import static org.schabi.newpipe.extractor.stream.AudioStream.UNKNOWN_BITRATE;
import static org.schabi.newpipe.extractor.stream.Stream.ID_UNKNOWN;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.MediaFormat;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.downloader.Downloader;
@ -10,18 +15,26 @@ import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.LinkHandler;
import org.schabi.newpipe.extractor.stream.AudioStream;
import org.schabi.newpipe.extractor.stream.DeliveryMethod;
import org.schabi.newpipe.extractor.stream.Description;
import org.schabi.newpipe.extractor.stream.Stream;
import org.schabi.newpipe.extractor.stream.StreamExtractor;
import org.schabi.newpipe.extractor.stream.StreamType;
import org.schabi.newpipe.extractor.stream.VideoStream;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.function.Function;
import java.util.stream.Collectors;
import javax.annotation.Nonnull;
public class MediaCCCLiveStreamExtractor extends StreamExtractor {
private static final String STREAMS = "streams";
private static final String URLS = "urls";
private static final String URL = "url";
private JsonObject conference = null;
private String group = "";
private JsonObject room = null;
@ -34,19 +47,22 @@ public class MediaCCCLiveStreamExtractor extends StreamExtractor {
@Override
public void onFetchPage(@Nonnull final Downloader downloader)
throws IOException, ExtractionException {
final JsonArray doc =
MediaCCCParsingHelper.getLiveStreams(downloader, getExtractorLocalization());
// find correct room
final JsonArray doc = MediaCCCParsingHelper.getLiveStreams(downloader,
getExtractorLocalization());
// Find the correct room
for (int c = 0; c < doc.size(); c++) {
conference = doc.getObject(c);
final JsonArray groups = conference.getArray("groups");
final JsonObject conferenceObject = doc.getObject(c);
final JsonArray groups = conferenceObject.getArray("groups");
for (int g = 0; g < groups.size(); g++) {
group = groups.getObject(g).getString("group");
final String groupObject = groups.getObject(g).getString("group");
final JsonArray rooms = groups.getObject(g).getArray("rooms");
for (int r = 0; r < rooms.size(); r++) {
room = rooms.getObject(r);
if (getId().equals(
conference.getString("slug") + "/" + room.getString("slug"))) {
final JsonObject roomObject = rooms.getObject(r);
if (getId().equals(conferenceObject.getString("slug") + "/"
+ roomObject.getString("slug"))) {
conference = conferenceObject;
group = groupObject;
room = roomObject;
return;
}
}
@ -63,8 +79,8 @@ public class MediaCCCLiveStreamExtractor extends StreamExtractor {
@Nonnull
@Override
public String getThumbnailUrl() throws ParsingException {
return room.getString("thumb");
public List<Image> getThumbnails() throws ParsingException {
return getThumbnailsFromLiveStreamItem(room);
}
@Nonnull
@ -91,69 +107,155 @@ public class MediaCCCLiveStreamExtractor extends StreamExtractor {
return conference.getString("conference");
}
/**
* Get the URL of the first DASH stream found.
*
* <p>
* There can be several DASH streams, so the URL of the first one found is returned by this
* method.
* </p>
*
* <p>
* You can find the other DASH video streams by using {@link #getVideoStreams()}
* </p>
*/
@Nonnull
@Override
public String getDashMpdUrl() throws ParsingException {
return getManifestOfDeliveryMethodWanted("dash");
}
/**
* Get the URL of the first HLS stream found.
*
* <p>
* There can be several HLS streams, so the URL of the first one found is returned by this
* method.
* </p>
*
* <p>
* You can find the other HLS video streams by using {@link #getVideoStreams()}
* </p>
*/
@Nonnull
@Override
public String getHlsUrl() {
// TODO: There are multiple HLS streams.
// Make getHlsUrl() and getDashMpdUrl() return lists of VideoStreams,
// so the user can choose a resolution.
for (int s = 0; s < room.getArray("streams").size(); s++) {
final JsonObject stream = room.getArray("streams").getObject(s);
if (stream.getString("type").equals("video")) {
if (stream.has("hls")) {
return stream.getObject("urls").getObject("hls").getString("url");
}
}
}
return "";
return getManifestOfDeliveryMethodWanted("hls");
}
@Nonnull
private String getManifestOfDeliveryMethodWanted(@Nonnull final String deliveryMethod) {
return room.getArray(STREAMS).stream()
.filter(JsonObject.class::isInstance)
.map(JsonObject.class::cast)
.map(streamObject -> streamObject.getObject(URLS))
.filter(urls -> urls.has(deliveryMethod))
.map(urls -> urls.getObject(deliveryMethod).getString(URL, ""))
.findFirst()
.orElse("");
}
@Override
public List<AudioStream> getAudioStreams() throws IOException, ExtractionException {
final List<AudioStream> audioStreams = new ArrayList<>();
for (int s = 0; s < room.getArray("streams").size(); s++) {
final JsonObject stream = room.getArray("streams").getObject(s);
if (stream.getString("type").equals("audio")) {
for (final String type : stream.getObject("urls").keySet()) {
final JsonObject url = stream.getObject("urls").getObject(type);
audioStreams.add(new AudioStream(url.getString("url"),
MediaFormat.getFromSuffix(type), -1));
}
}
}
return audioStreams;
return getStreams("audio",
dto -> {
final AudioStream.Builder builder = new AudioStream.Builder()
.setId(dto.urlValue.getString("tech", ID_UNKNOWN))
.setContent(dto.urlValue.getString(URL), true)
.setAverageBitrate(UNKNOWN_BITRATE);
if ("hls".equals(dto.urlKey)) {
// We don't know with the type string what media format will
// have HLS streams.
// However, the tech string may contain some information
// about the media format used.
return builder.setDeliveryMethod(DeliveryMethod.HLS)
.build();
}
return builder.setMediaFormat(MediaFormat.getFromSuffix(dto.urlKey))
.build();
});
}
@Override
public List<VideoStream> getVideoStreams() throws IOException, ExtractionException {
final List<VideoStream> videoStreams = new ArrayList<>();
for (int s = 0; s < room.getArray("streams").size(); s++) {
final JsonObject stream = room.getArray("streams").getObject(s);
if (stream.getString("type").equals("video")) {
final String resolution = stream.getArray("videoSize").getInt(0) + "x"
+ stream.getArray("videoSize").getInt(1);
for (final String type : stream.getObject("urls").keySet()) {
if (!type.equals("hls")) {
final JsonObject url = stream.getObject("urls").getObject(type);
videoStreams.add(new VideoStream(
url.getString("url"),
MediaFormat.getFromSuffix(type),
resolution));
return getStreams("video",
dto -> {
final JsonArray videoSize = dto.streamJsonObj.getArray("videoSize");
final VideoStream.Builder builder = new VideoStream.Builder()
.setId(dto.urlValue.getString("tech", ID_UNKNOWN))
.setContent(dto.urlValue.getString(URL), true)
.setIsVideoOnly(false)
.setResolution(videoSize.getInt(0) + "x" + videoSize.getInt(1));
if ("hls".equals(dto.urlKey)) {
// We don't know with the type string what media format will
// have HLS streams.
// However, the tech string may contain some information
// about the media format used.
return builder.setDeliveryMethod(DeliveryMethod.HLS)
.build();
}
}
}
return builder.setMediaFormat(MediaFormat.getFromSuffix(dto.urlKey))
.build();
});
}
/**
* This is just an internal class used in {@link #getStreams(String, Function)} to tie together
* the stream json object, its URL key and its URL value. An object of this class would be
* temporary and the three values it holds would be <b>convert</b>ed to a proper {@link Stream}
* object based on the wanted stream type.
*/
private static final class MediaCCCLiveStreamMapperDTO {
final JsonObject streamJsonObj;
final String urlKey;
final JsonObject urlValue;
MediaCCCLiveStreamMapperDTO(final JsonObject streamJsonObj,
final String urlKey,
final JsonObject urlValue) {
this.streamJsonObj = streamJsonObj;
this.urlKey = urlKey;
this.urlValue = urlValue;
}
return videoStreams;
}
private <T extends Stream> List<T> getStreams(
@Nonnull final String streamType,
@Nonnull final Function<MediaCCCLiveStreamMapperDTO, T> converter) {
return room.getArray(STREAMS).stream()
// Ensure that we use only process JsonObjects
.filter(JsonObject.class::isInstance)
.map(JsonObject.class::cast)
// Only process streams of requested type
.filter(streamJsonObj -> streamType.equals(streamJsonObj.getString("type")))
// Flatmap Urls and ensure that we use only process JsonObjects
.flatMap(streamJsonObj -> streamJsonObj.getObject(URLS).entrySet().stream()
.filter(e -> e.getValue() instanceof JsonObject)
.map(e -> new MediaCCCLiveStreamMapperDTO(
streamJsonObj,
e.getKey(),
(JsonObject) e.getValue())))
// The DASH manifest will be extracted with getDashMpdUrl
.filter(dto -> !"dash".equals(dto.urlKey))
// Convert
.map(converter)
.collect(Collectors.toList());
}
@Override
public List<VideoStream> getVideoOnlyStreams() {
return null;
return Collections.emptyList();
}
@Override
public StreamType getStreamType() throws ParsingException {
return StreamType.LIVE_STREAM; // TODO: video and audio only streams are both available
return StreamType.LIVE_STREAM;
}
@Nonnull

View File

@ -16,6 +16,8 @@ import javax.annotation.Nonnull;
import java.io.IOException;
public class MediaCCCLiveStreamKiosk extends KioskExtractor<StreamInfoItem> {
public static final String KIOSK_ID = "live";
private JsonArray doc;
public MediaCCCLiveStreamKiosk(final StreamingService streamingService,
@ -36,13 +38,16 @@ public class MediaCCCLiveStreamKiosk extends KioskExtractor<StreamInfoItem> {
final StreamInfoItemsCollector collector = new StreamInfoItemsCollector(getServiceId());
for (int c = 0; c < doc.size(); c++) {
final JsonObject conference = doc.getObject(c);
final JsonArray groups = conference.getArray("groups");
for (int g = 0; g < groups.size(); g++) {
final String group = groups.getObject(g).getString("group");
final JsonArray rooms = groups.getObject(g).getArray("rooms");
for (int r = 0; r < rooms.size(); r++) {
final JsonObject room = rooms.getObject(r);
collector.commit(new MediaCCCLiveStreamKioskExtractor(conference, group, room));
if (conference.getBoolean("isCurrentlyStreaming")) {
final JsonArray groups = conference.getArray("groups");
for (int g = 0; g < groups.size(); g++) {
final String group = groups.getObject(g).getString("group");
final JsonArray rooms = groups.getObject(g).getArray("rooms");
for (int r = 0; r < rooms.size(); r++) {
final JsonObject room = rooms.getObject(r);
collector.commit(new MediaCCCLiveStreamKioskExtractor(
conference, group, room));
}
}
}
@ -59,6 +64,6 @@ public class MediaCCCLiveStreamKiosk extends KioskExtractor<StreamInfoItem> {
@Nonnull
@Override
public String getName() throws ParsingException {
return "live";
return KIOSK_ID;
}
}

View File

@ -1,12 +1,17 @@
package org.schabi.newpipe.extractor.services.media_ccc.extractors;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.localization.DateWrapper;
import org.schabi.newpipe.extractor.stream.StreamInfoItemExtractor;
import org.schabi.newpipe.extractor.stream.StreamType;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.List;
import static org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCParsingHelper.getThumbnailsFromLiveStreamItem;
public class MediaCCCLiveStreamKioskExtractor implements StreamInfoItemExtractor {
@ -32,9 +37,10 @@ public class MediaCCCLiveStreamKioskExtractor implements StreamInfoItemExtractor
return roomInfo.getString("link");
}
@Nonnull
@Override
public String getThumbnailUrl() throws ParsingException {
return roomInfo.getString("thumb");
public List<Image> getThumbnails() throws ParsingException {
return getThumbnailsFromLiveStreamItem(roomInfo);
}
@Override
@ -75,12 +81,6 @@ public class MediaCCCLiveStreamKioskExtractor implements StreamInfoItemExtractor
return "https://media.ccc.de/c/" + conferenceInfo.getString("slug");
}
@Nullable
@Override
public String getUploaderAvatarUrl() {
return null;
}
@Override
public boolean isUploaderVerified() throws ParsingException {
return false;

View File

@ -1,21 +1,33 @@
package org.schabi.newpipe.extractor.services.media_ccc.extractors;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParser;
import com.grack.nanojson.JsonParserException;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.Image.ResolutionLevel;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.exceptions.ReCaptchaException;
import org.schabi.newpipe.extractor.localization.Localization;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.io.IOException;
import java.time.OffsetDateTime;
import java.time.format.DateTimeParseException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.regex.Pattern;
import static org.schabi.newpipe.extractor.Image.HEIGHT_UNKNOWN;
import static org.schabi.newpipe.extractor.Image.WIDTH_UNKNOWN;
import static org.schabi.newpipe.extractor.utils.Utils.isNullOrEmpty;
public final class MediaCCCParsingHelper {
// {conference_slug}/{room_slug}
// conference_slug/room_slug
private static final Pattern LIVE_STREAM_ID_PATTERN = Pattern.compile("\\w+/\\w+");
private static JsonArray liveStreams = null;
@ -69,4 +81,98 @@ public final class MediaCCCParsingHelper {
}
return liveStreams;
}
/**
* Get an {@link Image} list from a given image logo URL.
*
* <p>
* If the image URL is null or empty, an empty list is returned; otherwise, a singleton list is
* returned containing an {@link Image} with the image URL with its height, width and
* resolution unknown.
* </p>
*
* @param logoImageUrl a logo image URL, which can be null or empty
* @return an unmodifiable list of {@link Image}s, which is always empty or a singleton
*/
@Nonnull
public static List<Image> getImageListFromLogoImageUrl(@Nullable final String logoImageUrl) {
if (isNullOrEmpty(logoImageUrl)) {
return List.of();
}
return List.of(new Image(logoImageUrl, HEIGHT_UNKNOWN, WIDTH_UNKNOWN,
ResolutionLevel.UNKNOWN));
}
/**
* Get the {@link Image} list of thumbnails from a given stream item.
*
* <p>
* MediaCCC API provides two thumbnails for a stream item: a {@code thumb_url} one, which is
* medium quality and a {@code poster_url} one, which is high quality in most cases.
* </p>
*
* @param streamItem a stream JSON item of MediaCCC's API, which must not be null
* @return an unmodifiable list, which is never null but can be empty.
*/
@Nonnull
public static List<Image> getThumbnailsFromStreamItem(@Nonnull final JsonObject streamItem) {
return getThumbnailsFromObject(streamItem, "thumb_url", "poster_url");
}
/**
* Get the {@link Image} list of thumbnails from a given live stream item.
*
* <p>
* MediaCCC API provides two URL thumbnails for a livestream item: a {@code thumb} one,
* which should be medium quality and a {@code poster_url} one, which should be high quality.
* </p>
*
* @param liveStreamItem a stream JSON item of MediaCCC's API, which must not be null
* @return an unmodifiable list, which is never null but can be empty.
*/
@Nonnull
public static List<Image> getThumbnailsFromLiveStreamItem(
@Nonnull final JsonObject liveStreamItem) {
return getThumbnailsFromObject(liveStreamItem, "thumb", "poster");
}
/**
* Utility method to get an {@link Image} list of thumbnails from a stream or a livestream.
*
* <p>
* MediaCCC's API thumbnails come from two elements: a {@code thumb} element, which links to a
* medium thumbnail and a {@code poster} element, which links to a high thumbnail.
* </p>
* <p>
* Thumbnails are only added if their URLs are not null or empty.
* </p>
*
* @param streamOrLivestreamItem a (live)stream JSON item of MediaCCC's API, which must not be
* null
* @param thumbUrlKey the name of the {@code thumb} URL key
* @param posterUrlKey the name of the {@code poster} URL key
* @return an unmodifiable list, which is never null but can be empty.
*/
@Nonnull
private static List<Image> getThumbnailsFromObject(
@Nonnull final JsonObject streamOrLivestreamItem,
@Nonnull final String thumbUrlKey,
@Nonnull final String posterUrlKey) {
final List<Image> imageList = new ArrayList<>(2);
final String thumbUrl = streamOrLivestreamItem.getString(thumbUrlKey);
if (!isNullOrEmpty(thumbUrl)) {
imageList.add(new Image(thumbUrl, HEIGHT_UNKNOWN, WIDTH_UNKNOWN,
ResolutionLevel.MEDIUM));
}
final String posterUrl = streamOrLivestreamItem.getString(posterUrlKey);
if (!isNullOrEmpty(posterUrl)) {
imageList.add(new Image(posterUrl, HEIGHT_UNKNOWN, WIDTH_UNKNOWN,
ResolutionLevel.HIGH));
}
return Collections.unmodifiableList(imageList);
}
}

View File

@ -12,6 +12,7 @@ import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.kiosk.KioskExtractor;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.localization.DateWrapper;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import org.schabi.newpipe.extractor.stream.StreamInfoItemsCollector;
@ -22,6 +23,8 @@ import javax.annotation.Nonnull;
public class MediaCCCRecentKiosk extends KioskExtractor<StreamInfoItem> {
public static final String KIOSK_ID = "recent";
private JsonObject doc;
public MediaCCCRecentKiosk(final StreamingService streamingService,
@ -49,12 +52,12 @@ public class MediaCCCRecentKiosk extends KioskExtractor<StreamInfoItem> {
// Streams in the recent kiosk are not ordered by the release date.
// Sort them to have the latest stream at the beginning of the list.
Comparator<StreamInfoItem> comparator = Comparator.comparing(
streamInfoItem -> streamInfoItem.getUploadDate().offsetDateTime());
comparator = comparator.reversed();
final StreamInfoItemsCollector collector =
new StreamInfoItemsCollector(getServiceId(), comparator);
final Comparator<StreamInfoItem> comparator = Comparator
.comparing(StreamInfoItem::getUploadDate, Comparator
.nullsLast(Comparator.comparing(DateWrapper::offsetDateTime)))
.reversed();
final StreamInfoItemsCollector collector = new StreamInfoItemsCollector(getServiceId(),
comparator);
events.stream()
.filter(JsonObject.class::isInstance)
@ -76,6 +79,6 @@ public class MediaCCCRecentKiosk extends KioskExtractor<StreamInfoItem> {
@Nonnull
@Override
public String getName() throws ParsingException {
return "recent";
return KIOSK_ID;
}
}

View File

@ -2,6 +2,7 @@ package org.schabi.newpipe.extractor.services.media_ccc.extractors;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.localization.DateWrapper;
import org.schabi.newpipe.extractor.services.media_ccc.linkHandler.MediaCCCConferenceLinkHandlerFactory;
@ -10,9 +11,13 @@ import org.schabi.newpipe.extractor.stream.StreamType;
import java.time.ZonedDateTime;
import java.time.format.DateTimeFormatter;
import java.util.List;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import static org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCParsingHelper.getImageListFromLogoImageUrl;
public class MediaCCCRecentKioskExtractor implements StreamInfoItemExtractor {
private final JsonObject event;
@ -31,9 +36,10 @@ public class MediaCCCRecentKioskExtractor implements StreamInfoItemExtractor {
return event.getString("frontend_link");
}
@Nonnull
@Override
public String getThumbnailUrl() throws ParsingException {
return event.getString("thumb_url");
public List<Image> getThumbnails() throws ParsingException {
return getImageListFromLogoImageUrl(event.getString("poster_url"));
}
@Override
@ -65,17 +71,11 @@ public class MediaCCCRecentKioskExtractor implements StreamInfoItemExtractor {
@Override
public String getUploaderUrl() throws ParsingException {
return new MediaCCCConferenceLinkHandlerFactory()
return MediaCCCConferenceLinkHandlerFactory.getInstance()
.fromUrl(event.getString("conference_url")) // API URL
.getUrl(); // web URL
}
@Nullable
@Override
public String getUploaderAvatarUrl() {
return null;
}
@Override
public boolean isUploaderVerified() throws ParsingException {
return false;

View File

@ -9,8 +9,10 @@ import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParser;
import com.grack.nanojson.JsonParserException;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.MetaInfo;
import org.schabi.newpipe.extractor.MultiInfoItemsCollector;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.ChannelInfoItem;
@ -18,7 +20,6 @@ import org.schabi.newpipe.extractor.channel.ChannelInfoItemExtractor;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandler;
import org.schabi.newpipe.extractor.MultiInfoItemsCollector;
import org.schabi.newpipe.extractor.search.SearchExtractor;
import org.schabi.newpipe.extractor.services.media_ccc.extractors.infoItems.MediaCCCStreamInfoItemExtractor;
import org.schabi.newpipe.extractor.services.media_ccc.linkHandler.MediaCCCConferencesListLinkHandlerFactory;
@ -38,7 +39,8 @@ public class MediaCCCSearchExtractor extends SearchExtractor {
super(service, linkHandler);
try {
conferenceKiosk = new MediaCCCConferenceKiosk(service,
new MediaCCCConferencesListLinkHandlerFactory().fromId("conferences"),
MediaCCCConferencesListLinkHandlerFactory.getInstance()
.fromId("conferences"),
"conferences");
} catch (final Exception e) {
e.printStackTrace();
@ -156,9 +158,10 @@ public class MediaCCCSearchExtractor extends SearchExtractor {
return item.getUrl();
}
@Nonnull
@Override
public String getThumbnailUrl() {
return item.getThumbnailUrl();
public List<Image> getThumbnails() {
return item.getThumbnails();
}
});
}

View File

@ -1,10 +1,17 @@
package org.schabi.newpipe.extractor.services.media_ccc.extractors;
import static org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCParsingHelper.getImageListFromLogoImageUrl;
import static org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCParsingHelper.getThumbnailsFromStreamItem;
import static org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCParsingHelper.parseDateFrom;
import static org.schabi.newpipe.extractor.stream.AudioStream.UNKNOWN_BITRATE;
import static org.schabi.newpipe.extractor.stream.Stream.ID_UNKNOWN;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParser;
import com.grack.nanojson.JsonParserException;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.MediaFormat;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.downloader.Downloader;
@ -21,6 +28,7 @@ import org.schabi.newpipe.extractor.stream.StreamExtractor;
import org.schabi.newpipe.extractor.stream.StreamType;
import org.schabi.newpipe.extractor.stream.VideoStream;
import org.schabi.newpipe.extractor.utils.JsonUtils;
import org.schabi.newpipe.extractor.utils.LocaleCompat;
import java.io.IOException;
import java.util.ArrayList;
@ -47,13 +55,13 @@ public class MediaCCCStreamExtractor extends StreamExtractor {
@Nonnull
@Override
public DateWrapper getUploadDate() throws ParsingException {
return new DateWrapper(MediaCCCParsingHelper.parseDateFrom(getTextualUploadDate()));
return new DateWrapper(parseDateFrom(getTextualUploadDate()));
}
@Nonnull
@Override
public String getThumbnailUrl() {
return data.getString("thumb_url");
public List<Image> getThumbnails() {
return getThumbnailsFromStreamItem(data);
}
@Nonnull
@ -87,8 +95,8 @@ public class MediaCCCStreamExtractor extends StreamExtractor {
@Nonnull
@Override
public String getUploaderAvatarUrl() {
return conferenceData.getString("logo_url");
public List<Image> getUploaderAvatars() {
return getImageListFromLogoImageUrl(conferenceData.getString("logo_url"));
}
@Override
@ -99,7 +107,7 @@ public class MediaCCCStreamExtractor extends StreamExtractor {
final JsonObject recording = recordings.getObject(i);
final String mimeType = recording.getString("mime_type");
if (mimeType.startsWith("audio")) {
//first we need to resolve the actual video data from CDN
// First we need to resolve the actual video data from the CDN
final MediaFormat mediaFormat;
if (mimeType.endsWith("opus")) {
mediaFormat = MediaFormat.OPUS;
@ -108,11 +116,30 @@ public class MediaCCCStreamExtractor extends StreamExtractor {
} else if (mimeType.endsWith("ogg")) {
mediaFormat = MediaFormat.OGG;
} else {
throw new ExtractionException("Unknown media format: " + mimeType);
mediaFormat = null;
}
audioStreams.add(new AudioStream(recording.getString("recording_url"),
mediaFormat, -1));
final AudioStream.Builder builder = new AudioStream.Builder()
.setId(recording.getString("filename", ID_UNKNOWN))
.setContent(recording.getString("recording_url"), true)
.setMediaFormat(mediaFormat)
.setAverageBitrate(UNKNOWN_BITRATE);
final String language = recording.getString("language");
// If the language contains a - symbol, this means that the stream has an audio
// track with multiple languages, so there is no specific language for this stream
// Don't set the audio language in this case
if (language != null && !language.contains("-")) {
builder.setAudioLocale(LocaleCompat.forLanguageTag(language).orElseThrow(() ->
new ParsingException(
"Cannot convert this language to a locale: " + language)
));
}
// Not checking containsSimilarStream here, since MediaCCC does not provide enough
// information to decide whether two streams are similar. Hence that method would
// always return false, e.g. even for different language variations.
audioStreams.add(builder.build());
}
}
return audioStreams;
@ -126,21 +153,29 @@ public class MediaCCCStreamExtractor extends StreamExtractor {
final JsonObject recording = recordings.getObject(i);
final String mimeType = recording.getString("mime_type");
if (mimeType.startsWith("video")) {
//first we need to resolve the actual video data from CDN
// First we need to resolve the actual video data from the CDN
final MediaFormat mediaFormat;
if (mimeType.endsWith("webm")) {
mediaFormat = MediaFormat.WEBM;
} else if (mimeType.endsWith("mp4")) {
mediaFormat = MediaFormat.MPEG_4;
} else {
throw new ExtractionException("Unknown media format: " + mimeType);
mediaFormat = null;
}
videoStreams.add(new VideoStream(recording.getString("recording_url"),
mediaFormat, recording.getInt("height") + "p"));
// Not checking containsSimilarStream here, since MediaCCC does not provide enough
// information to decide whether two streams are similar. Hence that method would
// always return false, e.g. even for different language variations.
videoStreams.add(new VideoStream.Builder()
.setId(recording.getString("filename", ID_UNKNOWN))
.setContent(recording.getString("recording_url"), true)
.setIsVideoOnly(false)
.setMediaFormat(mediaFormat)
.setResolution(recording.getInt("height") + "p")
.build());
}
}
return videoStreams;
}
@ -163,7 +198,8 @@ public class MediaCCCStreamExtractor extends StreamExtractor {
conferenceData = JsonParser.object()
.from(downloader.get(data.getString("conference_url")).responseBody());
} catch (final JsonParserException jpe) {
throw new ExtractionException("Could not parse json returned by url: " + videoUrl, jpe);
throw new ExtractionException("Could not parse json returned by URL: " + videoUrl,
jpe);
}
}

View File

@ -2,10 +2,16 @@ package org.schabi.newpipe.extractor.services.media_ccc.extractors.infoItems;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.channel.ChannelInfoItemExtractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import javax.annotation.Nonnull;
import java.util.List;
import static org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCParsingHelper.getImageListFromLogoImageUrl;
public class MediaCCCConferenceInfoItemExtractor implements ChannelInfoItemExtractor {
private final JsonObject conference;
@ -43,8 +49,9 @@ public class MediaCCCConferenceInfoItemExtractor implements ChannelInfoItemExtra
return conference.getString("url");
}
@Nonnull
@Override
public String getThumbnailUrl() {
return conference.getString("logo_url");
public List<Image> getThumbnails() {
return getImageListFromLogoImageUrl(conference.getString("logo_url"));
}
}

View File

@ -1,13 +1,18 @@
package org.schabi.newpipe.extractor.services.media_ccc.extractors.infoItems;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.localization.DateWrapper;
import org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCParsingHelper;
import org.schabi.newpipe.extractor.stream.StreamInfoItemExtractor;
import org.schabi.newpipe.extractor.stream.StreamType;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.List;
import static org.schabi.newpipe.extractor.services.media_ccc.extractors.MediaCCCParsingHelper.getThumbnailsFromStreamItem;
public class MediaCCCStreamInfoItemExtractor implements StreamInfoItemExtractor {
private final JsonObject event;
@ -46,12 +51,6 @@ public class MediaCCCStreamInfoItemExtractor implements StreamInfoItemExtractor
return event.getString("conference_url");
}
@Nullable
@Override
public String getUploaderAvatarUrl() {
return null;
}
@Override
public boolean isUploaderVerified() throws ParsingException {
return false;
@ -84,8 +83,9 @@ public class MediaCCCStreamInfoItemExtractor implements StreamInfoItemExtractor
+ event.getString("guid");
}
@Nonnull
@Override
public String getThumbnailUrl() {
return event.getString("thumb_url");
public List<Image> getThumbnails() {
return getThumbnailsFromStreamItem(event);
}
}

View File

@ -1,12 +1,22 @@
package org.schabi.newpipe.extractor.services.media_ccc.linkHandler;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabs;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import org.schabi.newpipe.extractor.utils.Parser;
import java.util.List;
public class MediaCCCConferenceLinkHandlerFactory extends ListLinkHandlerFactory {
/**
* Since MediaCCC does not really have channel tabs (i.e. it only has one single "tab" with videos),
* this link handler acts both as the channel link handler and the channel tab link handler. That's
* why {@link #getAvailableContentFilter()} has been overridden.
*/
public final class MediaCCCConferenceLinkHandlerFactory extends ListLinkHandlerFactory {
private static final MediaCCCConferenceLinkHandlerFactory INSTANCE
= new MediaCCCConferenceLinkHandlerFactory();
public static final String CONFERENCE_API_ENDPOINT
= "https://api.media.ccc.de/public/conferences/";
public static final String CONFERENCE_PATH = "https://media.ccc.de/c/";
@ -14,15 +24,23 @@ public class MediaCCCConferenceLinkHandlerFactory extends ListLinkHandlerFactory
= "(?:(?:(?:api\\.)?media\\.ccc\\.de/public/conferences/)"
+ "|(?:media\\.ccc\\.de/[bc]/))([^/?&#]*)";
private MediaCCCConferenceLinkHandlerFactory() {
}
public static MediaCCCConferenceLinkHandlerFactory getInstance() {
return INSTANCE;
}
@Override
public String getUrl(final String id,
final List<String> contentFilter,
final String sortFilter) throws ParsingException {
final String sortFilter)
throws ParsingException, UnsupportedOperationException {
return CONFERENCE_PATH + id;
}
@Override
public String getId(final String url) throws ParsingException {
public String getId(final String url) throws ParsingException, UnsupportedOperationException {
return Parser.matchGroup1(ID_PATTERN, url);
}
@ -34,4 +52,15 @@ public class MediaCCCConferenceLinkHandlerFactory extends ListLinkHandlerFactory
return false;
}
}
/**
* @see MediaCCCConferenceLinkHandlerFactory
* @return MediaCCC's only channel "tab", i.e. {@link ChannelTabs#VIDEOS}
*/
@Override
public String[] getAvailableContentFilter() {
return new String[]{
ChannelTabs.VIDEOS,
};
}
}

View File

@ -5,15 +5,28 @@ import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import java.util.List;
public class MediaCCCConferencesListLinkHandlerFactory extends ListLinkHandlerFactory {
public final class MediaCCCConferencesListLinkHandlerFactory extends ListLinkHandlerFactory {
private static final MediaCCCConferencesListLinkHandlerFactory INSTANCE =
new MediaCCCConferencesListLinkHandlerFactory();
private MediaCCCConferencesListLinkHandlerFactory() {
}
public static MediaCCCConferencesListLinkHandlerFactory getInstance() {
return INSTANCE;
}
@Override
public String getId(final String url) throws ParsingException {
public String getId(final String url) throws ParsingException, UnsupportedOperationException {
return "conferences";
}
@Override
public String getUrl(final String id, final List<String> contentFilter,
final String sortFilter) throws ParsingException {
public String getUrl(final String id,
final List<String> contentFilter,
final String sortFilter)
throws ParsingException, UnsupportedOperationException {
return "https://media.ccc.de/public/conferences";
}

View File

@ -6,11 +6,22 @@ import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import java.util.List;
import java.util.regex.Pattern;
public class MediaCCCLiveListLinkHandlerFactory extends ListLinkHandlerFactory {
public final class MediaCCCLiveListLinkHandlerFactory extends ListLinkHandlerFactory {
private static final MediaCCCLiveListLinkHandlerFactory INSTANCE =
new MediaCCCLiveListLinkHandlerFactory();
private static final String STREAM_PATTERN = "^(?:https?://)?media\\.ccc\\.de/live$";
private MediaCCCLiveListLinkHandlerFactory() {
}
public static MediaCCCLiveListLinkHandlerFactory getInstance() {
return INSTANCE;
}
@Override
public String getId(final String url) throws ParsingException {
public String getId(final String url) throws ParsingException, UnsupportedOperationException {
return "live";
}
@ -22,7 +33,8 @@ public class MediaCCCLiveListLinkHandlerFactory extends ListLinkHandlerFactory {
@Override
public String getUrl(final String id,
final List<String> contentFilter,
final String sortFilter) throws ParsingException {
final String sortFilter)
throws ParsingException, UnsupportedOperationException {
// FIXME: wrong URL; should be https://streaming.media.ccc.de/{conference_slug}/{room_slug}
return "https://media.ccc.de/live";
}

View File

@ -1,32 +0,0 @@
package org.schabi.newpipe.extractor.services.media_ccc.linkHandler;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.LinkHandlerFactory;
import org.schabi.newpipe.extractor.utils.Parser;
public class MediaCCCLiveStreamLinkHandlerFactory extends LinkHandlerFactory {
public static final String VIDEO_API_ENDPOINT = "https://api.media.ccc.de/public/events/";
private static final String VIDEO_PATH = "https://streaming.media.ccc.de/v/";
private static final String ID_PATTERN
= "(?:(?:(?:api\\.)?media\\.ccc\\.de/public/events/)"
+ "|(?:media\\.ccc\\.de/v/))([^/?&#]*)";
@Override
public String getId(final String url) throws ParsingException {
return Parser.matchGroup1(ID_PATTERN, url);
}
@Override
public String getUrl(final String id) throws ParsingException {
return VIDEO_PATH + id;
}
@Override
public boolean onAcceptUrl(final String url) {
try {
return getId(url) != null;
} catch (final ParsingException e) {
return false;
}
}
}

View File

@ -1,15 +1,27 @@
package org.schabi.newpipe.extractor.services.media_ccc.linkHandler;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import java.util.List;
import java.util.regex.Pattern;
public class MediaCCCRecentListLinkHandlerFactory extends ListLinkHandlerFactory {
public final class MediaCCCRecentListLinkHandlerFactory extends ListLinkHandlerFactory {
private static final MediaCCCRecentListLinkHandlerFactory INSTANCE =
new MediaCCCRecentListLinkHandlerFactory();
private static final String PATTERN = "^(https?://)?media\\.ccc\\.de/recent/?$";
private MediaCCCRecentListLinkHandlerFactory() {
}
public static MediaCCCRecentListLinkHandlerFactory getInstance() {
return INSTANCE;
}
@Override
public String getId(final String url) {
public String getId(final String url) throws ParsingException, UnsupportedOperationException {
return "recent";
}
@ -21,7 +33,8 @@ public class MediaCCCRecentListLinkHandlerFactory extends ListLinkHandlerFactory
@Override
public String getUrl(final String id,
final List<String> contentFilter,
final String sortFilter) {
final String sortFilter)
throws ParsingException, UnsupportedOperationException {
return "https://media.ccc.de/recent";
}
}

Some files were not shown because too many files have changed in this diff Show More