Compare commits

...

1786 Commits
v0.19.0 ... dev

Author SHA1 Message Date
TobiGr fafd471606 [PeerTube] Fix test for like count
Number changed
2024-05-08 19:25:12 +02:00
TobiGr 4f477ad72b [PeerTube] Fix testing comment content
The comment is not available anymore.
2024-05-08 19:25:12 +02:00
Tobi 964e429978
Merge pull request #1166 from TeamNewPipe/dependabot/github_actions/peaceiris/actions-gh-pages-4
Bump peaceiris/actions-gh-pages from 3 to 4
2024-05-08 19:24:40 +02:00
dependabot[bot] 10c6965a28
Bump peaceiris/actions-gh-pages from 3 to 4
Bumps [peaceiris/actions-gh-pages](https://github.com/peaceiris/actions-gh-pages) from 3 to 4.
- [Release notes](https://github.com/peaceiris/actions-gh-pages/releases)
- [Changelog](https://github.com/peaceiris/actions-gh-pages/blob/main/CHANGELOG.md)
- [Commits](https://github.com/peaceiris/actions-gh-pages/compare/v3...v4)

---
updated-dependencies:
- dependency-name: peaceiris/actions-gh-pages
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-05-08 17:16:08 +00:00
TobiGr e54f38f5e7 [PeerTube] Fix test
UploaderName was changed by user
2024-05-08 19:13:49 +02:00
Tobi 5dd5c7a65b
Merge pull request #1173 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.8.5
Bump com.github.spotbugs:spotbugs-annotations from 4.8.3 to 4.8.5
2024-05-08 19:12:40 +02:00
Tobi 37438ff82f
Merge pull request #1174 from TeamNewPipe/dependabot/gradle/org.mozilla-rhino-1.7.15
Bump org.mozilla:rhino from 1.7.13 to 1.7.15
2024-05-08 19:02:57 +02:00
TobiGr bba3b6c69b version 0.24.0 2024-05-08 13:01:06 +02:00
TobiGr b40a5784ed Fix JDoc
YoutubePostLiveStreamDvrDashManifestCreator.java:90: error: unexpected end tag: </p>
2024-05-08 12:41:36 +02:00
dependabot[bot] c6da4004e2
Bump org.mozilla:rhino from 1.7.13 to 1.7.15
Bumps [org.mozilla:rhino](https://github.com/mozilla/rhino) from 1.7.13 to 1.7.15.
- [Release notes](https://github.com/mozilla/rhino/releases)
- [Changelog](https://github.com/mozilla/rhino/blob/master/RELEASE-NOTES.md)
- [Commits](https://github.com/mozilla/rhino/commits)

---
updated-dependencies:
- dependency-name: org.mozilla:rhino
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-05-06 09:43:05 +00:00
dependabot[bot] f26e84d39f
Bump com.github.spotbugs:spotbugs-annotations from 4.8.3 to 4.8.5
Bumps [com.github.spotbugs:spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.8.3 to 4.8.5.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.8.3...4.8.5)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-05-06 09:43:03 +00:00
Tobi ec3e8378c6
Merge pull request #1171 from TeamNewPipe/fix/jdoc
Fix JavaDocs and add Checkstyle exception for line length in JavaDocs' links
2024-04-23 21:22:12 +02:00
TobiGr 8d2a7a5281 [PeerTube] Fix test 2024-04-23 20:02:29 +02:00
TobiGr 7c29dbc965 Fix JDoc
Add Checkstyle exception for LineLength in JDoc links.

See https://github.com/TeamNewPipe/NewPipeExtractor/actions/runs/8804403691/job/24164703883
2024-04-23 19:55:51 +02:00
Stypox fbe9e6223a
Merge pull request #1168 from AudricV/yt_upd-cver-rm-keys-and-do-fixes
[YouTube] Update clients versions, restore access to some streams and more
2024-04-20 11:54:24 +02:00
Stypox 4e9e7cb29c
Improve assertTabsContain() to also check size 2024-04-20 11:48:36 +02:00
Stypox 9d0dd36034
[YouTube] Create constants for client names/versions 2024-04-20 11:43:54 +02:00
Stypox d4e6d22e64
[YouTube] Improve meta info code for review 2024-04-20 11:43:08 +02:00
AudricV 74bf000473
[YouTube] Update mocks 2024-04-11 18:59:11 +02:00
AudricV f9792cf3a9
[YouTube] Fix InteractiveTabbedHeader.testTabs test
YouTube now returns a Shorts tab for InteractiveTabbedHeader gaming channels,
which contains Shorts about the game of the topic channel but are not uploaded
on the game topic channel.

As this tab is already supported by the extractor, fetching a gaming topic
channel now returns a tab instead of none.

Channel name and channel URL of these Shorts needs to be set to null in a
separate commit, as Shorts on this tab do not have the topic channel as their
uploader.
2024-04-11 18:59:10 +02:00
AudricV f40fc0aa4f
[YouTube] Add support for new crisis meta info action data
The new action data can return multiple contact actions instead of only one,
which will be concatenated by a new line return.

This commit fixes tests of the CrisisResources test class of
YoutubeSearchExtractorTest.
2024-04-11 18:59:09 +02:00
AudricV 2a3c6f80d2
[YouTube] Fix YoutubeStreamExtractorRelatedMixTest 2024-04-10 21:19:03 +02:00
AudricV 657b4377aa
[YouTube] Fix YoutubeStreamExtractorDefaultTest tests 2024-04-10 21:19:03 +02:00
AudricV 7bf50bf1cb
[YouTube] Update Android client player parameters
YouTube disabled the effectiveness of the parameters which were used (the
player response we get redirects to another video), but new parameters which
work around Android's client integrity checks have been found.
2024-04-10 21:19:03 +02:00
AudricV 27dc1b1f50
[YouTube] Remove usage of API keys for InnerTube requests, bump versions
The API keys are not used anymore by official clients in almost all cases
(still used by the Android app until it gets a configuration) for all requests
we made.

Clients and device OS versions have been bumped to their latest stable version
known.

Methods and fields related to API keys have been renamed or deleted if they're
no longer relevant.
2024-04-10 21:19:02 +02:00
AudricV e380bb4bc3
[YouTube] Add missing prettyPrint query parameter to mixes continuations 2024-04-10 19:06:36 +02:00
Audric V 6c3c2e25d7
Merge pull request #1163 from AudricV/yt-fix_comments_extraction
[YouTube] Support new comments data
2024-04-10 18:19:59 +02:00
Stypox 02274d5395
[YouTube] Avoid XSS attacks in description or comments 2024-04-08 11:21:31 +02:00
Stypox 3f7b2653e3
[YouTube] Add YoutubeDescriptionHelperTest 2024-04-08 11:21:31 +02:00
Stypox a90237816a
[YouTube] Cleanup description helper
Remove unneeded isClose field, and make constants private
2024-04-08 11:21:31 +02:00
Stypox b80c3f5d51
[YouTube] Replace link text with accessibility label 2024-04-08 00:14:28 +02:00
Stypox 09732d6785
[YouTube] Add support for styles in attributed descriptions
Also refactor descriptions parsing.
2024-04-04 21:14:27 +02:00
AudricV 293c3e9e47
[YouTube] Support new A/B tested comments data
Also improve current comments code by removing outdated comment
renderer data.
2024-04-04 21:14:26 +02:00
Stypox e5b30ae8c3
Merge pull request #1151 from Profpatsch/localization-return-optional
LocaleCompat.forLanguageTag: return Optional if parsing fails
2024-03-29 13:50:48 +01:00
Stypox 23fc7aa209
Throw ParsingException instead of IllegalArg 2024-03-29 13:44:42 +01:00
Stypox fb468a23f4
Merge pull request #1142 from TeamNewPipe/peertube-v6
[PeerTube] Add support for PeerTube v6 features
2024-03-29 12:25:38 +01:00
Stypox 6589e2c15d
Merge pull request #1148 from Stypox/mediaccc-channel-tab-handler
[MediaCCC] Allow obtaining channel tab link handler
2024-03-28 13:45:05 +01:00
Tobi ad71864b23
Merge pull request #1162 from Stypox/fix-comment-description-npe
Make getCommentText @Nonnull
2024-03-27 20:04:17 +01:00
Stypox c57016b79b
Make getCommentText @Nonnull 2024-03-27 15:26:06 +01:00
Tobi adcc1f17ee
Merge pull request #1160 from TeamNewPipe/fix/tests
Fix some failing unit tests and detect new account termination messages for YouTube
2024-03-20 15:17:55 +01:00
TobiGr 51ddacc81d [SoundCloud] Fix SoundcloudSearchExtractorTest.NoNextPage
Search did not return no item at all, causing a NothingFoundException. New search query yields three items on first page
2024-03-20 15:10:39 +01:00
TobiGr 8392d50ba6 Update mocks for YoutubeChannelExtractorTest.NotAvailable 2024-03-20 14:59:44 +01:00
TobiGr aaccfecda8 [YouTube] Detect new account termination messages 2024-03-20 14:57:41 +01:00
TobiGr 73f0c63a9d [PeerTube] Fix tests for "What is PeerTube?" video 2024-03-20 14:44:06 +01:00
Tobi 896a55e319
Merge pull request #1139 from TeamNewPipe/dependabot/github_actions/actions/upload-artifact-4
Bump actions/upload-artifact from 3 to 4
2024-03-18 08:59:35 +01:00
dependabot[bot] e58fc652e0
Bump actions/upload-artifact from 3 to 4
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3 to 4.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-03-18 07:53:24 +00:00
Tobi e3f2c9aec7
Merge pull request #1153 from TeamNewPipe/dependabot/github_actions/actions/cache-4
Bump actions/cache from 3 to 4
2024-03-18 08:51:24 +01:00
Tobi 6b0fc14c04
Merge pull request #1156 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.10.2
Bump org.junit:junit-bom from 5.10.0 to 5.10.2
2024-03-18 08:46:13 +01:00
dependabot[bot] d579b608e5
Bump org.junit:junit-bom from 5.10.0 to 5.10.2
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.10.0 to 5.10.2.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.10.0...r5.10.2)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-02-05 09:15:26 +00:00
TobiGr fe47a4311f [PeerTube] Add test for segments and framesets 2024-01-29 10:22:06 +01:00
TobiGr 15e0e74b48 [PeerTube] Add support for stream frames/storyboards extraction
Implement PeerTubeStreamExtractor.getFrames()
2024-01-29 10:22:06 +01:00
dependabot[bot] da04eded5d
Bump actions/cache from 3 to 4
Bumps [actions/cache](https://github.com/actions/cache) from 3 to 4.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-01-18 09:32:42 +00:00
Profpatsch 7408173246 LocaleCompat.forLanguageTag: return Optional if parsing fails
It’s not obvious that the function will fail in some cases and throw
an `IllegalArgumentException`.

So instead of just failing if parsing fails, return an Optional that
all callers have to decide what to do (e.g. the YoutubeExtractor can
just ignore the locale in that case, like it does with most other
fields in the json if they are unexpected).
2024-01-07 14:31:34 +01:00
Stypox aaf3231fc7
[MediaCCC] Fix lambda link handler keeping reference to extractor
This caused problems in NewPipe, because extractors are not serializable, and well, keeping references to them is a bad idea anyway.
2023-12-30 23:23:19 +01:00
Stypox 137e924035
[MediaCCC] Add ChannelTabExtractorTest 2023-12-30 22:53:51 +01:00
Stypox cc9ade962e
[MediaCCC] Allow obtaining channel tab extractor from scratch
i.e. without needing to pass through the conference/channel extractor
This was needed because clients (like NewPipe) might rely on link handlers to hold as little data as possible, since they might be kept around for long or passed around in system transactions, so this commit allows obtaining a standalone link handler that does not hold a JsonObject within itself.
2023-12-30 22:53:27 +01:00
Stypox 3402cdb666
Merge pull request #1147 from petlyh/youtube-albums
[YouTube] Add Albums channel tab
2023-12-30 21:58:19 +01:00
petlyh 6dc25f7b97
[YouTube] Add Albums channel tab mocks 2023-12-30 14:46:39 +01:00
petlyh 4408e2d0ac
[YouTube] Add Albums channel tab 2023-12-30 14:01:30 +01:00
TobiGr 9ab932e394 Rename testDoNotAcceptNonURLs() -> assertDoNotAcceptNonURLs() 2023-12-29 16:38:11 +01:00
Stypox 9d66debf3c
Merge pull request #1132 from TeamNewPipe/dependabot/github_actions/actions/setup-java-4
Bump actions/setup-java from 3 to 4
2023-12-29 16:20:46 +01:00
Tobi 038ebdedc4
Merge pull request #1143 from petlyh/peertube-non-urls
Avoid PeerTube accepting non-URLs
2023-12-29 12:47:30 +01:00
TobiGr 61d237de02 [PeerTube] Test onAccept(String URL) in LinkHandlerFactories for non-URLs 2023-12-29 12:45:02 +01:00
petlyh 2b2c1546d1 Avoid PeerTube accepting non-URLs 2023-12-29 12:27:39 +01:00
Tobi 1e93b1dc20
Merge pull request #1135 from Stypox/yt-emergency-info
[YouTube] Implement emergency meta info
2023-12-29 12:01:40 +01:00
dependabot[bot] 3400af99b3
Bump actions/setup-java from 3 to 4
Bumps [actions/setup-java](https://github.com/actions/setup-java) from 3 to 4.
- [Release notes](https://github.com/actions/setup-java/releases)
- [Commits](https://github.com/actions/setup-java/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/setup-java
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-29 10:57:47 +00:00
Tobi 1f8a044462
Merge pull request #1138 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.8.3
Bump com.github.spotbugs:spotbugs-annotations from 4.8.0 to 4.8.3
2023-12-29 11:56:18 +01:00
dependabot[bot] 1470aa7303
Bump com.github.spotbugs:spotbugs-annotations from 4.8.0 to 4.8.3
Bumps [com.github.spotbugs:spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.8.0 to 4.8.3.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.8.0...4.8.3)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-29 10:53:28 +00:00
TobiGr 8f9ebdcb77 [PeerTube] Fix failing PeertubeTrendingLinkHandlerFactoryTest
The factory was updated in #1144
2023-12-29 11:52:19 +01:00
Stypox 1553931027
Merge pull request #1145 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.17.2
Bump org.jsoup:jsoup from 1.16.2 to 1.17.2
2023-12-29 11:27:01 +01:00
Stypox b2ec1b15fb
Merge pull request #1144 from dragfyre/patch-1
Update PeertubeTrendingLinkHandlerFactory.java
2023-12-29 11:21:08 +01:00
dependabot[bot] 151ee99da3
Bump org.jsoup:jsoup from 1.16.2 to 1.17.2
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup) from 1.16.2 to 1.17.2.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES.md)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.16.2...jsoup-1.17.2)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-29 09:47:00 +00:00
dragfyre 65e7bc5b95
Update PeertubeTrendingLinkHandlerFactory.java
correcting Peertube local trending api URL (per #10685 in main NewPipe repo); see https://docs.joinpeertube.org/api-rest-reference.html#tag/Video/operation/getVideos
2023-12-28 14:50:31 +07:00
Stypox f276caf54a
Release v0.23.1 2023-12-21 21:59:21 +01:00
Stypox fc54fb2fdb
Merge pull request #1140 from Stypox/yt-shorts-no-duration
[YouTube] Always return -1 as duration of Shorts returned inside reel items
2023-12-21 21:52:40 +01:00
Stypox 0518487d26
Fix SearchInfo's non-null MetaInfo being null when initialized or when an extraction error occurs (#1141)
The meta info might have been null either when SearchInfo is first initialized, or when extractor.getMetaInfo() throws an exception in getInfo().
This caused NewPipe to crash instead of showing a nice error in https://www.reddit.com/r/youtube/comments/184ttmw/what_exactly_about_blue_whales_has_youtube_so/.
2023-12-21 21:39:25 +01:00
Stypox 5b59a1a8c5
[YouTube] Move meta info extraction to separate file
YoutubeParsingHelper was longer than 2000 lines which caused checkstyle issues
2023-12-21 21:19:08 +01:00
Stypox b8e12dd76c
[YouTube] Implement emergency meta info
YouTube provides that meta info panel when users search for really sensitive content like suicide (e.g. "blue whale").

It contains:
- an encouragement as title (e.g. "We are with you")
- a phone number as action
- details about how to call the phone number (e.g. availability)
- an url pointing to the website of an association

Also add a test that just checks if a meta info is properly extracted
2023-12-21 21:19:08 +01:00
Stypox 83c1737f70
[YouTube] Update shorts channel tab mocks 2023-12-21 20:54:16 +01:00
Stypox 2938067c2c
[YouTube] Shorts don't provide a duration anymore 2023-12-21 20:41:01 +01:00
Stypox 91419ec6e8
Release v0.23.0 2023-12-10 15:59:32 +01:00
Audric V 678c98f24c
Merge pull request #1127 from AudricV/yt_improvements-and-fixes
[YouTube] Make some improvements and fixes
2023-12-09 14:07:31 +01:00
AudricV ec0194cfbf
[YouTube] Update mocks 2023-12-08 21:46:52 +01:00
AudricV 00a0f1a103
[YouTube] Add a blocking crisis resources bypass in search results test
This test only tests that search results are returned, when no content filters
are provided and crisis resources blocking search results should be returned.

Searches with blocking crisis resources and content filters should work too, as
the bypass has been implemented for them.
2023-12-08 21:46:49 +01:00
AudricV 06838d7245
[YouTube] Fix YoutubeSearchQHTest tests
As search parameters to bypass crisis resources blocking search results have
been implemented, they need to be added to search tests, in order to pass
them.
2023-12-08 21:46:49 +01:00
AudricV 642bb01388
[YouTube] Fix testSubscriberCount method of YoutubeChannelExtractorTest.CarouselHeader
The subscriber count is now lower than the expected count as some people
unsubscribed to the Sports system channel. The expected count has been so
lowered.
2023-12-08 21:46:49 +01:00
AudricV bedc9e5bc0
[YouTube] Remove Channels channel tab in tests 2023-12-08 21:46:48 +01:00
AudricV 5fa22ae25b
[YouTube] Enable and change testRelatedItems method of YoutubePlaylistExtractorTest.LearningPlaylist
This change makes the method test whether there is no more items instead of
being disabled.
2023-12-08 21:46:48 +01:00
AudricV 29dc7625f2
[YouTube] Fix YoutubeSearchExtractorTest.Suggestion
YouTube doesn't return anymore a suggestion for the query "algorythm", but does
for the query "on board ing" ("on boarding"). This search query is now used and
had to be URL-encoded.

URL encoding in the complete YoutubeSearchExtractorTest test class uses now
extractor's Util class instead of Java's URLDecoder class directly.
2023-12-08 21:46:48 +01:00
AudricV 56ab35423e
[YouTube] Fix potential NullPointerException in YoutubeSearchExtractor.getSearchSuggestion 2023-12-08 21:46:48 +01:00
AudricV 9dc1eab28c
[YouTube] Fix expected upload dates of StreamExtractor tests
YouTube is rolling out or A/B testing a new date format returned inside player
responses, which are precise to the second instead of the day.

This commit makes the StreamExtractor tests use these more precise dates.
2023-12-08 21:46:48 +01:00
AudricV ad3d187ac7
[YouTube] Fix testRelatedItems method of YoutubeStreamExtractorRelatedMixTest
This commit fixes the testRelatedItems test method by:
- accepting consent in the test class, in order to extract mixes in
recommendations;
- removing assertion of a music mix inside the recommendations, as YouTube
doesn't seem to return such mixes anymore, at least for the video used in the
test class.
2023-12-08 21:46:47 +01:00
AudricV e111814401
[YouTube] Fix YoutubeStreamExtractorDefaultTest.RatingsDisabledTest
Replace the video used in this test class with another one publicly available
and update the corresponding expected test values.

The test class's mocks will be updated in a different commit.
2023-12-08 21:46:47 +01:00
AudricV fc45941ead
[YouTube] Fix YoutubeChannelExtractorTest tests
- Change CarouselHeader test channel to Sports system one, as the Coachella one
doesn't return this channel header anymore;
- Fix InteractiveTabbedHeader test by checking whether the test's channel
description is not empty instead of containing some words, as it is changing
frequently.
2023-12-08 21:46:47 +01:00
AudricV 0bcb241c38
[YouTube] Fix expected data in YoutubeStreamExtractorRelatedMixTest
Video's title and tags have been changed by its uploader, so they have to be
updated.

Also make some package-private constants private, as they are not used outside
of the class, and remove unneeded test overrides.
2023-12-08 21:46:47 +01:00
AudricV 6ba8251be1
[YouTube] Bypass crisis resources blocking search results
These crisis resources are preventing search results to be returned. See
https://support.google.com/youtube/answer/10726080?hl=en for more info on them.

This commit changes search parameters to include the property allowing to show
search results.
2023-12-08 21:46:47 +01:00
AudricV 7dea2d0d27
[YouTube] Remove Channels channel tab support
This tab has been removed by YouTube.
2023-12-08 21:46:47 +01:00
AudricV 3782d9a02a
[YouTube] Support new A/B tested like data and avoid like count conversion from integer to long
Also make minor improvements to current like data extraction and remove
previous like count data support, as it is not returned anymore.
2023-12-08 21:46:46 +01:00
AudricV b71ce1123f
[YouTube] Extract only search results corresponding to a search type
YouTube returns sometimes videos inside channel search results. As we only want
results corresponding to the type we requested, this commits makes
YoutubeSearchExtractor ignoring non-requested search results we get, using the
extractor LinkHandler's first content filter value.

Also remove an unneeded exception throwing declaration in
YoutubeSearchExtractor.
2023-12-08 21:46:46 +01:00
AudricV ff8ed7247f
[YouTube] Switch to new consent cookie
Also move the documentation of the consent in its setter method in order to be
accessible publicly and improve it.
2023-12-08 21:46:46 +01:00
AudricV ec838d7421
[YouTube] Add missing prettyPrint query parameter to some test InnerTube requests
This query parameter for which its value is set to false was not added to two
requests made in test classes of YoutubeMixPlaylistExtractorTest.

Also remove an unneeded ParsingException exception throwing declaration in a
test method.
2023-12-08 21:46:46 +01:00
AudricV 2c941794c0
[YouTube] Add utcOffsetMinutes to all InnerTube payloads
This should make returned dates consistent between timezones and countries on
which the extractor is ran.

It was previously only set on YouTube Music search continuations.
2023-12-08 21:46:46 +01:00
AudricV d97c9e0db1
[YouTube] Improve payloads and URLs of InnerTube requests
For every InnerTube request:
- Always add a `request` object with the following properties:
  - "internalExperimentFlags" set to an empty array;
  - "useSsl" set to "true";
  - "lockedSafetyMode" set to "false".
- Use proper TODO comment to provide a way to enable restricted mode on every
request and add it on requests on which it wasn't present.

For YouTube Music:
- Remove alt query parameter, as it is not used anymore by the website;
- Add prettyPrint query parameter with false value on YouTube Music search
continuations.
2023-12-08 21:46:45 +01:00
AudricV 8a9ebcc373
[YouTube] Update InnerTube clients' version and devices' OS version and model 2023-12-08 21:46:45 +01:00
Tobi eac850ca10
Merge pull request #1114 from FineFindus/feat/comment-author-is-owner
[YouTube] Add channel owner to comments
2023-10-25 09:51:12 +02:00
Tobi 5ab1f784e8
Merge pull request #1117 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.16.2
Bump org.jsoup:jsoup from 1.16.1 to 1.16.2
2023-10-21 19:26:36 +02:00
dependabot[bot] 9d7bcba050
Bump org.jsoup:jsoup from 1.16.1 to 1.16.2
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup) from 1.16.1 to 1.16.2.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.16.1...jsoup-1.16.2)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-20 09:13:21 +00:00
dependabot[bot] e26065148a Bump com.github.spotbugs:spotbugs-annotations from 4.7.3 to 4.8.0
Bumps [com.github.spotbugs:spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.7.3 to 4.8.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.3...4.8.0)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-12 14:19:10 +02:00
FineFindus 34b05a0dda
feat(youtube/comments): support creator replies 2023-10-09 16:33:43 +02:00
TobiGr 0821f09114
Add missing mocks 2023-10-09 16:33:43 +02:00
FineFindus c1784a4bdb
[YouTube] Add channel owner to comments 2023-10-09 16:33:43 +02:00
TobiGr f9846352ea Fix wrong `@Nullable` annotation 2023-10-09 16:02:57 +02:00
Tobi d6f5cba6e2
Merge pull request #1111 from FineFindus/feat/creator-reply
Add `hasCreatorReply()` to CommentsInfoItem
2023-10-09 12:45:56 +02:00
TobiGr 9d63c75623 Add missing mocks 2023-10-09 11:24:39 +02:00
TobiGr d49f8411d7 [PeerTube] Implement CommentsInfoItemExtractor.hasCreatorReply() 2023-10-09 02:47:12 +02:00
Stypox bb132167d5
Merge pull request #1113 from AudricV/snd_fix-non-jpg-images
[SoundCloud] Fix extraction of non-JPG images
2023-10-02 19:40:57 +02:00
AudricV c98695fcea
[SoundCloud] Fix extraction of non-JPG images
Default image qualities were removed in image URLs with the jpg extension,
causing the addition of the image suffix to full non-JPG images URLs and so to
invalid image URLs.

Only the image quality name with its leading "-" character and the "."
character after the name is now removed and replaced by a string format
replaced itself with the image quality name for each quality.

As the image suffixes do not contain the image extension, the name of image
qualities lists has been adapted with these changes and some related comments
have been also improved.
2023-10-01 20:33:25 +02:00
AudricV ac00459c1a
Change requirement of image extensions in ImageSuffix class' Javadoc to a possibility
Some services may provide different image formats using the same suffix,
without we know what format the service provide. Enforcing an image extension
could so lead to provide invalid image URLs, like for SoundCloud PNG images
currently.

With this documentation change, it is now clear that users of this class decide
of whether they want to include image extensions in the suffix. The previous
behavior described in the Javadoc was not enforced.
2023-09-30 21:11:09 +02:00
FineFindus dd7b2d9798
feat(youtube/comments): support creator replies 2023-09-25 10:40:45 +02:00
Youssif Shaaban Alsager 917554acc4
[YouTube] Add support for ultralow audio formats (#1063) 2023-09-24 19:04:34 +02:00
Tobi 8b0068f8f4
Merge pull request #1110 from christian-2hu/chore/update-copyright
chore: Update copyright notices
2023-09-23 00:29:19 +02:00
Christian fc67d49f59 Update copyright notices
Update copyright notices to comply to GPLv3 and change NewPipe to NewPipe Extractor on some notices that were not updated.
2023-09-22 19:10:15 -03:00
Stypox 289db1178a
Merge pull request #1108 from AudricV/yt_refactor-js-usage
[YouTube] Refactor JavaScript usage and fix extraction of obfuscated signature deobfuscation function
2023-09-22 10:41:57 +02:00
AudricV 6ed22099a2
[YouTube] Update stream mocks 2023-09-21 21:59:34 +02:00
AudricV 714b141ecb
[YouTube] Catch any exception when extracting something from JavaScript's base player 2023-09-21 21:59:33 +02:00
AudricV 588c6a8422
[YouTube] Quote signature deobfuscation function name and add semicolon only where needed 2023-09-21 21:59:33 +02:00
AudricV 1fa85ec6ca
[YouTube] Add tests for signature timestamp extraction and signature deobfuscation function extraction and execution 2023-09-21 21:59:33 +02:00
AudricV a04bc320de
[YouTube] Convert signature timestamp to integer
The signature timestamp is used as a number by HTML5 clients, so it should be
used in the same way by the extractor too instead of being a string.

As the timestamp doesn't seem to exceed 5 digits, an integer is used to store
its value.
2023-09-21 21:59:32 +02:00
AudricV 7de3753a81
[YouTube] Refactor JavaScript player management API
This commit is introducing breaking changes.

For clients, everything is managed in a new class called
YoutubeJavaScriptPlayerManager:
- caching JavaScript base player code and its extracted code (functions and
variables);
- getting player signature timestamp;
- getting deobfuscated signatures of streaming URLs;
- getting streaming URLs with a throttling parameter deobfuscated, if
applicable.

The class delegates the extraction parts to external package-private classes:
- YoutubeJavaScriptExtractor, to extract and download YouTube's JavaScript base
player code: it always already present before and has been edited to mainly
remove the previous caching system and made it package-private;
- YoutubeSignatureUtils, for player signature timestamp and signature
deobfuscation function of streaming URLs, added in a recent commit;
- YoutubeThrottlingParameterUtils, which was originally
YoutubeThrottlingDecrypter, for throttling parameter of streaming URLs
deobfuscation function and checking whether this parameter is in a streaming
URL.

YoutubeJavaScriptPlayerManager caches and then runs the extracted code if it
has been executed successfully. The cache system of throttling parameters
deobfuscated values has been kept, its size can be get using the
getThrottlingParametersCacheSize method and can be cleared independently using
the clearThrottlingParametersCache method.

If an exception occurs during the extraction or the parsing of a function
property which is not related to JavaScript base player code fetching, it is
stored until caches are cleared, making subsequent failing extraction calls of
the requested function or property faster and consuming less resources, as the
result should be the same until the base player code changes.

All caches can be reset using the clearAllCaches method of
YoutubeJavaScriptPlayerManager.

Classes using JavaScript base player code and utilities directly (in the code
and its tests) have been also updated in this commit.
2023-09-21 21:59:32 +02:00
AudricV 6884d191cd
[YouTube] Add utility class around signatures and fix signature deobfuscation function extraction
The goal of this class is to decouple the extraction of signature timestamp and
signature deobfuscation function from YoutubeStreamExtractor.

The extraction of the signature deobfuscation function has been also adapted to
support the latest YouTube player versions.

This new class, YoutubeSignatureUtils, doens't store anything temporary such as
a copy of the player code, which has to be passed where required. It is not
public, as it will be used by a JavaScript player manager class in the future,
in order to handle in a better way fetching, caching and resetting cache of the
player code.
2023-09-21 21:59:26 +02:00
Tobi 3be76a6406
Merge pull request #1107 from Isira-Seneviratne/Locale_forLanguageTag
Use Locale.forLanguageTag() in tests
2023-09-18 16:48:30 +02:00
TobiGr 17790328cd Improve doc 2023-09-18 16:44:51 +02:00
Isira Seneviratne 4bc8ae7812 Use Locale.forLanguageTag() in tests 2023-09-18 08:59:13 +05:30
Tobi 90aed06a63
Merge pull request #1105 from TeamNewPipe/fix/bandcamp-streame-extractor-test
[Badcamp] Fix StreamExtractorTest
2023-09-18 01:49:04 +02:00
TobiGr cf49f4a31c [Badcamp] Fix StreamExtractorTest
The song was renamed and the URL changed
2023-09-17 23:58:07 +02:00
Tobi 7c7ceaceab
Merge pull request #1103 from TeamNewPipe/dependabot/github_actions/actions/checkout-4
Bump actions/checkout from 3 to 4
2023-09-17 22:42:01 +02:00
dependabot[bot] 72c475d944
Bump actions/checkout from 3 to 4
Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-09-05 10:00:13 +00:00
Stypox 1f08d28ae5
Merge pull request #889 from AudricV/multiple-images-support
Multiple images support
2023-08-13 11:35:11 +02:00
AudricV e8bfd20170
[MediaCCC] Apply changes in extractor tests
Also remove some public test methods modifiers.
2023-08-12 22:56:33 +02:00
AudricV 0292c4f3e8
[Bandcamp] Apply changes in extractor tests
Also remove some public test methods modifiers, add missing Test annotations on
old Junit 4 tests (and update them if needed), and use final in some places
where it was possible.

BandcampChannelExtractorTest.testLength has been removed as the test is always
true.
2023-08-12 22:56:32 +02:00
AudricV 2578f22054
[Bandcamp] Add utility test method to test images
This method, testImages(Collection<Image>), will use first the default image
collection test in DefaultTests and then will check that each image URL
contains f4.bcbits.com/img and ends with .jpg or .png.

To do so, a new non-instantiable final class has been added: BandcampTestUtils.
2023-08-12 22:56:32 +02:00
AudricV ba5315c72d
[PeerTube] Apply changes in extractor tests
Also remove some public test methods modifiers, add missing Test annotations on
old Junit 4 tests (and update them if needed), and improve some code.
2023-08-12 22:56:32 +02:00
AudricV 1d72bac53d
[SoundCloud] Apply changes in extractor tests 2023-08-12 22:56:32 +02:00
AudricV 93a210394d
[YouTube] Apply changes in extractor tests
Also remove some public test methods modifiers, add missing Test annotations on
old Junit 4 tests (and update them if needed), and use final in some places
where it was possible.
2023-08-12 22:56:31 +02:00
AudricV 2c436d428c
[YouTube] Add utility test method to test images in YoutubeTestsUtils
This method, testImages(Collection<Image>), will use first the default image
collection test in DefaultTests and then will check that each image URL
contains the string yt.

The JavaDoc of the class has been also updated to reflect the changes made in
it (it is now more general).
2023-08-12 22:56:31 +02:00
AudricV d381f3b70b
Update avatar, banners and thumbnail methods' name and apply changes in DefaultStreamExtractorTest 2023-08-12 22:56:31 +02:00
AudricV 434e885708
Add utility methods in ExtractorAsserts to check whether a collection is empty and to test image collections
Two new methods have been added in ExtractorAsserts to check if a collection is
empty:

- assertNotEmpty(String, Collection<?>), checking:
  - the non nullity of the collection;
  - its non emptiness (if that's not case, an exception will be thrown using
    the provided message).

- assertNotEmpty(Collection<?>), calling assertNotEmpty(String, Collection<?>)
  with null as the value of the string argument.

A new one has been added to this assertion class to check the contrary:
assertEmpty(Collection<?>), checking emptiness of the collection only if it is
not null.

Three new methods have been added in ExtractorAsserts as utility test methods
for image collections:

- assertContainsImageUrlInImageCollection(String, Collection<Image>), checking
that:
  - the provided URL and image collection are not null;
  - the image collection contains at least one image which has the provided
    string value as its URL (which is a string) property.

- assertContainsOnlyEquivalentImages(Collection<Image>, Collection<Image>),
  checking that:
  - both collections are not null;
  - they have the same size;
  - each image of the first collection has its equivalent in the second one.
    This means that the properties of an image in the first collection must be
    equal in an image of the second one.

- assertNotOnlyContainsEquivalentImages(Collection<Image>, Collection<Image>),
  checking that:
  - both collections are not null;
  - one of the following conditions is met:
    - they have different sizes;
    - an image of the first collection has not its equivalent in the second one.
      This means that the properties of an image in the first collection must
      be not equal in an image of the second one.

These methods will be used by services extractors tests (and default ones) to
test image collections.
2023-08-12 22:56:31 +02:00
AudricV 5158472852
Apply changes in DefaultTests and add utility method to test image lists
This new method, defaultTestImageList(List<Image), will check that the image
list is not null.

For each image, it will test that its URL is secure and its height and width
are more than or equal to their relevant unknown constants in the Image class
(HEIGHT_UNKNOWN and WIDTH_UNKNOWN).
2023-08-12 22:56:31 +02:00
AudricV 70fb3aa38e
Update BaseExtractorTests image methods' name
Also suppress unused warnings in BaseStreamExtractorTest, like it is done on
other BaseExtractorTests interfaces.
2023-08-12 22:56:30 +02:00
AudricV e16d521b7b
[MediaCCC] Apply changes in Extractors
Also remove usage of the conference logo as the banner of a conference, as it
is a logo and not a banner.
2023-08-12 22:56:30 +02:00
AudricV 306068a63b
[MediaCCC] Apply changes in InfoItemExtractors 2023-08-12 22:56:30 +02:00
AudricV 2f40861428
[MediaCCC] Add utility methods to get image lists from conference logos and streams
These three new methods, added in MediaCCCParsingHelper,
getImageListFromImageUrl(String), getThumbnailsFromStreamItem(JsonObject) and
getThumbnailsFromLiveStreamItem(JsonObject) (the last two are based on a common
method, getThumbnailsFromObject(JsonObject, String, String)), return an empty
list if the case no image URL could be extracted.

Images returned have their height and width unknown and a resolution level
depending on the image key of the JSON API response.
2023-08-12 22:56:30 +02:00
AudricV 71cda03c4c
[Bandcamp] Apply changes in Extractors 2023-08-12 22:56:29 +02:00
AudricV 7e01eaac33
[Bandcamp] Apply changes in InfoItemExtractors 2023-08-12 22:56:29 +02:00
AudricV 4b80d737a4
[Bandcamp] Add utility methods to get multiple images
Bandcamp images work with image IDs, which provide different resolutions.

Images on Bandcamp are not always squares, and some IDs respect aspect ratios
where some others not.

The extractor will only use the ones which preserve aspect ratio and will not
provide original images, for performance and size purposes.

Because of this aspect ratio preservation constraint, only one dimension will
be known at a time.

The image IDs with their respective dimension used are:

- 10: 1200w;
- 101: 90h;
- 170: 422h;
- 171: 646h;
- 20: 1024w;
- 200: 420h;
- 201: 280h;
- 202: 140h;
- 204: 360h;
- 205: 240h;
- 206: 180h;
- 207: 120h;
- 43: 100h;
- 44: 200h.

(Where w represents the width of the image and h the height of the image)

Note that these dimensions are theoretical because if the image size is less
than the dimensions of the image ID, it will be not upscaled but kept to its
original size.

All these resolutions are stored in a private static list of ThumbnailSuffixes
in BandcampExtractorHelper, in which the methods to get mutliple images have
been added:

- getImagesFromImageUrl(String): public method to get images from an image URL;
- getImagesFromImageId(long, boolean): public method to get images from an
  image ID;
- getImagesFromImageBaseUrl(String): private utility method to get images from
  the static list of ThumbnailSuffixes from a given image base URL, containing
  the path to the image, a "a" letter if it comes from an album, its ID and an
  underscore.

Some existing methods have been also edited:

- the documentation of getImageUrl(long, boolean) has been changed to reflect
  the Bandcamp images findings;
- getThumbnailUrlFromSearchResult has been renamed to
  getImagesFromSearchResult, and a documentation has been added to this method.

The method replaceHttpWithHttps of the Utils class has been also used in
BandcampExtractorHelper instead of doing manually what the method does.
2023-08-12 22:56:29 +02:00
AudricV 4e6fb368bc
[PeerTube] Apply changes in Extractors and remove usages of default avatar picture
The default avatar picture was used when no profile picture was found, but it
was removed and split in multiple images.

Thumbnails' size is not known, as this data is not provided by the API.
2023-08-12 22:56:29 +02:00
AudricV 0a6011a50e
[PeerTube] Apply changes in InfoItemExtractors
Also lower the visibility of attributes of channels and playlists InfoItems to
private.
2023-08-12 22:56:29 +02:00
AudricV 6f8331524b
[PeerTube] Add utility method to get thumbnails of playlists and videos
This method, getThumbnailsFromPlaylistOrVideoItem, has been added in
PeertubeParsingHelper and returns the two image variants for playlists and
videos.
2023-08-12 22:56:28 +02:00
AudricV 81c0d80a54
[PeerTube] Add utility methods to get avatars and banners of accounts and channels
Four new static methods have been added in PeertubeParsingHelper to do so:
- two public methods to get the corresponding image type:
  getAvatarsFromOwnerAccountOrVideoChannelObject(String, JsonObject) and
  getBannersFromAccountOrVideoChannelObject(String, JsonObject);
- two private methods as helper methods: getImagesFromAvatarsOrBanners(String,
  JsonObject, String, String) and getImagesFromAvatarOrBannerArray(String,
  JsonArray).
2023-08-12 22:56:28 +02:00
AudricV 31da5beb51
[SoundCloud] Apply changes in Extractors 2023-08-12 22:56:28 +02:00
AudricV a3a74cd566
[SoundCloud] Apply changes in InfoItemExtractors and return track user avatars as uploader avatars in SoundcloudStreamInfoItemExtractor 2023-08-12 22:56:28 +02:00
AudricV 7f818217d2
[SoundCloud] Add utility methods to get images from track JSON objects and image URLs
These new public and static methods, added in SoundcloudParsingHelper,
getAllImagesFromArtworkOrAvatarUrl(String) and
getAllImagesFromVisualUrl(String) (which call a common private method,
getAllImagesFromImageUrlReturned(String, List<ImageSuffix>, List<Image>)),
return an unmodifiable list of JPEG images containing almost every image
resolution provided by SoundCloud except the original size and the tiny
resolution (for artworks and avatars, as the image size is 20x20 for artworks
and 18x18 for avatars, so very close to or equal to the t20x20 resolution):

- for artworks and avatars:
  - mini: 16x16;
  - t20x20: 20x20;
  - small: 32x32;
  - badge: 47x47;
  - t50x50: 50x50;
  - t60x60: 60x60;
  - t67x67: 67x67;
  - large: 100x100;
  - t120x120: 120x120;
  - t200x200: 200x200;
  - t240x240: 240x240;
  - t250x250: 250x250;
  - t300x300: 300x300;
  - t500x500: 500x500.

- for visuals/user banners:
  - t1240x260: 1240x260;
  - t2480x520: 2480x520.

Duplicated code in two methods of SoundcloudParsingHelper
(getUsersFromApi(ChannelInfoItemsCollector, String) and
getStreamsFromApi(StreamInfoItemsCollector, String, boolean)) has been merged
into one common private method, getNextPageUrlFromResponseObject(JsonObject).
2023-08-12 22:56:28 +02:00
AudricV 266cd1f76b
[YouTube] Apply changes in YoutubeMusicSearchExtractor and split its InfoItemExtractors into separate classes
Splitting YoutubeMusicSearchExtractor's InfoItemExtractors into separate
classes (YoutubeMusicSongOrVideoInfoItemExtractor,
YoutubeMusicAlbumOrPlaylistInfoItemExtractor and
YoutubeMusicArtistInfoItemExtractor) allows to simplify
YoutubeMusicSearchExtractor,improves reading and applying changes to InfoItems
(no more losing at least quarter of a line due to indentations).

These InfoItems, in which the image changes have been applied, don't extend the
YouTube ones anymore, as most methods were overridden and the few ones that are
not don't apply in YouTube Music items responses, so it was useless to extend
them.

The code of YoutubeMusicSearchExtractor have been also improved a bit.
2023-08-12 22:56:27 +02:00
AudricV c1981ed54f
[YouTube] Apply changes in Extractors except YoutubeMusicSearchExtractor
Also improve a bit some code related to the changes.
2023-08-12 22:56:27 +02:00
AudricV 4cc99f9ce1
[YouTube] Apply changes in InfoItemExtractors except YouTube Music ones 2023-08-12 22:56:27 +02:00
AudricV adfad086ac
[YouTube] Add utility methods to get images from InfoItems and thumbnails arrays
Unmodifiable lists of Images are returned, parsed from a given YouTube
"thumbnails" JSON array.

These methods will be used in all YouTube extractors and InfoItems, as the
structures between content types (videos, channels, playlists, ...) are common.
2023-08-12 22:56:27 +02:00
AudricV d56b880cae
Replace avatar and thumbnail URLs attributes and methods to List<Image> in Infos 2023-08-12 22:56:26 +02:00
AudricV 9d8098576e
Replace avatar and thumbnail URLs attributes and methods to List<Image> in Extractors 2023-08-12 22:56:26 +02:00
AudricV 0f4a5a8184
Replace avatar and thumbnail URLs attributes and methods to List<Image> in InfoItemsCollectors 2023-08-12 22:56:26 +02:00
AudricV ca1d4a6fa4
Replace avatar and thumbnail URLs attributes and methods to List<Image> in InfoItemExtractors 2023-08-12 22:56:26 +02:00
AudricV 2f3ee8a3f2
Replace avatar and thumbnail URLs attributes and methods to List<Image> in InfoItems 2023-08-12 22:56:25 +02:00
AudricV 78ce65769f
Add an ImageSuffix class to the extractor
The goal of this utility class is to simply store suffixes which need to be
appended to image URLs, in order to get images at the suffix resolution.

This class contains four properties: the suffix (as a string), the height,
the width (as integers) and the estimated resolution level of the image
corresponding to the one represented by the suffix.
2023-08-12 22:56:25 +02:00
AudricV d85454186a
Add an Image class to the extractor
Objects of this serializable class contains four properties: a URL (as a
string), a width, a height (represented as integers) and an estimated
resolution level, which can be constructed from a given height.

Possible resolution levels are:
- UNKNOWN: for unknown heights or heights <= 0;
- LOW: for heights > 0 & < 175;
- MEDIUM: for heights >= 175 & < 720;
- HIGH: for heights >= 720.

Getters of these properties are available and the constructor needs these four
properties.
2023-08-12 22:56:25 +02:00
Stypox 7294675aea
Merge pull request #1093 from AudricV/yt_support-shorts-ui-playlists
[YouTube] Support Shorts UI in playlists
2023-08-12 11:11:36 +02:00
Stypox 93a90b816d
Merge pull request #1094 from AudricV/yt_support-more-channel-headers
[YouTube] Support more channel headers
2023-08-12 11:08:30 +02:00
Stypox 44b664af15
[YouTube] Simplify Optional chains in channel 2023-08-12 11:02:51 +02:00
AudricV 2f7bfd3e7f
[YouTube] Add mocks of interactiveTabbedHeaderRenderer channel header test 2023-08-08 19:12:29 +02:00
AudricV b147904571
[YouTube] Add test for interactiveTabbedHeaderRenderer channel header
This test uses the Minecraft game topic channel.
2023-08-08 19:12:28 +02:00
AudricV 1852031a0b
[YouTube] Support pageHeaderRenderer and interactiveTabbedHeaderRenderer channel headers
The addition of this support required to turn the isCarouselHeader boolean into
an enum containing all supported channel headers named HeaderType.

Also assert that the page has been fetched where needed to avoid
NullPointerExceptions when the channel page has been not fetched and remove the
getChannelHeaderJson method in YoutubeChannelExtractor, method for which its
code has been moved to its sole usage after the new headers support changes.
2023-08-08 19:12:27 +02:00
AudricV 698c710685
Do not require knowledge of uploader in default StreamInfoItems tests
This change is required as some services can return no uploader info, such as
YouTube for playlists with a Shorts UI.
2023-08-07 19:43:15 +02:00
AudricV 8237052ef5
Fix wrong assertion in assertNotEmpty
The non-null assertion was made on the exception message instead of the string
to check, causing a NullPointerException if the string to check was null.
2023-08-07 19:43:09 +02:00
AudricV 162c261577
[YouTube] Add mocks of the playlist with Shorts UI test 2023-08-07 19:07:53 +02:00
AudricV e2f4ee47b9
[YouTube] Add a playlist with Shorts UI test
The system Shorts videos uploads playlist of the YouTube official channel has
been chosen for this test.
2023-08-07 19:06:09 +02:00
AudricV e6f371fb94
[YouTube] Support Shorts UI in playlists
Also remove an outdated A/B test comment.
2023-08-07 19:01:08 +02:00
Stypox 7936987955
Merge pull request #1092 from Stypox/channel-tabs-improvements
Channel tabs code improvements
2023-08-06 21:44:10 +02:00
Stypox 6d2227111f
[YouTube] Assert that videos tab is ready after channel fetching 2023-08-06 21:14:57 +02:00
Stypox ee625c325c
Inherit from DefaultListExtractorTest in channel tab tests 2023-08-06 21:14:56 +02:00
Stypox 276c293889
Rename assertTabsContain 2023-08-06 21:14:56 +02:00
Stypox 9d3761a371
[YouTube] Directly use playlist collector in channel tabs wrapper
Note that this introduces a "Raw use of parameterized class 'InfoItemsPage'" warning, but it can be ignored since the type missing would be <InfoItem>, and StreamInfoItem extends InfoItem
2023-08-06 21:13:25 +02:00
Stypox 95a3cc0a17
Merge pull request #1083 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.10.0
Bump org.junit:junit-bom from 5.9.3 to 5.10.0
2023-08-06 18:58:27 +02:00
Stypox e34b4f1978
[YouTube] Avoid using Consumer 2023-08-06 13:02:31 +02:00
Stypox ef67c7cd74
[YouTube] Simplify usage of channel header json 2023-08-06 13:02:31 +02:00
Stypox a104cf3227
[YouTube] Fix docs in channel helper 2023-08-06 13:02:31 +02:00
Stypox bb47f05f89
Merge pull request #1091 from Stypox/update-mocks
[YouTube] Update stream mocks after #1087
2023-08-06 12:58:24 +02:00
Stypox 468bcc045d
[YouTube] Update mocks after #1087 2023-08-06 12:33:04 +02:00
Stypox 35f3a4ad01
Merge pull request #1082 from AudricV/channel-tabs-and-tags-support
Add support for channel tabs and channel tags
2023-08-06 12:21:42 +02:00
AudricV e7d64099a7
[YouTube] Update channel mocks and add channel tabs mocks 2023-08-06 12:15:06 +02:00
AudricV 684101c47d
[YouTube] Implement age-restricted channels support, link handlers and channels tabs and tags changes on tests
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV eaf2600ce0
[SoundCloud] Implement link handlers and channels tabs and tags changes on tests
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV 0ee2072de5
[PeerTube] Implement link handlers and channels tabs and tags changes on tests
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV d3801dd0e9
[MediaCCC] Implement link handlers and channels tabs and tags changes on tests
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV 8baec04611
[Bandcamp] Implement link handlers and channels tabs and tags changes on tests
Tests in BandcampChannelExtractorTest and BandcampChannelLinkHandlerFactoryTest
have been also fixed.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV e0ba29cd19
Add utility method to assert that given channel tabs are in the ones returned by a channel extractor
Only the first content filter of the ListLinkHandler instances provided is
used when collecting all channel tabs of the ListLinkHandler list, as channel
tabs implementations only use one content filter per ListLinkHandler instance.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV 18846baba7
Add tabs and tags methods in tests interfaces and annotate all methods with the Test JUnit annotation
These changes should help to detect tests as tests, when running a subset of
tests or all tests.
They should be also implemented in these interfaces' implementations (new and
existing ones).

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
ThetaDev c70a0e3543
Add a test for textual durations parsing using TimeAgoParser's patterns 2023-08-06 12:15:06 +02:00
AudricV 7366eab156
[YouTube] Add support for channel tabs and tags and age-restricted channels
Support of tags and videos, shorts, live, playlists and channels tabs has been
added for non-age restricted channels.

Age-restricted channels are now also supported and always returned the videos,
shorts and live tabs, accessible using system playlists. These tabs are the
only ones which can be accessed using YouTube's desktop website without being
logged-in.

The videos channel tab parameter has been updated to the one used by the
desktop website and when a channel extraction is fetched, this tab is returned
in the list of tabs as a cached one in the corresponding link handler.

Visitor data support per request has been added, as a valid visitor data is
required to fetch continuations with contents on the shorts tab. It is only
used in this case to enhance privacy.

A dedicated shorts UI elements (reelItemRenderers) extractor has been added,
YoutubeReelInfoItemExtractor. These elements do not provide the exact view
count, any uploader info (name, URL, avatar, verified status) and the upload
date.

All service's LinkHandlers are now using the singleton pattern and some code
has been also improved on the files changed.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:15:04 +02:00
AudricV 4586067934
Add utility method to parse textual durations using TimeAgoParser's patterns
This is required to parse duration of YouTube's reelItemRenderers, returned
only inside accessibility data.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:13:33 +02:00
AudricV d4bfe791ee
[SoundCloud] Add tabs support for users
Support of tracks, playlists and albums has been added for users.

Also add the declaration of the UnsupportedOperationException exception to the
service's LinkHandlers.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:13:32 +02:00
AudricV 6f7d1f079f
[Bandcamp] Add tabs support for artists
Support of tracks and albums has been added for artists.

Also use the singleton pattern and add the declaration of the
UnsupportedOperationException exception to the service's LinkHandlers and
improved some code in the files changed.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:12:19 +02:00
AudricV 1e8474b22d
[PeerTube] Add tabs support for accounts and video channels
Support of channels and videos has been added for accounts and support of
videos and playlists has been added for video channels.

The following changes have been also done:
- collectStreamsFrom method in PeertubeParsingHelper has been renamed to
collectItemsFrom;
- PeertubeChannelInfoItemExtractor.getStreamCount method has been fixed due to
ChannelExtractor's new inheritance;
- the declaration of the UnsupportedOperationException exception thrown has
been added to the service's LinkHandlers;
- a channel tab LinkHandlerFactory has been added,
PeertubeChannelTabLinkHandlerFactory;
- all service's LinkHandlers are now using properly the singleton pattern.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:12:15 +02:00
AudricV 652c2c8408
Add a ListLinkHandler which can be used to be returned from ChannelInfo.getTabs() when a specific tab's data has already been fetched
This new ListLinkHandler, ReadyChannelTabListLinkHandler, should help saving
clients data, energy and time by helping to reduce duplicate requests.

Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:11:12 +02:00
AudricV de823a6b68
Add an UnsupportedTabException exception class
This class makes easier for LinkHandlerFactory implementations to declare an
UnsupportedOperationException.
2023-08-06 12:11:12 +02:00
AudricV 76fb9dcdd7
Add UnsupportedOperationException to exceptions which can be thrown by getId and getUrl methods of LinkHandlerFactory and its base implementations
This change advertise to clients that channel tabs' link handler factories can
return an UnsupportedOperationException when a tab provided to them is
unsupported.
2023-08-06 12:11:12 +02:00
AudricV 946eb9bd91
Add structure of channel tags
Tags' getters and/or setters have been added in ChannelExtractor and
ChannelInfo to do so.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:11:12 +02:00
AudricV 356a888d6c
Add structure of channel tabs
This commit introduces the following breaking changes:

- Three new classes have been added:
  - ChannelTabExtractor, class extending ListExtractor<InfoItem>, which
  extracts InfoItems from a channel tab;
  - ChannelTabInfo extending ListInfo<InfoItem>, which extracts InfoItems from
  a ChannelTabExtractor and returns them as a ChannelTabInfo;
  - ChannelTabs, an immutable class containing all supported channel tabs.
- StreamingService implementations must implement new methods returning a
channel tab LinkHandlerFactory (getChannelTabsLHFactory) and a
ChannelTabExtractor (getChannelTabExtractor);
- ChannelExtractor inherits Extractor instead of ListExtractor<StreamInfoItem>
and ChannelInfo inherits Info instead of ListInfo<StreamInfoItem>;
- ChannelExtractor and ChannelInfo have now getters and/or setters of tabs.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:11:11 +02:00
Stypox 3faaf4301c
Merge pull request #1087 from AudricV/yt_js-extractor-improvements-and-fixes
[YouTube] Improve and fix YoutubeJavaScriptExtractor
2023-08-06 12:01:00 +02:00
Stypox 8fb6ba36fa
Merge pull request #1081 from TeamNewPipe/fix/sc/search-next-page
[SoundCloud] Detect whether there are any more search results
2023-08-06 11:49:35 +02:00
Stypox 2947257111
[SoundCloud] Properly calculate if results have finished 2023-08-06 11:38:22 +02:00
Stypox 485bfbca9d
[SoundCloud] Move try-catch inside getOffsetFromUrl 2023-08-06 11:35:37 +02:00
Stypox 7c70fef197
Merge pull request #1089 from TeamNewPipe/ccc
[media.ccc.de] Only extract kiosk live stream rooms if they are streaming
2023-08-06 10:12:04 +02:00
Stypox de0a9bb797
Merge pull request #1088 from FireMasterK/pseudo-rng
Replace cryptographically secure random with regular random
2023-08-05 11:37:49 +02:00
TobiGr 340095515d Make Kiosk IDs accessible if possible 2023-08-05 03:18:40 +02:00
TobiGr fe27d6a0ec [media.ccc.de] Only extract live streams if the conference is streaming 2023-08-05 01:53:43 +02:00
Kavin 25082d78b0
Replace SecureRandom with Random 2023-08-03 23:00:02 +01:00
TobiGr aa6c17dc77 [SoundCloud] Deduplicate some code 2023-08-03 14:41:30 +02:00
TobiGr 2fb9922a15 [SoundCloud] Detect whether there are any more search results
Add test for this edge case.
2023-08-03 14:37:13 +02:00
AudricV a3d160edab
[YouTube] Improve and fix YoutubeJavaScriptExtractor
- Enhance documentation;
- Fix the regular expression fallback on HTML embed watch page;
- Use HTML scripts tag search first instead of the regular expression approach,
now used as a last resort;
- Compile regular expressions only once, in order to improve the performance of
subsequent extraction calls when clearing the cache;
- Provide original exceptions when fetching or parsing pages on which the base
JavaScript's player could be found failed, allowing clients to detect network
errors when they are the cause of the failures for instance;
- Remove delegate method which was not taking a video ID and hardcoding one, as
we can provide the video ID in all cases or do not provide a video ID at worse;
- Rename and make extraction methods package-private, as they are not intended
to be used publicly.

These breaking internal changes have been applied where needed, in
YoutubeJavaScriptExtractorTest and YoutubeStreamExtractor (in which an unneeded
initStsFromPlayerJsIfNeeded call have been removed).
2023-08-02 23:05:08 +02:00
Stypox 5492343b8e
Pre-release v0.22.7 2023-08-02 20:02:29 +02:00
Tobi 9a59afbcf5
Merge pull request #1086 from AudricV/yt_no-exception-channels-without-banner
[YouTube] Don't throw an exception when there is no banner available on a channel
2023-08-02 18:56:26 +02:00
AudricV bb1ab166bf
[YouTube] Test that no banner is returned for carouselHeaderRenders 2023-08-01 22:19:43 +02:00
AudricV f1fa84b4e3
[YouTube] Don't throw an exception when there is no banner available on a channel
Channels may not have a banner, so no exception should be thrown if no banner
is found.
2023-08-01 12:40:20 +02:00
Tobi 39a911db9f
Merge pull request #1084 from AudricV/yt_android-403s-workaround-and-streams-tests-fixes
[YouTube] Workaround again 403 HTTP issues on the ANDROID InnerTube client and fix stream tests
2023-07-31 23:51:10 +02:00
dependabot[bot] bda3a3fc5d
Bump org.junit:junit-bom from 5.9.3 to 5.10.0
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.9.3 to 5.10.0.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.9.3...r5.10.0)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-07-24 09:53:05 +00:00
AudricV 522c78160f
[YouTube] Update stream tests mocks 2023-07-23 19:36:28 +02:00
AudricV 7528eb2bd9
[YouTube] Fix stream tests failures
- Fix testCheckAudioStreams test of
YoutubeStreamExtractorDefaultTest.AudioTrackLanguage test class, by updating
the excepted audio track name test to use the updated English audio track name
(audio track type info has been added on the video tested);
- Fix YoutubeStreamExtractorDefaultTest.PublicBroadcasterTest test class by
using a different video from a French and German public broadcast channel, as
the channel Dinge Erklärt – Kurzgesagt is not affiliated with a public
broadcast channel anymore;
- Fix YoutubeStreamExtractorLivestreamTest test class, by updating the excepted
name of the livestream to the current one.
2023-07-23 19:19:02 +02:00
AudricV 164c8e3abb
[YouTube] Workaround again 403 HTTP issues on the Android client by using new player parameters
These parameters are the only ones currently known to bypass 403 HTTP issues
related to failure of passing Android client integrity checks, as the ones of
stories (and the base of the shorts ones) do not work anymore, which may be
related to end of this format on the service.
2023-07-22 20:22:16 +02:00
Tobi 8ebbe1b7e1
Merge pull request #706 from FireMasterK/av1-itags
Add support for AV1 itags.
2023-07-22 13:31:45 +02:00
FireMasterK 6db0d116fe Add support for AV1 itags. 2023-07-22 13:23:44 +02:00
Tobi 986a3f7747
Merge pull request #1080 from TeamNewPipe/proguard-rules
Add proguard rules to README
2023-07-07 12:52:10 +02:00
Tobi d6132a9ed9
Merge pull request #1076 from AudricV/yt-fix-throttling-parameter-regex
[YouTube] Support multiple declarations for throttling parameter function name array
2023-07-07 12:51:26 +02:00
Tobi 87eb29a25d Add proguard rules to README 2023-07-05 08:25:23 +02:00
AudricV 4e22c5ee87
[YouTube] Support multiple declarations for throttling parameter function name array
Also moved the corresponding regex parts in static constants for easier future
modifications
2023-06-26 15:25:53 +02:00
Kavin d961d349c3
[YouTube] Check whether player responses are valid for all InnerTube clients used (#1070)
Co-authored-by: Audric V <74829229+AudricV@users.noreply.github.com>
2023-06-18 21:54:52 +02:00
ThetaDev ad97f08048
[YouTube] Fix parsing short relative date formats (English only) (#1068) 2023-06-18 21:41:29 +02:00
Tobi d294ccb433
Merge pull request #1071 from TeamNewPipe/feat/ServiceList
Init services at the correct place
2023-06-17 20:50:24 +02:00
Tobi 5d112aa772
Merge pull request #1074 from TeamNewPipe/imp/media-formats
Add more audio media formats and MediaFormat.getAllForMimeType(mimeType)
2023-06-17 20:49:38 +02:00
TobiGr 0e01d90562 Do not init services while creating services array 2023-06-17 18:08:09 +02:00
TobiGr 5809904cf7 Add annotations to MediaFormat 2023-06-17 18:01:54 +02:00
TobiGr 53dfd871e2 Add more audio media formats and MediaFormat.getAllForMimeType(mimeType) 2023-06-17 18:01:29 +02:00
Stypox 5eb886243f
Merge pull request #1072 from TeamNewPipe/fix/bandcamp_test
Fix BandcampRadioStreamExtractorTest.testGetAudioStreams()
2023-06-09 17:35:35 +02:00
TobiGr 19ce06fe00 Fix BandcampRadioStreamExtractorTest.testGetAudioStreams() 2023-06-06 23:07:56 +02:00
Audric V 533121fb81
Merge pull request #1045 from Theta-Dev/fix/trending-video-tab
[YouTube] Extract trends from A/B tested "Videos" tab and fix extraction of trends name from A/B tested new title design
2023-05-19 11:22:49 +02:00
Audric V 92a0024424
Merge pull request #1052 from TeamNewPipe/peertube/fix/nested-comment-replies
[PeerTube] Fix multi level comment replies
2023-05-18 18:49:06 +02:00
Tobi 3c036a9c03
Merge pull request #1061 from ChunkyProgrammer/develop/extract-yt-playlist-descrption
Extract playlists description
2023-05-17 22:43:17 +02:00
TobiGr c70bb83801 [Bandcamp] Implement PlaylistExtractor.getDescsription() 2023-05-15 15:23:03 +02:00
TobiGr ca0ce00753 Add PlaylistInfoItem.getDescription() and PlaylistInfoItemExtractor.getDescription()
[PeerTube] Implement the corresponding extractor method.
TODO: add tests
2023-05-12 01:43:59 +02:00
TobiGr b218bf69bd Implement PlaylistInfo.getDescription()
Implement PlaylistExtractor.getDescription() for PeerTube and SoundCloud.
Anotate   PlaylistExtractor.getDescription() as Nonnull
2023-05-12 00:44:10 +02:00
chunky programmer 81f29116ba switch from string to Description object 2023-05-11 00:36:57 -04:00
chunky programmer e147867d41 Add tests 2023-05-11 00:00:34 -04:00
chunky programmer 5ab6cd7420 Extract YouTube playlist description 2023-05-11 00:00:22 -04:00
TobiGr d358ba1c41 Improve PeertubeCommentsInfoItemExtractor constructor 2023-05-07 22:55:26 +02:00
TobiGr aff3e795f8 [PeerTube] Fix multi level comment replies 2023-05-07 22:49:14 +02:00
ThetaDev 3673d4ae01
fix: YouTube trending name extraction 2023-05-03 21:16:35 +02:00
ThetaDev 0addb98cd7
tests: update mocks 2023-05-03 21:16:35 +02:00
ThetaDev 24eba62305
fix: extract YouTube trends from new "Videos" tab 2023-05-03 21:16:23 +02:00
Kavin a9ca5c49e4
Merge pull request #1056 from AudricV/yt-improve-search-suggestions-extraction
[YouTube] Switch to new search suggestion domain and improve error handling
2023-05-02 20:17:48 +01:00
Stypox 00408db959
Merge pull request #1058 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.16.1
Bump org.jsoup:jsoup from 1.15.4 to 1.16.1
2023-05-01 19:59:02 +02:00
dependabot[bot] 108f8a7a17
Bump org.jsoup:jsoup from 1.15.4 to 1.16.1
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup) from 1.15.4 to 1.16.1.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.4...jsoup-1.16.1)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-05-01 09:58:32 +00:00
Tobi 11565db17f
Merge pull request #1055 from AudricV/peertube-no-description-channels
[PeerTube] Don't return "No description" when there is no description for a channel or an account
2023-04-30 23:38:10 +02:00
AudricV cf6040ddb3
Update YoutubeSuggestionExtractorTest mocks 2023-04-30 19:53:44 +02:00
AudricV 593122342f
[YouTube] Improve YoutubeSuggestionExtractorTest
- Remove useless concatenation on the downloader path;
- Remove unneeded public test modifier;
- Update license header;
- Specify the service class tested instead of the generic class.
2023-04-30 19:53:43 +02:00
AudricV e923fca440
[YouTube] Switch to new search suggestion domain and improve error handling
- Switch to the new domain used by YouTube for search suggestions,
suggestqueries-clients6.youtube.com, and add the xhr query parameter with the
t value, to allow getting responses without requiring trim;
- Use the Java 8 Stream API to collect search suggestions and improve invalid
response detection by checking whether the content type of the response
returned is JSON;
- Move the licence header at the top of the file.
2023-04-30 19:53:42 +02:00
AudricV 945165a3c0
[PeerTube] Don't return "No description" when there is no description for a channel or an account
When a description is missing, no description should be returned, even the ones
indicating there is no description. This behavior is represented by a null
return instead.

Also update PeertubeAccountExtractorTest to reflect these changes.
2023-04-30 18:41:38 +02:00
Stypox d61dc27406
Merge pull request #1054 from TeamNewPipe/fix-ci-variable
Make sure CI is not vulnerable to bash word splitting
2023-04-27 18:52:07 +02:00
Stypox 51cba5a287
Make sure CI is not vulnerable to bash word splitting
See https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-an-intermediate-environment-variable
2023-04-26 18:36:25 +02:00
Stypox f819c2fe34
Merge pull request #1053 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.3
Bump org.junit:junit-bom from 5.9.2 to 5.9.3
2023-04-26 15:42:19 +02:00
dependabot[bot] 4b92c5a34d
Bump org.junit:junit-bom from 5.9.2 to 5.9.3
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.9.2 to 5.9.3.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.9.2...r5.9.3)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-04-26 09:57:57 +00:00
Stypox 2deb023da4
Merge pull request #1050 from Theta-Dev/fix/channel-carousel-header
[YouTube] Add support for CarouselHeaderRenderer
2023-04-25 15:17:31 +02:00
ThetaDev 4aada7f91b refactor: rename carousel header channel test 2023-04-21 22:48:37 +02:00
Tobi b2b426048f
Merge pull request #1051 from Theta-Dev/fix/ytm-client-version-rg
[YouTube] fix: set musicClientVersion regex capture group
2023-04-16 23:48:19 +02:00
ThetaDev 47aa9fed40 fix: set musicClientVersion regex capture group 2023-04-16 19:25:05 +02:00
ThetaDev 20370395c5 fix: add support for CarouselHeaderRenderer 2023-04-16 17:40:13 +02:00
Stypox 7dba6e3891
Merge pull request #1033 from petlyh/bandcamp-paywalled-content
[Bandcamp] Handle paywalled tracks
2023-04-12 13:04:26 +02:00
petlyh e6aad117e7
[Bandcamp] Throw PaidContentException on paywalled albums 2023-04-03 19:27:09 +02:00
TobiGr 8495ad619e Bump version to 0.22.6 2023-04-02 22:35:02 +02:00
fynngodau 69705138e4
[Bandcamp] Fix extraction of related playlist items URL (#1047)
Small change in HTML structure
2023-04-02 22:24:29 +02:00
Björn Sigurbergsson 1b6fe5edd6
[YouTube] Fix ParsingException when comments are unavailable in a video (#1040)
Co-authored-by: bjs <bjs@elect-it.com>
Co-authored-by: Audric V. <74829229+AudricV@users.noreply.github.com>
Co-authored-by: Kavin <20838718+FireMasterK@users.noreply.github.com>
2023-03-30 19:58:06 +02:00
ThetaDev 8d1303e18f
Add track types to audio streams (#1041) 2023-03-28 00:02:20 +02:00
Tobi 44ae139d33
Merge pull request #1034 from AudricV/peertube-fix-commentsinfo-test
[PeerTube] Fix comments info test
2023-03-20 19:30:35 +01:00
Tobi d5e9df5381
Merge pull request #1038 from TeamNewPipe/docs-jdk-11
Update JDK to 11 in JDoc workflow
2023-03-20 19:25:21 +01:00
Tobi 0486ad7a08
Update JDK to 11 in JDoc workflow 2023-03-06 18:45:11 +01:00
AudricV 80a6fc2c63
[PeerTube] Fix testGetCommentsFromCommentsInfo test of PeertubeCommentsExtractorTest.Default
The tested comment has been removed, so it couldn't be found in the comments
list.

This comment has been replaced by a new one from the current comments of the
video.

Also, in the parent class PeertubeCommentsExtractorTest, final has been used as
much as possible and for-each loops of lists have been replaced by their
forEach method or the Stream API, in order to simplify code.
2023-03-04 16:49:10 +01:00
petlyh 5a9b6ed2e3
[Bandcamp] Support loading additional comments (#1030) 2023-03-04 14:01:06 +01:00
Stypox 6bdd698c25
Merge pull request #1026 from AudricV/audio-streams-descriptive-and-locale-properties
Add descriptive and locale properties to audio streams
2023-03-01 11:15:46 +01:00
Stypox 19e4b216c9
Merge pull request #1032 from AudricV/yt_fix-comments-hashtags-links-extraction
[YouTube] Fix hashtags links extraction and escape HTML links
2023-03-01 10:47:37 +01:00
Stypox b1298490c0
Merge pull request #1029 from AudricV/yt_fix-no-views-extraction-playlist-items
[YouTube] Fix partial non-extraction of "No views" string in stream items
2023-03-01 10:46:52 +01:00
petlyh 9dc1832733
[Bandcamp] Handle paywalled tracks 2023-02-28 17:51:30 +01:00
fynngodau 3fdb6ee476
Merge pull request #1031 from petlyh/bandcamp-fix-radio-comments
[Bandcamp] Show comments as disabled on radio streams
2023-02-27 20:50:12 +01:00
AudricV bd79b921e8
[YouTube] Refactor the code to get stream items' view count
This refactoring avoids code duplication as much as possible.
2023-02-27 10:25:46 +01:00
AudricV 51f9b39953
[YouTube] Fix partial non-extraction of no views string in stream items
As the "No views" string is returned in the case there is no view on a video, a
number cannot be parsed in this case, so -1 was returned.

This string is now detected in all methods to get the view count of a stream.
2023-02-27 10:18:45 +01:00
AudricV 95b3f5e391
[MediaCCC] Test audio language property extraction 2023-02-26 19:06:18 +01:00
AudricV 30a0f8c510
[MediaCCC] Extract audio language property for single language audio tracks 2023-02-26 19:06:18 +01:00
AudricV 7f0269c4c7
[YouTube] Edit YoutubeStreamExtractorDefaultTest.AudioTrackLanguage to test audio locale property
The Hindi audio track language presence test has been changed from audio track
label to audio locale.
2023-02-26 19:06:18 +01:00
AudricV 034f82dae7
[YouTube] Test language and descriptive audio in YoutubeDashManifestCreatorsTest 2023-02-26 19:06:17 +01:00
AudricV 05e8cb39f7
[YouTube] Add language and descriptive audio properties to DASH manifests 2023-02-26 19:06:17 +01:00
AudricV bf30d70152
[YouTube] Add descriptive audio test
This test uses video TjxC-evzxdk.

Also improve a bit YoutubeStreamExtractorDefaultTest.AudioTrackLanguage test.
2023-02-26 19:06:17 +01:00
AudricV 76b7c19c5d
[YouTube] Extract whether a track is a descriptive audio and audio locale when available
Also use audio track setters only for audio itags.
2023-02-26 19:06:17 +01:00
AudricV 3bb5eeef30
[YouTube] Add descriptive and locale audio support in ItagItem 2023-02-26 19:06:16 +01:00
AudricV 14bf3fb05b
Add ability to know the locale of an audio stream
Getting audio tracks locales by parsing their ID or their label, should not be
done by clients, but by the extractor.

This commit adds the ability to store the Locale of an AudioStream, which is
used to compare similar AudioStreams (in the equalStats method).
2023-02-26 19:06:16 +01:00
AudricV f92426560c
Add descriptive audio properties
Also improve AudioStream's audio language documentation
2023-02-26 19:06:16 +01:00
AudricV a63f289667
[YouTube] Update mocks of YoutubeCommentsExtractorTest.FormattingTest 2023-02-26 18:50:07 +01:00
AudricV 9483dcd9fa
[YouTube] Update mocks of YoutubeCommentsExtractorTest.RepliesTest 2023-02-26 18:43:36 +01:00
AudricV 1556adbb2d
[YouTube] Fix hashtags links extraction and escape text in attribute descriptions + HTML links
webCommandMetadata object is contained inside a commandMetadata one, so it is
not accessible from the root of the navigationEndpoint object.

The corresponding statement has been moved at the bottom of the specific
endpoints parsing, as the webCommandMetadata object is present almost
everywhere, otherwise URLs of some endpoints would have be changed, such as
uploader URLs (from channel IDs to handles).

As no ParsingException is now thrown by getUrlFromNavigationEndpoint, and so by
getTextFromObject, getUrlFromObject and getTextAtKey, the methods which were
catching ParsingExceptions thrown by these methods had to be updated.

URLs got in the HTML version of getTextFromObject are now escaped properly to
provide valid HTML to clients. This has been also done for attribute
descriptions, with the description text for this type of descriptions.

As YouTube descriptions are in HTML format (except for the fallback on the JSON
player response, which is plain text and only happens when there is no visual
metadata or a breaking change), all URLs returned are escaped, so tests which
are testing presence of URLs with escaped characters had to be updated (it was
only the case for YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing).
2023-02-26 18:43:36 +01:00
Stypox 99ab9777ad
Release 0.22.5 2023-02-26 16:33:25 +01:00
Stypox 39c2e1fda0
Merge pull request #1028 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.4
Bump org.jsoup:jsoup from 1.15.3 to 1.15.4
2023-02-26 16:27:31 +01:00
petlyh f7a7a236fb
[Bandcamp] Show comments as disabled on radio streams 2023-02-23 18:42:43 +01:00
dependabot[bot] f5599ff08d
Bump org.jsoup:jsoup from 1.15.3 to 1.15.4
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup) from 1.15.3 to 1.15.4.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.3...jsoup-1.15.4)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-20 10:05:26 +00:00
Stypox 7e793c11ae
Empty commit to make JitPack happy
The build for e920ab3f5f succeeded, but for some reason it is considered as failed...
2023-02-08 22:35:26 +01:00
Stypox e920ab3f5f
Merge pull request #1021 from TeamNewPipe/fix/yt-comment-text-hashtag
[YouTube] Fix getting the comment text if the comment contains a hashtag
2023-01-29 21:47:59 +01:00
TobiGr 3f7df9536e [YouTube] Fix getting the comment text if the comment contains a hashtag 2023-01-29 20:33:51 +01:00
Stypox 999fb7f812
Merge pull request #1024 from AudricV/snd_fix-tracks-like-count
[SoundCloud] Fix extraction of tracks like count
2023-01-29 10:52:54 +01:00
Stypox 3519d4c367
Merge pull request #1015 from AudricV/yt_fix-channel-id-rss-feeds
[YouTube] Fix channel ID extraction of YouTube channels RSS feeds
2023-01-29 10:41:38 +01:00
Stypox 9aca710e86
Merge pull request #1013 from Stypox/fix-music-mixes
[YouTube] Now music mixes can be treated as normal mixes
2023-01-29 09:48:51 +01:00
Stypox 76eeabac45
Merge pull request #1020 from TeamNewPipe/fix/yt-subscriber-count
[YouTube] Fix NPE in search when getting channel items without subscriber count
2023-01-29 09:44:22 +01:00
AudricV 676622f6df
[SoundCloud] Fix expectedLikeCountAtLeast tests of SoundcloudStreamExtractorTest test classes
As like count is now returned by the extractor, we need to assert a positive
minimum like count, which is close to the actual value, in order to avoid test
failures due to lower like counts than the ones excepted.
2023-01-29 01:08:02 +01:00
AudricV 2a24d407d5
[SoundCloud] Fix extraction of tracks like count
SoundCloud is using likes_count to return the like count of a track, like it
was the case before they switched to favoritings_count.
2023-01-29 01:00:49 +01:00
Tobi 0e4e6a9bac
Merge pull request #1022 from AudricV/yt_support-live-urls
[YouTube] Support live URLs
2023-01-28 21:52:26 +01:00
AudricV ba24976e41
[YouTube] Add live URLs test and do minor improvements to YoutubeStreamLinkHandlerFactoryTest
- Remove unused imports;
- Replace wildcard imports by single class imports;
- Suppress "HTTP links are not secured" warnings from IDEA IDEs;
- Replace removed video jZViOEv90dI by an existing video, 9Dpqou5cI08 (the
corresponding test has been of course renamed).
2023-01-28 19:36:21 +01:00
AudricV 57f850bc2d
[YouTube] Support live URLs and do minor improvements to YoutubeStreamLinkHandlerFactory
- Move license header at the top;
- Use an unmodifiable set for the subpaths instead of a modifiable list;
- Add missing Nonnull and Nullable annotations;
- Improve exception messages.
2023-01-28 19:36:20 +01:00
AudricV 1f4ed9dce9
[YouTube] Fix channel ID extraction of YouTube channel RSS feeds
The yt:channelId element doesn't provide the channel ID anymore and is empty,
like the id element, so we need now to extract it from the channel URL provided
in two elements: author -> uri and feed -> link.

Also avoid a NullPointerException in getUrl and getName methods.
2023-01-28 11:53:33 +01:00
Tobi c589a2c1a2
Merge pull request #1014 from TeamNewPipe/fix/yt-comments
[YouTube] Fix getting next comments pages
2023-01-27 11:14:55 +01:00
Tobi c2b36fd405
Merge pull request #1016 from lonewolf2208/bug-fix
[YouTube] Remove topStandaloneBadge check for view count of stream items
2023-01-25 08:30:36 +01:00
TobiGr 72573932cf [YouTube] Fix NPE in search when getting channel items without subscriber count 2023-01-24 23:03:45 +01:00
TobiGr f50b7275af [YouTube] Fix getting next comments pages 2023-01-24 22:39:08 +01:00
Kunal 9bdad40b06 Removed topStandaloneBadge 2023-01-20 02:41:21 +05:30
Stypox 5945057227
[YouTube] Add music mix test 2023-01-15 23:30:30 +01:00
Stypox 7293991832
[YouTube] Now music mixes can be treated as normal mixes
Using a playlist extractor on them would result in "Unviewable playlist" errors
2023-01-15 23:28:59 +01:00
Stypox ff94e9f30b
Merge pull request #1009 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.10.1
Bump gson from 2.10 to 2.10.1
2023-01-11 15:35:36 +01:00
Stypox c1040bccac
Merge pull request #794 from FireMasterK/comments-count
[YouTube] Add support to extract total comment count
2023-01-11 15:32:19 +01:00
Stypox 6ccc43e57e
Merge pull request #1010 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.2
Bump junit-bom from 5.9.1 to 5.9.2
2023-01-11 13:19:33 +01:00
dependabot[bot] 1bc44cb949
Bump junit-bom from 5.9.1 to 5.9.2
Bumps [junit-bom](https://github.com/junit-team/junit5) from 5.9.1 to 5.9.2.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.9.1...r5.9.2)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-01-11 09:04:12 +00:00
dependabot[bot] f43049985e
Bump gson from 2.10 to 2.10.1
Bumps [gson](https://github.com/google/gson) from 2.10 to 2.10.1.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.10...gson-parent-2.10.1)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-01-09 09:05:55 +00:00
Stypox 33f9f2cf66
Merge pull request #1006 from TeamNewPipe/yt/fix-html-escape
[YouTube] Fix escaping links in YouTubeParsingHelper.getTextFromObject
2023-01-05 19:02:23 +01:00
TobiGr 56aab4d971 [YouTube] Fix escaping links in YouTubeParsingHelper.getTextFromObject 2023-01-05 00:28:12 +01:00
Kavin 22a47da8c7
Fix requested change and remove outdated comment. 2023-01-02 20:42:32 +00:00
Kavin 98a90fd9c8
Don't cache comments count and return early on page fetch if no token. 2023-01-02 20:40:48 +00:00
Kavin 2974dfaa48
Only store ajaxJson for initial page and eager fetch the initial continuation. 2023-01-02 20:40:48 +00:00
Kavin 64d24aa09e
Fix request changes. 2023-01-02 20:40:48 +00:00
Kavin 67ef4f4c30
Cleanup and remove optional. 2023-01-02 20:40:48 +00:00
FireMasterK 22f71b010c
Fix for requested changes. 2023-01-02 20:40:48 +00:00
FireMasterK 656b7c1cd9
Improve method documentation. 2023-01-02 20:40:48 +00:00
FireMasterK 981aee4092
Add support to extract total comment count. 2023-01-02 20:40:48 +00:00
Stypox 45636b0d00
Merge pull request #986 from Isira-Seneviratne/Static_maps
Use immutable Map factory methods.
2023-01-02 18:11:14 +01:00
Stypox 219c5c5be5
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java 2023-01-02 18:11:03 +01:00
Stypox 259de3cba6
Merge pull request #995 from TeamNewPipe/feat/soundcloud-playlistinfoitemextractor
[SoundCloud] Implement getUploaderUrl() and isUploaderVerified() for PlaylistInfoItemExtractor
2023-01-02 15:10:40 +01:00
Stypox 991394b53a
Merge pull request #1005 from FireMasterK/fix-escaping-xss
Fix for potential XSS attacks and formatting issues
2023-01-02 15:06:17 +01:00
Stypox ce15f7cc50
Merge pull request #996 from TeamNewPipe/feat/peeertube-playlists
[PeerTube] Support searching for playlists and channels
2023-01-02 15:04:09 +01:00
Isira Seneviratne d8ce08d969 Use immutable Map factory methods. 2023-01-02 07:50:31 +05:30
Kavin 01acf79436
Fix for potential XSS attacks. 2022-12-31 20:05:32 +00:00
TobiGr 292e0d8ce7 [SoundCloud] Implement getUploaderUrl() and isUploaderVerified() for PlaylistInfoItemExtractor 2022-12-31 18:46:39 +01:00
TobiGr 2a8729aeb2 Apply suggestions
Co-authored-by: Stypox <stypox@pm.me>
2022-12-31 18:24:33 +01:00
TobiGr d75a997611 [PeerTube] Support searching for channels 2022-12-31 18:24:33 +01:00
TobiGr dea6d8ce4c [PeerTube] Support searching for playlists 2022-12-31 18:24:33 +01:00
Stypox 95cc6aefbb
Merge pull request #994 from TeamNewPipe/fix/peertube-subtitles-exception
[PeerTube] Report Exceptions thrown while getting a stream's subtitles
2022-12-31 15:01:39 +01:00
Stypox 7b54457789
Merge pull request #941 from TeamNewPipe/feat/peertube-comment-replies
[PeerTube]  Support comment replies
2022-12-31 14:57:51 +01:00
AudricV f45966d449
Merge pull request #910 from Isira-Seneviratne/Locale_forLanguageTag
Add compat Locale.forLanguageTag() implementation.
2022-12-24 23:53:30 +01:00
AudricV d5437e0bc5
Merge pull request #863 from AudricV/add-content-type-and-content-length-headers-to-post-requests
Add Content-Type header to all POST requests without an empty body
2022-12-16 19:32:56 +01:00
Tobi 88e07e555d
Merge pull request #1000 from AudricV/yt-streaminfoitemextractor-improvements
[YouTube] Improve YoutubeStreamInfoItemExtractor
2022-12-11 17:02:29 +01:00
AudricV 0766b1d211
[YouTube] Improve YoutubeStreamInfoItemExtractor
- Return duration of video premieres;
- Add another non-localized method to determine whether a stream is a running
livestream;
- Return view count and upload date of videos in playlists;
- Store isPremiere result;
- Remove shorts workaround code, as it was only useful on channels and shorts
have been moved into a separated channel tab;
- Improve some other code.
2022-12-08 13:59:12 +01:00
Tobi 896d7e09eb
Merge pull request #978 from Theta-Dev/fix/search-channel-handles
[YouTube] Fix search subscriber count extraction with channel handles
2022-12-05 17:52:05 +01:00
TobiGr cd3262745d [PeerTube] Report Exceptions thrown while getting a stream's subtitles 2022-12-03 16:11:21 +01:00
TobiGr 4e66b2287e [PeerTube] Add support for comment replies 2022-12-01 14:05:18 +01:00
Tobi 41c8dce452
Merge pull request #992 from Isira-Seneviratne/String_isBlank
Use String.isBlank().
2022-11-30 17:48:54 +01:00
Isira Seneviratne 2bca56f0df Use String.isBlank(). 2022-11-30 08:26:21 +05:30
Isira Seneviratne 3b80547976 Add code review suggestions. 2022-11-30 07:57:45 +05:30
Tobi a1128ecbdc
Merge pull request #991 from FireMasterK/fix-compile
Fix complication error in comment test
2022-11-29 19:20:01 +01:00
ThetaDev 016623131e docs: update comment in YoutubeChannelInfoItemExtractor 2022-11-29 19:06:03 +01:00
Kavin 2e08eaad96
Fix complication error in comment test. 2022-11-29 16:07:48 +00:00
Kavin abf08e1496
Merge pull request #990 from FireMasterK/bold-italic-strikethrough
[YouTube] Implement bold/italic/strike-through support
2022-11-29 15:59:38 +00:00
Tobi 72d5ed3318
Merge pull request #987 from FireMasterK/comments-text-description
Use Description object for comments text.
2022-11-29 16:54:56 +01:00
Kavin 57e7a6fb7c
Add mocks test. 2022-11-28 20:27:55 +00:00
Kavin 1d3d7fa5c3
Add test for formatting. 2022-11-28 20:26:37 +00:00
Kavin 52fda37915
Implement bold/italic/strike-through support. 2022-11-28 19:06:18 +00:00
Kavin b566084cac
Use Description object for comments text. 2022-11-28 17:02:19 +00:00
Tobi 40f1ec4a54
Merge pull request #989 from Stypox/forbid-wrong-nullable-notnull-imports
Block wrong nullable/nonnull imports in checkstyle
2022-11-28 14:57:46 +01:00
Stypox 5f6101698b
Block wrong nullable/nonnull imports in checkstyle 2022-11-28 14:42:18 +01:00
Tobi f8162b049d
Merge pull request #984 from FireMasterK/unused-dep
Remove unused autolink dependency
2022-11-28 11:28:42 +01:00
Tobi ad7c3c2374
Merge pull request #985 from FireMasterK/checkstyle-update
Update checkstyle dependency.
2022-11-28 11:22:40 +01:00
Tobi 1da0190056
Merge pull request #980 from TeamNewPipe/fix/yt/unavailable
[YouTube] Fix extracting the detailed error message for unavailable streams
2022-11-28 10:07:34 +01:00
Stypox 60fb30f835
Merge pull request #928 from FireMasterK/comment-urls
Parse YouTube comments as HTML
2022-11-27 19:16:34 +01:00
Kavin 5abea22225
Fix throwing correct reason. 2022-11-26 21:09:08 +00:00
Kavin 7874d3c8aa
Update checkstyle dependency.
Since we now use JDK 11
2022-11-26 20:22:25 +00:00
Kavin faf28f5c11
Remove unused dependency. 2022-11-26 20:17:25 +00:00
Kavin a3ed947b2d
Merge pull request #983 from FireMasterK/update-countries
Update supported countries list.
2022-11-26 20:06:00 +00:00
Kavin c043597255
Update supported countries list. 2022-11-26 19:01:33 +00:00
TobiGr 4680df0bdf Fix throwing correct reason 2022-11-23 17:03:22 +01:00
TobiGr 9de8405c9f [YouTube] Fix extracting the detailed error message of streams which are unavailable 2022-11-23 08:33:06 +01:00
Stypox 34d79bd267
[YouTube] Update mocks 2022-11-22 17:10:04 +01:00
AudricV 2ec296e674
Fix YoutubeSearchExtractorTest.MetaInfoTest
Not all the "learn more" button is uppercase anymore, that's only the case for
the first letter.
2022-11-22 16:34:54 +01:00
AudricV 3891542ca1
Use Downloader's postWithContentType and postWithContentTypeJson methods in services and extractors 2022-11-22 11:37:18 +01:00
AudricV b2862f3cd1
Add postWithContentType and postWithContentTypeJson utility methods in Downloader
Co-authored-by: Stypox <stypox@pm.me>
2022-11-22 11:37:17 +01:00
AudricV e9a0d3bd95
[YouTube] Send Content-Type header in all POST requests
This header was not sent partially before and was added and guessed by OkHttp. This can create issues when using other HTTP clients than OkHttp, such as Cronet.

Some code in the modified classes has been improved and / or deduplicated, and usages of the UTF_8 constant of the Utils class has been replaced by StandardCharsets.UTF_8 where possible.

Note that this header has been not added in except in YoutubeDashManifestCreatorsUtils, as an empty body is sent in the POST requests made by this class.
2022-11-22 11:37:16 +01:00
AudricV b9e463de49
[Bandcamp] Send Content-Type header in POST requests
This header was not sent before and was added and guessed by OkHttp. This can create issues when using other HTTP clients than OkHttp, such as Cronet.

Also make use of StandardCharsets.UTF_8 when getting bytes of bodies instead of the platform default's charset, to make sure to prevent some encoding issues on some JVMs.
2022-11-22 11:35:46 +01:00
AudricV 65d6321e3d
Fix typos in Downloader.post JavaDocs
Post methods in Downloader return the result of a POST request and not the one of a GET request.
2022-11-22 11:35:46 +01:00
ThetaDev 5daabd1793 fix: #976 search subscriber count extraction with channel handles 2022-11-22 02:17:10 +01:00
Kavin c953e23414
Merge pull request #968 from AudricV/yt-support-no-video-info-renderers-for-streams
[YouTube] Support lack of video info renderers for streams
2022-11-16 20:20:01 +00:00
Tobi 2211a24b69
Merge pull request #971 from lrusso96/patch-1
[YouTube] Improve duration parsing
2022-11-16 16:14:54 +01:00
Tobi f6eefdc76c
Merge pull request #975 from FireMasterK/jitpack
Use JDK 11 for Jitpack gradle builds
2022-11-15 20:23:21 +01:00
Kavin a1f22be0dd
Use JDK 11 for Jitpack gradle builds. 2022-11-15 18:43:19 +00:00
Kavin 9bdad55508
Merge pull request #973 from FireMasterK/audio-track
Add support for extracting audio track information
2022-11-15 08:19:47 +00:00
Kavin 86f06b333a
Address review. 2022-11-14 00:05:31 +00:00
Kavin b16e6082e1
Add test for audio stream languages. 2022-11-13 23:10:44 +00:00
Kavin 30909da1df
Fix audio track similar comparison for IDs. 2022-11-13 23:08:54 +00:00
Kavin 6d59cdbe3a
Add support for extracting audio tracks. 2022-11-13 21:39:29 +00:00
Tobi 43d1c1f8b1
Merge pull request #880 from Isira-Seneviratne/Use_StandardCharsets
Use StandardCharsets.UTF_8.
2022-11-12 03:23:44 +01:00
Isira Seneviratne e4d982c7ea Fix license. 2022-11-12 07:29:15 +05:30
Isira Seneviratne 416089146e Fix failing tests. 2022-11-12 07:29:15 +05:30
Isira Seneviratne ff5f223d3f Update README.md 2022-11-12 07:29:15 +05:30
Isira Seneviratne ddbce3b83d Add Utils methods for URL encoding/decoding using UTF-8. 2022-11-12 07:29:15 +05:30
Isira Seneviratne 366f5c1632 Use StandardCharsets.UTF_8. 2022-11-12 07:29:15 +05:30
Tobi 0ea16c0b73
Merge pull request #954 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.10
Bump gson from 2.9.1 to 2.10
2022-11-09 17:28:31 +01:00
dependabot[bot] 0b82fade51
Bump gson from 2.9.1 to 2.10
Bumps [gson](https://github.com/google/gson) from 2.9.1 to 2.10.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.9.1...gson-parent-2.10)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-09 16:21:16 +00:00
Luigi Russo c9635218e2
[YouTube] Improve duration parsing 2022-11-09 09:41:29 +01:00
Isira Seneviratne 1be270768f
Merge pull request #893 from Isira-Seneviratne/Static_sets
Use immutable sets in YoutubeParsingHelper.
2022-11-09 09:07:01 +05:30
Isira Seneviratne 316d8573fa Use immutable sets in YoutubeParsingHelper. 2022-11-07 07:50:26 +05:30
AudricV 6a2c680d8f
[YouTube] Add mocks for YoutubeStreamExtractorDefaultTest.NoVisualMetadataVideoTest 2022-11-04 19:43:06 +01:00
AudricV e66fed41d6
[YouTube] Add a StreamExtractor test for a video without visual metadata
The video "Makani’s first commercial-scale energy kite" (video ID:
An8vtD1FDqs), which has this behavior, is used for the new test,
NoVisualMetadataVideoTest, added in YoutubeStreamExtractorDefaultTest.

Tests of elements who throw an exception in this case (subscriber count, like
count, uploader avatar URL) test if the ParsingException exception is thrown by
YoutubeStreamExtractor.
2022-11-04 19:42:12 +01:00
AudricV aa9a8ca23c
[YouTube] Make non-extraction of videoPrimaryInfoRenderer and/or videoSecondaryInfoRenderer not fatal
Also de-duplicated common code related to the obtain of these video info renderers.

This change allows extraction of videos without visual metadata.
2022-11-04 18:35:53 +01:00
AudricV eb07d70a2c
Merge pull request #964 from AudricV/yt-support-handles-and-all-channel-usernames
[YouTube] Support handles and all channel usernames
2022-11-04 12:09:15 +01:00
AudricV 20cd8e8a4a
[YouTube] Update mocks of YoutubeChannelExtractorTest.Gronkh 2022-11-03 19:46:42 +01:00
AudricV a34f060ba0
[YouTube] Use a handle for YoutubeChannelExtractorTest.Gronkh 2022-11-03 19:46:42 +01:00
AudricV 724f669ff7
[YouTube] Add tests for handles and user IDs with non ASCII characters support
Unneeded public modifiers in test methods of
YoutubeChannelLinkHandlerFactoryTest have been also removed.
2022-11-03 19:46:42 +01:00
AudricV 61ce041bda
[YouTube] Support handles and all custom channel names
More non-channel paths have been also added to the excluded custom name paths,
documentation and exception messages have been improved and fixed in some
places, and the licence header of YoutubeChannelLinkHandlerFactory has been
moved to its beginning and updated.
2022-11-03 19:46:42 +01:00
AudricV ffffb04439
Merge pull request #953 from Theta-Dev/attributed-text-desc
[YouTube] Add support for attributed text description
2022-11-03 18:34:30 +01:00
ThetaDev 592e1d6386 fix: parsing attributed description with no command runs 2022-11-03 12:10:52 +01:00
AudricV 31bf7046b7
Merge pull request #944 from Theta-Dev/fix-rich-grid-renderer
[YouTube] Support richGridRenderer on channel page to fix extraction of videos in channels
2022-11-03 00:10:54 +01:00
ThetaDev 099b53cc4f
[YouTube] Add parser for attributedDescription
Also update the mock of the next InnerTube endpoint response of the
YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing test class with an
attributedDescription instead of a regular description
2022-11-02 23:11:33 +01:00
AudricV e4c24d4c36
[YouTube] Regenerate supported channels mocks 2022-11-02 19:13:59 +01:00
Theta-Dev 20e4a35814
[YouTube] Support richGridRenderer on channel pages
YouTube is deploying a new layout on their channel pages, which uses richGridRenderer JSON objects.
2022-11-02 19:01:29 +01:00
AudricV 4cae66f1f9
Merge pull request #946 from chowder/dev
Add ability to identify short-form `StreamInfoItem`s
2022-11-01 12:19:58 +01:00
Tobi eb40bb8458
Merge pull request #959 from FireMasterK/playlist-info-item-uploader
Add uploaderUrl and uploaderVerified to PlaylistInfoItem.
2022-10-31 13:10:33 +01:00
chowder b1a899fd47 Fix null pointer exception 2022-10-31 11:12:23 +00:00
Stypox a4db106a66
Merge pull request #960 from AudricV/yt-workaround-403-errors-android-client
[YouTube] Workaround 403 HTTP errors of ANDROID client streams
2022-10-30 21:53:26 +01:00
Kavin b441910257
Mark uploaderUrl as nullable. 2022-10-30 13:36:04 +00:00
chowder 3fdc0e72cc Line breaks for long docstrings 2022-10-30 13:28:39 +00:00
Kavin 6a256d0631
Add uploader url and verified to PlaylistInfoItem. 2022-10-30 13:00:19 +00:00
Tobi 430504b4b5
Merge pull request #958 from AudricV/yt-playlists-support-new-metadata-format
[YouTube] Support new metadata format of playlists
2022-10-30 12:31:43 +01:00
Kavin f9bd08c649 Address reviews. 2022-10-30 01:25:30 +00:00
Caleb 9282c3c13b Fix exception message for YoutubeStreamInfoItemExtractor#isShortFormContent
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
2022-10-30 01:23:15 +00:00
Caleb c5216f7c12 Update docstring for StreamExtractor#isShortFormContent
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
2022-10-30 01:23:15 +00:00
Caleb 04795fe5d2 Use Stream API for ShortFormContent#testShortFormContent
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
2022-10-30 01:23:15 +00:00
chowder 975b38a0b9 Regenerate test mocks 2022-10-30 01:23:15 +00:00
chowder 4cccd33f3d Implement isShortFormContent for StreamExtractor and StreamInfo 2022-10-30 01:23:15 +00:00
chowder 09544ceb23 Fix tests 2022-10-30 01:23:15 +00:00
chowder 644cc6cd76 Rename test function name 2022-10-30 01:23:15 +00:00
chowder daf5674951 Add ability to identify short-form StreamInfoItems 2022-10-30 01:23:12 +00:00
Tobi 3d314169b9
Merge pull request #943 from TeamNewPipe/fix/sc/comments
[SoundCloud] Fix getting more comments
2022-10-29 22:19:50 +02:00
AudricV 65125a3eb4
[YouTube] Fix YoutubePlaylistExtractorTest.LearningPlaylist
The video count is now returned for this playlist, so it isn't unknown.

The testStreamCount method of this test class asserts now that the stream count
is greater than 40.
2022-10-29 18:12:10 +02:00
AudricV 7258a53225
[YouTube] Support new playlist layout
This new layout doesn't provide author thumbnails and is completely different
for metadata, so the code to get them has been refactored.

The code of learning playlists video count check has been also removed, as it
seems to be not relevant anymore (the video count seems to be returned for
these playlists with both layouts).

Finally, unneeded overrides of subchannel methods, which don't apply to the
YouTube service, have been removed.
2022-10-29 18:12:10 +02:00
AudricV e37e8b358e
[YouTube] Update mocks that need to be updated 2022-10-29 18:10:18 +02:00
AudricV 60e97cd274
[YouTube] Workaround getting streaming URLs returning 403 HTTP response codes
Using the player parameters used to get stories seems to fix the issue, which
affects currently only certain countries such as UK.

This is a workaround and should be fixed in a better way (by changing the
InnerTube additional client used for videos or finding what is now required in
Android player requests).
2022-10-29 17:58:33 +02:00
AudricV 4bc90cd9d8
Merge pull request #957 from AudricV/fix-checkstyle-error
Fix Checkstyle error in YoutubeCommentsInfoItemExtractor
2022-10-29 17:55:33 +02:00
AudricV c230d84df1
Fix Checkstyle error in YoutubeCommentsInfoItemExtractor 2022-10-29 13:24:19 +02:00
Tobi 9ffdd0948b
Merge pull request #945 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.3
Bump spotbugs-annotations from 4.7.2 to 4.7.3
2022-10-18 22:55:24 +02:00
dependabot[bot] 24137bb15f
Bump spotbugs-annotations from 4.7.2 to 4.7.3
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.7.2 to 4.7.3.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.2...4.7.3)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-10-17 09:20:51 +00:00
xz-dev 0ffcb32d9c
[YouTube] Add comment reply count support (#936)
Add comment reply count support for YouTube and introduce `CommentsInfoItem.UNKNOWN_REPLY_COUNT` constant

Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2022-10-15 12:40:06 +02:00
Isira Seneviratne b90a566dd8 Add backport implementation of Locale.forLanguageTag(). 2022-10-12 09:21:39 +05:30
Isira Seneviratne b232c29d22 Use Locale.forLanguageTag(). 2022-10-12 09:21:38 +05:30
TobiGr 4d136599bd [SoundCloud] Fix getting more comments 2022-10-11 15:44:54 +02:00
Tobi a822e91909
Merge pull request #939 from TurtleArmyMc/fix/SoundcloudPlaylistExtractor_track_order
Fix SoundcloudPlaylistExtractor: tracks are in correct order
2022-10-10 22:28:18 +02:00
TobiGr 02810a7db7 Add a comment 2022-10-10 22:22:12 +02:00
Tobi 2d50369c77
Merge pull request #888 from Isira-Seneviratne/EnumMap
Use EnumMap in PatternsHolder.
2022-10-10 12:00:06 +02:00
Tobi 54092fc3c7
Merge pull request #930 from Isira-Seneviratne/Avoid_Comparator_NPE
Avoid possible NullPointerException in MediaCCCRecentKiosk.
2022-10-10 11:43:34 +02:00
Tobi d7c23ef118
Merge pull request #924 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.2
Bump spotbugs-annotations from 4.7.1 to 4.7.2
2022-10-10 11:35:40 +02:00
dependabot[bot] 08339d6b0c
Bump spotbugs-annotations from 4.7.1 to 4.7.2
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.7.1 to 4.7.2.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.1...4.7.2)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-10-10 09:32:17 +00:00
Tobi 5537e84100
Merge pull request #931 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.1
Bump junit-bom from 5.9.0 to 5.9.1
2022-10-10 11:30:53 +02:00
dependabot[bot] 719664593f
Bump junit-bom from 5.9.0 to 5.9.1
Bumps [junit-bom](https://github.com/junit-team/junit5) from 5.9.0 to 5.9.1.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.9.0...r5.9.1)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-10-10 09:25:42 +00:00
TurtleArmyMc bf70d32eb4 Fix SoundcloudPlaylistExtractor: tracks are in correct order 2022-09-30 15:26:25 -04:00
Stypox 5c710da160
Merge pull request #934 from AudricV/fix-yt-throttling-decryption-function-regex
[YouTube] Fix throttling decryption function regex
2022-09-25 13:26:08 +02:00
AudricV 8067c43837
[YouTube] Don't use a specific letter for the decryption function name pattern
Use the same possible characters for variables everywhere, in order to avoid
potential future throttling parameter decryption function name parsing issues
related to the usage of other letter(s) than b.
2022-09-24 21:49:22 +02:00
AudricV abcee87167
[YouTube] Fix throttling parameter decryption function regex
- Quote the function name, as it may contain special regex symbols, such as
  dollar;
- Support multiple lines;
- Use what looks like the end of the function for the end of the regex (this
  part is inspired from yt-dlp throttling parameter decryption regex);
- Move the throttling function body regex into a private and static constant.
2022-09-24 21:28:09 +02:00
ThetaDev 7244be7627
[YouTube] Add JavaScript lexer to parse completely throttling decryption function (#905) 2022-09-24 19:55:46 +02:00
Isira Seneviratne 0e31c86aee Avoid possible NullPointerException in MediaCCCRecentKiosk. 2022-09-21 05:40:07 +05:30
Stypox a99af9bb6e
Merge pull request #927 from Stypox/fix-feed-thumbnail
[YouTube] Return mqdefault thumbnails in fast feed
2022-09-19 08:52:01 +02:00
Kavin 4e14644c41
Ensure comments are parsed with URLs and timestamps. 2022-09-17 16:03:39 +05:30
Stypox 14ef430546
Merge pull request #920 from AudricV/fix-yt-trending-extraction
[YouTube] Fix extraction of the Trending kiosk with the new data model returned
2022-09-14 11:14:13 +02:00
Stypox 93ef73e2f2
[YouTube] Return mqdefault thumbnails in feed
The hqdefault thumbnails, which are used by default, have black bars at the top and at the bottom, and are thus not optimal.
2022-09-13 16:46:54 +02:00
AudricV d8ddeb549c
[YouTube] Support new trending structure and filter recently trending section
This new structure allow us to filter easily Trending shorts and Recently
trending sections.

On the previous one, this Recently trending section is now filtered, by
checking whether sections have a title, which isn't the case for normal trends
contrary to the other ones.

This makes that the extractor returns now only the real 50 "Now" YouTube
trends.

Elements inside arrays are now extracted dynamically instead of only the ones
of the first index, using Java 8's Stream API.

The getInitialPage() method of YoutubeTrendingExtractor can now throw a
ParsingException if no selected tab (corresponding to the one of the trends
type extracted) has been found.

Finally, the licence header has been moved to the top of the file and updated.
2022-09-11 18:52:12 +02:00
AudricV 884da40f65
Merge pull request #926 from AudricV/fix-yt-like-count-extraction
[YouTube] Fix extraction of like count with the new data model
2022-09-11 12:21:38 +02:00
AudricV 58b09cf78b
Merge pull request #925 from Theta-Dev/fix-td-regex
[YouTube] Fix throttling parameter decryption on Android
2022-09-10 23:30:50 +02:00
AudricV 119b9129e2
[YouTube] Fix extraction of like count with the new data model
In the new data model currently A/B tested or deployed, the like count button
data is the same, but its path has been changed.

The extraction of the like count has been also improved, by using the multiple
accessibility data available instead of the only one which was used before.
This accessibility data has been also deprioritized, because it is language
dependent when there is no view.

Like count is always known, even for age-restricted videos, so returning -1 if
the like count extraction fails on this type of videos is not needed and hides
an extraction error: that's the reason why this return has been removed and the
exception will always be thrown, even if a video is age-restricted.
2022-09-10 23:28:38 +02:00
ThetaDev 4905f74700
[YouTube] Fix throttling parameter decryption on Android
Escape the curly brace in the regular expression used to parse the throttling
parameter decryption function to allow its compatibility on Android.
2022-09-10 17:03:39 +02:00
Isira Seneviratne e101d58ae7 Use EnumMap in PatternsHolder. 2022-08-26 08:15:21 +05:30
Stypox 6a858368c8
Merge pull request #912 from Stypox/downgrade-rhino
Downgrade Rhino since it uses a class not available on Android
2022-08-25 10:10:27 +02:00
Stypox fa25a7280f
Downgrade Rhino since it uses a class not available on Android
Reverts #774, see TeamNewPipe/NewPipe#8876
2022-08-25 09:49:13 +02:00
Stypox 25f0ce7305
Merge pull request #911 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.3
Bump jsoup from 1.15.2 to 1.15.3
2022-08-24 14:23:44 +02:00
Stypox d8e10dd3ad
Merge pull request #907 from Isira-Seneviratne/Remove_EmptyString
Remove EMPTY_STRING.
2022-08-24 14:22:55 +02:00
dependabot[bot] 1d7a3a90a4
Bump jsoup from 1.15.2 to 1.15.3
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.15.2 to 1.15.3.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.2...jsoup-1.15.3)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-08-24 09:14:26 +00:00
Isira Seneviratne 41254ae12a Remove annotation. 2022-08-24 06:59:17 +05:30
Isira Seneviratne 943b7c033b Remove EMPTY_STRING. 2022-08-24 06:59:17 +05:30
litetex 20914a6035
Merge pull request #903 from TacoTheDank/updateGradle
Update gradle wrapper to 7.5.1
2022-08-23 16:21:56 +02:00
litetex d6577e5e0b
Merge pull request #882 from litetex/fix-all-tests
Fix all tests
2022-08-23 16:19:27 +02:00
litetex 0a5ad90724
Merge pull request #901 from AudricV/remove-deprecated-code-yt-throttling-decrypter
[YouTube] Remove deprecated throttling decryption code
2022-08-22 17:13:00 +02:00
AudricV 03d9a4fe9d
Fix typos in YoutubeStreamExtractor.tryDecryptUrl 2022-08-21 21:17:03 +02:00
litetex 0beb55a232 Improve ``YoutubeThrottlingDecrypterTest`` 2022-08-21 20:17:05 +02:00
litetex 9e93d6b193 YoutubeThrottlingDecrypter: Check if returned string is a valid JavaScript function 2022-08-21 20:16:45 +02:00
litetex ed0a07af4e YoutubeThrottlingDecrypter: Patch regex 2022-08-21 20:15:53 +02:00
litetex 641a447d9b Updated consent-related mock-data 2022-08-21 18:43:48 +02:00
litetex 8ff7a90f52 Improved consent cookie related constants and documentation 2022-08-21 18:41:40 +02:00
TacoTheDank d8339d4060 Update gradle wrapper to 7.5.1 2022-08-18 15:09:35 -04:00
AudricV cb64a480cd
[YouTube] Remove deprecated code in YoutubeThrottlingDecrypter
YoutubeThrottlingDecrypter is now a non-instantiable final class and non-static
attributes have been removed.

The static attributes of this class have been renamed, in order to respect the
naming specification used.
2022-08-16 15:01:05 +02:00
litetex 5bbea3a8f2 Fix ``YoutubeMixPlaylistExtractorTest`` again
Doesn't work for kurgesagt videos anymore, used music video (copyright free) instead
2022-08-14 15:12:27 +02:00
litetex da06166065 Updated mock data of ``YoutubeSearchExtractorTest$Suggestion`` 2022-08-14 15:12:26 +02:00
litetex 2e36ab1578 Fixed ``YoutubeSearchExtractorTest$Suggestion`` 2022-08-14 15:12:26 +02:00
litetex 2a8a623643 ``YoutubePlaylistLinkHandlerFactoryTest``: Use parameterized tests 2022-08-14 14:48:31 +02:00
litetex 938e69a16a Fixed ``YoutubePlaylistLinkHandlerFactoryTest`` 2022-08-14 14:48:30 +02:00
litetex 844de3e378 Fix ``YoutubeStreamLinkHandlerFactoryTest `` and parameterized the tests
used code from #833
2022-08-14 14:48:30 +02:00
litetex c1a72b8240 Use Android Studios code style 2022-08-14 14:48:29 +02:00
litetex fc27b8a5b8 Use preexisting ``ContentNotAvailableException`` 2022-08-14 14:48:29 +02:00
Stypox 17bad6cedf Fix test about exception type: NPE, not IllegalArgment
The relevant change was made in #877
2022-08-14 14:48:28 +02:00
litetex ecfc370685 Fixed all YTMixPlaylists
Added option to choose if you want to consent or not - currently this is done by a static variable in ``YoutubeParsingHelper`` - may not be the best long-term solution but for now the tests work again (in EU countries) 🥳
2022-08-14 14:48:27 +02:00
litetex 2b6fe294b2 Fixed ``YoutubeMusicSearchExtractorTest``
The renderers and queries got changed.
2022-08-14 14:48:27 +02:00
litetex d6586da614 Failing test works locally without any problems. Reformatted it a bit. 2022-08-14 14:48:27 +02:00
litetex bf3ae5e679 Fixed ``SoundcloudStreamLinkHandlerFactoryTest``
* Removed the unnecessary ``public``
* Use parameterized tests
* One song got removed (which caused the test failure), replaced it with another song
2022-08-14 14:48:27 +02:00
litetex 504f81036e Fixed ``PeertubePlaylistExtractorTest``
Also removed the unnecessary ``public`` and reformatted the code a bit
2022-08-14 14:48:26 +02:00
litetex e7c12258f4 Fixed ``BandcampSearchExtractorTest``
Also removed the unnecessary ``public``
2022-08-14 14:48:26 +02:00
litetex 9e8724df4d Fixed ``YoutubeStreamExtractorLivestreamTest``
The "Lofi Girl"-stream got interrupted by a copyright strike and had to be restarted. Because of this a new id is now used.
2022-08-14 14:48:26 +02:00
Tobi 76aad92fa5
Merge pull request #890 from AudricV/yt-clients-update-improvements-and-extraction-fixes
[YouTube] Update clients version and improve extraction of API key and client version of WEB client
2022-08-12 23:32:41 +02:00
AudricV 472f5d9e9c
[YouTube] Update mocks 2022-08-12 19:20:32 +02:00
AudricV 7bdca33a87
[YouTube] Ensure that an additional player response is the correct one
If YouTube detect that requests come from a third party client, they may
replace the real player response by another one of a video saying that this
content is not available on this app and to watch it on the latest version of
YouTube. We can detect this by checking whether the video ID of the player
response returned is the same as the one requested by the extractor.
2022-08-12 19:20:31 +02:00
AudricV c82317e318
[YouTube] Spoof more mobile clients
Additional parameters have been added to the player requests of ANDROID and IOS
clients:

- for both clients: osName and osVersion: their respective values are:
  - for the ANDROID one: Android and 12;
  - for the IOS one: iOS and 15.6.0.19G71.
- for the ANDROID client: androidTargetSdkVersion, with the Android SDK version
  corresponding to the Android version used in the player requests of this
  client. This parameter is now required with this client to be sure to get a
  correct player response, otherwise, the one of a video saying that this
  content is not available in this app and to watch it with the latest version
  of YouTube can be returned instead;
- for the IOS client: deviceMake, with Apple as its value.

The iOS version sent in the IOS client player requests has been also updated to
the version 15.6 of the OS.

Finally, a comment about the requirement to use the signature timestamp from
the player JavaScript base file for HTML5 player requests on videos with
obfuscated URLs has been added and replaces a previous one which may be not
true.
2022-08-12 19:20:31 +02:00
AudricV d0549a5a52
[YouTube] Update client versions and use a real version for the iOS client
The iOS version can be got easily in fact, by looking at the What's New section of the App Store' app page.
2022-08-12 19:20:31 +02:00
AudricV d7e678aca2
[YouTube] Improve WEB client version and API key HTML extraction
Common code in WEB client version HTML extraction has been deduplicated, usage of the Java 8 Stream API has been made and initial data fallback has been used as a last resort.
This means that the client version extraction from regexes will be used before this fallback, as it doesn't contain the full client version.
This can be used as a way to fingerprint the extractor, even if it seems to be not the case.
2022-08-12 19:20:30 +02:00
AudricV 6a885ef5ab
Merge pull request #891 from Theta-Dev/fix-throttling-decrypter
[YouTube] Fix extraction of more complex nsig functions
2022-08-12 18:24:21 +02:00
AudricV 5b548340e8
[YouTube] Catch any exception in YoutubeThrottlingDecrypter.apply and improve docs
This will prevent any future extractor break due to decryption failure, like it was excepted to be the case before.

Some documentation about the throttling decryption has been also improved.
2022-08-12 17:49:36 +02:00
ThetaDev 52ded6e3d7
Handle curly braces inside strings in StringUtils.matchToClosingParenthesis
This is required to extract fully more complex YouTube nsig functions.
2022-08-12 16:32:00 +02:00
Stypox d12003651b
Merge pull request #886 from Isira-Seneviratne/toArray_improvements
Make improvements to methods using toArray().
2022-08-06 22:34:23 +02:00
Isira Seneviratne 7daca10a06 Make improvements to methods using toArray(). 2022-08-06 05:21:12 +05:30
Stypox 2906be22af
Merge pull request #881 from Isira-Seneviratne/String_join
Use String.join() and Collectors.joining().
2022-08-04 12:09:33 +02:00
Stypox 4ddb96a86f
Merge pull request #883 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.9.1
Bump gson from 2.9.0 to 2.9.1
2022-08-04 11:20:41 +02:00
Isira Seneviratne 64771c5712 Use String.join() and Collectors.joining(). 2022-08-04 05:18:13 +05:30
Stypox fc8b5ebbc6
Merge pull request #878 from Isira-Seneviratne/Use_Collections
Use Collections methods.
2022-08-03 22:50:41 +02:00
Stypox 4a4939d89c
Merge pull request #877 from Isira-Seneviratne/Use_Objects_requireNonNull
Use Objects.requireNonNull().
2022-08-03 22:45:35 +02:00
Stypox c336bd58a5
Merge pull request #879 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.0
Bump junit-bom from 5.8.2 to 5.9.0
2022-08-03 21:06:20 +02:00
dependabot[bot] 325af31e5f
Bump gson from 2.9.0 to 2.9.1
Bumps [gson](https://github.com/google/gson) from 2.9.0 to 2.9.1.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.9.0...gson-parent-2.9.1)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-08-01 09:26:14 +00:00
dependabot[bot] d905636021
Bump junit-bom from 5.8.2 to 5.9.0
Bumps [junit-bom](https://github.com/junit-team/junit5) from 5.8.2 to 5.9.0.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.8.2...r5.9.0)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-07-27 09:20:12 +00:00
Isira Seneviratne 1af6b8eedb Use Collections.singletonList(). 2022-07-27 07:35:57 +05:30
Isira Seneviratne ff60e05c76 Use Collections.singletonMap(). 2022-07-27 07:35:57 +05:30
Isira Seneviratne 682a4263e5 Use Objects.requireNonNull(). 2022-07-27 06:55:26 +05:30
Stypox 8c5f014a6f
Merge pull request #869 from mhmdanas/add-workflow-permissions
Use minimum required permissions for GitHub workflows
2022-07-13 18:58:32 +02:00
Stypox 954a294e27
Merge pull request #870 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.2
Bump jsoup from 1.15.1 to 1.15.2
2022-07-13 17:48:06 +02:00
Mohammed Anas f57049d2c0
Use temurin instead of adopt (#868)
See
https://blog.adoptopenjdk.net/2021/08/goodbye-adoptopenjdk-hello-adoptium/.
2022-07-10 18:38:08 +03:00
Stypox 5bd7bf20cc
Merge pull request #867 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.1
Bump spotbugs-annotations from 4.7.0 to 4.7.1
2022-07-06 14:23:44 +02:00
Stypox 5ab74b3631
Merge pull request #857 from FireMasterK/video-title
Get original untranslated title for YouTube
2022-07-06 10:26:45 +02:00
dependabot[bot] 122365005a
Bump jsoup from 1.15.1 to 1.15.2
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.15.1 to 1.15.2.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.1...jsoup-1.15.2)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-07-04 09:38:16 +00:00
mhmdanas 414186cff5 Use minimum required permissions for GitHub workflows
This reduces the attack surface if the workflows are ever compromised.
2022-07-03 23:55:28 +03:00
dependabot[bot] e9b4be3e3c
Bump spotbugs-annotations from 4.7.0 to 4.7.1
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.7.0 to 4.7.1.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.0...4.7.1)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-06-28 09:15:07 +00:00
Stypox 5219a705ba
Merge pull request #864 from AudricV/fetch-android-client-for-ended/post-livestreams
[YouTube] Fetch the ANDROID client for ended/post livestreams
2022-06-24 14:55:20 +02:00
AudricV 090debd83b
[YouTube] Fetch the ANDROID client for ended/post livestreams
The ANDROID client was only fetched for video contents, where it can be useful on ended/post livestreams, if the n parameter of the WEB client cannot be decrypted, to avoid throttling issues (because the WEB client was only used before for ended/post livestreams).

It also provides an exclusive 48kbps M4A audio format in the adaptiveFormats array of the JSON player response, like other mobile clients (which can be also extracted from the response of the DASH manifest URL returned into the WEB client player's response, but the DASH manifest is not used by the extractor).

A note about non-fatality of fetching or parsing issues of the ANDROID and IOS clients has been added.
2022-06-21 18:53:49 +02:00
litetex a26bcc55c4
Merge pull request #845 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.0
Bump spotbugs-annotations from 4.6.0 to 4.7.0
2022-06-19 15:36:55 +02:00
dependabot[bot] 424eb1c559
Bump spotbugs-annotations from 4.6.0 to 4.7.0
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.6.0 to 4.7.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.6.0...4.7.0)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-06-19 13:29:18 +00:00
litetex 2712c3d788
Merge pull request #847 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.1
Bump jsoup from 1.14.3 to 1.15.1
2022-06-19 15:28:43 +02:00
dependabot[bot] 281d2b9f81
Bump jsoup from 1.14.3 to 1.15.1
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.14.3 to 1.15.1.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.14.3...jsoup-1.15.1)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-06-19 13:15:40 +00:00
litetex f775155d25
Merge pull request #846 from litetex/remove-unused-methods
Remove unused methods
2022-06-19 15:12:15 +02:00
litetex 9d625dd75f
Merge pull request #860 from gliptak/patch-1
Correct PeertubePlaylistExtractorTest unit tests
2022-06-19 15:09:27 +02:00
Gábor Lipták 28c9340d69
Correct unit tests 2022-06-18 11:18:38 -04:00
Stypox 1b51eab664
Merge pull request #859 from AudricV/delivery-methods-fixes-and-improvements
Fix extraction of some properties in ItagItems for YouTube livestreams and post-live streams and remove completely SoundCloud HLS workaround
2022-06-17 15:46:13 +02:00
AudricV 301a795ed3
[SoundCloud] Remove completely workaround for HLS streams
SoundCloud is currently removing this workaround completely, so there is no need to keep it, because it impacts the loading time (a HLS playlist was downloaded and parsed).
2022-06-16 12:12:54 +02:00
AudricV e960a417ec
[YouTube] Fix extraction of fps, audioSampleRate and audioChannels fields for ItagItems of live streams and post live streams
These values were only set before for video streams.

A fallback for the audio channels count has been added, in order to prevent exceptions when generating DASH manifests of audio streams: the fallback value is 2, because most audio streams on YouTube have 2 audio channels.
2022-06-16 12:12:54 +02:00
Stypox c8a77da2ab
Merge pull request #810 from TiA4f8R/delivery-methods-v2
Support delivery methods other than progressive HTTP
2022-06-02 22:44:24 +02:00
Kavin 7635aeed2c
Get original untranslated title for YouTube. 2022-06-02 09:57:52 +01:00
TiA4f8R 287d1dfd63
[SoundCloud] Use the HLS delivery method for all streams and extract only a single stream URL from HLS manifest for MP3 streams
SoundCloud broke the workaround used to get a single file from HLS manifests for Opus manifests, but it still works for MP3 ones.

The code has been adapted to prevent an unneeded request (the one to the Opus HLS manifest) and the HLS delivery method is now used for SoundCloud MP3 and Opus streams, plus the progressive one (for tracks which have a progressive stream (MP3) and for the ones which doesn't have one, it is still used by trying to get a progressive stream, using the workaround).

Streams extraction has been also moved to Java 8 Stream's API and the relevant test has been also updated.
2022-05-29 19:08:18 +02:00
Stypox b3c620f0d8
Apply code review and Streams rework 2022-05-28 12:00:58 +02:00
Stypox d652e05874
[MediaCCC] Fix comments about containsSimilarStream 2022-05-28 12:00:58 +02:00
Stypox 044639c32b
Solve some review comments 2022-05-28 12:00:57 +02:00
litetex c33d392958
Fixed typo XEE → XXE (Xml eXternal Entity attack)
See also:
https://en.wikipedia.org/wiki/XML_external_entity_attack
https://owasp.org/www-community/vulnerabilities/XML_External_Entity_(XXE)_Processing
2022-05-28 12:00:57 +02:00
TiA4f8R fffbbee7f3
[YouTube] Add missing Nonnull annotations in getCache method of YouTube DASH manifest creators 2022-05-28 12:00:56 +02:00
TiA4f8R f7b1515290
[YouTube] Refactor DASH manifests creation
Move DASH manifests creation into a new subpackage of the YouTube package, dashmanifestcreators.
This subpackage contains:

- CreationException, exception extending Java's RuntimeException, thrown by manifest creators when something goes wrong;
- YoutubeDashManifestCreatorsUtils, class which contains all common methods and constants of all or a part of the manifest creators;
- a manifest creator has been added per delivery type of YouTube streams:
  - YoutubeProgressiveDashManifestCreator, for progressive streams;
  - YoutubeOtfDashManifestCreator, for OTF streams;
  - YoutubePostLiveStreamDvrDashManifestCreator, for post-live DVR streams (which use the live delivery method).

Every DASH manifest creator has a getCache() static method, which returns the ManifestCreatorCache instance used to cache results.

DeliveryType has been also extracted from the YouTube DASH manifest creators part of the extractor and moved to the YouTube package.

YoutubeDashManifestCreatorTest has been updated and renamed to YoutubeDashManifestCreatorsTest, and YoutubeDashManifestCreator has been removed.

Finally, several documentation and exception messages fixes and improvements have been made.
2022-05-28 12:00:56 +02:00
TiA4f8R f17f7b9842
Apply requested changes in YoutubeParsingHelper 2022-05-28 12:00:55 +02:00
TiA4f8R 301b9fa024
Remove hashCode and equals methods overrides of Stream classes 2022-05-28 12:00:55 +02:00
TiA4f8R 2f3920c648
[YouTube] Return approxDurationMs value from YouTube's player response in ItagItems generated
This change allows to build DASH manifests using YoutubeDashManifestCreator with the real duration of streams and prevent potential cuts of the end of progressive streams, because the duration in YouTube's player response is in seconds and not milliseconds.
2022-05-28 12:00:54 +02:00
TiA4f8R 4158fc46a0
[Bandcamp] Fix regression of Opus radio streams extraction
When moving opus-lo into a constant, opus-lo was renamed to opus_lo and was only used if no MP3 stream was available (which was not the case before the changes in BandcampRadioStreamExtractor related to the addition of the support of all delivery methods), so these changes removed the ability to get Opus streams of Bandcamp radios.

This commit reverts this unwanted change.
2022-05-28 12:00:54 +02:00
TiA4f8R 54d323c2ae
Fix Checkstyle issue in YoutubeDashManifestCreator 2022-05-28 12:00:53 +02:00
Stypox 2321822844
Rename Stream's baseUrl to manifestUrl 2022-05-28 12:00:53 +02:00
Stypox cfc13f4a6f
[YouTube] Reduce exception generation code and move several attributes of MPD documents into constants
Rename YoutubeDashManifestCreationException to CreationException.
Also use these constants in YoutubeDashManifestCreatorTest.
2022-05-28 12:00:53 +02:00
Stypox 00bbe5eb4d
[YouTube] Make exception messages smaller 2022-05-28 12:00:52 +02:00
Stypox 4da05afe11
[YouTube] Inline collectSegmentsData in YoutubeDashManifestCreator 2022-05-28 12:00:52 +02:00
Stypox 3708ab9ed5
[YouTube] Refactor YoutubeDashManifestCreator
- Remove all of the methods used to access caches and replace them with three caches getters
- Rename caches to shorter and more meaningful names
- Remove redundant @throws tags that just say "if this method fails to do what it should do", which is obvious
2022-05-28 12:00:51 +02:00
Stypox 5c83409039
[YouTube] Rewrite manifest test and rename long methods 2022-05-28 12:00:51 +02:00
Stypox 8226fd044f
[YouTube] Suppress Sonar security warning for XEE 2022-05-28 12:00:50 +02:00
Stypox ba68b8c014
[YouTube] Secure DashManifestCreator against XEE attack
Also remove duplicate lines from javadoc comments, otherwise checkstyle complaints.
XEE prevention is inspired from https://github.com/ChuckerTeam/chucker/pull/201/files
For an explanation of XEE see https://rules.sonarsource.com/java/RSPEC-2755, though the solution reported there causes crashes on Android
2022-05-28 12:00:50 +02:00
Stypox 159d05c91b
Remove unused DashMpdParser (but kept in git history) 2022-05-28 12:00:49 +02:00
Stypox 50272db946
Apply reviews: improve comments, remove FILE, remove Stream#equals(Stream) 2022-05-28 12:00:49 +02:00
TiA4f8R 07b045f20d
[YouTube] Support the iOS client in YoutubeDashManifestCreator and decrypt again the n parameter, if present, for all clients
This commits reverts a new behavior introduced in this branch, which only applied the decryption if needed on streams from the WEB client.
Also fix rebase issues and documentations style in YoutubeDashManifestCreator.
2022-05-28 12:00:48 +02:00
TiA4f8R 436ddde29f
Use assertThrows in YoutubeDashManifestCreatorTest 2022-05-28 12:00:48 +02:00
TiA4f8R d64d7bbd01
Move ManifestCreatorCache tests to a separate class and remove override of equals and hashCode methods in ManifestCreatorCache
These methods don't need to be overriden, as they are not excepted to be used in collections.
Also improve the toString method of this class, which contains also now clearFactor and maximumSize attributes and for each operations.
2022-05-28 12:00:48 +02:00
TiA4f8R 2fb1a412a6
Fix Checkstyle issues, revert resolution string changes for YouTube video streams and don't return the rn parameter in DASH manifests
This parameter is still used to get the initialization sequence of OTF and POST-live streams, but is not returned anymore in the manifests.
It has been removed in order to avoid fingerprinting based on the number sent (e.g. when starting to play a stream close to the end and using 123 as the request number where it should be 1) and should be added dynamically by clients in their requests.
The relevant test has been also updated.

Checkstyle issues in YoutubeDashManifestCreator have been fixed, and the changes in the resolution string returned for video streams in YoutubeStreamExtractor have been reverted, as they create issues on NewPipe right now.
2022-05-28 12:00:47 +02:00
TiA4f8R f61e2092a1
[YouTube] Return a copy of the hardcoded ItagItem instead of returning the reference to the hardcoded one in ItagItem.getItag
To do so, a copy constructor has been added in the class.

This fixes, for instance, an issue in NewPipe, in which the ItagItem values where not the ones corresponsing to a stream but to another, when generating DASH manifests.
2022-05-28 12:00:47 +02:00
TiA4f8R aa4c10e751
Improve documentation and adress most of the requested changes
Also fix some issues in several places, in the code and the documentation.
2022-05-28 12:00:46 +02:00
TiA4f8R 6985167e63
Add tests for YoutubeDashManifestCreator and ManifestCreatorCache
The test added in YoutubeDashManifestCreator uses a video of the Creative Commons channel, licenced under the Creative Commons Attribution licence (reuse allowed).
Also remove public keywords of tests in UtilsTest, as suggested by SonarLint, because they are not needed with Junit 5.
2022-05-28 12:00:46 +02:00
TiA4f8R f6ec7f9a61
Update DefaultStreamExtractorTest and SoundcloudStreamExtractorTest to support changes made in Stream classes 2022-05-28 12:00:45 +02:00
TiA4f8R 7477ed0f3d
[YouTube] Add ability to generate manifests of progressive, OTF and post live streams
A new class has been added to do so: YoutubeDashManifestCreator.
It relies on a new class: ManifestCreatorCache, to cache the content, which relies on a new pair class named Pair.
Results are cached and there is a cache per delivery type, on which cache limit, clear factor, clearing and resetting can be applied to each cache and to all caches.
Look at code changes for more details.
2022-05-28 12:00:45 +02:00
TiA4f8R a857684442
Apply changes in YoutubeStreamExtractor
Extract post live DVR streams as post live streams instead of live streams.

A new class has been in order to improve code: ItagInfo, which stores an itag, the content (URL) extracted and if its an URL or not.
A functional interface has been added in order to abstract the stream building: StreamBuilderHelper.
Also add the cver parameter added by the desktop web client on the corresponding streams (a new method has been added in YoutubeParsingHelper to check this and another for Android streams).

Some code in these classes has been also refactored/improved/optimized.
2022-05-28 12:00:44 +02:00
TiA4f8R 4330b5f7be
Add POST_LIVE_STREAM and POST_LIVE_AUDIO_STREAM stream types
This allows the extractor to determine if a content is an ended audio or video livestream.
2022-05-28 12:00:43 +02:00
TiA4f8R 881969f1da
Apply changes in all StreamExtractors except YouTube's one and fix extraction of PeerTube audio streams as video streams
Some code in these classes has been also refactored/improved/optimized.
Also fix the extraction of PeerTube audio streams as video streams, which are now returned as audio streams.
2022-05-28 12:00:43 +02:00
TiA4f8R d5f3637fc3
[YouTube] Return more values returned inside the ItagItems of the player response and deprecate use of public audio and video fields
These fields can be now replaced by a getter and a setter.

New fields have been added and will allow the creation of DASH manifests for OTF and ended livestreams. There are:
- contentLength;
- approxDurationMs;
- targetDurationSec;
- sampleRate;
- audioChannels.
2022-05-28 12:00:42 +02:00
TiA4f8R 7c67d46e09
Move DashMpdParser to the YouTube package and fix extraction of streams
DashMpdParser is only working with YouTube streams, as it uses the ItagItem class.
Also update creation of AudioStreams and VideoStreams objects.
2022-05-28 12:00:41 +02:00
TiA4f8R ad993b920f
Remove fetching of the DASH manifest extracted when getting information of a content with StreamInfo
DashMpdParser is only working with YouTube streams, as it uses the ItagItem class.

Also improve code and comments of StreamInfo (especially final use where possible).
2022-05-28 12:00:41 +02:00
TiA4f8R 2f061b8dbd
Add support of other delivery methods than progressive HTTP in Stream classes
Stream constructors are now private and streams can be constructed with new Builder classes per stream class. This change has been made to prevent creating and using several constructors in stream classes.

Some default cases have been also added in these Builder classes, so not everything has to be set, depending of the service and the content.
2022-05-28 12:00:27 +02:00
Stypox 1dc80957d8
Merge pull request #848 from TiA4f8R/fix_yt_music_website_extraction_eu
[YouTube] Fix extracting YouTube Music client version and key in the EU
2022-05-22 22:05:05 +02:00
TiA4f8R c34b5e3a8b
[YouTube] Fix extraction of YouTube Music client version and API key when using YouTube Music's website in EU
Google returns now the consent page of YouTube for YouTube Music in EU, which can be also avoided by adding the ucbcb parameter to the URL with the value 1 ("?ucbcb=1").
2022-05-15 11:20:06 +02:00
litetex 2015eb374a Removed more unused methods 2022-05-09 21:05:03 +02:00
litetex f69b0ff77b Remove unused methods 2022-05-09 20:59:25 +02:00
Stypox 23fa31a1ec
Merge pull request #843 from TiA4f8R/fix-checkstyle-issues
Remove Checkstyle suppressions file and fix Checkstyle issues
2022-05-02 22:07:27 +02:00
TiA4f8R 3c3cd78676
Remove Checkstyle suppressions file and fix Checkstyle issues introduced in 24e8399 and 8c1041d
The Checkstyle suppressions file is now replaced by // CHECKSTYLE:OFF and // CHECKSTYLE:ON comments.
2022-05-02 21:51:25 +02:00
Stypox 2e1c5c119d
Merge pull request #822 from Stypox/more-refactors
More refactors
2022-05-02 19:03:54 +02:00
Stypox 598ebb92ea
Merge pull request #839 from TeamNewPipe/bandcamp/extract-length
Bandcamp: extract stream length
2022-05-02 15:49:41 +02:00
litetex 5db4d1faf3
Merge pull request #782 from litetex/cleanup-yt-stream-extractor
Cleanup of ``YoutubeStreamExtractor`` and some related classes
2022-05-01 16:44:11 +02:00
litetex fe30eb43a9 Cleanup ``YoutubeStreamExtractor`` and some related classes
* Fixed obvious sonar(lint) warnings
* Abstracted some code (get*Streams)
* Used some new lines to make code better readable
* Chopped down brace-jungle in some methods
* Use StandardCharset (Java 8 4tw)
2022-05-01 16:39:07 +02:00
Stypox c2b5370517
Apply suggestions: improve switch and use EMPTY_STRING 2022-04-30 16:39:51 +02:00
Stypox 7c78c39230
Merge pull request #821 from litetex/cleanup-TimeAgoParser-java
Cleanup ``TimeAgoParser``
2022-04-30 16:20:09 +02:00
Stypox ac1c22d81c
Release NewPipe Extractor 0.22.1 2022-04-28 11:05:52 +02:00
Stypox 3f5c8962ac
Merge pull request #841 from TiA4f8R/yt-respect-order-of-clients-for-streams
[YouTube] Use correct order of clients to get better streams first
2022-04-28 10:58:49 +02:00
TiA4f8R 9f9af35adb
[YouTube] Fix regression introduced in the order of streams used when adding more parameters to InnerTube requests, using the iOS client for livestreams and more 2022-04-25 20:23:04 +02:00
Fynn Godau c38c016de5 Bandcamp: extract stream length 2022-04-24 21:24:19 +02:00
litetex 5f8f3929bd
Merge pull request #838 from Nickoriginal/update_user_agent
Update User-Agent in NPE
2022-04-24 14:52:18 +02:00
Nickoriginal b5f3f9eb90
Update USER_AGENT to match the latest Firefox 2022-04-24 00:27:23 +03:00
litetex 9f665db4e1
Merge pull request #828 from TeamNewPipe/dependabot/github_actions/actions/setup-java-3
Bump actions/setup-java from 2 to 3
2022-04-23 00:09:44 +02:00
Stypox 52fa2d939a
Fix javadoc formatting error causing deployment to fail 2022-04-16 17:07:07 +02:00
Stypox 95a4b05acc
Release NewPipe Extractor 0.22.0 2022-04-16 16:57:58 +02:00
Tobi b77c72fb88
Merge pull request #832 from Stypox/hotfix-decrypter
Fix YouTube throttling decrypter function parsing
2022-04-15 13:35:10 +02:00
Stypox dcb7483dcf
Fix YouTube throttling decrypter function parsing 2022-04-15 13:10:19 +02:00
Robin bebdc55ad4
Merge pull request #829 from TiA4f8R/yt-subtitles-age-restricted-videos-fix
[YouTube] Extract subtitles for age-restricted videos
2022-04-14 10:31:16 +02:00
TiA4f8R ef49cd0007
[YouTube] Extract subtitles for age-restricted videos
Subtitles of age-restricted videos can be extracted since the InnerTube API migration, so there is no reason to not extract them anymore.
2022-04-11 22:09:56 +02:00
dependabot[bot] 00b51a2bf8
Bump actions/setup-java from 2 to 3
Bumps [actions/setup-java](https://github.com/actions/setup-java) from 2 to 3.
- [Release notes](https://github.com/actions/setup-java/releases)
- [Commits](https://github.com/actions/setup-java/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/setup-java
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-04-11 09:16:36 +00:00
TiA4f8R 2e92d718a2
Merge pull request #780 from TiA4f8R/yt-more-params-innertube-requests
[YouTube] Add more parameters to InnerTube requests, use the iOS client for livestreams and fix extraction of embeddable age-restricted videos and contents with a warning before playback
2022-04-09 18:15:18 +02:00
TiA4f8R b30e341559
Add link to learn more about the issue which makes YoutubeStreamExtractorRelatedMixTest.testRelatedItems test disabled 2022-04-04 20:36:59 +02:00
TiA4f8R 70812fa611
Update YouTube stream mocks and disable YoutubeStreamExtractorRelatedMixTest
Mixes seems to be not given by YouTube anymore if you use a PENDING consent cookie value.
As mocks needs to updated, the test is always failing because of this change.
2022-04-02 19:28:49 +02:00
TiA4f8R 67288a0191
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.

This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.

The outdated code for these contents has been removed.

Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.

Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 19:06:36 +02:00
TiA4f8R 11b5a222c4
Deduplicate code of getStringResultFromRegexArray methods in Utils
Also revert indentation in Utils.mixedNumberWordToLong.
2022-04-02 18:40:00 +02:00
TiA4f8R 9ca647a750
Update mocks 2022-03-27 22:10:59 +02:00
TiA4f8R dfa4239661
Fix missing imports and Checkstyle issues 2022-03-27 22:10:57 +02:00
TiA4f8R 6d27996ac4
Improve code of getStringResultFromRegexArray methods in Utils 2022-03-27 22:10:57 +02:00
TiA4f8R 2e3da445e6
[YouTube] Add documentation about parameters added and clients versions and key
Also move the iPhone device machine id to a constant, explain how it is used and move the licence in the header of the file, and fix missing imports in YoutubeStreamExtractor (due to a rebase issue).
2022-03-27 22:10:57 +02:00
TiA4f8R 1dad3bfe8b
[YouTube] Update again hardcoded client versions and update mobile user agents
Also provide ability to get mobile user-agents used for mobile InnerTube requests and deduplicate related code.
2022-03-27 20:52:40 +02:00
TiA4f8R 3d38459cf3
[YouTube] Reduce InnerTube response sizes by adding the prettyPrint parameter with the false value
InnerTube responses return pretty printed responses, which increase responses' size for nothing.

By using the prettyPrint parameter on requests and setting its value to false, responses are not pretty printed anymore, which reduces responses size, and so data transfer and processing times.
This usage has been recently deployed by YouTube on their websites.
2022-03-27 20:52:40 +02:00
litetex 349ba8db7f
Improve tests and randomness
- Use the existing RNG inside YoutubeParsingHelper
- Deduplicated test-setup for YouTube tests
- Minor improvements
2022-03-27 20:52:38 +02:00
TiA4f8R 52376949e5
Add setSeedForVideoTests method in YoutubeStreamExtractor tests
In order to use still use mocks with the generation of random strings in player requests, we need to use YoutubeParsingHelper.setSeedForVideoTests() method in every stream test.
2022-03-27 20:51:39 +02:00
TiA4f8R d0d91e6690
Adress requested changes 2022-03-27 20:51:39 +02:00
TiA4f8R b6bc521f0d
[YouTube] Update client versions again 2022-03-27 20:51:38 +02:00
TiA4f8R 26f93f5bb0
[YouTube] Extract streams of livestreams from the iOS client and disabled the Android client for livestreams
The iOS client is only enabled for livestreams and the Android client is now only enabled for videos, both by default.

A way to force, or not, the fetch of both clients have been added with two new static methods in YoutubeStreamExtractor.
2022-03-27 20:51:38 +02:00
TiA4f8R 7d07924de8
[YouTube] Try to use lighter requests when extracting client version and key from YouTube and YouTube Music
This is done by fetching https://www.youtube.com/sw.js for YouTube and https://music.youtube.com/sw.js for YouTube Music.

Two new methods in Utils class have been added which allow to try to get a match of regular expressions in a string array, or a Pattern array, on a content, on a specific index or 0.
Also some code refactoring has been made in this class.
2022-03-27 20:51:38 +02:00
TiA4f8R 05b7fee23b
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.

For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.

Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).

For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.

This commit also fixes a small bug with the Android User-Agent string.

Some code improvements have been also made.
2022-03-27 20:51:38 +02:00
TiA4f8R 83f374bff1
[YouTube] Update client versions and fix a bug when using resetClientVersionAndKey method
The boolean keyAndVersionExtracted in YoutubeParsingHelper was not set to false when resetting the client version and the key, which makes the extractor uses null on the next getting of the client version or the key if the clientVersion and the key were extracted before.
Also update client versions.
2022-03-27 20:51:38 +02:00
Stypox 8c1041def6
Add @ null annotations where Android Studio suggested it
That is, basically where the overriding function was missing an annotation from the base method.

Also apply renaming of emptyDescription to EMPTY_DESCRIPTION
2022-03-26 22:07:14 +01:00
Stypox adbbdc7a5b
[YouTube] Fix regex warning: use ' {2}' instead of ' ' 2022-03-26 22:07:14 +01:00
Stypox 24e83997b4
[Bandcamp] Add Java 8 streams 2022-03-26 22:07:12 +01:00
Stypox 349990fd48
Fix redundant escape \\ in regex in Utils 2022-03-26 22:01:30 +01:00
litetex 3bf7aa3762 Cleanup ``TimeAgoParser`` 2022-03-26 21:09:31 +01:00
litetex 5a18730845
Merge pull request #813 from litetex/fix-test-2022-03
Fixed tests (+ Youtube shorts in channels)
2022-03-26 21:09:10 +01:00
litetex af82edf9dc Fix checkstyle problems 2022-03-26 20:54:20 +01:00
litetex 29408f9356 ``YoutubeMusicSearchExtractorTest$Suggestion`` has no mock data 2022-03-26 20:52:28 +01:00
litetex fcee247f9f Switched to mockonly tests
* Made the reason why the tests are mockonly mandatory
2022-03-26 20:52:28 +01:00
litetex 0fceb4686b Added missing mock data 2022-03-26 20:52:27 +01:00
litetex 70b20f0d6b Updated mock data after conflicts 2022-03-26 20:52:27 +01:00
litetex 6a6c9359af Fixed kurzgesagt test - They changed their description... 2022-03-26 20:52:27 +01:00
litetex 26596215fa Refactored ``MediaCCCRecentListExtractorTest`` 2022-03-26 20:52:26 +01:00
litetex 53962bfd7d Disabled suggestion test for now
Has to be done once YT's backed works again...
2022-03-26 20:52:26 +01:00
litetex 66dc5e8bb8 API hardening against changes 2022-03-26 20:52:26 +01:00
litetex 5d58156cde Update mock data 2022-03-26 20:52:25 +01:00
litetex 7598b40957 Workaround for incorrect duration for "YT shorts" videos in channels
As a workaround 0 is returned as duration for such videos.
See also https://github.com/TeamNewPipe/NewPipe/issues/8034
2022-03-26 20:52:24 +01:00
litetex 164e21b5af Fixed ``MediaCCCRecentKiosk``
Ignore faulty data/items (with duration <= 0)
2022-03-26 20:52:23 +01:00
litetex 49dfdae5d2 Removed useless throw declaration 2022-03-26 20:51:38 +01:00
litetex 69a58fd3d1 Don't use sysout 2022-03-26 20:51:37 +01:00
litetex 639be7adda Minimized some code 2022-03-26 20:51:37 +01:00
litetex 2bd4299563 Fixed test: Peertube - Use new comments and videos
Old comments/videos got deleted
2022-03-26 20:49:39 +01:00
litetex 606a386a98 Fix tests: What's peertube has less comments now 2022-03-26 20:49:39 +01:00
litetex 4d1a1c8fb8 Better test for ``MediaCCCRecentListExtractorTest``
* Use assertAll
* Show which item is affected
2022-03-26 20:49:38 +01:00
litetex ba43dbaa28 Fix Tests: The channel lost abos 2022-03-26 20:49:38 +01:00
litetex ff436e5740 Fixed test: YT mix playlist 2022-03-26 20:49:38 +01:00
litetex bb49f7d857
Merge pull request #817 from Stypox/checkstyle
Add checkstyle
2022-03-26 20:46:49 +01:00
litetex 9c07e8a664 Fix useage of wrong object 2022-03-26 20:17:50 +01:00
litetex 93d6e5c4d5 CI: Upload test reports when failure occurs 2022-03-26 19:57:17 +01:00
litetex 804e57004f Fixed new checkstyle problems from dev 2022-03-26 19:46:10 +01:00
litetex 33347ac18b Removed unused methods
``contentFilters`` and ``sortfilter`` are get inside the ``ListLinkHandler`` and not the ``ListLinkHandlerFactory``
 ``ListLinkHandlerFactory`` only passes these values through when ``fromQuery`` is called
2022-03-26 19:43:11 +01:00
litetex ec5b54c38b Removed unused class 2022-03-26 19:43:10 +01:00
litetex 8771af7ba5 Restored original naming 2022-03-26 19:43:09 +01:00
litetex 01cfde0285 Fixed typo 2022-03-26 19:43:09 +01:00
Stypox bdadcfa1f7 Legitimately suppress remaining checkstyle warnings 2022-03-26 19:43:08 +01:00
Stypox 740a37a2de [YouTube] Fix checkstyle issues 2022-03-26 19:42:40 +01:00
Stypox 9dc17cd1ca [Soundcloud] Fix checkstyle issues 2022-03-26 19:40:20 +01:00
Stypox 9ab32cb2e7 [Peertube] Fix checkstyle issues 2022-03-26 19:40:19 +01:00
Stypox 9f7e06c817 [MediaCCC] Fix checkstyle issues 2022-03-26 19:40:18 +01:00
Stypox 3a94839359 [Bandcamp] Fix checkstyle issues 2022-03-26 19:40:17 +01:00
Stypox 08dff33002 Use Java 8 streams in NewPipe class 2022-03-26 19:40:15 +01:00
Stypox c2446ecff0 Use Java 8 streams and deduplicate code in MediaFormat class 2022-03-26 19:40:15 +01:00
Stypox d79e20340c Fix checkstyle issues in root package extractor/
Note: not all issues were fixed because MediaFormat and ServiceList use a specific formatting that makes sense for them
2022-03-26 19:40:14 +01:00
Stypox ca7c63f273 Fix remaining checkstyle issues in utils/ subpackage 2022-03-26 19:40:13 +01:00
Stypox 1d5f22e41f Fix checkstyle issues & more in JsonUtils
Also use Java 8 streams and extract duplicate code to getInstanceOf function
2022-03-26 19:40:13 +01:00
Stypox 87d2834986 Fix checkstyle issues & more in DonationLinkHelper
Also add comment about the class being unused and replace the fixLink function with Utils.stringToUrl()
2022-03-26 19:40:12 +01:00
Stypox bd7b362040 Fix checkstyle issues & more in DashMpdParser
Also remove useless null check on ItagItem.getItag() as that function already throws an exception if there is no itag
2022-03-26 19:40:11 +01:00
Stypox 8aba2b47b0 Fix checkstyle issues in subpackages with abstract classes 2022-03-26 19:40:10 +01:00
Stypox e4951a0623 Refactor code handling http headers in downloader.Request 2022-03-26 19:37:47 +01:00
Stypox 37690058d2 Add checkstyle to extractor gradle project
With respect to NewPipe's checkstyle.xml, checkstyle is disabled for javadoc comments. There is no need for strict rules over comments here in the extractor, as sometimes javadocs are just needed to clarify a small thing and having empty/meaningless @param or @throws is useless.
2022-03-26 19:37:46 +01:00
litetex 9284569c84
Merge pull request #774 from TeamNewPipe/dependabot/gradle/org.mozilla-rhino-1.7.14
Bump rhino from 1.7.13 to 1.7.14
2022-03-26 17:25:55 +01:00
litetex 358e619a27
Merge pull request #819 from TeamNewPipe/dependabot/github_actions/actions/cache-3
Bump actions/cache from 2 to 3
2022-03-23 21:02:18 +01:00
dependabot[bot] a2c9ad104c
Bump actions/cache from 2 to 3
Bumps [actions/cache](https://github.com/actions/cache) from 2 to 3.
- [Release notes](https://github.com/actions/cache/releases)
- [Commits](https://github.com/actions/cache/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-21 09:12:36 +00:00
XiangRongLin aa6b7272a4
Merge pull request #804 from Stypox/fix-yt-music-mix
[YouTube] Fix music mixes in some countries
2022-03-20 08:35:56 +01:00
Stypox 843945f2a6
Merge pull request #818 from litetex/update-license
Update the license to the latest version
2022-03-19 23:13:55 +01:00
litetex fa76098750
Delete copyright
was replace by LICENSE-file
2022-03-19 21:44:38 +01:00
litetex faaf4cbec3
Update the license to the latest version of https://www.gnu.org/licenses/gpl-3.0.txt 2022-03-19 17:59:27 +01:00
XiangRongLin 7f2ea133f0
Merge pull request #788 from Stypox/mix
Add MixInfoItem and extract YouTube mixes in related items
2022-03-19 16:46:01 +01:00
Stypox 09ddb6adbb
[YouTube] Add MockOnly to method testing mixes in related items 2022-03-19 10:54:38 +01:00
Stypox 8201b3b90e
[YouTube] Parse any playlist (including music mixes) in related items 2022-03-19 10:48:13 +01:00
Stypox 13f7900816
[YouTube] Add test for genre mix 2022-03-19 10:48:13 +01:00
Stypox 279f3a20fe
[YouTube] Fix mix tests with invalid video ids
Replaces mix tests based on a strange mix type RDQM{videoId} (only reference I could find is https://github.com/ytdl-org/youtube-dl/issues/26228) and with an invalid video id of 13 characters (the first two characters were QM, but even after removing QM there still wasn't a video available at that id).
Also updates mocks.
2022-03-19 10:48:13 +01:00
Stypox d660c04838
[YouTube] Also test playlist type in playlist tests 2022-03-19 10:48:13 +01:00
Stypox 401082abe4
[YouTube] Extract playlist type in playlist extractor 2022-03-19 10:48:12 +01:00
Stypox 63ed06a710
[YouTube] Differentiate genre mixes from normal mixes
Note: genre mixes already worked, now they are just considered as such in various video id extraction and in related items
Note 2: now extracting a mix id from a *normal* youtube mix id will fail if the video id wouldn't be exactly 11 characters long
2022-03-19 10:46:31 +01:00
Stypox f19660e7d9
[YouTube] Deduplicate code extracting video id from mix id 2022-03-19 10:46:30 +01:00
Stypox 8f9d5b858e
[YouTube] Remove useless comments about mixes 2022-03-19 10:44:06 +01:00
Stypox 34a4484c72
[YouTube] Add test for a video with a mix in related items 2022-03-19 10:44:06 +01:00
Stypox 50db871d89
[YouTube] Extract mixes from streams related items 2022-03-19 10:44:06 +01:00
Stypox 638da1756c
[Mix] Create MultiInfoItemsCollector
It is a collector that can handle many extractor types, to be used when a list contains items of different types (e.g. search). It was renamed from InfoItemsSearchCollector so that it can now be used not just for search but for any extractor needing it. It supports, streams, channels, playlists and *mixes*.
2022-03-19 10:44:06 +01:00
Stypox 53673d64c6
[Mix] Add type to playlists & playlist items, to distinguish mixes 2022-03-19 10:44:06 +01:00
Stypox d8f2031619
Merge pull request #816 from Stypox/mock-only-extension
Add `@MockOnly` Junit 5 extension
2022-03-19 10:40:38 +01:00
litetex cc2e4d7104
Merge pull request #815 from litetex/fix-soundcloud-id-once-and-for-all
Removed hardcoded soundcloud HARDCODED_CLIENT_ID
2022-03-17 13:54:08 +01:00
litetex cd8088b217
Merge pull request #811 from TiA4f8R/playlists-improvements-and-yt-playlists-fixes
[YouTube] Fix the extraction of series playlists and don't return the view count as the stream count for learning playlists
2022-03-17 13:52:24 +01:00
TiA4f8R c7757c0994
Apply requested changes 2022-03-16 20:14:08 +01:00
TiA4f8R 35e082248e
Fix YouTube and SoundCloud playlists tests 2022-03-16 19:40:30 +01:00
TiA4f8R 8b3f90eb7e
[YouTube] Fix extraction of series playlists and don't return the view count as the stream count for learning playlists
ITEM_COUNT_UNKNOWN is returned when the JSON array which contains usally the number of videos is less than 3 items.
Also apply the same type of optimizations done in other PlaylistExtractors in YoutubePlaylistExtractor.
2022-03-16 19:18:58 +01:00
TiA4f8R 58a247907e
Apply changes in all playlist extractors except YoutubePlaylistExtractor
Also fix some issues in the extractors, remove uneeded overrides, use the Java 8 Stream API where possible and replace usages of Utils.UTF_8 with StandardCharsets.UTF_8 in these classes.
2022-03-16 19:18:57 +01:00
TiA4f8R fc6b45ee36
Implement some methods in PlaylistExtractor
This will prevent their override in each child class where the values corresponding to the methods could not be extracted.
2022-03-16 19:18:36 +01:00
Stypox 73d1fd472f
Add MockOnly junit 5 test extension 2022-03-16 19:03:08 +01:00
Stypox ef71a5fa0f
static final instead of final static 2022-03-16 17:24:33 +01:00
Stypox 0c37c75981
Make getDownloader static & extract getDownloaderType 2022-03-16 17:22:42 +01:00
Stypox 40aa5104b1
Merge pull request #786 from XiangRongLin/throttling_resilience
[Youtube] Make throttling decryption more resilient to api change
2022-03-16 11:03:16 +01:00
litetex ba56be8ef1 Removed hardcoded soundcloud id
It never works (long enough) so let's simply remove it...
2022-03-15 21:19:19 +01:00
XiangRongLin e726437da3
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java
Co-authored-by: Stypox <stypox@pm.me>
2022-03-15 17:10:05 +01:00
litetex e7aee0ca57
Merge pull request #807 from FireMasterK/no-commentsinfo-instance
Remove the need for a CommentsInfo instance in CommentsInfo.getMoreItems and fix PeertubeCommentsExtractorTest.Default test
2022-03-15 15:06:56 +01:00
litetex d806984aa8
Merge pull request #800 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.9.0
Bump gson from 2.8.9 to 2.9.0
2022-03-14 19:47:54 +01:00
litetex 2aa5f98c26
Merge pull request #796 from FireMasterK/streaminfo-subscriber-count
Add support for extracting channel subscriber count in StreamInfo
2022-03-14 19:40:57 +01:00
Tobi a267093e76
Merge pull request #809 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.6.0
Bump spotbugs-annotations from 4.5.3 to 4.6.0
2022-03-12 00:02:51 +01:00
dependabot[bot] fdb603e9ac
Bump spotbugs-annotations from 4.5.3 to 4.6.0
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.5.3 to 4.6.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.5.3...4.6.0)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-08 09:16:28 +00:00
FireMasterK 60cc71e944
Remove the need for a CommentsInfo instance. 2022-03-03 11:48:41 +00:00
Stypox 3a6269bca6
Merge pull request #805 from TeamNewPipe/dependabot/github_actions/actions/checkout-3
Bump actions/checkout from 2 to 3
2022-03-02 11:09:28 +01:00
dependabot[bot] e953794703
Bump actions/checkout from 2 to 3
Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-02 09:25:59 +00:00
Stypox dd8687f9fe
[YouTube] Fix music mixes in some countries 2022-03-01 23:02:56 +01:00
FireMasterK e6d334765d
Apply requested codestyle improvements. 2022-02-26 17:12:51 +00:00
FireMasterK 6950b362f2
Add test values for failing tests. 2022-02-24 12:55:13 +00:00
FireMasterK 5b0ec694a6
Add requested changes. 2022-02-24 12:50:51 +00:00
FireMasterK d290d2e393
Move variable to super Extractor classes. 2022-02-24 12:50:51 +00:00
FireMasterK ab49cb6e18
Add requested changes. 2022-02-24 12:50:51 +00:00
Kavin 4bd59f65f4
Update exception message.
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2022-02-24 12:50:51 +00:00
FireMasterK 6f3f608ab6
Add support for extracting channel subscriber count in StreamInfo. 2022-02-24 12:50:51 +00:00
Stypox 69e18c80cb
Merge pull request #799 from litetex/imporve-yt-comments-extractor
Refactored YoutubeCommentsExtractor
2022-02-23 17:58:00 +01:00
litetex 1a67ea100a Refactored code
according to review
2022-02-21 19:56:12 +01:00
litetex f79ce1f52a Refactored YoutubeCommentsExtractor
* Use Java Streaming API
* Use StandardCharsets
* Prevented several NPEs/ArrayIndexOutOfBound
* Reformatted some code so that it's easier readable
2022-02-21 19:56:09 +01:00
TobiGr dfe8716f5f Merge branch 'dev' 2022-02-18 20:30:38 +01:00
TobiGr d07e16aadd Release NewPipe Extractor 0.21.14 2022-02-18 20:29:57 +01:00
TobiGr d337e537dc [YouTube] Add a comment clarifying why the panelIdentfier can be null 2022-02-18 20:26:19 +01:00
Tobi 5c8fbeb166
Fix NullPointerException in YouTubeStreamExtractor.getStreamSegments() (#797)
Fix NullPointerException in YouTubeStreamExtractor.getStreamSegments()

Co-authored-by: litetex <40789489+litetex@users.noreply.github.com>
2022-02-16 23:39:26 +01:00
dependabot[bot] 89900431ab
Bump gson from 2.8.9 to 2.9.0
Bumps [gson](https://github.com/google/gson) from 2.8.9 to 2.9.0.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.9...gson-parent-2.9.0)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-02-14 09:15:09 +00:00
Tobi 0d6aebce25
[SoundCloud] Update clientId (#798) 2022-02-11 17:16:32 +01:00
dependabot[bot] a2a41e3bf2
Bump rhino from 1.7.13 to 1.7.14
Bumps [rhino](https://github.com/mozilla/rhino) from 1.7.13 to 1.7.14.
- [Release notes](https://github.com/mozilla/rhino/releases)
- [Changelog](https://github.com/mozilla/rhino/blob/master/RELEASE-NOTES.md)
- [Commits](https://github.com/mozilla/rhino/commits)

---
updated-dependencies:
- dependency-name: org.mozilla:rhino
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-02-07 20:29:51 +00:00
XiangRongLin 545522b80f [Youtube] Make throttling decryption more resilient to api change 2022-02-01 19:57:34 +01:00
TobiGr ebc129c392 NewPipe Extractor 0.21.13 2022-02-01 13:20:34 +01:00
TobiGr 5306f7a988 Improve code style 2022-02-01 13:18:18 +01:00
Abdu Ameen 38e9da8d44 Adding removal of the braces 2022-02-01 13:18:10 +01:00
Abdu Ameen 36ebb633a4 Changed the regex to account for
nonword characters
2022-02-01 13:18:05 +01:00
Tobi 65129e65a8
Merge pull request #785 from AbduAmeen/regex_error
Regex error when parsing the Youtube JavaScript code
2022-02-01 13:17:19 +01:00
TobiGr 047d75a4c9 Improve code style 2022-02-01 12:56:59 +01:00
Abdu Ameen 50a4b026f8 Adding removal of the braces 2022-01-31 23:52:31 -08:00
Abdu Ameen a9174f6b9f Changed the regex to account for
nonword characters
2022-01-31 22:29:00 -08:00
dependabot[bot] a4399fd4ad Bump spotbugs-annotations from 4.5.2 to 4.5.3
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.5.2 to 4.5.3.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.5.2...4.5.3)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-01-24 18:45:21 +01:00
Stypox 59e0aeba88
Merge pull request #768 from litetex/junit-5
JUnit 5
2022-01-24 16:56:16 +01:00
Mohammed Anas 65df39b3e3
[YouTube] parse timestamps with >3 digits correctly (#775)
* [YouTube] parse timestamps with >3 digits correctly

Fixes https://github.com/TeamNewPipe/NewPipe/issues/7530; check the
issue for details.

* Remove outdated comment
2022-01-12 10:45:06 +01:00
litetex a579ef2651 Fixed Soundcloud tests 2022-01-07 14:27:24 +01:00
litetex a32edce750 Removed deprecation because AndroidSDK/API < 19 2022-01-07 14:15:55 +01:00
litetex 602068937d
Code cleanup
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2022-01-06 21:04:16 +01:00
litetex dd3f3da828 Kurzgesagt mock data is outdated 2022-01-04 17:44:12 +01:00
litetex a8ebd2cead Update mock resources for YT Mix playlist 2022-01-04 17:39:14 +01:00
litetex 066afd8629 Restored test 2022-01-04 17:33:08 +01:00
litetex 3712a669b1 Code cleanup / improvements
* Deprecated Utils#UTF-8; see StandardCharsets
* Added more helpful methods to ``ExtractorAsserts``
* Use parameterized (cool new) tests
* Restore functionality of some tests + updated mockdata
* Other code cleanups + Sonarlint improvements
2022-01-04 17:28:31 +01:00
litetex 4291a90251 Fixed SoundCloud's search(for tests) 2021-12-27 21:42:43 +01:00
litetex 4995709871 Fixed SoundCloud's search(for tests)
Getting the initial page was not returning initial page
2021-12-27 21:18:39 +01:00
litetex a2cbdf0991 Updated to JUnit 5 2021-12-27 21:08:08 +01:00
litetex 10f6cc7194
Merge pull request #755 from litetex/clean-up-yt-dis-likes
Cleanup: Remove old ways of getting YT dis/likes
2021-12-27 18:32:02 +01:00
litetex 148ba6641d Updated mock data 2021-12-27 17:08:14 +01:00
litetex 4d0a689cb6 Disabled dislike count in tests 2021-12-27 17:04:46 +01:00
litetex a6961c5efa Fixed NumberFormatException when processing likes
See also #7525
2021-12-27 16:54:47 +01:00
litetex 65687f3b9b Removed YT dislike count as it no longer works 2021-12-27 16:54:46 +01:00
litetex 1e2e0029fc [StreamExtractor] Deduplicated a ton of code by using default methods 2021-12-27 16:54:45 +01:00
litetex 15b98ffdb4 Remove old ways of getting YT dis/likes
* Added additional check for averageRating (in dislikes)
2021-12-27 16:54:44 +01:00
dependabot[bot] fe432425df Bump spotbugs-annotations from 4.5.1 to 4.5.2
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.5.1 to 4.5.2.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.5.1...4.5.2)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-12-14 10:35:32 +01:00
dependabot[bot] 36663dcb37 Bump spotbugs-annotations from 4.5.0 to 4.5.1
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.5.0 to 4.5.1.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.5.0...4.5.1)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-12-12 13:28:30 +01:00
litetex 12a6a48b15
Merge pull request #761 from litetex/fix-soundcloud-georestricted
Disabled ``SoundcloudStreamExtractorTest#SoundcloudGeoRestrictedTrack#testRelatedItems`` as it's unreliable
2021-12-11 18:06:17 +01:00
litetex 7484b6a098
Merge pull request #758 from litetex/release/v0.21.12
Release 0.21.12
2021-12-11 16:14:11 +01:00
litetex 795e6df596
Update README.md
Made usage section more abstract
2021-12-11 16:06:29 +01:00
litetex 4dd737c9a0 Updated version to v0.21.12 2021-12-11 15:34:46 +01:00
litetex 7489502cc8 Disabled ``testRelatedItems`` as it's unreliable 2021-12-11 15:23:11 +01:00
Tobi 1e0c409bec
Merge pull request #760 from litetex/fix-test
Fix test
2021-12-11 12:44:45 +01:00
litetex d9b2a2042a Updated mock data for YoutubeCommentsExtractorTest 2021-12-08 22:11:45 +01:00
litetex a3a0c1ee85 Fixed YoutubeCommentsExtractorTest
Old video was deleted / set to private
2021-12-08 22:11:34 +01:00
litetex 5c288eb8d7 Updated mock data for YoutubeChannelExtractorTest 2021-12-08 21:51:38 +01:00
litetex 75bb83a1b8 Fixed YoutubeChannelExtractorTest
Channel was restored, picked a new one
2021-12-08 21:51:15 +01:00
litetex f276bacce5 Fixed SoundcloudStreamExtractorTest 2021-12-08 21:41:27 +01:00
litetex 7f99c1c193 Fixed YoutubeStreamExtractorLivestreamTest 2021-12-08 21:16:39 +01:00
litetex 652fdf2c36 Updated mock-data for YoutubeStreamExtractorAgeRestrictedTest 2021-12-08 21:08:25 +01:00
litetex 3c669d3cb5 Fixed YoutubeStreamExtractorAgeRestrictedTest
Looks like a lot of stuff (description, tags, name) got changed on the existing video.
Use a better suited video.
2021-12-08 21:08:03 +01:00
litetex f735788a39 Fixed Soundcloud hardcoded client id 2021-12-08 21:00:53 +01:00
litetex f6f2724634
Merge pull request #753 from B0pol/dis-likes
Fix YouTube likes + dislikes
2021-11-29 20:42:17 +01:00
bopol 56c8af710e Code refactoring 2021-11-21 18:16:31 +01:00
bopol c4eca91be9 Fix YouTube likes + dislikes 2021-11-19 21:36:03 +01:00
TobiGr 5028396405 Improve names and formatting 2021-11-14 13:44:13 +01:00
dependabot[bot] 1e0c802d8d Bump spotbugs-annotations from 4.4.2 to 4.5.0
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.4.2 to 4.5.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.4.2...4.5.0)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-11-14 13:43:44 +01:00
Tobi 68d7e4e20a
Merge pull request #742 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.8.9
Bump gson from 2.8.8 to 2.8.9
2021-11-08 19:37:09 +01:00
bopol 8e4f9995bf
Merge pull request #745 from TeamNewPipe/update_SC_hardcoded_client_id
Update hardcoded SoundCloud client_id
2021-11-04 16:33:08 +01:00
opusforlife2 cb80a646d9
Update hardcoded SoundCloud client_id
Hah! I got here first, @TobiGr!
2021-11-04 15:28:26 +00:00
litetex 7e7b78f1b3
Merge pull request #743 from B0pol/peertube-shortlinks
PeerTube short links support
2021-11-04 16:10:23 +01:00
bopol 396aecef19 Make sure playlists aren't accepted by PeertubeStreamLinkHandlerFactory 2021-11-03 14:41:39 +01:00
bopol 38ad1eaac5 [PeerTube] Support /w/ short video links part 2
We also need to support it for comments
2021-11-03 13:54:37 +01:00
bopol 026751624f [PeerTube] Support /w/p/ short playlist links 2021-11-03 13:35:30 +01:00
bopol e0b8e142fc [PeerTube] Support /a/ and /c/ short links 2021-11-03 13:26:27 +01:00
bopol 599a91c88c [PeerTube] Support /w/ short video links 2021-11-03 11:51:58 +01:00
dependabot[bot] 198d37090b
Bump gson from 2.8.8 to 2.8.9
Bumps [gson](https://github.com/google/gson) from 2.8.8 to 2.8.9.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.8...gson-parent-2.8.9)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 09:12:53 +00:00
litetex 4f60225ddc
Merge pull request #741 from mhmdanas/add-y2ube-link-support
Add support for y2u.be links
2021-10-26 18:43:36 +02:00
mhmdanas 3e8e2a1532 Add support for y2u.be links 2021-10-22 22:48:18 +03:00
Tobi b425394e7a
Merge pull request #731 from FireMasterK/short-description
Extract Video Short Description in YouTube.
2021-10-15 23:10:00 +02:00
Tobi 4e9d9bffd0
Merge pull request #738 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.4.2
Bump spotbugs-annotations from 4.4.1 to 4.4.2
2021-10-12 12:09:36 +02:00
dependabot[bot] 1867a381a8
Bump spotbugs-annotations from 4.4.1 to 4.4.2
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.4.1 to 4.4.2.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.4.1...4.4.2)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-10-12 09:11:11 +00:00
FireMaskterK 62313962a0 Add default implementation for getShortDescription. 2021-10-04 01:42:29 +01:00
Kavin d1add05bc1 Add requested changes.
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
Add final.

Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-10-04 01:36:51 +01:00
Tobi 0b0c39aec2
Merge pull request #734 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.14.3
Bump jsoup from 1.14.2 to 1.14.3
2021-10-02 16:49:34 +02:00
TobiGr 8dcb74724a Release NewPipe Extractor 0.21.11 2021-10-02 16:25:35 +02:00
dependabot[bot] 3ca9f51b30
Bump jsoup from 1.14.2 to 1.14.3
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.14.2 to 1.14.3.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.14.2...jsoup-1.14.3)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-10-02 08:20:38 +00:00
TobiGr d8ec3ce370 [SoundCloud] Update client id 2021-10-01 02:51:38 +02:00
FireMasterK 94efe86c71
Extract Video Short Description in YouTube.
In Trending, and Search results.
2021-09-22 16:26:16 +01:00
Tobi a9d214478d
Merge pull request #703 from FireMasterK/comment-replies
Add support for extracting comment replies continuation
2021-09-14 23:58:14 +02:00
TobiGr ce8cabb9f8 Release NewPipe Exttractor 0.21.10 2021-09-13 13:59:35 +02:00
FireMasterK 6aabdc6d16
Fix for requested changes. 2021-09-12 01:15:19 +05:30
Kavin 0aad09fa22
Update JavaDocs for comment replies continuation.
Co-authored-by: Stypox <stypox@pm.me>
2021-09-12 01:05:20 +05:30
Tobi a031289a63
Merge pull request #727 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.4.1
Bump spotbugs-annotations from 4.4.0 to 4.4.1
2021-09-10 18:25:27 +02:00
dependabot[bot] 82ae322cb8
Bump spotbugs-annotations from 4.4.0 to 4.4.1
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.4.0 to 4.4.1.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.4.0...4.4.1)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-10 09:11:27 +00:00
Tobi 62b87552f5
Merge pull request #725 from FireMasterK/related-uploader-avatar
Extract Channel Avatar in related videos.
2021-09-03 22:14:28 +02:00
FireMasterK feb09e9997
Extract Channel Avatar in related videos. 2021-09-04 01:09:40 +05:30
Tobi f0aa46b008
Merge pull request #724 from FireMasterK/streams-uploader-verified
Extract uploaderVerified to StreamInfo.
2021-09-03 18:18:04 +02:00
Tobi c8037f5e00
Merge pull request #723 from FireMasterK/uploader-avatar
Extract Uploader's Avatar on YouTube and PeerTube
2021-09-03 18:13:59 +02:00
FireMasterK 857b2f39e8
Extract uploaderVerified to StreamInfo. 2021-09-03 21:27:58 +05:30
FireMasterK 1db463b55f
Add a default test for the Uploader's Avatar. 2021-09-03 21:05:36 +05:30
Kavin db6b3b2c29
Extract uploader's avatar in peertube.
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2021-09-02 16:14:45 +05:30
FireMasterK a0c1dcc8d8
Remove throws parsing exception. 2021-09-02 16:14:45 +05:30
FireMasterK 0ba03c552c
Improve based on review/feedback. 2021-09-02 00:35:56 +05:30
FireMasterK b9fad4fcc8
Extract Uploader's Avatar on YouTube. 2021-09-02 00:19:00 +05:30
Stypox 68f1fa994a
Merge pull request #721 from Stypox/yt-csv-import
[YouTube] Small improvements to subscription import
2021-08-31 12:02:59 +02:00
TobiGr b644f1d67b [SoundCloud] Update client id 2021-08-31 07:42:21 +02:00
Stypox a5e9eeb790
[YouTube] Small improvements to subscription import 2021-08-30 15:45:45 +02:00
Tobi 9570882c73
Merge pull request #712 from magicbrothers/add-eduvid-tubus
Add invidious instance EduVid Tubus
2021-08-24 16:07:07 +02:00
Stypox db6c729678
Merge pull request #709 from talanc/dev
[YouTube] Support csv and zip subscription import (Google Takeout)
2021-08-24 15:46:03 +02:00
talanc 94a29fd63f
[YouTube] csv and zip subscription import (Google Takeout)
csv:
Improved error messages
Exits early if it hasnt found any items in the first few lines

zip:
Now checks all CSV files instead of hard-coded paths

final qualifiers for immutable locals and parameters

Co-authored-by: litetex <40789489+litetex@users.noreply.github.com>
2021-08-24 15:42:49 +02:00
dependabot[bot] 816bc7137f Bump gson from 2.8.7 to 2.8.8
Bumps [gson](https://github.com/google/gson) from 2.8.7 to 2.8.8.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.7...gson-parent-2.8.8)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-08-23 13:04:10 +02:00
TobiGr 6d487e0dad Bump version to 0.21.9 2021-08-22 19:29:22 +02:00
Tobi 7823dcd99d
Merge pull request #715 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.4.0
Bump spotbugs-annotations from 4.2.3 to 4.4.0
2021-08-22 16:41:20 +02:00
dependabot[bot] d75d54e1c6
Bump spotbugs-annotations from 4.2.3 to 4.4.0
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.2.3 to 4.4.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.2.3...4.4.0)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-08-22 07:12:54 +00:00
dependabot[bot] 72393995bb
Bump jsoup from 1.13.1 to 1.14.2 (#716)
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.13.1 to 1.14.2.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.13.1...jsoup-1.14.2)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-22 09:12:14 +02:00
Stypox d8179dc35a
Merge pull request #713 from TiA4f8R/regexes-yt-signatureciphers
[YouTube] Consolidate the regular expressions used to find the cipher decryption function
2021-08-17 18:58:27 +02:00
Tobi 89bab241b9
Merge pull request #717 from XiangRongLin/fix_tests
Fix failing PeertubeStreamExtractorTest
2021-08-17 17:48:28 +02:00
XiangRongLin 61acdae4b8 Fix failing PeertubeStreamExtractorTest 2021-08-17 17:06:36 +02:00
FireMasterK 4f86b8ce43
Add verified info to ChannelInfo. (#714) 2021-08-17 13:08:49 +02:00
Stypox 52910ce970
Keep original exception when there is an error message (#710) 2021-08-13 16:13:58 +02:00
TiA4f8R f8309d546a
[YouTube] Consolidate current regexes of the cipher decryption function
Also add more regexes
2021-08-12 18:26:43 +02:00
magicfelix 0e16091ce0
Add invidious instance EduVid Tubus 2021-08-12 10:06:41 +02:00
Tobi b6f951edb4
Merge pull request #711 from gardenappl/dev
Fix minor typos in Javadoc
2021-08-11 18:16:34 +02:00
gardenapple 50bba32239
Fix typos in Javadoc 2021-08-11 12:01:25 +03:00
Tobi 78f2192cbb
Merge pull request #687 from TacoTheDank/gradle7
Upgrade to Gradle 7.1.1
2021-08-08 12:38:29 +02:00
TacoTheDank 4a7c1723a6 Use new 'maven-publish' plugin 2021-08-07 17:51:32 -04:00
TacoTheDank 8919a53d95 Update gradle wrapper to 7.1.1 2021-08-07 17:51:18 -04:00
TacoTheDank cf9e678223 Declare common dependency versions in root gradle 2021-08-07 17:51:09 -04:00
FireMasterK 71b9fd0076
Faster iframe api based player extraction. (#694)
* Faster iframe api based player extraction.

Uses the IFrame API to reduce the required download to less than 1/50 of the size.

* Remove debug code.

* Extract to two methods.

* Add tests for player URL extraction.

* Add assertThat for tests.
2021-08-07 12:35:48 +02:00
Tobi 4b147863ec
Merge pull request #705 from FireMasterK/suppored-countries
Update supported countries list for YouTube.
2021-08-06 10:14:27 +02:00
FireMasterK ee32317315
Update supported countries list for YouTube. 2021-08-05 23:48:38 +05:30
FireMasterK e8e9e6cd00
Fix comment reply continuations. 2021-08-04 22:14:04 +05:30
FireMasterK 666b45958e
Fix test and update mocks. 2021-08-04 21:10:02 +05:30
FireMasterK f6d054e5da
Add test for comments. 2021-08-04 21:03:00 +05:30
FireMasterK ed84658055
Get it working. 2021-08-04 18:35:14 +05:30
FireMasterK ea7e6526fd
Collect replies/page to CommentsInfoItem. 2021-08-04 18:09:42 +05:30
FireMasterK 4385cc1b7c
Add comment replies. 2021-08-04 17:56:04 +05:30
Stypox 8ca5e5e364
Merge pull request #702 from TeamNewPipe/master
This merges master into dev
2021-08-03 20:52:04 +02:00
Stypox bb3815d19b
Merge pull request #701 from Stypox/v0.21.8
V0.21.8
2021-08-03 20:43:15 +02:00
Stypox 5b38b3ae97
Merge branch 'master' into v0.21.8 2021-08-03 20:38:07 +02:00
Stypox 4b5907e18c
Release version 0.21.8 2021-08-03 20:33:24 +02:00
Stypox 5a88263785
Merge pull request #604 from TiA4f8R/youtubei-api
[YouTube] Use the new internal API in NewPipe Extractor
2021-08-03 19:01:14 +02:00
TiA4f8R 79cc5c8d12
Fix tests of YoutubePlaylistExtractorTest and YoutubeMixPlaylistExtractorTest
Without removing RunWith and SuiteClasses annotations (and the corresponding imports) in YoutubePlaylistExtractorTest and YoutubeMixPlaylistExtractorTest, some mocks cannot be generated, so the CI fails because of the missing mocks. Mocks of workings tests have been also updated.
2021-08-01 17:28:01 +02:00
TiA4f8R b74a39c176
Reformat some code and don't use the clickTrackingParams in continuations of YouTube Music search results
The clickTrackingParams of YouTube Music search results are not needed to get continuations. This commit removes their use, which may improve privacy.
2021-08-01 17:25:59 +02:00
TiA4f8R f8197da09e
Update YouTube comments and YouTube playlist continuations mocks 2021-08-01 12:39:05 +02:00
FireMasterK f3e4c9d689
Use the youtubei API for YouTube comments
Migrate YouTube comments to the desktop version by using the `next` endpoint of the InnerTube internal API.
With the desktop version, we are able to get the exact like count of YouTube comments (by parsing the accessibility data) (the current extraction is used as a fallback). We are also now able to get if the uploader of the comment is verified or not.

Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-08-01 12:39:05 +02:00
TiA4f8R 286d839a3b
Update mocks (except mocks of YoutubeCommentsExtractorTest) 2021-08-01 12:39:04 +02:00
TiA4f8R 693f654f02
Ignore a broken test of YoutubeMixPlaylistExtractorTest 2021-08-01 12:39:03 +02:00
TiA4f8R d55109258d
Fix YoutubeStreamExtractorAgeRestrictedTest 2021-08-01 12:39:03 +02:00
FireMasterK 2eeb0a3403
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:

- the desktop API is fetched.

If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status

If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.

If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.

We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.

Other code changes:

- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed

Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-08-01 12:39:03 +02:00
TiA4f8R 7753556e66
Adress the last requested changes + update YoutubeCommentsExtractor mocks 2021-08-01 12:39:03 +02:00
TiA4f8R 2320aecb7c
Update mocks 2021-08-01 12:39:02 +02:00
TiA4f8R 629dcd63b2
Ignore some broken tests 2021-08-01 12:39:01 +02:00
TiA4f8R 3adac6a150
Workaround for rate limits: always use the Android mobile API 2021-08-01 12:39:01 +02:00
TiA4f8R 8aa60d7e8f
Update clients version 2021-08-01 12:39:01 +02:00
TiA4f8R 27bd797f1e
Update mocks 2021-08-01 12:39:01 +02:00
TiA4f8R 609919db59
Adress again reviews, fix some rebase issues 2021-08-01 12:39:00 +02:00
TiA4f8R 4299d806a2
Adress changes 2021-08-01 12:38:59 +02:00
TiA4f8R 1a6b8da438
Annotate YoutubeParsingHelper methods with Nonnull when needed 2021-08-01 12:38:59 +02:00
TiA4f8R d8177b57f6
Loop in all formats to check if the stream has URLs protected by signatureCiphers 2021-08-01 12:38:59 +02:00
TiA4f8R a6a2c6eb80
Revert the use of Collections.singletonList instead of Arrays.asList in addCookieHeader of YoutubeParsingHelper 2021-08-01 12:38:59 +02:00
TiA4f8R 81013e5a8e
Add a static method to reset the YouTube deobfuscation code and use it in tests
This method is needed for YouTube stream tests, because when all YouTube tests are ran, the signatureTimestamp is known (the sts string) so a different body than the body present in the mocks is send by the extractor instance.
As a result, running all YouTube stream tests with the MockDownloader (like the CI does) will fail if this method is not called before fetching the page of a test.
2021-08-01 12:38:58 +02:00
TiA4f8R cc798523cd
Adress requested changes in other classes 2021-08-01 12:38:58 +02:00
TiA4f8R accd5ddef3
Adress requested changes in YoutubeStreamExtractor 2021-08-01 12:38:58 +02:00
TiA4f8R 632772d17f
Adress requested changes in YoutubeParsingHelper 2021-08-01 12:38:58 +02:00
TiA4f8R 657f165771
Update client version and mocks 2021-08-01 12:38:44 +02:00
TiA4f8R 8c1c7281b0
Real fix of the fetch of the JS player at each the signatureTimestamp is required
The strings playerJsUrl, sts and playerCode are now static in order to don't fetch again the JavaScript player at each time the signatureTimestamp is needed.
2021-08-01 12:38:43 +02:00
TiA4f8R e97a685989
Format the YoutubeMusicSearchExtractor class to be in the 100 characters per line limit 2021-08-01 12:38:43 +02:00
TiA4f8R 34a9ccb0fd
Adress requested changes 2021-08-01 12:38:42 +02:00
TiA4f8R 54d4551ca6
Adress requested changes in YoutubeParsingHelper and update mobile client version 2021-08-01 12:38:42 +02:00
TiA4f8R a59c2a3577
Catch every exception instead of only IOException and ExtractionException and add a Javadoc
Catch every exception instead of only IOException and ExtractionException.
Add JavaDoc for fetchAndroidMobileJsonPlayer method of YoutubeStreamExtractor
2021-08-01 12:38:42 +02:00
TiA4f8R 6921e80ded
Try again to don't fetch at each time the JavaScript player 2021-08-01 12:38:42 +02:00
TiA4f8R 70927ddade
Update client version and mocks 2021-08-01 12:38:40 +02:00
TiA4f8R 318bc46a8c
Readd the deleted code of views because watching count of livestreams was broken
The number shown was the total number of views that a livestream has. In order to fix this bug, the previous code is readded.
2021-08-01 12:38:39 +02:00
TiA4f8R 947baec805
Fetch again the desktop player JSON only if the content is protected by signatureCiphers 2021-08-01 12:38:39 +02:00
TiA4f8R 7474049fd1
Update mocks 2021-08-01 12:38:39 +02:00
TiA4f8R c32bc6e534
Try to don't fetch again the first page when requesting a channel continuation
Try to don't fetch again the first page of a YouTube channel when requesting a continuation of it by trying to store the channel name and the channel id into the next page using the ids field of the Page class.
2021-08-01 12:38:38 +02:00
TiA4f8R b52732a1c3
Fix the fetch of the playerCode at each time getStsFromPlayerJs is called
storePlayerJs was called even if sts was not empty in the getStsFromPlayerJs method. This commit fixes it.
2021-08-01 12:38:37 +02:00
TiA4f8R ae5abc0c5d
Fix a typo in YoutubeChannelExtractor 2021-08-01 12:38:37 +02:00
TiA4f8R 0f9e9b8b4b
Use the youtubei API for YouTube mixes + update the corresponding test + do some improvements
Use the youtubei API for YouTube mixes. The corresponding has been updated because the new API breaks the tests of YoutubeMixPlaylistExtractorTest.
Remove some deprecated code (the old search code with the pbj JSON) and do some other improvements.
2021-08-01 12:38:37 +02:00
TiA4f8R 14569c4aa9
Readd the return of itags 134 and 136 2021-08-01 12:38:37 +02:00
TiA4f8R 3017dde67e
Fix some typos 2021-08-01 12:38:36 +02:00
TiA4f8R 013b902535
Use the Android mobile API when there are OTF streams or the content is protected by signatureCiphers
Use the Android mobile API to get the itag 22 (720p with audio), removed when the content is protected by signatureCiphers.
Also use this API when they are OTF streams, to get the itag 17 and 36, low 3GPP quality streams but also the itag 139.
Update the web client version.
2021-08-01 12:38:36 +02:00
TiA4f8R e7d589edbf
Use the youtubei API for YouTube videos + update client version
Update the hardcoded client version to 2.20210520.09.00
Use the player and next endpoints of the Innertube API for YouTube videos
2021-08-01 12:38:36 +02:00
TiA4f8R f73c923f60
Don't use the youtubei.googleapis.com but the websites domains + update client version of the desktop internal API
Use again www.youtube.com and music.youtube.com domains instead of youtubei.googleapis.com domain because it spoofs more a web client of YouTube or YouTube Music and may reduce Google's detection of NewPipe Extractor users.
2021-08-01 12:38:34 +02:00
TiA4f8R effcdaa4f2
Update mocks, reenable a test and fix a test
This commit updates mocks and reenables the test invalidId of the NotAvailable class of the YoutubePlaylistExtractorTest class beacuse with the youtubei/innertube API, it returns "Not found" and doesn't redirect to the YouTube homepage.
The expectedMetaInfo test of the MetaInfoTest class of the YoutubeSearchExtractorTest class was broken because YouTube removes the vaccine progress link of the WHO from the meta info so this commit removes it and the test is now passing.
2021-08-01 12:38:04 +02:00
TiA4f8R 4d682834c3
Fix localization and update client version 2021-08-01 12:38:03 +02:00
TiA4f8R f46cfb0f26
Adress reviews and do some improvements
Adress changes requested in reviews.
Do some improvements, remove unused imports and format some code to be in the 100 characters line limit.
2021-08-01 12:38:03 +02:00
TiA4f8R e075dd5a63
Update client version, fix some tests, update mocks and do some improvements
Add the origin and the referer headers with the https://www.youtube.com value for YouTube JSON POST requests.
Don't add the consent cookie header for the requests which use the youtubei/innertube API because it's uneeded.
Fix some tests and update YouTube mocks
2021-08-01 12:38:02 +02:00
TiA4f8R b49ae547a3
Do some improvements to YoutubeStreamExtractor
Get the real name of the uploader (for autogenerated channels and music artist channels), like before the migration to the JSON pbj.
Do some other improvements, especially reformatting some code to be in the 100 characters line limit and use final where possible.
2021-08-01 12:38:01 +02:00
TiA4f8R 58ce9b04a1
Fix channel extraction when channel URL is youtube.com/c/username 2021-08-01 12:38:01 +02:00
TiA4f8R 991b2c7d73
Use lightweight requests when getting and checking YouTube API key and client version 2021-08-01 12:38:01 +02:00
TiA4f8R 9ab9c66ddf
Use the youtubei.googleapis.com domain for YouTube Music searches + change a check + update client version and mocks
Change the domain from music.youtube.com to youtubei.googleapis.com.
Use a lightweight request to check if the hardcoded YouTubeMusic keys are valid. Increase the length of the response to 500 because if the key is invalid, the length of the response returned is higher than 250 and the response when the key is valid is higher than 1500.
Format the YoutubeMusicSearchExtractor file.
Update YouTube web client version and mocks
2021-08-01 12:37:56 +02:00
TiA4f8R 77c031a88a
Use the youtubei API for YouTube trends 2021-08-01 12:36:28 +02:00
TiA4f8R f461224b2b
Use the youtubei API for YouTube searches + update mocks
Add getSearchParameter, a new method in YoutubeSearchQueryHandlerFactory class which returns the params field for a search, or an empty string if there is no one.
Update mocks of YoutubeSearchExtractorTest.
2021-08-01 12:36:28 +02:00
TiA4f8R a12c69da7d
Use the youtubei API for YouTube channels 2021-08-01 12:36:27 +02:00
TiA4f8R 5794eb2350
Use the youtubei API for YouTube playlists 2021-08-01 12:36:12 +02:00
Stypox c97a19d719
Merge pull request #696 from XiangRongLin/decrypt_pattern
Expand regex to match n param decrypt function
2021-08-01 10:24:22 +02:00
Stypox 06a5219c9c
Merge pull request #699 from FireMasterK/yt-music-search
Fix YouTube music search.
2021-08-01 10:17:15 +02:00
XiangRongLin 37df225556 Remove length check from StringUtils.matchToClosingParenthesis 2021-07-31 16:05:24 +02:00
FireMasterK f4aad8b014
Fix tests again. 2021-07-31 14:40:51 +05:30
FireMasterK 88c11db4c5
Fix isCorrectedSearch test. 2021-07-31 02:18:18 +05:30
FireMasterK fd19c53f65
Fix search suggestions test. 2021-07-31 02:08:50 +05:30
FireMasterK a685941bba
Fix YouTube music search. 2021-07-31 01:33:38 +05:30
XiangRongLin 852a65ff18 Add tests for StringUtils 2021-07-30 19:56:57 +02:00
XiangRongLin 48d897e6ad Add final and adjust utils class name 2021-07-29 21:33:45 +02:00
XiangRongLin 60794aea31 [YouTube] Add parenthesis matching as way to parse decrypt function
From @Stypox
2021-07-29 21:23:00 +02:00
XiangRongLin 2967d1ae6a [YouTube] Compile YoutubeThrottlingDecrypter pattern statically 2021-07-29 20:27:29 +02:00
XiangRongLin 1c78976900 [YouTube] Expand regex to match n param decrypt function
Temporary solution
2021-07-29 20:26:36 +02:00
TobiGr b62fe7141e Fix JDoc 2 2021-07-29 12:34:04 +02:00
TobiGr dbc4e01c17 Fix JDoc 2021-07-29 12:31:33 +02:00
TobiGr 6335823843 Merge branch 'master' into dev 2021-07-28 20:41:23 +02:00
TobiGr 7c6ff0a38f Release version 0.21.7 2021-07-28 20:38:45 +02:00
TobiGr 8bfcb0ad59 [SoundCloud] Fix hardcoded client id 2021-07-28 20:33:21 +02:00
Tobi 027dc65434 pull request #683 from XiangRongLin/yt_throttling
[YouTube] Fix buffering by decoding n parameter of stream urls
2021-07-28 20:32:38 +02:00
Tobi 14c179f343
Merge pull request #692 from TeamNewPipe/fix/sc/id
[SoundCloud] Fix hardcoded client id
2021-07-28 18:28:07 +02:00
Tobi 394c02ad06
Merge pull request #683 from XiangRongLin/yt_throttling
[YouTube] Fix buffering by decoding n parameter of stream urls
2021-07-28 18:01:57 +02:00
TobiGr 785ff4aa32 [SoundCloud] Fix hardcoded client id 2021-07-28 17:02:56 +02:00
TobiGr 79f2d74b04 Fix test
The decryption code changes over time. Only check whether the n parameter was changed in the URL
2021-07-28 16:45:31 +02:00
TobiGr d70adfdb8f Add methods for cache control to YoutubeThrottlingDecrypter. 2021-07-27 20:32:22 +02:00
TiA4f8R d13f531b6f Use YoutubeThrottlingDecrypter also in getAudioStreams and getVideoOnlyStreams methods of YoutubeStreamExtractor
Without this commit, the n param is only decrypted for streams extracted in getVideoStreams (so only for streams in the formats object of the player response).
2021-07-27 17:28:32 +02:00
TiA4f8R 32055147e0 Do some code improvements
Use final where possible, annotate some methods and parameters as Nonnull and format new code to be in the 100 characters limit per line.
2021-07-27 17:24:27 +02:00
TobiGr 1c30a2725e Cache nParams to prevent executing the JavaScript function for the same nParam multiple times.
Closes #689
2021-07-22 19:40:12 +02:00
XiangRongLin 3a3d1d7f2b Make YoutubeJavaScriptExtractor and JavaScript methods static
Also address review and rewrite some comments
2021-07-20 20:48:11 +02:00
XiangRongLin a683c8d278 Delete duplicated code to load youtube javascript file 2021-07-17 19:14:57 +02:00
XiangRongLin a02ee2e952 Rewrite youtube throttling solution and add tests 2021-07-17 19:10:09 +02:00
Tobi ada67d136a
Merge pull request #647 from litetex/playerSeekbarPreview
Code changes to enable player thumbnail seekbar preview in NewPipe
2021-07-17 17:49:54 +02:00
litetex 0c12b396e5 Review changes 2021-07-17 16:41:12 +02:00
litetex ecf4232ce3 Using ``Collections.emptyList()`` 2021-07-17 16:41:12 +02:00
litetex 0f9ed020a4 Frameset has to implement Serializable or NewPipe fails
java.lang.RuntimeException: Parcelable encountered IOException writing serializable object (name = org.schabi.newpipe.extractor.stream.StreamInfo)
...
Caused by: java.io.NotSerializableException: org.schabi.newpipe.extractor.stream.Frameset
...
2021-07-17 16:41:11 +02:00
litetex 17ccaf4b87 Update YoutubeStreamExtractor.java
To fix ``java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.String[] java.lang.String.split(java.lang.String)' on a null object reference``
2021-07-17 16:41:10 +02:00
litetex 4e41e172ea Update Frameset.java
Reformatted code
2021-07-17 16:41:10 +02:00
litetex 09b670d745 Renamed frames to previewFrames (so it's better understandable) 2021-07-17 16:41:09 +02:00
litetex b086655d54 Reverted useless change 2021-07-17 16:41:08 +02:00
litetex 93b0c5b15f Added seekbar data 2021-07-17 16:41:08 +02:00
XiangRongLin 6956b72af7 Fix YoutubeThrottlingDecoder.replaceNParam
Previously it replaced the parameter itself not the value of the parameter.
2021-07-17 10:05:14 +02:00
XiangRongLin a189f685dc cache player js code 2021-07-16 19:20:40 +02:00
XiangRongLin fcdb9bdbeb add url to sout 2021-07-15 20:05:26 +02:00
XiangRongLin 80cf8b3acd Extract separate YoutubeThrottlingDecoder 2021-07-15 20:05:25 +02:00
XiangRongLin a86a30103f [Youtube] bare bones version to solve throttling
Done by transforming the parameter "n" from videoplayback urls
https://github.com/ytdl-org/youtube-dl/issues/29326#issuecomment-865985377
2021-07-15 20:05:25 +02:00
bopol c38a06e8dc
Merge pull request #661 from B0pol/peertube-livestreams
[PeerTube] Support livestreams
2021-07-13 15:48:13 +02:00
Tobi b45bb411e8
Merge pull request #652 from litetex/fixYTCommentsAndAddDisabledComments
Fix yt comments and add disabled comments functionallity
2021-07-12 16:31:50 +02:00
TobiGr 6fd93cdb31 Release 0.21.6 2021-07-10 17:07:23 +02:00
litetex fdebf3c6cd Added javadoc warnings 2021-07-07 20:41:59 +02:00
litetex 6860543b07 Added more doc 2021-07-06 21:16:31 +02:00
litetex a59fead0d7 Fixed typos 2021-07-06 21:16:30 +02:00
litetex 688a1c316b Fixed Exception when YT comments are disabled and added ``commentsDisabled`` field
* Fixed code: Added missing finals (according to NewPipes Checkstyle guide)
* Fixed ``findValue`` method in ``YoutubeCommentsExtractor``
2021-07-06 21:16:29 +02:00
litetex ca33f4f60b Improved code format (final) and added ``isCommentsDisabled`` method 2021-07-06 21:16:28 +02:00
Tobi f324772254
Merge pull request #676 from TeamNewPipe/fix/consent-cookie
Fix invalid CONSENT cookie value
2021-07-04 17:41:47 +02:00
TobiGr dbdc962a07 Update mocks 2021-07-03 13:30:26 +02:00
TobiGr 9ca6dc26ed Fix invalid CONSENT cookie value 2021-07-03 13:27:40 +02:00
Tobi a023f08774
Merge pull request #673 from XiangRongLin/recording_downloader
Remove IPv4 addresses from response during mock recording
2021-07-03 13:25:41 +02:00
XiangRongLin b9549ace25 [Youtube] Fix lofi girl test and ignore mix with id test 2021-06-29 21:12:20 +02:00
XiangRongLin 5982431ffb Remove IPv4 addresses from response during mock recording 2021-06-29 21:03:33 +02:00
bopol 7d7cc087e2 [PeerTube] Support livestreams 2021-06-24 11:02:30 +02:00
bopol 3966178979
Merge pull request #665 from B0pol/agerestricted
[youtube] Fix extraction of age restricted videos
2021-06-23 21:07:14 +02:00
bopol b475f09ba7 don't do useless concatenation 2021-06-23 20:33:32 +02:00
TiA4f8R d1c56be744
Merge pull request #668 from B0pol/invidious-instances
Update Invidious instances and support Piped links
2021-06-23 18:36:48 +02:00
bopol b96b2a6eba Update invidious instances 2021-06-23 14:39:10 +02:00
bopol c5e8bd368d Update Mocks 2021-06-23 12:21:57 +02:00
bopol e9a992b0a9 [youtube] Fix extraction of age restricted videos 2021-06-23 12:06:02 +02:00
bopol 35c1bdd012
Merge pull request #655 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.2.3
Bump spotbugs-annotations from 4.0.2 to 4.2.3
2021-06-23 12:01:56 +02:00
dependabot[bot] ad403ce73e
Bump spotbugs-annotations from 4.0.2 to 4.2.3
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.0.2 to 4.2.3.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.0.2...4.2.3)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-06-23 09:57:42 +00:00
bopol 65fd8740aa
Merge pull request #658 from TeamNewPipe/revert-651-fix/streamType
[YouTube] Fix getting stream type of live streams
2021-06-23 11:33:24 +02:00
bopol 361f142621 Use badge style instead of label 2021-06-23 11:25:59 +02:00
bopol 58f109ddec adress stypox review 2021-06-23 11:15:40 +02:00
bopol fe29c78fe1
Merge pull request #656 from TeamNewPipe/dependabot/gradle/org.mozilla-rhino-1.7.13
Bump rhino from 1.7.12 to 1.7.13
2021-06-20 00:46:42 +02:00
bopol e9412fdadc
Merge pull request #654 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.8.7
Bump gson from 2.8.6 to 2.8.7
2021-06-20 00:35:24 +02:00
XiangRongLin 1510dd5187
Merge pull request #660 from TeamNewPipe/fix/sc/client-id
[SoundCloud] Update HARDCODED_CLIENT_ID
2021-06-18 15:48:27 +02:00
TobiGr 1f9a99c931 [SoundCloud] Update HARDCODED_CLIENT_ID 2021-06-18 15:30:54 +02:00
Tobi c998012c28
Revert "[YouTube] Fix getting stream type of live streams" 2021-06-18 09:20:32 +02:00
dependabot[bot] 38d0339753
Bump rhino from 1.7.12 to 1.7.13
Bumps [rhino](https://github.com/mozilla/rhino) from 1.7.12 to 1.7.13.
- [Release notes](https://github.com/mozilla/rhino/releases)
- [Changelog](https://github.com/mozilla/rhino/blob/master/RELEASE-NOTES.md)
- [Commits](https://github.com/mozilla/rhino/commits)

---
updated-dependencies:
- dependency-name: org.mozilla:rhino
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-06-17 21:49:07 +00:00
dependabot[bot] 5a46878b09
Bump gson from 2.8.6 to 2.8.7
Bumps [gson](https://github.com/google/gson) from 2.8.6 to 2.8.7.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.6...gson-parent-2.8.7)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-06-17 21:49:01 +00:00
bopol 598de2dd5f
Merge pull request #552 from FireMasterK/dependabot
Add dependabot
2021-06-17 23:48:41 +02:00
Tobi 0cb48e36bd
Merge pull request #651 from TeamNewPipe/fix/streamType
[YouTube] Fix getting stream type of live streams
2021-06-12 17:57:18 +02:00
TobiGr 6762fe3357 [YouTube] Fix getting stream type of live streams 2021-06-10 16:51:00 +02:00
Tobi a27575021d
Merge pull request #625 from TeamNewPipe/commentsStreamPosition
Add streamPosition for comments
2021-06-08 11:03:15 +02:00
TobiGr f7f727d19c Add constant NO_LIKE_COUNT to CommentsInfoItem 2021-06-08 10:28:44 +02:00
TobiGr b70c0f93c7 Add streamPosition for comments
SoundCloud is the only service which supports adding comments at a specific timestamp in the stream.
2021-06-08 10:27:52 +02:00
Tobi d4186d100b
Merge pull request #580 from TeamNewPipe/accountTerminated
Add AccountTerminatedException for terminated channels
2021-06-08 09:55:58 +02:00
TobiGr 89e0514d8b Fix Javadoc generation fix 2021-06-08 09:30:05 +02:00
TobiGr 80d3052033 Fix Javadoc generation 2021-06-08 09:26:01 +02:00
TobiGr 98825a2f01 Set version to 0.21.4 2021-06-07 23:45:12 +02:00
TobiGr fa444c8298 Remove assertations of exception messages 2021-06-06 11:38:43 +02:00
Tobi 8c42a48673
Merge pull request #646 from Stypox/get-service
Add utility method getService() to Info object
2021-06-05 14:29:09 +02:00
Stypox 2158ca4060
Remove two unneded warning suppressions 2021-06-05 14:15:48 +02:00
Stypox 13f192704d
Add utility method getService() to Info object 2021-06-05 14:14:19 +02:00
XiangRongLin 0ad51e76fb
Merge pull request #638 from TeamNewPipe/spelling
Improve exception message thrown when not accepting an URL in LinkHandlerFactory#fromUrl(String url, String baseUrl)
2021-06-04 18:23:31 +02:00
Tobi 143fd3c81c
Merge pull request #645 from XiangRongLin/fix_test
Fix PeertubeAccountExtractorTest
2021-06-03 20:49:59 +02:00
XiangRongLin faa503c7dd Fix PeertubeAccountExtractorTest 2021-06-03 19:54:42 +02:00
XiangRongLin db81384ae0
Merge pull request #644 from TiA4f8R/utf-8-encoding-for-mocks-windows
Specify UTF-8 file encoding in RecordingDownloader and MockDownloader
2021-06-03 18:55:00 +02:00
TiA4f8R ac31f3a883
Specify UTF-8 file encoding in RecordingDownloader and MockDownloader
On Windows, mocks are recorded and read with the Cp1252 encoding so it breaks the mocks on non ASCII characters for Linux OS (and so the CI).
The project is in Java 8, so we can't use FileReader(File, Charset) and FileReader(File, Charset) because these methods require Java 11. Instead of changing the Java version of the extractor, use OutputStreamWriter and FileOutputStream instead of FileWriter and InputStreamReader and FileInputStream instead of FileReader.
2021-06-03 18:38:25 +02:00
Tobi 7e4332e0cf
Update CI badge to representt the scheduled tests 2021-05-30 22:12:09 +02:00
bopol ff11c2df2a
Merge pull request #628 from litetex/fix-broken-yt-liked-comments
Fix broken yt likes in comments
2021-05-29 11:04:11 +02:00
TobiGr 1ff56a85df Update junit from 4.13.1 to 4.13.2
https://github.com/junit-team/junit4/blob/main/doc/ReleaseNotes4.13.2.md
2021-05-28 15:09:26 +02:00
TobiGr 858e007804 Improve exception message thrown when not accepting an URL in LinkHandlerFactory#fromUrl(String url, String baseUrl) 2021-05-28 14:59:44 +02:00
Tobi 519bba70b3
Merge pull request #630 from XiangRongLin/fix_tests
Fix MetaInfoTest.expectedMetaInfo
2021-05-28 14:16:53 +02:00
litetex 289b84e68b Removed unused code 2021-05-27 21:27:44 +02:00
litetex e81b0e2885 Tweaked code 2021-05-27 19:48:31 +02:00
TobiGr f4cb32cef5 Release version 0.21.3 2021-05-26 22:40:56 +02:00
XiangRongLin bf2af4f316
Merge pull request #635 from litetex/update-setup-java
Update setup java
2021-05-26 17:44:57 +02:00
litetex 8c96545e57 Merge branch 'TeamNewPipe:dev' into fix-broken-yt-liked-comments 2021-05-24 18:06:45 +02:00
litetex 12fb18c310 Updated mock data (only the affected) 2021-05-24 18:05:26 +02:00
litetex 8e08a2aac7 Improved docs 2021-05-24 18:05:26 +02:00
litetex 020acfed71 Fixed tests and added new ones 2021-05-24 18:05:26 +02:00
litetex b310922fc0 Better exception message 2021-05-24 18:05:25 +02:00
litetex 86bb9efb5d Added likeCount back to tests 2021-05-24 18:05:25 +02:00
litetex 46654f5fca Reformatted code 2021-05-24 18:05:25 +02:00
litetex 2174685c5c Reimplemented likeCount 2021-05-24 18:05:25 +02:00
litetex 01cfb55505 Update YoutubeCommentsInfoItemExtractor.java 2021-05-24 18:05:25 +02:00
litetex bedcd87abb Fixed occurrences where voteCount is not set (<=0 likes) 2021-05-24 18:05:25 +02:00
litetex 545c0a6f42 Fixed comment 2021-05-24 18:05:24 +02:00
Tobi ff005122bf
Merge pull request #627 from TiA4f8R/use-snd-api-v2-everywhere
[SoundCloud] Use a lightweight request to check if the hardcoded client_id is valid, fix the extraction of mobile URLs and more
2021-05-23 22:52:40 +02:00
TiA4f8R a00fdcbd3d
Format the indentation of the definition of the apiUrl string in SoundcloudParsingHelper.resolveFor method
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2021-05-23 20:27:45 +02:00
TiA4f8R 6b607eb38d
Update the hardcoded client_id 2021-05-23 18:08:44 +02:00
TiA4f8R 4552ea9c9f
Use the SOUNDCLOUD_API_V2_URL constant in all the SoundCloud package 2021-05-23 18:08:44 +02:00
TiA4f8R 86308d0603
Use a lightweight request to check if the client_id is valid
Request the api-v2 host with the client_id instead of checking if the streams of a SoundCloud track are not empty: if it is valid, the API returns 404, otherwise it should return 401.
2021-05-23 18:08:43 +02:00
TiA4f8R c5c190500c
Format the SoundCloud package of the extractor and fix some warnings
Use final where possible in the package and format code to be in the 100 caracters per line limit.
Fix some warnings generated by Android Studio and do some code improvements
2021-05-23 18:08:42 +02:00
litetex 89ecc4e2e8
Update docs.yml 2021-05-23 14:34:29 +02:00
litetex 7791907480
Update ci.yml 2021-05-23 14:34:16 +02:00
Tobi 636e27333b
Merge pull request #603 from TeamNewPipe/peertube_hls
Add PeerTube HLS streams
2021-05-23 11:49:40 +02:00
Tobi fadf677c42
Merge pull request #629 from TechComet/patch-1
[YouTube] Fix the fetch of the video info page
2021-05-23 09:41:24 +02:00
XiangRongLin 172394cd52 Update mocks 2021-05-22 20:33:20 +02:00
XiangRongLin 18c8cac40c Fix MetaInfoTest.expectedMetaInfo
Expected meta changed
2021-05-21 21:00:08 +02:00
Hassan Igbaria e8b5364745
Update YoutubeStreamExtractor.java 2021-05-21 16:22:24 +03:00
litetex c3b837fe3b
Deduped code; Using default values... 2021-05-20 20:22:40 +02:00
litetex b934c7ccbb
Removed unused import 2021-05-20 20:11:35 +02:00
litetex 10cf081145
Removed likeCount and added textualVoteCount 2021-05-20 20:10:14 +02:00
TobiGr 1acc53b8db [PeerTube] Add support for HLS streams 2021-05-15 12:39:23 +02:00
Tobi 3a3ade20f4
Merge pull request #620 from TeamNewPipe/bandcamp-comments
Add Bandcamp comments (first page)
2021-05-02 13:43:53 +02:00
Fynn Godau 85a4270bd9 [Bandcamp] Fix comments test 2021-05-02 13:07:24 +02:00
TobiGr ec50570053 Add mocks 2021-05-01 17:50:19 +02:00
TobiGr f4404b5cc8 Add tests for more reasons 2021-05-01 17:50:19 +02:00
TobiGr fc998589dc Detect channels which have been terminated due to copyright infringement 2021-05-01 17:50:19 +02:00
TobiGr bb3861ddce Add AccountTerminatedException for better error handling of terminated channels 2021-05-01 17:50:19 +02:00
Tobi 284362fc15
Merge pull request #608 from TeamNewPipe/youtube-fix-segment-test
[YouTube] fix tests
2021-05-01 17:24:41 +02:00
TobiGr dc678a9f6d Release version 0.21.2 2021-04-26 18:06:30 +02:00
Tobi 7cf8edd5ac
Merge pull request #618 from TeamNewPipe/soundcloud_id
Fix Soundcloud extraction
2021-04-26 18:03:26 +02:00
TobiGr 9c12dc5609 [SoundCloud] Fix SoundCloud ID extraction
resolveIdWithEmbedPlayer() does not work anymore because the JSON data has been extracted to an API call. For this reason, replace resolveIdWithEmbedPlayer() with resolveIdWithWidgetApi)( which performs the API call.
2021-04-26 17:58:34 +02:00
Fynn Godau 9dc7a1d5ee [Bandcamp] Add comments (first page) 2021-04-26 17:25:04 +02:00
TobiGr 8f023c1ec7 [SoundCloud] Update client id 2021-04-26 15:14:11 +02:00
Tobi 7f202db8b1
Fix Google URL detection (#617) 2021-04-26 11:49:47 +02:00
fynngodau 6db4bea8ca
Rework Bandcamp comments (#613) 2021-04-13 21:10:59 +02:00
Tobi c14f6db14a
Merge pull request #610 from TeamNewPipe/bandcamp_related_items
Extract related bandcamp items
2021-04-13 21:05:27 +02:00
Fynn Godau 3671876721 Fix bandcamp radio stream extractor 2021-04-13 19:30:11 +02:00
Fynn Godau 90b5c00599 Extract related items #593 2021-04-13 17:56:19 +02:00
TobiGr 1fe645704f [YouTube] FIx testMoreRelatedItems in RandomQueryNoMorePages 2021-04-11 23:21:35 +02:00
TobiGr 9a2b814d8f Update mocks 2021-04-11 22:59:01 +02:00
TobiGr 50005ce937 [YouTube] Replace unavailable YouTube Stream Segments test
The stream is unavailable in many countries.
2021-04-11 22:58:48 +02:00
Tobi 82d11386df
Merge pull request #607 from TeamNewPipe/bandcamp_radio_segments
[Bandcamp] Add Radio stream segments
2021-04-11 22:30:03 +02:00
TobiGr 318cec7625 [Bandcamp] Add Radio stream segments 2021-04-11 21:19:44 +02:00
TobiGr 55e0f8e725 Merge branch 'master' into dev 2021-04-10 11:33:34 +02:00
Tobi e747a89be2
Merge pull request #600 from TeamNewPipe/youtube-eu-cosent
[YouTube] Set EU consent cookie
2021-04-09 12:02:52 +02:00
TobiGr 33173eb3e6 Make CONSENT COOKIE constants private 2021-04-09 11:59:17 +02:00
TobiGr b94b316558 Update mocks 2021-04-09 11:59:17 +02:00
TobiGr 4b8d4a84b6 Set the NumberGenerator for th remaining tests 2021-04-09 11:59:17 +02:00
XiangRongLin db7ad9c27d Update Mocks 2021-04-08 16:37:29 +02:00
XiangRongLin 4833df60e5 Add method to inject Random into YoutubeParsingHelper and use in tests 2021-04-08 16:36:55 +02:00
TobiGr 682ec27737 Update mocks 2021-04-08 14:22:31 +02:00
XiangRongLin eafad3ad6b
Merge pull request #602 from XiangRongLin/soundcloud_go
Ignore unreliable SoundcloudGoPlusTrack.testRelatedItems
2021-04-07 19:41:44 +02:00
TobiGr 080d1e318d Code improvement 2021-04-07 14:11:47 +02:00
TobiGr 4ee270fe01 Fix LiveStream Test 2021-04-07 14:11:41 +02:00
TobiGr 883f16e0ad [YouTube] Set CONSENT cookie 2021-04-07 14:11:37 +02:00
XiangRongLin aee3838ed5 Ignore unreliable SoundcloudGoPlusTrack.testRelatedItems 2021-04-06 18:26:45 +02:00
XiangRongLin 90afd6162a
Merge pull request #599 from XiangRongLin/post_body
Add body field to Page and use it
2021-04-06 18:23:26 +02:00
XiangRongLin 7b06c696e2 Use Page.body for YoutubeChannelExtractor 2021-04-06 18:14:38 +02:00
XiangRongLin eda1b6e199 Use Page.body for YoutubePlaylistExtractor 2021-04-06 18:14:38 +02:00
XiangRongLin 3af26a2821 Add body field to Page 2021-04-04 11:39:58 +02:00
Tobi 4e0be60ddc
Merge pull request #601 from XiangRongLin/fix_tests
Fix SoundcloudGoPlusTrack Test
2021-04-04 09:55:54 +02:00
XiangRongLin 7b36469807 Fix SoundcloudGoPlusTrack
https://soundcloud.com/martinsolveig/places#t=0 has related streams
2021-04-04 09:39:26 +02:00
Tobi b481a228cd
Merge pull request #598 from XiangRongLin/fix_tests
Fix a test and ignore rest
2021-04-03 23:44:23 +02:00
XiangRongLin 3d70b6d46a Fix a test and ignore rest 2021-04-03 11:03:10 +02:00
XiangRongLin 1925dcf4dc
Merge pull request #597 from fynngodau/related-items
Rename getRelatedStreams to getRelatedItems and change return type
2021-04-02 19:54:22 +02:00
Fynn Godau c877712647 Add deprecated old method calls to avoid breaking API 2021-04-01 22:32:36 +02:00
Fynn Godau 14f6f1b7c3 Generify related streams calls and rename method 2021-03-31 20:24:40 +02:00
Stypox 9dbc152d44
Merge pull request #594 from TeamNewPipe/conersions
Reduce number of type conversions
2021-03-31 14:34:46 +02:00
TobiGr 1498e1905e Reduce number of type conversions in YouTubeSearchExtractor.collectStreamsFrom by introducing new variable 2021-03-31 13:30:38 +02:00
Tobi 7ea2cd73d3
Merge pull request #589 from fynngodau/bandcamp-more-recent-pages
Load more featured pages and fix featured cover arts not loading
2021-03-31 12:15:12 +02:00
Fynn Godau 705f6c6e33 Apply review 2021-03-31 11:02:30 +02:00
XiangRongLin 564d74c250
Merge pull request #578 from TeamNewPipe/code_improvements
Code improvements
2021-03-30 18:12:22 +02:00
TobiGr 657b00ca11 Smaller code improvements 2021-03-30 10:10:53 +02:00
TobiGr 8fbd1a2bcf FIx potential NullPointerException 2021-03-30 10:07:20 +02:00
TobiGr f2e7c7ebd6 Replace Vector with ArrayList for performance improvements
I could not figure out, why these two methods are necessary.
2021-03-30 10:07:20 +02:00
TobiGr 48fd8ca7b3 Remove redundant type declaration 2021-03-30 10:07:20 +02:00
TobiGr 22bd16fca3 Make vars private, final and static 2021-03-30 10:07:20 +02:00
TobiGr 6a0f6e846a Fix possible bug when language.id field is not present and this the languageCode an empty String. 2021-03-30 10:07:20 +02:00
TobiGr b9282bbe94 Remove checks which are always true 2021-03-30 10:07:20 +02:00
TobiGr 46eab1ec17 Remove declarations of exceptions which are not thrown from method signatures 2021-03-30 10:07:20 +02:00
TobiGr c3e23559d7 Remove unused imports 2021-03-30 10:07:20 +02:00
TobiGr e4c40ae6d1 Use List.isEmpty() 2021-03-30 10:07:20 +02:00
XiangRongLin af2183855a
Merge pull request #591 from TotalCaesar659/update-url-to-https
Update URL to HTTPS in README
2021-03-29 09:33:59 +02:00
TotalCaesar659 cf3f54f438 Update URL to HTTPS 2021-03-28 22:06:56 +03:00
Fynn Godau d663be5a78 [Bandcamp] Fix featured albums not showing thumbnails 2021-03-28 13:12:22 +02:00
Fynn Godau dbcf61c6f7 [Bandcamp] Load more featured pages 2021-03-28 12:53:05 +02:00
Stypox b4dee6d08f
Merge pull request #306 from B0pol/metadata
Extract metadata for youtube, soundcloud & mediaccc
2021-03-27 08:45:47 +01:00
TobiGr e61ceef005 Release 0.21.0 2021-03-25 21:59:15 +01:00
Tobi e2500fb7cc
Merge pull request #583 from TeamNewPipe/peertube_playlist
Fix some tests and bugs
2021-03-25 21:53:41 +01:00
TobiGr 070a40e181 [YouTube] Mix Playlist - Fix getting name 2021-03-25 21:47:16 +01:00
TobiGr b029779217 [media.cc.de] Disable unreliable tests for now 2021-03-25 19:44:44 +01:00
TobiGr bbee15474d [PeerTube] Fix playlist extraction 2021-03-25 19:19:19 +01:00
Tobi d4f83a1782
Merge pull request #575 from TiA4f8R/fix-peertube-test
Fix Peertube account subscribers extraction
2021-03-25 19:14:51 +01:00
TiA4f8R ae283314da
Add a comment explaining subscribers count calculation 2021-03-25 17:54:46 +01:00
TiA4f8R abb790f465
Remove println in getSubscriberCount from PeertubeAccountExtractor class 2021-03-25 17:49:22 +01:00
TiA4f8R 772de53a66
Increase the number of subscribers in PeertubeAccountExtractorTest 2021-03-25 17:49:20 +01:00
TiA4f8R efe2b964f0
Extract real subscribers of a Peertube account
Apply fix provided in the PR and move all accounts/ strings to a constant, ACCOUNTS.
2021-03-25 17:47:44 +01:00
Tobi 12835bfae1
Merge pull request #551 from FireMasterK/ff-privacy
Change UA to privacy.resistFingerprinting.
2021-03-24 17:58:34 +01:00
Tobi cae6cea551
Merge pull request #582 from Stypox/fix-trending
[YouTube] Fix trending getName()
2021-03-24 10:49:46 +01:00
Stypox ce7cbbc9a0
[YouTube] Fix trending getName() 2021-03-24 09:14:09 +01:00
XiangRongLin 9009f1e277
Merge pull request #517 from XiangRongLin/flaky_test
Add MockOnlyRule to allow skipping specific tests based on downloader
2021-03-20 18:04:14 +01:00
bopol 152221c7fb
ignore a test with encoding problems 2021-03-19 10:14:43 +01:00
bopol 8806fb4e6b
address reviews 2021-03-19 10:14:43 +01:00
bopol c47cc54908
Extract metadata for YouTube, SoundCloud & MediaCCC 2021-03-19 10:14:43 +01:00
Stypox f71cfd489c
Merge pull request #526 from TiA4f8R/snd-hls-workaround
Support SoundCloud HLS-only tracks by using a workaround
2021-03-19 10:09:54 +01:00
TiA4f8R 379d7312fa
Don't use a regular expression to find the last segment URL and do code improvements
Apply suggestions provided in the PR and remove a redundant import.
2021-03-14 17:54:22 +01:00
TiA4f8R 0e3e420a25
Fix tests 2021-03-14 17:54:21 +01:00
TiA4f8R d61d9d116d
Refactor getAudioStreams method of SoundcloudStreamExtractor
Split the method into private methods, in order to have a better reading.
2021-03-14 17:54:20 +01:00
TiA4f8R a7b15b51e6
Change t to transcodingJsonObject in SoundcloudStreamExtractor 2021-03-14 17:54:19 +01:00
TiA4f8R 0438828e36
Add a test for the number of audioStreams in CreativeCommonsPlaysWellWithOthers test
It should be only two audio streams for track "Plays Well with Others, Ep 2: What Do an Army of Ants and an Online Encyclopedia Have in Common?" by Creative Commons (https://soundcloud.com/wearecc/plays-well-with-others-ep-2-what-do-an-army-of-ants-and-an-online-encyclopedia-have-in-common):
- one which is a progressive stream, in MP3 format with a bitrate of 128 kbps
- one which is an HLS stream, in OPUS format with a bitrate of 64 kbps.
2021-03-14 17:54:18 +01:00
TiA4f8R 3bd08a2880
Adress requested changes and use final where possible in SoundcloudStreamExtractor
This commit moved the HLS parsing task to a separate method, did little performance improvements and used final where possible in the SoundcloudStreamExtractor file.
2021-03-14 17:54:17 +01:00
TiA4f8R cbacd3c0a5
Add a check to don't show MP3 128kbps stream twice and catch IOException when fetching the HLS Manifest
If a progressive stream is present in the transcodings, it's unnecessary to show twice an MP3 128kbps stream so if this is the case, the MP3 HLS stream will be not added to the audioStreams, else it will.
This commit also catch fetching errors in HLS manifests parsing and don't add the corresponding stream if an error occurs.
2021-03-14 17:54:16 +01:00
TiA4f8R 26f1b4e7dc
Support SoundCloud HLS by using a workaround
This commit tries to support SoundCloud HLS streams by parsing M3U manifests, get the last segment URL (in order to get track length) and request a segment URL equals to track's duration so it's a single URL.
2021-03-14 17:54:14 +01:00
Tobi def745b801
Merge pull request #579 from fynngodau/bandcamp-fix-linkhandler
[Bandcamp] Fix link handler acceptance behaviour
2021-03-14 12:56:10 +01:00
Fynn Godau 2e57a8f24f [Bandcamp] Fix link handler acceptance behaviour
* Test for bandcamp footer instead of meta tag (which is not present on
  all pages)
* Accept links to /music, not just /releases
* Correctly handle uppercase URLs
2021-03-14 09:51:30 +01:00
TobiGr 70d9e389b9 [Bandcamp] Add tests for external bandcamp artists
Necessary, because the external pages tested before were converted to bandcamp.com pages. See f9d06252f2
2021-03-13 20:51:59 +01:00
TobiGr f9d06252f2 [Bandcamp] Fix tests 2021-03-13 17:42:39 +01:00
Tobi 7e6f464407
Merge pull request #509 from TiA4f8R/soundcloud-improvements
Add new exceptions to be able to display different error messages in apps
2021-03-07 13:51:15 +01:00
Tobi ac51134aed
Merge pull request #546 from XiangRongLin/unignore2
Unignore more tests
2021-03-07 13:15:24 +01:00
Tobi 021da75f24
Merge pull request #232 from fynngodau/dev
Bandcamp support
2021-03-05 21:37:57 +01:00
TobiGr 22fa131922 Merge branch 'dev' into bandcamp 2021-03-05 19:45:37 +01:00
TiA4f8R e55284bb8f
Code style improvements
Use final where possible in YoutubeStreamExtractor and do some other code style improvements
2021-03-05 16:38:53 +01:00
TobiGr d0a1041afd
Prevent NullPointerException and use equalsIgnorecase 2021-03-05 16:38:52 +01:00
TiA4f8R 890cbba625
Update PaidContent exception 2021-03-05 16:38:51 +01:00
TobiGr da3cfa967d
Handle age-restricted videos 2021-03-05 16:38:50 +01:00
TiA4f8R 448b68700c
Use final in exceptions and add AgeRestrictedException
AgeRestrictedException will be thrown only if the reason message equals to "Sign in to confirm your age" and if the age limit is 18.
2021-03-05 16:38:49 +01:00
TobiGr 22e6f33f0a
Ignore SoundCloudGeoRestrictedTest until HLS stream extraction is added
The ContentNotSupportedException is thrown because no supported audio streams where extracted. However, SoundCLoud does not check, whether there are any streams available. 
This commit should be reverted in #526
2021-03-05 16:38:47 +01:00
TobiGr 21158744b1
Add expectedDescriptionIsEmpty() to DefaultStreamExtractorTest
Also check if related streams are empty if they are expected to be empty.
2021-03-05 16:38:46 +01:00
TiA4f8R f15d7837a1
Fix SoundCloud tests 2021-03-05 16:38:45 +01:00
TiA4f8R 4e45aef2b3
Fix SoundCloud CreativeCommonsPlaysWellWithOthers test 2021-03-05 16:38:43 +01:00
TiA4f8R 59d6d3f04e
Fix YouTube tests 2021-03-05 16:38:42 +01:00
TiA4f8R 3925204658
Add tests and rename SoundcloudGoPlusException to SoundcloudGoPlusContentException 2021-03-05 16:38:40 +01:00
TiA4f8R 771bb1a2cb
Add three new exceptions for unavailable YouTube contents
These exceptions are thrown on a test with the error messages text, because YouTube returns only "UNPLAYABLE" status in most error cases.
Tests are based with English strings, so changing the lang used by
extractor will throw the generic exception (ContentNotAvailableException).
2021-03-05 16:38:39 +01:00
TiA4f8R 35325d980d
Add GeographicRestrictionException and SoundCloudGoPlusException in NewPipe Extractor to be able to display different error messages
This commit adds two new exceptions in NewPipe Extractor: GeographicRestrictionException and SoundCloudGoPlusException (which extend to ContentNotAvailableException). These exceptions allow showing different error messages to user when a content isn't available in his/her/its country (only used for now by SoundCloudStreamExtractor) or when the content is a SoundCloud Go+ track.
2021-03-05 16:38:37 +01:00
TobiGr cb07ffa1eb Release 0.20.11 2021-03-05 15:35:11 +01:00
Tobi bc0cda68d1
Merge pull request #570 from TiA4f8R/fix-yt-channels-playlists
Fix YouTube channels contination
2021-03-05 15:26:49 +01:00
Tobi a3c6fceef5
Merge pull request #573 from B0pol/comments-performance
[youtube] improve comments extraction performance
2021-03-05 14:46:27 +01:00
bopol ff5273b882
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeCommentsExtractor.java
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2021-03-05 14:39:01 +01:00
Tobi ec1127dd6a
Merge pull request #571 from TeamNewPipe/soundcloud_client_id
[SoundCloud] Update client_id and fix extraction
2021-03-05 14:21:14 +01:00
bopol ed850d0688 [youtube] improve comments extraction performance
- do not parse responseBody twice for continuation
instead try to get commentsTokenInside with the new pattern ("sectionListRenderer")
and try again with the old pattern ("commentSectionRenderer") on failure
- do not unescape responseBody multiple times
   -> parse responseBody less times
2021-03-05 13:37:08 +01:00
TobiGr 440a808b8a Update mocks for channels 2021-03-05 13:13:36 +01:00
Tobi b608587e4d
Merge pull request #572 from golfinq/fixed-comments
Fixes missing comments under videos
2021-03-05 11:42:40 +01:00
golfinq 05213175cd
minor edit, made variable final 2021-03-05 02:49:15 -05:00
golfinq 8bceb57a6c
formatting fix 2021-03-05 02:46:06 -05:00
golfinq ae48bdea4c
Updated to reflect variable name change 2021-03-05 02:40:36 -05:00
TiA4f8R b27efdc9a4
Remove clickTrackingParams in getNextPage method of YoutubeChannelExtractor
Because it's unused with the new request.

Co-authored-by: bopol <bopol@e.email>
2021-03-04 22:50:20 +01:00
TobiGr 2b671b15ce [YouTube] Use new continuation API for channels
Co-authored-by: TiA4f8R <74829229+tia4f8r@users.noreply.github.com>
2021-03-04 21:47:16 +01:00
TobiGr 33594d4aff [SoundCloud] Fix client id extraction
SoundCloud changed their Javascript structure. That caused the client_id to be in the middle of the file (at byte ~43000 ). To have a little buffer, we now fetch the first 50000 bytes.
2021-03-04 21:32:34 +01:00
TobiGr fb2f37d223 [SoundCloud] Update hardcoded client id 2021-03-04 21:25:41 +01:00
XiangRongLin 9256b3b848
Merge pull request #567 from XiangRongLin/playlist_continuations
Playlist continuations
2021-03-04 19:05:32 +01:00
XiangRongLin 506cc5fbe9 regenerate mocks 2021-03-04 19:01:58 +01:00
XiangRongLin 03b00ff1d6 Move toJsonArray and toJsonObject to JsonUtils 2021-03-04 18:58:51 +01:00
FireMasterK 668b080b3a Use dynamic key. 2021-03-04 08:45:14 +01:00
XiangRongLin 43b46bd408 Use new youtube continuations api for playlists
Requires sending a POST request instead of GET.
clientName and clientVersion, which were required as headers previously now need to be part of the request payload.
continuation id also needs to be part of request body.

quick and dirty solution.
2021-03-03 19:49:26 +01:00
XiangRongLin beb05bd05c Replace test url, because playlist was deleted 2021-03-03 19:46:14 +01:00
XiangRongLin e13e237392 Fix typo and reword some explanations 2021-02-26 17:48:03 +01:00
Tobi 5d594cfded
Merge pull request #549 from FireMasterK/gradle-update
Update gradle wrapper.
2021-02-24 15:59:28 +01:00
Tobi 965a66bc87
Merge pull request #557 from TiA4f8R/fix-encoding-errors-windows
Set extractor encoding to UTF-8
2021-02-24 15:49:58 +01:00
TiA4f8R b2cf41496d
Set extractor encoding to UTF-8
When compiling the extractor on Windows, you get more than 100 errors with the message error: unmappable character for encoding Cp1252, especially in timeago-parser tasks. This commit fixes it by setting the encoding for all compileJava and compileTestJava tasks to UTF-8.
2021-02-23 14:38:20 +01:00
TobiGr a1688fe953 Move BandcampExtractorHelper.getJsonData(String, String) to JsonUtils 2021-02-21 13:51:12 +01:00
TobiGr 70814dcfef Fix Utils.nonEmptyAndNullJoin
When using the index here, it the index needs to be decremented once an element is removed. To cirecumvent this, the native Collections.removeIf() method is used.
2021-02-21 12:51:47 +01:00
TobiGr b9e8ee8450 Rename BandcampExtractorHelper.smartConcatenate(String[], String) to Utils.nonEmptyAndNullJoin(String, String[]) 2021-02-21 12:35:14 +01:00
TobiGr c07db80ef0 Add BASE_URL and BASE_API_URL to BandcampExtractorHelper 2021-02-21 12:15:45 +01:00
TobiGr adde4332d1 Code improvements 2021-02-20 23:58:54 +01:00
FireMasterK 42bdd8adc5
Update to fix failing mock tests. 2021-02-20 17:14:41 +05:30
FireMasterK e8bccfaf5d
Change UA to privacy.resistFingerprinting. 2021-02-20 17:14:41 +05:30
bopol a9aa385bb3
Merge pull request #554 from B0pol/test-verified
use DownloaderFactory instead of DownloaderTestImpl for search Channe…
2021-02-20 12:12:43 +01:00
bopol 0c831afa7b use DownloaderFactory instead of DownloaderTestImpl for search ChannelVerified 2021-02-20 11:41:30 +01:00
TobiGr ea120a4637 remove print stacktrace
use interface
2021-02-20 00:23:45 +01:00
XiangRongLin adf9d7d10f Add reason field to MockOnly
This enforces developers to document why a test is skipped
2021-02-19 18:54:18 +01:00
XiangRongLin ea52030613 Add MockOnlyRule to allow skipping specific tests based on downloader 2021-02-19 18:54:17 +01:00
TobiGr 02920fafa8 Add isUploaderVerified() 2021-02-19 15:51:23 +01:00
TobiGr 91e9309486 Merge remote-tracking branch 'origin/dev' into bandcamp 2021-02-19 13:37:45 +01:00
TobiGr fa61b864f2 Code improvements 2021-02-19 13:37:24 +01:00
TobiGr 98268e351c Move radio URL check into a function 2021-02-19 13:35:44 +01:00
FireMasterK 959d5d7db7
Add dependabot. 2021-02-19 11:52:56 +05:30
bopol 27a20e41cd
Merge pull request #550 from FireMasterK/add-more-data
Add more data.
2021-02-18 20:28:27 +01:00
FireMasterK 5333d8a98b
Add more data. 2021-02-19 00:55:03 +05:30
FireMasterK 88890d81a5
Update gradle wrapper. 2021-02-18 21:15:12 +05:30
bopol 9c1a7f7df6
Merge pull request #521 from B0pol/verified
Add uploader verified by service extraction
2021-02-18 16:39:53 +01:00
bopol 1a322ad8ed Add uploader verified by service extraction 2021-02-18 16:36:40 +01:00
bopol db253e202b
Merge pull request #535 from TeamPiped/adaptive-parsing
Adaptive Stream parsing
2021-02-18 15:55:27 +01:00
FireMasterK c24afa2cbb Permalink URL. 2021-02-18 20:16:53 +05:30
FireMasterK 8d54401233 Add fps field as well. 2021-02-18 20:16:53 +05:30
FireMasterK 4d096be14a Revert "Update gradle wrapper."
This reverts commit 591856e3a11998c4b8ba87b5e361e2b9556d760f.
2021-02-18 20:16:53 +05:30
FireMasterK 5b70645d85 Add data to muxed streams aswell. 2021-02-18 20:16:53 +05:30
FireMasterK e975d33fbe Change jdoc / comments. 2021-02-18 20:16:53 +05:30
FireMasterK f31b2a68fd use getters/setters. 2021-02-18 20:16:53 +05:30
FireMasterK d4945ac55c Constructor cleanup. 2021-02-18 20:16:52 +05:30
FireMasterK 3f6a601be8 Fix jdoc / comments. 2021-02-18 20:16:52 +05:30
FireMasterK a7c9905183 Fix requested changes. 2021-02-18 20:16:52 +05:30
FireMasterK 0c0f2d74bc Make fields private
Co-authored-by: bopol <bopol@e.email>
2021-02-18 20:16:52 +05:30
FireMasterK d1054338d3 Fix error with livestreams. 2021-02-18 20:16:52 +05:30
FireMasterK 11eb4932f4 Add data to respective classes. 2021-02-18 20:16:52 +05:30
FireMasterK 525e345ed8 remove duplicate field. 2021-02-18 20:16:52 +05:30
FireMasterK e844d2aed3 Add dash fields to the ItagItem. 2021-02-18 20:16:52 +05:30
FireMasterK b713a7af8c Update gradle wrapper. 2021-02-18 20:16:52 +05:30
bopol 64f6b0478f
Merge pull request #545 from XiangRongLin/comment_reset
Add comment explaining YoutubeParsingHelper.resetClientVersionAndKey
2021-02-18 15:14:03 +01:00
XiangRongLin a64579da2f Unignore some tests that work on Linux 2021-02-17 20:31:26 +01:00
XiangRongLin b99cb8624f Unignore PublicBroadcasterTest
Tests only work on linux, mocks have to be generated on linux too
2021-02-17 20:31:26 +01:00
XiangRongLin 37366c65ca Remove encoding of '&' to '&amp;' in test and unignore it 2021-02-17 20:31:26 +01:00
XiangRongLin c327922cc0 Add explanations to broken tests 2021-02-17 20:31:26 +01:00
XiangRongLin 26d50aca05 Add comment explaining YoutubeParsingHelper.resetClientVersionAndKey 2021-02-17 19:21:39 +01:00
TobiGr 54aa5b3042 Use propper structure in KioskExtractors
Made BandCampRadioExtractor a Kiosk which holds StreamInfoItems and not InfoItems.
2021-02-16 21:09:42 +01:00
TobiGr 54b8e54f80 Fix potential NPE 2021-02-16 20:59:13 +01:00
TobiGr 16973126a0 Use Collections.emptyList() instead of Collections.EMPTY_LIST or new ArrayList() 2021-02-16 20:25:48 +01:00
bopol 84380e4c3a
Merge pull request #536 from B0pol/watchendpoints
support /watch/, /v/ and /w/ subpaths
2021-02-16 20:23:27 +01:00
bopol 41f689b099 Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/linkHandler/YoutubeStreamLinkHandlerFactory.java
Co-authored-by: Stypox <stypox@pm.me>
2021-02-16 19:17:10 +01:00
XiangRongLin a9594f531f
Merge pull request #543 from XiangRongLin/unignore1
Unignore and fix some tests
2021-02-16 18:00:16 +01:00
XiangRongLin a28e0226c5
Merge pull request #519 from XiangRongLin/better_logging_error_handling
Better logging, error handling, doc for mock tests
2021-02-16 15:06:06 +01:00
XiangRongLin 05cb22c654 Unignore Gronkh.testDescription
Remove special character ü from assertion
2021-02-16 08:59:35 +01:00
XiangRongLin d9b4c2779f Unignore YoutubeSearchQHTest.testRegularValues 2021-02-16 08:59:35 +01:00
XiangRongLin 137272b7a4 Unignore YoutubeCommentsExtractorTest.EmptyComment
Mock files were missing
2021-02-16 08:59:34 +01:00
TobiGr 5bf9fddba9 Code improvements 2021-02-15 22:45:17 +01:00
TobiGr e062c8cb0d Merge branch 'dev' into bandcamp 2021-02-15 22:09:41 +01:00
bopol 416cf17b7a
Merge pull request #532 from B0pol/utf8
Code improvement
2021-02-15 18:56:28 +01:00
bopol 7a3d9bdb7d Description: add .equals() and .hashCode() 2021-02-15 18:51:47 +01:00
bopol 557934cb17 use EMPTY_STRING 2021-02-15 18:51:46 +01:00
bopol 9ca52ca68a use only one constant UTF-8 2021-02-15 18:47:54 +01:00
XiangRongLin 50e5718bd1
Merge pull request #518 from XiangRongLin/generate_mocks
Use and generate mocks for youtube tests
2021-02-15 06:39:41 +01:00
Tobi d7486f251a
Merge pull request #538 from B0pol/soundcloud-georestriction
Soundcloud: fix charts extraction when georestricted
2021-02-14 23:52:27 +01:00
TobiGr 432c68cdec Update okhttp from 3.12.11 to 3.12.13
Changelog https://square.github.io/okhttp/changelog_3x/#version-31213
2021-02-14 23:44:19 +01:00
TobiGr 5eb75d72d3 Release 0.20.10 2021-02-14 22:09:17 +01:00
XiangRongLin a88b2e6af4 Undo formatting of YoutubeStreamExtractorDefaultTest 2021-02-14 14:57:23 +01:00
bopol 8d89c82caa Soundcloud: fix charts extraction when georestricted 2021-02-13 22:28:31 +01:00
bopol ee2012c948 rename folder to subpath 2021-02-13 22:25:46 +01:00
bopol 8db40db4de use substring instead of split 2021-02-13 22:20:05 +01:00
bopol 81317cf6bc youtube: support /w/ folder 2021-02-13 22:20:05 +01:00
bopol f15c0fcfed support /watch/ and /v/ folders 2021-02-13 22:20:05 +01:00
XiangRongLin 971128c464 Ignore failing tests 2021-02-13 20:28:03 +01:00
XiangRongLin 92dea0806a Regenerate json files for youtube tests 2021-02-13 20:28:03 +01:00
XiangRongLin d6b9930c18 Use DownloaderFactory in YoutubeCommentsExtractorTest.Pinned 2021-02-13 20:12:39 +01:00
XiangRongLin b555f38fa5 Use real downloader for YoutubeSubscriptionExtractorTest
It doesn't make network requests anyway
2021-02-13 20:12:39 +01:00
XiangRongLin f352b0c7af YoutubeMusicSearchExtractorTest can't be used with mocks
Requires further research
2021-02-13 20:00:51 +01:00
XiangRongLin f45b5610bf Unignore YoutubeChannelLocalizationTest
With mocks time is not a concern anymore
2021-02-13 20:00:51 +01:00
XiangRongLin 19737e06ee Use DownloaderFactory in YoutubeChannelExtractorTest
KurzgesagtAdditional needed to be extracted because it ran a duplicated request.
2021-02-13 20:00:51 +01:00
XiangRongLin 84eb4b30c9 Use DownloaderFactory in YouTube tests 2021-02-13 20:00:51 +01:00
XiangRongLin b43f3474f5 Make resetClientVersionAndKey public 2021-02-13 20:00:49 +01:00
Tobi 8d7b62914c
Merge pull request #540 from B0pol/fix-tests
fix tests, fix links in YouTube description
2021-02-13 13:20:29 +01:00
bopol 2859c7ba5c add comment explaining why .substring(23) 2021-02-13 12:10:41 +01:00
bopol 73353996d7 fix more tests 2021-02-12 23:18:38 +01:00
bopol a931e31239 fix tests 2021-02-12 22:36:18 +01:00
TobiGr d116680ea3 Merge branch 'master' into dev 2021-02-12 22:26:35 +01:00
TobiGr f8d769617a Release 0.20.9 2021-02-12 22:14:23 +01:00
Tobi c2525916ac
Merge pull request #539 from B0pol/yt-channel-continuation
[YouTube]: Fixed channel continuation
2021-02-12 22:09:31 +01:00
bopol b0f356dd76 YT channels: fix continuation 2021-02-12 14:39:09 +01:00
bopol fcdb50b825 make getInfo from Extractor public
In StreamInfo and CommentsInfo
2021-02-09 14:58:43 +01:00
bopol 44c54d403a
fix Invidious channels being accepted by YoutubeStreamLinkHandlerFactory (#527)
fixes https://github.com/TeamNewPipe/NewPipeExtractor/issues/524, see it for more details
2021-02-05 18:57:58 +01:00
TiA4f8R 1414a6f178
Fix extraction of YouTube chapters (#531)
YouTube changed the name of the chapters in the JSON from engagement-panel-macro-markers to engagement-panel-macro-markers-description-chapters, so extracting chapters doesn't work.
2021-02-03 22:07:34 +01:00
Tobias Groza bfa639950e
Merge pull request #528 from Stypox/ytmusic-0
[YouTube Music] Fix extracting search item view/subscriber count when = 0
2021-01-29 23:23:23 +01:00
TobiGr d728c4fad0 Remove unused imports
I searched for "StandardCharsets" to prevent possible bugs with Android KitKat. The import is unused and is therefor removed.
There are other usages of the StandardCharsets class in the test implementation, but those are not relevant.
2021-01-29 23:16:40 +01:00
Stypox a64dfd7343
[YouTube Music] Fix extracting search item view/subscriber count when = 0 2021-01-29 16:44:24 +01:00
Tobias Groza c00ee75d74
Merge pull request #525 from B0pol/rename
rename getHeartedByCreator and getPinned to isPinned and isHeartedByC…
2021-01-23 13:45:24 +01:00
bopol 2662ceca22 rename getHeartedByCreator and getPinned to isPinned and isHeartedByCreator 2021-01-23 13:06:07 +01:00
Tobias Groza 10e8c543e8
Merge pull request #523 from B0pol/pinned
Extract whether the comment is pinned
2021-01-23 12:35:13 +01:00
bopol 66e4eb2f96 extract whether the comment is pinned 2021-01-22 23:59:22 +01:00
Tobias Groza 69f155d292
Merge pull request #522 from B0pol/invidious_instances
update invidious instances list
2021-01-22 21:04:10 +01:00
bopol 67ddfefdc6 keep invidious instances list in the same order
as suggested by @TobiGr
2021-01-22 20:58:14 +01:00
bopol 53d3f7989b update invidious instances list 2021-01-22 19:30:25 +01:00
XiangRongLin 88e4c8667b Add comments to RecordingDownloader about usage 2021-01-17 19:53:20 +01:00
XiangRongLin eecfe09f2c Check for non-existent folder in MockDownloader 2021-01-17 19:52:58 +01:00
XiangRongLin a0288b8fd6 Echo which downloader is used during CI process 2021-01-17 19:52:28 +01:00
Tobias Groza beb705013c
Merge pull request #516 from XiangRongLin/schedule_ci_with_real
Add if-else statement to run tests against real website on schedule
2021-01-16 12:59:05 +01:00
XiangRongLin 7a6d930e7b Add if-else statement to run tests against real website on schedule 2021-01-16 10:03:47 +01:00
TobiGr 07570554b6 Fix build and optimize imports 2021-01-15 21:55:40 +01:00
TobiGr 78c2113094 Merge remote-tracking branch 'origin/dev' into bandcamp 2021-01-15 21:49:58 +01:00
Stypox b9ba95614b
Merge pull request #514 from XiangRongLin/test_jvm_property
Pass on gradle JVM system property to test JVM
2021-01-15 20:44:23 +01:00
XiangRongLin a6cfe11ae9 Pass on gradle JVM system property to test JVM 2021-01-15 20:37:14 +01:00
Stypox 48a9993edd
Merge pull request #513 from XiangRongLin/ci_use_mock
Adjust CI workflow to run tests with mocks
2021-01-15 20:20:23 +01:00
Stypox 588fbcc61f
Merge pull request #512 from XiangRongLin/ignore_failing
Ignore all failing tests
2021-01-15 20:18:03 +01:00
XiangRongLin 6888e7773f Ignore all failing tests 2021-01-15 20:11:40 +01:00
XiangRongLin e5dc62d9bd Add argument -Ddownloader=MOCK to CI job
Now MockDownloader will be used in tests.
2021-01-15 18:04:33 +01:00
Stypox 7c774c84ce
Merge pull request #505 from TeamNewPipe/meida.ccc.de_search_fix
[media.ccc.de] Fix NPE in search results if they contain a future talk
2021-01-15 14:16:59 +01:00
Stypox c2c4d97efd
Merge pull request #482 from XiangRongLin/mock_mix_pl_test
Mock mix pl tests
2021-01-15 12:54:48 +01:00
XiangRongLin 4f81d9d689 Regenerate mock files 2021-01-15 09:51:23 +01:00
Stypox 951159f75f
Merge pull request #504 from TeamNewPipe/media.ccc.de_recent_order
[media.ccc.de] Recent kiosk: order streams by upload date
2021-01-14 23:31:09 +01:00
Tobias Groza cc51c5f6a3
Merge pull request #507 from vkay94/add-interval-to-stream-frames
Add durationPerFrame to Frameset
2021-01-14 20:27:40 +01:00
XiangRongLin 6ca7123d5d Add comment for usage of YoutubeParsingHelper.resetClientVersionAndKey 2021-01-14 20:01:52 +01:00
vkay94 11dcfe638b Add durationPerFrame to Frameset and getFrameBoundsAt method 2021-01-14 20:01:06 +01:00
Tobias Groza c2ff6723d7
Merge pull request #511 from B0pol/errors
Fix PeerTube tests by changing instance
2021-01-14 19:15:22 +01:00
Tobias Groza 6773ec8699
Merge pull request #510 from B0pol/heartedbycreator
comments: add heartedByUploader and extract it for YouTube
2021-01-12 19:00:04 +01:00
TobiGr 0c4836c800 [media.ccc.de] Recent kiosk: Add comment clarifying "length" and "duration" fields have the same value 2021-01-12 18:48:42 +01:00
XiangRongLin 4af50c6870 Use @link for downloader 2021-01-12 09:41:57 +01:00
XiangRongLin 3455f0f23c Add documentation to testing with mocks 2021-01-12 09:37:48 +01:00
bopol 5292cc1636 [peertube] test: remove usage of peertube.mastodon.host
It's down for months
2021-01-11 19:46:27 +01:00
bopol f3c22da618 comments: add heartedByCreator and extract it for youtube 2021-01-11 18:48:08 +01:00
Tobias Groza da968cf9c5
Merge pull request #508 from XiangRongLin/360_webm
[YouTube] Un-comment out 360p webm
2021-01-11 16:09:05 +01:00
XiangRongLin 63c237db41 Only create directories if they do not exist 2021-01-11 07:57:11 +01:00
XiangRongLin e82cb1efea Remove dependency to commons-io 2021-01-10 20:50:38 +01:00
XiangRongLin 1ea6c6ce54 Prefix mock file from RecordingDownloader with "generated_mock_"
Only read those files in MockDownloader
2021-01-10 20:46:42 +01:00
XiangRongLin f91916c017 Remove unnecessary file object creation 2021-01-10 20:39:22 +01:00
XiangRongLin 255c726f20 Add resource path base to DownloaderFactory 2021-01-10 20:27:34 +01:00
XiangRongLin 35e299759e Add method to allow resetting youtube client version and key
This is needed so that a request is made for each test class when running multiple at once. This way RecordingDownloader records all necessary requests.
This works as long as tests are run sequentially and not in parallel.
2021-01-10 20:24:50 +01:00
XiangRongLin f447a7a450 fix import due to downloader package move 2021-01-10 19:49:14 +01:00
XiangRongLin f8aa989c42 Add generated json and manually copy client_version.json 2021-01-10 19:38:47 +01:00
XiangRongLin 285c26eafa Adjust YoutubeMixPlaylistExtractorTest to use DownloaderFactory 2021-01-10 19:38:41 +01:00
XiangRongLin e6e8e39def Add DownloaderFactory to return a specific downloader based on 2 variables.
If the system property 'downloader' is set that use that specific downloader. This is used from gradle by appending `-Ddownloader=ABCD to the command.
ABCD is one of DownloaderType.
The other variable is the static property `DEFAULT_DOWNLOADER` in DownloaderFactory, which can be easily changed as needed inside the IDE according to development needs`.

Normal workflow would be to first use the recording downloader and afterwards only use mocks, if the requests are always staying the same.
2021-01-10 19:36:24 +01:00
XiangRongLin 7c40fb8bf7 Add additional downloader implementations
RecordingDownloader relies on the real downloader and saves the request/response pair into a json file.
MockDownloader uses json files from above and mocks responses for specific requests.
2021-01-10 19:36:24 +01:00
XiangRongLin 1bcb9c76a7 Generate equals and hashCode for Request 2021-01-10 19:36:24 +01:00
XiangRongLin 4dad3d60d6 Move DownloaderTestImpl into downloader package 2021-01-10 19:36:21 +01:00
XiangRongLin fe3902c669 Un-comment out 360p webm 2021-01-08 19:43:03 +01:00
TobiGr 2d93b23723 [media.ccc.de] Show conference title in search results instead of slug 2021-01-08 14:15:25 +01:00
TobiGr 52f3bd15f2 [media.ccc.de] Add conference title to "recent" kiosk
The "conference_title" field was added upstream recently (see https://github.com/voc/voctoweb/issues/495)
2021-01-08 12:04:39 +01:00
Tobias Groza ab62464e86
Merge pull request #495 from B0pol/sepiasearch
[PeerTube] Implement sepia search
2021-01-06 11:06:36 +01:00
bopol 0bcea9c2cd add JDoc 2021-01-01 14:37:32 +01:00
bopol 0e45c25ea9
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/peertube/extractors/PeertubeStreamInfoItemExtractor.java
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2021-01-01 14:20:47 +01:00
TobiGr fae67fbd45 Merge remote-tracking branch 'origin/master' into dev 2021-01-01 13:06:11 +01:00
Tobias Groza c4b9183684
Merge pull request #506 from TeamNewPipe/gh-actions
Migrate to GitHub actions from Travis
2021-01-01 10:35:33 +01:00
TobiGr 4671e68304 Update links to Java SE components to Java SE 8 2020-12-31 18:05:49 +01:00
TobiGr d63d20f943 Fix JDoc 2020-12-31 18:05:40 +01:00
FireMasterK 1ba689900e Migrate to GitHub actions from Travis. 2020-12-31 18:05:35 +01:00
TobiGr 2cbc3ccb1a [media.ccc.de] Fix live stream test if there are no current live streams 2020-12-31 17:03:23 +01:00
TobiGr a896ec41b9 [media.ccc.de] Use Pattern for getting the ive stream id 2020-12-31 17:03:23 +01:00
TobiGr 5dc9a76e3c [media.ccc.de] Recent kiosk: order streams by upload date 2020-12-30 18:38:09 +01:00
TobiGr edf8dd0e92 [media.ccc.de] Fix NPE in search results if they contain a future talk 2020-12-30 18:34:28 +01:00
Tobias Groza 3c8c8e7307
Merge pull request #497 from TiA4f8R/fix-player-js-url-redirect
Fix YouTube player JavaScript URL redirection
2020-12-27 18:04:40 +01:00
TobiGr 288ddab65b update version name 2020-12-27 18:02:16 +01:00
TiA4f8R d30caeb091
Fix YouTube player JavaScript URL redirection 2020-12-27 15:55:03 +01:00
TobiGr deb9af7bf5 [media.ccc.de] live kiosk - display info of the current talk 2020-12-27 15:52:09 +01:00
TobiGr 21c6a7884c Release 0.20.8 2020-12-27 14:50:34 +01:00
Tobias Groza 1259521d53
Merge pull request #500 from TeamNewPipe/media.ccc.de_live
[media.ccc.de] live stream kiosk
2020-12-27 14:47:55 +01:00
TobiGr b4e1913971 [media.ccc.de] Play live streams 2020-12-27 13:25:48 +01:00
TobiGr 80f4d42226 [Draft] Add live extractor 2020-12-27 11:01:22 +01:00
TobiGr ed9402c002 [media.ccc.de] Escape "." in id pattern 2020-12-27 11:01:10 +01:00
TobiGr 674f2227f3 make url pattern static 2020-12-27 01:27:37 +01:00
TobiGr 090dade5b3 [media.ccc.de] Add short test for recent kiosk 2020-12-27 01:03:08 +01:00
TobiGr 0cfefe222a Revert "[media.ccc.de] Get conference name from API URL"
This reverts commit d1a0686d2f.
2020-12-26 16:55:44 +01:00
TobiGr d1a0686d2f [media.ccc.de] Get conference name from API URL 2020-12-26 13:15:03 +01:00
TobiGr 41699798fe [media.ccc.de] Use web URL for uploaderUrl instead of API URL in recent 2020-12-26 12:47:32 +01:00
TobiGr b2154c6e87 [media.ccc.de] Fix conferences kiosk test 2020-12-25 23:55:45 +01:00
TobiGr ed6ae329b3 [media.ccc.de] Add "recent" kiosk 2020-12-25 22:26:58 +01:00
TobiGr b2837698f5 Release 0.20.7 2020-12-25 18:55:06 +01:00
Tobias Groza b5e50cc9fb
Merge pull request #484 from TeamNewPipe/yt_,music_search
Fix YouTube Music search
2020-12-25 18:30:25 +01:00
TobiGr d9e2da53c3 Replace explicit string checks whether a playlist íd is a certain YouTube Mix type with calling the dedicated methods 2020-12-25 15:00:34 +01:00
TobiGr abee0a8df1 Fix typos in YoutubePlaylistLinkHandlerFactory 2020-12-24 10:34:21 +01:00
TobiGr 3033c0b993 [YouTube] Music Mix: Fix playlist not being accepted
Regression introduced by YouTube Mix support (#280)
2020-12-23 21:08:36 +01:00
TobiGr b62144b49d [YouTube] Music: Fix search extraction caused by changed JSON response 2020-12-23 21:08:36 +01:00
TobiGr c9d9bd1e24 Fix typo in ListLinkHandlerFactory#getUrl 2020-12-23 21:05:58 +01:00
bopol 3ae924a7f1 Implement sepia search 2020-12-23 15:07:55 +01:00
TobiGr 942d840624 Fix typo in AffiliateService: rename "NO_AFILIATE" to "NO_AFFILIATE" 2020-12-23 12:50:45 +01:00
TobiGr 865c42e273 Fix line break 2020-12-23 12:49:50 +01:00
Tobias Groza c682ea0d18
Merge pull request #451 from TeamNewPipe/meta_info
Extract stream and search meta info for YouTube
2020-12-22 23:05:33 +01:00
Fynn Godau c9e9953bb0 [Bandcamp] Fix channel link handler factory 2020-12-22 20:02:53 +01:00
TobiGr 41a8ed625d Make some vars final and add annotations to methods 2020-12-22 18:19:26 +01:00
TobiGr bc6de14952 Extract stream and search meta info for YouTube
Add method to extract Google webcache URLs.
2020-12-22 18:19:26 +01:00
TobiGr 50903730b1 [Bandcamp] Fix accepting HTTP URLs 2020-12-22 08:53:10 +01:00
Tobias Groza 853a65a1a6
Merge pull request #491 from TeamNewPipe/dependecies
Update junit
2020-12-22 08:31:03 +01:00
TobiGr 838dc1ab54 Fix vulnerability in junit 2020-12-21 23:22:16 +01:00
Tobias Groza 7e481cf81a
Merge pull request #490 from B0pol/peertube_related
[peertube] fix related streams api url when no tags
2020-12-20 19:04:35 +01:00
bopol 8d3436565f [peertube] fix related streams api url when no tags
fixes #489
2020-12-19 23:25:37 +01:00
Tobias Groza 22a415156f
Merge pull request #479 from vkay94/stream-segments
Extract stream segments for YouTube
2020-12-19 11:12:22 +01:00
TobiGr 74b46fed2d [Bandcamp] Fix deprecation in parseDate helper function 2020-12-19 10:40:21 +01:00
TobiGr 116e921d6c Merge remote-tracking branch 'origin/dev' into bandcamp 2020-12-19 00:26:11 +01:00
Tobias Groza e8cc302d5f
Merge pull request #483 from XiangRongLin/cleanup-mix-pl
Cleanup mix pl
2020-12-19 00:13:17 +01:00
TobiGr df16a8646f [Bandcamp] Improve radio stream extractor test 2020-12-18 23:15:59 +01:00
XiangRongLin cdcb66b93e Clean up stuff from mix pl
Coming from unclean merge and missing hints from IDE
2020-12-16 08:35:28 +01:00
XiangRongLin ba8782a9ed Use lowercase string for extracting cookies
The map in a real response happens to ignore upper-/lowercase differences. Other maps used in unit test may not have that behaviour.
2020-12-16 08:32:04 +01:00
Stypox 85fa006214
Merge pull request #280 from XiangRongLin/mixPL
Extractor for youtube mix (auto-generated playlist)
2020-12-14 18:11:07 +01:00
XiangRongLin f90f6fcf92 [YouTube] Don't escape & in getUrlFromNavigationEndpoint for playlists 2020-12-12 20:40:13 +01:00
Xiang Rong Lin a338e4e08e [Youtube] Apply review suggestions and avoid channel mix edge case 2020-12-12 20:32:43 +01:00
Xiang Rong Lin 22d2f7e400 [Youtube] Add cookies to youtube mix request
This way youtube wont return duplicates when getting more items of the mix (but youtube can also track us)
2020-12-12 20:32:40 +01:00
Stypox 421935401f [Youtube] Add subchannel functions to mix and fix imports 2020-12-12 20:30:57 +01:00
Xiang Rong Lin 3ff8619bcc [Youtube] apply wb9688 suggestion (mix)
Channel mix adjusments and test
Don't accept youtube music mix urls as playlist
Don't override playlistData to keep getInitialPage()
Remove json constants
Indentation
2020-12-12 20:30:57 +01:00
Xiang Rong Lin 822cf307f7 [Youtube] Add _ITEMS constants and improve code style
Move thumbnail id exctraction code to getThumbnailUrlFromId
Add test for "My mix" detection to service tests
Use ITEM_COUNT_UNKNOWN everywhere instead of -1 and add some tests
2020-12-12 20:30:57 +01:00
Xiang Rong Lin df38b1926c [Youtube] Add tests and take thumbnail image always from first video of mix
Also fix getThumbnailUrl for "My Mix"
2020-12-12 20:30:57 +01:00
Xiang Rong Lin 68a3948af6 [Youtube] Fix get banner url 2020-12-12 20:30:57 +01:00
Xiang Rong Lin ec6b99c082 [Youtube] Adjust mix extractor to new user agent
Also extract continuation now
2020-12-12 20:30:57 +01:00
Xiang Rong Lin 0ff054acb4 [Youtube] Extract initial playlist info 2020-12-12 20:30:57 +01:00
Xiang Rong Lin a376792a5d [Youtube] Handle case where url is in "youtube.com/playlist?list=listID" format.
This occurs when sharing a mix from the official youtube app.
2020-12-12 20:30:57 +01:00
Xiang Rong Lin 327a5730a8 [Youtube] Add some comments to mix 2020-12-12 20:30:57 +01:00
Xiang Rong Lin d74265c846 [Youtube] Extract getThumbnailUrl into method and change getUploaderName 2020-12-12 20:30:57 +01:00
Xiang Rong Lin 0efb854d27 [Youtube] Implement mix extractor for auto-generated playlists.
-New YoutubeMixPlaylistExtractor, that extracts from a mix (auto-generated playlist).
-The url has the format of "youtube.com/watch?v=videoID&playlistID",
where playlistID always starts with "RD" and usually followed by the videoID.
-Change YoutubePlaylistLinkHandlerFactory to create a linkhandler with the given url if it is a mix.
-Change YoutubeService to return YoutubeMixPlaylistExtractor if the url is a mix.
2020-12-12 20:30:57 +01:00
vkay94 2ba27b39af Stream segments: Remove replaceAll from setPreviewUrl 2020-12-12 15:00:45 +01:00
Stypox 2b622fd2f1
Merge pull request #419 from Isira-Seneviratne/Use_Objects_methods
Use Objects methods.
2020-12-12 14:52:42 +01:00
vkay94 9a7a224a54 Add tests for stream segments 2020-12-12 10:24:55 +01:00
vkay94 d3eea4f4be Add stream segments to StreamInfo 2020-12-12 10:24:29 +01:00
Isira Seneviratne 57be1f1123 Use Objects.requireNonNull(). 2020-12-11 06:09:57 +05:30
Isira Seneviratne 4c19a88612 Use Objects.toString(). 2020-12-11 06:09:50 +05:30
Isira Seneviratne b8f64595a2 Use Objects' static equals() and hashCode() methods. 2020-12-11 06:09:44 +05:30
TobiGr b3835bd616 Release 0.20.6 2020-12-09 23:30:23 +01:00
Tobias Groza 6fbcdd24ee
Merge pull request #477 from vkay94/fix-comments-parsing
Fix comments parsing
2020-12-09 23:25:51 +01:00
vkay94 9dbacbc618 Fix comments parsing 2020-12-09 14:57:38 +01:00
Fynn Godau 04dd3d4d32 Rework link handlers to correctly accept external websites 2020-12-05 15:08:26 +01:00
Tobias Groza 8ade913d9b
Merge pull request #472 from TeamNewPipe/fix-media-ccc-de
Fix name of media.ccc.de service
2020-12-02 22:16:52 +01:00
TheAssassin ba3e2302bc Fix name of media.ccc.de service 2020-12-01 23:54:27 +01:00
TobiGr 175df679e0 Release 0.20.5 2020-12-01 08:18:29 +01:00
Tobias Groza b7a995187f
Merge pull request #463 from opusforlife2/update_invidious_URLs
Update Invidious URL List
2020-11-28 13:30:32 +01:00
Fynn Godau be562b8436 Change tests 2020-11-27 16:48:40 +01:00
TobiGr c91e21b37c [Bandcamp] Tests - Add finals and improve code formatting 2020-11-24 14:13:31 +01:00
TobiGr 99e7ef013e [Bandcamp] Apply small changes to code formatting and style
Make variables final when possible
Remove unused imports
Improve code formatting
2020-11-24 14:13:31 +01:00
fynngodau 8c369b0f79
Rephrase link in javadoc
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-11-24 14:01:31 +01:00
TobiGr 5a3f96b967 Release 0.20.4 2020-11-22 22:29:08 +01:00
Tobias Groza 650f0920fe
Merge pull request #465 from XiangRongLin/playlist_continuation
[YouTube] Fix playlist continuations extraction
2020-11-20 20:48:20 +01:00
TobiGr 334e1e9b53 Fix YouTube comments test 2020-11-20 19:53:53 +01:00
Fynn Godau 186936d041 Various changes regarding tests 2020-11-19 22:33:52 +01:00
Fynn Godau 6bc7e3420e Merge TNP/dev into fynngodau/dev 2020-11-19 21:32:08 +01:00
TobiGr 3582a5189f Release 0.20.3 2020-11-18 20:41:25 +01:00
XiangRongLin 5bceff0083 [YouTube] Fix extraction of next page url for the last page of playlist 2020-11-18 19:03:12 +01:00
XiangRongLin 8347e14952 [YouTube] Fix playlist continuations extraction 2020-11-17 21:04:53 +01:00
opusforlife2 2174055ef8
Update Invidious URL list in Parsing Helper 2020-11-11 15:12:31 +00:00
opusforlife2 c0afd5213a
Update Invidious URL list in Link Handler Factory 2020-11-11 14:54:16 +00:00
Tobias Groza 6701b0fe71
Merge pull request #447 from Stypox/better-deobfuscation
[YouTube] Cache deobfuscation code and more fixes
2020-11-08 00:32:36 +01:00
Stypox 89a77ae74a
[YouTube] Fix detection of ended livestreams and parse livestream upload date 2020-11-04 16:03:08 +01:00
Stypox 827f7bd137
[YouTube] Cache deobfuscation and improve requests made
Fix age restriction extraction
Automatically fixes more things
2020-11-04 16:02:12 +01:00
Stypox b42a196f35
Merge pull request #454 from Isira-Seneviratne/Fix_date_time_issues
Fix some issues related to the date/time changes.
2020-11-04 14:48:20 +01:00
Isira Seneviratne 4fe28d7e3a Fix YouTube parse error when only a date is present. 2020-11-04 05:49:42 +05:30
Isira Seneviratne 9cf9e7e980 Call existing constructor in DateWrapper. 2020-11-04 05:49:41 +05:30
Isira Seneviratne fe31a90cb3 Remove DateTimeFormatter.ISO_OFFSET_DATE_TIME usage. 2020-11-04 05:49:41 +05:30
Tobias Groza b13c7e1c1e
Merge pull request #452 from Stypox/yt-import
Implement YouTube subscription import from Google takeout
2020-11-03 20:32:17 +01:00
Tobias Groza ac50068bbe
Merge pull request #456 from B0pol/youtube_comments_test
create YouTubeCommentsLinkHandlerFactoryTest
2020-11-03 20:14:14 +01:00
bopol 345e136f6c create YouTubeCommentsLinkHandlerFactoryTest
and remove invidious test from YouTubeCommentsExtractorTest, because it was just testing if the URL is accepted, then the extractor does the same thing, we don't need to test the same thing twice
2020-11-03 19:10:10 +01:00
Stypox 501ec30152
Implement youtube subscription import from Google takeout 2020-11-02 11:04:48 +01:00
Tobias Groza 564a965810
Merge pull request #453 from bd0n4lds/fix-donation-typo
Fix typo in DonationLinkHelper and rewrote swtich statement
2020-11-02 00:15:04 +01:00
Bri@n 82746d172f Fix typo in DonationLinkHelper and rewrote swtich statement 2020-11-01 17:34:34 -05:00
Stypox 2f02c0e6a4
Merge pull request #450 from B0pol/fileutils
create FileUtils
2020-11-01 18:13:57 +01:00
bopol f69b3ef05d create FileUtils 2020-11-01 17:15:34 +01:00
TobiGr be9e160333 Fix build 2020-11-01 17:04:02 +01:00
Tobias Groza 8cbdec675b
Merge pull request #417 from Isira-Seneviratne/Use_Java_8_Date_Time_API
Use the Java 8 Date/Time API.
2020-11-01 16:49:06 +01:00
Isira Seneviratne fcdfe7d939 Update README.md. 2020-11-01 06:28:54 +05:30
Isira Seneviratne 4f04cfccca Switch from Calendar to OffsetDateTime in DateWrapper. 2020-11-01 06:28:54 +05:30
Isira Seneviratne ee3af63c04 Switch to ChronoUnit. 2020-11-01 06:28:53 +05:30
Isira Seneviratne 0526a5148d Set source and target versions to Java 8. 2020-11-01 06:28:52 +05:30
Isira Seneviratne b2d0c098a3 Merge branch 'dev' of https://github.com/TeamNewPipe/NewPipeExtractor into dev 2020-11-01 06:27:59 +05:30
Stypox 6cc50b57e3
Merge pull request #410 from Scrxtchy/sets-track-title
Fix parsing Soundcloud tracks that contain the term 'sets'
2020-10-31 21:43:21 +01:00
Stypox b242e1d113
Merge pull request #362 from Stypox/google-search-urls
Add support for Google search redirect url
2020-10-31 21:03:48 +01:00
Stypox 9e53cf0b56
Fix parameter reassignment and other style issues
Also remove left-behind debug statement
2020-10-31 20:58:28 +01:00
Stypox 3fe55b30ba
Add support for Google search redirect url 2020-10-31 20:58:28 +01:00
Stypox c190a3029b
Consider protocol as base url when it is a custom one (e.g. vnd.youtube) 2020-10-31 20:58:28 +01:00
Isira Seneviratne 35c7b5640c Merge branch 'dev' of https://github.com/TeamNewPipe/NewPipeExtractor into dev 2020-10-31 05:45:02 +05:30
Stypox a9303b24a5
Merge pull request #445 from B0pol/peertube_fix
[PeerTube] fix account and channel extractors
2020-10-30 19:35:51 +01:00
bopol b21e59925d [PeerTube] fix account and channel extractors 2020-10-29 19:52:29 +01:00
Isira Seneviratne 61dcbbb800 Merge branch 'dev' of https://github.com/TeamNewPipe/NewPipeExtractor into dev
 Conflicts:
	extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java
2020-10-29 12:20:16 +05:30
Stypox bcc01b99c7
Merge pull request #444 from Stypox/error-message
Remove any reference to decrypt and improve error message
2020-10-28 23:41:04 +01:00
Stypox 30ed4f2d63
Remove any reference to decrypt and improve error message 2020-10-28 20:26:00 +01:00
TobiGr 62912ee834 Merge branch 'master' into dev 2020-10-28 14:28:06 +01:00
TobiGr 94176c5d68 Merge branch 'master' into dev 2020-10-28 07:52:50 +01:00
Tobias Groza 19c0e8700d
0.20.2 2020-10-28 07:29:42 +01:00
Tobias Groza 354426fe5d
Merge pull request #441 from B0pol/fix-three-attempts
Support new YouTube JSON scheme
2020-10-28 07:24:31 +01:00
Scratch 947ce3ee10 Fix parsing Soundcloud tracks that contain the term 'sets' 2020-10-28 07:55:02 +11:00
bopol 6dc5ab4015 find playerUrl in another place when assetsPattern regex fails 2020-10-27 13:48:58 +01:00
bopol db0ef83d6b fix youtube decryption and three attemps bug
fixes teamnewpipe/newpipe#4572 fixes #439
2020-10-27 13:40:24 +01:00
Stypox 943334ba49
Merge pull request #440 from B0pol/fix-redirect-channels
Fix redirect channels
2020-10-26 20:00:06 +01:00
bopol c1e9857960 fix subscriber count when subscribe is disabled
fixes #305
2020-10-26 19:06:01 +01:00
bopol a39a2cca82 fix redirect channels 2020-10-26 19:05:19 +01:00
Tobias Groza a8665fd97f
Merge pull request #442 from B0pol/polish
Polish
2020-10-26 18:51:47 +01:00
bopol 0a12300c5e polish tests 2020-10-26 16:57:37 +01:00
bopol 01f49e8f66 polish strings 2020-10-26 16:32:39 +01:00
Tobias Groza 14c0c37d64
Merge pull request #309 from Stypox/improve-stream-tests
Improve stream extractor tests & various fixes
2020-10-25 18:49:53 +01:00
Stypox df26f61cfc
Merge pull request #394 from kirisakow/dev
Fix TeamNewPipe/NewPipe#4213: [YouTube] Ignore leading characters in video id
2020-10-25 14:39:03 +01:00
Kiril Isakov 366ed71523
[YouTube] Ignore leading characters in video id 2020-10-25 14:34:20 +01:00
Stypox 8026304a0a
Merge pull request #409 from Bartoshr/bugfix/short-channel-urls
Support short custom YouTube channel URLs
2020-10-25 11:43:50 +01:00
Stypox 57e7994c9e
Add some missing finals, nullables and comments 2020-10-25 08:12:41 +01:00
Stypox f11fe87688
[YouTube] Replace outdated PewDiePie video test with another one
The old video was made private, and this video (wedding) is probably never going to be removed.
2020-10-24 18:41:41 +02:00
Stypox 3c55ea9321
[PeerTube] Change age restricted video in tests
The old one wasn't available anymore
2020-10-24 18:41:41 +02:00
Stypox 880ff04a5c
[Test] Add stream dash mpd url test 2020-10-24 18:41:41 +02:00
Stypox d0b14644bb
[YouTube/MediaCCC] Consider dates as GMT and not as local 2020-10-24 18:41:41 +02:00
Stypox 8ce711f40f
[Test] Add sub channel name, url and thumbnail tests 2020-10-24 18:41:41 +02:00
Stypox 6127826571
[Test] Add stream metadata tests 2020-10-24 18:41:41 +02:00
Stypox a087b092b4
[Test] Improve code style and add final 2020-10-24 18:39:55 +02:00
Stypox a4097d8d01
[MediaCCC] Return empty list of video-only streams instead of null 2020-10-24 18:39:55 +02:00
Stypox 8dc3f28618
[PeerTube] Test one channel url with api and one without 2020-10-24 18:39:55 +02:00
Stypox 68d23defba
[YouTube] Do not catch every exception on getErrorMessage
@B0pol suggestion
2020-10-24 18:39:55 +02:00
Stypox 3191bd6c70
[YouTube] Use final when possible 2020-10-24 18:39:55 +02:00
Stypox 55bc01d1ce
[SoundCloud] Use final when possible, ide refactorings 2020-10-24 18:39:55 +02:00
Stypox 06430c4749
[PeerTube] Use final when possible, ide refactorings 2020-10-24 18:39:06 +02:00
Stypox fcb9b6f855
[MediaCCC] Use final when possible, ide refactorings
Refactorings on `throws` clause
2020-10-24 18:36:22 +02:00
Stypox af5b8b1915
[MediaCCC] Add tests for stream and conference link handlers 2020-10-24 18:36:21 +02:00
Stypox 07a90d116a
[MediaCCC] Use regex to parse stream and conference urls 2020-10-24 18:36:21 +02:00
Stypox 0c980b2d64
[PeerTube] Improve channel and stream link handler tests 2020-10-24 18:36:21 +02:00
Stypox d130fd79c3
[PeerTube] Prepend "accounts/" to channel id for backward compatibility 2020-10-24 18:36:21 +02:00
Stypox 492db83ccf
[MediaCCC] Return null instead of empty items collector
As per the documentation in the base getRelatedStreams()
2020-10-24 18:34:57 +02:00
Stypox 3b2a1829d4
[MediaCCC] Extract tags 2020-10-24 18:34:57 +02:00
Stypox b461da792f
[MediaCCC] Fix link handler inconsistency providing API links 2020-10-24 18:34:34 +02:00
Stypox aeeae87641
[PeerTube] Parse timestamp from url (previously unimplemented) 2020-10-24 18:33:42 +02:00
Stypox 7ae3cb6d07
[PeerTube] Fix link handler inconsistency providing API links 2020-10-24 18:33:11 +02:00
Stypox 4349be13af
[PeerTube] Return empty audio stream list instead of null 2020-10-24 18:25:56 +02:00
Stypox 3b2cfb4ca2
[SoundCloud] Return empty video stream list instead of null
Also replace every instance of `return new ArrayList<>();` with `return Collections.emptyList();`
2020-10-24 18:23:46 +02:00
Stypox 072bae321f
[YouTube] Fix frame extraction for livestreams
Use saved playerResponse instead of parsing json every time
2020-10-24 18:23:46 +02:00
Stypox 7cd410f3fc
[YouTube] Return 0 when there is no timestamp, not -2, as per javadoc 2020-10-24 18:23:45 +02:00
Stypox 7fb867c166
[YouTube] Fix error message obtaining when there is none 2020-10-24 18:22:44 +02:00
Stypox 1d7a86e664
[Test] Add base classes for stream extractor tests
Refactor all stream extractor tests to use new base class.

Remove check if upload date is in the past: this does not have to hold true: youtube premieres turn up in search results even though they are in the future
2020-10-24 18:22:44 +02:00
Bartosz Rumiński 29695aed0a Small field refactor 2020-10-23 16:42:30 +02:00
Bartosz Rumiński 0e67d820bc Use static regex pattern for excluded path segments 2020-10-22 20:13:29 +02:00
Bartosz Rumiński d3f80d1538 Exlude links which are not channels 2020-10-20 20:06:06 +02:00
Tobias Groza 3058af0e43
Merge pull request #418 from Stypox/yt-shorts
[YouTube] Handle URLs for Shorts
2020-10-18 13:06:10 +02:00
Stypox 6887d59570
[YouTube] Handle urls for Shorts 2020-10-18 12:03:01 +02:00
Scratch e945f711c1
Fix SoundCloud test artist account name (#416) 2020-10-16 23:42:46 +02:00
TobiGr 350eed6214 Release v0.20.1 2020-10-16 22:03:57 +02:00
Tobias Groza f82eda9f7d
Merge pull request #415 from wb9688/fix-ytinitialdata
Fix parsing new ytInitialData
2020-10-16 21:59:39 +02:00
wb9688 be9a6f931c Fix parsing new ytInitialData 2020-10-16 20:27:40 +02:00
Tobias Groza 1a9bc8caf1
Merge pull request #414 from Scrxtchy/soundcloud-slash
Remove tailing slash from SoundCloud URLs
2020-10-16 09:48:44 +02:00
Scratch 6a70cb9d50 Remove tailing slash from SoundCloud URLs
Fixes #412
2020-10-16 17:17:40 +11:00
Tobias Groza 9a785f9a7e
Merge pull request #413 from Scrxtchy/soundcloud-test-update
Fix SoundCloud test artist account name
2020-10-16 08:05:44 +02:00
Scratch 538f5d3973 Fix SoundCloud test artist account name 2020-10-16 13:45:05 +11:00
Tobias Groza 527945eadb
Merge pull request #406 from Stypox/fix-decrypt
[YouTube] Fix some decryption exceptions by retrying
2020-10-15 22:03:25 +02:00
Stypox 19e862657a
[YouTube] Fix some decryption exceptions by retrying 2020-10-15 20:16:44 +02:00
Bartosz Rumiński e3f996e014 Exlude links which are not channels 2020-10-12 20:59:56 +02:00
Bartosz Rumiński 5ab1b053d2
Update youtube channel link handler tests
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-10-12 20:11:28 +02:00
Bartosz Rumiński 7abb4b3713
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/linkHandler/YoutubeChannelLinkHandlerFactory.java
Fix typos

Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-10-12 19:57:45 +02:00
Bartosz Rumiński 9d63211a66
Fix typos
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-10-12 19:56:53 +02:00
Bartosz Rumiński 8c38a5509e Prevent attribution_link urls to be accepted by channel links handler 2020-10-09 19:07:38 +02:00
Bartosz Rumiński 4e04991762 Support short custom youtube channel urls 2020-10-09 17:06:59 +02:00
Fynn Godau 81b5e7cf3d Fix extractor 2020-10-08 17:56:03 +02:00
TobiGr acb04eb351 Update extractor version to 0.20.0 2020-10-04 13:37:19 +02:00
Stypox 2463884aa8
Merge pull request #400 from wb9688/skip-otf
Do not list YouTube's OTF streams
2020-09-29 20:16:34 +02:00
wb9688 ebbfe7f6d4 Skip YouTube's OTF streams 2020-09-29 10:48:02 +02:00
Tobias Groza 6633f26ec5
Merge pull request #386 from mhmdanas/remove-subtitles-stream-get-url
Remove SubtitlesStream#getURL()
2020-08-26 21:32:36 +02:00
Tobias Groza 070cd92857
Merge pull request #387 from wb9688/init-safe-standard-objects
Use initSafeStandardsObjects()
2020-08-26 21:25:46 +02:00
wb9688 7657c2ed1a Use initSafeStandardsObjects() 2020-08-15 17:08:07 +02:00
mhmdanas 1a63dcb355 Remove overriden field `url` 2020-08-12 22:37:36 +03:00
Tobias Groza de8edbe7a5
Merge pull request #382 from TobiGr/fix/empty_comment
[YouTube] Fix crash on empty  comment
2020-08-12 14:18:16 +02:00
mhmdanas f0f1c009b2 Remove SubtitlesStream#getURL() 2020-08-10 19:28:56 +03:00
Fynn Godau cfe88a74c1 Throw ContentNotSupportedException when opening radio uploader channel 2020-08-09 10:07:49 +02:00
TobiGr 0fb73301e3 [YouTube] Fix crash on empty comment
Closes #380
2020-08-05 18:25:40 +02:00
Fynn Godau 932d094d6a Upgrade featured playlist urls to https 2020-08-02 16:55:18 +02:00
Fynn Godau c12ef3a02d Merge TNP/dev into fynngodau/dev 2020-08-02 16:43:04 +02:00
TobiGr 5ac80624a4 Fix detection of YouTube's decryption function
Closes TeamNewPipe/NewPipe#3951
2020-07-28 01:01:10 +02:00
TobiGr b4481dfec2 Update version to 0.19.7 2020-07-26 14:24:07 +02:00
Tobias Groza 32dff1541f
Merge pull request #370 from wb9688/fix-yt-age-restricted
Fix age restricted YouTube videos
2020-07-26 14:17:19 +02:00
TobiGr ec778200d9 Add comments to the code 2020-07-26 14:15:13 +02:00
Tobias Groza 8627d01006
Merge pull request #373 from wb9688/fix-yt-continuations
Support YouTube's new continuations
2020-07-26 13:56:33 +02:00
wb9688 667dce034c Only use fallback methods for clientVersion when clientVersion hasn't been successfully extracted yet 2020-07-26 13:14:25 +02:00
wb9688 f345f667e2 Extract YouTube's key 2020-07-26 12:01:59 +02:00
wb9688 4c987a5302 Support YouTube's new continuations for search 2020-07-26 10:01:03 +02:00
wb9688 81459e289f Fix age restricted YouTube videos 2020-07-18 19:52:54 +02:00
Tobias Groza df28a087de
Merge pull request #337 from wb9688/next-stream
Remove getNextStream() from StreamExtractor
2020-07-18 11:55:58 +02:00
wb9688 fc3a63fec5 Update Javadoc for getRelatedStreams() 2020-07-15 18:48:37 +02:00
wb9688 ff560e907a Remove getNextStream() from StreamExtractor 2020-07-15 18:48:37 +02:00
TheAssassin 186193d366 Update version to 0.19.6 2020-07-14 17:04:03 +02:00
Tobias Groza a70cb0283f
Merge pull request #314 from wb9688/remove-getnextpageurl
Next page stuff
2020-07-07 20:49:38 +02:00
wb9688 0a5a905bc7 Add final at more places 2020-07-07 20:45:47 +02:00
Tobias Groza e5d23a89de
Merge pull request #366 from wb9688/yt-timestamp-start
Support start= timestamps for YouTube
2020-07-06 21:28:19 +02:00
wb9688 17ba8a57fa Clean up the code 2020-07-06 20:19:31 +02:00
wb9688 9b6fe1dea6 Throw IllegalArgumentException when Page is invalid 2020-07-06 20:19:31 +02:00
wb9688 4cc312086a Introduce Page class 2020-07-06 20:19:31 +02:00
wb9688 e3bfdba135 Remove getNextPageUrl() function from ListExtractor 2020-07-06 20:11:40 +02:00
wb9688 156a26b64b Support start= timestamps for YouTube 2020-07-06 14:08:40 +02:00
Tobias Groza 54d9e5a2f8
Merge pull request #365 from B0pol/update_inv_instances
Update inv instances
2020-07-02 23:13:07 +02:00
bopol 261471e7f4 remove invidiou.sh from tests
dead instance
2020-07-02 21:33:45 +02:00
bopol eb3901acee update invidious instances 2020-07-02 21:31:05 +02:00
Tobias Groza c0ceb5cb27
Merge pull request #340 from wb9688/peertube-deleted-comments
Handle isDeleted for PeerTube comments
2020-06-28 10:21:37 +02:00
Tobias Groza f4ed7ce922
Merge pull request #361 from yausername/gradleScope
expose extractor via api scope
2020-06-27 21:45:04 +02:00
Tobias Groza 339f1d9c43
Merge pull request #349 from wb9688/okhttp
Use OkHttp for tests like in NewPipe
2020-06-27 20:08:02 +02:00
Tobias Groza 970bc7f69d
Merge pull request #359 from mauriciocolli/fix-client-id-bug
[SoundCloud] Fix concurrency issue when getting the client id
2020-06-27 17:36:24 +02:00
Tobias Groza dbf6ab9e91
Merge pull request #355 from Redirion/patch-1
Added a comment
2020-06-27 11:47:50 +02:00
Robin 576754982e
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-06-27 11:35:01 +02:00
yausername 89e0e3ee2d
expose extractor via api scope 2020-06-27 03:02:43 +05:30
Mauricio Colli 7505e828cc
[SoundCloud] Update client id 2020-06-24 01:50:09 -03:00
Mauricio Colli d2f1c0f40d
[SoundCloud] Fix concurrency issue when getting the client id 2020-06-24 01:50:08 -03:00
Robin b63ae93495
Added a comment
and fixed a "typo" in the method parseDurationString replacing Long with Integer.

These are not the droids you are looking for wb9688 😁
2020-06-15 11:27:44 +02:00
Tobias Groza 92f6754f0f
Merge pull request #338 from wb9688/parse-duration-string-test
Add tests for parseDurationString()
2020-06-15 11:01:21 +02:00
wb9688 e02d3c37b2
Merge pull request #354 from Stypox/peertube-age-restricted
[PeerTube] Change age restricted video in tests
2020-06-14 20:09:36 +02:00
Stypox 5808439c3f
[PeerTube] Change age restricted video in tests
The old one wasn't available anymore
2020-06-14 19:57:39 +02:00
wb9688 3239aa84f2 Handle isDeleted for PeerTube comments 2020-06-14 19:27:20 +02:00
wb9688 a65e46e4b1 Use OkHttp for tests like in NewPipe 2020-06-11 15:06:08 +02:00
Tobias Groza 5f523254cb
Merge pull request #344 from B0pol/peertube_embeds
For #342: support PeerTube embeds
2020-06-06 18:53:07 +02:00
Fynn Godau 96de834b67 Bandcamp: parse date from radio info item 2020-06-04 19:36:58 +02:00
Fynn Godau 39b55b5b42 Bandcamp radio stream extractor: test uploader url 2020-06-04 19:32:19 +02:00
Fynn Godau 9a555d97e5 Remove useless bandcamp tests 2020-06-04 19:31:01 +02:00
Fynn Godau e13f341a45 Bandcamp stream extractor: test service id 2020-06-04 19:29:37 +02:00
Fynn Godau 4dd9540782 Bandcamp license switch order 2020-06-04 19:27:39 +02:00
Fynn Godau 52103ac61f Remove "useless" comments 2020-06-04 19:27:13 +02:00
Fynn Godau 8fa81537c0 Better bandcamp stream extractor description test 2020-06-04 19:17:51 +02:00
Fynn Godau 0950a95577 Fix for testing wrong object for null or empty 2020-06-04 19:16:08 +02:00
Fynn Godau e6ecd91929 Bandcamp: Test full date 2020-06-04 19:12:10 +02:00
Fynn Godau 27e7aad159 Bandcamp: Improve code style 2020-06-04 19:04:25 +02:00
fynngodau ea49202f64
Don't display internal license id
Co-authored-by: bopol <bopol@e.email>
2020-06-04 18:26:58 +02:00
bopol 063f4be766 For #342: support PeerTube embeds 2020-06-04 14:26:42 +02:00
Fynn Godau 34b6928124 Don't print stacktrace before throwing new exception 2020-06-04 14:25:38 +02:00
Fynn Godau 1be20ceeec Bandcamp: parse date 2020-06-04 14:09:21 +02:00
Fynn Godau 6822fe3dc8 Bandcamp: Get Stream upload date 2020-06-04 13:03:16 +02:00
Fynn Godau e98bff8ae9 Improved Bandcamp radio stream extractor tests 2020-06-04 12:30:53 +02:00
Fynn Godau 8f6c00f8d8 Use default test in BandcampSearchExtractorTest 2020-06-04 12:12:20 +02:00
Fynn Godau de776561b0 Bandcamp: fix loading uploader from streams in search 2020-06-04 12:02:18 +02:00
Fynn Godau 3940138fc5 Fix Bandcamp capitalization in one more spot 2020-06-03 22:41:27 +02:00
Fynn Godau 9fa9d920a9 Bandcamp: Implement new methods required due to interface changes 2020-06-03 21:35:11 +02:00
Fynn Godau dd955c7f0c Merge branch 'origin/dev' into dev 2020-06-03 21:13:40 +02:00
wb9688 33b1121fc7 Add tests for parseDurationString() 2020-05-30 17:20:54 +02:00
TobiGr 3aa6b93e91 Update version to 0.19.5 2020-05-30 10:45:39 +02:00
Tobias Groza bda83fe6a5
Merge pull request #336 from wb9688/fix-time-parsing
Fix YouTube video duration parsing
2020-05-30 10:36:46 +02:00
wb9688 e39147202c Fix YouTube video duration parsing 2020-05-30 09:31:08 +02:00
wb9688 98055a3c3c Revert "Add more decryption function regexes"
This reverts commit 384d6acd5d.
2020-05-29 18:41:40 +02:00
Tobias Groza bdb0f2ae6b
Merge pull request #332 from wb9688/learning-playlist
Support YouTube's learning playlists
2020-05-29 10:11:08 +02:00
Tobias Groza cf18cdb2f5
Merge pull request #329 from wb9688/decryption-regexes
Add more decryption function regexes
2020-05-29 10:10:50 +02:00
wb9688 ab77961b0a Extract trailer thumbnail in YT learning playlists 2020-05-29 09:55:24 +02:00
wb9688 4d683e7655 Support YouTube's learning playlists 2020-05-29 09:17:03 +02:00
Tobias Groza 0b4977bb0c
Merge pull request #333 from wb9688/playerresponse-dashmanifesturl
Extract dashManifestUrl in playerResponse
2020-05-28 21:17:07 +02:00
Tobias Groza 0f8057ae56
Merge pull request #322 from wb9688/upgrade-jsoup
Upgrade dependencies
2020-05-28 21:16:33 +02:00
wb9688 384d6acd5d Add more decryption function regexes 2020-05-28 11:40:38 +02:00
wb9688 8f71bde776 Fix bug for YT videos longer than 1000h 2020-05-28 11:34:42 +02:00
wb9688 aa277e1b6a Upgrade other dependencies 2020-05-28 11:34:42 +02:00
wb9688 036cc7e26e Upgrade jsoup 2020-05-28 11:34:41 +02:00
Fynn Godau 89b0639faa New Bandcamp tests for channel extractor 2020-05-25 20:22:23 +02:00
Fynn Godau e08256ebc6 Remove "url cleanup" in Bandcamp link handlers 2020-05-25 19:17:27 +02:00
Fynn Godau 692b2d06f4 Bandcamp: read info item data in place and not in advance 2020-05-25 18:53:15 +02:00
Fynn Godau fff21caa87 Fix Bandcamp no avatar test
I own the bandcamp artist account called "NewPipeExtractorTest", so it
shouldn't break again.
2020-05-25 18:53:10 +02:00
wb9688 7ef3e80493 Extract dashManifestUrl in playerResponse 2020-05-15 12:07:26 +02:00
Stypox 60e6df7e6b
Merge pull request #330 from B0pol/gitignore
add vscode / eclipse files to .gitignore
2020-05-13 10:39:33 +02:00
wb9688 efafbbb88a
Merge pull request #331 from wb9688/playlist-subchannel
Add sub-channel stuff to PlaylistInfo
2020-05-12 10:49:13 +02:00
wb9688 baf5dd3e8f
Merge pull request #311 from B0pol/fixSearchSuggestion
Search: add isCorrectedSearch() and fix YoutubeSearchExtractor#getSea…
2020-05-12 09:29:15 +02:00
wb9688 102f04e297 Add sub-channel stuff to PlaylistInfo 2020-05-12 09:14:29 +02:00
bopol cbf1163b80 ignore workspace and folder configs 2020-05-11 21:23:50 +02:00
Tobias Groza b40ccb5075
Merge pull request #307 from B0pol/playlists_peertube
[PeerTube] playlist support
2020-05-11 20:53:03 +02:00
bopol aeb2a6ff6f add vscode / eclipse files to .gitignore 2020-05-11 13:06:02 +02:00
B0pol 3a858e6640
Merge pull request #315 from B0pol/isnullorempty
refactor: add Utils.isNullOrEmpty()
2020-05-11 11:04:56 +00:00
bopol adaf196c99 remove unused import 2020-05-11 12:54:33 +02:00
bopol de26e00079 changed all == null || isEmpty() to isNullOrEmpty() 2020-05-11 11:40:24 +02:00
bopol 202a73516c refactor: add Utils.isNullOrEmpty() 2020-05-11 11:26:18 +02:00
Tobias Groza 3cae32b6db
Merge pull request #328 from wb9688/fix-327
Fix #327
2020-05-09 20:29:11 +02:00
wb9688 1de0876fb4 Fix #327 2020-05-09 20:06:50 +02:00
bopol 7cae95bf09 add tests for subchannel stuff in playlists 2020-05-09 10:15:35 +02:00
bopol 1a6e92ebf6 add getSubChannel Name, Url and AvatarUrl for playlists 2020-05-09 09:52:24 +02:00
bopol cf99107745 address yausername change 2020-05-09 00:52:04 +02:00
bopol 5bab9d9fc0 [PeerTube] playlist support & refactoring 2020-05-09 00:52:04 +02:00
Tobias Groza f3913e241e
Merge pull request #324 from Stypox/music-detector-url
[YouTube] Support stream urls in `vnd.youtube://ID` form
2020-05-08 17:43:04 +02:00
Tobias Groza 3ddd6e4d7b
Merge pull request #326 from wb9688/subchannel-parentchannel
Rename SubChannel in channels back to ParentChannel
2020-05-08 17:40:49 +02:00
wb9688 7320108c66 Rename SubChannel in channels back to ParentChannel 2020-05-08 16:02:59 +02:00
wb9688 44d382b4bd
Merge pull request #313 from Royosef/DisplayParentChannelDetails
Display parent channel details
2020-05-08 15:52:32 +02:00
Roy Yosef a408661771 Fix PR review
Rename "ParentChannel" to "SubChannel"
Add Tests for sub-channel info fetching
2020-05-07 20:20:12 +02:00
Roy Yosef 1de1f97cf1 Add parent channel info to StreamExtractor: name, url & avatar url 2020-05-07 20:20:12 +02:00
Roy Yosef 4234740baa Add parent channel info to ChannelExtractor: name, url & avatar url 2020-05-07 20:20:12 +02:00
bopol 6cff5de400 renaming methods for consistency 2020-05-07 15:40:41 +02:00
bopol 5760366d70 use JsonObject#isEmpty instead of JsonObject#equals(JsonUtils.EMPTY_OBJECT) 2020-05-07 15:40:41 +02:00
bopol 93e868634e clarified javadoc for isCorrectedSearch and getSearchSuggestions in SearchExtractor 2020-05-07 15:40:41 +02:00
bopol 3757541a9c adapt new nanojson version 2020-05-07 15:40:41 +02:00
bopol 79483a6dc0 «Use this in SearchInfo, for consistency's sake» @Stypox order xD 2020-05-07 15:40:41 +02:00
bopol e8d58e09c9 getSearchSuggestion now returns the corrected query if isCorrectedSearch() is true 2020-05-07 15:40:41 +02:00
bopol 5d1c3b3fa2 Use setters in SearchInfo 2020-05-07 15:39:57 +02:00
bopol 29b639b454 Search: add isCorrectedSearch() and fix YoutubeSearchExtractor#getSearchSuggestion()
isCorrectedSearch: see the javadoc
getSearchSuggestion: in YoutubeSearchExtractor, it was giving the corrected search query. It now gives the suggested query, as it should
2020-05-07 15:39:57 +02:00
wb9688 665c69b530
Merge pull request #310 from B0pol/soundcloudComments
Add support for soundcloud comments
2020-05-07 15:35:57 +02:00
Stypox a02bfbeae5
[YouTube] Support stream urls in `vnd.youtube://ID` form 2020-05-07 15:31:31 +02:00
bopol bc13e0c616 Merge remote-tracking branch 'upstream/dev' into soundcloudComments 2020-05-05 14:49:21 +02:00
TobiGr 4086715a68 Merge branch 'dev' 2020-05-03 21:21:30 +02:00
TobiGr 8a79b14077 Update extractor version to 0.19.4 2020-05-03 15:11:20 +02:00
Tobias Groza fc3a69ed54
Merge pull request #323 from wb9688/fix-yt-music-albums
Fix NPE in defaultAlertsCheck()
2020-05-03 14:08:47 +02:00
wb9688 6aae99f832 Fix NPE in defaultAlertsCheck() 2020-05-03 10:28:45 +02:00
Tobias Groza 2780e716b3
Merge pull request #321 from wb9688/fix-nanojson
Return null instead of "" in getTextFromObject()
2020-05-02 15:56:28 +02:00
wb9688 a1eabc7589 Return null instead of "" in getTextFromObject() 2020-05-02 08:02:25 +02:00
Fynn Godau e7b046db9d Merge remote-tracking branch 'origin/dev' into dev 2020-04-26 23:12:43 +02:00
Fynn Godau a3e8e1c70e Bandcamp: move own methods to helper class 2020-04-26 23:05:56 +02:00
Fynn Godau 9bc7a470ab Replace fori with foreach loop 2020-04-26 23:02:14 +02:00
Tobias Groza 49157fcf0d
Merge pull request #317 from wb9688/tnp-nanojson
Use our fork of nanojson
2020-04-24 21:54:35 +02:00
bopol 2564bcf399 refactor comments
rename methods in CommentsInfoItemExtractor interface to match the other method names across NewPipeExtractor
remove getName in (Youtube|SoundCloud|Peertube)CommentsExtractor and move it up in CommentsExtractor, return "Comments" instead
2020-04-21 20:05:54 +02:00
Fynn Godau f71bd29d58 Bandcamp playlist extractor: add more tests 2020-04-20 23:52:12 +02:00
Fynn Godau 5009d9f53b Bandcamp tests: remove certain tests 2020-04-20 23:25:00 +02:00
Fynn Godau b5e251c82f Bandcamp channel extractor: package-private access for getImageUrl(…) 2020-04-20 23:12:43 +02:00
Fynn Godau 8c70dab8c5 Bandcamp search query handler factory: inline variable 2020-04-20 23:11:16 +02:00
Fynn Godau 10ae3db118 Bandcamp javadoc: Replace br tags with p tags 2020-04-20 23:03:12 +02:00
Fynn Godau 9201e0d3ea Don't declare exceptions that can't actually be thrown 2020-04-20 22:46:54 +02:00
Fynn Godau 965bce00cf Bandcamp stream extractor: return NO_AGE_LIMIT 2020-04-20 22:44:45 +02:00
Fynn Godau 82099592c7 Bandcamp: Move code from SearchExtractor to InfoItemExtractors 2020-04-20 22:44:35 +02:00
Fynn Godau 00c0333059 Bandcamp: Fetch channelInfo in onFetchPage 2020-04-20 22:41:44 +02:00
Fynn Godau 433d72c26a Depend on nanojson fork in timeago-parser 2020-04-20 22:04:10 +02:00
Fynn Godau 67de0285e1 Bandcamp: Generate query JSON in a proper way 2020-04-20 22:03:29 +02:00
Fynn Godau c133190c53 Bandcamp capitalization 2020-04-20 21:55:35 +02:00
Fynn Godau b100b9873f Bandcmap: Use nanojson fork instead of org.json 2020-04-20 21:53:04 +02:00
wb9688 b51699a20e Implement @TobiGr's suggestions 2020-04-20 14:27:33 +02:00
wb9688 74f0ab1792
Merge pull request #301 from wb9688/media-ccc-non-api-urls
Use media.ccc.de instead of api.media.ccc.de
2020-04-19 17:06:57 +02:00
Fynn Godau 9c239371f4 Change nanojson dependency to @wb9688's fork 2020-04-19 14:29:39 +02:00
Fynn Godau 5c0a03328a Merge 'origin/dev' into dev 2020-04-19 14:26:51 +02:00
bopol 0cbbc2a1f9 remove unused import 2020-04-19 14:16:38 +02:00
wb9688 979c5a7502 Use our fork of nanojson 2020-04-17 19:36:42 +02:00
bopol d4352f9b84 support comments for SoundCloud 2020-04-10 20:41:41 +02:00
wb9688 1b522304c1
Merge pull request #308 from Royosef/PeerTubeChannelExtractor
[PeerTube] channels support
2020-04-10 19:07:51 +02:00
Roy Yosef 4afe657f6f Add tests for getUrlFromId and test for video-channels in acceptUrlTest 2020-04-10 19:05:38 +03:00
Roy Yosef 2c9f1260eb Fix wb9688 review comments
* Rename PeertubeUserExtractor to PeertubeAccountExtractor
* Add test for video-channels in PeertubeChannelLinkHandlerFactoryTest
* Compatibility support for older versions (use "accounts/" as default)
2020-04-10 16:40:53 +03:00
bopol 636c430743 refactor: create extractors and linkHandler packages for SoundCloud, move YoutubeParsingHelper to youtube package (not linkhandler anymore) 2020-04-10 10:51:05 +02:00
Roy Yosef b6e6f403a8 add support for PeerTube channels extraction 2020-04-09 20:37:49 +03:00
wb9688 a5155fb562
Merge pull request #300 from B0pol/hls_opus_not_supported
throw ContentNotSupportedException when content is known to be unsuppo…
2020-04-08 19:54:16 +02:00
bopol a20d53dd98 remove the mention of Opus on wb9688 advise when
ContentNotSupportedException is thrown
2020-04-08 15:31:11 +02:00
bopol 9a7c6b7ab0 use ContentNotSupportedException for channels without tabs 2020-04-07 14:33:43 +02:00
bopol a22104cbda add tests for ContentNotSupportedException 2020-04-07 14:33:43 +02:00
bopol 093762e793 throw ContentNotSupportedException when content is know to be unsupported 2020-04-07 14:33:43 +02:00
Tobias Groza 094b87c537
Merge pull request #299 from B0pol/sc_countries
add supported countries for soundcloud
2020-04-03 19:12:33 +02:00
Tobias Groza a7d3d14910
Add link to JDoc to readme 2020-04-03 18:14:14 +02:00
Tobias Groza e3263666bd
Merge pull request #303 from opusforlife2/correct-template
Corrected the pull request template
2020-04-03 17:56:02 +02:00
Tobias Groza b81e22ddc9
Merge pull request #291 from wb9688/yt-music-search
Add support for YouTube Music search
2020-04-03 17:55:34 +02:00
wb9688 bce27a0e22 Rename getValidResponseBody() to getValidJsonResponseBody() 2020-04-03 17:23:18 +02:00
opusforlife2 4a1d8c804c
Corrected the pull request template
It was a bit weird before.
2020-04-02 10:42:54 +00:00
bopol 1eb3deb7fd add supported countries for soundcloud 2020-04-02 11:26:19 +02:00
Tobias Groza cef7c96d7d
Merge pull request #302 from wb9688/readme-local-build
Add another way to test changes locally to README.md
2020-04-01 19:23:04 +02:00
wb9688 8a9e137385 Extract some code to getValidResponseBody() 2020-04-01 16:01:21 +02:00
wb9688 2d0c301440 Add another way to test changes locally to README.md 2020-04-01 15:43:11 +02:00
wb9688 bf24caa5f2 Use media.ccc.de instead of api.media.ccc.de 2020-04-01 13:25:10 +02:00
wb9688 2af610e3e7 Add tests for other YT Music search types 2020-04-01 10:30:51 +02:00
wb9688 c7f7bd2442 Fix error when YT Music videos has no uploader URL 2020-04-01 10:30:51 +02:00
wb9688 ac15df4548 Move YouTube Music search to its own class 2020-04-01 10:30:51 +02:00
wb9688 cf0f2aff3e Extract uploader url from certain YouTube Music videos 2020-04-01 10:30:51 +02:00
wb9688 aa8cea47f3 Refactor YouTube Music search tests 2020-04-01 10:30:51 +02:00
wb9688 dd434cca01 Fix issue when there is no didYouMeanRenderer in itemSectionRenderer 2020-04-01 10:30:51 +02:00
wb9688 5a775a4bbe Use new way of specifying stream count 2020-04-01 10:30:51 +02:00
wb9688 d58c0f230d Improve code for YouTube Music search 2020-04-01 10:30:51 +02:00
wb9688 dc29d87962 Extract YouTube search suggestions 2020-04-01 10:30:51 +02:00
wb9688 c852b13d5a Add Referer header so that it also works with HttpsUrlConnection 2020-04-01 10:30:51 +02:00
wb9688 eb48524411 Add tests for YouTube Music search 2020-04-01 10:30:51 +02:00
wb9688 2b9b2a78e8 Handle 100+ items in playlist 2020-04-01 10:30:51 +02:00
wb9688 fb9b9691b7 Improve getYoutubeMusicKeys() 2020-04-01 10:30:51 +02:00
wb9688 4ddbdf0aee Disable artist search for now 2020-04-01 10:30:51 +02:00
wb9688 1762a527c9 Add support for YouTube Music search 2020-04-01 10:30:51 +02:00
Tobias Groza 04af575764
Hey Travis, deploy the docs 2020-03-30 00:10:54 +02:00
Fynn Godau a1523eb293 Update Bandcamp service to latest interface changes 2020-03-17 20:40:25 +01:00
fynngodau b78f788017
Merge branch 'dev' into dev 2020-03-17 18:03:14 +01:00
Fynn Godau 36a316eea5
Merge branch 'dev' into dev 2020-02-08 09:43:13 +01:00
Fynn Godau 623e6a8c63
Merge branch 'dev' into dev 2020-01-17 22:37:42 +01:00
Fynn Godau 7b5702bdc2 Add bandcamp to list of supported services in README 2020-01-11 16:32:45 +01:00
Fynn Godau ba967f1a15 Workaround enourmous load times for long bandcamp playlists
Additionally, get cover art from json instead of html
2020-01-03 13:17:04 +01:00
Fynn Godau 46e1f3922c Refuse to load bandcamp playlists without content 2020-01-03 12:32:22 +01:00
Fynn Godau f0d36dfa7e Bandcamp radio: proper name, date shortened 2020-01-03 12:28:49 +01:00
Fynn Godau 808c8aa087 Add bandcamp weekly kiosk
Unfortunately a little messy
2020-01-03 12:28:49 +01:00
Fynn Godau 13ef11e9c8 Bandcamp featured kiosk 2020-01-03 12:28:49 +01:00
Fynn Godau db02850d4f Bandcamp playlist extractor: support playlists with just one stream 2020-01-03 12:28:49 +01:00
Fynn Godau 13e4908b83 Bandcamp search extractor: read track count from result page 2020-01-03 12:28:49 +01:00
Fynn Godau 9b16baffb7 Bandcamp search: multiple pages 2020-01-03 12:28:49 +01:00
Fynn Godau ce2a88e56f Add bandcamp search suggestion extractor 2020-01-03 12:28:49 +01:00
Fynn Godau e12ddaef7f Bandcamp playlist and search extractors: better and more tests 2020-01-03 12:28:49 +01:00
Fynn Godau 655df356a2 Various improvements concerning bandcamp playlists 2020-01-03 12:28:49 +01:00
Fynn Godau ba700bfb3e Add bandcamp playlists (albums) 2020-01-03 12:28:49 +01:00
Fynn Godau d05b14ae48 Add channels (artists) to search results 2020-01-03 12:28:49 +01:00
Fynn Godau 794ca5eeae Bandcamp channel extractor: handel nonexistent images better 2020-01-03 12:28:49 +01:00
Fynn Godau 91c0ec7cea Bandcamp stream extractor: catch NullPointerException 2020-01-03 12:28:49 +01:00
Fynn Godau d5cdc20be1 Bandcmap channel extractor: fix getting banner url 2020-01-03 12:28:49 +01:00
Fynn Godau 7730eb2ea1 Bandcamp channel extractor: enforce https when getting banner url 2020-01-03 12:28:49 +01:00
Fynn Godau a42c77425d Bandcamp channel extractor (ignoring everything but tracks) 2020-01-03 12:28:49 +01:00
Fynn Godau 5281456899 Bandcamp channel link handler factory 2020-01-03 12:28:49 +01:00
Fynn Godau c3d127ccd9 Bandcamp audio stream test and fix 2020-01-03 12:28:49 +01:00
Fynn Godau 80d67e22b7 Return mp3-128 audio stream [experimental] 2020-01-03 12:28:49 +01:00
Fynn Godau 43dc3c3d4c Return empty kiosk list instead of null 2020-01-03 12:28:49 +01:00
Fynn Godau a579337c9a Bandcamp service with support for streams and searches 2020-01-03 12:28:49 +01:00
788 changed files with 157212 additions and 10613 deletions

View File

@ -1,3 +1,3 @@
- [ ] I carefully read the [contribution guidelines](https://github.com/TeamNewPipe/NewPipe/blob/HEAD/.github/CONTRIBUTING.md) and agree to them.
- [ ] I did test the API against [NewPipe](https://github.com/TeamNewPipe/NewPipe).
- [ ] I agree to ASAP create a PULL request for [NewPipe](https://github.com/TeamNewPipe/NewPipe) for making in compatible when I changed the api.
- [ ] I have tested the API against [NewPipe](https://github.com/TeamNewPipe/NewPipe).
- [ ] I agree to create a pull request for [NewPipe](https://github.com/TeamNewPipe/NewPipe) as soon as possible to make it compatible with the changed API.

12
.github/dependabot.yml vendored Normal file
View File

@ -0,0 +1,12 @@
version: 2
updates:
# Maintain dependencies for Gradle
- package-ecosystem: "gradle"
directory: "/"
schedule:
interval: "daily"
# Maintain dependencies for GitHub Actions
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "daily"

51
.github/workflows/ci.yml vendored Normal file
View File

@ -0,0 +1,51 @@
name: CI
on:
schedule:
# once per day
- cron: 0 0 * * *
push:
branches:
- dev
- master
pull_request:
permissions:
contents: read
jobs:
build-and-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: set up JDK 11
uses: actions/setup-java@v4
with:
java-version: '11'
distribution: 'temurin'
- name: Cache Gradle dependencies
uses: actions/cache@v4
with:
path: ~/.gradle/caches
key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle') }}
restore-keys: ${{ runner.os }}-gradle
# See gradle file for difference between downloaders
- name: Build and run Tests
run: |
if [[ "$GITHUB_EVENT_NAME" == 'schedule' ]]; then
echo running with real downloader
./gradlew check --stacktrace -Ddownloader=REAL
else
echo running with mock downloader
./gradlew check --stacktrace -Ddownloader=MOCK
fi
- name: Upload test reports when failure occurs
uses: actions/upload-artifact@v4
if: failure()
with:
name: NewPipeExtractor-test-reports
path: extractor/build/reports/tests/test/**

38
.github/workflows/docs.yml vendored Normal file
View File

@ -0,0 +1,38 @@
name: Build and deploy JavaDocs
on:
push:
branches:
- master
permissions:
# The generated docs are written to the `gh-pages` branch.
contents: write
jobs:
build-and-deploy-docs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: set up JDK 11
uses: actions/setup-java@v4
with:
java-version: '11'
distribution: 'temurin'
- name: Cache Gradle dependencies
uses: actions/cache@v4
with:
path: ~/.gradle/caches
key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle') }}
restore-keys: ${{ runner.os }}-gradle
- name: Build JavaDocs
run: ./gradlew aggregatedJavadocs
- name: Deploy JavaDocs
uses: peaceiris/actions-gh-pages@v4
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: ./build/docs

8
.gitignore vendored
View File

@ -16,3 +16,11 @@ gradle-app.setting
# # Work around https://youtrack.jetbrains.com/issue/IDEA-116898
# gradle/wrapper/gradle-wrapper.properties
# vscode / eclipse files
*.classpath
*.project
*.settings
**/bin
**.vscode
*.code-workspace

View File

@ -1,15 +0,0 @@
language: java
jdk:
- openjdk8
script:
- ./gradlew check
- ./gradlew aggregatedJavadocs
deploy:
provider: pages
skip_cleanup: true
github_token: $GITHUB_TOKEN # Set in travis-ci.org dashboard
local_dir: build/docs
on:
branch: master

View File

@ -1,7 +1,7 @@
GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
@ -645,7 +645,7 @@ the "copyright" line and a pointer to where the full notice is found.
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
@ -664,11 +664,11 @@ might be different; for a GUI interface, you would use an "about box".
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU GPL, see
<http://www.gnu.org/licenses/>.
<https://www.gnu.org/licenses/>.
The GNU General Public License does not permit incorporating your program
into proprietary programs. If your program is a subroutine library, you
may consider it more useful to permit linking proprietary applications with
the library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License. But first, please read
<http://www.gnu.org/philosophy/why-not-lgpl.html>.
<https://www.gnu.org/licenses/why-not-lgpl.html>.

View File

@ -1,6 +1,6 @@
# NewPipe Extractor
[![Build Status](https://travis-ci.org/TeamNewPipe/NewPipeExtractor.svg?branch=master)](https://travis-ci.org/TeamNewPipe/NewPipeExtractor) [![JIT Pack Badge](https://jitpack.io/v/TeamNewPipe/NewPipeExtractor.svg)](https://jitpack.io/#TeamNewPipe/NewPipeExtractor) [Documentation](https://teamnewpipe.github.io/documentation/)
[![CI](https://github.com/TeamNewPipe/NewPipeExtractor/actions/workflows/ci.yml/badge.svg?branch=dev&event=schedule)](https://github.com/TeamNewPipe/NewPipeExtractor/actions/workflows/ci.yml) [![JIT Pack Badge](https://jitpack.io/v/TeamNewPipe/NewPipeExtractor.svg)](https://jitpack.io/#TeamNewPipe/NewPipeExtractor) [JDoc](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/) • [Documentation](https://teamnewpipe.github.io/documentation/)
NewPipe Extractor is a library for extracting things from streaming sites. It is a core component of [NewPipe](https://github.com/TeamNewPipe/NewPipe), but could be used independently.
@ -11,11 +11,31 @@ NewPipe Extractor is available at JitPack's Maven repo.
If you're using Gradle, you could add NewPipe Extractor as a dependency with the following steps:
1. Add `maven { url 'https://jitpack.io' }` to the `repositories` in your `build.gradle`.
2. Add `compile 'com.github.TeamNewPipe:NewPipeExtractor:v0.11.0'`the `dependencies` in your `build.gradle`. Replace `v0.11.0` with the latest release.
2. Add `implementation 'com.github.TeamNewPipe:NewPipeExtractor:INSERT_VERSION_HERE'` to the `dependencies` in your `build.gradle`. Replace `INSERT_VERSION_HERE` with the [latest release](https://github.com/TeamNewPipe/NewPipeExtractor/releases/latest).
3. If you are using tools to minimize your project, make sure to keep the files below, by e.g. adding the following lines to your proguard file:
```
## Rules for NewPipeExtractor
-keep class org.schabi.newpipe.extractor.timeago.patterns.** { *; }
-keep class org.mozilla.javascript.** { *; }
-keep class org.mozilla.classfile.ClassFileWriter
-dontwarn org.mozilla.javascript.tools.**
```
**Note:** To use NewPipe Extractor in Android projects with a `minSdk` below 26, [API desugaring](https://developer.android.com/studio/write/java8-support#library-desugaring) is required. If the `minSdk` is below 19, the `desugar_jdk_libs_nio` artifact is required, which requires Android Gradle Plugin (AGP) version 7.4.0.
### Testing changes
To test changes quickly you can build the library locally. Using the local Maven repository is a good approach, here's a gist of how to use it:
To test changes quickly you can build the library locally. A good approach would be to add something like the following to your `settings.gradle`:
```groovy
includeBuild('../NewPipeExtractor') {
dependencySubstitution {
substitute module('com.github.TeamNewPipe:NewPipeExtractor') with project(':extractor')
}
}
```
Another approach would be to use the local Maven repository, here's a gist of how to use it:
1. Add `mavenLocal()` in your project `repositories` list (usually as the first entry to give priority above the others).
2. It's _recommended_ that you change the `version` of this library (e.g. `LOCAL_SNAPSHOT`).
@ -30,14 +50,15 @@ The following sites are currently supported:
- YouTube
- SoundCloud
- MediaCCC
- media.ccc.de
- PeerTube (no P2P)
- Bandcamp
## License
[![GNU GPLv3 Image](https://www.gnu.org/graphics/gplv3-127x51.png)](http://www.gnu.org/licenses/gpl-3.0.en.html)
[![GNU GPLv3 Image](https://www.gnu.org/graphics/gplv3-127x51.png)](https://www.gnu.org/licenses/gpl-3.0.en.html)
NewPipe is Free Software: You can use, study share and improve it at your
NewPipe Extractor is Free Software: You can use, study share and improve it at your
will. Specifically you can redistribute and/or modify it under the terms of the
[GNU General Public License](https://www.gnu.org/licenses/gpl.html) as
published by the Free Software Foundation, either version 3 of the License, or

View File

@ -1,26 +1,47 @@
allprojects {
apply plugin: 'java-library'
apply plugin: 'maven'
apply plugin: 'maven-publish'
sourceCompatibility = 1.7
targetCompatibility = 1.7
compileJava.options.encoding = 'UTF-8'
compileTestJava.options.encoding = 'UTF-8'
version 'v0.19.0'
sourceCompatibility = JavaVersion.VERSION_11
targetCompatibility = JavaVersion.VERSION_11
version 'v0.24.0'
group 'com.github.TeamNewPipe'
repositories {
jcenter()
mavenCentral()
maven { url "https://jitpack.io" }
}
afterEvaluate {
publishing {
publications {
mavenJava(MavenPublication) {
from components.java
}
}
}
}
ext {
nanojsonVersion = "1d9e1aea9049fc9f85e68b43ba39fe7be1c1f751"
spotbugsVersion = "4.8.5"
junitVersion = "5.10.2"
checkstyleVersion = "10.4"
}
}
dependencies {
implementation project(':extractor')
api project(':extractor')
implementation project(':timeago-parser')
}
subprojects {
task sourcesJar(type: Jar, dependsOn: classes) {
classifier = 'sources'
archiveClassifier.set('sources')
from sourceSets.main.allSource
}
@ -42,8 +63,14 @@ task aggregatedJavadocs(type: Javadoc, group: 'Documentation') {
destinationDir = file("$buildDir/docs/javadoc")
title = "$project.name $version"
// options.memberLevel = JavadocMemberLevel.PRIVATE
options.links 'https://docs.oracle.com/javase/7/docs/api/'
options.links 'https://docs.oracle.com/javase/8/docs/api/'
options.encoding 'UTF-8'
// Fixes unknown tag @implNote; the other two were added precautionary
options.tags = [
"apiNote:a:API Note:",
"implSpec:a:Implementation Requirements:",
"implNote:a:Implementation Note:"
]
subprojects.each { project ->
project.tasks.withType(Javadoc).each { javadocTask ->

195
checkstyle/checkstyle.xml Normal file
View File

@ -0,0 +1,195 @@
<?xml version="1.0"?>
<!DOCTYPE module PUBLIC
"-//Checkstyle//DTD Checkstyle Configuration 1.3//EN"
"https://checkstyle.org/dtds/configuration_1_3.dtd">
<module name="Checker">
<!--
If you set the basedir property below, then all reported file
names will be relative to the specified directory. See
https://checkstyle.org/5.x/config.html#Checker
<property name="basedir" value="${basedir}"/>
-->
<property name="severity" value="error"/>
<property name="fileExtensions" value="java, properties, xml"/>
<!-- Excludes all 'module-info.java' files -->
<!-- See https://checkstyle.org/config_filefilters.html -->
<module name="BeforeExecutionExclusionFileFilter">
<property name="fileNamePattern" value="module\-info\.java$"/>
</module>
<!-- Checks that a package-info.java file exists for each package. -->
<!-- See https://checkstyle.org/config_javadoc.html#JavadocPackage -->
<!--<module name="JavadocPackage"/>-->
<!-- Checks whether files end with a new line. -->
<!-- See https://checkstyle.org/config_misc.html#NewlineAtEndOfFile -->
<module name="NewlineAtEndOfFile"/>
<!-- Checks that property files contain the same keys. -->
<!-- See https://checkstyle.org/config_misc.html#Translation -->
<module name="Translation"/>
<!-- Checks for Size Violations. -->
<!-- See https://checkstyle.org/config_sizes.html -->
<module name="FileLength"/>
<module name="LineLength">
<property name="max" value="100"/>
<property name="fileExtensions" value="java"/>
<!-- Also allow links in javadocs to be longer (the default would just cover imports) -->
<property name="ignorePattern" value="^((package|import) .*)|( *\* (@see )?&lt;a href ?\= ?&quot;.*&quot;&gt;)$"/>
</module>
<!-- Checks for whitespace -->
<!-- See https://checkstyle.org/config_whitespace.html -->
<module name="FileTabCharacter"/>
<!-- Miscellaneous other checks. -->
<!-- See https://checkstyle.org/config_misc.html -->
<module name="RegexpSingleline">
<property name="format" value="\s+$"/>
<property name="minimum" value="0"/>
<property name="maximum" value="0"/>
<property name="message" value="Line has trailing spaces."/>
</module>
<!-- Checks for Headers -->
<!-- See https://checkstyle.org/config_header.html -->
<!-- <module name="Header"> -->
<!-- <property name="headerFile" value="${checkstyle.header.file}"/> -->
<!-- <property name="fileExtensions" value="java"/> -->
<!-- </module> -->
<module name="SuppressWarningsFilter" />
<module name="SuppressWithPlainTextCommentFilter"/>
<module name="TreeWalker">
<!-- Checks for Javadoc comments. -->
<!-- See https://checkstyle.org/config_javadoc.html -->
<module name="InvalidJavadocPosition"/>
<module name="JavadocMethod">
<property name="allowMissingParamTags" value="true"/>
<property name="allowMissingReturnTag" value="true"/>
</module>
<module name="JavadocType"/>
<!--<module name="JavadocVariable"/>-->
<module name="JavadocStyle">
<property name="checkFirstSentence" value="false"/>
</module>
<!--<module name="MissingJavadocMethod"/>-->
<!-- Checks for Naming Conventions. -->
<!-- See https://checkstyle.org/config_naming.html -->
<module name="ConstantName"/>
<module name="LocalFinalVariableName"/>
<module name="LocalVariableName"/>
<module name="MemberName">
<property name="format" value="^(TAG|DEBUG|[a-z][a-zA-Z0-9]*)$"/>
</module>
<module name="MethodName"/>
<module name="PackageName"/>
<module name="ParameterName"/>
<module name="StaticVariableName"/>
<module name="TypeName"/>
<!-- Checks for imports -->
<!-- See https://checkstyle.org/config_import.html -->
<module name="AvoidStarImport"/>
<module name="IllegalImport"> <!-- defaults to sun.* packages -->
<property name="illegalClasses" value="
org.jetbrains.annotations.Nullable,
org.jetbrains.annotations.NotNull,
androidx.annotation.Nullable,
androidx.annotation.NonNull,
io.reactivex.rxjava3.annotations.NonNull,
io.reactivex.rxjava3.annotations.Nullable" />
</module>
<module name="RedundantImport"/>
<module name="UnusedImports"/>
<!-- Checks for Size Violations. -->
<!-- See https://checkstyle.org/config_sizes.html -->
<module name="MethodLength">
<property name="severity" value="warning"/>
</module>
<module name="ParameterNumber">
<property name="severity" value="warning"/>
</module>
<!-- Checks for whitespace -->
<!-- See https://checkstyle.org/config_whitespace.html -->
<module name="EmptyForIteratorPad"/>
<module name="GenericWhitespace"/>
<module name="MethodParamPad"/>
<module name="NoWhitespaceAfter"/>
<module name="NoWhitespaceBefore"/>
<module name="OperatorWrap"/>
<module name="ParenPad"/>
<module name="TypecastParenPad"/>
<module name="WhitespaceAfter"/>
<module name="WhitespaceAround"/>
<!-- Modifier Checks -->
<!-- See https://checkstyle.org/config_modifiers.html -->
<module name="ModifierOrder"/>
<module name="RedundantModifier"/>
<!-- Checks for blocks. You know, those {}'s -->
<!-- See https://checkstyle.org/config_blocks.html -->
<module name="AvoidNestedBlocks"/>
<module name="EmptyBlock"/>
<module name="LeftCurly"/>
<module name="NeedBraces"/>
<module name="RightCurly"/>
<!-- Checks for common coding problems -->
<!-- See https://checkstyle.org/config_coding.html -->
<module name="EmptyStatement"/>
<module name="EqualsHashCode">
<property name="severity" value="warning"/>
</module>
<module name="HiddenField">
<property name="ignoreConstructorParameter" value="true"/>
<property name="ignoreSetter" value="true"/>
</module>
<module name="IllegalInstantiation"/>
<module name="InnerAssignment"/>
<!--<module name="MagicNumber"/>-->
<!--<module name="MissingSwitchDefault">
<property name="severity" value="warning"/>
</module>-->
<module name="MultipleVariableDeclarations"/>
<module name="SimplifyBooleanExpression"/>
<module name="SimplifyBooleanReturn"/>
<module name="FinalLocalVariable">
<property name="tokens" value="VARIABLE_DEF,PARAMETER_DEF"/>
<property name="validateEnhancedForLoopVariable" value="true"/>
</module>
<!-- Checks for class design -->
<!-- See https://checkstyle.org/config_design.html -->
<!--<module name="DesignForExtension"/>-->
<module name="FinalClass"/>
<module name="HideUtilityClassConstructor"/>
<module name="InterfaceIsType"/>
<!--<module name="VisibilityModifier">
<property name="ignoreAnnotationCanonicalNames" value="State,ColumnInfo"/>
<property name="severity" value="warning"/>
</module>-->
<!-- Miscellaneous other checks. -->
<!-- See https://checkstyle.org/config_misc.html -->
<module name="ArrayTypeStyle"/>
<module name="FinalParameters"/>
<!--<module name="TodoComment">
<property name="format" value="(TODO:|FIXME:)"/>
<property name="severity" value="warning"/>
</module>-->
<module name="UpperEll"/>
<module name="SuppressWarningsHolder" />
</module>
</module>

View File

@ -1,15 +0,0 @@
Copyright: 2017 Christian Schabesberger <chris.schabesberger@mailbox.com>
License: GPL-3.0+
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.

View File

@ -1,11 +1,45 @@
plugins {
id 'checkstyle'
}
test {
// Pass on downloader type to tests for different CI jobs. See DownloaderFactory.java and ci.yml
if (System.properties.containsKey('downloader')) {
systemProperty('downloader', System.getProperty('downloader'))
}
useJUnitPlatform()
dependsOn checkstyleMain // run checkstyle when testing
}
checkstyle {
getConfigDirectory().set(rootProject.file("checkstyle"))
ignoreFailures false
showViolations true
toolVersion checkstyleVersion
}
checkstyleTest {
enabled false // do not checkstyle test files
}
dependencies {
implementation project(':timeago-parser')
implementation 'com.grack:nanojson:1.1'
implementation 'org.jsoup:jsoup:1.9.2'
implementation 'org.mozilla:rhino:1.7.7.1'
implementation 'com.github.spotbugs:spotbugs-annotations:3.1.0'
implementation 'org.nibor.autolink:autolink:0.8.0'
implementation "com.github.TeamNewPipe:nanojson:$nanojsonVersion"
implementation 'org.jsoup:jsoup:1.17.2'
implementation "com.github.spotbugs:spotbugs-annotations:$spotbugsVersion"
testImplementation 'junit:junit:4.12'
}
// do not upgrade to 1.7.14, since in 1.7.14 Rhino uses the `SourceVersion` class, which is not
// available on Android (even when using desugaring), and `NoClassDefFoundError` is thrown
implementation 'org.mozilla:rhino:1.7.15'
checkstyle "com.puppycrawl.tools:checkstyle:$checkstyleVersion"
testImplementation platform("org.junit:junit-bom:$junitVersion")
testImplementation 'org.junit.jupiter:junit-jupiter-api'
testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine'
testImplementation 'org.junit.jupiter:junit-jupiter-params'
testImplementation "com.squareup.okhttp3:okhttp:3.12.13"
testImplementation 'com.google.code.gson:gson:2.10.1'
}

View File

@ -10,12 +10,15 @@ import org.schabi.newpipe.extractor.localization.TimeAgoParser;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.io.IOException;
import java.util.Objects;
public abstract class Extractor {
/**
* {@link StreamingService} currently related to this extractor.<br>
* Useful for getting other things from a service (like the url handlers for cleaning/accepting/get id from urls).
* Useful for getting other things from a service (like the url handlers for
* cleaning/accepting/get id from urls).
*/
private final StreamingService service;
private final LinkHandler linkHandler;
@ -26,19 +29,18 @@ public abstract class Extractor {
private ContentCountry forcedContentCountry = null;
private boolean pageFetched = false;
// called like this to prevent checkstyle errors about "hiding a field"
private final Downloader downloader;
public Extractor(final StreamingService service, final LinkHandler linkHandler) {
if (service == null) throw new NullPointerException("service is null");
if (linkHandler == null) throw new NullPointerException("LinkHandler is null");
this.service = service;
this.linkHandler = linkHandler;
this.downloader = NewPipe.getDownloader();
if (downloader == null) throw new NullPointerException("downloader is null");
protected Extractor(final StreamingService service, final LinkHandler linkHandler) {
this.service = Objects.requireNonNull(service, "service is null");
this.linkHandler = Objects.requireNonNull(linkHandler, "LinkHandler is null");
this.downloader = Objects.requireNonNull(NewPipe.getDownloader(), "downloader is null");
}
/**
* @return The {@link LinkHandler} of the current extractor object (e.g. a ChannelExtractor should return a channel url handler).
* @return The {@link LinkHandler} of the current extractor object (e.g. a ChannelExtractor
* should return a channel url handler).
*/
@Nonnull
public LinkHandler getLinkHandler() {
@ -52,13 +54,17 @@ public abstract class Extractor {
* @throws ExtractionException if the pages content is not understood
*/
public void fetchPage() throws IOException, ExtractionException {
if (pageFetched) return;
if (pageFetched) {
return;
}
onFetchPage(downloader);
pageFetched = true;
}
protected void assertPageFetched() {
if (!pageFetched) throw new IllegalStateException("Page is not fetched. Make sure you call fetchPage()");
if (!pageFetched) {
throw new IllegalStateException("Page is not fetched. Make sure you call fetchPage()");
}
}
protected boolean isPageFetched() {
@ -68,11 +74,13 @@ public abstract class Extractor {
/**
* Fetch the current page.
*
* @param downloader the download to use
* @param downloader the downloader to use
* @throws IOException if the page can not be loaded
* @throws ExtractionException if the pages content is not understood
*/
public abstract void onFetchPage(@Nonnull Downloader downloader) throws IOException, ExtractionException;
@SuppressWarnings("HiddenField")
public abstract void onFetchPage(@Nonnull Downloader downloader)
throws IOException, ExtractionException;
@Nonnull
public String getId() throws ParsingException {
@ -120,11 +128,11 @@ public abstract class Extractor {
// Localization
//////////////////////////////////////////////////////////////////////////*/
public void forceLocalization(Localization localization) {
public void forceLocalization(final Localization localization) {
this.forcedLocalization = localization;
}
public void forceContentCountry(ContentCountry contentCountry) {
public void forceContentCountry(final ContentCountry contentCountry) {
this.forcedContentCountry = contentCountry;
}
@ -135,7 +143,8 @@ public abstract class Extractor {
@Nonnull
public ContentCountry getExtractorContentCountry() {
return forcedContentCountry == null ? getService().getContentCountry() : forcedContentCountry;
return forcedContentCountry == null ? getService().getContentCountry()
: forcedContentCountry;
}
@Nonnull

View File

@ -0,0 +1,211 @@
package org.schabi.newpipe.extractor;
import javax.annotation.Nonnull;
import java.io.Serializable;
import java.util.Objects;
/**
* Class representing images in the extractor.
*
* <p>
* An image has four properties: its URL, its height, its width and its estimated quality level.
* </p>
*
* <p>
* Depending of the services, the height, the width or both properties may be not known.
* Implementations <b>must use</b> the relevant unknown constants in this case
* ({@link #HEIGHT_UNKNOWN} and {@link #WIDTH_UNKNOWN}), to ensure properly the lack of knowledge
* of one or both of these properties to extractor clients.
* </p>
*
* <p>
* They should also respect the ranges defined in the estimated image resolution levels as much as
* possible, to ensure consistency to extractor clients.
* </p>
*/
public final class Image implements Serializable {
/**
* Constant representing that the height of an {@link Image} is unknown.
*/
public static final int HEIGHT_UNKNOWN = -1;
/**
* Constant representing that the width of an {@link Image} is unknown.
*/
public static final int WIDTH_UNKNOWN = -1;
@Nonnull
private final String url;
private final int height;
private final int width;
@Nonnull
private final ResolutionLevel estimatedResolutionLevel;
/**
* Construct an {@link Image} instance.
*
* @param url the URL to the image, which should be not null or empty
* @param height the image's height
* @param width the image's width
* @param estimatedResolutionLevel the image's estimated resolution level, which must not be
* null
* @throws NullPointerException if {@code estimatedResolutionLevel} is null
*/
public Image(@Nonnull final String url,
final int height,
final int width,
@Nonnull final ResolutionLevel estimatedResolutionLevel)
throws NullPointerException {
this.url = url;
this.height = height;
this.width = width;
this.estimatedResolutionLevel = Objects.requireNonNull(
estimatedResolutionLevel, "estimatedResolutionLevel is null");
}
/**
* Get the URL of this {@link Image}.
*
* @return the {@link Image}'s URL.
*/
@Nonnull
public String getUrl() {
return url;
}
/**
* Get the height of this {@link Image}.
*
* <p>
* If it is unknown, {@link #HEIGHT_UNKNOWN} is returned instead.
* </p>
*
* @return the {@link Image}'s height or {@link #HEIGHT_UNKNOWN}
*/
public int getHeight() {
return height;
}
/**
* Get the width of this {@link Image}.
*
* <p>
* If it is unknown, {@link #WIDTH_UNKNOWN} is returned instead.
* </p>
*
* @return the {@link Image}'s width or {@link #WIDTH_UNKNOWN}
*/
public int getWidth() {
return width;
}
/**
* Get the estimated resolution level of this image.
*
* <p>
* If it is unknown, {@link ResolutionLevel#UNKNOWN} is returned instead.
* </p>
*
* @return the estimated resolution level, which is never {@code null}
* @see ResolutionLevel
*/
@Nonnull
public ResolutionLevel getEstimatedResolutionLevel() {
return estimatedResolutionLevel;
}
/**
* Get a string representation of this {@link Image} instance.
*
* <p>
* The representation will be in the following format, where {@code url}, {@code height},
* {@code width} and {@code estimatedResolutionLevel} represent the corresponding properties:
* <br>
* <br>
* {@code Image {url=url, height='height, width=width,
* estimatedResolutionLevel=estimatedResolutionLevel}'}
* </p>
*
* @return a string representation of this {@link Image} instance
*/
@Nonnull
@Override
public String toString() {
return "Image {" + "url=" + url + ", height=" + height + ", width=" + width
+ ", estimatedResolutionLevel=" + estimatedResolutionLevel + "}";
}
/**
* The estimated resolution level of an {@link Image}.
*
* <p>
* Some services don't return the size of their images, but we may know for a specific image
* type that a service returns, according to real data, an approximation of the resolution
* level.
* </p>
*/
public enum ResolutionLevel {
/**
* The high resolution level.
*
* <p>
* This level applies to images with a height greater than or equal to 720px.
* </p>
*/
HIGH,
/**
* The medium resolution level.
*
* <p>
* This level applies to images with a height between 175px inclusive and 720px exclusive.
* </p>
*/
MEDIUM,
/**
* The low resolution level.
*
* <p>
* This level applies to images with a height between 1px inclusive and 175px exclusive.
* </p>
*/
LOW,
/**
* The unknown resolution level.
*
* <p>
* This value is returned when the extractor doesn't know what resolution level an image
* could have, for example if the extractor loops in an array of images with different
* resolution levels without knowing the height.
* </p>
*/
UNKNOWN;
/**
* Get a {@link ResolutionLevel} based from the given height.
*
* @param heightPx the height from which returning the good {@link ResolutionLevel}
* @return the {@link ResolutionLevel} corresponding to the height provided. See the
* {@link ResolutionLevel} values for details about what value is returned.
*/
public static ResolutionLevel fromHeight(final int heightPx) {
if (heightPx <= 0) {
return UNKNOWN;
}
if (heightPx < 175) {
return LOW;
}
if (heightPx < 720) {
return MEDIUM;
}
return HIGH;
}
}
}

View File

@ -1,5 +1,6 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.LinkHandler;
import java.io.Serializable;
@ -16,7 +17,8 @@ public abstract class Info implements Serializable {
*/
private final String id;
/**
* Different than the {@link #originalUrl} in the sense that it <i>may</i> be set as a cleaned url.
* Different than the {@link #originalUrl} in the sense that it <i>may</i> be set as a cleaned
* url.
*
* @see LinkHandler#getUrl()
* @see Extractor#getOriginalUrl()
@ -32,15 +34,19 @@ public abstract class Info implements Serializable {
private final List<Throwable> errors = new ArrayList<>();
public void addError(Throwable throwable) {
public void addError(final Throwable throwable) {
this.errors.add(throwable);
}
public void addAllErrors(Collection<Throwable> errors) {
this.errors.addAll(errors);
public void addAllErrors(final Collection<Throwable> throwables) {
this.errors.addAll(throwables);
}
public Info(int serviceId, String id, String url, String originalUrl, String name) {
public Info(final int serviceId,
final String id,
final String url,
final String originalUrl,
final String name) {
this.serviceId = serviceId;
this.id = id;
this.url = url;
@ -48,7 +54,7 @@ public abstract class Info implements Serializable {
this.name = name;
}
public Info(int serviceId, LinkHandler linkHandler, String name) {
public Info(final int serviceId, final LinkHandler linkHandler, final String name) {
this(serviceId,
linkHandler.getId(),
linkHandler.getUrl(),
@ -58,20 +64,31 @@ public abstract class Info implements Serializable {
@Override
public String toString() {
final String ifDifferentString = !url.equals(originalUrl) ? " (originalUrl=\"" + originalUrl + "\")" : "";
return getClass().getSimpleName() + "[url=\"" + url + "\"" + ifDifferentString + ", name=\"" + name + "\"]";
final String ifDifferentString
= url.equals(originalUrl) ? "" : " (originalUrl=\"" + originalUrl + "\")";
return getClass().getSimpleName() + "[url=\"" + url + "\"" + ifDifferentString
+ ", name=\"" + name + "\"]";
}
// if you use an api and want to handle the website url
// overriding original url is essential
public void setOriginalUrl(String url) {
originalUrl = url;
public void setOriginalUrl(final String originalUrl) {
this.originalUrl = originalUrl;
}
public int getServiceId() {
return serviceId;
}
public StreamingService getService() {
try {
return NewPipe.getService(serviceId);
} catch (final ExtractionException e) {
// this should be unreachable, as serviceId certainly refers to a valid service
throw new RuntimeException("Info object has invalid service id", e);
}
}
public String getId() {
return id;
}

View File

@ -1,35 +1,41 @@
package org.schabi.newpipe.extractor;
/*
* Created by Christian Schabesberger on 11.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* InfoItem.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* InfoItem.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <https://www.gnu.org/licenses/>.
*/
package org.schabi.newpipe.extractor;
import javax.annotation.Nonnull;
import java.io.Serializable;
import java.util.List;
public abstract class InfoItem implements Serializable {
private final InfoType infoType;
private final int serviceId;
private final String url;
private final String name;
private String thumbnailUrl;
@Nonnull
private List<Image> thumbnails = List.of();
public InfoItem(InfoType infoType, int serviceId, String url, String name) {
public InfoItem(final InfoType infoType,
final int serviceId,
final String url,
final String name) {
this.infoType = infoType;
this.serviceId = serviceId;
this.url = url;
@ -52,12 +58,13 @@ public abstract class InfoItem implements Serializable {
return name;
}
public void setThumbnailUrl(String thumbnailUrl) {
this.thumbnailUrl = thumbnailUrl;
public void setThumbnails(@Nonnull final List<Image> thumbnails) {
this.thumbnails = thumbnails;
}
public String getThumbnailUrl() {
return thumbnailUrl;
@Nonnull
public List<Image> getThumbnails() {
return thumbnails;
}
@Override

View File

@ -2,8 +2,12 @@ package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import javax.annotation.Nonnull;
import java.util.List;
public interface InfoItemExtractor {
String getName() throws ParsingException;
String getUrl() throws ParsingException;
String getThumbnailUrl() throws ParsingException;
@Nonnull
List<Image> getThumbnails() throws ParsingException;
}

View File

@ -3,46 +3,63 @@ package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.exceptions.FoundAdException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import javax.annotation.Nullable;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.List;
/*
* Created by Christian Schabesberger on 12.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* InfoItemsCollector.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* InfoItemsCollector.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public abstract class InfoItemsCollector<I extends InfoItem, E extends InfoItemExtractor> implements Collector<I, E> {
public abstract class InfoItemsCollector<I extends InfoItem, E extends InfoItemExtractor>
implements Collector<I, E> {
private final List<I> itemList = new ArrayList<>();
private final List<Throwable> errors = new ArrayList<>();
private final int serviceId;
@Nullable
private final Comparator<I> comparator;
/**
* Create a new collector with no comparator / sorting function
* @param serviceId the service id
*/
public InfoItemsCollector(final int serviceId) {
this(serviceId, null);
}
/**
* Create a new collector
* @param serviceId the service id
*/
public InfoItemsCollector(int serviceId) {
public InfoItemsCollector(final int serviceId, @Nullable final Comparator<I> comparator) {
this.serviceId = serviceId;
this.comparator = comparator;
}
@Override
public List<I> getItems() {
if (comparator != null) {
itemList.sort(comparator);
}
return Collections.unmodifiableList(itemList);
}
@ -61,7 +78,7 @@ public abstract class InfoItemsCollector<I extends InfoItem, E extends InfoItemE
* Add an error
* @param error the error
*/
protected void addError(Exception error) {
protected void addError(final Exception error) {
errors.add(error);
}
@ -69,7 +86,7 @@ public abstract class InfoItemsCollector<I extends InfoItem, E extends InfoItemE
* Add an item
* @param item the item
*/
protected void addItem(I item) {
protected void addItem(final I item) {
itemList.add(item);
}
@ -82,12 +99,12 @@ public abstract class InfoItemsCollector<I extends InfoItem, E extends InfoItemE
}
@Override
public void commit(E extractor) {
public void commit(final E extractor) {
try {
addItem(extract(extractor));
} catch (FoundAdException ae) {
} catch (final FoundAdException ae) {
// found an ad. Maybe a debug line could be placed here
} catch (ParsingException e) {
} catch (final ParsingException e) {
addError(e);
}
}

View File

@ -3,53 +3,57 @@ package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import javax.annotation.Nonnull;
import java.io.IOException;
import java.util.Collections;
import java.util.List;
import javax.annotation.Nonnull;
/**
* Base class to extractors that have a list (e.g. playlists, users).
* @param <R> the info item type this list extractor provides
*/
public abstract class ListExtractor<R extends InfoItem> extends Extractor {
/**
* Constant that should be returned whenever
* a list has an unknown number of items.
*/
public static final long ITEM_COUNT_UNKNOWN = -1;
/**
* Constant that should be returned whenever a list has an
* infinite number of items. For example a YouTube mix.
*/
public static final long ITEM_COUNT_INFINITE = -2;
/**
* Constant that should be returned whenever a list
* has an unknown number of items bigger than 100.
*/
public static final long ITEM_COUNT_MORE_THAN_100 = -3;
public ListExtractor(StreamingService service, ListLinkHandler linkHandler) {
public ListExtractor(final StreamingService service, final ListLinkHandler linkHandler) {
super(service, linkHandler);
}
/**
* A {@link InfoItemsPage InfoItemsPage} corresponding to the initial page where the items are from the initial request and
* the nextPageUrl relative to it.
* A {@link InfoItemsPage InfoItemsPage} corresponding to the initial page
* where the items are from the initial request and the nextPage relative to it.
*
* @return a {@link InfoItemsPage} corresponding to the initial page
*/
@Nonnull
public abstract InfoItemsPage<R> getInitialPage() throws IOException, ExtractionException;
/**
* Returns an url that can be used to get the next page relative to the initial one.
* <p>Usually, these links will only work in the implementation itself.</p>
*
* @return an url pointing to the next page relative to the initial page
* @see #getPage(String)
*/
public abstract String getNextPageUrl() throws IOException, ExtractionException;
/**
* Get a list of items corresponding to the specific requested page.
*
* @param pageUrl any page url got from the exclusive implementation of the list extractor
* @param page any page got from the exclusive implementation of the list extractor
* @return a {@link InfoItemsPage} corresponding to the requested page
* @see #getNextPageUrl()
* @see InfoItemsPage#getNextPageUrl()
* @see InfoItemsPage#getNextPage()
*/
public abstract InfoItemsPage<R> getPage(final String pageUrl) throws IOException, ExtractionException;
public boolean hasNextPage() throws IOException, ExtractionException {
final String nextPageUrl = getNextPageUrl();
return nextPageUrl != null && !nextPageUrl.isEmpty();
}
public abstract InfoItemsPage<R> getPage(Page page) throws IOException, ExtractionException;
@Nonnull
@Override
public ListLinkHandler getLinkHandler() {
return (ListLinkHandler) super.getLinkHandler();
@ -61,23 +65,24 @@ public abstract class ListExtractor<R extends InfoItem> extends Extractor {
/**
* A class that is used to wrap a list of gathered items and eventual errors, it
* also contains a field that points to the next available page ({@link #nextPageUrl}).
* also contains a field that points to the next available page ({@link #nextPage}).
* @param <T> the info item type that this page is supposed to store and provide
*/
public static class InfoItemsPage<T extends InfoItem> {
private static final InfoItemsPage<InfoItem> EMPTY =
new InfoItemsPage<>(Collections.<InfoItem>emptyList(), "", Collections.<Throwable>emptyList());
new InfoItemsPage<>(Collections.emptyList(), null, Collections.emptyList());
/**
* A convenient method that returns a representation of an empty page.
*
* @return a type-safe page with the list of items and errors empty and the nextPageUrl set to an empty string.
* @return a type-safe page with the list of items and errors empty and the nextPage set to
* {@code null}.
*/
public static <T extends InfoItem> InfoItemsPage<T> emptyPage() {
//noinspection unchecked
return (InfoItemsPage<T>) EMPTY;
}
/**
* The current list of items of this page
*/
@ -86,40 +91,42 @@ public abstract class ListExtractor<R extends InfoItem> extends Extractor {
/**
* Url pointing to the next page relative to this one
*
* @see ListExtractor#getPage(String)
* @see ListExtractor#getPage(Page)
* @see Page
*/
private final String nextPageUrl;
private final Page nextPage;
/**
* Errors that happened during the extraction
*/
private final List<Throwable> errors;
public InfoItemsPage(InfoItemsCollector<T, ?> collector, String nextPageUrl) {
this(collector.getItems(), nextPageUrl, collector.getErrors());
public InfoItemsPage(final InfoItemsCollector<T, ?> collector, final Page nextPage) {
this(collector.getItems(), nextPage, collector.getErrors());
}
public InfoItemsPage(List<T> itemsList, String nextPageUrl, List<Throwable> errors) {
public InfoItemsPage(final List<T> itemsList,
final Page nextPage,
final List<Throwable> errors) {
this.itemsList = itemsList;
this.nextPageUrl = nextPageUrl;
this.nextPage = nextPage;
this.errors = errors;
}
public boolean hasNextPage() {
return nextPageUrl != null && !nextPageUrl.isEmpty();
return Page.isValid(nextPage);
}
public List<T> getItems() {
return itemsList;
}
public String getNextPageUrl() {
return nextPageUrl;
public Page getNextPage() {
return nextPage;
}
public List<Throwable> getErrors() {
return errors;
}
}
}

View File

@ -6,23 +6,25 @@ import java.util.List;
public abstract class ListInfo<T extends InfoItem> extends Info {
private List<T> relatedItems;
private String nextPageUrl = null;
private Page nextPage = null;
private final List<String> contentFilters;
private final String sortFilter;
public ListInfo(int serviceId,
String id,
String url,
String originalUrl,
String name,
List<String> contentFilter,
String sortFilter) {
public ListInfo(final int serviceId,
final String id,
final String url,
final String originalUrl,
final String name,
final List<String> contentFilter,
final String sortFilter) {
super(serviceId, id, url, originalUrl, name);
this.contentFilters = contentFilter;
this.sortFilter = sortFilter;
}
public ListInfo(int serviceId, ListLinkHandler listUrlIdHandler, String name) {
public ListInfo(final int serviceId,
final ListLinkHandler listUrlIdHandler,
final String name) {
super(serviceId, listUrlIdHandler, name);
this.contentFilters = listUrlIdHandler.getContentFilters();
this.sortFilter = listUrlIdHandler.getSortFilter();
@ -32,20 +34,20 @@ public abstract class ListInfo<T extends InfoItem> extends Info {
return relatedItems;
}
public void setRelatedItems(List<T> relatedItems) {
public void setRelatedItems(final List<T> relatedItems) {
this.relatedItems = relatedItems;
}
public boolean hasNextPage() {
return nextPageUrl != null && !nextPageUrl.isEmpty();
return Page.isValid(nextPage);
}
public String getNextPageUrl() {
return nextPageUrl;
public Page getNextPage() {
return nextPage;
}
public void setNextPageUrl(String pageUrl) {
this.nextPageUrl = pageUrl;
public void setNextPage(final Page page) {
this.nextPage = page;
}
public List<String> getContentFilters() {

View File

@ -3,115 +3,158 @@ package org.schabi.newpipe.extractor;
/*
* Created by Adam Howard on 08/11/15.
*
* Copyright (c) Christian Schabesberger <chris.schabesberger@mailbox.org>
* and Adam Howard <achdisposable1@gmail.com> 2015
* Copyright (c) 2015 Christian Schabesberger <chris.schabesberger@mailbox.org>
* and Adam Howard <achdisposable1@gmail.com>
*
* MediaFormat.java is part of NewPipe.
* MediaFormat.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.Arrays;
import java.util.List;
import java.util.function.Function;
import java.util.stream.Collectors;
/**
* Static data about various media formats support by NewPipe, eg mime type, extension
*/
@SuppressWarnings("MethodParamPad") // we want the media format table below to be aligned
public enum MediaFormat {
// @formatter:off
//video and audio combined formats
// id name suffix mime type
MPEG_4 (0x0, "MPEG-4", "mp4", "video/mp4"),
v3GPP (0x10, "3GPP", "3gp", "video/3gpp"),
WEBM (0x20, "WebM", "webm", "video/webm"),
// id name suffix mimeType
MPEG_4 (0x0, "MPEG-4", "mp4", "video/mp4"),
v3GPP (0x10, "3GPP", "3gp", "video/3gpp"),
WEBM (0x20, "WebM", "webm", "video/webm"),
// audio formats
M4A (0x100, "m4a", "m4a", "audio/mp4"),
WEBMA (0x200, "WebM", "webm", "audio/webm"),
MP3 (0x300, "MP3", "mp3", "audio/mpeg"),
OPUS (0x400, "opus", "opus", "audio/opus"),
OGG (0x500, "ogg", "ogg", "audio/ogg"),
WEBMA_OPUS (0x200, "WebM Opus", "webm", "audio/webm"),
M4A (0x100, "m4a", "m4a", "audio/mp4"),
WEBMA (0x200, "WebM", "webm", "audio/webm"),
MP3 (0x300, "MP3", "mp3", "audio/mpeg"),
MP2 (0x310, "MP2", "mp2", "audio/mpeg"),
OPUS (0x400, "opus", "opus", "audio/opus"),
OGG (0x500, "ogg", "ogg", "audio/ogg"),
WEBMA_OPUS(0x200, "WebM Opus", "webm", "audio/webm"),
AIFF (0x600, "AIFF", "aiff", "audio/aiff"),
/**
* Same as {@link MediaFormat#AIFF}, just with the shorter suffix/file extension
*/
AIF (0x600, "AIFF", "aif", "audio/aiff"),
WAV (0x700, "WAV", "wav", "audio/wav"),
FLAC (0x800, "FLAC", "flac", "audio/flac"),
ALAC (0x900, "ALAC", "alac", "audio/alac"),
// subtitles formats
VTT (0x1000, "WebVTT", "vtt", "text/vtt"),
TTML (0x2000, "Timed Text Markup Language", "ttml", "application/ttml+xml"),
TRANSCRIPT1 (0x3000, "TranScript v1", "srv1", "text/xml"),
TRANSCRIPT2 (0x4000, "TranScript v2", "srv2", "text/xml"),
TRANSCRIPT3 (0x5000, "TranScript v3", "srv3", "text/xml"),
SRT (0x6000, "SubRip file format", "srt", "text/srt");
VTT (0x1000, "WebVTT", "vtt", "text/vtt"),
TTML (0x2000, "Timed Text Markup Language", "ttml", "application/ttml+xml"),
TRANSCRIPT1(0x3000, "TranScript v1", "srv1", "text/xml"),
TRANSCRIPT2(0x4000, "TranScript v2", "srv2", "text/xml"),
TRANSCRIPT3(0x5000, "TranScript v3", "srv3", "text/xml"),
SRT (0x6000, "SubRip file format", "srt", "text/srt");
// @formatter:on
public final int id;
@Nonnull
public final String name;
@Nonnull
public final String suffix;
@Nonnull
public final String mimeType;
MediaFormat(int id, String name, String suffix, String mimeType) {
MediaFormat(final int id, @Nonnull final String name,
@Nonnull final String suffix, @Nonnull final String mimeType) {
this.id = id;
this.name = name;
this.suffix = suffix;
this.mimeType = mimeType;
}
private static <T> T getById(final int id,
final Function<MediaFormat, T> field,
final T orElse) {
return Arrays.stream(MediaFormat.values())
.filter(mediaFormat -> mediaFormat.id == id)
.map(field)
.findFirst()
.orElse(orElse);
}
/**
* Return the friendly name of the media format with the supplied id
*
* @param ident the id of the media format. Currently an arbitrary, NewPipe-specific number.
* @param id the id of the media format. Currently an arbitrary, NewPipe-specific number.
* @return the friendly name of the MediaFormat associated with this ids,
* or an empty String if none match it.
*/
public static String getNameById(int ident) {
for (MediaFormat vf : MediaFormat.values()) {
if (vf.id == ident) return vf.name;
}
return "";
@Nonnull
public static String getNameById(final int id) {
return getById(id, MediaFormat::getName, "");
}
/**
* Return the file extension of the media format with the supplied id
*
* @param ident the id of the media format. Currently an arbitrary, NewPipe-specific number.
* @param id the id of the media format. Currently an arbitrary, NewPipe-specific number.
* @return the file extension of the MediaFormat associated with this ids,
* or an empty String if none match it.
*/
public static String getSuffixById(int ident) {
for (MediaFormat vf : MediaFormat.values()) {
if (vf.id == ident) return vf.suffix;
}
return "";
@Nonnull
public static String getSuffixById(final int id) {
return getById(id, MediaFormat::getSuffix, "");
}
/**
* Return the MIME type of the media format with the supplied id
*
* @param ident the id of the media format. Currently an arbitrary, NewPipe-specific number.
* @param id the id of the media format. Currently an arbitrary, NewPipe-specific number.
* @return the MIME type of the MediaFormat associated with this ids,
* or an empty String if none match it.
*/
public static String getMimeById(int ident) {
for (MediaFormat vf : MediaFormat.values()) {
if (vf.id == ident) return vf.mimeType;
}
return "";
@Nullable
public static String getMimeById(final int id) {
return getById(id, MediaFormat::getMimeType, null);
}
/**
* Return the MediaFormat with the supplied mime type
* Return the first {@link MediaFormat} with the supplied mime type.
* There might be more formats which have the same mime type.
* To retrieve those, use {@link #getAllFromMimeType(String)}.
*
* @return MediaFormat associated with this mime type,
* or null if none match it.
*/
public static MediaFormat getFromMimeType(String mimeType) {
for (MediaFormat vf : MediaFormat.values()) {
if (vf.mimeType.equals(mimeType)) return vf;
}
return null;
@Nullable
public static MediaFormat getFromMimeType(final String mimeType) {
return Arrays.stream(MediaFormat.values())
.filter(mediaFormat -> mediaFormat.mimeType.equals(mimeType))
.findFirst()
.orElse(null);
}
/**
* Get all media formats which have the given mime type.
* @param mimeType the mime type to search for
* @return a modifiable {@link List} which contains the {@link MediaFormat}s
* that have the given mime type.
*/
@Nonnull
public static List<MediaFormat> getAllFromMimeType(final String mimeType) {
return Arrays.stream(MediaFormat.values())
.filter(mediaFormat -> mediaFormat.mimeType.equals(mimeType))
.collect(Collectors.toList());
}
/**
@ -120,18 +163,21 @@ public enum MediaFormat {
* @param id the id
* @return the id of the media format or null.
*/
public static MediaFormat getFormatById(int id) {
for (MediaFormat vf : values()) {
if (vf.id == id) return vf;
}
return null;
@Nullable
public static MediaFormat getFormatById(final int id) {
return getById(id, mediaFormat -> mediaFormat, null);
}
public static MediaFormat getFromSuffix(String suffix) {
for (MediaFormat vf : values()) {
if (vf.suffix.equals(suffix)) return vf;
}
return null;
/**
* Get the first media format that has the given suffix/file extension.
* @return the matching {@link MediaFormat} or {@code null} if no associated format is found
*/
@Nullable
public static MediaFormat getFromSuffix(final String suffix) {
return Arrays.stream(MediaFormat.values())
.filter(mediaFormat -> mediaFormat.suffix.equals(suffix))
.findFirst()
.orElse(null);
}
/**
@ -139,6 +185,7 @@ public enum MediaFormat {
*
* @return the name of the format
*/
@Nonnull
public String getName() {
return name;
}
@ -148,6 +195,7 @@ public enum MediaFormat {
*
* @return the filename extension
*/
@Nonnull
public String getSuffix() {
return suffix;
}
@ -157,6 +205,7 @@ public enum MediaFormat {
*
* @return the mime type
*/
@Nonnull
public String getMimeType() {
return mimeType;
}

View File

@ -0,0 +1,78 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.stream.Description;
import java.io.Serializable;
import java.net.URL;
import java.util.ArrayList;
import java.util.List;
import javax.annotation.Nonnull;
public class MetaInfo implements Serializable {
private String title = "";
private Description content;
private List<URL> urls = new ArrayList<>();
private List<String> urlTexts = new ArrayList<>();
public MetaInfo(@Nonnull final String title,
@Nonnull final Description content,
@Nonnull final List<URL> urls,
@Nonnull final List<String> urlTexts) {
this.title = title;
this.content = content;
this.urls = urls;
this.urlTexts = urlTexts;
}
public MetaInfo() {
}
/**
* @return Title of the info. Can be empty.
*/
@Nonnull
public String getTitle() {
return title;
}
public void setTitle(@Nonnull final String title) {
this.title = title;
}
@Nonnull
public Description getContent() {
return content;
}
public void setContent(@Nonnull final Description content) {
this.content = content;
}
@Nonnull
public List<URL> getUrls() {
return urls;
}
public void setUrls(@Nonnull final List<URL> urls) {
this.urls = urls;
}
public void addUrl(@Nonnull final URL url) {
urls.add(url);
}
@Nonnull
public List<String> getUrlTexts() {
return urlTexts;
}
public void setUrlTexts(@Nonnull final List<String> urlTexts) {
this.urlTexts = urlTexts;
}
public void addUrlText(@Nonnull final String urlText) {
urlTexts.add(urlText);
}
}

View File

@ -1,8 +1,5 @@
package org.schabi.newpipe.extractor.search;
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.InfoItemExtractor;
import org.schabi.newpipe.extractor.InfoItemsCollector;
import org.schabi.newpipe.extractor.channel.ChannelInfoItemExtractor;
import org.schabi.newpipe.extractor.channel.ChannelInfoItemsCollector;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
@ -18,25 +15,26 @@ import java.util.List;
/*
* Created by Christian Schabesberger on 12.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* InfoItemsSearchCollector.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* InfoItemsSearchCollector.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
/**
* Collector for search results
* A collector that can handle many extractor types, to be used when a list contains items of
* different types (e.g. search)
* <p>
* This collector can handle the following extractor types:
* <ul>
@ -44,15 +42,15 @@ import java.util.List;
* <li>{@link ChannelInfoItemExtractor}</li>
* <li>{@link PlaylistInfoItemExtractor}</li>
* </ul>
* Calling {@link #extract(InfoItemExtractor)} or {@link #commit(Object)} with any
* Calling {@link #extract(InfoItemExtractor)} or {@link #commit(InfoItemExtractor)} with any
* other extractor type will raise an exception.
*/
public class InfoItemsSearchCollector extends InfoItemsCollector<InfoItem, InfoItemExtractor> {
public class MultiInfoItemsCollector extends InfoItemsCollector<InfoItem, InfoItemExtractor> {
private final StreamInfoItemsCollector streamCollector;
private final ChannelInfoItemsCollector userCollector;
private final PlaylistInfoItemsCollector playlistCollector;
public InfoItemsSearchCollector(int serviceId) {
public MultiInfoItemsCollector(final int serviceId) {
super(serviceId);
streamCollector = new StreamInfoItemsCollector(serviceId);
userCollector = new ChannelInfoItemsCollector(serviceId);
@ -78,7 +76,7 @@ public class InfoItemsSearchCollector extends InfoItemsCollector<InfoItem, InfoI
}
@Override
public InfoItem extract(InfoItemExtractor extractor) throws ParsingException {
public InfoItem extract(final InfoItemExtractor extractor) throws ParsingException {
// Use the corresponding collector for each item extractor type
if (extractor instanceof StreamInfoItemExtractor) {
return streamCollector.extract((StreamInfoItemExtractor) extractor);

View File

@ -3,21 +3,21 @@ package org.schabi.newpipe.extractor;
/*
* Created by Christian Schabesberger on 23.08.15.
*
* Copyright (C) Christian Schabesberger 2015 <chris.schabesberger@mailbox.org>
* NewPipe.java is part of NewPipe.
* Copyright (C) 2015 Christian Schabesberger <chris.schabesberger@mailbox.org>
* NewPipe Extractor.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
import org.schabi.newpipe.extractor.downloader.Downloader;
@ -25,14 +25,15 @@ import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.localization.ContentCountry;
import org.schabi.newpipe.extractor.localization.Localization;
import java.util.List;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.List;
/**
* Provides access to streaming services supported by NewPipe.
*/
public class NewPipe {
public final class NewPipe {
private static Downloader downloader;
private static Localization preferredLocalization;
private static ContentCountry preferredContentCountry;
@ -40,19 +41,16 @@ public class NewPipe {
private NewPipe() {
}
public static void init(Downloader d) {
downloader = d;
preferredLocalization = Localization.DEFAULT;
preferredContentCountry = ContentCountry.DEFAULT;
public static void init(final Downloader d) {
init(d, Localization.DEFAULT);
}
public static void init(Downloader d, Localization l) {
downloader = d;
preferredLocalization = l;
preferredContentCountry = l.getCountryCode().isEmpty() ? ContentCountry.DEFAULT : new ContentCountry(l.getCountryCode());
public static void init(final Downloader d, final Localization l) {
init(d, l, l.getCountryCode().isEmpty()
? ContentCountry.DEFAULT : new ContentCountry(l.getCountryCode()));
}
public static void init(Downloader d, Localization l, ContentCountry c) {
public static void init(final Downloader d, final Localization l, final ContentCountry c) {
downloader = d;
preferredLocalization = l;
preferredContentCountry = c;
@ -70,26 +68,24 @@ public class NewPipe {
return ServiceList.all();
}
public static StreamingService getService(int serviceId) throws ExtractionException {
for (StreamingService service : ServiceList.all()) {
if (service.getServiceId() == serviceId) {
return service;
}
}
throw new ExtractionException("There's no service with the id = \"" + serviceId + "\"");
public static StreamingService getService(final int serviceId) throws ExtractionException {
return ServiceList.all().stream()
.filter(service -> service.getServiceId() == serviceId)
.findFirst()
.orElseThrow(() -> new ExtractionException(
"There's no service with the id = \"" + serviceId + "\""));
}
public static StreamingService getService(String serviceName) throws ExtractionException {
for (StreamingService service : ServiceList.all()) {
if (service.getServiceInfo().getName().equals(serviceName)) {
return service;
}
}
throw new ExtractionException("There's no service with the name = \"" + serviceName + "\"");
public static StreamingService getService(final String serviceName) throws ExtractionException {
return ServiceList.all().stream()
.filter(service -> service.getServiceInfo().getName().equals(serviceName))
.findFirst()
.orElseThrow(() -> new ExtractionException(
"There's no service with the name = \"" + serviceName + "\""));
}
public static StreamingService getServiceByUrl(String url) throws ExtractionException {
for (StreamingService service : ServiceList.all()) {
public static StreamingService getServiceByUrl(final String url) throws ExtractionException {
for (final StreamingService service : ServiceList.all()) {
if (service.getLinkTypeByUrl(url) != StreamingService.LinkType.NONE) {
return service;
}
@ -97,43 +93,25 @@ public class NewPipe {
throw new ExtractionException("No service can handle the url = \"" + url + "\"");
}
public static int getIdOfService(String serviceName) {
try {
//noinspection ConstantConditions
return getService(serviceName).getServiceId();
} catch (ExtractionException ignored) {
return -1;
}
}
public static String getNameOfService(int id) {
try {
//noinspection ConstantConditions
return getService(id).getServiceInfo().getName();
} catch (Exception e) {
System.err.println("Service id not known");
e.printStackTrace();
return "<unknown>";
}
}
/*//////////////////////////////////////////////////////////////////////////
// Localization
//////////////////////////////////////////////////////////////////////////*/
public static void setupLocalization(Localization preferredLocalization) {
setupLocalization(preferredLocalization, null);
public static void setupLocalization(final Localization thePreferredLocalization) {
setupLocalization(thePreferredLocalization, null);
}
public static void setupLocalization(Localization preferredLocalization, @Nullable ContentCountry preferredContentCountry) {
NewPipe.preferredLocalization = preferredLocalization;
public static void setupLocalization(
final Localization thePreferredLocalization,
@Nullable final ContentCountry thePreferredContentCountry) {
NewPipe.preferredLocalization = thePreferredLocalization;
if (preferredContentCountry != null) {
NewPipe.preferredContentCountry = preferredContentCountry;
if (thePreferredContentCountry != null) {
NewPipe.preferredContentCountry = thePreferredContentCountry;
} else {
NewPipe.preferredContentCountry = preferredLocalization.getCountryCode().isEmpty()
NewPipe.preferredContentCountry = thePreferredLocalization.getCountryCode().isEmpty()
? ContentCountry.DEFAULT
: new ContentCountry(preferredLocalization.getCountryCode());
: new ContentCountry(thePreferredLocalization.getCountryCode());
}
}
@ -142,7 +120,7 @@ public class NewPipe {
return preferredLocalization == null ? Localization.DEFAULT : preferredLocalization;
}
public static void setPreferredLocalization(Localization preferredLocalization) {
public static void setPreferredLocalization(final Localization preferredLocalization) {
NewPipe.preferredLocalization = preferredLocalization;
}
@ -151,7 +129,7 @@ public class NewPipe {
return preferredContentCountry == null ? ContentCountry.DEFAULT : preferredContentCountry;
}
public static void setPreferredContentCountry(ContentCountry preferredContentCountry) {
public static void setPreferredContentCountry(final ContentCountry preferredContentCountry) {
NewPipe.preferredContentCountry = preferredContentCountry;
}
}

View File

@ -0,0 +1,85 @@
package org.schabi.newpipe.extractor;
import java.io.Serializable;
import java.util.List;
import java.util.Map;
import javax.annotation.Nullable;
import static org.schabi.newpipe.extractor.utils.Utils.isNullOrEmpty;
public class Page implements Serializable {
private final String url;
private final String id;
private final List<String> ids;
private final Map<String, String> cookies;
@Nullable
private final byte[] body;
public Page(final String url,
final String id,
final List<String> ids,
final Map<String, String> cookies,
@Nullable final byte[] body) {
this.url = url;
this.id = id;
this.ids = ids;
this.cookies = cookies;
this.body = body;
}
public Page(final String url) {
this(url, null, null, null, null);
}
public Page(final String url, final String id) {
this(url, id, null, null, null);
}
public Page(final String url, final String id, final byte[] body) {
this(url, id, null, null, body);
}
public Page(final String url, final byte[] body) {
this(url, null, null, null, body);
}
public Page(final String url, final Map<String, String> cookies) {
this(url, null, null, cookies, null);
}
public Page(final List<String> ids) {
this(null, null, ids, null, null);
}
public Page(final List<String> ids, final Map<String, String> cookies) {
this(null, null, ids, cookies, null);
}
public String getUrl() {
return url;
}
public String getId() {
return id;
}
public List<String> getIds() {
return ids;
}
public Map<String, String> getCookies() {
return cookies;
}
public static boolean isValid(final Page page) {
return page != null && (!isNullOrEmpty(page.getUrl())
|| !isNullOrEmpty(page.getIds()));
}
@Nullable
public byte[] getBody() {
return body;
}
}

View File

@ -1,56 +1,52 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.services.bandcamp.BandcampService;
import org.schabi.newpipe.extractor.services.media_ccc.MediaCCCService;
import org.schabi.newpipe.extractor.services.peertube.PeertubeService;
import org.schabi.newpipe.extractor.services.soundcloud.SoundcloudService;
import org.schabi.newpipe.extractor.services.youtube.YoutubeService;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
/*
* Copyright (C) Christian Schabesberger 2018 <chris.schabesberger@mailbox.org>
* ServiceList.java is part of NewPipe.
* Copyright (C) 2018 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ServiceList.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
/**
* A list of supported services.
*/
@SuppressWarnings({"ConstantName", "InnerAssignment"}) // keep unusual names and inner assignments
public final class ServiceList {
private ServiceList() {
//no instance
// no instance
}
public static final YoutubeService YouTube;
public static final SoundcloudService SoundCloud;
public static final MediaCCCService MediaCCC;
public static final PeertubeService PeerTube;
public static final YoutubeService YouTube = new YoutubeService(0);
public static final SoundcloudService SoundCloud = new SoundcloudService(1);
public static final MediaCCCService MediaCCC = new MediaCCCService(2);
public static final PeertubeService PeerTube = new PeertubeService(3);
public static final BandcampService Bandcamp = new BandcampService(4);
/**
* When creating a new service, put this service in the end of this list,
* and give it the next free id.
*/
private static final List<StreamingService> SERVICES = Collections.unmodifiableList(
Arrays.asList(
YouTube = new YoutubeService(0),
SoundCloud = new SoundcloudService(1),
MediaCCC = new MediaCCCService(2),
PeerTube = new PeertubeService(3)
));
private static final List<StreamingService> SERVICES = List.of(
YouTube, SoundCloud, MediaCCC, PeerTube, Bandcamp);
/**
* Get all the supported services.

View File

@ -1,12 +1,18 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.channel.ChannelExtractor;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import org.schabi.newpipe.extractor.comments.CommentsExtractor;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.feed.FeedExtractor;
import org.schabi.newpipe.extractor.kiosk.KioskList;
import org.schabi.newpipe.extractor.linkhandler.*;
import org.schabi.newpipe.extractor.linkhandler.LinkHandler;
import org.schabi.newpipe.extractor.linkhandler.LinkHandlerFactory;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandler;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandlerFactory;
import org.schabi.newpipe.extractor.localization.ContentCountry;
import org.schabi.newpipe.extractor.localization.Localization;
import org.schabi.newpipe.extractor.localization.TimeAgoParser;
@ -16,27 +22,28 @@ import org.schabi.newpipe.extractor.search.SearchExtractor;
import org.schabi.newpipe.extractor.stream.StreamExtractor;
import org.schabi.newpipe.extractor.subscription.SubscriptionExtractor;
import org.schabi.newpipe.extractor.suggestion.SuggestionExtractor;
import org.schabi.newpipe.extractor.utils.Utils;
import javax.annotation.Nullable;
import java.util.Collections;
import java.util.List;
/*
* Copyright (C) Christian Schabesberger 2018 <chris.schabesberger@mailbox.org>
* StreamingService.java is part of NewPipe.
* Copyright (C) 2018 Christian Schabesberger <chris.schabesberger@mailbox.org>
* StreamingService.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public abstract class StreamingService {
@ -54,7 +61,7 @@ public abstract class StreamingService {
* @param name the name of the service
* @param mediaCapabilities the type of media this service can handle
*/
public ServiceInfo(String name, List<MediaCapability> mediaCapabilities) {
public ServiceInfo(final String name, final List<MediaCapability> mediaCapabilities) {
this.name = name;
this.mediaCapabilities = Collections.unmodifiableList(mediaCapabilities);
}
@ -73,8 +80,8 @@ public abstract class StreamingService {
}
/**
* LinkType will be used to determine which type of URL you are handling, and therefore which part
* of NewPipe should handle a certain URL.
* LinkType will be used to determine which type of URL you are handling, and therefore which
* part of NewPipe should handle a certain URL.
*/
public enum LinkType {
NONE,
@ -89,14 +96,15 @@ public abstract class StreamingService {
/**
* Creates a new Streaming service.
* If you Implement one do not set id within your implementation of this extractor, instead
* set the id when you put the extractor into
* <a href="https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/ServiceList.html">ServiceList</a>.
* set the id when you put the extractor into {@link ServiceList}
* All other parameters can be set directly from the overriding constructor.
* @param id the number of the service to identify him within the NewPipe frontend
* @param name the name of the service
* @param capabilities the type of media this service can handle
*/
public StreamingService(int id, String name, List<ServiceInfo.MediaCapability> capabilities) {
public StreamingService(final int id,
final String name,
final List<ServiceInfo.MediaCapability> capabilities) {
this.serviceId = id;
this.serviceInfo = new ServiceInfo(name, capabilities);
}
@ -133,6 +141,14 @@ public abstract class StreamingService {
*/
public abstract ListLinkHandlerFactory getChannelLHFactory();
/**
* Must return a new instance of an implementation of ListLinkHandlerFactory for channel tabs.
* If support for channel tabs is not given null must be returned.
*
* @return an instance of a ListLinkHandlerFactory for channels or null
*/
public abstract ListLinkHandlerFactory getChannelTabLHFactory();
/**
* Must return a new instance of an implementation of ListLinkHandlerFactory for playlists.
* If support for playlists is not given null must be returned.
@ -171,22 +187,21 @@ public abstract class StreamingService {
public abstract SubscriptionExtractor getSubscriptionExtractor();
/**
* This method decides which strategy will be chosen to fetch the feed. In YouTube, for example, a separate feed
* exists which is lightweight and made specifically to be used like this.
* This method decides which strategy will be chosen to fetch the feed. In YouTube, for example,
* a separate feed exists which is lightweight and made specifically to be used like this.
* <p>
* In services which there's no other way to retrieve them, null should be returned.
*
* @return a {@link FeedExtractor} instance or null.
*/
@Nullable
public FeedExtractor getFeedExtractor(String url) throws ExtractionException {
public FeedExtractor getFeedExtractor(final String url) throws ExtractionException {
return null;
}
/**
* Must create a new instance of a KioskList implementation.
* @return a new KioskList instance
* @throws ExtractionException
*/
public abstract KioskList getKioskList() throws ExtractionException;
@ -194,49 +209,61 @@ public abstract class StreamingService {
* Must create a new instance of a ChannelExtractor implementation.
* @param linkHandler is pointing to the channel which should be handled by this new instance.
* @return a new ChannelExtractor
* @throws ExtractionException
*/
public abstract ChannelExtractor getChannelExtractor(ListLinkHandler linkHandler) throws ExtractionException;
public abstract ChannelExtractor getChannelExtractor(ListLinkHandler linkHandler)
throws ExtractionException;
/**
* Must create a new instance of a ChannelTabExtractor implementation.
*
* @param linkHandler is pointing to the channel which should be handled by this new instance.
* @return a new ChannelTabExtractor
*/
public abstract ChannelTabExtractor getChannelTabExtractor(ListLinkHandler linkHandler)
throws ExtractionException;
/**
* Must crete a new instance of a PlaylistExtractor implementation.
* @param linkHandler is pointing to the playlist which should be handled by this new instance.
* @return a new PlaylistExtractor
* @throws ExtractionException
*/
public abstract PlaylistExtractor getPlaylistExtractor(ListLinkHandler linkHandler) throws ExtractionException;
public abstract PlaylistExtractor getPlaylistExtractor(ListLinkHandler linkHandler)
throws ExtractionException;
/**
* Must create a new instance of a StreamExtractor implementation.
* @param linkHandler is pointing to the stream which should be handled by this new instance.
* @return a new StreamExtractor
* @throws ExtractionException
*/
public abstract StreamExtractor getStreamExtractor(LinkHandler linkHandler) throws ExtractionException;
public abstract StreamExtractor getStreamExtractor(LinkHandler linkHandler)
throws ExtractionException;
public abstract CommentsExtractor getCommentsExtractor(ListLinkHandler linkHandler) throws ExtractionException;
public abstract CommentsExtractor getCommentsExtractor(ListLinkHandler linkHandler)
throws ExtractionException;
/*//////////////////////////////////////////////////////////////////////////
// Extractors without link handler
//////////////////////////////////////////////////////////////////////////*/
public SearchExtractor getSearchExtractor(String query,
List<String> contentFilter,
String sortFilter) throws ExtractionException {
public SearchExtractor getSearchExtractor(final String query,
final List<String> contentFilter,
final String sortFilter) throws ExtractionException {
return getSearchExtractor(getSearchQHFactory()
.fromQuery(query, contentFilter, sortFilter));
}
public ChannelExtractor getChannelExtractor(String id,
List<String> contentFilter,
String sortFilter) throws ExtractionException {
public ChannelExtractor getChannelExtractor(final String id,
final List<String> contentFilter,
final String sortFilter)
throws ExtractionException {
return getChannelExtractor(getChannelLHFactory()
.fromQuery(id, contentFilter, sortFilter));
}
public PlaylistExtractor getPlaylistExtractor(String id,
List<String> contentFilter,
String sortFilter) throws ExtractionException {
public PlaylistExtractor getPlaylistExtractor(final String id,
final List<String> contentFilter,
final String sortFilter)
throws ExtractionException {
return getPlaylistExtractor(getPlaylistLHFactory()
.fromQuery(id, contentFilter, sortFilter));
}
@ -245,28 +272,42 @@ public abstract class StreamingService {
// Short extractors overloads
//////////////////////////////////////////////////////////////////////////*/
public SearchExtractor getSearchExtractor(String query) throws ExtractionException {
public SearchExtractor getSearchExtractor(final String query) throws ExtractionException {
return getSearchExtractor(getSearchQHFactory().fromQuery(query));
}
public ChannelExtractor getChannelExtractor(String url) throws ExtractionException {
public ChannelExtractor getChannelExtractor(final String url) throws ExtractionException {
return getChannelExtractor(getChannelLHFactory().fromUrl(url));
}
public PlaylistExtractor getPlaylistExtractor(String url) throws ExtractionException {
public ChannelTabExtractor getChannelTabExtractorFromId(final String id, final String tab)
throws ExtractionException {
return getChannelTabExtractor(getChannelTabLHFactory().fromQuery(
id, Collections.singletonList(tab), ""));
}
public ChannelTabExtractor getChannelTabExtractorFromIdAndBaseUrl(final String id,
final String tab,
final String baseUrl)
throws ExtractionException {
return getChannelTabExtractor(getChannelTabLHFactory().fromQuery(
id, Collections.singletonList(tab), "", baseUrl));
}
public PlaylistExtractor getPlaylistExtractor(final String url) throws ExtractionException {
return getPlaylistExtractor(getPlaylistLHFactory().fromUrl(url));
}
public StreamExtractor getStreamExtractor(String url) throws ExtractionException {
public StreamExtractor getStreamExtractor(final String url) throws ExtractionException {
return getStreamExtractor(getStreamLHFactory().fromUrl(url));
}
public CommentsExtractor getCommentsExtractor(String url) throws ExtractionException {
ListLinkHandlerFactory llhf = getCommentsLHFactory();
if (llhf == null) {
public CommentsExtractor getCommentsExtractor(final String url) throws ExtractionException {
final ListLinkHandlerFactory listLinkHandlerFactory = getCommentsLHFactory();
if (listLinkHandlerFactory == null) {
return null;
}
return getCommentsExtractor(llhf.fromUrl(url));
return getCommentsExtractor(listLinkHandlerFactory.fromUrl(url));
}
/*//////////////////////////////////////////////////////////////////////////
@ -277,18 +318,19 @@ public abstract class StreamingService {
* Figures out where the link is pointing to (a channel, a video, a playlist, etc.)
* @param url the url on which it should be decided of which link type it is
* @return the link type of url
* @throws ParsingException
*/
public final LinkType getLinkTypeByUrl(String url) throws ParsingException {
LinkHandlerFactory sH = getStreamLHFactory();
LinkHandlerFactory cH = getChannelLHFactory();
LinkHandlerFactory pH = getPlaylistLHFactory();
public final LinkType getLinkTypeByUrl(final String url) throws ParsingException {
final String polishedUrl = Utils.followGoogleRedirectIfNeeded(url);
if (sH != null && sH.acceptUrl(url)) {
final LinkHandlerFactory sH = getStreamLHFactory();
final LinkHandlerFactory cH = getChannelLHFactory();
final LinkHandlerFactory pH = getPlaylistLHFactory();
if (sH != null && sH.acceptUrl(polishedUrl)) {
return LinkType.STREAM;
} else if (cH != null && cH.acceptUrl(url)) {
} else if (cH != null && cH.acceptUrl(polishedUrl)) {
return LinkType.CHANNEL;
} else if (pH != null && pH.acceptUrl(url)) {
} else if (pH != null && pH.acceptUrl(polishedUrl)) {
return LinkType.PLAYLIST;
} else {
return LinkType.NONE;
@ -318,7 +360,8 @@ public abstract class StreamingService {
* the user prefer (using {@link NewPipe#getPreferredLocalization()}), then it will:
* <ul>
* <li>Check if the exactly localization is supported by this service.</li>
* <li>If not, check if a less specific localization is available, using only the language code.</li>
* <li>If not, check if a less specific localization is available, using only the language
* code.</li>
* <li>Fallback to the {@link Localization#DEFAULT default} localization.</li>
* </ul>
*/
@ -331,8 +374,9 @@ public abstract class StreamingService {
}
// Fallback to the first supported language that matches the preferred language
for (Localization supportedLanguage : getSupportedLocalizations()) {
if (supportedLanguage.getLanguageCode().equals(preferredLocalization.getLanguageCode())) {
for (final Localization supportedLanguage : getSupportedLocalizations()) {
if (supportedLanguage.getLanguageCode()
.equals(preferredLocalization.getLanguageCode())) {
return supportedLanguage;
}
}
@ -341,8 +385,8 @@ public abstract class StreamingService {
}
/**
* Returns the country that should be used to fetch content in this service. It will get which country
* the user prefer (using {@link NewPipe#getPreferredContentCountry()}), then it will:
* Returns the country that should be used to fetch content in this service. It will get which
* country the user prefer (using {@link NewPipe#getPreferredContentCountry()}), then it will:
* <ul>
* <li>Check if the country is supported by this service.</li>
* <li>If not, fallback to the {@link ContentCountry#DEFAULT default} country.</li>
@ -359,14 +403,15 @@ public abstract class StreamingService {
}
/**
* Get an instance of the time ago parser using the patterns related to the passed localization.<br>
* <br>
* Just like {@link #getLocalization()}, it will also try to fallback to a less specific localization if
* the exact one is not available/supported.
* Get an instance of the time ago parser using the patterns related to the passed localization.
* <br><br>
* Just like {@link #getLocalization()}, it will also try to fallback to a less specific
* localization if the exact one is not available/supported.
*
* @throws IllegalArgumentException if the localization is not supported (parsing patterns are not present).
* @throws IllegalArgumentException if the localization is not supported (parsing patterns are
* not present).
*/
public TimeAgoParser getTimeAgoParser(Localization localization) {
public TimeAgoParser getTimeAgoParser(final Localization localization) {
final TimeAgoParser targetParser = TimeAgoPatternsManager.getTimeAgoParserFor(localization);
if (targetParser != null) {
@ -374,15 +419,18 @@ public abstract class StreamingService {
}
if (!localization.getCountryCode().isEmpty()) {
final Localization lessSpecificLocalization = new Localization(localization.getLanguageCode());
final TimeAgoParser lessSpecificParser = TimeAgoPatternsManager.getTimeAgoParserFor(lessSpecificLocalization);
final Localization lessSpecificLocalization
= new Localization(localization.getLanguageCode());
final TimeAgoParser lessSpecificParser
= TimeAgoPatternsManager.getTimeAgoParserFor(lessSpecificLocalization);
if (lessSpecificParser != null) {
return lessSpecificParser;
}
}
throw new IllegalArgumentException("Localization is not supported (\"" + localization.toString() + "\")");
throw new IllegalArgumentException(
"Localization is not supported (\"" + localization + "\")");
}
}

View File

@ -1,40 +1,58 @@
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
/*
* Created by Christian Schabesberger on 25.07.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* ChannelExtractor.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ChannelExtractor.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <https://www.gnu.org/licenses/>.
*/
public abstract class ChannelExtractor extends ListExtractor<StreamInfoItem> {
package org.schabi.newpipe.extractor.channel;
public ChannelExtractor(StreamingService service, ListLinkHandler linkHandler) {
import org.schabi.newpipe.extractor.Extractor;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import javax.annotation.Nonnull;
import java.util.List;
public abstract class ChannelExtractor extends Extractor {
public static final long UNKNOWN_SUBSCRIBER_COUNT = -1;
protected ChannelExtractor(final StreamingService service, final ListLinkHandler linkHandler) {
super(service, linkHandler);
}
public abstract String getAvatarUrl() throws ParsingException;
public abstract String getBannerUrl() throws ParsingException;
@Nonnull
public abstract List<Image> getAvatars() throws ParsingException;
@Nonnull
public abstract List<Image> getBanners() throws ParsingException;
public abstract String getFeedUrl() throws ParsingException;
public abstract long getSubscriberCount() throws ParsingException;
public abstract String getDescription() throws ParsingException;
public abstract String getParentChannelName() throws ParsingException;
public abstract String getParentChannelUrl() throws ParsingException;
@Nonnull
public abstract List<Image> getParentChannelAvatars() throws ParsingException;
public abstract boolean isVerified() throws ParsingException;
@Nonnull
public abstract List<ListLinkHandler> getTabs() throws ParsingException;
@Nonnull
public List<String> getTags() throws ParsingException {
return List.of();
}
}

View File

@ -1,59 +1,60 @@
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.ListExtractor.InfoItemsPage;
import org.schabi.newpipe.extractor.ListInfo;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import org.schabi.newpipe.extractor.utils.ExtractorHelper;
import java.io.IOException;
/*
* Created by Christian Schabesberger on 31.07.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* ChannelInfo.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ChannelInfo.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <https://www.gnu.org/licenses/>.
*/
public class ChannelInfo extends ListInfo<StreamInfoItem> {
package org.schabi.newpipe.extractor.channel;
public ChannelInfo(int serviceId, String id, String url, String originalUrl, String name, ListLinkHandler listLinkHandler) {
super(serviceId, id, url, originalUrl, name, listLinkHandler.getContentFilters(), listLinkHandler.getSortFilter());
import org.schabi.newpipe.extractor.Info;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import java.io.IOException;
import java.util.List;
import javax.annotation.Nonnull;
public class ChannelInfo extends Info {
public ChannelInfo(final int serviceId,
final String id,
final String url,
final String originalUrl,
final String name) {
super(serviceId, id, url, originalUrl, name);
}
public static ChannelInfo getInfo(String url) throws IOException, ExtractionException {
public static ChannelInfo getInfo(final String url) throws IOException, ExtractionException {
return getInfo(NewPipe.getServiceByUrl(url), url);
}
public static ChannelInfo getInfo(StreamingService service, String url) throws IOException, ExtractionException {
ChannelExtractor extractor = service.getChannelExtractor(url);
public static ChannelInfo getInfo(final StreamingService service, final String url)
throws IOException, ExtractionException {
final ChannelExtractor extractor = service.getChannelExtractor(url);
extractor.fetchPage();
return getInfo(extractor);
}
public static InfoItemsPage<StreamInfoItem> getMoreItems(StreamingService service,
String url,
String pageUrl) throws IOException, ExtractionException {
return service.getChannelExtractor(url).getPage(pageUrl);
}
public static ChannelInfo getInfo(ChannelExtractor extractor) throws IOException, ExtractionException {
public static ChannelInfo getInfo(final ChannelExtractor extractor)
throws IOException, ExtractionException {
final int serviceId = extractor.getServiceId();
final String id = extractor.getId();
@ -61,70 +62,141 @@ public class ChannelInfo extends ListInfo<StreamInfoItem> {
final String originalUrl = extractor.getOriginalUrl();
final String name = extractor.getName();
final ChannelInfo info = new ChannelInfo(serviceId, id, url, originalUrl, name, extractor.getLinkHandler());
final ChannelInfo info = new ChannelInfo(serviceId, id, url, originalUrl, name);
try {
info.setAvatarUrl(extractor.getAvatarUrl());
} catch (Exception e) {
info.setAvatars(extractor.getAvatars());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setBannerUrl(extractor.getBannerUrl());
} catch (Exception e) {
info.setBanners(extractor.getBanners());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setFeedUrl(extractor.getFeedUrl());
} catch (Exception e) {
} catch (final Exception e) {
info.addError(e);
}
final InfoItemsPage<StreamInfoItem> itemsPage = ExtractorHelper.getItemsPageOrLogError(info, extractor);
info.setRelatedItems(itemsPage.getItems());
info.setNextPageUrl(itemsPage.getNextPageUrl());
try {
info.setSubscriberCount(extractor.getSubscriberCount());
} catch (Exception e) {
} catch (final Exception e) {
info.addError(e);
}
try {
info.setDescription(extractor.getDescription());
} catch (Exception e) {
} catch (final Exception e) {
info.addError(e);
}
try {
info.setParentChannelName(extractor.getParentChannelName());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setParentChannelUrl(extractor.getParentChannelUrl());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setParentChannelAvatars(extractor.getParentChannelAvatars());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setVerified(extractor.isVerified());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setTabs(extractor.getTabs());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setTags(extractor.getTags());
} catch (final Exception e) {
info.addError(e);
}
return info;
}
private String avatarUrl;
private String bannerUrl;
private String parentChannelName;
private String parentChannelUrl;
private String feedUrl;
private long subscriberCount = -1;
private String description;
private String[] donationLinks;
@Nonnull
private List<Image> avatars = List.of();
@Nonnull
private List<Image> banners = List.of();
@Nonnull
private List<Image> parentChannelAvatars = List.of();
private boolean verified;
private List<ListLinkHandler> tabs = List.of();
private List<String> tags = List.of();
public String getAvatarUrl() {
return avatarUrl;
public String getParentChannelName() {
return parentChannelName;
}
public void setAvatarUrl(String avatarUrl) {
this.avatarUrl = avatarUrl;
public void setParentChannelName(final String parentChannelName) {
this.parentChannelName = parentChannelName;
}
public String getBannerUrl() {
return bannerUrl;
public String getParentChannelUrl() {
return parentChannelUrl;
}
public void setBannerUrl(String bannerUrl) {
this.bannerUrl = bannerUrl;
public void setParentChannelUrl(final String parentChannelUrl) {
this.parentChannelUrl = parentChannelUrl;
}
@Nonnull
public List<Image> getParentChannelAvatars() {
return parentChannelAvatars;
}
public void setParentChannelAvatars(@Nonnull final List<Image> parentChannelAvatars) {
this.parentChannelAvatars = parentChannelAvatars;
}
@Nonnull
public List<Image> getAvatars() {
return avatars;
}
public void setAvatars(@Nonnull final List<Image> avatars) {
this.avatars = avatars;
}
@Nonnull
public List<Image> getBanners() {
return banners;
}
public void setBanners(@Nonnull final List<Image> banners) {
this.banners = banners;
}
public String getFeedUrl() {
return feedUrl;
}
public void setFeedUrl(String feedUrl) {
public void setFeedUrl(final String feedUrl) {
this.feedUrl = feedUrl;
}
@ -132,7 +204,7 @@ public class ChannelInfo extends ListInfo<StreamInfoItem> {
return subscriberCount;
}
public void setSubscriberCount(long subscriberCount) {
public void setSubscriberCount(final long subscriberCount) {
this.subscriberCount = subscriberCount;
}
@ -140,7 +212,7 @@ public class ChannelInfo extends ListInfo<StreamInfoItem> {
return description;
}
public void setDescription(String description) {
public void setDescription(final String description) {
this.description = description;
}
@ -148,7 +220,33 @@ public class ChannelInfo extends ListInfo<StreamInfoItem> {
return donationLinks;
}
public void setDonationLinks(String[] donationLinks) {
public void setDonationLinks(final String[] donationLinks) {
this.donationLinks = donationLinks;
}
public boolean isVerified() {
return verified;
}
public void setVerified(final boolean verified) {
this.verified = verified;
}
@Nonnull
public List<ListLinkHandler> getTabs() {
return tabs;
}
public void setTabs(@Nonnull final List<ListLinkHandler> tabs) {
this.tabs = tabs;
}
@Nonnull
public List<String> getTags() {
return tags;
}
public void setTags(@Nonnull final List<String> tags) {
this.tags = tags;
}
}

View File

@ -5,21 +5,21 @@ import org.schabi.newpipe.extractor.InfoItem;
/*
* Created by Christian Schabesberger on 11.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* ChannelInfoItem.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ChannelInfoItem.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public class ChannelInfoItem extends InfoItem {
@ -27,9 +27,9 @@ public class ChannelInfoItem extends InfoItem {
private String description;
private long subscriberCount = -1;
private long streamCount = -1;
private boolean verified = false;
public ChannelInfoItem(int serviceId, String url, String name) {
public ChannelInfoItem(final int serviceId, final String url, final String name) {
super(InfoType.CHANNEL, serviceId, url, name);
}
@ -37,7 +37,7 @@ public class ChannelInfoItem extends InfoItem {
return description;
}
public void setDescription(String description) {
public void setDescription(final String description) {
this.description = description;
}
@ -45,15 +45,23 @@ public class ChannelInfoItem extends InfoItem {
return subscriberCount;
}
public void setSubscriberCount(long subscriber_count) {
this.subscriberCount = subscriber_count;
public void setSubscriberCount(final long subscriberCount) {
this.subscriberCount = subscriberCount;
}
public long getStreamCount() {
return streamCount;
}
public void setStreamCount(long stream_count) {
this.streamCount = stream_count;
public void setStreamCount(final long streamCount) {
this.streamCount = streamCount;
}
public boolean isVerified() {
return verified;
}
public void setVerified(final boolean verified) {
this.verified = verified;
}
}

View File

@ -6,26 +6,29 @@ import org.schabi.newpipe.extractor.exceptions.ParsingException;
/*
* Created by Christian Schabesberger on 12.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* ChannelInfoItemExtractor.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ChannelInfoItemExtractor.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public interface ChannelInfoItemExtractor extends InfoItemExtractor {
String getDescription() throws ParsingException;
long getSubscriberCount() throws ParsingException;
long getStreamCount() throws ParsingException;
boolean isVerified() throws ParsingException;
}

View File

@ -1,64 +1,67 @@
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.InfoItemsCollector;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
/*
* Created by Christian Schabesberger on 12.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* ChannelInfoItemsCollector.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ChannelInfoItemsCollector.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <https://www.gnu.org/licenses/>.
*/
public class ChannelInfoItemsCollector extends InfoItemsCollector<ChannelInfoItem, ChannelInfoItemExtractor> {
public ChannelInfoItemsCollector(int serviceId) {
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.InfoItemsCollector;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
public final class ChannelInfoItemsCollector
extends InfoItemsCollector<ChannelInfoItem, ChannelInfoItemExtractor> {
public ChannelInfoItemsCollector(final int serviceId) {
super(serviceId);
}
@Override
public ChannelInfoItem extract(ChannelInfoItemExtractor extractor) throws ParsingException {
// important information
int serviceId = getServiceId();
String name = extractor.getName();
String url = extractor.getUrl();
ChannelInfoItem resultItem = new ChannelInfoItem(serviceId, url, name);
public ChannelInfoItem extract(final ChannelInfoItemExtractor extractor)
throws ParsingException {
final ChannelInfoItem resultItem = new ChannelInfoItem(
getServiceId(), extractor.getUrl(), extractor.getName());
// optional information
try {
resultItem.setSubscriberCount(extractor.getSubscriberCount());
} catch (Exception e) {
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setStreamCount(extractor.getStreamCount());
} catch (Exception e) {
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setThumbnailUrl(extractor.getThumbnailUrl());
} catch (Exception e) {
resultItem.setThumbnails(extractor.getThumbnails());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setDescription(extractor.getDescription());
} catch (Exception e) {
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setVerified(extractor.isVerified());
} catch (final Exception e) {
addError(e);
}
return resultItem;
}
}

View File

@ -0,0 +1,25 @@
package org.schabi.newpipe.extractor.channel.tabs;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import javax.annotation.Nonnull;
/**
* A {@link ListExtractor} of {@link InfoItem}s for tabs of channels.
*/
public abstract class ChannelTabExtractor extends ListExtractor<InfoItem> {
protected ChannelTabExtractor(@Nonnull final StreamingService service,
@Nonnull final ListLinkHandler linkHandler) {
super(service, linkHandler);
}
@Nonnull
@Override
public String getName() {
return getLinkHandler().getContentFilters().get(0);
}
}

View File

@ -0,0 +1,70 @@
package org.schabi.newpipe.extractor.channel.tabs;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.ListInfo;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.ChannelInfo;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.utils.ExtractorHelper;
import javax.annotation.Nonnull;
import java.io.IOException;
public class ChannelTabInfo extends ListInfo<InfoItem> {
public ChannelTabInfo(final int serviceId,
@Nonnull final ListLinkHandler linkHandler) {
super(serviceId, linkHandler, linkHandler.getContentFilters().get(0));
}
/**
* Get a {@link ChannelTabInfo} instance from the given service and tab handler.
*
* @param service streaming service
* @param linkHandler Channel tab handler (from {@link ChannelInfo})
* @return the extracted {@link ChannelTabInfo}
*/
@Nonnull
public static ChannelTabInfo getInfo(@Nonnull final StreamingService service,
@Nonnull final ListLinkHandler linkHandler)
throws ExtractionException, IOException {
final ChannelTabExtractor extractor = service.getChannelTabExtractor(linkHandler);
extractor.fetchPage();
return getInfo(extractor);
}
/**
* Get a {@link ChannelTabInfo} instance from a {@link ChannelTabExtractor}.
*
* @param extractor an extractor where {@code fetchPage()} was already got called on
* @return the extracted {@link ChannelTabInfo}
*/
@Nonnull
public static ChannelTabInfo getInfo(@Nonnull final ChannelTabExtractor extractor) {
final ChannelTabInfo info =
new ChannelTabInfo(extractor.getServiceId(), extractor.getLinkHandler());
try {
info.setOriginalUrl(extractor.getOriginalUrl());
} catch (final Exception e) {
info.addError(e);
}
final ListExtractor.InfoItemsPage<InfoItem> page
= ExtractorHelper.getItemsPageOrLogError(info, extractor);
info.setRelatedItems(page.getItems());
info.setNextPage(page.getNextPage());
return info;
}
public static ListExtractor.InfoItemsPage<InfoItem> getMoreItems(
@Nonnull final StreamingService service,
@Nonnull final ListLinkHandler linkHandler,
@Nonnull final Page page) throws ExtractionException, IOException {
return service.getChannelTabExtractor(linkHandler).getPage(page);
}
}

View File

@ -0,0 +1,17 @@
package org.schabi.newpipe.extractor.channel.tabs;
/**
* Constants of channel tabs supported by the extractor.
*/
public final class ChannelTabs {
public static final String VIDEOS = "videos";
public static final String TRACKS = "tracks";
public static final String SHORTS = "shorts";
public static final String LIVESTREAMS = "livestreams";
public static final String CHANNELS = "channels";
public static final String PLAYLISTS = "playlists";
public static final String ALBUMS = "albums";
private ChannelTabs() {
}
}

View File

@ -2,13 +2,36 @@ package org.schabi.newpipe.extractor.comments;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import javax.annotation.Nonnull;
public abstract class CommentsExtractor extends ListExtractor<CommentsInfoItem> {
public CommentsExtractor(StreamingService service, ListLinkHandler uiHandler) {
public CommentsExtractor(final StreamingService service, final ListLinkHandler uiHandler) {
super(service, uiHandler);
// TODO Auto-generated constructor stub
}
/**
* @apiNote Warning: This method is experimental and may get removed in a future release.
* @return <code>true</code> if the comments are disabled otherwise <code>false</code> (default)
*/
public boolean isCommentsDisabled() throws ExtractionException {
return false;
}
/**
* @return the total number of comments
*/
public int getCommentsCount() throws ExtractionException {
return -1;
}
@Nonnull
@Override
public String getName() throws ParsingException {
return "Comments";
}
}

View File

@ -3,6 +3,7 @@ package org.schabi.newpipe.extractor.comments;
import org.schabi.newpipe.extractor.ListExtractor.InfoItemsPage;
import org.schabi.newpipe.extractor.ListInfo;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
@ -10,62 +11,116 @@ import org.schabi.newpipe.extractor.utils.ExtractorHelper;
import java.io.IOException;
public class CommentsInfo extends ListInfo<CommentsInfoItem> {
public final class CommentsInfo extends ListInfo<CommentsInfoItem> {
private CommentsInfo(int serviceId, ListLinkHandler listUrlIdHandler, String name) {
private CommentsInfo(
final int serviceId,
final ListLinkHandler listUrlIdHandler,
final String name) {
super(serviceId, listUrlIdHandler, name);
}
public static CommentsInfo getInfo(String url) throws IOException, ExtractionException {
public static CommentsInfo getInfo(final String url) throws IOException, ExtractionException {
return getInfo(NewPipe.getServiceByUrl(url), url);
}
public static CommentsInfo getInfo(StreamingService serviceByUrl, String url) throws ExtractionException, IOException {
return getInfo(serviceByUrl.getCommentsExtractor(url));
public static CommentsInfo getInfo(final StreamingService service, final String url)
throws ExtractionException, IOException {
return getInfo(service.getCommentsExtractor(url));
}
private static CommentsInfo getInfo(CommentsExtractor commentsExtractor) throws IOException, ExtractionException {
public static CommentsInfo getInfo(final CommentsExtractor commentsExtractor)
throws IOException, ExtractionException {
// for services which do not have a comments extractor
if (null == commentsExtractor) {
if (commentsExtractor == null) {
return null;
}
commentsExtractor.fetchPage();
String name = commentsExtractor.getName();
int serviceId = commentsExtractor.getServiceId();
ListLinkHandler listUrlIdHandler = commentsExtractor.getLinkHandler();
CommentsInfo commentsInfo = new CommentsInfo(serviceId, listUrlIdHandler, name);
final String name = commentsExtractor.getName();
final int serviceId = commentsExtractor.getServiceId();
final ListLinkHandler listUrlIdHandler = commentsExtractor.getLinkHandler();
final CommentsInfo commentsInfo = new CommentsInfo(serviceId, listUrlIdHandler, name);
commentsInfo.setCommentsExtractor(commentsExtractor);
InfoItemsPage<CommentsInfoItem> initialCommentsPage = ExtractorHelper.getItemsPageOrLogError(commentsInfo,
commentsExtractor);
final InfoItemsPage<CommentsInfoItem> initialCommentsPage =
ExtractorHelper.getItemsPageOrLogError(commentsInfo, commentsExtractor);
commentsInfo.setCommentsDisabled(commentsExtractor.isCommentsDisabled());
commentsInfo.setRelatedItems(initialCommentsPage.getItems());
commentsInfo.setNextPageUrl(initialCommentsPage.getNextPageUrl());
try {
commentsInfo.setCommentsCount(commentsExtractor.getCommentsCount());
} catch (final Exception e) {
commentsInfo.addError(e);
}
commentsInfo.setNextPage(initialCommentsPage.getNextPage());
return commentsInfo;
}
public static InfoItemsPage<CommentsInfoItem> getMoreItems(CommentsInfo commentsInfo, String pageUrl)
throws ExtractionException, IOException {
return getMoreItems(NewPipe.getService(commentsInfo.getServiceId()), commentsInfo, pageUrl);
public static InfoItemsPage<CommentsInfoItem> getMoreItems(
final CommentsInfo commentsInfo,
final Page page) throws ExtractionException, IOException {
return getMoreItems(NewPipe.getService(commentsInfo.getServiceId()), commentsInfo.getUrl(),
page);
}
public static InfoItemsPage<CommentsInfoItem> getMoreItems(StreamingService service, CommentsInfo commentsInfo,
String pageUrl) throws IOException, ExtractionException {
if (null == commentsInfo.getCommentsExtractor()) {
commentsInfo.setCommentsExtractor(service.getCommentsExtractor(commentsInfo.getUrl()));
commentsInfo.getCommentsExtractor().fetchPage();
}
return commentsInfo.getCommentsExtractor().getPage(pageUrl);
public static InfoItemsPage<CommentsInfoItem> getMoreItems(
final StreamingService service,
final CommentsInfo commentsInfo,
final Page page) throws IOException, ExtractionException {
return getMoreItems(service, commentsInfo.getUrl(), page);
}
public static InfoItemsPage<CommentsInfoItem> getMoreItems(
final StreamingService service,
final String url,
final Page page) throws IOException, ExtractionException {
return service.getCommentsExtractor(url).getPage(page);
}
private transient CommentsExtractor commentsExtractor;
private boolean commentsDisabled = false;
private int commentsCount;
public CommentsExtractor getCommentsExtractor() {
return commentsExtractor;
}
public void setCommentsExtractor(CommentsExtractor commentsExtractor) {
public void setCommentsExtractor(final CommentsExtractor commentsExtractor) {
this.commentsExtractor = commentsExtractor;
}
/**
* @return {@code true} if the comments are disabled otherwise {@code false} (default)
* @see CommentsExtractor#isCommentsDisabled()
*/
public boolean isCommentsDisabled() {
return commentsDisabled;
}
/**
* @param commentsDisabled {@code true} if the comments are disabled otherwise {@code false}
*/
public void setCommentsDisabled(final boolean commentsDisabled) {
this.commentsDisabled = commentsDisabled;
}
/**
* Returns the total number of comments.
*
* @return the total number of comments
*/
public int getCommentsCount() {
return commentsCount;
}
/**
* Sets the total number of comments.
*
* @param commentsCount the commentsCount to set.
*/
public void setCommentsCount(final int commentsCount) {
this.commentsCount = commentsCount;
}
}

View File

@ -1,22 +1,45 @@
package org.schabi.newpipe.extractor.comments;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.localization.DateWrapper;
import org.schabi.newpipe.extractor.stream.Description;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.List;
public class CommentsInfoItem extends InfoItem {
private String commentId;
private String commentText;
private String authorName;
private String authorThumbnail;
private String authorEndpoint;
private String textualPublishedTime;
@Nullable private DateWrapper publishedTime;
@Nonnull
private Description commentText = Description.EMPTY_DESCRIPTION;
private String uploaderName;
@Nonnull
private List<Image> uploaderAvatars = List.of();
private String uploaderUrl;
private boolean uploaderVerified;
private String textualUploadDate;
@Nullable
private DateWrapper uploadDate;
private int likeCount;
private String textualLikeCount;
private boolean heartedByUploader;
private boolean pinned;
private int streamPosition;
private int replyCount;
@Nullable
private Page replies;
private boolean isChannelOwner;
private boolean creatorReply;
public CommentsInfoItem(int serviceId, String url, String name) {
public static final int NO_LIKE_COUNT = -1;
public static final int NO_STREAM_POSITION = -1;
public static final int UNKNOWN_REPLY_COUNT = -1;
public CommentsInfoItem(final int serviceId, final String url, final String name) {
super(InfoType.COMMENT, serviceId, url, name);
}
@ -24,64 +47,151 @@ public class CommentsInfoItem extends InfoItem {
return commentId;
}
public void setCommentId(String commentId) {
public void setCommentId(final String commentId) {
this.commentId = commentId;
}
public String getCommentText() {
@Nonnull
public Description getCommentText() {
return commentText;
}
public void setCommentText(String commentText) {
public void setCommentText(@Nonnull final Description commentText) {
this.commentText = commentText;
}
public String getAuthorName() {
return authorName;
public String getUploaderName() {
return uploaderName;
}
public void setAuthorName(String authorName) {
this.authorName = authorName;
public void setUploaderName(final String uploaderName) {
this.uploaderName = uploaderName;
}
public String getAuthorThumbnail() {
return authorThumbnail;
@Nonnull
public List<Image> getUploaderAvatars() {
return uploaderAvatars;
}
public void setAuthorThumbnail(String authorThumbnail) {
this.authorThumbnail = authorThumbnail;
public void setUploaderAvatars(@Nonnull final List<Image> uploaderAvatars) {
this.uploaderAvatars = uploaderAvatars;
}
public String getAuthorEndpoint() {
return authorEndpoint;
public String getUploaderUrl() {
return uploaderUrl;
}
public void setAuthorEndpoint(String authorEndpoint) {
this.authorEndpoint = authorEndpoint;
public void setUploaderUrl(final String uploaderUrl) {
this.uploaderUrl = uploaderUrl;
}
public String getTextualPublishedTime() {
return textualPublishedTime;
public String getTextualUploadDate() {
return textualUploadDate;
}
public void setTextualPublishedTime(String textualPublishedTime) {
this.textualPublishedTime = textualPublishedTime;
public void setTextualUploadDate(final String textualUploadDate) {
this.textualUploadDate = textualUploadDate;
}
@Nullable
public DateWrapper getPublishedTime() {
return publishedTime;
public DateWrapper getUploadDate() {
return uploadDate;
}
public void setPublishedTime(@Nullable DateWrapper publishedTime) {
this.publishedTime = publishedTime;
public void setUploadDate(@Nullable final DateWrapper uploadDate) {
this.uploadDate = uploadDate;
}
/**
* @return the comment's like count or {@link CommentsInfoItem#NO_LIKE_COUNT} if it is
* unavailable
*/
public int getLikeCount() {
return likeCount;
}
public void setLikeCount(int likeCount) {
public void setLikeCount(final int likeCount) {
this.likeCount = likeCount;
}
public String getTextualLikeCount() {
return textualLikeCount;
}
public void setTextualLikeCount(final String textualLikeCount) {
this.textualLikeCount = textualLikeCount;
}
public void setHeartedByUploader(final boolean isHeartedByUploader) {
this.heartedByUploader = isHeartedByUploader;
}
public boolean isHeartedByUploader() {
return this.heartedByUploader;
}
public boolean isPinned() {
return pinned;
}
public void setPinned(final boolean pinned) {
this.pinned = pinned;
}
public void setUploaderVerified(final boolean uploaderVerified) {
this.uploaderVerified = uploaderVerified;
}
public boolean isUploaderVerified() {
return uploaderVerified;
}
public void setStreamPosition(final int streamPosition) {
this.streamPosition = streamPosition;
}
/**
* Get the playback position of the stream to which this comment belongs.
* This is not supported by all services.
*
* @return the playback position in seconds or {@link #NO_STREAM_POSITION} if not available
*/
public int getStreamPosition() {
return streamPosition;
}
public void setReplyCount(final int replyCount) {
this.replyCount = replyCount;
}
public int getReplyCount() {
return replyCount;
}
public void setReplies(@Nullable final Page replies) {
this.replies = replies;
}
@Nullable
public Page getReplies() {
return this.replies;
}
public void setChannelOwner(final boolean channelOwner) {
this.isChannelOwner = channelOwner;
}
public boolean isChannelOwner() {
return isChannelOwner;
}
public void setCreatorReply(final boolean creatorReply) {
this.creatorReply = creatorReply;
}
public boolean hasCreatorReply() {
return creatorReply;
}
}

View File

@ -1,19 +1,152 @@
package org.schabi.newpipe.extractor.comments;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.InfoItemExtractor;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.localization.DateWrapper;
import org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeCommentsInfoItemExtractor;
import org.schabi.newpipe.extractor.stream.Description;
import org.schabi.newpipe.extractor.stream.StreamExtractor;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.List;
public interface CommentsInfoItemExtractor extends InfoItemExtractor {
String getCommentId() throws ParsingException;
String getCommentText() throws ParsingException;
String getAuthorName() throws ParsingException;
String getAuthorThumbnail() throws ParsingException;
String getAuthorEndpoint() throws ParsingException;
String getTextualPublishedTime() throws ParsingException;
/**
* Return the like count of the comment,
* or {@link CommentsInfoItem#NO_LIKE_COUNT} if it is unavailable.
*
* <br>
* <p>
* NOTE: Currently only implemented for YT {@link
* YoutubeCommentsInfoItemExtractor#getLikeCount()}
* with limitations (only approximate like count is returned)
*
* @return the comment's like count
* or {@link CommentsInfoItem#NO_LIKE_COUNT} if it is unavailable
* @see StreamExtractor#getLikeCount()
*/
default int getLikeCount() throws ParsingException {
return CommentsInfoItem.NO_LIKE_COUNT;
}
/**
* The unmodified like count given by the service
* <br>
* It may be language dependent
*/
default String getTextualLikeCount() throws ParsingException {
return "";
}
/**
* The text of the comment
*/
@Nonnull
default Description getCommentText() throws ParsingException {
return Description.EMPTY_DESCRIPTION;
}
/**
* The upload date given by the service, unmodified
*
* @see StreamExtractor#getTextualUploadDate()
*/
default String getTextualUploadDate() throws ParsingException {
return "";
}
/**
* The upload date wrapped with DateWrapper class
*
* @see StreamExtractor#getUploadDate()
*/
@Nullable
DateWrapper getPublishedTime() throws ParsingException;
int getLikeCount() throws ParsingException;
default DateWrapper getUploadDate() throws ParsingException {
return null;
}
default String getCommentId() throws ParsingException {
return "";
}
default String getUploaderUrl() throws ParsingException {
return "";
}
default String getUploaderName() throws ParsingException {
return "";
}
@Nonnull
default List<Image> getUploaderAvatars() throws ParsingException {
return List.of();
}
/**
* Whether the comment has been hearted by the uploader
*/
default boolean isHeartedByUploader() throws ParsingException {
return false;
}
/**
* Whether the comment is pinned
*/
default boolean isPinned() throws ParsingException {
return false;
}
/**
* Whether the uploader is verified by the service
*/
default boolean isUploaderVerified() throws ParsingException {
return false;
}
/**
* The playback position of the stream to which this comment belongs.
*
* @see CommentsInfoItem#getStreamPosition()
*/
default int getStreamPosition() throws ParsingException {
return CommentsInfoItem.NO_STREAM_POSITION;
}
/**
* The count of comment replies.
*
* @return the count of the replies
* or {@link CommentsInfoItem#UNKNOWN_REPLY_COUNT} if replies are not supported
*/
default int getReplyCount() throws ParsingException {
return CommentsInfoItem.UNKNOWN_REPLY_COUNT;
}
/**
* The continuation page which is used to get comment replies from.
*
* @return the continuation Page for the replies, or null if replies are not supported
*/
@Nullable
default Page getReplies() throws ParsingException {
return null;
}
/**
* Whether the comment was made by the channel owner.
*/
default boolean isChannelOwner() throws ParsingException {
return false;
}
/**
* Whether the comment was replied to by the creator.
*/
default boolean hasCreatorReply() throws ParsingException {
return false;
}
}

View File

@ -1,94 +1,133 @@
package org.schabi.newpipe.extractor.comments;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.InfoItemsCollector;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import java.util.ArrayList;
import java.util.List;
import java.util.Vector;
public class CommentsInfoItemsCollector extends InfoItemsCollector<CommentsInfoItem, CommentsInfoItemExtractor> {
public final class CommentsInfoItemsCollector
extends InfoItemsCollector<CommentsInfoItem, CommentsInfoItemExtractor> {
public CommentsInfoItemsCollector(int serviceId) {
public CommentsInfoItemsCollector(final int serviceId) {
super(serviceId);
}
@Override
public CommentsInfoItem extract(CommentsInfoItemExtractor extractor) throws ParsingException {
// important information
int serviceId = getServiceId();
String url = extractor.getUrl();
String name = extractor.getName();
CommentsInfoItem resultItem = new CommentsInfoItem(serviceId, url, name);
public CommentsInfoItem extract(final CommentsInfoItemExtractor extractor)
throws ParsingException {
final CommentsInfoItem resultItem = new CommentsInfoItem(
getServiceId(), extractor.getUrl(), extractor.getName());
// optional information
try {
resultItem.setCommentId(extractor.getCommentId());
} catch (Exception e) {
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setCommentText(extractor.getCommentText());
} catch (Exception e) {
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setAuthorName(extractor.getAuthorName());
} catch (Exception e) {
resultItem.setUploaderName(extractor.getUploaderName());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setAuthorThumbnail(extractor.getAuthorThumbnail());
} catch (Exception e) {
resultItem.setUploaderAvatars(extractor.getUploaderAvatars());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setAuthorEndpoint(extractor.getAuthorEndpoint());
} catch (Exception e) {
resultItem.setUploaderUrl(extractor.getUploaderUrl());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setTextualPublishedTime(extractor.getTextualPublishedTime());
} catch (Exception e) {
resultItem.setTextualUploadDate(extractor.getTextualUploadDate());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setPublishedTime(extractor.getPublishedTime());
} catch (Exception e) {
resultItem.setUploadDate(extractor.getUploadDate());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setLikeCount(extractor.getLikeCount());
} catch (Exception e) {
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setThumbnailUrl(extractor.getThumbnailUrl());
} catch (Exception e) {
resultItem.setTextualLikeCount(extractor.getTextualLikeCount());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setThumbnails(extractor.getThumbnails());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setHeartedByUploader(extractor.isHeartedByUploader());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setPinned(extractor.isPinned());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setStreamPosition(extractor.getStreamPosition());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setReplyCount(extractor.getReplyCount());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setReplies(extractor.getReplies());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setChannelOwner(extractor.isChannelOwner());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setCreatorReply(extractor.hasCreatorReply());
} catch (final Exception e) {
addError(e);
}
return resultItem;
}
@Override
public void commit(CommentsInfoItemExtractor extractor) {
public void commit(final CommentsInfoItemExtractor extractor) {
try {
addItem(extract(extractor));
} catch (Exception e) {
} catch (final Exception e) {
addError(e);
}
}
public List<CommentsInfoItem> getCommentsInfoItemList() {
List<CommentsInfoItem> siiList = new Vector<>();
for (InfoItem ii : super.getItems()) {
if (ii instanceof CommentsInfoItem) {
siiList.add((CommentsInfoItem) ii);
}
}
return siiList;
return new ArrayList<>(super.getItems());
}
}

View File

@ -7,6 +7,8 @@ import org.schabi.newpipe.extractor.localization.Localization;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.io.IOException;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
@ -19,13 +21,14 @@ public abstract class Downloader {
/**
* Do a GET request to get the resource that the url is pointing to.<br>
* <br>
* This method calls {@link #get(String, Map, Localization)} with the default preferred localization. It should only be
* used when the resource that will be fetched won't be affected by the localization.
* This method calls {@link #get(String, Map, Localization)} with the default preferred
* localization. It should only be used when the resource that will be fetched won't be affected
* by the localization.
*
* @param url the URL that is pointing to the wanted resource
* @return the result of the GET request
*/
public Response get(String url) throws IOException, ReCaptchaException {
public Response get(final String url) throws IOException, ReCaptchaException {
return get(url, null, NewPipe.getPreferredLocalization());
}
@ -38,7 +41,8 @@ public abstract class Downloader {
* @param localization the source of the value of the {@code Accept-Language} header
* @return the result of the GET request
*/
public Response get(String url, @Nullable Localization localization) throws IOException, ReCaptchaException {
public Response get(final String url, final Localization localization)
throws IOException, ReCaptchaException {
return get(url, null, localization);
}
@ -50,7 +54,8 @@ public abstract class Downloader {
* Any default headers <b>should</b> be overridden by these.
* @return the result of the GET request
*/
public Response get(String url, @Nullable Map<String, List<String>> headers) throws IOException, ReCaptchaException {
public Response get(final String url, @Nullable final Map<String, List<String>> headers)
throws IOException, ReCaptchaException {
return get(url, headers, NewPipe.getPreferredLocalization());
}
@ -65,7 +70,9 @@ public abstract class Downloader {
* @param localization the source of the value of the {@code Accept-Language} header
* @return the result of the GET request
*/
public Response get(String url, @Nullable Map<String, List<String>> headers, @Nullable Localization localization)
public Response get(final String url,
@Nullable final Map<String, List<String>> headers,
final Localization localization)
throws IOException, ReCaptchaException {
return execute(Request.newBuilder()
.get(url)
@ -80,7 +87,7 @@ public abstract class Downloader {
* @param url the URL that is pointing to the wanted resource
* @return the result of the HEAD request
*/
public Response head(String url) throws IOException, ReCaptchaException {
public Response head(final String url) throws IOException, ReCaptchaException {
return head(url, null);
}
@ -92,7 +99,7 @@ public abstract class Downloader {
* Any default headers <b>should</b> be overridden by these.
* @return the result of the HEAD request
*/
public Response head(String url, @Nullable Map<String, List<String>> headers)
public Response head(final String url, @Nullable final Map<String, List<String>> headers)
throws IOException, ReCaptchaException {
return execute(Request.newBuilder()
.head(url)
@ -107,9 +114,11 @@ public abstract class Downloader {
* @param headers a list of headers that will be used in the request.
* Any default headers <b>should</b> be overridden by these.
* @param dataToSend byte array that will be sent when doing the request.
* @return the result of the GET request
* @return the result of the POST request
*/
public Response post(String url, @Nullable Map<String, List<String>> headers, @Nullable byte[] dataToSend)
public Response post(final String url,
@Nullable final Map<String, List<String>> headers,
@Nullable final byte[] dataToSend)
throws IOException, ReCaptchaException {
return post(url, headers, dataToSend, NewPipe.getPreferredLocalization());
}
@ -124,9 +133,12 @@ public abstract class Downloader {
* Any default headers <b>should</b> be overridden by these.
* @param dataToSend byte array that will be sent when doing the request.
* @param localization the source of the value of the {@code Accept-Language} header
* @return the result of the GET request
* @return the result of the POST request
*/
public Response post(String url, @Nullable Map<String, List<String>> headers, @Nullable byte[] dataToSend, @Nullable Localization localization)
public Response post(final String url,
@Nullable final Map<String, List<String>> headers,
@Nullable final byte[] dataToSend,
final Localization localization)
throws IOException, ReCaptchaException {
return execute(Request.newBuilder()
.post(url, dataToSend)
@ -135,10 +147,100 @@ public abstract class Downloader {
.build());
}
/**
* Convenient method to send a POST request using the specified value of the
* {@code Content-Type} header with a given {@link Localization}.
*
* @param url the URL that is pointing to the wanted resource
* @param headers a list of headers that will be used in the request.
* Any default headers <b>should</b> be overridden by these.
* @param dataToSend byte array that will be sent when doing the request.
* @param localization the source of the value of the {@code Accept-Language} header
* @param contentType the mime type of the body sent, which will be set as the value of the
* {@code Content-Type} header
* @return the result of the POST request
* @see #post(String, Map, byte[], Localization)
*/
public Response postWithContentType(final String url,
@Nullable final Map<String, List<String>> headers,
@Nullable final byte[] dataToSend,
final Localization localization,
final String contentType)
throws IOException, ReCaptchaException {
final Map<String, List<String>> actualHeaders = new HashMap<>();
if (headers != null) {
actualHeaders.putAll(headers);
}
actualHeaders.put("Content-Type", Collections.singletonList(contentType));
return post(url, actualHeaders, dataToSend, localization);
}
/**
* Convenient method to send a POST request using the specified value of the
* {@code Content-Type} header.
*
* @param url the URL that is pointing to the wanted resource
* @param headers a list of headers that will be used in the request.
* Any default headers <b>should</b> be overridden by these.
* @param dataToSend byte array that will be sent when doing the request.
* @param contentType the mime type of the body sent, which will be set as the value of the
* {@code Content-Type} header
* @return the result of the POST request
* @see #post(String, Map, byte[], Localization)
*/
public Response postWithContentType(final String url,
@Nullable final Map<String, List<String>> headers,
@Nullable final byte[] dataToSend,
final String contentType)
throws IOException, ReCaptchaException {
return postWithContentType(url, headers, dataToSend, NewPipe.getPreferredLocalization(),
contentType);
}
/**
* Convenient method to send a POST request the JSON mime type as the value of the
* {@code Content-Type} header with a given {@link Localization}.
*
* @param url the URL that is pointing to the wanted resource
* @param headers a list of headers that will be used in the request.
* Any default headers <b>should</b> be overridden by these.
* @param dataToSend byte array that will be sent when doing the request.
* @param localization the source of the value of the {@code Accept-Language} header
* @return the result of the POST request
* @see #post(String, Map, byte[], Localization)
*/
public Response postWithContentTypeJson(final String url,
@Nullable final Map<String, List<String>> headers,
@Nullable final byte[] dataToSend,
final Localization localization)
throws IOException, ReCaptchaException {
return postWithContentType(url, headers, dataToSend, localization, "application/json");
}
/**
* Convenient method to send a POST request the JSON mime type as the value of the
* {@code Content-Type} header.
*
* @param url the URL that is pointing to the wanted resource
* @param headers a list of headers that will be used in the request.
* Any default headers <b>should</b> be overridden by these.
* @param dataToSend byte array that will be sent when doing the request.
* @return the result of the POST request
* @see #post(String, Map, byte[], Localization)
*/
public Response postWithContentTypeJson(final String url,
@Nullable final Map<String, List<String>> headers,
@Nullable final byte[] dataToSend)
throws IOException, ReCaptchaException {
return postWithContentTypeJson(url, headers, dataToSend,
NewPipe.getPreferredLocalization());
}
/**
* Do a request using the specified {@link Request} object.
*
* @return the result of the request
*/
public abstract Response execute(@Nonnull Request request) throws IOException, ReCaptchaException;
public abstract Response execute(@Nonnull Request request)
throws IOException, ReCaptchaException;
}

View File

@ -2,42 +2,53 @@ package org.schabi.newpipe.extractor.downloader;
import org.schabi.newpipe.extractor.localization.Localization;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.*;
/**
* An object that holds request information used when {@link Downloader#execute(Request) executing} a request.
* An object that holds request information used when {@link Downloader#execute(Request) executing}
* a request.
*/
public class Request {
private final String httpMethod;
private final String url;
private final Map<String, List<String>> headers;
@Nullable private final byte[] dataToSend;
@Nullable private final Localization localization;
@Nullable
private final byte[] dataToSend;
@Nullable
private final Localization localization;
public Request(String httpMethod, String url, Map<String, List<String>> headers, @Nullable byte[] dataToSend,
@Nullable Localization localization, boolean automaticLocalizationHeader) {
if (httpMethod == null) throw new IllegalArgumentException("Request's httpMethod is null");
if (url == null) throw new IllegalArgumentException("Request's url is null");
this.httpMethod = httpMethod;
this.url = url;
public Request(final String httpMethod,
final String url,
@Nullable final Map<String, List<String>> headers,
@Nullable final byte[] dataToSend,
@Nullable final Localization localization,
final boolean automaticLocalizationHeader) {
this.httpMethod = Objects.requireNonNull(httpMethod, "Request's httpMethod is null");
this.url = Objects.requireNonNull(url, "Request's url is null");
this.dataToSend = dataToSend;
this.localization = localization;
Map<String, List<String>> headersToSet = null;
if (headers == null) headers = Collections.emptyMap();
final Map<String, List<String>> actualHeaders = new LinkedHashMap<>();
if (headers != null) {
actualHeaders.putAll(headers);
}
if (automaticLocalizationHeader && localization != null) {
headersToSet = new LinkedHashMap<>(headersFromLocalization(localization));
headersToSet.putAll(headers);
actualHeaders.putAll(getHeadersFromLocalization(localization));
}
this.headers = Collections.unmodifiableMap(headersToSet == null ? headers : headersToSet);
this.headers = Collections.unmodifiableMap(actualHeaders);
}
private Request(Builder builder) {
private Request(final Builder builder) {
this(builder.httpMethod, builder.url, builder.headers, builder.dataToSend,
builder.localization, builder.automaticLocalizationHeader);
}
@ -80,7 +91,7 @@ public class Request {
* A localization object that should be used when executing a request.<br>
* <br>
* Usually the {@code Accept-Language} will be set to this value (a helper
* method to do this easily: {@link Request#headersFromLocalization(Localization)}).
* method to do this easily: {@link Request#getHeadersFromLocalization(Localization)}).
*/
@Nullable
public Localization localization() {
@ -94,7 +105,7 @@ public class Request {
public static final class Builder {
private String httpMethod;
private String url;
private Map<String, List<String>> headers = new LinkedHashMap<>();
private final Map<String, List<String>> headers = new LinkedHashMap<>();
private byte[] dataToSend;
private Localization localization;
private boolean automaticLocalizationHeader = true;
@ -105,30 +116,29 @@ public class Request {
/**
* A http method (i.e. {@code GET, POST, HEAD}).
*/
public Builder httpMethod(String httpMethod) {
this.httpMethod = httpMethod;
public Builder httpMethod(final String httpMethodToSet) {
this.httpMethod = httpMethodToSet;
return this;
}
/**
* The URL that is pointing to the wanted resource.
*/
public Builder url(String url) {
this.url = url;
public Builder url(final String urlToSet) {
this.url = urlToSet;
return this;
}
/**
* A list of headers that will be used in the request.<br>
* Any default headers that the implementation may have, <b>should</b> be overridden by these.
* Any default headers that the implementation may have, <b>should</b> be overridden by
* these.
*/
public Builder headers(@Nullable Map<String, List<String>> headers) {
if (headers == null) {
this.headers.clear();
return this;
}
public Builder headers(@Nullable final Map<String, List<String>> headersToSet) {
this.headers.clear();
this.headers.putAll(headers);
if (headersToSet != null) {
this.headers.putAll(headersToSet);
}
return this;
}
@ -139,8 +149,8 @@ public class Request {
* The implementation should make note of some recommended headers
* (for example, {@code Content-Length} in a post request).
*/
public Builder dataToSend(byte[] dataToSend) {
this.dataToSend = dataToSend;
public Builder dataToSend(final byte[] dataToSendToSet) {
this.dataToSend = dataToSendToSet;
return this;
}
@ -148,18 +158,18 @@ public class Request {
* A localization object that should be used when executing a request.<br>
* <br>
* Usually the {@code Accept-Language} will be set to this value (a helper
* method to do this easily: {@link Request#headersFromLocalization(Localization)}).
* method to do this easily: {@link Request#getHeadersFromLocalization(Localization)}).
*/
public Builder localization(Localization localization) {
this.localization = localization;
public Builder localization(final Localization localizationToSet) {
this.localization = localizationToSet;
return this;
}
/**
* If localization headers should automatically be included in the request.
*/
public Builder automaticLocalizationHeader(boolean automaticLocalizationHeader) {
this.automaticLocalizationHeader = automaticLocalizationHeader;
public Builder automaticLocalizationHeader(final boolean automaticLocalizationHeaderToSet) {
this.automaticLocalizationHeader = automaticLocalizationHeaderToSet;
return this;
}
@ -172,22 +182,22 @@ public class Request {
// Http Methods Utils
//////////////////////////////////////////////////////////////////////////*/
public Builder get(String url) {
public Builder get(final String urlToSet) {
this.httpMethod = "GET";
this.url = url;
this.url = urlToSet;
return this;
}
public Builder head(String url) {
public Builder head(final String urlToSet) {
this.httpMethod = "HEAD";
this.url = url;
this.url = urlToSet;
return this;
}
public Builder post(String url, @Nullable byte[] dataToSend) {
public Builder post(final String urlToSet, @Nullable final byte[] dataToSendToSet) {
this.httpMethod = "POST";
this.url = url;
this.dataToSend = dataToSend;
this.url = urlToSet;
this.dataToSend = dataToSendToSet;
return this;
}
@ -195,13 +205,13 @@ public class Request {
// Additional Headers Utils
//////////////////////////////////////////////////////////////////////////*/
public Builder setHeaders(String headerName, List<String> headerValueList) {
public Builder setHeaders(final String headerName, final List<String> headerValueList) {
this.headers.remove(headerName);
this.headers.put(headerName, headerValueList);
return this;
}
public Builder addHeaders(String headerName, List<String> headerValueList) {
public Builder addHeaders(final String headerName, final List<String> headerValueList) {
@Nullable List<String> currentHeaderValueList = this.headers.get(headerName);
if (currentHeaderValueList == null) {
currentHeaderValueList = new ArrayList<>();
@ -212,11 +222,11 @@ public class Request {
return this;
}
public Builder setHeader(String headerName, String headerValue) {
public Builder setHeader(final String headerName, final String headerValue) {
return setHeaders(headerName, Collections.singletonList(headerValue));
}
public Builder addHeader(String headerName, String headerValue) {
public Builder addHeader(final String headerName, final String headerValue) {
return addHeaders(headerName, Collections.singletonList(headerValue));
}
@ -228,17 +238,43 @@ public class Request {
@SuppressWarnings("WeakerAccess")
@Nonnull
public static Map<String, List<String>> headersFromLocalization(@Nullable Localization localization) {
if (localization == null) return Collections.emptyMap();
final Map<String, List<String>> headers = new LinkedHashMap<>();
if (!localization.getCountryCode().isEmpty()) {
headers.put("Accept-Language", Collections.singletonList(localization.getLocalizationCode() +
", " + localization.getLanguageCode() + ";q=0.9"));
} else {
headers.put("Accept-Language", Collections.singletonList(localization.getLanguageCode()));
public static Map<String, List<String>> getHeadersFromLocalization(
@Nullable final Localization localization) {
if (localization == null) {
return Collections.emptyMap();
}
return headers;
final String languageCode = localization.getLanguageCode();
final List<String> languageCodeList = Collections.singletonList(
localization.getCountryCode().isEmpty() ? languageCode
: localization.getLocalizationCode() + ", " + languageCode + ";q=0.9");
return Collections.singletonMap("Accept-Language", languageCodeList);
}
/*//////////////////////////////////////////////////////////////////////////
// Generated
//////////////////////////////////////////////////////////////////////////*/
@Override
public boolean equals(final Object o) {
if (this == o) {
return true;
}
if (o == null || getClass() != o.getClass()) {
return false;
}
final Request request = (Request) o;
return httpMethod.equals(request.httpMethod)
&& url.equals(request.url)
&& headers.equals(request.headers)
&& Arrays.equals(dataToSend, request.dataToSend)
&& Objects.equals(localization, request.localization);
}
@Override
public int hashCode() {
int result = Objects.hash(httpMethod, url, headers, localization);
result = 31 * result + Arrays.hashCode(dataToSend);
return result;
}
}

View File

@ -17,11 +17,14 @@ public class Response {
private final String latestUrl;
public Response(int responseCode, String responseMessage, Map<String, List<String>> responseHeaders,
@Nullable String responseBody, @Nullable String latestUrl) {
public Response(final int responseCode,
final String responseMessage,
@Nullable final Map<String, List<String>> responseHeaders,
@Nullable final String responseBody,
@Nullable final String latestUrl) {
this.responseCode = responseCode;
this.responseMessage = responseMessage;
this.responseHeaders = responseHeaders != null ? responseHeaders : Collections.<String, List<String>>emptyMap();
this.responseHeaders = responseHeaders == null ? Collections.emptyMap() : responseHeaders;
this.responseBody = responseBody == null ? "" : responseBody;
this.latestUrl = latestUrl;
@ -60,19 +63,18 @@ public class Response {
/**
* For easy access to some header value that (usually) don't repeat itself.
* <p>For getting all the values associated to the header, use {@link #responseHeaders()} (e.g. {@code Set-Cookie}).
* <p>For getting all the values associated to the header, use {@link #responseHeaders()} (e.g.
* {@code Set-Cookie}).
*
* @param name the name of the header
* @return the first value assigned to this header
*/
@Nullable
public String getHeader(String name) {
for (Map.Entry<String, List<String>> headerEntry : responseHeaders.entrySet()) {
public String getHeader(final String name) {
for (final Map.Entry<String, List<String>> headerEntry : responseHeaders.entrySet()) {
final String key = headerEntry.getKey();
if (key != null && key.equalsIgnoreCase(name)) {
if (headerEntry.getValue().size() > 0) {
return headerEntry.getValue().get(0);
}
if (key != null && key.equalsIgnoreCase(name) && !headerEntry.getValue().isEmpty()) {
return headerEntry.getValue().get(0);
}
}

View File

@ -0,0 +1,31 @@
package org.schabi.newpipe.extractor.exceptions;
public class AccountTerminatedException extends ContentNotAvailableException {
private Reason reason = Reason.UNKNOWN;
public AccountTerminatedException(final String message) {
super(message);
}
public AccountTerminatedException(final String message, final Reason reason) {
super(message);
this.reason = reason;
}
public AccountTerminatedException(final String message, final Throwable cause) {
super(message, cause);
}
/**
* The reason for the violation. There should also be more info in the exception's message.
*/
public Reason getReason() {
return reason;
}
public enum Reason {
UNKNOWN,
VIOLATION
}
}

View File

@ -0,0 +1,11 @@
package org.schabi.newpipe.extractor.exceptions;
public class AgeRestrictedContentException extends ContentNotAvailableException {
public AgeRestrictedContentException(final String message) {
super(message);
}
public AgeRestrictedContentException(final String message, final Throwable cause) {
super(message, cause);
}
}

View File

@ -1,11 +1,11 @@
package org.schabi.newpipe.extractor.exceptions;
public class ContentNotAvailableException extends ParsingException {
public ContentNotAvailableException(String message) {
public ContentNotAvailableException(final String message) {
super(message);
}
public ContentNotAvailableException(String message, Throwable cause) {
public ContentNotAvailableException(final String message, final Throwable cause) {
super(message, cause);
}
}

View File

@ -0,0 +1,11 @@
package org.schabi.newpipe.extractor.exceptions;
public class ContentNotSupportedException extends ParsingException {
public ContentNotSupportedException(final String message) {
super(message);
}
public ContentNotSupportedException(final String message, final Throwable cause) {
super(message, cause);
}
}

View File

@ -3,33 +3,33 @@ package org.schabi.newpipe.extractor.exceptions;
/*
* Created by Christian Schabesberger on 30.01.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* ExtractionException.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ExtractionException.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public class ExtractionException extends Exception {
public ExtractionException(String message) {
public ExtractionException(final String message) {
super(message);
}
public ExtractionException(Throwable cause) {
public ExtractionException(final Throwable cause) {
super(cause);
}
public ExtractionException(String message, Throwable cause) {
public ExtractionException(final String message, final Throwable cause) {
super(message, cause);
}
}
}

View File

@ -3,29 +3,29 @@ package org.schabi.newpipe.extractor.exceptions;
/*
* Created by Christian Schabesberger on 12.09.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* FoundAdException.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* FoundAdException.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public class FoundAdException extends ParsingException {
public FoundAdException(String message) {
public FoundAdException(final String message) {
super(message);
}
public FoundAdException(String message, Throwable cause) {
public FoundAdException(final String message, final Throwable cause) {
super(message, cause);
}
}

View File

@ -0,0 +1,11 @@
package org.schabi.newpipe.extractor.exceptions;
public class GeographicRestrictionException extends ContentNotAvailableException {
public GeographicRestrictionException(final String message) {
super(message);
}
public GeographicRestrictionException(final String message, final Throwable cause) {
super(message, cause);
}
}

View File

@ -0,0 +1,11 @@
package org.schabi.newpipe.extractor.exceptions;
public class PaidContentException extends ContentNotAvailableException {
public PaidContentException(final String message) {
super(message);
}
public PaidContentException(final String message, final Throwable cause) {
super(message, cause);
}
}

View File

@ -3,30 +3,30 @@ package org.schabi.newpipe.extractor.exceptions;
/*
* Created by Christian Schabesberger on 31.01.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* ParsingException.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ParsingException.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public class ParsingException extends ExtractionException {
public ParsingException(String message) {
public ParsingException(final String message) {
super(message);
}
public ParsingException(String message, Throwable cause) {
public ParsingException(final String message, final Throwable cause) {
super(message, cause);
}
}
}

View File

@ -0,0 +1,11 @@
package org.schabi.newpipe.extractor.exceptions;
public class PrivateContentException extends ContentNotAvailableException {
public PrivateContentException(final String message) {
super(message);
}
public PrivateContentException(final String message, final Throwable cause) {
super(message, cause);
}
}

View File

@ -3,27 +3,27 @@ package org.schabi.newpipe.extractor.exceptions;
/*
* Created by beneth <bmauduit@beneth.fr> on 07.12.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* ReCaptchaException.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ReCaptchaException.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public class ReCaptchaException extends ExtractionException {
private String url;
private final String url;
public ReCaptchaException(String message, String url) {
public ReCaptchaException(final String message, final String url) {
super(message);
this.url = url;
}

View File

@ -0,0 +1,11 @@
package org.schabi.newpipe.extractor.exceptions;
public class SoundCloudGoPlusContentException extends ContentNotAvailableException {
public SoundCloudGoPlusContentException() {
super("This track is a SoundCloud Go+ track");
}
public SoundCloudGoPlusContentException(final Throwable cause) {
super("This track is a SoundCloud Go+ track", cause);
}
}

View File

@ -0,0 +1,7 @@
package org.schabi.newpipe.extractor.exceptions;
public final class UnsupportedTabException extends UnsupportedOperationException {
public UnsupportedTabException(final String unsupportedTab) {
super("Unsupported tab " + unsupportedTab);
}
}

View File

@ -0,0 +1,11 @@
package org.schabi.newpipe.extractor.exceptions;
public class YoutubeMusicPremiumContentException extends ContentNotAvailableException {
public YoutubeMusicPremiumContentException() {
super("This video is a YouTube Music Premium video");
}
public YoutubeMusicPremiumContentException(final Throwable cause) {
super("This video is a YouTube Music Premium video", cause);
}
}

View File

@ -11,7 +11,7 @@ import org.schabi.newpipe.extractor.stream.StreamInfoItem;
* YouTube is an example of a service that has this alternative available.
*/
public abstract class FeedExtractor extends ListExtractor<StreamInfoItem> {
public FeedExtractor(StreamingService service, ListLinkHandler listLinkHandler) {
public FeedExtractor(final StreamingService service, final ListLinkHandler listLinkHandler) {
super(service, listLinkHandler);
}
}

View File

@ -13,26 +13,35 @@ import java.util.List;
public class FeedInfo extends ListInfo<StreamInfoItem> {
public FeedInfo(int serviceId, String id, String url, String originalUrl, String name, List<String> contentFilter, String sortFilter) {
public FeedInfo(final int serviceId,
final String id,
final String url,
final String originalUrl,
final String name,
final List<String> contentFilter,
final String sortFilter) {
super(serviceId, id, url, originalUrl, name, contentFilter, sortFilter);
}
public static FeedInfo getInfo(String url) throws IOException, ExtractionException {
public static FeedInfo getInfo(final String url) throws IOException, ExtractionException {
return getInfo(NewPipe.getServiceByUrl(url), url);
}
public static FeedInfo getInfo(StreamingService service, String url) throws IOException, ExtractionException {
public static FeedInfo getInfo(final StreamingService service, final String url)
throws IOException, ExtractionException {
final FeedExtractor extractor = service.getFeedExtractor(url);
if (extractor == null) {
throw new IllegalArgumentException("Service \"" + service.getServiceInfo().getName() + "\" doesn't support FeedExtractor.");
throw new IllegalArgumentException("Service \"" + service.getServiceInfo().getName()
+ "\" doesn't support FeedExtractor.");
}
extractor.fetchPage();
return getInfo(extractor);
}
public static FeedInfo getInfo(FeedExtractor extractor) throws IOException, ExtractionException {
public static FeedInfo getInfo(final FeedExtractor extractor)
throws IOException, ExtractionException {
extractor.fetchPage();
final int serviceId = extractor.getServiceId();
@ -43,9 +52,10 @@ public class FeedInfo extends ListInfo<StreamInfoItem> {
final FeedInfo info = new FeedInfo(serviceId, id, url, originalUrl, name, null, null);
final InfoItemsPage<StreamInfoItem> itemsPage = ExtractorHelper.getItemsPageOrLogError(info, extractor);
final InfoItemsPage<StreamInfoItem> itemsPage
= ExtractorHelper.getItemsPageOrLogError(info, extractor);
info.setRelatedItems(itemsPage.getItems());
info.setNextPageUrl(itemsPage.getNextPageUrl());
info.setNextPage(itemsPage.getNextPage());
return info;
}

View File

@ -3,21 +3,21 @@ package org.schabi.newpipe.extractor.kiosk;
/*
* Created by Christian Schabesberger on 12.08.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* KioskExtractor.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* KioskExtractor.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
import org.schabi.newpipe.extractor.InfoItem;
@ -31,9 +31,9 @@ import javax.annotation.Nonnull;
public abstract class KioskExtractor<T extends InfoItem> extends ListExtractor<T> {
private final String id;
public KioskExtractor(StreamingService streamingService,
ListLinkHandler linkHandler,
String kioskId) {
public KioskExtractor(final StreamingService streamingService,
final ListLinkHandler linkHandler,
final String kioskId) {
super(streamingService, linkHandler);
this.id = kioskId;
}
@ -45,12 +45,11 @@ public abstract class KioskExtractor<T extends InfoItem> extends ListExtractor<T
}
/**
* Id should be the name of the kiosk, tho Id is used for identifing it in the frontend,
* Id should be the name of the kiosk, tho Id is used for identifying it in the frontend,
* so id should be kept in english.
* In order to get the name of the kiosk in the desired language we have to
* crawl if from the website.
* @return the tranlsated version of id
* @throws ParsingException
* @return the translated version of id
*/
@Nonnull
@Override

View File

@ -3,57 +3,53 @@ package org.schabi.newpipe.extractor.kiosk;
/*
* Created by Christian Schabesberger on 12.08.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* KioskInfo.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* KioskInfo.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.ListInfo;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import org.schabi.newpipe.extractor.utils.ExtractorHelper;
import java.io.IOException;
public class KioskInfo extends ListInfo<StreamInfoItem> {
private KioskInfo(int serviceId, ListLinkHandler linkHandler, String name) throws ParsingException {
public final class KioskInfo extends ListInfo<StreamInfoItem> {
private KioskInfo(final int serviceId, final ListLinkHandler linkHandler, final String name) {
super(serviceId, linkHandler, name);
}
public static ListExtractor.InfoItemsPage<StreamInfoItem> getMoreItems(StreamingService service,
String url,
String pageUrl)
public static ListExtractor.InfoItemsPage<StreamInfoItem> getMoreItems(
final StreamingService service, final String url, final Page page)
throws IOException, ExtractionException {
KioskList kl = service.getKioskList();
KioskExtractor extractor = kl.getExtractorByUrl(url, pageUrl);
return extractor.getPage(pageUrl);
return service.getKioskList().getExtractorByUrl(url, page).getPage(page);
}
public static KioskInfo getInfo(String url) throws IOException, ExtractionException {
public static KioskInfo getInfo(final String url) throws IOException, ExtractionException {
return getInfo(NewPipe.getServiceByUrl(url), url);
}
public static KioskInfo getInfo(StreamingService service, String url) throws IOException, ExtractionException {
KioskList kl = service.getKioskList();
KioskExtractor extractor = kl.getExtractorByUrl(url, null);
public static KioskInfo getInfo(final StreamingService service, final String url)
throws IOException, ExtractionException {
final KioskExtractor extractor = service.getKioskList().getExtractorByUrl(url, null);
extractor.fetchPage();
return getInfo(extractor);
}
@ -63,15 +59,16 @@ public class KioskInfo extends ListInfo<StreamInfoItem> {
*
* @param extractor an extractor where fetchPage() was already got called on.
*/
public static KioskInfo getInfo(KioskExtractor extractor) throws ExtractionException {
public static KioskInfo getInfo(final KioskExtractor extractor) throws ExtractionException {
final KioskInfo info = new KioskInfo(extractor.getServiceId(),
extractor.getLinkHandler(),
extractor.getName());
final ListExtractor.InfoItemsPage<StreamInfoItem> itemsPage = ExtractorHelper.getItemsPageOrLogError(info, extractor);
final ListExtractor.InfoItemsPage<StreamInfoItem> itemsPage
= ExtractorHelper.getItemsPageOrLogError(info, extractor);
info.setRelatedItems(itemsPage.getItems());
info.setNextPageUrl(itemsPage.getNextPageUrl());
info.setNextPage(itemsPage.getNextPage());
return info;
}

View File

@ -1,24 +1,28 @@
package org.schabi.newpipe.extractor.kiosk;
import static org.schabi.newpipe.extractor.utils.Utils.isNullOrEmpty;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import org.schabi.newpipe.extractor.localization.ContentCountry;
import org.schabi.newpipe.extractor.localization.Localization;
import javax.annotation.Nullable;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import java.util.Set;
import javax.annotation.Nullable;
public class KioskList {
public interface KioskExtractorFactory {
KioskExtractor createNewKiosk(final StreamingService streamingService,
final String url,
final String kioskId)
KioskExtractor createNewKiosk(StreamingService streamingService,
String url,
String kioskId)
throws ExtractionException, IOException;
}
@ -31,8 +35,8 @@ public class KioskList {
@Nullable
private ContentCountry forcedContentCountry;
private class KioskEntry {
public KioskEntry(KioskExtractorFactory ef, ListLinkHandlerFactory h) {
private static class KioskEntry {
KioskEntry(final KioskExtractorFactory ef, final ListLinkHandlerFactory h) {
extractorFactory = ef;
handlerFactory = h;
}
@ -41,11 +45,13 @@ public class KioskList {
final ListLinkHandlerFactory handlerFactory;
}
public KioskList(StreamingService service) {
public KioskList(final StreamingService service) {
this.service = service;
}
public void addKioskEntry(KioskExtractorFactory extractorFactory, ListLinkHandlerFactory handlerFactory, String id)
public void addKioskEntry(final KioskExtractorFactory extractorFactory,
final ListLinkHandlerFactory handlerFactory,
final String id)
throws Exception {
if (kioskList.get(id) != null) {
throw new Exception("Kiosk with type " + id + " already exists.");
@ -53,29 +59,30 @@ public class KioskList {
kioskList.put(id, new KioskEntry(extractorFactory, handlerFactory));
}
public void setDefaultKiosk(String kioskType) {
public void setDefaultKiosk(final String kioskType) {
defaultKiosk = kioskType;
}
public KioskExtractor getDefaultKioskExtractor()
throws ExtractionException, IOException {
return getDefaultKioskExtractor("");
return getDefaultKioskExtractor(null);
}
public KioskExtractor getDefaultKioskExtractor(String nextPageUrl)
public KioskExtractor getDefaultKioskExtractor(final Page nextPage)
throws ExtractionException, IOException {
return getDefaultKioskExtractor(nextPageUrl, NewPipe.getPreferredLocalization());
return getDefaultKioskExtractor(nextPage, NewPipe.getPreferredLocalization());
}
public KioskExtractor getDefaultKioskExtractor(String nextPageUrl, Localization localization)
public KioskExtractor getDefaultKioskExtractor(final Page nextPage,
final Localization localization)
throws ExtractionException, IOException {
if (defaultKiosk != null && !defaultKiosk.equals("")) {
return getExtractorById(defaultKiosk, nextPageUrl, localization);
if (!isNullOrEmpty(defaultKiosk)) {
return getExtractorById(defaultKiosk, nextPage, localization);
} else {
if (!kioskList.isEmpty()) {
final String first = kioskList.keySet().stream().findAny().orElse(null);
if (first != null) {
// if not set get any entry
Object[] keySet = kioskList.keySet().toArray();
return getExtractorById(keySet[0].toString(), nextPageUrl, localization);
return getExtractorById(first, nextPage, localization);
} else {
return null;
}
@ -86,22 +93,28 @@ public class KioskList {
return defaultKiosk;
}
public KioskExtractor getExtractorById(String kioskId, String nextPageUrl)
public KioskExtractor getExtractorById(final String kioskId, final Page nextPage)
throws ExtractionException, IOException {
return getExtractorById(kioskId, nextPageUrl, NewPipe.getPreferredLocalization());
return getExtractorById(kioskId, nextPage, NewPipe.getPreferredLocalization());
}
public KioskExtractor getExtractorById(String kioskId, String nextPageUrl, Localization localization)
public KioskExtractor getExtractorById(final String kioskId,
final Page nextPage,
final Localization localization)
throws ExtractionException, IOException {
KioskEntry ke = kioskList.get(kioskId);
final KioskEntry ke = kioskList.get(kioskId);
if (ke == null) {
throw new ExtractionException("No kiosk found with the type: " + kioskId);
} else {
final KioskExtractor kioskExtractor = ke.extractorFactory.createNewKiosk(service,
ke.handlerFactory.fromId(kioskId).getUrl(), kioskId);
if (forcedLocalization != null) kioskExtractor.forceLocalization(forcedLocalization);
if (forcedContentCountry != null) kioskExtractor.forceContentCountry(forcedContentCountry);
if (forcedLocalization != null) {
kioskExtractor.forceLocalization(forcedLocalization);
}
if (forcedContentCountry != null) {
kioskExtractor.forceContentCountry(forcedContentCountry);
}
return kioskExtractor;
}
@ -111,31 +124,33 @@ public class KioskList {
return kioskList.keySet();
}
public KioskExtractor getExtractorByUrl(String url, String nextPageUrl)
public KioskExtractor getExtractorByUrl(final String url, final Page nextPage)
throws ExtractionException, IOException {
return getExtractorByUrl(url, nextPageUrl, NewPipe.getPreferredLocalization());
return getExtractorByUrl(url, nextPage, NewPipe.getPreferredLocalization());
}
public KioskExtractor getExtractorByUrl(String url, String nextPageUrl, Localization localization)
public KioskExtractor getExtractorByUrl(final String url,
final Page nextPage,
final Localization localization)
throws ExtractionException, IOException {
for (Map.Entry<String, KioskEntry> e : kioskList.entrySet()) {
KioskEntry ke = e.getValue();
for (final Map.Entry<String, KioskEntry> e : kioskList.entrySet()) {
final KioskEntry ke = e.getValue();
if (ke.handlerFactory.acceptUrl(url)) {
return getExtractorById(ke.handlerFactory.getId(url), nextPageUrl, localization);
return getExtractorById(ke.handlerFactory.getId(url), nextPage, localization);
}
}
throw new ExtractionException("Could not find a kiosk that fits to the url: " + url);
}
public ListLinkHandlerFactory getListLinkHandlerFactoryByType(String type) {
public ListLinkHandlerFactory getListLinkHandlerFactoryByType(final String type) {
return kioskList.get(type).handlerFactory;
}
public void forceLocalization(@Nullable Localization localization) {
public void forceLocalization(@Nullable final Localization localization) {
this.forcedLocalization = localization;
}
public void forceContentCountry(@Nullable ContentCountry contentCountry) {
public void forceContentCountry(@Nullable final ContentCountry contentCountry) {
this.forcedContentCountry = contentCountry;
}
}

View File

@ -10,13 +10,13 @@ public class LinkHandler implements Serializable {
protected final String url;
protected final String id;
public LinkHandler(String originalUrl, String url, String id) {
public LinkHandler(final String originalUrl, final String url, final String id) {
this.originalUrl = originalUrl;
this.url = url;
this.id = id;
}
public LinkHandler(LinkHandler handler) {
public LinkHandler(final LinkHandler handler) {
this(handler.originalUrl, handler.url, handler.id);
}

View File

@ -1,27 +1,28 @@
package org.schabi.newpipe.extractor.linkhandler;
import org.schabi.newpipe.extractor.exceptions.FoundAdException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.utils.Utils;
import java.util.Objects;
/*
* Created by Christian Schabesberger on 26.07.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* LinkHandlerFactory.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* LinkHandlerFactory.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public abstract class LinkHandlerFactory {
@ -30,11 +31,14 @@ public abstract class LinkHandlerFactory {
// To Override
///////////////////////////////////
public abstract String getId(String url) throws ParsingException;
public abstract String getUrl(String id) throws ParsingException;
public abstract boolean onAcceptUrl(final String url) throws ParsingException;
public abstract String getId(String url) throws ParsingException, UnsupportedOperationException;
public String getUrl(String id, String baseUrl) throws ParsingException {
public abstract String getUrl(String id) throws ParsingException, UnsupportedOperationException;
public abstract boolean onAcceptUrl(String url) throws ParsingException;
public String getUrl(final String id, final String baseUrl)
throws ParsingException, UnsupportedOperationException {
return getUrl(id);
}
@ -42,30 +46,52 @@ public abstract class LinkHandlerFactory {
// Logic
///////////////////////////////////
public LinkHandler fromUrl(String url) throws ParsingException {
if (url == null) throw new IllegalArgumentException("url can not be null");
final String baseUrl = Utils.getBaseUrl(url);
return fromUrl(url, baseUrl);
/**
* Builds a {@link LinkHandler} from a url.<br>
* Be sure to call {@link Utils#followGoogleRedirectIfNeeded(String)} on the url if overriding
* this function.
*
* @param url the url to extract path and id from
* @return a {@link LinkHandler} complete with information
*/
public LinkHandler fromUrl(final String url) throws ParsingException {
if (Utils.isNullOrEmpty(url)) {
throw new IllegalArgumentException("The url is null or empty");
}
final String polishedUrl = Utils.followGoogleRedirectIfNeeded(url);
final String baseUrl = Utils.getBaseUrl(polishedUrl);
return fromUrl(polishedUrl, baseUrl);
}
public LinkHandler fromUrl(String url, String baseUrl) throws ParsingException {
if (url == null) throw new IllegalArgumentException("url can not be null");
/**
* Builds a {@link LinkHandler} from an URL and a base URL. The URL is expected to be already
* polished from Google search redirects (otherwise how could {@code baseUrl} have been
* extracted?).<br>
* So do not call {@link Utils#followGoogleRedirectIfNeeded(String)} on the URL if overriding
* this function, since that should be done in {@link #fromUrl(String)}.
*
* @param url the URL without Google search redirects to extract id from
* @param baseUrl the base URL
* @return a {@link LinkHandler} complete with information
*/
public LinkHandler fromUrl(final String url, final String baseUrl) throws ParsingException {
Objects.requireNonNull(url, "URL cannot be null");
if (!acceptUrl(url)) {
throw new ParsingException("Malformed unacceptable url: " + url);
throw new ParsingException("URL not accepted: " + url);
}
final String id = getId(url);
return new LinkHandler(url, getUrl(id, baseUrl), id);
}
public LinkHandler fromId(String id) throws ParsingException {
if (id == null) throw new IllegalArgumentException("id can not be null");
public LinkHandler fromId(final String id) throws ParsingException {
Objects.requireNonNull(id, "ID cannot be null");
final String url = getUrl(id);
return new LinkHandler(url, url, id);
}
public LinkHandler fromId(String id, String baseUrl) throws ParsingException {
if (id == null) throw new IllegalArgumentException("id can not be null");
public LinkHandler fromId(final String id, final String baseUrl) throws ParsingException {
Objects.requireNonNull(id, "ID cannot be null");
final String url = getUrl(id, baseUrl);
return new LinkHandler(url, url, id);
}
@ -76,11 +102,6 @@ public abstract class LinkHandlerFactory {
* Return false if this service shall not allow to be called through ACTIONs.
*/
public boolean acceptUrl(final String url) throws ParsingException {
try {
return onAcceptUrl(url);
} catch (FoundAdException fe) {
throw fe;
}
return onAcceptUrl(url);
}
}

View File

@ -7,17 +7,17 @@ public class ListLinkHandler extends LinkHandler {
protected final List<String> contentFilters;
protected final String sortFilter;
public ListLinkHandler(String originalUrl,
String url,
String id,
List<String> contentFilters,
String sortFilter) {
public ListLinkHandler(final String originalUrl,
final String url,
final String id,
final List<String> contentFilters,
final String sortFilter) {
super(originalUrl, url, id);
this.contentFilters = Collections.unmodifiableList(contentFilters);
this.sortFilter = sortFilter;
}
public ListLinkHandler(ListLinkHandler handler) {
public ListLinkHandler(final ListLinkHandler handler) {
this(handler.originalUrl,
handler.url,
handler.id,
@ -25,14 +25,12 @@ public class ListLinkHandler extends LinkHandler {
handler.sortFilter);
}
public ListLinkHandler(LinkHandler handler,
List<String> contentFilters,
String sortFilter) {
public ListLinkHandler(final LinkHandler handler) {
this(handler.originalUrl,
handler.url,
handler.id,
contentFilters,
sortFilter);
Collections.emptyList(),
"");
}
public List<String> getContentFilters() {

View File

@ -5,6 +5,7 @@ import org.schabi.newpipe.extractor.utils.Utils;
import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
public abstract class ListLinkHandlerFactory extends LinkHandlerFactory {
@ -12,17 +13,14 @@ public abstract class ListLinkHandlerFactory extends LinkHandlerFactory {
// To Override
///////////////////////////////////
public List<String> getContentFilter(String url) throws ParsingException {
return new ArrayList<>(0);
}
public abstract String getUrl(String id, List<String> contentFilter, String sortFilter)
throws ParsingException, UnsupportedOperationException;
public String getSortFilter(String url) throws ParsingException {
return "";
}
public abstract String getUrl(String id, List<String> contentFilter, String sortFilter) throws ParsingException;
public String getUrl(String id, List<String> contentFilter, String sortFilter, String baseUrl) throws ParsingException {
public String getUrl(final String id,
final List<String> contentFilter,
final String sortFilter,
final String baseUrl)
throws ParsingException, UnsupportedOperationException {
return getUrl(id, contentFilter, sortFilter);
}
@ -31,61 +29,62 @@ public abstract class ListLinkHandlerFactory extends LinkHandlerFactory {
///////////////////////////////////
@Override
public ListLinkHandler fromUrl(String url) throws ParsingException {
String baseUrl = Utils.getBaseUrl(url);
return fromUrl(url, baseUrl);
public ListLinkHandler fromUrl(final String url) throws ParsingException {
final String polishedUrl = Utils.followGoogleRedirectIfNeeded(url);
final String baseUrl = Utils.getBaseUrl(polishedUrl);
return fromUrl(polishedUrl, baseUrl);
}
@Override
public ListLinkHandler fromUrl(String url, String baseUrl) throws ParsingException {
if (url == null) throw new IllegalArgumentException("url may not be null");
return new ListLinkHandler(super.fromUrl(url, baseUrl), getContentFilter(url), getSortFilter(url));
public ListLinkHandler fromUrl(final String url, final String baseUrl) throws ParsingException {
Objects.requireNonNull(url, "URL may not be null");
return new ListLinkHandler(super.fromUrl(url, baseUrl));
}
@Override
public ListLinkHandler fromId(String id) throws ParsingException {
return new ListLinkHandler(super.fromId(id), new ArrayList<String>(0), "");
public ListLinkHandler fromId(final String id) throws ParsingException {
return new ListLinkHandler(super.fromId(id));
}
@Override
public ListLinkHandler fromId(String id, String baseUrl) throws ParsingException {
return new ListLinkHandler(super.fromId(id, baseUrl), new ArrayList<String>(0), "");
public ListLinkHandler fromId(final String id, final String baseUrl) throws ParsingException {
return new ListLinkHandler(super.fromId(id, baseUrl));
}
public ListLinkHandler fromQuery(String id,
List<String> contentFilters,
String sortFilter) throws ParsingException {
public ListLinkHandler fromQuery(final String id,
final List<String> contentFilters,
final String sortFilter) throws ParsingException {
final String url = getUrl(id, contentFilters, sortFilter);
return new ListLinkHandler(url, url, id, contentFilters, sortFilter);
}
public ListLinkHandler fromQuery(String id,
List<String> contentFilters,
String sortFilter, String baseUrl) throws ParsingException {
public ListLinkHandler fromQuery(final String id,
final List<String> contentFilters,
final String sortFilter,
final String baseUrl) throws ParsingException {
final String url = getUrl(id, contentFilters, sortFilter, baseUrl);
return new ListLinkHandler(url, url, id, contentFilters, sortFilter);
}
/**
* For making ListLinkHandlerFactory compatible with LinkHandlerFactory we need to override this,
* however it should not be overridden by the actual implementation.
* For making ListLinkHandlerFactory compatible with LinkHandlerFactory we need to override
* this, however it should not be overridden by the actual implementation.
*
* @param id
* @return the url coresponding to id without any filters applied
* @return the url corresponding to id without any filters applied
*/
public String getUrl(String id) throws ParsingException {
return getUrl(id, new ArrayList<String>(0), "");
public String getUrl(final String id) throws ParsingException, UnsupportedOperationException {
return getUrl(id, new ArrayList<>(0), "");
}
@Override
public String getUrl(String id, String baseUrl) throws ParsingException {
return getUrl(id, new ArrayList<String>(0), "", baseUrl);
public String getUrl(final String id, final String baseUrl) throws ParsingException {
return getUrl(id, new ArrayList<>(0), "", baseUrl);
}
/**
* Will returns content filter the corresponding extractor can handle like "channels", "videos", "music", etc.
* Will returns content filter the corresponding extractor can handle like "channels", "videos",
* "music", etc.
*
* @return filter that can be applied when building a query for getting a list
*/
@ -94,7 +93,8 @@ public abstract class ListLinkHandlerFactory extends LinkHandlerFactory {
}
/**
* Will returns sort filter the corresponding extractor can handle like "A-Z", "oldest first", "size", etc.
* Will returns sort filter the corresponding extractor can handle like "A-Z", "oldest first",
* "size", etc.
*
* @return filter that can be applied when building a query for getting a list
*/

View File

@ -0,0 +1,54 @@
package org.schabi.newpipe.extractor.linkhandler;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import javax.annotation.Nonnull;
import java.io.Serializable;
import java.util.List;
/**
* A {@link ListLinkHandler} which can be used to be returned from {@link
* org.schabi.newpipe.extractor.channel.ChannelInfo#getTabs() ChannelInfo#getTabs()} when a
* specific tab's data has already been fetched.
*
* <p>
* This class allows passing a builder for a {@link ChannelTabExtractor} that can hold references
* to variables.
* </p>
*
* <p>
* Note: a service that wishes to use this class in one of its {@link
* org.schabi.newpipe.extractor.channel.ChannelExtractor ChannelExtractor}s must also add the
* following snippet of code in the service's
* {@link StreamingService#getChannelTabExtractor(ListLinkHandler)}:
* <pre>
* if (linkHandler instanceof ReadyChannelTabListLinkHandler) {
* return ((ReadyChannelTabListLinkHandler) linkHandler).getChannelTabExtractor(this);
* }
* </pre>
*/
public class ReadyChannelTabListLinkHandler extends ListLinkHandler {
public interface ChannelTabExtractorBuilder extends Serializable {
@Nonnull
ChannelTabExtractor build(@Nonnull StreamingService service,
@Nonnull ListLinkHandler linkHandler);
}
private final ChannelTabExtractorBuilder extractorBuilder;
public ReadyChannelTabListLinkHandler(
final String url,
final String channelId,
@Nonnull final String channelTab,
@Nonnull final ChannelTabExtractorBuilder extractorBuilder) {
super(url, url, channelId, List.of(channelTab), "");
this.extractorBuilder = extractorBuilder;
}
@Nonnull
public ChannelTabExtractor getChannelTabExtractor(@Nonnull final StreamingService service) {
return extractorBuilder.build(service, new ListLinkHandler(this));
}
}

View File

@ -4,15 +4,15 @@ import java.util.List;
public class SearchQueryHandler extends ListLinkHandler {
public SearchQueryHandler(String originalUrl,
String url,
String searchString,
List<String> contentFilters,
String sortFilter) {
public SearchQueryHandler(final String originalUrl,
final String url,
final String searchString,
final List<String> contentFilters,
final String sortFilter) {
super(originalUrl, url, searchString, contentFilters, sortFilter);
}
public SearchQueryHandler(ListLinkHandler handler) {
public SearchQueryHandler(final ListLinkHandler handler) {
this(handler.originalUrl,
handler.url,
handler.id,

View File

@ -2,7 +2,7 @@ package org.schabi.newpipe.extractor.linkhandler;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
public abstract class SearchQueryHandlerFactory extends ListLinkHandlerFactory {
@ -12,9 +12,11 @@ public abstract class SearchQueryHandlerFactory extends ListLinkHandlerFactory {
///////////////////////////////////
@Override
public abstract String getUrl(String querry, List<String> contentFilter, String sortFilter) throws ParsingException;
public abstract String getUrl(String query, List<String> contentFilter, String sortFilter)
throws ParsingException, UnsupportedOperationException;
public String getSearchString(String url) {
@SuppressWarnings("unused")
public String getSearchString(final String url) {
return "";
}
@ -23,29 +25,26 @@ public abstract class SearchQueryHandlerFactory extends ListLinkHandlerFactory {
///////////////////////////////////
@Override
public String getId(String url) {
public String getId(final String url) throws ParsingException, UnsupportedOperationException {
return getSearchString(url);
}
@Override
public SearchQueryHandler fromQuery(String querry,
List<String> contentFilter,
String sortFilter) throws ParsingException {
return new SearchQueryHandler(super.fromQuery(querry, contentFilter, sortFilter));
public SearchQueryHandler fromQuery(final String query,
final List<String> contentFilter,
final String sortFilter) throws ParsingException {
return new SearchQueryHandler(super.fromQuery(query, contentFilter, sortFilter));
}
public SearchQueryHandler fromQuery(String querry) throws ParsingException {
return fromQuery(querry, new ArrayList<String>(0), "");
public SearchQueryHandler fromQuery(final String query) throws ParsingException {
return fromQuery(query, Collections.emptyList(), "");
}
/**
* It's not mandatory for NewPipe to handle the Url
*
* @param url
* @return
*/
@Override
public boolean onAcceptUrl(String url) {
public boolean onAcceptUrl(final String url) {
return false;
}
}

View File

@ -1,6 +1,7 @@
package org.schabi.newpipe.extractor.localization;
import javax.annotation.Nonnull;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.Collections;
@ -9,23 +10,26 @@ import java.util.List;
/**
* Represents a country that should be used when fetching content.
* <p>
* YouTube, for example, give different results in their feed depending on which country is selected.
* YouTube, for example, give different results in their feed depending on which country is
* selected.
* </p>
*/
public class ContentCountry implements Serializable {
public static final ContentCountry DEFAULT = new ContentCountry(Localization.DEFAULT.getCountryCode());
public static final ContentCountry DEFAULT =
new ContentCountry(Localization.DEFAULT.getCountryCode());
@Nonnull private final String countryCode;
@Nonnull
private final String countryCode;
public static List<ContentCountry> listFrom(String... countryCodeList) {
public static List<ContentCountry> listFrom(final String... countryCodeList) {
final List<ContentCountry> toReturn = new ArrayList<>();
for (String countryCode : countryCodeList) {
for (final String countryCode : countryCodeList) {
toReturn.add(new ContentCountry(countryCode));
}
return Collections.unmodifiableList(toReturn);
}
public ContentCountry(@Nonnull String countryCode) {
public ContentCountry(@Nonnull final String countryCode) {
this.countryCode = countryCode;
}
@ -40,11 +44,15 @@ public class ContentCountry implements Serializable {
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (!(o instanceof ContentCountry)) return false;
public boolean equals(final Object o) {
if (this == o) {
return true;
}
if (!(o instanceof ContentCountry)) {
return false;
}
ContentCountry that = (ContentCountry) o;
final ContentCountry that = (ContentCountry) o;
return countryCode.equals(that.countryCode);
}

View File

@ -1,37 +1,70 @@
package org.schabi.newpipe.extractor.localization;
import edu.umd.cs.findbugs.annotations.NonNull;
import javax.annotation.Nonnull;
import java.io.Serializable;
import java.time.OffsetDateTime;
import java.time.ZoneOffset;
import java.util.Calendar;
import java.util.GregorianCalendar;
/**
* A wrapper class that provides a field to describe if the date is precise or just an approximation.
* A wrapper class that provides a field to describe if the date/time is precise or just an
* approximation.
*/
public class DateWrapper implements Serializable {
@NonNull private final Calendar date;
@Nonnull
private final OffsetDateTime offsetDateTime;
private final boolean isApproximation;
public DateWrapper(@NonNull Calendar date) {
this(date, false);
/**
* @deprecated Use {@link #DateWrapper(OffsetDateTime)} instead.
*/
@Deprecated
public DateWrapper(@Nonnull final Calendar calendar) {
//noinspection deprecation
this(calendar, false);
}
public DateWrapper(@NonNull Calendar date, boolean isApproximation) {
this.date = date;
/**
* @deprecated Use {@link #DateWrapper(OffsetDateTime, boolean)} instead.
*/
@Deprecated
public DateWrapper(@Nonnull final Calendar calendar, final boolean isApproximation) {
this(OffsetDateTime.ofInstant(calendar.toInstant(), ZoneOffset.UTC), isApproximation);
}
public DateWrapper(@Nonnull final OffsetDateTime offsetDateTime) {
this(offsetDateTime, false);
}
public DateWrapper(@Nonnull final OffsetDateTime offsetDateTime,
final boolean isApproximation) {
this.offsetDateTime = offsetDateTime.withOffsetSameInstant(ZoneOffset.UTC);
this.isApproximation = isApproximation;
}
/**
* @return the wrapped date.
* @return the wrapped date/time as a {@link Calendar}.
* @deprecated use {@link #offsetDateTime()} instead.
*/
@NonNull
@Deprecated
@Nonnull
public Calendar date() {
return date;
return GregorianCalendar.from(offsetDateTime.toZonedDateTime());
}
/**
* @return if the date is considered is precise or just an approximation (e.g. service only returns an approximation
* like 2 weeks ago instead of a precise date).
* @return the wrapped date/time.
*/
@Nonnull
public OffsetDateTime offsetDateTime() {
return offsetDateTime;
}
/**
* @return if the date is considered is precise or just an approximation (e.g. service only
* returns an approximation like 2 weeks ago instead of a precise date).
*/
public boolean isApproximation() {
return isApproximation;

View File

@ -1,57 +1,67 @@
package org.schabi.newpipe.extractor.localization;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.utils.LocaleCompat;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
public class Localization implements Serializable {
public static final Localization DEFAULT = new Localization("en", "GB");
@Nonnull private final String languageCode;
@Nullable private final String countryCode;
@Nonnull
private final String languageCode;
@Nullable
private final String countryCode;
/**
* @param localizationCodeList a list of localization code, formatted like {@link #getLocalizationCode()}
* @param localizationCodeList a list of localization code, formatted like {@link
* #getLocalizationCode()}
* @throws IllegalArgumentException If any of the localizationCodeList is formatted incorrectly
* @return list of Localization objects
*/
public static List<Localization> listFrom(String... localizationCodeList) {
@Nonnull
public static List<Localization> listFrom(final String... localizationCodeList) {
final List<Localization> toReturn = new ArrayList<>();
for (String localizationCode : localizationCodeList) {
toReturn.add(fromLocalizationCode(localizationCode));
for (final String localizationCode : localizationCodeList) {
toReturn.add(fromLocalizationCode(localizationCode)
.orElseThrow(() -> new IllegalArgumentException(
"Not a localization code: " + localizationCode
)));
}
return Collections.unmodifiableList(toReturn);
}
/**
* @param localizationCode a localization code, formatted like {@link #getLocalizationCode()}
* @return A Localization, if the code was valid.
*/
public static Localization fromLocalizationCode(String localizationCode) {
final int indexSeparator = localizationCode.indexOf("-");
final String languageCode, countryCode;
if (indexSeparator != -1) {
languageCode = localizationCode.substring(0, indexSeparator);
countryCode = localizationCode.substring(indexSeparator + 1);
} else {
languageCode = localizationCode;
countryCode = null;
}
return new Localization(languageCode, countryCode);
@Nonnull
public static Optional<Localization> fromLocalizationCode(final String localizationCode) {
return LocaleCompat.forLanguageTag(localizationCode).map(Localization::fromLocale);
}
public Localization(@Nonnull String languageCode, @Nullable String countryCode) {
public Localization(@Nonnull final String languageCode, @Nullable final String countryCode) {
this.languageCode = languageCode;
this.countryCode = countryCode;
}
public Localization(@Nonnull String languageCode) {
public Localization(@Nonnull final String languageCode) {
this(languageCode, null);
}
@Nonnull
public String getLanguageCode() {
return languageCode;
}
@ -61,17 +71,15 @@ public class Localization implements Serializable {
return countryCode == null ? "" : countryCode;
}
public Locale asLocale() {
return new Locale(getLanguageCode(), getCountryCode());
}
public static Localization fromLocale(@Nonnull Locale locale) {
public static Localization fromLocale(@Nonnull final Locale locale) {
return new Localization(locale.getLanguage(), locale.getCountry());
}
/**
* Return a formatted string in the form of: {@code language-Country}, or
* just {@code language} if country is {@code null}.
*
* @return A correctly formatted localizationCode for this localization.
*/
public String getLocalizationCode() {
return languageCode + (countryCode == null ? "" : "-" + countryCode);
@ -83,20 +91,47 @@ public class Localization implements Serializable {
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (!(o instanceof Localization)) return false;
public boolean equals(final Object o) {
if (this == o) {
return true;
}
if (!(o instanceof Localization)) {
return false;
}
Localization that = (Localization) o;
final Localization that = (Localization) o;
if (!languageCode.equals(that.languageCode)) return false;
return countryCode != null ? countryCode.equals(that.countryCode) : that.countryCode == null;
return languageCode.equals(that.languageCode)
&& Objects.equals(countryCode, that.countryCode);
}
@Override
public int hashCode() {
int result = languageCode.hashCode();
result = 31 * result + (countryCode != null ? countryCode.hashCode() : 0);
result = 31 * result + Objects.hashCode(countryCode);
return result;
}
/**
* Converts a three letter language code (ISO 639-2/T) to a Locale
* because limits of Java Locale class.
*
* @param code a three letter language code
* @return the Locale corresponding
*/
public static Locale getLocaleFromThreeLetterCode(@Nonnull final String code)
throws ParsingException {
final String[] languages = Locale.getISOLanguages();
final Map<String, Locale> localeMap = new HashMap<>(languages.length);
for (final String language : languages) {
final Locale locale = new Locale(language);
localeMap.put(locale.getISO3Language(), locale);
}
if (localeMap.containsKey(code)) {
return localeMap.get(code);
} else {
throw new ParsingException(
"Could not get Locale from this three letter language code" + code);
}
}
}

View File

@ -2,21 +2,28 @@ package org.schabi.newpipe.extractor.localization;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.timeago.PatternsHolder;
import org.schabi.newpipe.extractor.timeago.TimeAgoUnit;
import org.schabi.newpipe.extractor.utils.Parser;
import java.util.Calendar;
import java.util.Collection;
import java.time.OffsetDateTime;
import java.time.ZoneOffset;
import java.time.temporal.ChronoUnit;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.regex.Pattern;
import java.util.regex.Matcher;
import java.util.regex.MatchResult;
/**
* A helper class that is meant to be used by services that need to parse upload dates in the
* format '2 days ago' or similar.
* A helper class that is meant to be used by services that need to parse durations such as
* {@code 23 seconds} and/or upload dates in the format {@code 2 days ago} or similar.
*/
public class TimeAgoParser {
private static final Pattern DURATION_PATTERN = Pattern.compile("(?:(\\d+) )?([A-z]+)");
private final PatternsHolder patternsHolder;
private final Calendar consistentNow;
private final OffsetDateTime now;
/**
* Creates a helper to parse upload dates in the format '2 days ago'.
@ -24,16 +31,17 @@ public class TimeAgoParser {
* Instantiate a new {@link TimeAgoParser} every time you extract a new batch of items.
* </p>
*
* @param patternsHolder An object that holds the "time ago" patterns, special cases, and the language word separator.
* @param patternsHolder An object that holds the "time ago" patterns, special cases, and the
* language word separator.
*/
public TimeAgoParser(PatternsHolder patternsHolder) {
public TimeAgoParser(final PatternsHolder patternsHolder) {
this.patternsHolder = patternsHolder;
consistentNow = Calendar.getInstance();
now = OffsetDateTime.now(ZoneOffset.UTC);
}
/**
* Parses a textual date in the format '2 days ago' into a Calendar representation which is then wrapped in a
* {@link DateWrapper} object.
* Parses a textual date in the format '2 days ago' into a Calendar representation which is then
* wrapped in a {@link DateWrapper} object.
* <p>
* Beginning with days ago, the date is considered as an approximation.
*
@ -41,136 +49,140 @@ public class TimeAgoParser {
* @return The parsed time (can be approximated)
* @throws ParsingException if the time unit could not be recognized
*/
public DateWrapper parse(String textualDate) throws ParsingException {
for (Map.Entry<TimeAgoUnit, Map<String, Integer>> caseUnitEntry : patternsHolder.specialCases().entrySet()) {
final TimeAgoUnit timeAgoUnit = caseUnitEntry.getKey();
for (Map.Entry<String, Integer> caseMapToAmountEntry : caseUnitEntry.getValue().entrySet()) {
public DateWrapper parse(final String textualDate) throws ParsingException {
for (final Map.Entry<ChronoUnit, Map<String, Integer>> caseUnitEntry
: patternsHolder.specialCases().entrySet()) {
final ChronoUnit chronoUnit = caseUnitEntry.getKey();
for (final Map.Entry<String, Integer> caseMapToAmountEntry
: caseUnitEntry.getValue().entrySet()) {
final String caseText = caseMapToAmountEntry.getKey();
final Integer caseAmount = caseMapToAmountEntry.getValue();
if (textualDateMatches(textualDate, caseText)) {
return getResultFor(caseAmount, timeAgoUnit);
return getResultFor(caseAmount, chronoUnit);
}
}
}
int timeAgoAmount;
return getResultFor(parseTimeAgoAmount(textualDate), parseChronoUnit(textualDate));
}
/**
* Parses a textual duration into a duration computer number.
*
* @param textualDuration the textual duration to parse
* @return the textual duration parsed, as a primitive {@code long}
* @throws ParsingException if the textual duration could not be parsed
*/
public long parseDuration(final String textualDuration) throws ParsingException {
// We can't use Matcher.results, as it is only available on Android 14 and above
final Matcher matcher = DURATION_PATTERN.matcher(textualDuration);
final List<MatchResult> results = new ArrayList<>();
while (matcher.find()) {
results.add(matcher.toMatchResult());
}
return results.stream()
.map(match -> {
final String digits = match.group(1);
final String word = match.group(2);
int amount;
try {
amount = Integer.parseInt(digits);
} catch (final NumberFormatException ignored) {
amount = 1;
}
final ChronoUnit unit;
try {
unit = parseChronoUnit(word);
} catch (final ParsingException ignored) {
return 0L;
}
return amount * unit.getDuration().getSeconds();
})
.filter(n -> n > 0)
.reduce(Long::sum)
.orElseThrow(() -> new ParsingException(
"Could not parse duration \"" + textualDuration + "\""));
}
private int parseTimeAgoAmount(final String textualDate) {
try {
timeAgoAmount = parseTimeAgoAmount(textualDate);
} catch (NumberFormatException e) {
return Integer.parseInt(textualDate.replaceAll("\\D+", ""));
} catch (final NumberFormatException ignored) {
// If there is no valid number in the textual date,
// assume it is 1 (as in 'a second ago').
timeAgoAmount = 1;
return 1;
}
final TimeAgoUnit timeAgoUnit = parseTimeAgoUnit(textualDate);
return getResultFor(timeAgoAmount, timeAgoUnit);
}
private int parseTimeAgoAmount(String textualDate) throws NumberFormatException {
String timeValueStr = textualDate.replaceAll("\\D+", "");
return Integer.parseInt(timeValueStr);
private ChronoUnit parseChronoUnit(final String textualDate) throws ParsingException {
return patternsHolder.asMap().entrySet().stream()
.filter(e -> e.getValue().stream()
.anyMatch(agoPhrase -> textualDateMatches(textualDate, agoPhrase)))
.map(Map.Entry::getKey)
.findFirst()
.orElseThrow(() ->
new ParsingException("Unable to parse the date: " + textualDate));
}
private TimeAgoUnit parseTimeAgoUnit(String textualDate) throws ParsingException {
for (Map.Entry<TimeAgoUnit, Collection<String>> entry : patternsHolder.asMap().entrySet()) {
final TimeAgoUnit timeAgoUnit = entry.getKey();
for (String agoPhrase : entry.getValue()) {
if (textualDateMatches(textualDate, agoPhrase)) {
return timeAgoUnit;
}
}
}
throw new ParsingException("Unable to parse the date: " + textualDate);
}
private boolean textualDateMatches(String textualDate, String agoPhrase) {
private boolean textualDateMatches(final String textualDate, final String agoPhrase) {
if (textualDate.equals(agoPhrase)) {
return true;
}
if (patternsHolder.wordSeparator().isEmpty()) {
return textualDate.toLowerCase().contains(agoPhrase.toLowerCase());
} else {
final String escapedPhrase = Pattern.quote(agoPhrase.toLowerCase());
final String escapedSeparator;
if (patternsHolder.wordSeparator().equals(" ")) {
// From JDK8 \h - Treat horizontal spaces as a normal one (non-breaking space, thin space, etc.)
escapedSeparator = "[ \\t\\xA0\\u1680\\u180e\\u2000-\\u200a\\u202f\\u205f\\u3000]";
} else {
escapedSeparator = Pattern.quote(patternsHolder.wordSeparator());
}
// (^|separator)pattern($|separator)
// Check if the pattern is surrounded by separators or start/end of the string.
final String pattern =
"(^|" + escapedSeparator + ")" + escapedPhrase + "($|" + escapedSeparator + ")";
return Parser.isMatch(pattern, textualDate.toLowerCase());
}
final String escapedPhrase = Pattern.quote(agoPhrase.toLowerCase());
final String escapedSeparator = patternsHolder.wordSeparator().equals(" ")
// From JDK8 \h - Treat horizontal spaces as a normal one
// (non-breaking space, thin space, etc.)
// Also split the string on numbers to be able to parse strings like "2wk"
? "[ \\t\\xA0\\u1680\\u180e\\u2000-\\u200a\\u202f\\u205f\\u3000\\d]"
: Pattern.quote(patternsHolder.wordSeparator());
// (^|separator)pattern($|separator)
// Check if the pattern is surrounded by separators or start/end of the string.
final String pattern =
"(^|" + escapedSeparator + ")" + escapedPhrase + "($|" + escapedSeparator + ")";
return Parser.isMatch(pattern, textualDate.toLowerCase());
}
private DateWrapper getResultFor(int timeAgoAmount, TimeAgoUnit timeAgoUnit) {
final Calendar calendarTime = getNow();
private DateWrapper getResultFor(final int timeAgoAmount, final ChronoUnit chronoUnit) {
OffsetDateTime offsetDateTime = now;
boolean isApproximation = false;
switch (timeAgoUnit) {
switch (chronoUnit) {
case SECONDS:
calendarTime.add(Calendar.SECOND, -timeAgoAmount);
break;
case MINUTES:
calendarTime.add(Calendar.MINUTE, -timeAgoAmount);
break;
case HOURS:
calendarTime.add(Calendar.HOUR_OF_DAY, -timeAgoAmount);
offsetDateTime = offsetDateTime.minus(timeAgoAmount, chronoUnit);
break;
case DAYS:
calendarTime.add(Calendar.DAY_OF_MONTH, -timeAgoAmount);
isApproximation = true;
break;
case WEEKS:
calendarTime.add(Calendar.WEEK_OF_YEAR, -timeAgoAmount);
isApproximation = true;
break;
case MONTHS:
calendarTime.add(Calendar.MONTH, -timeAgoAmount);
offsetDateTime = offsetDateTime.minus(timeAgoAmount, chronoUnit);
isApproximation = true;
break;
case YEARS:
calendarTime.add(Calendar.YEAR, -timeAgoAmount);
// Prevent `PrettyTime` from showing '12 months ago'.
calendarTime.add(Calendar.DAY_OF_MONTH, -1);
// minusDays is needed to prevent `PrettyTime` from showing '12 months ago'.
offsetDateTime = offsetDateTime.minusYears(timeAgoAmount).minusDays(1);
isApproximation = true;
break;
}
if (isApproximation) {
markApproximatedTime(calendarTime);
offsetDateTime = offsetDateTime.truncatedTo(ChronoUnit.HOURS);
}
return new DateWrapper(calendarTime, isApproximation);
}
private Calendar getNow() {
return (Calendar) consistentNow.clone();
}
/**
* Marks the time as approximated by setting minutes, seconds and milliseconds to 0.
*
* @param calendarTime Time to be marked as approximated
*/
private void markApproximatedTime(Calendar calendarTime) {
calendarTime.set(Calendar.MINUTE, 0);
calendarTime.set(Calendar.SECOND, 0);
calendarTime.set(Calendar.MILLISECOND, 0);
return new DateWrapper(offsetDateTime, isApproximation);
}
}

View File

@ -6,14 +6,18 @@ import org.schabi.newpipe.extractor.timeago.PatternsManager;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
public class TimeAgoPatternsManager {
@Nullable
private static PatternsHolder getPatternsFor(@Nonnull Localization localization) {
return PatternsManager.getPatterns(localization.getLanguageCode(), localization.getCountryCode());
public final class TimeAgoPatternsManager {
private TimeAgoPatternsManager() {
}
@Nullable
public static TimeAgoParser getTimeAgoParserFor(@Nonnull Localization localization) {
private static PatternsHolder getPatternsFor(@Nonnull final Localization localization) {
return PatternsManager.getPatterns(localization.getLanguageCode(),
localization.getCountryCode());
}
@Nullable
public static TimeAgoParser getTimeAgoParserFor(@Nonnull final Localization localization) {
final PatternsHolder holder = getPatternsFor(localization);
if (holder == null) {

View File

@ -1,23 +1,61 @@
package org.schabi.newpipe.extractor.playlist;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.stream.Description;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import javax.annotation.Nonnull;
import java.util.Collections;
import java.util.List;
public abstract class PlaylistExtractor extends ListExtractor<StreamInfoItem> {
public PlaylistExtractor(StreamingService service, ListLinkHandler linkHandler) {
public PlaylistExtractor(final StreamingService service, final ListLinkHandler linkHandler) {
super(service, linkHandler);
}
public abstract String getThumbnailUrl() throws ParsingException;
public abstract String getBannerUrl() throws ParsingException;
public abstract String getUploaderUrl() throws ParsingException;
public abstract String getUploaderName() throws ParsingException;
public abstract String getUploaderAvatarUrl() throws ParsingException;
@Nonnull
public abstract List<Image> getUploaderAvatars() throws ParsingException;
public abstract boolean isUploaderVerified() throws ParsingException;
public abstract long getStreamCount() throws ParsingException;
@Nonnull
public abstract Description getDescription() throws ParsingException;
@Nonnull
public List<Image> getThumbnails() throws ParsingException {
return Collections.emptyList();
}
@Nonnull
public List<Image> getBanners() throws ParsingException {
return List.of();
}
@Nonnull
public String getSubChannelName() throws ParsingException {
return "";
}
@Nonnull
public String getSubChannelUrl() throws ParsingException {
return "";
}
@Nonnull
public List<Image> getSubChannelAvatars() throws ParsingException {
return List.of();
}
public PlaylistInfo.PlaylistType getPlaylistType() throws ParsingException {
return PlaylistInfo.PlaylistType.NORMAL;
}
}

View File

@ -1,39 +1,82 @@
package org.schabi.newpipe.extractor.playlist;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.ListExtractor.InfoItemsPage;
import org.schabi.newpipe.extractor.ListInfo;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.stream.Description;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import org.schabi.newpipe.extractor.utils.ExtractorHelper;
import javax.annotation.Nonnull;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
public class PlaylistInfo extends ListInfo<StreamInfoItem> {
public final class PlaylistInfo extends ListInfo<StreamInfoItem> {
private PlaylistInfo(int serviceId, ListLinkHandler linkHandler, String name) throws ParsingException {
/**
* Mixes are handled as particular playlists in NewPipeExtractor. {@link PlaylistType#NORMAL} is
* for non-mixes, while other values are for the different types of mixes. The type of a mix
* depends on how its contents are autogenerated.
*/
public enum PlaylistType {
/**
* A normal playlist (not a mix)
*/
NORMAL,
/**
* A mix made only of streams related to a particular stream, for example YouTube mixes
*/
MIX_STREAM,
/**
* A mix made only of music streams related to a particular stream, for example YouTube
* music mixes
*/
MIX_MUSIC,
/**
* A mix made only of streams from (or related to) the same channel, for example YouTube
* channel mixes
*/
MIX_CHANNEL,
/**
* A mix made only of streams related to a particular (musical) genre, for example YouTube
* genre mixes
*/
MIX_GENRE,
}
@SuppressWarnings("RedundantThrows")
private PlaylistInfo(final int serviceId, final ListLinkHandler linkHandler, final String name)
throws ParsingException {
super(serviceId, linkHandler, name);
}
public static PlaylistInfo getInfo(String url) throws IOException, ExtractionException {
public static PlaylistInfo getInfo(final String url) throws IOException, ExtractionException {
return getInfo(NewPipe.getServiceByUrl(url), url);
}
public static PlaylistInfo getInfo(StreamingService service, String url) throws IOException, ExtractionException {
PlaylistExtractor extractor = service.getPlaylistExtractor(url);
public static PlaylistInfo getInfo(final StreamingService service, final String url)
throws IOException, ExtractionException {
final PlaylistExtractor extractor = service.getPlaylistExtractor(url);
extractor.fetchPage();
return getInfo(extractor);
}
public static InfoItemsPage<StreamInfoItem> getMoreItems(StreamingService service,
String url,
String pageUrl) throws IOException, ExtractionException {
return service.getPlaylistExtractor(url).getPage(pageUrl);
public static InfoItemsPage<StreamInfoItem> getMoreItems(final StreamingService service,
final String url,
final Page page)
throws IOException, ExtractionException {
return service.getPlaylistExtractor(url).getPage(page);
}
/**
@ -41,7 +84,8 @@ public class PlaylistInfo extends ListInfo<StreamInfoItem> {
*
* @param extractor an extractor where fetchPage() was already got called on.
*/
public static PlaylistInfo getInfo(PlaylistExtractor extractor) throws ExtractionException {
public static PlaylistInfo getInfo(final PlaylistExtractor extractor)
throws ExtractionException {
final PlaylistInfo info = new PlaylistInfo(
extractor.getServiceId(),
@ -49,87 +93,122 @@ public class PlaylistInfo extends ListInfo<StreamInfoItem> {
extractor.getName());
// collect uploader extraction failures until we are sure this is not
// just a playlist without an uploader
List<Throwable> uploaderParsingErrors = new ArrayList<Throwable>(3);
final List<Throwable> uploaderParsingErrors = new ArrayList<>();
try {
info.setOriginalUrl(extractor.getOriginalUrl());
} catch (Exception e) {
} catch (final Exception e) {
info.addError(e);
}
try {
info.setStreamCount(extractor.getStreamCount());
} catch (Exception e) {
} catch (final Exception e) {
info.addError(e);
}
try {
info.setThumbnailUrl(extractor.getThumbnailUrl());
} catch (Exception e) {
info.setDescription(extractor.getDescription());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setThumbnails(extractor.getThumbnails());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setUploaderUrl(extractor.getUploaderUrl());
} catch (Exception e) {
info.setUploaderUrl("");
} catch (final Exception e) {
uploaderParsingErrors.add(e);
}
try {
info.setUploaderName(extractor.getUploaderName());
} catch (Exception e) {
info.setUploaderName("");
} catch (final Exception e) {
uploaderParsingErrors.add(e);
}
try {
info.setUploaderAvatarUrl(extractor.getUploaderAvatarUrl());
} catch (Exception e) {
info.setUploaderAvatarUrl("");
info.setUploaderAvatars(extractor.getUploaderAvatars());
} catch (final Exception e) {
uploaderParsingErrors.add(e);
}
try {
info.setBannerUrl(extractor.getBannerUrl());
} catch (Exception e) {
info.setSubChannelUrl(extractor.getSubChannelUrl());
} catch (final Exception e) {
uploaderParsingErrors.add(e);
}
try {
info.setSubChannelName(extractor.getSubChannelName());
} catch (final Exception e) {
uploaderParsingErrors.add(e);
}
try {
info.setSubChannelAvatars(extractor.getSubChannelAvatars());
} catch (final Exception e) {
uploaderParsingErrors.add(e);
}
try {
info.setBanners(extractor.getBanners());
} catch (final Exception e) {
info.addError(e);
}
// do not fail if everything but the uploader infos could be collected
if (uploaderParsingErrors.size() > 0 &&
(!info.getErrors().isEmpty() || uploaderParsingErrors.size() < 3)) {
try {
info.setPlaylistType(extractor.getPlaylistType());
} catch (final Exception e) {
info.addError(e);
}
// do not fail if everything but the uploader infos could be collected (TODO better comment)
if (!uploaderParsingErrors.isEmpty()
&& (!info.getErrors().isEmpty() || uploaderParsingErrors.size() < 3)) {
info.addAllErrors(uploaderParsingErrors);
}
final InfoItemsPage<StreamInfoItem> itemsPage = ExtractorHelper.getItemsPageOrLogError(info, extractor);
final InfoItemsPage<StreamInfoItem> itemsPage
= ExtractorHelper.getItemsPageOrLogError(info, extractor);
info.setRelatedItems(itemsPage.getItems());
info.setNextPageUrl(itemsPage.getNextPageUrl());
info.setNextPage(itemsPage.getNextPage());
return info;
}
private String thumbnailUrl;
private String bannerUrl;
private String uploaderUrl;
private String uploaderName;
private String uploaderAvatarUrl;
private long streamCount = 0;
private String uploaderUrl = "";
private String uploaderName = "";
private String subChannelUrl;
private String subChannelName;
private Description description;
@Nonnull
private List<Image> banners = List.of();
@Nonnull
private List<Image> subChannelAvatars = List.of();
@Nonnull
private List<Image> thumbnails = List.of();
@Nonnull
private List<Image> uploaderAvatars = List.of();
private long streamCount;
private PlaylistType playlistType;
public String getThumbnailUrl() {
return thumbnailUrl;
@Nonnull
public List<Image> getThumbnails() {
return thumbnails;
}
public void setThumbnailUrl(String thumbnailUrl) {
this.thumbnailUrl = thumbnailUrl;
public void setThumbnails(@Nonnull final List<Image> thumbnails) {
this.thumbnails = thumbnails;
}
public String getBannerUrl() {
return bannerUrl;
@Nonnull
public List<Image> getBanners() {
return banners;
}
public void setBannerUrl(String bannerUrl) {
this.bannerUrl = bannerUrl;
public void setBanners(@Nonnull final List<Image> banners) {
this.banners = banners;
}
public String getUploaderUrl() {
return uploaderUrl;
}
public void setUploaderUrl(String uploaderUrl) {
public void setUploaderUrl(final String uploaderUrl) {
this.uploaderUrl = uploaderUrl;
}
@ -137,23 +216,65 @@ public class PlaylistInfo extends ListInfo<StreamInfoItem> {
return uploaderName;
}
public void setUploaderName(String uploaderName) {
public void setUploaderName(final String uploaderName) {
this.uploaderName = uploaderName;
}
public String getUploaderAvatarUrl() {
return uploaderAvatarUrl;
@Nonnull
public List<Image> getUploaderAvatars() {
return uploaderAvatars;
}
public void setUploaderAvatarUrl(String uploaderAvatarUrl) {
this.uploaderAvatarUrl = uploaderAvatarUrl;
public void setUploaderAvatars(@Nonnull final List<Image> uploaderAvatars) {
this.uploaderAvatars = uploaderAvatars;
}
public String getSubChannelUrl() {
return subChannelUrl;
}
public void setSubChannelUrl(final String subChannelUrl) {
this.subChannelUrl = subChannelUrl;
}
public String getSubChannelName() {
return subChannelName;
}
public void setSubChannelName(final String subChannelName) {
this.subChannelName = subChannelName;
}
@Nonnull
public List<Image> getSubChannelAvatars() {
return subChannelAvatars;
}
public void setSubChannelAvatars(@Nonnull final List<Image> subChannelAvatars) {
this.subChannelAvatars = subChannelAvatars;
}
public long getStreamCount() {
return streamCount;
}
public void setStreamCount(long streamCount) {
public void setStreamCount(final long streamCount) {
this.streamCount = streamCount;
}
public Description getDescription() {
return description;
}
public void setDescription(final Description description) {
this.description = description;
}
public PlaylistType getPlaylistType() {
return playlistType;
}
public void setPlaylistType(final PlaylistType playlistType) {
this.playlistType = playlistType;
}
}

View File

@ -1,16 +1,23 @@
package org.schabi.newpipe.extractor.playlist;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.stream.Description;
import javax.annotation.Nullable;
public class PlaylistInfoItem extends InfoItem {
private String uploaderName;
private String uploaderUrl;
private boolean uploaderVerified;
/**
* How many streams this playlist have
*/
private long streamCount = 0;
private Description description;
private PlaylistInfo.PlaylistType playlistType;
public PlaylistInfoItem(int serviceId, String url, String name) {
public PlaylistInfoItem(final int serviceId, final String url, final String name) {
super(InfoType.PLAYLIST, serviceId, url, name);
}
@ -18,15 +25,48 @@ public class PlaylistInfoItem extends InfoItem {
return uploaderName;
}
public void setUploaderName(String uploader_name) {
this.uploaderName = uploader_name;
public void setUploaderName(final String uploaderName) {
this.uploaderName = uploaderName;
}
@Nullable
public String getUploaderUrl() {
return uploaderUrl;
}
public void setUploaderUrl(@Nullable final String uploaderUrl) {
this.uploaderUrl = uploaderUrl;
}
public boolean isUploaderVerified() {
return uploaderVerified;
}
public void setUploaderVerified(final boolean uploaderVerified) {
this.uploaderVerified = uploaderVerified;
}
public long getStreamCount() {
return streamCount;
}
public void setStreamCount(long stream_count) {
this.streamCount = stream_count;
public void setStreamCount(final long streamCount) {
this.streamCount = streamCount;
}
public Description getDescription() {
return description;
}
public void setDescription(final Description description) {
this.description = description;
}
public PlaylistInfo.PlaylistType getPlaylistType() {
return playlistType;
}
public void setPlaylistType(final PlaylistInfo.PlaylistType playlistType) {
this.playlistType = playlistType;
}
}

View File

@ -2,20 +2,52 @@ package org.schabi.newpipe.extractor.playlist;
import org.schabi.newpipe.extractor.InfoItemExtractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.stream.Description;
import javax.annotation.Nonnull;
public interface PlaylistInfoItemExtractor extends InfoItemExtractor {
/**
* Get the uploader name
* @return the uploader name
* @throws ParsingException
*/
String getUploaderName() throws ParsingException;
/**
* Get the uploader url
* @return the uploader url
*/
String getUploaderUrl() throws ParsingException;
/**
* Get whether the uploader is verified
* @return whether the uploader is verified
*/
boolean isUploaderVerified() throws ParsingException;
/**
* Get the number of streams
* @return the number of streams
* @throws ParsingException
*/
long getStreamCount() throws ParsingException;
/**
* Get the description of the playlist if there is any.
* Otherwise, an {@link Description#EMPTY_DESCRIPTION EMPTY_DESCRIPTION} is returned.
* @return the playlist's description
*/
@Nonnull
default Description getDescription() throws ParsingException {
return Description.EMPTY_DESCRIPTION;
}
/**
* @return the type of this playlist, see {@link PlaylistInfo.PlaylistType} for a description
* of types. If not overridden always returns {@link PlaylistInfo.PlaylistType#NORMAL}.
*/
@Nonnull
default PlaylistInfo.PlaylistType getPlaylistType() throws ParsingException {
return PlaylistInfo.PlaylistType.NORMAL;
}
}

View File

@ -3,34 +3,52 @@ package org.schabi.newpipe.extractor.playlist;
import org.schabi.newpipe.extractor.InfoItemsCollector;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
public class PlaylistInfoItemsCollector extends InfoItemsCollector<PlaylistInfoItem, PlaylistInfoItemExtractor> {
public class PlaylistInfoItemsCollector
extends InfoItemsCollector<PlaylistInfoItem, PlaylistInfoItemExtractor> {
public PlaylistInfoItemsCollector(int serviceId) {
public PlaylistInfoItemsCollector(final int serviceId) {
super(serviceId);
}
@Override
public PlaylistInfoItem extract(PlaylistInfoItemExtractor extractor) throws ParsingException {
String name = extractor.getName();
int serviceId = getServiceId();
String url = extractor.getUrl();
PlaylistInfoItem resultItem = new PlaylistInfoItem(serviceId, url, name);
public PlaylistInfoItem extract(final PlaylistInfoItemExtractor extractor)
throws ParsingException {
final PlaylistInfoItem resultItem = new PlaylistInfoItem(
getServiceId(), extractor.getUrl(), extractor.getName());
try {
resultItem.setUploaderName(extractor.getUploaderName());
} catch (Exception e) {
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setThumbnailUrl(extractor.getThumbnailUrl());
} catch (Exception e) {
resultItem.setUploaderUrl(extractor.getUploaderUrl());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setUploaderVerified(extractor.isUploaderVerified());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setThumbnails(extractor.getThumbnails());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setStreamCount(extractor.getStreamCount());
} catch (Exception e) {
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setDescription(extractor.getDescription());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setPlaylistType(extractor.getPlaylistType());
} catch (final Exception e) {
addError(e);
}
return resultItem;

View File

@ -2,22 +2,24 @@ package org.schabi.newpipe.extractor.search;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.MetaInfo;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandler;
import javax.annotation.Nonnull;
import java.util.List;
public abstract class SearchExtractor extends ListExtractor<InfoItem> {
public static class NothingFoundException extends ExtractionException {
public NothingFoundException(String message) {
public NothingFoundException(final String message) {
super(message);
}
}
public SearchExtractor(StreamingService service, SearchQueryHandler linkHandler) {
public SearchExtractor(final StreamingService service, final SearchQueryHandler linkHandler) {
super(service, linkHandler);
}
@ -25,8 +27,18 @@ public abstract class SearchExtractor extends ListExtractor<InfoItem> {
return getLinkHandler().getSearchString();
}
/**
* The search suggestion provided by the service.
* <p>
* This method also returns the corrected query if
* {@link SearchExtractor#isCorrectedSearch()} is true.
*
* @return a suggestion to another query, the corrected query, or an empty String.
*/
@Nonnull
public abstract String getSearchSuggestion() throws ParsingException;
@Nonnull
@Override
public SearchQueryHandler getLinkHandler() {
return (SearchQueryHandler) super.getLinkHandler();
@ -37,4 +49,24 @@ public abstract class SearchExtractor extends ListExtractor<InfoItem> {
public String getName() {
return getLinkHandler().getSearchString();
}
/**
* Tell if the search was corrected by the service (if it's not exactly the search you typed).
* <p>
* Example: on YouTube, if you search for "pewdeipie",
* it will give you results for "pewdiepie", then isCorrectedSearch should return true.
*
* @return whether the results comes from a corrected query or not.
*/
public abstract boolean isCorrectedSearch() throws ParsingException;
/**
* Meta information about the search query.
* <p>
* Example: on YouTube, if you search for "Covid-19",
* there is a box with information from the WHO about Covid-19 and a link to the WHO's website.
* @return additional meta information about the search query
*/
@Nonnull
public abstract List<MetaInfo> getMetaInfo() throws ParsingException;
}

View File

@ -3,34 +3,42 @@ package org.schabi.newpipe.extractor.search;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.ListInfo;
import org.schabi.newpipe.extractor.MetaInfo;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandler;
import org.schabi.newpipe.extractor.utils.ExtractorHelper;
import java.io.IOException;
import java.util.List;
import javax.annotation.Nonnull;
public class SearchInfo extends ListInfo<InfoItem> {
private String searchString;
private final String searchString;
private String searchSuggestion;
private boolean isCorrectedSearch;
private List<MetaInfo> metaInfo = List.of();
public SearchInfo(int serviceId,
SearchQueryHandler qIHandler,
String searchString) {
public SearchInfo(final int serviceId,
final SearchQueryHandler qIHandler,
final String searchString) {
super(serviceId, qIHandler, "Search");
this.searchString = searchString;
}
public static SearchInfo getInfo(StreamingService service, SearchQueryHandler searchQuery) throws ExtractionException, IOException {
SearchExtractor extractor = service.getSearchExtractor(searchQuery);
public static SearchInfo getInfo(final StreamingService service,
final SearchQueryHandler searchQuery)
throws ExtractionException, IOException {
final SearchExtractor extractor = service.getSearchExtractor(searchQuery);
extractor.fetchPage();
return getInfo(extractor);
}
public static SearchInfo getInfo(SearchExtractor extractor) throws ExtractionException, IOException {
public static SearchInfo getInfo(final SearchExtractor extractor)
throws ExtractionException, IOException {
final SearchInfo info = new SearchInfo(
extractor.getServiceId(),
extractor.getLinkHandler(),
@ -38,36 +46,68 @@ public class SearchInfo extends ListInfo<InfoItem> {
try {
info.setOriginalUrl(extractor.getOriginalUrl());
} catch (Exception e) {
} catch (final Exception e) {
info.addError(e);
}
try {
info.searchSuggestion = extractor.getSearchSuggestion();
} catch (Exception e) {
info.setSearchSuggestion(extractor.getSearchSuggestion());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setIsCorrectedSearch(extractor.isCorrectedSearch());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setMetaInfo(extractor.getMetaInfo());
} catch (final Exception e) {
info.addError(e);
}
ListExtractor.InfoItemsPage<InfoItem> page = ExtractorHelper.getItemsPageOrLogError(info, extractor);
final ListExtractor.InfoItemsPage<InfoItem> page
= ExtractorHelper.getItemsPageOrLogError(info, extractor);
info.setRelatedItems(page.getItems());
info.setNextPageUrl(page.getNextPageUrl());
info.setNextPage(page.getNextPage());
return info;
}
public static ListExtractor.InfoItemsPage<InfoItem> getMoreItems(StreamingService service,
SearchQueryHandler query,
String pageUrl)
public static ListExtractor.InfoItemsPage<InfoItem> getMoreItems(final StreamingService service,
final SearchQueryHandler query,
final Page page)
throws IOException, ExtractionException {
return service.getSearchExtractor(query).getPage(pageUrl);
return service.getSearchExtractor(query).getPage(page);
}
// Getter
public String getSearchString() {
return searchString;
return this.searchString;
}
public String getSearchSuggestion() {
return searchSuggestion;
return this.searchSuggestion;
}
public boolean isCorrectedSearch() {
return this.isCorrectedSearch;
}
public void setIsCorrectedSearch(final boolean isCorrectedSearch) {
this.isCorrectedSearch = isCorrectedSearch;
}
public void setSearchSuggestion(final String searchSuggestion) {
this.searchSuggestion = searchSuggestion;
}
@Nonnull
public List<MetaInfo> getMetaInfo() {
return metaInfo;
}
public void setMetaInfo(@Nonnull final List<MetaInfo> metaInfo) {
this.metaInfo = metaInfo;
}
}

View File

@ -0,0 +1,174 @@
// Created by Fynn Godau 2019, licensed GNU GPL version 3 or later
package org.schabi.newpipe.extractor.services.bandcamp;
import static org.schabi.newpipe.extractor.StreamingService.ServiceInfo.MediaCapability.AUDIO;
import static org.schabi.newpipe.extractor.StreamingService.ServiceInfo.MediaCapability.COMMENTS;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_URL;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampFeaturedExtractor.FEATURED_API_URL;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampFeaturedExtractor.KIOSK_FEATURED;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampRadioExtractor.KIOSK_RADIO;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampRadioExtractor.RADIO_API_URL;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.ChannelExtractor;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import org.schabi.newpipe.extractor.comments.CommentsExtractor;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.kiosk.KioskList;
import org.schabi.newpipe.extractor.linkhandler.LinkHandler;
import org.schabi.newpipe.extractor.linkhandler.LinkHandlerFactory;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import org.schabi.newpipe.extractor.linkhandler.ReadyChannelTabListLinkHandler;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandler;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandlerFactory;
import org.schabi.newpipe.extractor.playlist.PlaylistExtractor;
import org.schabi.newpipe.extractor.search.SearchExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampChannelExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampChannelTabExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampCommentsExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampFeaturedExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampPlaylistExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampRadioExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampRadioStreamExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampSearchExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampStreamExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampSuggestionExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampChannelLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampChannelTabLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampCommentsLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampFeaturedLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampPlaylistLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampSearchQueryHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampStreamLinkHandlerFactory;
import org.schabi.newpipe.extractor.stream.StreamExtractor;
import org.schabi.newpipe.extractor.subscription.SubscriptionExtractor;
import org.schabi.newpipe.extractor.suggestion.SuggestionExtractor;
import java.util.Arrays;
public class BandcampService extends StreamingService {
public BandcampService(final int id) {
super(id, "Bandcamp", Arrays.asList(AUDIO, COMMENTS));
}
@Override
public String getBaseUrl() {
return BASE_URL;
}
@Override
public LinkHandlerFactory getStreamLHFactory() {
return BandcampStreamLinkHandlerFactory.getInstance();
}
@Override
public ListLinkHandlerFactory getChannelLHFactory() {
return BandcampChannelLinkHandlerFactory.getInstance();
}
@Override
public ListLinkHandlerFactory getChannelTabLHFactory() {
return BandcampChannelTabLinkHandlerFactory.getInstance();
}
@Override
public ListLinkHandlerFactory getPlaylistLHFactory() {
return BandcampPlaylistLinkHandlerFactory.getInstance();
}
@Override
public SearchQueryHandlerFactory getSearchQHFactory() {
return BandcampSearchQueryHandlerFactory.getInstance();
}
@Override
public ListLinkHandlerFactory getCommentsLHFactory() {
return BandcampCommentsLinkHandlerFactory.getInstance();
}
@Override
public SearchExtractor getSearchExtractor(final SearchQueryHandler queryHandler) {
return new BandcampSearchExtractor(this, queryHandler);
}
@Override
public SuggestionExtractor getSuggestionExtractor() {
return new BandcampSuggestionExtractor(this);
}
@Override
public SubscriptionExtractor getSubscriptionExtractor() {
return null;
}
@Override
public KioskList getKioskList() throws ExtractionException {
final KioskList kioskList = new KioskList(this);
final ListLinkHandlerFactory h = BandcampFeaturedLinkHandlerFactory.getInstance();
try {
kioskList.addKioskEntry(
(streamingService, url, kioskId) -> new BandcampFeaturedExtractor(
BandcampService.this,
h.fromUrl(FEATURED_API_URL),
kioskId
),
h,
KIOSK_FEATURED
);
kioskList.addKioskEntry(
(streamingService, url, kioskId) -> new BandcampRadioExtractor(
BandcampService.this,
h.fromUrl(RADIO_API_URL),
kioskId
),
h,
KIOSK_RADIO
);
kioskList.setDefaultKiosk(KIOSK_FEATURED);
} catch (final Exception e) {
throw new ExtractionException(e);
}
return kioskList;
}
@Override
public ChannelExtractor getChannelExtractor(final ListLinkHandler linkHandler) {
return new BandcampChannelExtractor(this, linkHandler);
}
@Override
public ChannelTabExtractor getChannelTabExtractor(final ListLinkHandler linkHandler) {
if (linkHandler instanceof ReadyChannelTabListLinkHandler) {
return ((ReadyChannelTabListLinkHandler) linkHandler).getChannelTabExtractor(this);
} else {
return new BandcampChannelTabExtractor(this, linkHandler);
}
}
@Override
public PlaylistExtractor getPlaylistExtractor(final ListLinkHandler linkHandler) {
return new BandcampPlaylistExtractor(this, linkHandler);
}
@Override
public StreamExtractor getStreamExtractor(final LinkHandler linkHandler) {
if (BandcampExtractorHelper.isRadioUrl(linkHandler.getUrl())) {
return new BandcampRadioStreamExtractor(this, linkHandler);
}
return new BandcampStreamExtractor(this, linkHandler);
}
@Override
public CommentsExtractor getCommentsExtractor(final ListLinkHandler linkHandler) {
return new BandcampCommentsExtractor(this, linkHandler);
}
}

View File

@ -0,0 +1,62 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.playlist.PlaylistInfoItemExtractor;
import java.util.List;
import javax.annotation.Nonnull;
public class BandcampAlbumInfoItemExtractor implements PlaylistInfoItemExtractor {
private final JsonObject albumInfoItem;
private final String uploaderUrl;
public BandcampAlbumInfoItemExtractor(final JsonObject albumInfoItem,
final String uploaderUrl) {
this.albumInfoItem = albumInfoItem;
this.uploaderUrl = uploaderUrl;
}
@Override
public String getName() throws ParsingException {
return albumInfoItem.getString("title");
}
@Override
public String getUrl() throws ParsingException {
return BandcampExtractorHelper.getStreamUrlFromIds(
albumInfoItem.getLong("band_id"),
albumInfoItem.getLong("item_id"),
albumInfoItem.getString("item_type"));
}
@Nonnull
@Override
public List<Image> getThumbnails() throws ParsingException {
return BandcampExtractorHelper.getImagesFromImageId(albumInfoItem.getLong("art_id"), true);
}
@Override
public String getUploaderName() throws ParsingException {
return albumInfoItem.getString("band_name");
}
@Override
public String getUploaderUrl() {
return uploaderUrl;
}
@Override
public boolean isUploaderVerified() {
return false;
}
@Override
public long getStreamCount() {
return ListExtractor.ITEM_COUNT_UNKNOWN;
}
}

View File

@ -0,0 +1,192 @@
// Created by Fynn Godau 2019, licensed GNU GPL version 3 or later
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import static org.schabi.newpipe.extractor.Image.HEIGHT_UNKNOWN;
import static org.schabi.newpipe.extractor.Image.WIDTH_UNKNOWN;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getArtistDetails;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
import static org.schabi.newpipe.extractor.utils.Utils.replaceHttpWithHttps;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import org.jsoup.Jsoup;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.Image.ResolutionLevel;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.ChannelExtractor;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.exceptions.ReCaptchaException;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabs;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.linkhandler.ReadyChannelTabListLinkHandler;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampChannelTabLinkHandlerFactory;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Objects;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import javax.annotation.Nonnull;
public class BandcampChannelExtractor extends ChannelExtractor {
private JsonObject channelInfo;
public BandcampChannelExtractor(final StreamingService service,
final ListLinkHandler linkHandler) {
super(service, linkHandler);
}
@Nonnull
@Override
public List<Image> getAvatars() {
return getImagesFromImageId(channelInfo.getLong("bio_image_id"), false);
}
@Nonnull
@Override
public List<Image> getBanners() throws ParsingException {
/*
* Mobile API does not return the header or not the correct header.
* Therefore, we need to query the website
*/
try {
final String html = getDownloader()
.get(replaceHttpWithHttps(channelInfo.getString("bandcamp_url")))
.responseBody();
return Stream.of(Jsoup.parse(html).getElementById("customHeader"))
.filter(Objects::nonNull)
.flatMap(element -> element.getElementsByTag("img").stream())
.map(element -> element.attr("src"))
.filter(url -> !url.isEmpty())
.map(url -> new Image(
replaceHttpWithHttps(url), HEIGHT_UNKNOWN, WIDTH_UNKNOWN,
ResolutionLevel.UNKNOWN))
.collect(Collectors.toUnmodifiableList());
} catch (final IOException | ReCaptchaException e) {
throw new ParsingException("Could not download artist web site", e);
}
}
/**
* Bandcamp discontinued their RSS feeds because it hadn't been used enough.
*/
@Override
public String getFeedUrl() {
return null;
}
@Override
public long getSubscriberCount() {
return -1;
}
@Override
public String getDescription() {
return channelInfo.getString("bio");
}
@Override
public String getParentChannelName() {
return null;
}
@Override
public String getParentChannelUrl() {
return null;
}
@Nonnull
@Override
public List<Image> getParentChannelAvatars() {
return List.of();
}
@Override
public boolean isVerified() throws ParsingException {
return false;
}
@Nonnull
@Override
public List<ListLinkHandler> getTabs() throws ParsingException {
final JsonArray discography = channelInfo.getArray("discography");
final TabExtractorBuilder builder = new TabExtractorBuilder(discography);
final List<ListLinkHandler> tabs = new ArrayList<>();
boolean foundTrackItem = false;
boolean foundAlbumItem = false;
for (final Object discographyItem : discography) {
if (foundTrackItem && foundAlbumItem) {
break;
}
if (!(discographyItem instanceof JsonObject)) {
continue;
}
final JsonObject discographyJsonItem = (JsonObject) discographyItem;
final String itemType = discographyJsonItem.getString("item_type");
if (!foundTrackItem && "track".equals(itemType)) {
foundTrackItem = true;
tabs.add(new ReadyChannelTabListLinkHandler(getUrl()
+ BandcampChannelTabLinkHandlerFactory.getUrlSuffix(ChannelTabs.TRACKS),
getId(),
ChannelTabs.TRACKS,
builder));
}
if (!foundAlbumItem && "album".equals(itemType)) {
foundAlbumItem = true;
tabs.add(new ReadyChannelTabListLinkHandler(getUrl()
+ BandcampChannelTabLinkHandlerFactory.getUrlSuffix(ChannelTabs.ALBUMS),
getId(),
ChannelTabs.ALBUMS,
builder));
}
}
return Collections.unmodifiableList(tabs);
}
@Override
public void onFetchPage(@Nonnull final Downloader downloader)
throws IOException, ExtractionException {
channelInfo = getArtistDetails(getId());
}
@Nonnull
@Override
public String getName() {
return channelInfo.getString("name");
}
private static final class TabExtractorBuilder
implements ReadyChannelTabListLinkHandler.ChannelTabExtractorBuilder {
private final JsonArray discography;
TabExtractorBuilder(final JsonArray discography) {
this.discography = discography;
}
@Nonnull
@Override
public ChannelTabExtractor build(@Nonnull final StreamingService service,
@Nonnull final ListLinkHandler linkHandler) {
return BandcampChannelTabExtractor.fromDiscography(service, linkHandler, discography);
}
}
}

View File

@ -0,0 +1,60 @@
// Created by Fynn Godau 2019, licensed GNU GPL version 3 or later
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import org.jsoup.nodes.Element;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.channel.ChannelInfoItemExtractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import javax.annotation.Nonnull;
import java.util.List;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromSearchResult;
public class BandcampChannelInfoItemExtractor implements ChannelInfoItemExtractor {
private final Element resultInfo;
private final Element searchResult;
public BandcampChannelInfoItemExtractor(final Element searchResult) {
this.searchResult = searchResult;
resultInfo = searchResult.getElementsByClass("result-info").first();
}
@Override
public String getName() throws ParsingException {
return resultInfo.getElementsByClass("heading").text();
}
@Override
public String getUrl() throws ParsingException {
return resultInfo.getElementsByClass("itemurl").text();
}
@Nonnull
@Override
public List<Image> getThumbnails() throws ParsingException {
return getImagesFromSearchResult(searchResult);
}
@Override
public String getDescription() {
return resultInfo.getElementsByClass("subhead").text();
}
@Override
public long getSubscriberCount() {
return -1;
}
@Override
public long getStreamCount() {
return -1;
}
@Override
public boolean isVerified() throws ParsingException {
return false;
}
}

View File

@ -0,0 +1,95 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.MultiInfoItemsCollector;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabs;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.streaminfoitem.BandcampDiscographStreamInfoItemExtractor;
import javax.annotation.Nonnull;
import java.io.IOException;
public class BandcampChannelTabExtractor extends ChannelTabExtractor {
private JsonArray discography;
private final String filter;
public BandcampChannelTabExtractor(final StreamingService service,
final ListLinkHandler linkHandler) {
super(service, linkHandler);
final String tab = linkHandler.getContentFilters().get(0);
switch (tab) {
case ChannelTabs.TRACKS:
filter = "track";
break;
case ChannelTabs.ALBUMS:
filter = "album";
break;
default:
throw new IllegalArgumentException("Unsupported channel tab: " + tab);
}
}
public static BandcampChannelTabExtractor fromDiscography(final StreamingService service,
final ListLinkHandler linkHandler,
final JsonArray discography) {
final BandcampChannelTabExtractor tabExtractor =
new BandcampChannelTabExtractor(service, linkHandler);
tabExtractor.discography = discography;
return tabExtractor;
}
@Override
public void onFetchPage(@Nonnull final Downloader downloader) throws ParsingException {
if (discography == null) {
discography = BandcampExtractorHelper.getArtistDetails(getId())
.getArray("discography");
}
}
@Nonnull
@Override
public InfoItemsPage<InfoItem> getInitialPage() throws IOException, ExtractionException {
final MultiInfoItemsCollector collector = new MultiInfoItemsCollector(getServiceId());
for (final Object discograph : discography) {
// A discograph is as an item appears in a discography
if (!(discograph instanceof JsonObject)) {
continue;
}
final JsonObject discographJsonObject = (JsonObject) discograph;
final String itemType = discographJsonObject.getString("item_type", "");
if (!itemType.equals(filter)) {
continue;
}
switch (itemType) {
case "track":
collector.commit(new BandcampDiscographStreamInfoItemExtractor(
discographJsonObject, getUrl()));
break;
case "album":
collector.commit(new BandcampAlbumInfoItemExtractor(
discographJsonObject, getUrl()));
break;
}
}
return new InfoItemsPage<>(collector, null);
}
@Override
public InfoItemsPage<InfoItem> getPage(final Page page) {
return null;
}
}

View File

@ -0,0 +1,136 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_API_URL;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonWriter;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.comments.CommentsExtractor;
import org.schabi.newpipe.extractor.comments.CommentsInfoItem;
import org.schabi.newpipe.extractor.comments.CommentsInfoItemsCollector;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.exceptions.ReCaptchaException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.utils.JsonUtils;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.Collections;
import java.util.List;
import javax.annotation.Nonnull;
public class BandcampCommentsExtractor extends CommentsExtractor {
private static final String REVIEWS_API_URL = BASE_API_URL + "/tralbumcollectors/2/reviews";
private Document document;
public BandcampCommentsExtractor(final StreamingService service,
final ListLinkHandler linkHandler) {
super(service, linkHandler);
}
@Override
public void onFetchPage(@Nonnull final Downloader downloader)
throws IOException, ExtractionException {
document = Jsoup.parse(downloader.get(getLinkHandler().getUrl()).responseBody());
}
@Nonnull
@Override
public InfoItemsPage<CommentsInfoItem> getInitialPage()
throws IOException, ExtractionException {
final CommentsInfoItemsCollector collector = new CommentsInfoItemsCollector(getServiceId());
final JsonObject collectorsData = JsonUtils.toJsonObject(
document.getElementById("collectors-data").attr("data-blob"));
final JsonArray reviews = collectorsData.getArray("reviews");
for (final Object review : reviews) {
collector.commit(
new BandcampCommentsInfoItemExtractor((JsonObject) review, getUrl()));
}
if (!collectorsData.getBoolean("more_reviews_available")) {
return new InfoItemsPage<>(collector, null);
}
final String trackId = getTrackId();
final String token = getNextPageToken(reviews);
return new InfoItemsPage<>(collector, new Page(List.of(trackId, token)));
}
@Override
public InfoItemsPage<CommentsInfoItem> getPage(final Page page)
throws IOException, ExtractionException {
final CommentsInfoItemsCollector collector = new CommentsInfoItemsCollector(getServiceId());
final List<String> pageIds = page.getIds();
final String trackId = pageIds.get(0);
final String token = pageIds.get(1);
final JsonObject reviewsData = fetchReviewsData(trackId, token);
final JsonArray reviews = reviewsData.getArray("results");
for (final Object review : reviews) {
collector.commit(
new BandcampCommentsInfoItemExtractor((JsonObject) review, getUrl()));
}
if (!reviewsData.getBoolean("more_available")) {
return new InfoItemsPage<>(collector, null);
}
return new InfoItemsPage<>(collector,
new Page(List.of(trackId, getNextPageToken(reviews))));
}
private JsonObject fetchReviewsData(final String trackId, final String token)
throws ParsingException {
try {
return JsonUtils.toJsonObject(getDownloader().postWithContentTypeJson(
REVIEWS_API_URL,
Collections.emptyMap(),
JsonWriter.string().object()
.value("tralbum_type", "t")
.value("tralbum_id", trackId)
.value("token", token)
.value("count", 7)
.array("exclude_fan_ids").end()
.end().done().getBytes(StandardCharsets.UTF_8)).responseBody());
} catch (final IOException | ReCaptchaException e) {
throw new ParsingException("Could not fetch reviews", e);
}
}
private String getNextPageToken(final JsonArray reviews) throws ParsingException {
return reviews.stream()
.filter(JsonObject.class::isInstance)
.map(JsonObject.class::cast)
.map(review -> review.getString("token"))
.reduce((a, b) -> b) // keep only the last element
.orElseThrow(() -> new ParsingException("Could not get token"));
}
private String getTrackId() throws ParsingException {
final JsonObject pageProperties = JsonUtils.toJsonObject(
document.selectFirst("meta[name=bc-page-properties]")
.attr("content"));
return Long.toString(pageProperties.getLong("item_id"));
}
@Override
public boolean isCommentsDisabled() throws ExtractionException {
return BandcampExtractorHelper.isRadioUrl(getUrl());
}
}

View File

@ -0,0 +1,57 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.comments.CommentsInfoItemExtractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.stream.Description;
import javax.annotation.Nonnull;
import java.util.List;
public class BandcampCommentsInfoItemExtractor implements CommentsInfoItemExtractor {
private final JsonObject review;
private final String url;
public BandcampCommentsInfoItemExtractor(final JsonObject review, final String url) {
this.review = review;
this.url = url;
}
@Override
public String getName() throws ParsingException {
return getCommentText().getContent();
}
@Override
public String getUrl() {
return url;
}
@Nonnull
@Override
public List<Image> getThumbnails() throws ParsingException {
return getUploaderAvatars();
}
@Nonnull
@Override
public Description getCommentText() throws ParsingException {
return new Description(review.getString("why"), Description.PLAIN_TEXT);
}
@Override
public String getUploaderName() throws ParsingException {
return review.getString("name");
}
@Nonnull
@Override
public List<Image> getUploaderAvatars() {
return getImagesFromImageId(review.getLong("image_id"), false);
}
}

View File

@ -0,0 +1,293 @@
// Created by Fynn Godau 2019, licensed GNU GPL version 3 or later
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParser;
import com.grack.nanojson.JsonParserException;
import com.grack.nanojson.JsonWriter;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Element;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.Image.ResolutionLevel;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.exceptions.ReCaptchaException;
import org.schabi.newpipe.extractor.localization.DateWrapper;
import org.schabi.newpipe.extractor.utils.ImageSuffix;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.time.DateTimeException;
import java.time.ZonedDateTime;
import java.time.format.DateTimeFormatter;
import java.util.Collections;
import java.util.List;
import java.util.Locale;
import java.util.stream.Collectors;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import static org.schabi.newpipe.extractor.Image.HEIGHT_UNKNOWN;
import static org.schabi.newpipe.extractor.Image.WIDTH_UNKNOWN;
import static org.schabi.newpipe.extractor.utils.Utils.isNullOrEmpty;
import static org.schabi.newpipe.extractor.utils.Utils.replaceHttpWithHttps;
public final class BandcampExtractorHelper {
/**
* List of image IDs which preserve aspect ratio with their theoretical dimension known.
*
* <p>
* Bandcamp images are not always squares, so images which preserve aspect ratio are only used.
* </p>
*
* <p>
* One of the direct consequences of this specificity is that only one dimension of images is
* known at time, depending of the image ID.
* </p>
*
* <p>
* Note also that dimensions are only theoretical because if the image size is less than the
* dimensions of the image ID, it will be not upscaled but kept to its original size.
* </p>
*
* <p>
* IDs come from <a href="https://gist.github.com/f2k1de/06f5fd0ae9c919a7c3693a44ee522213">the
* GitHub Gist "Bandcamp File Format Parameters" by f2k1de</a>
* </p>
*/
private static final List<ImageSuffix> IMAGE_URL_SUFFIXES_AND_RESOLUTIONS = List.of(
// ID | HEIGHT | WIDTH
new ImageSuffix("10.jpg", HEIGHT_UNKNOWN, 1200, ResolutionLevel.HIGH),
new ImageSuffix("101.jpg", 90, WIDTH_UNKNOWN, ResolutionLevel.LOW),
new ImageSuffix("170.jpg", 422, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM),
// 180 returns the same image aspect ratio and size as 171
new ImageSuffix("171.jpg", 646, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM),
new ImageSuffix("20.jpg", HEIGHT_UNKNOWN, 1024, ResolutionLevel.HIGH),
// 203 returns the same image aspect ratio and size as 200
new ImageSuffix("200.jpg", 420, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM),
new ImageSuffix("201.jpg", 280, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM),
new ImageSuffix("202.jpg", 140, WIDTH_UNKNOWN, ResolutionLevel.LOW),
new ImageSuffix("204.jpg", 360, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM),
new ImageSuffix("205.jpg", 240, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM),
new ImageSuffix("206.jpg", 180, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM),
new ImageSuffix("207.jpg", 120, WIDTH_UNKNOWN, ResolutionLevel.LOW),
new ImageSuffix("43.jpg", 100, WIDTH_UNKNOWN, ResolutionLevel.LOW),
new ImageSuffix("44.jpg", 200, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM));
private static final String IMAGE_URL_APPENDIX_AND_EXTENSION_REGEX = "_\\d+\\.\\w+";
private static final String IMAGES_DOMAIN_AND_PATH = "https://f4.bcbits.com/img/";
public static final String BASE_URL = "https://bandcamp.com";
public static final String BASE_API_URL = BASE_URL + "/api";
private BandcampExtractorHelper() {
}
/**
* Translate all these parameters together to the URL of the corresponding album or track
* using the mobile API
*/
public static String getStreamUrlFromIds(final long bandId,
final long itemId,
final String itemType) throws ParsingException {
try {
final String jsonString = NewPipe.getDownloader().get(
BASE_API_URL + "/mobile/22/tralbum_details?band_id=" + bandId
+ "&tralbum_id=" + itemId + "&tralbum_type=" + itemType.charAt(0))
.responseBody();
return replaceHttpWithHttps(JsonParser.object().from(jsonString)
.getString("bandcamp_url"));
} catch (final JsonParserException | ReCaptchaException | IOException e) {
throw new ParsingException("Ids could not be translated to URL", e);
}
}
/**
* Fetch artist details from mobile endpoint.
* <a href="https://notabug.org/fynngodau/bandcampDirect/wiki/rewindBandcamp+%E2%80%93+Fetching+artist+details">
* More technical info.</a>
*/
public static JsonObject getArtistDetails(final String id) throws ParsingException {
try {
return JsonParser.object().from(NewPipe.getDownloader().postWithContentTypeJson(
BASE_API_URL + "/mobile/22/band_details",
Collections.emptyMap(),
JsonWriter.string()
.object()
.value("band_id", id)
.end()
.done()
.getBytes(StandardCharsets.UTF_8)).responseBody());
} catch (final IOException | ReCaptchaException | JsonParserException e) {
throw new ParsingException("Could not download band details", e);
}
}
/**
* Generate an image url from an image ID.
*
* <p>
* The image ID {@code 10} was chosen because it provides images wide up to 1200px (when
* the original image width is more than or equal this resolution).
* </p>
*
* <p>
* Other integer values are possible as well (e.g. 0 is a very large resolution, possibly the
* original); see {@link #IMAGE_URL_SUFFIXES_AND_RESOLUTIONS} for more details about image
* resolution IDs.
* </p>
*
* @param id the image ID
* @param isAlbum whether the image is the cover of an album or a track
* @return a URL of the image with this ID with a width up to 1200px
*/
@Nonnull
public static String getImageUrl(final long id, final boolean isAlbum) {
return IMAGES_DOMAIN_AND_PATH + (isAlbum ? 'a' : "") + id + "_10.jpg";
}
/**
* @return <code>true</code> if the given URL looks like it comes from a bandcamp custom domain
* or if it comes from <code>bandcamp.com</code> itself
*/
public static boolean isSupportedDomain(final String url) throws ParsingException {
// Accept all bandcamp.com URLs
if (url.toLowerCase().matches("https?://.+\\.bandcamp\\.com(/.*)?")) {
return true;
}
try {
// Test other URLs for whether they contain a footer that links to bandcamp
return Jsoup.parse(NewPipe.getDownloader().get(url).responseBody())
.getElementById("pgFt")
.getElementById("pgFt-inner")
.getElementById("footer-logo-wrapper")
.getElementById("footer-logo")
.getElementsByClass("hiddenAccess")
.text().equals("Bandcamp");
} catch (final NullPointerException e) {
return false;
} catch (final IOException | ReCaptchaException e) {
throw new ParsingException("Could not determine whether URL is custom domain "
+ "(not available? network error?)");
}
}
/**
* Whether the URL points to a radio kiosk.
* @param url the URL to check
* @return true if the URL matches {@code https://bandcamp.com/?show=SHOW_ID}
*/
public static boolean isRadioUrl(final String url) {
return url.toLowerCase().matches("https?://bandcamp\\.com/\\?show=\\d+");
}
public static DateWrapper parseDate(final String textDate) throws ParsingException {
try {
final ZonedDateTime zonedDateTime = ZonedDateTime.parse(textDate,
DateTimeFormatter.ofPattern("dd MMM yyyy HH:mm:ss zzz", Locale.ENGLISH));
return new DateWrapper(zonedDateTime.toOffsetDateTime(), false);
} catch (final DateTimeException e) {
throw new ParsingException("Could not parse date '" + textDate + "'", e);
}
}
/**
* Get a list of images from a search result {@link Element}.
*
* <p>
* This method will call {@link #getImagesFromImageUrl(String)} using the first non null and
* non empty image URL found from the {@code src} attribute of {@code img} HTML elements, or an
* empty string if no valid image URL was found.
* </p>
*
* @param searchResult a search result {@link Element}
* @return an unmodifiable list of {@link Image}s, which is never null but can be empty, in the
* case where no valid image URL was found
*/
@Nonnull
public static List<Image> getImagesFromSearchResult(@Nonnull final Element searchResult) {
return getImagesFromImageUrl(searchResult.getElementsByClass("art")
.stream()
.flatMap(element -> element.getElementsByTag("img").stream())
.map(element -> element.attr("src"))
.filter(imageUrl -> !isNullOrEmpty(imageUrl))
.findFirst()
.orElse(""));
}
/**
* Get all images which have resolutions preserving aspect ratio from an image URL.
*
* <p>
* This method will remove the image ID and its extension from the end of the URL and then call
* {@link #getImagesFromImageBaseUrl(String)}.
* </p>
*
* @param imageUrl the full URL of an image provided by Bandcamp, such as in its HTML code
* @return an unmodifiable list of {@link Image}s, which is never null but can be empty, in the
* case where the image URL has been not extracted (and so is null or empty)
*/
@Nonnull
public static List<Image> getImagesFromImageUrl(@Nullable final String imageUrl) {
if (isNullOrEmpty(imageUrl)) {
return List.of();
}
return getImagesFromImageBaseUrl(
imageUrl.replaceFirst(IMAGE_URL_APPENDIX_AND_EXTENSION_REGEX, "_"));
}
/**
* Get all images which have resolutions preserving aspect ratio from an image ID.
*
* <p>
* This method will call {@link #getImagesFromImageBaseUrl(String)}.
* </p>
*
* @param id the id of an image provided by Bandcamp
* @param isAlbum whether the image is the cover of an album
* @return an unmodifiable list of {@link Image}s, which is never null but can be empty, in the
* case where the image ID has been not extracted (and so equal to 0)
*/
@Nonnull
public static List<Image> getImagesFromImageId(final long id, final boolean isAlbum) {
if (id == 0) {
return List.of();
}
return getImagesFromImageBaseUrl(IMAGES_DOMAIN_AND_PATH + (isAlbum ? 'a' : "") + id + "_");
}
/**
* Get all images resolutions preserving aspect ratio from a base image URL.
*
* <p>
* Base image URLs are images containing the image path, a {@code a} letter if it comes from an
* album, its ID and an underscore.
* </p>
*
* <p>
* Images resolutions returned are the ones of {@link #IMAGE_URL_SUFFIXES_AND_RESOLUTIONS}.
* </p>
*
* @param baseUrl the base URL of the image
* @return an unmodifiable and non-empty list of {@link Image}s
*/
@Nonnull
private static List<Image> getImagesFromImageBaseUrl(@Nonnull final String baseUrl) {
return IMAGE_URL_SUFFIXES_AND_RESOLUTIONS.stream()
.map(imageSuffix -> new Image(baseUrl + imageSuffix.getSuffix(),
imageSuffix.getHeight(), imageSuffix.getWidth(),
imageSuffix.getResolutionLevel()))
.collect(Collectors.toUnmodifiableList());
}
}

View File

@ -0,0 +1,118 @@
// Created by Fynn Godau 2019, licensed GNU GPL version 3 or later
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParser;
import com.grack.nanojson.JsonParserException;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.kiosk.KioskExtractor;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.playlist.PlaylistInfoItem;
import org.schabi.newpipe.extractor.playlist.PlaylistInfoItemsCollector;
import javax.annotation.Nonnull;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.Collections;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_API_URL;
public class BandcampFeaturedExtractor extends KioskExtractor<PlaylistInfoItem> {
public static final String KIOSK_FEATURED = "Featured";
public static final String FEATURED_API_URL = BASE_API_URL + "/mobile/24/bootstrap_data";
public static final String MORE_FEATURED_API_URL
= BASE_API_URL + "/mobile/24/feed_older_logged_out";
private JsonObject json;
public BandcampFeaturedExtractor(final StreamingService streamingService,
final ListLinkHandler listLinkHandler,
final String kioskId) {
super(streamingService, listLinkHandler, kioskId);
}
@Override
public void onFetchPage(@Nonnull final Downloader downloader)
throws IOException, ExtractionException {
try {
json = JsonParser.object().from(getDownloader().postWithContentTypeJson(
FEATURED_API_URL,
Collections.emptyMap(),
"{\"platform\":\"\",\"version\":0}".getBytes(StandardCharsets.UTF_8))
.responseBody());
} catch (final JsonParserException e) {
throw new ParsingException("Could not parse Bandcamp featured API response", e);
}
}
@Nonnull
@Override
public String getName() throws ParsingException {
return KIOSK_FEATURED;
}
@Nonnull
@Override
public InfoItemsPage<PlaylistInfoItem> getInitialPage()
throws IOException, ExtractionException {
final JsonArray featuredStories = json.getObject("feed_content")
.getObject("stories")
.getArray("featured");
return extractItems(featuredStories);
}
private InfoItemsPage<PlaylistInfoItem> extractItems(final JsonArray featuredStories) {
final PlaylistInfoItemsCollector c = new PlaylistInfoItemsCollector(getServiceId());
for (int i = 0; i < featuredStories.size(); i++) {
final JsonObject featuredStory = featuredStories.getObject(i);
if (featuredStory.isNull("album_title")) {
// Is not an album, ignore
continue;
}
c.commit(new BandcampPlaylistInfoItemFeaturedExtractor(featuredStory));
}
final JsonObject lastFeaturedStory = featuredStories.getObject(featuredStories.size() - 1);
return new InfoItemsPage<>(c, getNextPageFrom(lastFeaturedStory));
}
/**
* Next Page can be generated from metadata of last featured story
*/
private Page getNextPageFrom(final JsonObject lastFeaturedStory) {
final long lastStoryDate = lastFeaturedStory.getLong("story_date");
final long lastStoryId = lastFeaturedStory.getLong("ntid");
final String lastStoryType = lastFeaturedStory.getString("story_type");
return new Page(
MORE_FEATURED_API_URL + "?story_groups=featured"
+ ':' + lastStoryDate + ':' + lastStoryType + ':' + lastStoryId
);
}
@Override
public InfoItemsPage<PlaylistInfoItem> getPage(final Page page)
throws IOException, ExtractionException {
final JsonObject response;
try {
response = JsonParser.object().from(
getDownloader().get(page.getUrl()).responseBody()
);
} catch (final JsonParserException e) {
throw new ParsingException("Could not parse Bandcamp featured API response", e);
}
return extractItems(response.getObject("stories").getArray("featured"));
}
}

View File

@ -0,0 +1,179 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageUrl;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampStreamExtractor.getAlbumInfoJson;
import static org.schabi.newpipe.extractor.utils.JsonUtils.getJsonData;
import static org.schabi.newpipe.extractor.utils.Utils.HTTPS;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParserException;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.PaidContentException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.playlist.PlaylistExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.streaminfoitem.BandcampPlaylistStreamInfoItemExtractor;
import org.schabi.newpipe.extractor.stream.Description;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import org.schabi.newpipe.extractor.stream.StreamInfoItemsCollector;
import java.io.IOException;
import java.util.Objects;
import java.util.List;
import javax.annotation.Nonnull;
public class BandcampPlaylistExtractor extends PlaylistExtractor {
/**
* An arbitrarily chosen number above which cover arts won't be fetched individually for each
* track; instead, it will be assumed that every track has the same cover art as the album,
* which is not always the case.
*/
private static final int MAXIMUM_INDIVIDUAL_COVER_ARTS = 10;
private Document document;
private JsonObject albumJson;
private JsonArray trackInfo;
private String name;
public BandcampPlaylistExtractor(final StreamingService service,
final ListLinkHandler linkHandler) {
super(service, linkHandler);
}
@Override
public void onFetchPage(@Nonnull final Downloader downloader)
throws IOException, ExtractionException {
final String html = downloader.get(getLinkHandler().getUrl()).responseBody();
document = Jsoup.parse(html);
albumJson = getAlbumInfoJson(html);
trackInfo = albumJson.getArray("trackinfo");
try {
name = getJsonData(html, "data-embed").getString("album_title");
} catch (final JsonParserException e) {
throw new ParsingException("Faulty JSON; page likely does not contain album data", e);
} catch (final ArrayIndexOutOfBoundsException e) {
throw new ParsingException("JSON does not exist", e);
}
if (trackInfo.isEmpty()) {
// Albums without trackInfo need to be purchased before they can be played
throw new PaidContentException("Album needs to be purchased");
}
}
@Nonnull
@Override
public List<Image> getThumbnails() throws ParsingException {
if (albumJson.isNull("art_id")) {
return List.of();
} else {
return getImagesFromImageId(albumJson.getLong("art_id"), true);
}
}
@Override
public String getUploaderUrl() throws ParsingException {
final String[] parts = getUrl().split("/");
// https: (/) (/) * .bandcamp.com (/) and leave out the rest
return HTTPS + parts[2] + "/";
}
@Override
public String getUploaderName() {
return albumJson.getString("artist");
}
@Nonnull
@Override
public List<Image> getUploaderAvatars() {
return getImagesFromImageUrl(document.getElementsByClass("band-photo")
.stream()
.map(element -> element.attr("src"))
.findFirst()
.orElse(""));
}
@Override
public boolean isUploaderVerified() throws ParsingException {
return false;
}
@Override
public long getStreamCount() {
return trackInfo.size();
}
@Nonnull
@Override
public Description getDescription() throws ParsingException {
final Element tInfo = document.getElementById("trackInfo");
if (tInfo == null) {
throw new ParsingException("Could not find trackInfo in document");
}
final Elements about = tInfo.getElementsByClass("tralbum-about");
final Elements credits = tInfo.getElementsByClass("tralbum-credits");
final Element license = document.getElementById("license");
if (about.isEmpty() && credits.isEmpty() && license == null) {
return Description.EMPTY_DESCRIPTION;
}
final StringBuilder sb = new StringBuilder();
if (!about.isEmpty()) {
sb.append(Objects.requireNonNull(about.first()).html());
}
if (!credits.isEmpty()) {
sb.append(Objects.requireNonNull(credits.first()).html());
}
if (license != null) {
sb.append(license.html());
}
return new Description(sb.toString(), Description.HTML);
}
@Nonnull
@Override
public InfoItemsPage<StreamInfoItem> getInitialPage() throws ExtractionException {
final StreamInfoItemsCollector collector = new StreamInfoItemsCollector(getServiceId());
for (int i = 0; i < trackInfo.size(); i++) {
final JsonObject track = trackInfo.getObject(i);
if (trackInfo.size() < MAXIMUM_INDIVIDUAL_COVER_ARTS) {
// Load cover art of every track individually
collector.commit(new BandcampPlaylistStreamInfoItemExtractor(
track, getUploaderUrl(), getService()));
} else {
// Pretend every track has the same cover art as the album
collector.commit(new BandcampPlaylistStreamInfoItemExtractor(
track, getUploaderUrl(), getThumbnails()));
}
}
return new InfoItemsPage<>(collector, null);
}
@Override
public InfoItemsPage<StreamInfoItem> getPage(final Page page) {
return null;
}
@Nonnull
@Override
public String getName() throws ParsingException {
return name;
}
}

View File

@ -0,0 +1,58 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import org.jsoup.nodes.Element;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.playlist.PlaylistInfoItemExtractor;
import javax.annotation.Nonnull;
import java.util.List;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromSearchResult;
public class BandcampPlaylistInfoItemExtractor implements PlaylistInfoItemExtractor {
private final Element searchResult;
private final Element resultInfo;
public BandcampPlaylistInfoItemExtractor(@Nonnull final Element searchResult) {
this.searchResult = searchResult;
resultInfo = searchResult.getElementsByClass("result-info").first();
}
@Override
public String getUploaderName() {
return resultInfo.getElementsByClass("subhead").text()
.split(" by")[0];
}
@Override
public String getUploaderUrl() {
return null;
}
@Override
public boolean isUploaderVerified() {
return false;
}
@Override
public long getStreamCount() {
final String length = resultInfo.getElementsByClass("length").text();
return Integer.parseInt(length.split(" track")[0]);
}
@Override
public String getName() {
return resultInfo.getElementsByClass("heading").text();
}
@Override
public String getUrl() {
return resultInfo.getElementsByClass("itemurl").text();
}
@Nonnull
@Override
public List<Image> getThumbnails() {
return getImagesFromSearchResult(searchResult);
}
}

View File

@ -0,0 +1,58 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.playlist.PlaylistInfoItemExtractor;
import org.schabi.newpipe.extractor.utils.Utils;
import javax.annotation.Nonnull;
import java.util.List;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
public class BandcampPlaylistInfoItemFeaturedExtractor implements PlaylistInfoItemExtractor {
private final JsonObject featuredStory;
public BandcampPlaylistInfoItemFeaturedExtractor(final JsonObject featuredStory) {
this.featuredStory = featuredStory;
}
@Override
public String getUploaderName() {
return featuredStory.getString("band_name");
}
@Override
public String getUploaderUrl() {
return null;
}
@Override
public boolean isUploaderVerified() {
return false;
}
@Override
public long getStreamCount() {
return featuredStory.getInt("num_streamable_tracks");
}
@Override
public String getName() {
return featuredStory.getString("album_title");
}
@Override
public String getUrl() {
return Utils.replaceHttpWithHttps(featuredStory.getString("item_url"));
}
@Nonnull
@Override
public List<Image> getThumbnails() {
return featuredStory.has("art_id")
? getImagesFromImageId(featuredStory.getLong("art_id"), true)
: getImagesFromImageId(featuredStory.getLong("item_art_id"), true);
}
}

View File

@ -0,0 +1,73 @@
// Created by Fynn Godau 2019, licensed GNU GPL version 3 or later
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParser;
import com.grack.nanojson.JsonParserException;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.kiosk.KioskExtractor;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import org.schabi.newpipe.extractor.stream.StreamInfoItemsCollector;
import javax.annotation.Nonnull;
import java.io.IOException;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_API_URL;
public class BandcampRadioExtractor extends KioskExtractor<StreamInfoItem> {
public static final String KIOSK_RADIO = "Radio";
public static final String RADIO_API_URL = BASE_API_URL + "/bcweekly/1/list";
private JsonObject json = null;
public BandcampRadioExtractor(final StreamingService streamingService,
final ListLinkHandler linkHandler,
final String kioskId) {
super(streamingService, linkHandler, kioskId);
}
@Override
public void onFetchPage(@Nonnull final Downloader downloader)
throws IOException, ExtractionException {
try {
json = JsonParser.object().from(
getDownloader().get(RADIO_API_URL).responseBody());
} catch (final JsonParserException e) {
throw new ExtractionException("Could not parse Bandcamp Radio API response", e);
}
}
@Nonnull
@Override
public String getName() throws ParsingException {
return KIOSK_RADIO;
}
@Nonnull
@Override
public InfoItemsPage<StreamInfoItem> getInitialPage() {
final StreamInfoItemsCollector collector = new StreamInfoItemsCollector(getServiceId());
final JsonArray radioShows = json.getArray("results");
for (int i = 0; i < radioShows.size(); i++) {
final JsonObject radioShow = radioShows.getObject(i);
collector.commit(new BandcampRadioInfoItemExtractor(radioShow));
}
return new InfoItemsPage<>(collector, null);
}
@Override
public InfoItemsPage<StreamInfoItem> getPage(final Page page) {
return null;
}
}

View File

@ -0,0 +1,96 @@
// Created by Fynn Godau 2019, licensed GNU GPL version 3 or later
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.localization.DateWrapper;
import org.schabi.newpipe.extractor.stream.StreamInfoItemExtractor;
import org.schabi.newpipe.extractor.stream.StreamType;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.List;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_URL;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.parseDate;
public class BandcampRadioInfoItemExtractor implements StreamInfoItemExtractor {
private final JsonObject show;
public BandcampRadioInfoItemExtractor(final JsonObject radioShow) {
show = radioShow;
}
@Override
public long getDuration() {
/* Duration is only present in the more detailed information that has to be queried
separately. Therefore, over 300 queries would be needed every time the kiosk is opened if we
were to display the real value. */
//return query(show.getInt("id")).getLong("audio_duration");
return 0;
}
@Nullable
@Override
public String getTextualUploadDate() {
return show.getString("date");
}
@Nullable
@Override
public DateWrapper getUploadDate() throws ParsingException {
return parseDate(getTextualUploadDate());
}
@Override
public String getName() throws ParsingException {
return show.getString("subtitle");
}
@Override
public String getUrl() {
return BASE_URL + "/?show=" + show.getInt("id");
}
@Nonnull
@Override
public List<Image> getThumbnails() {
return getImagesFromImageId(show.getLong("image_id"), false);
}
@Override
public StreamType getStreamType() {
return StreamType.AUDIO_STREAM;
}
@Override
public long getViewCount() {
return -1;
}
@Override
public String getUploaderName() {
// JSON does not contain uploader name
return "";
}
@Override
public String getUploaderUrl() {
return "";
}
@Override
public boolean isUploaderVerified() throws ParsingException {
return false;
}
@Override
public boolean isAd() {
return false;
}
}

View File

@ -0,0 +1,194 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_API_URL;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_URL;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImageUrl;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParser;
import com.grack.nanojson.JsonParserException;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Element;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.Image.ResolutionLevel;
import org.schabi.newpipe.extractor.MediaFormat;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ContentNotSupportedException;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.exceptions.ReCaptchaException;
import org.schabi.newpipe.extractor.linkhandler.LinkHandler;
import org.schabi.newpipe.extractor.playlist.PlaylistInfoItemsCollector;
import org.schabi.newpipe.extractor.stream.AudioStream;
import org.schabi.newpipe.extractor.stream.Description;
import org.schabi.newpipe.extractor.stream.StreamSegment;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
public class BandcampRadioStreamExtractor extends BandcampStreamExtractor {
private static final String OPUS_LO = "opus-lo";
private static final String MP3_128 = "mp3-128";
private JsonObject showInfo;
public BandcampRadioStreamExtractor(final StreamingService service,
final LinkHandler linkHandler) {
super(service, linkHandler);
}
static JsonObject query(final int id) throws ParsingException {
try {
return JsonParser.object().from(NewPipe.getDownloader()
.get(BASE_API_URL + "/bcweekly/1/get?id=" + id).responseBody());
} catch (final IOException | ReCaptchaException | JsonParserException e) {
throw new ParsingException("could not get show data", e);
}
}
@Override
public void onFetchPage(@Nonnull final Downloader downloader)
throws IOException, ExtractionException {
showInfo = query(Integer.parseInt(getId()));
}
@Nonnull
@Override
public String getName() throws ParsingException {
/* Select "subtitle" and not "audio_title", as the latter would cause a lot of
* items to show the same title, e.g. "Bandcamp Weekly".
*/
return showInfo.getString("subtitle");
}
@Nonnull
@Override
public String getUploaderUrl() throws ContentNotSupportedException {
throw new ContentNotSupportedException("Fan pages are not supported");
}
@Nonnull
@Override
public String getUrl() throws ParsingException {
return getLinkHandler().getUrl();
}
@Nonnull
@Override
public String getUploaderName() throws ParsingException {
return Jsoup.parse(showInfo.getString("image_caption")).getElementsByTag("a").stream()
.map(Element::text)
.findFirst()
.orElseThrow(() -> new ParsingException("Could not get uploader name"));
}
@Nullable
@Override
public String getTextualUploadDate() {
return showInfo.getString("published_date");
}
@Nonnull
@Override
public List<Image> getThumbnails() throws ParsingException {
return getImagesFromImageId(showInfo.getLong("show_image_id"), false);
}
@Nonnull
@Override
public List<Image> getUploaderAvatars() {
return Collections.singletonList(
new Image(BASE_URL + "/img/buttons/bandcamp-button-circle-whitecolor-512.png",
512, 512, ResolutionLevel.MEDIUM));
}
@Nonnull
@Override
public Description getDescription() {
return new Description(showInfo.getString("desc"), Description.PLAIN_TEXT);
}
@Override
public long getLength() {
return showInfo.getLong("audio_duration");
}
@Override
public List<AudioStream> getAudioStreams() {
final List<AudioStream> audioStreams = new ArrayList<>();
final JsonObject streams = showInfo.getObject("audio_stream");
if (streams.has(MP3_128)) {
audioStreams.add(new AudioStream.Builder()
.setId(MP3_128)
.setContent(streams.getString(MP3_128), true)
.setMediaFormat(MediaFormat.MP3)
.setAverageBitrate(128)
.build());
}
if (streams.has(OPUS_LO)) {
audioStreams.add(new AudioStream.Builder()
.setId(OPUS_LO)
.setContent(streams.getString(OPUS_LO), true)
.setMediaFormat(MediaFormat.OPUS)
.setAverageBitrate(100).build());
}
return audioStreams;
}
@Nonnull
@Override
public List<StreamSegment> getStreamSegments() throws ParsingException {
final JsonArray tracks = showInfo.getArray("tracks");
final List<StreamSegment> segments = new ArrayList<>(tracks.size());
for (final Object t : tracks) {
final JsonObject track = (JsonObject) t;
final StreamSegment segment = new StreamSegment(
track.getString("title"), track.getInt("timecode"));
// "track art" is the track's album cover
segment.setPreviewUrl(getImageUrl(track.getLong("track_art_id"), true));
segment.setChannelName(track.getString("artist"));
segments.add(segment);
}
return segments;
}
@Nonnull
@Override
public String getLicence() {
// Contrary to other Bandcamp streams, radio streams don't have a license
return "";
}
@Nonnull
@Override
public String getCategory() {
// Contrary to other Bandcamp streams, radio streams don't have categories
return "";
}
@Nonnull
@Override
public List<String> getTags() {
// Contrary to other Bandcamp streams, radio streams don't have tags
return Collections.emptyList();
}
@Override
public PlaylistInfoItemsCollector getRelatedItems() {
// Contrary to other Bandcamp streams, radio streams don't have related items
return null;
}
}

View File

@ -0,0 +1,60 @@
// Created by Fynn Godau 2021, licensed GNU GPL version 3 or later
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import org.jsoup.nodes.Element;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.playlist.PlaylistInfoItemExtractor;
import javax.annotation.Nonnull;
import java.util.List;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageUrl;
/**
* Extracts recommended albums from tracks' website
*/
public class BandcampRelatedPlaylistInfoItemExtractor implements PlaylistInfoItemExtractor {
private final Element relatedAlbum;
public BandcampRelatedPlaylistInfoItemExtractor(@Nonnull final Element relatedAlbum) {
this.relatedAlbum = relatedAlbum;
}
@Override
public String getName() throws ParsingException {
return relatedAlbum.getElementsByClass("release-title").text();
}
@Override
public String getUrl() throws ParsingException {
return relatedAlbum.getElementsByClass("album-link").attr("abs:href");
}
@Nonnull
@Override
public List<Image> getThumbnails() throws ParsingException {
return getImagesFromImageUrl(relatedAlbum.getElementsByClass("album-art").attr("src"));
}
@Override
public String getUploaderName() throws ParsingException {
return relatedAlbum.getElementsByClass("by-artist").text().replace("by ", "");
}
@Override
public String getUploaderUrl() throws ParsingException {
return null;
}
@Override
public boolean isUploaderVerified() throws ParsingException {
return false;
}
@Override
public long getStreamCount() throws ParsingException {
return -1;
}
}

View File

@ -0,0 +1,122 @@
// Created by Fynn Godau 2019, licensed GNU GPL version 3 or later
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.MetaInfo;
import org.schabi.newpipe.extractor.MultiInfoItemsCollector;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandler;
import org.schabi.newpipe.extractor.search.SearchExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.streaminfoitem.BandcampSearchStreamInfoItemExtractor;
import java.io.IOException;
import java.util.Collections;
import java.util.List;
import javax.annotation.Nonnull;
public class BandcampSearchExtractor extends SearchExtractor {
public BandcampSearchExtractor(final StreamingService service,
final SearchQueryHandler linkHandler) {
super(service, linkHandler);
}
@Nonnull
@Override
public String getSearchSuggestion() {
return "";
}
@Override
public boolean isCorrectedSearch() {
return false;
}
@Nonnull
@Override
public List<MetaInfo> getMetaInfo() throws ParsingException {
return Collections.emptyList();
}
public InfoItemsPage<InfoItem> getPage(final Page page)
throws IOException, ExtractionException {
final MultiInfoItemsCollector collector = new MultiInfoItemsCollector(getServiceId());
final Document d = Jsoup.parse(getDownloader().get(page.getUrl()).responseBody());
for (final Element searchResult : d.getElementsByClass("searchresult")) {
final String type = searchResult.getElementsByClass("result-info").stream()
.flatMap(element -> element.getElementsByClass("itemtype").stream())
.map(Element::text)
.findFirst()
.orElse("");
switch (type) {
case "ARTIST":
collector.commit(new BandcampChannelInfoItemExtractor(searchResult));
break;
case "ALBUM":
collector.commit(new BandcampPlaylistInfoItemExtractor(searchResult));
break;
case "TRACK":
collector.commit(new BandcampSearchStreamInfoItemExtractor(searchResult, null));
break;
default:
// don't display fan results ("FAN") or other things
break;
}
}
// Count pages
final Elements pageLists = d.getElementsByClass("pagelist");
if (pageLists.isEmpty()) {
return new InfoItemsPage<>(collector, null);
}
final Elements pages = pageLists.stream()
.map(element -> element.getElementsByTag("li"))
.findFirst()
.orElseGet(Elements::new);
// Find current page
int currentPage = -1;
for (int i = 0; i < pages.size(); i++) {
final Element pageElement = pages.get(i);
if (!pageElement.getElementsByTag("span").isEmpty()) {
currentPage = i + 1;
break;
}
}
// Search results appear to be capped at six pages
assert pages.size() < 10;
String nextUrl = null;
if (currentPage < pages.size()) {
nextUrl = page.getUrl().substring(0, page.getUrl().length() - 1) + (currentPage + 1);
}
return new InfoItemsPage<>(collector, new Page(nextUrl));
}
@Nonnull
@Override
public InfoItemsPage<InfoItem> getInitialPage() throws IOException, ExtractionException {
return getPage(new Page(getUrl()));
}
@Override
public void onFetchPage(@Nonnull final Downloader downloader)
throws IOException, ExtractionException {
}
}

View File

@ -0,0 +1,247 @@
// Created by Fynn Godau 2019, licensed GNU GPL version 3 or later
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageUrl;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.parseDate;
import static org.schabi.newpipe.extractor.utils.Utils.HTTPS;
import static org.schabi.newpipe.extractor.utils.Utils.replaceHttpWithHttps;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParserException;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.MediaFormat;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.PaidContentException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.LinkHandler;
import org.schabi.newpipe.extractor.localization.DateWrapper;
import org.schabi.newpipe.extractor.playlist.PlaylistInfoItemsCollector;
import org.schabi.newpipe.extractor.stream.AudioStream;
import org.schabi.newpipe.extractor.stream.Description;
import org.schabi.newpipe.extractor.stream.StreamExtractor;
import org.schabi.newpipe.extractor.stream.StreamType;
import org.schabi.newpipe.extractor.stream.VideoStream;
import org.schabi.newpipe.extractor.utils.JsonUtils;
import org.schabi.newpipe.extractor.utils.Utils;
import java.io.IOException;
import java.util.Collections;
import java.util.List;
import java.util.stream.Collectors;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
public class BandcampStreamExtractor extends StreamExtractor {
private JsonObject albumJson;
private JsonObject current;
private Document document;
public BandcampStreamExtractor(final StreamingService service, final LinkHandler linkHandler) {
super(service, linkHandler);
}
@Override
public void onFetchPage(@Nonnull final Downloader downloader)
throws IOException, ExtractionException {
final String html = downloader.get(getLinkHandler().getUrl()).responseBody();
document = Jsoup.parse(html);
albumJson = getAlbumInfoJson(html);
current = albumJson.getObject("current");
if (albumJson.getArray("trackinfo").size() > 1) {
// In this case, we are actually viewing an album page!
throw new ExtractionException("Page is actually an album, not a track");
}
if (albumJson.getArray("trackinfo").getObject(0).isNull("file")) {
throw new PaidContentException("This track is not available without being purchased");
}
}
/**
* Get the JSON that contains album's metadata from page
*
* @param html Website
* @return Album metadata JSON
* @throws ParsingException In case of a faulty website
*/
public static JsonObject getAlbumInfoJson(final String html) throws ParsingException {
try {
return JsonUtils.getJsonData(html, "data-tralbum");
} catch (final JsonParserException e) {
throw new ParsingException("Faulty JSON; page likely does not contain album data", e);
} catch (final ArrayIndexOutOfBoundsException e) {
throw new ParsingException("JSON does not exist", e);
}
}
@Nonnull
@Override
public String getName() throws ParsingException {
return current.getString("title");
}
@Nonnull
@Override
public String getUploaderUrl() throws ParsingException {
final String[] parts = getUrl().split("/");
// https: (/) (/) * .bandcamp.com (/) and leave out the rest
return HTTPS + parts[2] + "/";
}
@Nonnull
@Override
public String getUrl() throws ParsingException {
return replaceHttpWithHttps(albumJson.getString("url"));
}
@Nonnull
@Override
public String getUploaderName() throws ParsingException {
return albumJson.getString("artist");
}
@Nullable
@Override
public String getTextualUploadDate() {
return current.getString("publish_date");
}
@Nullable
@Override
public DateWrapper getUploadDate() throws ParsingException {
return parseDate(getTextualUploadDate());
}
@Nonnull
@Override
public List<Image> getThumbnails() throws ParsingException {
if (albumJson.isNull("art_id")) {
return List.of();
}
return getImagesFromImageId(albumJson.getLong("art_id"), true);
}
@Nonnull
@Override
public List<Image> getUploaderAvatars() {
return getImagesFromImageUrl(document.getElementsByClass("band-photo")
.stream()
.map(element -> element.attr("src"))
.findFirst()
.orElse(""));
}
@Nonnull
@Override
public Description getDescription() {
final String s = Utils.nonEmptyAndNullJoin("\n\n", current.getString("about"),
current.getString("lyrics"), current.getString("credits"));
return new Description(s, Description.PLAIN_TEXT);
}
@Override
public List<AudioStream> getAudioStreams() {
return Collections.singletonList(new AudioStream.Builder()
.setId("mp3-128")
.setContent(albumJson.getArray("trackinfo")
.getObject(0)
.getObject("file")
.getString("mp3-128"), true)
.setMediaFormat(MediaFormat.MP3)
.setAverageBitrate(128)
.build());
}
@Override
public long getLength() throws ParsingException {
return (long) albumJson.getArray("trackinfo").getObject(0)
.getDouble("duration");
}
@Override
public List<VideoStream> getVideoStreams() {
return Collections.emptyList();
}
@Override
public List<VideoStream> getVideoOnlyStreams() {
return Collections.emptyList();
}
@Override
public StreamType getStreamType() {
return StreamType.AUDIO_STREAM;
}
@Override
public PlaylistInfoItemsCollector getRelatedItems() {
final PlaylistInfoItemsCollector collector = new PlaylistInfoItemsCollector(getServiceId());
document.getElementsByClass("recommended-album")
.stream()
.map(BandcampRelatedPlaylistInfoItemExtractor::new)
.forEach(collector::commit);
return collector;
}
@Nonnull
@Override
public String getCategory() {
// Get first tag from html, which is the artist's Genre
return document.getElementsByClass("tralbum-tags").stream()
.flatMap(element -> element.getElementsByClass("tag").stream())
.map(Element::text)
.findFirst()
.orElse("");
}
@Nonnull
@Override
public String getLicence() {
/*
Tests resulted in this mapping of ints to licence:
https://cloud.disroot.org/s/ZTWBxbQ9fKRmRWJ/preview (screenshot from a Bandcamp artist's
account)
*/
switch (current.getInt("license_type")) {
case 1:
return "All rights reserved ©";
case 2:
return "CC BY-NC-ND 3.0";
case 3:
return "CC BY-NC-SA 3.0";
case 4:
return "CC BY-NC 3.0";
case 5:
return "CC BY-ND 3.0";
case 6:
return "CC BY 3.0";
case 8:
return "CC BY-SA 3.0";
default:
return "Unknown";
}
}
@Nonnull
@Override
public List<String> getTags() {
return document.getElementsByAttributeValue("itemprop", "keywords")
.stream()
.map(Element::text)
.collect(Collectors.toList());
}
}

View File

@ -0,0 +1,48 @@
// Created by Fynn Godau 2019, licensed GNU GPL version 3 or later
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_API_URL;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParser;
import com.grack.nanojson.JsonParserException;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.suggestion.SuggestionExtractor;
import org.schabi.newpipe.extractor.utils.Utils;
import java.io.IOException;
import java.util.Collections;
import java.util.List;
import java.util.stream.Collectors;
public class BandcampSuggestionExtractor extends SuggestionExtractor {
private static final String AUTOCOMPLETE_URL = BASE_API_URL + "/fuzzysearch/1/autocomplete?q=";
public BandcampSuggestionExtractor(final StreamingService service) {
super(service);
}
@Override
public List<String> suggestionList(final String query) throws IOException, ExtractionException {
final Downloader downloader = NewPipe.getDownloader();
try {
final JsonObject fuzzyResults = JsonParser.object().from(downloader
.get(AUTOCOMPLETE_URL + Utils.encodeUrlUtf8(query)).responseBody());
return fuzzyResults.getObject("auto").getArray("results").stream()
.filter(JsonObject.class::isInstance)
.map(JsonObject.class::cast)
.map(jsonObject -> jsonObject.getString("name"))
.distinct()
.collect(Collectors.toList());
} catch (final JsonParserException e) {
return Collections.emptyList();
}
}
}

Some files were not shown because too many files have changed in this diff Show More