Compare commits

...

2598 Commits
v0.9.8 ... dev

Author SHA1 Message Date
TobiGr fafd471606 [PeerTube] Fix test for like count
Number changed
2024-05-08 19:25:12 +02:00
TobiGr 4f477ad72b [PeerTube] Fix testing comment content
The comment is not available anymore.
2024-05-08 19:25:12 +02:00
Tobi 964e429978
Merge pull request #1166 from TeamNewPipe/dependabot/github_actions/peaceiris/actions-gh-pages-4
Bump peaceiris/actions-gh-pages from 3 to 4
2024-05-08 19:24:40 +02:00
dependabot[bot] 10c6965a28
Bump peaceiris/actions-gh-pages from 3 to 4
Bumps [peaceiris/actions-gh-pages](https://github.com/peaceiris/actions-gh-pages) from 3 to 4.
- [Release notes](https://github.com/peaceiris/actions-gh-pages/releases)
- [Changelog](https://github.com/peaceiris/actions-gh-pages/blob/main/CHANGELOG.md)
- [Commits](https://github.com/peaceiris/actions-gh-pages/compare/v3...v4)

---
updated-dependencies:
- dependency-name: peaceiris/actions-gh-pages
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-05-08 17:16:08 +00:00
TobiGr e54f38f5e7 [PeerTube] Fix test
UploaderName was changed by user
2024-05-08 19:13:49 +02:00
Tobi 5dd5c7a65b
Merge pull request #1173 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.8.5
Bump com.github.spotbugs:spotbugs-annotations from 4.8.3 to 4.8.5
2024-05-08 19:12:40 +02:00
Tobi 37438ff82f
Merge pull request #1174 from TeamNewPipe/dependabot/gradle/org.mozilla-rhino-1.7.15
Bump org.mozilla:rhino from 1.7.13 to 1.7.15
2024-05-08 19:02:57 +02:00
TobiGr bba3b6c69b version 0.24.0 2024-05-08 13:01:06 +02:00
TobiGr b40a5784ed Fix JDoc
YoutubePostLiveStreamDvrDashManifestCreator.java:90: error: unexpected end tag: </p>
2024-05-08 12:41:36 +02:00
dependabot[bot] c6da4004e2
Bump org.mozilla:rhino from 1.7.13 to 1.7.15
Bumps [org.mozilla:rhino](https://github.com/mozilla/rhino) from 1.7.13 to 1.7.15.
- [Release notes](https://github.com/mozilla/rhino/releases)
- [Changelog](https://github.com/mozilla/rhino/blob/master/RELEASE-NOTES.md)
- [Commits](https://github.com/mozilla/rhino/commits)

---
updated-dependencies:
- dependency-name: org.mozilla:rhino
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-05-06 09:43:05 +00:00
dependabot[bot] f26e84d39f
Bump com.github.spotbugs:spotbugs-annotations from 4.8.3 to 4.8.5
Bumps [com.github.spotbugs:spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.8.3 to 4.8.5.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.8.3...4.8.5)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-05-06 09:43:03 +00:00
Tobi ec3e8378c6
Merge pull request #1171 from TeamNewPipe/fix/jdoc
Fix JavaDocs and add Checkstyle exception for line length in JavaDocs' links
2024-04-23 21:22:12 +02:00
TobiGr 8d2a7a5281 [PeerTube] Fix test 2024-04-23 20:02:29 +02:00
TobiGr 7c29dbc965 Fix JDoc
Add Checkstyle exception for LineLength in JDoc links.

See https://github.com/TeamNewPipe/NewPipeExtractor/actions/runs/8804403691/job/24164703883
2024-04-23 19:55:51 +02:00
Stypox fbe9e6223a
Merge pull request #1168 from AudricV/yt_upd-cver-rm-keys-and-do-fixes
[YouTube] Update clients versions, restore access to some streams and more
2024-04-20 11:54:24 +02:00
Stypox 4e9e7cb29c
Improve assertTabsContain() to also check size 2024-04-20 11:48:36 +02:00
Stypox 9d0dd36034
[YouTube] Create constants for client names/versions 2024-04-20 11:43:54 +02:00
Stypox d4e6d22e64
[YouTube] Improve meta info code for review 2024-04-20 11:43:08 +02:00
AudricV 74bf000473
[YouTube] Update mocks 2024-04-11 18:59:11 +02:00
AudricV f9792cf3a9
[YouTube] Fix InteractiveTabbedHeader.testTabs test
YouTube now returns a Shorts tab for InteractiveTabbedHeader gaming channels,
which contains Shorts about the game of the topic channel but are not uploaded
on the game topic channel.

As this tab is already supported by the extractor, fetching a gaming topic
channel now returns a tab instead of none.

Channel name and channel URL of these Shorts needs to be set to null in a
separate commit, as Shorts on this tab do not have the topic channel as their
uploader.
2024-04-11 18:59:10 +02:00
AudricV f40fc0aa4f
[YouTube] Add support for new crisis meta info action data
The new action data can return multiple contact actions instead of only one,
which will be concatenated by a new line return.

This commit fixes tests of the CrisisResources test class of
YoutubeSearchExtractorTest.
2024-04-11 18:59:09 +02:00
AudricV 2a3c6f80d2
[YouTube] Fix YoutubeStreamExtractorRelatedMixTest 2024-04-10 21:19:03 +02:00
AudricV 657b4377aa
[YouTube] Fix YoutubeStreamExtractorDefaultTest tests 2024-04-10 21:19:03 +02:00
AudricV 7bf50bf1cb
[YouTube] Update Android client player parameters
YouTube disabled the effectiveness of the parameters which were used (the
player response we get redirects to another video), but new parameters which
work around Android's client integrity checks have been found.
2024-04-10 21:19:03 +02:00
AudricV 27dc1b1f50
[YouTube] Remove usage of API keys for InnerTube requests, bump versions
The API keys are not used anymore by official clients in almost all cases
(still used by the Android app until it gets a configuration) for all requests
we made.

Clients and device OS versions have been bumped to their latest stable version
known.

Methods and fields related to API keys have been renamed or deleted if they're
no longer relevant.
2024-04-10 21:19:02 +02:00
AudricV e380bb4bc3
[YouTube] Add missing prettyPrint query parameter to mixes continuations 2024-04-10 19:06:36 +02:00
Audric V 6c3c2e25d7
Merge pull request #1163 from AudricV/yt-fix_comments_extraction
[YouTube] Support new comments data
2024-04-10 18:19:59 +02:00
Stypox 02274d5395
[YouTube] Avoid XSS attacks in description or comments 2024-04-08 11:21:31 +02:00
Stypox 3f7b2653e3
[YouTube] Add YoutubeDescriptionHelperTest 2024-04-08 11:21:31 +02:00
Stypox a90237816a
[YouTube] Cleanup description helper
Remove unneeded isClose field, and make constants private
2024-04-08 11:21:31 +02:00
Stypox b80c3f5d51
[YouTube] Replace link text with accessibility label 2024-04-08 00:14:28 +02:00
Stypox 09732d6785
[YouTube] Add support for styles in attributed descriptions
Also refactor descriptions parsing.
2024-04-04 21:14:27 +02:00
AudricV 293c3e9e47
[YouTube] Support new A/B tested comments data
Also improve current comments code by removing outdated comment
renderer data.
2024-04-04 21:14:26 +02:00
Stypox e5b30ae8c3
Merge pull request #1151 from Profpatsch/localization-return-optional
LocaleCompat.forLanguageTag: return Optional if parsing fails
2024-03-29 13:50:48 +01:00
Stypox 23fc7aa209
Throw ParsingException instead of IllegalArg 2024-03-29 13:44:42 +01:00
Stypox fb468a23f4
Merge pull request #1142 from TeamNewPipe/peertube-v6
[PeerTube] Add support for PeerTube v6 features
2024-03-29 12:25:38 +01:00
Stypox 6589e2c15d
Merge pull request #1148 from Stypox/mediaccc-channel-tab-handler
[MediaCCC] Allow obtaining channel tab link handler
2024-03-28 13:45:05 +01:00
Tobi ad71864b23
Merge pull request #1162 from Stypox/fix-comment-description-npe
Make getCommentText @Nonnull
2024-03-27 20:04:17 +01:00
Stypox c57016b79b
Make getCommentText @Nonnull 2024-03-27 15:26:06 +01:00
Tobi adcc1f17ee
Merge pull request #1160 from TeamNewPipe/fix/tests
Fix some failing unit tests and detect new account termination messages for YouTube
2024-03-20 15:17:55 +01:00
TobiGr 51ddacc81d [SoundCloud] Fix SoundcloudSearchExtractorTest.NoNextPage
Search did not return no item at all, causing a NothingFoundException. New search query yields three items on first page
2024-03-20 15:10:39 +01:00
TobiGr 8392d50ba6 Update mocks for YoutubeChannelExtractorTest.NotAvailable 2024-03-20 14:59:44 +01:00
TobiGr aaccfecda8 [YouTube] Detect new account termination messages 2024-03-20 14:57:41 +01:00
TobiGr 73f0c63a9d [PeerTube] Fix tests for "What is PeerTube?" video 2024-03-20 14:44:06 +01:00
Tobi 896a55e319
Merge pull request #1139 from TeamNewPipe/dependabot/github_actions/actions/upload-artifact-4
Bump actions/upload-artifact from 3 to 4
2024-03-18 08:59:35 +01:00
dependabot[bot] e58fc652e0
Bump actions/upload-artifact from 3 to 4
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3 to 4.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-03-18 07:53:24 +00:00
Tobi e3f2c9aec7
Merge pull request #1153 from TeamNewPipe/dependabot/github_actions/actions/cache-4
Bump actions/cache from 3 to 4
2024-03-18 08:51:24 +01:00
Tobi 6b0fc14c04
Merge pull request #1156 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.10.2
Bump org.junit:junit-bom from 5.10.0 to 5.10.2
2024-03-18 08:46:13 +01:00
dependabot[bot] d579b608e5
Bump org.junit:junit-bom from 5.10.0 to 5.10.2
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.10.0 to 5.10.2.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.10.0...r5.10.2)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-02-05 09:15:26 +00:00
TobiGr fe47a4311f [PeerTube] Add test for segments and framesets 2024-01-29 10:22:06 +01:00
TobiGr 15e0e74b48 [PeerTube] Add support for stream frames/storyboards extraction
Implement PeerTubeStreamExtractor.getFrames()
2024-01-29 10:22:06 +01:00
dependabot[bot] da04eded5d
Bump actions/cache from 3 to 4
Bumps [actions/cache](https://github.com/actions/cache) from 3 to 4.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-01-18 09:32:42 +00:00
Profpatsch 7408173246 LocaleCompat.forLanguageTag: return Optional if parsing fails
It’s not obvious that the function will fail in some cases and throw
an `IllegalArgumentException`.

So instead of just failing if parsing fails, return an Optional that
all callers have to decide what to do (e.g. the YoutubeExtractor can
just ignore the locale in that case, like it does with most other
fields in the json if they are unexpected).
2024-01-07 14:31:34 +01:00
Stypox aaf3231fc7
[MediaCCC] Fix lambda link handler keeping reference to extractor
This caused problems in NewPipe, because extractors are not serializable, and well, keeping references to them is a bad idea anyway.
2023-12-30 23:23:19 +01:00
Stypox 137e924035
[MediaCCC] Add ChannelTabExtractorTest 2023-12-30 22:53:51 +01:00
Stypox cc9ade962e
[MediaCCC] Allow obtaining channel tab extractor from scratch
i.e. without needing to pass through the conference/channel extractor
This was needed because clients (like NewPipe) might rely on link handlers to hold as little data as possible, since they might be kept around for long or passed around in system transactions, so this commit allows obtaining a standalone link handler that does not hold a JsonObject within itself.
2023-12-30 22:53:27 +01:00
Stypox 3402cdb666
Merge pull request #1147 from petlyh/youtube-albums
[YouTube] Add Albums channel tab
2023-12-30 21:58:19 +01:00
petlyh 6dc25f7b97
[YouTube] Add Albums channel tab mocks 2023-12-30 14:46:39 +01:00
petlyh 4408e2d0ac
[YouTube] Add Albums channel tab 2023-12-30 14:01:30 +01:00
TobiGr 9ab932e394 Rename testDoNotAcceptNonURLs() -> assertDoNotAcceptNonURLs() 2023-12-29 16:38:11 +01:00
Stypox 9d66debf3c
Merge pull request #1132 from TeamNewPipe/dependabot/github_actions/actions/setup-java-4
Bump actions/setup-java from 3 to 4
2023-12-29 16:20:46 +01:00
Tobi 038ebdedc4
Merge pull request #1143 from petlyh/peertube-non-urls
Avoid PeerTube accepting non-URLs
2023-12-29 12:47:30 +01:00
TobiGr 61d237de02 [PeerTube] Test onAccept(String URL) in LinkHandlerFactories for non-URLs 2023-12-29 12:45:02 +01:00
petlyh 2b2c1546d1 Avoid PeerTube accepting non-URLs 2023-12-29 12:27:39 +01:00
Tobi 1e93b1dc20
Merge pull request #1135 from Stypox/yt-emergency-info
[YouTube] Implement emergency meta info
2023-12-29 12:01:40 +01:00
dependabot[bot] 3400af99b3
Bump actions/setup-java from 3 to 4
Bumps [actions/setup-java](https://github.com/actions/setup-java) from 3 to 4.
- [Release notes](https://github.com/actions/setup-java/releases)
- [Commits](https://github.com/actions/setup-java/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/setup-java
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-29 10:57:47 +00:00
Tobi 1f8a044462
Merge pull request #1138 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.8.3
Bump com.github.spotbugs:spotbugs-annotations from 4.8.0 to 4.8.3
2023-12-29 11:56:18 +01:00
dependabot[bot] 1470aa7303
Bump com.github.spotbugs:spotbugs-annotations from 4.8.0 to 4.8.3
Bumps [com.github.spotbugs:spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.8.0 to 4.8.3.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.8.0...4.8.3)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-29 10:53:28 +00:00
TobiGr 8f9ebdcb77 [PeerTube] Fix failing PeertubeTrendingLinkHandlerFactoryTest
The factory was updated in #1144
2023-12-29 11:52:19 +01:00
Stypox 1553931027
Merge pull request #1145 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.17.2
Bump org.jsoup:jsoup from 1.16.2 to 1.17.2
2023-12-29 11:27:01 +01:00
Stypox b2ec1b15fb
Merge pull request #1144 from dragfyre/patch-1
Update PeertubeTrendingLinkHandlerFactory.java
2023-12-29 11:21:08 +01:00
dependabot[bot] 151ee99da3
Bump org.jsoup:jsoup from 1.16.2 to 1.17.2
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup) from 1.16.2 to 1.17.2.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES.md)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.16.2...jsoup-1.17.2)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-29 09:47:00 +00:00
dragfyre 65e7bc5b95
Update PeertubeTrendingLinkHandlerFactory.java
correcting Peertube local trending api URL (per #10685 in main NewPipe repo); see https://docs.joinpeertube.org/api-rest-reference.html#tag/Video/operation/getVideos
2023-12-28 14:50:31 +07:00
Stypox f276caf54a
Release v0.23.1 2023-12-21 21:59:21 +01:00
Stypox fc54fb2fdb
Merge pull request #1140 from Stypox/yt-shorts-no-duration
[YouTube] Always return -1 as duration of Shorts returned inside reel items
2023-12-21 21:52:40 +01:00
Stypox 0518487d26
Fix SearchInfo's non-null MetaInfo being null when initialized or when an extraction error occurs (#1141)
The meta info might have been null either when SearchInfo is first initialized, or when extractor.getMetaInfo() throws an exception in getInfo().
This caused NewPipe to crash instead of showing a nice error in https://www.reddit.com/r/youtube/comments/184ttmw/what_exactly_about_blue_whales_has_youtube_so/.
2023-12-21 21:39:25 +01:00
Stypox 5b59a1a8c5
[YouTube] Move meta info extraction to separate file
YoutubeParsingHelper was longer than 2000 lines which caused checkstyle issues
2023-12-21 21:19:08 +01:00
Stypox b8e12dd76c
[YouTube] Implement emergency meta info
YouTube provides that meta info panel when users search for really sensitive content like suicide (e.g. "blue whale").

It contains:
- an encouragement as title (e.g. "We are with you")
- a phone number as action
- details about how to call the phone number (e.g. availability)
- an url pointing to the website of an association

Also add a test that just checks if a meta info is properly extracted
2023-12-21 21:19:08 +01:00
Stypox 83c1737f70
[YouTube] Update shorts channel tab mocks 2023-12-21 20:54:16 +01:00
Stypox 2938067c2c
[YouTube] Shorts don't provide a duration anymore 2023-12-21 20:41:01 +01:00
Stypox 91419ec6e8
Release v0.23.0 2023-12-10 15:59:32 +01:00
Audric V 678c98f24c
Merge pull request #1127 from AudricV/yt_improvements-and-fixes
[YouTube] Make some improvements and fixes
2023-12-09 14:07:31 +01:00
AudricV ec0194cfbf
[YouTube] Update mocks 2023-12-08 21:46:52 +01:00
AudricV 00a0f1a103
[YouTube] Add a blocking crisis resources bypass in search results test
This test only tests that search results are returned, when no content filters
are provided and crisis resources blocking search results should be returned.

Searches with blocking crisis resources and content filters should work too, as
the bypass has been implemented for them.
2023-12-08 21:46:49 +01:00
AudricV 06838d7245
[YouTube] Fix YoutubeSearchQHTest tests
As search parameters to bypass crisis resources blocking search results have
been implemented, they need to be added to search tests, in order to pass
them.
2023-12-08 21:46:49 +01:00
AudricV 642bb01388
[YouTube] Fix testSubscriberCount method of YoutubeChannelExtractorTest.CarouselHeader
The subscriber count is now lower than the expected count as some people
unsubscribed to the Sports system channel. The expected count has been so
lowered.
2023-12-08 21:46:49 +01:00
AudricV bedc9e5bc0
[YouTube] Remove Channels channel tab in tests 2023-12-08 21:46:48 +01:00
AudricV 5fa22ae25b
[YouTube] Enable and change testRelatedItems method of YoutubePlaylistExtractorTest.LearningPlaylist
This change makes the method test whether there is no more items instead of
being disabled.
2023-12-08 21:46:48 +01:00
AudricV 29dc7625f2
[YouTube] Fix YoutubeSearchExtractorTest.Suggestion
YouTube doesn't return anymore a suggestion for the query "algorythm", but does
for the query "on board ing" ("on boarding"). This search query is now used and
had to be URL-encoded.

URL encoding in the complete YoutubeSearchExtractorTest test class uses now
extractor's Util class instead of Java's URLDecoder class directly.
2023-12-08 21:46:48 +01:00
AudricV 56ab35423e
[YouTube] Fix potential NullPointerException in YoutubeSearchExtractor.getSearchSuggestion 2023-12-08 21:46:48 +01:00
AudricV 9dc1eab28c
[YouTube] Fix expected upload dates of StreamExtractor tests
YouTube is rolling out or A/B testing a new date format returned inside player
responses, which are precise to the second instead of the day.

This commit makes the StreamExtractor tests use these more precise dates.
2023-12-08 21:46:48 +01:00
AudricV ad3d187ac7
[YouTube] Fix testRelatedItems method of YoutubeStreamExtractorRelatedMixTest
This commit fixes the testRelatedItems test method by:
- accepting consent in the test class, in order to extract mixes in
recommendations;
- removing assertion of a music mix inside the recommendations, as YouTube
doesn't seem to return such mixes anymore, at least for the video used in the
test class.
2023-12-08 21:46:47 +01:00
AudricV e111814401
[YouTube] Fix YoutubeStreamExtractorDefaultTest.RatingsDisabledTest
Replace the video used in this test class with another one publicly available
and update the corresponding expected test values.

The test class's mocks will be updated in a different commit.
2023-12-08 21:46:47 +01:00
AudricV fc45941ead
[YouTube] Fix YoutubeChannelExtractorTest tests
- Change CarouselHeader test channel to Sports system one, as the Coachella one
doesn't return this channel header anymore;
- Fix InteractiveTabbedHeader test by checking whether the test's channel
description is not empty instead of containing some words, as it is changing
frequently.
2023-12-08 21:46:47 +01:00
AudricV 0bcb241c38
[YouTube] Fix expected data in YoutubeStreamExtractorRelatedMixTest
Video's title and tags have been changed by its uploader, so they have to be
updated.

Also make some package-private constants private, as they are not used outside
of the class, and remove unneeded test overrides.
2023-12-08 21:46:47 +01:00
AudricV 6ba8251be1
[YouTube] Bypass crisis resources blocking search results
These crisis resources are preventing search results to be returned. See
https://support.google.com/youtube/answer/10726080?hl=en for more info on them.

This commit changes search parameters to include the property allowing to show
search results.
2023-12-08 21:46:47 +01:00
AudricV 7dea2d0d27
[YouTube] Remove Channels channel tab support
This tab has been removed by YouTube.
2023-12-08 21:46:47 +01:00
AudricV 3782d9a02a
[YouTube] Support new A/B tested like data and avoid like count conversion from integer to long
Also make minor improvements to current like data extraction and remove
previous like count data support, as it is not returned anymore.
2023-12-08 21:46:46 +01:00
AudricV b71ce1123f
[YouTube] Extract only search results corresponding to a search type
YouTube returns sometimes videos inside channel search results. As we only want
results corresponding to the type we requested, this commits makes
YoutubeSearchExtractor ignoring non-requested search results we get, using the
extractor LinkHandler's first content filter value.

Also remove an unneeded exception throwing declaration in
YoutubeSearchExtractor.
2023-12-08 21:46:46 +01:00
AudricV ff8ed7247f
[YouTube] Switch to new consent cookie
Also move the documentation of the consent in its setter method in order to be
accessible publicly and improve it.
2023-12-08 21:46:46 +01:00
AudricV ec838d7421
[YouTube] Add missing prettyPrint query parameter to some test InnerTube requests
This query parameter for which its value is set to false was not added to two
requests made in test classes of YoutubeMixPlaylistExtractorTest.

Also remove an unneeded ParsingException exception throwing declaration in a
test method.
2023-12-08 21:46:46 +01:00
AudricV 2c941794c0
[YouTube] Add utcOffsetMinutes to all InnerTube payloads
This should make returned dates consistent between timezones and countries on
which the extractor is ran.

It was previously only set on YouTube Music search continuations.
2023-12-08 21:46:46 +01:00
AudricV d97c9e0db1
[YouTube] Improve payloads and URLs of InnerTube requests
For every InnerTube request:
- Always add a `request` object with the following properties:
  - "internalExperimentFlags" set to an empty array;
  - "useSsl" set to "true";
  - "lockedSafetyMode" set to "false".
- Use proper TODO comment to provide a way to enable restricted mode on every
request and add it on requests on which it wasn't present.

For YouTube Music:
- Remove alt query parameter, as it is not used anymore by the website;
- Add prettyPrint query parameter with false value on YouTube Music search
continuations.
2023-12-08 21:46:45 +01:00
AudricV 8a9ebcc373
[YouTube] Update InnerTube clients' version and devices' OS version and model 2023-12-08 21:46:45 +01:00
Tobi eac850ca10
Merge pull request #1114 from FineFindus/feat/comment-author-is-owner
[YouTube] Add channel owner to comments
2023-10-25 09:51:12 +02:00
Tobi 5ab1f784e8
Merge pull request #1117 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.16.2
Bump org.jsoup:jsoup from 1.16.1 to 1.16.2
2023-10-21 19:26:36 +02:00
dependabot[bot] 9d7bcba050
Bump org.jsoup:jsoup from 1.16.1 to 1.16.2
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup) from 1.16.1 to 1.16.2.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.16.1...jsoup-1.16.2)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-20 09:13:21 +00:00
dependabot[bot] e26065148a Bump com.github.spotbugs:spotbugs-annotations from 4.7.3 to 4.8.0
Bumps [com.github.spotbugs:spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.7.3 to 4.8.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.3...4.8.0)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-12 14:19:10 +02:00
FineFindus 34b05a0dda
feat(youtube/comments): support creator replies 2023-10-09 16:33:43 +02:00
TobiGr 0821f09114
Add missing mocks 2023-10-09 16:33:43 +02:00
FineFindus c1784a4bdb
[YouTube] Add channel owner to comments 2023-10-09 16:33:43 +02:00
TobiGr f9846352ea Fix wrong `@Nullable` annotation 2023-10-09 16:02:57 +02:00
Tobi d6f5cba6e2
Merge pull request #1111 from FineFindus/feat/creator-reply
Add `hasCreatorReply()` to CommentsInfoItem
2023-10-09 12:45:56 +02:00
TobiGr 9d63c75623 Add missing mocks 2023-10-09 11:24:39 +02:00
TobiGr d49f8411d7 [PeerTube] Implement CommentsInfoItemExtractor.hasCreatorReply() 2023-10-09 02:47:12 +02:00
Stypox bb132167d5
Merge pull request #1113 from AudricV/snd_fix-non-jpg-images
[SoundCloud] Fix extraction of non-JPG images
2023-10-02 19:40:57 +02:00
AudricV c98695fcea
[SoundCloud] Fix extraction of non-JPG images
Default image qualities were removed in image URLs with the jpg extension,
causing the addition of the image suffix to full non-JPG images URLs and so to
invalid image URLs.

Only the image quality name with its leading "-" character and the "."
character after the name is now removed and replaced by a string format
replaced itself with the image quality name for each quality.

As the image suffixes do not contain the image extension, the name of image
qualities lists has been adapted with these changes and some related comments
have been also improved.
2023-10-01 20:33:25 +02:00
AudricV ac00459c1a
Change requirement of image extensions in ImageSuffix class' Javadoc to a possibility
Some services may provide different image formats using the same suffix,
without we know what format the service provide. Enforcing an image extension
could so lead to provide invalid image URLs, like for SoundCloud PNG images
currently.

With this documentation change, it is now clear that users of this class decide
of whether they want to include image extensions in the suffix. The previous
behavior described in the Javadoc was not enforced.
2023-09-30 21:11:09 +02:00
FineFindus dd7b2d9798
feat(youtube/comments): support creator replies 2023-09-25 10:40:45 +02:00
Youssif Shaaban Alsager 917554acc4
[YouTube] Add support for ultralow audio formats (#1063) 2023-09-24 19:04:34 +02:00
Tobi 8b0068f8f4
Merge pull request #1110 from christian-2hu/chore/update-copyright
chore: Update copyright notices
2023-09-23 00:29:19 +02:00
Christian fc67d49f59 Update copyright notices
Update copyright notices to comply to GPLv3 and change NewPipe to NewPipe Extractor on some notices that were not updated.
2023-09-22 19:10:15 -03:00
Stypox 289db1178a
Merge pull request #1108 from AudricV/yt_refactor-js-usage
[YouTube] Refactor JavaScript usage and fix extraction of obfuscated signature deobfuscation function
2023-09-22 10:41:57 +02:00
AudricV 6ed22099a2
[YouTube] Update stream mocks 2023-09-21 21:59:34 +02:00
AudricV 714b141ecb
[YouTube] Catch any exception when extracting something from JavaScript's base player 2023-09-21 21:59:33 +02:00
AudricV 588c6a8422
[YouTube] Quote signature deobfuscation function name and add semicolon only where needed 2023-09-21 21:59:33 +02:00
AudricV 1fa85ec6ca
[YouTube] Add tests for signature timestamp extraction and signature deobfuscation function extraction and execution 2023-09-21 21:59:33 +02:00
AudricV a04bc320de
[YouTube] Convert signature timestamp to integer
The signature timestamp is used as a number by HTML5 clients, so it should be
used in the same way by the extractor too instead of being a string.

As the timestamp doesn't seem to exceed 5 digits, an integer is used to store
its value.
2023-09-21 21:59:32 +02:00
AudricV 7de3753a81
[YouTube] Refactor JavaScript player management API
This commit is introducing breaking changes.

For clients, everything is managed in a new class called
YoutubeJavaScriptPlayerManager:
- caching JavaScript base player code and its extracted code (functions and
variables);
- getting player signature timestamp;
- getting deobfuscated signatures of streaming URLs;
- getting streaming URLs with a throttling parameter deobfuscated, if
applicable.

The class delegates the extraction parts to external package-private classes:
- YoutubeJavaScriptExtractor, to extract and download YouTube's JavaScript base
player code: it always already present before and has been edited to mainly
remove the previous caching system and made it package-private;
- YoutubeSignatureUtils, for player signature timestamp and signature
deobfuscation function of streaming URLs, added in a recent commit;
- YoutubeThrottlingParameterUtils, which was originally
YoutubeThrottlingDecrypter, for throttling parameter of streaming URLs
deobfuscation function and checking whether this parameter is in a streaming
URL.

YoutubeJavaScriptPlayerManager caches and then runs the extracted code if it
has been executed successfully. The cache system of throttling parameters
deobfuscated values has been kept, its size can be get using the
getThrottlingParametersCacheSize method and can be cleared independently using
the clearThrottlingParametersCache method.

If an exception occurs during the extraction or the parsing of a function
property which is not related to JavaScript base player code fetching, it is
stored until caches are cleared, making subsequent failing extraction calls of
the requested function or property faster and consuming less resources, as the
result should be the same until the base player code changes.

All caches can be reset using the clearAllCaches method of
YoutubeJavaScriptPlayerManager.

Classes using JavaScript base player code and utilities directly (in the code
and its tests) have been also updated in this commit.
2023-09-21 21:59:32 +02:00
AudricV 6884d191cd
[YouTube] Add utility class around signatures and fix signature deobfuscation function extraction
The goal of this class is to decouple the extraction of signature timestamp and
signature deobfuscation function from YoutubeStreamExtractor.

The extraction of the signature deobfuscation function has been also adapted to
support the latest YouTube player versions.

This new class, YoutubeSignatureUtils, doens't store anything temporary such as
a copy of the player code, which has to be passed where required. It is not
public, as it will be used by a JavaScript player manager class in the future,
in order to handle in a better way fetching, caching and resetting cache of the
player code.
2023-09-21 21:59:26 +02:00
Tobi 3be76a6406
Merge pull request #1107 from Isira-Seneviratne/Locale_forLanguageTag
Use Locale.forLanguageTag() in tests
2023-09-18 16:48:30 +02:00
TobiGr 17790328cd Improve doc 2023-09-18 16:44:51 +02:00
Isira Seneviratne 4bc8ae7812 Use Locale.forLanguageTag() in tests 2023-09-18 08:59:13 +05:30
Tobi 90aed06a63
Merge pull request #1105 from TeamNewPipe/fix/bandcamp-streame-extractor-test
[Badcamp] Fix StreamExtractorTest
2023-09-18 01:49:04 +02:00
TobiGr cf49f4a31c [Badcamp] Fix StreamExtractorTest
The song was renamed and the URL changed
2023-09-17 23:58:07 +02:00
Tobi 7c7ceaceab
Merge pull request #1103 from TeamNewPipe/dependabot/github_actions/actions/checkout-4
Bump actions/checkout from 3 to 4
2023-09-17 22:42:01 +02:00
dependabot[bot] 72c475d944
Bump actions/checkout from 3 to 4
Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-09-05 10:00:13 +00:00
Stypox 1f08d28ae5
Merge pull request #889 from AudricV/multiple-images-support
Multiple images support
2023-08-13 11:35:11 +02:00
AudricV e8bfd20170
[MediaCCC] Apply changes in extractor tests
Also remove some public test methods modifiers.
2023-08-12 22:56:33 +02:00
AudricV 0292c4f3e8
[Bandcamp] Apply changes in extractor tests
Also remove some public test methods modifiers, add missing Test annotations on
old Junit 4 tests (and update them if needed), and use final in some places
where it was possible.

BandcampChannelExtractorTest.testLength has been removed as the test is always
true.
2023-08-12 22:56:32 +02:00
AudricV 2578f22054
[Bandcamp] Add utility test method to test images
This method, testImages(Collection<Image>), will use first the default image
collection test in DefaultTests and then will check that each image URL
contains f4.bcbits.com/img and ends with .jpg or .png.

To do so, a new non-instantiable final class has been added: BandcampTestUtils.
2023-08-12 22:56:32 +02:00
AudricV ba5315c72d
[PeerTube] Apply changes in extractor tests
Also remove some public test methods modifiers, add missing Test annotations on
old Junit 4 tests (and update them if needed), and improve some code.
2023-08-12 22:56:32 +02:00
AudricV 1d72bac53d
[SoundCloud] Apply changes in extractor tests 2023-08-12 22:56:32 +02:00
AudricV 93a210394d
[YouTube] Apply changes in extractor tests
Also remove some public test methods modifiers, add missing Test annotations on
old Junit 4 tests (and update them if needed), and use final in some places
where it was possible.
2023-08-12 22:56:31 +02:00
AudricV 2c436d428c
[YouTube] Add utility test method to test images in YoutubeTestsUtils
This method, testImages(Collection<Image>), will use first the default image
collection test in DefaultTests and then will check that each image URL
contains the string yt.

The JavaDoc of the class has been also updated to reflect the changes made in
it (it is now more general).
2023-08-12 22:56:31 +02:00
AudricV d381f3b70b
Update avatar, banners and thumbnail methods' name and apply changes in DefaultStreamExtractorTest 2023-08-12 22:56:31 +02:00
AudricV 434e885708
Add utility methods in ExtractorAsserts to check whether a collection is empty and to test image collections
Two new methods have been added in ExtractorAsserts to check if a collection is
empty:

- assertNotEmpty(String, Collection<?>), checking:
  - the non nullity of the collection;
  - its non emptiness (if that's not case, an exception will be thrown using
    the provided message).

- assertNotEmpty(Collection<?>), calling assertNotEmpty(String, Collection<?>)
  with null as the value of the string argument.

A new one has been added to this assertion class to check the contrary:
assertEmpty(Collection<?>), checking emptiness of the collection only if it is
not null.

Three new methods have been added in ExtractorAsserts as utility test methods
for image collections:

- assertContainsImageUrlInImageCollection(String, Collection<Image>), checking
that:
  - the provided URL and image collection are not null;
  - the image collection contains at least one image which has the provided
    string value as its URL (which is a string) property.

- assertContainsOnlyEquivalentImages(Collection<Image>, Collection<Image>),
  checking that:
  - both collections are not null;
  - they have the same size;
  - each image of the first collection has its equivalent in the second one.
    This means that the properties of an image in the first collection must be
    equal in an image of the second one.

- assertNotOnlyContainsEquivalentImages(Collection<Image>, Collection<Image>),
  checking that:
  - both collections are not null;
  - one of the following conditions is met:
    - they have different sizes;
    - an image of the first collection has not its equivalent in the second one.
      This means that the properties of an image in the first collection must
      be not equal in an image of the second one.

These methods will be used by services extractors tests (and default ones) to
test image collections.
2023-08-12 22:56:31 +02:00
AudricV 5158472852
Apply changes in DefaultTests and add utility method to test image lists
This new method, defaultTestImageList(List<Image), will check that the image
list is not null.

For each image, it will test that its URL is secure and its height and width
are more than or equal to their relevant unknown constants in the Image class
(HEIGHT_UNKNOWN and WIDTH_UNKNOWN).
2023-08-12 22:56:31 +02:00
AudricV 70fb3aa38e
Update BaseExtractorTests image methods' name
Also suppress unused warnings in BaseStreamExtractorTest, like it is done on
other BaseExtractorTests interfaces.
2023-08-12 22:56:30 +02:00
AudricV e16d521b7b
[MediaCCC] Apply changes in Extractors
Also remove usage of the conference logo as the banner of a conference, as it
is a logo and not a banner.
2023-08-12 22:56:30 +02:00
AudricV 306068a63b
[MediaCCC] Apply changes in InfoItemExtractors 2023-08-12 22:56:30 +02:00
AudricV 2f40861428
[MediaCCC] Add utility methods to get image lists from conference logos and streams
These three new methods, added in MediaCCCParsingHelper,
getImageListFromImageUrl(String), getThumbnailsFromStreamItem(JsonObject) and
getThumbnailsFromLiveStreamItem(JsonObject) (the last two are based on a common
method, getThumbnailsFromObject(JsonObject, String, String)), return an empty
list if the case no image URL could be extracted.

Images returned have their height and width unknown and a resolution level
depending on the image key of the JSON API response.
2023-08-12 22:56:30 +02:00
AudricV 71cda03c4c
[Bandcamp] Apply changes in Extractors 2023-08-12 22:56:29 +02:00
AudricV 7e01eaac33
[Bandcamp] Apply changes in InfoItemExtractors 2023-08-12 22:56:29 +02:00
AudricV 4b80d737a4
[Bandcamp] Add utility methods to get multiple images
Bandcamp images work with image IDs, which provide different resolutions.

Images on Bandcamp are not always squares, and some IDs respect aspect ratios
where some others not.

The extractor will only use the ones which preserve aspect ratio and will not
provide original images, for performance and size purposes.

Because of this aspect ratio preservation constraint, only one dimension will
be known at a time.

The image IDs with their respective dimension used are:

- 10: 1200w;
- 101: 90h;
- 170: 422h;
- 171: 646h;
- 20: 1024w;
- 200: 420h;
- 201: 280h;
- 202: 140h;
- 204: 360h;
- 205: 240h;
- 206: 180h;
- 207: 120h;
- 43: 100h;
- 44: 200h.

(Where w represents the width of the image and h the height of the image)

Note that these dimensions are theoretical because if the image size is less
than the dimensions of the image ID, it will be not upscaled but kept to its
original size.

All these resolutions are stored in a private static list of ThumbnailSuffixes
in BandcampExtractorHelper, in which the methods to get mutliple images have
been added:

- getImagesFromImageUrl(String): public method to get images from an image URL;
- getImagesFromImageId(long, boolean): public method to get images from an
  image ID;
- getImagesFromImageBaseUrl(String): private utility method to get images from
  the static list of ThumbnailSuffixes from a given image base URL, containing
  the path to the image, a "a" letter if it comes from an album, its ID and an
  underscore.

Some existing methods have been also edited:

- the documentation of getImageUrl(long, boolean) has been changed to reflect
  the Bandcamp images findings;
- getThumbnailUrlFromSearchResult has been renamed to
  getImagesFromSearchResult, and a documentation has been added to this method.

The method replaceHttpWithHttps of the Utils class has been also used in
BandcampExtractorHelper instead of doing manually what the method does.
2023-08-12 22:56:29 +02:00
AudricV 4e6fb368bc
[PeerTube] Apply changes in Extractors and remove usages of default avatar picture
The default avatar picture was used when no profile picture was found, but it
was removed and split in multiple images.

Thumbnails' size is not known, as this data is not provided by the API.
2023-08-12 22:56:29 +02:00
AudricV 0a6011a50e
[PeerTube] Apply changes in InfoItemExtractors
Also lower the visibility of attributes of channels and playlists InfoItems to
private.
2023-08-12 22:56:29 +02:00
AudricV 6f8331524b
[PeerTube] Add utility method to get thumbnails of playlists and videos
This method, getThumbnailsFromPlaylistOrVideoItem, has been added in
PeertubeParsingHelper and returns the two image variants for playlists and
videos.
2023-08-12 22:56:28 +02:00
AudricV 81c0d80a54
[PeerTube] Add utility methods to get avatars and banners of accounts and channels
Four new static methods have been added in PeertubeParsingHelper to do so:
- two public methods to get the corresponding image type:
  getAvatarsFromOwnerAccountOrVideoChannelObject(String, JsonObject) and
  getBannersFromAccountOrVideoChannelObject(String, JsonObject);
- two private methods as helper methods: getImagesFromAvatarsOrBanners(String,
  JsonObject, String, String) and getImagesFromAvatarOrBannerArray(String,
  JsonArray).
2023-08-12 22:56:28 +02:00
AudricV 31da5beb51
[SoundCloud] Apply changes in Extractors 2023-08-12 22:56:28 +02:00
AudricV a3a74cd566
[SoundCloud] Apply changes in InfoItemExtractors and return track user avatars as uploader avatars in SoundcloudStreamInfoItemExtractor 2023-08-12 22:56:28 +02:00
AudricV 7f818217d2
[SoundCloud] Add utility methods to get images from track JSON objects and image URLs
These new public and static methods, added in SoundcloudParsingHelper,
getAllImagesFromArtworkOrAvatarUrl(String) and
getAllImagesFromVisualUrl(String) (which call a common private method,
getAllImagesFromImageUrlReturned(String, List<ImageSuffix>, List<Image>)),
return an unmodifiable list of JPEG images containing almost every image
resolution provided by SoundCloud except the original size and the tiny
resolution (for artworks and avatars, as the image size is 20x20 for artworks
and 18x18 for avatars, so very close to or equal to the t20x20 resolution):

- for artworks and avatars:
  - mini: 16x16;
  - t20x20: 20x20;
  - small: 32x32;
  - badge: 47x47;
  - t50x50: 50x50;
  - t60x60: 60x60;
  - t67x67: 67x67;
  - large: 100x100;
  - t120x120: 120x120;
  - t200x200: 200x200;
  - t240x240: 240x240;
  - t250x250: 250x250;
  - t300x300: 300x300;
  - t500x500: 500x500.

- for visuals/user banners:
  - t1240x260: 1240x260;
  - t2480x520: 2480x520.

Duplicated code in two methods of SoundcloudParsingHelper
(getUsersFromApi(ChannelInfoItemsCollector, String) and
getStreamsFromApi(StreamInfoItemsCollector, String, boolean)) has been merged
into one common private method, getNextPageUrlFromResponseObject(JsonObject).
2023-08-12 22:56:28 +02:00
AudricV 266cd1f76b
[YouTube] Apply changes in YoutubeMusicSearchExtractor and split its InfoItemExtractors into separate classes
Splitting YoutubeMusicSearchExtractor's InfoItemExtractors into separate
classes (YoutubeMusicSongOrVideoInfoItemExtractor,
YoutubeMusicAlbumOrPlaylistInfoItemExtractor and
YoutubeMusicArtistInfoItemExtractor) allows to simplify
YoutubeMusicSearchExtractor,improves reading and applying changes to InfoItems
(no more losing at least quarter of a line due to indentations).

These InfoItems, in which the image changes have been applied, don't extend the
YouTube ones anymore, as most methods were overridden and the few ones that are
not don't apply in YouTube Music items responses, so it was useless to extend
them.

The code of YoutubeMusicSearchExtractor have been also improved a bit.
2023-08-12 22:56:27 +02:00
AudricV c1981ed54f
[YouTube] Apply changes in Extractors except YoutubeMusicSearchExtractor
Also improve a bit some code related to the changes.
2023-08-12 22:56:27 +02:00
AudricV 4cc99f9ce1
[YouTube] Apply changes in InfoItemExtractors except YouTube Music ones 2023-08-12 22:56:27 +02:00
AudricV adfad086ac
[YouTube] Add utility methods to get images from InfoItems and thumbnails arrays
Unmodifiable lists of Images are returned, parsed from a given YouTube
"thumbnails" JSON array.

These methods will be used in all YouTube extractors and InfoItems, as the
structures between content types (videos, channels, playlists, ...) are common.
2023-08-12 22:56:27 +02:00
AudricV d56b880cae
Replace avatar and thumbnail URLs attributes and methods to List<Image> in Infos 2023-08-12 22:56:26 +02:00
AudricV 9d8098576e
Replace avatar and thumbnail URLs attributes and methods to List<Image> in Extractors 2023-08-12 22:56:26 +02:00
AudricV 0f4a5a8184
Replace avatar and thumbnail URLs attributes and methods to List<Image> in InfoItemsCollectors 2023-08-12 22:56:26 +02:00
AudricV ca1d4a6fa4
Replace avatar and thumbnail URLs attributes and methods to List<Image> in InfoItemExtractors 2023-08-12 22:56:26 +02:00
AudricV 2f3ee8a3f2
Replace avatar and thumbnail URLs attributes and methods to List<Image> in InfoItems 2023-08-12 22:56:25 +02:00
AudricV 78ce65769f
Add an ImageSuffix class to the extractor
The goal of this utility class is to simply store suffixes which need to be
appended to image URLs, in order to get images at the suffix resolution.

This class contains four properties: the suffix (as a string), the height,
the width (as integers) and the estimated resolution level of the image
corresponding to the one represented by the suffix.
2023-08-12 22:56:25 +02:00
AudricV d85454186a
Add an Image class to the extractor
Objects of this serializable class contains four properties: a URL (as a
string), a width, a height (represented as integers) and an estimated
resolution level, which can be constructed from a given height.

Possible resolution levels are:
- UNKNOWN: for unknown heights or heights <= 0;
- LOW: for heights > 0 & < 175;
- MEDIUM: for heights >= 175 & < 720;
- HIGH: for heights >= 720.

Getters of these properties are available and the constructor needs these four
properties.
2023-08-12 22:56:25 +02:00
Stypox 7294675aea
Merge pull request #1093 from AudricV/yt_support-shorts-ui-playlists
[YouTube] Support Shorts UI in playlists
2023-08-12 11:11:36 +02:00
Stypox 93a90b816d
Merge pull request #1094 from AudricV/yt_support-more-channel-headers
[YouTube] Support more channel headers
2023-08-12 11:08:30 +02:00
Stypox 44b664af15
[YouTube] Simplify Optional chains in channel 2023-08-12 11:02:51 +02:00
AudricV 2f7bfd3e7f
[YouTube] Add mocks of interactiveTabbedHeaderRenderer channel header test 2023-08-08 19:12:29 +02:00
AudricV b147904571
[YouTube] Add test for interactiveTabbedHeaderRenderer channel header
This test uses the Minecraft game topic channel.
2023-08-08 19:12:28 +02:00
AudricV 1852031a0b
[YouTube] Support pageHeaderRenderer and interactiveTabbedHeaderRenderer channel headers
The addition of this support required to turn the isCarouselHeader boolean into
an enum containing all supported channel headers named HeaderType.

Also assert that the page has been fetched where needed to avoid
NullPointerExceptions when the channel page has been not fetched and remove the
getChannelHeaderJson method in YoutubeChannelExtractor, method for which its
code has been moved to its sole usage after the new headers support changes.
2023-08-08 19:12:27 +02:00
AudricV 698c710685
Do not require knowledge of uploader in default StreamInfoItems tests
This change is required as some services can return no uploader info, such as
YouTube for playlists with a Shorts UI.
2023-08-07 19:43:15 +02:00
AudricV 8237052ef5
Fix wrong assertion in assertNotEmpty
The non-null assertion was made on the exception message instead of the string
to check, causing a NullPointerException if the string to check was null.
2023-08-07 19:43:09 +02:00
AudricV 162c261577
[YouTube] Add mocks of the playlist with Shorts UI test 2023-08-07 19:07:53 +02:00
AudricV e2f4ee47b9
[YouTube] Add a playlist with Shorts UI test
The system Shorts videos uploads playlist of the YouTube official channel has
been chosen for this test.
2023-08-07 19:06:09 +02:00
AudricV e6f371fb94
[YouTube] Support Shorts UI in playlists
Also remove an outdated A/B test comment.
2023-08-07 19:01:08 +02:00
Stypox 7936987955
Merge pull request #1092 from Stypox/channel-tabs-improvements
Channel tabs code improvements
2023-08-06 21:44:10 +02:00
Stypox 6d2227111f
[YouTube] Assert that videos tab is ready after channel fetching 2023-08-06 21:14:57 +02:00
Stypox ee625c325c
Inherit from DefaultListExtractorTest in channel tab tests 2023-08-06 21:14:56 +02:00
Stypox 276c293889
Rename assertTabsContain 2023-08-06 21:14:56 +02:00
Stypox 9d3761a371
[YouTube] Directly use playlist collector in channel tabs wrapper
Note that this introduces a "Raw use of parameterized class 'InfoItemsPage'" warning, but it can be ignored since the type missing would be <InfoItem>, and StreamInfoItem extends InfoItem
2023-08-06 21:13:25 +02:00
Stypox 95a3cc0a17
Merge pull request #1083 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.10.0
Bump org.junit:junit-bom from 5.9.3 to 5.10.0
2023-08-06 18:58:27 +02:00
Stypox e34b4f1978
[YouTube] Avoid using Consumer 2023-08-06 13:02:31 +02:00
Stypox ef67c7cd74
[YouTube] Simplify usage of channel header json 2023-08-06 13:02:31 +02:00
Stypox a104cf3227
[YouTube] Fix docs in channel helper 2023-08-06 13:02:31 +02:00
Stypox bb47f05f89
Merge pull request #1091 from Stypox/update-mocks
[YouTube] Update stream mocks after #1087
2023-08-06 12:58:24 +02:00
Stypox 468bcc045d
[YouTube] Update mocks after #1087 2023-08-06 12:33:04 +02:00
Stypox 35f3a4ad01
Merge pull request #1082 from AudricV/channel-tabs-and-tags-support
Add support for channel tabs and channel tags
2023-08-06 12:21:42 +02:00
AudricV e7d64099a7
[YouTube] Update channel mocks and add channel tabs mocks 2023-08-06 12:15:06 +02:00
AudricV 684101c47d
[YouTube] Implement age-restricted channels support, link handlers and channels tabs and tags changes on tests
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV eaf2600ce0
[SoundCloud] Implement link handlers and channels tabs and tags changes on tests
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV 0ee2072de5
[PeerTube] Implement link handlers and channels tabs and tags changes on tests
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV d3801dd0e9
[MediaCCC] Implement link handlers and channels tabs and tags changes on tests
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV 8baec04611
[Bandcamp] Implement link handlers and channels tabs and tags changes on tests
Tests in BandcampChannelExtractorTest and BandcampChannelLinkHandlerFactoryTest
have been also fixed.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV e0ba29cd19
Add utility method to assert that given channel tabs are in the ones returned by a channel extractor
Only the first content filter of the ListLinkHandler instances provided is
used when collecting all channel tabs of the ListLinkHandler list, as channel
tabs implementations only use one content filter per ListLinkHandler instance.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV 18846baba7
Add tabs and tags methods in tests interfaces and annotate all methods with the Test JUnit annotation
These changes should help to detect tests as tests, when running a subset of
tests or all tests.
They should be also implemented in these interfaces' implementations (new and
existing ones).

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
ThetaDev c70a0e3543
Add a test for textual durations parsing using TimeAgoParser's patterns 2023-08-06 12:15:06 +02:00
AudricV 7366eab156
[YouTube] Add support for channel tabs and tags and age-restricted channels
Support of tags and videos, shorts, live, playlists and channels tabs has been
added for non-age restricted channels.

Age-restricted channels are now also supported and always returned the videos,
shorts and live tabs, accessible using system playlists. These tabs are the
only ones which can be accessed using YouTube's desktop website without being
logged-in.

The videos channel tab parameter has been updated to the one used by the
desktop website and when a channel extraction is fetched, this tab is returned
in the list of tabs as a cached one in the corresponding link handler.

Visitor data support per request has been added, as a valid visitor data is
required to fetch continuations with contents on the shorts tab. It is only
used in this case to enhance privacy.

A dedicated shorts UI elements (reelItemRenderers) extractor has been added,
YoutubeReelInfoItemExtractor. These elements do not provide the exact view
count, any uploader info (name, URL, avatar, verified status) and the upload
date.

All service's LinkHandlers are now using the singleton pattern and some code
has been also improved on the files changed.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:15:04 +02:00
AudricV 4586067934
Add utility method to parse textual durations using TimeAgoParser's patterns
This is required to parse duration of YouTube's reelItemRenderers, returned
only inside accessibility data.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:13:33 +02:00
AudricV d4bfe791ee
[SoundCloud] Add tabs support for users
Support of tracks, playlists and albums has been added for users.

Also add the declaration of the UnsupportedOperationException exception to the
service's LinkHandlers.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:13:32 +02:00
AudricV 6f7d1f079f
[Bandcamp] Add tabs support for artists
Support of tracks and albums has been added for artists.

Also use the singleton pattern and add the declaration of the
UnsupportedOperationException exception to the service's LinkHandlers and
improved some code in the files changed.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:12:19 +02:00
AudricV 1e8474b22d
[PeerTube] Add tabs support for accounts and video channels
Support of channels and videos has been added for accounts and support of
videos and playlists has been added for video channels.

The following changes have been also done:
- collectStreamsFrom method in PeertubeParsingHelper has been renamed to
collectItemsFrom;
- PeertubeChannelInfoItemExtractor.getStreamCount method has been fixed due to
ChannelExtractor's new inheritance;
- the declaration of the UnsupportedOperationException exception thrown has
been added to the service's LinkHandlers;
- a channel tab LinkHandlerFactory has been added,
PeertubeChannelTabLinkHandlerFactory;
- all service's LinkHandlers are now using properly the singleton pattern.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:12:15 +02:00
AudricV 652c2c8408
Add a ListLinkHandler which can be used to be returned from ChannelInfo.getTabs() when a specific tab's data has already been fetched
This new ListLinkHandler, ReadyChannelTabListLinkHandler, should help saving
clients data, energy and time by helping to reduce duplicate requests.

Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:11:12 +02:00
AudricV de823a6b68
Add an UnsupportedTabException exception class
This class makes easier for LinkHandlerFactory implementations to declare an
UnsupportedOperationException.
2023-08-06 12:11:12 +02:00
AudricV 76fb9dcdd7
Add UnsupportedOperationException to exceptions which can be thrown by getId and getUrl methods of LinkHandlerFactory and its base implementations
This change advertise to clients that channel tabs' link handler factories can
return an UnsupportedOperationException when a tab provided to them is
unsupported.
2023-08-06 12:11:12 +02:00
AudricV 946eb9bd91
Add structure of channel tags
Tags' getters and/or setters have been added in ChannelExtractor and
ChannelInfo to do so.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:11:12 +02:00
AudricV 356a888d6c
Add structure of channel tabs
This commit introduces the following breaking changes:

- Three new classes have been added:
  - ChannelTabExtractor, class extending ListExtractor<InfoItem>, which
  extracts InfoItems from a channel tab;
  - ChannelTabInfo extending ListInfo<InfoItem>, which extracts InfoItems from
  a ChannelTabExtractor and returns them as a ChannelTabInfo;
  - ChannelTabs, an immutable class containing all supported channel tabs.
- StreamingService implementations must implement new methods returning a
channel tab LinkHandlerFactory (getChannelTabsLHFactory) and a
ChannelTabExtractor (getChannelTabExtractor);
- ChannelExtractor inherits Extractor instead of ListExtractor<StreamInfoItem>
and ChannelInfo inherits Info instead of ListInfo<StreamInfoItem>;
- ChannelExtractor and ChannelInfo have now getters and/or setters of tabs.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:11:11 +02:00
Stypox 3faaf4301c
Merge pull request #1087 from AudricV/yt_js-extractor-improvements-and-fixes
[YouTube] Improve and fix YoutubeJavaScriptExtractor
2023-08-06 12:01:00 +02:00
Stypox 8fb6ba36fa
Merge pull request #1081 from TeamNewPipe/fix/sc/search-next-page
[SoundCloud] Detect whether there are any more search results
2023-08-06 11:49:35 +02:00
Stypox 2947257111
[SoundCloud] Properly calculate if results have finished 2023-08-06 11:38:22 +02:00
Stypox 485bfbca9d
[SoundCloud] Move try-catch inside getOffsetFromUrl 2023-08-06 11:35:37 +02:00
Stypox 7c70fef197
Merge pull request #1089 from TeamNewPipe/ccc
[media.ccc.de] Only extract kiosk live stream rooms if they are streaming
2023-08-06 10:12:04 +02:00
Stypox de0a9bb797
Merge pull request #1088 from FireMasterK/pseudo-rng
Replace cryptographically secure random with regular random
2023-08-05 11:37:49 +02:00
TobiGr 340095515d Make Kiosk IDs accessible if possible 2023-08-05 03:18:40 +02:00
TobiGr fe27d6a0ec [media.ccc.de] Only extract live streams if the conference is streaming 2023-08-05 01:53:43 +02:00
Kavin 25082d78b0
Replace SecureRandom with Random 2023-08-03 23:00:02 +01:00
TobiGr aa6c17dc77 [SoundCloud] Deduplicate some code 2023-08-03 14:41:30 +02:00
TobiGr 2fb9922a15 [SoundCloud] Detect whether there are any more search results
Add test for this edge case.
2023-08-03 14:37:13 +02:00
AudricV a3d160edab
[YouTube] Improve and fix YoutubeJavaScriptExtractor
- Enhance documentation;
- Fix the regular expression fallback on HTML embed watch page;
- Use HTML scripts tag search first instead of the regular expression approach,
now used as a last resort;
- Compile regular expressions only once, in order to improve the performance of
subsequent extraction calls when clearing the cache;
- Provide original exceptions when fetching or parsing pages on which the base
JavaScript's player could be found failed, allowing clients to detect network
errors when they are the cause of the failures for instance;
- Remove delegate method which was not taking a video ID and hardcoding one, as
we can provide the video ID in all cases or do not provide a video ID at worse;
- Rename and make extraction methods package-private, as they are not intended
to be used publicly.

These breaking internal changes have been applied where needed, in
YoutubeJavaScriptExtractorTest and YoutubeStreamExtractor (in which an unneeded
initStsFromPlayerJsIfNeeded call have been removed).
2023-08-02 23:05:08 +02:00
Stypox 5492343b8e
Pre-release v0.22.7 2023-08-02 20:02:29 +02:00
Tobi 9a59afbcf5
Merge pull request #1086 from AudricV/yt_no-exception-channels-without-banner
[YouTube] Don't throw an exception when there is no banner available on a channel
2023-08-02 18:56:26 +02:00
AudricV bb1ab166bf
[YouTube] Test that no banner is returned for carouselHeaderRenders 2023-08-01 22:19:43 +02:00
AudricV f1fa84b4e3
[YouTube] Don't throw an exception when there is no banner available on a channel
Channels may not have a banner, so no exception should be thrown if no banner
is found.
2023-08-01 12:40:20 +02:00
Tobi 39a911db9f
Merge pull request #1084 from AudricV/yt_android-403s-workaround-and-streams-tests-fixes
[YouTube] Workaround again 403 HTTP issues on the ANDROID InnerTube client and fix stream tests
2023-07-31 23:51:10 +02:00
dependabot[bot] bda3a3fc5d
Bump org.junit:junit-bom from 5.9.3 to 5.10.0
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.9.3 to 5.10.0.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.9.3...r5.10.0)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-07-24 09:53:05 +00:00
AudricV 522c78160f
[YouTube] Update stream tests mocks 2023-07-23 19:36:28 +02:00
AudricV 7528eb2bd9
[YouTube] Fix stream tests failures
- Fix testCheckAudioStreams test of
YoutubeStreamExtractorDefaultTest.AudioTrackLanguage test class, by updating
the excepted audio track name test to use the updated English audio track name
(audio track type info has been added on the video tested);
- Fix YoutubeStreamExtractorDefaultTest.PublicBroadcasterTest test class by
using a different video from a French and German public broadcast channel, as
the channel Dinge Erklärt – Kurzgesagt is not affiliated with a public
broadcast channel anymore;
- Fix YoutubeStreamExtractorLivestreamTest test class, by updating the excepted
name of the livestream to the current one.
2023-07-23 19:19:02 +02:00
AudricV 164c8e3abb
[YouTube] Workaround again 403 HTTP issues on the Android client by using new player parameters
These parameters are the only ones currently known to bypass 403 HTTP issues
related to failure of passing Android client integrity checks, as the ones of
stories (and the base of the shorts ones) do not work anymore, which may be
related to end of this format on the service.
2023-07-22 20:22:16 +02:00
Tobi 8ebbe1b7e1
Merge pull request #706 from FireMasterK/av1-itags
Add support for AV1 itags.
2023-07-22 13:31:45 +02:00
FireMasterK 6db0d116fe Add support for AV1 itags. 2023-07-22 13:23:44 +02:00
Tobi 986a3f7747
Merge pull request #1080 from TeamNewPipe/proguard-rules
Add proguard rules to README
2023-07-07 12:52:10 +02:00
Tobi d6132a9ed9
Merge pull request #1076 from AudricV/yt-fix-throttling-parameter-regex
[YouTube] Support multiple declarations for throttling parameter function name array
2023-07-07 12:51:26 +02:00
Tobi 87eb29a25d Add proguard rules to README 2023-07-05 08:25:23 +02:00
AudricV 4e22c5ee87
[YouTube] Support multiple declarations for throttling parameter function name array
Also moved the corresponding regex parts in static constants for easier future
modifications
2023-06-26 15:25:53 +02:00
Kavin d961d349c3
[YouTube] Check whether player responses are valid for all InnerTube clients used (#1070)
Co-authored-by: Audric V <74829229+AudricV@users.noreply.github.com>
2023-06-18 21:54:52 +02:00
ThetaDev ad97f08048
[YouTube] Fix parsing short relative date formats (English only) (#1068) 2023-06-18 21:41:29 +02:00
Tobi d294ccb433
Merge pull request #1071 from TeamNewPipe/feat/ServiceList
Init services at the correct place
2023-06-17 20:50:24 +02:00
Tobi 5d112aa772
Merge pull request #1074 from TeamNewPipe/imp/media-formats
Add more audio media formats and MediaFormat.getAllForMimeType(mimeType)
2023-06-17 20:49:38 +02:00
TobiGr 0e01d90562 Do not init services while creating services array 2023-06-17 18:08:09 +02:00
TobiGr 5809904cf7 Add annotations to MediaFormat 2023-06-17 18:01:54 +02:00
TobiGr 53dfd871e2 Add more audio media formats and MediaFormat.getAllForMimeType(mimeType) 2023-06-17 18:01:29 +02:00
Stypox 5eb886243f
Merge pull request #1072 from TeamNewPipe/fix/bandcamp_test
Fix BandcampRadioStreamExtractorTest.testGetAudioStreams()
2023-06-09 17:35:35 +02:00
TobiGr 19ce06fe00 Fix BandcampRadioStreamExtractorTest.testGetAudioStreams() 2023-06-06 23:07:56 +02:00
Audric V 533121fb81
Merge pull request #1045 from Theta-Dev/fix/trending-video-tab
[YouTube] Extract trends from A/B tested "Videos" tab and fix extraction of trends name from A/B tested new title design
2023-05-19 11:22:49 +02:00
Audric V 92a0024424
Merge pull request #1052 from TeamNewPipe/peertube/fix/nested-comment-replies
[PeerTube] Fix multi level comment replies
2023-05-18 18:49:06 +02:00
Tobi 3c036a9c03
Merge pull request #1061 from ChunkyProgrammer/develop/extract-yt-playlist-descrption
Extract playlists description
2023-05-17 22:43:17 +02:00
TobiGr c70bb83801 [Bandcamp] Implement PlaylistExtractor.getDescsription() 2023-05-15 15:23:03 +02:00
TobiGr ca0ce00753 Add PlaylistInfoItem.getDescription() and PlaylistInfoItemExtractor.getDescription()
[PeerTube] Implement the corresponding extractor method.
TODO: add tests
2023-05-12 01:43:59 +02:00
TobiGr b218bf69bd Implement PlaylistInfo.getDescription()
Implement PlaylistExtractor.getDescription() for PeerTube and SoundCloud.
Anotate   PlaylistExtractor.getDescription() as Nonnull
2023-05-12 00:44:10 +02:00
chunky programmer 81f29116ba switch from string to Description object 2023-05-11 00:36:57 -04:00
chunky programmer e147867d41 Add tests 2023-05-11 00:00:34 -04:00
chunky programmer 5ab6cd7420 Extract YouTube playlist description 2023-05-11 00:00:22 -04:00
TobiGr d358ba1c41 Improve PeertubeCommentsInfoItemExtractor constructor 2023-05-07 22:55:26 +02:00
TobiGr aff3e795f8 [PeerTube] Fix multi level comment replies 2023-05-07 22:49:14 +02:00
ThetaDev 3673d4ae01
fix: YouTube trending name extraction 2023-05-03 21:16:35 +02:00
ThetaDev 0addb98cd7
tests: update mocks 2023-05-03 21:16:35 +02:00
ThetaDev 24eba62305
fix: extract YouTube trends from new "Videos" tab 2023-05-03 21:16:23 +02:00
Kavin a9ca5c49e4
Merge pull request #1056 from AudricV/yt-improve-search-suggestions-extraction
[YouTube] Switch to new search suggestion domain and improve error handling
2023-05-02 20:17:48 +01:00
Stypox 00408db959
Merge pull request #1058 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.16.1
Bump org.jsoup:jsoup from 1.15.4 to 1.16.1
2023-05-01 19:59:02 +02:00
dependabot[bot] 108f8a7a17
Bump org.jsoup:jsoup from 1.15.4 to 1.16.1
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup) from 1.15.4 to 1.16.1.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.4...jsoup-1.16.1)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-05-01 09:58:32 +00:00
Tobi 11565db17f
Merge pull request #1055 from AudricV/peertube-no-description-channels
[PeerTube] Don't return "No description" when there is no description for a channel or an account
2023-04-30 23:38:10 +02:00
AudricV cf6040ddb3
Update YoutubeSuggestionExtractorTest mocks 2023-04-30 19:53:44 +02:00
AudricV 593122342f
[YouTube] Improve YoutubeSuggestionExtractorTest
- Remove useless concatenation on the downloader path;
- Remove unneeded public test modifier;
- Update license header;
- Specify the service class tested instead of the generic class.
2023-04-30 19:53:43 +02:00
AudricV e923fca440
[YouTube] Switch to new search suggestion domain and improve error handling
- Switch to the new domain used by YouTube for search suggestions,
suggestqueries-clients6.youtube.com, and add the xhr query parameter with the
t value, to allow getting responses without requiring trim;
- Use the Java 8 Stream API to collect search suggestions and improve invalid
response detection by checking whether the content type of the response
returned is JSON;
- Move the licence header at the top of the file.
2023-04-30 19:53:42 +02:00
AudricV 945165a3c0
[PeerTube] Don't return "No description" when there is no description for a channel or an account
When a description is missing, no description should be returned, even the ones
indicating there is no description. This behavior is represented by a null
return instead.

Also update PeertubeAccountExtractorTest to reflect these changes.
2023-04-30 18:41:38 +02:00
Stypox d61dc27406
Merge pull request #1054 from TeamNewPipe/fix-ci-variable
Make sure CI is not vulnerable to bash word splitting
2023-04-27 18:52:07 +02:00
Stypox 51cba5a287
Make sure CI is not vulnerable to bash word splitting
See https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-an-intermediate-environment-variable
2023-04-26 18:36:25 +02:00
Stypox f819c2fe34
Merge pull request #1053 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.3
Bump org.junit:junit-bom from 5.9.2 to 5.9.3
2023-04-26 15:42:19 +02:00
dependabot[bot] 4b92c5a34d
Bump org.junit:junit-bom from 5.9.2 to 5.9.3
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.9.2 to 5.9.3.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.9.2...r5.9.3)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-04-26 09:57:57 +00:00
Stypox 2deb023da4
Merge pull request #1050 from Theta-Dev/fix/channel-carousel-header
[YouTube] Add support for CarouselHeaderRenderer
2023-04-25 15:17:31 +02:00
ThetaDev 4aada7f91b refactor: rename carousel header channel test 2023-04-21 22:48:37 +02:00
Tobi b2b426048f
Merge pull request #1051 from Theta-Dev/fix/ytm-client-version-rg
[YouTube] fix: set musicClientVersion regex capture group
2023-04-16 23:48:19 +02:00
ThetaDev 47aa9fed40 fix: set musicClientVersion regex capture group 2023-04-16 19:25:05 +02:00
ThetaDev 20370395c5 fix: add support for CarouselHeaderRenderer 2023-04-16 17:40:13 +02:00
Stypox 7dba6e3891
Merge pull request #1033 from petlyh/bandcamp-paywalled-content
[Bandcamp] Handle paywalled tracks
2023-04-12 13:04:26 +02:00
petlyh e6aad117e7
[Bandcamp] Throw PaidContentException on paywalled albums 2023-04-03 19:27:09 +02:00
TobiGr 8495ad619e Bump version to 0.22.6 2023-04-02 22:35:02 +02:00
fynngodau 69705138e4
[Bandcamp] Fix extraction of related playlist items URL (#1047)
Small change in HTML structure
2023-04-02 22:24:29 +02:00
Björn Sigurbergsson 1b6fe5edd6
[YouTube] Fix ParsingException when comments are unavailable in a video (#1040)
Co-authored-by: bjs <bjs@elect-it.com>
Co-authored-by: Audric V. <74829229+AudricV@users.noreply.github.com>
Co-authored-by: Kavin <20838718+FireMasterK@users.noreply.github.com>
2023-03-30 19:58:06 +02:00
ThetaDev 8d1303e18f
Add track types to audio streams (#1041) 2023-03-28 00:02:20 +02:00
Tobi 44ae139d33
Merge pull request #1034 from AudricV/peertube-fix-commentsinfo-test
[PeerTube] Fix comments info test
2023-03-20 19:30:35 +01:00
Tobi d5e9df5381
Merge pull request #1038 from TeamNewPipe/docs-jdk-11
Update JDK to 11 in JDoc workflow
2023-03-20 19:25:21 +01:00
Tobi 0486ad7a08
Update JDK to 11 in JDoc workflow 2023-03-06 18:45:11 +01:00
AudricV 80a6fc2c63
[PeerTube] Fix testGetCommentsFromCommentsInfo test of PeertubeCommentsExtractorTest.Default
The tested comment has been removed, so it couldn't be found in the comments
list.

This comment has been replaced by a new one from the current comments of the
video.

Also, in the parent class PeertubeCommentsExtractorTest, final has been used as
much as possible and for-each loops of lists have been replaced by their
forEach method or the Stream API, in order to simplify code.
2023-03-04 16:49:10 +01:00
petlyh 5a9b6ed2e3
[Bandcamp] Support loading additional comments (#1030) 2023-03-04 14:01:06 +01:00
Stypox 6bdd698c25
Merge pull request #1026 from AudricV/audio-streams-descriptive-and-locale-properties
Add descriptive and locale properties to audio streams
2023-03-01 11:15:46 +01:00
Stypox 19e4b216c9
Merge pull request #1032 from AudricV/yt_fix-comments-hashtags-links-extraction
[YouTube] Fix hashtags links extraction and escape HTML links
2023-03-01 10:47:37 +01:00
Stypox b1298490c0
Merge pull request #1029 from AudricV/yt_fix-no-views-extraction-playlist-items
[YouTube] Fix partial non-extraction of "No views" string in stream items
2023-03-01 10:46:52 +01:00
petlyh 9dc1832733
[Bandcamp] Handle paywalled tracks 2023-02-28 17:51:30 +01:00
fynngodau 3fdb6ee476
Merge pull request #1031 from petlyh/bandcamp-fix-radio-comments
[Bandcamp] Show comments as disabled on radio streams
2023-02-27 20:50:12 +01:00
AudricV bd79b921e8
[YouTube] Refactor the code to get stream items' view count
This refactoring avoids code duplication as much as possible.
2023-02-27 10:25:46 +01:00
AudricV 51f9b39953
[YouTube] Fix partial non-extraction of no views string in stream items
As the "No views" string is returned in the case there is no view on a video, a
number cannot be parsed in this case, so -1 was returned.

This string is now detected in all methods to get the view count of a stream.
2023-02-27 10:18:45 +01:00
AudricV 95b3f5e391
[MediaCCC] Test audio language property extraction 2023-02-26 19:06:18 +01:00
AudricV 30a0f8c510
[MediaCCC] Extract audio language property for single language audio tracks 2023-02-26 19:06:18 +01:00
AudricV 7f0269c4c7
[YouTube] Edit YoutubeStreamExtractorDefaultTest.AudioTrackLanguage to test audio locale property
The Hindi audio track language presence test has been changed from audio track
label to audio locale.
2023-02-26 19:06:18 +01:00
AudricV 034f82dae7
[YouTube] Test language and descriptive audio in YoutubeDashManifestCreatorsTest 2023-02-26 19:06:17 +01:00
AudricV 05e8cb39f7
[YouTube] Add language and descriptive audio properties to DASH manifests 2023-02-26 19:06:17 +01:00
AudricV bf30d70152
[YouTube] Add descriptive audio test
This test uses video TjxC-evzxdk.

Also improve a bit YoutubeStreamExtractorDefaultTest.AudioTrackLanguage test.
2023-02-26 19:06:17 +01:00
AudricV 76b7c19c5d
[YouTube] Extract whether a track is a descriptive audio and audio locale when available
Also use audio track setters only for audio itags.
2023-02-26 19:06:17 +01:00
AudricV 3bb5eeef30
[YouTube] Add descriptive and locale audio support in ItagItem 2023-02-26 19:06:16 +01:00
AudricV 14bf3fb05b
Add ability to know the locale of an audio stream
Getting audio tracks locales by parsing their ID or their label, should not be
done by clients, but by the extractor.

This commit adds the ability to store the Locale of an AudioStream, which is
used to compare similar AudioStreams (in the equalStats method).
2023-02-26 19:06:16 +01:00
AudricV f92426560c
Add descriptive audio properties
Also improve AudioStream's audio language documentation
2023-02-26 19:06:16 +01:00
AudricV a63f289667
[YouTube] Update mocks of YoutubeCommentsExtractorTest.FormattingTest 2023-02-26 18:50:07 +01:00
AudricV 9483dcd9fa
[YouTube] Update mocks of YoutubeCommentsExtractorTest.RepliesTest 2023-02-26 18:43:36 +01:00
AudricV 1556adbb2d
[YouTube] Fix hashtags links extraction and escape text in attribute descriptions + HTML links
webCommandMetadata object is contained inside a commandMetadata one, so it is
not accessible from the root of the navigationEndpoint object.

The corresponding statement has been moved at the bottom of the specific
endpoints parsing, as the webCommandMetadata object is present almost
everywhere, otherwise URLs of some endpoints would have be changed, such as
uploader URLs (from channel IDs to handles).

As no ParsingException is now thrown by getUrlFromNavigationEndpoint, and so by
getTextFromObject, getUrlFromObject and getTextAtKey, the methods which were
catching ParsingExceptions thrown by these methods had to be updated.

URLs got in the HTML version of getTextFromObject are now escaped properly to
provide valid HTML to clients. This has been also done for attribute
descriptions, with the description text for this type of descriptions.

As YouTube descriptions are in HTML format (except for the fallback on the JSON
player response, which is plain text and only happens when there is no visual
metadata or a breaking change), all URLs returned are escaped, so tests which
are testing presence of URLs with escaped characters had to be updated (it was
only the case for YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing).
2023-02-26 18:43:36 +01:00
Stypox 99ab9777ad
Release 0.22.5 2023-02-26 16:33:25 +01:00
Stypox 39c2e1fda0
Merge pull request #1028 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.4
Bump org.jsoup:jsoup from 1.15.3 to 1.15.4
2023-02-26 16:27:31 +01:00
petlyh f7a7a236fb
[Bandcamp] Show comments as disabled on radio streams 2023-02-23 18:42:43 +01:00
dependabot[bot] f5599ff08d
Bump org.jsoup:jsoup from 1.15.3 to 1.15.4
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup) from 1.15.3 to 1.15.4.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.3...jsoup-1.15.4)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-20 10:05:26 +00:00
Stypox 7e793c11ae
Empty commit to make JitPack happy
The build for e920ab3f5f succeeded, but for some reason it is considered as failed...
2023-02-08 22:35:26 +01:00
Stypox e920ab3f5f
Merge pull request #1021 from TeamNewPipe/fix/yt-comment-text-hashtag
[YouTube] Fix getting the comment text if the comment contains a hashtag
2023-01-29 21:47:59 +01:00
TobiGr 3f7df9536e [YouTube] Fix getting the comment text if the comment contains a hashtag 2023-01-29 20:33:51 +01:00
Stypox 999fb7f812
Merge pull request #1024 from AudricV/snd_fix-tracks-like-count
[SoundCloud] Fix extraction of tracks like count
2023-01-29 10:52:54 +01:00
Stypox 3519d4c367
Merge pull request #1015 from AudricV/yt_fix-channel-id-rss-feeds
[YouTube] Fix channel ID extraction of YouTube channels RSS feeds
2023-01-29 10:41:38 +01:00
Stypox 9aca710e86
Merge pull request #1013 from Stypox/fix-music-mixes
[YouTube] Now music mixes can be treated as normal mixes
2023-01-29 09:48:51 +01:00
Stypox 76eeabac45
Merge pull request #1020 from TeamNewPipe/fix/yt-subscriber-count
[YouTube] Fix NPE in search when getting channel items without subscriber count
2023-01-29 09:44:22 +01:00
AudricV 676622f6df
[SoundCloud] Fix expectedLikeCountAtLeast tests of SoundcloudStreamExtractorTest test classes
As like count is now returned by the extractor, we need to assert a positive
minimum like count, which is close to the actual value, in order to avoid test
failures due to lower like counts than the ones excepted.
2023-01-29 01:08:02 +01:00
AudricV 2a24d407d5
[SoundCloud] Fix extraction of tracks like count
SoundCloud is using likes_count to return the like count of a track, like it
was the case before they switched to favoritings_count.
2023-01-29 01:00:49 +01:00
Tobi 0e4e6a9bac
Merge pull request #1022 from AudricV/yt_support-live-urls
[YouTube] Support live URLs
2023-01-28 21:52:26 +01:00
AudricV ba24976e41
[YouTube] Add live URLs test and do minor improvements to YoutubeStreamLinkHandlerFactoryTest
- Remove unused imports;
- Replace wildcard imports by single class imports;
- Suppress "HTTP links are not secured" warnings from IDEA IDEs;
- Replace removed video jZViOEv90dI by an existing video, 9Dpqou5cI08 (the
corresponding test has been of course renamed).
2023-01-28 19:36:21 +01:00
AudricV 57f850bc2d
[YouTube] Support live URLs and do minor improvements to YoutubeStreamLinkHandlerFactory
- Move license header at the top;
- Use an unmodifiable set for the subpaths instead of a modifiable list;
- Add missing Nonnull and Nullable annotations;
- Improve exception messages.
2023-01-28 19:36:20 +01:00
AudricV 1f4ed9dce9
[YouTube] Fix channel ID extraction of YouTube channel RSS feeds
The yt:channelId element doesn't provide the channel ID anymore and is empty,
like the id element, so we need now to extract it from the channel URL provided
in two elements: author -> uri and feed -> link.

Also avoid a NullPointerException in getUrl and getName methods.
2023-01-28 11:53:33 +01:00
Tobi c589a2c1a2
Merge pull request #1014 from TeamNewPipe/fix/yt-comments
[YouTube] Fix getting next comments pages
2023-01-27 11:14:55 +01:00
Tobi c2b36fd405
Merge pull request #1016 from lonewolf2208/bug-fix
[YouTube] Remove topStandaloneBadge check for view count of stream items
2023-01-25 08:30:36 +01:00
TobiGr 72573932cf [YouTube] Fix NPE in search when getting channel items without subscriber count 2023-01-24 23:03:45 +01:00
TobiGr f50b7275af [YouTube] Fix getting next comments pages 2023-01-24 22:39:08 +01:00
Kunal 9bdad40b06 Removed topStandaloneBadge 2023-01-20 02:41:21 +05:30
Stypox 5945057227
[YouTube] Add music mix test 2023-01-15 23:30:30 +01:00
Stypox 7293991832
[YouTube] Now music mixes can be treated as normal mixes
Using a playlist extractor on them would result in "Unviewable playlist" errors
2023-01-15 23:28:59 +01:00
Stypox ff94e9f30b
Merge pull request #1009 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.10.1
Bump gson from 2.10 to 2.10.1
2023-01-11 15:35:36 +01:00
Stypox c1040bccac
Merge pull request #794 from FireMasterK/comments-count
[YouTube] Add support to extract total comment count
2023-01-11 15:32:19 +01:00
Stypox 6ccc43e57e
Merge pull request #1010 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.2
Bump junit-bom from 5.9.1 to 5.9.2
2023-01-11 13:19:33 +01:00
dependabot[bot] 1bc44cb949
Bump junit-bom from 5.9.1 to 5.9.2
Bumps [junit-bom](https://github.com/junit-team/junit5) from 5.9.1 to 5.9.2.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.9.1...r5.9.2)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-01-11 09:04:12 +00:00
dependabot[bot] f43049985e
Bump gson from 2.10 to 2.10.1
Bumps [gson](https://github.com/google/gson) from 2.10 to 2.10.1.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.10...gson-parent-2.10.1)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-01-09 09:05:55 +00:00
Stypox 33f9f2cf66
Merge pull request #1006 from TeamNewPipe/yt/fix-html-escape
[YouTube] Fix escaping links in YouTubeParsingHelper.getTextFromObject
2023-01-05 19:02:23 +01:00
TobiGr 56aab4d971 [YouTube] Fix escaping links in YouTubeParsingHelper.getTextFromObject 2023-01-05 00:28:12 +01:00
Kavin 22a47da8c7
Fix requested change and remove outdated comment. 2023-01-02 20:42:32 +00:00
Kavin 98a90fd9c8
Don't cache comments count and return early on page fetch if no token. 2023-01-02 20:40:48 +00:00
Kavin 2974dfaa48
Only store ajaxJson for initial page and eager fetch the initial continuation. 2023-01-02 20:40:48 +00:00
Kavin 64d24aa09e
Fix request changes. 2023-01-02 20:40:48 +00:00
Kavin 67ef4f4c30
Cleanup and remove optional. 2023-01-02 20:40:48 +00:00
FireMasterK 22f71b010c
Fix for requested changes. 2023-01-02 20:40:48 +00:00
FireMasterK 656b7c1cd9
Improve method documentation. 2023-01-02 20:40:48 +00:00
FireMasterK 981aee4092
Add support to extract total comment count. 2023-01-02 20:40:48 +00:00
Stypox 45636b0d00
Merge pull request #986 from Isira-Seneviratne/Static_maps
Use immutable Map factory methods.
2023-01-02 18:11:14 +01:00
Stypox 219c5c5be5
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java 2023-01-02 18:11:03 +01:00
Stypox 259de3cba6
Merge pull request #995 from TeamNewPipe/feat/soundcloud-playlistinfoitemextractor
[SoundCloud] Implement getUploaderUrl() and isUploaderVerified() for PlaylistInfoItemExtractor
2023-01-02 15:10:40 +01:00
Stypox 991394b53a
Merge pull request #1005 from FireMasterK/fix-escaping-xss
Fix for potential XSS attacks and formatting issues
2023-01-02 15:06:17 +01:00
Stypox ce15f7cc50
Merge pull request #996 from TeamNewPipe/feat/peeertube-playlists
[PeerTube] Support searching for playlists and channels
2023-01-02 15:04:09 +01:00
Isira Seneviratne d8ce08d969 Use immutable Map factory methods. 2023-01-02 07:50:31 +05:30
Kavin 01acf79436
Fix for potential XSS attacks. 2022-12-31 20:05:32 +00:00
TobiGr 292e0d8ce7 [SoundCloud] Implement getUploaderUrl() and isUploaderVerified() for PlaylistInfoItemExtractor 2022-12-31 18:46:39 +01:00
TobiGr 2a8729aeb2 Apply suggestions
Co-authored-by: Stypox <stypox@pm.me>
2022-12-31 18:24:33 +01:00
TobiGr d75a997611 [PeerTube] Support searching for channels 2022-12-31 18:24:33 +01:00
TobiGr dea6d8ce4c [PeerTube] Support searching for playlists 2022-12-31 18:24:33 +01:00
Stypox 95cc6aefbb
Merge pull request #994 from TeamNewPipe/fix/peertube-subtitles-exception
[PeerTube] Report Exceptions thrown while getting a stream's subtitles
2022-12-31 15:01:39 +01:00
Stypox 7b54457789
Merge pull request #941 from TeamNewPipe/feat/peertube-comment-replies
[PeerTube]  Support comment replies
2022-12-31 14:57:51 +01:00
AudricV f45966d449
Merge pull request #910 from Isira-Seneviratne/Locale_forLanguageTag
Add compat Locale.forLanguageTag() implementation.
2022-12-24 23:53:30 +01:00
AudricV d5437e0bc5
Merge pull request #863 from AudricV/add-content-type-and-content-length-headers-to-post-requests
Add Content-Type header to all POST requests without an empty body
2022-12-16 19:32:56 +01:00
Tobi 88e07e555d
Merge pull request #1000 from AudricV/yt-streaminfoitemextractor-improvements
[YouTube] Improve YoutubeStreamInfoItemExtractor
2022-12-11 17:02:29 +01:00
AudricV 0766b1d211
[YouTube] Improve YoutubeStreamInfoItemExtractor
- Return duration of video premieres;
- Add another non-localized method to determine whether a stream is a running
livestream;
- Return view count and upload date of videos in playlists;
- Store isPremiere result;
- Remove shorts workaround code, as it was only useful on channels and shorts
have been moved into a separated channel tab;
- Improve some other code.
2022-12-08 13:59:12 +01:00
Tobi 896d7e09eb
Merge pull request #978 from Theta-Dev/fix/search-channel-handles
[YouTube] Fix search subscriber count extraction with channel handles
2022-12-05 17:52:05 +01:00
TobiGr cd3262745d [PeerTube] Report Exceptions thrown while getting a stream's subtitles 2022-12-03 16:11:21 +01:00
TobiGr 4e66b2287e [PeerTube] Add support for comment replies 2022-12-01 14:05:18 +01:00
Tobi 41c8dce452
Merge pull request #992 from Isira-Seneviratne/String_isBlank
Use String.isBlank().
2022-11-30 17:48:54 +01:00
Isira Seneviratne 2bca56f0df Use String.isBlank(). 2022-11-30 08:26:21 +05:30
Isira Seneviratne 3b80547976 Add code review suggestions. 2022-11-30 07:57:45 +05:30
Tobi a1128ecbdc
Merge pull request #991 from FireMasterK/fix-compile
Fix complication error in comment test
2022-11-29 19:20:01 +01:00
ThetaDev 016623131e docs: update comment in YoutubeChannelInfoItemExtractor 2022-11-29 19:06:03 +01:00
Kavin 2e08eaad96
Fix complication error in comment test. 2022-11-29 16:07:48 +00:00
Kavin abf08e1496
Merge pull request #990 from FireMasterK/bold-italic-strikethrough
[YouTube] Implement bold/italic/strike-through support
2022-11-29 15:59:38 +00:00
Tobi 72d5ed3318
Merge pull request #987 from FireMasterK/comments-text-description
Use Description object for comments text.
2022-11-29 16:54:56 +01:00
Kavin 57e7a6fb7c
Add mocks test. 2022-11-28 20:27:55 +00:00
Kavin 1d3d7fa5c3
Add test for formatting. 2022-11-28 20:26:37 +00:00
Kavin 52fda37915
Implement bold/italic/strike-through support. 2022-11-28 19:06:18 +00:00
Kavin b566084cac
Use Description object for comments text. 2022-11-28 17:02:19 +00:00
Tobi 40f1ec4a54
Merge pull request #989 from Stypox/forbid-wrong-nullable-notnull-imports
Block wrong nullable/nonnull imports in checkstyle
2022-11-28 14:57:46 +01:00
Stypox 5f6101698b
Block wrong nullable/nonnull imports in checkstyle 2022-11-28 14:42:18 +01:00
Tobi f8162b049d
Merge pull request #984 from FireMasterK/unused-dep
Remove unused autolink dependency
2022-11-28 11:28:42 +01:00
Tobi ad7c3c2374
Merge pull request #985 from FireMasterK/checkstyle-update
Update checkstyle dependency.
2022-11-28 11:22:40 +01:00
Tobi 1da0190056
Merge pull request #980 from TeamNewPipe/fix/yt/unavailable
[YouTube] Fix extracting the detailed error message for unavailable streams
2022-11-28 10:07:34 +01:00
Stypox 60fb30f835
Merge pull request #928 from FireMasterK/comment-urls
Parse YouTube comments as HTML
2022-11-27 19:16:34 +01:00
Kavin 5abea22225
Fix throwing correct reason. 2022-11-26 21:09:08 +00:00
Kavin 7874d3c8aa
Update checkstyle dependency.
Since we now use JDK 11
2022-11-26 20:22:25 +00:00
Kavin faf28f5c11
Remove unused dependency. 2022-11-26 20:17:25 +00:00
Kavin a3ed947b2d
Merge pull request #983 from FireMasterK/update-countries
Update supported countries list.
2022-11-26 20:06:00 +00:00
Kavin c043597255
Update supported countries list. 2022-11-26 19:01:33 +00:00
TobiGr 4680df0bdf Fix throwing correct reason 2022-11-23 17:03:22 +01:00
TobiGr 9de8405c9f [YouTube] Fix extracting the detailed error message of streams which are unavailable 2022-11-23 08:33:06 +01:00
Stypox 34d79bd267
[YouTube] Update mocks 2022-11-22 17:10:04 +01:00
AudricV 2ec296e674
Fix YoutubeSearchExtractorTest.MetaInfoTest
Not all the "learn more" button is uppercase anymore, that's only the case for
the first letter.
2022-11-22 16:34:54 +01:00
AudricV 3891542ca1
Use Downloader's postWithContentType and postWithContentTypeJson methods in services and extractors 2022-11-22 11:37:18 +01:00
AudricV b2862f3cd1
Add postWithContentType and postWithContentTypeJson utility methods in Downloader
Co-authored-by: Stypox <stypox@pm.me>
2022-11-22 11:37:17 +01:00
AudricV e9a0d3bd95
[YouTube] Send Content-Type header in all POST requests
This header was not sent partially before and was added and guessed by OkHttp. This can create issues when using other HTTP clients than OkHttp, such as Cronet.

Some code in the modified classes has been improved and / or deduplicated, and usages of the UTF_8 constant of the Utils class has been replaced by StandardCharsets.UTF_8 where possible.

Note that this header has been not added in except in YoutubeDashManifestCreatorsUtils, as an empty body is sent in the POST requests made by this class.
2022-11-22 11:37:16 +01:00
AudricV b9e463de49
[Bandcamp] Send Content-Type header in POST requests
This header was not sent before and was added and guessed by OkHttp. This can create issues when using other HTTP clients than OkHttp, such as Cronet.

Also make use of StandardCharsets.UTF_8 when getting bytes of bodies instead of the platform default's charset, to make sure to prevent some encoding issues on some JVMs.
2022-11-22 11:35:46 +01:00
AudricV 65d6321e3d
Fix typos in Downloader.post JavaDocs
Post methods in Downloader return the result of a POST request and not the one of a GET request.
2022-11-22 11:35:46 +01:00
ThetaDev 5daabd1793 fix: #976 search subscriber count extraction with channel handles 2022-11-22 02:17:10 +01:00
Kavin c953e23414
Merge pull request #968 from AudricV/yt-support-no-video-info-renderers-for-streams
[YouTube] Support lack of video info renderers for streams
2022-11-16 20:20:01 +00:00
Tobi 2211a24b69
Merge pull request #971 from lrusso96/patch-1
[YouTube] Improve duration parsing
2022-11-16 16:14:54 +01:00
Tobi f6eefdc76c
Merge pull request #975 from FireMasterK/jitpack
Use JDK 11 for Jitpack gradle builds
2022-11-15 20:23:21 +01:00
Kavin a1f22be0dd
Use JDK 11 for Jitpack gradle builds. 2022-11-15 18:43:19 +00:00
Kavin 9bdad55508
Merge pull request #973 from FireMasterK/audio-track
Add support for extracting audio track information
2022-11-15 08:19:47 +00:00
Kavin 86f06b333a
Address review. 2022-11-14 00:05:31 +00:00
Kavin b16e6082e1
Add test for audio stream languages. 2022-11-13 23:10:44 +00:00
Kavin 30909da1df
Fix audio track similar comparison for IDs. 2022-11-13 23:08:54 +00:00
Kavin 6d59cdbe3a
Add support for extracting audio tracks. 2022-11-13 21:39:29 +00:00
Tobi 43d1c1f8b1
Merge pull request #880 from Isira-Seneviratne/Use_StandardCharsets
Use StandardCharsets.UTF_8.
2022-11-12 03:23:44 +01:00
Isira Seneviratne e4d982c7ea Fix license. 2022-11-12 07:29:15 +05:30
Isira Seneviratne 416089146e Fix failing tests. 2022-11-12 07:29:15 +05:30
Isira Seneviratne ff5f223d3f Update README.md 2022-11-12 07:29:15 +05:30
Isira Seneviratne ddbce3b83d Add Utils methods for URL encoding/decoding using UTF-8. 2022-11-12 07:29:15 +05:30
Isira Seneviratne 366f5c1632 Use StandardCharsets.UTF_8. 2022-11-12 07:29:15 +05:30
Tobi 0ea16c0b73
Merge pull request #954 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.10
Bump gson from 2.9.1 to 2.10
2022-11-09 17:28:31 +01:00
dependabot[bot] 0b82fade51
Bump gson from 2.9.1 to 2.10
Bumps [gson](https://github.com/google/gson) from 2.9.1 to 2.10.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.9.1...gson-parent-2.10)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-09 16:21:16 +00:00
Luigi Russo c9635218e2
[YouTube] Improve duration parsing 2022-11-09 09:41:29 +01:00
Isira Seneviratne 1be270768f
Merge pull request #893 from Isira-Seneviratne/Static_sets
Use immutable sets in YoutubeParsingHelper.
2022-11-09 09:07:01 +05:30
Isira Seneviratne 316d8573fa Use immutable sets in YoutubeParsingHelper. 2022-11-07 07:50:26 +05:30
AudricV 6a2c680d8f
[YouTube] Add mocks for YoutubeStreamExtractorDefaultTest.NoVisualMetadataVideoTest 2022-11-04 19:43:06 +01:00
AudricV e66fed41d6
[YouTube] Add a StreamExtractor test for a video without visual metadata
The video "Makani’s first commercial-scale energy kite" (video ID:
An8vtD1FDqs), which has this behavior, is used for the new test,
NoVisualMetadataVideoTest, added in YoutubeStreamExtractorDefaultTest.

Tests of elements who throw an exception in this case (subscriber count, like
count, uploader avatar URL) test if the ParsingException exception is thrown by
YoutubeStreamExtractor.
2022-11-04 19:42:12 +01:00
AudricV aa9a8ca23c
[YouTube] Make non-extraction of videoPrimaryInfoRenderer and/or videoSecondaryInfoRenderer not fatal
Also de-duplicated common code related to the obtain of these video info renderers.

This change allows extraction of videos without visual metadata.
2022-11-04 18:35:53 +01:00
AudricV eb07d70a2c
Merge pull request #964 from AudricV/yt-support-handles-and-all-channel-usernames
[YouTube] Support handles and all channel usernames
2022-11-04 12:09:15 +01:00
AudricV 20cd8e8a4a
[YouTube] Update mocks of YoutubeChannelExtractorTest.Gronkh 2022-11-03 19:46:42 +01:00
AudricV a34f060ba0
[YouTube] Use a handle for YoutubeChannelExtractorTest.Gronkh 2022-11-03 19:46:42 +01:00
AudricV 724f669ff7
[YouTube] Add tests for handles and user IDs with non ASCII characters support
Unneeded public modifiers in test methods of
YoutubeChannelLinkHandlerFactoryTest have been also removed.
2022-11-03 19:46:42 +01:00
AudricV 61ce041bda
[YouTube] Support handles and all custom channel names
More non-channel paths have been also added to the excluded custom name paths,
documentation and exception messages have been improved and fixed in some
places, and the licence header of YoutubeChannelLinkHandlerFactory has been
moved to its beginning and updated.
2022-11-03 19:46:42 +01:00
AudricV ffffb04439
Merge pull request #953 from Theta-Dev/attributed-text-desc
[YouTube] Add support for attributed text description
2022-11-03 18:34:30 +01:00
ThetaDev 592e1d6386 fix: parsing attributed description with no command runs 2022-11-03 12:10:52 +01:00
AudricV 31bf7046b7
Merge pull request #944 from Theta-Dev/fix-rich-grid-renderer
[YouTube] Support richGridRenderer on channel page to fix extraction of videos in channels
2022-11-03 00:10:54 +01:00
ThetaDev 099b53cc4f
[YouTube] Add parser for attributedDescription
Also update the mock of the next InnerTube endpoint response of the
YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing test class with an
attributedDescription instead of a regular description
2022-11-02 23:11:33 +01:00
AudricV e4c24d4c36
[YouTube] Regenerate supported channels mocks 2022-11-02 19:13:59 +01:00
Theta-Dev 20e4a35814
[YouTube] Support richGridRenderer on channel pages
YouTube is deploying a new layout on their channel pages, which uses richGridRenderer JSON objects.
2022-11-02 19:01:29 +01:00
AudricV 4cae66f1f9
Merge pull request #946 from chowder/dev
Add ability to identify short-form `StreamInfoItem`s
2022-11-01 12:19:58 +01:00
Tobi eb40bb8458
Merge pull request #959 from FireMasterK/playlist-info-item-uploader
Add uploaderUrl and uploaderVerified to PlaylistInfoItem.
2022-10-31 13:10:33 +01:00
chowder b1a899fd47 Fix null pointer exception 2022-10-31 11:12:23 +00:00
Stypox a4db106a66
Merge pull request #960 from AudricV/yt-workaround-403-errors-android-client
[YouTube] Workaround 403 HTTP errors of ANDROID client streams
2022-10-30 21:53:26 +01:00
Kavin b441910257
Mark uploaderUrl as nullable. 2022-10-30 13:36:04 +00:00
chowder 3fdc0e72cc Line breaks for long docstrings 2022-10-30 13:28:39 +00:00
Kavin 6a256d0631
Add uploader url and verified to PlaylistInfoItem. 2022-10-30 13:00:19 +00:00
Tobi 430504b4b5
Merge pull request #958 from AudricV/yt-playlists-support-new-metadata-format
[YouTube] Support new metadata format of playlists
2022-10-30 12:31:43 +01:00
Kavin f9bd08c649 Address reviews. 2022-10-30 01:25:30 +00:00
Caleb 9282c3c13b Fix exception message for YoutubeStreamInfoItemExtractor#isShortFormContent
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
2022-10-30 01:23:15 +00:00
Caleb c5216f7c12 Update docstring for StreamExtractor#isShortFormContent
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
2022-10-30 01:23:15 +00:00
Caleb 04795fe5d2 Use Stream API for ShortFormContent#testShortFormContent
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
2022-10-30 01:23:15 +00:00
chowder 975b38a0b9 Regenerate test mocks 2022-10-30 01:23:15 +00:00
chowder 4cccd33f3d Implement isShortFormContent for StreamExtractor and StreamInfo 2022-10-30 01:23:15 +00:00
chowder 09544ceb23 Fix tests 2022-10-30 01:23:15 +00:00
chowder 644cc6cd76 Rename test function name 2022-10-30 01:23:15 +00:00
chowder daf5674951 Add ability to identify short-form StreamInfoItems 2022-10-30 01:23:12 +00:00
Tobi 3d314169b9
Merge pull request #943 from TeamNewPipe/fix/sc/comments
[SoundCloud] Fix getting more comments
2022-10-29 22:19:50 +02:00
AudricV 65125a3eb4
[YouTube] Fix YoutubePlaylistExtractorTest.LearningPlaylist
The video count is now returned for this playlist, so it isn't unknown.

The testStreamCount method of this test class asserts now that the stream count
is greater than 40.
2022-10-29 18:12:10 +02:00
AudricV 7258a53225
[YouTube] Support new playlist layout
This new layout doesn't provide author thumbnails and is completely different
for metadata, so the code to get them has been refactored.

The code of learning playlists video count check has been also removed, as it
seems to be not relevant anymore (the video count seems to be returned for
these playlists with both layouts).

Finally, unneeded overrides of subchannel methods, which don't apply to the
YouTube service, have been removed.
2022-10-29 18:12:10 +02:00
AudricV e37e8b358e
[YouTube] Update mocks that need to be updated 2022-10-29 18:10:18 +02:00
AudricV 60e97cd274
[YouTube] Workaround getting streaming URLs returning 403 HTTP response codes
Using the player parameters used to get stories seems to fix the issue, which
affects currently only certain countries such as UK.

This is a workaround and should be fixed in a better way (by changing the
InnerTube additional client used for videos or finding what is now required in
Android player requests).
2022-10-29 17:58:33 +02:00
AudricV 4bc90cd9d8
Merge pull request #957 from AudricV/fix-checkstyle-error
Fix Checkstyle error in YoutubeCommentsInfoItemExtractor
2022-10-29 17:55:33 +02:00
AudricV c230d84df1
Fix Checkstyle error in YoutubeCommentsInfoItemExtractor 2022-10-29 13:24:19 +02:00
Tobi 9ffdd0948b
Merge pull request #945 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.3
Bump spotbugs-annotations from 4.7.2 to 4.7.3
2022-10-18 22:55:24 +02:00
dependabot[bot] 24137bb15f
Bump spotbugs-annotations from 4.7.2 to 4.7.3
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.7.2 to 4.7.3.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.2...4.7.3)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-10-17 09:20:51 +00:00
xz-dev 0ffcb32d9c
[YouTube] Add comment reply count support (#936)
Add comment reply count support for YouTube and introduce `CommentsInfoItem.UNKNOWN_REPLY_COUNT` constant

Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2022-10-15 12:40:06 +02:00
Isira Seneviratne b90a566dd8 Add backport implementation of Locale.forLanguageTag(). 2022-10-12 09:21:39 +05:30
Isira Seneviratne b232c29d22 Use Locale.forLanguageTag(). 2022-10-12 09:21:38 +05:30
TobiGr 4d136599bd [SoundCloud] Fix getting more comments 2022-10-11 15:44:54 +02:00
Tobi a822e91909
Merge pull request #939 from TurtleArmyMc/fix/SoundcloudPlaylistExtractor_track_order
Fix SoundcloudPlaylistExtractor: tracks are in correct order
2022-10-10 22:28:18 +02:00
TobiGr 02810a7db7 Add a comment 2022-10-10 22:22:12 +02:00
Tobi 2d50369c77
Merge pull request #888 from Isira-Seneviratne/EnumMap
Use EnumMap in PatternsHolder.
2022-10-10 12:00:06 +02:00
Tobi 54092fc3c7
Merge pull request #930 from Isira-Seneviratne/Avoid_Comparator_NPE
Avoid possible NullPointerException in MediaCCCRecentKiosk.
2022-10-10 11:43:34 +02:00
Tobi d7c23ef118
Merge pull request #924 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.2
Bump spotbugs-annotations from 4.7.1 to 4.7.2
2022-10-10 11:35:40 +02:00
dependabot[bot] 08339d6b0c
Bump spotbugs-annotations from 4.7.1 to 4.7.2
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.7.1 to 4.7.2.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.1...4.7.2)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-10-10 09:32:17 +00:00
Tobi 5537e84100
Merge pull request #931 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.1
Bump junit-bom from 5.9.0 to 5.9.1
2022-10-10 11:30:53 +02:00
dependabot[bot] 719664593f
Bump junit-bom from 5.9.0 to 5.9.1
Bumps [junit-bom](https://github.com/junit-team/junit5) from 5.9.0 to 5.9.1.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.9.0...r5.9.1)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-10-10 09:25:42 +00:00
TurtleArmyMc bf70d32eb4 Fix SoundcloudPlaylistExtractor: tracks are in correct order 2022-09-30 15:26:25 -04:00
Stypox 5c710da160
Merge pull request #934 from AudricV/fix-yt-throttling-decryption-function-regex
[YouTube] Fix throttling decryption function regex
2022-09-25 13:26:08 +02:00
AudricV 8067c43837
[YouTube] Don't use a specific letter for the decryption function name pattern
Use the same possible characters for variables everywhere, in order to avoid
potential future throttling parameter decryption function name parsing issues
related to the usage of other letter(s) than b.
2022-09-24 21:49:22 +02:00
AudricV abcee87167
[YouTube] Fix throttling parameter decryption function regex
- Quote the function name, as it may contain special regex symbols, such as
  dollar;
- Support multiple lines;
- Use what looks like the end of the function for the end of the regex (this
  part is inspired from yt-dlp throttling parameter decryption regex);
- Move the throttling function body regex into a private and static constant.
2022-09-24 21:28:09 +02:00
ThetaDev 7244be7627
[YouTube] Add JavaScript lexer to parse completely throttling decryption function (#905) 2022-09-24 19:55:46 +02:00
Isira Seneviratne 0e31c86aee Avoid possible NullPointerException in MediaCCCRecentKiosk. 2022-09-21 05:40:07 +05:30
Stypox a99af9bb6e
Merge pull request #927 from Stypox/fix-feed-thumbnail
[YouTube] Return mqdefault thumbnails in fast feed
2022-09-19 08:52:01 +02:00
Kavin 4e14644c41
Ensure comments are parsed with URLs and timestamps. 2022-09-17 16:03:39 +05:30
Stypox 14ef430546
Merge pull request #920 from AudricV/fix-yt-trending-extraction
[YouTube] Fix extraction of the Trending kiosk with the new data model returned
2022-09-14 11:14:13 +02:00
Stypox 93ef73e2f2
[YouTube] Return mqdefault thumbnails in feed
The hqdefault thumbnails, which are used by default, have black bars at the top and at the bottom, and are thus not optimal.
2022-09-13 16:46:54 +02:00
AudricV d8ddeb549c
[YouTube] Support new trending structure and filter recently trending section
This new structure allow us to filter easily Trending shorts and Recently
trending sections.

On the previous one, this Recently trending section is now filtered, by
checking whether sections have a title, which isn't the case for normal trends
contrary to the other ones.

This makes that the extractor returns now only the real 50 "Now" YouTube
trends.

Elements inside arrays are now extracted dynamically instead of only the ones
of the first index, using Java 8's Stream API.

The getInitialPage() method of YoutubeTrendingExtractor can now throw a
ParsingException if no selected tab (corresponding to the one of the trends
type extracted) has been found.

Finally, the licence header has been moved to the top of the file and updated.
2022-09-11 18:52:12 +02:00
AudricV 884da40f65
Merge pull request #926 from AudricV/fix-yt-like-count-extraction
[YouTube] Fix extraction of like count with the new data model
2022-09-11 12:21:38 +02:00
AudricV 58b09cf78b
Merge pull request #925 from Theta-Dev/fix-td-regex
[YouTube] Fix throttling parameter decryption on Android
2022-09-10 23:30:50 +02:00
AudricV 119b9129e2
[YouTube] Fix extraction of like count with the new data model
In the new data model currently A/B tested or deployed, the like count button
data is the same, but its path has been changed.

The extraction of the like count has been also improved, by using the multiple
accessibility data available instead of the only one which was used before.
This accessibility data has been also deprioritized, because it is language
dependent when there is no view.

Like count is always known, even for age-restricted videos, so returning -1 if
the like count extraction fails on this type of videos is not needed and hides
an extraction error: that's the reason why this return has been removed and the
exception will always be thrown, even if a video is age-restricted.
2022-09-10 23:28:38 +02:00
ThetaDev 4905f74700
[YouTube] Fix throttling parameter decryption on Android
Escape the curly brace in the regular expression used to parse the throttling
parameter decryption function to allow its compatibility on Android.
2022-09-10 17:03:39 +02:00
Isira Seneviratne e101d58ae7 Use EnumMap in PatternsHolder. 2022-08-26 08:15:21 +05:30
Stypox 6a858368c8
Merge pull request #912 from Stypox/downgrade-rhino
Downgrade Rhino since it uses a class not available on Android
2022-08-25 10:10:27 +02:00
Stypox fa25a7280f
Downgrade Rhino since it uses a class not available on Android
Reverts #774, see TeamNewPipe/NewPipe#8876
2022-08-25 09:49:13 +02:00
Stypox 25f0ce7305
Merge pull request #911 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.3
Bump jsoup from 1.15.2 to 1.15.3
2022-08-24 14:23:44 +02:00
Stypox d8e10dd3ad
Merge pull request #907 from Isira-Seneviratne/Remove_EmptyString
Remove EMPTY_STRING.
2022-08-24 14:22:55 +02:00
dependabot[bot] 1d7a3a90a4
Bump jsoup from 1.15.2 to 1.15.3
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.15.2 to 1.15.3.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.2...jsoup-1.15.3)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-08-24 09:14:26 +00:00
Isira Seneviratne 41254ae12a Remove annotation. 2022-08-24 06:59:17 +05:30
Isira Seneviratne 943b7c033b Remove EMPTY_STRING. 2022-08-24 06:59:17 +05:30
litetex 20914a6035
Merge pull request #903 from TacoTheDank/updateGradle
Update gradle wrapper to 7.5.1
2022-08-23 16:21:56 +02:00
litetex d6577e5e0b
Merge pull request #882 from litetex/fix-all-tests
Fix all tests
2022-08-23 16:19:27 +02:00
litetex 0a5ad90724
Merge pull request #901 from AudricV/remove-deprecated-code-yt-throttling-decrypter
[YouTube] Remove deprecated throttling decryption code
2022-08-22 17:13:00 +02:00
AudricV 03d9a4fe9d
Fix typos in YoutubeStreamExtractor.tryDecryptUrl 2022-08-21 21:17:03 +02:00
litetex 0beb55a232 Improve ``YoutubeThrottlingDecrypterTest`` 2022-08-21 20:17:05 +02:00
litetex 9e93d6b193 YoutubeThrottlingDecrypter: Check if returned string is a valid JavaScript function 2022-08-21 20:16:45 +02:00
litetex ed0a07af4e YoutubeThrottlingDecrypter: Patch regex 2022-08-21 20:15:53 +02:00
litetex 641a447d9b Updated consent-related mock-data 2022-08-21 18:43:48 +02:00
litetex 8ff7a90f52 Improved consent cookie related constants and documentation 2022-08-21 18:41:40 +02:00
TacoTheDank d8339d4060 Update gradle wrapper to 7.5.1 2022-08-18 15:09:35 -04:00
AudricV cb64a480cd
[YouTube] Remove deprecated code in YoutubeThrottlingDecrypter
YoutubeThrottlingDecrypter is now a non-instantiable final class and non-static
attributes have been removed.

The static attributes of this class have been renamed, in order to respect the
naming specification used.
2022-08-16 15:01:05 +02:00
litetex 5bbea3a8f2 Fix ``YoutubeMixPlaylistExtractorTest`` again
Doesn't work for kurgesagt videos anymore, used music video (copyright free) instead
2022-08-14 15:12:27 +02:00
litetex da06166065 Updated mock data of ``YoutubeSearchExtractorTest$Suggestion`` 2022-08-14 15:12:26 +02:00
litetex 2e36ab1578 Fixed ``YoutubeSearchExtractorTest$Suggestion`` 2022-08-14 15:12:26 +02:00
litetex 2a8a623643 ``YoutubePlaylistLinkHandlerFactoryTest``: Use parameterized tests 2022-08-14 14:48:31 +02:00
litetex 938e69a16a Fixed ``YoutubePlaylistLinkHandlerFactoryTest`` 2022-08-14 14:48:30 +02:00
litetex 844de3e378 Fix ``YoutubeStreamLinkHandlerFactoryTest `` and parameterized the tests
used code from #833
2022-08-14 14:48:30 +02:00
litetex c1a72b8240 Use Android Studios code style 2022-08-14 14:48:29 +02:00
litetex fc27b8a5b8 Use preexisting ``ContentNotAvailableException`` 2022-08-14 14:48:29 +02:00
Stypox 17bad6cedf Fix test about exception type: NPE, not IllegalArgment
The relevant change was made in #877
2022-08-14 14:48:28 +02:00
litetex ecfc370685 Fixed all YTMixPlaylists
Added option to choose if you want to consent or not - currently this is done by a static variable in ``YoutubeParsingHelper`` - may not be the best long-term solution but for now the tests work again (in EU countries) 🥳
2022-08-14 14:48:27 +02:00
litetex 2b6fe294b2 Fixed ``YoutubeMusicSearchExtractorTest``
The renderers and queries got changed.
2022-08-14 14:48:27 +02:00
litetex d6586da614 Failing test works locally without any problems. Reformatted it a bit. 2022-08-14 14:48:27 +02:00
litetex bf3ae5e679 Fixed ``SoundcloudStreamLinkHandlerFactoryTest``
* Removed the unnecessary ``public``
* Use parameterized tests
* One song got removed (which caused the test failure), replaced it with another song
2022-08-14 14:48:27 +02:00
litetex 504f81036e Fixed ``PeertubePlaylistExtractorTest``
Also removed the unnecessary ``public`` and reformatted the code a bit
2022-08-14 14:48:26 +02:00
litetex e7c12258f4 Fixed ``BandcampSearchExtractorTest``
Also removed the unnecessary ``public``
2022-08-14 14:48:26 +02:00
litetex 9e8724df4d Fixed ``YoutubeStreamExtractorLivestreamTest``
The "Lofi Girl"-stream got interrupted by a copyright strike and had to be restarted. Because of this a new id is now used.
2022-08-14 14:48:26 +02:00
Tobi 76aad92fa5
Merge pull request #890 from AudricV/yt-clients-update-improvements-and-extraction-fixes
[YouTube] Update clients version and improve extraction of API key and client version of WEB client
2022-08-12 23:32:41 +02:00
AudricV 472f5d9e9c
[YouTube] Update mocks 2022-08-12 19:20:32 +02:00
AudricV 7bdca33a87
[YouTube] Ensure that an additional player response is the correct one
If YouTube detect that requests come from a third party client, they may
replace the real player response by another one of a video saying that this
content is not available on this app and to watch it on the latest version of
YouTube. We can detect this by checking whether the video ID of the player
response returned is the same as the one requested by the extractor.
2022-08-12 19:20:31 +02:00
AudricV c82317e318
[YouTube] Spoof more mobile clients
Additional parameters have been added to the player requests of ANDROID and IOS
clients:

- for both clients: osName and osVersion: their respective values are:
  - for the ANDROID one: Android and 12;
  - for the IOS one: iOS and 15.6.0.19G71.
- for the ANDROID client: androidTargetSdkVersion, with the Android SDK version
  corresponding to the Android version used in the player requests of this
  client. This parameter is now required with this client to be sure to get a
  correct player response, otherwise, the one of a video saying that this
  content is not available in this app and to watch it with the latest version
  of YouTube can be returned instead;
- for the IOS client: deviceMake, with Apple as its value.

The iOS version sent in the IOS client player requests has been also updated to
the version 15.6 of the OS.

Finally, a comment about the requirement to use the signature timestamp from
the player JavaScript base file for HTML5 player requests on videos with
obfuscated URLs has been added and replaces a previous one which may be not
true.
2022-08-12 19:20:31 +02:00
AudricV d0549a5a52
[YouTube] Update client versions and use a real version for the iOS client
The iOS version can be got easily in fact, by looking at the What's New section of the App Store' app page.
2022-08-12 19:20:31 +02:00
AudricV d7e678aca2
[YouTube] Improve WEB client version and API key HTML extraction
Common code in WEB client version HTML extraction has been deduplicated, usage of the Java 8 Stream API has been made and initial data fallback has been used as a last resort.
This means that the client version extraction from regexes will be used before this fallback, as it doesn't contain the full client version.
This can be used as a way to fingerprint the extractor, even if it seems to be not the case.
2022-08-12 19:20:30 +02:00
AudricV 6a885ef5ab
Merge pull request #891 from Theta-Dev/fix-throttling-decrypter
[YouTube] Fix extraction of more complex nsig functions
2022-08-12 18:24:21 +02:00
AudricV 5b548340e8
[YouTube] Catch any exception in YoutubeThrottlingDecrypter.apply and improve docs
This will prevent any future extractor break due to decryption failure, like it was excepted to be the case before.

Some documentation about the throttling decryption has been also improved.
2022-08-12 17:49:36 +02:00
ThetaDev 52ded6e3d7
Handle curly braces inside strings in StringUtils.matchToClosingParenthesis
This is required to extract fully more complex YouTube nsig functions.
2022-08-12 16:32:00 +02:00
Stypox d12003651b
Merge pull request #886 from Isira-Seneviratne/toArray_improvements
Make improvements to methods using toArray().
2022-08-06 22:34:23 +02:00
Isira Seneviratne 7daca10a06 Make improvements to methods using toArray(). 2022-08-06 05:21:12 +05:30
Stypox 2906be22af
Merge pull request #881 from Isira-Seneviratne/String_join
Use String.join() and Collectors.joining().
2022-08-04 12:09:33 +02:00
Stypox 4ddb96a86f
Merge pull request #883 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.9.1
Bump gson from 2.9.0 to 2.9.1
2022-08-04 11:20:41 +02:00
Isira Seneviratne 64771c5712 Use String.join() and Collectors.joining(). 2022-08-04 05:18:13 +05:30
Stypox fc8b5ebbc6
Merge pull request #878 from Isira-Seneviratne/Use_Collections
Use Collections methods.
2022-08-03 22:50:41 +02:00
Stypox 4a4939d89c
Merge pull request #877 from Isira-Seneviratne/Use_Objects_requireNonNull
Use Objects.requireNonNull().
2022-08-03 22:45:35 +02:00
Stypox c336bd58a5
Merge pull request #879 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.0
Bump junit-bom from 5.8.2 to 5.9.0
2022-08-03 21:06:20 +02:00
dependabot[bot] 325af31e5f
Bump gson from 2.9.0 to 2.9.1
Bumps [gson](https://github.com/google/gson) from 2.9.0 to 2.9.1.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.9.0...gson-parent-2.9.1)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-08-01 09:26:14 +00:00
dependabot[bot] d905636021
Bump junit-bom from 5.8.2 to 5.9.0
Bumps [junit-bom](https://github.com/junit-team/junit5) from 5.8.2 to 5.9.0.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.8.2...r5.9.0)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-07-27 09:20:12 +00:00
Isira Seneviratne 1af6b8eedb Use Collections.singletonList(). 2022-07-27 07:35:57 +05:30
Isira Seneviratne ff60e05c76 Use Collections.singletonMap(). 2022-07-27 07:35:57 +05:30
Isira Seneviratne 682a4263e5 Use Objects.requireNonNull(). 2022-07-27 06:55:26 +05:30
Stypox 8c5f014a6f
Merge pull request #869 from mhmdanas/add-workflow-permissions
Use minimum required permissions for GitHub workflows
2022-07-13 18:58:32 +02:00
Stypox 954a294e27
Merge pull request #870 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.2
Bump jsoup from 1.15.1 to 1.15.2
2022-07-13 17:48:06 +02:00
Mohammed Anas f57049d2c0
Use temurin instead of adopt (#868)
See
https://blog.adoptopenjdk.net/2021/08/goodbye-adoptopenjdk-hello-adoptium/.
2022-07-10 18:38:08 +03:00
Stypox 5bd7bf20cc
Merge pull request #867 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.1
Bump spotbugs-annotations from 4.7.0 to 4.7.1
2022-07-06 14:23:44 +02:00
Stypox 5ab74b3631
Merge pull request #857 from FireMasterK/video-title
Get original untranslated title for YouTube
2022-07-06 10:26:45 +02:00
dependabot[bot] 122365005a
Bump jsoup from 1.15.1 to 1.15.2
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.15.1 to 1.15.2.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.1...jsoup-1.15.2)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-07-04 09:38:16 +00:00
mhmdanas 414186cff5 Use minimum required permissions for GitHub workflows
This reduces the attack surface if the workflows are ever compromised.
2022-07-03 23:55:28 +03:00
dependabot[bot] e9b4be3e3c
Bump spotbugs-annotations from 4.7.0 to 4.7.1
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.7.0 to 4.7.1.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.0...4.7.1)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-06-28 09:15:07 +00:00
Stypox 5219a705ba
Merge pull request #864 from AudricV/fetch-android-client-for-ended/post-livestreams
[YouTube] Fetch the ANDROID client for ended/post livestreams
2022-06-24 14:55:20 +02:00
AudricV 090debd83b
[YouTube] Fetch the ANDROID client for ended/post livestreams
The ANDROID client was only fetched for video contents, where it can be useful on ended/post livestreams, if the n parameter of the WEB client cannot be decrypted, to avoid throttling issues (because the WEB client was only used before for ended/post livestreams).

It also provides an exclusive 48kbps M4A audio format in the adaptiveFormats array of the JSON player response, like other mobile clients (which can be also extracted from the response of the DASH manifest URL returned into the WEB client player's response, but the DASH manifest is not used by the extractor).

A note about non-fatality of fetching or parsing issues of the ANDROID and IOS clients has been added.
2022-06-21 18:53:49 +02:00
litetex a26bcc55c4
Merge pull request #845 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.0
Bump spotbugs-annotations from 4.6.0 to 4.7.0
2022-06-19 15:36:55 +02:00
dependabot[bot] 424eb1c559
Bump spotbugs-annotations from 4.6.0 to 4.7.0
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.6.0 to 4.7.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.6.0...4.7.0)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-06-19 13:29:18 +00:00
litetex 2712c3d788
Merge pull request #847 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.1
Bump jsoup from 1.14.3 to 1.15.1
2022-06-19 15:28:43 +02:00
dependabot[bot] 281d2b9f81
Bump jsoup from 1.14.3 to 1.15.1
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.14.3 to 1.15.1.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.14.3...jsoup-1.15.1)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-06-19 13:15:40 +00:00
litetex f775155d25
Merge pull request #846 from litetex/remove-unused-methods
Remove unused methods
2022-06-19 15:12:15 +02:00
litetex 9d625dd75f
Merge pull request #860 from gliptak/patch-1
Correct PeertubePlaylistExtractorTest unit tests
2022-06-19 15:09:27 +02:00
Gábor Lipták 28c9340d69
Correct unit tests 2022-06-18 11:18:38 -04:00
Stypox 1b51eab664
Merge pull request #859 from AudricV/delivery-methods-fixes-and-improvements
Fix extraction of some properties in ItagItems for YouTube livestreams and post-live streams and remove completely SoundCloud HLS workaround
2022-06-17 15:46:13 +02:00
AudricV 301a795ed3
[SoundCloud] Remove completely workaround for HLS streams
SoundCloud is currently removing this workaround completely, so there is no need to keep it, because it impacts the loading time (a HLS playlist was downloaded and parsed).
2022-06-16 12:12:54 +02:00
AudricV e960a417ec
[YouTube] Fix extraction of fps, audioSampleRate and audioChannels fields for ItagItems of live streams and post live streams
These values were only set before for video streams.

A fallback for the audio channels count has been added, in order to prevent exceptions when generating DASH manifests of audio streams: the fallback value is 2, because most audio streams on YouTube have 2 audio channels.
2022-06-16 12:12:54 +02:00
Stypox c8a77da2ab
Merge pull request #810 from TiA4f8R/delivery-methods-v2
Support delivery methods other than progressive HTTP
2022-06-02 22:44:24 +02:00
Kavin 7635aeed2c
Get original untranslated title for YouTube. 2022-06-02 09:57:52 +01:00
TiA4f8R 287d1dfd63
[SoundCloud] Use the HLS delivery method for all streams and extract only a single stream URL from HLS manifest for MP3 streams
SoundCloud broke the workaround used to get a single file from HLS manifests for Opus manifests, but it still works for MP3 ones.

The code has been adapted to prevent an unneeded request (the one to the Opus HLS manifest) and the HLS delivery method is now used for SoundCloud MP3 and Opus streams, plus the progressive one (for tracks which have a progressive stream (MP3) and for the ones which doesn't have one, it is still used by trying to get a progressive stream, using the workaround).

Streams extraction has been also moved to Java 8 Stream's API and the relevant test has been also updated.
2022-05-29 19:08:18 +02:00
Stypox b3c620f0d8
Apply code review and Streams rework 2022-05-28 12:00:58 +02:00
Stypox d652e05874
[MediaCCC] Fix comments about containsSimilarStream 2022-05-28 12:00:58 +02:00
Stypox 044639c32b
Solve some review comments 2022-05-28 12:00:57 +02:00
litetex c33d392958
Fixed typo XEE → XXE (Xml eXternal Entity attack)
See also:
https://en.wikipedia.org/wiki/XML_external_entity_attack
https://owasp.org/www-community/vulnerabilities/XML_External_Entity_(XXE)_Processing
2022-05-28 12:00:57 +02:00
TiA4f8R fffbbee7f3
[YouTube] Add missing Nonnull annotations in getCache method of YouTube DASH manifest creators 2022-05-28 12:00:56 +02:00
TiA4f8R f7b1515290
[YouTube] Refactor DASH manifests creation
Move DASH manifests creation into a new subpackage of the YouTube package, dashmanifestcreators.
This subpackage contains:

- CreationException, exception extending Java's RuntimeException, thrown by manifest creators when something goes wrong;
- YoutubeDashManifestCreatorsUtils, class which contains all common methods and constants of all or a part of the manifest creators;
- a manifest creator has been added per delivery type of YouTube streams:
  - YoutubeProgressiveDashManifestCreator, for progressive streams;
  - YoutubeOtfDashManifestCreator, for OTF streams;
  - YoutubePostLiveStreamDvrDashManifestCreator, for post-live DVR streams (which use the live delivery method).

Every DASH manifest creator has a getCache() static method, which returns the ManifestCreatorCache instance used to cache results.

DeliveryType has been also extracted from the YouTube DASH manifest creators part of the extractor and moved to the YouTube package.

YoutubeDashManifestCreatorTest has been updated and renamed to YoutubeDashManifestCreatorsTest, and YoutubeDashManifestCreator has been removed.

Finally, several documentation and exception messages fixes and improvements have been made.
2022-05-28 12:00:56 +02:00
TiA4f8R f17f7b9842
Apply requested changes in YoutubeParsingHelper 2022-05-28 12:00:55 +02:00
TiA4f8R 301b9fa024
Remove hashCode and equals methods overrides of Stream classes 2022-05-28 12:00:55 +02:00
TiA4f8R 2f3920c648
[YouTube] Return approxDurationMs value from YouTube's player response in ItagItems generated
This change allows to build DASH manifests using YoutubeDashManifestCreator with the real duration of streams and prevent potential cuts of the end of progressive streams, because the duration in YouTube's player response is in seconds and not milliseconds.
2022-05-28 12:00:54 +02:00
TiA4f8R 4158fc46a0
[Bandcamp] Fix regression of Opus radio streams extraction
When moving opus-lo into a constant, opus-lo was renamed to opus_lo and was only used if no MP3 stream was available (which was not the case before the changes in BandcampRadioStreamExtractor related to the addition of the support of all delivery methods), so these changes removed the ability to get Opus streams of Bandcamp radios.

This commit reverts this unwanted change.
2022-05-28 12:00:54 +02:00
TiA4f8R 54d323c2ae
Fix Checkstyle issue in YoutubeDashManifestCreator 2022-05-28 12:00:53 +02:00
Stypox 2321822844
Rename Stream's baseUrl to manifestUrl 2022-05-28 12:00:53 +02:00
Stypox cfc13f4a6f
[YouTube] Reduce exception generation code and move several attributes of MPD documents into constants
Rename YoutubeDashManifestCreationException to CreationException.
Also use these constants in YoutubeDashManifestCreatorTest.
2022-05-28 12:00:53 +02:00
Stypox 00bbe5eb4d
[YouTube] Make exception messages smaller 2022-05-28 12:00:52 +02:00
Stypox 4da05afe11
[YouTube] Inline collectSegmentsData in YoutubeDashManifestCreator 2022-05-28 12:00:52 +02:00
Stypox 3708ab9ed5
[YouTube] Refactor YoutubeDashManifestCreator
- Remove all of the methods used to access caches and replace them with three caches getters
- Rename caches to shorter and more meaningful names
- Remove redundant @throws tags that just say "if this method fails to do what it should do", which is obvious
2022-05-28 12:00:51 +02:00
Stypox 5c83409039
[YouTube] Rewrite manifest test and rename long methods 2022-05-28 12:00:51 +02:00
Stypox 8226fd044f
[YouTube] Suppress Sonar security warning for XEE 2022-05-28 12:00:50 +02:00
Stypox ba68b8c014
[YouTube] Secure DashManifestCreator against XEE attack
Also remove duplicate lines from javadoc comments, otherwise checkstyle complaints.
XEE prevention is inspired from https://github.com/ChuckerTeam/chucker/pull/201/files
For an explanation of XEE see https://rules.sonarsource.com/java/RSPEC-2755, though the solution reported there causes crashes on Android
2022-05-28 12:00:50 +02:00
Stypox 159d05c91b
Remove unused DashMpdParser (but kept in git history) 2022-05-28 12:00:49 +02:00
Stypox 50272db946
Apply reviews: improve comments, remove FILE, remove Stream#equals(Stream) 2022-05-28 12:00:49 +02:00
TiA4f8R 07b045f20d
[YouTube] Support the iOS client in YoutubeDashManifestCreator and decrypt again the n parameter, if present, for all clients
This commits reverts a new behavior introduced in this branch, which only applied the decryption if needed on streams from the WEB client.
Also fix rebase issues and documentations style in YoutubeDashManifestCreator.
2022-05-28 12:00:48 +02:00
TiA4f8R 436ddde29f
Use assertThrows in YoutubeDashManifestCreatorTest 2022-05-28 12:00:48 +02:00
TiA4f8R d64d7bbd01
Move ManifestCreatorCache tests to a separate class and remove override of equals and hashCode methods in ManifestCreatorCache
These methods don't need to be overriden, as they are not excepted to be used in collections.
Also improve the toString method of this class, which contains also now clearFactor and maximumSize attributes and for each operations.
2022-05-28 12:00:48 +02:00
TiA4f8R 2fb1a412a6
Fix Checkstyle issues, revert resolution string changes for YouTube video streams and don't return the rn parameter in DASH manifests
This parameter is still used to get the initialization sequence of OTF and POST-live streams, but is not returned anymore in the manifests.
It has been removed in order to avoid fingerprinting based on the number sent (e.g. when starting to play a stream close to the end and using 123 as the request number where it should be 1) and should be added dynamically by clients in their requests.
The relevant test has been also updated.

Checkstyle issues in YoutubeDashManifestCreator have been fixed, and the changes in the resolution string returned for video streams in YoutubeStreamExtractor have been reverted, as they create issues on NewPipe right now.
2022-05-28 12:00:47 +02:00
TiA4f8R f61e2092a1
[YouTube] Return a copy of the hardcoded ItagItem instead of returning the reference to the hardcoded one in ItagItem.getItag
To do so, a copy constructor has been added in the class.

This fixes, for instance, an issue in NewPipe, in which the ItagItem values where not the ones corresponsing to a stream but to another, when generating DASH manifests.
2022-05-28 12:00:47 +02:00
TiA4f8R aa4c10e751
Improve documentation and adress most of the requested changes
Also fix some issues in several places, in the code and the documentation.
2022-05-28 12:00:46 +02:00
TiA4f8R 6985167e63
Add tests for YoutubeDashManifestCreator and ManifestCreatorCache
The test added in YoutubeDashManifestCreator uses a video of the Creative Commons channel, licenced under the Creative Commons Attribution licence (reuse allowed).
Also remove public keywords of tests in UtilsTest, as suggested by SonarLint, because they are not needed with Junit 5.
2022-05-28 12:00:46 +02:00
TiA4f8R f6ec7f9a61
Update DefaultStreamExtractorTest and SoundcloudStreamExtractorTest to support changes made in Stream classes 2022-05-28 12:00:45 +02:00
TiA4f8R 7477ed0f3d
[YouTube] Add ability to generate manifests of progressive, OTF and post live streams
A new class has been added to do so: YoutubeDashManifestCreator.
It relies on a new class: ManifestCreatorCache, to cache the content, which relies on a new pair class named Pair.
Results are cached and there is a cache per delivery type, on which cache limit, clear factor, clearing and resetting can be applied to each cache and to all caches.
Look at code changes for more details.
2022-05-28 12:00:45 +02:00
TiA4f8R a857684442
Apply changes in YoutubeStreamExtractor
Extract post live DVR streams as post live streams instead of live streams.

A new class has been in order to improve code: ItagInfo, which stores an itag, the content (URL) extracted and if its an URL or not.
A functional interface has been added in order to abstract the stream building: StreamBuilderHelper.
Also add the cver parameter added by the desktop web client on the corresponding streams (a new method has been added in YoutubeParsingHelper to check this and another for Android streams).

Some code in these classes has been also refactored/improved/optimized.
2022-05-28 12:00:44 +02:00
TiA4f8R 4330b5f7be
Add POST_LIVE_STREAM and POST_LIVE_AUDIO_STREAM stream types
This allows the extractor to determine if a content is an ended audio or video livestream.
2022-05-28 12:00:43 +02:00
TiA4f8R 881969f1da
Apply changes in all StreamExtractors except YouTube's one and fix extraction of PeerTube audio streams as video streams
Some code in these classes has been also refactored/improved/optimized.
Also fix the extraction of PeerTube audio streams as video streams, which are now returned as audio streams.
2022-05-28 12:00:43 +02:00
TiA4f8R d5f3637fc3
[YouTube] Return more values returned inside the ItagItems of the player response and deprecate use of public audio and video fields
These fields can be now replaced by a getter and a setter.

New fields have been added and will allow the creation of DASH manifests for OTF and ended livestreams. There are:
- contentLength;
- approxDurationMs;
- targetDurationSec;
- sampleRate;
- audioChannels.
2022-05-28 12:00:42 +02:00
TiA4f8R 7c67d46e09
Move DashMpdParser to the YouTube package and fix extraction of streams
DashMpdParser is only working with YouTube streams, as it uses the ItagItem class.
Also update creation of AudioStreams and VideoStreams objects.
2022-05-28 12:00:41 +02:00
TiA4f8R ad993b920f
Remove fetching of the DASH manifest extracted when getting information of a content with StreamInfo
DashMpdParser is only working with YouTube streams, as it uses the ItagItem class.

Also improve code and comments of StreamInfo (especially final use where possible).
2022-05-28 12:00:41 +02:00
TiA4f8R 2f061b8dbd
Add support of other delivery methods than progressive HTTP in Stream classes
Stream constructors are now private and streams can be constructed with new Builder classes per stream class. This change has been made to prevent creating and using several constructors in stream classes.

Some default cases have been also added in these Builder classes, so not everything has to be set, depending of the service and the content.
2022-05-28 12:00:27 +02:00
Stypox 1dc80957d8
Merge pull request #848 from TiA4f8R/fix_yt_music_website_extraction_eu
[YouTube] Fix extracting YouTube Music client version and key in the EU
2022-05-22 22:05:05 +02:00
TiA4f8R c34b5e3a8b
[YouTube] Fix extraction of YouTube Music client version and API key when using YouTube Music's website in EU
Google returns now the consent page of YouTube for YouTube Music in EU, which can be also avoided by adding the ucbcb parameter to the URL with the value 1 ("?ucbcb=1").
2022-05-15 11:20:06 +02:00
litetex 2015eb374a Removed more unused methods 2022-05-09 21:05:03 +02:00
litetex f69b0ff77b Remove unused methods 2022-05-09 20:59:25 +02:00
Stypox 23fa31a1ec
Merge pull request #843 from TiA4f8R/fix-checkstyle-issues
Remove Checkstyle suppressions file and fix Checkstyle issues
2022-05-02 22:07:27 +02:00
TiA4f8R 3c3cd78676
Remove Checkstyle suppressions file and fix Checkstyle issues introduced in 24e8399 and 8c1041d
The Checkstyle suppressions file is now replaced by // CHECKSTYLE:OFF and // CHECKSTYLE:ON comments.
2022-05-02 21:51:25 +02:00
Stypox 2e1c5c119d
Merge pull request #822 from Stypox/more-refactors
More refactors
2022-05-02 19:03:54 +02:00
Stypox 598ebb92ea
Merge pull request #839 from TeamNewPipe/bandcamp/extract-length
Bandcamp: extract stream length
2022-05-02 15:49:41 +02:00
litetex 5db4d1faf3
Merge pull request #782 from litetex/cleanup-yt-stream-extractor
Cleanup of ``YoutubeStreamExtractor`` and some related classes
2022-05-01 16:44:11 +02:00
litetex fe30eb43a9 Cleanup ``YoutubeStreamExtractor`` and some related classes
* Fixed obvious sonar(lint) warnings
* Abstracted some code (get*Streams)
* Used some new lines to make code better readable
* Chopped down brace-jungle in some methods
* Use StandardCharset (Java 8 4tw)
2022-05-01 16:39:07 +02:00
Stypox c2b5370517
Apply suggestions: improve switch and use EMPTY_STRING 2022-04-30 16:39:51 +02:00
Stypox 7c78c39230
Merge pull request #821 from litetex/cleanup-TimeAgoParser-java
Cleanup ``TimeAgoParser``
2022-04-30 16:20:09 +02:00
Stypox ac1c22d81c
Release NewPipe Extractor 0.22.1 2022-04-28 11:05:52 +02:00
Stypox 3f5c8962ac
Merge pull request #841 from TiA4f8R/yt-respect-order-of-clients-for-streams
[YouTube] Use correct order of clients to get better streams first
2022-04-28 10:58:49 +02:00
TiA4f8R 9f9af35adb
[YouTube] Fix regression introduced in the order of streams used when adding more parameters to InnerTube requests, using the iOS client for livestreams and more 2022-04-25 20:23:04 +02:00
Fynn Godau c38c016de5 Bandcamp: extract stream length 2022-04-24 21:24:19 +02:00
litetex 5f8f3929bd
Merge pull request #838 from Nickoriginal/update_user_agent
Update User-Agent in NPE
2022-04-24 14:52:18 +02:00
Nickoriginal b5f3f9eb90
Update USER_AGENT to match the latest Firefox 2022-04-24 00:27:23 +03:00
litetex 9f665db4e1
Merge pull request #828 from TeamNewPipe/dependabot/github_actions/actions/setup-java-3
Bump actions/setup-java from 2 to 3
2022-04-23 00:09:44 +02:00
Stypox 52fa2d939a
Fix javadoc formatting error causing deployment to fail 2022-04-16 17:07:07 +02:00
Stypox 95a4b05acc
Release NewPipe Extractor 0.22.0 2022-04-16 16:57:58 +02:00
Tobi b77c72fb88
Merge pull request #832 from Stypox/hotfix-decrypter
Fix YouTube throttling decrypter function parsing
2022-04-15 13:35:10 +02:00
Stypox dcb7483dcf
Fix YouTube throttling decrypter function parsing 2022-04-15 13:10:19 +02:00
Robin bebdc55ad4
Merge pull request #829 from TiA4f8R/yt-subtitles-age-restricted-videos-fix
[YouTube] Extract subtitles for age-restricted videos
2022-04-14 10:31:16 +02:00
TiA4f8R ef49cd0007
[YouTube] Extract subtitles for age-restricted videos
Subtitles of age-restricted videos can be extracted since the InnerTube API migration, so there is no reason to not extract them anymore.
2022-04-11 22:09:56 +02:00
dependabot[bot] 00b51a2bf8
Bump actions/setup-java from 2 to 3
Bumps [actions/setup-java](https://github.com/actions/setup-java) from 2 to 3.
- [Release notes](https://github.com/actions/setup-java/releases)
- [Commits](https://github.com/actions/setup-java/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/setup-java
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-04-11 09:16:36 +00:00
TiA4f8R 2e92d718a2
Merge pull request #780 from TiA4f8R/yt-more-params-innertube-requests
[YouTube] Add more parameters to InnerTube requests, use the iOS client for livestreams and fix extraction of embeddable age-restricted videos and contents with a warning before playback
2022-04-09 18:15:18 +02:00
TiA4f8R b30e341559
Add link to learn more about the issue which makes YoutubeStreamExtractorRelatedMixTest.testRelatedItems test disabled 2022-04-04 20:36:59 +02:00
TiA4f8R 70812fa611
Update YouTube stream mocks and disable YoutubeStreamExtractorRelatedMixTest
Mixes seems to be not given by YouTube anymore if you use a PENDING consent cookie value.
As mocks needs to updated, the test is always failing because of this change.
2022-04-02 19:28:49 +02:00
TiA4f8R 67288a0191
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.

This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.

The outdated code for these contents has been removed.

Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.

Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 19:06:36 +02:00
TiA4f8R 11b5a222c4
Deduplicate code of getStringResultFromRegexArray methods in Utils
Also revert indentation in Utils.mixedNumberWordToLong.
2022-04-02 18:40:00 +02:00
TiA4f8R 9ca647a750
Update mocks 2022-03-27 22:10:59 +02:00
TiA4f8R dfa4239661
Fix missing imports and Checkstyle issues 2022-03-27 22:10:57 +02:00
TiA4f8R 6d27996ac4
Improve code of getStringResultFromRegexArray methods in Utils 2022-03-27 22:10:57 +02:00
TiA4f8R 2e3da445e6
[YouTube] Add documentation about parameters added and clients versions and key
Also move the iPhone device machine id to a constant, explain how it is used and move the licence in the header of the file, and fix missing imports in YoutubeStreamExtractor (due to a rebase issue).
2022-03-27 22:10:57 +02:00
TiA4f8R 1dad3bfe8b
[YouTube] Update again hardcoded client versions and update mobile user agents
Also provide ability to get mobile user-agents used for mobile InnerTube requests and deduplicate related code.
2022-03-27 20:52:40 +02:00
TiA4f8R 3d38459cf3
[YouTube] Reduce InnerTube response sizes by adding the prettyPrint parameter with the false value
InnerTube responses return pretty printed responses, which increase responses' size for nothing.

By using the prettyPrint parameter on requests and setting its value to false, responses are not pretty printed anymore, which reduces responses size, and so data transfer and processing times.
This usage has been recently deployed by YouTube on their websites.
2022-03-27 20:52:40 +02:00
litetex 349ba8db7f
Improve tests and randomness
- Use the existing RNG inside YoutubeParsingHelper
- Deduplicated test-setup for YouTube tests
- Minor improvements
2022-03-27 20:52:38 +02:00
TiA4f8R 52376949e5
Add setSeedForVideoTests method in YoutubeStreamExtractor tests
In order to use still use mocks with the generation of random strings in player requests, we need to use YoutubeParsingHelper.setSeedForVideoTests() method in every stream test.
2022-03-27 20:51:39 +02:00
TiA4f8R d0d91e6690
Adress requested changes 2022-03-27 20:51:39 +02:00
TiA4f8R b6bc521f0d
[YouTube] Update client versions again 2022-03-27 20:51:38 +02:00
TiA4f8R 26f93f5bb0
[YouTube] Extract streams of livestreams from the iOS client and disabled the Android client for livestreams
The iOS client is only enabled for livestreams and the Android client is now only enabled for videos, both by default.

A way to force, or not, the fetch of both clients have been added with two new static methods in YoutubeStreamExtractor.
2022-03-27 20:51:38 +02:00
TiA4f8R 7d07924de8
[YouTube] Try to use lighter requests when extracting client version and key from YouTube and YouTube Music
This is done by fetching https://www.youtube.com/sw.js for YouTube and https://music.youtube.com/sw.js for YouTube Music.

Two new methods in Utils class have been added which allow to try to get a match of regular expressions in a string array, or a Pattern array, on a content, on a specific index or 0.
Also some code refactoring has been made in this class.
2022-03-27 20:51:38 +02:00
TiA4f8R 05b7fee23b
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.

For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.

Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).

For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.

This commit also fixes a small bug with the Android User-Agent string.

Some code improvements have been also made.
2022-03-27 20:51:38 +02:00
TiA4f8R 83f374bff1
[YouTube] Update client versions and fix a bug when using resetClientVersionAndKey method
The boolean keyAndVersionExtracted in YoutubeParsingHelper was not set to false when resetting the client version and the key, which makes the extractor uses null on the next getting of the client version or the key if the clientVersion and the key were extracted before.
Also update client versions.
2022-03-27 20:51:38 +02:00
Stypox 8c1041def6
Add @ null annotations where Android Studio suggested it
That is, basically where the overriding function was missing an annotation from the base method.

Also apply renaming of emptyDescription to EMPTY_DESCRIPTION
2022-03-26 22:07:14 +01:00
Stypox adbbdc7a5b
[YouTube] Fix regex warning: use ' {2}' instead of ' ' 2022-03-26 22:07:14 +01:00
Stypox 24e83997b4
[Bandcamp] Add Java 8 streams 2022-03-26 22:07:12 +01:00
Stypox 349990fd48
Fix redundant escape \\ in regex in Utils 2022-03-26 22:01:30 +01:00
litetex 3bf7aa3762 Cleanup ``TimeAgoParser`` 2022-03-26 21:09:31 +01:00
litetex 5a18730845
Merge pull request #813 from litetex/fix-test-2022-03
Fixed tests (+ Youtube shorts in channels)
2022-03-26 21:09:10 +01:00
litetex af82edf9dc Fix checkstyle problems 2022-03-26 20:54:20 +01:00
litetex 29408f9356 ``YoutubeMusicSearchExtractorTest$Suggestion`` has no mock data 2022-03-26 20:52:28 +01:00
litetex fcee247f9f Switched to mockonly tests
* Made the reason why the tests are mockonly mandatory
2022-03-26 20:52:28 +01:00
litetex 0fceb4686b Added missing mock data 2022-03-26 20:52:27 +01:00
litetex 70b20f0d6b Updated mock data after conflicts 2022-03-26 20:52:27 +01:00
litetex 6a6c9359af Fixed kurzgesagt test - They changed their description... 2022-03-26 20:52:27 +01:00
litetex 26596215fa Refactored ``MediaCCCRecentListExtractorTest`` 2022-03-26 20:52:26 +01:00
litetex 53962bfd7d Disabled suggestion test for now
Has to be done once YT's backed works again...
2022-03-26 20:52:26 +01:00
litetex 66dc5e8bb8 API hardening against changes 2022-03-26 20:52:26 +01:00
litetex 5d58156cde Update mock data 2022-03-26 20:52:25 +01:00
litetex 7598b40957 Workaround for incorrect duration for "YT shorts" videos in channels
As a workaround 0 is returned as duration for such videos.
See also https://github.com/TeamNewPipe/NewPipe/issues/8034
2022-03-26 20:52:24 +01:00
litetex 164e21b5af Fixed ``MediaCCCRecentKiosk``
Ignore faulty data/items (with duration <= 0)
2022-03-26 20:52:23 +01:00
litetex 49dfdae5d2 Removed useless throw declaration 2022-03-26 20:51:38 +01:00
litetex 69a58fd3d1 Don't use sysout 2022-03-26 20:51:37 +01:00
litetex 639be7adda Minimized some code 2022-03-26 20:51:37 +01:00
litetex 2bd4299563 Fixed test: Peertube - Use new comments and videos
Old comments/videos got deleted
2022-03-26 20:49:39 +01:00
litetex 606a386a98 Fix tests: What's peertube has less comments now 2022-03-26 20:49:39 +01:00
litetex 4d1a1c8fb8 Better test for ``MediaCCCRecentListExtractorTest``
* Use assertAll
* Show which item is affected
2022-03-26 20:49:38 +01:00
litetex ba43dbaa28 Fix Tests: The channel lost abos 2022-03-26 20:49:38 +01:00
litetex ff436e5740 Fixed test: YT mix playlist 2022-03-26 20:49:38 +01:00
litetex bb49f7d857
Merge pull request #817 from Stypox/checkstyle
Add checkstyle
2022-03-26 20:46:49 +01:00
litetex 9c07e8a664 Fix useage of wrong object 2022-03-26 20:17:50 +01:00
litetex 93d6e5c4d5 CI: Upload test reports when failure occurs 2022-03-26 19:57:17 +01:00
litetex 804e57004f Fixed new checkstyle problems from dev 2022-03-26 19:46:10 +01:00
litetex 33347ac18b Removed unused methods
``contentFilters`` and ``sortfilter`` are get inside the ``ListLinkHandler`` and not the ``ListLinkHandlerFactory``
 ``ListLinkHandlerFactory`` only passes these values through when ``fromQuery`` is called
2022-03-26 19:43:11 +01:00
litetex ec5b54c38b Removed unused class 2022-03-26 19:43:10 +01:00
litetex 8771af7ba5 Restored original naming 2022-03-26 19:43:09 +01:00
litetex 01cfde0285 Fixed typo 2022-03-26 19:43:09 +01:00
Stypox bdadcfa1f7 Legitimately suppress remaining checkstyle warnings 2022-03-26 19:43:08 +01:00
Stypox 740a37a2de [YouTube] Fix checkstyle issues 2022-03-26 19:42:40 +01:00
Stypox 9dc17cd1ca [Soundcloud] Fix checkstyle issues 2022-03-26 19:40:20 +01:00
Stypox 9ab32cb2e7 [Peertube] Fix checkstyle issues 2022-03-26 19:40:19 +01:00
Stypox 9f7e06c817 [MediaCCC] Fix checkstyle issues 2022-03-26 19:40:18 +01:00
Stypox 3a94839359 [Bandcamp] Fix checkstyle issues 2022-03-26 19:40:17 +01:00
Stypox 08dff33002 Use Java 8 streams in NewPipe class 2022-03-26 19:40:15 +01:00
Stypox c2446ecff0 Use Java 8 streams and deduplicate code in MediaFormat class 2022-03-26 19:40:15 +01:00
Stypox d79e20340c Fix checkstyle issues in root package extractor/
Note: not all issues were fixed because MediaFormat and ServiceList use a specific formatting that makes sense for them
2022-03-26 19:40:14 +01:00
Stypox ca7c63f273 Fix remaining checkstyle issues in utils/ subpackage 2022-03-26 19:40:13 +01:00
Stypox 1d5f22e41f Fix checkstyle issues & more in JsonUtils
Also use Java 8 streams and extract duplicate code to getInstanceOf function
2022-03-26 19:40:13 +01:00
Stypox 87d2834986 Fix checkstyle issues & more in DonationLinkHelper
Also add comment about the class being unused and replace the fixLink function with Utils.stringToUrl()
2022-03-26 19:40:12 +01:00
Stypox bd7b362040 Fix checkstyle issues & more in DashMpdParser
Also remove useless null check on ItagItem.getItag() as that function already throws an exception if there is no itag
2022-03-26 19:40:11 +01:00
Stypox 8aba2b47b0 Fix checkstyle issues in subpackages with abstract classes 2022-03-26 19:40:10 +01:00
Stypox e4951a0623 Refactor code handling http headers in downloader.Request 2022-03-26 19:37:47 +01:00
Stypox 37690058d2 Add checkstyle to extractor gradle project
With respect to NewPipe's checkstyle.xml, checkstyle is disabled for javadoc comments. There is no need for strict rules over comments here in the extractor, as sometimes javadocs are just needed to clarify a small thing and having empty/meaningless @param or @throws is useless.
2022-03-26 19:37:46 +01:00
litetex 9284569c84
Merge pull request #774 from TeamNewPipe/dependabot/gradle/org.mozilla-rhino-1.7.14
Bump rhino from 1.7.13 to 1.7.14
2022-03-26 17:25:55 +01:00
litetex 358e619a27
Merge pull request #819 from TeamNewPipe/dependabot/github_actions/actions/cache-3
Bump actions/cache from 2 to 3
2022-03-23 21:02:18 +01:00
dependabot[bot] a2c9ad104c
Bump actions/cache from 2 to 3
Bumps [actions/cache](https://github.com/actions/cache) from 2 to 3.
- [Release notes](https://github.com/actions/cache/releases)
- [Commits](https://github.com/actions/cache/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-21 09:12:36 +00:00
XiangRongLin aa6b7272a4
Merge pull request #804 from Stypox/fix-yt-music-mix
[YouTube] Fix music mixes in some countries
2022-03-20 08:35:56 +01:00
Stypox 843945f2a6
Merge pull request #818 from litetex/update-license
Update the license to the latest version
2022-03-19 23:13:55 +01:00
litetex fa76098750
Delete copyright
was replace by LICENSE-file
2022-03-19 21:44:38 +01:00
litetex faaf4cbec3
Update the license to the latest version of https://www.gnu.org/licenses/gpl-3.0.txt 2022-03-19 17:59:27 +01:00
XiangRongLin 7f2ea133f0
Merge pull request #788 from Stypox/mix
Add MixInfoItem and extract YouTube mixes in related items
2022-03-19 16:46:01 +01:00
Stypox 09ddb6adbb
[YouTube] Add MockOnly to method testing mixes in related items 2022-03-19 10:54:38 +01:00
Stypox 8201b3b90e
[YouTube] Parse any playlist (including music mixes) in related items 2022-03-19 10:48:13 +01:00
Stypox 13f7900816
[YouTube] Add test for genre mix 2022-03-19 10:48:13 +01:00
Stypox 279f3a20fe
[YouTube] Fix mix tests with invalid video ids
Replaces mix tests based on a strange mix type RDQM{videoId} (only reference I could find is https://github.com/ytdl-org/youtube-dl/issues/26228) and with an invalid video id of 13 characters (the first two characters were QM, but even after removing QM there still wasn't a video available at that id).
Also updates mocks.
2022-03-19 10:48:13 +01:00
Stypox d660c04838
[YouTube] Also test playlist type in playlist tests 2022-03-19 10:48:13 +01:00
Stypox 401082abe4
[YouTube] Extract playlist type in playlist extractor 2022-03-19 10:48:12 +01:00
Stypox 63ed06a710
[YouTube] Differentiate genre mixes from normal mixes
Note: genre mixes already worked, now they are just considered as such in various video id extraction and in related items
Note 2: now extracting a mix id from a *normal* youtube mix id will fail if the video id wouldn't be exactly 11 characters long
2022-03-19 10:46:31 +01:00
Stypox f19660e7d9
[YouTube] Deduplicate code extracting video id from mix id 2022-03-19 10:46:30 +01:00
Stypox 8f9d5b858e
[YouTube] Remove useless comments about mixes 2022-03-19 10:44:06 +01:00
Stypox 34a4484c72
[YouTube] Add test for a video with a mix in related items 2022-03-19 10:44:06 +01:00
Stypox 50db871d89
[YouTube] Extract mixes from streams related items 2022-03-19 10:44:06 +01:00
Stypox 638da1756c
[Mix] Create MultiInfoItemsCollector
It is a collector that can handle many extractor types, to be used when a list contains items of different types (e.g. search). It was renamed from InfoItemsSearchCollector so that it can now be used not just for search but for any extractor needing it. It supports, streams, channels, playlists and *mixes*.
2022-03-19 10:44:06 +01:00
Stypox 53673d64c6
[Mix] Add type to playlists & playlist items, to distinguish mixes 2022-03-19 10:44:06 +01:00
Stypox d8f2031619
Merge pull request #816 from Stypox/mock-only-extension
Add `@MockOnly` Junit 5 extension
2022-03-19 10:40:38 +01:00
litetex cc2e4d7104
Merge pull request #815 from litetex/fix-soundcloud-id-once-and-for-all
Removed hardcoded soundcloud HARDCODED_CLIENT_ID
2022-03-17 13:54:08 +01:00
litetex cd8088b217
Merge pull request #811 from TiA4f8R/playlists-improvements-and-yt-playlists-fixes
[YouTube] Fix the extraction of series playlists and don't return the view count as the stream count for learning playlists
2022-03-17 13:52:24 +01:00
TiA4f8R c7757c0994
Apply requested changes 2022-03-16 20:14:08 +01:00
TiA4f8R 35e082248e
Fix YouTube and SoundCloud playlists tests 2022-03-16 19:40:30 +01:00
TiA4f8R 8b3f90eb7e
[YouTube] Fix extraction of series playlists and don't return the view count as the stream count for learning playlists
ITEM_COUNT_UNKNOWN is returned when the JSON array which contains usally the number of videos is less than 3 items.
Also apply the same type of optimizations done in other PlaylistExtractors in YoutubePlaylistExtractor.
2022-03-16 19:18:58 +01:00
TiA4f8R 58a247907e
Apply changes in all playlist extractors except YoutubePlaylistExtractor
Also fix some issues in the extractors, remove uneeded overrides, use the Java 8 Stream API where possible and replace usages of Utils.UTF_8 with StandardCharsets.UTF_8 in these classes.
2022-03-16 19:18:57 +01:00
TiA4f8R fc6b45ee36
Implement some methods in PlaylistExtractor
This will prevent their override in each child class where the values corresponding to the methods could not be extracted.
2022-03-16 19:18:36 +01:00
Stypox 73d1fd472f
Add MockOnly junit 5 test extension 2022-03-16 19:03:08 +01:00
Stypox ef71a5fa0f
static final instead of final static 2022-03-16 17:24:33 +01:00
Stypox 0c37c75981
Make getDownloader static & extract getDownloaderType 2022-03-16 17:22:42 +01:00
Stypox 40aa5104b1
Merge pull request #786 from XiangRongLin/throttling_resilience
[Youtube] Make throttling decryption more resilient to api change
2022-03-16 11:03:16 +01:00
litetex ba56be8ef1 Removed hardcoded soundcloud id
It never works (long enough) so let's simply remove it...
2022-03-15 21:19:19 +01:00
XiangRongLin e726437da3
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java
Co-authored-by: Stypox <stypox@pm.me>
2022-03-15 17:10:05 +01:00
litetex e7aee0ca57
Merge pull request #807 from FireMasterK/no-commentsinfo-instance
Remove the need for a CommentsInfo instance in CommentsInfo.getMoreItems and fix PeertubeCommentsExtractorTest.Default test
2022-03-15 15:06:56 +01:00
litetex d806984aa8
Merge pull request #800 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.9.0
Bump gson from 2.8.9 to 2.9.0
2022-03-14 19:47:54 +01:00
litetex 2aa5f98c26
Merge pull request #796 from FireMasterK/streaminfo-subscriber-count
Add support for extracting channel subscriber count in StreamInfo
2022-03-14 19:40:57 +01:00
Tobi a267093e76
Merge pull request #809 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.6.0
Bump spotbugs-annotations from 4.5.3 to 4.6.0
2022-03-12 00:02:51 +01:00
dependabot[bot] fdb603e9ac
Bump spotbugs-annotations from 4.5.3 to 4.6.0
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.5.3 to 4.6.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.5.3...4.6.0)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-08 09:16:28 +00:00
FireMasterK 60cc71e944
Remove the need for a CommentsInfo instance. 2022-03-03 11:48:41 +00:00
Stypox 3a6269bca6
Merge pull request #805 from TeamNewPipe/dependabot/github_actions/actions/checkout-3
Bump actions/checkout from 2 to 3
2022-03-02 11:09:28 +01:00
dependabot[bot] e953794703
Bump actions/checkout from 2 to 3
Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-02 09:25:59 +00:00
Stypox dd8687f9fe
[YouTube] Fix music mixes in some countries 2022-03-01 23:02:56 +01:00
FireMasterK e6d334765d
Apply requested codestyle improvements. 2022-02-26 17:12:51 +00:00
FireMasterK 6950b362f2
Add test values for failing tests. 2022-02-24 12:55:13 +00:00
FireMasterK 5b0ec694a6
Add requested changes. 2022-02-24 12:50:51 +00:00
FireMasterK d290d2e393
Move variable to super Extractor classes. 2022-02-24 12:50:51 +00:00
FireMasterK ab49cb6e18
Add requested changes. 2022-02-24 12:50:51 +00:00
Kavin 4bd59f65f4
Update exception message.
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2022-02-24 12:50:51 +00:00
FireMasterK 6f3f608ab6
Add support for extracting channel subscriber count in StreamInfo. 2022-02-24 12:50:51 +00:00
Stypox 69e18c80cb
Merge pull request #799 from litetex/imporve-yt-comments-extractor
Refactored YoutubeCommentsExtractor
2022-02-23 17:58:00 +01:00
litetex 1a67ea100a Refactored code
according to review
2022-02-21 19:56:12 +01:00
litetex f79ce1f52a Refactored YoutubeCommentsExtractor
* Use Java Streaming API
* Use StandardCharsets
* Prevented several NPEs/ArrayIndexOutOfBound
* Reformatted some code so that it's easier readable
2022-02-21 19:56:09 +01:00
TobiGr dfe8716f5f Merge branch 'dev' 2022-02-18 20:30:38 +01:00
TobiGr d07e16aadd Release NewPipe Extractor 0.21.14 2022-02-18 20:29:57 +01:00
TobiGr d337e537dc [YouTube] Add a comment clarifying why the panelIdentfier can be null 2022-02-18 20:26:19 +01:00
Tobi 5c8fbeb166
Fix NullPointerException in YouTubeStreamExtractor.getStreamSegments() (#797)
Fix NullPointerException in YouTubeStreamExtractor.getStreamSegments()

Co-authored-by: litetex <40789489+litetex@users.noreply.github.com>
2022-02-16 23:39:26 +01:00
dependabot[bot] 89900431ab
Bump gson from 2.8.9 to 2.9.0
Bumps [gson](https://github.com/google/gson) from 2.8.9 to 2.9.0.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.9...gson-parent-2.9.0)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-02-14 09:15:09 +00:00
Tobi 0d6aebce25
[SoundCloud] Update clientId (#798) 2022-02-11 17:16:32 +01:00
dependabot[bot] a2a41e3bf2
Bump rhino from 1.7.13 to 1.7.14
Bumps [rhino](https://github.com/mozilla/rhino) from 1.7.13 to 1.7.14.
- [Release notes](https://github.com/mozilla/rhino/releases)
- [Changelog](https://github.com/mozilla/rhino/blob/master/RELEASE-NOTES.md)
- [Commits](https://github.com/mozilla/rhino/commits)

---
updated-dependencies:
- dependency-name: org.mozilla:rhino
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-02-07 20:29:51 +00:00
XiangRongLin 545522b80f [Youtube] Make throttling decryption more resilient to api change 2022-02-01 19:57:34 +01:00
TobiGr ebc129c392 NewPipe Extractor 0.21.13 2022-02-01 13:20:34 +01:00
TobiGr 5306f7a988 Improve code style 2022-02-01 13:18:18 +01:00
Abdu Ameen 38e9da8d44 Adding removal of the braces 2022-02-01 13:18:10 +01:00
Abdu Ameen 36ebb633a4 Changed the regex to account for
nonword characters
2022-02-01 13:18:05 +01:00
Tobi 65129e65a8
Merge pull request #785 from AbduAmeen/regex_error
Regex error when parsing the Youtube JavaScript code
2022-02-01 13:17:19 +01:00
TobiGr 047d75a4c9 Improve code style 2022-02-01 12:56:59 +01:00
Abdu Ameen 50a4b026f8 Adding removal of the braces 2022-01-31 23:52:31 -08:00
Abdu Ameen a9174f6b9f Changed the regex to account for
nonword characters
2022-01-31 22:29:00 -08:00
dependabot[bot] a4399fd4ad Bump spotbugs-annotations from 4.5.2 to 4.5.3
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.5.2 to 4.5.3.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.5.2...4.5.3)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-01-24 18:45:21 +01:00
Stypox 59e0aeba88
Merge pull request #768 from litetex/junit-5
JUnit 5
2022-01-24 16:56:16 +01:00
Mohammed Anas 65df39b3e3
[YouTube] parse timestamps with >3 digits correctly (#775)
* [YouTube] parse timestamps with >3 digits correctly

Fixes https://github.com/TeamNewPipe/NewPipe/issues/7530; check the
issue for details.

* Remove outdated comment
2022-01-12 10:45:06 +01:00
litetex a579ef2651 Fixed Soundcloud tests 2022-01-07 14:27:24 +01:00
litetex a32edce750 Removed deprecation because AndroidSDK/API < 19 2022-01-07 14:15:55 +01:00
litetex 602068937d
Code cleanup
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2022-01-06 21:04:16 +01:00
litetex dd3f3da828 Kurzgesagt mock data is outdated 2022-01-04 17:44:12 +01:00
litetex a8ebd2cead Update mock resources for YT Mix playlist 2022-01-04 17:39:14 +01:00
litetex 066afd8629 Restored test 2022-01-04 17:33:08 +01:00
litetex 3712a669b1 Code cleanup / improvements
* Deprecated Utils#UTF-8; see StandardCharsets
* Added more helpful methods to ``ExtractorAsserts``
* Use parameterized (cool new) tests
* Restore functionality of some tests + updated mockdata
* Other code cleanups + Sonarlint improvements
2022-01-04 17:28:31 +01:00
litetex 4291a90251 Fixed SoundCloud's search(for tests) 2021-12-27 21:42:43 +01:00
litetex 4995709871 Fixed SoundCloud's search(for tests)
Getting the initial page was not returning initial page
2021-12-27 21:18:39 +01:00
litetex a2cbdf0991 Updated to JUnit 5 2021-12-27 21:08:08 +01:00
litetex 10f6cc7194
Merge pull request #755 from litetex/clean-up-yt-dis-likes
Cleanup: Remove old ways of getting YT dis/likes
2021-12-27 18:32:02 +01:00
litetex 148ba6641d Updated mock data 2021-12-27 17:08:14 +01:00
litetex 4d0a689cb6 Disabled dislike count in tests 2021-12-27 17:04:46 +01:00
litetex a6961c5efa Fixed NumberFormatException when processing likes
See also #7525
2021-12-27 16:54:47 +01:00
litetex 65687f3b9b Removed YT dislike count as it no longer works 2021-12-27 16:54:46 +01:00
litetex 1e2e0029fc [StreamExtractor] Deduplicated a ton of code by using default methods 2021-12-27 16:54:45 +01:00
litetex 15b98ffdb4 Remove old ways of getting YT dis/likes
* Added additional check for averageRating (in dislikes)
2021-12-27 16:54:44 +01:00
dependabot[bot] fe432425df Bump spotbugs-annotations from 4.5.1 to 4.5.2
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.5.1 to 4.5.2.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.5.1...4.5.2)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-12-14 10:35:32 +01:00
dependabot[bot] 36663dcb37 Bump spotbugs-annotations from 4.5.0 to 4.5.1
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.5.0 to 4.5.1.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.5.0...4.5.1)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-12-12 13:28:30 +01:00
litetex 12a6a48b15
Merge pull request #761 from litetex/fix-soundcloud-georestricted
Disabled ``SoundcloudStreamExtractorTest#SoundcloudGeoRestrictedTrack#testRelatedItems`` as it's unreliable
2021-12-11 18:06:17 +01:00
litetex 7484b6a098
Merge pull request #758 from litetex/release/v0.21.12
Release 0.21.12
2021-12-11 16:14:11 +01:00
litetex 795e6df596
Update README.md
Made usage section more abstract
2021-12-11 16:06:29 +01:00
litetex 4dd737c9a0 Updated version to v0.21.12 2021-12-11 15:34:46 +01:00
litetex 7489502cc8 Disabled ``testRelatedItems`` as it's unreliable 2021-12-11 15:23:11 +01:00
Tobi 1e0c409bec
Merge pull request #760 from litetex/fix-test
Fix test
2021-12-11 12:44:45 +01:00
litetex d9b2a2042a Updated mock data for YoutubeCommentsExtractorTest 2021-12-08 22:11:45 +01:00
litetex a3a0c1ee85 Fixed YoutubeCommentsExtractorTest
Old video was deleted / set to private
2021-12-08 22:11:34 +01:00
litetex 5c288eb8d7 Updated mock data for YoutubeChannelExtractorTest 2021-12-08 21:51:38 +01:00
litetex 75bb83a1b8 Fixed YoutubeChannelExtractorTest
Channel was restored, picked a new one
2021-12-08 21:51:15 +01:00
litetex f276bacce5 Fixed SoundcloudStreamExtractorTest 2021-12-08 21:41:27 +01:00
litetex 7f99c1c193 Fixed YoutubeStreamExtractorLivestreamTest 2021-12-08 21:16:39 +01:00
litetex 652fdf2c36 Updated mock-data for YoutubeStreamExtractorAgeRestrictedTest 2021-12-08 21:08:25 +01:00
litetex 3c669d3cb5 Fixed YoutubeStreamExtractorAgeRestrictedTest
Looks like a lot of stuff (description, tags, name) got changed on the existing video.
Use a better suited video.
2021-12-08 21:08:03 +01:00
litetex f735788a39 Fixed Soundcloud hardcoded client id 2021-12-08 21:00:53 +01:00
litetex f6f2724634
Merge pull request #753 from B0pol/dis-likes
Fix YouTube likes + dislikes
2021-11-29 20:42:17 +01:00
bopol 56c8af710e Code refactoring 2021-11-21 18:16:31 +01:00
bopol c4eca91be9 Fix YouTube likes + dislikes 2021-11-19 21:36:03 +01:00
TobiGr 5028396405 Improve names and formatting 2021-11-14 13:44:13 +01:00
dependabot[bot] 1e0c802d8d Bump spotbugs-annotations from 4.4.2 to 4.5.0
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.4.2 to 4.5.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.4.2...4.5.0)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-11-14 13:43:44 +01:00
Tobi 68d7e4e20a
Merge pull request #742 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.8.9
Bump gson from 2.8.8 to 2.8.9
2021-11-08 19:37:09 +01:00
bopol 8e4f9995bf
Merge pull request #745 from TeamNewPipe/update_SC_hardcoded_client_id
Update hardcoded SoundCloud client_id
2021-11-04 16:33:08 +01:00
opusforlife2 cb80a646d9
Update hardcoded SoundCloud client_id
Hah! I got here first, @TobiGr!
2021-11-04 15:28:26 +00:00
litetex 7e7b78f1b3
Merge pull request #743 from B0pol/peertube-shortlinks
PeerTube short links support
2021-11-04 16:10:23 +01:00
bopol 396aecef19 Make sure playlists aren't accepted by PeertubeStreamLinkHandlerFactory 2021-11-03 14:41:39 +01:00
bopol 38ad1eaac5 [PeerTube] Support /w/ short video links part 2
We also need to support it for comments
2021-11-03 13:54:37 +01:00
bopol 026751624f [PeerTube] Support /w/p/ short playlist links 2021-11-03 13:35:30 +01:00
bopol e0b8e142fc [PeerTube] Support /a/ and /c/ short links 2021-11-03 13:26:27 +01:00
bopol 599a91c88c [PeerTube] Support /w/ short video links 2021-11-03 11:51:58 +01:00
dependabot[bot] 198d37090b
Bump gson from 2.8.8 to 2.8.9
Bumps [gson](https://github.com/google/gson) from 2.8.8 to 2.8.9.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.8...gson-parent-2.8.9)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 09:12:53 +00:00
litetex 4f60225ddc
Merge pull request #741 from mhmdanas/add-y2ube-link-support
Add support for y2u.be links
2021-10-26 18:43:36 +02:00
mhmdanas 3e8e2a1532 Add support for y2u.be links 2021-10-22 22:48:18 +03:00
Tobi b425394e7a
Merge pull request #731 from FireMasterK/short-description
Extract Video Short Description in YouTube.
2021-10-15 23:10:00 +02:00
Tobi 4e9d9bffd0
Merge pull request #738 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.4.2
Bump spotbugs-annotations from 4.4.1 to 4.4.2
2021-10-12 12:09:36 +02:00
dependabot[bot] 1867a381a8
Bump spotbugs-annotations from 4.4.1 to 4.4.2
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.4.1 to 4.4.2.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.4.1...4.4.2)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-10-12 09:11:11 +00:00
FireMaskterK 62313962a0 Add default implementation for getShortDescription. 2021-10-04 01:42:29 +01:00
Kavin d1add05bc1 Add requested changes.
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
Add final.

Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-10-04 01:36:51 +01:00
Tobi 0b0c39aec2
Merge pull request #734 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.14.3
Bump jsoup from 1.14.2 to 1.14.3
2021-10-02 16:49:34 +02:00
TobiGr 8dcb74724a Release NewPipe Extractor 0.21.11 2021-10-02 16:25:35 +02:00
dependabot[bot] 3ca9f51b30
Bump jsoup from 1.14.2 to 1.14.3
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.14.2 to 1.14.3.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.14.2...jsoup-1.14.3)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-10-02 08:20:38 +00:00
TobiGr d8ec3ce370 [SoundCloud] Update client id 2021-10-01 02:51:38 +02:00
FireMasterK 94efe86c71
Extract Video Short Description in YouTube.
In Trending, and Search results.
2021-09-22 16:26:16 +01:00
Tobi a9d214478d
Merge pull request #703 from FireMasterK/comment-replies
Add support for extracting comment replies continuation
2021-09-14 23:58:14 +02:00
TobiGr ce8cabb9f8 Release NewPipe Exttractor 0.21.10 2021-09-13 13:59:35 +02:00
FireMasterK 6aabdc6d16
Fix for requested changes. 2021-09-12 01:15:19 +05:30
Kavin 0aad09fa22
Update JavaDocs for comment replies continuation.
Co-authored-by: Stypox <stypox@pm.me>
2021-09-12 01:05:20 +05:30
Tobi a031289a63
Merge pull request #727 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.4.1
Bump spotbugs-annotations from 4.4.0 to 4.4.1
2021-09-10 18:25:27 +02:00
dependabot[bot] 82ae322cb8
Bump spotbugs-annotations from 4.4.0 to 4.4.1
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.4.0 to 4.4.1.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.4.0...4.4.1)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-10 09:11:27 +00:00
Tobi 62b87552f5
Merge pull request #725 from FireMasterK/related-uploader-avatar
Extract Channel Avatar in related videos.
2021-09-03 22:14:28 +02:00
FireMasterK feb09e9997
Extract Channel Avatar in related videos. 2021-09-04 01:09:40 +05:30
Tobi f0aa46b008
Merge pull request #724 from FireMasterK/streams-uploader-verified
Extract uploaderVerified to StreamInfo.
2021-09-03 18:18:04 +02:00
Tobi c8037f5e00
Merge pull request #723 from FireMasterK/uploader-avatar
Extract Uploader's Avatar on YouTube and PeerTube
2021-09-03 18:13:59 +02:00
FireMasterK 857b2f39e8
Extract uploaderVerified to StreamInfo. 2021-09-03 21:27:58 +05:30
FireMasterK 1db463b55f
Add a default test for the Uploader's Avatar. 2021-09-03 21:05:36 +05:30
Kavin db6b3b2c29
Extract uploader's avatar in peertube.
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2021-09-02 16:14:45 +05:30
FireMasterK a0c1dcc8d8
Remove throws parsing exception. 2021-09-02 16:14:45 +05:30
FireMasterK 0ba03c552c
Improve based on review/feedback. 2021-09-02 00:35:56 +05:30
FireMasterK b9fad4fcc8
Extract Uploader's Avatar on YouTube. 2021-09-02 00:19:00 +05:30
Stypox 68f1fa994a
Merge pull request #721 from Stypox/yt-csv-import
[YouTube] Small improvements to subscription import
2021-08-31 12:02:59 +02:00
TobiGr b644f1d67b [SoundCloud] Update client id 2021-08-31 07:42:21 +02:00
Stypox a5e9eeb790
[YouTube] Small improvements to subscription import 2021-08-30 15:45:45 +02:00
Tobi 9570882c73
Merge pull request #712 from magicbrothers/add-eduvid-tubus
Add invidious instance EduVid Tubus
2021-08-24 16:07:07 +02:00
Stypox db6c729678
Merge pull request #709 from talanc/dev
[YouTube] Support csv and zip subscription import (Google Takeout)
2021-08-24 15:46:03 +02:00
talanc 94a29fd63f
[YouTube] csv and zip subscription import (Google Takeout)
csv:
Improved error messages
Exits early if it hasnt found any items in the first few lines

zip:
Now checks all CSV files instead of hard-coded paths

final qualifiers for immutable locals and parameters

Co-authored-by: litetex <40789489+litetex@users.noreply.github.com>
2021-08-24 15:42:49 +02:00
dependabot[bot] 816bc7137f Bump gson from 2.8.7 to 2.8.8
Bumps [gson](https://github.com/google/gson) from 2.8.7 to 2.8.8.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.7...gson-parent-2.8.8)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-08-23 13:04:10 +02:00
TobiGr 6d487e0dad Bump version to 0.21.9 2021-08-22 19:29:22 +02:00
Tobi 7823dcd99d
Merge pull request #715 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.4.0
Bump spotbugs-annotations from 4.2.3 to 4.4.0
2021-08-22 16:41:20 +02:00
dependabot[bot] d75d54e1c6
Bump spotbugs-annotations from 4.2.3 to 4.4.0
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.2.3 to 4.4.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.2.3...4.4.0)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-08-22 07:12:54 +00:00
dependabot[bot] 72393995bb
Bump jsoup from 1.13.1 to 1.14.2 (#716)
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.13.1 to 1.14.2.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.13.1...jsoup-1.14.2)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-22 09:12:14 +02:00
Stypox d8179dc35a
Merge pull request #713 from TiA4f8R/regexes-yt-signatureciphers
[YouTube] Consolidate the regular expressions used to find the cipher decryption function
2021-08-17 18:58:27 +02:00
Tobi 89bab241b9
Merge pull request #717 from XiangRongLin/fix_tests
Fix failing PeertubeStreamExtractorTest
2021-08-17 17:48:28 +02:00
XiangRongLin 61acdae4b8 Fix failing PeertubeStreamExtractorTest 2021-08-17 17:06:36 +02:00
FireMasterK 4f86b8ce43
Add verified info to ChannelInfo. (#714) 2021-08-17 13:08:49 +02:00
Stypox 52910ce970
Keep original exception when there is an error message (#710) 2021-08-13 16:13:58 +02:00
TiA4f8R f8309d546a
[YouTube] Consolidate current regexes of the cipher decryption function
Also add more regexes
2021-08-12 18:26:43 +02:00
magicfelix 0e16091ce0
Add invidious instance EduVid Tubus 2021-08-12 10:06:41 +02:00
Tobi b6f951edb4
Merge pull request #711 from gardenappl/dev
Fix minor typos in Javadoc
2021-08-11 18:16:34 +02:00
gardenapple 50bba32239
Fix typos in Javadoc 2021-08-11 12:01:25 +03:00
Tobi 78f2192cbb
Merge pull request #687 from TacoTheDank/gradle7
Upgrade to Gradle 7.1.1
2021-08-08 12:38:29 +02:00
TacoTheDank 4a7c1723a6 Use new 'maven-publish' plugin 2021-08-07 17:51:32 -04:00
TacoTheDank 8919a53d95 Update gradle wrapper to 7.1.1 2021-08-07 17:51:18 -04:00
TacoTheDank cf9e678223 Declare common dependency versions in root gradle 2021-08-07 17:51:09 -04:00
FireMasterK 71b9fd0076
Faster iframe api based player extraction. (#694)
* Faster iframe api based player extraction.

Uses the IFrame API to reduce the required download to less than 1/50 of the size.

* Remove debug code.

* Extract to two methods.

* Add tests for player URL extraction.

* Add assertThat for tests.
2021-08-07 12:35:48 +02:00
Tobi 4b147863ec
Merge pull request #705 from FireMasterK/suppored-countries
Update supported countries list for YouTube.
2021-08-06 10:14:27 +02:00
FireMasterK ee32317315
Update supported countries list for YouTube. 2021-08-05 23:48:38 +05:30
FireMasterK e8e9e6cd00
Fix comment reply continuations. 2021-08-04 22:14:04 +05:30
FireMasterK 666b45958e
Fix test and update mocks. 2021-08-04 21:10:02 +05:30
FireMasterK f6d054e5da
Add test for comments. 2021-08-04 21:03:00 +05:30
FireMasterK ed84658055
Get it working. 2021-08-04 18:35:14 +05:30
FireMasterK ea7e6526fd
Collect replies/page to CommentsInfoItem. 2021-08-04 18:09:42 +05:30
FireMasterK 4385cc1b7c
Add comment replies. 2021-08-04 17:56:04 +05:30
Stypox 8ca5e5e364
Merge pull request #702 from TeamNewPipe/master
This merges master into dev
2021-08-03 20:52:04 +02:00
Stypox bb3815d19b
Merge pull request #701 from Stypox/v0.21.8
V0.21.8
2021-08-03 20:43:15 +02:00
Stypox 5b38b3ae97
Merge branch 'master' into v0.21.8 2021-08-03 20:38:07 +02:00
Stypox 4b5907e18c
Release version 0.21.8 2021-08-03 20:33:24 +02:00
Stypox 5a88263785
Merge pull request #604 from TiA4f8R/youtubei-api
[YouTube] Use the new internal API in NewPipe Extractor
2021-08-03 19:01:14 +02:00
TiA4f8R 79cc5c8d12
Fix tests of YoutubePlaylistExtractorTest and YoutubeMixPlaylistExtractorTest
Without removing RunWith and SuiteClasses annotations (and the corresponding imports) in YoutubePlaylistExtractorTest and YoutubeMixPlaylistExtractorTest, some mocks cannot be generated, so the CI fails because of the missing mocks. Mocks of workings tests have been also updated.
2021-08-01 17:28:01 +02:00
TiA4f8R b74a39c176
Reformat some code and don't use the clickTrackingParams in continuations of YouTube Music search results
The clickTrackingParams of YouTube Music search results are not needed to get continuations. This commit removes their use, which may improve privacy.
2021-08-01 17:25:59 +02:00
TiA4f8R f8197da09e
Update YouTube comments and YouTube playlist continuations mocks 2021-08-01 12:39:05 +02:00
FireMasterK f3e4c9d689
Use the youtubei API for YouTube comments
Migrate YouTube comments to the desktop version by using the `next` endpoint of the InnerTube internal API.
With the desktop version, we are able to get the exact like count of YouTube comments (by parsing the accessibility data) (the current extraction is used as a fallback). We are also now able to get if the uploader of the comment is verified or not.

Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-08-01 12:39:05 +02:00
TiA4f8R 286d839a3b
Update mocks (except mocks of YoutubeCommentsExtractorTest) 2021-08-01 12:39:04 +02:00
TiA4f8R 693f654f02
Ignore a broken test of YoutubeMixPlaylistExtractorTest 2021-08-01 12:39:03 +02:00
TiA4f8R d55109258d
Fix YoutubeStreamExtractorAgeRestrictedTest 2021-08-01 12:39:03 +02:00
FireMasterK 2eeb0a3403
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:

- the desktop API is fetched.

If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status

If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.

If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.

We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.

Other code changes:

- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed

Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-08-01 12:39:03 +02:00
TiA4f8R 7753556e66
Adress the last requested changes + update YoutubeCommentsExtractor mocks 2021-08-01 12:39:03 +02:00
TiA4f8R 2320aecb7c
Update mocks 2021-08-01 12:39:02 +02:00
TiA4f8R 629dcd63b2
Ignore some broken tests 2021-08-01 12:39:01 +02:00
TiA4f8R 3adac6a150
Workaround for rate limits: always use the Android mobile API 2021-08-01 12:39:01 +02:00
TiA4f8R 8aa60d7e8f
Update clients version 2021-08-01 12:39:01 +02:00
TiA4f8R 27bd797f1e
Update mocks 2021-08-01 12:39:01 +02:00
TiA4f8R 609919db59
Adress again reviews, fix some rebase issues 2021-08-01 12:39:00 +02:00
TiA4f8R 4299d806a2
Adress changes 2021-08-01 12:38:59 +02:00
TiA4f8R 1a6b8da438
Annotate YoutubeParsingHelper methods with Nonnull when needed 2021-08-01 12:38:59 +02:00
TiA4f8R d8177b57f6
Loop in all formats to check if the stream has URLs protected by signatureCiphers 2021-08-01 12:38:59 +02:00
TiA4f8R a6a2c6eb80
Revert the use of Collections.singletonList instead of Arrays.asList in addCookieHeader of YoutubeParsingHelper 2021-08-01 12:38:59 +02:00
TiA4f8R 81013e5a8e
Add a static method to reset the YouTube deobfuscation code and use it in tests
This method is needed for YouTube stream tests, because when all YouTube tests are ran, the signatureTimestamp is known (the sts string) so a different body than the body present in the mocks is send by the extractor instance.
As a result, running all YouTube stream tests with the MockDownloader (like the CI does) will fail if this method is not called before fetching the page of a test.
2021-08-01 12:38:58 +02:00
TiA4f8R cc798523cd
Adress requested changes in other classes 2021-08-01 12:38:58 +02:00
TiA4f8R accd5ddef3
Adress requested changes in YoutubeStreamExtractor 2021-08-01 12:38:58 +02:00
TiA4f8R 632772d17f
Adress requested changes in YoutubeParsingHelper 2021-08-01 12:38:58 +02:00
TiA4f8R 657f165771
Update client version and mocks 2021-08-01 12:38:44 +02:00
TiA4f8R 8c1c7281b0
Real fix of the fetch of the JS player at each the signatureTimestamp is required
The strings playerJsUrl, sts and playerCode are now static in order to don't fetch again the JavaScript player at each time the signatureTimestamp is needed.
2021-08-01 12:38:43 +02:00
TiA4f8R e97a685989
Format the YoutubeMusicSearchExtractor class to be in the 100 characters per line limit 2021-08-01 12:38:43 +02:00
TiA4f8R 34a9ccb0fd
Adress requested changes 2021-08-01 12:38:42 +02:00
TiA4f8R 54d4551ca6
Adress requested changes in YoutubeParsingHelper and update mobile client version 2021-08-01 12:38:42 +02:00
TiA4f8R a59c2a3577
Catch every exception instead of only IOException and ExtractionException and add a Javadoc
Catch every exception instead of only IOException and ExtractionException.
Add JavaDoc for fetchAndroidMobileJsonPlayer method of YoutubeStreamExtractor
2021-08-01 12:38:42 +02:00
TiA4f8R 6921e80ded
Try again to don't fetch at each time the JavaScript player 2021-08-01 12:38:42 +02:00
TiA4f8R 70927ddade
Update client version and mocks 2021-08-01 12:38:40 +02:00
TiA4f8R 318bc46a8c
Readd the deleted code of views because watching count of livestreams was broken
The number shown was the total number of views that a livestream has. In order to fix this bug, the previous code is readded.
2021-08-01 12:38:39 +02:00
TiA4f8R 947baec805
Fetch again the desktop player JSON only if the content is protected by signatureCiphers 2021-08-01 12:38:39 +02:00
TiA4f8R 7474049fd1
Update mocks 2021-08-01 12:38:39 +02:00
TiA4f8R c32bc6e534
Try to don't fetch again the first page when requesting a channel continuation
Try to don't fetch again the first page of a YouTube channel when requesting a continuation of it by trying to store the channel name and the channel id into the next page using the ids field of the Page class.
2021-08-01 12:38:38 +02:00
TiA4f8R b52732a1c3
Fix the fetch of the playerCode at each time getStsFromPlayerJs is called
storePlayerJs was called even if sts was not empty in the getStsFromPlayerJs method. This commit fixes it.
2021-08-01 12:38:37 +02:00
TiA4f8R ae5abc0c5d
Fix a typo in YoutubeChannelExtractor 2021-08-01 12:38:37 +02:00
TiA4f8R 0f9e9b8b4b
Use the youtubei API for YouTube mixes + update the corresponding test + do some improvements
Use the youtubei API for YouTube mixes. The corresponding has been updated because the new API breaks the tests of YoutubeMixPlaylistExtractorTest.
Remove some deprecated code (the old search code with the pbj JSON) and do some other improvements.
2021-08-01 12:38:37 +02:00
TiA4f8R 14569c4aa9
Readd the return of itags 134 and 136 2021-08-01 12:38:37 +02:00
TiA4f8R 3017dde67e
Fix some typos 2021-08-01 12:38:36 +02:00
TiA4f8R 013b902535
Use the Android mobile API when there are OTF streams or the content is protected by signatureCiphers
Use the Android mobile API to get the itag 22 (720p with audio), removed when the content is protected by signatureCiphers.
Also use this API when they are OTF streams, to get the itag 17 and 36, low 3GPP quality streams but also the itag 139.
Update the web client version.
2021-08-01 12:38:36 +02:00
TiA4f8R e7d589edbf
Use the youtubei API for YouTube videos + update client version
Update the hardcoded client version to 2.20210520.09.00
Use the player and next endpoints of the Innertube API for YouTube videos
2021-08-01 12:38:36 +02:00
TiA4f8R f73c923f60
Don't use the youtubei.googleapis.com but the websites domains + update client version of the desktop internal API
Use again www.youtube.com and music.youtube.com domains instead of youtubei.googleapis.com domain because it spoofs more a web client of YouTube or YouTube Music and may reduce Google's detection of NewPipe Extractor users.
2021-08-01 12:38:34 +02:00
TiA4f8R effcdaa4f2
Update mocks, reenable a test and fix a test
This commit updates mocks and reenables the test invalidId of the NotAvailable class of the YoutubePlaylistExtractorTest class beacuse with the youtubei/innertube API, it returns "Not found" and doesn't redirect to the YouTube homepage.
The expectedMetaInfo test of the MetaInfoTest class of the YoutubeSearchExtractorTest class was broken because YouTube removes the vaccine progress link of the WHO from the meta info so this commit removes it and the test is now passing.
2021-08-01 12:38:04 +02:00
TiA4f8R 4d682834c3
Fix localization and update client version 2021-08-01 12:38:03 +02:00
TiA4f8R f46cfb0f26
Adress reviews and do some improvements
Adress changes requested in reviews.
Do some improvements, remove unused imports and format some code to be in the 100 characters line limit.
2021-08-01 12:38:03 +02:00
TiA4f8R e075dd5a63
Update client version, fix some tests, update mocks and do some improvements
Add the origin and the referer headers with the https://www.youtube.com value for YouTube JSON POST requests.
Don't add the consent cookie header for the requests which use the youtubei/innertube API because it's uneeded.
Fix some tests and update YouTube mocks
2021-08-01 12:38:02 +02:00
TiA4f8R b49ae547a3
Do some improvements to YoutubeStreamExtractor
Get the real name of the uploader (for autogenerated channels and music artist channels), like before the migration to the JSON pbj.
Do some other improvements, especially reformatting some code to be in the 100 characters line limit and use final where possible.
2021-08-01 12:38:01 +02:00
TiA4f8R 58ce9b04a1
Fix channel extraction when channel URL is youtube.com/c/username 2021-08-01 12:38:01 +02:00
TiA4f8R 991b2c7d73
Use lightweight requests when getting and checking YouTube API key and client version 2021-08-01 12:38:01 +02:00
TiA4f8R 9ab9c66ddf
Use the youtubei.googleapis.com domain for YouTube Music searches + change a check + update client version and mocks
Change the domain from music.youtube.com to youtubei.googleapis.com.
Use a lightweight request to check if the hardcoded YouTubeMusic keys are valid. Increase the length of the response to 500 because if the key is invalid, the length of the response returned is higher than 250 and the response when the key is valid is higher than 1500.
Format the YoutubeMusicSearchExtractor file.
Update YouTube web client version and mocks
2021-08-01 12:37:56 +02:00
TiA4f8R 77c031a88a
Use the youtubei API for YouTube trends 2021-08-01 12:36:28 +02:00
TiA4f8R f461224b2b
Use the youtubei API for YouTube searches + update mocks
Add getSearchParameter, a new method in YoutubeSearchQueryHandlerFactory class which returns the params field for a search, or an empty string if there is no one.
Update mocks of YoutubeSearchExtractorTest.
2021-08-01 12:36:28 +02:00
TiA4f8R a12c69da7d
Use the youtubei API for YouTube channels 2021-08-01 12:36:27 +02:00
TiA4f8R 5794eb2350
Use the youtubei API for YouTube playlists 2021-08-01 12:36:12 +02:00
Stypox c97a19d719
Merge pull request #696 from XiangRongLin/decrypt_pattern
Expand regex to match n param decrypt function
2021-08-01 10:24:22 +02:00
Stypox 06a5219c9c
Merge pull request #699 from FireMasterK/yt-music-search
Fix YouTube music search.
2021-08-01 10:17:15 +02:00
XiangRongLin 37df225556 Remove length check from StringUtils.matchToClosingParenthesis 2021-07-31 16:05:24 +02:00
FireMasterK f4aad8b014
Fix tests again. 2021-07-31 14:40:51 +05:30
FireMasterK 88c11db4c5
Fix isCorrectedSearch test. 2021-07-31 02:18:18 +05:30
FireMasterK fd19c53f65
Fix search suggestions test. 2021-07-31 02:08:50 +05:30
FireMasterK a685941bba
Fix YouTube music search. 2021-07-31 01:33:38 +05:30
XiangRongLin 852a65ff18 Add tests for StringUtils 2021-07-30 19:56:57 +02:00
XiangRongLin 48d897e6ad Add final and adjust utils class name 2021-07-29 21:33:45 +02:00
XiangRongLin 60794aea31 [YouTube] Add parenthesis matching as way to parse decrypt function
From @Stypox
2021-07-29 21:23:00 +02:00
XiangRongLin 2967d1ae6a [YouTube] Compile YoutubeThrottlingDecrypter pattern statically 2021-07-29 20:27:29 +02:00
XiangRongLin 1c78976900 [YouTube] Expand regex to match n param decrypt function
Temporary solution
2021-07-29 20:26:36 +02:00
TobiGr b62fe7141e Fix JDoc 2 2021-07-29 12:34:04 +02:00
TobiGr dbc4e01c17 Fix JDoc 2021-07-29 12:31:33 +02:00
TobiGr 6335823843 Merge branch 'master' into dev 2021-07-28 20:41:23 +02:00
TobiGr 7c6ff0a38f Release version 0.21.7 2021-07-28 20:38:45 +02:00
TobiGr 8bfcb0ad59 [SoundCloud] Fix hardcoded client id 2021-07-28 20:33:21 +02:00
Tobi 027dc65434 pull request #683 from XiangRongLin/yt_throttling
[YouTube] Fix buffering by decoding n parameter of stream urls
2021-07-28 20:32:38 +02:00
Tobi 14c179f343
Merge pull request #692 from TeamNewPipe/fix/sc/id
[SoundCloud] Fix hardcoded client id
2021-07-28 18:28:07 +02:00
Tobi 394c02ad06
Merge pull request #683 from XiangRongLin/yt_throttling
[YouTube] Fix buffering by decoding n parameter of stream urls
2021-07-28 18:01:57 +02:00
TobiGr 785ff4aa32 [SoundCloud] Fix hardcoded client id 2021-07-28 17:02:56 +02:00
TobiGr 79f2d74b04 Fix test
The decryption code changes over time. Only check whether the n parameter was changed in the URL
2021-07-28 16:45:31 +02:00
TobiGr d70adfdb8f Add methods for cache control to YoutubeThrottlingDecrypter. 2021-07-27 20:32:22 +02:00
TiA4f8R d13f531b6f Use YoutubeThrottlingDecrypter also in getAudioStreams and getVideoOnlyStreams methods of YoutubeStreamExtractor
Without this commit, the n param is only decrypted for streams extracted in getVideoStreams (so only for streams in the formats object of the player response).
2021-07-27 17:28:32 +02:00
TiA4f8R 32055147e0 Do some code improvements
Use final where possible, annotate some methods and parameters as Nonnull and format new code to be in the 100 characters limit per line.
2021-07-27 17:24:27 +02:00
TobiGr 1c30a2725e Cache nParams to prevent executing the JavaScript function for the same nParam multiple times.
Closes #689
2021-07-22 19:40:12 +02:00
XiangRongLin 3a3d1d7f2b Make YoutubeJavaScriptExtractor and JavaScript methods static
Also address review and rewrite some comments
2021-07-20 20:48:11 +02:00
XiangRongLin a683c8d278 Delete duplicated code to load youtube javascript file 2021-07-17 19:14:57 +02:00
XiangRongLin a02ee2e952 Rewrite youtube throttling solution and add tests 2021-07-17 19:10:09 +02:00
Tobi ada67d136a
Merge pull request #647 from litetex/playerSeekbarPreview
Code changes to enable player thumbnail seekbar preview in NewPipe
2021-07-17 17:49:54 +02:00
litetex 0c12b396e5 Review changes 2021-07-17 16:41:12 +02:00
litetex ecf4232ce3 Using ``Collections.emptyList()`` 2021-07-17 16:41:12 +02:00
litetex 0f9ed020a4 Frameset has to implement Serializable or NewPipe fails
java.lang.RuntimeException: Parcelable encountered IOException writing serializable object (name = org.schabi.newpipe.extractor.stream.StreamInfo)
...
Caused by: java.io.NotSerializableException: org.schabi.newpipe.extractor.stream.Frameset
...
2021-07-17 16:41:11 +02:00
litetex 17ccaf4b87 Update YoutubeStreamExtractor.java
To fix ``java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.String[] java.lang.String.split(java.lang.String)' on a null object reference``
2021-07-17 16:41:10 +02:00
litetex 4e41e172ea Update Frameset.java
Reformatted code
2021-07-17 16:41:10 +02:00
litetex 09b670d745 Renamed frames to previewFrames (so it's better understandable) 2021-07-17 16:41:09 +02:00
litetex b086655d54 Reverted useless change 2021-07-17 16:41:08 +02:00
litetex 93b0c5b15f Added seekbar data 2021-07-17 16:41:08 +02:00
XiangRongLin 6956b72af7 Fix YoutubeThrottlingDecoder.replaceNParam
Previously it replaced the parameter itself not the value of the parameter.
2021-07-17 10:05:14 +02:00
XiangRongLin a189f685dc cache player js code 2021-07-16 19:20:40 +02:00
XiangRongLin fcdb9bdbeb add url to sout 2021-07-15 20:05:26 +02:00
XiangRongLin 80cf8b3acd Extract separate YoutubeThrottlingDecoder 2021-07-15 20:05:25 +02:00
XiangRongLin a86a30103f [Youtube] bare bones version to solve throttling
Done by transforming the parameter "n" from videoplayback urls
https://github.com/ytdl-org/youtube-dl/issues/29326#issuecomment-865985377
2021-07-15 20:05:25 +02:00
bopol c38a06e8dc
Merge pull request #661 from B0pol/peertube-livestreams
[PeerTube] Support livestreams
2021-07-13 15:48:13 +02:00
Tobi b45bb411e8
Merge pull request #652 from litetex/fixYTCommentsAndAddDisabledComments
Fix yt comments and add disabled comments functionallity
2021-07-12 16:31:50 +02:00
TobiGr 6fd93cdb31 Release 0.21.6 2021-07-10 17:07:23 +02:00
litetex fdebf3c6cd Added javadoc warnings 2021-07-07 20:41:59 +02:00
litetex 6860543b07 Added more doc 2021-07-06 21:16:31 +02:00
litetex a59fead0d7 Fixed typos 2021-07-06 21:16:30 +02:00
litetex 688a1c316b Fixed Exception when YT comments are disabled and added ``commentsDisabled`` field
* Fixed code: Added missing finals (according to NewPipes Checkstyle guide)
* Fixed ``findValue`` method in ``YoutubeCommentsExtractor``
2021-07-06 21:16:29 +02:00
litetex ca33f4f60b Improved code format (final) and added ``isCommentsDisabled`` method 2021-07-06 21:16:28 +02:00
Tobi f324772254
Merge pull request #676 from TeamNewPipe/fix/consent-cookie
Fix invalid CONSENT cookie value
2021-07-04 17:41:47 +02:00
TobiGr dbdc962a07 Update mocks 2021-07-03 13:30:26 +02:00
TobiGr 9ca6dc26ed Fix invalid CONSENT cookie value 2021-07-03 13:27:40 +02:00
Tobi a023f08774
Merge pull request #673 from XiangRongLin/recording_downloader
Remove IPv4 addresses from response during mock recording
2021-07-03 13:25:41 +02:00
XiangRongLin b9549ace25 [Youtube] Fix lofi girl test and ignore mix with id test 2021-06-29 21:12:20 +02:00
XiangRongLin 5982431ffb Remove IPv4 addresses from response during mock recording 2021-06-29 21:03:33 +02:00
bopol 7d7cc087e2 [PeerTube] Support livestreams 2021-06-24 11:02:30 +02:00
bopol 3966178979
Merge pull request #665 from B0pol/agerestricted
[youtube] Fix extraction of age restricted videos
2021-06-23 21:07:14 +02:00
bopol b475f09ba7 don't do useless concatenation 2021-06-23 20:33:32 +02:00
TiA4f8R d1c56be744
Merge pull request #668 from B0pol/invidious-instances
Update Invidious instances and support Piped links
2021-06-23 18:36:48 +02:00
bopol b96b2a6eba Update invidious instances 2021-06-23 14:39:10 +02:00
bopol c5e8bd368d Update Mocks 2021-06-23 12:21:57 +02:00
bopol e9a992b0a9 [youtube] Fix extraction of age restricted videos 2021-06-23 12:06:02 +02:00
bopol 35c1bdd012
Merge pull request #655 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.2.3
Bump spotbugs-annotations from 4.0.2 to 4.2.3
2021-06-23 12:01:56 +02:00
dependabot[bot] ad403ce73e
Bump spotbugs-annotations from 4.0.2 to 4.2.3
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.0.2 to 4.2.3.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.0.2...4.2.3)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-06-23 09:57:42 +00:00
bopol 65fd8740aa
Merge pull request #658 from TeamNewPipe/revert-651-fix/streamType
[YouTube] Fix getting stream type of live streams
2021-06-23 11:33:24 +02:00
bopol 361f142621 Use badge style instead of label 2021-06-23 11:25:59 +02:00
bopol 58f109ddec adress stypox review 2021-06-23 11:15:40 +02:00
bopol fe29c78fe1
Merge pull request #656 from TeamNewPipe/dependabot/gradle/org.mozilla-rhino-1.7.13
Bump rhino from 1.7.12 to 1.7.13
2021-06-20 00:46:42 +02:00
bopol e9412fdadc
Merge pull request #654 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.8.7
Bump gson from 2.8.6 to 2.8.7
2021-06-20 00:35:24 +02:00
XiangRongLin 1510dd5187
Merge pull request #660 from TeamNewPipe/fix/sc/client-id
[SoundCloud] Update HARDCODED_CLIENT_ID
2021-06-18 15:48:27 +02:00
TobiGr 1f9a99c931 [SoundCloud] Update HARDCODED_CLIENT_ID 2021-06-18 15:30:54 +02:00
Tobi c998012c28
Revert "[YouTube] Fix getting stream type of live streams" 2021-06-18 09:20:32 +02:00
dependabot[bot] 38d0339753
Bump rhino from 1.7.12 to 1.7.13
Bumps [rhino](https://github.com/mozilla/rhino) from 1.7.12 to 1.7.13.
- [Release notes](https://github.com/mozilla/rhino/releases)
- [Changelog](https://github.com/mozilla/rhino/blob/master/RELEASE-NOTES.md)
- [Commits](https://github.com/mozilla/rhino/commits)

---
updated-dependencies:
- dependency-name: org.mozilla:rhino
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-06-17 21:49:07 +00:00
dependabot[bot] 5a46878b09
Bump gson from 2.8.6 to 2.8.7
Bumps [gson](https://github.com/google/gson) from 2.8.6 to 2.8.7.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.6...gson-parent-2.8.7)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-06-17 21:49:01 +00:00
bopol 598de2dd5f
Merge pull request #552 from FireMasterK/dependabot
Add dependabot
2021-06-17 23:48:41 +02:00
Tobi 0cb48e36bd
Merge pull request #651 from TeamNewPipe/fix/streamType
[YouTube] Fix getting stream type of live streams
2021-06-12 17:57:18 +02:00
TobiGr 6762fe3357 [YouTube] Fix getting stream type of live streams 2021-06-10 16:51:00 +02:00
Tobi a27575021d
Merge pull request #625 from TeamNewPipe/commentsStreamPosition
Add streamPosition for comments
2021-06-08 11:03:15 +02:00
TobiGr f7f727d19c Add constant NO_LIKE_COUNT to CommentsInfoItem 2021-06-08 10:28:44 +02:00
TobiGr b70c0f93c7 Add streamPosition for comments
SoundCloud is the only service which supports adding comments at a specific timestamp in the stream.
2021-06-08 10:27:52 +02:00
Tobi d4186d100b
Merge pull request #580 from TeamNewPipe/accountTerminated
Add AccountTerminatedException for terminated channels
2021-06-08 09:55:58 +02:00
TobiGr 89e0514d8b Fix Javadoc generation fix 2021-06-08 09:30:05 +02:00
TobiGr 80d3052033 Fix Javadoc generation 2021-06-08 09:26:01 +02:00
TobiGr 98825a2f01 Set version to 0.21.4 2021-06-07 23:45:12 +02:00
TobiGr fa444c8298 Remove assertations of exception messages 2021-06-06 11:38:43 +02:00
Tobi 8c42a48673
Merge pull request #646 from Stypox/get-service
Add utility method getService() to Info object
2021-06-05 14:29:09 +02:00
Stypox 2158ca4060
Remove two unneded warning suppressions 2021-06-05 14:15:48 +02:00
Stypox 13f192704d
Add utility method getService() to Info object 2021-06-05 14:14:19 +02:00
XiangRongLin 0ad51e76fb
Merge pull request #638 from TeamNewPipe/spelling
Improve exception message thrown when not accepting an URL in LinkHandlerFactory#fromUrl(String url, String baseUrl)
2021-06-04 18:23:31 +02:00
Tobi 143fd3c81c
Merge pull request #645 from XiangRongLin/fix_test
Fix PeertubeAccountExtractorTest
2021-06-03 20:49:59 +02:00
XiangRongLin faa503c7dd Fix PeertubeAccountExtractorTest 2021-06-03 19:54:42 +02:00
XiangRongLin db81384ae0
Merge pull request #644 from TiA4f8R/utf-8-encoding-for-mocks-windows
Specify UTF-8 file encoding in RecordingDownloader and MockDownloader
2021-06-03 18:55:00 +02:00
TiA4f8R ac31f3a883
Specify UTF-8 file encoding in RecordingDownloader and MockDownloader
On Windows, mocks are recorded and read with the Cp1252 encoding so it breaks the mocks on non ASCII characters for Linux OS (and so the CI).
The project is in Java 8, so we can't use FileReader(File, Charset) and FileReader(File, Charset) because these methods require Java 11. Instead of changing the Java version of the extractor, use OutputStreamWriter and FileOutputStream instead of FileWriter and InputStreamReader and FileInputStream instead of FileReader.
2021-06-03 18:38:25 +02:00
Tobi 7e4332e0cf
Update CI badge to representt the scheduled tests 2021-05-30 22:12:09 +02:00
bopol ff11c2df2a
Merge pull request #628 from litetex/fix-broken-yt-liked-comments
Fix broken yt likes in comments
2021-05-29 11:04:11 +02:00
TobiGr 1ff56a85df Update junit from 4.13.1 to 4.13.2
https://github.com/junit-team/junit4/blob/main/doc/ReleaseNotes4.13.2.md
2021-05-28 15:09:26 +02:00
TobiGr 858e007804 Improve exception message thrown when not accepting an URL in LinkHandlerFactory#fromUrl(String url, String baseUrl) 2021-05-28 14:59:44 +02:00
Tobi 519bba70b3
Merge pull request #630 from XiangRongLin/fix_tests
Fix MetaInfoTest.expectedMetaInfo
2021-05-28 14:16:53 +02:00
litetex 289b84e68b Removed unused code 2021-05-27 21:27:44 +02:00
litetex e81b0e2885 Tweaked code 2021-05-27 19:48:31 +02:00
TobiGr f4cb32cef5 Release version 0.21.3 2021-05-26 22:40:56 +02:00
XiangRongLin bf2af4f316
Merge pull request #635 from litetex/update-setup-java
Update setup java
2021-05-26 17:44:57 +02:00
litetex 8c96545e57 Merge branch 'TeamNewPipe:dev' into fix-broken-yt-liked-comments 2021-05-24 18:06:45 +02:00
litetex 12fb18c310 Updated mock data (only the affected) 2021-05-24 18:05:26 +02:00
litetex 8e08a2aac7 Improved docs 2021-05-24 18:05:26 +02:00
litetex 020acfed71 Fixed tests and added new ones 2021-05-24 18:05:26 +02:00
litetex b310922fc0 Better exception message 2021-05-24 18:05:25 +02:00
litetex 86bb9efb5d Added likeCount back to tests 2021-05-24 18:05:25 +02:00
litetex 46654f5fca Reformatted code 2021-05-24 18:05:25 +02:00
litetex 2174685c5c Reimplemented likeCount 2021-05-24 18:05:25 +02:00
litetex 01cfb55505 Update YoutubeCommentsInfoItemExtractor.java 2021-05-24 18:05:25 +02:00
litetex bedcd87abb Fixed occurrences where voteCount is not set (<=0 likes) 2021-05-24 18:05:25 +02:00
litetex 545c0a6f42 Fixed comment 2021-05-24 18:05:24 +02:00
Tobi ff005122bf
Merge pull request #627 from TiA4f8R/use-snd-api-v2-everywhere
[SoundCloud] Use a lightweight request to check if the hardcoded client_id is valid, fix the extraction of mobile URLs and more
2021-05-23 22:52:40 +02:00
TiA4f8R a00fdcbd3d
Format the indentation of the definition of the apiUrl string in SoundcloudParsingHelper.resolveFor method
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2021-05-23 20:27:45 +02:00
TiA4f8R 6b607eb38d
Update the hardcoded client_id 2021-05-23 18:08:44 +02:00
TiA4f8R 4552ea9c9f
Use the SOUNDCLOUD_API_V2_URL constant in all the SoundCloud package 2021-05-23 18:08:44 +02:00
TiA4f8R 86308d0603
Use a lightweight request to check if the client_id is valid
Request the api-v2 host with the client_id instead of checking if the streams of a SoundCloud track are not empty: if it is valid, the API returns 404, otherwise it should return 401.
2021-05-23 18:08:43 +02:00
TiA4f8R c5c190500c
Format the SoundCloud package of the extractor and fix some warnings
Use final where possible in the package and format code to be in the 100 caracters per line limit.
Fix some warnings generated by Android Studio and do some code improvements
2021-05-23 18:08:42 +02:00
litetex 89ecc4e2e8
Update docs.yml 2021-05-23 14:34:29 +02:00
litetex 7791907480
Update ci.yml 2021-05-23 14:34:16 +02:00
Tobi 636e27333b
Merge pull request #603 from TeamNewPipe/peertube_hls
Add PeerTube HLS streams
2021-05-23 11:49:40 +02:00
Tobi fadf677c42
Merge pull request #629 from TechComet/patch-1
[YouTube] Fix the fetch of the video info page
2021-05-23 09:41:24 +02:00
XiangRongLin 172394cd52 Update mocks 2021-05-22 20:33:20 +02:00
XiangRongLin 18c8cac40c Fix MetaInfoTest.expectedMetaInfo
Expected meta changed
2021-05-21 21:00:08 +02:00
Hassan Igbaria e8b5364745
Update YoutubeStreamExtractor.java 2021-05-21 16:22:24 +03:00
litetex c3b837fe3b
Deduped code; Using default values... 2021-05-20 20:22:40 +02:00
litetex b934c7ccbb
Removed unused import 2021-05-20 20:11:35 +02:00
litetex 10cf081145
Removed likeCount and added textualVoteCount 2021-05-20 20:10:14 +02:00
TobiGr 1acc53b8db [PeerTube] Add support for HLS streams 2021-05-15 12:39:23 +02:00
Tobi 3a3ade20f4
Merge pull request #620 from TeamNewPipe/bandcamp-comments
Add Bandcamp comments (first page)
2021-05-02 13:43:53 +02:00
Fynn Godau 85a4270bd9 [Bandcamp] Fix comments test 2021-05-02 13:07:24 +02:00
TobiGr ec50570053 Add mocks 2021-05-01 17:50:19 +02:00
TobiGr f4404b5cc8 Add tests for more reasons 2021-05-01 17:50:19 +02:00
TobiGr fc998589dc Detect channels which have been terminated due to copyright infringement 2021-05-01 17:50:19 +02:00
TobiGr bb3861ddce Add AccountTerminatedException for better error handling of terminated channels 2021-05-01 17:50:19 +02:00
Tobi 284362fc15
Merge pull request #608 from TeamNewPipe/youtube-fix-segment-test
[YouTube] fix tests
2021-05-01 17:24:41 +02:00
TobiGr dc678a9f6d Release version 0.21.2 2021-04-26 18:06:30 +02:00
Tobi 7cf8edd5ac
Merge pull request #618 from TeamNewPipe/soundcloud_id
Fix Soundcloud extraction
2021-04-26 18:03:26 +02:00
TobiGr 9c12dc5609 [SoundCloud] Fix SoundCloud ID extraction
resolveIdWithEmbedPlayer() does not work anymore because the JSON data has been extracted to an API call. For this reason, replace resolveIdWithEmbedPlayer() with resolveIdWithWidgetApi)( which performs the API call.
2021-04-26 17:58:34 +02:00
Fynn Godau 9dc7a1d5ee [Bandcamp] Add comments (first page) 2021-04-26 17:25:04 +02:00
TobiGr 8f023c1ec7 [SoundCloud] Update client id 2021-04-26 15:14:11 +02:00
Tobi 7f202db8b1
Fix Google URL detection (#617) 2021-04-26 11:49:47 +02:00
fynngodau 6db4bea8ca
Rework Bandcamp comments (#613) 2021-04-13 21:10:59 +02:00
Tobi c14f6db14a
Merge pull request #610 from TeamNewPipe/bandcamp_related_items
Extract related bandcamp items
2021-04-13 21:05:27 +02:00
Fynn Godau 3671876721 Fix bandcamp radio stream extractor 2021-04-13 19:30:11 +02:00
Fynn Godau 90b5c00599 Extract related items #593 2021-04-13 17:56:19 +02:00
TobiGr 1fe645704f [YouTube] FIx testMoreRelatedItems in RandomQueryNoMorePages 2021-04-11 23:21:35 +02:00
TobiGr 9a2b814d8f Update mocks 2021-04-11 22:59:01 +02:00
TobiGr 50005ce937 [YouTube] Replace unavailable YouTube Stream Segments test
The stream is unavailable in many countries.
2021-04-11 22:58:48 +02:00
Tobi 82d11386df
Merge pull request #607 from TeamNewPipe/bandcamp_radio_segments
[Bandcamp] Add Radio stream segments
2021-04-11 22:30:03 +02:00
TobiGr 318cec7625 [Bandcamp] Add Radio stream segments 2021-04-11 21:19:44 +02:00
TobiGr 55e0f8e725 Merge branch 'master' into dev 2021-04-10 11:33:34 +02:00
Tobi e747a89be2
Merge pull request #600 from TeamNewPipe/youtube-eu-cosent
[YouTube] Set EU consent cookie
2021-04-09 12:02:52 +02:00
TobiGr 33173eb3e6 Make CONSENT COOKIE constants private 2021-04-09 11:59:17 +02:00
TobiGr b94b316558 Update mocks 2021-04-09 11:59:17 +02:00
TobiGr 4b8d4a84b6 Set the NumberGenerator for th remaining tests 2021-04-09 11:59:17 +02:00
XiangRongLin db7ad9c27d Update Mocks 2021-04-08 16:37:29 +02:00
XiangRongLin 4833df60e5 Add method to inject Random into YoutubeParsingHelper and use in tests 2021-04-08 16:36:55 +02:00
TobiGr 682ec27737 Update mocks 2021-04-08 14:22:31 +02:00
XiangRongLin eafad3ad6b
Merge pull request #602 from XiangRongLin/soundcloud_go
Ignore unreliable SoundcloudGoPlusTrack.testRelatedItems
2021-04-07 19:41:44 +02:00
TobiGr 080d1e318d Code improvement 2021-04-07 14:11:47 +02:00
TobiGr 4ee270fe01 Fix LiveStream Test 2021-04-07 14:11:41 +02:00
TobiGr 883f16e0ad [YouTube] Set CONSENT cookie 2021-04-07 14:11:37 +02:00
XiangRongLin aee3838ed5 Ignore unreliable SoundcloudGoPlusTrack.testRelatedItems 2021-04-06 18:26:45 +02:00
XiangRongLin 90afd6162a
Merge pull request #599 from XiangRongLin/post_body
Add body field to Page and use it
2021-04-06 18:23:26 +02:00
XiangRongLin 7b06c696e2 Use Page.body for YoutubeChannelExtractor 2021-04-06 18:14:38 +02:00
XiangRongLin eda1b6e199 Use Page.body for YoutubePlaylistExtractor 2021-04-06 18:14:38 +02:00
XiangRongLin 3af26a2821 Add body field to Page 2021-04-04 11:39:58 +02:00
Tobi 4e0be60ddc
Merge pull request #601 from XiangRongLin/fix_tests
Fix SoundcloudGoPlusTrack Test
2021-04-04 09:55:54 +02:00
XiangRongLin 7b36469807 Fix SoundcloudGoPlusTrack
https://soundcloud.com/martinsolveig/places#t=0 has related streams
2021-04-04 09:39:26 +02:00
Tobi b481a228cd
Merge pull request #598 from XiangRongLin/fix_tests
Fix a test and ignore rest
2021-04-03 23:44:23 +02:00
XiangRongLin 3d70b6d46a Fix a test and ignore rest 2021-04-03 11:03:10 +02:00
XiangRongLin 1925dcf4dc
Merge pull request #597 from fynngodau/related-items
Rename getRelatedStreams to getRelatedItems and change return type
2021-04-02 19:54:22 +02:00
Fynn Godau c877712647 Add deprecated old method calls to avoid breaking API 2021-04-01 22:32:36 +02:00
Fynn Godau 14f6f1b7c3 Generify related streams calls and rename method 2021-03-31 20:24:40 +02:00
Stypox 9dbc152d44
Merge pull request #594 from TeamNewPipe/conersions
Reduce number of type conversions
2021-03-31 14:34:46 +02:00
TobiGr 1498e1905e Reduce number of type conversions in YouTubeSearchExtractor.collectStreamsFrom by introducing new variable 2021-03-31 13:30:38 +02:00
Tobi 7ea2cd73d3
Merge pull request #589 from fynngodau/bandcamp-more-recent-pages
Load more featured pages and fix featured cover arts not loading
2021-03-31 12:15:12 +02:00
Fynn Godau 705f6c6e33 Apply review 2021-03-31 11:02:30 +02:00
XiangRongLin 564d74c250
Merge pull request #578 from TeamNewPipe/code_improvements
Code improvements
2021-03-30 18:12:22 +02:00
TobiGr 657b00ca11 Smaller code improvements 2021-03-30 10:10:53 +02:00
TobiGr 8fbd1a2bcf FIx potential NullPointerException 2021-03-30 10:07:20 +02:00
TobiGr f2e7c7ebd6 Replace Vector with ArrayList for performance improvements
I could not figure out, why these two methods are necessary.
2021-03-30 10:07:20 +02:00
TobiGr 48fd8ca7b3 Remove redundant type declaration 2021-03-30 10:07:20 +02:00
TobiGr 22bd16fca3 Make vars private, final and static 2021-03-30 10:07:20 +02:00
TobiGr 6a0f6e846a Fix possible bug when language.id field is not present and this the languageCode an empty String. 2021-03-30 10:07:20 +02:00
TobiGr b9282bbe94 Remove checks which are always true 2021-03-30 10:07:20 +02:00
TobiGr 46eab1ec17 Remove declarations of exceptions which are not thrown from method signatures 2021-03-30 10:07:20 +02:00
TobiGr c3e23559d7 Remove unused imports 2021-03-30 10:07:20 +02:00
TobiGr e4c40ae6d1 Use List.isEmpty() 2021-03-30 10:07:20 +02:00
XiangRongLin af2183855a
Merge pull request #591 from TotalCaesar659/update-url-to-https
Update URL to HTTPS in README
2021-03-29 09:33:59 +02:00
TotalCaesar659 cf3f54f438 Update URL to HTTPS 2021-03-28 22:06:56 +03:00
Fynn Godau d663be5a78 [Bandcamp] Fix featured albums not showing thumbnails 2021-03-28 13:12:22 +02:00
Fynn Godau dbcf61c6f7 [Bandcamp] Load more featured pages 2021-03-28 12:53:05 +02:00
Stypox b4dee6d08f
Merge pull request #306 from B0pol/metadata
Extract metadata for youtube, soundcloud & mediaccc
2021-03-27 08:45:47 +01:00
TobiGr e61ceef005 Release 0.21.0 2021-03-25 21:59:15 +01:00
Tobi e2500fb7cc
Merge pull request #583 from TeamNewPipe/peertube_playlist
Fix some tests and bugs
2021-03-25 21:53:41 +01:00
TobiGr 070a40e181 [YouTube] Mix Playlist - Fix getting name 2021-03-25 21:47:16 +01:00
TobiGr b029779217 [media.cc.de] Disable unreliable tests for now 2021-03-25 19:44:44 +01:00
TobiGr bbee15474d [PeerTube] Fix playlist extraction 2021-03-25 19:19:19 +01:00
Tobi d4f83a1782
Merge pull request #575 from TiA4f8R/fix-peertube-test
Fix Peertube account subscribers extraction
2021-03-25 19:14:51 +01:00
TiA4f8R ae283314da
Add a comment explaining subscribers count calculation 2021-03-25 17:54:46 +01:00
TiA4f8R abb790f465
Remove println in getSubscriberCount from PeertubeAccountExtractor class 2021-03-25 17:49:22 +01:00
TiA4f8R 772de53a66
Increase the number of subscribers in PeertubeAccountExtractorTest 2021-03-25 17:49:20 +01:00
TiA4f8R efe2b964f0
Extract real subscribers of a Peertube account
Apply fix provided in the PR and move all accounts/ strings to a constant, ACCOUNTS.
2021-03-25 17:47:44 +01:00
Tobi 12835bfae1
Merge pull request #551 from FireMasterK/ff-privacy
Change UA to privacy.resistFingerprinting.
2021-03-24 17:58:34 +01:00
Tobi cae6cea551
Merge pull request #582 from Stypox/fix-trending
[YouTube] Fix trending getName()
2021-03-24 10:49:46 +01:00
Stypox ce7cbbc9a0
[YouTube] Fix trending getName() 2021-03-24 09:14:09 +01:00
XiangRongLin 9009f1e277
Merge pull request #517 from XiangRongLin/flaky_test
Add MockOnlyRule to allow skipping specific tests based on downloader
2021-03-20 18:04:14 +01:00
bopol 152221c7fb
ignore a test with encoding problems 2021-03-19 10:14:43 +01:00
bopol 8806fb4e6b
address reviews 2021-03-19 10:14:43 +01:00
bopol c47cc54908
Extract metadata for YouTube, SoundCloud & MediaCCC 2021-03-19 10:14:43 +01:00
Stypox f71cfd489c
Merge pull request #526 from TiA4f8R/snd-hls-workaround
Support SoundCloud HLS-only tracks by using a workaround
2021-03-19 10:09:54 +01:00
TiA4f8R 379d7312fa
Don't use a regular expression to find the last segment URL and do code improvements
Apply suggestions provided in the PR and remove a redundant import.
2021-03-14 17:54:22 +01:00
TiA4f8R 0e3e420a25
Fix tests 2021-03-14 17:54:21 +01:00
TiA4f8R d61d9d116d
Refactor getAudioStreams method of SoundcloudStreamExtractor
Split the method into private methods, in order to have a better reading.
2021-03-14 17:54:20 +01:00
TiA4f8R a7b15b51e6
Change t to transcodingJsonObject in SoundcloudStreamExtractor 2021-03-14 17:54:19 +01:00
TiA4f8R 0438828e36
Add a test for the number of audioStreams in CreativeCommonsPlaysWellWithOthers test
It should be only two audio streams for track "Plays Well with Others, Ep 2: What Do an Army of Ants and an Online Encyclopedia Have in Common?" by Creative Commons (https://soundcloud.com/wearecc/plays-well-with-others-ep-2-what-do-an-army-of-ants-and-an-online-encyclopedia-have-in-common):
- one which is a progressive stream, in MP3 format with a bitrate of 128 kbps
- one which is an HLS stream, in OPUS format with a bitrate of 64 kbps.
2021-03-14 17:54:18 +01:00
TiA4f8R 3bd08a2880
Adress requested changes and use final where possible in SoundcloudStreamExtractor
This commit moved the HLS parsing task to a separate method, did little performance improvements and used final where possible in the SoundcloudStreamExtractor file.
2021-03-14 17:54:17 +01:00
TiA4f8R cbacd3c0a5
Add a check to don't show MP3 128kbps stream twice and catch IOException when fetching the HLS Manifest
If a progressive stream is present in the transcodings, it's unnecessary to show twice an MP3 128kbps stream so if this is the case, the MP3 HLS stream will be not added to the audioStreams, else it will.
This commit also catch fetching errors in HLS manifests parsing and don't add the corresponding stream if an error occurs.
2021-03-14 17:54:16 +01:00
TiA4f8R 26f1b4e7dc
Support SoundCloud HLS by using a workaround
This commit tries to support SoundCloud HLS streams by parsing M3U manifests, get the last segment URL (in order to get track length) and request a segment URL equals to track's duration so it's a single URL.
2021-03-14 17:54:14 +01:00
Tobi def745b801
Merge pull request #579 from fynngodau/bandcamp-fix-linkhandler
[Bandcamp] Fix link handler acceptance behaviour
2021-03-14 12:56:10 +01:00
Fynn Godau 2e57a8f24f [Bandcamp] Fix link handler acceptance behaviour
* Test for bandcamp footer instead of meta tag (which is not present on
  all pages)
* Accept links to /music, not just /releases
* Correctly handle uppercase URLs
2021-03-14 09:51:30 +01:00
TobiGr 70d9e389b9 [Bandcamp] Add tests for external bandcamp artists
Necessary, because the external pages tested before were converted to bandcamp.com pages. See f9d06252f2
2021-03-13 20:51:59 +01:00
TobiGr f9d06252f2 [Bandcamp] Fix tests 2021-03-13 17:42:39 +01:00
Tobi 7e6f464407
Merge pull request #509 from TiA4f8R/soundcloud-improvements
Add new exceptions to be able to display different error messages in apps
2021-03-07 13:51:15 +01:00
Tobi ac51134aed
Merge pull request #546 from XiangRongLin/unignore2
Unignore more tests
2021-03-07 13:15:24 +01:00
Tobi 021da75f24
Merge pull request #232 from fynngodau/dev
Bandcamp support
2021-03-05 21:37:57 +01:00
TobiGr 22fa131922 Merge branch 'dev' into bandcamp 2021-03-05 19:45:37 +01:00
TiA4f8R e55284bb8f
Code style improvements
Use final where possible in YoutubeStreamExtractor and do some other code style improvements
2021-03-05 16:38:53 +01:00
TobiGr d0a1041afd
Prevent NullPointerException and use equalsIgnorecase 2021-03-05 16:38:52 +01:00
TiA4f8R 890cbba625
Update PaidContent exception 2021-03-05 16:38:51 +01:00
TobiGr da3cfa967d
Handle age-restricted videos 2021-03-05 16:38:50 +01:00
TiA4f8R 448b68700c
Use final in exceptions and add AgeRestrictedException
AgeRestrictedException will be thrown only if the reason message equals to "Sign in to confirm your age" and if the age limit is 18.
2021-03-05 16:38:49 +01:00
TobiGr 22e6f33f0a
Ignore SoundCloudGeoRestrictedTest until HLS stream extraction is added
The ContentNotSupportedException is thrown because no supported audio streams where extracted. However, SoundCLoud does not check, whether there are any streams available. 
This commit should be reverted in #526
2021-03-05 16:38:47 +01:00
TobiGr 21158744b1
Add expectedDescriptionIsEmpty() to DefaultStreamExtractorTest
Also check if related streams are empty if they are expected to be empty.
2021-03-05 16:38:46 +01:00
TiA4f8R f15d7837a1
Fix SoundCloud tests 2021-03-05 16:38:45 +01:00
TiA4f8R 4e45aef2b3
Fix SoundCloud CreativeCommonsPlaysWellWithOthers test 2021-03-05 16:38:43 +01:00
TiA4f8R 59d6d3f04e
Fix YouTube tests 2021-03-05 16:38:42 +01:00
TiA4f8R 3925204658
Add tests and rename SoundcloudGoPlusException to SoundcloudGoPlusContentException 2021-03-05 16:38:40 +01:00
TiA4f8R 771bb1a2cb
Add three new exceptions for unavailable YouTube contents
These exceptions are thrown on a test with the error messages text, because YouTube returns only "UNPLAYABLE" status in most error cases.
Tests are based with English strings, so changing the lang used by
extractor will throw the generic exception (ContentNotAvailableException).
2021-03-05 16:38:39 +01:00
TiA4f8R 35325d980d
Add GeographicRestrictionException and SoundCloudGoPlusException in NewPipe Extractor to be able to display different error messages
This commit adds two new exceptions in NewPipe Extractor: GeographicRestrictionException and SoundCloudGoPlusException (which extend to ContentNotAvailableException). These exceptions allow showing different error messages to user when a content isn't available in his/her/its country (only used for now by SoundCloudStreamExtractor) or when the content is a SoundCloud Go+ track.
2021-03-05 16:38:37 +01:00
TobiGr cb07ffa1eb Release 0.20.11 2021-03-05 15:35:11 +01:00
Tobi bc0cda68d1
Merge pull request #570 from TiA4f8R/fix-yt-channels-playlists
Fix YouTube channels contination
2021-03-05 15:26:49 +01:00
Tobi a3c6fceef5
Merge pull request #573 from B0pol/comments-performance
[youtube] improve comments extraction performance
2021-03-05 14:46:27 +01:00
bopol ff5273b882
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeCommentsExtractor.java
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2021-03-05 14:39:01 +01:00
Tobi ec1127dd6a
Merge pull request #571 from TeamNewPipe/soundcloud_client_id
[SoundCloud] Update client_id and fix extraction
2021-03-05 14:21:14 +01:00
bopol ed850d0688 [youtube] improve comments extraction performance
- do not parse responseBody twice for continuation
instead try to get commentsTokenInside with the new pattern ("sectionListRenderer")
and try again with the old pattern ("commentSectionRenderer") on failure
- do not unescape responseBody multiple times
   -> parse responseBody less times
2021-03-05 13:37:08 +01:00
TobiGr 440a808b8a Update mocks for channels 2021-03-05 13:13:36 +01:00
Tobi b608587e4d
Merge pull request #572 from golfinq/fixed-comments
Fixes missing comments under videos
2021-03-05 11:42:40 +01:00
golfinq 05213175cd
minor edit, made variable final 2021-03-05 02:49:15 -05:00
golfinq 8bceb57a6c
formatting fix 2021-03-05 02:46:06 -05:00
golfinq ae48bdea4c
Updated to reflect variable name change 2021-03-05 02:40:36 -05:00
TiA4f8R b27efdc9a4
Remove clickTrackingParams in getNextPage method of YoutubeChannelExtractor
Because it's unused with the new request.

Co-authored-by: bopol <bopol@e.email>
2021-03-04 22:50:20 +01:00
TobiGr 2b671b15ce [YouTube] Use new continuation API for channels
Co-authored-by: TiA4f8R <74829229+tia4f8r@users.noreply.github.com>
2021-03-04 21:47:16 +01:00
TobiGr 33594d4aff [SoundCloud] Fix client id extraction
SoundCloud changed their Javascript structure. That caused the client_id to be in the middle of the file (at byte ~43000 ). To have a little buffer, we now fetch the first 50000 bytes.
2021-03-04 21:32:34 +01:00
TobiGr fb2f37d223 [SoundCloud] Update hardcoded client id 2021-03-04 21:25:41 +01:00
XiangRongLin 9256b3b848
Merge pull request #567 from XiangRongLin/playlist_continuations
Playlist continuations
2021-03-04 19:05:32 +01:00
XiangRongLin 506cc5fbe9 regenerate mocks 2021-03-04 19:01:58 +01:00
XiangRongLin 03b00ff1d6 Move toJsonArray and toJsonObject to JsonUtils 2021-03-04 18:58:51 +01:00
FireMasterK 668b080b3a Use dynamic key. 2021-03-04 08:45:14 +01:00
XiangRongLin 43b46bd408 Use new youtube continuations api for playlists
Requires sending a POST request instead of GET.
clientName and clientVersion, which were required as headers previously now need to be part of the request payload.
continuation id also needs to be part of request body.

quick and dirty solution.
2021-03-03 19:49:26 +01:00
XiangRongLin beb05bd05c Replace test url, because playlist was deleted 2021-03-03 19:46:14 +01:00
XiangRongLin e13e237392 Fix typo and reword some explanations 2021-02-26 17:48:03 +01:00
Tobi 5d594cfded
Merge pull request #549 from FireMasterK/gradle-update
Update gradle wrapper.
2021-02-24 15:59:28 +01:00
Tobi 965a66bc87
Merge pull request #557 from TiA4f8R/fix-encoding-errors-windows
Set extractor encoding to UTF-8
2021-02-24 15:49:58 +01:00
TiA4f8R b2cf41496d
Set extractor encoding to UTF-8
When compiling the extractor on Windows, you get more than 100 errors with the message error: unmappable character for encoding Cp1252, especially in timeago-parser tasks. This commit fixes it by setting the encoding for all compileJava and compileTestJava tasks to UTF-8.
2021-02-23 14:38:20 +01:00
TobiGr a1688fe953 Move BandcampExtractorHelper.getJsonData(String, String) to JsonUtils 2021-02-21 13:51:12 +01:00
TobiGr 70814dcfef Fix Utils.nonEmptyAndNullJoin
When using the index here, it the index needs to be decremented once an element is removed. To cirecumvent this, the native Collections.removeIf() method is used.
2021-02-21 12:51:47 +01:00
TobiGr b9e8ee8450 Rename BandcampExtractorHelper.smartConcatenate(String[], String) to Utils.nonEmptyAndNullJoin(String, String[]) 2021-02-21 12:35:14 +01:00
TobiGr c07db80ef0 Add BASE_URL and BASE_API_URL to BandcampExtractorHelper 2021-02-21 12:15:45 +01:00
TobiGr adde4332d1 Code improvements 2021-02-20 23:58:54 +01:00
FireMasterK 42bdd8adc5
Update to fix failing mock tests. 2021-02-20 17:14:41 +05:30
FireMasterK e8bccfaf5d
Change UA to privacy.resistFingerprinting. 2021-02-20 17:14:41 +05:30
bopol a9aa385bb3
Merge pull request #554 from B0pol/test-verified
use DownloaderFactory instead of DownloaderTestImpl for search Channe…
2021-02-20 12:12:43 +01:00
bopol 0c831afa7b use DownloaderFactory instead of DownloaderTestImpl for search ChannelVerified 2021-02-20 11:41:30 +01:00
TobiGr ea120a4637 remove print stacktrace
use interface
2021-02-20 00:23:45 +01:00
XiangRongLin adf9d7d10f Add reason field to MockOnly
This enforces developers to document why a test is skipped
2021-02-19 18:54:18 +01:00
XiangRongLin ea52030613 Add MockOnlyRule to allow skipping specific tests based on downloader 2021-02-19 18:54:17 +01:00
TobiGr 02920fafa8 Add isUploaderVerified() 2021-02-19 15:51:23 +01:00
TobiGr 91e9309486 Merge remote-tracking branch 'origin/dev' into bandcamp 2021-02-19 13:37:45 +01:00
TobiGr fa61b864f2 Code improvements 2021-02-19 13:37:24 +01:00
TobiGr 98268e351c Move radio URL check into a function 2021-02-19 13:35:44 +01:00
FireMasterK 959d5d7db7
Add dependabot. 2021-02-19 11:52:56 +05:30
bopol 27a20e41cd
Merge pull request #550 from FireMasterK/add-more-data
Add more data.
2021-02-18 20:28:27 +01:00
FireMasterK 5333d8a98b
Add more data. 2021-02-19 00:55:03 +05:30
FireMasterK 88890d81a5
Update gradle wrapper. 2021-02-18 21:15:12 +05:30
bopol 9c1a7f7df6
Merge pull request #521 from B0pol/verified
Add uploader verified by service extraction
2021-02-18 16:39:53 +01:00
bopol 1a322ad8ed Add uploader verified by service extraction 2021-02-18 16:36:40 +01:00
bopol db253e202b
Merge pull request #535 from TeamPiped/adaptive-parsing
Adaptive Stream parsing
2021-02-18 15:55:27 +01:00
FireMasterK c24afa2cbb Permalink URL. 2021-02-18 20:16:53 +05:30
FireMasterK 8d54401233 Add fps field as well. 2021-02-18 20:16:53 +05:30
FireMasterK 4d096be14a Revert "Update gradle wrapper."
This reverts commit 591856e3a11998c4b8ba87b5e361e2b9556d760f.
2021-02-18 20:16:53 +05:30
FireMasterK 5b70645d85 Add data to muxed streams aswell. 2021-02-18 20:16:53 +05:30
FireMasterK e975d33fbe Change jdoc / comments. 2021-02-18 20:16:53 +05:30
FireMasterK f31b2a68fd use getters/setters. 2021-02-18 20:16:53 +05:30
FireMasterK d4945ac55c Constructor cleanup. 2021-02-18 20:16:52 +05:30
FireMasterK 3f6a601be8 Fix jdoc / comments. 2021-02-18 20:16:52 +05:30
FireMasterK a7c9905183 Fix requested changes. 2021-02-18 20:16:52 +05:30
FireMasterK 0c0f2d74bc Make fields private
Co-authored-by: bopol <bopol@e.email>
2021-02-18 20:16:52 +05:30
FireMasterK d1054338d3 Fix error with livestreams. 2021-02-18 20:16:52 +05:30
FireMasterK 11eb4932f4 Add data to respective classes. 2021-02-18 20:16:52 +05:30
FireMasterK 525e345ed8 remove duplicate field. 2021-02-18 20:16:52 +05:30
FireMasterK e844d2aed3 Add dash fields to the ItagItem. 2021-02-18 20:16:52 +05:30
FireMasterK b713a7af8c Update gradle wrapper. 2021-02-18 20:16:52 +05:30
bopol 64f6b0478f
Merge pull request #545 from XiangRongLin/comment_reset
Add comment explaining YoutubeParsingHelper.resetClientVersionAndKey
2021-02-18 15:14:03 +01:00
XiangRongLin a64579da2f Unignore some tests that work on Linux 2021-02-17 20:31:26 +01:00
XiangRongLin b99cb8624f Unignore PublicBroadcasterTest
Tests only work on linux, mocks have to be generated on linux too
2021-02-17 20:31:26 +01:00
XiangRongLin 37366c65ca Remove encoding of '&' to '&amp;' in test and unignore it 2021-02-17 20:31:26 +01:00
XiangRongLin c327922cc0 Add explanations to broken tests 2021-02-17 20:31:26 +01:00
XiangRongLin 26d50aca05 Add comment explaining YoutubeParsingHelper.resetClientVersionAndKey 2021-02-17 19:21:39 +01:00
TobiGr 54aa5b3042 Use propper structure in KioskExtractors
Made BandCampRadioExtractor a Kiosk which holds StreamInfoItems and not InfoItems.
2021-02-16 21:09:42 +01:00
TobiGr 54b8e54f80 Fix potential NPE 2021-02-16 20:59:13 +01:00
TobiGr 16973126a0 Use Collections.emptyList() instead of Collections.EMPTY_LIST or new ArrayList() 2021-02-16 20:25:48 +01:00
bopol 84380e4c3a
Merge pull request #536 from B0pol/watchendpoints
support /watch/, /v/ and /w/ subpaths
2021-02-16 20:23:27 +01:00
bopol 41f689b099 Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/linkHandler/YoutubeStreamLinkHandlerFactory.java
Co-authored-by: Stypox <stypox@pm.me>
2021-02-16 19:17:10 +01:00
XiangRongLin a9594f531f
Merge pull request #543 from XiangRongLin/unignore1
Unignore and fix some tests
2021-02-16 18:00:16 +01:00
XiangRongLin a28e0226c5
Merge pull request #519 from XiangRongLin/better_logging_error_handling
Better logging, error handling, doc for mock tests
2021-02-16 15:06:06 +01:00
XiangRongLin 05cb22c654 Unignore Gronkh.testDescription
Remove special character ü from assertion
2021-02-16 08:59:35 +01:00
XiangRongLin d9b4c2779f Unignore YoutubeSearchQHTest.testRegularValues 2021-02-16 08:59:35 +01:00
XiangRongLin 137272b7a4 Unignore YoutubeCommentsExtractorTest.EmptyComment
Mock files were missing
2021-02-16 08:59:34 +01:00
TobiGr 5bf9fddba9 Code improvements 2021-02-15 22:45:17 +01:00
TobiGr e062c8cb0d Merge branch 'dev' into bandcamp 2021-02-15 22:09:41 +01:00
bopol 416cf17b7a
Merge pull request #532 from B0pol/utf8
Code improvement
2021-02-15 18:56:28 +01:00
bopol 7a3d9bdb7d Description: add .equals() and .hashCode() 2021-02-15 18:51:47 +01:00
bopol 557934cb17 use EMPTY_STRING 2021-02-15 18:51:46 +01:00
bopol 9ca52ca68a use only one constant UTF-8 2021-02-15 18:47:54 +01:00
XiangRongLin 50e5718bd1
Merge pull request #518 from XiangRongLin/generate_mocks
Use and generate mocks for youtube tests
2021-02-15 06:39:41 +01:00
Tobi d7486f251a
Merge pull request #538 from B0pol/soundcloud-georestriction
Soundcloud: fix charts extraction when georestricted
2021-02-14 23:52:27 +01:00
TobiGr 432c68cdec Update okhttp from 3.12.11 to 3.12.13
Changelog https://square.github.io/okhttp/changelog_3x/#version-31213
2021-02-14 23:44:19 +01:00
TobiGr 5eb75d72d3 Release 0.20.10 2021-02-14 22:09:17 +01:00
XiangRongLin a88b2e6af4 Undo formatting of YoutubeStreamExtractorDefaultTest 2021-02-14 14:57:23 +01:00
bopol 8d89c82caa Soundcloud: fix charts extraction when georestricted 2021-02-13 22:28:31 +01:00
bopol ee2012c948 rename folder to subpath 2021-02-13 22:25:46 +01:00
bopol 8db40db4de use substring instead of split 2021-02-13 22:20:05 +01:00
bopol 81317cf6bc youtube: support /w/ folder 2021-02-13 22:20:05 +01:00
bopol f15c0fcfed support /watch/ and /v/ folders 2021-02-13 22:20:05 +01:00
XiangRongLin 971128c464 Ignore failing tests 2021-02-13 20:28:03 +01:00
XiangRongLin 92dea0806a Regenerate json files for youtube tests 2021-02-13 20:28:03 +01:00
XiangRongLin d6b9930c18 Use DownloaderFactory in YoutubeCommentsExtractorTest.Pinned 2021-02-13 20:12:39 +01:00
XiangRongLin b555f38fa5 Use real downloader for YoutubeSubscriptionExtractorTest
It doesn't make network requests anyway
2021-02-13 20:12:39 +01:00
XiangRongLin f352b0c7af YoutubeMusicSearchExtractorTest can't be used with mocks
Requires further research
2021-02-13 20:00:51 +01:00
XiangRongLin f45b5610bf Unignore YoutubeChannelLocalizationTest
With mocks time is not a concern anymore
2021-02-13 20:00:51 +01:00
XiangRongLin 19737e06ee Use DownloaderFactory in YoutubeChannelExtractorTest
KurzgesagtAdditional needed to be extracted because it ran a duplicated request.
2021-02-13 20:00:51 +01:00
XiangRongLin 84eb4b30c9 Use DownloaderFactory in YouTube tests 2021-02-13 20:00:51 +01:00
XiangRongLin b43f3474f5 Make resetClientVersionAndKey public 2021-02-13 20:00:49 +01:00
Tobi 8d7b62914c
Merge pull request #540 from B0pol/fix-tests
fix tests, fix links in YouTube description
2021-02-13 13:20:29 +01:00
bopol 2859c7ba5c add comment explaining why .substring(23) 2021-02-13 12:10:41 +01:00
bopol 73353996d7 fix more tests 2021-02-12 23:18:38 +01:00
bopol a931e31239 fix tests 2021-02-12 22:36:18 +01:00
TobiGr d116680ea3 Merge branch 'master' into dev 2021-02-12 22:26:35 +01:00
TobiGr f8d769617a Release 0.20.9 2021-02-12 22:14:23 +01:00
Tobi c2525916ac
Merge pull request #539 from B0pol/yt-channel-continuation
[YouTube]: Fixed channel continuation
2021-02-12 22:09:31 +01:00
bopol b0f356dd76 YT channels: fix continuation 2021-02-12 14:39:09 +01:00
bopol fcdb50b825 make getInfo from Extractor public
In StreamInfo and CommentsInfo
2021-02-09 14:58:43 +01:00
bopol 44c54d403a
fix Invidious channels being accepted by YoutubeStreamLinkHandlerFactory (#527)
fixes https://github.com/TeamNewPipe/NewPipeExtractor/issues/524, see it for more details
2021-02-05 18:57:58 +01:00
TiA4f8R 1414a6f178
Fix extraction of YouTube chapters (#531)
YouTube changed the name of the chapters in the JSON from engagement-panel-macro-markers to engagement-panel-macro-markers-description-chapters, so extracting chapters doesn't work.
2021-02-03 22:07:34 +01:00
Tobias Groza bfa639950e
Merge pull request #528 from Stypox/ytmusic-0
[YouTube Music] Fix extracting search item view/subscriber count when = 0
2021-01-29 23:23:23 +01:00
TobiGr d728c4fad0 Remove unused imports
I searched for "StandardCharsets" to prevent possible bugs with Android KitKat. The import is unused and is therefor removed.
There are other usages of the StandardCharsets class in the test implementation, but those are not relevant.
2021-01-29 23:16:40 +01:00
Stypox a64dfd7343
[YouTube Music] Fix extracting search item view/subscriber count when = 0 2021-01-29 16:44:24 +01:00
Tobias Groza c00ee75d74
Merge pull request #525 from B0pol/rename
rename getHeartedByCreator and getPinned to isPinned and isHeartedByC…
2021-01-23 13:45:24 +01:00
bopol 2662ceca22 rename getHeartedByCreator and getPinned to isPinned and isHeartedByCreator 2021-01-23 13:06:07 +01:00
Tobias Groza 10e8c543e8
Merge pull request #523 from B0pol/pinned
Extract whether the comment is pinned
2021-01-23 12:35:13 +01:00
bopol 66e4eb2f96 extract whether the comment is pinned 2021-01-22 23:59:22 +01:00
Tobias Groza 69f155d292
Merge pull request #522 from B0pol/invidious_instances
update invidious instances list
2021-01-22 21:04:10 +01:00
bopol 67ddfefdc6 keep invidious instances list in the same order
as suggested by @TobiGr
2021-01-22 20:58:14 +01:00
bopol 53d3f7989b update invidious instances list 2021-01-22 19:30:25 +01:00
XiangRongLin 88e4c8667b Add comments to RecordingDownloader about usage 2021-01-17 19:53:20 +01:00
XiangRongLin eecfe09f2c Check for non-existent folder in MockDownloader 2021-01-17 19:52:58 +01:00
XiangRongLin a0288b8fd6 Echo which downloader is used during CI process 2021-01-17 19:52:28 +01:00
Tobias Groza beb705013c
Merge pull request #516 from XiangRongLin/schedule_ci_with_real
Add if-else statement to run tests against real website on schedule
2021-01-16 12:59:05 +01:00
XiangRongLin 7a6d930e7b Add if-else statement to run tests against real website on schedule 2021-01-16 10:03:47 +01:00
TobiGr 07570554b6 Fix build and optimize imports 2021-01-15 21:55:40 +01:00
TobiGr 78c2113094 Merge remote-tracking branch 'origin/dev' into bandcamp 2021-01-15 21:49:58 +01:00
Stypox b9ba95614b
Merge pull request #514 from XiangRongLin/test_jvm_property
Pass on gradle JVM system property to test JVM
2021-01-15 20:44:23 +01:00
XiangRongLin a6cfe11ae9 Pass on gradle JVM system property to test JVM 2021-01-15 20:37:14 +01:00
Stypox 48a9993edd
Merge pull request #513 from XiangRongLin/ci_use_mock
Adjust CI workflow to run tests with mocks
2021-01-15 20:20:23 +01:00
Stypox 588fbcc61f
Merge pull request #512 from XiangRongLin/ignore_failing
Ignore all failing tests
2021-01-15 20:18:03 +01:00
XiangRongLin 6888e7773f Ignore all failing tests 2021-01-15 20:11:40 +01:00
XiangRongLin e5dc62d9bd Add argument -Ddownloader=MOCK to CI job
Now MockDownloader will be used in tests.
2021-01-15 18:04:33 +01:00
Stypox 7c774c84ce
Merge pull request #505 from TeamNewPipe/meida.ccc.de_search_fix
[media.ccc.de] Fix NPE in search results if they contain a future talk
2021-01-15 14:16:59 +01:00
Stypox c2c4d97efd
Merge pull request #482 from XiangRongLin/mock_mix_pl_test
Mock mix pl tests
2021-01-15 12:54:48 +01:00
XiangRongLin 4f81d9d689 Regenerate mock files 2021-01-15 09:51:23 +01:00
Stypox 951159f75f
Merge pull request #504 from TeamNewPipe/media.ccc.de_recent_order
[media.ccc.de] Recent kiosk: order streams by upload date
2021-01-14 23:31:09 +01:00
Tobias Groza cc51c5f6a3
Merge pull request #507 from vkay94/add-interval-to-stream-frames
Add durationPerFrame to Frameset
2021-01-14 20:27:40 +01:00
XiangRongLin 6ca7123d5d Add comment for usage of YoutubeParsingHelper.resetClientVersionAndKey 2021-01-14 20:01:52 +01:00
vkay94 11dcfe638b Add durationPerFrame to Frameset and getFrameBoundsAt method 2021-01-14 20:01:06 +01:00
Tobias Groza c2ff6723d7
Merge pull request #511 from B0pol/errors
Fix PeerTube tests by changing instance
2021-01-14 19:15:22 +01:00
Tobias Groza 6773ec8699
Merge pull request #510 from B0pol/heartedbycreator
comments: add heartedByUploader and extract it for YouTube
2021-01-12 19:00:04 +01:00
TobiGr 0c4836c800 [media.ccc.de] Recent kiosk: Add comment clarifying "length" and "duration" fields have the same value 2021-01-12 18:48:42 +01:00
XiangRongLin 4af50c6870 Use @link for downloader 2021-01-12 09:41:57 +01:00
XiangRongLin 3455f0f23c Add documentation to testing with mocks 2021-01-12 09:37:48 +01:00
bopol 5292cc1636 [peertube] test: remove usage of peertube.mastodon.host
It's down for months
2021-01-11 19:46:27 +01:00
bopol f3c22da618 comments: add heartedByCreator and extract it for youtube 2021-01-11 18:48:08 +01:00
Tobias Groza da968cf9c5
Merge pull request #508 from XiangRongLin/360_webm
[YouTube] Un-comment out 360p webm
2021-01-11 16:09:05 +01:00
XiangRongLin 63c237db41 Only create directories if they do not exist 2021-01-11 07:57:11 +01:00
XiangRongLin e82cb1efea Remove dependency to commons-io 2021-01-10 20:50:38 +01:00
XiangRongLin 1ea6c6ce54 Prefix mock file from RecordingDownloader with "generated_mock_"
Only read those files in MockDownloader
2021-01-10 20:46:42 +01:00
XiangRongLin f91916c017 Remove unnecessary file object creation 2021-01-10 20:39:22 +01:00
XiangRongLin 255c726f20 Add resource path base to DownloaderFactory 2021-01-10 20:27:34 +01:00
XiangRongLin 35e299759e Add method to allow resetting youtube client version and key
This is needed so that a request is made for each test class when running multiple at once. This way RecordingDownloader records all necessary requests.
This works as long as tests are run sequentially and not in parallel.
2021-01-10 20:24:50 +01:00
XiangRongLin f447a7a450 fix import due to downloader package move 2021-01-10 19:49:14 +01:00
XiangRongLin f8aa989c42 Add generated json and manually copy client_version.json 2021-01-10 19:38:47 +01:00
XiangRongLin 285c26eafa Adjust YoutubeMixPlaylistExtractorTest to use DownloaderFactory 2021-01-10 19:38:41 +01:00
XiangRongLin e6e8e39def Add DownloaderFactory to return a specific downloader based on 2 variables.
If the system property 'downloader' is set that use that specific downloader. This is used from gradle by appending `-Ddownloader=ABCD to the command.
ABCD is one of DownloaderType.
The other variable is the static property `DEFAULT_DOWNLOADER` in DownloaderFactory, which can be easily changed as needed inside the IDE according to development needs`.

Normal workflow would be to first use the recording downloader and afterwards only use mocks, if the requests are always staying the same.
2021-01-10 19:36:24 +01:00
XiangRongLin 7c40fb8bf7 Add additional downloader implementations
RecordingDownloader relies on the real downloader and saves the request/response pair into a json file.
MockDownloader uses json files from above and mocks responses for specific requests.
2021-01-10 19:36:24 +01:00
XiangRongLin 1bcb9c76a7 Generate equals and hashCode for Request 2021-01-10 19:36:24 +01:00
XiangRongLin 4dad3d60d6 Move DownloaderTestImpl into downloader package 2021-01-10 19:36:21 +01:00
XiangRongLin fe3902c669 Un-comment out 360p webm 2021-01-08 19:43:03 +01:00
TobiGr 2d93b23723 [media.ccc.de] Show conference title in search results instead of slug 2021-01-08 14:15:25 +01:00
TobiGr 52f3bd15f2 [media.ccc.de] Add conference title to "recent" kiosk
The "conference_title" field was added upstream recently (see https://github.com/voc/voctoweb/issues/495)
2021-01-08 12:04:39 +01:00
Tobias Groza ab62464e86
Merge pull request #495 from B0pol/sepiasearch
[PeerTube] Implement sepia search
2021-01-06 11:06:36 +01:00
bopol 0bcea9c2cd add JDoc 2021-01-01 14:37:32 +01:00
bopol 0e45c25ea9
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/peertube/extractors/PeertubeStreamInfoItemExtractor.java
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2021-01-01 14:20:47 +01:00
TobiGr fae67fbd45 Merge remote-tracking branch 'origin/master' into dev 2021-01-01 13:06:11 +01:00
Tobias Groza c4b9183684
Merge pull request #506 from TeamNewPipe/gh-actions
Migrate to GitHub actions from Travis
2021-01-01 10:35:33 +01:00
TobiGr 4671e68304 Update links to Java SE components to Java SE 8 2020-12-31 18:05:49 +01:00
TobiGr d63d20f943 Fix JDoc 2020-12-31 18:05:40 +01:00
FireMasterK 1ba689900e Migrate to GitHub actions from Travis. 2020-12-31 18:05:35 +01:00
TobiGr 2cbc3ccb1a [media.ccc.de] Fix live stream test if there are no current live streams 2020-12-31 17:03:23 +01:00
TobiGr a896ec41b9 [media.ccc.de] Use Pattern for getting the ive stream id 2020-12-31 17:03:23 +01:00
TobiGr 5dc9a76e3c [media.ccc.de] Recent kiosk: order streams by upload date 2020-12-30 18:38:09 +01:00
TobiGr edf8dd0e92 [media.ccc.de] Fix NPE in search results if they contain a future talk 2020-12-30 18:34:28 +01:00
Tobias Groza 3c8c8e7307
Merge pull request #497 from TiA4f8R/fix-player-js-url-redirect
Fix YouTube player JavaScript URL redirection
2020-12-27 18:04:40 +01:00
TobiGr 288ddab65b update version name 2020-12-27 18:02:16 +01:00
TiA4f8R d30caeb091
Fix YouTube player JavaScript URL redirection 2020-12-27 15:55:03 +01:00
TobiGr deb9af7bf5 [media.ccc.de] live kiosk - display info of the current talk 2020-12-27 15:52:09 +01:00
TobiGr 21c6a7884c Release 0.20.8 2020-12-27 14:50:34 +01:00
Tobias Groza 1259521d53
Merge pull request #500 from TeamNewPipe/media.ccc.de_live
[media.ccc.de] live stream kiosk
2020-12-27 14:47:55 +01:00
TobiGr b4e1913971 [media.ccc.de] Play live streams 2020-12-27 13:25:48 +01:00
TobiGr 80f4d42226 [Draft] Add live extractor 2020-12-27 11:01:22 +01:00
TobiGr ed9402c002 [media.ccc.de] Escape "." in id pattern 2020-12-27 11:01:10 +01:00
TobiGr 674f2227f3 make url pattern static 2020-12-27 01:27:37 +01:00
TobiGr 090dade5b3 [media.ccc.de] Add short test for recent kiosk 2020-12-27 01:03:08 +01:00
TobiGr 0cfefe222a Revert "[media.ccc.de] Get conference name from API URL"
This reverts commit d1a0686d2f.
2020-12-26 16:55:44 +01:00
TobiGr d1a0686d2f [media.ccc.de] Get conference name from API URL 2020-12-26 13:15:03 +01:00
TobiGr 41699798fe [media.ccc.de] Use web URL for uploaderUrl instead of API URL in recent 2020-12-26 12:47:32 +01:00
TobiGr b2154c6e87 [media.ccc.de] Fix conferences kiosk test 2020-12-25 23:55:45 +01:00
TobiGr ed6ae329b3 [media.ccc.de] Add "recent" kiosk 2020-12-25 22:26:58 +01:00
TobiGr b2837698f5 Release 0.20.7 2020-12-25 18:55:06 +01:00
Tobias Groza b5e50cc9fb
Merge pull request #484 from TeamNewPipe/yt_,music_search
Fix YouTube Music search
2020-12-25 18:30:25 +01:00
TobiGr d9e2da53c3 Replace explicit string checks whether a playlist íd is a certain YouTube Mix type with calling the dedicated methods 2020-12-25 15:00:34 +01:00
TobiGr abee0a8df1 Fix typos in YoutubePlaylistLinkHandlerFactory 2020-12-24 10:34:21 +01:00
TobiGr 3033c0b993 [YouTube] Music Mix: Fix playlist not being accepted
Regression introduced by YouTube Mix support (#280)
2020-12-23 21:08:36 +01:00
TobiGr b62144b49d [YouTube] Music: Fix search extraction caused by changed JSON response 2020-12-23 21:08:36 +01:00
TobiGr c9d9bd1e24 Fix typo in ListLinkHandlerFactory#getUrl 2020-12-23 21:05:58 +01:00
bopol 3ae924a7f1 Implement sepia search 2020-12-23 15:07:55 +01:00
TobiGr 942d840624 Fix typo in AffiliateService: rename "NO_AFILIATE" to "NO_AFFILIATE" 2020-12-23 12:50:45 +01:00
TobiGr 865c42e273 Fix line break 2020-12-23 12:49:50 +01:00
Tobias Groza c682ea0d18
Merge pull request #451 from TeamNewPipe/meta_info
Extract stream and search meta info for YouTube
2020-12-22 23:05:33 +01:00
Fynn Godau c9e9953bb0 [Bandcamp] Fix channel link handler factory 2020-12-22 20:02:53 +01:00
TobiGr 41a8ed625d Make some vars final and add annotations to methods 2020-12-22 18:19:26 +01:00
TobiGr bc6de14952 Extract stream and search meta info for YouTube
Add method to extract Google webcache URLs.
2020-12-22 18:19:26 +01:00
TobiGr 50903730b1 [Bandcamp] Fix accepting HTTP URLs 2020-12-22 08:53:10 +01:00
Tobias Groza 853a65a1a6
Merge pull request #491 from TeamNewPipe/dependecies
Update junit
2020-12-22 08:31:03 +01:00
TobiGr 838dc1ab54 Fix vulnerability in junit 2020-12-21 23:22:16 +01:00
Tobias Groza 7e481cf81a
Merge pull request #490 from B0pol/peertube_related
[peertube] fix related streams api url when no tags
2020-12-20 19:04:35 +01:00
bopol 8d3436565f [peertube] fix related streams api url when no tags
fixes #489
2020-12-19 23:25:37 +01:00
Tobias Groza 22a415156f
Merge pull request #479 from vkay94/stream-segments
Extract stream segments for YouTube
2020-12-19 11:12:22 +01:00
TobiGr 74b46fed2d [Bandcamp] Fix deprecation in parseDate helper function 2020-12-19 10:40:21 +01:00
TobiGr 116e921d6c Merge remote-tracking branch 'origin/dev' into bandcamp 2020-12-19 00:26:11 +01:00
Tobias Groza e8cc302d5f
Merge pull request #483 from XiangRongLin/cleanup-mix-pl
Cleanup mix pl
2020-12-19 00:13:17 +01:00
TobiGr df16a8646f [Bandcamp] Improve radio stream extractor test 2020-12-18 23:15:59 +01:00
XiangRongLin cdcb66b93e Clean up stuff from mix pl
Coming from unclean merge and missing hints from IDE
2020-12-16 08:35:28 +01:00
XiangRongLin ba8782a9ed Use lowercase string for extracting cookies
The map in a real response happens to ignore upper-/lowercase differences. Other maps used in unit test may not have that behaviour.
2020-12-16 08:32:04 +01:00
Stypox 85fa006214
Merge pull request #280 from XiangRongLin/mixPL
Extractor for youtube mix (auto-generated playlist)
2020-12-14 18:11:07 +01:00
XiangRongLin f90f6fcf92 [YouTube] Don't escape & in getUrlFromNavigationEndpoint for playlists 2020-12-12 20:40:13 +01:00
Xiang Rong Lin a338e4e08e [Youtube] Apply review suggestions and avoid channel mix edge case 2020-12-12 20:32:43 +01:00
Xiang Rong Lin 22d2f7e400 [Youtube] Add cookies to youtube mix request
This way youtube wont return duplicates when getting more items of the mix (but youtube can also track us)
2020-12-12 20:32:40 +01:00
Stypox 421935401f [Youtube] Add subchannel functions to mix and fix imports 2020-12-12 20:30:57 +01:00
Xiang Rong Lin 3ff8619bcc [Youtube] apply wb9688 suggestion (mix)
Channel mix adjusments and test
Don't accept youtube music mix urls as playlist
Don't override playlistData to keep getInitialPage()
Remove json constants
Indentation
2020-12-12 20:30:57 +01:00
Xiang Rong Lin 822cf307f7 [Youtube] Add _ITEMS constants and improve code style
Move thumbnail id exctraction code to getThumbnailUrlFromId
Add test for "My mix" detection to service tests
Use ITEM_COUNT_UNKNOWN everywhere instead of -1 and add some tests
2020-12-12 20:30:57 +01:00
Xiang Rong Lin df38b1926c [Youtube] Add tests and take thumbnail image always from first video of mix
Also fix getThumbnailUrl for "My Mix"
2020-12-12 20:30:57 +01:00
Xiang Rong Lin 68a3948af6 [Youtube] Fix get banner url 2020-12-12 20:30:57 +01:00
Xiang Rong Lin ec6b99c082 [Youtube] Adjust mix extractor to new user agent
Also extract continuation now
2020-12-12 20:30:57 +01:00
Xiang Rong Lin 0ff054acb4 [Youtube] Extract initial playlist info 2020-12-12 20:30:57 +01:00
Xiang Rong Lin a376792a5d [Youtube] Handle case where url is in "youtube.com/playlist?list=listID" format.
This occurs when sharing a mix from the official youtube app.
2020-12-12 20:30:57 +01:00
Xiang Rong Lin 327a5730a8 [Youtube] Add some comments to mix 2020-12-12 20:30:57 +01:00
Xiang Rong Lin d74265c846 [Youtube] Extract getThumbnailUrl into method and change getUploaderName 2020-12-12 20:30:57 +01:00
Xiang Rong Lin 0efb854d27 [Youtube] Implement mix extractor for auto-generated playlists.
-New YoutubeMixPlaylistExtractor, that extracts from a mix (auto-generated playlist).
-The url has the format of "youtube.com/watch?v=videoID&playlistID",
where playlistID always starts with "RD" and usually followed by the videoID.
-Change YoutubePlaylistLinkHandlerFactory to create a linkhandler with the given url if it is a mix.
-Change YoutubeService to return YoutubeMixPlaylistExtractor if the url is a mix.
2020-12-12 20:30:57 +01:00
vkay94 2ba27b39af Stream segments: Remove replaceAll from setPreviewUrl 2020-12-12 15:00:45 +01:00
Stypox 2b622fd2f1
Merge pull request #419 from Isira-Seneviratne/Use_Objects_methods
Use Objects methods.
2020-12-12 14:52:42 +01:00
vkay94 9a7a224a54 Add tests for stream segments 2020-12-12 10:24:55 +01:00
vkay94 d3eea4f4be Add stream segments to StreamInfo 2020-12-12 10:24:29 +01:00
Isira Seneviratne 57be1f1123 Use Objects.requireNonNull(). 2020-12-11 06:09:57 +05:30
Isira Seneviratne 4c19a88612 Use Objects.toString(). 2020-12-11 06:09:50 +05:30
Isira Seneviratne b8f64595a2 Use Objects' static equals() and hashCode() methods. 2020-12-11 06:09:44 +05:30
TobiGr b3835bd616 Release 0.20.6 2020-12-09 23:30:23 +01:00
Tobias Groza 6fbcdd24ee
Merge pull request #477 from vkay94/fix-comments-parsing
Fix comments parsing
2020-12-09 23:25:51 +01:00
vkay94 9dbacbc618 Fix comments parsing 2020-12-09 14:57:38 +01:00
Fynn Godau 04dd3d4d32 Rework link handlers to correctly accept external websites 2020-12-05 15:08:26 +01:00
Tobias Groza 8ade913d9b
Merge pull request #472 from TeamNewPipe/fix-media-ccc-de
Fix name of media.ccc.de service
2020-12-02 22:16:52 +01:00
TheAssassin ba3e2302bc Fix name of media.ccc.de service 2020-12-01 23:54:27 +01:00
TobiGr 175df679e0 Release 0.20.5 2020-12-01 08:18:29 +01:00
Tobias Groza b7a995187f
Merge pull request #463 from opusforlife2/update_invidious_URLs
Update Invidious URL List
2020-11-28 13:30:32 +01:00
Fynn Godau be562b8436 Change tests 2020-11-27 16:48:40 +01:00
TobiGr c91e21b37c [Bandcamp] Tests - Add finals and improve code formatting 2020-11-24 14:13:31 +01:00
TobiGr 99e7ef013e [Bandcamp] Apply small changes to code formatting and style
Make variables final when possible
Remove unused imports
Improve code formatting
2020-11-24 14:13:31 +01:00
fynngodau 8c369b0f79
Rephrase link in javadoc
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-11-24 14:01:31 +01:00
TobiGr 5a3f96b967 Release 0.20.4 2020-11-22 22:29:08 +01:00
Tobias Groza 650f0920fe
Merge pull request #465 from XiangRongLin/playlist_continuation
[YouTube] Fix playlist continuations extraction
2020-11-20 20:48:20 +01:00
TobiGr 334e1e9b53 Fix YouTube comments test 2020-11-20 19:53:53 +01:00
Fynn Godau 186936d041 Various changes regarding tests 2020-11-19 22:33:52 +01:00
Fynn Godau 6bc7e3420e Merge TNP/dev into fynngodau/dev 2020-11-19 21:32:08 +01:00
TobiGr 3582a5189f Release 0.20.3 2020-11-18 20:41:25 +01:00
XiangRongLin 5bceff0083 [YouTube] Fix extraction of next page url for the last page of playlist 2020-11-18 19:03:12 +01:00
XiangRongLin 8347e14952 [YouTube] Fix playlist continuations extraction 2020-11-17 21:04:53 +01:00
opusforlife2 2174055ef8
Update Invidious URL list in Parsing Helper 2020-11-11 15:12:31 +00:00
opusforlife2 c0afd5213a
Update Invidious URL list in Link Handler Factory 2020-11-11 14:54:16 +00:00
Tobias Groza 6701b0fe71
Merge pull request #447 from Stypox/better-deobfuscation
[YouTube] Cache deobfuscation code and more fixes
2020-11-08 00:32:36 +01:00
Stypox 89a77ae74a
[YouTube] Fix detection of ended livestreams and parse livestream upload date 2020-11-04 16:03:08 +01:00
Stypox 827f7bd137
[YouTube] Cache deobfuscation and improve requests made
Fix age restriction extraction
Automatically fixes more things
2020-11-04 16:02:12 +01:00
Stypox b42a196f35
Merge pull request #454 from Isira-Seneviratne/Fix_date_time_issues
Fix some issues related to the date/time changes.
2020-11-04 14:48:20 +01:00
Isira Seneviratne 4fe28d7e3a Fix YouTube parse error when only a date is present. 2020-11-04 05:49:42 +05:30
Isira Seneviratne 9cf9e7e980 Call existing constructor in DateWrapper. 2020-11-04 05:49:41 +05:30
Isira Seneviratne fe31a90cb3 Remove DateTimeFormatter.ISO_OFFSET_DATE_TIME usage. 2020-11-04 05:49:41 +05:30
Tobias Groza b13c7e1c1e
Merge pull request #452 from Stypox/yt-import
Implement YouTube subscription import from Google takeout
2020-11-03 20:32:17 +01:00
Tobias Groza ac50068bbe
Merge pull request #456 from B0pol/youtube_comments_test
create YouTubeCommentsLinkHandlerFactoryTest
2020-11-03 20:14:14 +01:00
bopol 345e136f6c create YouTubeCommentsLinkHandlerFactoryTest
and remove invidious test from YouTubeCommentsExtractorTest, because it was just testing if the URL is accepted, then the extractor does the same thing, we don't need to test the same thing twice
2020-11-03 19:10:10 +01:00
Stypox 501ec30152
Implement youtube subscription import from Google takeout 2020-11-02 11:04:48 +01:00
Tobias Groza 564a965810
Merge pull request #453 from bd0n4lds/fix-donation-typo
Fix typo in DonationLinkHelper and rewrote swtich statement
2020-11-02 00:15:04 +01:00
Bri@n 82746d172f Fix typo in DonationLinkHelper and rewrote swtich statement 2020-11-01 17:34:34 -05:00
Stypox 2f02c0e6a4
Merge pull request #450 from B0pol/fileutils
create FileUtils
2020-11-01 18:13:57 +01:00
bopol f69b3ef05d create FileUtils 2020-11-01 17:15:34 +01:00
TobiGr be9e160333 Fix build 2020-11-01 17:04:02 +01:00
Tobias Groza 8cbdec675b
Merge pull request #417 from Isira-Seneviratne/Use_Java_8_Date_Time_API
Use the Java 8 Date/Time API.
2020-11-01 16:49:06 +01:00
Isira Seneviratne fcdfe7d939 Update README.md. 2020-11-01 06:28:54 +05:30
Isira Seneviratne 4f04cfccca Switch from Calendar to OffsetDateTime in DateWrapper. 2020-11-01 06:28:54 +05:30
Isira Seneviratne ee3af63c04 Switch to ChronoUnit. 2020-11-01 06:28:53 +05:30
Isira Seneviratne 0526a5148d Set source and target versions to Java 8. 2020-11-01 06:28:52 +05:30
Isira Seneviratne b2d0c098a3 Merge branch 'dev' of https://github.com/TeamNewPipe/NewPipeExtractor into dev 2020-11-01 06:27:59 +05:30
Stypox 6cc50b57e3
Merge pull request #410 from Scrxtchy/sets-track-title
Fix parsing Soundcloud tracks that contain the term 'sets'
2020-10-31 21:43:21 +01:00
Stypox b242e1d113
Merge pull request #362 from Stypox/google-search-urls
Add support for Google search redirect url
2020-10-31 21:03:48 +01:00
Stypox 9e53cf0b56
Fix parameter reassignment and other style issues
Also remove left-behind debug statement
2020-10-31 20:58:28 +01:00
Stypox 3fe55b30ba
Add support for Google search redirect url 2020-10-31 20:58:28 +01:00
Stypox c190a3029b
Consider protocol as base url when it is a custom one (e.g. vnd.youtube) 2020-10-31 20:58:28 +01:00
Isira Seneviratne 35c7b5640c Merge branch 'dev' of https://github.com/TeamNewPipe/NewPipeExtractor into dev 2020-10-31 05:45:02 +05:30
Stypox a9303b24a5
Merge pull request #445 from B0pol/peertube_fix
[PeerTube] fix account and channel extractors
2020-10-30 19:35:51 +01:00
bopol b21e59925d [PeerTube] fix account and channel extractors 2020-10-29 19:52:29 +01:00
Isira Seneviratne 61dcbbb800 Merge branch 'dev' of https://github.com/TeamNewPipe/NewPipeExtractor into dev
 Conflicts:
	extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java
2020-10-29 12:20:16 +05:30
Stypox bcc01b99c7
Merge pull request #444 from Stypox/error-message
Remove any reference to decrypt and improve error message
2020-10-28 23:41:04 +01:00
Stypox 30ed4f2d63
Remove any reference to decrypt and improve error message 2020-10-28 20:26:00 +01:00
TobiGr 62912ee834 Merge branch 'master' into dev 2020-10-28 14:28:06 +01:00
TobiGr 94176c5d68 Merge branch 'master' into dev 2020-10-28 07:52:50 +01:00
Tobias Groza 19c0e8700d
0.20.2 2020-10-28 07:29:42 +01:00
Tobias Groza 354426fe5d
Merge pull request #441 from B0pol/fix-three-attempts
Support new YouTube JSON scheme
2020-10-28 07:24:31 +01:00
Scratch 947ce3ee10 Fix parsing Soundcloud tracks that contain the term 'sets' 2020-10-28 07:55:02 +11:00
bopol 6dc5ab4015 find playerUrl in another place when assetsPattern regex fails 2020-10-27 13:48:58 +01:00
bopol db0ef83d6b fix youtube decryption and three attemps bug
fixes teamnewpipe/newpipe#4572 fixes #439
2020-10-27 13:40:24 +01:00
Stypox 943334ba49
Merge pull request #440 from B0pol/fix-redirect-channels
Fix redirect channels
2020-10-26 20:00:06 +01:00
bopol c1e9857960 fix subscriber count when subscribe is disabled
fixes #305
2020-10-26 19:06:01 +01:00
bopol a39a2cca82 fix redirect channels 2020-10-26 19:05:19 +01:00
Tobias Groza a8665fd97f
Merge pull request #442 from B0pol/polish
Polish
2020-10-26 18:51:47 +01:00
bopol 0a12300c5e polish tests 2020-10-26 16:57:37 +01:00
bopol 01f49e8f66 polish strings 2020-10-26 16:32:39 +01:00
Tobias Groza 14c0c37d64
Merge pull request #309 from Stypox/improve-stream-tests
Improve stream extractor tests & various fixes
2020-10-25 18:49:53 +01:00
Stypox df26f61cfc
Merge pull request #394 from kirisakow/dev
Fix TeamNewPipe/NewPipe#4213: [YouTube] Ignore leading characters in video id
2020-10-25 14:39:03 +01:00
Kiril Isakov 366ed71523
[YouTube] Ignore leading characters in video id 2020-10-25 14:34:20 +01:00
Stypox 8026304a0a
Merge pull request #409 from Bartoshr/bugfix/short-channel-urls
Support short custom YouTube channel URLs
2020-10-25 11:43:50 +01:00
Stypox 57e7994c9e
Add some missing finals, nullables and comments 2020-10-25 08:12:41 +01:00
Stypox f11fe87688
[YouTube] Replace outdated PewDiePie video test with another one
The old video was made private, and this video (wedding) is probably never going to be removed.
2020-10-24 18:41:41 +02:00
Stypox 3c55ea9321
[PeerTube] Change age restricted video in tests
The old one wasn't available anymore
2020-10-24 18:41:41 +02:00
Stypox 880ff04a5c
[Test] Add stream dash mpd url test 2020-10-24 18:41:41 +02:00
Stypox d0b14644bb
[YouTube/MediaCCC] Consider dates as GMT and not as local 2020-10-24 18:41:41 +02:00
Stypox 8ce711f40f
[Test] Add sub channel name, url and thumbnail tests 2020-10-24 18:41:41 +02:00
Stypox 6127826571
[Test] Add stream metadata tests 2020-10-24 18:41:41 +02:00
Stypox a087b092b4
[Test] Improve code style and add final 2020-10-24 18:39:55 +02:00
Stypox a4097d8d01
[MediaCCC] Return empty list of video-only streams instead of null 2020-10-24 18:39:55 +02:00
Stypox 8dc3f28618
[PeerTube] Test one channel url with api and one without 2020-10-24 18:39:55 +02:00
Stypox 68d23defba
[YouTube] Do not catch every exception on getErrorMessage
@B0pol suggestion
2020-10-24 18:39:55 +02:00
Stypox 3191bd6c70
[YouTube] Use final when possible 2020-10-24 18:39:55 +02:00
Stypox 55bc01d1ce
[SoundCloud] Use final when possible, ide refactorings 2020-10-24 18:39:55 +02:00
Stypox 06430c4749
[PeerTube] Use final when possible, ide refactorings 2020-10-24 18:39:06 +02:00
Stypox fcb9b6f855
[MediaCCC] Use final when possible, ide refactorings
Refactorings on `throws` clause
2020-10-24 18:36:22 +02:00
Stypox af5b8b1915
[MediaCCC] Add tests for stream and conference link handlers 2020-10-24 18:36:21 +02:00
Stypox 07a90d116a
[MediaCCC] Use regex to parse stream and conference urls 2020-10-24 18:36:21 +02:00
Stypox 0c980b2d64
[PeerTube] Improve channel and stream link handler tests 2020-10-24 18:36:21 +02:00
Stypox d130fd79c3
[PeerTube] Prepend "accounts/" to channel id for backward compatibility 2020-10-24 18:36:21 +02:00
Stypox 492db83ccf
[MediaCCC] Return null instead of empty items collector
As per the documentation in the base getRelatedStreams()
2020-10-24 18:34:57 +02:00
Stypox 3b2a1829d4
[MediaCCC] Extract tags 2020-10-24 18:34:57 +02:00
Stypox b461da792f
[MediaCCC] Fix link handler inconsistency providing API links 2020-10-24 18:34:34 +02:00
Stypox aeeae87641
[PeerTube] Parse timestamp from url (previously unimplemented) 2020-10-24 18:33:42 +02:00
Stypox 7ae3cb6d07
[PeerTube] Fix link handler inconsistency providing API links 2020-10-24 18:33:11 +02:00
Stypox 4349be13af
[PeerTube] Return empty audio stream list instead of null 2020-10-24 18:25:56 +02:00
Stypox 3b2cfb4ca2
[SoundCloud] Return empty video stream list instead of null
Also replace every instance of `return new ArrayList<>();` with `return Collections.emptyList();`
2020-10-24 18:23:46 +02:00
Stypox 072bae321f
[YouTube] Fix frame extraction for livestreams
Use saved playerResponse instead of parsing json every time
2020-10-24 18:23:46 +02:00
Stypox 7cd410f3fc
[YouTube] Return 0 when there is no timestamp, not -2, as per javadoc 2020-10-24 18:23:45 +02:00
Stypox 7fb867c166
[YouTube] Fix error message obtaining when there is none 2020-10-24 18:22:44 +02:00
Stypox 1d7a86e664
[Test] Add base classes for stream extractor tests
Refactor all stream extractor tests to use new base class.

Remove check if upload date is in the past: this does not have to hold true: youtube premieres turn up in search results even though they are in the future
2020-10-24 18:22:44 +02:00
Bartosz Rumiński 29695aed0a Small field refactor 2020-10-23 16:42:30 +02:00
Bartosz Rumiński 0e67d820bc Use static regex pattern for excluded path segments 2020-10-22 20:13:29 +02:00
Bartosz Rumiński d3f80d1538 Exlude links which are not channels 2020-10-20 20:06:06 +02:00
Tobias Groza 3058af0e43
Merge pull request #418 from Stypox/yt-shorts
[YouTube] Handle URLs for Shorts
2020-10-18 13:06:10 +02:00
Stypox 6887d59570
[YouTube] Handle urls for Shorts 2020-10-18 12:03:01 +02:00
Scratch e945f711c1
Fix SoundCloud test artist account name (#416) 2020-10-16 23:42:46 +02:00
TobiGr 350eed6214 Release v0.20.1 2020-10-16 22:03:57 +02:00
Tobias Groza f82eda9f7d
Merge pull request #415 from wb9688/fix-ytinitialdata
Fix parsing new ytInitialData
2020-10-16 21:59:39 +02:00
wb9688 be9a6f931c Fix parsing new ytInitialData 2020-10-16 20:27:40 +02:00
Tobias Groza 1a9bc8caf1
Merge pull request #414 from Scrxtchy/soundcloud-slash
Remove tailing slash from SoundCloud URLs
2020-10-16 09:48:44 +02:00
Scratch 6a70cb9d50 Remove tailing slash from SoundCloud URLs
Fixes #412
2020-10-16 17:17:40 +11:00
Tobias Groza 9a785f9a7e
Merge pull request #413 from Scrxtchy/soundcloud-test-update
Fix SoundCloud test artist account name
2020-10-16 08:05:44 +02:00
Scratch 538f5d3973 Fix SoundCloud test artist account name 2020-10-16 13:45:05 +11:00
Tobias Groza 527945eadb
Merge pull request #406 from Stypox/fix-decrypt
[YouTube] Fix some decryption exceptions by retrying
2020-10-15 22:03:25 +02:00
Stypox 19e862657a
[YouTube] Fix some decryption exceptions by retrying 2020-10-15 20:16:44 +02:00
Bartosz Rumiński e3f996e014 Exlude links which are not channels 2020-10-12 20:59:56 +02:00
Bartosz Rumiński 5ab1b053d2
Update youtube channel link handler tests
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-10-12 20:11:28 +02:00
Bartosz Rumiński 7abb4b3713
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/linkHandler/YoutubeChannelLinkHandlerFactory.java
Fix typos

Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-10-12 19:57:45 +02:00
Bartosz Rumiński 9d63211a66
Fix typos
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-10-12 19:56:53 +02:00
Bartosz Rumiński 8c38a5509e Prevent attribution_link urls to be accepted by channel links handler 2020-10-09 19:07:38 +02:00
Bartosz Rumiński 4e04991762 Support short custom youtube channel urls 2020-10-09 17:06:59 +02:00
Fynn Godau 81b5e7cf3d Fix extractor 2020-10-08 17:56:03 +02:00
TobiGr acb04eb351 Update extractor version to 0.20.0 2020-10-04 13:37:19 +02:00
Stypox 2463884aa8
Merge pull request #400 from wb9688/skip-otf
Do not list YouTube's OTF streams
2020-09-29 20:16:34 +02:00
wb9688 ebbfe7f6d4 Skip YouTube's OTF streams 2020-09-29 10:48:02 +02:00
Tobias Groza 6633f26ec5
Merge pull request #386 from mhmdanas/remove-subtitles-stream-get-url
Remove SubtitlesStream#getURL()
2020-08-26 21:32:36 +02:00
Tobias Groza 070cd92857
Merge pull request #387 from wb9688/init-safe-standard-objects
Use initSafeStandardsObjects()
2020-08-26 21:25:46 +02:00
wb9688 7657c2ed1a Use initSafeStandardsObjects() 2020-08-15 17:08:07 +02:00
mhmdanas 1a63dcb355 Remove overriden field `url` 2020-08-12 22:37:36 +03:00
Tobias Groza de8edbe7a5
Merge pull request #382 from TobiGr/fix/empty_comment
[YouTube] Fix crash on empty  comment
2020-08-12 14:18:16 +02:00
mhmdanas f0f1c009b2 Remove SubtitlesStream#getURL() 2020-08-10 19:28:56 +03:00
Fynn Godau cfe88a74c1 Throw ContentNotSupportedException when opening radio uploader channel 2020-08-09 10:07:49 +02:00
TobiGr 0fb73301e3 [YouTube] Fix crash on empty comment
Closes #380
2020-08-05 18:25:40 +02:00
Fynn Godau 932d094d6a Upgrade featured playlist urls to https 2020-08-02 16:55:18 +02:00
Fynn Godau c12ef3a02d Merge TNP/dev into fynngodau/dev 2020-08-02 16:43:04 +02:00
TobiGr 5ac80624a4 Fix detection of YouTube's decryption function
Closes TeamNewPipe/NewPipe#3951
2020-07-28 01:01:10 +02:00
TobiGr b4481dfec2 Update version to 0.19.7 2020-07-26 14:24:07 +02:00
Tobias Groza 32dff1541f
Merge pull request #370 from wb9688/fix-yt-age-restricted
Fix age restricted YouTube videos
2020-07-26 14:17:19 +02:00
TobiGr ec778200d9 Add comments to the code 2020-07-26 14:15:13 +02:00
Tobias Groza 8627d01006
Merge pull request #373 from wb9688/fix-yt-continuations
Support YouTube's new continuations
2020-07-26 13:56:33 +02:00
wb9688 667dce034c Only use fallback methods for clientVersion when clientVersion hasn't been successfully extracted yet 2020-07-26 13:14:25 +02:00
wb9688 f345f667e2 Extract YouTube's key 2020-07-26 12:01:59 +02:00
wb9688 4c987a5302 Support YouTube's new continuations for search 2020-07-26 10:01:03 +02:00
wb9688 81459e289f Fix age restricted YouTube videos 2020-07-18 19:52:54 +02:00
Tobias Groza df28a087de
Merge pull request #337 from wb9688/next-stream
Remove getNextStream() from StreamExtractor
2020-07-18 11:55:58 +02:00
wb9688 fc3a63fec5 Update Javadoc for getRelatedStreams() 2020-07-15 18:48:37 +02:00
wb9688 ff560e907a Remove getNextStream() from StreamExtractor 2020-07-15 18:48:37 +02:00
TheAssassin 186193d366 Update version to 0.19.6 2020-07-14 17:04:03 +02:00
Tobias Groza a70cb0283f
Merge pull request #314 from wb9688/remove-getnextpageurl
Next page stuff
2020-07-07 20:49:38 +02:00
wb9688 0a5a905bc7 Add final at more places 2020-07-07 20:45:47 +02:00
Tobias Groza e5d23a89de
Merge pull request #366 from wb9688/yt-timestamp-start
Support start= timestamps for YouTube
2020-07-06 21:28:19 +02:00
wb9688 17ba8a57fa Clean up the code 2020-07-06 20:19:31 +02:00
wb9688 9b6fe1dea6 Throw IllegalArgumentException when Page is invalid 2020-07-06 20:19:31 +02:00
wb9688 4cc312086a Introduce Page class 2020-07-06 20:19:31 +02:00
wb9688 e3bfdba135 Remove getNextPageUrl() function from ListExtractor 2020-07-06 20:11:40 +02:00
wb9688 156a26b64b Support start= timestamps for YouTube 2020-07-06 14:08:40 +02:00
Tobias Groza 54d9e5a2f8
Merge pull request #365 from B0pol/update_inv_instances
Update inv instances
2020-07-02 23:13:07 +02:00
bopol 261471e7f4 remove invidiou.sh from tests
dead instance
2020-07-02 21:33:45 +02:00
bopol eb3901acee update invidious instances 2020-07-02 21:31:05 +02:00
Tobias Groza c0ceb5cb27
Merge pull request #340 from wb9688/peertube-deleted-comments
Handle isDeleted for PeerTube comments
2020-06-28 10:21:37 +02:00
Tobias Groza f4ed7ce922
Merge pull request #361 from yausername/gradleScope
expose extractor via api scope
2020-06-27 21:45:04 +02:00
Tobias Groza 339f1d9c43
Merge pull request #349 from wb9688/okhttp
Use OkHttp for tests like in NewPipe
2020-06-27 20:08:02 +02:00
Tobias Groza 970bc7f69d
Merge pull request #359 from mauriciocolli/fix-client-id-bug
[SoundCloud] Fix concurrency issue when getting the client id
2020-06-27 17:36:24 +02:00
Tobias Groza dbf6ab9e91
Merge pull request #355 from Redirion/patch-1
Added a comment
2020-06-27 11:47:50 +02:00
Robin 576754982e
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-06-27 11:35:01 +02:00
yausername 89e0e3ee2d
expose extractor via api scope 2020-06-27 03:02:43 +05:30
Mauricio Colli 7505e828cc
[SoundCloud] Update client id 2020-06-24 01:50:09 -03:00
Mauricio Colli d2f1c0f40d
[SoundCloud] Fix concurrency issue when getting the client id 2020-06-24 01:50:08 -03:00
Robin b63ae93495
Added a comment
and fixed a "typo" in the method parseDurationString replacing Long with Integer.

These are not the droids you are looking for wb9688 😁
2020-06-15 11:27:44 +02:00
Tobias Groza 92f6754f0f
Merge pull request #338 from wb9688/parse-duration-string-test
Add tests for parseDurationString()
2020-06-15 11:01:21 +02:00
wb9688 e02d3c37b2
Merge pull request #354 from Stypox/peertube-age-restricted
[PeerTube] Change age restricted video in tests
2020-06-14 20:09:36 +02:00
Stypox 5808439c3f
[PeerTube] Change age restricted video in tests
The old one wasn't available anymore
2020-06-14 19:57:39 +02:00
wb9688 3239aa84f2 Handle isDeleted for PeerTube comments 2020-06-14 19:27:20 +02:00
wb9688 a65e46e4b1 Use OkHttp for tests like in NewPipe 2020-06-11 15:06:08 +02:00
Tobias Groza 5f523254cb
Merge pull request #344 from B0pol/peertube_embeds
For #342: support PeerTube embeds
2020-06-06 18:53:07 +02:00
Fynn Godau 96de834b67 Bandcamp: parse date from radio info item 2020-06-04 19:36:58 +02:00
Fynn Godau 39b55b5b42 Bandcamp radio stream extractor: test uploader url 2020-06-04 19:32:19 +02:00
Fynn Godau 9a555d97e5 Remove useless bandcamp tests 2020-06-04 19:31:01 +02:00
Fynn Godau e13f341a45 Bandcamp stream extractor: test service id 2020-06-04 19:29:37 +02:00
Fynn Godau 4dd9540782 Bandcamp license switch order 2020-06-04 19:27:39 +02:00
Fynn Godau 52103ac61f Remove "useless" comments 2020-06-04 19:27:13 +02:00
Fynn Godau 8fa81537c0 Better bandcamp stream extractor description test 2020-06-04 19:17:51 +02:00
Fynn Godau 0950a95577 Fix for testing wrong object for null or empty 2020-06-04 19:16:08 +02:00
Fynn Godau e6ecd91929 Bandcamp: Test full date 2020-06-04 19:12:10 +02:00
Fynn Godau 27e7aad159 Bandcamp: Improve code style 2020-06-04 19:04:25 +02:00
fynngodau ea49202f64
Don't display internal license id
Co-authored-by: bopol <bopol@e.email>
2020-06-04 18:26:58 +02:00
bopol 063f4be766 For #342: support PeerTube embeds 2020-06-04 14:26:42 +02:00
Fynn Godau 34b6928124 Don't print stacktrace before throwing new exception 2020-06-04 14:25:38 +02:00
Fynn Godau 1be20ceeec Bandcamp: parse date 2020-06-04 14:09:21 +02:00
Fynn Godau 6822fe3dc8 Bandcamp: Get Stream upload date 2020-06-04 13:03:16 +02:00
Fynn Godau e98bff8ae9 Improved Bandcamp radio stream extractor tests 2020-06-04 12:30:53 +02:00
Fynn Godau 8f6c00f8d8 Use default test in BandcampSearchExtractorTest 2020-06-04 12:12:20 +02:00
Fynn Godau de776561b0 Bandcamp: fix loading uploader from streams in search 2020-06-04 12:02:18 +02:00
Fynn Godau 3940138fc5 Fix Bandcamp capitalization in one more spot 2020-06-03 22:41:27 +02:00
Fynn Godau 9fa9d920a9 Bandcamp: Implement new methods required due to interface changes 2020-06-03 21:35:11 +02:00
Fynn Godau dd955c7f0c Merge branch 'origin/dev' into dev 2020-06-03 21:13:40 +02:00
wb9688 33b1121fc7 Add tests for parseDurationString() 2020-05-30 17:20:54 +02:00
TobiGr 3aa6b93e91 Update version to 0.19.5 2020-05-30 10:45:39 +02:00
Tobias Groza bda83fe6a5
Merge pull request #336 from wb9688/fix-time-parsing
Fix YouTube video duration parsing
2020-05-30 10:36:46 +02:00
wb9688 e39147202c Fix YouTube video duration parsing 2020-05-30 09:31:08 +02:00
wb9688 98055a3c3c Revert "Add more decryption function regexes"
This reverts commit 384d6acd5d.
2020-05-29 18:41:40 +02:00
Tobias Groza bdb0f2ae6b
Merge pull request #332 from wb9688/learning-playlist
Support YouTube's learning playlists
2020-05-29 10:11:08 +02:00
Tobias Groza cf18cdb2f5
Merge pull request #329 from wb9688/decryption-regexes
Add more decryption function regexes
2020-05-29 10:10:50 +02:00
wb9688 ab77961b0a Extract trailer thumbnail in YT learning playlists 2020-05-29 09:55:24 +02:00
wb9688 4d683e7655 Support YouTube's learning playlists 2020-05-29 09:17:03 +02:00
Tobias Groza 0b4977bb0c
Merge pull request #333 from wb9688/playerresponse-dashmanifesturl
Extract dashManifestUrl in playerResponse
2020-05-28 21:17:07 +02:00
Tobias Groza 0f8057ae56
Merge pull request #322 from wb9688/upgrade-jsoup
Upgrade dependencies
2020-05-28 21:16:33 +02:00
wb9688 384d6acd5d Add more decryption function regexes 2020-05-28 11:40:38 +02:00
wb9688 8f71bde776 Fix bug for YT videos longer than 1000h 2020-05-28 11:34:42 +02:00
wb9688 aa277e1b6a Upgrade other dependencies 2020-05-28 11:34:42 +02:00
wb9688 036cc7e26e Upgrade jsoup 2020-05-28 11:34:41 +02:00
Fynn Godau 89b0639faa New Bandcamp tests for channel extractor 2020-05-25 20:22:23 +02:00
Fynn Godau e08256ebc6 Remove "url cleanup" in Bandcamp link handlers 2020-05-25 19:17:27 +02:00
Fynn Godau 692b2d06f4 Bandcamp: read info item data in place and not in advance 2020-05-25 18:53:15 +02:00
Fynn Godau fff21caa87 Fix Bandcamp no avatar test
I own the bandcamp artist account called "NewPipeExtractorTest", so it
shouldn't break again.
2020-05-25 18:53:10 +02:00
wb9688 7ef3e80493 Extract dashManifestUrl in playerResponse 2020-05-15 12:07:26 +02:00
Stypox 60e6df7e6b
Merge pull request #330 from B0pol/gitignore
add vscode / eclipse files to .gitignore
2020-05-13 10:39:33 +02:00
wb9688 efafbbb88a
Merge pull request #331 from wb9688/playlist-subchannel
Add sub-channel stuff to PlaylistInfo
2020-05-12 10:49:13 +02:00
wb9688 baf5dd3e8f
Merge pull request #311 from B0pol/fixSearchSuggestion
Search: add isCorrectedSearch() and fix YoutubeSearchExtractor#getSea…
2020-05-12 09:29:15 +02:00
wb9688 102f04e297 Add sub-channel stuff to PlaylistInfo 2020-05-12 09:14:29 +02:00
bopol cbf1163b80 ignore workspace and folder configs 2020-05-11 21:23:50 +02:00
Tobias Groza b40ccb5075
Merge pull request #307 from B0pol/playlists_peertube
[PeerTube] playlist support
2020-05-11 20:53:03 +02:00
bopol aeb2a6ff6f add vscode / eclipse files to .gitignore 2020-05-11 13:06:02 +02:00
B0pol 3a858e6640
Merge pull request #315 from B0pol/isnullorempty
refactor: add Utils.isNullOrEmpty()
2020-05-11 11:04:56 +00:00
bopol adaf196c99 remove unused import 2020-05-11 12:54:33 +02:00
bopol de26e00079 changed all == null || isEmpty() to isNullOrEmpty() 2020-05-11 11:40:24 +02:00
bopol 202a73516c refactor: add Utils.isNullOrEmpty() 2020-05-11 11:26:18 +02:00
Tobias Groza 3cae32b6db
Merge pull request #328 from wb9688/fix-327
Fix #327
2020-05-09 20:29:11 +02:00
wb9688 1de0876fb4 Fix #327 2020-05-09 20:06:50 +02:00
bopol 7cae95bf09 add tests for subchannel stuff in playlists 2020-05-09 10:15:35 +02:00
bopol 1a6e92ebf6 add getSubChannel Name, Url and AvatarUrl for playlists 2020-05-09 09:52:24 +02:00
bopol cf99107745 address yausername change 2020-05-09 00:52:04 +02:00
bopol 5bab9d9fc0 [PeerTube] playlist support & refactoring 2020-05-09 00:52:04 +02:00
Tobias Groza f3913e241e
Merge pull request #324 from Stypox/music-detector-url
[YouTube] Support stream urls in `vnd.youtube://ID` form
2020-05-08 17:43:04 +02:00
Tobias Groza 3ddd6e4d7b
Merge pull request #326 from wb9688/subchannel-parentchannel
Rename SubChannel in channels back to ParentChannel
2020-05-08 17:40:49 +02:00
wb9688 7320108c66 Rename SubChannel in channels back to ParentChannel 2020-05-08 16:02:59 +02:00
wb9688 44d382b4bd
Merge pull request #313 from Royosef/DisplayParentChannelDetails
Display parent channel details
2020-05-08 15:52:32 +02:00
Roy Yosef a408661771 Fix PR review
Rename "ParentChannel" to "SubChannel"
Add Tests for sub-channel info fetching
2020-05-07 20:20:12 +02:00
Roy Yosef 1de1f97cf1 Add parent channel info to StreamExtractor: name, url & avatar url 2020-05-07 20:20:12 +02:00
Roy Yosef 4234740baa Add parent channel info to ChannelExtractor: name, url & avatar url 2020-05-07 20:20:12 +02:00
bopol 6cff5de400 renaming methods for consistency 2020-05-07 15:40:41 +02:00
bopol 5760366d70 use JsonObject#isEmpty instead of JsonObject#equals(JsonUtils.EMPTY_OBJECT) 2020-05-07 15:40:41 +02:00
bopol 93e868634e clarified javadoc for isCorrectedSearch and getSearchSuggestions in SearchExtractor 2020-05-07 15:40:41 +02:00
bopol 3757541a9c adapt new nanojson version 2020-05-07 15:40:41 +02:00
bopol 79483a6dc0 «Use this in SearchInfo, for consistency's sake» @Stypox order xD 2020-05-07 15:40:41 +02:00
bopol e8d58e09c9 getSearchSuggestion now returns the corrected query if isCorrectedSearch() is true 2020-05-07 15:40:41 +02:00
bopol 5d1c3b3fa2 Use setters in SearchInfo 2020-05-07 15:39:57 +02:00
bopol 29b639b454 Search: add isCorrectedSearch() and fix YoutubeSearchExtractor#getSearchSuggestion()
isCorrectedSearch: see the javadoc
getSearchSuggestion: in YoutubeSearchExtractor, it was giving the corrected search query. It now gives the suggested query, as it should
2020-05-07 15:39:57 +02:00
wb9688 665c69b530
Merge pull request #310 from B0pol/soundcloudComments
Add support for soundcloud comments
2020-05-07 15:35:57 +02:00
Stypox a02bfbeae5
[YouTube] Support stream urls in `vnd.youtube://ID` form 2020-05-07 15:31:31 +02:00
bopol bc13e0c616 Merge remote-tracking branch 'upstream/dev' into soundcloudComments 2020-05-05 14:49:21 +02:00
TobiGr 4086715a68 Merge branch 'dev' 2020-05-03 21:21:30 +02:00
TobiGr 8a79b14077 Update extractor version to 0.19.4 2020-05-03 15:11:20 +02:00
Tobias Groza fc3a69ed54
Merge pull request #323 from wb9688/fix-yt-music-albums
Fix NPE in defaultAlertsCheck()
2020-05-03 14:08:47 +02:00
wb9688 6aae99f832 Fix NPE in defaultAlertsCheck() 2020-05-03 10:28:45 +02:00
Tobias Groza 2780e716b3
Merge pull request #321 from wb9688/fix-nanojson
Return null instead of "" in getTextFromObject()
2020-05-02 15:56:28 +02:00
wb9688 a1eabc7589 Return null instead of "" in getTextFromObject() 2020-05-02 08:02:25 +02:00
Fynn Godau e7b046db9d Merge remote-tracking branch 'origin/dev' into dev 2020-04-26 23:12:43 +02:00
Fynn Godau a3e8e1c70e Bandcamp: move own methods to helper class 2020-04-26 23:05:56 +02:00
Fynn Godau 9bc7a470ab Replace fori with foreach loop 2020-04-26 23:02:14 +02:00
Tobias Groza 49157fcf0d
Merge pull request #317 from wb9688/tnp-nanojson
Use our fork of nanojson
2020-04-24 21:54:35 +02:00
bopol 2564bcf399 refactor comments
rename methods in CommentsInfoItemExtractor interface to match the other method names across NewPipeExtractor
remove getName in (Youtube|SoundCloud|Peertube)CommentsExtractor and move it up in CommentsExtractor, return "Comments" instead
2020-04-21 20:05:54 +02:00
Fynn Godau f71bd29d58 Bandcamp playlist extractor: add more tests 2020-04-20 23:52:12 +02:00
Fynn Godau 5009d9f53b Bandcamp tests: remove certain tests 2020-04-20 23:25:00 +02:00
Fynn Godau b5e251c82f Bandcamp channel extractor: package-private access for getImageUrl(…) 2020-04-20 23:12:43 +02:00
Fynn Godau 8c70dab8c5 Bandcamp search query handler factory: inline variable 2020-04-20 23:11:16 +02:00
Fynn Godau 10ae3db118 Bandcamp javadoc: Replace br tags with p tags 2020-04-20 23:03:12 +02:00
Fynn Godau 9201e0d3ea Don't declare exceptions that can't actually be thrown 2020-04-20 22:46:54 +02:00
Fynn Godau 965bce00cf Bandcamp stream extractor: return NO_AGE_LIMIT 2020-04-20 22:44:45 +02:00
Fynn Godau 82099592c7 Bandcamp: Move code from SearchExtractor to InfoItemExtractors 2020-04-20 22:44:35 +02:00
Fynn Godau 00c0333059 Bandcamp: Fetch channelInfo in onFetchPage 2020-04-20 22:41:44 +02:00
Fynn Godau 433d72c26a Depend on nanojson fork in timeago-parser 2020-04-20 22:04:10 +02:00
Fynn Godau 67de0285e1 Bandcamp: Generate query JSON in a proper way 2020-04-20 22:03:29 +02:00
Fynn Godau c133190c53 Bandcamp capitalization 2020-04-20 21:55:35 +02:00
Fynn Godau b100b9873f Bandcmap: Use nanojson fork instead of org.json 2020-04-20 21:53:04 +02:00
wb9688 b51699a20e Implement @TobiGr's suggestions 2020-04-20 14:27:33 +02:00
wb9688 74f0ab1792
Merge pull request #301 from wb9688/media-ccc-non-api-urls
Use media.ccc.de instead of api.media.ccc.de
2020-04-19 17:06:57 +02:00
Fynn Godau 9c239371f4 Change nanojson dependency to @wb9688's fork 2020-04-19 14:29:39 +02:00
Fynn Godau 5c0a03328a Merge 'origin/dev' into dev 2020-04-19 14:26:51 +02:00
bopol 0cbbc2a1f9 remove unused import 2020-04-19 14:16:38 +02:00
wb9688 979c5a7502 Use our fork of nanojson 2020-04-17 19:36:42 +02:00
bopol d4352f9b84 support comments for SoundCloud 2020-04-10 20:41:41 +02:00
wb9688 1b522304c1
Merge pull request #308 from Royosef/PeerTubeChannelExtractor
[PeerTube] channels support
2020-04-10 19:07:51 +02:00
Roy Yosef 4afe657f6f Add tests for getUrlFromId and test for video-channels in acceptUrlTest 2020-04-10 19:05:38 +03:00
Roy Yosef 2c9f1260eb Fix wb9688 review comments
* Rename PeertubeUserExtractor to PeertubeAccountExtractor
* Add test for video-channels in PeertubeChannelLinkHandlerFactoryTest
* Compatibility support for older versions (use "accounts/" as default)
2020-04-10 16:40:53 +03:00
bopol 636c430743 refactor: create extractors and linkHandler packages for SoundCloud, move YoutubeParsingHelper to youtube package (not linkhandler anymore) 2020-04-10 10:51:05 +02:00
Roy Yosef b6e6f403a8 add support for PeerTube channels extraction 2020-04-09 20:37:49 +03:00
wb9688 a5155fb562
Merge pull request #300 from B0pol/hls_opus_not_supported
throw ContentNotSupportedException when content is known to be unsuppo…
2020-04-08 19:54:16 +02:00
bopol a20d53dd98 remove the mention of Opus on wb9688 advise when
ContentNotSupportedException is thrown
2020-04-08 15:31:11 +02:00
bopol 9a7c6b7ab0 use ContentNotSupportedException for channels without tabs 2020-04-07 14:33:43 +02:00
bopol a22104cbda add tests for ContentNotSupportedException 2020-04-07 14:33:43 +02:00
bopol 093762e793 throw ContentNotSupportedException when content is know to be unsupported 2020-04-07 14:33:43 +02:00
Tobias Groza 094b87c537
Merge pull request #299 from B0pol/sc_countries
add supported countries for soundcloud
2020-04-03 19:12:33 +02:00
Tobias Groza a7d3d14910
Add link to JDoc to readme 2020-04-03 18:14:14 +02:00
Tobias Groza e3263666bd
Merge pull request #303 from opusforlife2/correct-template
Corrected the pull request template
2020-04-03 17:56:02 +02:00
Tobias Groza b81e22ddc9
Merge pull request #291 from wb9688/yt-music-search
Add support for YouTube Music search
2020-04-03 17:55:34 +02:00
wb9688 bce27a0e22 Rename getValidResponseBody() to getValidJsonResponseBody() 2020-04-03 17:23:18 +02:00
opusforlife2 4a1d8c804c
Corrected the pull request template
It was a bit weird before.
2020-04-02 10:42:54 +00:00
bopol 1eb3deb7fd add supported countries for soundcloud 2020-04-02 11:26:19 +02:00
Tobias Groza cef7c96d7d
Merge pull request #302 from wb9688/readme-local-build
Add another way to test changes locally to README.md
2020-04-01 19:23:04 +02:00
wb9688 8a9e137385 Extract some code to getValidResponseBody() 2020-04-01 16:01:21 +02:00
wb9688 2d0c301440 Add another way to test changes locally to README.md 2020-04-01 15:43:11 +02:00
wb9688 bf24caa5f2 Use media.ccc.de instead of api.media.ccc.de 2020-04-01 13:25:10 +02:00
wb9688 2af610e3e7 Add tests for other YT Music search types 2020-04-01 10:30:51 +02:00
wb9688 c7f7bd2442 Fix error when YT Music videos has no uploader URL 2020-04-01 10:30:51 +02:00
wb9688 ac15df4548 Move YouTube Music search to its own class 2020-04-01 10:30:51 +02:00
wb9688 cf0f2aff3e Extract uploader url from certain YouTube Music videos 2020-04-01 10:30:51 +02:00
wb9688 aa8cea47f3 Refactor YouTube Music search tests 2020-04-01 10:30:51 +02:00
wb9688 dd434cca01 Fix issue when there is no didYouMeanRenderer in itemSectionRenderer 2020-04-01 10:30:51 +02:00
wb9688 5a775a4bbe Use new way of specifying stream count 2020-04-01 10:30:51 +02:00
wb9688 d58c0f230d Improve code for YouTube Music search 2020-04-01 10:30:51 +02:00
wb9688 dc29d87962 Extract YouTube search suggestions 2020-04-01 10:30:51 +02:00
wb9688 c852b13d5a Add Referer header so that it also works with HttpsUrlConnection 2020-04-01 10:30:51 +02:00
wb9688 eb48524411 Add tests for YouTube Music search 2020-04-01 10:30:51 +02:00
wb9688 2b9b2a78e8 Handle 100+ items in playlist 2020-04-01 10:30:51 +02:00
wb9688 fb9b9691b7 Improve getYoutubeMusicKeys() 2020-04-01 10:30:51 +02:00
wb9688 4ddbdf0aee Disable artist search for now 2020-04-01 10:30:51 +02:00
wb9688 1762a527c9 Add support for YouTube Music search 2020-04-01 10:30:51 +02:00
Tobias Groza 04af575764
Hey Travis, deploy the docs 2020-03-30 00:10:54 +02:00
TobiGr c8abc1abd5 v0.19.0 2020-03-29 23:47:26 +02:00
Tobias Groza 69e0624e35
Merge pull request #298 from B0pol/feed_peertube
[peertube] add feed url in channels
2020-03-28 12:53:47 +01:00
bopol 0ae53c1e50 [peertube] adapt test for channel feed url 2020-03-28 11:48:24 +01:00
bopol bc43f8977d [peertube] add feed url in channels 2020-03-28 10:22:47 +01:00
TobiGr 647e7cd450 Accept YouTube Music playlists but not YouTube Mixes 2020-03-25 22:24:18 +01:00
Tobias Groza bff595c6c6
Merge pull request #276 from wb9688/no-yt-mix
Don't accept YouTube Mix playlists
2020-03-25 11:18:07 +01:00
wb9688 bcd2a1092b Don't accept YouTube Mix playlists 2020-03-25 11:07:28 +01:00
Tobias Groza e4874d554e
Merge pull request #296 from mauriciocolli/fix-search-errors-detection
Fix search errors detection and refactor search tests
2020-03-24 21:53:32 +01:00
Tobias Groza 8a9ae32e6b
Merge pull request #292 from wb9688/fix-yt-no-views
Fix YouTube videos with no views
2020-03-24 20:27:08 +01:00
Mauricio Colli 0a20c53f1a
Use easier-to-read asserts statements in tests 2020-03-23 18:08:59 -03:00
Mauricio Colli c921e5d6a5
Remove unnecessary collector creation function from search extractor 2020-03-23 18:08:51 -03:00
Tobias Groza 6fd9b38ad9
Merge pull request #287 from mauriciocolli/fix-channel-redirect
[YouTube] Fix channel with redirects directly in the response
2020-03-23 18:41:56 +01:00
Mauricio Colli 04bfa0ec6a
[PeerTube] Update video used in age limit test (was 404 not found) 2020-03-21 15:55:40 -03:00
Mauricio Colli 921bf30bb7
[YouTube] Add check for channel items with no video count in search 2020-03-21 15:55:39 -03:00
Mauricio Colli b7f8001a49
[YouTube] Add check for channel items without description in search 2020-03-21 15:55:38 -03:00
Mauricio Colli 9b7999fe54
[YouTube] Check if channel item has subscription count in search 2020-03-21 15:55:37 -03:00
Mauricio Colli 9704fc9952
Improve search extractor tests for services 2020-03-21 15:55:36 -03:00
Mauricio Colli d72130edae
Always return a new item collector in the search extractor 2020-03-21 15:55:34 -03:00
Mauricio Colli 0c27198ba1
Include errors from child collectors in the search collector 2020-03-21 15:55:29 -03:00
TobiGr 2cf8cbf0df Merge branch 'master' into dev 2020-03-18 22:29:10 +01:00
Tobias Groza 6f03c6e877
Merge pull request #293 from Stypox/fix-soundcloud
[SoundCloud] Fix by migrating to `api-v2` and other improvements
2020-03-18 17:49:54 +01:00
wb9688 222d659d9e
[SoundCloud] Don't make separate request for getAudioStreams() in StreamExtractor
Signed-off-by: Stypox <stypox@pm.me>
2020-03-18 11:30:15 +01:00
wb9688 904c9d681f Fix YouTube videos with no views 2020-03-18 10:07:49 +01:00
Stypox 265cfb61f3
[SoundCloud] Add test for small playlist, use defaultTestGetPageInNewExtractor 2020-03-17 21:03:50 +01:00
Stypox 45bb646480
[SoundCloud] Do not overwrite nextPageUrl in PlaylistExtractor
Consistent with YouTube and the documentation
2020-03-17 20:56:47 +01:00
Fynn Godau a1523eb293 Update Bandcamp service to latest interface changes 2020-03-17 20:40:25 +01:00
Stypox c505d4e2b7
[SoundCloud] Remove trailing , in playlist page urls 2020-03-17 20:36:59 +01:00
Stypox ae47c9587c
[SoundCloud] Optimize imports in edited files 2020-03-17 20:32:04 +01:00
Stypox 4389fd3b7b
[SoundCloud] Migrate StreamExtractor to api-v2 2020-03-17 18:06:13 +01:00
Stypox 5e4ddb368f
[SoundCloud] Fix extractors built from next playlist pages
They didn't have the information to calculate another next page url. So now `nextPageUrl` contains a full link with all video ids, and `getPage` takes the first part of the url (containing 15 streams) and produces another `nextPageUrl` with the remaining streams.
Also add a test for this.
2020-03-17 18:04:40 +01:00
fynngodau b78f788017
Merge branch 'dev' into dev 2020-03-17 18:03:14 +01:00
Stypox 0e1b4bbf17
[SoundCloud] Test playlists banner: it should not exist 2020-03-17 15:53:25 +01:00
Stypox 65bdb3bc9d
[SoundCloud] Unignore ignored playlist tests 2020-03-17 15:49:58 +01:00
Stypox 1558da6f6b
[SoundCloud] Fix playlist next page generation 2020-03-17 15:46:35 +01:00
Stypox ca8bf53b61
[SoundCloud] Fix playlist test: number of streams changed 2020-03-17 15:35:33 +01:00
Stypox d4aa4a0763
[SoundCloud] Fix typo in ChartsExtractor 2020-03-17 15:25:11 +01:00
Stypox 4b1121aac7
[SoundCloud] Add tests for api-v2 channel urls 2020-03-17 15:23:13 +01:00
Stypox c3d811fde5
[SoundCloud] Use api-v2 in SubscriptionExtractor
Also added
2020-03-17 15:22:25 +01:00
Stypox d0e66cc600
[SoundCloud] Improve thumbnail url extraction in playlists
Prevent NullPointerExceptions and remove duplicate code
2020-03-17 15:13:28 +01:00
Stypox f3095713f9
[SoundCloud] Use api-v2 in PlaylistExtractor
Rewrote methods to calculate next page url and to get items from it. `api-v2` is different from `api` since the initial playlist page contains (usually) the full info of the first 3 streams and only the id of the other. Then the single tracks can be requested in batch using `/tracks?ids=id1,id2,...`.
2020-03-17 15:12:13 +01:00
TobiGr 9eca7df947 Forget to fetch the page 2020-03-16 20:50:08 +01:00
TobiGr edb57840d7 [SoundCloud] Update hard-coded client_id 2020-03-16 19:43:36 +01:00
TobiGr efad0be99f Test if the extractor can get AudiStreams with the hard-coded cliend_id
SoundCloud client_id are changed on a regular basis now. We might want to remove the hardcoded id completely on a later point of time when client_id is changed to often
2020-03-16 19:42:35 +01:00
TobiGr dc20f2aa0c version 0.18.7 2020-03-16 17:27:26 +01:00
TobiGr 5f39bc4e99 [SoundCloud] Update hardcoded client id 2020-03-16 17:26:30 +01:00
Tobias Groza 65a7eda446
Merge pull request #256 from B0pol/mediaccc
Support for non-api mediaccc links, invidious shortened links, refactor mediaccc tests
2020-03-15 17:04:09 +01:00
TobiGr 40b3bc138f Remove hack to fix YoutubeStreamLinkHandlerFactoryTest of vnd.youtube videos 2020-03-14 21:54:54 +01:00
Tobias Groza 1c954fbded
Merge pull request #277 from XiangRongLin/vnd
Handle youtube stream urls in "vnd.youtube:videoId" format.
2020-03-14 21:52:49 +01:00
Tobias Groza 9edd0f2675
Merge pull request #288 from Stypox/fix-unlisted-video
[YouTube] Fix videos without next stream throwing error and add test for unlisted videos
2020-03-14 21:36:13 +01:00
Stypox 957b78ed9b
[YouTube] Add test for an unlisted video, which has no next stream 2020-03-14 14:04:01 +01:00
Stypox 325a51f73a
[YouTube] Do not crash if there is no next stream, but return null
Like every other extractor does if there is no next stream
2020-03-14 14:02:48 +01:00
TobiGr aea0962cfe [SoundCloud] Update hardcoded client id 2020-03-14 12:41:44 +01:00
Mauricio Colli b086e9db3f
[YouTube] Fix id extraction for some channels
Some channels had no reliable way to get the redirected id in the response,
so saving it for later was a valid alternative.
2020-03-14 02:39:49 -03:00
Mauricio Colli 00d1ed439b
[YouTube] Fix channel extraction when redirects are in the response
Some redirects were embed directly into the response as instructions
for the page, instead of the usual http redirects.
2020-03-14 02:39:48 -03:00
bopol 66518ec444 check wether mediaccc streams url are secure 2020-03-13 20:07:36 +01:00
bopol 0cd5e05b7b MediaCCCLH: use substring instead of replace; improve a bit tests, return Collections.emptyList(); instead of null where it's annotated @NonNull 2020-03-13 20:00:49 +01:00
bopol f742a6bd3e code optimization 2020-03-13 20:00:49 +01:00
bopol 9701c7d800 invidious shortened links 2020-03-13 20:00:49 +01:00
bopol e8e535b815 mediaccc: update linkhandlers & refactor tests 2020-03-13 20:00:49 +01:00
Tobias Groza e7be952fbf
Merge pull request #281 from mauriciocolli/add-check-live
[YouTube] Add additional check for live videos
2020-03-07 22:07:34 +01:00
Mauricio Colli 70abd57852
[YouTube] Cache commonly used stream type result 2020-03-07 16:48:32 -03:00
Mauricio Colli 1ef706f567
[YouTube] Detect LIVE videos in the trending page 2020-03-07 16:48:31 -03:00
Xiang Rong Lin cc8fb486ee Handle youtube stream urls in "vnd.youtube:videoId" format.
The official YouTube app accept intents in this format, which causes other developers to use it.
2020-03-02 20:40:33 +01:00
Tobias Groza 6446abc6d1
Merge pull request #271 from mauriciocolli/improve-status-checking
[YouTube] Improve response status checks
2020-03-01 15:30:53 +01:00
Mauricio Colli 6aa4d59b91
[PeerTube] Update video used in age limit test (the previous was 404) 2020-03-01 10:29:30 -03:00
Mauricio Colli cfc278317d
[YouTube] Fail-fast if status exist and is anything other than "OK" 2020-03-01 10:29:29 -03:00
Tobias Groza 65f0ec6057
Merge pull request #265 from mauriciocolli/improve-tests
Test if services recognizes their own items urls
2020-03-01 12:59:12 +01:00
Mauricio Colli 90ae5fbea2 Improve kiosk tests across services 2020-03-01 12:57:18 +01:00
Mauricio Colli e590417cc4 Test if services recognizes their own items urls 2020-03-01 12:57:18 +01:00
TobiGr 35252235b0 Go through all badges when checking if a StreamInfoItem is a live stream 2020-03-01 12:52:37 +01:00
Tobias Groza 830b7d3dbd
Merge pull request #267 from mauriciocolli/hotfixes
Hotfixes for YouTube and improve unavailable cases
2020-03-01 12:50:10 +01:00
Mauricio Colli 5686a6f562 [YouTube] Detect when a stream is deleted or doesn't exist
Added a test case as well.
2020-03-01 12:49:13 +01:00
Mauricio Colli e65333c3ce [YouTube] Detect deleted/nonexistent/invalid channels and playlists
- Added tests for these cases.
2020-03-01 12:49:13 +01:00
Mauricio Colli 98e359438a [YouTube] Detect simple 404s in the standard fetch method 2020-03-01 12:49:13 +01:00
Mauricio Colli 408f042127 [YouTube] Fix bug when url isn't present in the browseEndpoint object 2020-03-01 12:49:13 +01:00
Mauricio Colli 342bdbb852 [YouTube] Avoid crashing by letting exceptions bubble up 2020-03-01 12:49:13 +01:00
Mauricio Colli e9644e6216 [YouTube] Handle video premiere's date and duration 2020-03-01 12:49:13 +01:00
Mauricio Colli 2a470ac4f7 [YouTube] Handle videos with no views or with "Recommended to you" text 2020-03-01 12:49:13 +01:00
Mauricio Colli da2c562eb0 [YouTube] Take into account videos that have their views hidden
Also remove catch because it may hide bugs when the page changes.
2020-03-01 12:49:13 +01:00
Mauricio Colli f66c20de54 Ignore null-keyed entries when iterating through the response headers 2020-03-01 12:49:13 +01:00
Mauricio Colli 5edd774fc4 Add latest url to the response to make detection of a redirect possible
Will be latest one in this commit because there's need to check the
history of redirects as of now.
2020-03-01 12:49:13 +01:00
Mauricio Colli 3441946bea Make test downloader return a response instead of throwing an exception
The test implementation was throwing an exception instead of just
returning the response and letting the caller handle it.
2020-03-01 12:49:13 +01:00
Tobias Groza fcbc96a86c
Merge pull request #269 from wb9688/charset
Fix for Android versions earlier than KitKat
2020-03-01 11:31:26 +01:00
wb9688 df219a9453 Don't use StandardCharsets class 2020-03-01 09:41:06 +01:00
TobiGr 705896aca0 Version 0.18.6 2020-02-29 21:39:45 +01:00
Tobias Groza 5a101fd17f
Merge pull request #262 from wb9688/pbj
Improve yt_new
2020-02-29 21:39:02 +01:00
wb9688 2807079531 Catch NumberFormatException instead of checking whether viewCount equals "Recommended for you" 2020-02-29 21:28:38 +01:00
wb9688 3e4d2fd64e Improve testChannelUrl() 2020-02-29 18:20:35 +01:00
wb9688 bfe3eb1409 Use suggested try-if code style 2020-02-29 17:18:50 +01:00
wb9688 a6a63e9570 Let isHardcodedClientVersionValid() throw IOException 2020-02-29 17:01:36 +01:00
wb9688 45df8248bf Use Localization in getJsonResponse() 2020-02-29 16:55:07 +01:00
wb9688 157055fbfd Create getJsonResponse() function 2020-02-29 16:42:04 +01:00
wb9688 4f406512c2 Use YoutubeChannelLinkHandlerFactory 2020-02-29 16:26:28 +01:00
wb9688 4ff655352d Reimplement getAgeLimit() 2020-02-28 17:14:26 +01:00
wb9688 6ba03e542b Cache getVideoTab() 2020-02-28 17:05:31 +01:00
wb9688 8bfbae904d Cache getVideo(Primary|Secondary)InfoRenderer() 2020-02-28 17:03:21 +01:00
wb9688 0b940f5cf4 Use "getAgeLimit() != NO_AGE_LIMIT" instead of "getAgeLimit() == 18" 2020-02-28 16:40:50 +01:00
wb9688 beb07a4852 Add test to check whether hardcoded client version is still valid 2020-02-28 16:36:14 +01:00
wb9688 b89b6b1db2 Check if hard-coded client version is valid in separate function 2020-02-28 16:35:24 +01:00
wb9688 7898ba230d Use pbj in YoutubeStreamExtractor 2020-02-28 15:17:47 +01:00
wb9688 612ec06839 Actually fix thumbnail URLs starting with // 2020-02-28 09:36:33 +01:00
wb9688 f946310af5 Fix search queries with a promotedSparklesTextSearchRenderer returning 0 results 2020-02-28 08:57:44 +01:00
wb9688 94a2d5be46 Fix thumbnail URLs starting with // 2020-02-27 19:08:46 +01:00
wb9688 838d834ce2 Get Uploads view of Videos tab 2020-02-27 18:54:27 +01:00
wb9688 365b0329f3 Implement getTextFromObject() function 2020-02-27 17:39:23 +01:00
wb9688 0798bdd5cd Fix getSubscriberCount() for 0 subscribers 2020-02-27 16:26:22 +01:00
wb9688 8ebd971648 Fix channel descriptions consisting of multiple parts 2020-02-27 10:06:35 +01:00
wb9688 880b951088 Fix getUploaderAvatarUrl() in YoutubeStreamExtractor 2020-02-27 09:53:59 +01:00
wb9688 8aa560bef4 Fix video titles consisting of multiple parts due to @ 2020-02-27 09:42:22 +01:00
wb9688 951ed9f0d8 Don't call getUploaderInfo() in onFetchPage() for playlists
This way YouTube Music playlists will keep working in NewPipe
2020-02-27 09:27:56 +01:00
wb9688 2399d46ab0 Fix YoutubeChannelExtractor for channels without videos 2020-02-27 09:21:25 +01:00
wb9688 cdbc751b87 Fix getUploaderUrl() in YoutubeStreamInfoItemExtractor 2020-02-27 09:05:41 +01:00
wb9688 db305408e9 Get all expandedShelfContentsRenderers in YoutubeTrendingExtractor 2020-02-26 20:37:20 +01:00
wb9688 a2d8489e16 Parse timestamps in description 2020-02-26 20:27:30 +01:00
wb9688 8d503407ef Parse browseEndpoint and watchEndpoint in descriptions 2020-02-26 19:37:58 +01:00
wb9688 8dc39d517a Fix tests
Except testGetFullLinksInDescription() in DescriptionTestUnboxing, since that requires other changes first.
2020-02-26 17:31:01 +01:00
wb9688 88282761f4 Fix YoutubeStreamExtractor not being able to getAgeLimit() in onFetchPage() 2020-02-26 16:19:42 +01:00
wb9688 bb3a3d70bf Use pbj in YoutubeTrendingExtractor 2020-02-26 16:10:24 +01:00
wb9688 1c3a4e710b Check if there are no related streams in getAgeLimit() 2020-02-26 16:03:06 +01:00
wb9688 985c3ec877 Improve getClientVersion() 2020-02-26 15:27:50 +01:00
wb9688 15afbea3e1 Use pbj in YoutubePlaylistExtractor 2020-02-26 15:27:50 +01:00
wb9688 89a41a7f69 Use pbj in YoutubeChannelExtractor 2020-02-26 15:27:50 +01:00
wb9688 0973263aab Use pbj in YoutubeSearchExtractor 2020-02-26 15:27:46 +01:00
TobiGr fc465c8bf7 Fix getUploadDate() for premiered videos with a given date
Detect publishDate and uploadDate
2020-02-25 22:54:37 +01:00
TobiGr d1b7461d0e Version 0.18.5 2020-02-25 22:00:47 +01:00
Tobias Groza 8838e2d136
Merge pull request #261 from TeamNewPipe/yt_new
Update YouTube to material version
2020-02-25 21:57:18 +01:00
TobiGr 96285e0815 Fix getUploadDate() for premiered videos with a given date 2020-02-25 21:51:30 +01:00
wb9688 5a35300a5e Fix parsing upload date for premiered streams 2020-02-25 21:51:30 +01:00
wb9688 729fd2eaee Fix parsing description for navigationEndpoint without urlEndpoint 2020-02-25 21:51:30 +01:00
wb9688 26fb44595f Fix parsing description 2020-02-25 21:51:30 +01:00
TobiGr 0fff03038e Get description with correct links 2020-02-25 21:51:30 +01:00
wb9688 5d08c34322 Fix parsing upload date in YoutubeStreamExtractor 2020-02-25 21:51:30 +01:00
wb9688 bbe1a3cd62 Use getVideo(Primary|Secondary)InfoRenderer() 2020-02-25 21:51:30 +01:00
wb9688 8e6be880d7 Simplify getId() in YoutubeChannelExtractor 2020-02-25 21:51:30 +01:00
TobiGr ecfc27a92c Implement fallback methods for getDescription() and getTextualUploadDate() in YouTubeStreamExtractor 2020-02-25 21:51:30 +01:00
wb9688 eed29ea734 Return null for related streams at age restricted videos 2020-02-25 21:51:30 +01:00
TobiGr 9efcc61ca6 Adapt some tests to match the new URLs 2020-02-25 21:51:30 +01:00
TobiGr f39603f6ef Implement getUploadDate() in YouTubeStreamInfoItemExtractor 2020-02-25 21:51:30 +01:00
wb9688 26ea3dceb6 Fix getViewCount() in YoutubeStreamExtractor 2020-02-25 21:51:30 +01:00
wb9688 76d54abdbf Reimplement more methods in YoutubeStreamExtractor 2020-02-25 21:51:30 +01:00
wb9688 f13c0288cc Reimplement some methods in YoutubeStreamExtractor 2020-02-25 21:51:30 +01:00
wb9688 02b59903fa Remove useless code 2020-02-25 21:51:30 +01:00
TobiGr dd4dd849dc Get client version dynamically in YouTubeSearchExtractor 2020-02-25 21:51:30 +01:00
TobiGr 583e9c1571 Fix getPage()
Someone should check if there are unstaged changes before committing...
2020-02-25 21:51:30 +01:00
TobiGr 5842b9ad37 Add getClientVersion() and HARDCODED_CLIENT_VERSION to YouTubeParsingHelper
Prefer hardcoded client version above the current one when making requests to retrieve the same JSON structure for each request.
2020-02-25 21:51:30 +01:00
wb9688 5d883d100c Implement pagination in YoutubeSearchExtractor 2020-02-25 21:51:30 +01:00
wb9688 c0a8e01889 Implement pagination in YoutubePlaylistExtractor 2020-02-25 21:51:30 +01:00
wb9688 4039409820 Fix getLikeCount() for certain streams 2020-02-25 21:51:30 +01:00
wb9688 627ab2459f Uncomment fetchPage() 2020-02-25 21:51:30 +01:00
wb9688 2dfa2187ff Implement pagination in YoutubeChannelExtractor 2020-02-25 21:51:30 +01:00
wb9688 a38ab9b791 Return null if channel has no banner 2020-02-25 21:51:30 +01:00
wb9688 1eca63089b Count private and deleted videos as ad 2020-02-25 21:51:30 +01:00
wb9688 c7360e4a46 Fix getThumbnailUrl() in YoutubeChannelInfoItemExtractor 2020-02-25 21:51:30 +01:00
wb9688 51fb26625a Implement getSubscriberCount() in YoutubeChannelInfoItemExtractor 2020-02-25 21:51:30 +01:00
TobiGr 6d20b2b1d0 Fix NPE on collecting page items 2020-02-25 21:51:30 +01:00
wb9688 4462cbe3f1 Reimplement YoutubePlaylistInfoItemExtractor 2020-02-25 21:51:30 +01:00
wb9688 8aea4d445b Reimplement YoutubeChannelInfoItemExtractor 2020-02-25 21:51:30 +01:00
wb9688 3187116a63 Handle premium videos 2020-02-25 21:51:30 +01:00
wb9688 7442f91aa6 Don't assume which channel tab is videos 2020-02-25 21:51:30 +01:00
wb9688 316fe0109d Fix first playlist page 2020-02-25 21:51:30 +01:00
wb9688 4e57e589ce Fix first channel page 2020-02-25 21:51:30 +01:00
TobiGr 5816202cc7 Move getInitialData() method to YouTubeParsingHelper
Rename ytInitialData to initialData
2020-02-25 21:51:30 +01:00
wb9688 38aabc6aca Use plain YoutubeStreamInfoItemExtractor in YoutubeTrendingExtractor 2020-02-25 21:51:30 +01:00
wb9688 957db062ef Fix getting uploader for videoRenderer 2020-02-25 21:51:30 +01:00
wb9688 b88188d419 Move stuff from extractVideoPreviewInfo() into YoutubeStreamInfoItemExtractor and partially fix search 2020-02-25 21:51:30 +01:00
wb9688 af49b3c487 Fix live streams in extractVideoPreviewInfo() 2020-02-25 21:51:30 +01:00
TobiGr f33f9466ce Fix testDescription() in ChannelExtractor 2020-02-25 21:51:30 +01:00
TobiGr 10e9c16d8c Fix LiveStream view count 2020-02-25 21:51:30 +01:00
wb9688 127c4d5893 Fix getNextStream() in YoutubeStreamExtractor 2020-02-25 21:51:30 +01:00
wb9688 7bfc0e62c6 Fix getRelatedStreams() in YoutubeStreamExtractor 2020-02-25 21:51:30 +01:00
TobiGr 12166ca60c [StreamExtractor] Add fallback method for uploader avatar url 2020-02-25 21:51:30 +01:00
wb9688 3ec422a542 Fix like/dislike parsing 2020-02-25 21:51:30 +01:00
TobiGr a34e5c18e1 Trending StreamInfoItemCollecetor 2020-02-25 21:51:30 +01:00
TobiGr b705515da0 Adapt most playlist fields 2020-02-25 21:51:30 +01:00
TobiGr 21253abd9b Use Firefox ESR User-Agent: 68 2020-02-25 21:51:30 +01:00
TobiGr 655c999795 Detect disabled subriber count correctly
Fix parsing of kiosk name
2020-02-25 21:51:30 +01:00
TobiGr 7dcc9f159b Stream: fix uploader avatar url 2020-02-25 21:51:30 +01:00
TobiGr be3e20e263 Adapt most channel changes 2020-02-25 21:51:30 +01:00
TobiGr 39bf1ff1eb Add ytInitialData regex 2020-02-25 21:51:30 +01:00
TobiGr a63371a2c5 Version 0.18.4 2020-02-21 22:29:20 +01:00
TobiGr 468092894f Merge branch 'dev' 2020-02-21 22:27:54 +01:00
Tobias Groza 4bc5b8d33f
Merge pull request #254 from TeamNewPipe/formatting
Improve code formatting and optimize imports
2020-02-17 08:38:01 +01:00
Tobias Groza a129c65c63
more improvements of comments
Co-Authored-By: Stypox <stypox@pm.me>
2020-02-16 23:59:31 +01:00
TobiGr 0a4115286b Apply one suggestion 2020-02-15 13:36:54 +01:00
Tobias Groza 6822574588 Fix typos
Co-Authored-By: B0pol <bopol@e.email>
2020-02-15 13:36:54 +01:00
TobiGr 030465b5d4 Improve code formatting and optimise imports 2020-02-15 13:36:54 +01:00
TobiGr fc9f03178f Merge branch 'master' into dev 2020-02-15 00:35:45 +01:00
TobiGr 33961b2092 Version 0.18.3 2020-02-10 21:12:39 +01:00
TobiGr aae3476ee4 [SoundCloud] Update client_id 2020-02-10 21:11:09 +01:00
TobiGr 9112a106ea [SoundCloud] Update client_id 2020-02-08 23:09:03 +01:00
TobiGr 008b6f505c [PeerTube] Fix subscriber count in ChannelExtractorTest 2020-02-08 23:08:37 +01:00
Tobias Groza 3add704f5d
Merge pull request #239 from B0pol/peertube
[PeerTube] added metadata, fix descriptions, fix thumbnail, fix upload date, fix age limit, update tests.
2020-02-08 22:31:02 +01:00
TobiGr 0e33249c85 Fix SoundCloud description test 2020-02-08 22:20:20 +01:00
Fynn Godau 36a316eea5
Merge branch 'dev' into dev 2020-02-08 09:43:13 +01:00
bopol 11bcc78d9c Description implements Serializable. fix NotSerializableException 2020-02-07 14:10:54 +01:00
bopol 0f8a7f9df2 fix testGetUploadDate for PeerTubeStreamExtractor 2020-02-07 13:33:50 +01:00
bopol 70a40e7388 Description: rm constructor by serviceId 2020-02-07 13:28:27 +01:00
bopol 26c65b2948 Create class Description 2020-02-06 23:35:46 +01:00
bopol 5756df8dc7 Use GMT as base time (actually fix upload date) 2020-02-06 22:33:48 +01:00
B0pol b671a4ba18
Merge branch 'dev' into peertube 2020-02-01 18:18:23 +00:00
Tobias Groza 62b81c3607
Merge pull request #222 from mauriciocolli/feed-extractor
Introduce FeedExtractor to fetch from dedicated feeds when available
2020-02-01 18:41:26 +01:00
Mauricio Colli 26234a1c0f
Introduce FeedExtractor making fetching from dedicated feeds possible
YouTube, for example, has a dedicated feed which was built to be used
like this.
2020-01-28 02:44:18 -03:00
Mauricio Colli be81f2945c
[YouTube] Use correct upload date flag in StreamExtractor 2020-01-28 02:44:17 -03:00
Mauricio Colli 5c0c35064c
Use clean url and id when creating a channel info 2020-01-28 02:44:16 -03:00
TobiGr 033a9fb8e3 Revert change which I accidentally committed while fixing the decryption
function in 7b72fd2a7d
2020-01-25 21:08:17 +01:00
B0pol e392b6c68f getLanguageInfo returns Locale instead of String
so that java can automatically translate with Locale.getDisplayLanguage(), instead of always having English name of the language
2020-01-25 14:12:38 +01:00
B0pol 341372c0d0 reindenting (ctrl alt l) on JsonUtils and PeertubeStreamExtractor 2020-01-24 20:16:24 +01:00
B0pol b816e48216
Merge branch 'dev' into peertube 2020-01-24 20:10:47 +01:00
TobiGr f8d117f6cd Merge remote-tracking branch 'origin/master' into dev 2020-01-24 19:22:40 +01:00
Tobias Groza ff61e28424
Merge pull request #243 from TeamNewPipe/fix/yt_decryption
Fix/yt decryption
2020-01-24 02:18:40 +01:00
TobiGr 7b72fd2a7d [YouTube] Add new decryption function 2020-01-24 02:10:56 +01:00
TobiGr bce87f3d01 Improve getDescriptionFuncName by removing catches and adding a loop 2020-01-24 02:10:38 +01:00
TobiGr 9b45c61103 Typos, comments and formatting 2020-01-24 02:10:18 +01:00
TobiGr 29a4cc78bf Fix typo in decryption constant 2020-01-24 02:10:08 +01:00
B0pol 812c8e0ad2 authorName in comments now follow PeerTube website
https://peertube.cpy.re/videos/watch/d2a5ec78-5f85-4090-8ec5-dc1102e022ea
anonyme zirbeldrüse is his displayName, what was displayed in NewPipe.
but on the website, it's shown 777@mastodon.xyz
2020-01-24 00:22:05 +01:00
bopol c790261eed update test 2020-01-23 19:08:41 +01:00
bopol a691d6d0fc fix upload date: there was a one hour delay
on peertube.co.uk: https://i.imgur.com/8Pai1rb.png
on newpipe, before this commit: https://i.imgur.com/NIRbs4Z.png
on newpipe, with this commit: https://i.ibb.co/mhKKtRD/Screenshot-20200123-185422-New-Pipe-Debug.png
2020-01-23 19:02:40 +01:00
bopol 20da4750f8 empty support returns "", same for empty description 2020-01-23 14:37:14 +01:00
bopol 74439f692a add extraction for support info and rename getLanguageInfo function 2020-01-23 14:19:22 +01:00
B0pol 1a15c0e750 agelimit now returns 18 if the video is marked as nsfw, 0 otherwise
+ created getBoolean method in JsonUtils.java
2020-01-23 04:42:54 +01:00
B0pol bcfe7be4e6
Merge branch 'dev' into peertube 2020-01-22 19:19:41 +01:00
Tobias Groza b9afc9807c
Merge pull request #233 from Stypox/yt-stream-ext
Youtube stream extractor improvements by using playerResponse json
2020-01-20 23:17:36 +01:00
Tobias Groza 82eff772fa
Merge pull request #234 from TeamNewPipe/fix/YT_search_test
Fix and improve tests
2020-01-20 23:16:27 +01:00
Stypox 1cfdc4547a
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java
Co-Authored-By: Tobias Groza <TobiGr@users.noreply.github.com>
2020-01-20 22:52:48 +01:00
Stypox 1e1100ef76
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java
Co-Authored-By: Tobias Groza <TobiGr@users.noreply.github.com>
2020-01-20 22:52:36 +01:00
TobiGr 2308b074f0 [MediaCCC] Fix testGetTextualUploadDate() 2020-01-20 22:12:39 +01:00
TobiGr 089dbe8afb Fix StreamLinkHandlerFactoryTest by passing baseUrl to vdn.youtube urls 2020-01-20 22:10:18 +01:00
B0pol f403490bd9
Refactoring
Co-Authored-By: Tobias Groza <TobiGr@users.noreply.github.com>
2020-01-20 21:28:26 +01:00
TobiGr 69ccb7e69d Update subtitle tests comments 2020-01-20 21:25:47 +01:00
TobiGr 7943130307 Throw IllegalArgumentException when url is null in LinkHandlerFactory 2020-01-20 21:25:16 +01:00
TobiGr 39de55dcd3 [YouTube] Fix and move streamCountTest 2020-01-20 21:24:51 +01:00
TobiGr 51fd9ba491 Merge branch 'master' into dev 2020-01-20 20:31:32 +01:00
bopol ad7f97ae83 fix PeerTube description and add more description tests
actually, the max description length is 250 after request with our extractor.
during my tests, I made API requests with Firefox, copy/pasted into echo "insert description" | wc, and it was giving a wrong length, maybe due to the escapers, I have no idea
anyway, it's now fixed
2020-01-20 15:14:23 +01:00
bopol b382416372 changed the way to extract peertube description
if the description length is above 254, and ends with ..., it means the description we got from the first request is shortened.
why above 254: because in fact, shortened descriptions length are between 255 : https://framatube.org/videos/watch/24efbe1b-73c0-4d72-a3ff-77c8b32d3fcd
https://framatube.org/videos/watch/1ca99f40-fb5b-4fa4-abe5-4d54325df7fc
and 269: https://framatube.org/videos/watch/4d661d5f-a884-4510-bca8-15cb19aa3fe5

also fixed a typo in StreamExtractor.java
2020-01-20 14:36:12 +01:00
bopol a281519e77 added metadata, fix descriptions, fix thumbnail, update tests
thumbnail: quality before: https://peertube.cpy.re/static/thumbnails/d2a5ec78-5f85-4090-8ec5-dc1102e022ea.jpg
quality after: https://peertube.cpy.re/static/previews/d2a5ec78-5f85-4090-8ec5-dc1102e022ea.jpg

description: we were getting about the first 260 characters, we now get full description (with fallback to first 260 chars if the get request for full description fails)
test: updated tests to match description, also changed some test: it was assertEquals(extracted, expected), but the proper way to do it is assertEquals(expected, extracted)

metadata: got host, privacy (public, private, unlisted), licence, language, tags
2020-01-20 12:22:33 +01:00
TobiGr c1e1ac1f57 Merge branch 'dev' 2020-01-19 10:13:14 +01:00
Tobias Groza 2ee558fbe7
Merge pull request #237 from B0pol/fix_invidious_comments
Fix invidious comments
2020-01-17 22:55:51 +01:00
Tobias Groza 5a259fca94
Merge pull request #238 from B0pol/c_links_support
add support to /c/shortened_url channel links
2020-01-17 22:53:46 +01:00
Tobias Groza 57cf47b9f3
Merge pull request #236 from B0pol/fix_typos
fix: typos in Test
2020-01-17 22:52:29 +01:00
Fynn Godau 623e6a8c63
Merge branch 'dev' into dev 2020-01-17 22:37:42 +01:00
bopol 221e8dd171 changed assertTrue(string.equals(string)) to assertEquals(string, string) 2020-01-17 21:24:52 +01:00
bopol 5e81ed1dff add support to /c/shortened_url channel links 2020-01-17 19:27:54 +01:00
B0pol 7e8d00981f changed getId() to remove duplicate code 2020-01-17 13:44:55 +01:00
B0pol 02930d08f1 added comments test for invidious 2020-01-17 12:58:50 +01:00
B0pol a12c0e2a37 fix:invidious: getID and onAccpetURl for comments 2020-01-17 12:09:59 +01:00
B0pol 3743c60695 fix: typos in Test 2020-01-15 14:27:52 +01:00
TobiGr 4aaf123f24 Add PeerTube to supported services in README 2020-01-14 15:36:25 +01:00
Fynn Godau 7b5702bdc2 Add bandcamp to list of supported services in README 2020-01-11 16:32:45 +01:00
Stypox 0567c20549
Merge branch 'dev' into yt-stream-ext 2020-01-07 00:10:06 +01:00
Stypox 7ba04836eb
Add tests for ratings in yt stream extractor tests 2020-01-06 23:57:08 +01:00
Stypox 1bb6cdee22
Enable commented-out test for yt stream extractor 2020-01-06 21:43:24 +01:00
Stypox d5ca02f3f6
Fix test failing because music channel could be Vevo
In yt stream extractor test.
The Vevo and the normal music channel are equivalent, so Youtube picks one of them at random, and in playerResponse the channel id is Vevo's one.
2020-01-06 21:41:15 +01:00
Stypox 1689037b01
Fix getLength() tests now that the value is rounded (not floor-ed)
In yt stream extractor
2020-01-06 20:49:08 +01:00
Stypox 1ed89aad3e
Use more often playerResponse in yt stream extractor
This enhances performance and should make the extractor more reliable since it get info from a stable json structure that shouldn't be subject to many changes. Fallback html methods have been kept. In case of error the thrown exception contains the data about the playerResponse failure, that should be clearer than a NPE caused by not-found html tags.
2020-01-06 20:45:57 +01:00
Tobias Groza bdbfa26835
Merge pull request #231 from pihug12/dev
[Soundcloud] Update client_id
2020-01-06 11:21:27 +01:00
Fynn Godau ba967f1a15 Workaround enourmous load times for long bandcamp playlists
Additionally, get cover art from json instead of html
2020-01-03 13:17:04 +01:00
Fynn Godau 46e1f3922c Refuse to load bandcamp playlists without content 2020-01-03 12:32:22 +01:00
Fynn Godau f0d36dfa7e Bandcamp radio: proper name, date shortened 2020-01-03 12:28:49 +01:00
Fynn Godau 808c8aa087 Add bandcamp weekly kiosk
Unfortunately a little messy
2020-01-03 12:28:49 +01:00
Fynn Godau 13ef11e9c8 Bandcamp featured kiosk 2020-01-03 12:28:49 +01:00
Fynn Godau db02850d4f Bandcamp playlist extractor: support playlists with just one stream 2020-01-03 12:28:49 +01:00
Fynn Godau 13e4908b83 Bandcamp search extractor: read track count from result page 2020-01-03 12:28:49 +01:00
Fynn Godau 9b16baffb7 Bandcamp search: multiple pages 2020-01-03 12:28:49 +01:00
Fynn Godau ce2a88e56f Add bandcamp search suggestion extractor 2020-01-03 12:28:49 +01:00
Fynn Godau e12ddaef7f Bandcamp playlist and search extractors: better and more tests 2020-01-03 12:28:49 +01:00
Fynn Godau 655df356a2 Various improvements concerning bandcamp playlists 2020-01-03 12:28:49 +01:00
Fynn Godau ba700bfb3e Add bandcamp playlists (albums) 2020-01-03 12:28:49 +01:00
Fynn Godau d05b14ae48 Add channels (artists) to search results 2020-01-03 12:28:49 +01:00
Fynn Godau 794ca5eeae Bandcamp channel extractor: handel nonexistent images better 2020-01-03 12:28:49 +01:00
Fynn Godau 91c0ec7cea Bandcamp stream extractor: catch NullPointerException 2020-01-03 12:28:49 +01:00
Fynn Godau d5cdc20be1 Bandcmap channel extractor: fix getting banner url 2020-01-03 12:28:49 +01:00
Fynn Godau 7730eb2ea1 Bandcamp channel extractor: enforce https when getting banner url 2020-01-03 12:28:49 +01:00
Fynn Godau a42c77425d Bandcamp channel extractor (ignoring everything but tracks) 2020-01-03 12:28:49 +01:00
Fynn Godau 5281456899 Bandcamp channel link handler factory 2020-01-03 12:28:49 +01:00
Fynn Godau c3d127ccd9 Bandcamp audio stream test and fix 2020-01-03 12:28:49 +01:00
Fynn Godau 80d67e22b7 Return mp3-128 audio stream [experimental] 2020-01-03 12:28:49 +01:00
Fynn Godau 43dc3c3d4c Return empty kiosk list instead of null 2020-01-03 12:28:49 +01:00
Fynn Godau a579337c9a Bandcamp service with support for streams and searches 2020-01-03 12:28:49 +01:00
pihug12 d4e0930f44 [Soundcloud] Update client_id 2019-12-31 23:36:23 +01:00
Tobias Groza d83787a5ca
Merge pull request #226 from 7FM/dev
fix switching on null causing NPE
2019-12-31 00:26:27 +01:00
Tobias Groza 1c0a5db6da
Merge pull request #215 from gzsombor/fix-search-paging
Fix search result paging
2019-12-31 00:25:23 +01:00
TobiGr 0b6e37e71f Improve YouTubeSearchPagingtest 2019-12-31 00:18:53 +01:00
Zsombor Gegesy 64729e5357 Improve the tests 2019-12-29 23:14:55 +01:00
Zsombor Gegesy 68b0fd9650 Add test for search paging 2019-12-29 23:14:55 +01:00
Zsombor Gegesy 2123b3abd3 Fix search result paging - due to the way as the InfoItemsSearchCollector are re-used,
the returned item list just grows, which cause that same videos are returned.
2019-12-29 23:14:55 +01:00
TobiGr 305ff21770 use openjdk8 for travis build 2019-12-26 23:38:40 +01:00
TobiGr f826c24749 Use UTF-8 encoding for docs and bump version to v0.18.0 2019-12-26 22:46:19 +01:00
TobiGr 01eb7a9658 Fix syntax 2019-12-26 22:22:46 +01:00
TobiGr 147322b526 Deploy JDoc even when tests on master fail 2019-12-26 22:11:12 +01:00
Zsombor Gegesy 0a3422a591 Fix invalid jsoup query 2019-12-26 14:03:40 +01:00
7FM fd5ed07d33
Merge branch 'dev' into dev 2019-12-22 12:33:26 +01:00
TobiGr 8e53fdad16 [SoundCloud] Fix getAudioStreams 401 response
Fix TeamNewPipe/NewPipe#2823
2019-12-22 00:42:07 +01:00
7FM f57d9d43ce fix switching on null causing NPE 2019-12-21 14:15:42 +01:00
Tobias Groza 8cb32501e7
Merge pull request #217 from TeamNewPipe/fix/SC_client_id
Fix SoundCloud client_id extraction and update SoundCloud client_id
2019-12-12 18:11:42 +01:00
TobiGr 54d1a1a831 Fix SoundCloud client_id extraction and update SoundCloud client_id
Don't throw exception when hard coded client_id is wrong.
Addresses TeamNewPipe/NewPipe#2823
2019-12-12 18:01:48 +01:00
Tobias Groza be79f20217
Merge pull request #219 from Redirion/patch-1
Allow PlayLists with missing uploader
2019-12-12 17:58:57 +01:00
Redirion 035994b3cf
Update PlaylistInfo.java
added missing imports
2019-12-09 16:55:28 +01:00
Redirion e98112ea21
Update PlaylistInfo.java
Fixed typo
2019-12-07 14:34:15 +01:00
Redirion cdf41b868c
Allow PlayLists with missing uploader
This supports PR 2724 in NewPipe
2019-12-07 12:49:42 +01:00
Tobias Groza 43b54ccc3c
Merge pull request #145 from yausername/peertube
Basic PeerTube support (without P2P)
2019-12-02 23:08:52 +01:00
yausername 00c23683b1
Merge remote-tracking branch 'upstream/dev' into peertube 2019-12-03 00:45:03 +05:30
yausername 5115541268
fix peertube trending, added most-liked kiosk 2019-12-03 00:42:06 +05:30
Tobias Groza b6d3252290
Merge pull request #199 from kapodamy/yt-webm-opus
use webm as format for opus audio streams
2019-11-24 21:39:59 +01:00
kapodamy c1d39f692d
Merge branch 'dev' into yt-webm-opus 2019-11-24 11:56:16 -03:00
yausername 6a7680ce17 added instance getter 2019-11-24 03:38:27 +05:30
yausername 281ccea39f [peertube] constant name for service 2019-11-24 02:35:41 +05:30
yausername bc75c6603b added getBaseUrl method to linkhandler 2019-11-23 00:05:49 +05:30
yausername 279f175693 validate peertube instance 2019-11-22 22:59:14 +05:30
yausername 4e0adbefbc [peertube] use baseUrl instead of global instance url 2019-11-21 05:35:22 +05:30
yausername 8a7aa6b9a9 added baseUrl param in linkhandlers. Required for multi instance services 2019-11-21 05:00:09 +05:30
yausername f3a59a6cdc merged upstream/dev 2019-11-20 03:08:17 +05:30
Mauricio Colli 0c6e2c8f23 Revert "Temp Commit: Ignore Test (Remove this before merge)"
This reverts commit 6ca4c8986a.
2019-11-18 10:09:18 +01:00
Tobias Groza 5c420340ce
Merge pull request #158 from mauriciocolli/timeago-localization
Implement time ago parser and improve localization handling
2019-11-16 00:31:35 +01:00
yausername 318f600527 add subtitles support for peertube 2019-11-16 04:30:13 +05:30
yausername bb5ad49fac fix html comments 2019-11-16 03:20:35 +05:30
yausername 193442d01c add original url for share 2019-11-16 02:42:39 +05:30
yausername d7582734e7 merged upstream/dev 2019-11-16 00:07:45 +05:30
Mauricio Colli 3d21ef5dba
Introduce class that indicates when the time ago is an approximation 2019-11-03 15:46:28 -03:00
Mauricio Colli 6ca4c8986a
Temp Commit: Ignore Test (Remove this before merge) 2019-11-03 15:46:27 -03:00
Mauricio Colli 41b51adceb
[YouTube] Force english as the only supported language (#196) 2019-11-03 15:46:25 -03:00
Mauricio Colli b42bebdee8
Add instructions to quickly test changes in the extractor
- Make use of the local Maven repository
2019-11-03 15:46:24 -03:00
Mauricio Colli cb1e327a6e
[YouTube] Fix parsing of video reminders 2019-11-03 15:46:23 -03:00
Mauricio Colli d8280ce0da
[YouTube] Parse watching count in live streams items 2019-11-03 15:46:22 -03:00
Mauricio Colli 3638f0e0ea
Implement time ago parser and improve localization handling
- Handle special cases for languages where the number is not shown
- Rework the Downloader base implementation, allowing for more
advanced things to be done
- Separate the localization from the content country (just like
YouTube let's the user choose both).
2019-11-03 15:46:21 -03:00
wojcik-online 180836c180
Base Implementation: Parse the upload date of StreamInfoItems
In the format '2 days ago' (in English) on a YouTube channel page.
(Parser extensible to other pages.)
2019-11-03 15:46:20 -03:00
Tobias Groza 514ed7bdc1
Merge pull request #211 from mauriciocolli/improve-recaptcha-detection
[YouTube] Improve reCAPTCHA detection
2019-11-01 12:05:49 +01:00
Mauricio Colli c4fe2183ce
[YouTube] Improve detection of reCAPTCHA pages 2019-10-29 02:00:41 -03:00
Mauricio Colli 9a325b280d
[YouTube] Make detection of age restricted pages more reliable 2019-10-29 02:00:40 -03:00
Mauricio Colli 91c360df5e
Remove section of dead code 2019-10-29 02:00:39 -03:00
Mauricio Colli ddd563fe78 [SoundCloud] Add test for hardcoded client id 2019-10-23 16:54:20 +02:00
Mauricio Colli 4fc18a6994 [SoundCloud] Fix extraction of client id
- Hardcoded id and check at the first usage.
- As a fallback, and considering that the scripts containing the client
id were all split up, try searching it in each of them.
2019-10-23 16:54:20 +02:00
Mauricio Colli 250c0bb1e8 Add head request to the current downloader implementation 2019-10-23 16:54:20 +02:00
kapodamy 3c6e93c96a
Merge branch 'dev' into yt-webm-opus 2019-10-12 14:01:50 -03:00
Tobias Groza 06f2144e4d
[YouTube] Remove GEMA exception handling (#202) 2019-10-05 14:59:05 +02:00
kapodamy 95f93bc71c
Merge branch 'dev' into yt-webm-opus 2019-10-01 13:11:43 -03:00
Tobias Groza b9c0d71210
Merge pull request #190 from nv95/feature/frames
Fetching frames preview from youtube
2019-09-30 19:48:27 +02:00
Stypox fbb9a86c9f
Merge branch 'dev' into yt-webm-opus 2019-09-29 18:16:14 +02:00
Vasiliy d8279f91f6
Merge branch 'dev' of https://github.com/TeamNewPipe/NewPipeExtractor into feature/frames 2019-09-28 14:46:14 +03:00
TobiGr 8ab48c62b9 [YouTube] Fix NPE in ChennelExtractor.getSubsciberCount() 2019-09-25 08:57:08 +02:00
TobiGr 0710f31a39 Fix TeamNewPipe/NewPipeExtractor#197 2019-09-25 08:37:26 +02:00
Stypox d1cd341592 Change comment 2019-09-25 08:32:00 +02:00
Stypox f6088c4fc1 Add test for Eminem channel (it has no "Subscribe" button) 2019-09-25 08:32:00 +02:00
Stypox 35921345d9 Use Exception instead of Throwable (more consistent) 2019-09-25 08:32:00 +02:00
Stypox db3596c818 Fix "Could not get id" for channels w/o "Subscribe" button 2019-09-25 08:32:00 +02:00
Stypox 94e7f0d3ab Fix fallback method is not tried on exception
in YoutubeChannelInfoItem.getUrl()
2019-09-25 08:32:00 +02:00
kapodamy 12008fc6bb Update MediaFormat.java
declare WEBMA_OPUS
2019-09-23 21:25:19 -03:00
kapodamy 5f3c2f5622 Update ItagItem.java 2019-09-23 21:25:19 -03:00
TobiGr 6d504e0883 Add test for mixedNumberWordToLong method
Add Billion to mixedNumberWordToLong
2019-09-17 09:15:05 +02:00
TobiGr 06016d1ae3 Fix YouTube subscriber count
Modify test to fail on too small subscriber count
2019-09-16 23:16:05 +02:00
Tobias Groza dbdd9ed083
Merge pull request #195 from Stypox/livestream-fix
Fix livestreams
2019-09-12 22:40:44 +02:00
Tobias Groza acf2b4c5fa
Merge pull request #194 from toehead2001/youtube-music
Add support for music.youtube.com to link handler
2019-09-12 22:36:04 +02:00
toehead2001 b709529cb6 Add link handler tests for music.youtube.com 2019-09-12 12:16:30 -06:00
toehead2001 075e6d51d6 Add music.youtube.com to link handler 2019-09-12 12:16:30 -06:00
Stypox 1a1672248a
Eliminate Android Studio warnings in livestream test 2019-09-12 15:11:23 +02:00
Stypox 3f1ba93be5
Fix NPE when extracting itags with non-existing streamingData key 2019-09-12 15:08:17 +02:00
Stypox 0d8fb65003
Fix NPE on determining whether stream is live on Youtube 2019-09-12 15:07:07 +02:00
Stypox 4453a63447
Add test for YouTube livestreams
The current livestream is https://www.youtube.com/watch?v=EcEMX-63PKY
2019-09-12 15:05:22 +02:00
Stypox 5f8e76eb87
Move stream-related youtube tests to subfolder 2019-09-12 14:36:42 +02:00
Tobias Groza bfa6b4d14b
Merge pull request #182 from 2secslater/patch-1
Add Invidious instances
2019-09-12 09:19:11 +02:00
Tobias Groza ec3554a2ea
Merge pull request #193 from Stypox/unavailable-video-fix
Fix TeamNewPipe/NewPipe#2615
2019-09-11 21:44:05 +02:00
Stypox 24a37b88a9
Use pre-generated playerResponse field in yt's getHlsUrl()
Also refactored code to always throw exception when the url can't be found
2019-09-11 20:12:30 +02:00
Stypox 9c423a0a40
Use FORMATS to get video+audio streams on yt
Not ADAPTIVE_FORMATS
2019-09-11 20:04:28 +02:00
Stypox e5e8c66686
Readd signature decryption in YtStreamExtractor 2019-09-11 19:56:16 +02:00
Stypox d9570d8634
Use pre-generated playerResponse field everywhere in YtStreamExtractor 2019-09-11 19:35:08 +02:00
Stypox 63a37c48e3
Remove println left behind 2019-09-11 19:31:39 +02:00
Stypox bf017bf5b9
Fix TeamNewPipe/NewPipe#2615 2019-09-11 19:05:41 +02:00
Vasiliy ecb8ad85a1
Update comments 2019-09-11 19:03:53 +03:00
Andrew d0f1c31b34
Add Invidious instances to parsing helper for YouTube 2019-09-10 17:54:32 +01:00
Vasiliy b55c05d04f
Update frameset extractor test 2019-09-10 19:42:55 +03:00
Vasiliy ab32b0ed84
Merge remote-tracking branch 'upstream/dev' into feature/frames 2019-09-10 19:39:58 +03:00
Vasiliy f084cfec24
Refactor frames extraction 2019-09-10 19:38:51 +03:00
Tobias Groza c7e9ad57c4
Merge branch 'dev' into patch-1 2019-09-09 19:05:00 +02:00
Tobias Groza 7169bcfb75
Merge pull request #192 from gzsombor/fix-illegal-expression
Fix 'java.lang.IllegalArgumentException
2019-09-03 09:12:21 +02:00
Zsombor Gegesy 93aeb19bbc Fix 'java.lang.IllegalArgumentException: Did not find balanced marker at 'class*="yt-lockup-video"'
at org.jsoup.helper.Validate.fail(Validate.java:110)'
2019-09-02 23:45:37 +02:00
Vasiliy a6c94c7a9d
Grub frames preview from youtube 2019-08-26 19:14:09 +03:00
Christian Schabesberger cfc72d8dc9
Merge pull request #188 from jimbo1qaz/fix-description-timestamps
Fix timestamp links in Youtube video descriptions
2019-08-22 11:25:16 +02:00
jimbo1qaz e38d906ff9 Fix timestamp links in Youtube video descriptions
For some reason, in NewPipeExtractor,
comments were loaded from JSON by YoutubeCommentsInfoItemExtractor as text,
sent via CommentsInfoItem#getCommentText to NewPipe,
where timestamps are converted to hyperlinks using Linkify:
https://github.com/TeamNewPipe/NewPipe/pull/2168

On the other hand, video descriptions are handled in NewPipeExtractor
by scraping the watch-page HTML.
There, timestamp links were previously mangled (and now properly parsed),
before being sent as HTML via YoutubeStreamExtractor#getDescription
to NewPipe (where HTML gets converted to Spanned).

The logic introduced in this commit is different from the above PR,
since it operates in the extractor, and mutates the HTML DOM
rather than identifying via regex.
2019-08-18 01:05:41 -07:00
Christian Schabesberger 430da57350
Merge pull request #184 from Stypox/duplicate-subscription-fix
Fix inconsistency in youtube channel urls
2019-08-18 00:42:31 +02:00
Christian Schabesberger 58e3996edc
Merge branch 'dev' into duplicate-subscription-fix 2019-08-18 00:42:24 +02:00
Christian Schabesberger f9da8bc0c8
Merge pull request #186 from Stypox/recaptcha-with-url
Add url to ReCaptchaException
2019-08-18 00:42:15 +02:00
Christian Schabesberger 3e2df12f12
Merge branch 'dev' into recaptcha-with-url 2019-08-18 00:41:27 +02:00
Christian Schabesberger 92846076ce
Merge branch 'dev' into duplicate-subscription-fix 2019-08-18 00:41:16 +02:00
Christian Schabesberger a13304248f
Merge pull request #185 from Stypox/search-error-fix
Fix casual search error when playlists are in results
2019-08-18 00:37:59 +02:00
Stypox 06689a2f27
Add url to ReCaptchaException
Sometimes YouTube introduces recaptchas only on some pages. By adding an url to the ReCaptchaException the NewPipe app is able to use that url to load the page that originally caused the problem.
Also removed every instance of exception caught and rethrown with a different description: it makes no sense and it removes part of the useful stacktrace.
2019-08-17 09:09:07 +02:00
Stypox d4e975e4fa
Fix search error with some playlists
Somtimes there were two divs with class "yt-lockup-meta", so the extractor couldn't get the correct one.
2019-08-16 22:47:02 +02:00
Stypox 216a4eb1f5
Complete fix inconsistency in youtube channel urls
It is not always possible to get the url in the form "https://www.youtube.com/channel/...", so a not has been added whenever that happens to be the case (i.e. only in InfoStreamItems).
2019-08-16 21:17:03 +02:00
Stypox 315c5c262f
Typo 2019-08-16 21:14:52 +02:00
Stypox d14c45c948
Fix SoundCloud tests 2019-08-12 17:15:21 +02:00
Stypox b09e402d4f
Fix wrong regex when channel id is at the end of the url
It had no "&" at the end.
2019-08-12 16:55:39 +02:00
Stypox 31e74253f8
Fix tests 2019-08-12 16:38:56 +02:00
Stypox 09c6822b1d
Change youtube channel url test 2019-08-12 13:13:41 +02:00
Stypox b8bc57c53f
Add tests for youtube channel urls
They have to be in the form "https://www.youtube.com/channel/ID"
2019-08-12 11:58:50 +02:00
Stypox 6aa69a2df8
Fix inconsistency in youtube channel urls
Urls from the youtube search extractor were "https://www.youtube.com/user/NAME" instead of "https://www.youtube.com/channel/ID". This fixes TeamNewPipe/NewPipe#2167
2019-08-12 11:57:29 +02:00
Andrew 7fb17684f5
Fixed missing colon causing builds to fail 2019-08-08 00:25:42 +00:00
Andrew 5a2dcdce16
Add Invidious instances
Added all publicly listed Invidious instances from omarroth/invidious wiki page to the link handler factory for YouTube.
2019-08-08 00:19:02 +00:00
Christian Schabesberger 5f65788a2f
Merge pull request #177 from TeamNewPipe/duration_fix
fix duration can not be paresd
2019-07-31 18:03:41 +02:00
Christian Schabesberger 5798c8fdf5 fix duration can not be paresd
update gradle to version 5.1

fix sts issue for agegated videos

GOD DAMN FUCKING BULLSHIT

add duratin for controversal/age gated videos

bring back sts

remove ignores

fix ogg test
2019-07-31 16:11:42 +02:00
Christian Schabesberger 4488c21765
Merge pull request #159 from ritiek/better-soundcloud-thumbnail
Improve thumbnail quality for Soundcloud service
2019-06-27 14:27:23 +02:00
Tobias Groza cfdbc4e1d1
Merge pull request #170 from TobiGr/dev
Fix SoundCloud playlists parsing exception
2019-06-26 01:02:34 +02:00
TobiGr 0d09a9fd61 Fix SoundCloud playlists parsing exception
Closes TeamNewPipe/NewPipe#2344
2019-06-26 00:56:03 +02:00
Tobias Groza c64c90a567
Merge pull request #168 from masozzi/dev
soundcloud parsing helper: fixed id parser regex
2019-06-02 21:06:57 +02:00
Tobias Groza 796c1b7bc0
Merge pull request #164 from TeamNewPipe/fix/test_comments
Fix failing YouTube comments tests
2019-06-02 11:36:16 +02:00
Matteo Sozzi 93d4299f07 soundcloud parsing helper: fixed id parser regex 2019-05-31 20:15:36 +02:00
Tobias Groza 2ac713e70f
Merge pull request #160 from Stypox/invalid-youtube-subscription-fix
Fixed youtube subscription import: ignore ones with invalid url and keep ones with empty title.
2019-05-14 22:26:25 +02:00
Tobias Groza 867ca1cabf Fix failing YouTube comments tests
The comment function has been disabled for the video on which we ran the test. We are testing the comments of a different video now.
2019-05-14 22:11:40 +02:00
Tobias Groza acebbaf073
Merge pull request #163 from Stypox/youtube-url-signature-fix
Fix invalid yt url: signature tag name is not always "signature"
2019-05-14 21:29:31 +02:00
Stypox c70d28597b
Add fallback for urls not conaining the "sp" tag
If ever YouTube changes thing again (or uses old urls for some unknown reason), this prevents the extractor from crashing.
As suggested here: 133cc032d9 (r283529811)
2019-05-14 13:57:45 +02:00
Stypox 133cc032d9
Fix invalid yt url: signature tag name is not always "signature"
Thanks to @omarroth for the suggestion: see TeamNewPipe/NewPipeExtractor#155
2019-05-13 21:25:35 +02:00
Stypox 0eaca52c15
Add test for subscription with invalid url.
Also modified the test for empty title, since now  subscriptions with empty title are not ignored anymore.
2019-04-28 14:19:33 +02:00
Stypox 171f2c49fe
Ignore subscriptions with invalid url and keep ones with empty title.
if a channel if deleted (thus it has an empty title), it is imported in NewPipe anyway, so that if it becomes undeleted in the future, it will be shown in the app.
2019-04-28 14:17:52 +02:00
Stypox d5043cdf49
Add test for subscriptions with empty title.
(youtube subscription extractor)
2019-04-26 19:59:23 +02:00
Stypox 03893abd91
Fixed TeamNewPipe/NewPipe#2226.
(in the youtube subscription extractor)
Ignore subscriptions that have an empty title instead of throwing an error: the youtube subscription_manager XML file can sometimes contain those (i.e. deleted channels).
2019-04-26 18:54:30 +02:00
Ritiek Malhotra c7974b2aed Fetch better quality thumbnails
and fallback to avatar thumbnail if track thumbnail isn't found
2019-04-26 02:59:15 +05:30
yausername f60c973928 more javadoc 2019-03-23 19:32:06 +05:30
yausername e4e6d44770 reordered services 2019-03-23 19:12:26 +05:30
yausername aabc65b57d added java doc for stream 2019-03-23 19:11:51 +05:30
yausername 07a8129a73 fixed peertube search test 2019-03-23 19:11:19 +05:30
yausername bd22b55885 fixed youtube channel name test 2019-03-23 19:10:34 +05:30
yausername 2193119fab youtube's comment censorship breaks test 2019-03-23 19:09:40 +05:30
Christian Schabesberger d22786b817 Merge branch 'master' into dev 2019-03-23 10:07:28 +01:00
Christian Schabesberger 4c633c2166
Merge pull request #137 from connectety/invidious-support
Invidious support
2019-03-23 10:01:51 +01:00
Christian Schabesberger 5b93db04e6
Merge pull request #152 from yausername/fixMissingAuthorName
fix empty author name
2019-03-23 09:57:44 +01:00
yausername 4effd0b36d fix empty author name 2019-03-23 00:13:18 +05:30
Christian Schabesberger 560c648e92 fix decrypt regex for akamai 2 times in file 2019-03-14 16:49:30 +01:00
Connectety ed78f7dadc
Merge branch 'master' into invidious-support 2019-03-14 16:07:04 +01:00
Christian Schabesberger dd61d66cf5 speed up finding decrypt function 2019-03-14 09:07:19 +01:00
Christian Schabesberger 5eac266a16 Merge branch 'dev' of github.com:TeamNewPipe/NewPipeExtractor into dev 2019-03-14 08:50:57 +01:00
Ritvik Saraf 0a7d42f58d set soundcloud default kiosk 2019-03-14 08:49:37 +01:00
Ritvik Saraf 53058802e2 fix comment url 2019-03-14 08:49:37 +01:00
Christian Schabesberger e072bf6461 fix dercrypt error due to wrong dollar sign detection 2019-03-14 08:49:11 +01:00
Christian Schabesberger 5f44031ce7
Merge pull request #146 from yausername/soundcloudDefaultKiosk
set soundcloud default kiosk
2019-03-10 22:28:29 +01:00
Ritvik Saraf cb30b336ab set soundcloud default kiosk 2019-03-11 02:14:58 +05:30
Ritvik Saraf c220700c94 fixed tests 2019-03-10 00:37:01 +05:30
Ritvik Saraf 36139558c6 handle stream 404 error, fixed tests 2019-03-09 23:33:51 +05:30
Ritvik Saraf e66d188381 throw content not available 2019-03-08 04:54:39 +05:30
Ritvik Saraf 22dac63698 merged upstream/dev 2019-03-08 03:09:05 +05:30
Christian Schabesberger aa4f03a385
Merge pull request #144 from yausername/fixCommentUrl
fix comment url
2019-03-02 23:38:28 +01:00
Connectety c255fdcdf7
Merge branch 'master' into invidious-support 2019-03-02 13:03:31 +01:00
Ritvik Saraf a2735c4f97 fix comment url 2019-03-02 02:48:05 +05:30
Christian Schabesberger 536365c99e
Merge pull request #142 from yausername/serviceByUrlNPEfix
fix npe in fetching service by url
2019-03-01 09:57:52 +01:00
Christian Schabesberger 16d693ea7a
Merge branch 'master' into serviceByUrlNPEfix 2019-03-01 09:54:41 +01:00
Christian Schabesberger 8425936b67
Merge pull request #143 from yausername/sanitizeComment
Sanitize youtube comment
2019-03-01 09:52:22 +01:00
Ritvik Saraf 72262707bf fixed youtube comment utf8 bom 2019-03-01 04:21:45 +05:30
Ritvik Saraf 9eff18252b fix npe in fetching service by url 2019-02-28 22:55:18 +05:30
Christian Schabesberger cffbd468cc
Merge branch 'master' into invidious-support 2019-02-24 21:58:59 +01:00
Christian Schabesberger e7e411dc29
Merge pull request #141 from yausername/invalidChannelUrlFix
Invalid channel url fix
2019-02-24 21:58:26 +01:00
Ritvik Saraf 948a7826ab fetch channel id for channel url 2019-02-23 19:03:52 +05:30
Connectety 28c5b86f6d
Merge branch 'master' into invidious-support 2019-02-20 17:58:21 +01:00
Christian Schabesberger 8de53111d9
Merge pull request #101 from yausername/master
added comments extractor
2019-02-19 17:31:25 +01:00
Christian Schabesberger 021e4e06b6
Merge branch 'master' into master 2019-02-19 17:28:38 +01:00
Christian Schabesberger 04460bba2c
add mediaCCC to supported sites in description. 2019-02-19 16:10:46 +01:00
Ritvik Saraf 3dadf63028 refactored comments capability 2019-02-16 00:57:00 +05:30
Ritvik Saraf a97b6dfa7a mediaccc merge 2019-02-16 00:41:23 +05:30
Ritvik Saraf f8fb7b6500 Merge remote-tracking branch 'upstream/master' 2019-02-16 00:18:39 +05:30
Connectety 5079d5efe1
Merge branch 'master' into invidious-support 2019-02-07 23:06:19 +01:00
Christian Schabesberger d15f32925b
Merge pull request #139 from TeamNewPipe/ccc
Ccc
2019-02-05 18:27:47 +01:00
Ritvik Saraf f97d5ab4cd Merge remote-tracking branch 'upstream/master' 2019-01-29 22:10:07 +05:30
Christian Schabesberger 79b0a19d1a make search filter empty be equal to all 2019-01-29 16:49:37 +01:00
Christian Schabesberger d0efe8bd47 fix ogg 2019-01-29 16:04:19 +01:00
Christian Schabesberger 644da4fc3c add ogg as audioformat 2019-01-28 15:43:21 +01:00
Christian Schabesberger 14e1ccdb86 add conferences search 2019-01-28 15:30:17 +01:00
Christian Schabesberger 1503459292 add kiosk for conferences 2019-01-28 12:18:16 +01:00
Connectety-W 6390eb268b
fixed YoutubePlaylistLinkHandlerFactory accepting invalid links 2019-01-27 12:05:36 +01:00
Connectety-W 1ab7a1f930
added tests for YoutubePlaylistLinkHandlerFactory 2019-01-27 12:00:23 +01:00
Connectety-L ec4aa9e0cd
added Invidio subdomain test and fixed error in test caused by Invidio playlist support 2019-01-27 02:12:12 +01:00
Connectety-L 10939efcce
added support playlists on Invidio 2019-01-27 01:44:46 +01:00
Connectety-L cce5e4ad33
added support and tests for Invidio Trending-URLs 2019-01-27 01:29:23 +01:00
Connectety-L 7493ed903b
split isYoutubeALikeURL into multiple methods 2019-01-27 01:28:51 +01:00
Christian Schabesberger cb51254aa3 return one null less 2019-01-26 22:55:34 +01:00
Christian Schabesberger 73232a7bad stream names may not contain dots 2019-01-26 17:47:58 +01:00
Christian Schabesberger 50a17afc05 fix issue with originalUrl 2019-01-26 17:47:58 +01:00
Christian Schabesberger cf3ea22753 make stream size have a p in the end 2019-01-26 17:47:58 +01:00
Christian Schabesberger 7fcb3f5790 fix crash when return null as a related streams collector 2019-01-26 17:47:58 +01:00
Christian Schabesberger 2148edc7d7 add channel and strem extractor for mediaccc 2019-01-26 17:47:58 +01:00
Christian Schabesberger ef576341c8 implement searchextractor for mediaccc 2019-01-26 15:22:16 +01:00
Connectety-L 2ede47d36c
added hooktube and invidio http test 2019-01-24 14:19:44 +01:00
Connectety-L 2ae23a6f79
added support for videos on invidio.us 2019-01-24 11:13:01 +01:00
Connectety-L 399b4f2eef
added support for "vnd.youtube.launch" URI-scheme 2019-01-24 10:53:03 +01:00
Connectety-L ae23059d66
added support for channels on invidio.us 2019-01-24 10:48:29 +01:00
Connectety-L 79ba6aba95
added tests from https://github.com/TeamNewPipe/NewPipeExtractor/pull/128 2019-01-24 10:27:01 +01:00
Christian Schabesberger 8114ce6ae4
Merge pull request #132 from connectety/master
refactored YouTube-linkHandler
2019-01-20 14:46:06 +01:00
Connectety-W 19288c1456
added forgotten break statements 2019-01-20 14:39:06 +01:00
Christian Schabesberger 76a3742084
Merge branch 'master' into master 2019-01-20 14:34:16 +01:00
Christian Schabesberger f7c7b9df1a
Merge pull request #135 from yausername/ytlivestream-fix
[Youtube Live Streams] fixed extracting live HLS URL
2019-01-20 13:55:32 +01:00
Connectety-W a6c972eff8
fixed tests by prepending HTTP to URLs without protocol and adding a check for null. 2019-01-20 01:31:30 +01:00
Ritvik Saraf c60d6ec0ab removed unused import 2019-01-19 18:20:02 +05:30
Ritvik Saraf 50c4783f71 fixed extracting hls url 2019-01-19 18:07:35 +05:30
Ritvik Saraf c4e502b9d2 Merge branch 'master' into peertube 2019-01-19 14:33:01 +05:30
Ritvik Saraf 25ed9deaf3 Merge remote-tracking branch 'upstream/master' 2019-01-19 14:28:12 +05:30
Christian Schabesberger 97d72590fc
Merge branch 'master' into master 2019-01-18 22:31:37 +01:00
Christian Schabesberger 99915e4527 fix failing tests 2019-01-18 12:44:34 +01:00
Christian Schabesberger 6776abdd55 Merge https://github.com/The-Twisted-Samurai/NewPipeExtractor 2019-01-18 12:25:20 +01:00
Christian Schabesberger 09bafe0660 fix failing viewcount test 2019-01-18 12:24:30 +01:00
Kunal Sareen b6790bbe6b Updating regex 2019-01-18 21:47:34 +11:00
Connectety 43ffeac023
Merge branch 'master' into master 2019-01-13 13:38:35 +01:00
Connectety-W 98f49852d7
refactored YouTube-linkHandler to use less regex and more URL-methods 2019-01-13 12:52:07 +01:00
Ritvik Saraf fade7bd749 getter for torrent url 2018-12-30 15:23:24 +05:30
Ritvik Saraf ef0ffb2229 added torrent url in streams 2018-12-30 15:00:57 +05:30
Ritvik Saraf b1a77fa484 update peertube instance 2018-12-29 22:23:35 +05:30
Ritvik Saraf a83342de0f update peertube instance 2018-12-29 21:58:43 +05:30
Ritvik Saraf b05fa45080 resolved review comments 2018-12-28 13:02:00 +05:30
Ritvik Saraf b2c6928459 added logic for fetching related streams 2018-12-26 14:29:23 +05:30
Ritvik Saraf bc82a53f2c replace base url 2018-12-26 13:07:06 +05:30
Ritvik Saraf 203c884d51 strip base url 2018-12-26 13:02:04 +05:30
Ritvik Saraf 14cd1dbcbb fixed kiosk by url 2018-12-26 12:50:43 +05:30
Ritvik Saraf 8755c25349 format date of comment published. changed default instance 2018-12-26 11:08:39 +05:30
Ritvik Saraf df0db8468d merged master 2018-12-25 19:12:22 +05:30
Ritvik Saraf c77050dc70 handle empty comments 2018-12-23 08:24:36 +05:30
Ritvik Saraf 9ad102df3f fixed fetching youtube client name 2018-12-23 08:24:36 +05:30
Ritvik Saraf 4c49a347f6 merged upstream/master 2018-12-04 21:20:26 +05:30
Christian Schabesberger 91b1efc97e add documentation of StreamingService 2018-11-10 10:50:13 +01:00
Christian Schabesberger a78ad16235 rebase correctly 2018-11-09 19:55:40 +01:00
kapodamy b40a1f3657 Delete Subtitles.java
change class name
2018-11-08 17:06:41 -03:00
kapodamy 388f4fd38b Delete SubtitlesFormat.java
merge into MediaFormat
2018-11-08 16:29:06 -03:00
kapodamy e4afb21862 make Subtitle object Stream compliant
* merge SubtitlesFormat into MediaFormat
* implement Stream interface on Subtitle class
* misc fixes: dont show a snackbar error on age-restricted videos, lint fix.
2018-11-08 16:29:05 -03:00
Christian Schabesberger 7129d6db55 add inline documentation for StreamExtractor 2018-11-07 18:28:44 +01:00
Christian Schabesberger 4de99ae28f add support for segmented streams 2018-11-06 14:24:47 +01:00
Christian Schabesberger 32d316330c fix broken search results 2018-10-25 15:46:47 +02:00
Ritvik Saraf f58c914e73 merge fix 2018-10-19 20:17:26 +05:30
Ritvik Saraf 05ef80c2eb Merge remote-tracking branch 'upstream/master' 2018-10-19 20:09:51 +05:30
Ritvik Saraf 99a0134b1c removed unchecked cast 2018-10-19 19:33:36 +05:30
Ritvik Saraf 4883b6fb48 return suggestion extractor 2018-10-12 13:04:51 +05:30
Ritvik Saraf 706d4271a6 empty suggestions for search 2018-10-12 12:45:09 +05:30
Ritvik Saraf c9fba9cb1e fallback avatar for channel 2018-10-12 02:27:32 +05:30
Ritvik Saraf 26da3088ba fallback avatar and description 2018-10-12 01:59:13 +05:30
Ritvik Saraf 9dee8d5fcd updated kiosk accept link 2018-10-12 01:37:47 +05:30
Ritvik Saraf 20f280cb57 added peertube extractor 2018-10-12 00:40:22 +05:30
Christian Schabesberger 24cb684e23 fix failing test 2018-10-06 17:11:12 +02:00
Christian Schabesberger 0920680708 fix search country for next page 2018-10-06 17:03:55 +02:00
Christian Schabesberger 0c3dc86b07 add support for yt content country 2018-10-06 12:22:37 +02:00
Christian Schabesberger 3a865749e8 fix issues Kiosk issue about localization, nad rename to preferred localization 2018-10-05 16:01:10 +02:00
Ritvik Saraf e85958b180 merged upstream/master 2018-09-29 13:19:00 +05:30
Ritvik Saraf d1ff1c7589 using mobile website since it is faster 2018-09-28 04:54:57 +05:30
Ritvik Saraf c2ed99b1b7 no comments 2018-09-27 23:01:27 +05:30
Ritvik Saraf ad8066830d getting client version and name from initial page 2018-09-27 22:32:13 +05:30
Ritvik Saraf 6b620914b6 moved cookie logic outside 2018-09-27 02:04:12 +05:30
Ritvik Saraf fb1419608a using getDataString 2018-09-27 00:44:55 +05:30
Ritvik Saraf ce76885553 removed generics 2018-09-26 04:21:58 +05:30
Ritvik Saraf 8e27801183 removed jackson and java 8 2018-09-26 03:20:29 +05:30
Christian Schabesberger 5223aece7b fix failing unit test 2018-09-24 15:17:57 +02:00
Christian Schabesberger f334dd07de fix youtub.be links in description 2018-09-24 15:06:00 +02:00
Christian Schabesberger ad4b15ac66
Merge pull request #114 from TeamNewPipe/localization
Localization
2018-09-24 14:00:20 +02:00
Ritvik Saraf c1199c8fcf added isCommentsSupported method 2018-09-23 01:12:11 +05:30
Ritvik Saraf 0e86475891 removed commentsInfo from streamInfo 2018-09-22 23:36:41 +05:30
Ritvik Saraf 7ed0da0493 more testing 2018-09-22 19:09:31 +05:30
Ritvik Saraf 6a0341d59c testing 2018-09-22 14:55:59 +05:30
Christian Schabesberger 49c2eb5185 fix failing watch links 2018-09-20 13:51:06 +02:00
Ritvik Saraf e04787f340 fixed NPE for services where comments is not available 2018-09-19 05:32:14 +05:30
Ritvik Saraf 4794e16dcb removed comment details form streamInfo. added commentsInfo instead 2018-09-19 04:58:40 +05:30
Ritvik Saraf ee239985ae added CommentsInfo 2018-09-19 04:58:40 +05:30
Ritvik Saraf 4ca23ab5c3 revering serialzable change 2018-09-19 04:58:40 +05:30
Ritvik Saraf bf2d3deb03 make extractor serializable 2018-09-19 04:58:40 +05:30
Ritvik Saraf 823551170d added loadable comments in stream info 2018-09-19 04:58:40 +05:30
Ritvik Saraf 9fb0622a24 better quality thumbnails 2018-09-19 04:58:40 +05:30
Ritvik Saraf f62f147ea0 added comments to stream info 2018-09-19 04:58:40 +05:30
Ritvik Saraf 06898f47b2 close resource 2018-09-19 04:58:40 +05:30
Ritvik Saraf 95575756ee added http post method in downloader, formatting 2018-09-19 04:58:40 +05:30
Ritvik Saraf 8b8779b176 removing catch exception 2018-09-19 04:58:40 +05:30
Ritvik Saraf 53d3997904 added comments extractor 2018-09-19 04:58:40 +05:30
Christian Schabesberger ce2bbee1e9 add propper localization 2018-09-15 22:29:28 +02:00
Christian Schabesberger dc0d0bda24 replace uiHandler name with linkhandler 2018-09-15 22:29:28 +02:00
Christian Schabesberger 1e7bcfbd66
Merge pull request #97 from MadderRagax/master
Fixed  #1548 and #1546 - Playlist links not getting parsed properly
2018-09-15 22:29:08 +02:00
Christian Schabesberger ae8107b182
Merge branch 'master' into master 2018-09-15 22:18:54 +02:00
Christian Schabesberger 96b4c97cee
Merge pull request #109 from karyogamy/sig-fix-update
Decryption Signature Regex Update
2018-09-15 22:10:31 +02:00
Christian Schabesberger 217d13b102 fix wrong subscription count 2018-09-11 15:14:22 +02:00
Christian Schabesberger 8a4afe2548 refactor split time parsing 2018-09-09 14:01:39 +02:00
Christian Schabesberger 49b72cbe5d
Merge pull request #112 from skil3z/master
Fix time formatting for different countries
2018-09-09 13:58:34 +02:00
skil3z a83d020944
Accommodate time formatting for different countries
If there's a . in the time format, this detects it and uses is instead of : 
This removes errors and lag related to "Could not get duration" while using NewPipe in countries with official time formatting with . (dot) instead of : (punctuation colon)

Tested to compile and work on real device
2018-09-09 12:53:10 +03:00
John Zhen Mo ed73ae55f1 -Added more decrypt function name matching regex.
-Cleaned up decryption code generation method.
2018-09-07 22:25:07 -07:00
Christian Schabesberger 66c3c3f452 fix channel links in description part 2 2018-09-07 22:18:22 +02:00
Christian Schabesberger 4469d11307 fix channel links in description 2018-09-07 21:40:36 +02:00
John Zhen Mo 119843bfac -Fix signature extraction parse exception. 2018-09-07 11:34:14 -07:00
Mauricio Colli 850670917f
Quick fix for the kiosks in SoundCloud 2018-09-05 07:25:24 -03:00
Mauricio Colli 834382111b
Fix NPE when playlist is empty 2018-09-03 22:37:31 -03:00
Mauricio Colli 0400ae026f
Fix channels with subscription count disabled
Related: TeamNewPipe/NewPipe#1649
2018-09-03 22:37:31 -03:00
Mauricio Colli f6be85e2f5
Merge branch 'dev' into 'master' 2018-09-03 22:37:31 -03:00
Christian Schabesberger f498dd7875 fix failing unit tests
fix yt share url error
2018-08-28 17:19:22 +02:00
Christian Schabesberger fef71aeccc roleback using href for description links 2018-08-28 11:55:12 +02:00
Christian Schabesberger 3aa7ff6d0e make ytdescription alter href instead of the text of links 2018-08-27 20:45:43 +02:00
Christian Schabesberger e662c97433 make dash parser ignore segmented streams 2018-08-21 17:23:56 +02:00
Christian Schabesberger c4e16c7337 fix hashtag parsing 2018-08-17 16:22:50 +02:00
Christian Schabesberger 981cb333b5 fix link parsing for yt internal links
blub
2018-08-16 17:44:09 +02:00
Christian Schabesberger 389a87fc89 add link parsing for youtube description 2018-08-16 17:11:18 +02:00
Christian Schabesberger aeb813840d fix search querry tests 2018-08-05 14:14:36 +02:00
Christian Schabesberger 701666f498 block yt premium videos
fix logig error for block yt premium videos
2018-08-05 13:04:49 +02:00
Christian Schabesberger af1dddb79e
Merge pull request #96 from Bleuzen/enable_opus
Enable opus format 251
2018-08-01 18:21:50 +02:00
oscar a077cb9777 Small YoutubeStreamLinkHandlerFactory refactoring 2018-07-25 17:21:42 +02:00
oscar a9ce0b29d0 Fixed improper handling of playlist deep links 2018-07-25 17:15:56 +02:00
Christian Schabesberger dcca11fbeb Merge branch 'dev' 2018-07-25 14:13:16 +02:00
Bleuzen 9af07a2562 re-enable opus 2018-07-24 17:25:42 +02:00
Christian Schabesberger 1eff8c5708 rename filter playlist to playlists 2018-07-19 16:16:37 +02:00
Christian Schabesberger 558a973e43 rename search filters 2018-07-19 16:06:44 +02:00
Christian Schabesberger 8ee068fd98 fix names in kiosklist
blub
2018-07-13 21:50:49 +02:00
Christian Schabesberger 28788a05db rename uih and remove afiliate link foo 2018-07-13 18:02:40 +02:00
Christian Schabesberger a1aaca1bea get info from initial page 2018-07-08 20:49:13 +02:00
Christian Schabesberger 53f0bc9d8d add getInfo without need for search extractor to searchinfo 2018-07-08 20:15:14 +02:00
Christian Schabesberger 35b46900c1 add outcomming items of searchextractor to searchinfo 2018-07-08 19:53:25 +02:00
Christian Schabesberger 4746a1c48a upgrade kisok uriIdhandler to ListUIHandler 2018-07-08 15:27:35 +02:00
Christian Schabesberger 31b0480e22 fix failing tests 2018-07-08 14:43:49 +02:00
Christian Schabesberger 2f351be772 add getMoreInfo to SearchInfo 2018-07-08 14:32:31 +02:00
Christian Schabesberger 39332a283b remove searchengine 2018-07-02 13:49:14 +02:00
Christian Schabesberger 812a785811 remove searchengine 2018-07-02 13:47:39 +02:00
Christian Schabesberger 291de18418 getUIHandler() to getUIHFactory()
handler to handlerFactory in kiosk
2018-07-01 19:01:39 +02:00
Christian Schabesberger cf542db853 getUIHandler() to getUIHFactory() 2018-07-01 18:50:21 +02:00
Christian Schabesberger bd5423fe2a make less tests fail 2018-07-01 16:55:48 +02:00
Christian Schabesberger 17f46b8b12 rename UrlIdHandler to UIHFactory 2018-07-01 16:55:48 +02:00
Christian Schabesberger 9bda761103 add test for soundcloud search extractor 2018-07-01 16:55:48 +02:00
Christian Schabesberger ef2ce6854c add soundcloud SearchExtractor 2018-07-01 16:55:48 +02:00
Christian Schabesberger 5718d5b8b4 add tests for searchextractor 2018-07-01 16:55:48 +02:00
Christian Schabesberger 06c67763d2 remove soundcloud and make first search test work 2018-07-01 16:55:48 +02:00
Christian Schabesberger b4544a67e8 add SearchExtractor 2018-07-01 16:55:48 +02:00
Christian Schabesberger 91aa52f65c Merge branch 'dev' 2018-05-15 22:38:54 +02:00
Christian Schabesberger 0501a2f543 fix donation link parsing 2018-05-08 21:58:35 +02:00
Christian Schabesberger 94d1eb6f32 restructure youtube service 2018-05-08 21:19:03 +02:00
Christian Schabesberger e5125c15f1
Merge pull request #78 from TeamNewPipe/donation
enable Donations for streams
2018-05-07 10:08:36 +02:00
Christian Schabesberger 08f0247d73 make UrlIdHandler actual wrapper for the url/id
fix tests

fix tests
2018-05-06 19:51:24 +02:00
Christian Schabesberger b7c1f251d8 add filter to stream lists 2018-05-05 20:07:47 +02:00
Christian Schabesberger bf1c771662 fix could not parese metadata despide available error 2018-05-04 16:11:25 +02:00
Christian Schabesberger bad67f0bc3 Merge branch 'dev' of github.com:TeamNewPipe/NewPipeExtractor into dev 2018-04-29 21:31:45 +02:00
Christian Schabesberger 466865385e add support for affiliate 2018-04-29 21:31:24 +02:00
Christian Schabesberger 016c2fc549 add channel donations for youtube 2018-04-08 15:58:42 +02:00
Mauricio Colli 1c8dafe4c6
Lightweight method to get the client id on SoundCloud
- Instead of downloading the whole ~1MB file, download just the first 16KB. With fallback to the old way in case of fail
- I've come across some technical difficulties as to how to limit the download (current implementation is too basic/simple for that), but in the end, a simple http header (Range) was enough
2018-04-07 22:12:27 -03:00
Christian Schabesberger 17e7e5a359 enable donations for streams 2018-04-07 21:36:20 +02:00
Christian Schabesberger 1d94e7d477 add ability to get donation links from channel 2018-04-07 16:45:07 +02:00
Mauricio Colli 77a74b8d20
Add "originalUrl" field to Info's object 2018-04-06 03:06:46 -03:00
Mauricio Colli 3fe9922d49
Fix failing test due to an invalid path 2018-04-06 03:06:33 -03:00
Christian Schabesberger a6b6235644 return propper name for soundcloud kisok extractor 2018-04-01 20:27:44 +02:00
Christian Schabesberger 22c59a4895 changed implement me to SoundCloud Charts 2018-03-31 18:38:22 +02:00
Christian Schabesberger 9664658914 Merge branch 'master' of github.com:TeamNewPipe/NewPipeExtractor 2018-03-25 16:08:35 +02:00
Christian Schabesberger f2067338db move on to version 0.13.0, and made ListCollector only accept InfoItemExtracto 2018-03-25 16:08:14 +02:00
Mauricio Colli 27f61e049d
Add time units strings and utilities 2018-03-22 20:31:02 -03:00
Mauricio Colli fa3974bf3a
Fix javadoc generation and some errors
- Generate javadoc containing all the classes from all submodules
2018-03-18 16:18:11 -03:00
Mauricio Colli f787b375e5
Created gradle module and moved existing code to new one 2018-03-14 00:44:02 -03:00
Mauricio Colli 94e24a6e1c
Remove RunWith from tests
- It was causing tests to repeat
2018-03-13 23:50:57 -03:00
Mauricio Colli d71a42d6cd
Ignore some tests due to some recent changes 2018-03-13 23:43:13 -03:00
Mauricio Colli 37f2e5cfda
Make some methods return the specific InfoItem type
- Some methods were returning a broader range of InfoItem types than it should be. For example, a ChannelInfo should return a List containing only StreamInfoItem, instead of the more general InfoItem.
- Renamed and changed return type of ListExtractor.getInfoItems to getInitialPage returning a InfoItemsPage, to be consistent with getPage(url)
2018-03-13 23:32:31 -03:00
Mauricio Colli 5dd2daad37 Finish transition to use getters on Info classes
- Renamed Collector method
2018-03-13 23:28:58 -03:00
Mauricio Colli 6c07b7851b Fix Java 7 incompatibility 2018-03-13 23:25:23 -03:00
Mauricio Colli 29afd0c29c Fix some bugs discovered by the new tests 2018-03-13 23:25:23 -03:00
Mauricio Colli d9cf1fd631 Improve Channel and Playlist tests 2018-03-13 23:25:23 -03:00
Mauricio Colli 11216f361f Fix bug and some re-structure 2018-03-13 23:24:51 -03:00
Christian Schabesberger 5e34556ac3 removed more nextStreams keywords 2018-03-13 23:24:51 -03:00
Christian Schabesberger dc0ef3b89a remove unused ExtractionExceptions 2018-03-13 23:24:51 -03:00
Christian Schabesberger d812a114c5 made getPage() function be stand alone 2018-03-13 23:24:51 -03:00
Mauricio Colli 4366b7321f
Fail-fast in case of unsuccessful tests 2018-03-13 23:16:55 -03:00
Christian Schabesberger fce324d1bc fix playlist banner foo 2018-03-12 16:14:33 +01:00
Christian Schabesberger b1130629bb
Merge pull request #73 from karyogamy/live
HLS Livestream Extraction
2018-03-06 19:39:18 +01:00
John Zhen Mo cc3f3b8292 -Fixed Youtube page extraction on flagged / offensive content urls.
-Added test for flagged content url extraction.
2018-03-01 16:44:15 -08:00
John Zhen Mo b4206479cb -Fixed Youtube page extraction on flagged / offensive content urls. 2018-03-01 16:31:36 -08:00
John Zhen Mo 4cf4ee394f -Fixed youtube stream type to no longer return none.
-Fixed search result errors on empty.
2018-02-27 18:16:29 -08:00
John Zhen Mo 837dbd6b86 -Fixed live stream info using video stream type. 2018-02-25 14:31:42 -08:00
John Zhen Mo 690f241357 -Added hls manifest url extraction.
-Changed manifest url fields to nonnull.
2018-02-25 13:03:51 -08:00
John Zhen Mo e851c12230 -Removed live stream exceptions 2018-02-25 12:34:09 -08:00
Christian Schabesberger 86db415b18 architecture refacturing 2018-02-24 22:20:50 +01:00
Mauricio Colli 9dfcb3be06 Implement SubscriptionExtractor for services
- YouTube supports extracting from an export (from subscriptions manager)
- SoundCloud supports extracting the "followings" from an user
2018-02-22 11:52:38 -03:00
Christian Schabesberger a99f466c28
Merge pull request #72 from wb9688/extractor-to-parsing
Move SoundcloudExtractorHelper methods to SoundcloudParsingHelper
2018-02-21 10:45:21 +01:00
wb9688 c08cef9331 Move SoundcloudExtractorHelper methods to SoundcloudParsingHelper 2018-02-21 09:23:57 +01:00
Christian Schabesberger 7716b14378
Merge pull request #70 from wb9688/fix-okhttp
Fix OkHttp
2018-02-20 13:46:00 +01:00
wb9688 4ac70e6966 Fix OkHttp 2018-02-15 14:23:45 +01:00
Christian Schabesberger 232b8699d6 add contribution notes to extractor 2018-02-12 20:34:24 +01:00
Christian Schabesberger e51bc58a85 fix compatibility issue 2018-02-12 20:23:43 +01:00
Christian Schabesberger 09aca7cf90
Merge pull request #68 from TeamNewPipe/hook
add support for hooktube links
2018-02-12 20:06:06 +01:00
Christian Schabesberger 055cc41834 add support for hooktube links
fix some links
2018-02-12 20:04:55 +01:00
Mauricio Colli f3865445f5 Improve ServiceList class
- [Minor] Fix wrong arguments order in KioskInfo
2018-02-10 00:46:05 -02:00
Mauricio Colli 42411099db Revert and fix Idea's IML file 2018-02-10 00:37:40 -02:00
Mauricio Colli 4fb49d54b5
Merge pull request #66 from karyogamy/subtitles_fix
Subtitles Extraction NPE Fix
2018-02-09 00:42:25 -02:00
John Zhen Mo ccc8cde80e -Fix subtitles extraction NPE on certain video that contains no caption info.
-Moved subtitles to optional data extraction.
2018-02-06 10:45:58 -08:00
John Zhen Mo 7fd21ec085 -Modified subtitles to use locale rather than language code.
-Fixed locale parsing for youtube.
2018-02-03 13:53:03 -08:00
John Zhen Mo 02dd281395 -Added display name to subtitles that resolves language code into locale. 2018-02-03 13:28:28 -08:00
John Zhen Mo 502fd53cc4 -Refactored subtitle storing to reduce redundancy. 2018-02-01 23:24:22 -08:00
John Zhen Mo c1cfb2b223 -Fixed url format property.
-Removed unused code.
2018-02-01 23:03:06 -08:00
John Zhen Mo 5b79ef3557 -Fixed extraction to only output manual and autogenerated scripts, omitting autotranslated scripts. 2018-02-01 22:51:53 -08:00
John Zhen Mo aafe543334 -Removed new method in favor of current method.
-Fixed current method to output all available captions.
-Fixed tests.
2018-02-01 21:48:34 -08:00
John Zhen Mo b3699cdcfc -Fixed tests
-Simplified subtitle extraction url
2018-02-01 19:07:57 -08:00
John Zhen Mo 1f1bbaad57 -Fixed subtitles extraction to use method from youtube-dl.
-Expose subtitles during extraction.
-Make subtitle lists return nonnull empty collections instead of null.
2018-02-01 13:27:22 -08:00
Mauricio Colli 0061131d39
Merge pull request #60 from wb9688/fix-m
Fix mobile SoundCloud links
2018-01-21 05:11:30 -02:00
wb9688 6453805288 Fix mobile SoundCloud links 2018-01-20 18:48:15 +01:00
Christian Schabesberger 1c97da8b51 roleback fix warning of generics (broke frontend compartiblity
actual fix

another actual fix
2018-01-20 15:45:49 +01:00
Mauricio Colli 179c38c619
Merge pull request #59 from coffeemakr/bugfix-test-urls-and-relative-fix
Add more tests and fix 2 bugs
2018-01-15 11:31:44 -02:00
Coffeemakr 978a33fae2
Use 'abs:' prefix 2018-01-15 12:33:09 +01:00
Mauricio Colli a75137be2e Remove unnecessary check 2018-01-13 17:26:11 -02:00
Mauricio Colli a693c0d61d Fix getNextStreams from Youtube's channel extractor 2018-01-13 17:09:31 -02:00
Mauricio Colli cfd3ab3500 Fix warning of generics 2018-01-13 17:08:48 -02:00
Christian Schabesberger 9de63f8c0a horrible hack nextinfoitem for channel 2018-01-06 20:24:55 +01:00
Coffeemakr 7cfdaf39a4
Fix relative url in uploader URL 2018-01-04 18:48:15 +01:00
Coffeemakr 06ea74cbb8
Add better tests for youtube search items 2018-01-04 18:48:15 +01:00
Coffeemakr 0ffd4d9743
Improve tests for search results and reuse code 2018-01-04 18:47:27 +01:00
Coffeemakr fe2583f3d7
Generate and verify Https links 2018-01-04 18:47:27 +01:00
Christian Schabesberger 5f2d0cf6b5 fix yt trending contains to less items 2018-01-04 04:46:27 +01:00
Christian Schabesberger a0e5c88b13
Merge pull request #58 from karyogamy/playlist_info_fix
Playlist Info Fix
2018-01-04 04:33:07 +01:00
John Zhen Mo 32dbb3d2df -Fixed reversed order of inputs on playlist info. 2018-01-03 16:42:55 -08:00
Christian Schabesberger f40447fa40
Merge pull request #57 from karyogamy/channel_extractor_fix
Fix channel extractor error
2017-12-30 01:35:29 +01:00
John Zhen Mo c426140811 -Fix channel extractor error. 2017-12-29 16:13:24 -08:00
Schabi 044b8fe32f fix fetch page 2017-12-29 14:40:42 +01:00
Schabi c5ce8ec906 Merge branch 'master' of github.com:TeamNewPipe/NewPipeExtractor 2017-12-29 01:13:05 +01:00
Schabi 33bd9d9a7a fix kiosk content language 2017-12-29 01:12:55 +01:00
wb9688 eae7504526 Fix 'New & hot' chart (#55) 2017-12-24 23:15:40 +01:00
Schabi 2d191c4ca7 remove debug prints 2017-12-23 18:00:44 +01:00
John Zhen Mo e0315ca2af -Fixes age-gated video info retrieval.
-Fixes youtube video stream test passing even when no stream is fetched on age-gated videos.
2017-12-18 14:05:58 -08:00
Christian Schabesberger d6a00228e8 fix exception on found ads 2017-12-15 22:12:32 +01:00
Christian Schabesberger 647605c5a6 Merge branch 'fast-tests' of https://github.com/Jawnnypoo/NewPipeExtractor into fast 2017-12-07 13:46:43 +01:00
Christian Schabesberger 17ce9f537e
Merge pull request #53 from coffeemakr/bugfix-channel-extractor
Fix channel info item collector
2017-12-07 11:19:14 +01:00
Coffeemakr f76cc4922e
Fix channel info item collector 2017-12-07 09:53:18 +01:00
John 93d9469bf6 Before -> BeforeClass to speed up tests 2017-12-06 19:09:46 -06:00
John d70c0792f7 Ignore .iml files 2017-12-06 19:06:21 -06:00
Christian Schabesberger 4f62d96cc0
add documentation link 2017-12-02 20:54:17 +01:00
Christian Schabesberger 5cae16fa98
Merge pull request #51 from coffeemakr/patch-1
Update README.md
2017-12-02 20:28:10 +01:00
Cyril Müller 2d00b7180f
Update README.md
* Add JitPack badge with link to installation instructions
 * Use tag instead of commit for proposed usage
2017-12-01 14:15:36 +01:00
Christian Schabesberger a5ba482871 fix kiosk id test for germany 2017-11-30 16:33:57 +01:00
Christian Schabesberger 18e486da30 Merge branch 'bugfix-dont-call-fetch-page-in-constructor' of https://github.com/coffeemakr/NewPipeExtractor into f 2017-11-30 15:39:27 +01:00
Christian Schabesberger 1cb025e483
Merge pull request #50 from coffeemakr/master
Automatically deploy Javadocs
2017-11-30 15:38:14 +01:00
Coffeemakr 9842744f22
Add travis configuration 2017-11-30 12:24:51 +01:00
Coffeemakr f8d2794a1d
Fix javadoc errors. (warnings still not fixed) 2017-11-30 12:24:50 +01:00
Coffeemakr 4fa14b31b9
Remove unused imports 2017-11-30 11:39:09 +01:00
Coffeemakr 58df281975
Make some fields final 2017-11-30 11:39:03 +01:00
Coffeemakr 89f8678912
Fix age restriction test - make fields NOT STATIC 2017-11-30 11:21:22 +01:00
Coffeemakr 4b092828db
Correct typo 2017-11-30 10:51:00 +01:00
Coffeemakr f037f67a15
Add checks if page is fetched 2017-11-30 10:49:52 +01:00
Coffeemakr 23231732ca
Be more verbose 2017-11-30 10:49:52 +01:00
Coffeemakr 9a46992285
Don't call fetch page in constructor 2017-11-30 10:49:46 +01:00
Christian Schabesberger d84a64debd
Merge pull request #47 from coffeemakr/bugfix-correct-dependency-for-annotaions
Include stopbugs annotations only
2017-11-29 11:51:24 +01:00
Coffeemakr 665f38215d
Include stopbugs annotations only 2017-11-28 12:22:42 +01:00
Christian Schabesberger 5272468898
Merge pull request #46 from coffeemakr/bugfix-uploader-url
Fix uploader url for related streams
2017-11-26 20:33:34 +01:00
Coffeemakr 421550e0c4
Fix uploader url for related streams 2017-11-26 17:27:30 +01:00
Christian Schabesberger 16b77b56cc fix failing tests 2017-11-25 19:21:33 +01:00
Coffeemakr dacffda194
Override getUploaderUrl fro YoutubeTrendingExtractor 2017-11-25 04:00:19 +01:00
Coffeemakr d76c8e1773
Add check for subtitles in restricted videos 2017-11-25 03:45:03 +01:00
Coffeemakr 239df49df3
Fix some test source code 2017-11-25 03:42:31 +01:00
Coffeemakr d59618d0e7
Ignore deleted playlist items and add uploader url 2017-11-25 03:14:15 +01:00
Coffeemakr c9eb790d1c
Some tests extension 2017-11-25 03:13:26 +01:00
Coffeemakr 95120641a9
Fix channel url for channel items 2017-11-25 02:38:46 +01:00
Coffeemakr 9cd11a7b6b
Encode paramters and small changes 2017-11-25 02:20:40 +01:00
Coffeemakr b5b9f8645c
Convert assertTrue(!..) to assertFalse 2017-11-25 02:20:33 +01:00
Coffeemakr d4b72f539b
Add @Nonnull annotations 2017-11-25 02:03:30 +01:00
Coffeemakr d0a05706e3
Ignore get audio streams for age restricted video 2017-11-25 01:56:37 +01:00
Coffeemakr dc109d78f8
Fix bugs and present some in the future 2017-11-25 01:44:56 +01:00
Coffeemakr 775011db77
Make info item fields final 2017-11-25 01:44:56 +01:00
Coffeemakr ceb556384b
Refactor collectors
* Define a common iterface
  * Use generic types
  * Remove some duplicated code
  * Simplify InfoItemSearchCollector and remove unused methods
  * SearchResult: Make fields final
2017-11-25 01:44:56 +01:00
Coffeemakr bc44557bdb
Add and use setters 2017-11-25 01:44:56 +01:00
Coffeemakr e11c6e35f6
Code refactoring
* Add getters
 * Add javadoc
 * Use Mediaformat instead of the id of media format
2017-11-25 01:44:11 +01:00
Christian Schabesberger 98358cb0f9 Merge branch 'feature-YTsubtitles' of https://github.com/tonakriz/NewPipeExtractor into refactor 2017-11-24 21:02:13 +01:00
tonakriz bd1914612f Added comment 2017-11-24 13:59:52 +01:00
tonakriz 2c7acc74f5 Fixed exceptions as requested 2017-11-24 13:57:54 +01:00
Christian Schabesberger 3c9c5c1004 Merge branch 'channel-url' of https://github.com/wb9688/NewPipeExtractor into up 2017-11-24 12:22:43 +01:00
tonakriz 310b34558b Changed the way of getting subtitles data type, removed language name from Subtitles class 2017-11-23 16:33:03 +01:00
tonakriz 79e49ce374 Merge branch 'feature-YTsubtitles' of https://github.com/tonakriz/NewPipeExtractor into feature-YTsubtitles 2017-11-23 12:31:13 +01:00
tonakriz 320c0d8c99 Fixed typos in tests 2017-11-23 12:30:19 +01:00
tona_kriz d657e38cd2
Merge branch 'master' into feature-YTsubtitles 2017-11-23 12:26:35 +01:00
tona_kriz 57769788fc
Merge branch 'master' into feature-YTsubtitles 2017-11-23 11:54:27 +01:00
tonakriz f09b4e68d0 Using Class instead of HashMap, removed downloadSubtitles method 2017-11-23 11:47:08 +01:00
tonakriz 72f9ac223e Using Class instead of HashMap, removed downloadSubtitles method 2017-11-23 11:47:05 +01:00
tonakriz 6e3651fdf5 Added method to download subtitles as VTT 2017-11-23 00:10:12 +01:00
Christian Schabesberger b37dbbf424
Merge pull request #43 from tonakriz/refactor
Possible removation of duplicated code
2017-11-22 20:42:39 +01:00
tonakriz 3902de00a6 Removed duplicated code 2017-11-22 18:45:49 +01:00
tonakriz 6031695343 Addede fetching subtitles for youtube and its tests 2017-11-22 18:39:38 +01:00
Coffeemakr acf05b748c
Add uploaderUrl to StreamInfoItem 2017-11-11 12:46:14 +01:00
wb9688 6e42cb87e8 Add support for getting uploader URL 2017-11-08 10:19:55 +01:00
Christian Schabesberger b9d0941411 fix N.pointer exception for getUploaderName() 2017-10-31 14:10:52 +01:00
Christian Schabesberger 8baeb3dcae
Merge pull request #37 from jlelse/master
Updated Gradle wrapper
2017-10-29 19:52:39 +01:00
Christian Schabesberger 7899cd1323 fix Kisok spelling error 2017-10-22 21:44:24 +02:00
Jan-Lk Else 4764bca9c6 Updated Gradle wrapper 2017-10-14 11:28:39 +02:00
Mauricio Colli 1df3f67074 Fix YouTube's channel extractor 2017-09-26 17:46:21 -03:00
Christian Schabesberger 466d87ceb4 remove type from kiosk and make getName() crawl the translated kiosk name 2017-09-25 12:43:40 +02:00
Christian Schabesberger 7beb90bf8a Merge pull request #34 from wb9688/readme
[WiP] Improve README
2017-09-23 17:55:45 +02:00
Christian Schabesberger 7fffef5d4f messing kiosk up in order to make it work 2017-09-23 17:14:21 +02:00
Christian Schabesberger b37b24511f set default kiosk for youtube 2017-09-23 15:59:00 +02:00
Christian Schabesberger 78d461ed89 improve kiosk function
improve kiosk function
2017-09-23 15:48:08 +02:00
wb9688 bcae90d7ed Add sentence about it being a core component of NewPipe 2017-09-23 14:53:54 +02:00
wb9688 d459ab546c Improve instructions for adding this as a dependency 2017-09-22 11:04:22 +02:00
Christian Schabesberger 82ae72900a add license to readme 2017-09-21 23:24:25 +02:00
Christian Schabesberger 0839eae916 Add maven link to readme 2017-09-21 23:23:13 +02:00
Christian Schabesberger 0774eff775 Merge pull request #32 from wb9688/fix-soundcloud
Fix SoundCloud
2017-09-16 13:50:10 +02:00
wb9688 306e31a2a4 Fix likes count of SoundCloud 2017-09-16 11:08:08 +02:00
wb9688 b93910081f Fix NullPointerException 2017-09-16 10:23:20 +02:00
Mauricio Colli 4a91e29dcf Merge pull request #31 from dosvald/yt-search-suggestions
Change search suggestions to match what is seen at youtube.com
2017-09-15 19:27:09 -03:00
dosvald 24a8df7c49 Change search suggestions to match what is seen at youtube.com 2017-09-15 23:50:54 +02:00
Mauricio Colli 841955a85e Fix disabled likes/dislikes of SoundCloud 2017-09-15 14:36:43 -03:00
Mauricio Colli e9156ee499 Fix parsing of SoundCloud stream info items 2017-09-11 13:18:17 -03:00
Christian Schabesberger 0c632e195b Merge pull request #27 from TeamNewPipe/kiosk
add Kiosk support
2017-09-11 16:26:25 +02:00
Mauricio Colli 7ae274b299 Fix uploader name when requesting next streams 2017-09-11 11:04:32 -03:00
Mauricio Colli bd3db34092 Ignore YouTube mix when searching
- We can't extract them properly (at least for now)
2017-09-11 11:04:06 -03:00
Mauricio Colli 390e5727d6 Reduce bandwidth consume by StreamExtractor
- Don't download the not needed info page
- The difference is very small (~20-30 KB)
2017-09-11 10:18:13 -03:00
Christian Schabesberger f64c82f04c Merge pull request #28 from wb9688/kiosk
Use content country if supported by SoundCloud
2017-09-10 15:52:26 +02:00
Christian Schabesberger 725b35ef20 merge with changes from master 2017-09-10 15:49:32 +02:00
Christian Schabesberger f308755359 try YoutubeChannelExtractor.getName() hack try 2 2017-09-09 22:11:16 +02:00
Christian Schabesberger a6da202438 try YoutubeChannelExtractor.getName() hack 2017-09-09 21:56:27 +02:00
Christian Schabesberger 540237d8fc make YoutubeChannelExtractor.getName() exception more expose 2017-09-09 21:42:42 +02:00
Mauricio Colli 6cdaef53f9 Youtube playlist's id minimum length regex 2017-09-08 11:22:49 -03:00
Christian Schabesberger 5b25ded1b7 Merge pull request #30 from karyogamy/id_fix
Fix Playlist id extraction when length is not 34
2017-09-08 16:14:09 +02:00
John Zhen M 3c1ccb5b8a Fix Playlist id extraction when length is less than 34 2017-09-08 07:03:37 -07:00
Mauricio Colli 13fab09810 Fix encrypted signature error
- Closes #29
2017-09-08 10:45:18 -03:00
Mauricio Colli 44c80e68db Remove unnecessary error when dashMpd fails
- Also fix some failing tests
2017-09-08 10:43:40 -03:00
Christian Schabesberger 97ad1a2052 add NewPipeExtractor.iml 2017-08-29 14:08:42 +02:00
Christian Schabesberger e1efbcc2c9 make this gradle module loadable to other projects 2017-08-28 17:33:29 +02:00
wb9688 d07f6cdef7 Use content country if supported by SoundCloud 2017-08-22 14:47:51 +02:00
wb9688 e2b7cb9c69 Add SoundcloudChartsExtractor 2017-08-20 10:11:00 +02:00
Mauricio Colli 34547a37b6 Fix YouTubeChannelExtractor
It was given some inconsistent results (/user and /channel), now it only returns /channel urls don't matter what the original is (at least when calling the getCleanUrl() method).
2017-08-15 23:59:19 -03:00
Mauricio Colli a8a4eaf81b Change Json dependency again
The other had some incompatibilities with android.
2017-08-15 23:40:03 -03:00
Mauricio Colli aff595c40e Merge pull request #25 from karyogamy/playlist_info
Playlist info extraction for search engine
2017-08-15 22:23:56 -03:00
John Zhen M 645ee5a5ff -Added playlist info item extraction for Youtube and Soundcloud.
-Added playlist collection into search engine.
-Fixed stream info duration exception when parsing 0 second video.
2017-08-15 17:11:38 -07:00
Christian Schabesberger 5119dabb63 add contentCountry to Kiosk 2017-08-14 12:48:51 +02:00
Christian Schabesberger c76f39c81b fix minor details 2017-08-13 01:10:22 +02:00
Christian Schabesberger 7b7f6d2cbf made YoutubeTrendingExtractor work 2017-08-13 00:58:29 +02:00
Christian Schabesberger 88d2fff091 add test for YoutubeTrendingExtractor 2017-08-12 23:19:35 +02:00
Christian Schabesberger 8dabda293b add test for YoutubeTrendingUrlIdHandler 2017-08-12 23:03:34 +02:00
Mauricio Colli 9184fc509c Store errors that can happen during extraction of the next page
- Closes #24
2017-08-12 17:19:56 -03:00
Christian Schabesberger b89f5a9b42 add youtube trending extractor 2017-08-12 21:10:21 +02:00
Christian Schabesberger a6c1728dac add copyright header to kiosklist 2017-08-12 17:35:43 +02:00
Christian Schabesberger 316b479199 add basic setup for kiosk 2017-08-12 17:29:28 +02:00
Mauricio Colli 82824cdd72 Create InfoItemExtractor 2017-08-11 15:21:49 -03:00
Mauricio Colli b719e59fae Change json dependency
The previous was conflicting with the built-in version of android, this should solve it as it has a different package name (with the advantage of being able to receive updates).
2017-08-11 14:29:11 -03:00
Mauricio Colli 03634a2b6a Rollback classes naming 2017-08-10 22:23:09 -03:00
Mauricio Colli 8d88093eb7 Fix getRealIdFromShared occasionally failing 2017-08-10 19:59:41 -03:00
Mauricio Colli c4f521fbb4 Fix bugs and improve InfoItemExtractors
- Improve livestream detection
2017-08-10 14:50:59 -03:00
Mauricio Colli 5bf2e95d7b Renaming and new fields/methods
- Mainly in PlaylistInfoItem and the collector
2017-08-09 23:50:29 -03:00
Mauricio Colli b1eefa040e Build sources for jitpack.io 2017-08-09 10:25:02 -03:00
Mauricio Colli 2e5e4e91c9 Revert recent travis changes 2017-08-08 18:58:25 -03:00
Mauricio Colli 01eaefdbdf Use Java 7 on travis 2017-08-08 18:52:54 -03:00
Mauricio Colli 5ebb5301c9 Rename some variables and methods 2017-08-08 18:36:11 -03:00
Mauricio Colli 1a2a6df7ec Merge pull request #22 from wb9688/soundcloud-test
Add tests for SoundCloud
2017-08-08 12:10:10 -03:00
Mauricio Colli 81c2a3cb39 Fix some tests
- Use JSONObject#optString instead of JSONObject#getString to some optional values
- Check if url match the pattern before doing anything (for getId and cleanUrl in the url handlers)
- Ignore the suggestion tests
2017-08-08 12:01:50 -03:00
wb9688 bf845d85aa Add tests for SoundCloud 2017-08-08 08:42:28 +02:00
Mauricio Colli 845c16b5a0 Merge pull request #21 from karyogamy/channel_description
Added description parsing for channel info.
2017-08-07 14:13:03 -03:00
John Zhen M e0f1b5cd79 Added description parsing for channel info. 2017-08-07 10:00:36 -07:00
Mauricio Colli 28d12cd90c Merge pull request #20 from wb9688/rename
Rename some stuff
2017-08-07 13:33:17 -03:00
wb9688 c33ee3e409 Rename some stuff 2017-08-07 18:19:10 +02:00
Mauricio Colli cfc2b9b3e2 Merge pull request #19 from mauriciocolli/refactor
Refactor and improvements
2017-08-06 18:07:39 -03:00
Mauricio Colli ea5b08db4c Refactor and improvements 2017-08-06 17:20:15 -03:00
Mauricio Colli 731102be39 Update README.md 2017-08-05 11:36:38 -03:00
Mauricio Colli b01ad7f864 Merge pull request #17 from wb9688/gradle
Use Gradle
2017-08-05 11:25:00 -03:00
Mauricio Colli 171c00a891 Try to fix tests
- Fix double youtube package
2017-08-05 11:15:36 -03:00
wb9688 d88fe691cf Use Gradle 2017-08-05 10:03:56 +02:00
wb9688 f314bec396 Add SoundCloud service (#15)
Add SoundCloud service
2017-08-04 11:21:45 -03:00
Mauricio Colli 12bfdf5234 Change dash parser exception handling 2017-07-12 13:54:48 -03:00
Mauricio Colli b5b25a4188 Merge pull request #14 from mauriciocolli/refactor-extractor
Fix dash parser and more refactor
2017-07-11 22:26:48 -03:00
Mauricio Colli b1989c0a83 Fix dash parser and more refactor
- Add new itags
2017-07-11 11:42:38 -03:00
Mauricio Colli bda65e83d6 Refactor Extractor
- Renaming and removal of duplicate code
- New base class for list extractors
2017-07-09 19:56:35 -03:00
Mauricio Colli 729930802e Merge pull request #11 from mauriciocolli/refactor-extractor
Refactor extractor
2017-07-09 19:11:01 -03:00
Mauricio Colli 21e542e7d2 Refactor extractor
- Refactor info classes and extractors
- Reformat and fix indentation
- Organize packages and classes
- Rename variables/methods and fix regex
- Change the capitalization
- Add methods to playlist extractor
2017-06-29 15:12:55 -03:00
Coffeemaker 7581c1200b Fix typos (#12) 2017-06-29 19:34:21 +02:00
901 changed files with 189569 additions and 4527 deletions

3
.github/PULL_REQUEST_TEMPLATE.md vendored Normal file
View File

@ -0,0 +1,3 @@
- [ ] I carefully read the [contribution guidelines](https://github.com/TeamNewPipe/NewPipe/blob/HEAD/.github/CONTRIBUTING.md) and agree to them.
- [ ] I have tested the API against [NewPipe](https://github.com/TeamNewPipe/NewPipe).
- [ ] I agree to create a pull request for [NewPipe](https://github.com/TeamNewPipe/NewPipe) as soon as possible to make it compatible with the changed API.

12
.github/dependabot.yml vendored Normal file
View File

@ -0,0 +1,12 @@
version: 2
updates:
# Maintain dependencies for Gradle
- package-ecosystem: "gradle"
directory: "/"
schedule:
interval: "daily"
# Maintain dependencies for GitHub Actions
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "daily"

51
.github/workflows/ci.yml vendored Normal file
View File

@ -0,0 +1,51 @@
name: CI
on:
schedule:
# once per day
- cron: 0 0 * * *
push:
branches:
- dev
- master
pull_request:
permissions:
contents: read
jobs:
build-and-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: set up JDK 11
uses: actions/setup-java@v4
with:
java-version: '11'
distribution: 'temurin'
- name: Cache Gradle dependencies
uses: actions/cache@v4
with:
path: ~/.gradle/caches
key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle') }}
restore-keys: ${{ runner.os }}-gradle
# See gradle file for difference between downloaders
- name: Build and run Tests
run: |
if [[ "$GITHUB_EVENT_NAME" == 'schedule' ]]; then
echo running with real downloader
./gradlew check --stacktrace -Ddownloader=REAL
else
echo running with mock downloader
./gradlew check --stacktrace -Ddownloader=MOCK
fi
- name: Upload test reports when failure occurs
uses: actions/upload-artifact@v4
if: failure()
with:
name: NewPipeExtractor-test-reports
path: extractor/build/reports/tests/test/**

38
.github/workflows/docs.yml vendored Normal file
View File

@ -0,0 +1,38 @@
name: Build and deploy JavaDocs
on:
push:
branches:
- master
permissions:
# The generated docs are written to the `gh-pages` branch.
contents: write
jobs:
build-and-deploy-docs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: set up JDK 11
uses: actions/setup-java@v4
with:
java-version: '11'
distribution: 'temurin'
- name: Cache Gradle dependencies
uses: actions/cache@v4
with:
path: ~/.gradle/caches
key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle') }}
restore-keys: ${{ runner.os }}-gradle
- name: Build JavaDocs
run: ./gradlew aggregatedJavadocs
- name: Deploy JavaDocs
uses: peaceiris/actions-gh-pages@v4
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: ./build/docs

26
.gitignore vendored Normal file
View File

@ -0,0 +1,26 @@
.gradle
build/
.idea
local.properties
out/
*.iml
# Ignore Gradle GUI config
gradle-app.setting
# Avoid ignoring Gradle wrapper jar file (.jar files are usually ignored)
!gradle-wrapper.jar
# Cache of project
.gradletasknamecache
# # Work around https://youtrack.jetbrains.com/issue/IDEA-116898
# gradle/wrapper/gradle-wrapper.properties
# vscode / eclipse files
*.classpath
*.project
*.settings
**/bin
**.vscode
*.code-workspace

View File

@ -1,43 +0,0 @@
package org.schabi.newpipe.extractor;
/**
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* AbstractStreamInfo.java is part of NewPipe.
*
* NewPipe is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
*/
import java.io.Serializable;
/**Common properties between StreamInfo and StreamInfoItem.*/
public abstract class AbstractStreamInfo implements Serializable{
public enum StreamType {
NONE, // placeholder to check if stream type was checked or not
VIDEO_STREAM,
AUDIO_STREAM,
LIVE_STREAM,
AUDIO_LIVE_STREAM,
FILE
}
public StreamType stream_type;
public int service_id = -1;
public String id = "";
public String title = "";
public String uploader = "";
public String thumbnail_url = "";
public String webpage_url = "";
public String upload_date = "";
public long view_count = -1;
}

View File

@ -1,93 +0,0 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.exceptions.ReCaptchaException;
import org.schabi.newpipe.extractor.stream_info.AudioStream;
import org.w3c.dom.Document;
import org.w3c.dom.Element;
import org.w3c.dom.NodeList;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.List;
import java.util.Vector;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
/**
* Created by Christian Schabesberger on 02.02.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* DashMpdParser.java is part of NewPipe.
*
* NewPipe is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
*/
public class DashMpdParser {
private DashMpdParser() {
}
static class DashMpdParsingException extends ParsingException {
DashMpdParsingException(String message, Exception e) {
super(message, e);
}
}
public static List<AudioStream> getAudioStreams(String dashManifestUrl)
throws DashMpdParsingException, ReCaptchaException {
String dashDoc;
Downloader downloader = NewPipe.getDownloader();
try {
dashDoc = downloader.download(dashManifestUrl);
} catch(IOException ioe) {
throw new DashMpdParsingException("Could not get dash mpd: " + dashManifestUrl, ioe);
} catch (ReCaptchaException e) {
throw new ReCaptchaException("reCaptcha Challenge needed");
}
Vector<AudioStream> audioStreams = new Vector<>();
try {
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
DocumentBuilder builder = factory.newDocumentBuilder();
InputStream stream = new ByteArrayInputStream(dashDoc.getBytes());
Document doc = builder.parse(stream);
NodeList adaptationSetList = doc.getElementsByTagName("AdaptationSet");
for(int i = 0; i < adaptationSetList.getLength(); i++) {
Element adaptationSet = (Element) adaptationSetList.item(i);
String memeType = adaptationSet.getAttribute("mimeType");
if(memeType.contains("audio")) {
Element representation = (Element) adaptationSet.getElementsByTagName("Representation").item(0);
String url = representation.getElementsByTagName("BaseURL").item(0).getTextContent();
int bandwidth = Integer.parseInt(representation.getAttribute("bandwidth"));
int samplingRate = Integer.parseInt(representation.getAttribute("audioSamplingRate"));
int format = -1;
if(memeType.equals(MediaFormat.WEBMA.mimeType)) {
format = MediaFormat.WEBMA.id;
} else if(memeType.equals(MediaFormat.M4A.mimeType)) {
format = MediaFormat.M4A.id;
}
audioStreams.add(new AudioStream(url, format, 0, bandwidth, samplingRate));
}
}
}
catch(Exception e) {
throw new DashMpdParsingException("Could not parse Dash mpd", e);
}
return audioStreams;
}
}

View File

@ -1,52 +0,0 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.exceptions.ReCaptchaException;
import java.io.IOException;
import java.util.Map;
/**
* Created by Christian Schabesberger on 28.01.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* Downloader.java is part of NewPipe.
*
* NewPipe is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
*/
public interface Downloader {
/**Download the text file at the supplied URL as in download(String),
* but set the HTTP header field "Accept-Language" to the supplied string.
* @param siteUrl the URL of the text file to return the contents of
* @param language the language (usually a 2-character code) to set as the preferred language
* @return the contents of the specified text file
* @throws IOException*/
String download(String siteUrl, String language) throws IOException, ReCaptchaException;
/**Download the text file at the supplied URL as in download(String),
* but set the HTTP header field "Accept-Language" to the supplied string.
* @param siteUrl the URL of the text file to return the contents of
* @param customProperties set request header properties
* @return the contents of the specified text file
* @throws IOException*/
String download(String siteUrl, Map<String, String> customProperties) throws IOException, ReCaptchaException;
/**Download (via HTTP) the text file located at the supplied URL, and return its contents.
* Primarily intended for downloading web pages.
* @param siteUrl the URL of the text file to download
* @return the contents of the specified text file
* @throws IOException*/
String download(String siteUrl) throws IOException, ReCaptchaException;
}

View File

@ -1,35 +0,0 @@
package org.schabi.newpipe.extractor;
import java.io.Serializable;
/**
* Created by the-scrabi on 11.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* InfoItem.java is part of NewPipe.
*
* NewPipe is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
*/
public interface InfoItem extends Serializable{
enum InfoType {
STREAM,
PLAYLIST,
CHANNEL
}
InfoType infoType();
String getTitle();
String getLink();
}

View File

@ -1,61 +0,0 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import java.util.List;
import java.util.Vector;
/**
* Created by Christian Schabesberger on 12.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* InfoItemCollector.java is part of NewPipe.
*
* NewPipe is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
*/
public class InfoItemCollector {
private List<InfoItem> itemList = new Vector<>();
private List<Throwable> errors = new Vector<>();
private int serviceId = -1;
public InfoItemCollector(int serviceId) {
this.serviceId = serviceId;
}
public List<InfoItem> getItemList() {
return itemList;
}
public List<Throwable> getErrors() {
return errors;
}
protected void addFromCollector(InfoItemCollector otherC) throws ExtractionException {
if(serviceId != otherC.serviceId) {
throw new ExtractionException("Service Id does not equal: "
+ NewPipe.getNameOfService(serviceId)
+ " and " + NewPipe.getNameOfService(otherC.serviceId));
}
errors.addAll(otherC.errors);
itemList.addAll(otherC.itemList);
}
protected void addError(Exception e) {
errors.add(e);
}
protected void addItem(InfoItem item) {
itemList.add(item);
}
protected int getServiceId() {
return serviceId;
}
}

View File

@ -1,7 +1,7 @@
GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
@ -645,7 +645,7 @@ the "copyright" line and a pointer to where the full notice is found.
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
@ -664,11 +664,11 @@ might be different; for a GUI interface, you would use an "about box".
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU GPL, see
<http://www.gnu.org/licenses/>.
<https://www.gnu.org/licenses/>.
The GNU General Public License does not permit incorporating your program
into proprietary programs. If your program is a subroutine library, you
may consider it more useful to permit linking proprietary applications with
the library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License. But first, please read
<http://www.gnu.org/philosophy/why-not-lgpl.html>.
<https://www.gnu.org/licenses/why-not-lgpl.html>.

View File

@ -1,83 +0,0 @@
package org.schabi.newpipe.extractor;
/**
* Created by Adam Howard on 08/11/15.
*
* Copyright (c) Christian Schabesberger <chris.schabesberger@mailbox.org>
* and Adam Howard <achdisposable1@gmail.com> 2015
*
* MediaFormat.java is part of NewPipe.
*
* NewPipe is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
*/
/**Static data about various media formats support by Newpipe, eg mime type, extension*/
public enum MediaFormat {
//video and audio combined formats
// id name suffix mime type
MPEG_4 (0x0, "MPEG-4", "mp4", "video/mp4"),
v3GPP (0x1, "3GPP", "3gp", "video/3gpp"),
WEBM (0x2, "WebM", "webm", "video/webm"),
// audio formats
M4A (0x3, "m4a", "m4a", "audio/mp4"),
WEBMA (0x4, "WebM", "webm", "audio/webm");
public final int id;
@SuppressWarnings("WeakerAccess")
public final String name;
@SuppressWarnings("WeakerAccess")
public final String suffix;
public final String mimeType;
MediaFormat(int id, String name, String suffix, String mimeType) {
this.id = id;
this.name = name;
this.suffix = suffix;
this.mimeType = mimeType;
}
/**Return the friendly name of the media format with the supplied id
* @param ident the id of the media format. Currently an arbitrary, NewPipe-specific number.
* @return the friendly name of the MediaFormat associated with this ids,
* or an empty String if none match it.*/
public static String getNameById(int ident) {
for (MediaFormat vf : MediaFormat.values()) {
if(vf.id == ident) return vf.name;
}
return "";
}
/**Return the file extension of the media format with the supplied id
* @param ident the id of the media format. Currently an arbitrary, NewPipe-specific number.
* @return the file extension of the MediaFormat associated with this ids,
* or an empty String if none match it.*/
public static String getSuffixById(int ident) {
for (MediaFormat vf : MediaFormat.values()) {
if(vf.id == ident) return vf.suffix;
}
return "";
}
/**Return the MIME type of the media format with the supplied id
* @param ident the id of the media format. Currently an arbitrary, NewPipe-specific number.
* @return the MIME type of the MediaFormat associated with this ids,
* or an empty String if none match it.*/
public static String getMimeById(int ident) {
for (MediaFormat vf : MediaFormat.values()) {
if(vf.id == ident) return vf.mimeType;
}
return "";
}
}

View File

@ -1,88 +0,0 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.services.youtube.YoutubeService;
/**
* Created by Christian Schabesberger on 23.08.15.
*
* Copyright (C) Christian Schabesberger 2015 <chris.schabesberger@mailbox.org>
* NewPipe.java is part of NewPipe.
*
* NewPipe is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
*/
/**Provides access to the video streaming services supported by NewPipe.
* Currently only Youtube until the API becomes more stable.*/
@SuppressWarnings("ALL")
public class NewPipe {
private NewPipe() {
}
private static final String TAG = NewPipe.class.toString();
private static Downloader downloader = null;
public static StreamingService[] getServices() {
return ServiceList.serviceList;
}
public static StreamingService getService(int serviceId)throws ExtractionException {
for(StreamingService s : ServiceList.serviceList) {
if(s.getServiceId() == serviceId) {
return s;
}
}
return null;
}
public static StreamingService getService(String serviceName) throws ExtractionException {
return ServiceList.serviceList[getIdOfService(serviceName)];
}
public static String getNameOfService(int id) {
try {
return getService(id).getServiceInfo().name;
} catch (Exception e) {
System.err.println("Service id not known");
e.printStackTrace();
return "";
}
}
public static int getIdOfService(String serviceName) {
for(int i = 0; i < ServiceList.serviceList.length; i++) {
if(ServiceList.serviceList[i].getServiceInfo().name.equals(serviceName)) {
return i;
}
}
return -1;
}
public static void init(Downloader d) {
downloader = d;
}
public static Downloader getDownloader() {
return downloader;
}
public static StreamingService getServiceByUrl(String url) {
for(StreamingService s : ServiceList.serviceList) {
if(s.getLinkTypeByUrl(url) != StreamingService.LinkType.NONE) {
return s;
}
}
return null;
}
}

View File

@ -1,77 +0,0 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import java.io.UnsupportedEncodingException;
import java.net.URLDecoder;
import java.util.HashMap;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
/**
* Created by Christian Schabesberger on 02.02.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* Parser.java is part of NewPipe.
*
* NewPipe is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
*/
/** avoid using regex !!! */
public class Parser {
private Parser() {
}
public static class RegexException extends ParsingException {
public RegexException(String message) {
super(message);
}
}
public static String matchGroup1(String pattern, String input) throws RegexException {
return matchGroup(pattern, input, 1);
}
public static String matchGroup(String pattern, String input, int group) throws RegexException {
Pattern pat = Pattern.compile(pattern);
Matcher mat = pat.matcher(input);
boolean foundMatch = mat.find();
if (foundMatch) {
return mat.group(group);
}
else {
//Log.e(TAG, "failed to find pattern \""+pattern+"\" inside of \""+input+"\"");
if(input.length() > 1024) {
throw new RegexException("failed to find pattern \""+pattern);
} else {
throw new RegexException("failed to find pattern \"" + pattern + " inside of " + input + "\"");
}
}
}
public static Map<String, String> compatParseMap(final String input) throws UnsupportedEncodingException {
Map<String, String> map = new HashMap<>();
for(String arg : input.split("&")) {
String[] splitArg = arg.split("=");
if(splitArg.length > 1) {
map.put(splitArg[0], URLDecoder.decode(splitArg[1], "UTF-8"));
} else {
map.put(splitArg[0], "");
}
}
return map;
}
}

View File

@ -1,7 +1,65 @@
NewPipe Extractor
=================
# NewPipe Extractor
This is the system independent core of [NewPipe](https://github.com/TeamNewPipe/NewPipe).
It can be used to create your own java based NewPipe client.
[![CI](https://github.com/TeamNewPipe/NewPipeExtractor/actions/workflows/ci.yml/badge.svg?branch=dev&event=schedule)](https://github.com/TeamNewPipe/NewPipeExtractor/actions/workflows/ci.yml) [![JIT Pack Badge](https://jitpack.io/v/TeamNewPipe/NewPipeExtractor.svg)](https://jitpack.io/#TeamNewPipe/NewPipeExtractor) [JDoc](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/) • [Documentation](https://teamnewpipe.github.io/documentation/)
A small exmaple showing how this extractor can be used can be found [here](https://github.com/TeamNewPipe/np-cli).
NewPipe Extractor is a library for extracting things from streaming sites. It is a core component of [NewPipe](https://github.com/TeamNewPipe/NewPipe), but could be used independently.
## Usage
NewPipe Extractor is available at JitPack's Maven repo.
If you're using Gradle, you could add NewPipe Extractor as a dependency with the following steps:
1. Add `maven { url 'https://jitpack.io' }` to the `repositories` in your `build.gradle`.
2. Add `implementation 'com.github.TeamNewPipe:NewPipeExtractor:INSERT_VERSION_HERE'` to the `dependencies` in your `build.gradle`. Replace `INSERT_VERSION_HERE` with the [latest release](https://github.com/TeamNewPipe/NewPipeExtractor/releases/latest).
3. If you are using tools to minimize your project, make sure to keep the files below, by e.g. adding the following lines to your proguard file:
```
## Rules for NewPipeExtractor
-keep class org.schabi.newpipe.extractor.timeago.patterns.** { *; }
-keep class org.mozilla.javascript.** { *; }
-keep class org.mozilla.classfile.ClassFileWriter
-dontwarn org.mozilla.javascript.tools.**
```
**Note:** To use NewPipe Extractor in Android projects with a `minSdk` below 26, [API desugaring](https://developer.android.com/studio/write/java8-support#library-desugaring) is required. If the `minSdk` is below 19, the `desugar_jdk_libs_nio` artifact is required, which requires Android Gradle Plugin (AGP) version 7.4.0.
### Testing changes
To test changes quickly you can build the library locally. A good approach would be to add something like the following to your `settings.gradle`:
```groovy
includeBuild('../NewPipeExtractor') {
dependencySubstitution {
substitute module('com.github.TeamNewPipe:NewPipeExtractor') with project(':extractor')
}
}
```
Another approach would be to use the local Maven repository, here's a gist of how to use it:
1. Add `mavenLocal()` in your project `repositories` list (usually as the first entry to give priority above the others).
2. It's _recommended_ that you change the `version` of this library (e.g. `LOCAL_SNAPSHOT`).
3. Run gradle's `ìnstall` task to deploy this library to your local repository (using the wrapper, present in the root of this project: `./gradlew install`)
4. Change the dependency version used in your project to match the one you chose in step 2 (`implementation 'com.github.TeamNewPipe:NewPipeExtractor:LOCAL_SNAPSHOT'`)
> Tip for Android Studio users: After you make changes and run the `install` task, use the menu option `File → "Sync with File System"` to refresh the library in your project.
## Supported sites
The following sites are currently supported:
- YouTube
- SoundCloud
- media.ccc.de
- PeerTube (no P2P)
- Bandcamp
## License
[![GNU GPLv3 Image](https://www.gnu.org/graphics/gplv3-127x51.png)](https://www.gnu.org/licenses/gpl-3.0.en.html)
NewPipe Extractor is Free Software: You can use, study share and improve it at your
will. Specifically you can redistribute and/or modify it under the terms of the
[GNU General Public License](https://www.gnu.org/licenses/gpl.html) as
published by the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

View File

@ -1,13 +0,0 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.services.youtube.YoutubeService;
/**
* Created by the-scrabi on 18.02.17.
*/
class ServiceList {
public static final StreamingService[] serviceList = {
new YoutubeService(0)
};
}

View File

@ -1,85 +0,0 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.channel.ChannelExtractor;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.playlist.PlayListExtractor;
import org.schabi.newpipe.extractor.search.SearchEngine;
import org.schabi.newpipe.extractor.stream_info.StreamExtractor;
import java.io.IOException;
/**
* Created by Christian Schabesberger on 23.08.15.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* StreamingService.java is part of NewPipe.
*
* NewPipe is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
*/
public abstract class StreamingService {
public class ServiceInfo {
public String name = "";
}
public enum LinkType {
NONE,
STREAM,
CHANNEL,
PLAYLIST
}
private int serviceId;
public StreamingService(int id) {
serviceId = id;
}
public abstract ServiceInfo getServiceInfo();
public abstract StreamExtractor getExtractorInstance(String url)
throws IOException, ExtractionException;
public abstract SearchEngine getSearchEngineInstance();
public abstract UrlIdHandler getStreamUrlIdHandlerInstance();
public abstract UrlIdHandler getChannelUrlIdHandlerInstance();
public abstract UrlIdHandler getPlayListUrlIdHandlerInstance();
public abstract ChannelExtractor getChannelExtractorInstance(String url, int page)
throws ExtractionException, IOException;
public abstract PlayListExtractor getPlayListExtractorInstance(String url, int page)
throws ExtractionException, IOException;
public abstract SuggestionExtractor getSuggestionExtractorInstance();
public final int getServiceId() {
return serviceId;
}
/**
* figure out where the link is pointing to (a channel, video, playlist, etc.)
*/
public final LinkType getLinkTypeByUrl(String url) {
UrlIdHandler sH = getStreamUrlIdHandlerInstance();
UrlIdHandler cH = getChannelUrlIdHandlerInstance();
UrlIdHandler pH = getPlayListUrlIdHandlerInstance();
if(sH.acceptUrl(url)) {
return LinkType.STREAM;
} else if(cH.acceptUrl(url)) {
return LinkType.CHANNEL;
} else if (pH.acceptUrl(url)) {
return LinkType.PLAYLIST;
} else {
return LinkType.NONE;
}
}
}

View File

@ -1,43 +0,0 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import java.io.IOException;
import java.util.List;
/**
* Created by Christian Schabesberger on 28.09.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* SuggestionExtractor.java is part of NewPipe.
*
* NewPipe is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
*/
public abstract class SuggestionExtractor {
private int serviceId;
public SuggestionExtractor(int serviceId) {
this.serviceId = serviceId;
}
public abstract List<String> suggestionList(
String query,String contentCountry)
throws ExtractionException, IOException;
public int getServiceId() {
return serviceId;
}
}

View File

@ -1,35 +0,0 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
/**
* Created by Christian Schabesberger on 26.07.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* UrlIdHandler.java is part of NewPipe.
*
* NewPipe is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
*/
public interface UrlIdHandler {
String getUrl(String videoId);
String getId(String siteUrl) throws ParsingException;
String cleanUrl(String siteUrl) throws ParsingException;
/**When a VIEW_ACTION is caught this function will test if the url delivered within the calling
Intent was meant to be watched with this Service.
Return false if this service shall not allow to be called through ACTIONs.*/
boolean acceptUrl(String videoUrl);
}

83
build.gradle Normal file
View File

@ -0,0 +1,83 @@
allprojects {
apply plugin: 'java-library'
apply plugin: 'maven-publish'
compileJava.options.encoding = 'UTF-8'
compileTestJava.options.encoding = 'UTF-8'
sourceCompatibility = JavaVersion.VERSION_11
targetCompatibility = JavaVersion.VERSION_11
version 'v0.24.0'
group 'com.github.TeamNewPipe'
repositories {
mavenCentral()
maven { url "https://jitpack.io" }
}
afterEvaluate {
publishing {
publications {
mavenJava(MavenPublication) {
from components.java
}
}
}
}
ext {
nanojsonVersion = "1d9e1aea9049fc9f85e68b43ba39fe7be1c1f751"
spotbugsVersion = "4.8.5"
junitVersion = "5.10.2"
checkstyleVersion = "10.4"
}
}
dependencies {
api project(':extractor')
implementation project(':timeago-parser')
}
subprojects {
task sourcesJar(type: Jar, dependsOn: classes) {
archiveClassifier.set('sources')
from sourceSets.main.allSource
}
tasks.withType(Test) {
testLogging {
events "skipped", "failed"
showStandardStreams = true
exceptionFormat = 'full'
}
}
artifacts {
archives sourcesJar
}
}
// https://discuss.gradle.org/t/best-approach-gradle-multi-module-project-generate-just-one-global-javadoc/18657/21
task aggregatedJavadocs(type: Javadoc, group: 'Documentation') {
destinationDir = file("$buildDir/docs/javadoc")
title = "$project.name $version"
// options.memberLevel = JavadocMemberLevel.PRIVATE
options.links 'https://docs.oracle.com/javase/8/docs/api/'
options.encoding 'UTF-8'
// Fixes unknown tag @implNote; the other two were added precautionary
options.tags = [
"apiNote:a:API Note:",
"implSpec:a:Implementation Requirements:",
"implNote:a:Implementation Note:"
]
subprojects.each { project ->
project.tasks.withType(Javadoc).each { javadocTask ->
source += javadocTask.source
classpath += javadocTask.classpath
excludes += javadocTask.excludes
includes += javadocTask.includes
}
}
}

View File

@ -1,62 +0,0 @@
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.UrlIdHandler;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.stream_info.StreamInfoItemCollector;
import java.io.IOException;
/**
* Created by Christian Schabesberger on 25.07.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* ChannelExtractor.java is part of NewPipe.
*
* NewPipe is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
*/
public abstract class ChannelExtractor {
private int serviceId;
private String url;
private UrlIdHandler urlIdHandler;
private StreamInfoItemCollector previewInfoCollector;
private int page = -1;
public ChannelExtractor(UrlIdHandler urlIdHandler, String url, int page, int serviceId)
throws ExtractionException, IOException {
this.url = url;
this.page = page;
this.serviceId = serviceId;
this.urlIdHandler = urlIdHandler;
previewInfoCollector = new StreamInfoItemCollector(urlIdHandler, serviceId);
}
public String getUrl() { return url; }
public UrlIdHandler getUrlIdHandler() { return urlIdHandler; }
public StreamInfoItemCollector getStreamPreviewInfoCollector() {
return previewInfoCollector;
}
public abstract String getChannelName() throws ParsingException;
public abstract String getAvatarUrl() throws ParsingException;
public abstract String getBannerUrl() throws ParsingException;
public abstract String getFeedUrl() throws ParsingException;
public abstract StreamInfoItemCollector getStreams() throws ParsingException;
public abstract long getSubscriberCount() throws ParsingException;
public abstract boolean hasNextPage() throws ParsingException;
public int getServiceId() {
return serviceId;
}
}

View File

@ -1,86 +0,0 @@
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.stream_info.StreamInfoItemCollector;
import java.io.Serializable;
import java.util.List;
import java.util.Vector;
/**
* Created by Christian Schabesberger on 31.07.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* ChannelInfo.java is part of NewPipe.
*
* NewPipe is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
*/
public class ChannelInfo implements Serializable{
public void addException(Exception e) {
errors.add(e);
}
public static ChannelInfo getInfo(ChannelExtractor extractor)
throws ParsingException {
ChannelInfo info = new ChannelInfo();
// importand data
info.service_id = extractor.getServiceId();
info.channel_name = extractor.getChannelName();
info.hasNextPage = extractor.hasNextPage();
try {
info.avatar_url = extractor.getAvatarUrl();
} catch (Exception e) {
info.errors.add(e);
}
try {
info.banner_url = extractor.getBannerUrl();
} catch (Exception e) {
info.errors.add(e);
}
try {
info.feed_url = extractor.getFeedUrl();
} catch(Exception e) {
info.errors.add(e);
}
try {
StreamInfoItemCollector c = extractor.getStreams();
info.related_streams = c.getItemList();
info.errors.addAll(c.getErrors());
} catch(Exception e) {
info.errors.add(e);
}
try {
info.subscriberCount = extractor.getSubscriberCount();
} catch (Exception e) {
info.errors.add(e);
}
return info;
}
public int service_id = -1;
public String channel_name = "";
public String avatar_url = "";
public String banner_url = "";
public String feed_url = "";
public List<InfoItem> related_streams = null;
public long subscriberCount = -1;
public boolean hasNextPage = false;
public List<Throwable> errors = new Vector<>();
}

View File

@ -1,44 +0,0 @@
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.InfoItem;
/**
* Created by Christian Schabesberger on 11.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* ChannelInfoItem.java is part of NewPipe.
*
* NewPipe is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
*/
public class ChannelInfoItem implements InfoItem {
public int serviceId = -1;
public String channelName = "";
public String thumbnailUrl = "";
public String webPageUrl = "";
public String description = "";
public long subscriberCount = -1;
public int videoAmount = -1;
public InfoType infoType() {
return InfoType.CHANNEL;
}
public String getTitle() {
return channelName;
}
public String getLink() {
return webPageUrl;
}
}

View File

@ -1,74 +0,0 @@
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.InfoItemCollector;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.exceptions.FoundAdException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.stream_info.StreamInfoItem;
import org.schabi.newpipe.extractor.stream_info.StreamInfoItemExtractor;
/**
* Created by Christian Schabesberger on 12.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* ChannelInfoItemCollector.java is part of NewPipe.
*
* NewPipe is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
*/
public class ChannelInfoItemCollector extends InfoItemCollector {
public ChannelInfoItemCollector(int serviceId) {
super(serviceId);
}
public ChannelInfoItem extract(ChannelInfoItemExtractor extractor) throws ParsingException {
ChannelInfoItem resultItem = new ChannelInfoItem();
// importand information
resultItem.channelName = extractor.getChannelName();
resultItem.serviceId = getServiceId();
resultItem.webPageUrl = extractor.getWebPageUrl();
// optional information
try {
resultItem.subscriberCount = extractor.getSubscriberCount();
} catch (Exception e) {
addError(e);
}
try {
resultItem.videoAmount = extractor.getVideoAmount();
} catch (Exception e) {
addError(e);
}
try {
resultItem.thumbnailUrl = extractor.getThumbnailUrl();
} catch (Exception e) {
addError(e);
}
try {
resultItem.description = extractor.getDescription();
} catch (Exception e) {
addError(e);
}
return resultItem;
}
public void commit(ChannelInfoItemExtractor extractor) throws ParsingException {
try {
addItem(extract(extractor));
} catch (Exception e) {
addError(e);
}
}
}

195
checkstyle/checkstyle.xml Normal file
View File

@ -0,0 +1,195 @@
<?xml version="1.0"?>
<!DOCTYPE module PUBLIC
"-//Checkstyle//DTD Checkstyle Configuration 1.3//EN"
"https://checkstyle.org/dtds/configuration_1_3.dtd">
<module name="Checker">
<!--
If you set the basedir property below, then all reported file
names will be relative to the specified directory. See
https://checkstyle.org/5.x/config.html#Checker
<property name="basedir" value="${basedir}"/>
-->
<property name="severity" value="error"/>
<property name="fileExtensions" value="java, properties, xml"/>
<!-- Excludes all 'module-info.java' files -->
<!-- See https://checkstyle.org/config_filefilters.html -->
<module name="BeforeExecutionExclusionFileFilter">
<property name="fileNamePattern" value="module\-info\.java$"/>
</module>
<!-- Checks that a package-info.java file exists for each package. -->
<!-- See https://checkstyle.org/config_javadoc.html#JavadocPackage -->
<!--<module name="JavadocPackage"/>-->
<!-- Checks whether files end with a new line. -->
<!-- See https://checkstyle.org/config_misc.html#NewlineAtEndOfFile -->
<module name="NewlineAtEndOfFile"/>
<!-- Checks that property files contain the same keys. -->
<!-- See https://checkstyle.org/config_misc.html#Translation -->
<module name="Translation"/>
<!-- Checks for Size Violations. -->
<!-- See https://checkstyle.org/config_sizes.html -->
<module name="FileLength"/>
<module name="LineLength">
<property name="max" value="100"/>
<property name="fileExtensions" value="java"/>
<!-- Also allow links in javadocs to be longer (the default would just cover imports) -->
<property name="ignorePattern" value="^((package|import) .*)|( *\* (@see )?&lt;a href ?\= ?&quot;.*&quot;&gt;)$"/>
</module>
<!-- Checks for whitespace -->
<!-- See https://checkstyle.org/config_whitespace.html -->
<module name="FileTabCharacter"/>
<!-- Miscellaneous other checks. -->
<!-- See https://checkstyle.org/config_misc.html -->
<module name="RegexpSingleline">
<property name="format" value="\s+$"/>
<property name="minimum" value="0"/>
<property name="maximum" value="0"/>
<property name="message" value="Line has trailing spaces."/>
</module>
<!-- Checks for Headers -->
<!-- See https://checkstyle.org/config_header.html -->
<!-- <module name="Header"> -->
<!-- <property name="headerFile" value="${checkstyle.header.file}"/> -->
<!-- <property name="fileExtensions" value="java"/> -->
<!-- </module> -->
<module name="SuppressWarningsFilter" />
<module name="SuppressWithPlainTextCommentFilter"/>
<module name="TreeWalker">
<!-- Checks for Javadoc comments. -->
<!-- See https://checkstyle.org/config_javadoc.html -->
<module name="InvalidJavadocPosition"/>
<module name="JavadocMethod">
<property name="allowMissingParamTags" value="true"/>
<property name="allowMissingReturnTag" value="true"/>
</module>
<module name="JavadocType"/>
<!--<module name="JavadocVariable"/>-->
<module name="JavadocStyle">
<property name="checkFirstSentence" value="false"/>
</module>
<!--<module name="MissingJavadocMethod"/>-->
<!-- Checks for Naming Conventions. -->
<!-- See https://checkstyle.org/config_naming.html -->
<module name="ConstantName"/>
<module name="LocalFinalVariableName"/>
<module name="LocalVariableName"/>
<module name="MemberName">
<property name="format" value="^(TAG|DEBUG|[a-z][a-zA-Z0-9]*)$"/>
</module>
<module name="MethodName"/>
<module name="PackageName"/>
<module name="ParameterName"/>
<module name="StaticVariableName"/>
<module name="TypeName"/>
<!-- Checks for imports -->
<!-- See https://checkstyle.org/config_import.html -->
<module name="AvoidStarImport"/>
<module name="IllegalImport"> <!-- defaults to sun.* packages -->
<property name="illegalClasses" value="
org.jetbrains.annotations.Nullable,
org.jetbrains.annotations.NotNull,
androidx.annotation.Nullable,
androidx.annotation.NonNull,
io.reactivex.rxjava3.annotations.NonNull,
io.reactivex.rxjava3.annotations.Nullable" />
</module>
<module name="RedundantImport"/>
<module name="UnusedImports"/>
<!-- Checks for Size Violations. -->
<!-- See https://checkstyle.org/config_sizes.html -->
<module name="MethodLength">
<property name="severity" value="warning"/>
</module>
<module name="ParameterNumber">
<property name="severity" value="warning"/>
</module>
<!-- Checks for whitespace -->
<!-- See https://checkstyle.org/config_whitespace.html -->
<module name="EmptyForIteratorPad"/>
<module name="GenericWhitespace"/>
<module name="MethodParamPad"/>
<module name="NoWhitespaceAfter"/>
<module name="NoWhitespaceBefore"/>
<module name="OperatorWrap"/>
<module name="ParenPad"/>
<module name="TypecastParenPad"/>
<module name="WhitespaceAfter"/>
<module name="WhitespaceAround"/>
<!-- Modifier Checks -->
<!-- See https://checkstyle.org/config_modifiers.html -->
<module name="ModifierOrder"/>
<module name="RedundantModifier"/>
<!-- Checks for blocks. You know, those {}'s -->
<!-- See https://checkstyle.org/config_blocks.html -->
<module name="AvoidNestedBlocks"/>
<module name="EmptyBlock"/>
<module name="LeftCurly"/>
<module name="NeedBraces"/>
<module name="RightCurly"/>
<!-- Checks for common coding problems -->
<!-- See https://checkstyle.org/config_coding.html -->
<module name="EmptyStatement"/>
<module name="EqualsHashCode">
<property name="severity" value="warning"/>
</module>
<module name="HiddenField">
<property name="ignoreConstructorParameter" value="true"/>
<property name="ignoreSetter" value="true"/>
</module>
<module name="IllegalInstantiation"/>
<module name="InnerAssignment"/>
<!--<module name="MagicNumber"/>-->
<!--<module name="MissingSwitchDefault">
<property name="severity" value="warning"/>
</module>-->
<module name="MultipleVariableDeclarations"/>
<module name="SimplifyBooleanExpression"/>
<module name="SimplifyBooleanReturn"/>
<module name="FinalLocalVariable">
<property name="tokens" value="VARIABLE_DEF,PARAMETER_DEF"/>
<property name="validateEnhancedForLoopVariable" value="true"/>
</module>
<!-- Checks for class design -->
<!-- See https://checkstyle.org/config_design.html -->
<!--<module name="DesignForExtension"/>-->
<module name="FinalClass"/>
<module name="HideUtilityClassConstructor"/>
<module name="InterfaceIsType"/>
<!--<module name="VisibilityModifier">
<property name="ignoreAnnotationCanonicalNames" value="State,ColumnInfo"/>
<property name="severity" value="warning"/>
</module>-->
<!-- Miscellaneous other checks. -->
<!-- See https://checkstyle.org/config_misc.html -->
<module name="ArrayTypeStyle"/>
<module name="FinalParameters"/>
<!--<module name="TodoComment">
<property name="format" value="(TODO:|FIXME:)"/>
<property name="severity" value="warning"/>
</module>-->
<module name="UpperEll"/>
<module name="SuppressWarningsHolder" />
</module>
</module>

View File

@ -1,15 +0,0 @@
Copyright: 2017 Christian Schabesberger <chris.schabesberger@mailbox.com>
License: GPL-3.0+
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.

45
extractor/build.gradle Normal file
View File

@ -0,0 +1,45 @@
plugins {
id 'checkstyle'
}
test {
// Pass on downloader type to tests for different CI jobs. See DownloaderFactory.java and ci.yml
if (System.properties.containsKey('downloader')) {
systemProperty('downloader', System.getProperty('downloader'))
}
useJUnitPlatform()
dependsOn checkstyleMain // run checkstyle when testing
}
checkstyle {
getConfigDirectory().set(rootProject.file("checkstyle"))
ignoreFailures false
showViolations true
toolVersion checkstyleVersion
}
checkstyleTest {
enabled false // do not checkstyle test files
}
dependencies {
implementation project(':timeago-parser')
implementation "com.github.TeamNewPipe:nanojson:$nanojsonVersion"
implementation 'org.jsoup:jsoup:1.17.2'
implementation "com.github.spotbugs:spotbugs-annotations:$spotbugsVersion"
// do not upgrade to 1.7.14, since in 1.7.14 Rhino uses the `SourceVersion` class, which is not
// available on Android (even when using desugaring), and `NoClassDefFoundError` is thrown
implementation 'org.mozilla:rhino:1.7.15'
checkstyle "com.puppycrawl.tools:checkstyle:$checkstyleVersion"
testImplementation platform("org.junit:junit-bom:$junitVersion")
testImplementation 'org.junit.jupiter:junit-jupiter-api'
testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine'
testImplementation 'org.junit.jupiter:junit-jupiter-params'
testImplementation "com.squareup.okhttp3:okhttp:3.12.13"
testImplementation 'com.google.code.gson:gson:2.10.1'
}

View File

@ -0,0 +1,46 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import java.util.List;
/**
* Collectors are used to simplify the collection of information
* from extractors
* @param <I> the item type
* @param <E> the extractor type
*/
public interface Collector<I, E> {
/**
* Try to add an extractor to the collection
* @param extractor the extractor to add
*/
void commit(E extractor);
/**
* Try to extract the item from an extractor without adding it to the collection
* @param extractor the extractor to use
* @return the item
* @throws ParsingException thrown if there is an error extracting the
* <b>required</b> fields of the item.
*/
I extract(E extractor) throws ParsingException;
/**
* Get all items
* @return the items
*/
List<I> getItems();
/**
* Get all errors
* @return the errors
*/
List<Throwable> getErrors();
/**
* Reset all collected items and errors
*/
void reset();
}

View File

@ -0,0 +1,154 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.LinkHandler;
import org.schabi.newpipe.extractor.localization.ContentCountry;
import org.schabi.newpipe.extractor.localization.Localization;
import org.schabi.newpipe.extractor.localization.TimeAgoParser;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.io.IOException;
import java.util.Objects;
public abstract class Extractor {
/**
* {@link StreamingService} currently related to this extractor.<br>
* Useful for getting other things from a service (like the url handlers for
* cleaning/accepting/get id from urls).
*/
private final StreamingService service;
private final LinkHandler linkHandler;
@Nullable
private Localization forcedLocalization = null;
@Nullable
private ContentCountry forcedContentCountry = null;
private boolean pageFetched = false;
// called like this to prevent checkstyle errors about "hiding a field"
private final Downloader downloader;
protected Extractor(final StreamingService service, final LinkHandler linkHandler) {
this.service = Objects.requireNonNull(service, "service is null");
this.linkHandler = Objects.requireNonNull(linkHandler, "LinkHandler is null");
this.downloader = Objects.requireNonNull(NewPipe.getDownloader(), "downloader is null");
}
/**
* @return The {@link LinkHandler} of the current extractor object (e.g. a ChannelExtractor
* should return a channel url handler).
*/
@Nonnull
public LinkHandler getLinkHandler() {
return linkHandler;
}
/**
* Fetch the current page.
*
* @throws IOException if the page can not be loaded
* @throws ExtractionException if the pages content is not understood
*/
public void fetchPage() throws IOException, ExtractionException {
if (pageFetched) {
return;
}
onFetchPage(downloader);
pageFetched = true;
}
protected void assertPageFetched() {
if (!pageFetched) {
throw new IllegalStateException("Page is not fetched. Make sure you call fetchPage()");
}
}
protected boolean isPageFetched() {
return pageFetched;
}
/**
* Fetch the current page.
*
* @param downloader the downloader to use
* @throws IOException if the page can not be loaded
* @throws ExtractionException if the pages content is not understood
*/
@SuppressWarnings("HiddenField")
public abstract void onFetchPage(@Nonnull Downloader downloader)
throws IOException, ExtractionException;
@Nonnull
public String getId() throws ParsingException {
return linkHandler.getId();
}
/**
* Get the name
*
* @return the name
* @throws ParsingException if the name cannot be extracted
*/
@Nonnull
public abstract String getName() throws ParsingException;
@Nonnull
public String getOriginalUrl() throws ParsingException {
return linkHandler.getOriginalUrl();
}
@Nonnull
public String getUrl() throws ParsingException {
return linkHandler.getUrl();
}
@Nonnull
public String getBaseUrl() throws ParsingException {
return linkHandler.getBaseUrl();
}
@Nonnull
public StreamingService getService() {
return service;
}
public int getServiceId() {
return service.getServiceId();
}
public Downloader getDownloader() {
return downloader;
}
/*//////////////////////////////////////////////////////////////////////////
// Localization
//////////////////////////////////////////////////////////////////////////*/
public void forceLocalization(final Localization localization) {
this.forcedLocalization = localization;
}
public void forceContentCountry(final ContentCountry contentCountry) {
this.forcedContentCountry = contentCountry;
}
@Nonnull
public Localization getExtractorLocalization() {
return forcedLocalization == null ? getService().getLocalization() : forcedLocalization;
}
@Nonnull
public ContentCountry getExtractorContentCountry() {
return forcedContentCountry == null ? getService().getContentCountry()
: forcedContentCountry;
}
@Nonnull
public TimeAgoParser getTimeAgoParser() {
return getService().getTimeAgoParser(getExtractorLocalization());
}
}

View File

@ -0,0 +1,211 @@
package org.schabi.newpipe.extractor;
import javax.annotation.Nonnull;
import java.io.Serializable;
import java.util.Objects;
/**
* Class representing images in the extractor.
*
* <p>
* An image has four properties: its URL, its height, its width and its estimated quality level.
* </p>
*
* <p>
* Depending of the services, the height, the width or both properties may be not known.
* Implementations <b>must use</b> the relevant unknown constants in this case
* ({@link #HEIGHT_UNKNOWN} and {@link #WIDTH_UNKNOWN}), to ensure properly the lack of knowledge
* of one or both of these properties to extractor clients.
* </p>
*
* <p>
* They should also respect the ranges defined in the estimated image resolution levels as much as
* possible, to ensure consistency to extractor clients.
* </p>
*/
public final class Image implements Serializable {
/**
* Constant representing that the height of an {@link Image} is unknown.
*/
public static final int HEIGHT_UNKNOWN = -1;
/**
* Constant representing that the width of an {@link Image} is unknown.
*/
public static final int WIDTH_UNKNOWN = -1;
@Nonnull
private final String url;
private final int height;
private final int width;
@Nonnull
private final ResolutionLevel estimatedResolutionLevel;
/**
* Construct an {@link Image} instance.
*
* @param url the URL to the image, which should be not null or empty
* @param height the image's height
* @param width the image's width
* @param estimatedResolutionLevel the image's estimated resolution level, which must not be
* null
* @throws NullPointerException if {@code estimatedResolutionLevel} is null
*/
public Image(@Nonnull final String url,
final int height,
final int width,
@Nonnull final ResolutionLevel estimatedResolutionLevel)
throws NullPointerException {
this.url = url;
this.height = height;
this.width = width;
this.estimatedResolutionLevel = Objects.requireNonNull(
estimatedResolutionLevel, "estimatedResolutionLevel is null");
}
/**
* Get the URL of this {@link Image}.
*
* @return the {@link Image}'s URL.
*/
@Nonnull
public String getUrl() {
return url;
}
/**
* Get the height of this {@link Image}.
*
* <p>
* If it is unknown, {@link #HEIGHT_UNKNOWN} is returned instead.
* </p>
*
* @return the {@link Image}'s height or {@link #HEIGHT_UNKNOWN}
*/
public int getHeight() {
return height;
}
/**
* Get the width of this {@link Image}.
*
* <p>
* If it is unknown, {@link #WIDTH_UNKNOWN} is returned instead.
* </p>
*
* @return the {@link Image}'s width or {@link #WIDTH_UNKNOWN}
*/
public int getWidth() {
return width;
}
/**
* Get the estimated resolution level of this image.
*
* <p>
* If it is unknown, {@link ResolutionLevel#UNKNOWN} is returned instead.
* </p>
*
* @return the estimated resolution level, which is never {@code null}
* @see ResolutionLevel
*/
@Nonnull
public ResolutionLevel getEstimatedResolutionLevel() {
return estimatedResolutionLevel;
}
/**
* Get a string representation of this {@link Image} instance.
*
* <p>
* The representation will be in the following format, where {@code url}, {@code height},
* {@code width} and {@code estimatedResolutionLevel} represent the corresponding properties:
* <br>
* <br>
* {@code Image {url=url, height='height, width=width,
* estimatedResolutionLevel=estimatedResolutionLevel}'}
* </p>
*
* @return a string representation of this {@link Image} instance
*/
@Nonnull
@Override
public String toString() {
return "Image {" + "url=" + url + ", height=" + height + ", width=" + width
+ ", estimatedResolutionLevel=" + estimatedResolutionLevel + "}";
}
/**
* The estimated resolution level of an {@link Image}.
*
* <p>
* Some services don't return the size of their images, but we may know for a specific image
* type that a service returns, according to real data, an approximation of the resolution
* level.
* </p>
*/
public enum ResolutionLevel {
/**
* The high resolution level.
*
* <p>
* This level applies to images with a height greater than or equal to 720px.
* </p>
*/
HIGH,
/**
* The medium resolution level.
*
* <p>
* This level applies to images with a height between 175px inclusive and 720px exclusive.
* </p>
*/
MEDIUM,
/**
* The low resolution level.
*
* <p>
* This level applies to images with a height between 1px inclusive and 175px exclusive.
* </p>
*/
LOW,
/**
* The unknown resolution level.
*
* <p>
* This value is returned when the extractor doesn't know what resolution level an image
* could have, for example if the extractor loops in an array of images with different
* resolution levels without knowing the height.
* </p>
*/
UNKNOWN;
/**
* Get a {@link ResolutionLevel} based from the given height.
*
* @param heightPx the height from which returning the good {@link ResolutionLevel}
* @return the {@link ResolutionLevel} corresponding to the height provided. See the
* {@link ResolutionLevel} values for details about what value is returned.
*/
public static ResolutionLevel fromHeight(final int heightPx) {
if (heightPx <= 0) {
return UNKNOWN;
}
if (heightPx < 175) {
return LOW;
}
if (heightPx < 720) {
return MEDIUM;
}
return HIGH;
}
}
}

View File

@ -0,0 +1,111 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.LinkHandler;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
public abstract class Info implements Serializable {
private final int serviceId;
/**
* Id of this Info object <br>
* e.g. Youtube: https://www.youtube.com/watch?v=RER5qCTzZ7 &gt; RER5qCTzZ7
*/
private final String id;
/**
* Different than the {@link #originalUrl} in the sense that it <i>may</i> be set as a cleaned
* url.
*
* @see LinkHandler#getUrl()
* @see Extractor#getOriginalUrl()
*/
private final String url;
/**
* The url used to start the extraction of this {@link Info} object.
*
* @see Extractor#getOriginalUrl()
*/
private String originalUrl;
private final String name;
private final List<Throwable> errors = new ArrayList<>();
public void addError(final Throwable throwable) {
this.errors.add(throwable);
}
public void addAllErrors(final Collection<Throwable> throwables) {
this.errors.addAll(throwables);
}
public Info(final int serviceId,
final String id,
final String url,
final String originalUrl,
final String name) {
this.serviceId = serviceId;
this.id = id;
this.url = url;
this.originalUrl = originalUrl;
this.name = name;
}
public Info(final int serviceId, final LinkHandler linkHandler, final String name) {
this(serviceId,
linkHandler.getId(),
linkHandler.getUrl(),
linkHandler.getOriginalUrl(),
name);
}
@Override
public String toString() {
final String ifDifferentString
= url.equals(originalUrl) ? "" : " (originalUrl=\"" + originalUrl + "\")";
return getClass().getSimpleName() + "[url=\"" + url + "\"" + ifDifferentString
+ ", name=\"" + name + "\"]";
}
// if you use an api and want to handle the website url
// overriding original url is essential
public void setOriginalUrl(final String originalUrl) {
this.originalUrl = originalUrl;
}
public int getServiceId() {
return serviceId;
}
public StreamingService getService() {
try {
return NewPipe.getService(serviceId);
} catch (final ExtractionException e) {
// this should be unreachable, as serviceId certainly refers to a valid service
throw new RuntimeException("Info object has invalid service id", e);
}
}
public String getId() {
return id;
}
public String getUrl() {
return url;
}
public String getOriginalUrl() {
return originalUrl;
}
public String getName() {
return name;
}
public List<Throwable> getErrors() {
return errors;
}
}

View File

@ -0,0 +1,81 @@
/*
* Created by Christian Schabesberger on 11.02.17.
*
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* InfoItem.java is part of NewPipe Extractor.
*
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe Extractor. If not, see <https://www.gnu.org/licenses/>.
*/
package org.schabi.newpipe.extractor;
import javax.annotation.Nonnull;
import java.io.Serializable;
import java.util.List;
public abstract class InfoItem implements Serializable {
private final InfoType infoType;
private final int serviceId;
private final String url;
private final String name;
@Nonnull
private List<Image> thumbnails = List.of();
public InfoItem(final InfoType infoType,
final int serviceId,
final String url,
final String name) {
this.infoType = infoType;
this.serviceId = serviceId;
this.url = url;
this.name = name;
}
public InfoType getInfoType() {
return infoType;
}
public int getServiceId() {
return serviceId;
}
public String getUrl() {
return url;
}
public String getName() {
return name;
}
public void setThumbnails(@Nonnull final List<Image> thumbnails) {
this.thumbnails = thumbnails;
}
@Nonnull
public List<Image> getThumbnails() {
return thumbnails;
}
@Override
public String toString() {
return getClass().getSimpleName() + "[url=\"" + url + "\", name=\"" + name + "\"]";
}
public enum InfoType {
STREAM,
PLAYLIST,
CHANNEL,
COMMENT
}
}

View File

@ -0,0 +1,13 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import javax.annotation.Nonnull;
import java.util.List;
public interface InfoItemExtractor {
String getName() throws ParsingException;
String getUrl() throws ParsingException;
@Nonnull
List<Image> getThumbnails() throws ParsingException;
}

View File

@ -0,0 +1,111 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.exceptions.FoundAdException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import javax.annotation.Nullable;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.List;
/*
* Created by Christian Schabesberger on 12.02.17.
*
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* InfoItemsCollector.java is part of NewPipe Extractor.
*
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public abstract class InfoItemsCollector<I extends InfoItem, E extends InfoItemExtractor>
implements Collector<I, E> {
private final List<I> itemList = new ArrayList<>();
private final List<Throwable> errors = new ArrayList<>();
private final int serviceId;
@Nullable
private final Comparator<I> comparator;
/**
* Create a new collector with no comparator / sorting function
* @param serviceId the service id
*/
public InfoItemsCollector(final int serviceId) {
this(serviceId, null);
}
/**
* Create a new collector
* @param serviceId the service id
*/
public InfoItemsCollector(final int serviceId, @Nullable final Comparator<I> comparator) {
this.serviceId = serviceId;
this.comparator = comparator;
}
@Override
public List<I> getItems() {
if (comparator != null) {
itemList.sort(comparator);
}
return Collections.unmodifiableList(itemList);
}
@Override
public List<Throwable> getErrors() {
return Collections.unmodifiableList(errors);
}
@Override
public void reset() {
itemList.clear();
errors.clear();
}
/**
* Add an error
* @param error the error
*/
protected void addError(final Exception error) {
errors.add(error);
}
/**
* Add an item
* @param item the item
*/
protected void addItem(final I item) {
itemList.add(item);
}
/**
* Get the service id
* @return the service id
*/
public int getServiceId() {
return serviceId;
}
@Override
public void commit(final E extractor) {
try {
addItem(extract(extractor));
} catch (final FoundAdException ae) {
// found an ad. Maybe a debug line could be placed here
} catch (final ParsingException e) {
addError(e);
}
}
}

View File

@ -0,0 +1,132 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import java.io.IOException;
import java.util.Collections;
import java.util.List;
import javax.annotation.Nonnull;
/**
* Base class to extractors that have a list (e.g. playlists, users).
* @param <R> the info item type this list extractor provides
*/
public abstract class ListExtractor<R extends InfoItem> extends Extractor {
/**
* Constant that should be returned whenever
* a list has an unknown number of items.
*/
public static final long ITEM_COUNT_UNKNOWN = -1;
/**
* Constant that should be returned whenever a list has an
* infinite number of items. For example a YouTube mix.
*/
public static final long ITEM_COUNT_INFINITE = -2;
/**
* Constant that should be returned whenever a list
* has an unknown number of items bigger than 100.
*/
public static final long ITEM_COUNT_MORE_THAN_100 = -3;
public ListExtractor(final StreamingService service, final ListLinkHandler linkHandler) {
super(service, linkHandler);
}
/**
* A {@link InfoItemsPage InfoItemsPage} corresponding to the initial page
* where the items are from the initial request and the nextPage relative to it.
*
* @return a {@link InfoItemsPage} corresponding to the initial page
*/
@Nonnull
public abstract InfoItemsPage<R> getInitialPage() throws IOException, ExtractionException;
/**
* Get a list of items corresponding to the specific requested page.
*
* @param page any page got from the exclusive implementation of the list extractor
* @return a {@link InfoItemsPage} corresponding to the requested page
* @see InfoItemsPage#getNextPage()
*/
public abstract InfoItemsPage<R> getPage(Page page) throws IOException, ExtractionException;
@Nonnull
@Override
public ListLinkHandler getLinkHandler() {
return (ListLinkHandler) super.getLinkHandler();
}
/*//////////////////////////////////////////////////////////////////////////
// Inner
//////////////////////////////////////////////////////////////////////////*/
/**
* A class that is used to wrap a list of gathered items and eventual errors, it
* also contains a field that points to the next available page ({@link #nextPage}).
* @param <T> the info item type that this page is supposed to store and provide
*/
public static class InfoItemsPage<T extends InfoItem> {
private static final InfoItemsPage<InfoItem> EMPTY =
new InfoItemsPage<>(Collections.emptyList(), null, Collections.emptyList());
/**
* A convenient method that returns a representation of an empty page.
*
* @return a type-safe page with the list of items and errors empty and the nextPage set to
* {@code null}.
*/
public static <T extends InfoItem> InfoItemsPage<T> emptyPage() {
//noinspection unchecked
return (InfoItemsPage<T>) EMPTY;
}
/**
* The current list of items of this page
*/
private final List<T> itemsList;
/**
* Url pointing to the next page relative to this one
*
* @see ListExtractor#getPage(Page)
* @see Page
*/
private final Page nextPage;
/**
* Errors that happened during the extraction
*/
private final List<Throwable> errors;
public InfoItemsPage(final InfoItemsCollector<T, ?> collector, final Page nextPage) {
this(collector.getItems(), nextPage, collector.getErrors());
}
public InfoItemsPage(final List<T> itemsList,
final Page nextPage,
final List<Throwable> errors) {
this.itemsList = itemsList;
this.nextPage = nextPage;
this.errors = errors;
}
public boolean hasNextPage() {
return Page.isValid(nextPage);
}
public List<T> getItems() {
return itemsList;
}
public Page getNextPage() {
return nextPage;
}
public List<Throwable> getErrors() {
return errors;
}
}
}

View File

@ -0,0 +1,60 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import java.util.List;
public abstract class ListInfo<T extends InfoItem> extends Info {
private List<T> relatedItems;
private Page nextPage = null;
private final List<String> contentFilters;
private final String sortFilter;
public ListInfo(final int serviceId,
final String id,
final String url,
final String originalUrl,
final String name,
final List<String> contentFilter,
final String sortFilter) {
super(serviceId, id, url, originalUrl, name);
this.contentFilters = contentFilter;
this.sortFilter = sortFilter;
}
public ListInfo(final int serviceId,
final ListLinkHandler listUrlIdHandler,
final String name) {
super(serviceId, listUrlIdHandler, name);
this.contentFilters = listUrlIdHandler.getContentFilters();
this.sortFilter = listUrlIdHandler.getSortFilter();
}
public List<T> getRelatedItems() {
return relatedItems;
}
public void setRelatedItems(final List<T> relatedItems) {
this.relatedItems = relatedItems;
}
public boolean hasNextPage() {
return Page.isValid(nextPage);
}
public Page getNextPage() {
return nextPage;
}
public void setNextPage(final Page page) {
this.nextPage = page;
}
public List<String> getContentFilters() {
return contentFilters;
}
public String getSortFilter() {
return sortFilter;
}
}

View File

@ -0,0 +1,213 @@
package org.schabi.newpipe.extractor;
/*
* Created by Adam Howard on 08/11/15.
*
* Copyright (c) 2015 Christian Schabesberger <chris.schabesberger@mailbox.org>
* and Adam Howard <achdisposable1@gmail.com>
*
* MediaFormat.java is part of NewPipe Extractor.
*
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.Arrays;
import java.util.List;
import java.util.function.Function;
import java.util.stream.Collectors;
/**
* Static data about various media formats support by NewPipe, eg mime type, extension
*/
@SuppressWarnings("MethodParamPad") // we want the media format table below to be aligned
public enum MediaFormat {
// @formatter:off
//video and audio combined formats
// id name suffix mimeType
MPEG_4 (0x0, "MPEG-4", "mp4", "video/mp4"),
v3GPP (0x10, "3GPP", "3gp", "video/3gpp"),
WEBM (0x20, "WebM", "webm", "video/webm"),
// audio formats
M4A (0x100, "m4a", "m4a", "audio/mp4"),
WEBMA (0x200, "WebM", "webm", "audio/webm"),
MP3 (0x300, "MP3", "mp3", "audio/mpeg"),
MP2 (0x310, "MP2", "mp2", "audio/mpeg"),
OPUS (0x400, "opus", "opus", "audio/opus"),
OGG (0x500, "ogg", "ogg", "audio/ogg"),
WEBMA_OPUS(0x200, "WebM Opus", "webm", "audio/webm"),
AIFF (0x600, "AIFF", "aiff", "audio/aiff"),
/**
* Same as {@link MediaFormat#AIFF}, just with the shorter suffix/file extension
*/
AIF (0x600, "AIFF", "aif", "audio/aiff"),
WAV (0x700, "WAV", "wav", "audio/wav"),
FLAC (0x800, "FLAC", "flac", "audio/flac"),
ALAC (0x900, "ALAC", "alac", "audio/alac"),
// subtitles formats
VTT (0x1000, "WebVTT", "vtt", "text/vtt"),
TTML (0x2000, "Timed Text Markup Language", "ttml", "application/ttml+xml"),
TRANSCRIPT1(0x3000, "TranScript v1", "srv1", "text/xml"),
TRANSCRIPT2(0x4000, "TranScript v2", "srv2", "text/xml"),
TRANSCRIPT3(0x5000, "TranScript v3", "srv3", "text/xml"),
SRT (0x6000, "SubRip file format", "srt", "text/srt");
// @formatter:on
public final int id;
@Nonnull
public final String name;
@Nonnull
public final String suffix;
@Nonnull
public final String mimeType;
MediaFormat(final int id, @Nonnull final String name,
@Nonnull final String suffix, @Nonnull final String mimeType) {
this.id = id;
this.name = name;
this.suffix = suffix;
this.mimeType = mimeType;
}
private static <T> T getById(final int id,
final Function<MediaFormat, T> field,
final T orElse) {
return Arrays.stream(MediaFormat.values())
.filter(mediaFormat -> mediaFormat.id == id)
.map(field)
.findFirst()
.orElse(orElse);
}
/**
* Return the friendly name of the media format with the supplied id
*
* @param id the id of the media format. Currently an arbitrary, NewPipe-specific number.
* @return the friendly name of the MediaFormat associated with this ids,
* or an empty String if none match it.
*/
@Nonnull
public static String getNameById(final int id) {
return getById(id, MediaFormat::getName, "");
}
/**
* Return the file extension of the media format with the supplied id
*
* @param id the id of the media format. Currently an arbitrary, NewPipe-specific number.
* @return the file extension of the MediaFormat associated with this ids,
* or an empty String if none match it.
*/
@Nonnull
public static String getSuffixById(final int id) {
return getById(id, MediaFormat::getSuffix, "");
}
/**
* Return the MIME type of the media format with the supplied id
*
* @param id the id of the media format. Currently an arbitrary, NewPipe-specific number.
* @return the MIME type of the MediaFormat associated with this ids,
* or an empty String if none match it.
*/
@Nullable
public static String getMimeById(final int id) {
return getById(id, MediaFormat::getMimeType, null);
}
/**
* Return the first {@link MediaFormat} with the supplied mime type.
* There might be more formats which have the same mime type.
* To retrieve those, use {@link #getAllFromMimeType(String)}.
*
* @return MediaFormat associated with this mime type,
* or null if none match it.
*/
@Nullable
public static MediaFormat getFromMimeType(final String mimeType) {
return Arrays.stream(MediaFormat.values())
.filter(mediaFormat -> mediaFormat.mimeType.equals(mimeType))
.findFirst()
.orElse(null);
}
/**
* Get all media formats which have the given mime type.
* @param mimeType the mime type to search for
* @return a modifiable {@link List} which contains the {@link MediaFormat}s
* that have the given mime type.
*/
@Nonnull
public static List<MediaFormat> getAllFromMimeType(final String mimeType) {
return Arrays.stream(MediaFormat.values())
.filter(mediaFormat -> mediaFormat.mimeType.equals(mimeType))
.collect(Collectors.toList());
}
/**
* Get the media format by its id.
*
* @param id the id
* @return the id of the media format or null.
*/
@Nullable
public static MediaFormat getFormatById(final int id) {
return getById(id, mediaFormat -> mediaFormat, null);
}
/**
* Get the first media format that has the given suffix/file extension.
* @return the matching {@link MediaFormat} or {@code null} if no associated format is found
*/
@Nullable
public static MediaFormat getFromSuffix(final String suffix) {
return Arrays.stream(MediaFormat.values())
.filter(mediaFormat -> mediaFormat.suffix.equals(suffix))
.findFirst()
.orElse(null);
}
/**
* Get the name of the format
*
* @return the name of the format
*/
@Nonnull
public String getName() {
return name;
}
/**
* Get the filename extension
*
* @return the filename extension
*/
@Nonnull
public String getSuffix() {
return suffix;
}
/**
* Get the mime type
*
* @return the mime type
*/
@Nonnull
public String getMimeType() {
return mimeType;
}
}

View File

@ -0,0 +1,78 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.stream.Description;
import java.io.Serializable;
import java.net.URL;
import java.util.ArrayList;
import java.util.List;
import javax.annotation.Nonnull;
public class MetaInfo implements Serializable {
private String title = "";
private Description content;
private List<URL> urls = new ArrayList<>();
private List<String> urlTexts = new ArrayList<>();
public MetaInfo(@Nonnull final String title,
@Nonnull final Description content,
@Nonnull final List<URL> urls,
@Nonnull final List<String> urlTexts) {
this.title = title;
this.content = content;
this.urls = urls;
this.urlTexts = urlTexts;
}
public MetaInfo() {
}
/**
* @return Title of the info. Can be empty.
*/
@Nonnull
public String getTitle() {
return title;
}
public void setTitle(@Nonnull final String title) {
this.title = title;
}
@Nonnull
public Description getContent() {
return content;
}
public void setContent(@Nonnull final Description content) {
this.content = content;
}
@Nonnull
public List<URL> getUrls() {
return urls;
}
public void setUrls(@Nonnull final List<URL> urls) {
this.urls = urls;
}
public void addUrl(@Nonnull final URL url) {
urls.add(url);
}
@Nonnull
public List<String> getUrlTexts() {
return urlTexts;
}
public void setUrlTexts(@Nonnull final List<String> urlTexts) {
this.urlTexts = urlTexts;
}
public void addUrlText(@Nonnull final String urlText) {
urlTexts.add(urlText);
}
}

View File

@ -0,0 +1,91 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.channel.ChannelInfoItemExtractor;
import org.schabi.newpipe.extractor.channel.ChannelInfoItemsCollector;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.playlist.PlaylistInfoItemExtractor;
import org.schabi.newpipe.extractor.playlist.PlaylistInfoItemsCollector;
import org.schabi.newpipe.extractor.stream.StreamInfoItemExtractor;
import org.schabi.newpipe.extractor.stream.StreamInfoItemsCollector;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
/*
* Created by Christian Schabesberger on 12.02.17.
*
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* InfoItemsSearchCollector.java is part of NewPipe Extractor.
*
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
/**
* A collector that can handle many extractor types, to be used when a list contains items of
* different types (e.g. search)
* <p>
* This collector can handle the following extractor types:
* <ul>
* <li>{@link StreamInfoItemExtractor}</li>
* <li>{@link ChannelInfoItemExtractor}</li>
* <li>{@link PlaylistInfoItemExtractor}</li>
* </ul>
* Calling {@link #extract(InfoItemExtractor)} or {@link #commit(InfoItemExtractor)} with any
* other extractor type will raise an exception.
*/
public class MultiInfoItemsCollector extends InfoItemsCollector<InfoItem, InfoItemExtractor> {
private final StreamInfoItemsCollector streamCollector;
private final ChannelInfoItemsCollector userCollector;
private final PlaylistInfoItemsCollector playlistCollector;
public MultiInfoItemsCollector(final int serviceId) {
super(serviceId);
streamCollector = new StreamInfoItemsCollector(serviceId);
userCollector = new ChannelInfoItemsCollector(serviceId);
playlistCollector = new PlaylistInfoItemsCollector(serviceId);
}
@Override
public List<Throwable> getErrors() {
final List<Throwable> errors = new ArrayList<>(super.getErrors());
errors.addAll(streamCollector.getErrors());
errors.addAll(userCollector.getErrors());
errors.addAll(playlistCollector.getErrors());
return Collections.unmodifiableList(errors);
}
@Override
public void reset() {
super.reset();
streamCollector.reset();
userCollector.reset();
playlistCollector.reset();
}
@Override
public InfoItem extract(final InfoItemExtractor extractor) throws ParsingException {
// Use the corresponding collector for each item extractor type
if (extractor instanceof StreamInfoItemExtractor) {
return streamCollector.extract((StreamInfoItemExtractor) extractor);
} else if (extractor instanceof ChannelInfoItemExtractor) {
return userCollector.extract((ChannelInfoItemExtractor) extractor);
} else if (extractor instanceof PlaylistInfoItemExtractor) {
return playlistCollector.extract((PlaylistInfoItemExtractor) extractor);
} else {
throw new IllegalArgumentException("Invalid extractor type: " + extractor);
}
}
}

View File

@ -0,0 +1,135 @@
package org.schabi.newpipe.extractor;
/*
* Created by Christian Schabesberger on 23.08.15.
*
* Copyright (C) 2015 Christian Schabesberger <chris.schabesberger@mailbox.org>
* NewPipe Extractor.java is part of NewPipe Extractor.
*
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.localization.ContentCountry;
import org.schabi.newpipe.extractor.localization.Localization;
import java.util.List;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
/**
* Provides access to streaming services supported by NewPipe.
*/
public final class NewPipe {
private static Downloader downloader;
private static Localization preferredLocalization;
private static ContentCountry preferredContentCountry;
private NewPipe() {
}
public static void init(final Downloader d) {
init(d, Localization.DEFAULT);
}
public static void init(final Downloader d, final Localization l) {
init(d, l, l.getCountryCode().isEmpty()
? ContentCountry.DEFAULT : new ContentCountry(l.getCountryCode()));
}
public static void init(final Downloader d, final Localization l, final ContentCountry c) {
downloader = d;
preferredLocalization = l;
preferredContentCountry = c;
}
public static Downloader getDownloader() {
return downloader;
}
/*//////////////////////////////////////////////////////////////////////////
// Utils
//////////////////////////////////////////////////////////////////////////*/
public static List<StreamingService> getServices() {
return ServiceList.all();
}
public static StreamingService getService(final int serviceId) throws ExtractionException {
return ServiceList.all().stream()
.filter(service -> service.getServiceId() == serviceId)
.findFirst()
.orElseThrow(() -> new ExtractionException(
"There's no service with the id = \"" + serviceId + "\""));
}
public static StreamingService getService(final String serviceName) throws ExtractionException {
return ServiceList.all().stream()
.filter(service -> service.getServiceInfo().getName().equals(serviceName))
.findFirst()
.orElseThrow(() -> new ExtractionException(
"There's no service with the name = \"" + serviceName + "\""));
}
public static StreamingService getServiceByUrl(final String url) throws ExtractionException {
for (final StreamingService service : ServiceList.all()) {
if (service.getLinkTypeByUrl(url) != StreamingService.LinkType.NONE) {
return service;
}
}
throw new ExtractionException("No service can handle the url = \"" + url + "\"");
}
/*//////////////////////////////////////////////////////////////////////////
// Localization
//////////////////////////////////////////////////////////////////////////*/
public static void setupLocalization(final Localization thePreferredLocalization) {
setupLocalization(thePreferredLocalization, null);
}
public static void setupLocalization(
final Localization thePreferredLocalization,
@Nullable final ContentCountry thePreferredContentCountry) {
NewPipe.preferredLocalization = thePreferredLocalization;
if (thePreferredContentCountry != null) {
NewPipe.preferredContentCountry = thePreferredContentCountry;
} else {
NewPipe.preferredContentCountry = thePreferredLocalization.getCountryCode().isEmpty()
? ContentCountry.DEFAULT
: new ContentCountry(thePreferredLocalization.getCountryCode());
}
}
@Nonnull
public static Localization getPreferredLocalization() {
return preferredLocalization == null ? Localization.DEFAULT : preferredLocalization;
}
public static void setPreferredLocalization(final Localization preferredLocalization) {
NewPipe.preferredLocalization = preferredLocalization;
}
@Nonnull
public static ContentCountry getPreferredContentCountry() {
return preferredContentCountry == null ? ContentCountry.DEFAULT : preferredContentCountry;
}
public static void setPreferredContentCountry(final ContentCountry preferredContentCountry) {
NewPipe.preferredContentCountry = preferredContentCountry;
}
}

View File

@ -0,0 +1,85 @@
package org.schabi.newpipe.extractor;
import java.io.Serializable;
import java.util.List;
import java.util.Map;
import javax.annotation.Nullable;
import static org.schabi.newpipe.extractor.utils.Utils.isNullOrEmpty;
public class Page implements Serializable {
private final String url;
private final String id;
private final List<String> ids;
private final Map<String, String> cookies;
@Nullable
private final byte[] body;
public Page(final String url,
final String id,
final List<String> ids,
final Map<String, String> cookies,
@Nullable final byte[] body) {
this.url = url;
this.id = id;
this.ids = ids;
this.cookies = cookies;
this.body = body;
}
public Page(final String url) {
this(url, null, null, null, null);
}
public Page(final String url, final String id) {
this(url, id, null, null, null);
}
public Page(final String url, final String id, final byte[] body) {
this(url, id, null, null, body);
}
public Page(final String url, final byte[] body) {
this(url, null, null, null, body);
}
public Page(final String url, final Map<String, String> cookies) {
this(url, null, null, cookies, null);
}
public Page(final List<String> ids) {
this(null, null, ids, null, null);
}
public Page(final List<String> ids, final Map<String, String> cookies) {
this(null, null, ids, cookies, null);
}
public String getUrl() {
return url;
}
public String getId() {
return id;
}
public List<String> getIds() {
return ids;
}
public Map<String, String> getCookies() {
return cookies;
}
public static boolean isValid(final Page page) {
return page != null && (!isNullOrEmpty(page.getUrl())
|| !isNullOrEmpty(page.getIds()));
}
@Nullable
public byte[] getBody() {
return body;
}
}

View File

@ -0,0 +1,59 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.services.bandcamp.BandcampService;
import org.schabi.newpipe.extractor.services.media_ccc.MediaCCCService;
import org.schabi.newpipe.extractor.services.peertube.PeertubeService;
import org.schabi.newpipe.extractor.services.soundcloud.SoundcloudService;
import org.schabi.newpipe.extractor.services.youtube.YoutubeService;
import java.util.List;
/*
* Copyright (C) 2018 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ServiceList.java is part of NewPipe Extractor.
*
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
/**
* A list of supported services.
*/
@SuppressWarnings({"ConstantName", "InnerAssignment"}) // keep unusual names and inner assignments
public final class ServiceList {
private ServiceList() {
// no instance
}
public static final YoutubeService YouTube = new YoutubeService(0);
public static final SoundcloudService SoundCloud = new SoundcloudService(1);
public static final MediaCCCService MediaCCC = new MediaCCCService(2);
public static final PeertubeService PeerTube = new PeertubeService(3);
public static final BandcampService Bandcamp = new BandcampService(4);
/**
* When creating a new service, put this service in the end of this list,
* and give it the next free id.
*/
private static final List<StreamingService> SERVICES = List.of(
YouTube, SoundCloud, MediaCCC, PeerTube, Bandcamp);
/**
* Get all the supported services.
*
* @return a unmodifiable list of all the supported services
*/
public static List<StreamingService> all() {
return SERVICES;
}
}

View File

@ -0,0 +1,436 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.channel.ChannelExtractor;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import org.schabi.newpipe.extractor.comments.CommentsExtractor;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.feed.FeedExtractor;
import org.schabi.newpipe.extractor.kiosk.KioskList;
import org.schabi.newpipe.extractor.linkhandler.LinkHandler;
import org.schabi.newpipe.extractor.linkhandler.LinkHandlerFactory;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandler;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandlerFactory;
import org.schabi.newpipe.extractor.localization.ContentCountry;
import org.schabi.newpipe.extractor.localization.Localization;
import org.schabi.newpipe.extractor.localization.TimeAgoParser;
import org.schabi.newpipe.extractor.localization.TimeAgoPatternsManager;
import org.schabi.newpipe.extractor.playlist.PlaylistExtractor;
import org.schabi.newpipe.extractor.search.SearchExtractor;
import org.schabi.newpipe.extractor.stream.StreamExtractor;
import org.schabi.newpipe.extractor.subscription.SubscriptionExtractor;
import org.schabi.newpipe.extractor.suggestion.SuggestionExtractor;
import org.schabi.newpipe.extractor.utils.Utils;
import javax.annotation.Nullable;
import java.util.Collections;
import java.util.List;
/*
* Copyright (C) 2018 Christian Schabesberger <chris.schabesberger@mailbox.org>
* StreamingService.java is part of NewPipe Extractor.
*
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public abstract class StreamingService {
/**
* This class holds meta information about the service implementation.
*/
public static class ServiceInfo {
private final String name;
private final List<MediaCapability> mediaCapabilities;
/**
* Creates a new instance of a ServiceInfo
* @param name the name of the service
* @param mediaCapabilities the type of media this service can handle
*/
public ServiceInfo(final String name, final List<MediaCapability> mediaCapabilities) {
this.name = name;
this.mediaCapabilities = Collections.unmodifiableList(mediaCapabilities);
}
public String getName() {
return name;
}
public List<MediaCapability> getMediaCapabilities() {
return mediaCapabilities;
}
public enum MediaCapability {
AUDIO, VIDEO, LIVE, COMMENTS
}
}
/**
* LinkType will be used to determine which type of URL you are handling, and therefore which
* part of NewPipe should handle a certain URL.
*/
public enum LinkType {
NONE,
STREAM,
CHANNEL,
PLAYLIST
}
private final int serviceId;
private final ServiceInfo serviceInfo;
/**
* Creates a new Streaming service.
* If you Implement one do not set id within your implementation of this extractor, instead
* set the id when you put the extractor into {@link ServiceList}
* All other parameters can be set directly from the overriding constructor.
* @param id the number of the service to identify him within the NewPipe frontend
* @param name the name of the service
* @param capabilities the type of media this service can handle
*/
public StreamingService(final int id,
final String name,
final List<ServiceInfo.MediaCapability> capabilities) {
this.serviceId = id;
this.serviceInfo = new ServiceInfo(name, capabilities);
}
public final int getServiceId() {
return serviceId;
}
public ServiceInfo getServiceInfo() {
return serviceInfo;
}
@Override
public String toString() {
return serviceId + ":" + serviceInfo.getName();
}
public abstract String getBaseUrl();
/*//////////////////////////////////////////////////////////////////////////
// Url Id handler
//////////////////////////////////////////////////////////////////////////*/
/**
* Must return a new instance of an implementation of LinkHandlerFactory for streams.
* @return an instance of a LinkHandlerFactory for streams
*/
public abstract LinkHandlerFactory getStreamLHFactory();
/**
* Must return a new instance of an implementation of ListLinkHandlerFactory for channels.
* If support for channels is not given null must be returned.
* @return an instance of a ListLinkHandlerFactory for channels or null
*/
public abstract ListLinkHandlerFactory getChannelLHFactory();
/**
* Must return a new instance of an implementation of ListLinkHandlerFactory for channel tabs.
* If support for channel tabs is not given null must be returned.
*
* @return an instance of a ListLinkHandlerFactory for channels or null
*/
public abstract ListLinkHandlerFactory getChannelTabLHFactory();
/**
* Must return a new instance of an implementation of ListLinkHandlerFactory for playlists.
* If support for playlists is not given null must be returned.
* @return an instance of a ListLinkHandlerFactory for playlists or null
*/
public abstract ListLinkHandlerFactory getPlaylistLHFactory();
/**
* Must return an instance of an implementation of SearchQueryHandlerFactory.
* @return an instance of a SearchQueryHandlerFactory
*/
public abstract SearchQueryHandlerFactory getSearchQHFactory();
public abstract ListLinkHandlerFactory getCommentsLHFactory();
/*//////////////////////////////////////////////////////////////////////////
// Extractors
//////////////////////////////////////////////////////////////////////////*/
/**
* Must create a new instance of a SearchExtractor implementation.
* @param queryHandler specifies the keyword lock for, and the filters which should be applied.
* @return a new SearchExtractor instance
*/
public abstract SearchExtractor getSearchExtractor(SearchQueryHandler queryHandler);
/**
* Must create a new instance of a SuggestionExtractor implementation.
* @return a new SuggestionExtractor instance
*/
public abstract SuggestionExtractor getSuggestionExtractor();
/**
* Outdated or obsolete. null can be returned.
* @return just null
*/
public abstract SubscriptionExtractor getSubscriptionExtractor();
/**
* This method decides which strategy will be chosen to fetch the feed. In YouTube, for example,
* a separate feed exists which is lightweight and made specifically to be used like this.
* <p>
* In services which there's no other way to retrieve them, null should be returned.
*
* @return a {@link FeedExtractor} instance or null.
*/
@Nullable
public FeedExtractor getFeedExtractor(final String url) throws ExtractionException {
return null;
}
/**
* Must create a new instance of a KioskList implementation.
* @return a new KioskList instance
*/
public abstract KioskList getKioskList() throws ExtractionException;
/**
* Must create a new instance of a ChannelExtractor implementation.
* @param linkHandler is pointing to the channel which should be handled by this new instance.
* @return a new ChannelExtractor
*/
public abstract ChannelExtractor getChannelExtractor(ListLinkHandler linkHandler)
throws ExtractionException;
/**
* Must create a new instance of a ChannelTabExtractor implementation.
*
* @param linkHandler is pointing to the channel which should be handled by this new instance.
* @return a new ChannelTabExtractor
*/
public abstract ChannelTabExtractor getChannelTabExtractor(ListLinkHandler linkHandler)
throws ExtractionException;
/**
* Must crete a new instance of a PlaylistExtractor implementation.
* @param linkHandler is pointing to the playlist which should be handled by this new instance.
* @return a new PlaylistExtractor
*/
public abstract PlaylistExtractor getPlaylistExtractor(ListLinkHandler linkHandler)
throws ExtractionException;
/**
* Must create a new instance of a StreamExtractor implementation.
* @param linkHandler is pointing to the stream which should be handled by this new instance.
* @return a new StreamExtractor
*/
public abstract StreamExtractor getStreamExtractor(LinkHandler linkHandler)
throws ExtractionException;
public abstract CommentsExtractor getCommentsExtractor(ListLinkHandler linkHandler)
throws ExtractionException;
/*//////////////////////////////////////////////////////////////////////////
// Extractors without link handler
//////////////////////////////////////////////////////////////////////////*/
public SearchExtractor getSearchExtractor(final String query,
final List<String> contentFilter,
final String sortFilter) throws ExtractionException {
return getSearchExtractor(getSearchQHFactory()
.fromQuery(query, contentFilter, sortFilter));
}
public ChannelExtractor getChannelExtractor(final String id,
final List<String> contentFilter,
final String sortFilter)
throws ExtractionException {
return getChannelExtractor(getChannelLHFactory()
.fromQuery(id, contentFilter, sortFilter));
}
public PlaylistExtractor getPlaylistExtractor(final String id,
final List<String> contentFilter,
final String sortFilter)
throws ExtractionException {
return getPlaylistExtractor(getPlaylistLHFactory()
.fromQuery(id, contentFilter, sortFilter));
}
/*//////////////////////////////////////////////////////////////////////////
// Short extractors overloads
//////////////////////////////////////////////////////////////////////////*/
public SearchExtractor getSearchExtractor(final String query) throws ExtractionException {
return getSearchExtractor(getSearchQHFactory().fromQuery(query));
}
public ChannelExtractor getChannelExtractor(final String url) throws ExtractionException {
return getChannelExtractor(getChannelLHFactory().fromUrl(url));
}
public ChannelTabExtractor getChannelTabExtractorFromId(final String id, final String tab)
throws ExtractionException {
return getChannelTabExtractor(getChannelTabLHFactory().fromQuery(
id, Collections.singletonList(tab), ""));
}
public ChannelTabExtractor getChannelTabExtractorFromIdAndBaseUrl(final String id,
final String tab,
final String baseUrl)
throws ExtractionException {
return getChannelTabExtractor(getChannelTabLHFactory().fromQuery(
id, Collections.singletonList(tab), "", baseUrl));
}
public PlaylistExtractor getPlaylistExtractor(final String url) throws ExtractionException {
return getPlaylistExtractor(getPlaylistLHFactory().fromUrl(url));
}
public StreamExtractor getStreamExtractor(final String url) throws ExtractionException {
return getStreamExtractor(getStreamLHFactory().fromUrl(url));
}
public CommentsExtractor getCommentsExtractor(final String url) throws ExtractionException {
final ListLinkHandlerFactory listLinkHandlerFactory = getCommentsLHFactory();
if (listLinkHandlerFactory == null) {
return null;
}
return getCommentsExtractor(listLinkHandlerFactory.fromUrl(url));
}
/*//////////////////////////////////////////////////////////////////////////
// Utils
//////////////////////////////////////////////////////////////////////////*/
/**
* Figures out where the link is pointing to (a channel, a video, a playlist, etc.)
* @param url the url on which it should be decided of which link type it is
* @return the link type of url
*/
public final LinkType getLinkTypeByUrl(final String url) throws ParsingException {
final String polishedUrl = Utils.followGoogleRedirectIfNeeded(url);
final LinkHandlerFactory sH = getStreamLHFactory();
final LinkHandlerFactory cH = getChannelLHFactory();
final LinkHandlerFactory pH = getPlaylistLHFactory();
if (sH != null && sH.acceptUrl(polishedUrl)) {
return LinkType.STREAM;
} else if (cH != null && cH.acceptUrl(polishedUrl)) {
return LinkType.CHANNEL;
} else if (pH != null && pH.acceptUrl(polishedUrl)) {
return LinkType.PLAYLIST;
} else {
return LinkType.NONE;
}
}
/*//////////////////////////////////////////////////////////////////////////
// Localization
//////////////////////////////////////////////////////////////////////////*/
/**
* Returns a list of localizations that this service supports.
*/
public List<Localization> getSupportedLocalizations() {
return Collections.singletonList(Localization.DEFAULT);
}
/**
* Returns a list of countries that this service supports.<br>
*/
public List<ContentCountry> getSupportedCountries() {
return Collections.singletonList(ContentCountry.DEFAULT);
}
/**
* Returns the localization that should be used in this service. It will get which localization
* the user prefer (using {@link NewPipe#getPreferredLocalization()}), then it will:
* <ul>
* <li>Check if the exactly localization is supported by this service.</li>
* <li>If not, check if a less specific localization is available, using only the language
* code.</li>
* <li>Fallback to the {@link Localization#DEFAULT default} localization.</li>
* </ul>
*/
public Localization getLocalization() {
final Localization preferredLocalization = NewPipe.getPreferredLocalization();
// Check the localization's language and country
if (getSupportedLocalizations().contains(preferredLocalization)) {
return preferredLocalization;
}
// Fallback to the first supported language that matches the preferred language
for (final Localization supportedLanguage : getSupportedLocalizations()) {
if (supportedLanguage.getLanguageCode()
.equals(preferredLocalization.getLanguageCode())) {
return supportedLanguage;
}
}
return Localization.DEFAULT;
}
/**
* Returns the country that should be used to fetch content in this service. It will get which
* country the user prefer (using {@link NewPipe#getPreferredContentCountry()}), then it will:
* <ul>
* <li>Check if the country is supported by this service.</li>
* <li>If not, fallback to the {@link ContentCountry#DEFAULT default} country.</li>
* </ul>
*/
public ContentCountry getContentCountry() {
final ContentCountry preferredContentCountry = NewPipe.getPreferredContentCountry();
if (getSupportedCountries().contains(preferredContentCountry)) {
return preferredContentCountry;
}
return ContentCountry.DEFAULT;
}
/**
* Get an instance of the time ago parser using the patterns related to the passed localization.
* <br><br>
* Just like {@link #getLocalization()}, it will also try to fallback to a less specific
* localization if the exact one is not available/supported.
*
* @throws IllegalArgumentException if the localization is not supported (parsing patterns are
* not present).
*/
public TimeAgoParser getTimeAgoParser(final Localization localization) {
final TimeAgoParser targetParser = TimeAgoPatternsManager.getTimeAgoParserFor(localization);
if (targetParser != null) {
return targetParser;
}
if (!localization.getCountryCode().isEmpty()) {
final Localization lessSpecificLocalization
= new Localization(localization.getLanguageCode());
final TimeAgoParser lessSpecificParser
= TimeAgoPatternsManager.getTimeAgoParserFor(lessSpecificLocalization);
if (lessSpecificParser != null) {
return lessSpecificParser;
}
}
throw new IllegalArgumentException(
"Localization is not supported (\"" + localization + "\")");
}
}

View File

@ -0,0 +1,58 @@
/*
* Created by Christian Schabesberger on 25.07.16.
*
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ChannelExtractor.java is part of NewPipe Extractor.
*
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe Extractor. If not, see <https://www.gnu.org/licenses/>.
*/
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.Extractor;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import javax.annotation.Nonnull;
import java.util.List;
public abstract class ChannelExtractor extends Extractor {
public static final long UNKNOWN_SUBSCRIBER_COUNT = -1;
protected ChannelExtractor(final StreamingService service, final ListLinkHandler linkHandler) {
super(service, linkHandler);
}
@Nonnull
public abstract List<Image> getAvatars() throws ParsingException;
@Nonnull
public abstract List<Image> getBanners() throws ParsingException;
public abstract String getFeedUrl() throws ParsingException;
public abstract long getSubscriberCount() throws ParsingException;
public abstract String getDescription() throws ParsingException;
public abstract String getParentChannelName() throws ParsingException;
public abstract String getParentChannelUrl() throws ParsingException;
@Nonnull
public abstract List<Image> getParentChannelAvatars() throws ParsingException;
public abstract boolean isVerified() throws ParsingException;
@Nonnull
public abstract List<ListLinkHandler> getTabs() throws ParsingException;
@Nonnull
public List<String> getTags() throws ParsingException {
return List.of();
}
}

View File

@ -0,0 +1,252 @@
/*
* Created by Christian Schabesberger on 31.07.16.
*
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ChannelInfo.java is part of NewPipe Extractor.
*
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe Extractor. If not, see <https://www.gnu.org/licenses/>.
*/
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.Info;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import java.io.IOException;
import java.util.List;
import javax.annotation.Nonnull;
public class ChannelInfo extends Info {
public ChannelInfo(final int serviceId,
final String id,
final String url,
final String originalUrl,
final String name) {
super(serviceId, id, url, originalUrl, name);
}
public static ChannelInfo getInfo(final String url) throws IOException, ExtractionException {
return getInfo(NewPipe.getServiceByUrl(url), url);
}
public static ChannelInfo getInfo(final StreamingService service, final String url)
throws IOException, ExtractionException {
final ChannelExtractor extractor = service.getChannelExtractor(url);
extractor.fetchPage();
return getInfo(extractor);
}
public static ChannelInfo getInfo(final ChannelExtractor extractor)
throws IOException, ExtractionException {
final int serviceId = extractor.getServiceId();
final String id = extractor.getId();
final String url = extractor.getUrl();
final String originalUrl = extractor.getOriginalUrl();
final String name = extractor.getName();
final ChannelInfo info = new ChannelInfo(serviceId, id, url, originalUrl, name);
try {
info.setAvatars(extractor.getAvatars());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setBanners(extractor.getBanners());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setFeedUrl(extractor.getFeedUrl());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setSubscriberCount(extractor.getSubscriberCount());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setDescription(extractor.getDescription());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setParentChannelName(extractor.getParentChannelName());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setParentChannelUrl(extractor.getParentChannelUrl());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setParentChannelAvatars(extractor.getParentChannelAvatars());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setVerified(extractor.isVerified());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setTabs(extractor.getTabs());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setTags(extractor.getTags());
} catch (final Exception e) {
info.addError(e);
}
return info;
}
private String parentChannelName;
private String parentChannelUrl;
private String feedUrl;
private long subscriberCount = -1;
private String description;
private String[] donationLinks;
@Nonnull
private List<Image> avatars = List.of();
@Nonnull
private List<Image> banners = List.of();
@Nonnull
private List<Image> parentChannelAvatars = List.of();
private boolean verified;
private List<ListLinkHandler> tabs = List.of();
private List<String> tags = List.of();
public String getParentChannelName() {
return parentChannelName;
}
public void setParentChannelName(final String parentChannelName) {
this.parentChannelName = parentChannelName;
}
public String getParentChannelUrl() {
return parentChannelUrl;
}
public void setParentChannelUrl(final String parentChannelUrl) {
this.parentChannelUrl = parentChannelUrl;
}
@Nonnull
public List<Image> getParentChannelAvatars() {
return parentChannelAvatars;
}
public void setParentChannelAvatars(@Nonnull final List<Image> parentChannelAvatars) {
this.parentChannelAvatars = parentChannelAvatars;
}
@Nonnull
public List<Image> getAvatars() {
return avatars;
}
public void setAvatars(@Nonnull final List<Image> avatars) {
this.avatars = avatars;
}
@Nonnull
public List<Image> getBanners() {
return banners;
}
public void setBanners(@Nonnull final List<Image> banners) {
this.banners = banners;
}
public String getFeedUrl() {
return feedUrl;
}
public void setFeedUrl(final String feedUrl) {
this.feedUrl = feedUrl;
}
public long getSubscriberCount() {
return subscriberCount;
}
public void setSubscriberCount(final long subscriberCount) {
this.subscriberCount = subscriberCount;
}
public String getDescription() {
return description;
}
public void setDescription(final String description) {
this.description = description;
}
public String[] getDonationLinks() {
return donationLinks;
}
public void setDonationLinks(final String[] donationLinks) {
this.donationLinks = donationLinks;
}
public boolean isVerified() {
return verified;
}
public void setVerified(final boolean verified) {
this.verified = verified;
}
@Nonnull
public List<ListLinkHandler> getTabs() {
return tabs;
}
public void setTabs(@Nonnull final List<ListLinkHandler> tabs) {
this.tabs = tabs;
}
@Nonnull
public List<String> getTags() {
return tags;
}
public void setTags(@Nonnull final List<String> tags) {
this.tags = tags;
}
}

View File

@ -0,0 +1,67 @@
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.InfoItem;
/*
* Created by Christian Schabesberger on 11.02.17.
*
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ChannelInfoItem.java is part of NewPipe Extractor.
*
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public class ChannelInfoItem extends InfoItem {
private String description;
private long subscriberCount = -1;
private long streamCount = -1;
private boolean verified = false;
public ChannelInfoItem(final int serviceId, final String url, final String name) {
super(InfoType.CHANNEL, serviceId, url, name);
}
public String getDescription() {
return description;
}
public void setDescription(final String description) {
this.description = description;
}
public long getSubscriberCount() {
return subscriberCount;
}
public void setSubscriberCount(final long subscriberCount) {
this.subscriberCount = subscriberCount;
}
public long getStreamCount() {
return streamCount;
}
public void setStreamCount(final long streamCount) {
this.streamCount = streamCount;
}
public boolean isVerified() {
return verified;
}
public void setVerified(final boolean verified) {
this.verified = verified;
}
}

View File

@ -1,32 +1,34 @@
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.InfoItemExtractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
/**
/*
* Created by Christian Schabesberger on 12.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* ChannelInfoItemExtractor.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ChannelInfoItemExtractor.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public interface ChannelInfoItemExtractor {
String getThumbnailUrl() throws ParsingException;
String getChannelName() throws ParsingException;
String getWebPageUrl() throws ParsingException;
public interface ChannelInfoItemExtractor extends InfoItemExtractor {
String getDescription() throws ParsingException;
long getSubscriberCount() throws ParsingException;
int getVideoAmount() throws ParsingException;
long getStreamCount() throws ParsingException;
boolean isVerified() throws ParsingException;
}

View File

@ -0,0 +1,67 @@
/*
* Created by Christian Schabesberger on 12.02.17.
*
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ChannelInfoItemsCollector.java is part of NewPipe Extractor.
*
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe Extractor. If not, see <https://www.gnu.org/licenses/>.
*/
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.InfoItemsCollector;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
public final class ChannelInfoItemsCollector
extends InfoItemsCollector<ChannelInfoItem, ChannelInfoItemExtractor> {
public ChannelInfoItemsCollector(final int serviceId) {
super(serviceId);
}
@Override
public ChannelInfoItem extract(final ChannelInfoItemExtractor extractor)
throws ParsingException {
final ChannelInfoItem resultItem = new ChannelInfoItem(
getServiceId(), extractor.getUrl(), extractor.getName());
// optional information
try {
resultItem.setSubscriberCount(extractor.getSubscriberCount());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setStreamCount(extractor.getStreamCount());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setThumbnails(extractor.getThumbnails());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setDescription(extractor.getDescription());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setVerified(extractor.isVerified());
} catch (final Exception e) {
addError(e);
}
return resultItem;
}
}

View File

@ -0,0 +1,25 @@
package org.schabi.newpipe.extractor.channel.tabs;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import javax.annotation.Nonnull;
/**
* A {@link ListExtractor} of {@link InfoItem}s for tabs of channels.
*/
public abstract class ChannelTabExtractor extends ListExtractor<InfoItem> {
protected ChannelTabExtractor(@Nonnull final StreamingService service,
@Nonnull final ListLinkHandler linkHandler) {
super(service, linkHandler);
}
@Nonnull
@Override
public String getName() {
return getLinkHandler().getContentFilters().get(0);
}
}

View File

@ -0,0 +1,70 @@
package org.schabi.newpipe.extractor.channel.tabs;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.ListInfo;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.ChannelInfo;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.utils.ExtractorHelper;
import javax.annotation.Nonnull;
import java.io.IOException;
public class ChannelTabInfo extends ListInfo<InfoItem> {
public ChannelTabInfo(final int serviceId,
@Nonnull final ListLinkHandler linkHandler) {
super(serviceId, linkHandler, linkHandler.getContentFilters().get(0));
}
/**
* Get a {@link ChannelTabInfo} instance from the given service and tab handler.
*
* @param service streaming service
* @param linkHandler Channel tab handler (from {@link ChannelInfo})
* @return the extracted {@link ChannelTabInfo}
*/
@Nonnull
public static ChannelTabInfo getInfo(@Nonnull final StreamingService service,
@Nonnull final ListLinkHandler linkHandler)
throws ExtractionException, IOException {
final ChannelTabExtractor extractor = service.getChannelTabExtractor(linkHandler);
extractor.fetchPage();
return getInfo(extractor);
}
/**
* Get a {@link ChannelTabInfo} instance from a {@link ChannelTabExtractor}.
*
* @param extractor an extractor where {@code fetchPage()} was already got called on
* @return the extracted {@link ChannelTabInfo}
*/
@Nonnull
public static ChannelTabInfo getInfo(@Nonnull final ChannelTabExtractor extractor) {
final ChannelTabInfo info =
new ChannelTabInfo(extractor.getServiceId(), extractor.getLinkHandler());
try {
info.setOriginalUrl(extractor.getOriginalUrl());
} catch (final Exception e) {
info.addError(e);
}
final ListExtractor.InfoItemsPage<InfoItem> page
= ExtractorHelper.getItemsPageOrLogError(info, extractor);
info.setRelatedItems(page.getItems());
info.setNextPage(page.getNextPage());
return info;
}
public static ListExtractor.InfoItemsPage<InfoItem> getMoreItems(
@Nonnull final StreamingService service,
@Nonnull final ListLinkHandler linkHandler,
@Nonnull final Page page) throws ExtractionException, IOException {
return service.getChannelTabExtractor(linkHandler).getPage(page);
}
}

View File

@ -0,0 +1,17 @@
package org.schabi.newpipe.extractor.channel.tabs;
/**
* Constants of channel tabs supported by the extractor.
*/
public final class ChannelTabs {
public static final String VIDEOS = "videos";
public static final String TRACKS = "tracks";
public static final String SHORTS = "shorts";
public static final String LIVESTREAMS = "livestreams";
public static final String CHANNELS = "channels";
public static final String PLAYLISTS = "playlists";
public static final String ALBUMS = "albums";
private ChannelTabs() {
}
}

View File

@ -0,0 +1,37 @@
package org.schabi.newpipe.extractor.comments;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import javax.annotation.Nonnull;
public abstract class CommentsExtractor extends ListExtractor<CommentsInfoItem> {
public CommentsExtractor(final StreamingService service, final ListLinkHandler uiHandler) {
super(service, uiHandler);
}
/**
* @apiNote Warning: This method is experimental and may get removed in a future release.
* @return <code>true</code> if the comments are disabled otherwise <code>false</code> (default)
*/
public boolean isCommentsDisabled() throws ExtractionException {
return false;
}
/**
* @return the total number of comments
*/
public int getCommentsCount() throws ExtractionException {
return -1;
}
@Nonnull
@Override
public String getName() throws ParsingException {
return "Comments";
}
}

View File

@ -0,0 +1,126 @@
package org.schabi.newpipe.extractor.comments;
import org.schabi.newpipe.extractor.ListExtractor.InfoItemsPage;
import org.schabi.newpipe.extractor.ListInfo;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.utils.ExtractorHelper;
import java.io.IOException;
public final class CommentsInfo extends ListInfo<CommentsInfoItem> {
private CommentsInfo(
final int serviceId,
final ListLinkHandler listUrlIdHandler,
final String name) {
super(serviceId, listUrlIdHandler, name);
}
public static CommentsInfo getInfo(final String url) throws IOException, ExtractionException {
return getInfo(NewPipe.getServiceByUrl(url), url);
}
public static CommentsInfo getInfo(final StreamingService service, final String url)
throws ExtractionException, IOException {
return getInfo(service.getCommentsExtractor(url));
}
public static CommentsInfo getInfo(final CommentsExtractor commentsExtractor)
throws IOException, ExtractionException {
// for services which do not have a comments extractor
if (commentsExtractor == null) {
return null;
}
commentsExtractor.fetchPage();
final String name = commentsExtractor.getName();
final int serviceId = commentsExtractor.getServiceId();
final ListLinkHandler listUrlIdHandler = commentsExtractor.getLinkHandler();
final CommentsInfo commentsInfo = new CommentsInfo(serviceId, listUrlIdHandler, name);
commentsInfo.setCommentsExtractor(commentsExtractor);
final InfoItemsPage<CommentsInfoItem> initialCommentsPage =
ExtractorHelper.getItemsPageOrLogError(commentsInfo, commentsExtractor);
commentsInfo.setCommentsDisabled(commentsExtractor.isCommentsDisabled());
commentsInfo.setRelatedItems(initialCommentsPage.getItems());
try {
commentsInfo.setCommentsCount(commentsExtractor.getCommentsCount());
} catch (final Exception e) {
commentsInfo.addError(e);
}
commentsInfo.setNextPage(initialCommentsPage.getNextPage());
return commentsInfo;
}
public static InfoItemsPage<CommentsInfoItem> getMoreItems(
final CommentsInfo commentsInfo,
final Page page) throws ExtractionException, IOException {
return getMoreItems(NewPipe.getService(commentsInfo.getServiceId()), commentsInfo.getUrl(),
page);
}
public static InfoItemsPage<CommentsInfoItem> getMoreItems(
final StreamingService service,
final CommentsInfo commentsInfo,
final Page page) throws IOException, ExtractionException {
return getMoreItems(service, commentsInfo.getUrl(), page);
}
public static InfoItemsPage<CommentsInfoItem> getMoreItems(
final StreamingService service,
final String url,
final Page page) throws IOException, ExtractionException {
return service.getCommentsExtractor(url).getPage(page);
}
private transient CommentsExtractor commentsExtractor;
private boolean commentsDisabled = false;
private int commentsCount;
public CommentsExtractor getCommentsExtractor() {
return commentsExtractor;
}
public void setCommentsExtractor(final CommentsExtractor commentsExtractor) {
this.commentsExtractor = commentsExtractor;
}
/**
* @return {@code true} if the comments are disabled otherwise {@code false} (default)
* @see CommentsExtractor#isCommentsDisabled()
*/
public boolean isCommentsDisabled() {
return commentsDisabled;
}
/**
* @param commentsDisabled {@code true} if the comments are disabled otherwise {@code false}
*/
public void setCommentsDisabled(final boolean commentsDisabled) {
this.commentsDisabled = commentsDisabled;
}
/**
* Returns the total number of comments.
*
* @return the total number of comments
*/
public int getCommentsCount() {
return commentsCount;
}
/**
* Sets the total number of comments.
*
* @param commentsCount the commentsCount to set.
*/
public void setCommentsCount(final int commentsCount) {
this.commentsCount = commentsCount;
}
}

View File

@ -0,0 +1,197 @@
package org.schabi.newpipe.extractor.comments;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.localization.DateWrapper;
import org.schabi.newpipe.extractor.stream.Description;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.List;
public class CommentsInfoItem extends InfoItem {
private String commentId;
@Nonnull
private Description commentText = Description.EMPTY_DESCRIPTION;
private String uploaderName;
@Nonnull
private List<Image> uploaderAvatars = List.of();
private String uploaderUrl;
private boolean uploaderVerified;
private String textualUploadDate;
@Nullable
private DateWrapper uploadDate;
private int likeCount;
private String textualLikeCount;
private boolean heartedByUploader;
private boolean pinned;
private int streamPosition;
private int replyCount;
@Nullable
private Page replies;
private boolean isChannelOwner;
private boolean creatorReply;
public static final int NO_LIKE_COUNT = -1;
public static final int NO_STREAM_POSITION = -1;
public static final int UNKNOWN_REPLY_COUNT = -1;
public CommentsInfoItem(final int serviceId, final String url, final String name) {
super(InfoType.COMMENT, serviceId, url, name);
}
public String getCommentId() {
return commentId;
}
public void setCommentId(final String commentId) {
this.commentId = commentId;
}
@Nonnull
public Description getCommentText() {
return commentText;
}
public void setCommentText(@Nonnull final Description commentText) {
this.commentText = commentText;
}
public String getUploaderName() {
return uploaderName;
}
public void setUploaderName(final String uploaderName) {
this.uploaderName = uploaderName;
}
@Nonnull
public List<Image> getUploaderAvatars() {
return uploaderAvatars;
}
public void setUploaderAvatars(@Nonnull final List<Image> uploaderAvatars) {
this.uploaderAvatars = uploaderAvatars;
}
public String getUploaderUrl() {
return uploaderUrl;
}
public void setUploaderUrl(final String uploaderUrl) {
this.uploaderUrl = uploaderUrl;
}
public String getTextualUploadDate() {
return textualUploadDate;
}
public void setTextualUploadDate(final String textualUploadDate) {
this.textualUploadDate = textualUploadDate;
}
@Nullable
public DateWrapper getUploadDate() {
return uploadDate;
}
public void setUploadDate(@Nullable final DateWrapper uploadDate) {
this.uploadDate = uploadDate;
}
/**
* @return the comment's like count or {@link CommentsInfoItem#NO_LIKE_COUNT} if it is
* unavailable
*/
public int getLikeCount() {
return likeCount;
}
public void setLikeCount(final int likeCount) {
this.likeCount = likeCount;
}
public String getTextualLikeCount() {
return textualLikeCount;
}
public void setTextualLikeCount(final String textualLikeCount) {
this.textualLikeCount = textualLikeCount;
}
public void setHeartedByUploader(final boolean isHeartedByUploader) {
this.heartedByUploader = isHeartedByUploader;
}
public boolean isHeartedByUploader() {
return this.heartedByUploader;
}
public boolean isPinned() {
return pinned;
}
public void setPinned(final boolean pinned) {
this.pinned = pinned;
}
public void setUploaderVerified(final boolean uploaderVerified) {
this.uploaderVerified = uploaderVerified;
}
public boolean isUploaderVerified() {
return uploaderVerified;
}
public void setStreamPosition(final int streamPosition) {
this.streamPosition = streamPosition;
}
/**
* Get the playback position of the stream to which this comment belongs.
* This is not supported by all services.
*
* @return the playback position in seconds or {@link #NO_STREAM_POSITION} if not available
*/
public int getStreamPosition() {
return streamPosition;
}
public void setReplyCount(final int replyCount) {
this.replyCount = replyCount;
}
public int getReplyCount() {
return replyCount;
}
public void setReplies(@Nullable final Page replies) {
this.replies = replies;
}
@Nullable
public Page getReplies() {
return this.replies;
}
public void setChannelOwner(final boolean channelOwner) {
this.isChannelOwner = channelOwner;
}
public boolean isChannelOwner() {
return isChannelOwner;
}
public void setCreatorReply(final boolean creatorReply) {
this.creatorReply = creatorReply;
}
public boolean hasCreatorReply() {
return creatorReply;
}
}

View File

@ -0,0 +1,152 @@
package org.schabi.newpipe.extractor.comments;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.InfoItemExtractor;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.localization.DateWrapper;
import org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeCommentsInfoItemExtractor;
import org.schabi.newpipe.extractor.stream.Description;
import org.schabi.newpipe.extractor.stream.StreamExtractor;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.List;
public interface CommentsInfoItemExtractor extends InfoItemExtractor {
/**
* Return the like count of the comment,
* or {@link CommentsInfoItem#NO_LIKE_COUNT} if it is unavailable.
*
* <br>
* <p>
* NOTE: Currently only implemented for YT {@link
* YoutubeCommentsInfoItemExtractor#getLikeCount()}
* with limitations (only approximate like count is returned)
*
* @return the comment's like count
* or {@link CommentsInfoItem#NO_LIKE_COUNT} if it is unavailable
* @see StreamExtractor#getLikeCount()
*/
default int getLikeCount() throws ParsingException {
return CommentsInfoItem.NO_LIKE_COUNT;
}
/**
* The unmodified like count given by the service
* <br>
* It may be language dependent
*/
default String getTextualLikeCount() throws ParsingException {
return "";
}
/**
* The text of the comment
*/
@Nonnull
default Description getCommentText() throws ParsingException {
return Description.EMPTY_DESCRIPTION;
}
/**
* The upload date given by the service, unmodified
*
* @see StreamExtractor#getTextualUploadDate()
*/
default String getTextualUploadDate() throws ParsingException {
return "";
}
/**
* The upload date wrapped with DateWrapper class
*
* @see StreamExtractor#getUploadDate()
*/
@Nullable
default DateWrapper getUploadDate() throws ParsingException {
return null;
}
default String getCommentId() throws ParsingException {
return "";
}
default String getUploaderUrl() throws ParsingException {
return "";
}
default String getUploaderName() throws ParsingException {
return "";
}
@Nonnull
default List<Image> getUploaderAvatars() throws ParsingException {
return List.of();
}
/**
* Whether the comment has been hearted by the uploader
*/
default boolean isHeartedByUploader() throws ParsingException {
return false;
}
/**
* Whether the comment is pinned
*/
default boolean isPinned() throws ParsingException {
return false;
}
/**
* Whether the uploader is verified by the service
*/
default boolean isUploaderVerified() throws ParsingException {
return false;
}
/**
* The playback position of the stream to which this comment belongs.
*
* @see CommentsInfoItem#getStreamPosition()
*/
default int getStreamPosition() throws ParsingException {
return CommentsInfoItem.NO_STREAM_POSITION;
}
/**
* The count of comment replies.
*
* @return the count of the replies
* or {@link CommentsInfoItem#UNKNOWN_REPLY_COUNT} if replies are not supported
*/
default int getReplyCount() throws ParsingException {
return CommentsInfoItem.UNKNOWN_REPLY_COUNT;
}
/**
* The continuation page which is used to get comment replies from.
*
* @return the continuation Page for the replies, or null if replies are not supported
*/
@Nullable
default Page getReplies() throws ParsingException {
return null;
}
/**
* Whether the comment was made by the channel owner.
*/
default boolean isChannelOwner() throws ParsingException {
return false;
}
/**
* Whether the comment was replied to by the creator.
*/
default boolean hasCreatorReply() throws ParsingException {
return false;
}
}

View File

@ -0,0 +1,133 @@
package org.schabi.newpipe.extractor.comments;
import org.schabi.newpipe.extractor.InfoItemsCollector;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import java.util.ArrayList;
import java.util.List;
public final class CommentsInfoItemsCollector
extends InfoItemsCollector<CommentsInfoItem, CommentsInfoItemExtractor> {
public CommentsInfoItemsCollector(final int serviceId) {
super(serviceId);
}
@Override
public CommentsInfoItem extract(final CommentsInfoItemExtractor extractor)
throws ParsingException {
final CommentsInfoItem resultItem = new CommentsInfoItem(
getServiceId(), extractor.getUrl(), extractor.getName());
// optional information
try {
resultItem.setCommentId(extractor.getCommentId());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setCommentText(extractor.getCommentText());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setUploaderName(extractor.getUploaderName());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setUploaderAvatars(extractor.getUploaderAvatars());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setUploaderUrl(extractor.getUploaderUrl());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setTextualUploadDate(extractor.getTextualUploadDate());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setUploadDate(extractor.getUploadDate());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setLikeCount(extractor.getLikeCount());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setTextualLikeCount(extractor.getTextualLikeCount());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setThumbnails(extractor.getThumbnails());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setHeartedByUploader(extractor.isHeartedByUploader());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setPinned(extractor.isPinned());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setStreamPosition(extractor.getStreamPosition());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setReplyCount(extractor.getReplyCount());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setReplies(extractor.getReplies());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setChannelOwner(extractor.isChannelOwner());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setCreatorReply(extractor.hasCreatorReply());
} catch (final Exception e) {
addError(e);
}
return resultItem;
}
@Override
public void commit(final CommentsInfoItemExtractor extractor) {
try {
addItem(extract(extractor));
} catch (final Exception e) {
addError(e);
}
}
public List<CommentsInfoItem> getCommentsInfoItemList() {
return new ArrayList<>(super.getItems());
}
}

View File

@ -0,0 +1,246 @@
package org.schabi.newpipe.extractor.downloader;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.exceptions.ReCaptchaException;
import org.schabi.newpipe.extractor.localization.Localization;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.io.IOException;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
/**
* A base for downloader implementations that NewPipe will use
* to download needed resources during extraction.
*/
public abstract class Downloader {
/**
* Do a GET request to get the resource that the url is pointing to.<br>
* <br>
* This method calls {@link #get(String, Map, Localization)} with the default preferred
* localization. It should only be used when the resource that will be fetched won't be affected
* by the localization.
*
* @param url the URL that is pointing to the wanted resource
* @return the result of the GET request
*/
public Response get(final String url) throws IOException, ReCaptchaException {
return get(url, null, NewPipe.getPreferredLocalization());
}
/**
* Do a GET request to get the resource that the url is pointing to.<br>
* <br>
* It will set the {@code Accept-Language} header to the language of the localization parameter.
*
* @param url the URL that is pointing to the wanted resource
* @param localization the source of the value of the {@code Accept-Language} header
* @return the result of the GET request
*/
public Response get(final String url, final Localization localization)
throws IOException, ReCaptchaException {
return get(url, null, localization);
}
/**
* Do a GET request with the specified headers.
*
* @param url the URL that is pointing to the wanted resource
* @param headers a list of headers that will be used in the request.
* Any default headers <b>should</b> be overridden by these.
* @return the result of the GET request
*/
public Response get(final String url, @Nullable final Map<String, List<String>> headers)
throws IOException, ReCaptchaException {
return get(url, headers, NewPipe.getPreferredLocalization());
}
/**
* Do a GET request with the specified headers.<br>
* <br>
* It will set the {@code Accept-Language} header to the language of the localization parameter.
*
* @param url the URL that is pointing to the wanted resource
* @param headers a list of headers that will be used in the request.
* Any default headers <b>should</b> be overridden by these.
* @param localization the source of the value of the {@code Accept-Language} header
* @return the result of the GET request
*/
public Response get(final String url,
@Nullable final Map<String, List<String>> headers,
final Localization localization)
throws IOException, ReCaptchaException {
return execute(Request.newBuilder()
.get(url)
.headers(headers)
.localization(localization)
.build());
}
/**
* Do a HEAD request.
*
* @param url the URL that is pointing to the wanted resource
* @return the result of the HEAD request
*/
public Response head(final String url) throws IOException, ReCaptchaException {
return head(url, null);
}
/**
* Do a HEAD request with the specified headers.
*
* @param url the URL that is pointing to the wanted resource
* @param headers a list of headers that will be used in the request.
* Any default headers <b>should</b> be overridden by these.
* @return the result of the HEAD request
*/
public Response head(final String url, @Nullable final Map<String, List<String>> headers)
throws IOException, ReCaptchaException {
return execute(Request.newBuilder()
.head(url)
.headers(headers)
.build());
}
/**
* Do a POST request with the specified headers, sending the data array.
*
* @param url the URL that is pointing to the wanted resource
* @param headers a list of headers that will be used in the request.
* Any default headers <b>should</b> be overridden by these.
* @param dataToSend byte array that will be sent when doing the request.
* @return the result of the POST request
*/
public Response post(final String url,
@Nullable final Map<String, List<String>> headers,
@Nullable final byte[] dataToSend)
throws IOException, ReCaptchaException {
return post(url, headers, dataToSend, NewPipe.getPreferredLocalization());
}
/**
* Do a POST request with the specified headers, sending the data array.
* <br>
* It will set the {@code Accept-Language} header to the language of the localization parameter.
*
* @param url the URL that is pointing to the wanted resource
* @param headers a list of headers that will be used in the request.
* Any default headers <b>should</b> be overridden by these.
* @param dataToSend byte array that will be sent when doing the request.
* @param localization the source of the value of the {@code Accept-Language} header
* @return the result of the POST request
*/
public Response post(final String url,
@Nullable final Map<String, List<String>> headers,
@Nullable final byte[] dataToSend,
final Localization localization)
throws IOException, ReCaptchaException {
return execute(Request.newBuilder()
.post(url, dataToSend)
.headers(headers)
.localization(localization)
.build());
}
/**
* Convenient method to send a POST request using the specified value of the
* {@code Content-Type} header with a given {@link Localization}.
*
* @param url the URL that is pointing to the wanted resource
* @param headers a list of headers that will be used in the request.
* Any default headers <b>should</b> be overridden by these.
* @param dataToSend byte array that will be sent when doing the request.
* @param localization the source of the value of the {@code Accept-Language} header
* @param contentType the mime type of the body sent, which will be set as the value of the
* {@code Content-Type} header
* @return the result of the POST request
* @see #post(String, Map, byte[], Localization)
*/
public Response postWithContentType(final String url,
@Nullable final Map<String, List<String>> headers,
@Nullable final byte[] dataToSend,
final Localization localization,
final String contentType)
throws IOException, ReCaptchaException {
final Map<String, List<String>> actualHeaders = new HashMap<>();
if (headers != null) {
actualHeaders.putAll(headers);
}
actualHeaders.put("Content-Type", Collections.singletonList(contentType));
return post(url, actualHeaders, dataToSend, localization);
}
/**
* Convenient method to send a POST request using the specified value of the
* {@code Content-Type} header.
*
* @param url the URL that is pointing to the wanted resource
* @param headers a list of headers that will be used in the request.
* Any default headers <b>should</b> be overridden by these.
* @param dataToSend byte array that will be sent when doing the request.
* @param contentType the mime type of the body sent, which will be set as the value of the
* {@code Content-Type} header
* @return the result of the POST request
* @see #post(String, Map, byte[], Localization)
*/
public Response postWithContentType(final String url,
@Nullable final Map<String, List<String>> headers,
@Nullable final byte[] dataToSend,
final String contentType)
throws IOException, ReCaptchaException {
return postWithContentType(url, headers, dataToSend, NewPipe.getPreferredLocalization(),
contentType);
}
/**
* Convenient method to send a POST request the JSON mime type as the value of the
* {@code Content-Type} header with a given {@link Localization}.
*
* @param url the URL that is pointing to the wanted resource
* @param headers a list of headers that will be used in the request.
* Any default headers <b>should</b> be overridden by these.
* @param dataToSend byte array that will be sent when doing the request.
* @param localization the source of the value of the {@code Accept-Language} header
* @return the result of the POST request
* @see #post(String, Map, byte[], Localization)
*/
public Response postWithContentTypeJson(final String url,
@Nullable final Map<String, List<String>> headers,
@Nullable final byte[] dataToSend,
final Localization localization)
throws IOException, ReCaptchaException {
return postWithContentType(url, headers, dataToSend, localization, "application/json");
}
/**
* Convenient method to send a POST request the JSON mime type as the value of the
* {@code Content-Type} header.
*
* @param url the URL that is pointing to the wanted resource
* @param headers a list of headers that will be used in the request.
* Any default headers <b>should</b> be overridden by these.
* @param dataToSend byte array that will be sent when doing the request.
* @return the result of the POST request
* @see #post(String, Map, byte[], Localization)
*/
public Response postWithContentTypeJson(final String url,
@Nullable final Map<String, List<String>> headers,
@Nullable final byte[] dataToSend)
throws IOException, ReCaptchaException {
return postWithContentTypeJson(url, headers, dataToSend,
NewPipe.getPreferredLocalization());
}
/**
* Do a request using the specified {@link Request} object.
*
* @return the result of the request
*/
public abstract Response execute(@Nonnull Request request)
throws IOException, ReCaptchaException;
}

View File

@ -0,0 +1,280 @@
package org.schabi.newpipe.extractor.downloader;
import org.schabi.newpipe.extractor.localization.Localization;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
/**
* An object that holds request information used when {@link Downloader#execute(Request) executing}
* a request.
*/
public class Request {
private final String httpMethod;
private final String url;
private final Map<String, List<String>> headers;
@Nullable
private final byte[] dataToSend;
@Nullable
private final Localization localization;
public Request(final String httpMethod,
final String url,
@Nullable final Map<String, List<String>> headers,
@Nullable final byte[] dataToSend,
@Nullable final Localization localization,
final boolean automaticLocalizationHeader) {
this.httpMethod = Objects.requireNonNull(httpMethod, "Request's httpMethod is null");
this.url = Objects.requireNonNull(url, "Request's url is null");
this.dataToSend = dataToSend;
this.localization = localization;
final Map<String, List<String>> actualHeaders = new LinkedHashMap<>();
if (headers != null) {
actualHeaders.putAll(headers);
}
if (automaticLocalizationHeader && localization != null) {
actualHeaders.putAll(getHeadersFromLocalization(localization));
}
this.headers = Collections.unmodifiableMap(actualHeaders);
}
private Request(final Builder builder) {
this(builder.httpMethod, builder.url, builder.headers, builder.dataToSend,
builder.localization, builder.automaticLocalizationHeader);
}
/**
* A http method (i.e. {@code GET, POST, HEAD}).
*/
public String httpMethod() {
return httpMethod;
}
/**
* The URL that is pointing to the wanted resource.
*/
public String url() {
return url;
}
/**
* A list of headers that will be used in the request.<br>
* Any default headers that the implementation may have, <b>should</b> be overridden by these.
*/
public Map<String, List<String>> headers() {
return headers;
}
/**
* An optional byte array that will be sent when doing the request, very commonly used in
* {@code POST} requests.<br>
* <br>
* The implementation should make note of some recommended headers
* (for example, {@code Content-Length} in a post request).
*/
@Nullable
public byte[] dataToSend() {
return dataToSend;
}
/**
* A localization object that should be used when executing a request.<br>
* <br>
* Usually the {@code Accept-Language} will be set to this value (a helper
* method to do this easily: {@link Request#getHeadersFromLocalization(Localization)}).
*/
@Nullable
public Localization localization() {
return localization;
}
public static Builder newBuilder() {
return new Builder();
}
public static final class Builder {
private String httpMethod;
private String url;
private final Map<String, List<String>> headers = new LinkedHashMap<>();
private byte[] dataToSend;
private Localization localization;
private boolean automaticLocalizationHeader = true;
public Builder() {
}
/**
* A http method (i.e. {@code GET, POST, HEAD}).
*/
public Builder httpMethod(final String httpMethodToSet) {
this.httpMethod = httpMethodToSet;
return this;
}
/**
* The URL that is pointing to the wanted resource.
*/
public Builder url(final String urlToSet) {
this.url = urlToSet;
return this;
}
/**
* A list of headers that will be used in the request.<br>
* Any default headers that the implementation may have, <b>should</b> be overridden by
* these.
*/
public Builder headers(@Nullable final Map<String, List<String>> headersToSet) {
this.headers.clear();
if (headersToSet != null) {
this.headers.putAll(headersToSet);
}
return this;
}
/**
* An optional byte array that will be sent when doing the request, very commonly used in
* {@code POST} requests.<br>
* <br>
* The implementation should make note of some recommended headers
* (for example, {@code Content-Length} in a post request).
*/
public Builder dataToSend(final byte[] dataToSendToSet) {
this.dataToSend = dataToSendToSet;
return this;
}
/**
* A localization object that should be used when executing a request.<br>
* <br>
* Usually the {@code Accept-Language} will be set to this value (a helper
* method to do this easily: {@link Request#getHeadersFromLocalization(Localization)}).
*/
public Builder localization(final Localization localizationToSet) {
this.localization = localizationToSet;
return this;
}
/**
* If localization headers should automatically be included in the request.
*/
public Builder automaticLocalizationHeader(final boolean automaticLocalizationHeaderToSet) {
this.automaticLocalizationHeader = automaticLocalizationHeaderToSet;
return this;
}
public Request build() {
return new Request(this);
}
/*//////////////////////////////////////////////////////////////////////////
// Http Methods Utils
//////////////////////////////////////////////////////////////////////////*/
public Builder get(final String urlToSet) {
this.httpMethod = "GET";
this.url = urlToSet;
return this;
}
public Builder head(final String urlToSet) {
this.httpMethod = "HEAD";
this.url = urlToSet;
return this;
}
public Builder post(final String urlToSet, @Nullable final byte[] dataToSendToSet) {
this.httpMethod = "POST";
this.url = urlToSet;
this.dataToSend = dataToSendToSet;
return this;
}
/*//////////////////////////////////////////////////////////////////////////
// Additional Headers Utils
//////////////////////////////////////////////////////////////////////////*/
public Builder setHeaders(final String headerName, final List<String> headerValueList) {
this.headers.remove(headerName);
this.headers.put(headerName, headerValueList);
return this;
}
public Builder addHeaders(final String headerName, final List<String> headerValueList) {
@Nullable List<String> currentHeaderValueList = this.headers.get(headerName);
if (currentHeaderValueList == null) {
currentHeaderValueList = new ArrayList<>();
}
currentHeaderValueList.addAll(headerValueList);
this.headers.put(headerName, headerValueList);
return this;
}
public Builder setHeader(final String headerName, final String headerValue) {
return setHeaders(headerName, Collections.singletonList(headerValue));
}
public Builder addHeader(final String headerName, final String headerValue) {
return addHeaders(headerName, Collections.singletonList(headerValue));
}
}
/*//////////////////////////////////////////////////////////////////////////
// Utils
//////////////////////////////////////////////////////////////////////////*/
@SuppressWarnings("WeakerAccess")
@Nonnull
public static Map<String, List<String>> getHeadersFromLocalization(
@Nullable final Localization localization) {
if (localization == null) {
return Collections.emptyMap();
}
final String languageCode = localization.getLanguageCode();
final List<String> languageCodeList = Collections.singletonList(
localization.getCountryCode().isEmpty() ? languageCode
: localization.getLocalizationCode() + ", " + languageCode + ";q=0.9");
return Collections.singletonMap("Accept-Language", languageCodeList);
}
/*//////////////////////////////////////////////////////////////////////////
// Generated
//////////////////////////////////////////////////////////////////////////*/
@Override
public boolean equals(final Object o) {
if (this == o) {
return true;
}
if (o == null || getClass() != o.getClass()) {
return false;
}
final Request request = (Request) o;
return httpMethod.equals(request.httpMethod)
&& url.equals(request.url)
&& headers.equals(request.headers)
&& Arrays.equals(dataToSend, request.dataToSend)
&& Objects.equals(localization, request.localization);
}
@Override
public int hashCode() {
int result = Objects.hash(httpMethod, url, headers, localization);
result = 31 * result + Arrays.hashCode(dataToSend);
return result;
}
}

View File

@ -0,0 +1,83 @@
package org.schabi.newpipe.extractor.downloader;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.Collections;
import java.util.List;
import java.util.Map;
/**
* A Data class used to hold the results from requests made by the Downloader implementation.
*/
public class Response {
private final int responseCode;
private final String responseMessage;
private final Map<String, List<String>> responseHeaders;
private final String responseBody;
private final String latestUrl;
public Response(final int responseCode,
final String responseMessage,
@Nullable final Map<String, List<String>> responseHeaders,
@Nullable final String responseBody,
@Nullable final String latestUrl) {
this.responseCode = responseCode;
this.responseMessage = responseMessage;
this.responseHeaders = responseHeaders == null ? Collections.emptyMap() : responseHeaders;
this.responseBody = responseBody == null ? "" : responseBody;
this.latestUrl = latestUrl;
}
public int responseCode() {
return responseCode;
}
public String responseMessage() {
return responseMessage;
}
public Map<String, List<String>> responseHeaders() {
return responseHeaders;
}
@Nonnull
public String responseBody() {
return responseBody;
}
/**
* Used for detecting a possible redirection, limited to the latest one.
*
* @return latest url known right before this response object was created
*/
@Nonnull
public String latestUrl() {
return latestUrl;
}
/*//////////////////////////////////////////////////////////////////////////
// Utils
//////////////////////////////////////////////////////////////////////////*/
/**
* For easy access to some header value that (usually) don't repeat itself.
* <p>For getting all the values associated to the header, use {@link #responseHeaders()} (e.g.
* {@code Set-Cookie}).
*
* @param name the name of the header
* @return the first value assigned to this header
*/
@Nullable
public String getHeader(final String name) {
for (final Map.Entry<String, List<String>> headerEntry : responseHeaders.entrySet()) {
final String key = headerEntry.getKey();
if (key != null && key.equalsIgnoreCase(name) && !headerEntry.getValue().isEmpty()) {
return headerEntry.getValue().get(0);
}
}
return null;
}
}

View File

@ -0,0 +1,31 @@
package org.schabi.newpipe.extractor.exceptions;
public class AccountTerminatedException extends ContentNotAvailableException {
private Reason reason = Reason.UNKNOWN;
public AccountTerminatedException(final String message) {
super(message);
}
public AccountTerminatedException(final String message, final Reason reason) {
super(message);
this.reason = reason;
}
public AccountTerminatedException(final String message, final Throwable cause) {
super(message, cause);
}
/**
* The reason for the violation. There should also be more info in the exception's message.
*/
public Reason getReason() {
return reason;
}
public enum Reason {
UNKNOWN,
VIOLATION
}
}

View File

@ -0,0 +1,11 @@
package org.schabi.newpipe.extractor.exceptions;
public class AgeRestrictedContentException extends ContentNotAvailableException {
public AgeRestrictedContentException(final String message) {
super(message);
}
public AgeRestrictedContentException(final String message, final Throwable cause) {
super(message, cause);
}
}

View File

@ -0,0 +1,11 @@
package org.schabi.newpipe.extractor.exceptions;
public class ContentNotAvailableException extends ParsingException {
public ContentNotAvailableException(final String message) {
super(message);
}
public ContentNotAvailableException(final String message, final Throwable cause) {
super(message, cause);
}
}

View File

@ -0,0 +1,11 @@
package org.schabi.newpipe.extractor.exceptions;
public class ContentNotSupportedException extends ParsingException {
public ContentNotSupportedException(final String message) {
super(message);
}
public ContentNotSupportedException(final String message, final Throwable cause) {
super(message, cause);
}
}

View File

@ -1,33 +1,35 @@
package org.schabi.newpipe.extractor.exceptions;
/**
/*
* Created by Christian Schabesberger on 30.01.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* ExtractionException.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ExtractionException.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public class ExtractionException extends Exception {
public ExtractionException(String message) {
public ExtractionException(final String message) {
super(message);
}
public ExtractionException(Throwable cause) {
public ExtractionException(final Throwable cause) {
super(cause);
}
public ExtractionException(String message, Throwable cause) {
public ExtractionException(final String message, final Throwable cause) {
super(message, cause);
}
}
}

View File

@ -1,30 +1,31 @@
package org.schabi.newpipe.extractor.exceptions;
/**
/*
* Created by Christian Schabesberger on 12.09.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* FoundAdException.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* FoundAdException.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public class FoundAdException extends ParsingException {
public FoundAdException(String message) {
public FoundAdException(final String message) {
super(message);
}
public FoundAdException(String message, Throwable cause) {
public FoundAdException(final String message, final Throwable cause) {
super(message, cause);
}
}

View File

@ -0,0 +1,11 @@
package org.schabi.newpipe.extractor.exceptions;
public class GeographicRestrictionException extends ContentNotAvailableException {
public GeographicRestrictionException(final String message) {
super(message);
}
public GeographicRestrictionException(final String message, final Throwable cause) {
super(message, cause);
}
}

View File

@ -0,0 +1,11 @@
package org.schabi.newpipe.extractor.exceptions;
public class PaidContentException extends ContentNotAvailableException {
public PaidContentException(final String message) {
super(message);
}
public PaidContentException(final String message, final Throwable cause) {
super(message, cause);
}
}

View File

@ -1,31 +1,32 @@
package org.schabi.newpipe.extractor.exceptions;
/**
/*
* Created by Christian Schabesberger on 31.01.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* ParsingException.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ParsingException.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public class ParsingException extends ExtractionException {
public ParsingException(String message) {
public ParsingException(final String message) {
super(message);
}
public ParsingException(String message, Throwable cause) {
public ParsingException(final String message, final Throwable cause) {
super(message, cause);
}
}
}

View File

@ -0,0 +1,11 @@
package org.schabi.newpipe.extractor.exceptions;
public class PrivateContentException extends ContentNotAvailableException {
public PrivateContentException(final String message) {
super(message);
}
public PrivateContentException(final String message, final Throwable cause) {
super(message, cause);
}
}

View File

@ -1,27 +1,34 @@
package org.schabi.newpipe.extractor.exceptions;
/**
/*
* Created by beneth <bmauduit@beneth.fr> on 07.12.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* ReCaptchaException.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ReCaptchaException.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public class ReCaptchaException extends ExtractionException {
public ReCaptchaException(String message) {
private final String url;
public ReCaptchaException(final String message, final String url) {
super(message);
this.url = url;
}
public String getUrl() {
return url;
}
}

View File

@ -0,0 +1,11 @@
package org.schabi.newpipe.extractor.exceptions;
public class SoundCloudGoPlusContentException extends ContentNotAvailableException {
public SoundCloudGoPlusContentException() {
super("This track is a SoundCloud Go+ track");
}
public SoundCloudGoPlusContentException(final Throwable cause) {
super("This track is a SoundCloud Go+ track", cause);
}
}

View File

@ -0,0 +1,7 @@
package org.schabi.newpipe.extractor.exceptions;
public final class UnsupportedTabException extends UnsupportedOperationException {
public UnsupportedTabException(final String unsupportedTab) {
super("Unsupported tab " + unsupportedTab);
}
}

View File

@ -0,0 +1,11 @@
package org.schabi.newpipe.extractor.exceptions;
public class YoutubeMusicPremiumContentException extends ContentNotAvailableException {
public YoutubeMusicPremiumContentException() {
super("This video is a YouTube Music Premium video");
}
public YoutubeMusicPremiumContentException(final Throwable cause) {
super("This video is a YouTube Music Premium video", cause);
}
}

View File

@ -0,0 +1,17 @@
package org.schabi.newpipe.extractor.feed;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
/**
* This class helps to extract items from lightweight feeds that the services may provide.
* <p>
* YouTube is an example of a service that has this alternative available.
*/
public abstract class FeedExtractor extends ListExtractor<StreamInfoItem> {
public FeedExtractor(final StreamingService service, final ListLinkHandler listLinkHandler) {
super(service, listLinkHandler);
}
}

View File

@ -0,0 +1,62 @@
package org.schabi.newpipe.extractor.feed;
import org.schabi.newpipe.extractor.ListExtractor.InfoItemsPage;
import org.schabi.newpipe.extractor.ListInfo;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import org.schabi.newpipe.extractor.utils.ExtractorHelper;
import java.io.IOException;
import java.util.List;
public class FeedInfo extends ListInfo<StreamInfoItem> {
public FeedInfo(final int serviceId,
final String id,
final String url,
final String originalUrl,
final String name,
final List<String> contentFilter,
final String sortFilter) {
super(serviceId, id, url, originalUrl, name, contentFilter, sortFilter);
}
public static FeedInfo getInfo(final String url) throws IOException, ExtractionException {
return getInfo(NewPipe.getServiceByUrl(url), url);
}
public static FeedInfo getInfo(final StreamingService service, final String url)
throws IOException, ExtractionException {
final FeedExtractor extractor = service.getFeedExtractor(url);
if (extractor == null) {
throw new IllegalArgumentException("Service \"" + service.getServiceInfo().getName()
+ "\" doesn't support FeedExtractor.");
}
extractor.fetchPage();
return getInfo(extractor);
}
public static FeedInfo getInfo(final FeedExtractor extractor)
throws IOException, ExtractionException {
extractor.fetchPage();
final int serviceId = extractor.getServiceId();
final String id = extractor.getId();
final String url = extractor.getUrl();
final String originalUrl = extractor.getOriginalUrl();
final String name = extractor.getName();
final FeedInfo info = new FeedInfo(serviceId, id, url, originalUrl, name, null, null);
final InfoItemsPage<StreamInfoItem> itemsPage
= ExtractorHelper.getItemsPageOrLogError(info, extractor);
info.setRelatedItems(itemsPage.getItems());
info.setNextPage(itemsPage.getNextPage());
return info;
}
}

View File

@ -0,0 +1,57 @@
package org.schabi.newpipe.extractor.kiosk;
/*
* Created by Christian Schabesberger on 12.08.17.
*
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* KioskExtractor.java is part of NewPipe Extractor.
*
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import javax.annotation.Nonnull;
public abstract class KioskExtractor<T extends InfoItem> extends ListExtractor<T> {
private final String id;
public KioskExtractor(final StreamingService streamingService,
final ListLinkHandler linkHandler,
final String kioskId) {
super(streamingService, linkHandler);
this.id = kioskId;
}
@Nonnull
@Override
public String getId() {
return id;
}
/**
* Id should be the name of the kiosk, tho Id is used for identifying it in the frontend,
* so id should be kept in english.
* In order to get the name of the kiosk in the desired language we have to
* crawl if from the website.
* @return the translated version of id
*/
@Nonnull
@Override
public abstract String getName() throws ParsingException;
}

View File

@ -0,0 +1,75 @@
package org.schabi.newpipe.extractor.kiosk;
/*
* Created by Christian Schabesberger on 12.08.17.
*
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* KioskInfo.java is part of NewPipe Extractor.
*
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.ListInfo;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import org.schabi.newpipe.extractor.utils.ExtractorHelper;
import java.io.IOException;
public final class KioskInfo extends ListInfo<StreamInfoItem> {
private KioskInfo(final int serviceId, final ListLinkHandler linkHandler, final String name) {
super(serviceId, linkHandler, name);
}
public static ListExtractor.InfoItemsPage<StreamInfoItem> getMoreItems(
final StreamingService service, final String url, final Page page)
throws IOException, ExtractionException {
return service.getKioskList().getExtractorByUrl(url, page).getPage(page);
}
public static KioskInfo getInfo(final String url) throws IOException, ExtractionException {
return getInfo(NewPipe.getServiceByUrl(url), url);
}
public static KioskInfo getInfo(final StreamingService service, final String url)
throws IOException, ExtractionException {
final KioskExtractor extractor = service.getKioskList().getExtractorByUrl(url, null);
extractor.fetchPage();
return getInfo(extractor);
}
/**
* Get KioskInfo from KioskExtractor
*
* @param extractor an extractor where fetchPage() was already got called on.
*/
public static KioskInfo getInfo(final KioskExtractor extractor) throws ExtractionException {
final KioskInfo info = new KioskInfo(extractor.getServiceId(),
extractor.getLinkHandler(),
extractor.getName());
final ListExtractor.InfoItemsPage<StreamInfoItem> itemsPage
= ExtractorHelper.getItemsPageOrLogError(info, extractor);
info.setRelatedItems(itemsPage.getItems());
info.setNextPage(itemsPage.getNextPage());
return info;
}
}

View File

@ -0,0 +1,156 @@
package org.schabi.newpipe.extractor.kiosk;
import static org.schabi.newpipe.extractor.utils.Utils.isNullOrEmpty;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import org.schabi.newpipe.extractor.localization.ContentCountry;
import org.schabi.newpipe.extractor.localization.Localization;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import java.util.Set;
import javax.annotation.Nullable;
public class KioskList {
public interface KioskExtractorFactory {
KioskExtractor createNewKiosk(StreamingService streamingService,
String url,
String kioskId)
throws ExtractionException, IOException;
}
private final StreamingService service;
private final HashMap<String, KioskEntry> kioskList = new HashMap<>();
private String defaultKiosk = null;
@Nullable
private Localization forcedLocalization;
@Nullable
private ContentCountry forcedContentCountry;
private static class KioskEntry {
KioskEntry(final KioskExtractorFactory ef, final ListLinkHandlerFactory h) {
extractorFactory = ef;
handlerFactory = h;
}
final KioskExtractorFactory extractorFactory;
final ListLinkHandlerFactory handlerFactory;
}
public KioskList(final StreamingService service) {
this.service = service;
}
public void addKioskEntry(final KioskExtractorFactory extractorFactory,
final ListLinkHandlerFactory handlerFactory,
final String id)
throws Exception {
if (kioskList.get(id) != null) {
throw new Exception("Kiosk with type " + id + " already exists.");
}
kioskList.put(id, new KioskEntry(extractorFactory, handlerFactory));
}
public void setDefaultKiosk(final String kioskType) {
defaultKiosk = kioskType;
}
public KioskExtractor getDefaultKioskExtractor()
throws ExtractionException, IOException {
return getDefaultKioskExtractor(null);
}
public KioskExtractor getDefaultKioskExtractor(final Page nextPage)
throws ExtractionException, IOException {
return getDefaultKioskExtractor(nextPage, NewPipe.getPreferredLocalization());
}
public KioskExtractor getDefaultKioskExtractor(final Page nextPage,
final Localization localization)
throws ExtractionException, IOException {
if (!isNullOrEmpty(defaultKiosk)) {
return getExtractorById(defaultKiosk, nextPage, localization);
} else {
final String first = kioskList.keySet().stream().findAny().orElse(null);
if (first != null) {
// if not set get any entry
return getExtractorById(first, nextPage, localization);
} else {
return null;
}
}
}
public String getDefaultKioskId() {
return defaultKiosk;
}
public KioskExtractor getExtractorById(final String kioskId, final Page nextPage)
throws ExtractionException, IOException {
return getExtractorById(kioskId, nextPage, NewPipe.getPreferredLocalization());
}
public KioskExtractor getExtractorById(final String kioskId,
final Page nextPage,
final Localization localization)
throws ExtractionException, IOException {
final KioskEntry ke = kioskList.get(kioskId);
if (ke == null) {
throw new ExtractionException("No kiosk found with the type: " + kioskId);
} else {
final KioskExtractor kioskExtractor = ke.extractorFactory.createNewKiosk(service,
ke.handlerFactory.fromId(kioskId).getUrl(), kioskId);
if (forcedLocalization != null) {
kioskExtractor.forceLocalization(forcedLocalization);
}
if (forcedContentCountry != null) {
kioskExtractor.forceContentCountry(forcedContentCountry);
}
return kioskExtractor;
}
}
public Set<String> getAvailableKiosks() {
return kioskList.keySet();
}
public KioskExtractor getExtractorByUrl(final String url, final Page nextPage)
throws ExtractionException, IOException {
return getExtractorByUrl(url, nextPage, NewPipe.getPreferredLocalization());
}
public KioskExtractor getExtractorByUrl(final String url,
final Page nextPage,
final Localization localization)
throws ExtractionException, IOException {
for (final Map.Entry<String, KioskEntry> e : kioskList.entrySet()) {
final KioskEntry ke = e.getValue();
if (ke.handlerFactory.acceptUrl(url)) {
return getExtractorById(ke.handlerFactory.getId(url), nextPage, localization);
}
}
throw new ExtractionException("Could not find a kiosk that fits to the url: " + url);
}
public ListLinkHandlerFactory getListLinkHandlerFactoryByType(final String type) {
return kioskList.get(type).handlerFactory;
}
public void forceLocalization(@Nullable final Localization localization) {
this.forcedLocalization = localization;
}
public void forceContentCountry(@Nullable final ContentCountry contentCountry) {
this.forcedContentCountry = contentCountry;
}
}

View File

@ -0,0 +1,38 @@
package org.schabi.newpipe.extractor.linkhandler;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.utils.Utils;
import java.io.Serializable;
public class LinkHandler implements Serializable {
protected final String originalUrl;
protected final String url;
protected final String id;
public LinkHandler(final String originalUrl, final String url, final String id) {
this.originalUrl = originalUrl;
this.url = url;
this.id = id;
}
public LinkHandler(final LinkHandler handler) {
this(handler.originalUrl, handler.url, handler.id);
}
public String getOriginalUrl() {
return originalUrl;
}
public String getUrl() {
return url;
}
public String getId() {
return id;
}
public String getBaseUrl() throws ParsingException {
return Utils.getBaseUrl(url);
}
}

View File

@ -0,0 +1,107 @@
package org.schabi.newpipe.extractor.linkhandler;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.utils.Utils;
import java.util.Objects;
/*
* Created by Christian Schabesberger on 26.07.16.
*
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* LinkHandlerFactory.java is part of NewPipe Extractor.
*
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public abstract class LinkHandlerFactory {
///////////////////////////////////
// To Override
///////////////////////////////////
public abstract String getId(String url) throws ParsingException, UnsupportedOperationException;
public abstract String getUrl(String id) throws ParsingException, UnsupportedOperationException;
public abstract boolean onAcceptUrl(String url) throws ParsingException;
public String getUrl(final String id, final String baseUrl)
throws ParsingException, UnsupportedOperationException {
return getUrl(id);
}
///////////////////////////////////
// Logic
///////////////////////////////////
/**
* Builds a {@link LinkHandler} from a url.<br>
* Be sure to call {@link Utils#followGoogleRedirectIfNeeded(String)} on the url if overriding
* this function.
*
* @param url the url to extract path and id from
* @return a {@link LinkHandler} complete with information
*/
public LinkHandler fromUrl(final String url) throws ParsingException {
if (Utils.isNullOrEmpty(url)) {
throw new IllegalArgumentException("The url is null or empty");
}
final String polishedUrl = Utils.followGoogleRedirectIfNeeded(url);
final String baseUrl = Utils.getBaseUrl(polishedUrl);
return fromUrl(polishedUrl, baseUrl);
}
/**
* Builds a {@link LinkHandler} from an URL and a base URL. The URL is expected to be already
* polished from Google search redirects (otherwise how could {@code baseUrl} have been
* extracted?).<br>
* So do not call {@link Utils#followGoogleRedirectIfNeeded(String)} on the URL if overriding
* this function, since that should be done in {@link #fromUrl(String)}.
*
* @param url the URL without Google search redirects to extract id from
* @param baseUrl the base URL
* @return a {@link LinkHandler} complete with information
*/
public LinkHandler fromUrl(final String url, final String baseUrl) throws ParsingException {
Objects.requireNonNull(url, "URL cannot be null");
if (!acceptUrl(url)) {
throw new ParsingException("URL not accepted: " + url);
}
final String id = getId(url);
return new LinkHandler(url, getUrl(id, baseUrl), id);
}
public LinkHandler fromId(final String id) throws ParsingException {
Objects.requireNonNull(id, "ID cannot be null");
final String url = getUrl(id);
return new LinkHandler(url, url, id);
}
public LinkHandler fromId(final String id, final String baseUrl) throws ParsingException {
Objects.requireNonNull(id, "ID cannot be null");
final String url = getUrl(id, baseUrl);
return new LinkHandler(url, url, id);
}
/**
* When a VIEW_ACTION is caught this function will test if the url delivered within the calling
* Intent was meant to be watched with this Service.
* Return false if this service shall not allow to be called through ACTIONs.
*/
public boolean acceptUrl(final String url) throws ParsingException {
return onAcceptUrl(url);
}
}

View File

@ -0,0 +1,43 @@
package org.schabi.newpipe.extractor.linkhandler;
import java.util.Collections;
import java.util.List;
public class ListLinkHandler extends LinkHandler {
protected final List<String> contentFilters;
protected final String sortFilter;
public ListLinkHandler(final String originalUrl,
final String url,
final String id,
final List<String> contentFilters,
final String sortFilter) {
super(originalUrl, url, id);
this.contentFilters = Collections.unmodifiableList(contentFilters);
this.sortFilter = sortFilter;
}
public ListLinkHandler(final ListLinkHandler handler) {
this(handler.originalUrl,
handler.url,
handler.id,
handler.contentFilters,
handler.sortFilter);
}
public ListLinkHandler(final LinkHandler handler) {
this(handler.originalUrl,
handler.url,
handler.id,
Collections.emptyList(),
"");
}
public List<String> getContentFilters() {
return contentFilters;
}
public String getSortFilter() {
return sortFilter;
}
}

View File

@ -0,0 +1,104 @@
package org.schabi.newpipe.extractor.linkhandler;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.utils.Utils;
import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
public abstract class ListLinkHandlerFactory extends LinkHandlerFactory {
///////////////////////////////////
// To Override
///////////////////////////////////
public abstract String getUrl(String id, List<String> contentFilter, String sortFilter)
throws ParsingException, UnsupportedOperationException;
public String getUrl(final String id,
final List<String> contentFilter,
final String sortFilter,
final String baseUrl)
throws ParsingException, UnsupportedOperationException {
return getUrl(id, contentFilter, sortFilter);
}
///////////////////////////////////
// Logic
///////////////////////////////////
@Override
public ListLinkHandler fromUrl(final String url) throws ParsingException {
final String polishedUrl = Utils.followGoogleRedirectIfNeeded(url);
final String baseUrl = Utils.getBaseUrl(polishedUrl);
return fromUrl(polishedUrl, baseUrl);
}
@Override
public ListLinkHandler fromUrl(final String url, final String baseUrl) throws ParsingException {
Objects.requireNonNull(url, "URL may not be null");
return new ListLinkHandler(super.fromUrl(url, baseUrl));
}
@Override
public ListLinkHandler fromId(final String id) throws ParsingException {
return new ListLinkHandler(super.fromId(id));
}
@Override
public ListLinkHandler fromId(final String id, final String baseUrl) throws ParsingException {
return new ListLinkHandler(super.fromId(id, baseUrl));
}
public ListLinkHandler fromQuery(final String id,
final List<String> contentFilters,
final String sortFilter) throws ParsingException {
final String url = getUrl(id, contentFilters, sortFilter);
return new ListLinkHandler(url, url, id, contentFilters, sortFilter);
}
public ListLinkHandler fromQuery(final String id,
final List<String> contentFilters,
final String sortFilter,
final String baseUrl) throws ParsingException {
final String url = getUrl(id, contentFilters, sortFilter, baseUrl);
return new ListLinkHandler(url, url, id, contentFilters, sortFilter);
}
/**
* For making ListLinkHandlerFactory compatible with LinkHandlerFactory we need to override
* this, however it should not be overridden by the actual implementation.
*
* @return the url corresponding to id without any filters applied
*/
public String getUrl(final String id) throws ParsingException, UnsupportedOperationException {
return getUrl(id, new ArrayList<>(0), "");
}
@Override
public String getUrl(final String id, final String baseUrl) throws ParsingException {
return getUrl(id, new ArrayList<>(0), "", baseUrl);
}
/**
* Will returns content filter the corresponding extractor can handle like "channels", "videos",
* "music", etc.
*
* @return filter that can be applied when building a query for getting a list
*/
public String[] getAvailableContentFilter() {
return new String[0];
}
/**
* Will returns sort filter the corresponding extractor can handle like "A-Z", "oldest first",
* "size", etc.
*
* @return filter that can be applied when building a query for getting a list
*/
public String[] getAvailableSortFilter() {
return new String[0];
}
}

View File

@ -0,0 +1,54 @@
package org.schabi.newpipe.extractor.linkhandler;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import javax.annotation.Nonnull;
import java.io.Serializable;
import java.util.List;
/**
* A {@link ListLinkHandler} which can be used to be returned from {@link
* org.schabi.newpipe.extractor.channel.ChannelInfo#getTabs() ChannelInfo#getTabs()} when a
* specific tab's data has already been fetched.
*
* <p>
* This class allows passing a builder for a {@link ChannelTabExtractor} that can hold references
* to variables.
* </p>
*
* <p>
* Note: a service that wishes to use this class in one of its {@link
* org.schabi.newpipe.extractor.channel.ChannelExtractor ChannelExtractor}s must also add the
* following snippet of code in the service's
* {@link StreamingService#getChannelTabExtractor(ListLinkHandler)}:
* <pre>
* if (linkHandler instanceof ReadyChannelTabListLinkHandler) {
* return ((ReadyChannelTabListLinkHandler) linkHandler).getChannelTabExtractor(this);
* }
* </pre>
*/
public class ReadyChannelTabListLinkHandler extends ListLinkHandler {
public interface ChannelTabExtractorBuilder extends Serializable {
@Nonnull
ChannelTabExtractor build(@Nonnull StreamingService service,
@Nonnull ListLinkHandler linkHandler);
}
private final ChannelTabExtractorBuilder extractorBuilder;
public ReadyChannelTabListLinkHandler(
final String url,
final String channelId,
@Nonnull final String channelTab,
@Nonnull final ChannelTabExtractorBuilder extractorBuilder) {
super(url, url, channelId, List.of(channelTab), "");
this.extractorBuilder = extractorBuilder;
}
@Nonnull
public ChannelTabExtractor getChannelTabExtractor(@Nonnull final StreamingService service) {
return extractorBuilder.build(service, new ListLinkHandler(this));
}
}

View File

@ -0,0 +1,33 @@
package org.schabi.newpipe.extractor.linkhandler;
import java.util.List;
public class SearchQueryHandler extends ListLinkHandler {
public SearchQueryHandler(final String originalUrl,
final String url,
final String searchString,
final List<String> contentFilters,
final String sortFilter) {
super(originalUrl, url, searchString, contentFilters, sortFilter);
}
public SearchQueryHandler(final ListLinkHandler handler) {
this(handler.originalUrl,
handler.url,
handler.id,
handler.contentFilters,
handler.sortFilter);
}
/**
* Returns the search string. Since ListQIHandler is based on ListLinkHandler
* getSearchString() is equivalent to calling getId().
*
* @return the search string
*/
public String getSearchString() {
return getId();
}
}

View File

@ -0,0 +1,50 @@
package org.schabi.newpipe.extractor.linkhandler;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import java.util.Collections;
import java.util.List;
public abstract class SearchQueryHandlerFactory extends ListLinkHandlerFactory {
///////////////////////////////////
// To Override
///////////////////////////////////
@Override
public abstract String getUrl(String query, List<String> contentFilter, String sortFilter)
throws ParsingException, UnsupportedOperationException;
@SuppressWarnings("unused")
public String getSearchString(final String url) {
return "";
}
///////////////////////////////////
// Logic
///////////////////////////////////
@Override
public String getId(final String url) throws ParsingException, UnsupportedOperationException {
return getSearchString(url);
}
@Override
public SearchQueryHandler fromQuery(final String query,
final List<String> contentFilter,
final String sortFilter) throws ParsingException {
return new SearchQueryHandler(super.fromQuery(query, contentFilter, sortFilter));
}
public SearchQueryHandler fromQuery(final String query) throws ParsingException {
return fromQuery(query, Collections.emptyList(), "");
}
/**
* It's not mandatory for NewPipe to handle the Url
*/
@Override
public boolean onAcceptUrl(final String url) {
return false;
}
}

View File

@ -0,0 +1,64 @@
package org.schabi.newpipe.extractor.localization;
import javax.annotation.Nonnull;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
/**
* Represents a country that should be used when fetching content.
* <p>
* YouTube, for example, give different results in their feed depending on which country is
* selected.
* </p>
*/
public class ContentCountry implements Serializable {
public static final ContentCountry DEFAULT =
new ContentCountry(Localization.DEFAULT.getCountryCode());
@Nonnull
private final String countryCode;
public static List<ContentCountry> listFrom(final String... countryCodeList) {
final List<ContentCountry> toReturn = new ArrayList<>();
for (final String countryCode : countryCodeList) {
toReturn.add(new ContentCountry(countryCode));
}
return Collections.unmodifiableList(toReturn);
}
public ContentCountry(@Nonnull final String countryCode) {
this.countryCode = countryCode;
}
@Nonnull
public String getCountryCode() {
return countryCode;
}
@Override
public String toString() {
return getCountryCode();
}
@Override
public boolean equals(final Object o) {
if (this == o) {
return true;
}
if (!(o instanceof ContentCountry)) {
return false;
}
final ContentCountry that = (ContentCountry) o;
return countryCode.equals(that.countryCode);
}
@Override
public int hashCode() {
return countryCode.hashCode();
}
}

View File

@ -0,0 +1,72 @@
package org.schabi.newpipe.extractor.localization;
import javax.annotation.Nonnull;
import java.io.Serializable;
import java.time.OffsetDateTime;
import java.time.ZoneOffset;
import java.util.Calendar;
import java.util.GregorianCalendar;
/**
* A wrapper class that provides a field to describe if the date/time is precise or just an
* approximation.
*/
public class DateWrapper implements Serializable {
@Nonnull
private final OffsetDateTime offsetDateTime;
private final boolean isApproximation;
/**
* @deprecated Use {@link #DateWrapper(OffsetDateTime)} instead.
*/
@Deprecated
public DateWrapper(@Nonnull final Calendar calendar) {
//noinspection deprecation
this(calendar, false);
}
/**
* @deprecated Use {@link #DateWrapper(OffsetDateTime, boolean)} instead.
*/
@Deprecated
public DateWrapper(@Nonnull final Calendar calendar, final boolean isApproximation) {
this(OffsetDateTime.ofInstant(calendar.toInstant(), ZoneOffset.UTC), isApproximation);
}
public DateWrapper(@Nonnull final OffsetDateTime offsetDateTime) {
this(offsetDateTime, false);
}
public DateWrapper(@Nonnull final OffsetDateTime offsetDateTime,
final boolean isApproximation) {
this.offsetDateTime = offsetDateTime.withOffsetSameInstant(ZoneOffset.UTC);
this.isApproximation = isApproximation;
}
/**
* @return the wrapped date/time as a {@link Calendar}.
* @deprecated use {@link #offsetDateTime()} instead.
*/
@Deprecated
@Nonnull
public Calendar date() {
return GregorianCalendar.from(offsetDateTime.toZonedDateTime());
}
/**
* @return the wrapped date/time.
*/
@Nonnull
public OffsetDateTime offsetDateTime() {
return offsetDateTime;
}
/**
* @return if the date is considered is precise or just an approximation (e.g. service only
* returns an approximation like 2 weeks ago instead of a precise date).
*/
public boolean isApproximation() {
return isApproximation;
}
}

View File

@ -0,0 +1,137 @@
package org.schabi.newpipe.extractor.localization;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.utils.LocaleCompat;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
public class Localization implements Serializable {
public static final Localization DEFAULT = new Localization("en", "GB");
@Nonnull
private final String languageCode;
@Nullable
private final String countryCode;
/**
* @param localizationCodeList a list of localization code, formatted like {@link
* #getLocalizationCode()}
* @throws IllegalArgumentException If any of the localizationCodeList is formatted incorrectly
* @return list of Localization objects
*/
@Nonnull
public static List<Localization> listFrom(final String... localizationCodeList) {
final List<Localization> toReturn = new ArrayList<>();
for (final String localizationCode : localizationCodeList) {
toReturn.add(fromLocalizationCode(localizationCode)
.orElseThrow(() -> new IllegalArgumentException(
"Not a localization code: " + localizationCode
)));
}
return Collections.unmodifiableList(toReturn);
}
/**
* @param localizationCode a localization code, formatted like {@link #getLocalizationCode()}
* @return A Localization, if the code was valid.
*/
@Nonnull
public static Optional<Localization> fromLocalizationCode(final String localizationCode) {
return LocaleCompat.forLanguageTag(localizationCode).map(Localization::fromLocale);
}
public Localization(@Nonnull final String languageCode, @Nullable final String countryCode) {
this.languageCode = languageCode;
this.countryCode = countryCode;
}
public Localization(@Nonnull final String languageCode) {
this(languageCode, null);
}
@Nonnull
public String getLanguageCode() {
return languageCode;
}
@Nonnull
public String getCountryCode() {
return countryCode == null ? "" : countryCode;
}
public static Localization fromLocale(@Nonnull final Locale locale) {
return new Localization(locale.getLanguage(), locale.getCountry());
}
/**
* Return a formatted string in the form of: {@code language-Country}, or
* just {@code language} if country is {@code null}.
*
* @return A correctly formatted localizationCode for this localization.
*/
public String getLocalizationCode() {
return languageCode + (countryCode == null ? "" : "-" + countryCode);
}
@Override
public String toString() {
return "Localization[" + getLocalizationCode() + "]";
}
@Override
public boolean equals(final Object o) {
if (this == o) {
return true;
}
if (!(o instanceof Localization)) {
return false;
}
final Localization that = (Localization) o;
return languageCode.equals(that.languageCode)
&& Objects.equals(countryCode, that.countryCode);
}
@Override
public int hashCode() {
int result = languageCode.hashCode();
result = 31 * result + Objects.hashCode(countryCode);
return result;
}
/**
* Converts a three letter language code (ISO 639-2/T) to a Locale
* because limits of Java Locale class.
*
* @param code a three letter language code
* @return the Locale corresponding
*/
public static Locale getLocaleFromThreeLetterCode(@Nonnull final String code)
throws ParsingException {
final String[] languages = Locale.getISOLanguages();
final Map<String, Locale> localeMap = new HashMap<>(languages.length);
for (final String language : languages) {
final Locale locale = new Locale(language);
localeMap.put(locale.getISO3Language(), locale);
}
if (localeMap.containsKey(code)) {
return localeMap.get(code);
} else {
throw new ParsingException(
"Could not get Locale from this three letter language code" + code);
}
}
}

View File

@ -0,0 +1,188 @@
package org.schabi.newpipe.extractor.localization;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.timeago.PatternsHolder;
import org.schabi.newpipe.extractor.utils.Parser;
import java.time.OffsetDateTime;
import java.time.ZoneOffset;
import java.time.temporal.ChronoUnit;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.regex.Pattern;
import java.util.regex.Matcher;
import java.util.regex.MatchResult;
/**
* A helper class that is meant to be used by services that need to parse durations such as
* {@code 23 seconds} and/or upload dates in the format {@code 2 days ago} or similar.
*/
public class TimeAgoParser {
private static final Pattern DURATION_PATTERN = Pattern.compile("(?:(\\d+) )?([A-z]+)");
private final PatternsHolder patternsHolder;
private final OffsetDateTime now;
/**
* Creates a helper to parse upload dates in the format '2 days ago'.
* <p>
* Instantiate a new {@link TimeAgoParser} every time you extract a new batch of items.
* </p>
*
* @param patternsHolder An object that holds the "time ago" patterns, special cases, and the
* language word separator.
*/
public TimeAgoParser(final PatternsHolder patternsHolder) {
this.patternsHolder = patternsHolder;
now = OffsetDateTime.now(ZoneOffset.UTC);
}
/**
* Parses a textual date in the format '2 days ago' into a Calendar representation which is then
* wrapped in a {@link DateWrapper} object.
* <p>
* Beginning with days ago, the date is considered as an approximation.
*
* @param textualDate The original date as provided by the streaming service
* @return The parsed time (can be approximated)
* @throws ParsingException if the time unit could not be recognized
*/
public DateWrapper parse(final String textualDate) throws ParsingException {
for (final Map.Entry<ChronoUnit, Map<String, Integer>> caseUnitEntry
: patternsHolder.specialCases().entrySet()) {
final ChronoUnit chronoUnit = caseUnitEntry.getKey();
for (final Map.Entry<String, Integer> caseMapToAmountEntry
: caseUnitEntry.getValue().entrySet()) {
final String caseText = caseMapToAmountEntry.getKey();
final Integer caseAmount = caseMapToAmountEntry.getValue();
if (textualDateMatches(textualDate, caseText)) {
return getResultFor(caseAmount, chronoUnit);
}
}
}
return getResultFor(parseTimeAgoAmount(textualDate), parseChronoUnit(textualDate));
}
/**
* Parses a textual duration into a duration computer number.
*
* @param textualDuration the textual duration to parse
* @return the textual duration parsed, as a primitive {@code long}
* @throws ParsingException if the textual duration could not be parsed
*/
public long parseDuration(final String textualDuration) throws ParsingException {
// We can't use Matcher.results, as it is only available on Android 14 and above
final Matcher matcher = DURATION_PATTERN.matcher(textualDuration);
final List<MatchResult> results = new ArrayList<>();
while (matcher.find()) {
results.add(matcher.toMatchResult());
}
return results.stream()
.map(match -> {
final String digits = match.group(1);
final String word = match.group(2);
int amount;
try {
amount = Integer.parseInt(digits);
} catch (final NumberFormatException ignored) {
amount = 1;
}
final ChronoUnit unit;
try {
unit = parseChronoUnit(word);
} catch (final ParsingException ignored) {
return 0L;
}
return amount * unit.getDuration().getSeconds();
})
.filter(n -> n > 0)
.reduce(Long::sum)
.orElseThrow(() -> new ParsingException(
"Could not parse duration \"" + textualDuration + "\""));
}
private int parseTimeAgoAmount(final String textualDate) {
try {
return Integer.parseInt(textualDate.replaceAll("\\D+", ""));
} catch (final NumberFormatException ignored) {
// If there is no valid number in the textual date,
// assume it is 1 (as in 'a second ago').
return 1;
}
}
private ChronoUnit parseChronoUnit(final String textualDate) throws ParsingException {
return patternsHolder.asMap().entrySet().stream()
.filter(e -> e.getValue().stream()
.anyMatch(agoPhrase -> textualDateMatches(textualDate, agoPhrase)))
.map(Map.Entry::getKey)
.findFirst()
.orElseThrow(() ->
new ParsingException("Unable to parse the date: " + textualDate));
}
private boolean textualDateMatches(final String textualDate, final String agoPhrase) {
if (textualDate.equals(agoPhrase)) {
return true;
}
if (patternsHolder.wordSeparator().isEmpty()) {
return textualDate.toLowerCase().contains(agoPhrase.toLowerCase());
}
final String escapedPhrase = Pattern.quote(agoPhrase.toLowerCase());
final String escapedSeparator = patternsHolder.wordSeparator().equals(" ")
// From JDK8 \h - Treat horizontal spaces as a normal one
// (non-breaking space, thin space, etc.)
// Also split the string on numbers to be able to parse strings like "2wk"
? "[ \\t\\xA0\\u1680\\u180e\\u2000-\\u200a\\u202f\\u205f\\u3000\\d]"
: Pattern.quote(patternsHolder.wordSeparator());
// (^|separator)pattern($|separator)
// Check if the pattern is surrounded by separators or start/end of the string.
final String pattern =
"(^|" + escapedSeparator + ")" + escapedPhrase + "($|" + escapedSeparator + ")";
return Parser.isMatch(pattern, textualDate.toLowerCase());
}
private DateWrapper getResultFor(final int timeAgoAmount, final ChronoUnit chronoUnit) {
OffsetDateTime offsetDateTime = now;
boolean isApproximation = false;
switch (chronoUnit) {
case SECONDS:
case MINUTES:
case HOURS:
offsetDateTime = offsetDateTime.minus(timeAgoAmount, chronoUnit);
break;
case DAYS:
case WEEKS:
case MONTHS:
offsetDateTime = offsetDateTime.minus(timeAgoAmount, chronoUnit);
isApproximation = true;
break;
case YEARS:
// minusDays is needed to prevent `PrettyTime` from showing '12 months ago'.
offsetDateTime = offsetDateTime.minusYears(timeAgoAmount).minusDays(1);
isApproximation = true;
break;
}
if (isApproximation) {
offsetDateTime = offsetDateTime.truncatedTo(ChronoUnit.HOURS);
}
return new DateWrapper(offsetDateTime, isApproximation);
}
}

View File

@ -0,0 +1,29 @@
package org.schabi.newpipe.extractor.localization;
import org.schabi.newpipe.extractor.timeago.PatternsHolder;
import org.schabi.newpipe.extractor.timeago.PatternsManager;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
public final class TimeAgoPatternsManager {
private TimeAgoPatternsManager() {
}
@Nullable
private static PatternsHolder getPatternsFor(@Nonnull final Localization localization) {
return PatternsManager.getPatterns(localization.getLanguageCode(),
localization.getCountryCode());
}
@Nullable
public static TimeAgoParser getTimeAgoParserFor(@Nonnull final Localization localization) {
final PatternsHolder holder = getPatternsFor(localization);
if (holder == null) {
return null;
}
return new TimeAgoParser(holder);
}
}

View File

@ -0,0 +1,61 @@
package org.schabi.newpipe.extractor.playlist;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.stream.Description;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import javax.annotation.Nonnull;
import java.util.Collections;
import java.util.List;
public abstract class PlaylistExtractor extends ListExtractor<StreamInfoItem> {
public PlaylistExtractor(final StreamingService service, final ListLinkHandler linkHandler) {
super(service, linkHandler);
}
public abstract String getUploaderUrl() throws ParsingException;
public abstract String getUploaderName() throws ParsingException;
@Nonnull
public abstract List<Image> getUploaderAvatars() throws ParsingException;
public abstract boolean isUploaderVerified() throws ParsingException;
public abstract long getStreamCount() throws ParsingException;
@Nonnull
public abstract Description getDescription() throws ParsingException;
@Nonnull
public List<Image> getThumbnails() throws ParsingException {
return Collections.emptyList();
}
@Nonnull
public List<Image> getBanners() throws ParsingException {
return List.of();
}
@Nonnull
public String getSubChannelName() throws ParsingException {
return "";
}
@Nonnull
public String getSubChannelUrl() throws ParsingException {
return "";
}
@Nonnull
public List<Image> getSubChannelAvatars() throws ParsingException {
return List.of();
}
public PlaylistInfo.PlaylistType getPlaylistType() throws ParsingException {
return PlaylistInfo.PlaylistType.NORMAL;
}
}

View File

@ -0,0 +1,280 @@
package org.schabi.newpipe.extractor.playlist;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.ListExtractor.InfoItemsPage;
import org.schabi.newpipe.extractor.ListInfo;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.stream.Description;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import org.schabi.newpipe.extractor.utils.ExtractorHelper;
import javax.annotation.Nonnull;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
public final class PlaylistInfo extends ListInfo<StreamInfoItem> {
/**
* Mixes are handled as particular playlists in NewPipeExtractor. {@link PlaylistType#NORMAL} is
* for non-mixes, while other values are for the different types of mixes. The type of a mix
* depends on how its contents are autogenerated.
*/
public enum PlaylistType {
/**
* A normal playlist (not a mix)
*/
NORMAL,
/**
* A mix made only of streams related to a particular stream, for example YouTube mixes
*/
MIX_STREAM,
/**
* A mix made only of music streams related to a particular stream, for example YouTube
* music mixes
*/
MIX_MUSIC,
/**
* A mix made only of streams from (or related to) the same channel, for example YouTube
* channel mixes
*/
MIX_CHANNEL,
/**
* A mix made only of streams related to a particular (musical) genre, for example YouTube
* genre mixes
*/
MIX_GENRE,
}
@SuppressWarnings("RedundantThrows")
private PlaylistInfo(final int serviceId, final ListLinkHandler linkHandler, final String name)
throws ParsingException {
super(serviceId, linkHandler, name);
}
public static PlaylistInfo getInfo(final String url) throws IOException, ExtractionException {
return getInfo(NewPipe.getServiceByUrl(url), url);
}
public static PlaylistInfo getInfo(final StreamingService service, final String url)
throws IOException, ExtractionException {
final PlaylistExtractor extractor = service.getPlaylistExtractor(url);
extractor.fetchPage();
return getInfo(extractor);
}
public static InfoItemsPage<StreamInfoItem> getMoreItems(final StreamingService service,
final String url,
final Page page)
throws IOException, ExtractionException {
return service.getPlaylistExtractor(url).getPage(page);
}
/**
* Get PlaylistInfo from PlaylistExtractor
*
* @param extractor an extractor where fetchPage() was already got called on.
*/
public static PlaylistInfo getInfo(final PlaylistExtractor extractor)
throws ExtractionException {
final PlaylistInfo info = new PlaylistInfo(
extractor.getServiceId(),
extractor.getLinkHandler(),
extractor.getName());
// collect uploader extraction failures until we are sure this is not
// just a playlist without an uploader
final List<Throwable> uploaderParsingErrors = new ArrayList<>();
try {
info.setOriginalUrl(extractor.getOriginalUrl());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setStreamCount(extractor.getStreamCount());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setDescription(extractor.getDescription());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setThumbnails(extractor.getThumbnails());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setUploaderUrl(extractor.getUploaderUrl());
} catch (final Exception e) {
uploaderParsingErrors.add(e);
}
try {
info.setUploaderName(extractor.getUploaderName());
} catch (final Exception e) {
uploaderParsingErrors.add(e);
}
try {
info.setUploaderAvatars(extractor.getUploaderAvatars());
} catch (final Exception e) {
uploaderParsingErrors.add(e);
}
try {
info.setSubChannelUrl(extractor.getSubChannelUrl());
} catch (final Exception e) {
uploaderParsingErrors.add(e);
}
try {
info.setSubChannelName(extractor.getSubChannelName());
} catch (final Exception e) {
uploaderParsingErrors.add(e);
}
try {
info.setSubChannelAvatars(extractor.getSubChannelAvatars());
} catch (final Exception e) {
uploaderParsingErrors.add(e);
}
try {
info.setBanners(extractor.getBanners());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setPlaylistType(extractor.getPlaylistType());
} catch (final Exception e) {
info.addError(e);
}
// do not fail if everything but the uploader infos could be collected (TODO better comment)
if (!uploaderParsingErrors.isEmpty()
&& (!info.getErrors().isEmpty() || uploaderParsingErrors.size() < 3)) {
info.addAllErrors(uploaderParsingErrors);
}
final InfoItemsPage<StreamInfoItem> itemsPage
= ExtractorHelper.getItemsPageOrLogError(info, extractor);
info.setRelatedItems(itemsPage.getItems());
info.setNextPage(itemsPage.getNextPage());
return info;
}
private String uploaderUrl = "";
private String uploaderName = "";
private String subChannelUrl;
private String subChannelName;
private Description description;
@Nonnull
private List<Image> banners = List.of();
@Nonnull
private List<Image> subChannelAvatars = List.of();
@Nonnull
private List<Image> thumbnails = List.of();
@Nonnull
private List<Image> uploaderAvatars = List.of();
private long streamCount;
private PlaylistType playlistType;
@Nonnull
public List<Image> getThumbnails() {
return thumbnails;
}
public void setThumbnails(@Nonnull final List<Image> thumbnails) {
this.thumbnails = thumbnails;
}
@Nonnull
public List<Image> getBanners() {
return banners;
}
public void setBanners(@Nonnull final List<Image> banners) {
this.banners = banners;
}
public String getUploaderUrl() {
return uploaderUrl;
}
public void setUploaderUrl(final String uploaderUrl) {
this.uploaderUrl = uploaderUrl;
}
public String getUploaderName() {
return uploaderName;
}
public void setUploaderName(final String uploaderName) {
this.uploaderName = uploaderName;
}
@Nonnull
public List<Image> getUploaderAvatars() {
return uploaderAvatars;
}
public void setUploaderAvatars(@Nonnull final List<Image> uploaderAvatars) {
this.uploaderAvatars = uploaderAvatars;
}
public String getSubChannelUrl() {
return subChannelUrl;
}
public void setSubChannelUrl(final String subChannelUrl) {
this.subChannelUrl = subChannelUrl;
}
public String getSubChannelName() {
return subChannelName;
}
public void setSubChannelName(final String subChannelName) {
this.subChannelName = subChannelName;
}
@Nonnull
public List<Image> getSubChannelAvatars() {
return subChannelAvatars;
}
public void setSubChannelAvatars(@Nonnull final List<Image> subChannelAvatars) {
this.subChannelAvatars = subChannelAvatars;
}
public long getStreamCount() {
return streamCount;
}
public void setStreamCount(final long streamCount) {
this.streamCount = streamCount;
}
public Description getDescription() {
return description;
}
public void setDescription(final Description description) {
this.description = description;
}
public PlaylistType getPlaylistType() {
return playlistType;
}
public void setPlaylistType(final PlaylistType playlistType) {
this.playlistType = playlistType;
}
}

View File

@ -0,0 +1,72 @@
package org.schabi.newpipe.extractor.playlist;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.stream.Description;
import javax.annotation.Nullable;
public class PlaylistInfoItem extends InfoItem {
private String uploaderName;
private String uploaderUrl;
private boolean uploaderVerified;
/**
* How many streams this playlist have
*/
private long streamCount = 0;
private Description description;
private PlaylistInfo.PlaylistType playlistType;
public PlaylistInfoItem(final int serviceId, final String url, final String name) {
super(InfoType.PLAYLIST, serviceId, url, name);
}
public String getUploaderName() {
return uploaderName;
}
public void setUploaderName(final String uploaderName) {
this.uploaderName = uploaderName;
}
@Nullable
public String getUploaderUrl() {
return uploaderUrl;
}
public void setUploaderUrl(@Nullable final String uploaderUrl) {
this.uploaderUrl = uploaderUrl;
}
public boolean isUploaderVerified() {
return uploaderVerified;
}
public void setUploaderVerified(final boolean uploaderVerified) {
this.uploaderVerified = uploaderVerified;
}
public long getStreamCount() {
return streamCount;
}
public void setStreamCount(final long streamCount) {
this.streamCount = streamCount;
}
public Description getDescription() {
return description;
}
public void setDescription(final Description description) {
this.description = description;
}
public PlaylistInfo.PlaylistType getPlaylistType() {
return playlistType;
}
public void setPlaylistType(final PlaylistInfo.PlaylistType playlistType) {
this.playlistType = playlistType;
}
}

View File

@ -0,0 +1,53 @@
package org.schabi.newpipe.extractor.playlist;
import org.schabi.newpipe.extractor.InfoItemExtractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.stream.Description;
import javax.annotation.Nonnull;
public interface PlaylistInfoItemExtractor extends InfoItemExtractor {
/**
* Get the uploader name
* @return the uploader name
*/
String getUploaderName() throws ParsingException;
/**
* Get the uploader url
* @return the uploader url
*/
String getUploaderUrl() throws ParsingException;
/**
* Get whether the uploader is verified
* @return whether the uploader is verified
*/
boolean isUploaderVerified() throws ParsingException;
/**
* Get the number of streams
* @return the number of streams
*/
long getStreamCount() throws ParsingException;
/**
* Get the description of the playlist if there is any.
* Otherwise, an {@link Description#EMPTY_DESCRIPTION EMPTY_DESCRIPTION} is returned.
* @return the playlist's description
*/
@Nonnull
default Description getDescription() throws ParsingException {
return Description.EMPTY_DESCRIPTION;
}
/**
* @return the type of this playlist, see {@link PlaylistInfo.PlaylistType} for a description
* of types. If not overridden always returns {@link PlaylistInfo.PlaylistType#NORMAL}.
*/
@Nonnull
default PlaylistInfo.PlaylistType getPlaylistType() throws ParsingException {
return PlaylistInfo.PlaylistType.NORMAL;
}
}

View File

@ -0,0 +1,56 @@
package org.schabi.newpipe.extractor.playlist;
import org.schabi.newpipe.extractor.InfoItemsCollector;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
public class PlaylistInfoItemsCollector
extends InfoItemsCollector<PlaylistInfoItem, PlaylistInfoItemExtractor> {
public PlaylistInfoItemsCollector(final int serviceId) {
super(serviceId);
}
@Override
public PlaylistInfoItem extract(final PlaylistInfoItemExtractor extractor)
throws ParsingException {
final PlaylistInfoItem resultItem = new PlaylistInfoItem(
getServiceId(), extractor.getUrl(), extractor.getName());
try {
resultItem.setUploaderName(extractor.getUploaderName());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setUploaderUrl(extractor.getUploaderUrl());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setUploaderVerified(extractor.isUploaderVerified());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setThumbnails(extractor.getThumbnails());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setStreamCount(extractor.getStreamCount());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setDescription(extractor.getDescription());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setPlaylistType(extractor.getPlaylistType());
} catch (final Exception e) {
addError(e);
}
return resultItem;
}
}

View File

@ -0,0 +1,72 @@
package org.schabi.newpipe.extractor.search;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.MetaInfo;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandler;
import javax.annotation.Nonnull;
import java.util.List;
public abstract class SearchExtractor extends ListExtractor<InfoItem> {
public static class NothingFoundException extends ExtractionException {
public NothingFoundException(final String message) {
super(message);
}
}
public SearchExtractor(final StreamingService service, final SearchQueryHandler linkHandler) {
super(service, linkHandler);
}
public String getSearchString() {
return getLinkHandler().getSearchString();
}
/**
* The search suggestion provided by the service.
* <p>
* This method also returns the corrected query if
* {@link SearchExtractor#isCorrectedSearch()} is true.
*
* @return a suggestion to another query, the corrected query, or an empty String.
*/
@Nonnull
public abstract String getSearchSuggestion() throws ParsingException;
@Nonnull
@Override
public SearchQueryHandler getLinkHandler() {
return (SearchQueryHandler) super.getLinkHandler();
}
@Nonnull
@Override
public String getName() {
return getLinkHandler().getSearchString();
}
/**
* Tell if the search was corrected by the service (if it's not exactly the search you typed).
* <p>
* Example: on YouTube, if you search for "pewdeipie",
* it will give you results for "pewdiepie", then isCorrectedSearch should return true.
*
* @return whether the results comes from a corrected query or not.
*/
public abstract boolean isCorrectedSearch() throws ParsingException;
/**
* Meta information about the search query.
* <p>
* Example: on YouTube, if you search for "Covid-19",
* there is a box with information from the WHO about Covid-19 and a link to the WHO's website.
* @return additional meta information about the search query
*/
@Nonnull
public abstract List<MetaInfo> getMetaInfo() throws ParsingException;
}

View File

@ -0,0 +1,113 @@
package org.schabi.newpipe.extractor.search;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.ListInfo;
import org.schabi.newpipe.extractor.MetaInfo;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandler;
import org.schabi.newpipe.extractor.utils.ExtractorHelper;
import java.io.IOException;
import java.util.List;
import javax.annotation.Nonnull;
public class SearchInfo extends ListInfo<InfoItem> {
private final String searchString;
private String searchSuggestion;
private boolean isCorrectedSearch;
private List<MetaInfo> metaInfo = List.of();
public SearchInfo(final int serviceId,
final SearchQueryHandler qIHandler,
final String searchString) {
super(serviceId, qIHandler, "Search");
this.searchString = searchString;
}
public static SearchInfo getInfo(final StreamingService service,
final SearchQueryHandler searchQuery)
throws ExtractionException, IOException {
final SearchExtractor extractor = service.getSearchExtractor(searchQuery);
extractor.fetchPage();
return getInfo(extractor);
}
public static SearchInfo getInfo(final SearchExtractor extractor)
throws ExtractionException, IOException {
final SearchInfo info = new SearchInfo(
extractor.getServiceId(),
extractor.getLinkHandler(),
extractor.getSearchString());
try {
info.setOriginalUrl(extractor.getOriginalUrl());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setSearchSuggestion(extractor.getSearchSuggestion());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setIsCorrectedSearch(extractor.isCorrectedSearch());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setMetaInfo(extractor.getMetaInfo());
} catch (final Exception e) {
info.addError(e);
}
final ListExtractor.InfoItemsPage<InfoItem> page
= ExtractorHelper.getItemsPageOrLogError(info, extractor);
info.setRelatedItems(page.getItems());
info.setNextPage(page.getNextPage());
return info;
}
public static ListExtractor.InfoItemsPage<InfoItem> getMoreItems(final StreamingService service,
final SearchQueryHandler query,
final Page page)
throws IOException, ExtractionException {
return service.getSearchExtractor(query).getPage(page);
}
// Getter
public String getSearchString() {
return this.searchString;
}
public String getSearchSuggestion() {
return this.searchSuggestion;
}
public boolean isCorrectedSearch() {
return this.isCorrectedSearch;
}
public void setIsCorrectedSearch(final boolean isCorrectedSearch) {
this.isCorrectedSearch = isCorrectedSearch;
}
public void setSearchSuggestion(final String searchSuggestion) {
this.searchSuggestion = searchSuggestion;
}
@Nonnull
public List<MetaInfo> getMetaInfo() {
return metaInfo;
}
public void setMetaInfo(@Nonnull final List<MetaInfo> metaInfo) {
this.metaInfo = metaInfo;
}
}

View File

@ -0,0 +1,174 @@
// Created by Fynn Godau 2019, licensed GNU GPL version 3 or later
package org.schabi.newpipe.extractor.services.bandcamp;
import static org.schabi.newpipe.extractor.StreamingService.ServiceInfo.MediaCapability.AUDIO;
import static org.schabi.newpipe.extractor.StreamingService.ServiceInfo.MediaCapability.COMMENTS;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_URL;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampFeaturedExtractor.FEATURED_API_URL;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampFeaturedExtractor.KIOSK_FEATURED;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampRadioExtractor.KIOSK_RADIO;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampRadioExtractor.RADIO_API_URL;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.ChannelExtractor;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import org.schabi.newpipe.extractor.comments.CommentsExtractor;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.kiosk.KioskList;
import org.schabi.newpipe.extractor.linkhandler.LinkHandler;
import org.schabi.newpipe.extractor.linkhandler.LinkHandlerFactory;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import org.schabi.newpipe.extractor.linkhandler.ReadyChannelTabListLinkHandler;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandler;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandlerFactory;
import org.schabi.newpipe.extractor.playlist.PlaylistExtractor;
import org.schabi.newpipe.extractor.search.SearchExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampChannelExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampChannelTabExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampCommentsExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampFeaturedExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampPlaylistExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampRadioExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampRadioStreamExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampSearchExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampStreamExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampSuggestionExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampChannelLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampChannelTabLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampCommentsLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampFeaturedLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampPlaylistLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampSearchQueryHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampStreamLinkHandlerFactory;
import org.schabi.newpipe.extractor.stream.StreamExtractor;
import org.schabi.newpipe.extractor.subscription.SubscriptionExtractor;
import org.schabi.newpipe.extractor.suggestion.SuggestionExtractor;
import java.util.Arrays;
public class BandcampService extends StreamingService {
public BandcampService(final int id) {
super(id, "Bandcamp", Arrays.asList(AUDIO, COMMENTS));
}
@Override
public String getBaseUrl() {
return BASE_URL;
}
@Override
public LinkHandlerFactory getStreamLHFactory() {
return BandcampStreamLinkHandlerFactory.getInstance();
}
@Override
public ListLinkHandlerFactory getChannelLHFactory() {
return BandcampChannelLinkHandlerFactory.getInstance();
}
@Override
public ListLinkHandlerFactory getChannelTabLHFactory() {
return BandcampChannelTabLinkHandlerFactory.getInstance();
}
@Override
public ListLinkHandlerFactory getPlaylistLHFactory() {
return BandcampPlaylistLinkHandlerFactory.getInstance();
}
@Override
public SearchQueryHandlerFactory getSearchQHFactory() {
return BandcampSearchQueryHandlerFactory.getInstance();
}
@Override
public ListLinkHandlerFactory getCommentsLHFactory() {
return BandcampCommentsLinkHandlerFactory.getInstance();
}
@Override
public SearchExtractor getSearchExtractor(final SearchQueryHandler queryHandler) {
return new BandcampSearchExtractor(this, queryHandler);
}
@Override
public SuggestionExtractor getSuggestionExtractor() {
return new BandcampSuggestionExtractor(this);
}
@Override
public SubscriptionExtractor getSubscriptionExtractor() {
return null;
}
@Override
public KioskList getKioskList() throws ExtractionException {
final KioskList kioskList = new KioskList(this);
final ListLinkHandlerFactory h = BandcampFeaturedLinkHandlerFactory.getInstance();
try {
kioskList.addKioskEntry(
(streamingService, url, kioskId) -> new BandcampFeaturedExtractor(
BandcampService.this,
h.fromUrl(FEATURED_API_URL),
kioskId
),
h,
KIOSK_FEATURED
);
kioskList.addKioskEntry(
(streamingService, url, kioskId) -> new BandcampRadioExtractor(
BandcampService.this,
h.fromUrl(RADIO_API_URL),
kioskId
),
h,
KIOSK_RADIO
);
kioskList.setDefaultKiosk(KIOSK_FEATURED);
} catch (final Exception e) {
throw new ExtractionException(e);
}
return kioskList;
}
@Override
public ChannelExtractor getChannelExtractor(final ListLinkHandler linkHandler) {
return new BandcampChannelExtractor(this, linkHandler);
}
@Override
public ChannelTabExtractor getChannelTabExtractor(final ListLinkHandler linkHandler) {
if (linkHandler instanceof ReadyChannelTabListLinkHandler) {
return ((ReadyChannelTabListLinkHandler) linkHandler).getChannelTabExtractor(this);
} else {
return new BandcampChannelTabExtractor(this, linkHandler);
}
}
@Override
public PlaylistExtractor getPlaylistExtractor(final ListLinkHandler linkHandler) {
return new BandcampPlaylistExtractor(this, linkHandler);
}
@Override
public StreamExtractor getStreamExtractor(final LinkHandler linkHandler) {
if (BandcampExtractorHelper.isRadioUrl(linkHandler.getUrl())) {
return new BandcampRadioStreamExtractor(this, linkHandler);
}
return new BandcampStreamExtractor(this, linkHandler);
}
@Override
public CommentsExtractor getCommentsExtractor(final ListLinkHandler linkHandler) {
return new BandcampCommentsExtractor(this, linkHandler);
}
}

View File

@ -0,0 +1,62 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.playlist.PlaylistInfoItemExtractor;
import java.util.List;
import javax.annotation.Nonnull;
public class BandcampAlbumInfoItemExtractor implements PlaylistInfoItemExtractor {
private final JsonObject albumInfoItem;
private final String uploaderUrl;
public BandcampAlbumInfoItemExtractor(final JsonObject albumInfoItem,
final String uploaderUrl) {
this.albumInfoItem = albumInfoItem;
this.uploaderUrl = uploaderUrl;
}
@Override
public String getName() throws ParsingException {
return albumInfoItem.getString("title");
}
@Override
public String getUrl() throws ParsingException {
return BandcampExtractorHelper.getStreamUrlFromIds(
albumInfoItem.getLong("band_id"),
albumInfoItem.getLong("item_id"),
albumInfoItem.getString("item_type"));
}
@Nonnull
@Override
public List<Image> getThumbnails() throws ParsingException {
return BandcampExtractorHelper.getImagesFromImageId(albumInfoItem.getLong("art_id"), true);
}
@Override
public String getUploaderName() throws ParsingException {
return albumInfoItem.getString("band_name");
}
@Override
public String getUploaderUrl() {
return uploaderUrl;
}
@Override
public boolean isUploaderVerified() {
return false;
}
@Override
public long getStreamCount() {
return ListExtractor.ITEM_COUNT_UNKNOWN;
}
}

View File

@ -0,0 +1,192 @@
// Created by Fynn Godau 2019, licensed GNU GPL version 3 or later
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import static org.schabi.newpipe.extractor.Image.HEIGHT_UNKNOWN;
import static org.schabi.newpipe.extractor.Image.WIDTH_UNKNOWN;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getArtistDetails;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
import static org.schabi.newpipe.extractor.utils.Utils.replaceHttpWithHttps;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import org.jsoup.Jsoup;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.Image.ResolutionLevel;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.ChannelExtractor;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.exceptions.ReCaptchaException;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabs;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.linkhandler.ReadyChannelTabListLinkHandler;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampChannelTabLinkHandlerFactory;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Objects;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import javax.annotation.Nonnull;
public class BandcampChannelExtractor extends ChannelExtractor {
private JsonObject channelInfo;
public BandcampChannelExtractor(final StreamingService service,
final ListLinkHandler linkHandler) {
super(service, linkHandler);
}
@Nonnull
@Override
public List<Image> getAvatars() {
return getImagesFromImageId(channelInfo.getLong("bio_image_id"), false);
}
@Nonnull
@Override
public List<Image> getBanners() throws ParsingException {
/*
* Mobile API does not return the header or not the correct header.
* Therefore, we need to query the website
*/
try {
final String html = getDownloader()
.get(replaceHttpWithHttps(channelInfo.getString("bandcamp_url")))
.responseBody();
return Stream.of(Jsoup.parse(html).getElementById("customHeader"))
.filter(Objects::nonNull)
.flatMap(element -> element.getElementsByTag("img").stream())
.map(element -> element.attr("src"))
.filter(url -> !url.isEmpty())
.map(url -> new Image(
replaceHttpWithHttps(url), HEIGHT_UNKNOWN, WIDTH_UNKNOWN,
ResolutionLevel.UNKNOWN))
.collect(Collectors.toUnmodifiableList());
} catch (final IOException | ReCaptchaException e) {
throw new ParsingException("Could not download artist web site", e);
}
}
/**
* Bandcamp discontinued their RSS feeds because it hadn't been used enough.
*/
@Override
public String getFeedUrl() {
return null;
}
@Override
public long getSubscriberCount() {
return -1;
}
@Override
public String getDescription() {
return channelInfo.getString("bio");
}
@Override
public String getParentChannelName() {
return null;
}
@Override
public String getParentChannelUrl() {
return null;
}
@Nonnull
@Override
public List<Image> getParentChannelAvatars() {
return List.of();
}
@Override
public boolean isVerified() throws ParsingException {
return false;
}
@Nonnull
@Override
public List<ListLinkHandler> getTabs() throws ParsingException {
final JsonArray discography = channelInfo.getArray("discography");
final TabExtractorBuilder builder = new TabExtractorBuilder(discography);
final List<ListLinkHandler> tabs = new ArrayList<>();
boolean foundTrackItem = false;
boolean foundAlbumItem = false;
for (final Object discographyItem : discography) {
if (foundTrackItem && foundAlbumItem) {
break;
}
if (!(discographyItem instanceof JsonObject)) {
continue;
}
final JsonObject discographyJsonItem = (JsonObject) discographyItem;
final String itemType = discographyJsonItem.getString("item_type");
if (!foundTrackItem && "track".equals(itemType)) {
foundTrackItem = true;
tabs.add(new ReadyChannelTabListLinkHandler(getUrl()
+ BandcampChannelTabLinkHandlerFactory.getUrlSuffix(ChannelTabs.TRACKS),
getId(),
ChannelTabs.TRACKS,
builder));
}
if (!foundAlbumItem && "album".equals(itemType)) {
foundAlbumItem = true;
tabs.add(new ReadyChannelTabListLinkHandler(getUrl()
+ BandcampChannelTabLinkHandlerFactory.getUrlSuffix(ChannelTabs.ALBUMS),
getId(),
ChannelTabs.ALBUMS,
builder));
}
}
return Collections.unmodifiableList(tabs);
}
@Override
public void onFetchPage(@Nonnull final Downloader downloader)
throws IOException, ExtractionException {
channelInfo = getArtistDetails(getId());
}
@Nonnull
@Override
public String getName() {
return channelInfo.getString("name");
}
private static final class TabExtractorBuilder
implements ReadyChannelTabListLinkHandler.ChannelTabExtractorBuilder {
private final JsonArray discography;
TabExtractorBuilder(final JsonArray discography) {
this.discography = discography;
}
@Nonnull
@Override
public ChannelTabExtractor build(@Nonnull final StreamingService service,
@Nonnull final ListLinkHandler linkHandler) {
return BandcampChannelTabExtractor.fromDiscography(service, linkHandler, discography);
}
}
}

Some files were not shown because too many files have changed in this diff Show More