TobiGr
fafd471606
[PeerTube] Fix test for like count
...
Number changed
2024-05-08 19:25:12 +02:00
TobiGr
4f477ad72b
[PeerTube] Fix testing comment content
...
The comment is not available anymore.
2024-05-08 19:25:12 +02:00
Tobi
964e429978
Merge pull request #1166 from TeamNewPipe/dependabot/github_actions/peaceiris/actions-gh-pages-4
...
Bump peaceiris/actions-gh-pages from 3 to 4
2024-05-08 19:24:40 +02:00
dependabot[bot]
10c6965a28
Bump peaceiris/actions-gh-pages from 3 to 4
...
Bumps [peaceiris/actions-gh-pages](https://github.com/peaceiris/actions-gh-pages ) from 3 to 4.
- [Release notes](https://github.com/peaceiris/actions-gh-pages/releases )
- [Changelog](https://github.com/peaceiris/actions-gh-pages/blob/main/CHANGELOG.md )
- [Commits](https://github.com/peaceiris/actions-gh-pages/compare/v3...v4 )
---
updated-dependencies:
- dependency-name: peaceiris/actions-gh-pages
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-05-08 17:16:08 +00:00
TobiGr
e54f38f5e7
[PeerTube] Fix test
...
UploaderName was changed by user
2024-05-08 19:13:49 +02:00
Tobi
5dd5c7a65b
Merge pull request #1173 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.8.5
...
Bump com.github.spotbugs:spotbugs-annotations from 4.8.3 to 4.8.5
2024-05-08 19:12:40 +02:00
Tobi
37438ff82f
Merge pull request #1174 from TeamNewPipe/dependabot/gradle/org.mozilla-rhino-1.7.15
...
Bump org.mozilla:rhino from 1.7.13 to 1.7.15
2024-05-08 19:02:57 +02:00
TobiGr
bba3b6c69b
version 0.24.0
2024-05-08 13:01:06 +02:00
TobiGr
b40a5784ed
Fix JDoc
...
YoutubePostLiveStreamDvrDashManifestCreator.java:90: error: unexpected end tag: </p>
2024-05-08 12:41:36 +02:00
dependabot[bot]
c6da4004e2
Bump org.mozilla:rhino from 1.7.13 to 1.7.15
...
Bumps [org.mozilla:rhino](https://github.com/mozilla/rhino ) from 1.7.13 to 1.7.15.
- [Release notes](https://github.com/mozilla/rhino/releases )
- [Changelog](https://github.com/mozilla/rhino/blob/master/RELEASE-NOTES.md )
- [Commits](https://github.com/mozilla/rhino/commits )
---
updated-dependencies:
- dependency-name: org.mozilla:rhino
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-05-06 09:43:05 +00:00
dependabot[bot]
f26e84d39f
Bump com.github.spotbugs:spotbugs-annotations from 4.8.3 to 4.8.5
...
Bumps [com.github.spotbugs:spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.8.3 to 4.8.5.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.8.3...4.8.5 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-05-06 09:43:03 +00:00
Tobi
ec3e8378c6
Merge pull request #1171 from TeamNewPipe/fix/jdoc
...
Fix JavaDocs and add Checkstyle exception for line length in JavaDocs' links
2024-04-23 21:22:12 +02:00
TobiGr
8d2a7a5281
[PeerTube] Fix test
2024-04-23 20:02:29 +02:00
TobiGr
7c29dbc965
Fix JDoc
...
Add Checkstyle exception for LineLength in JDoc links.
See https://github.com/TeamNewPipe/NewPipeExtractor/actions/runs/8804403691/job/24164703883
2024-04-23 19:55:51 +02:00
Stypox
fbe9e6223a
Merge pull request #1168 from AudricV/yt_upd-cver-rm-keys-and-do-fixes
...
[YouTube] Update clients versions, restore access to some streams and more
2024-04-20 11:54:24 +02:00
Stypox
4e9e7cb29c
Improve assertTabsContain() to also check size
2024-04-20 11:48:36 +02:00
Stypox
9d0dd36034
[YouTube] Create constants for client names/versions
2024-04-20 11:43:54 +02:00
Stypox
d4e6d22e64
[YouTube] Improve meta info code for review
2024-04-20 11:43:08 +02:00
AudricV
74bf000473
[YouTube] Update mocks
2024-04-11 18:59:11 +02:00
AudricV
f9792cf3a9
[YouTube] Fix InteractiveTabbedHeader.testTabs test
...
YouTube now returns a Shorts tab for InteractiveTabbedHeader gaming channels,
which contains Shorts about the game of the topic channel but are not uploaded
on the game topic channel.
As this tab is already supported by the extractor, fetching a gaming topic
channel now returns a tab instead of none.
Channel name and channel URL of these Shorts needs to be set to null in a
separate commit, as Shorts on this tab do not have the topic channel as their
uploader.
2024-04-11 18:59:10 +02:00
AudricV
f40fc0aa4f
[YouTube] Add support for new crisis meta info action data
...
The new action data can return multiple contact actions instead of only one,
which will be concatenated by a new line return.
This commit fixes tests of the CrisisResources test class of
YoutubeSearchExtractorTest.
2024-04-11 18:59:09 +02:00
AudricV
2a3c6f80d2
[YouTube] Fix YoutubeStreamExtractorRelatedMixTest
2024-04-10 21:19:03 +02:00
AudricV
657b4377aa
[YouTube] Fix YoutubeStreamExtractorDefaultTest tests
2024-04-10 21:19:03 +02:00
AudricV
7bf50bf1cb
[YouTube] Update Android client player parameters
...
YouTube disabled the effectiveness of the parameters which were used (the
player response we get redirects to another video), but new parameters which
work around Android's client integrity checks have been found.
2024-04-10 21:19:03 +02:00
AudricV
27dc1b1f50
[YouTube] Remove usage of API keys for InnerTube requests, bump versions
...
The API keys are not used anymore by official clients in almost all cases
(still used by the Android app until it gets a configuration) for all requests
we made.
Clients and device OS versions have been bumped to their latest stable version
known.
Methods and fields related to API keys have been renamed or deleted if they're
no longer relevant.
2024-04-10 21:19:02 +02:00
AudricV
e380bb4bc3
[YouTube] Add missing prettyPrint query parameter to mixes continuations
2024-04-10 19:06:36 +02:00
Audric V
6c3c2e25d7
Merge pull request #1163 from AudricV/yt-fix_comments_extraction
...
[YouTube] Support new comments data
2024-04-10 18:19:59 +02:00
Stypox
02274d5395
[YouTube] Avoid XSS attacks in description or comments
2024-04-08 11:21:31 +02:00
Stypox
3f7b2653e3
[YouTube] Add YoutubeDescriptionHelperTest
2024-04-08 11:21:31 +02:00
Stypox
a90237816a
[YouTube] Cleanup description helper
...
Remove unneeded isClose field, and make constants private
2024-04-08 11:21:31 +02:00
Stypox
b80c3f5d51
[YouTube] Replace link text with accessibility label
2024-04-08 00:14:28 +02:00
Stypox
09732d6785
[YouTube] Add support for styles in attributed descriptions
...
Also refactor descriptions parsing.
2024-04-04 21:14:27 +02:00
AudricV
293c3e9e47
[YouTube] Support new A/B tested comments data
...
Also improve current comments code by removing outdated comment
renderer data.
2024-04-04 21:14:26 +02:00
Stypox
e5b30ae8c3
Merge pull request #1151 from Profpatsch/localization-return-optional
...
LocaleCompat.forLanguageTag: return Optional if parsing fails
2024-03-29 13:50:48 +01:00
Stypox
23fc7aa209
Throw ParsingException instead of IllegalArg
2024-03-29 13:44:42 +01:00
Stypox
fb468a23f4
Merge pull request #1142 from TeamNewPipe/peertube-v6
...
[PeerTube] Add support for PeerTube v6 features
2024-03-29 12:25:38 +01:00
Stypox
6589e2c15d
Merge pull request #1148 from Stypox/mediaccc-channel-tab-handler
...
[MediaCCC] Allow obtaining channel tab link handler
2024-03-28 13:45:05 +01:00
Tobi
ad71864b23
Merge pull request #1162 from Stypox/fix-comment-description-npe
...
Make getCommentText @Nonnull
2024-03-27 20:04:17 +01:00
Stypox
c57016b79b
Make getCommentText @Nonnull
2024-03-27 15:26:06 +01:00
Tobi
adcc1f17ee
Merge pull request #1160 from TeamNewPipe/fix/tests
...
Fix some failing unit tests and detect new account termination messages for YouTube
2024-03-20 15:17:55 +01:00
TobiGr
51ddacc81d
[SoundCloud] Fix SoundcloudSearchExtractorTest.NoNextPage
...
Search did not return no item at all, causing a NothingFoundException. New search query yields three items on first page
2024-03-20 15:10:39 +01:00
TobiGr
8392d50ba6
Update mocks for YoutubeChannelExtractorTest.NotAvailable
2024-03-20 14:59:44 +01:00
TobiGr
aaccfecda8
[YouTube] Detect new account termination messages
2024-03-20 14:57:41 +01:00
TobiGr
73f0c63a9d
[PeerTube] Fix tests for "What is PeerTube?" video
2024-03-20 14:44:06 +01:00
Tobi
896a55e319
Merge pull request #1139 from TeamNewPipe/dependabot/github_actions/actions/upload-artifact-4
...
Bump actions/upload-artifact from 3 to 4
2024-03-18 08:59:35 +01:00
dependabot[bot]
e58fc652e0
Bump actions/upload-artifact from 3 to 4
...
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact ) from 3 to 4.
- [Release notes](https://github.com/actions/upload-artifact/releases )
- [Commits](https://github.com/actions/upload-artifact/compare/v3...v4 )
---
updated-dependencies:
- dependency-name: actions/upload-artifact
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-03-18 07:53:24 +00:00
Tobi
e3f2c9aec7
Merge pull request #1153 from TeamNewPipe/dependabot/github_actions/actions/cache-4
...
Bump actions/cache from 3 to 4
2024-03-18 08:51:24 +01:00
Tobi
6b0fc14c04
Merge pull request #1156 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.10.2
...
Bump org.junit:junit-bom from 5.10.0 to 5.10.2
2024-03-18 08:46:13 +01:00
dependabot[bot]
d579b608e5
Bump org.junit:junit-bom from 5.10.0 to 5.10.2
...
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5 ) from 5.10.0 to 5.10.2.
- [Release notes](https://github.com/junit-team/junit5/releases )
- [Commits](https://github.com/junit-team/junit5/compare/r5.10.0...r5.10.2 )
---
updated-dependencies:
- dependency-name: org.junit:junit-bom
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-02-05 09:15:26 +00:00
TobiGr
fe47a4311f
[PeerTube] Add test for segments and framesets
2024-01-29 10:22:06 +01:00
TobiGr
15e0e74b48
[PeerTube] Add support for stream frames/storyboards extraction
...
Implement PeerTubeStreamExtractor.getFrames()
2024-01-29 10:22:06 +01:00
dependabot[bot]
da04eded5d
Bump actions/cache from 3 to 4
...
Bumps [actions/cache](https://github.com/actions/cache ) from 3 to 4.
- [Release notes](https://github.com/actions/cache/releases )
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md )
- [Commits](https://github.com/actions/cache/compare/v3...v4 )
---
updated-dependencies:
- dependency-name: actions/cache
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-01-18 09:32:42 +00:00
Profpatsch
7408173246
LocaleCompat.forLanguageTag: return Optional if parsing fails
...
It’s not obvious that the function will fail in some cases and throw
an `IllegalArgumentException`.
So instead of just failing if parsing fails, return an Optional that
all callers have to decide what to do (e.g. the YoutubeExtractor can
just ignore the locale in that case, like it does with most other
fields in the json if they are unexpected).
2024-01-07 14:31:34 +01:00
Stypox
aaf3231fc7
[MediaCCC] Fix lambda link handler keeping reference to extractor
...
This caused problems in NewPipe, because extractors are not serializable, and well, keeping references to them is a bad idea anyway.
2023-12-30 23:23:19 +01:00
Stypox
137e924035
[MediaCCC] Add ChannelTabExtractorTest
2023-12-30 22:53:51 +01:00
Stypox
cc9ade962e
[MediaCCC] Allow obtaining channel tab extractor from scratch
...
i.e. without needing to pass through the conference/channel extractor
This was needed because clients (like NewPipe) might rely on link handlers to hold as little data as possible, since they might be kept around for long or passed around in system transactions, so this commit allows obtaining a standalone link handler that does not hold a JsonObject within itself.
2023-12-30 22:53:27 +01:00
Stypox
3402cdb666
Merge pull request #1147 from petlyh/youtube-albums
...
[YouTube] Add Albums channel tab
2023-12-30 21:58:19 +01:00
petlyh
6dc25f7b97
[YouTube] Add Albums channel tab mocks
2023-12-30 14:46:39 +01:00
petlyh
4408e2d0ac
[YouTube] Add Albums channel tab
2023-12-30 14:01:30 +01:00
TobiGr
9ab932e394
Rename testDoNotAcceptNonURLs() -> assertDoNotAcceptNonURLs()
2023-12-29 16:38:11 +01:00
Stypox
9d66debf3c
Merge pull request #1132 from TeamNewPipe/dependabot/github_actions/actions/setup-java-4
...
Bump actions/setup-java from 3 to 4
2023-12-29 16:20:46 +01:00
Tobi
038ebdedc4
Merge pull request #1143 from petlyh/peertube-non-urls
...
Avoid PeerTube accepting non-URLs
2023-12-29 12:47:30 +01:00
TobiGr
61d237de02
[PeerTube] Test onAccept(String URL) in LinkHandlerFactories for non-URLs
2023-12-29 12:45:02 +01:00
petlyh
2b2c1546d1
Avoid PeerTube accepting non-URLs
2023-12-29 12:27:39 +01:00
Tobi
1e93b1dc20
Merge pull request #1135 from Stypox/yt-emergency-info
...
[YouTube] Implement emergency meta info
2023-12-29 12:01:40 +01:00
dependabot[bot]
3400af99b3
Bump actions/setup-java from 3 to 4
...
Bumps [actions/setup-java](https://github.com/actions/setup-java ) from 3 to 4.
- [Release notes](https://github.com/actions/setup-java/releases )
- [Commits](https://github.com/actions/setup-java/compare/v3...v4 )
---
updated-dependencies:
- dependency-name: actions/setup-java
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-12-29 10:57:47 +00:00
Tobi
1f8a044462
Merge pull request #1138 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.8.3
...
Bump com.github.spotbugs:spotbugs-annotations from 4.8.0 to 4.8.3
2023-12-29 11:56:18 +01:00
dependabot[bot]
1470aa7303
Bump com.github.spotbugs:spotbugs-annotations from 4.8.0 to 4.8.3
...
Bumps [com.github.spotbugs:spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.8.0 to 4.8.3.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.8.0...4.8.3 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-12-29 10:53:28 +00:00
TobiGr
8f9ebdcb77
[PeerTube] Fix failing PeertubeTrendingLinkHandlerFactoryTest
...
The factory was updated in #1144
2023-12-29 11:52:19 +01:00
Stypox
1553931027
Merge pull request #1145 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.17.2
...
Bump org.jsoup:jsoup from 1.16.2 to 1.17.2
2023-12-29 11:27:01 +01:00
Stypox
b2ec1b15fb
Merge pull request #1144 from dragfyre/patch-1
...
Update PeertubeTrendingLinkHandlerFactory.java
2023-12-29 11:21:08 +01:00
dependabot[bot]
151ee99da3
Bump org.jsoup:jsoup from 1.16.2 to 1.17.2
...
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup ) from 1.16.2 to 1.17.2.
- [Release notes](https://github.com/jhy/jsoup/releases )
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES.md )
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.16.2...jsoup-1.17.2 )
---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-12-29 09:47:00 +00:00
dragfyre
65e7bc5b95
Update PeertubeTrendingLinkHandlerFactory.java
...
correcting Peertube local trending api URL (per #10685 in main NewPipe repo); see https://docs.joinpeertube.org/api-rest-reference.html#tag/Video/operation/getVideos
2023-12-28 14:50:31 +07:00
Stypox
f276caf54a
Release v0.23.1
2023-12-21 21:59:21 +01:00
Stypox
fc54fb2fdb
Merge pull request #1140 from Stypox/yt-shorts-no-duration
...
[YouTube] Always return -1 as duration of Shorts returned inside reel items
2023-12-21 21:52:40 +01:00
Stypox
0518487d26
Fix SearchInfo's non-null MetaInfo being null when initialized or when an extraction error occurs ( #1141 )
...
The meta info might have been null either when SearchInfo is first initialized, or when extractor.getMetaInfo() throws an exception in getInfo().
This caused NewPipe to crash instead of showing a nice error in https://www.reddit.com/r/youtube/comments/184ttmw/what_exactly_about_blue_whales_has_youtube_so/ .
2023-12-21 21:39:25 +01:00
Stypox
5b59a1a8c5
[YouTube] Move meta info extraction to separate file
...
YoutubeParsingHelper was longer than 2000 lines which caused checkstyle issues
2023-12-21 21:19:08 +01:00
Stypox
b8e12dd76c
[YouTube] Implement emergency meta info
...
YouTube provides that meta info panel when users search for really sensitive content like suicide (e.g. "blue whale").
It contains:
- an encouragement as title (e.g. "We are with you")
- a phone number as action
- details about how to call the phone number (e.g. availability)
- an url pointing to the website of an association
Also add a test that just checks if a meta info is properly extracted
2023-12-21 21:19:08 +01:00
Stypox
83c1737f70
[YouTube] Update shorts channel tab mocks
2023-12-21 20:54:16 +01:00
Stypox
2938067c2c
[YouTube] Shorts don't provide a duration anymore
2023-12-21 20:41:01 +01:00
Stypox
91419ec6e8
Release v0.23.0
2023-12-10 15:59:32 +01:00
Audric V
678c98f24c
Merge pull request #1127 from AudricV/yt_improvements-and-fixes
...
[YouTube] Make some improvements and fixes
2023-12-09 14:07:31 +01:00
AudricV
ec0194cfbf
[YouTube] Update mocks
2023-12-08 21:46:52 +01:00
AudricV
00a0f1a103
[YouTube] Add a blocking crisis resources bypass in search results test
...
This test only tests that search results are returned, when no content filters
are provided and crisis resources blocking search results should be returned.
Searches with blocking crisis resources and content filters should work too, as
the bypass has been implemented for them.
2023-12-08 21:46:49 +01:00
AudricV
06838d7245
[YouTube] Fix YoutubeSearchQHTest tests
...
As search parameters to bypass crisis resources blocking search results have
been implemented, they need to be added to search tests, in order to pass
them.
2023-12-08 21:46:49 +01:00
AudricV
642bb01388
[YouTube] Fix testSubscriberCount method of YoutubeChannelExtractorTest.CarouselHeader
...
The subscriber count is now lower than the expected count as some people
unsubscribed to the Sports system channel. The expected count has been so
lowered.
2023-12-08 21:46:49 +01:00
AudricV
bedc9e5bc0
[YouTube] Remove Channels channel tab in tests
2023-12-08 21:46:48 +01:00
AudricV
5fa22ae25b
[YouTube] Enable and change testRelatedItems method of YoutubePlaylistExtractorTest.LearningPlaylist
...
This change makes the method test whether there is no more items instead of
being disabled.
2023-12-08 21:46:48 +01:00
AudricV
29dc7625f2
[YouTube] Fix YoutubeSearchExtractorTest.Suggestion
...
YouTube doesn't return anymore a suggestion for the query "algorythm", but does
for the query "on board ing" ("on boarding"). This search query is now used and
had to be URL-encoded.
URL encoding in the complete YoutubeSearchExtractorTest test class uses now
extractor's Util class instead of Java's URLDecoder class directly.
2023-12-08 21:46:48 +01:00
AudricV
56ab35423e
[YouTube] Fix potential NullPointerException in YoutubeSearchExtractor.getSearchSuggestion
2023-12-08 21:46:48 +01:00
AudricV
9dc1eab28c
[YouTube] Fix expected upload dates of StreamExtractor tests
...
YouTube is rolling out or A/B testing a new date format returned inside player
responses, which are precise to the second instead of the day.
This commit makes the StreamExtractor tests use these more precise dates.
2023-12-08 21:46:48 +01:00
AudricV
ad3d187ac7
[YouTube] Fix testRelatedItems method of YoutubeStreamExtractorRelatedMixTest
...
This commit fixes the testRelatedItems test method by:
- accepting consent in the test class, in order to extract mixes in
recommendations;
- removing assertion of a music mix inside the recommendations, as YouTube
doesn't seem to return such mixes anymore, at least for the video used in the
test class.
2023-12-08 21:46:47 +01:00
AudricV
e111814401
[YouTube] Fix YoutubeStreamExtractorDefaultTest.RatingsDisabledTest
...
Replace the video used in this test class with another one publicly available
and update the corresponding expected test values.
The test class's mocks will be updated in a different commit.
2023-12-08 21:46:47 +01:00
AudricV
fc45941ead
[YouTube] Fix YoutubeChannelExtractorTest tests
...
- Change CarouselHeader test channel to Sports system one, as the Coachella one
doesn't return this channel header anymore;
- Fix InteractiveTabbedHeader test by checking whether the test's channel
description is not empty instead of containing some words, as it is changing
frequently.
2023-12-08 21:46:47 +01:00
AudricV
0bcb241c38
[YouTube] Fix expected data in YoutubeStreamExtractorRelatedMixTest
...
Video's title and tags have been changed by its uploader, so they have to be
updated.
Also make some package-private constants private, as they are not used outside
of the class, and remove unneeded test overrides.
2023-12-08 21:46:47 +01:00
AudricV
6ba8251be1
[YouTube] Bypass crisis resources blocking search results
...
These crisis resources are preventing search results to be returned. See
https://support.google.com/youtube/answer/10726080?hl=en for more info on them.
This commit changes search parameters to include the property allowing to show
search results.
2023-12-08 21:46:47 +01:00
AudricV
7dea2d0d27
[YouTube] Remove Channels channel tab support
...
This tab has been removed by YouTube.
2023-12-08 21:46:47 +01:00
AudricV
3782d9a02a
[YouTube] Support new A/B tested like data and avoid like count conversion from integer to long
...
Also make minor improvements to current like data extraction and remove
previous like count data support, as it is not returned anymore.
2023-12-08 21:46:46 +01:00
AudricV
b71ce1123f
[YouTube] Extract only search results corresponding to a search type
...
YouTube returns sometimes videos inside channel search results. As we only want
results corresponding to the type we requested, this commits makes
YoutubeSearchExtractor ignoring non-requested search results we get, using the
extractor LinkHandler's first content filter value.
Also remove an unneeded exception throwing declaration in
YoutubeSearchExtractor.
2023-12-08 21:46:46 +01:00
AudricV
ff8ed7247f
[YouTube] Switch to new consent cookie
...
Also move the documentation of the consent in its setter method in order to be
accessible publicly and improve it.
2023-12-08 21:46:46 +01:00
AudricV
ec838d7421
[YouTube] Add missing prettyPrint query parameter to some test InnerTube requests
...
This query parameter for which its value is set to false was not added to two
requests made in test classes of YoutubeMixPlaylistExtractorTest.
Also remove an unneeded ParsingException exception throwing declaration in a
test method.
2023-12-08 21:46:46 +01:00
AudricV
2c941794c0
[YouTube] Add utcOffsetMinutes to all InnerTube payloads
...
This should make returned dates consistent between timezones and countries on
which the extractor is ran.
It was previously only set on YouTube Music search continuations.
2023-12-08 21:46:46 +01:00
AudricV
d97c9e0db1
[YouTube] Improve payloads and URLs of InnerTube requests
...
For every InnerTube request:
- Always add a `request` object with the following properties:
- "internalExperimentFlags" set to an empty array;
- "useSsl" set to "true";
- "lockedSafetyMode" set to "false".
- Use proper TODO comment to provide a way to enable restricted mode on every
request and add it on requests on which it wasn't present.
For YouTube Music:
- Remove alt query parameter, as it is not used anymore by the website;
- Add prettyPrint query parameter with false value on YouTube Music search
continuations.
2023-12-08 21:46:45 +01:00
AudricV
8a9ebcc373
[YouTube] Update InnerTube clients' version and devices' OS version and model
2023-12-08 21:46:45 +01:00
Tobi
eac850ca10
Merge pull request #1114 from FineFindus/feat/comment-author-is-owner
...
[YouTube] Add channel owner to comments
2023-10-25 09:51:12 +02:00
Tobi
5ab1f784e8
Merge pull request #1117 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.16.2
...
Bump org.jsoup:jsoup from 1.16.1 to 1.16.2
2023-10-21 19:26:36 +02:00
dependabot[bot]
9d7bcba050
Bump org.jsoup:jsoup from 1.16.1 to 1.16.2
...
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup ) from 1.16.1 to 1.16.2.
- [Release notes](https://github.com/jhy/jsoup/releases )
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES )
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.16.1...jsoup-1.16.2 )
---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-10-20 09:13:21 +00:00
dependabot[bot]
e26065148a
Bump com.github.spotbugs:spotbugs-annotations from 4.7.3 to 4.8.0
...
Bumps [com.github.spotbugs:spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.7.3 to 4.8.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.3...4.8.0 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-10-12 14:19:10 +02:00
FineFindus
34b05a0dda
feat(youtube/comments): support creator replies
2023-10-09 16:33:43 +02:00
TobiGr
0821f09114
Add missing mocks
2023-10-09 16:33:43 +02:00
FineFindus
c1784a4bdb
[YouTube] Add channel owner to comments
2023-10-09 16:33:43 +02:00
TobiGr
f9846352ea
Fix wrong `@Nullable` annotation
2023-10-09 16:02:57 +02:00
Tobi
d6f5cba6e2
Merge pull request #1111 from FineFindus/feat/creator-reply
...
Add `hasCreatorReply()` to CommentsInfoItem
2023-10-09 12:45:56 +02:00
TobiGr
9d63c75623
Add missing mocks
2023-10-09 11:24:39 +02:00
TobiGr
d49f8411d7
[PeerTube] Implement CommentsInfoItemExtractor.hasCreatorReply()
2023-10-09 02:47:12 +02:00
Stypox
bb132167d5
Merge pull request #1113 from AudricV/snd_fix-non-jpg-images
...
[SoundCloud] Fix extraction of non-JPG images
2023-10-02 19:40:57 +02:00
AudricV
c98695fcea
[SoundCloud] Fix extraction of non-JPG images
...
Default image qualities were removed in image URLs with the jpg extension,
causing the addition of the image suffix to full non-JPG images URLs and so to
invalid image URLs.
Only the image quality name with its leading "-" character and the "."
character after the name is now removed and replaced by a string format
replaced itself with the image quality name for each quality.
As the image suffixes do not contain the image extension, the name of image
qualities lists has been adapted with these changes and some related comments
have been also improved.
2023-10-01 20:33:25 +02:00
AudricV
ac00459c1a
Change requirement of image extensions in ImageSuffix class' Javadoc to a possibility
...
Some services may provide different image formats using the same suffix,
without we know what format the service provide. Enforcing an image extension
could so lead to provide invalid image URLs, like for SoundCloud PNG images
currently.
With this documentation change, it is now clear that users of this class decide
of whether they want to include image extensions in the suffix. The previous
behavior described in the Javadoc was not enforced.
2023-09-30 21:11:09 +02:00
FineFindus
dd7b2d9798
feat(youtube/comments): support creator replies
2023-09-25 10:40:45 +02:00
Youssif Shaaban Alsager
917554acc4
[YouTube] Add support for ultralow audio formats ( #1063 )
2023-09-24 19:04:34 +02:00
Tobi
8b0068f8f4
Merge pull request #1110 from christian-2hu/chore/update-copyright
...
chore: Update copyright notices
2023-09-23 00:29:19 +02:00
Christian
fc67d49f59
Update copyright notices
...
Update copyright notices to comply to GPLv3 and change NewPipe to NewPipe Extractor on some notices that were not updated.
2023-09-22 19:10:15 -03:00
Stypox
289db1178a
Merge pull request #1108 from AudricV/yt_refactor-js-usage
...
[YouTube] Refactor JavaScript usage and fix extraction of obfuscated signature deobfuscation function
2023-09-22 10:41:57 +02:00
AudricV
6ed22099a2
[YouTube] Update stream mocks
2023-09-21 21:59:34 +02:00
AudricV
714b141ecb
[YouTube] Catch any exception when extracting something from JavaScript's base player
2023-09-21 21:59:33 +02:00
AudricV
588c6a8422
[YouTube] Quote signature deobfuscation function name and add semicolon only where needed
2023-09-21 21:59:33 +02:00
AudricV
1fa85ec6ca
[YouTube] Add tests for signature timestamp extraction and signature deobfuscation function extraction and execution
2023-09-21 21:59:33 +02:00
AudricV
a04bc320de
[YouTube] Convert signature timestamp to integer
...
The signature timestamp is used as a number by HTML5 clients, so it should be
used in the same way by the extractor too instead of being a string.
As the timestamp doesn't seem to exceed 5 digits, an integer is used to store
its value.
2023-09-21 21:59:32 +02:00
AudricV
7de3753a81
[YouTube] Refactor JavaScript player management API
...
This commit is introducing breaking changes.
For clients, everything is managed in a new class called
YoutubeJavaScriptPlayerManager:
- caching JavaScript base player code and its extracted code (functions and
variables);
- getting player signature timestamp;
- getting deobfuscated signatures of streaming URLs;
- getting streaming URLs with a throttling parameter deobfuscated, if
applicable.
The class delegates the extraction parts to external package-private classes:
- YoutubeJavaScriptExtractor, to extract and download YouTube's JavaScript base
player code: it always already present before and has been edited to mainly
remove the previous caching system and made it package-private;
- YoutubeSignatureUtils, for player signature timestamp and signature
deobfuscation function of streaming URLs, added in a recent commit;
- YoutubeThrottlingParameterUtils, which was originally
YoutubeThrottlingDecrypter, for throttling parameter of streaming URLs
deobfuscation function and checking whether this parameter is in a streaming
URL.
YoutubeJavaScriptPlayerManager caches and then runs the extracted code if it
has been executed successfully. The cache system of throttling parameters
deobfuscated values has been kept, its size can be get using the
getThrottlingParametersCacheSize method and can be cleared independently using
the clearThrottlingParametersCache method.
If an exception occurs during the extraction or the parsing of a function
property which is not related to JavaScript base player code fetching, it is
stored until caches are cleared, making subsequent failing extraction calls of
the requested function or property faster and consuming less resources, as the
result should be the same until the base player code changes.
All caches can be reset using the clearAllCaches method of
YoutubeJavaScriptPlayerManager.
Classes using JavaScript base player code and utilities directly (in the code
and its tests) have been also updated in this commit.
2023-09-21 21:59:32 +02:00
AudricV
6884d191cd
[YouTube] Add utility class around signatures and fix signature deobfuscation function extraction
...
The goal of this class is to decouple the extraction of signature timestamp and
signature deobfuscation function from YoutubeStreamExtractor.
The extraction of the signature deobfuscation function has been also adapted to
support the latest YouTube player versions.
This new class, YoutubeSignatureUtils, doens't store anything temporary such as
a copy of the player code, which has to be passed where required. It is not
public, as it will be used by a JavaScript player manager class in the future,
in order to handle in a better way fetching, caching and resetting cache of the
player code.
2023-09-21 21:59:26 +02:00
Tobi
3be76a6406
Merge pull request #1107 from Isira-Seneviratne/Locale_forLanguageTag
...
Use Locale.forLanguageTag() in tests
2023-09-18 16:48:30 +02:00
TobiGr
17790328cd
Improve doc
2023-09-18 16:44:51 +02:00
Isira Seneviratne
4bc8ae7812
Use Locale.forLanguageTag() in tests
2023-09-18 08:59:13 +05:30
Tobi
90aed06a63
Merge pull request #1105 from TeamNewPipe/fix/bandcamp-streame-extractor-test
...
[Badcamp] Fix StreamExtractorTest
2023-09-18 01:49:04 +02:00
TobiGr
cf49f4a31c
[Badcamp] Fix StreamExtractorTest
...
The song was renamed and the URL changed
2023-09-17 23:58:07 +02:00
Tobi
7c7ceaceab
Merge pull request #1103 from TeamNewPipe/dependabot/github_actions/actions/checkout-4
...
Bump actions/checkout from 3 to 4
2023-09-17 22:42:01 +02:00
dependabot[bot]
72c475d944
Bump actions/checkout from 3 to 4
...
Bumps [actions/checkout](https://github.com/actions/checkout ) from 3 to 4.
- [Release notes](https://github.com/actions/checkout/releases )
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md )
- [Commits](https://github.com/actions/checkout/compare/v3...v4 )
---
updated-dependencies:
- dependency-name: actions/checkout
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-09-05 10:00:13 +00:00
Stypox
1f08d28ae5
Merge pull request #889 from AudricV/multiple-images-support
...
Multiple images support
2023-08-13 11:35:11 +02:00
AudricV
e8bfd20170
[MediaCCC] Apply changes in extractor tests
...
Also remove some public test methods modifiers.
2023-08-12 22:56:33 +02:00
AudricV
0292c4f3e8
[Bandcamp] Apply changes in extractor tests
...
Also remove some public test methods modifiers, add missing Test annotations on
old Junit 4 tests (and update them if needed), and use final in some places
where it was possible.
BandcampChannelExtractorTest.testLength has been removed as the test is always
true.
2023-08-12 22:56:32 +02:00
AudricV
2578f22054
[Bandcamp] Add utility test method to test images
...
This method, testImages(Collection<Image>), will use first the default image
collection test in DefaultTests and then will check that each image URL
contains f4.bcbits.com/img and ends with .jpg or .png.
To do so, a new non-instantiable final class has been added: BandcampTestUtils.
2023-08-12 22:56:32 +02:00
AudricV
ba5315c72d
[PeerTube] Apply changes in extractor tests
...
Also remove some public test methods modifiers, add missing Test annotations on
old Junit 4 tests (and update them if needed), and improve some code.
2023-08-12 22:56:32 +02:00
AudricV
1d72bac53d
[SoundCloud] Apply changes in extractor tests
2023-08-12 22:56:32 +02:00
AudricV
93a210394d
[YouTube] Apply changes in extractor tests
...
Also remove some public test methods modifiers, add missing Test annotations on
old Junit 4 tests (and update them if needed), and use final in some places
where it was possible.
2023-08-12 22:56:31 +02:00
AudricV
2c436d428c
[YouTube] Add utility test method to test images in YoutubeTestsUtils
...
This method, testImages(Collection<Image>), will use first the default image
collection test in DefaultTests and then will check that each image URL
contains the string yt.
The JavaDoc of the class has been also updated to reflect the changes made in
it (it is now more general).
2023-08-12 22:56:31 +02:00
AudricV
d381f3b70b
Update avatar, banners and thumbnail methods' name and apply changes in DefaultStreamExtractorTest
2023-08-12 22:56:31 +02:00
AudricV
434e885708
Add utility methods in ExtractorAsserts to check whether a collection is empty and to test image collections
...
Two new methods have been added in ExtractorAsserts to check if a collection is
empty:
- assertNotEmpty(String, Collection<?>), checking:
- the non nullity of the collection;
- its non emptiness (if that's not case, an exception will be thrown using
the provided message).
- assertNotEmpty(Collection<?>), calling assertNotEmpty(String, Collection<?>)
with null as the value of the string argument.
A new one has been added to this assertion class to check the contrary:
assertEmpty(Collection<?>), checking emptiness of the collection only if it is
not null.
Three new methods have been added in ExtractorAsserts as utility test methods
for image collections:
- assertContainsImageUrlInImageCollection(String, Collection<Image>), checking
that:
- the provided URL and image collection are not null;
- the image collection contains at least one image which has the provided
string value as its URL (which is a string) property.
- assertContainsOnlyEquivalentImages(Collection<Image>, Collection<Image>),
checking that:
- both collections are not null;
- they have the same size;
- each image of the first collection has its equivalent in the second one.
This means that the properties of an image in the first collection must be
equal in an image of the second one.
- assertNotOnlyContainsEquivalentImages(Collection<Image>, Collection<Image>),
checking that:
- both collections are not null;
- one of the following conditions is met:
- they have different sizes;
- an image of the first collection has not its equivalent in the second one.
This means that the properties of an image in the first collection must
be not equal in an image of the second one.
These methods will be used by services extractors tests (and default ones) to
test image collections.
2023-08-12 22:56:31 +02:00
AudricV
5158472852
Apply changes in DefaultTests and add utility method to test image lists
...
This new method, defaultTestImageList(List<Image), will check that the image
list is not null.
For each image, it will test that its URL is secure and its height and width
are more than or equal to their relevant unknown constants in the Image class
(HEIGHT_UNKNOWN and WIDTH_UNKNOWN).
2023-08-12 22:56:31 +02:00
AudricV
70fb3aa38e
Update BaseExtractorTests image methods' name
...
Also suppress unused warnings in BaseStreamExtractorTest, like it is done on
other BaseExtractorTests interfaces.
2023-08-12 22:56:30 +02:00
AudricV
e16d521b7b
[MediaCCC] Apply changes in Extractors
...
Also remove usage of the conference logo as the banner of a conference, as it
is a logo and not a banner.
2023-08-12 22:56:30 +02:00
AudricV
306068a63b
[MediaCCC] Apply changes in InfoItemExtractors
2023-08-12 22:56:30 +02:00
AudricV
2f40861428
[MediaCCC] Add utility methods to get image lists from conference logos and streams
...
These three new methods, added in MediaCCCParsingHelper,
getImageListFromImageUrl(String), getThumbnailsFromStreamItem(JsonObject) and
getThumbnailsFromLiveStreamItem(JsonObject) (the last two are based on a common
method, getThumbnailsFromObject(JsonObject, String, String)), return an empty
list if the case no image URL could be extracted.
Images returned have their height and width unknown and a resolution level
depending on the image key of the JSON API response.
2023-08-12 22:56:30 +02:00
AudricV
71cda03c4c
[Bandcamp] Apply changes in Extractors
2023-08-12 22:56:29 +02:00
AudricV
7e01eaac33
[Bandcamp] Apply changes in InfoItemExtractors
2023-08-12 22:56:29 +02:00
AudricV
4b80d737a4
[Bandcamp] Add utility methods to get multiple images
...
Bandcamp images work with image IDs, which provide different resolutions.
Images on Bandcamp are not always squares, and some IDs respect aspect ratios
where some others not.
The extractor will only use the ones which preserve aspect ratio and will not
provide original images, for performance and size purposes.
Because of this aspect ratio preservation constraint, only one dimension will
be known at a time.
The image IDs with their respective dimension used are:
- 10: 1200w;
- 101: 90h;
- 170: 422h;
- 171: 646h;
- 20: 1024w;
- 200: 420h;
- 201: 280h;
- 202: 140h;
- 204: 360h;
- 205: 240h;
- 206: 180h;
- 207: 120h;
- 43: 100h;
- 44: 200h.
(Where w represents the width of the image and h the height of the image)
Note that these dimensions are theoretical because if the image size is less
than the dimensions of the image ID, it will be not upscaled but kept to its
original size.
All these resolutions are stored in a private static list of ThumbnailSuffixes
in BandcampExtractorHelper, in which the methods to get mutliple images have
been added:
- getImagesFromImageUrl(String): public method to get images from an image URL;
- getImagesFromImageId(long, boolean): public method to get images from an
image ID;
- getImagesFromImageBaseUrl(String): private utility method to get images from
the static list of ThumbnailSuffixes from a given image base URL, containing
the path to the image, a "a" letter if it comes from an album, its ID and an
underscore.
Some existing methods have been also edited:
- the documentation of getImageUrl(long, boolean) has been changed to reflect
the Bandcamp images findings;
- getThumbnailUrlFromSearchResult has been renamed to
getImagesFromSearchResult, and a documentation has been added to this method.
The method replaceHttpWithHttps of the Utils class has been also used in
BandcampExtractorHelper instead of doing manually what the method does.
2023-08-12 22:56:29 +02:00
AudricV
4e6fb368bc
[PeerTube] Apply changes in Extractors and remove usages of default avatar picture
...
The default avatar picture was used when no profile picture was found, but it
was removed and split in multiple images.
Thumbnails' size is not known, as this data is not provided by the API.
2023-08-12 22:56:29 +02:00
AudricV
0a6011a50e
[PeerTube] Apply changes in InfoItemExtractors
...
Also lower the visibility of attributes of channels and playlists InfoItems to
private.
2023-08-12 22:56:29 +02:00
AudricV
6f8331524b
[PeerTube] Add utility method to get thumbnails of playlists and videos
...
This method, getThumbnailsFromPlaylistOrVideoItem, has been added in
PeertubeParsingHelper and returns the two image variants for playlists and
videos.
2023-08-12 22:56:28 +02:00
AudricV
81c0d80a54
[PeerTube] Add utility methods to get avatars and banners of accounts and channels
...
Four new static methods have been added in PeertubeParsingHelper to do so:
- two public methods to get the corresponding image type:
getAvatarsFromOwnerAccountOrVideoChannelObject(String, JsonObject) and
getBannersFromAccountOrVideoChannelObject(String, JsonObject);
- two private methods as helper methods: getImagesFromAvatarsOrBanners(String,
JsonObject, String, String) and getImagesFromAvatarOrBannerArray(String,
JsonArray).
2023-08-12 22:56:28 +02:00
AudricV
31da5beb51
[SoundCloud] Apply changes in Extractors
2023-08-12 22:56:28 +02:00
AudricV
a3a74cd566
[SoundCloud] Apply changes in InfoItemExtractors and return track user avatars as uploader avatars in SoundcloudStreamInfoItemExtractor
2023-08-12 22:56:28 +02:00
AudricV
7f818217d2
[SoundCloud] Add utility methods to get images from track JSON objects and image URLs
...
These new public and static methods, added in SoundcloudParsingHelper,
getAllImagesFromArtworkOrAvatarUrl(String) and
getAllImagesFromVisualUrl(String) (which call a common private method,
getAllImagesFromImageUrlReturned(String, List<ImageSuffix>, List<Image>)),
return an unmodifiable list of JPEG images containing almost every image
resolution provided by SoundCloud except the original size and the tiny
resolution (for artworks and avatars, as the image size is 20x20 for artworks
and 18x18 for avatars, so very close to or equal to the t20x20 resolution):
- for artworks and avatars:
- mini: 16x16;
- t20x20: 20x20;
- small: 32x32;
- badge: 47x47;
- t50x50: 50x50;
- t60x60: 60x60;
- t67x67: 67x67;
- large: 100x100;
- t120x120: 120x120;
- t200x200: 200x200;
- t240x240: 240x240;
- t250x250: 250x250;
- t300x300: 300x300;
- t500x500: 500x500.
- for visuals/user banners:
- t1240x260: 1240x260;
- t2480x520: 2480x520.
Duplicated code in two methods of SoundcloudParsingHelper
(getUsersFromApi(ChannelInfoItemsCollector, String) and
getStreamsFromApi(StreamInfoItemsCollector, String, boolean)) has been merged
into one common private method, getNextPageUrlFromResponseObject(JsonObject).
2023-08-12 22:56:28 +02:00
AudricV
266cd1f76b
[YouTube] Apply changes in YoutubeMusicSearchExtractor and split its InfoItemExtractors into separate classes
...
Splitting YoutubeMusicSearchExtractor's InfoItemExtractors into separate
classes (YoutubeMusicSongOrVideoInfoItemExtractor,
YoutubeMusicAlbumOrPlaylistInfoItemExtractor and
YoutubeMusicArtistInfoItemExtractor) allows to simplify
YoutubeMusicSearchExtractor,improves reading and applying changes to InfoItems
(no more losing at least quarter of a line due to indentations).
These InfoItems, in which the image changes have been applied, don't extend the
YouTube ones anymore, as most methods were overridden and the few ones that are
not don't apply in YouTube Music items responses, so it was useless to extend
them.
The code of YoutubeMusicSearchExtractor have been also improved a bit.
2023-08-12 22:56:27 +02:00
AudricV
c1981ed54f
[YouTube] Apply changes in Extractors except YoutubeMusicSearchExtractor
...
Also improve a bit some code related to the changes.
2023-08-12 22:56:27 +02:00
AudricV
4cc99f9ce1
[YouTube] Apply changes in InfoItemExtractors except YouTube Music ones
2023-08-12 22:56:27 +02:00
AudricV
adfad086ac
[YouTube] Add utility methods to get images from InfoItems and thumbnails arrays
...
Unmodifiable lists of Images are returned, parsed from a given YouTube
"thumbnails" JSON array.
These methods will be used in all YouTube extractors and InfoItems, as the
structures between content types (videos, channels, playlists, ...) are common.
2023-08-12 22:56:27 +02:00
AudricV
d56b880cae
Replace avatar and thumbnail URLs attributes and methods to List<Image> in Infos
2023-08-12 22:56:26 +02:00
AudricV
9d8098576e
Replace avatar and thumbnail URLs attributes and methods to List<Image> in Extractors
2023-08-12 22:56:26 +02:00
AudricV
0f4a5a8184
Replace avatar and thumbnail URLs attributes and methods to List<Image> in InfoItemsCollectors
2023-08-12 22:56:26 +02:00
AudricV
ca1d4a6fa4
Replace avatar and thumbnail URLs attributes and methods to List<Image> in InfoItemExtractors
2023-08-12 22:56:26 +02:00
AudricV
2f3ee8a3f2
Replace avatar and thumbnail URLs attributes and methods to List<Image> in InfoItems
2023-08-12 22:56:25 +02:00
AudricV
78ce65769f
Add an ImageSuffix class to the extractor
...
The goal of this utility class is to simply store suffixes which need to be
appended to image URLs, in order to get images at the suffix resolution.
This class contains four properties: the suffix (as a string), the height,
the width (as integers) and the estimated resolution level of the image
corresponding to the one represented by the suffix.
2023-08-12 22:56:25 +02:00
AudricV
d85454186a
Add an Image class to the extractor
...
Objects of this serializable class contains four properties: a URL (as a
string), a width, a height (represented as integers) and an estimated
resolution level, which can be constructed from a given height.
Possible resolution levels are:
- UNKNOWN: for unknown heights or heights <= 0;
- LOW: for heights > 0 & < 175;
- MEDIUM: for heights >= 175 & < 720;
- HIGH: for heights >= 720.
Getters of these properties are available and the constructor needs these four
properties.
2023-08-12 22:56:25 +02:00
Stypox
7294675aea
Merge pull request #1093 from AudricV/yt_support-shorts-ui-playlists
...
[YouTube] Support Shorts UI in playlists
2023-08-12 11:11:36 +02:00
Stypox
93a90b816d
Merge pull request #1094 from AudricV/yt_support-more-channel-headers
...
[YouTube] Support more channel headers
2023-08-12 11:08:30 +02:00
Stypox
44b664af15
[YouTube] Simplify Optional chains in channel
2023-08-12 11:02:51 +02:00
AudricV
2f7bfd3e7f
[YouTube] Add mocks of interactiveTabbedHeaderRenderer channel header test
2023-08-08 19:12:29 +02:00
AudricV
b147904571
[YouTube] Add test for interactiveTabbedHeaderRenderer channel header
...
This test uses the Minecraft game topic channel.
2023-08-08 19:12:28 +02:00
AudricV
1852031a0b
[YouTube] Support pageHeaderRenderer and interactiveTabbedHeaderRenderer channel headers
...
The addition of this support required to turn the isCarouselHeader boolean into
an enum containing all supported channel headers named HeaderType.
Also assert that the page has been fetched where needed to avoid
NullPointerExceptions when the channel page has been not fetched and remove the
getChannelHeaderJson method in YoutubeChannelExtractor, method for which its
code has been moved to its sole usage after the new headers support changes.
2023-08-08 19:12:27 +02:00
AudricV
698c710685
Do not require knowledge of uploader in default StreamInfoItems tests
...
This change is required as some services can return no uploader info, such as
YouTube for playlists with a Shorts UI.
2023-08-07 19:43:15 +02:00
AudricV
8237052ef5
Fix wrong assertion in assertNotEmpty
...
The non-null assertion was made on the exception message instead of the string
to check, causing a NullPointerException if the string to check was null.
2023-08-07 19:43:09 +02:00
AudricV
162c261577
[YouTube] Add mocks of the playlist with Shorts UI test
2023-08-07 19:07:53 +02:00
AudricV
e2f4ee47b9
[YouTube] Add a playlist with Shorts UI test
...
The system Shorts videos uploads playlist of the YouTube official channel has
been chosen for this test.
2023-08-07 19:06:09 +02:00
AudricV
e6f371fb94
[YouTube] Support Shorts UI in playlists
...
Also remove an outdated A/B test comment.
2023-08-07 19:01:08 +02:00
Stypox
7936987955
Merge pull request #1092 from Stypox/channel-tabs-improvements
...
Channel tabs code improvements
2023-08-06 21:44:10 +02:00
Stypox
6d2227111f
[YouTube] Assert that videos tab is ready after channel fetching
2023-08-06 21:14:57 +02:00
Stypox
ee625c325c
Inherit from DefaultListExtractorTest in channel tab tests
2023-08-06 21:14:56 +02:00
Stypox
276c293889
Rename assertTabsContain
2023-08-06 21:14:56 +02:00
Stypox
9d3761a371
[YouTube] Directly use playlist collector in channel tabs wrapper
...
Note that this introduces a "Raw use of parameterized class 'InfoItemsPage'" warning, but it can be ignored since the type missing would be <InfoItem>, and StreamInfoItem extends InfoItem
2023-08-06 21:13:25 +02:00
Stypox
95a3cc0a17
Merge pull request #1083 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.10.0
...
Bump org.junit:junit-bom from 5.9.3 to 5.10.0
2023-08-06 18:58:27 +02:00
Stypox
e34b4f1978
[YouTube] Avoid using Consumer
2023-08-06 13:02:31 +02:00
Stypox
ef67c7cd74
[YouTube] Simplify usage of channel header json
2023-08-06 13:02:31 +02:00
Stypox
a104cf3227
[YouTube] Fix docs in channel helper
2023-08-06 13:02:31 +02:00
Stypox
bb47f05f89
Merge pull request #1091 from Stypox/update-mocks
...
[YouTube] Update stream mocks after #1087
2023-08-06 12:58:24 +02:00
Stypox
468bcc045d
[YouTube] Update mocks after #1087
2023-08-06 12:33:04 +02:00
Stypox
35f3a4ad01
Merge pull request #1082 from AudricV/channel-tabs-and-tags-support
...
Add support for channel tabs and channel tags
2023-08-06 12:21:42 +02:00
AudricV
e7d64099a7
[YouTube] Update channel mocks and add channel tabs mocks
2023-08-06 12:15:06 +02:00
AudricV
684101c47d
[YouTube] Implement age-restricted channels support, link handlers and channels tabs and tags changes on tests
...
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV
eaf2600ce0
[SoundCloud] Implement link handlers and channels tabs and tags changes on tests
...
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV
0ee2072de5
[PeerTube] Implement link handlers and channels tabs and tags changes on tests
...
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV
d3801dd0e9
[MediaCCC] Implement link handlers and channels tabs and tags changes on tests
...
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV
8baec04611
[Bandcamp] Implement link handlers and channels tabs and tags changes on tests
...
Tests in BandcampChannelExtractorTest and BandcampChannelLinkHandlerFactoryTest
have been also fixed.
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV
e0ba29cd19
Add utility method to assert that given channel tabs are in the ones returned by a channel extractor
...
Only the first content filter of the ListLinkHandler instances provided is
used when collecting all channel tabs of the ListLinkHandler list, as channel
tabs implementations only use one content filter per ListLinkHandler instance.
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV
18846baba7
Add tabs and tags methods in tests interfaces and annotate all methods with the Test JUnit annotation
...
These changes should help to detect tests as tests, when running a subset of
tests or all tests.
They should be also implemented in these interfaces' implementations (new and
existing ones).
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
ThetaDev
c70a0e3543
Add a test for textual durations parsing using TimeAgoParser's patterns
2023-08-06 12:15:06 +02:00
AudricV
7366eab156
[YouTube] Add support for channel tabs and tags and age-restricted channels
...
Support of tags and videos, shorts, live, playlists and channels tabs has been
added for non-age restricted channels.
Age-restricted channels are now also supported and always returned the videos,
shorts and live tabs, accessible using system playlists. These tabs are the
only ones which can be accessed using YouTube's desktop website without being
logged-in.
The videos channel tab parameter has been updated to the one used by the
desktop website and when a channel extraction is fetched, this tab is returned
in the list of tabs as a cached one in the corresponding link handler.
Visitor data support per request has been added, as a valid visitor data is
required to fetch continuations with contents on the shorts tab. It is only
used in this case to enhance privacy.
A dedicated shorts UI elements (reelItemRenderers) extractor has been added,
YoutubeReelInfoItemExtractor. These elements do not provide the exact view
count, any uploader info (name, URL, avatar, verified status) and the upload
date.
All service's LinkHandlers are now using the singleton pattern and some code
has been also improved on the files changed.
Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:15:04 +02:00
AudricV
4586067934
Add utility method to parse textual durations using TimeAgoParser's patterns
...
This is required to parse duration of YouTube's reelItemRenderers, returned
only inside accessibility data.
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:13:33 +02:00
AudricV
d4bfe791ee
[SoundCloud] Add tabs support for users
...
Support of tracks, playlists and albums has been added for users.
Also add the declaration of the UnsupportedOperationException exception to the
service's LinkHandlers.
Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:13:32 +02:00
AudricV
6f7d1f079f
[Bandcamp] Add tabs support for artists
...
Support of tracks and albums has been added for artists.
Also use the singleton pattern and add the declaration of the
UnsupportedOperationException exception to the service's LinkHandlers and
improved some code in the files changed.
Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:12:19 +02:00
AudricV
1e8474b22d
[PeerTube] Add tabs support for accounts and video channels
...
Support of channels and videos has been added for accounts and support of
videos and playlists has been added for video channels.
The following changes have been also done:
- collectStreamsFrom method in PeertubeParsingHelper has been renamed to
collectItemsFrom;
- PeertubeChannelInfoItemExtractor.getStreamCount method has been fixed due to
ChannelExtractor's new inheritance;
- the declaration of the UnsupportedOperationException exception thrown has
been added to the service's LinkHandlers;
- a channel tab LinkHandlerFactory has been added,
PeertubeChannelTabLinkHandlerFactory;
- all service's LinkHandlers are now using properly the singleton pattern.
Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:12:15 +02:00
AudricV
652c2c8408
Add a ListLinkHandler which can be used to be returned from ChannelInfo.getTabs() when a specific tab's data has already been fetched
...
This new ListLinkHandler, ReadyChannelTabListLinkHandler, should help saving
clients data, energy and time by helping to reduce duplicate requests.
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:11:12 +02:00
AudricV
de823a6b68
Add an UnsupportedTabException exception class
...
This class makes easier for LinkHandlerFactory implementations to declare an
UnsupportedOperationException.
2023-08-06 12:11:12 +02:00
AudricV
76fb9dcdd7
Add UnsupportedOperationException to exceptions which can be thrown by getId and getUrl methods of LinkHandlerFactory and its base implementations
...
This change advertise to clients that channel tabs' link handler factories can
return an UnsupportedOperationException when a tab provided to them is
unsupported.
2023-08-06 12:11:12 +02:00
AudricV
946eb9bd91
Add structure of channel tags
...
Tags' getters and/or setters have been added in ChannelExtractor and
ChannelInfo to do so.
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:11:12 +02:00
AudricV
356a888d6c
Add structure of channel tabs
...
This commit introduces the following breaking changes:
- Three new classes have been added:
- ChannelTabExtractor, class extending ListExtractor<InfoItem>, which
extracts InfoItems from a channel tab;
- ChannelTabInfo extending ListInfo<InfoItem>, which extracts InfoItems from
a ChannelTabExtractor and returns them as a ChannelTabInfo;
- ChannelTabs, an immutable class containing all supported channel tabs.
- StreamingService implementations must implement new methods returning a
channel tab LinkHandlerFactory (getChannelTabsLHFactory) and a
ChannelTabExtractor (getChannelTabExtractor);
- ChannelExtractor inherits Extractor instead of ListExtractor<StreamInfoItem>
and ChannelInfo inherits Info instead of ListInfo<StreamInfoItem>;
- ChannelExtractor and ChannelInfo have now getters and/or setters of tabs.
Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:11:11 +02:00
Stypox
3faaf4301c
Merge pull request #1087 from AudricV/yt_js-extractor-improvements-and-fixes
...
[YouTube] Improve and fix YoutubeJavaScriptExtractor
2023-08-06 12:01:00 +02:00
Stypox
8fb6ba36fa
Merge pull request #1081 from TeamNewPipe/fix/sc/search-next-page
...
[SoundCloud] Detect whether there are any more search results
2023-08-06 11:49:35 +02:00
Stypox
2947257111
[SoundCloud] Properly calculate if results have finished
2023-08-06 11:38:22 +02:00
Stypox
485bfbca9d
[SoundCloud] Move try-catch inside getOffsetFromUrl
2023-08-06 11:35:37 +02:00
Stypox
7c70fef197
Merge pull request #1089 from TeamNewPipe/ccc
...
[media.ccc.de] Only extract kiosk live stream rooms if they are streaming
2023-08-06 10:12:04 +02:00
Stypox
de0a9bb797
Merge pull request #1088 from FireMasterK/pseudo-rng
...
Replace cryptographically secure random with regular random
2023-08-05 11:37:49 +02:00
TobiGr
340095515d
Make Kiosk IDs accessible if possible
2023-08-05 03:18:40 +02:00
TobiGr
fe27d6a0ec
[media.ccc.de] Only extract live streams if the conference is streaming
2023-08-05 01:53:43 +02:00
Kavin
25082d78b0
Replace SecureRandom with Random
2023-08-03 23:00:02 +01:00
TobiGr
aa6c17dc77
[SoundCloud] Deduplicate some code
2023-08-03 14:41:30 +02:00
TobiGr
2fb9922a15
[SoundCloud] Detect whether there are any more search results
...
Add test for this edge case.
2023-08-03 14:37:13 +02:00
AudricV
a3d160edab
[YouTube] Improve and fix YoutubeJavaScriptExtractor
...
- Enhance documentation;
- Fix the regular expression fallback on HTML embed watch page;
- Use HTML scripts tag search first instead of the regular expression approach,
now used as a last resort;
- Compile regular expressions only once, in order to improve the performance of
subsequent extraction calls when clearing the cache;
- Provide original exceptions when fetching or parsing pages on which the base
JavaScript's player could be found failed, allowing clients to detect network
errors when they are the cause of the failures for instance;
- Remove delegate method which was not taking a video ID and hardcoding one, as
we can provide the video ID in all cases or do not provide a video ID at worse;
- Rename and make extraction methods package-private, as they are not intended
to be used publicly.
These breaking internal changes have been applied where needed, in
YoutubeJavaScriptExtractorTest and YoutubeStreamExtractor (in which an unneeded
initStsFromPlayerJsIfNeeded call have been removed).
2023-08-02 23:05:08 +02:00
Stypox
5492343b8e
Pre-release v0.22.7
2023-08-02 20:02:29 +02:00
Tobi
9a59afbcf5
Merge pull request #1086 from AudricV/yt_no-exception-channels-without-banner
...
[YouTube] Don't throw an exception when there is no banner available on a channel
2023-08-02 18:56:26 +02:00
AudricV
bb1ab166bf
[YouTube] Test that no banner is returned for carouselHeaderRenders
2023-08-01 22:19:43 +02:00
AudricV
f1fa84b4e3
[YouTube] Don't throw an exception when there is no banner available on a channel
...
Channels may not have a banner, so no exception should be thrown if no banner
is found.
2023-08-01 12:40:20 +02:00
Tobi
39a911db9f
Merge pull request #1084 from AudricV/yt_android-403s-workaround-and-streams-tests-fixes
...
[YouTube] Workaround again 403 HTTP issues on the ANDROID InnerTube client and fix stream tests
2023-07-31 23:51:10 +02:00
dependabot[bot]
bda3a3fc5d
Bump org.junit:junit-bom from 5.9.3 to 5.10.0
...
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5 ) from 5.9.3 to 5.10.0.
- [Release notes](https://github.com/junit-team/junit5/releases )
- [Commits](https://github.com/junit-team/junit5/compare/r5.9.3...r5.10.0 )
---
updated-dependencies:
- dependency-name: org.junit:junit-bom
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-07-24 09:53:05 +00:00
AudricV
522c78160f
[YouTube] Update stream tests mocks
2023-07-23 19:36:28 +02:00
AudricV
7528eb2bd9
[YouTube] Fix stream tests failures
...
- Fix testCheckAudioStreams test of
YoutubeStreamExtractorDefaultTest.AudioTrackLanguage test class, by updating
the excepted audio track name test to use the updated English audio track name
(audio track type info has been added on the video tested);
- Fix YoutubeStreamExtractorDefaultTest.PublicBroadcasterTest test class by
using a different video from a French and German public broadcast channel, as
the channel Dinge Erklärt – Kurzgesagt is not affiliated with a public
broadcast channel anymore;
- Fix YoutubeStreamExtractorLivestreamTest test class, by updating the excepted
name of the livestream to the current one.
2023-07-23 19:19:02 +02:00
AudricV
164c8e3abb
[YouTube] Workaround again 403 HTTP issues on the Android client by using new player parameters
...
These parameters are the only ones currently known to bypass 403 HTTP issues
related to failure of passing Android client integrity checks, as the ones of
stories (and the base of the shorts ones) do not work anymore, which may be
related to end of this format on the service.
2023-07-22 20:22:16 +02:00
Tobi
8ebbe1b7e1
Merge pull request #706 from FireMasterK/av1-itags
...
Add support for AV1 itags.
2023-07-22 13:31:45 +02:00
FireMasterK
6db0d116fe
Add support for AV1 itags.
2023-07-22 13:23:44 +02:00
Tobi
986a3f7747
Merge pull request #1080 from TeamNewPipe/proguard-rules
...
Add proguard rules to README
2023-07-07 12:52:10 +02:00
Tobi
d6132a9ed9
Merge pull request #1076 from AudricV/yt-fix-throttling-parameter-regex
...
[YouTube] Support multiple declarations for throttling parameter function name array
2023-07-07 12:51:26 +02:00
Tobi
87eb29a25d
Add proguard rules to README
2023-07-05 08:25:23 +02:00
AudricV
4e22c5ee87
[YouTube] Support multiple declarations for throttling parameter function name array
...
Also moved the corresponding regex parts in static constants for easier future
modifications
2023-06-26 15:25:53 +02:00
Kavin
d961d349c3
[YouTube] Check whether player responses are valid for all InnerTube clients used ( #1070 )
...
Co-authored-by: Audric V <74829229+AudricV@users.noreply.github.com>
2023-06-18 21:54:52 +02:00
ThetaDev
ad97f08048
[YouTube] Fix parsing short relative date formats (English only) ( #1068 )
2023-06-18 21:41:29 +02:00
Tobi
d294ccb433
Merge pull request #1071 from TeamNewPipe/feat/ServiceList
...
Init services at the correct place
2023-06-17 20:50:24 +02:00
Tobi
5d112aa772
Merge pull request #1074 from TeamNewPipe/imp/media-formats
...
Add more audio media formats and MediaFormat.getAllForMimeType(mimeType)
2023-06-17 20:49:38 +02:00
TobiGr
0e01d90562
Do not init services while creating services array
2023-06-17 18:08:09 +02:00
TobiGr
5809904cf7
Add annotations to MediaFormat
2023-06-17 18:01:54 +02:00
TobiGr
53dfd871e2
Add more audio media formats and MediaFormat.getAllForMimeType(mimeType)
2023-06-17 18:01:29 +02:00
Stypox
5eb886243f
Merge pull request #1072 from TeamNewPipe/fix/bandcamp_test
...
Fix BandcampRadioStreamExtractorTest.testGetAudioStreams()
2023-06-09 17:35:35 +02:00
TobiGr
19ce06fe00
Fix BandcampRadioStreamExtractorTest.testGetAudioStreams()
2023-06-06 23:07:56 +02:00
Audric V
533121fb81
Merge pull request #1045 from Theta-Dev/fix/trending-video-tab
...
[YouTube] Extract trends from A/B tested "Videos" tab and fix extraction of trends name from A/B tested new title design
2023-05-19 11:22:49 +02:00
Audric V
92a0024424
Merge pull request #1052 from TeamNewPipe/peertube/fix/nested-comment-replies
...
[PeerTube] Fix multi level comment replies
2023-05-18 18:49:06 +02:00
Tobi
3c036a9c03
Merge pull request #1061 from ChunkyProgrammer/develop/extract-yt-playlist-descrption
...
Extract playlists description
2023-05-17 22:43:17 +02:00
TobiGr
c70bb83801
[Bandcamp] Implement PlaylistExtractor.getDescsription()
2023-05-15 15:23:03 +02:00
TobiGr
ca0ce00753
Add PlaylistInfoItem.getDescription() and PlaylistInfoItemExtractor.getDescription()
...
[PeerTube] Implement the corresponding extractor method.
TODO: add tests
2023-05-12 01:43:59 +02:00
TobiGr
b218bf69bd
Implement PlaylistInfo.getDescription()
...
Implement PlaylistExtractor.getDescription() for PeerTube and SoundCloud.
Anotate PlaylistExtractor.getDescription() as Nonnull
2023-05-12 00:44:10 +02:00
chunky programmer
81f29116ba
switch from string to Description object
2023-05-11 00:36:57 -04:00
chunky programmer
e147867d41
Add tests
2023-05-11 00:00:34 -04:00
chunky programmer
5ab6cd7420
Extract YouTube playlist description
2023-05-11 00:00:22 -04:00
TobiGr
d358ba1c41
Improve PeertubeCommentsInfoItemExtractor constructor
2023-05-07 22:55:26 +02:00
TobiGr
aff3e795f8
[PeerTube] Fix multi level comment replies
2023-05-07 22:49:14 +02:00
ThetaDev
3673d4ae01
fix: YouTube trending name extraction
2023-05-03 21:16:35 +02:00
ThetaDev
0addb98cd7
tests: update mocks
2023-05-03 21:16:35 +02:00
ThetaDev
24eba62305
fix: extract YouTube trends from new "Videos" tab
2023-05-03 21:16:23 +02:00
Kavin
a9ca5c49e4
Merge pull request #1056 from AudricV/yt-improve-search-suggestions-extraction
...
[YouTube] Switch to new search suggestion domain and improve error handling
2023-05-02 20:17:48 +01:00
Stypox
00408db959
Merge pull request #1058 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.16.1
...
Bump org.jsoup:jsoup from 1.15.4 to 1.16.1
2023-05-01 19:59:02 +02:00
dependabot[bot]
108f8a7a17
Bump org.jsoup:jsoup from 1.15.4 to 1.16.1
...
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup ) from 1.15.4 to 1.16.1.
- [Release notes](https://github.com/jhy/jsoup/releases )
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES )
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.4...jsoup-1.16.1 )
---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-05-01 09:58:32 +00:00
Tobi
11565db17f
Merge pull request #1055 from AudricV/peertube-no-description-channels
...
[PeerTube] Don't return "No description" when there is no description for a channel or an account
2023-04-30 23:38:10 +02:00
AudricV
cf6040ddb3
Update YoutubeSuggestionExtractorTest mocks
2023-04-30 19:53:44 +02:00
AudricV
593122342f
[YouTube] Improve YoutubeSuggestionExtractorTest
...
- Remove useless concatenation on the downloader path;
- Remove unneeded public test modifier;
- Update license header;
- Specify the service class tested instead of the generic class.
2023-04-30 19:53:43 +02:00
AudricV
e923fca440
[YouTube] Switch to new search suggestion domain and improve error handling
...
- Switch to the new domain used by YouTube for search suggestions,
suggestqueries-clients6.youtube.com, and add the xhr query parameter with the
t value, to allow getting responses without requiring trim;
- Use the Java 8 Stream API to collect search suggestions and improve invalid
response detection by checking whether the content type of the response
returned is JSON;
- Move the licence header at the top of the file.
2023-04-30 19:53:42 +02:00
AudricV
945165a3c0
[PeerTube] Don't return "No description" when there is no description for a channel or an account
...
When a description is missing, no description should be returned, even the ones
indicating there is no description. This behavior is represented by a null
return instead.
Also update PeertubeAccountExtractorTest to reflect these changes.
2023-04-30 18:41:38 +02:00
Stypox
d61dc27406
Merge pull request #1054 from TeamNewPipe/fix-ci-variable
...
Make sure CI is not vulnerable to bash word splitting
2023-04-27 18:52:07 +02:00
Stypox
51cba5a287
Make sure CI is not vulnerable to bash word splitting
...
See https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-an-intermediate-environment-variable
2023-04-26 18:36:25 +02:00
Stypox
f819c2fe34
Merge pull request #1053 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.3
...
Bump org.junit:junit-bom from 5.9.2 to 5.9.3
2023-04-26 15:42:19 +02:00
dependabot[bot]
4b92c5a34d
Bump org.junit:junit-bom from 5.9.2 to 5.9.3
...
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5 ) from 5.9.2 to 5.9.3.
- [Release notes](https://github.com/junit-team/junit5/releases )
- [Commits](https://github.com/junit-team/junit5/compare/r5.9.2...r5.9.3 )
---
updated-dependencies:
- dependency-name: org.junit:junit-bom
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-04-26 09:57:57 +00:00
Stypox
2deb023da4
Merge pull request #1050 from Theta-Dev/fix/channel-carousel-header
...
[YouTube] Add support for CarouselHeaderRenderer
2023-04-25 15:17:31 +02:00
ThetaDev
4aada7f91b
refactor: rename carousel header channel test
2023-04-21 22:48:37 +02:00
Tobi
b2b426048f
Merge pull request #1051 from Theta-Dev/fix/ytm-client-version-rg
...
[YouTube] fix: set musicClientVersion regex capture group
2023-04-16 23:48:19 +02:00
ThetaDev
47aa9fed40
fix: set musicClientVersion regex capture group
2023-04-16 19:25:05 +02:00
ThetaDev
20370395c5
fix: add support for CarouselHeaderRenderer
2023-04-16 17:40:13 +02:00
Stypox
7dba6e3891
Merge pull request #1033 from petlyh/bandcamp-paywalled-content
...
[Bandcamp] Handle paywalled tracks
2023-04-12 13:04:26 +02:00
petlyh
e6aad117e7
[Bandcamp] Throw PaidContentException on paywalled albums
2023-04-03 19:27:09 +02:00
TobiGr
8495ad619e
Bump version to 0.22.6
2023-04-02 22:35:02 +02:00
fynngodau
69705138e4
[Bandcamp] Fix extraction of related playlist items URL ( #1047 )
...
Small change in HTML structure
2023-04-02 22:24:29 +02:00
Björn Sigurbergsson
1b6fe5edd6
[YouTube] Fix ParsingException when comments are unavailable in a video ( #1040 )
...
Co-authored-by: bjs <bjs@elect-it.com>
Co-authored-by: Audric V. <74829229+AudricV@users.noreply.github.com>
Co-authored-by: Kavin <20838718+FireMasterK@users.noreply.github.com>
2023-03-30 19:58:06 +02:00
ThetaDev
8d1303e18f
Add track types to audio streams ( #1041 )
2023-03-28 00:02:20 +02:00
Tobi
44ae139d33
Merge pull request #1034 from AudricV/peertube-fix-commentsinfo-test
...
[PeerTube] Fix comments info test
2023-03-20 19:30:35 +01:00
Tobi
d5e9df5381
Merge pull request #1038 from TeamNewPipe/docs-jdk-11
...
Update JDK to 11 in JDoc workflow
2023-03-20 19:25:21 +01:00
Tobi
0486ad7a08
Update JDK to 11 in JDoc workflow
2023-03-06 18:45:11 +01:00
AudricV
80a6fc2c63
[PeerTube] Fix testGetCommentsFromCommentsInfo test of PeertubeCommentsExtractorTest.Default
...
The tested comment has been removed, so it couldn't be found in the comments
list.
This comment has been replaced by a new one from the current comments of the
video.
Also, in the parent class PeertubeCommentsExtractorTest, final has been used as
much as possible and for-each loops of lists have been replaced by their
forEach method or the Stream API, in order to simplify code.
2023-03-04 16:49:10 +01:00
petlyh
5a9b6ed2e3
[Bandcamp] Support loading additional comments ( #1030 )
2023-03-04 14:01:06 +01:00
Stypox
6bdd698c25
Merge pull request #1026 from AudricV/audio-streams-descriptive-and-locale-properties
...
Add descriptive and locale properties to audio streams
2023-03-01 11:15:46 +01:00
Stypox
19e4b216c9
Merge pull request #1032 from AudricV/yt_fix-comments-hashtags-links-extraction
...
[YouTube] Fix hashtags links extraction and escape HTML links
2023-03-01 10:47:37 +01:00
Stypox
b1298490c0
Merge pull request #1029 from AudricV/yt_fix-no-views-extraction-playlist-items
...
[YouTube] Fix partial non-extraction of "No views" string in stream items
2023-03-01 10:46:52 +01:00
petlyh
9dc1832733
[Bandcamp] Handle paywalled tracks
2023-02-28 17:51:30 +01:00
fynngodau
3fdb6ee476
Merge pull request #1031 from petlyh/bandcamp-fix-radio-comments
...
[Bandcamp] Show comments as disabled on radio streams
2023-02-27 20:50:12 +01:00
AudricV
bd79b921e8
[YouTube] Refactor the code to get stream items' view count
...
This refactoring avoids code duplication as much as possible.
2023-02-27 10:25:46 +01:00
AudricV
51f9b39953
[YouTube] Fix partial non-extraction of no views string in stream items
...
As the "No views" string is returned in the case there is no view on a video, a
number cannot be parsed in this case, so -1 was returned.
This string is now detected in all methods to get the view count of a stream.
2023-02-27 10:18:45 +01:00
AudricV
95b3f5e391
[MediaCCC] Test audio language property extraction
2023-02-26 19:06:18 +01:00
AudricV
30a0f8c510
[MediaCCC] Extract audio language property for single language audio tracks
2023-02-26 19:06:18 +01:00
AudricV
7f0269c4c7
[YouTube] Edit YoutubeStreamExtractorDefaultTest.AudioTrackLanguage to test audio locale property
...
The Hindi audio track language presence test has been changed from audio track
label to audio locale.
2023-02-26 19:06:18 +01:00
AudricV
034f82dae7
[YouTube] Test language and descriptive audio in YoutubeDashManifestCreatorsTest
2023-02-26 19:06:17 +01:00
AudricV
05e8cb39f7
[YouTube] Add language and descriptive audio properties to DASH manifests
2023-02-26 19:06:17 +01:00
AudricV
bf30d70152
[YouTube] Add descriptive audio test
...
This test uses video TjxC-evzxdk.
Also improve a bit YoutubeStreamExtractorDefaultTest.AudioTrackLanguage test.
2023-02-26 19:06:17 +01:00
AudricV
76b7c19c5d
[YouTube] Extract whether a track is a descriptive audio and audio locale when available
...
Also use audio track setters only for audio itags.
2023-02-26 19:06:17 +01:00
AudricV
3bb5eeef30
[YouTube] Add descriptive and locale audio support in ItagItem
2023-02-26 19:06:16 +01:00
AudricV
14bf3fb05b
Add ability to know the locale of an audio stream
...
Getting audio tracks locales by parsing their ID or their label, should not be
done by clients, but by the extractor.
This commit adds the ability to store the Locale of an AudioStream, which is
used to compare similar AudioStreams (in the equalStats method).
2023-02-26 19:06:16 +01:00
AudricV
f92426560c
Add descriptive audio properties
...
Also improve AudioStream's audio language documentation
2023-02-26 19:06:16 +01:00
AudricV
a63f289667
[YouTube] Update mocks of YoutubeCommentsExtractorTest.FormattingTest
2023-02-26 18:50:07 +01:00
AudricV
9483dcd9fa
[YouTube] Update mocks of YoutubeCommentsExtractorTest.RepliesTest
2023-02-26 18:43:36 +01:00
AudricV
1556adbb2d
[YouTube] Fix hashtags links extraction and escape text in attribute descriptions + HTML links
...
webCommandMetadata object is contained inside a commandMetadata one, so it is
not accessible from the root of the navigationEndpoint object.
The corresponding statement has been moved at the bottom of the specific
endpoints parsing, as the webCommandMetadata object is present almost
everywhere, otherwise URLs of some endpoints would have be changed, such as
uploader URLs (from channel IDs to handles).
As no ParsingException is now thrown by getUrlFromNavigationEndpoint, and so by
getTextFromObject, getUrlFromObject and getTextAtKey, the methods which were
catching ParsingExceptions thrown by these methods had to be updated.
URLs got in the HTML version of getTextFromObject are now escaped properly to
provide valid HTML to clients. This has been also done for attribute
descriptions, with the description text for this type of descriptions.
As YouTube descriptions are in HTML format (except for the fallback on the JSON
player response, which is plain text and only happens when there is no visual
metadata or a breaking change), all URLs returned are escaped, so tests which
are testing presence of URLs with escaped characters had to be updated (it was
only the case for YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing).
2023-02-26 18:43:36 +01:00
Stypox
99ab9777ad
Release 0.22.5
2023-02-26 16:33:25 +01:00
Stypox
39c2e1fda0
Merge pull request #1028 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.4
...
Bump org.jsoup:jsoup from 1.15.3 to 1.15.4
2023-02-26 16:27:31 +01:00
petlyh
f7a7a236fb
[Bandcamp] Show comments as disabled on radio streams
2023-02-23 18:42:43 +01:00
dependabot[bot]
f5599ff08d
Bump org.jsoup:jsoup from 1.15.3 to 1.15.4
...
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup ) from 1.15.3 to 1.15.4.
- [Release notes](https://github.com/jhy/jsoup/releases )
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES )
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.3...jsoup-1.15.4 )
---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-02-20 10:05:26 +00:00
Stypox
7e793c11ae
Empty commit to make JitPack happy
...
The build for e920ab3f5f
succeeded, but for some reason it is considered as failed...
2023-02-08 22:35:26 +01:00
Stypox
e920ab3f5f
Merge pull request #1021 from TeamNewPipe/fix/yt-comment-text-hashtag
...
[YouTube] Fix getting the comment text if the comment contains a hashtag
2023-01-29 21:47:59 +01:00
TobiGr
3f7df9536e
[YouTube] Fix getting the comment text if the comment contains a hashtag
2023-01-29 20:33:51 +01:00
Stypox
999fb7f812
Merge pull request #1024 from AudricV/snd_fix-tracks-like-count
...
[SoundCloud] Fix extraction of tracks like count
2023-01-29 10:52:54 +01:00
Stypox
3519d4c367
Merge pull request #1015 from AudricV/yt_fix-channel-id-rss-feeds
...
[YouTube] Fix channel ID extraction of YouTube channels RSS feeds
2023-01-29 10:41:38 +01:00
Stypox
9aca710e86
Merge pull request #1013 from Stypox/fix-music-mixes
...
[YouTube] Now music mixes can be treated as normal mixes
2023-01-29 09:48:51 +01:00
Stypox
76eeabac45
Merge pull request #1020 from TeamNewPipe/fix/yt-subscriber-count
...
[YouTube] Fix NPE in search when getting channel items without subscriber count
2023-01-29 09:44:22 +01:00
AudricV
676622f6df
[SoundCloud] Fix expectedLikeCountAtLeast tests of SoundcloudStreamExtractorTest test classes
...
As like count is now returned by the extractor, we need to assert a positive
minimum like count, which is close to the actual value, in order to avoid test
failures due to lower like counts than the ones excepted.
2023-01-29 01:08:02 +01:00
AudricV
2a24d407d5
[SoundCloud] Fix extraction of tracks like count
...
SoundCloud is using likes_count to return the like count of a track, like it
was the case before they switched to favoritings_count.
2023-01-29 01:00:49 +01:00
Tobi
0e4e6a9bac
Merge pull request #1022 from AudricV/yt_support-live-urls
...
[YouTube] Support live URLs
2023-01-28 21:52:26 +01:00
AudricV
ba24976e41
[YouTube] Add live URLs test and do minor improvements to YoutubeStreamLinkHandlerFactoryTest
...
- Remove unused imports;
- Replace wildcard imports by single class imports;
- Suppress "HTTP links are not secured" warnings from IDEA IDEs;
- Replace removed video jZViOEv90dI by an existing video, 9Dpqou5cI08 (the
corresponding test has been of course renamed).
2023-01-28 19:36:21 +01:00
AudricV
57f850bc2d
[YouTube] Support live URLs and do minor improvements to YoutubeStreamLinkHandlerFactory
...
- Move license header at the top;
- Use an unmodifiable set for the subpaths instead of a modifiable list;
- Add missing Nonnull and Nullable annotations;
- Improve exception messages.
2023-01-28 19:36:20 +01:00
AudricV
1f4ed9dce9
[YouTube] Fix channel ID extraction of YouTube channel RSS feeds
...
The yt:channelId element doesn't provide the channel ID anymore and is empty,
like the id element, so we need now to extract it from the channel URL provided
in two elements: author -> uri and feed -> link.
Also avoid a NullPointerException in getUrl and getName methods.
2023-01-28 11:53:33 +01:00
Tobi
c589a2c1a2
Merge pull request #1014 from TeamNewPipe/fix/yt-comments
...
[YouTube] Fix getting next comments pages
2023-01-27 11:14:55 +01:00
Tobi
c2b36fd405
Merge pull request #1016 from lonewolf2208/bug-fix
...
[YouTube] Remove topStandaloneBadge check for view count of stream items
2023-01-25 08:30:36 +01:00
TobiGr
72573932cf
[YouTube] Fix NPE in search when getting channel items without subscriber count
2023-01-24 23:03:45 +01:00
TobiGr
f50b7275af
[YouTube] Fix getting next comments pages
2023-01-24 22:39:08 +01:00
Kunal
9bdad40b06
Removed topStandaloneBadge
2023-01-20 02:41:21 +05:30
Stypox
5945057227
[YouTube] Add music mix test
2023-01-15 23:30:30 +01:00
Stypox
7293991832
[YouTube] Now music mixes can be treated as normal mixes
...
Using a playlist extractor on them would result in "Unviewable playlist" errors
2023-01-15 23:28:59 +01:00
Stypox
ff94e9f30b
Merge pull request #1009 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.10.1
...
Bump gson from 2.10 to 2.10.1
2023-01-11 15:35:36 +01:00
Stypox
c1040bccac
Merge pull request #794 from FireMasterK/comments-count
...
[YouTube] Add support to extract total comment count
2023-01-11 15:32:19 +01:00
Stypox
6ccc43e57e
Merge pull request #1010 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.2
...
Bump junit-bom from 5.9.1 to 5.9.2
2023-01-11 13:19:33 +01:00
dependabot[bot]
1bc44cb949
Bump junit-bom from 5.9.1 to 5.9.2
...
Bumps [junit-bom](https://github.com/junit-team/junit5 ) from 5.9.1 to 5.9.2.
- [Release notes](https://github.com/junit-team/junit5/releases )
- [Commits](https://github.com/junit-team/junit5/compare/r5.9.1...r5.9.2 )
---
updated-dependencies:
- dependency-name: org.junit:junit-bom
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-01-11 09:04:12 +00:00
dependabot[bot]
f43049985e
Bump gson from 2.10 to 2.10.1
...
Bumps [gson](https://github.com/google/gson ) from 2.10 to 2.10.1.
- [Release notes](https://github.com/google/gson/releases )
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md )
- [Commits](https://github.com/google/gson/compare/gson-parent-2.10...gson-parent-2.10.1 )
---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-01-09 09:05:55 +00:00
Stypox
33f9f2cf66
Merge pull request #1006 from TeamNewPipe/yt/fix-html-escape
...
[YouTube] Fix escaping links in YouTubeParsingHelper.getTextFromObject
2023-01-05 19:02:23 +01:00
TobiGr
56aab4d971
[YouTube] Fix escaping links in YouTubeParsingHelper.getTextFromObject
2023-01-05 00:28:12 +01:00
Kavin
22a47da8c7
Fix requested change and remove outdated comment.
2023-01-02 20:42:32 +00:00
Kavin
98a90fd9c8
Don't cache comments count and return early on page fetch if no token.
2023-01-02 20:40:48 +00:00
Kavin
2974dfaa48
Only store ajaxJson for initial page and eager fetch the initial continuation.
2023-01-02 20:40:48 +00:00
Kavin
64d24aa09e
Fix request changes.
2023-01-02 20:40:48 +00:00
Kavin
67ef4f4c30
Cleanup and remove optional.
2023-01-02 20:40:48 +00:00
FireMasterK
22f71b010c
Fix for requested changes.
2023-01-02 20:40:48 +00:00
FireMasterK
656b7c1cd9
Improve method documentation.
2023-01-02 20:40:48 +00:00
FireMasterK
981aee4092
Add support to extract total comment count.
2023-01-02 20:40:48 +00:00
Stypox
45636b0d00
Merge pull request #986 from Isira-Seneviratne/Static_maps
...
Use immutable Map factory methods.
2023-01-02 18:11:14 +01:00
Stypox
219c5c5be5
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java
2023-01-02 18:11:03 +01:00
Stypox
259de3cba6
Merge pull request #995 from TeamNewPipe/feat/soundcloud-playlistinfoitemextractor
...
[SoundCloud] Implement getUploaderUrl() and isUploaderVerified() for PlaylistInfoItemExtractor
2023-01-02 15:10:40 +01:00
Stypox
991394b53a
Merge pull request #1005 from FireMasterK/fix-escaping-xss
...
Fix for potential XSS attacks and formatting issues
2023-01-02 15:06:17 +01:00
Stypox
ce15f7cc50
Merge pull request #996 from TeamNewPipe/feat/peeertube-playlists
...
[PeerTube] Support searching for playlists and channels
2023-01-02 15:04:09 +01:00
Isira Seneviratne
d8ce08d969
Use immutable Map factory methods.
2023-01-02 07:50:31 +05:30
Kavin
01acf79436
Fix for potential XSS attacks.
2022-12-31 20:05:32 +00:00
TobiGr
292e0d8ce7
[SoundCloud] Implement getUploaderUrl() and isUploaderVerified() for PlaylistInfoItemExtractor
2022-12-31 18:46:39 +01:00
TobiGr
2a8729aeb2
Apply suggestions
...
Co-authored-by: Stypox <stypox@pm.me>
2022-12-31 18:24:33 +01:00
TobiGr
d75a997611
[PeerTube] Support searching for channels
2022-12-31 18:24:33 +01:00
TobiGr
dea6d8ce4c
[PeerTube] Support searching for playlists
2022-12-31 18:24:33 +01:00
Stypox
95cc6aefbb
Merge pull request #994 from TeamNewPipe/fix/peertube-subtitles-exception
...
[PeerTube] Report Exceptions thrown while getting a stream's subtitles
2022-12-31 15:01:39 +01:00
Stypox
7b54457789
Merge pull request #941 from TeamNewPipe/feat/peertube-comment-replies
...
[PeerTube] Support comment replies
2022-12-31 14:57:51 +01:00
AudricV
f45966d449
Merge pull request #910 from Isira-Seneviratne/Locale_forLanguageTag
...
Add compat Locale.forLanguageTag() implementation.
2022-12-24 23:53:30 +01:00
AudricV
d5437e0bc5
Merge pull request #863 from AudricV/add-content-type-and-content-length-headers-to-post-requests
...
Add Content-Type header to all POST requests without an empty body
2022-12-16 19:32:56 +01:00
Tobi
88e07e555d
Merge pull request #1000 from AudricV/yt-streaminfoitemextractor-improvements
...
[YouTube] Improve YoutubeStreamInfoItemExtractor
2022-12-11 17:02:29 +01:00
AudricV
0766b1d211
[YouTube] Improve YoutubeStreamInfoItemExtractor
...
- Return duration of video premieres;
- Add another non-localized method to determine whether a stream is a running
livestream;
- Return view count and upload date of videos in playlists;
- Store isPremiere result;
- Remove shorts workaround code, as it was only useful on channels and shorts
have been moved into a separated channel tab;
- Improve some other code.
2022-12-08 13:59:12 +01:00
Tobi
896d7e09eb
Merge pull request #978 from Theta-Dev/fix/search-channel-handles
...
[YouTube] Fix search subscriber count extraction with channel handles
2022-12-05 17:52:05 +01:00
TobiGr
cd3262745d
[PeerTube] Report Exceptions thrown while getting a stream's subtitles
2022-12-03 16:11:21 +01:00
TobiGr
4e66b2287e
[PeerTube] Add support for comment replies
2022-12-01 14:05:18 +01:00
Tobi
41c8dce452
Merge pull request #992 from Isira-Seneviratne/String_isBlank
...
Use String.isBlank().
2022-11-30 17:48:54 +01:00
Isira Seneviratne
2bca56f0df
Use String.isBlank().
2022-11-30 08:26:21 +05:30
Isira Seneviratne
3b80547976
Add code review suggestions.
2022-11-30 07:57:45 +05:30
Tobi
a1128ecbdc
Merge pull request #991 from FireMasterK/fix-compile
...
Fix complication error in comment test
2022-11-29 19:20:01 +01:00
ThetaDev
016623131e
docs: update comment in YoutubeChannelInfoItemExtractor
2022-11-29 19:06:03 +01:00
Kavin
2e08eaad96
Fix complication error in comment test.
2022-11-29 16:07:48 +00:00
Kavin
abf08e1496
Merge pull request #990 from FireMasterK/bold-italic-strikethrough
...
[YouTube] Implement bold/italic/strike-through support
2022-11-29 15:59:38 +00:00
Tobi
72d5ed3318
Merge pull request #987 from FireMasterK/comments-text-description
...
Use Description object for comments text.
2022-11-29 16:54:56 +01:00
Kavin
57e7a6fb7c
Add mocks test.
2022-11-28 20:27:55 +00:00
Kavin
1d3d7fa5c3
Add test for formatting.
2022-11-28 20:26:37 +00:00
Kavin
52fda37915
Implement bold/italic/strike-through support.
2022-11-28 19:06:18 +00:00
Kavin
b566084cac
Use Description object for comments text.
2022-11-28 17:02:19 +00:00
Tobi
40f1ec4a54
Merge pull request #989 from Stypox/forbid-wrong-nullable-notnull-imports
...
Block wrong nullable/nonnull imports in checkstyle
2022-11-28 14:57:46 +01:00
Stypox
5f6101698b
Block wrong nullable/nonnull imports in checkstyle
2022-11-28 14:42:18 +01:00
Tobi
f8162b049d
Merge pull request #984 from FireMasterK/unused-dep
...
Remove unused autolink dependency
2022-11-28 11:28:42 +01:00
Tobi
ad7c3c2374
Merge pull request #985 from FireMasterK/checkstyle-update
...
Update checkstyle dependency.
2022-11-28 11:22:40 +01:00
Tobi
1da0190056
Merge pull request #980 from TeamNewPipe/fix/yt/unavailable
...
[YouTube] Fix extracting the detailed error message for unavailable streams
2022-11-28 10:07:34 +01:00
Stypox
60fb30f835
Merge pull request #928 from FireMasterK/comment-urls
...
Parse YouTube comments as HTML
2022-11-27 19:16:34 +01:00
Kavin
5abea22225
Fix throwing correct reason.
2022-11-26 21:09:08 +00:00
Kavin
7874d3c8aa
Update checkstyle dependency.
...
Since we now use JDK 11
2022-11-26 20:22:25 +00:00
Kavin
faf28f5c11
Remove unused dependency.
2022-11-26 20:17:25 +00:00
Kavin
a3ed947b2d
Merge pull request #983 from FireMasterK/update-countries
...
Update supported countries list.
2022-11-26 20:06:00 +00:00
Kavin
c043597255
Update supported countries list.
2022-11-26 19:01:33 +00:00
TobiGr
4680df0bdf
Fix throwing correct reason
2022-11-23 17:03:22 +01:00
TobiGr
9de8405c9f
[YouTube] Fix extracting the detailed error message of streams which are unavailable
2022-11-23 08:33:06 +01:00
Stypox
34d79bd267
[YouTube] Update mocks
2022-11-22 17:10:04 +01:00
AudricV
2ec296e674
Fix YoutubeSearchExtractorTest.MetaInfoTest
...
Not all the "learn more" button is uppercase anymore, that's only the case for
the first letter.
2022-11-22 16:34:54 +01:00
AudricV
3891542ca1
Use Downloader's postWithContentType and postWithContentTypeJson methods in services and extractors
2022-11-22 11:37:18 +01:00
AudricV
b2862f3cd1
Add postWithContentType and postWithContentTypeJson utility methods in Downloader
...
Co-authored-by: Stypox <stypox@pm.me>
2022-11-22 11:37:17 +01:00
AudricV
e9a0d3bd95
[YouTube] Send Content-Type header in all POST requests
...
This header was not sent partially before and was added and guessed by OkHttp. This can create issues when using other HTTP clients than OkHttp, such as Cronet.
Some code in the modified classes has been improved and / or deduplicated, and usages of the UTF_8 constant of the Utils class has been replaced by StandardCharsets.UTF_8 where possible.
Note that this header has been not added in except in YoutubeDashManifestCreatorsUtils, as an empty body is sent in the POST requests made by this class.
2022-11-22 11:37:16 +01:00
AudricV
b9e463de49
[Bandcamp] Send Content-Type header in POST requests
...
This header was not sent before and was added and guessed by OkHttp. This can create issues when using other HTTP clients than OkHttp, such as Cronet.
Also make use of StandardCharsets.UTF_8 when getting bytes of bodies instead of the platform default's charset, to make sure to prevent some encoding issues on some JVMs.
2022-11-22 11:35:46 +01:00
AudricV
65d6321e3d
Fix typos in Downloader.post JavaDocs
...
Post methods in Downloader return the result of a POST request and not the one of a GET request.
2022-11-22 11:35:46 +01:00
ThetaDev
5daabd1793
fix : #976 search subscriber count extraction with channel handles
2022-11-22 02:17:10 +01:00
Kavin
c953e23414
Merge pull request #968 from AudricV/yt-support-no-video-info-renderers-for-streams
...
[YouTube] Support lack of video info renderers for streams
2022-11-16 20:20:01 +00:00
Tobi
2211a24b69
Merge pull request #971 from lrusso96/patch-1
...
[YouTube] Improve duration parsing
2022-11-16 16:14:54 +01:00
Tobi
f6eefdc76c
Merge pull request #975 from FireMasterK/jitpack
...
Use JDK 11 for Jitpack gradle builds
2022-11-15 20:23:21 +01:00
Kavin
a1f22be0dd
Use JDK 11 for Jitpack gradle builds.
2022-11-15 18:43:19 +00:00
Kavin
9bdad55508
Merge pull request #973 from FireMasterK/audio-track
...
Add support for extracting audio track information
2022-11-15 08:19:47 +00:00
Kavin
86f06b333a
Address review.
2022-11-14 00:05:31 +00:00
Kavin
b16e6082e1
Add test for audio stream languages.
2022-11-13 23:10:44 +00:00
Kavin
30909da1df
Fix audio track similar comparison for IDs.
2022-11-13 23:08:54 +00:00
Kavin
6d59cdbe3a
Add support for extracting audio tracks.
2022-11-13 21:39:29 +00:00
Tobi
43d1c1f8b1
Merge pull request #880 from Isira-Seneviratne/Use_StandardCharsets
...
Use StandardCharsets.UTF_8.
2022-11-12 03:23:44 +01:00
Isira Seneviratne
e4d982c7ea
Fix license.
2022-11-12 07:29:15 +05:30
Isira Seneviratne
416089146e
Fix failing tests.
2022-11-12 07:29:15 +05:30
Isira Seneviratne
ff5f223d3f
Update README.md
2022-11-12 07:29:15 +05:30
Isira Seneviratne
ddbce3b83d
Add Utils methods for URL encoding/decoding using UTF-8.
2022-11-12 07:29:15 +05:30
Isira Seneviratne
366f5c1632
Use StandardCharsets.UTF_8.
2022-11-12 07:29:15 +05:30
Tobi
0ea16c0b73
Merge pull request #954 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.10
...
Bump gson from 2.9.1 to 2.10
2022-11-09 17:28:31 +01:00
dependabot[bot]
0b82fade51
Bump gson from 2.9.1 to 2.10
...
Bumps [gson](https://github.com/google/gson ) from 2.9.1 to 2.10.
- [Release notes](https://github.com/google/gson/releases )
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md )
- [Commits](https://github.com/google/gson/compare/gson-parent-2.9.1...gson-parent-2.10 )
---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-11-09 16:21:16 +00:00
Luigi Russo
c9635218e2
[YouTube] Improve duration parsing
2022-11-09 09:41:29 +01:00
Isira Seneviratne
1be270768f
Merge pull request #893 from Isira-Seneviratne/Static_sets
...
Use immutable sets in YoutubeParsingHelper.
2022-11-09 09:07:01 +05:30
Isira Seneviratne
316d8573fa
Use immutable sets in YoutubeParsingHelper.
2022-11-07 07:50:26 +05:30
AudricV
6a2c680d8f
[YouTube] Add mocks for YoutubeStreamExtractorDefaultTest.NoVisualMetadataVideoTest
2022-11-04 19:43:06 +01:00
AudricV
e66fed41d6
[YouTube] Add a StreamExtractor test for a video without visual metadata
...
The video "Makani’s first commercial-scale energy kite" (video ID:
An8vtD1FDqs), which has this behavior, is used for the new test,
NoVisualMetadataVideoTest, added in YoutubeStreamExtractorDefaultTest.
Tests of elements who throw an exception in this case (subscriber count, like
count, uploader avatar URL) test if the ParsingException exception is thrown by
YoutubeStreamExtractor.
2022-11-04 19:42:12 +01:00
AudricV
aa9a8ca23c
[YouTube] Make non-extraction of videoPrimaryInfoRenderer and/or videoSecondaryInfoRenderer not fatal
...
Also de-duplicated common code related to the obtain of these video info renderers.
This change allows extraction of videos without visual metadata.
2022-11-04 18:35:53 +01:00
AudricV
eb07d70a2c
Merge pull request #964 from AudricV/yt-support-handles-and-all-channel-usernames
...
[YouTube] Support handles and all channel usernames
2022-11-04 12:09:15 +01:00
AudricV
20cd8e8a4a
[YouTube] Update mocks of YoutubeChannelExtractorTest.Gronkh
2022-11-03 19:46:42 +01:00
AudricV
a34f060ba0
[YouTube] Use a handle for YoutubeChannelExtractorTest.Gronkh
2022-11-03 19:46:42 +01:00
AudricV
724f669ff7
[YouTube] Add tests for handles and user IDs with non ASCII characters support
...
Unneeded public modifiers in test methods of
YoutubeChannelLinkHandlerFactoryTest have been also removed.
2022-11-03 19:46:42 +01:00
AudricV
61ce041bda
[YouTube] Support handles and all custom channel names
...
More non-channel paths have been also added to the excluded custom name paths,
documentation and exception messages have been improved and fixed in some
places, and the licence header of YoutubeChannelLinkHandlerFactory has been
moved to its beginning and updated.
2022-11-03 19:46:42 +01:00
AudricV
ffffb04439
Merge pull request #953 from Theta-Dev/attributed-text-desc
...
[YouTube] Add support for attributed text description
2022-11-03 18:34:30 +01:00
ThetaDev
592e1d6386
fix: parsing attributed description with no command runs
2022-11-03 12:10:52 +01:00
AudricV
31bf7046b7
Merge pull request #944 from Theta-Dev/fix-rich-grid-renderer
...
[YouTube] Support richGridRenderer on channel page to fix extraction of videos in channels
2022-11-03 00:10:54 +01:00
ThetaDev
099b53cc4f
[YouTube] Add parser for attributedDescription
...
Also update the mock of the next InnerTube endpoint response of the
YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing test class with an
attributedDescription instead of a regular description
2022-11-02 23:11:33 +01:00
AudricV
e4c24d4c36
[YouTube] Regenerate supported channels mocks
2022-11-02 19:13:59 +01:00
Theta-Dev
20e4a35814
[YouTube] Support richGridRenderer on channel pages
...
YouTube is deploying a new layout on their channel pages, which uses richGridRenderer JSON objects.
2022-11-02 19:01:29 +01:00
AudricV
4cae66f1f9
Merge pull request #946 from chowder/dev
...
Add ability to identify short-form `StreamInfoItem`s
2022-11-01 12:19:58 +01:00
Tobi
eb40bb8458
Merge pull request #959 from FireMasterK/playlist-info-item-uploader
...
Add uploaderUrl and uploaderVerified to PlaylistInfoItem.
2022-10-31 13:10:33 +01:00
chowder
b1a899fd47
Fix null pointer exception
2022-10-31 11:12:23 +00:00
Stypox
a4db106a66
Merge pull request #960 from AudricV/yt-workaround-403-errors-android-client
...
[YouTube] Workaround 403 HTTP errors of ANDROID client streams
2022-10-30 21:53:26 +01:00
Kavin
b441910257
Mark uploaderUrl as nullable.
2022-10-30 13:36:04 +00:00
chowder
3fdc0e72cc
Line breaks for long docstrings
2022-10-30 13:28:39 +00:00
Kavin
6a256d0631
Add uploader url and verified to PlaylistInfoItem.
2022-10-30 13:00:19 +00:00
Tobi
430504b4b5
Merge pull request #958 from AudricV/yt-playlists-support-new-metadata-format
...
[YouTube] Support new metadata format of playlists
2022-10-30 12:31:43 +01:00
Kavin
f9bd08c649
Address reviews.
2022-10-30 01:25:30 +00:00
Caleb
9282c3c13b
Fix exception message for YoutubeStreamInfoItemExtractor#isShortFormContent
...
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
2022-10-30 01:23:15 +00:00
Caleb
c5216f7c12
Update docstring for StreamExtractor#isShortFormContent
...
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
2022-10-30 01:23:15 +00:00
Caleb
04795fe5d2
Use Stream API for ShortFormContent#testShortFormContent
...
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
2022-10-30 01:23:15 +00:00
chowder
975b38a0b9
Regenerate test mocks
2022-10-30 01:23:15 +00:00
chowder
4cccd33f3d
Implement isShortFormContent for StreamExtractor and StreamInfo
2022-10-30 01:23:15 +00:00
chowder
09544ceb23
Fix tests
2022-10-30 01:23:15 +00:00
chowder
644cc6cd76
Rename test function name
2022-10-30 01:23:15 +00:00
chowder
daf5674951
Add ability to identify short-form StreamInfoItems
2022-10-30 01:23:12 +00:00
Tobi
3d314169b9
Merge pull request #943 from TeamNewPipe/fix/sc/comments
...
[SoundCloud] Fix getting more comments
2022-10-29 22:19:50 +02:00
AudricV
65125a3eb4
[YouTube] Fix YoutubePlaylistExtractorTest.LearningPlaylist
...
The video count is now returned for this playlist, so it isn't unknown.
The testStreamCount method of this test class asserts now that the stream count
is greater than 40.
2022-10-29 18:12:10 +02:00
AudricV
7258a53225
[YouTube] Support new playlist layout
...
This new layout doesn't provide author thumbnails and is completely different
for metadata, so the code to get them has been refactored.
The code of learning playlists video count check has been also removed, as it
seems to be not relevant anymore (the video count seems to be returned for
these playlists with both layouts).
Finally, unneeded overrides of subchannel methods, which don't apply to the
YouTube service, have been removed.
2022-10-29 18:12:10 +02:00
AudricV
e37e8b358e
[YouTube] Update mocks that need to be updated
2022-10-29 18:10:18 +02:00
AudricV
60e97cd274
[YouTube] Workaround getting streaming URLs returning 403 HTTP response codes
...
Using the player parameters used to get stories seems to fix the issue, which
affects currently only certain countries such as UK.
This is a workaround and should be fixed in a better way (by changing the
InnerTube additional client used for videos or finding what is now required in
Android player requests).
2022-10-29 17:58:33 +02:00
AudricV
4bc90cd9d8
Merge pull request #957 from AudricV/fix-checkstyle-error
...
Fix Checkstyle error in YoutubeCommentsInfoItemExtractor
2022-10-29 17:55:33 +02:00
AudricV
c230d84df1
Fix Checkstyle error in YoutubeCommentsInfoItemExtractor
2022-10-29 13:24:19 +02:00
Tobi
9ffdd0948b
Merge pull request #945 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.3
...
Bump spotbugs-annotations from 4.7.2 to 4.7.3
2022-10-18 22:55:24 +02:00
dependabot[bot]
24137bb15f
Bump spotbugs-annotations from 4.7.2 to 4.7.3
...
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.7.2 to 4.7.3.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.2...4.7.3 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-10-17 09:20:51 +00:00
xz-dev
0ffcb32d9c
[YouTube] Add comment reply count support ( #936 )
...
Add comment reply count support for YouTube and introduce `CommentsInfoItem.UNKNOWN_REPLY_COUNT` constant
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2022-10-15 12:40:06 +02:00
Isira Seneviratne
b90a566dd8
Add backport implementation of Locale.forLanguageTag().
2022-10-12 09:21:39 +05:30
Isira Seneviratne
b232c29d22
Use Locale.forLanguageTag().
2022-10-12 09:21:38 +05:30
TobiGr
4d136599bd
[SoundCloud] Fix getting more comments
2022-10-11 15:44:54 +02:00
Tobi
a822e91909
Merge pull request #939 from TurtleArmyMc/fix/SoundcloudPlaylistExtractor_track_order
...
Fix SoundcloudPlaylistExtractor: tracks are in correct order
2022-10-10 22:28:18 +02:00
TobiGr
02810a7db7
Add a comment
2022-10-10 22:22:12 +02:00
Tobi
2d50369c77
Merge pull request #888 from Isira-Seneviratne/EnumMap
...
Use EnumMap in PatternsHolder.
2022-10-10 12:00:06 +02:00
Tobi
54092fc3c7
Merge pull request #930 from Isira-Seneviratne/Avoid_Comparator_NPE
...
Avoid possible NullPointerException in MediaCCCRecentKiosk.
2022-10-10 11:43:34 +02:00
Tobi
d7c23ef118
Merge pull request #924 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.2
...
Bump spotbugs-annotations from 4.7.1 to 4.7.2
2022-10-10 11:35:40 +02:00
dependabot[bot]
08339d6b0c
Bump spotbugs-annotations from 4.7.1 to 4.7.2
...
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.7.1 to 4.7.2.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.1...4.7.2 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-10-10 09:32:17 +00:00
Tobi
5537e84100
Merge pull request #931 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.1
...
Bump junit-bom from 5.9.0 to 5.9.1
2022-10-10 11:30:53 +02:00
dependabot[bot]
719664593f
Bump junit-bom from 5.9.0 to 5.9.1
...
Bumps [junit-bom](https://github.com/junit-team/junit5 ) from 5.9.0 to 5.9.1.
- [Release notes](https://github.com/junit-team/junit5/releases )
- [Commits](https://github.com/junit-team/junit5/compare/r5.9.0...r5.9.1 )
---
updated-dependencies:
- dependency-name: org.junit:junit-bom
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-10-10 09:25:42 +00:00
TurtleArmyMc
bf70d32eb4
Fix SoundcloudPlaylistExtractor: tracks are in correct order
2022-09-30 15:26:25 -04:00
Stypox
5c710da160
Merge pull request #934 from AudricV/fix-yt-throttling-decryption-function-regex
...
[YouTube] Fix throttling decryption function regex
2022-09-25 13:26:08 +02:00
AudricV
8067c43837
[YouTube] Don't use a specific letter for the decryption function name pattern
...
Use the same possible characters for variables everywhere, in order to avoid
potential future throttling parameter decryption function name parsing issues
related to the usage of other letter(s) than b.
2022-09-24 21:49:22 +02:00
AudricV
abcee87167
[YouTube] Fix throttling parameter decryption function regex
...
- Quote the function name, as it may contain special regex symbols, such as
dollar;
- Support multiple lines;
- Use what looks like the end of the function for the end of the regex (this
part is inspired from yt-dlp throttling parameter decryption regex);
- Move the throttling function body regex into a private and static constant.
2022-09-24 21:28:09 +02:00
ThetaDev
7244be7627
[YouTube] Add JavaScript lexer to parse completely throttling decryption function ( #905 )
2022-09-24 19:55:46 +02:00
Isira Seneviratne
0e31c86aee
Avoid possible NullPointerException in MediaCCCRecentKiosk.
2022-09-21 05:40:07 +05:30
Stypox
a99af9bb6e
Merge pull request #927 from Stypox/fix-feed-thumbnail
...
[YouTube] Return mqdefault thumbnails in fast feed
2022-09-19 08:52:01 +02:00
Kavin
4e14644c41
Ensure comments are parsed with URLs and timestamps.
2022-09-17 16:03:39 +05:30
Stypox
14ef430546
Merge pull request #920 from AudricV/fix-yt-trending-extraction
...
[YouTube] Fix extraction of the Trending kiosk with the new data model returned
2022-09-14 11:14:13 +02:00
Stypox
93ef73e2f2
[YouTube] Return mqdefault thumbnails in feed
...
The hqdefault thumbnails, which are used by default, have black bars at the top and at the bottom, and are thus not optimal.
2022-09-13 16:46:54 +02:00
AudricV
d8ddeb549c
[YouTube] Support new trending structure and filter recently trending section
...
This new structure allow us to filter easily Trending shorts and Recently
trending sections.
On the previous one, this Recently trending section is now filtered, by
checking whether sections have a title, which isn't the case for normal trends
contrary to the other ones.
This makes that the extractor returns now only the real 50 "Now" YouTube
trends.
Elements inside arrays are now extracted dynamically instead of only the ones
of the first index, using Java 8's Stream API.
The getInitialPage() method of YoutubeTrendingExtractor can now throw a
ParsingException if no selected tab (corresponding to the one of the trends
type extracted) has been found.
Finally, the licence header has been moved to the top of the file and updated.
2022-09-11 18:52:12 +02:00
AudricV
884da40f65
Merge pull request #926 from AudricV/fix-yt-like-count-extraction
...
[YouTube] Fix extraction of like count with the new data model
2022-09-11 12:21:38 +02:00
AudricV
58b09cf78b
Merge pull request #925 from Theta-Dev/fix-td-regex
...
[YouTube] Fix throttling parameter decryption on Android
2022-09-10 23:30:50 +02:00
AudricV
119b9129e2
[YouTube] Fix extraction of like count with the new data model
...
In the new data model currently A/B tested or deployed, the like count button
data is the same, but its path has been changed.
The extraction of the like count has been also improved, by using the multiple
accessibility data available instead of the only one which was used before.
This accessibility data has been also deprioritized, because it is language
dependent when there is no view.
Like count is always known, even for age-restricted videos, so returning -1 if
the like count extraction fails on this type of videos is not needed and hides
an extraction error: that's the reason why this return has been removed and the
exception will always be thrown, even if a video is age-restricted.
2022-09-10 23:28:38 +02:00
ThetaDev
4905f74700
[YouTube] Fix throttling parameter decryption on Android
...
Escape the curly brace in the regular expression used to parse the throttling
parameter decryption function to allow its compatibility on Android.
2022-09-10 17:03:39 +02:00
Isira Seneviratne
e101d58ae7
Use EnumMap in PatternsHolder.
2022-08-26 08:15:21 +05:30
Stypox
6a858368c8
Merge pull request #912 from Stypox/downgrade-rhino
...
Downgrade Rhino since it uses a class not available on Android
2022-08-25 10:10:27 +02:00
Stypox
fa25a7280f
Downgrade Rhino since it uses a class not available on Android
...
Reverts #774 , see TeamNewPipe/NewPipe#8876
2022-08-25 09:49:13 +02:00
Stypox
25f0ce7305
Merge pull request #911 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.3
...
Bump jsoup from 1.15.2 to 1.15.3
2022-08-24 14:23:44 +02:00
Stypox
d8e10dd3ad
Merge pull request #907 from Isira-Seneviratne/Remove_EmptyString
...
Remove EMPTY_STRING.
2022-08-24 14:22:55 +02:00
dependabot[bot]
1d7a3a90a4
Bump jsoup from 1.15.2 to 1.15.3
...
Bumps [jsoup](https://github.com/jhy/jsoup ) from 1.15.2 to 1.15.3.
- [Release notes](https://github.com/jhy/jsoup/releases )
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES )
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.2...jsoup-1.15.3 )
---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-08-24 09:14:26 +00:00
Isira Seneviratne
41254ae12a
Remove annotation.
2022-08-24 06:59:17 +05:30
Isira Seneviratne
943b7c033b
Remove EMPTY_STRING.
2022-08-24 06:59:17 +05:30
litetex
20914a6035
Merge pull request #903 from TacoTheDank/updateGradle
...
Update gradle wrapper to 7.5.1
2022-08-23 16:21:56 +02:00
litetex
d6577e5e0b
Merge pull request #882 from litetex/fix-all-tests
...
Fix all tests
2022-08-23 16:19:27 +02:00
litetex
0a5ad90724
Merge pull request #901 from AudricV/remove-deprecated-code-yt-throttling-decrypter
...
[YouTube] Remove deprecated throttling decryption code
2022-08-22 17:13:00 +02:00
AudricV
03d9a4fe9d
Fix typos in YoutubeStreamExtractor.tryDecryptUrl
2022-08-21 21:17:03 +02:00
litetex
0beb55a232
Improve ``YoutubeThrottlingDecrypterTest``
2022-08-21 20:17:05 +02:00
litetex
9e93d6b193
YoutubeThrottlingDecrypter: Check if returned string is a valid JavaScript function
2022-08-21 20:16:45 +02:00
litetex
ed0a07af4e
YoutubeThrottlingDecrypter: Patch regex
2022-08-21 20:15:53 +02:00
litetex
641a447d9b
Updated consent-related mock-data
2022-08-21 18:43:48 +02:00
litetex
8ff7a90f52
Improved consent cookie related constants and documentation
2022-08-21 18:41:40 +02:00
TacoTheDank
d8339d4060
Update gradle wrapper to 7.5.1
2022-08-18 15:09:35 -04:00
AudricV
cb64a480cd
[YouTube] Remove deprecated code in YoutubeThrottlingDecrypter
...
YoutubeThrottlingDecrypter is now a non-instantiable final class and non-static
attributes have been removed.
The static attributes of this class have been renamed, in order to respect the
naming specification used.
2022-08-16 15:01:05 +02:00
litetex
5bbea3a8f2
Fix ``YoutubeMixPlaylistExtractorTest`` again
...
Doesn't work for kurgesagt videos anymore, used music video (copyright free) instead
2022-08-14 15:12:27 +02:00
litetex
da06166065
Updated mock data of ``YoutubeSearchExtractorTest$Suggestion``
2022-08-14 15:12:26 +02:00
litetex
2e36ab1578
Fixed ``YoutubeSearchExtractorTest$Suggestion``
2022-08-14 15:12:26 +02:00
litetex
2a8a623643
``YoutubePlaylistLinkHandlerFactoryTest``: Use parameterized tests
2022-08-14 14:48:31 +02:00
litetex
938e69a16a
Fixed ``YoutubePlaylistLinkHandlerFactoryTest``
2022-08-14 14:48:30 +02:00
litetex
844de3e378
Fix ``YoutubeStreamLinkHandlerFactoryTest `` and parameterized the tests
...
used code from #833
2022-08-14 14:48:30 +02:00
litetex
c1a72b8240
Use Android Studios code style
2022-08-14 14:48:29 +02:00
litetex
fc27b8a5b8
Use preexisting ``ContentNotAvailableException``
2022-08-14 14:48:29 +02:00
Stypox
17bad6cedf
Fix test about exception type: NPE, not IllegalArgment
...
The relevant change was made in #877
2022-08-14 14:48:28 +02:00
litetex
ecfc370685
Fixed all YTMixPlaylists
...
Added option to choose if you want to consent or not - currently this is done by a static variable in ``YoutubeParsingHelper`` - may not be the best long-term solution but for now the tests work again (in EU countries) 🥳
2022-08-14 14:48:27 +02:00
litetex
2b6fe294b2
Fixed ``YoutubeMusicSearchExtractorTest``
...
The renderers and queries got changed.
2022-08-14 14:48:27 +02:00
litetex
d6586da614
Failing test works locally without any problems. Reformatted it a bit.
2022-08-14 14:48:27 +02:00
litetex
bf3ae5e679
Fixed ``SoundcloudStreamLinkHandlerFactoryTest``
...
* Removed the unnecessary ``public``
* Use parameterized tests
* One song got removed (which caused the test failure), replaced it with another song
2022-08-14 14:48:27 +02:00
litetex
504f81036e
Fixed ``PeertubePlaylistExtractorTest``
...
Also removed the unnecessary ``public`` and reformatted the code a bit
2022-08-14 14:48:26 +02:00
litetex
e7c12258f4
Fixed ``BandcampSearchExtractorTest``
...
Also removed the unnecessary ``public``
2022-08-14 14:48:26 +02:00
litetex
9e8724df4d
Fixed ``YoutubeStreamExtractorLivestreamTest``
...
The "Lofi Girl"-stream got interrupted by a copyright strike and had to be restarted. Because of this a new id is now used.
2022-08-14 14:48:26 +02:00
Tobi
76aad92fa5
Merge pull request #890 from AudricV/yt-clients-update-improvements-and-extraction-fixes
...
[YouTube] Update clients version and improve extraction of API key and client version of WEB client
2022-08-12 23:32:41 +02:00
AudricV
472f5d9e9c
[YouTube] Update mocks
2022-08-12 19:20:32 +02:00
AudricV
7bdca33a87
[YouTube] Ensure that an additional player response is the correct one
...
If YouTube detect that requests come from a third party client, they may
replace the real player response by another one of a video saying that this
content is not available on this app and to watch it on the latest version of
YouTube. We can detect this by checking whether the video ID of the player
response returned is the same as the one requested by the extractor.
2022-08-12 19:20:31 +02:00
AudricV
c82317e318
[YouTube] Spoof more mobile clients
...
Additional parameters have been added to the player requests of ANDROID and IOS
clients:
- for both clients: osName and osVersion: their respective values are:
- for the ANDROID one: Android and 12;
- for the IOS one: iOS and 15.6.0.19G71.
- for the ANDROID client: androidTargetSdkVersion, with the Android SDK version
corresponding to the Android version used in the player requests of this
client. This parameter is now required with this client to be sure to get a
correct player response, otherwise, the one of a video saying that this
content is not available in this app and to watch it with the latest version
of YouTube can be returned instead;
- for the IOS client: deviceMake, with Apple as its value.
The iOS version sent in the IOS client player requests has been also updated to
the version 15.6 of the OS.
Finally, a comment about the requirement to use the signature timestamp from
the player JavaScript base file for HTML5 player requests on videos with
obfuscated URLs has been added and replaces a previous one which may be not
true.
2022-08-12 19:20:31 +02:00
AudricV
d0549a5a52
[YouTube] Update client versions and use a real version for the iOS client
...
The iOS version can be got easily in fact, by looking at the What's New section of the App Store' app page.
2022-08-12 19:20:31 +02:00
AudricV
d7e678aca2
[YouTube] Improve WEB client version and API key HTML extraction
...
Common code in WEB client version HTML extraction has been deduplicated, usage of the Java 8 Stream API has been made and initial data fallback has been used as a last resort.
This means that the client version extraction from regexes will be used before this fallback, as it doesn't contain the full client version.
This can be used as a way to fingerprint the extractor, even if it seems to be not the case.
2022-08-12 19:20:30 +02:00
AudricV
6a885ef5ab
Merge pull request #891 from Theta-Dev/fix-throttling-decrypter
...
[YouTube] Fix extraction of more complex nsig functions
2022-08-12 18:24:21 +02:00
AudricV
5b548340e8
[YouTube] Catch any exception in YoutubeThrottlingDecrypter.apply and improve docs
...
This will prevent any future extractor break due to decryption failure, like it was excepted to be the case before.
Some documentation about the throttling decryption has been also improved.
2022-08-12 17:49:36 +02:00
ThetaDev
52ded6e3d7
Handle curly braces inside strings in StringUtils.matchToClosingParenthesis
...
This is required to extract fully more complex YouTube nsig functions.
2022-08-12 16:32:00 +02:00
Stypox
d12003651b
Merge pull request #886 from Isira-Seneviratne/toArray_improvements
...
Make improvements to methods using toArray().
2022-08-06 22:34:23 +02:00
Isira Seneviratne
7daca10a06
Make improvements to methods using toArray().
2022-08-06 05:21:12 +05:30
Stypox
2906be22af
Merge pull request #881 from Isira-Seneviratne/String_join
...
Use String.join() and Collectors.joining().
2022-08-04 12:09:33 +02:00
Stypox
4ddb96a86f
Merge pull request #883 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.9.1
...
Bump gson from 2.9.0 to 2.9.1
2022-08-04 11:20:41 +02:00
Isira Seneviratne
64771c5712
Use String.join() and Collectors.joining().
2022-08-04 05:18:13 +05:30
Stypox
fc8b5ebbc6
Merge pull request #878 from Isira-Seneviratne/Use_Collections
...
Use Collections methods.
2022-08-03 22:50:41 +02:00
Stypox
4a4939d89c
Merge pull request #877 from Isira-Seneviratne/Use_Objects_requireNonNull
...
Use Objects.requireNonNull().
2022-08-03 22:45:35 +02:00
Stypox
c336bd58a5
Merge pull request #879 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.0
...
Bump junit-bom from 5.8.2 to 5.9.0
2022-08-03 21:06:20 +02:00
dependabot[bot]
325af31e5f
Bump gson from 2.9.0 to 2.9.1
...
Bumps [gson](https://github.com/google/gson ) from 2.9.0 to 2.9.1.
- [Release notes](https://github.com/google/gson/releases )
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md )
- [Commits](https://github.com/google/gson/compare/gson-parent-2.9.0...gson-parent-2.9.1 )
---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-08-01 09:26:14 +00:00
dependabot[bot]
d905636021
Bump junit-bom from 5.8.2 to 5.9.0
...
Bumps [junit-bom](https://github.com/junit-team/junit5 ) from 5.8.2 to 5.9.0.
- [Release notes](https://github.com/junit-team/junit5/releases )
- [Commits](https://github.com/junit-team/junit5/compare/r5.8.2...r5.9.0 )
---
updated-dependencies:
- dependency-name: org.junit:junit-bom
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-07-27 09:20:12 +00:00
Isira Seneviratne
1af6b8eedb
Use Collections.singletonList().
2022-07-27 07:35:57 +05:30
Isira Seneviratne
ff60e05c76
Use Collections.singletonMap().
2022-07-27 07:35:57 +05:30
Isira Seneviratne
682a4263e5
Use Objects.requireNonNull().
2022-07-27 06:55:26 +05:30
Stypox
8c5f014a6f
Merge pull request #869 from mhmdanas/add-workflow-permissions
...
Use minimum required permissions for GitHub workflows
2022-07-13 18:58:32 +02:00
Stypox
954a294e27
Merge pull request #870 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.2
...
Bump jsoup from 1.15.1 to 1.15.2
2022-07-13 17:48:06 +02:00
Mohammed Anas
f57049d2c0
Use temurin instead of adopt ( #868 )
...
See
https://blog.adoptopenjdk.net/2021/08/goodbye-adoptopenjdk-hello-adoptium/ .
2022-07-10 18:38:08 +03:00
Stypox
5bd7bf20cc
Merge pull request #867 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.1
...
Bump spotbugs-annotations from 4.7.0 to 4.7.1
2022-07-06 14:23:44 +02:00
Stypox
5ab74b3631
Merge pull request #857 from FireMasterK/video-title
...
Get original untranslated title for YouTube
2022-07-06 10:26:45 +02:00
dependabot[bot]
122365005a
Bump jsoup from 1.15.1 to 1.15.2
...
Bumps [jsoup](https://github.com/jhy/jsoup ) from 1.15.1 to 1.15.2.
- [Release notes](https://github.com/jhy/jsoup/releases )
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES )
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.1...jsoup-1.15.2 )
---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-07-04 09:38:16 +00:00
mhmdanas
414186cff5
Use minimum required permissions for GitHub workflows
...
This reduces the attack surface if the workflows are ever compromised.
2022-07-03 23:55:28 +03:00
dependabot[bot]
e9b4be3e3c
Bump spotbugs-annotations from 4.7.0 to 4.7.1
...
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.7.0 to 4.7.1.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.0...4.7.1 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-06-28 09:15:07 +00:00
Stypox
5219a705ba
Merge pull request #864 from AudricV/fetch-android-client-for-ended/post-livestreams
...
[YouTube] Fetch the ANDROID client for ended/post livestreams
2022-06-24 14:55:20 +02:00
AudricV
090debd83b
[YouTube] Fetch the ANDROID client for ended/post livestreams
...
The ANDROID client was only fetched for video contents, where it can be useful on ended/post livestreams, if the n parameter of the WEB client cannot be decrypted, to avoid throttling issues (because the WEB client was only used before for ended/post livestreams).
It also provides an exclusive 48kbps M4A audio format in the adaptiveFormats array of the JSON player response, like other mobile clients (which can be also extracted from the response of the DASH manifest URL returned into the WEB client player's response, but the DASH manifest is not used by the extractor).
A note about non-fatality of fetching or parsing issues of the ANDROID and IOS clients has been added.
2022-06-21 18:53:49 +02:00
litetex
a26bcc55c4
Merge pull request #845 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.0
...
Bump spotbugs-annotations from 4.6.0 to 4.7.0
2022-06-19 15:36:55 +02:00
dependabot[bot]
424eb1c559
Bump spotbugs-annotations from 4.6.0 to 4.7.0
...
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.6.0 to 4.7.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.6.0...4.7.0 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-06-19 13:29:18 +00:00
litetex
2712c3d788
Merge pull request #847 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.1
...
Bump jsoup from 1.14.3 to 1.15.1
2022-06-19 15:28:43 +02:00
dependabot[bot]
281d2b9f81
Bump jsoup from 1.14.3 to 1.15.1
...
Bumps [jsoup](https://github.com/jhy/jsoup ) from 1.14.3 to 1.15.1.
- [Release notes](https://github.com/jhy/jsoup/releases )
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES )
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.14.3...jsoup-1.15.1 )
---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-06-19 13:15:40 +00:00
litetex
f775155d25
Merge pull request #846 from litetex/remove-unused-methods
...
Remove unused methods
2022-06-19 15:12:15 +02:00
litetex
9d625dd75f
Merge pull request #860 from gliptak/patch-1
...
Correct PeertubePlaylistExtractorTest unit tests
2022-06-19 15:09:27 +02:00
Gábor Lipták
28c9340d69
Correct unit tests
2022-06-18 11:18:38 -04:00
Stypox
1b51eab664
Merge pull request #859 from AudricV/delivery-methods-fixes-and-improvements
...
Fix extraction of some properties in ItagItems for YouTube livestreams and post-live streams and remove completely SoundCloud HLS workaround
2022-06-17 15:46:13 +02:00
AudricV
301a795ed3
[SoundCloud] Remove completely workaround for HLS streams
...
SoundCloud is currently removing this workaround completely, so there is no need to keep it, because it impacts the loading time (a HLS playlist was downloaded and parsed).
2022-06-16 12:12:54 +02:00
AudricV
e960a417ec
[YouTube] Fix extraction of fps, audioSampleRate and audioChannels fields for ItagItems of live streams and post live streams
...
These values were only set before for video streams.
A fallback for the audio channels count has been added, in order to prevent exceptions when generating DASH manifests of audio streams: the fallback value is 2, because most audio streams on YouTube have 2 audio channels.
2022-06-16 12:12:54 +02:00
Stypox
c8a77da2ab
Merge pull request #810 from TiA4f8R/delivery-methods-v2
...
Support delivery methods other than progressive HTTP
2022-06-02 22:44:24 +02:00
Kavin
7635aeed2c
Get original untranslated title for YouTube.
2022-06-02 09:57:52 +01:00
TiA4f8R
287d1dfd63
[SoundCloud] Use the HLS delivery method for all streams and extract only a single stream URL from HLS manifest for MP3 streams
...
SoundCloud broke the workaround used to get a single file from HLS manifests for Opus manifests, but it still works for MP3 ones.
The code has been adapted to prevent an unneeded request (the one to the Opus HLS manifest) and the HLS delivery method is now used for SoundCloud MP3 and Opus streams, plus the progressive one (for tracks which have a progressive stream (MP3) and for the ones which doesn't have one, it is still used by trying to get a progressive stream, using the workaround).
Streams extraction has been also moved to Java 8 Stream's API and the relevant test has been also updated.
2022-05-29 19:08:18 +02:00
Stypox
b3c620f0d8
Apply code review and Streams rework
2022-05-28 12:00:58 +02:00
Stypox
d652e05874
[MediaCCC] Fix comments about containsSimilarStream
2022-05-28 12:00:58 +02:00
Stypox
044639c32b
Solve some review comments
2022-05-28 12:00:57 +02:00
litetex
c33d392958
Fixed typo XEE → XXE (Xml eXternal Entity attack)
...
See also:
https://en.wikipedia.org/wiki/XML_external_entity_attack
https://owasp.org/www-community/vulnerabilities/XML_External_Entity_(XXE)_Processing
2022-05-28 12:00:57 +02:00
TiA4f8R
fffbbee7f3
[YouTube] Add missing Nonnull annotations in getCache method of YouTube DASH manifest creators
2022-05-28 12:00:56 +02:00
TiA4f8R
f7b1515290
[YouTube] Refactor DASH manifests creation
...
Move DASH manifests creation into a new subpackage of the YouTube package, dashmanifestcreators.
This subpackage contains:
- CreationException, exception extending Java's RuntimeException, thrown by manifest creators when something goes wrong;
- YoutubeDashManifestCreatorsUtils, class which contains all common methods and constants of all or a part of the manifest creators;
- a manifest creator has been added per delivery type of YouTube streams:
- YoutubeProgressiveDashManifestCreator, for progressive streams;
- YoutubeOtfDashManifestCreator, for OTF streams;
- YoutubePostLiveStreamDvrDashManifestCreator, for post-live DVR streams (which use the live delivery method).
Every DASH manifest creator has a getCache() static method, which returns the ManifestCreatorCache instance used to cache results.
DeliveryType has been also extracted from the YouTube DASH manifest creators part of the extractor and moved to the YouTube package.
YoutubeDashManifestCreatorTest has been updated and renamed to YoutubeDashManifestCreatorsTest, and YoutubeDashManifestCreator has been removed.
Finally, several documentation and exception messages fixes and improvements have been made.
2022-05-28 12:00:56 +02:00
TiA4f8R
f17f7b9842
Apply requested changes in YoutubeParsingHelper
2022-05-28 12:00:55 +02:00
TiA4f8R
301b9fa024
Remove hashCode and equals methods overrides of Stream classes
2022-05-28 12:00:55 +02:00
TiA4f8R
2f3920c648
[YouTube] Return approxDurationMs value from YouTube's player response in ItagItems generated
...
This change allows to build DASH manifests using YoutubeDashManifestCreator with the real duration of streams and prevent potential cuts of the end of progressive streams, because the duration in YouTube's player response is in seconds and not milliseconds.
2022-05-28 12:00:54 +02:00
TiA4f8R
4158fc46a0
[Bandcamp] Fix regression of Opus radio streams extraction
...
When moving opus-lo into a constant, opus-lo was renamed to opus_lo and was only used if no MP3 stream was available (which was not the case before the changes in BandcampRadioStreamExtractor related to the addition of the support of all delivery methods), so these changes removed the ability to get Opus streams of Bandcamp radios.
This commit reverts this unwanted change.
2022-05-28 12:00:54 +02:00
TiA4f8R
54d323c2ae
Fix Checkstyle issue in YoutubeDashManifestCreator
2022-05-28 12:00:53 +02:00
Stypox
2321822844
Rename Stream's baseUrl to manifestUrl
2022-05-28 12:00:53 +02:00
Stypox
cfc13f4a6f
[YouTube] Reduce exception generation code and move several attributes of MPD documents into constants
...
Rename YoutubeDashManifestCreationException to CreationException.
Also use these constants in YoutubeDashManifestCreatorTest.
2022-05-28 12:00:53 +02:00
Stypox
00bbe5eb4d
[YouTube] Make exception messages smaller
2022-05-28 12:00:52 +02:00
Stypox
4da05afe11
[YouTube] Inline collectSegmentsData in YoutubeDashManifestCreator
2022-05-28 12:00:52 +02:00
Stypox
3708ab9ed5
[YouTube] Refactor YoutubeDashManifestCreator
...
- Remove all of the methods used to access caches and replace them with three caches getters
- Rename caches to shorter and more meaningful names
- Remove redundant @throws tags that just say "if this method fails to do what it should do", which is obvious
2022-05-28 12:00:51 +02:00
Stypox
5c83409039
[YouTube] Rewrite manifest test and rename long methods
2022-05-28 12:00:51 +02:00
Stypox
8226fd044f
[YouTube] Suppress Sonar security warning for XEE
2022-05-28 12:00:50 +02:00
Stypox
ba68b8c014
[YouTube] Secure DashManifestCreator against XEE attack
...
Also remove duplicate lines from javadoc comments, otherwise checkstyle complaints.
XEE prevention is inspired from https://github.com/ChuckerTeam/chucker/pull/201/files
For an explanation of XEE see https://rules.sonarsource.com/java/RSPEC-2755 , though the solution reported there causes crashes on Android
2022-05-28 12:00:50 +02:00
Stypox
159d05c91b
Remove unused DashMpdParser (but kept in git history)
2022-05-28 12:00:49 +02:00
Stypox
50272db946
Apply reviews: improve comments, remove FILE, remove Stream#equals(Stream)
2022-05-28 12:00:49 +02:00
TiA4f8R
07b045f20d
[YouTube] Support the iOS client in YoutubeDashManifestCreator and decrypt again the n parameter, if present, for all clients
...
This commits reverts a new behavior introduced in this branch, which only applied the decryption if needed on streams from the WEB client.
Also fix rebase issues and documentations style in YoutubeDashManifestCreator.
2022-05-28 12:00:48 +02:00
TiA4f8R
436ddde29f
Use assertThrows in YoutubeDashManifestCreatorTest
2022-05-28 12:00:48 +02:00
TiA4f8R
d64d7bbd01
Move ManifestCreatorCache tests to a separate class and remove override of equals and hashCode methods in ManifestCreatorCache
...
These methods don't need to be overriden, as they are not excepted to be used in collections.
Also improve the toString method of this class, which contains also now clearFactor and maximumSize attributes and for each operations.
2022-05-28 12:00:48 +02:00
TiA4f8R
2fb1a412a6
Fix Checkstyle issues, revert resolution string changes for YouTube video streams and don't return the rn parameter in DASH manifests
...
This parameter is still used to get the initialization sequence of OTF and POST-live streams, but is not returned anymore in the manifests.
It has been removed in order to avoid fingerprinting based on the number sent (e.g. when starting to play a stream close to the end and using 123 as the request number where it should be 1) and should be added dynamically by clients in their requests.
The relevant test has been also updated.
Checkstyle issues in YoutubeDashManifestCreator have been fixed, and the changes in the resolution string returned for video streams in YoutubeStreamExtractor have been reverted, as they create issues on NewPipe right now.
2022-05-28 12:00:47 +02:00
TiA4f8R
f61e2092a1
[YouTube] Return a copy of the hardcoded ItagItem instead of returning the reference to the hardcoded one in ItagItem.getItag
...
To do so, a copy constructor has been added in the class.
This fixes, for instance, an issue in NewPipe, in which the ItagItem values where not the ones corresponsing to a stream but to another, when generating DASH manifests.
2022-05-28 12:00:47 +02:00
TiA4f8R
aa4c10e751
Improve documentation and adress most of the requested changes
...
Also fix some issues in several places, in the code and the documentation.
2022-05-28 12:00:46 +02:00
TiA4f8R
6985167e63
Add tests for YoutubeDashManifestCreator and ManifestCreatorCache
...
The test added in YoutubeDashManifestCreator uses a video of the Creative Commons channel, licenced under the Creative Commons Attribution licence (reuse allowed).
Also remove public keywords of tests in UtilsTest, as suggested by SonarLint, because they are not needed with Junit 5.
2022-05-28 12:00:46 +02:00
TiA4f8R
f6ec7f9a61
Update DefaultStreamExtractorTest and SoundcloudStreamExtractorTest to support changes made in Stream classes
2022-05-28 12:00:45 +02:00
TiA4f8R
7477ed0f3d
[YouTube] Add ability to generate manifests of progressive, OTF and post live streams
...
A new class has been added to do so: YoutubeDashManifestCreator.
It relies on a new class: ManifestCreatorCache, to cache the content, which relies on a new pair class named Pair.
Results are cached and there is a cache per delivery type, on which cache limit, clear factor, clearing and resetting can be applied to each cache and to all caches.
Look at code changes for more details.
2022-05-28 12:00:45 +02:00
TiA4f8R
a857684442
Apply changes in YoutubeStreamExtractor
...
Extract post live DVR streams as post live streams instead of live streams.
A new class has been in order to improve code: ItagInfo, which stores an itag, the content (URL) extracted and if its an URL or not.
A functional interface has been added in order to abstract the stream building: StreamBuilderHelper.
Also add the cver parameter added by the desktop web client on the corresponding streams (a new method has been added in YoutubeParsingHelper to check this and another for Android streams).
Some code in these classes has been also refactored/improved/optimized.
2022-05-28 12:00:44 +02:00
TiA4f8R
4330b5f7be
Add POST_LIVE_STREAM and POST_LIVE_AUDIO_STREAM stream types
...
This allows the extractor to determine if a content is an ended audio or video livestream.
2022-05-28 12:00:43 +02:00
TiA4f8R
881969f1da
Apply changes in all StreamExtractors except YouTube's one and fix extraction of PeerTube audio streams as video streams
...
Some code in these classes has been also refactored/improved/optimized.
Also fix the extraction of PeerTube audio streams as video streams, which are now returned as audio streams.
2022-05-28 12:00:43 +02:00
TiA4f8R
d5f3637fc3
[YouTube] Return more values returned inside the ItagItems of the player response and deprecate use of public audio and video fields
...
These fields can be now replaced by a getter and a setter.
New fields have been added and will allow the creation of DASH manifests for OTF and ended livestreams. There are:
- contentLength;
- approxDurationMs;
- targetDurationSec;
- sampleRate;
- audioChannels.
2022-05-28 12:00:42 +02:00
TiA4f8R
7c67d46e09
Move DashMpdParser to the YouTube package and fix extraction of streams
...
DashMpdParser is only working with YouTube streams, as it uses the ItagItem class.
Also update creation of AudioStreams and VideoStreams objects.
2022-05-28 12:00:41 +02:00
TiA4f8R
ad993b920f
Remove fetching of the DASH manifest extracted when getting information of a content with StreamInfo
...
DashMpdParser is only working with YouTube streams, as it uses the ItagItem class.
Also improve code and comments of StreamInfo (especially final use where possible).
2022-05-28 12:00:41 +02:00
TiA4f8R
2f061b8dbd
Add support of other delivery methods than progressive HTTP in Stream classes
...
Stream constructors are now private and streams can be constructed with new Builder classes per stream class. This change has been made to prevent creating and using several constructors in stream classes.
Some default cases have been also added in these Builder classes, so not everything has to be set, depending of the service and the content.
2022-05-28 12:00:27 +02:00
Stypox
1dc80957d8
Merge pull request #848 from TiA4f8R/fix_yt_music_website_extraction_eu
...
[YouTube] Fix extracting YouTube Music client version and key in the EU
2022-05-22 22:05:05 +02:00
TiA4f8R
c34b5e3a8b
[YouTube] Fix extraction of YouTube Music client version and API key when using YouTube Music's website in EU
...
Google returns now the consent page of YouTube for YouTube Music in EU, which can be also avoided by adding the ucbcb parameter to the URL with the value 1 ("?ucbcb=1").
2022-05-15 11:20:06 +02:00
litetex
2015eb374a
Removed more unused methods
2022-05-09 21:05:03 +02:00
litetex
f69b0ff77b
Remove unused methods
2022-05-09 20:59:25 +02:00
Stypox
23fa31a1ec
Merge pull request #843 from TiA4f8R/fix-checkstyle-issues
...
Remove Checkstyle suppressions file and fix Checkstyle issues
2022-05-02 22:07:27 +02:00
TiA4f8R
3c3cd78676
Remove Checkstyle suppressions file and fix Checkstyle issues introduced in 24e8399
and 8c1041d
...
The Checkstyle suppressions file is now replaced by // CHECKSTYLE:OFF and // CHECKSTYLE:ON comments.
2022-05-02 21:51:25 +02:00
Stypox
2e1c5c119d
Merge pull request #822 from Stypox/more-refactors
...
More refactors
2022-05-02 19:03:54 +02:00
Stypox
598ebb92ea
Merge pull request #839 from TeamNewPipe/bandcamp/extract-length
...
Bandcamp: extract stream length
2022-05-02 15:49:41 +02:00
litetex
5db4d1faf3
Merge pull request #782 from litetex/cleanup-yt-stream-extractor
...
Cleanup of ``YoutubeStreamExtractor`` and some related classes
2022-05-01 16:44:11 +02:00
litetex
fe30eb43a9
Cleanup ``YoutubeStreamExtractor`` and some related classes
...
* Fixed obvious sonar(lint) warnings
* Abstracted some code (get*Streams)
* Used some new lines to make code better readable
* Chopped down brace-jungle in some methods
* Use StandardCharset (Java 8 4tw)
2022-05-01 16:39:07 +02:00
Stypox
c2b5370517
Apply suggestions: improve switch and use EMPTY_STRING
2022-04-30 16:39:51 +02:00
Stypox
7c78c39230
Merge pull request #821 from litetex/cleanup-TimeAgoParser-java
...
Cleanup ``TimeAgoParser``
2022-04-30 16:20:09 +02:00
Stypox
ac1c22d81c
Release NewPipe Extractor 0.22.1
2022-04-28 11:05:52 +02:00
Stypox
3f5c8962ac
Merge pull request #841 from TiA4f8R/yt-respect-order-of-clients-for-streams
...
[YouTube] Use correct order of clients to get better streams first
2022-04-28 10:58:49 +02:00
TiA4f8R
9f9af35adb
[YouTube] Fix regression introduced in the order of streams used when adding more parameters to InnerTube requests, using the iOS client for livestreams and more
2022-04-25 20:23:04 +02:00
Fynn Godau
c38c016de5
Bandcamp: extract stream length
2022-04-24 21:24:19 +02:00
litetex
5f8f3929bd
Merge pull request #838 from Nickoriginal/update_user_agent
...
Update User-Agent in NPE
2022-04-24 14:52:18 +02:00
Nickoriginal
b5f3f9eb90
Update USER_AGENT to match the latest Firefox
2022-04-24 00:27:23 +03:00
litetex
9f665db4e1
Merge pull request #828 from TeamNewPipe/dependabot/github_actions/actions/setup-java-3
...
Bump actions/setup-java from 2 to 3
2022-04-23 00:09:44 +02:00
Stypox
52fa2d939a
Fix javadoc formatting error causing deployment to fail
2022-04-16 17:07:07 +02:00
Stypox
95a4b05acc
Release NewPipe Extractor 0.22.0
2022-04-16 16:57:58 +02:00
Tobi
b77c72fb88
Merge pull request #832 from Stypox/hotfix-decrypter
...
Fix YouTube throttling decrypter function parsing
2022-04-15 13:35:10 +02:00
Stypox
dcb7483dcf
Fix YouTube throttling decrypter function parsing
2022-04-15 13:10:19 +02:00
Robin
bebdc55ad4
Merge pull request #829 from TiA4f8R/yt-subtitles-age-restricted-videos-fix
...
[YouTube] Extract subtitles for age-restricted videos
2022-04-14 10:31:16 +02:00
TiA4f8R
ef49cd0007
[YouTube] Extract subtitles for age-restricted videos
...
Subtitles of age-restricted videos can be extracted since the InnerTube API migration, so there is no reason to not extract them anymore.
2022-04-11 22:09:56 +02:00
dependabot[bot]
00b51a2bf8
Bump actions/setup-java from 2 to 3
...
Bumps [actions/setup-java](https://github.com/actions/setup-java ) from 2 to 3.
- [Release notes](https://github.com/actions/setup-java/releases )
- [Commits](https://github.com/actions/setup-java/compare/v2...v3 )
---
updated-dependencies:
- dependency-name: actions/setup-java
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-04-11 09:16:36 +00:00
TiA4f8R
2e92d718a2
Merge pull request #780 from TiA4f8R/yt-more-params-innertube-requests
...
[YouTube] Add more parameters to InnerTube requests, use the iOS client for livestreams and fix extraction of embeddable age-restricted videos and contents with a warning before playback
2022-04-09 18:15:18 +02:00
TiA4f8R
b30e341559
Add link to learn more about the issue which makes YoutubeStreamExtractorRelatedMixTest.testRelatedItems test disabled
2022-04-04 20:36:59 +02:00
TiA4f8R
70812fa611
Update YouTube stream mocks and disable YoutubeStreamExtractorRelatedMixTest
...
Mixes seems to be not given by YouTube anymore if you use a PENDING consent cookie value.
As mocks needs to updated, the test is always failing because of this change.
2022-04-02 19:28:49 +02:00
TiA4f8R
67288a0191
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
...
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 19:06:36 +02:00
TiA4f8R
11b5a222c4
Deduplicate code of getStringResultFromRegexArray methods in Utils
...
Also revert indentation in Utils.mixedNumberWordToLong.
2022-04-02 18:40:00 +02:00
TiA4f8R
9ca647a750
Update mocks
2022-03-27 22:10:59 +02:00
TiA4f8R
dfa4239661
Fix missing imports and Checkstyle issues
2022-03-27 22:10:57 +02:00
TiA4f8R
6d27996ac4
Improve code of getStringResultFromRegexArray methods in Utils
2022-03-27 22:10:57 +02:00
TiA4f8R
2e3da445e6
[YouTube] Add documentation about parameters added and clients versions and key
...
Also move the iPhone device machine id to a constant, explain how it is used and move the licence in the header of the file, and fix missing imports in YoutubeStreamExtractor (due to a rebase issue).
2022-03-27 22:10:57 +02:00
TiA4f8R
1dad3bfe8b
[YouTube] Update again hardcoded client versions and update mobile user agents
...
Also provide ability to get mobile user-agents used for mobile InnerTube requests and deduplicate related code.
2022-03-27 20:52:40 +02:00
TiA4f8R
3d38459cf3
[YouTube] Reduce InnerTube response sizes by adding the prettyPrint parameter with the false value
...
InnerTube responses return pretty printed responses, which increase responses' size for nothing.
By using the prettyPrint parameter on requests and setting its value to false, responses are not pretty printed anymore, which reduces responses size, and so data transfer and processing times.
This usage has been recently deployed by YouTube on their websites.
2022-03-27 20:52:40 +02:00
litetex
349ba8db7f
Improve tests and randomness
...
- Use the existing RNG inside YoutubeParsingHelper
- Deduplicated test-setup for YouTube tests
- Minor improvements
2022-03-27 20:52:38 +02:00
TiA4f8R
52376949e5
Add setSeedForVideoTests method in YoutubeStreamExtractor tests
...
In order to use still use mocks with the generation of random strings in player requests, we need to use YoutubeParsingHelper.setSeedForVideoTests() method in every stream test.
2022-03-27 20:51:39 +02:00
TiA4f8R
d0d91e6690
Adress requested changes
2022-03-27 20:51:39 +02:00
TiA4f8R
b6bc521f0d
[YouTube] Update client versions again
2022-03-27 20:51:38 +02:00
TiA4f8R
26f93f5bb0
[YouTube] Extract streams of livestreams from the iOS client and disabled the Android client for livestreams
...
The iOS client is only enabled for livestreams and the Android client is now only enabled for videos, both by default.
A way to force, or not, the fetch of both clients have been added with two new static methods in YoutubeStreamExtractor.
2022-03-27 20:51:38 +02:00
TiA4f8R
7d07924de8
[YouTube] Try to use lighter requests when extracting client version and key from YouTube and YouTube Music
...
This is done by fetching https://www.youtube.com/sw.js for YouTube and https://music.youtube.com/sw.js for YouTube Music.
Two new methods in Utils class have been added which allow to try to get a match of regular expressions in a string array, or a Pattern array, on a content, on a specific index or 0.
Also some code refactoring has been made in this class.
2022-03-27 20:51:38 +02:00
TiA4f8R
05b7fee23b
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
...
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
2022-03-27 20:51:38 +02:00
TiA4f8R
83f374bff1
[YouTube] Update client versions and fix a bug when using resetClientVersionAndKey method
...
The boolean keyAndVersionExtracted in YoutubeParsingHelper was not set to false when resetting the client version and the key, which makes the extractor uses null on the next getting of the client version or the key if the clientVersion and the key were extracted before.
Also update client versions.
2022-03-27 20:51:38 +02:00
Stypox
8c1041def6
Add @ null annotations where Android Studio suggested it
...
That is, basically where the overriding function was missing an annotation from the base method.
Also apply renaming of emptyDescription to EMPTY_DESCRIPTION
2022-03-26 22:07:14 +01:00
Stypox
adbbdc7a5b
[YouTube] Fix regex warning: use ' {2}' instead of ' '
2022-03-26 22:07:14 +01:00
Stypox
24e83997b4
[Bandcamp] Add Java 8 streams
2022-03-26 22:07:12 +01:00
Stypox
349990fd48
Fix redundant escape \\ in regex in Utils
2022-03-26 22:01:30 +01:00
litetex
3bf7aa3762
Cleanup ``TimeAgoParser``
2022-03-26 21:09:31 +01:00
litetex
5a18730845
Merge pull request #813 from litetex/fix-test-2022-03
...
Fixed tests (+ Youtube shorts in channels)
2022-03-26 21:09:10 +01:00
litetex
af82edf9dc
Fix checkstyle problems
2022-03-26 20:54:20 +01:00
litetex
29408f9356
``YoutubeMusicSearchExtractorTest$Suggestion`` has no mock data
2022-03-26 20:52:28 +01:00
litetex
fcee247f9f
Switched to mockonly tests
...
* Made the reason why the tests are mockonly mandatory
2022-03-26 20:52:28 +01:00
litetex
0fceb4686b
Added missing mock data
2022-03-26 20:52:27 +01:00
litetex
70b20f0d6b
Updated mock data after conflicts
2022-03-26 20:52:27 +01:00
litetex
6a6c9359af
Fixed kurzgesagt test - They changed their description...
2022-03-26 20:52:27 +01:00
litetex
26596215fa
Refactored ``MediaCCCRecentListExtractorTest``
2022-03-26 20:52:26 +01:00
litetex
53962bfd7d
Disabled suggestion test for now
...
Has to be done once YT's backed works again...
2022-03-26 20:52:26 +01:00
litetex
66dc5e8bb8
API hardening against changes
2022-03-26 20:52:26 +01:00
litetex
5d58156cde
Update mock data
2022-03-26 20:52:25 +01:00
litetex
7598b40957
Workaround for incorrect duration for "YT shorts" videos in channels
...
As a workaround 0 is returned as duration for such videos.
See also https://github.com/TeamNewPipe/NewPipe/issues/8034
2022-03-26 20:52:24 +01:00
litetex
164e21b5af
Fixed ``MediaCCCRecentKiosk``
...
Ignore faulty data/items (with duration <= 0)
2022-03-26 20:52:23 +01:00
litetex
49dfdae5d2
Removed useless throw declaration
2022-03-26 20:51:38 +01:00
litetex
69a58fd3d1
Don't use sysout
2022-03-26 20:51:37 +01:00
litetex
639be7adda
Minimized some code
2022-03-26 20:51:37 +01:00
litetex
2bd4299563
Fixed test: Peertube - Use new comments and videos
...
Old comments/videos got deleted
2022-03-26 20:49:39 +01:00
litetex
606a386a98
Fix tests: What's peertube has less comments now
2022-03-26 20:49:39 +01:00
litetex
4d1a1c8fb8
Better test for ``MediaCCCRecentListExtractorTest``
...
* Use assertAll
* Show which item is affected
2022-03-26 20:49:38 +01:00
litetex
ba43dbaa28
Fix Tests: The channel lost abos
2022-03-26 20:49:38 +01:00
litetex
ff436e5740
Fixed test: YT mix playlist
2022-03-26 20:49:38 +01:00
litetex
bb49f7d857
Merge pull request #817 from Stypox/checkstyle
...
Add checkstyle
2022-03-26 20:46:49 +01:00
litetex
9c07e8a664
Fix useage of wrong object
2022-03-26 20:17:50 +01:00
litetex
93d6e5c4d5
CI: Upload test reports when failure occurs
2022-03-26 19:57:17 +01:00
litetex
804e57004f
Fixed new checkstyle problems from dev
2022-03-26 19:46:10 +01:00
litetex
33347ac18b
Removed unused methods
...
``contentFilters`` and ``sortfilter`` are get inside the ``ListLinkHandler`` and not the ``ListLinkHandlerFactory``
``ListLinkHandlerFactory`` only passes these values through when ``fromQuery`` is called
2022-03-26 19:43:11 +01:00
litetex
ec5b54c38b
Removed unused class
2022-03-26 19:43:10 +01:00
litetex
8771af7ba5
Restored original naming
2022-03-26 19:43:09 +01:00
litetex
01cfde0285
Fixed typo
2022-03-26 19:43:09 +01:00
Stypox
bdadcfa1f7
Legitimately suppress remaining checkstyle warnings
2022-03-26 19:43:08 +01:00
Stypox
740a37a2de
[YouTube] Fix checkstyle issues
2022-03-26 19:42:40 +01:00
Stypox
9dc17cd1ca
[Soundcloud] Fix checkstyle issues
2022-03-26 19:40:20 +01:00
Stypox
9ab32cb2e7
[Peertube] Fix checkstyle issues
2022-03-26 19:40:19 +01:00
Stypox
9f7e06c817
[MediaCCC] Fix checkstyle issues
2022-03-26 19:40:18 +01:00
Stypox
3a94839359
[Bandcamp] Fix checkstyle issues
2022-03-26 19:40:17 +01:00
Stypox
08dff33002
Use Java 8 streams in NewPipe class
2022-03-26 19:40:15 +01:00
Stypox
c2446ecff0
Use Java 8 streams and deduplicate code in MediaFormat class
2022-03-26 19:40:15 +01:00
Stypox
d79e20340c
Fix checkstyle issues in root package extractor/
...
Note: not all issues were fixed because MediaFormat and ServiceList use a specific formatting that makes sense for them
2022-03-26 19:40:14 +01:00
Stypox
ca7c63f273
Fix remaining checkstyle issues in utils/ subpackage
2022-03-26 19:40:13 +01:00
Stypox
1d5f22e41f
Fix checkstyle issues & more in JsonUtils
...
Also use Java 8 streams and extract duplicate code to getInstanceOf function
2022-03-26 19:40:13 +01:00
Stypox
87d2834986
Fix checkstyle issues & more in DonationLinkHelper
...
Also add comment about the class being unused and replace the fixLink function with Utils.stringToUrl()
2022-03-26 19:40:12 +01:00
Stypox
bd7b362040
Fix checkstyle issues & more in DashMpdParser
...
Also remove useless null check on ItagItem.getItag() as that function already throws an exception if there is no itag
2022-03-26 19:40:11 +01:00
Stypox
8aba2b47b0
Fix checkstyle issues in subpackages with abstract classes
2022-03-26 19:40:10 +01:00
Stypox
e4951a0623
Refactor code handling http headers in downloader.Request
2022-03-26 19:37:47 +01:00
Stypox
37690058d2
Add checkstyle to extractor gradle project
...
With respect to NewPipe's checkstyle.xml, checkstyle is disabled for javadoc comments. There is no need for strict rules over comments here in the extractor, as sometimes javadocs are just needed to clarify a small thing and having empty/meaningless @param or @throws is useless.
2022-03-26 19:37:46 +01:00
litetex
9284569c84
Merge pull request #774 from TeamNewPipe/dependabot/gradle/org.mozilla-rhino-1.7.14
...
Bump rhino from 1.7.13 to 1.7.14
2022-03-26 17:25:55 +01:00
litetex
358e619a27
Merge pull request #819 from TeamNewPipe/dependabot/github_actions/actions/cache-3
...
Bump actions/cache from 2 to 3
2022-03-23 21:02:18 +01:00
dependabot[bot]
a2c9ad104c
Bump actions/cache from 2 to 3
...
Bumps [actions/cache](https://github.com/actions/cache ) from 2 to 3.
- [Release notes](https://github.com/actions/cache/releases )
- [Commits](https://github.com/actions/cache/compare/v2...v3 )
---
updated-dependencies:
- dependency-name: actions/cache
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-03-21 09:12:36 +00:00
XiangRongLin
aa6b7272a4
Merge pull request #804 from Stypox/fix-yt-music-mix
...
[YouTube] Fix music mixes in some countries
2022-03-20 08:35:56 +01:00
Stypox
843945f2a6
Merge pull request #818 from litetex/update-license
...
Update the license to the latest version
2022-03-19 23:13:55 +01:00
litetex
fa76098750
Delete copyright
...
was replace by LICENSE-file
2022-03-19 21:44:38 +01:00
litetex
faaf4cbec3
Update the license to the latest version of https://www.gnu.org/licenses/gpl-3.0.txt
2022-03-19 17:59:27 +01:00
XiangRongLin
7f2ea133f0
Merge pull request #788 from Stypox/mix
...
Add MixInfoItem and extract YouTube mixes in related items
2022-03-19 16:46:01 +01:00
Stypox
09ddb6adbb
[YouTube] Add MockOnly to method testing mixes in related items
2022-03-19 10:54:38 +01:00
Stypox
8201b3b90e
[YouTube] Parse any playlist (including music mixes) in related items
2022-03-19 10:48:13 +01:00
Stypox
13f7900816
[YouTube] Add test for genre mix
2022-03-19 10:48:13 +01:00
Stypox
279f3a20fe
[YouTube] Fix mix tests with invalid video ids
...
Replaces mix tests based on a strange mix type RDQM{videoId} (only reference I could find is https://github.com/ytdl-org/youtube-dl/issues/26228 ) and with an invalid video id of 13 characters (the first two characters were QM, but even after removing QM there still wasn't a video available at that id).
Also updates mocks.
2022-03-19 10:48:13 +01:00
Stypox
d660c04838
[YouTube] Also test playlist type in playlist tests
2022-03-19 10:48:13 +01:00
Stypox
401082abe4
[YouTube] Extract playlist type in playlist extractor
2022-03-19 10:48:12 +01:00
Stypox
63ed06a710
[YouTube] Differentiate genre mixes from normal mixes
...
Note: genre mixes already worked, now they are just considered as such in various video id extraction and in related items
Note 2: now extracting a mix id from a *normal* youtube mix id will fail if the video id wouldn't be exactly 11 characters long
2022-03-19 10:46:31 +01:00
Stypox
f19660e7d9
[YouTube] Deduplicate code extracting video id from mix id
2022-03-19 10:46:30 +01:00
Stypox
8f9d5b858e
[YouTube] Remove useless comments about mixes
2022-03-19 10:44:06 +01:00
Stypox
34a4484c72
[YouTube] Add test for a video with a mix in related items
2022-03-19 10:44:06 +01:00
Stypox
50db871d89
[YouTube] Extract mixes from streams related items
2022-03-19 10:44:06 +01:00
Stypox
638da1756c
[Mix] Create MultiInfoItemsCollector
...
It is a collector that can handle many extractor types, to be used when a list contains items of different types (e.g. search). It was renamed from InfoItemsSearchCollector so that it can now be used not just for search but for any extractor needing it. It supports, streams, channels, playlists and *mixes*.
2022-03-19 10:44:06 +01:00
Stypox
53673d64c6
[Mix] Add type to playlists & playlist items, to distinguish mixes
2022-03-19 10:44:06 +01:00
Stypox
d8f2031619
Merge pull request #816 from Stypox/mock-only-extension
...
Add `@MockOnly` Junit 5 extension
2022-03-19 10:40:38 +01:00
litetex
cc2e4d7104
Merge pull request #815 from litetex/fix-soundcloud-id-once-and-for-all
...
Removed hardcoded soundcloud HARDCODED_CLIENT_ID
2022-03-17 13:54:08 +01:00
litetex
cd8088b217
Merge pull request #811 from TiA4f8R/playlists-improvements-and-yt-playlists-fixes
...
[YouTube] Fix the extraction of series playlists and don't return the view count as the stream count for learning playlists
2022-03-17 13:52:24 +01:00
TiA4f8R
c7757c0994
Apply requested changes
2022-03-16 20:14:08 +01:00
TiA4f8R
35e082248e
Fix YouTube and SoundCloud playlists tests
2022-03-16 19:40:30 +01:00
TiA4f8R
8b3f90eb7e
[YouTube] Fix extraction of series playlists and don't return the view count as the stream count for learning playlists
...
ITEM_COUNT_UNKNOWN is returned when the JSON array which contains usally the number of videos is less than 3 items.
Also apply the same type of optimizations done in other PlaylistExtractors in YoutubePlaylistExtractor.
2022-03-16 19:18:58 +01:00
TiA4f8R
58a247907e
Apply changes in all playlist extractors except YoutubePlaylistExtractor
...
Also fix some issues in the extractors, remove uneeded overrides, use the Java 8 Stream API where possible and replace usages of Utils.UTF_8 with StandardCharsets.UTF_8 in these classes.
2022-03-16 19:18:57 +01:00
TiA4f8R
fc6b45ee36
Implement some methods in PlaylistExtractor
...
This will prevent their override in each child class where the values corresponding to the methods could not be extracted.
2022-03-16 19:18:36 +01:00
Stypox
73d1fd472f
Add MockOnly junit 5 test extension
2022-03-16 19:03:08 +01:00
Stypox
ef71a5fa0f
static final instead of final static
2022-03-16 17:24:33 +01:00
Stypox
0c37c75981
Make getDownloader static & extract getDownloaderType
2022-03-16 17:22:42 +01:00
Stypox
40aa5104b1
Merge pull request #786 from XiangRongLin/throttling_resilience
...
[Youtube] Make throttling decryption more resilient to api change
2022-03-16 11:03:16 +01:00
litetex
ba56be8ef1
Removed hardcoded soundcloud id
...
It never works (long enough) so let's simply remove it...
2022-03-15 21:19:19 +01:00
XiangRongLin
e726437da3
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java
...
Co-authored-by: Stypox <stypox@pm.me>
2022-03-15 17:10:05 +01:00
litetex
e7aee0ca57
Merge pull request #807 from FireMasterK/no-commentsinfo-instance
...
Remove the need for a CommentsInfo instance in CommentsInfo.getMoreItems and fix PeertubeCommentsExtractorTest.Default test
2022-03-15 15:06:56 +01:00
litetex
d806984aa8
Merge pull request #800 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.9.0
...
Bump gson from 2.8.9 to 2.9.0
2022-03-14 19:47:54 +01:00
litetex
2aa5f98c26
Merge pull request #796 from FireMasterK/streaminfo-subscriber-count
...
Add support for extracting channel subscriber count in StreamInfo
2022-03-14 19:40:57 +01:00
Tobi
a267093e76
Merge pull request #809 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.6.0
...
Bump spotbugs-annotations from 4.5.3 to 4.6.0
2022-03-12 00:02:51 +01:00
dependabot[bot]
fdb603e9ac
Bump spotbugs-annotations from 4.5.3 to 4.6.0
...
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.5.3 to 4.6.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.5.3...4.6.0 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-03-08 09:16:28 +00:00
FireMasterK
60cc71e944
Remove the need for a CommentsInfo instance.
2022-03-03 11:48:41 +00:00
Stypox
3a6269bca6
Merge pull request #805 from TeamNewPipe/dependabot/github_actions/actions/checkout-3
...
Bump actions/checkout from 2 to 3
2022-03-02 11:09:28 +01:00
dependabot[bot]
e953794703
Bump actions/checkout from 2 to 3
...
Bumps [actions/checkout](https://github.com/actions/checkout ) from 2 to 3.
- [Release notes](https://github.com/actions/checkout/releases )
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md )
- [Commits](https://github.com/actions/checkout/compare/v2...v3 )
---
updated-dependencies:
- dependency-name: actions/checkout
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-03-02 09:25:59 +00:00
Stypox
dd8687f9fe
[YouTube] Fix music mixes in some countries
2022-03-01 23:02:56 +01:00
FireMasterK
e6d334765d
Apply requested codestyle improvements.
2022-02-26 17:12:51 +00:00
FireMasterK
6950b362f2
Add test values for failing tests.
2022-02-24 12:55:13 +00:00
FireMasterK
5b0ec694a6
Add requested changes.
2022-02-24 12:50:51 +00:00
FireMasterK
d290d2e393
Move variable to super Extractor classes.
2022-02-24 12:50:51 +00:00
FireMasterK
ab49cb6e18
Add requested changes.
2022-02-24 12:50:51 +00:00
Kavin
4bd59f65f4
Update exception message.
...
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2022-02-24 12:50:51 +00:00
FireMasterK
6f3f608ab6
Add support for extracting channel subscriber count in StreamInfo.
2022-02-24 12:50:51 +00:00
Stypox
69e18c80cb
Merge pull request #799 from litetex/imporve-yt-comments-extractor
...
Refactored YoutubeCommentsExtractor
2022-02-23 17:58:00 +01:00
litetex
1a67ea100a
Refactored code
...
according to review
2022-02-21 19:56:12 +01:00
litetex
f79ce1f52a
Refactored YoutubeCommentsExtractor
...
* Use Java Streaming API
* Use StandardCharsets
* Prevented several NPEs/ArrayIndexOutOfBound
* Reformatted some code so that it's easier readable
2022-02-21 19:56:09 +01:00
TobiGr
dfe8716f5f
Merge branch 'dev'
2022-02-18 20:30:38 +01:00
TobiGr
d07e16aadd
Release NewPipe Extractor 0.21.14
2022-02-18 20:29:57 +01:00
TobiGr
d337e537dc
[YouTube] Add a comment clarifying why the panelIdentfier can be null
2022-02-18 20:26:19 +01:00
Tobi
5c8fbeb166
Fix NullPointerException in YouTubeStreamExtractor.getStreamSegments() ( #797 )
...
Fix NullPointerException in YouTubeStreamExtractor.getStreamSegments()
Co-authored-by: litetex <40789489+litetex@users.noreply.github.com>
2022-02-16 23:39:26 +01:00
dependabot[bot]
89900431ab
Bump gson from 2.8.9 to 2.9.0
...
Bumps [gson](https://github.com/google/gson ) from 2.8.9 to 2.9.0.
- [Release notes](https://github.com/google/gson/releases )
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md )
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.9...gson-parent-2.9.0 )
---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-02-14 09:15:09 +00:00
Tobi
0d6aebce25
[SoundCloud] Update clientId ( #798 )
2022-02-11 17:16:32 +01:00
dependabot[bot]
a2a41e3bf2
Bump rhino from 1.7.13 to 1.7.14
...
Bumps [rhino](https://github.com/mozilla/rhino ) from 1.7.13 to 1.7.14.
- [Release notes](https://github.com/mozilla/rhino/releases )
- [Changelog](https://github.com/mozilla/rhino/blob/master/RELEASE-NOTES.md )
- [Commits](https://github.com/mozilla/rhino/commits )
---
updated-dependencies:
- dependency-name: org.mozilla:rhino
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-02-07 20:29:51 +00:00
XiangRongLin
545522b80f
[Youtube] Make throttling decryption more resilient to api change
2022-02-01 19:57:34 +01:00
TobiGr
ebc129c392
NewPipe Extractor 0.21.13
2022-02-01 13:20:34 +01:00
TobiGr
5306f7a988
Improve code style
2022-02-01 13:18:18 +01:00
Abdu Ameen
38e9da8d44
Adding removal of the braces
2022-02-01 13:18:10 +01:00
Abdu Ameen
36ebb633a4
Changed the regex to account for
...
nonword characters
2022-02-01 13:18:05 +01:00
Tobi
65129e65a8
Merge pull request #785 from AbduAmeen/regex_error
...
Regex error when parsing the Youtube JavaScript code
2022-02-01 13:17:19 +01:00
TobiGr
047d75a4c9
Improve code style
2022-02-01 12:56:59 +01:00
Abdu Ameen
50a4b026f8
Adding removal of the braces
2022-01-31 23:52:31 -08:00
Abdu Ameen
a9174f6b9f
Changed the regex to account for
...
nonword characters
2022-01-31 22:29:00 -08:00
dependabot[bot]
a4399fd4ad
Bump spotbugs-annotations from 4.5.2 to 4.5.3
...
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.5.2 to 4.5.3.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.5.2...4.5.3 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-01-24 18:45:21 +01:00
Stypox
59e0aeba88
Merge pull request #768 from litetex/junit-5
...
JUnit 5
2022-01-24 16:56:16 +01:00
Mohammed Anas
65df39b3e3
[YouTube] parse timestamps with >3 digits correctly ( #775 )
...
* [YouTube] parse timestamps with >3 digits correctly
Fixes https://github.com/TeamNewPipe/NewPipe/issues/7530 ; check the
issue for details.
* Remove outdated comment
2022-01-12 10:45:06 +01:00
litetex
a579ef2651
Fixed Soundcloud tests
2022-01-07 14:27:24 +01:00
litetex
a32edce750
Removed deprecation because AndroidSDK/API < 19
2022-01-07 14:15:55 +01:00
litetex
602068937d
Code cleanup
...
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2022-01-06 21:04:16 +01:00
litetex
dd3f3da828
Kurzgesagt mock data is outdated
2022-01-04 17:44:12 +01:00
litetex
a8ebd2cead
Update mock resources for YT Mix playlist
2022-01-04 17:39:14 +01:00
litetex
066afd8629
Restored test
2022-01-04 17:33:08 +01:00
litetex
3712a669b1
Code cleanup / improvements
...
* Deprecated Utils#UTF-8; see StandardCharsets
* Added more helpful methods to ``ExtractorAsserts``
* Use parameterized (cool new) tests
* Restore functionality of some tests + updated mockdata
* Other code cleanups + Sonarlint improvements
2022-01-04 17:28:31 +01:00
litetex
4291a90251
Fixed SoundCloud's search(for tests)
2021-12-27 21:42:43 +01:00
litetex
4995709871
Fixed SoundCloud's search(for tests)
...
Getting the initial page was not returning initial page
2021-12-27 21:18:39 +01:00
litetex
a2cbdf0991
Updated to JUnit 5
2021-12-27 21:08:08 +01:00
litetex
10f6cc7194
Merge pull request #755 from litetex/clean-up-yt-dis-likes
...
Cleanup: Remove old ways of getting YT dis/likes
2021-12-27 18:32:02 +01:00
litetex
148ba6641d
Updated mock data
2021-12-27 17:08:14 +01:00
litetex
4d0a689cb6
Disabled dislike count in tests
2021-12-27 17:04:46 +01:00
litetex
a6961c5efa
Fixed NumberFormatException when processing likes
...
See also #7525
2021-12-27 16:54:47 +01:00
litetex
65687f3b9b
Removed YT dislike count as it no longer works
2021-12-27 16:54:46 +01:00
litetex
1e2e0029fc
[StreamExtractor] Deduplicated a ton of code by using default methods
2021-12-27 16:54:45 +01:00
litetex
15b98ffdb4
Remove old ways of getting YT dis/likes
...
* Added additional check for averageRating (in dislikes)
2021-12-27 16:54:44 +01:00
dependabot[bot]
fe432425df
Bump spotbugs-annotations from 4.5.1 to 4.5.2
...
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.5.1 to 4.5.2.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.5.1...4.5.2 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-12-14 10:35:32 +01:00
dependabot[bot]
36663dcb37
Bump spotbugs-annotations from 4.5.0 to 4.5.1
...
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.5.0 to 4.5.1.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.5.0...4.5.1 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-12-12 13:28:30 +01:00
litetex
12a6a48b15
Merge pull request #761 from litetex/fix-soundcloud-georestricted
...
Disabled ``SoundcloudStreamExtractorTest#SoundcloudGeoRestrictedTrack#testRelatedItems`` as it's unreliable
2021-12-11 18:06:17 +01:00
litetex
7484b6a098
Merge pull request #758 from litetex/release/v0.21.12
...
Release 0.21.12
2021-12-11 16:14:11 +01:00
litetex
795e6df596
Update README.md
...
Made usage section more abstract
2021-12-11 16:06:29 +01:00
litetex
4dd737c9a0
Updated version to v0.21.12
2021-12-11 15:34:46 +01:00
litetex
7489502cc8
Disabled ``testRelatedItems`` as it's unreliable
2021-12-11 15:23:11 +01:00
Tobi
1e0c409bec
Merge pull request #760 from litetex/fix-test
...
Fix test
2021-12-11 12:44:45 +01:00
litetex
d9b2a2042a
Updated mock data for YoutubeCommentsExtractorTest
2021-12-08 22:11:45 +01:00
litetex
a3a0c1ee85
Fixed YoutubeCommentsExtractorTest
...
Old video was deleted / set to private
2021-12-08 22:11:34 +01:00
litetex
5c288eb8d7
Updated mock data for YoutubeChannelExtractorTest
2021-12-08 21:51:38 +01:00
litetex
75bb83a1b8
Fixed YoutubeChannelExtractorTest
...
Channel was restored, picked a new one
2021-12-08 21:51:15 +01:00
litetex
f276bacce5
Fixed SoundcloudStreamExtractorTest
2021-12-08 21:41:27 +01:00
litetex
7f99c1c193
Fixed YoutubeStreamExtractorLivestreamTest
2021-12-08 21:16:39 +01:00
litetex
652fdf2c36
Updated mock-data for YoutubeStreamExtractorAgeRestrictedTest
2021-12-08 21:08:25 +01:00
litetex
3c669d3cb5
Fixed YoutubeStreamExtractorAgeRestrictedTest
...
Looks like a lot of stuff (description, tags, name) got changed on the existing video.
Use a better suited video.
2021-12-08 21:08:03 +01:00
litetex
f735788a39
Fixed Soundcloud hardcoded client id
2021-12-08 21:00:53 +01:00
litetex
f6f2724634
Merge pull request #753 from B0pol/dis-likes
...
Fix YouTube likes + dislikes
2021-11-29 20:42:17 +01:00
bopol
56c8af710e
Code refactoring
2021-11-21 18:16:31 +01:00
bopol
c4eca91be9
Fix YouTube likes + dislikes
2021-11-19 21:36:03 +01:00
TobiGr
5028396405
Improve names and formatting
2021-11-14 13:44:13 +01:00
dependabot[bot]
1e0c802d8d
Bump spotbugs-annotations from 4.4.2 to 4.5.0
...
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.4.2 to 4.5.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.4.2...4.5.0 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-14 13:43:44 +01:00
Tobi
68d7e4e20a
Merge pull request #742 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.8.9
...
Bump gson from 2.8.8 to 2.8.9
2021-11-08 19:37:09 +01:00
bopol
8e4f9995bf
Merge pull request #745 from TeamNewPipe/update_SC_hardcoded_client_id
...
Update hardcoded SoundCloud client_id
2021-11-04 16:33:08 +01:00
opusforlife2
cb80a646d9
Update hardcoded SoundCloud client_id
...
Hah! I got here first, @TobiGr!
2021-11-04 15:28:26 +00:00
litetex
7e7b78f1b3
Merge pull request #743 from B0pol/peertube-shortlinks
...
PeerTube short links support
2021-11-04 16:10:23 +01:00
bopol
396aecef19
Make sure playlists aren't accepted by PeertubeStreamLinkHandlerFactory
2021-11-03 14:41:39 +01:00
bopol
38ad1eaac5
[PeerTube] Support /w/ short video links part 2
...
We also need to support it for comments
2021-11-03 13:54:37 +01:00
bopol
026751624f
[PeerTube] Support /w/p/ short playlist links
2021-11-03 13:35:30 +01:00
bopol
e0b8e142fc
[PeerTube] Support /a/ and /c/ short links
2021-11-03 13:26:27 +01:00
bopol
599a91c88c
[PeerTube] Support /w/ short video links
2021-11-03 11:51:58 +01:00
dependabot[bot]
198d37090b
Bump gson from 2.8.8 to 2.8.9
...
Bumps [gson](https://github.com/google/gson ) from 2.8.8 to 2.8.9.
- [Release notes](https://github.com/google/gson/releases )
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md )
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.8...gson-parent-2.8.9 )
---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 09:12:53 +00:00
litetex
4f60225ddc
Merge pull request #741 from mhmdanas/add-y2ube-link-support
...
Add support for y2u.be links
2021-10-26 18:43:36 +02:00
mhmdanas
3e8e2a1532
Add support for y2u.be links
2021-10-22 22:48:18 +03:00
Tobi
b425394e7a
Merge pull request #731 from FireMasterK/short-description
...
Extract Video Short Description in YouTube.
2021-10-15 23:10:00 +02:00
Tobi
4e9d9bffd0
Merge pull request #738 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.4.2
...
Bump spotbugs-annotations from 4.4.1 to 4.4.2
2021-10-12 12:09:36 +02:00
dependabot[bot]
1867a381a8
Bump spotbugs-annotations from 4.4.1 to 4.4.2
...
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.4.1 to 4.4.2.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.4.1...4.4.2 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-12 09:11:11 +00:00
FireMaskterK
62313962a0
Add default implementation for getShortDescription.
2021-10-04 01:42:29 +01:00
Kavin
d1add05bc1
Add requested changes.
...
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
Add final.
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-10-04 01:36:51 +01:00
Tobi
0b0c39aec2
Merge pull request #734 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.14.3
...
Bump jsoup from 1.14.2 to 1.14.3
2021-10-02 16:49:34 +02:00
TobiGr
8dcb74724a
Release NewPipe Extractor 0.21.11
2021-10-02 16:25:35 +02:00
dependabot[bot]
3ca9f51b30
Bump jsoup from 1.14.2 to 1.14.3
...
Bumps [jsoup](https://github.com/jhy/jsoup ) from 1.14.2 to 1.14.3.
- [Release notes](https://github.com/jhy/jsoup/releases )
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES )
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.14.2...jsoup-1.14.3 )
---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-02 08:20:38 +00:00
TobiGr
d8ec3ce370
[SoundCloud] Update client id
2021-10-01 02:51:38 +02:00
FireMasterK
94efe86c71
Extract Video Short Description in YouTube.
...
In Trending, and Search results.
2021-09-22 16:26:16 +01:00
Tobi
a9d214478d
Merge pull request #703 from FireMasterK/comment-replies
...
Add support for extracting comment replies continuation
2021-09-14 23:58:14 +02:00
TobiGr
ce8cabb9f8
Release NewPipe Exttractor 0.21.10
2021-09-13 13:59:35 +02:00
FireMasterK
6aabdc6d16
Fix for requested changes.
2021-09-12 01:15:19 +05:30
Kavin
0aad09fa22
Update JavaDocs for comment replies continuation.
...
Co-authored-by: Stypox <stypox@pm.me>
2021-09-12 01:05:20 +05:30
Tobi
a031289a63
Merge pull request #727 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.4.1
...
Bump spotbugs-annotations from 4.4.0 to 4.4.1
2021-09-10 18:25:27 +02:00
dependabot[bot]
82ae322cb8
Bump spotbugs-annotations from 4.4.0 to 4.4.1
...
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.4.0 to 4.4.1.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.4.0...4.4.1 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-10 09:11:27 +00:00
Tobi
62b87552f5
Merge pull request #725 from FireMasterK/related-uploader-avatar
...
Extract Channel Avatar in related videos.
2021-09-03 22:14:28 +02:00
FireMasterK
feb09e9997
Extract Channel Avatar in related videos.
2021-09-04 01:09:40 +05:30
Tobi
f0aa46b008
Merge pull request #724 from FireMasterK/streams-uploader-verified
...
Extract uploaderVerified to StreamInfo.
2021-09-03 18:18:04 +02:00
Tobi
c8037f5e00
Merge pull request #723 from FireMasterK/uploader-avatar
...
Extract Uploader's Avatar on YouTube and PeerTube
2021-09-03 18:13:59 +02:00
FireMasterK
857b2f39e8
Extract uploaderVerified to StreamInfo.
2021-09-03 21:27:58 +05:30
FireMasterK
1db463b55f
Add a default test for the Uploader's Avatar.
2021-09-03 21:05:36 +05:30
Kavin
db6b3b2c29
Extract uploader's avatar in peertube.
...
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2021-09-02 16:14:45 +05:30
FireMasterK
a0c1dcc8d8
Remove throws parsing exception.
2021-09-02 16:14:45 +05:30
FireMasterK
0ba03c552c
Improve based on review/feedback.
2021-09-02 00:35:56 +05:30
FireMasterK
b9fad4fcc8
Extract Uploader's Avatar on YouTube.
2021-09-02 00:19:00 +05:30
Stypox
68f1fa994a
Merge pull request #721 from Stypox/yt-csv-import
...
[YouTube] Small improvements to subscription import
2021-08-31 12:02:59 +02:00
TobiGr
b644f1d67b
[SoundCloud] Update client id
2021-08-31 07:42:21 +02:00
Stypox
a5e9eeb790
[YouTube] Small improvements to subscription import
2021-08-30 15:45:45 +02:00
Tobi
9570882c73
Merge pull request #712 from magicbrothers/add-eduvid-tubus
...
Add invidious instance EduVid Tubus
2021-08-24 16:07:07 +02:00
Stypox
db6c729678
Merge pull request #709 from talanc/dev
...
[YouTube] Support csv and zip subscription import (Google Takeout)
2021-08-24 15:46:03 +02:00
talanc
94a29fd63f
[YouTube] csv and zip subscription import (Google Takeout)
...
csv:
Improved error messages
Exits early if it hasnt found any items in the first few lines
zip:
Now checks all CSV files instead of hard-coded paths
final qualifiers for immutable locals and parameters
Co-authored-by: litetex <40789489+litetex@users.noreply.github.com>
2021-08-24 15:42:49 +02:00
dependabot[bot]
816bc7137f
Bump gson from 2.8.7 to 2.8.8
...
Bumps [gson](https://github.com/google/gson ) from 2.8.7 to 2.8.8.
- [Release notes](https://github.com/google/gson/releases )
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md )
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.7...gson-parent-2.8.8 )
---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-23 13:04:10 +02:00
TobiGr
6d487e0dad
Bump version to 0.21.9
2021-08-22 19:29:22 +02:00
Tobi
7823dcd99d
Merge pull request #715 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.4.0
...
Bump spotbugs-annotations from 4.2.3 to 4.4.0
2021-08-22 16:41:20 +02:00
dependabot[bot]
d75d54e1c6
Bump spotbugs-annotations from 4.2.3 to 4.4.0
...
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.2.3 to 4.4.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.2.3...4.4.0 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-22 07:12:54 +00:00
dependabot[bot]
72393995bb
Bump jsoup from 1.13.1 to 1.14.2 ( #716 )
...
Bumps [jsoup](https://github.com/jhy/jsoup ) from 1.13.1 to 1.14.2.
- [Release notes](https://github.com/jhy/jsoup/releases )
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES )
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.13.1...jsoup-1.14.2 )
---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-22 09:12:14 +02:00
Stypox
d8179dc35a
Merge pull request #713 from TiA4f8R/regexes-yt-signatureciphers
...
[YouTube] Consolidate the regular expressions used to find the cipher decryption function
2021-08-17 18:58:27 +02:00
Tobi
89bab241b9
Merge pull request #717 from XiangRongLin/fix_tests
...
Fix failing PeertubeStreamExtractorTest
2021-08-17 17:48:28 +02:00
XiangRongLin
61acdae4b8
Fix failing PeertubeStreamExtractorTest
2021-08-17 17:06:36 +02:00
FireMasterK
4f86b8ce43
Add verified info to ChannelInfo. ( #714 )
2021-08-17 13:08:49 +02:00
Stypox
52910ce970
Keep original exception when there is an error message ( #710 )
2021-08-13 16:13:58 +02:00
TiA4f8R
f8309d546a
[YouTube] Consolidate current regexes of the cipher decryption function
...
Also add more regexes
2021-08-12 18:26:43 +02:00
magicfelix
0e16091ce0
Add invidious instance EduVid Tubus
2021-08-12 10:06:41 +02:00
Tobi
b6f951edb4
Merge pull request #711 from gardenappl/dev
...
Fix minor typos in Javadoc
2021-08-11 18:16:34 +02:00
gardenapple
50bba32239
Fix typos in Javadoc
2021-08-11 12:01:25 +03:00
Tobi
78f2192cbb
Merge pull request #687 from TacoTheDank/gradle7
...
Upgrade to Gradle 7.1.1
2021-08-08 12:38:29 +02:00
TacoTheDank
4a7c1723a6
Use new 'maven-publish' plugin
2021-08-07 17:51:32 -04:00
TacoTheDank
8919a53d95
Update gradle wrapper to 7.1.1
2021-08-07 17:51:18 -04:00
TacoTheDank
cf9e678223
Declare common dependency versions in root gradle
2021-08-07 17:51:09 -04:00
FireMasterK
71b9fd0076
Faster iframe api based player extraction. ( #694 )
...
* Faster iframe api based player extraction.
Uses the IFrame API to reduce the required download to less than 1/50 of the size.
* Remove debug code.
* Extract to two methods.
* Add tests for player URL extraction.
* Add assertThat for tests.
2021-08-07 12:35:48 +02:00
Tobi
4b147863ec
Merge pull request #705 from FireMasterK/suppored-countries
...
Update supported countries list for YouTube.
2021-08-06 10:14:27 +02:00
FireMasterK
ee32317315
Update supported countries list for YouTube.
2021-08-05 23:48:38 +05:30
FireMasterK
e8e9e6cd00
Fix comment reply continuations.
2021-08-04 22:14:04 +05:30
FireMasterK
666b45958e
Fix test and update mocks.
2021-08-04 21:10:02 +05:30
FireMasterK
f6d054e5da
Add test for comments.
2021-08-04 21:03:00 +05:30
FireMasterK
ed84658055
Get it working.
2021-08-04 18:35:14 +05:30
FireMasterK
ea7e6526fd
Collect replies/page to CommentsInfoItem.
2021-08-04 18:09:42 +05:30
FireMasterK
4385cc1b7c
Add comment replies.
2021-08-04 17:56:04 +05:30
Stypox
8ca5e5e364
Merge pull request #702 from TeamNewPipe/master
...
This merges master into dev
2021-08-03 20:52:04 +02:00
Stypox
bb3815d19b
Merge pull request #701 from Stypox/v0.21.8
...
V0.21.8
2021-08-03 20:43:15 +02:00
Stypox
5b38b3ae97
Merge branch 'master' into v0.21.8
2021-08-03 20:38:07 +02:00
Stypox
4b5907e18c
Release version 0.21.8
2021-08-03 20:33:24 +02:00
Stypox
5a88263785
Merge pull request #604 from TiA4f8R/youtubei-api
...
[YouTube] Use the new internal API in NewPipe Extractor
2021-08-03 19:01:14 +02:00
TiA4f8R
79cc5c8d12
Fix tests of YoutubePlaylistExtractorTest and YoutubeMixPlaylistExtractorTest
...
Without removing RunWith and SuiteClasses annotations (and the corresponding imports) in YoutubePlaylistExtractorTest and YoutubeMixPlaylistExtractorTest, some mocks cannot be generated, so the CI fails because of the missing mocks. Mocks of workings tests have been also updated.
2021-08-01 17:28:01 +02:00
TiA4f8R
b74a39c176
Reformat some code and don't use the clickTrackingParams in continuations of YouTube Music search results
...
The clickTrackingParams of YouTube Music search results are not needed to get continuations. This commit removes their use, which may improve privacy.
2021-08-01 17:25:59 +02:00
TiA4f8R
f8197da09e
Update YouTube comments and YouTube playlist continuations mocks
2021-08-01 12:39:05 +02:00
FireMasterK
f3e4c9d689
Use the youtubei API for YouTube comments
...
Migrate YouTube comments to the desktop version by using the `next` endpoint of the InnerTube internal API.
With the desktop version, we are able to get the exact like count of YouTube comments (by parsing the accessibility data) (the current extraction is used as a fallback). We are also now able to get if the uploader of the comment is verified or not.
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-08-01 12:39:05 +02:00
TiA4f8R
286d839a3b
Update mocks (except mocks of YoutubeCommentsExtractorTest)
2021-08-01 12:39:04 +02:00
TiA4f8R
693f654f02
Ignore a broken test of YoutubeMixPlaylistExtractorTest
2021-08-01 12:39:03 +02:00
TiA4f8R
d55109258d
Fix YoutubeStreamExtractorAgeRestrictedTest
2021-08-01 12:39:03 +02:00
FireMasterK
2eeb0a3403
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
...
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:
- the desktop API is fetched.
If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status
If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.
If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.
We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.
Other code changes:
- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-08-01 12:39:03 +02:00
TiA4f8R
7753556e66
Adress the last requested changes + update YoutubeCommentsExtractor mocks
2021-08-01 12:39:03 +02:00
TiA4f8R
2320aecb7c
Update mocks
2021-08-01 12:39:02 +02:00
TiA4f8R
629dcd63b2
Ignore some broken tests
2021-08-01 12:39:01 +02:00
TiA4f8R
3adac6a150
Workaround for rate limits: always use the Android mobile API
2021-08-01 12:39:01 +02:00
TiA4f8R
8aa60d7e8f
Update clients version
2021-08-01 12:39:01 +02:00
TiA4f8R
27bd797f1e
Update mocks
2021-08-01 12:39:01 +02:00
TiA4f8R
609919db59
Adress again reviews, fix some rebase issues
2021-08-01 12:39:00 +02:00
TiA4f8R
4299d806a2
Adress changes
2021-08-01 12:38:59 +02:00
TiA4f8R
1a6b8da438
Annotate YoutubeParsingHelper methods with Nonnull when needed
2021-08-01 12:38:59 +02:00
TiA4f8R
d8177b57f6
Loop in all formats to check if the stream has URLs protected by signatureCiphers
2021-08-01 12:38:59 +02:00
TiA4f8R
a6a2c6eb80
Revert the use of Collections.singletonList instead of Arrays.asList in addCookieHeader of YoutubeParsingHelper
2021-08-01 12:38:59 +02:00
TiA4f8R
81013e5a8e
Add a static method to reset the YouTube deobfuscation code and use it in tests
...
This method is needed for YouTube stream tests, because when all YouTube tests are ran, the signatureTimestamp is known (the sts string) so a different body than the body present in the mocks is send by the extractor instance.
As a result, running all YouTube stream tests with the MockDownloader (like the CI does) will fail if this method is not called before fetching the page of a test.
2021-08-01 12:38:58 +02:00
TiA4f8R
cc798523cd
Adress requested changes in other classes
2021-08-01 12:38:58 +02:00
TiA4f8R
accd5ddef3
Adress requested changes in YoutubeStreamExtractor
2021-08-01 12:38:58 +02:00
TiA4f8R
632772d17f
Adress requested changes in YoutubeParsingHelper
2021-08-01 12:38:58 +02:00
TiA4f8R
657f165771
Update client version and mocks
2021-08-01 12:38:44 +02:00
TiA4f8R
8c1c7281b0
Real fix of the fetch of the JS player at each the signatureTimestamp is required
...
The strings playerJsUrl, sts and playerCode are now static in order to don't fetch again the JavaScript player at each time the signatureTimestamp is needed.
2021-08-01 12:38:43 +02:00
TiA4f8R
e97a685989
Format the YoutubeMusicSearchExtractor class to be in the 100 characters per line limit
2021-08-01 12:38:43 +02:00
TiA4f8R
34a9ccb0fd
Adress requested changes
2021-08-01 12:38:42 +02:00
TiA4f8R
54d4551ca6
Adress requested changes in YoutubeParsingHelper and update mobile client version
2021-08-01 12:38:42 +02:00
TiA4f8R
a59c2a3577
Catch every exception instead of only IOException and ExtractionException and add a Javadoc
...
Catch every exception instead of only IOException and ExtractionException.
Add JavaDoc for fetchAndroidMobileJsonPlayer method of YoutubeStreamExtractor
2021-08-01 12:38:42 +02:00
TiA4f8R
6921e80ded
Try again to don't fetch at each time the JavaScript player
2021-08-01 12:38:42 +02:00
TiA4f8R
70927ddade
Update client version and mocks
2021-08-01 12:38:40 +02:00
TiA4f8R
318bc46a8c
Readd the deleted code of views because watching count of livestreams was broken
...
The number shown was the total number of views that a livestream has. In order to fix this bug, the previous code is readded.
2021-08-01 12:38:39 +02:00
TiA4f8R
947baec805
Fetch again the desktop player JSON only if the content is protected by signatureCiphers
2021-08-01 12:38:39 +02:00
TiA4f8R
7474049fd1
Update mocks
2021-08-01 12:38:39 +02:00
TiA4f8R
c32bc6e534
Try to don't fetch again the first page when requesting a channel continuation
...
Try to don't fetch again the first page of a YouTube channel when requesting a continuation of it by trying to store the channel name and the channel id into the next page using the ids field of the Page class.
2021-08-01 12:38:38 +02:00
TiA4f8R
b52732a1c3
Fix the fetch of the playerCode at each time getStsFromPlayerJs is called
...
storePlayerJs was called even if sts was not empty in the getStsFromPlayerJs method. This commit fixes it.
2021-08-01 12:38:37 +02:00
TiA4f8R
ae5abc0c5d
Fix a typo in YoutubeChannelExtractor
2021-08-01 12:38:37 +02:00
TiA4f8R
0f9e9b8b4b
Use the youtubei API for YouTube mixes + update the corresponding test + do some improvements
...
Use the youtubei API for YouTube mixes. The corresponding has been updated because the new API breaks the tests of YoutubeMixPlaylistExtractorTest.
Remove some deprecated code (the old search code with the pbj JSON) and do some other improvements.
2021-08-01 12:38:37 +02:00
TiA4f8R
14569c4aa9
Readd the return of itags 134 and 136
2021-08-01 12:38:37 +02:00
TiA4f8R
3017dde67e
Fix some typos
2021-08-01 12:38:36 +02:00
TiA4f8R
013b902535
Use the Android mobile API when there are OTF streams or the content is protected by signatureCiphers
...
Use the Android mobile API to get the itag 22 (720p with audio), removed when the content is protected by signatureCiphers.
Also use this API when they are OTF streams, to get the itag 17 and 36, low 3GPP quality streams but also the itag 139.
Update the web client version.
2021-08-01 12:38:36 +02:00
TiA4f8R
e7d589edbf
Use the youtubei API for YouTube videos + update client version
...
Update the hardcoded client version to 2.20210520.09.00
Use the player and next endpoints of the Innertube API for YouTube videos
2021-08-01 12:38:36 +02:00
TiA4f8R
f73c923f60
Don't use the youtubei.googleapis.com but the websites domains + update client version of the desktop internal API
...
Use again www.youtube.com and music.youtube.com domains instead of youtubei.googleapis.com domain because it spoofs more a web client of YouTube or YouTube Music and may reduce Google's detection of NewPipe Extractor users.
2021-08-01 12:38:34 +02:00
TiA4f8R
effcdaa4f2
Update mocks, reenable a test and fix a test
...
This commit updates mocks and reenables the test invalidId of the NotAvailable class of the YoutubePlaylistExtractorTest class beacuse with the youtubei/innertube API, it returns "Not found" and doesn't redirect to the YouTube homepage.
The expectedMetaInfo test of the MetaInfoTest class of the YoutubeSearchExtractorTest class was broken because YouTube removes the vaccine progress link of the WHO from the meta info so this commit removes it and the test is now passing.
2021-08-01 12:38:04 +02:00
TiA4f8R
4d682834c3
Fix localization and update client version
2021-08-01 12:38:03 +02:00
TiA4f8R
f46cfb0f26
Adress reviews and do some improvements
...
Adress changes requested in reviews.
Do some improvements, remove unused imports and format some code to be in the 100 characters line limit.
2021-08-01 12:38:03 +02:00
TiA4f8R
e075dd5a63
Update client version, fix some tests, update mocks and do some improvements
...
Add the origin and the referer headers with the https://www.youtube.com value for YouTube JSON POST requests.
Don't add the consent cookie header for the requests which use the youtubei/innertube API because it's uneeded.
Fix some tests and update YouTube mocks
2021-08-01 12:38:02 +02:00
TiA4f8R
b49ae547a3
Do some improvements to YoutubeStreamExtractor
...
Get the real name of the uploader (for autogenerated channels and music artist channels), like before the migration to the JSON pbj.
Do some other improvements, especially reformatting some code to be in the 100 characters line limit and use final where possible.
2021-08-01 12:38:01 +02:00
TiA4f8R
58ce9b04a1
Fix channel extraction when channel URL is youtube.com/c/username
2021-08-01 12:38:01 +02:00
TiA4f8R
991b2c7d73
Use lightweight requests when getting and checking YouTube API key and client version
2021-08-01 12:38:01 +02:00
TiA4f8R
9ab9c66ddf
Use the youtubei.googleapis.com domain for YouTube Music searches + change a check + update client version and mocks
...
Change the domain from music.youtube.com to youtubei.googleapis.com.
Use a lightweight request to check if the hardcoded YouTubeMusic keys are valid. Increase the length of the response to 500 because if the key is invalid, the length of the response returned is higher than 250 and the response when the key is valid is higher than 1500.
Format the YoutubeMusicSearchExtractor file.
Update YouTube web client version and mocks
2021-08-01 12:37:56 +02:00
TiA4f8R
77c031a88a
Use the youtubei API for YouTube trends
2021-08-01 12:36:28 +02:00
TiA4f8R
f461224b2b
Use the youtubei API for YouTube searches + update mocks
...
Add getSearchParameter, a new method in YoutubeSearchQueryHandlerFactory class which returns the params field for a search, or an empty string if there is no one.
Update mocks of YoutubeSearchExtractorTest.
2021-08-01 12:36:28 +02:00
TiA4f8R
a12c69da7d
Use the youtubei API for YouTube channels
2021-08-01 12:36:27 +02:00
TiA4f8R
5794eb2350
Use the youtubei API for YouTube playlists
2021-08-01 12:36:12 +02:00
Stypox
c97a19d719
Merge pull request #696 from XiangRongLin/decrypt_pattern
...
Expand regex to match n param decrypt function
2021-08-01 10:24:22 +02:00
Stypox
06a5219c9c
Merge pull request #699 from FireMasterK/yt-music-search
...
Fix YouTube music search.
2021-08-01 10:17:15 +02:00
XiangRongLin
37df225556
Remove length check from StringUtils.matchToClosingParenthesis
2021-07-31 16:05:24 +02:00
FireMasterK
f4aad8b014
Fix tests again.
2021-07-31 14:40:51 +05:30
FireMasterK
88c11db4c5
Fix isCorrectedSearch test.
2021-07-31 02:18:18 +05:30
FireMasterK
fd19c53f65
Fix search suggestions test.
2021-07-31 02:08:50 +05:30
FireMasterK
a685941bba
Fix YouTube music search.
2021-07-31 01:33:38 +05:30
XiangRongLin
852a65ff18
Add tests for StringUtils
2021-07-30 19:56:57 +02:00
XiangRongLin
48d897e6ad
Add final and adjust utils class name
2021-07-29 21:33:45 +02:00
XiangRongLin
60794aea31
[YouTube] Add parenthesis matching as way to parse decrypt function
...
From @Stypox
2021-07-29 21:23:00 +02:00
XiangRongLin
2967d1ae6a
[YouTube] Compile YoutubeThrottlingDecrypter pattern statically
2021-07-29 20:27:29 +02:00
XiangRongLin
1c78976900
[YouTube] Expand regex to match n param decrypt function
...
Temporary solution
2021-07-29 20:26:36 +02:00
TobiGr
b62fe7141e
Fix JDoc 2
2021-07-29 12:34:04 +02:00
TobiGr
dbc4e01c17
Fix JDoc
2021-07-29 12:31:33 +02:00
TobiGr
6335823843
Merge branch 'master' into dev
2021-07-28 20:41:23 +02:00
TobiGr
7c6ff0a38f
Release version 0.21.7
2021-07-28 20:38:45 +02:00
TobiGr
8bfcb0ad59
[SoundCloud] Fix hardcoded client id
2021-07-28 20:33:21 +02:00
Tobi
027dc65434
pull request #683 from XiangRongLin/yt_throttling
...
[YouTube] Fix buffering by decoding n parameter of stream urls
2021-07-28 20:32:38 +02:00
Tobi
14c179f343
Merge pull request #692 from TeamNewPipe/fix/sc/id
...
[SoundCloud] Fix hardcoded client id
2021-07-28 18:28:07 +02:00
Tobi
394c02ad06
Merge pull request #683 from XiangRongLin/yt_throttling
...
[YouTube] Fix buffering by decoding n parameter of stream urls
2021-07-28 18:01:57 +02:00
TobiGr
785ff4aa32
[SoundCloud] Fix hardcoded client id
2021-07-28 17:02:56 +02:00
TobiGr
79f2d74b04
Fix test
...
The decryption code changes over time. Only check whether the n parameter was changed in the URL
2021-07-28 16:45:31 +02:00
TobiGr
d70adfdb8f
Add methods for cache control to YoutubeThrottlingDecrypter.
2021-07-27 20:32:22 +02:00
TiA4f8R
d13f531b6f
Use YoutubeThrottlingDecrypter also in getAudioStreams and getVideoOnlyStreams methods of YoutubeStreamExtractor
...
Without this commit, the n param is only decrypted for streams extracted in getVideoStreams (so only for streams in the formats object of the player response).
2021-07-27 17:28:32 +02:00
TiA4f8R
32055147e0
Do some code improvements
...
Use final where possible, annotate some methods and parameters as Nonnull and format new code to be in the 100 characters limit per line.
2021-07-27 17:24:27 +02:00
TobiGr
1c30a2725e
Cache nParams to prevent executing the JavaScript function for the same nParam multiple times.
...
Closes #689
2021-07-22 19:40:12 +02:00
XiangRongLin
3a3d1d7f2b
Make YoutubeJavaScriptExtractor and JavaScript methods static
...
Also address review and rewrite some comments
2021-07-20 20:48:11 +02:00
XiangRongLin
a683c8d278
Delete duplicated code to load youtube javascript file
2021-07-17 19:14:57 +02:00
XiangRongLin
a02ee2e952
Rewrite youtube throttling solution and add tests
2021-07-17 19:10:09 +02:00
Tobi
ada67d136a
Merge pull request #647 from litetex/playerSeekbarPreview
...
Code changes to enable player thumbnail seekbar preview in NewPipe
2021-07-17 17:49:54 +02:00
litetex
0c12b396e5
Review changes
2021-07-17 16:41:12 +02:00
litetex
ecf4232ce3
Using ``Collections.emptyList()``
2021-07-17 16:41:12 +02:00
litetex
0f9ed020a4
Frameset has to implement Serializable or NewPipe fails
...
java.lang.RuntimeException: Parcelable encountered IOException writing serializable object (name = org.schabi.newpipe.extractor.stream.StreamInfo)
...
Caused by: java.io.NotSerializableException: org.schabi.newpipe.extractor.stream.Frameset
...
2021-07-17 16:41:11 +02:00
litetex
17ccaf4b87
Update YoutubeStreamExtractor.java
...
To fix ``java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.String[] java.lang.String.split(java.lang.String)' on a null object reference``
2021-07-17 16:41:10 +02:00
litetex
4e41e172ea
Update Frameset.java
...
Reformatted code
2021-07-17 16:41:10 +02:00
litetex
09b670d745
Renamed frames to previewFrames (so it's better understandable)
2021-07-17 16:41:09 +02:00
litetex
b086655d54
Reverted useless change
2021-07-17 16:41:08 +02:00
litetex
93b0c5b15f
Added seekbar data
2021-07-17 16:41:08 +02:00
XiangRongLin
6956b72af7
Fix YoutubeThrottlingDecoder.replaceNParam
...
Previously it replaced the parameter itself not the value of the parameter.
2021-07-17 10:05:14 +02:00
XiangRongLin
a189f685dc
cache player js code
2021-07-16 19:20:40 +02:00
XiangRongLin
fcdb9bdbeb
add url to sout
2021-07-15 20:05:26 +02:00
XiangRongLin
80cf8b3acd
Extract separate YoutubeThrottlingDecoder
2021-07-15 20:05:25 +02:00
XiangRongLin
a86a30103f
[Youtube] bare bones version to solve throttling
...
Done by transforming the parameter "n" from videoplayback urls
https://github.com/ytdl-org/youtube-dl/issues/29326#issuecomment-865985377
2021-07-15 20:05:25 +02:00
bopol
c38a06e8dc
Merge pull request #661 from B0pol/peertube-livestreams
...
[PeerTube] Support livestreams
2021-07-13 15:48:13 +02:00
Tobi
b45bb411e8
Merge pull request #652 from litetex/fixYTCommentsAndAddDisabledComments
...
Fix yt comments and add disabled comments functionallity
2021-07-12 16:31:50 +02:00
TobiGr
6fd93cdb31
Release 0.21.6
2021-07-10 17:07:23 +02:00
litetex
fdebf3c6cd
Added javadoc warnings
2021-07-07 20:41:59 +02:00
litetex
6860543b07
Added more doc
2021-07-06 21:16:31 +02:00
litetex
a59fead0d7
Fixed typos
2021-07-06 21:16:30 +02:00
litetex
688a1c316b
Fixed Exception when YT comments are disabled and added ``commentsDisabled`` field
...
* Fixed code: Added missing finals (according to NewPipes Checkstyle guide)
* Fixed ``findValue`` method in ``YoutubeCommentsExtractor``
2021-07-06 21:16:29 +02:00
litetex
ca33f4f60b
Improved code format (final) and added ``isCommentsDisabled`` method
2021-07-06 21:16:28 +02:00
Tobi
f324772254
Merge pull request #676 from TeamNewPipe/fix/consent-cookie
...
Fix invalid CONSENT cookie value
2021-07-04 17:41:47 +02:00
TobiGr
dbdc962a07
Update mocks
2021-07-03 13:30:26 +02:00
TobiGr
9ca6dc26ed
Fix invalid CONSENT cookie value
2021-07-03 13:27:40 +02:00
Tobi
a023f08774
Merge pull request #673 from XiangRongLin/recording_downloader
...
Remove IPv4 addresses from response during mock recording
2021-07-03 13:25:41 +02:00
XiangRongLin
b9549ace25
[Youtube] Fix lofi girl test and ignore mix with id test
2021-06-29 21:12:20 +02:00
XiangRongLin
5982431ffb
Remove IPv4 addresses from response during mock recording
2021-06-29 21:03:33 +02:00
bopol
7d7cc087e2
[PeerTube] Support livestreams
2021-06-24 11:02:30 +02:00
bopol
3966178979
Merge pull request #665 from B0pol/agerestricted
...
[youtube] Fix extraction of age restricted videos
2021-06-23 21:07:14 +02:00
bopol
b475f09ba7
don't do useless concatenation
2021-06-23 20:33:32 +02:00
TiA4f8R
d1c56be744
Merge pull request #668 from B0pol/invidious-instances
...
Update Invidious instances and support Piped links
2021-06-23 18:36:48 +02:00
bopol
b96b2a6eba
Update invidious instances
2021-06-23 14:39:10 +02:00
bopol
c5e8bd368d
Update Mocks
2021-06-23 12:21:57 +02:00
bopol
e9a992b0a9
[youtube] Fix extraction of age restricted videos
2021-06-23 12:06:02 +02:00
bopol
35c1bdd012
Merge pull request #655 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.2.3
...
Bump spotbugs-annotations from 4.0.2 to 4.2.3
2021-06-23 12:01:56 +02:00
dependabot[bot]
ad403ce73e
Bump spotbugs-annotations from 4.0.2 to 4.2.3
...
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.0.2 to 4.2.3.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.0.2...4.2.3 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-06-23 09:57:42 +00:00
bopol
65fd8740aa
Merge pull request #658 from TeamNewPipe/revert-651-fix/streamType
...
[YouTube] Fix getting stream type of live streams
2021-06-23 11:33:24 +02:00
bopol
361f142621
Use badge style instead of label
2021-06-23 11:25:59 +02:00
bopol
58f109ddec
adress stypox review
2021-06-23 11:15:40 +02:00
bopol
fe29c78fe1
Merge pull request #656 from TeamNewPipe/dependabot/gradle/org.mozilla-rhino-1.7.13
...
Bump rhino from 1.7.12 to 1.7.13
2021-06-20 00:46:42 +02:00
bopol
e9412fdadc
Merge pull request #654 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.8.7
...
Bump gson from 2.8.6 to 2.8.7
2021-06-20 00:35:24 +02:00
XiangRongLin
1510dd5187
Merge pull request #660 from TeamNewPipe/fix/sc/client-id
...
[SoundCloud] Update HARDCODED_CLIENT_ID
2021-06-18 15:48:27 +02:00
TobiGr
1f9a99c931
[SoundCloud] Update HARDCODED_CLIENT_ID
2021-06-18 15:30:54 +02:00
Tobi
c998012c28
Revert "[YouTube] Fix getting stream type of live streams"
2021-06-18 09:20:32 +02:00
dependabot[bot]
38d0339753
Bump rhino from 1.7.12 to 1.7.13
...
Bumps [rhino](https://github.com/mozilla/rhino ) from 1.7.12 to 1.7.13.
- [Release notes](https://github.com/mozilla/rhino/releases )
- [Changelog](https://github.com/mozilla/rhino/blob/master/RELEASE-NOTES.md )
- [Commits](https://github.com/mozilla/rhino/commits )
---
updated-dependencies:
- dependency-name: org.mozilla:rhino
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-06-17 21:49:07 +00:00
dependabot[bot]
5a46878b09
Bump gson from 2.8.6 to 2.8.7
...
Bumps [gson](https://github.com/google/gson ) from 2.8.6 to 2.8.7.
- [Release notes](https://github.com/google/gson/releases )
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md )
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.6...gson-parent-2.8.7 )
---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-06-17 21:49:01 +00:00
bopol
598de2dd5f
Merge pull request #552 from FireMasterK/dependabot
...
Add dependabot
2021-06-17 23:48:41 +02:00
Tobi
0cb48e36bd
Merge pull request #651 from TeamNewPipe/fix/streamType
...
[YouTube] Fix getting stream type of live streams
2021-06-12 17:57:18 +02:00
TobiGr
6762fe3357
[YouTube] Fix getting stream type of live streams
2021-06-10 16:51:00 +02:00
Tobi
a27575021d
Merge pull request #625 from TeamNewPipe/commentsStreamPosition
...
Add streamPosition for comments
2021-06-08 11:03:15 +02:00
TobiGr
f7f727d19c
Add constant NO_LIKE_COUNT to CommentsInfoItem
2021-06-08 10:28:44 +02:00
TobiGr
b70c0f93c7
Add streamPosition for comments
...
SoundCloud is the only service which supports adding comments at a specific timestamp in the stream.
2021-06-08 10:27:52 +02:00
Tobi
d4186d100b
Merge pull request #580 from TeamNewPipe/accountTerminated
...
Add AccountTerminatedException for terminated channels
2021-06-08 09:55:58 +02:00
TobiGr
89e0514d8b
Fix Javadoc generation fix
2021-06-08 09:30:05 +02:00
TobiGr
80d3052033
Fix Javadoc generation
2021-06-08 09:26:01 +02:00
TobiGr
98825a2f01
Set version to 0.21.4
2021-06-07 23:45:12 +02:00
TobiGr
fa444c8298
Remove assertations of exception messages
2021-06-06 11:38:43 +02:00
Tobi
8c42a48673
Merge pull request #646 from Stypox/get-service
...
Add utility method getService() to Info object
2021-06-05 14:29:09 +02:00
Stypox
2158ca4060
Remove two unneded warning suppressions
2021-06-05 14:15:48 +02:00
Stypox
13f192704d
Add utility method getService() to Info object
2021-06-05 14:14:19 +02:00
XiangRongLin
0ad51e76fb
Merge pull request #638 from TeamNewPipe/spelling
...
Improve exception message thrown when not accepting an URL in LinkHandlerFactory#fromUrl(String url, String baseUrl)
2021-06-04 18:23:31 +02:00
Tobi
143fd3c81c
Merge pull request #645 from XiangRongLin/fix_test
...
Fix PeertubeAccountExtractorTest
2021-06-03 20:49:59 +02:00
XiangRongLin
faa503c7dd
Fix PeertubeAccountExtractorTest
2021-06-03 19:54:42 +02:00
XiangRongLin
db81384ae0
Merge pull request #644 from TiA4f8R/utf-8-encoding-for-mocks-windows
...
Specify UTF-8 file encoding in RecordingDownloader and MockDownloader
2021-06-03 18:55:00 +02:00
TiA4f8R
ac31f3a883
Specify UTF-8 file encoding in RecordingDownloader and MockDownloader
...
On Windows, mocks are recorded and read with the Cp1252 encoding so it breaks the mocks on non ASCII characters for Linux OS (and so the CI).
The project is in Java 8, so we can't use FileReader(File, Charset) and FileReader(File, Charset) because these methods require Java 11. Instead of changing the Java version of the extractor, use OutputStreamWriter and FileOutputStream instead of FileWriter and InputStreamReader and FileInputStream instead of FileReader.
2021-06-03 18:38:25 +02:00
Tobi
7e4332e0cf
Update CI badge to representt the scheduled tests
2021-05-30 22:12:09 +02:00
bopol
ff11c2df2a
Merge pull request #628 from litetex/fix-broken-yt-liked-comments
...
Fix broken yt likes in comments
2021-05-29 11:04:11 +02:00
TobiGr
1ff56a85df
Update junit from 4.13.1 to 4.13.2
...
https://github.com/junit-team/junit4/blob/main/doc/ReleaseNotes4.13.2.md
2021-05-28 15:09:26 +02:00
TobiGr
858e007804
Improve exception message thrown when not accepting an URL in LinkHandlerFactory#fromUrl(String url, String baseUrl)
2021-05-28 14:59:44 +02:00
Tobi
519bba70b3
Merge pull request #630 from XiangRongLin/fix_tests
...
Fix MetaInfoTest.expectedMetaInfo
2021-05-28 14:16:53 +02:00
litetex
289b84e68b
Removed unused code
2021-05-27 21:27:44 +02:00
litetex
e81b0e2885
Tweaked code
2021-05-27 19:48:31 +02:00
TobiGr
f4cb32cef5
Release version 0.21.3
2021-05-26 22:40:56 +02:00
XiangRongLin
bf2af4f316
Merge pull request #635 from litetex/update-setup-java
...
Update setup java
2021-05-26 17:44:57 +02:00
litetex
8c96545e57
Merge branch 'TeamNewPipe:dev' into fix-broken-yt-liked-comments
2021-05-24 18:06:45 +02:00
litetex
12fb18c310
Updated mock data (only the affected)
2021-05-24 18:05:26 +02:00
litetex
8e08a2aac7
Improved docs
2021-05-24 18:05:26 +02:00
litetex
020acfed71
Fixed tests and added new ones
2021-05-24 18:05:26 +02:00
litetex
b310922fc0
Better exception message
2021-05-24 18:05:25 +02:00
litetex
86bb9efb5d
Added likeCount back to tests
2021-05-24 18:05:25 +02:00
litetex
46654f5fca
Reformatted code
2021-05-24 18:05:25 +02:00
litetex
2174685c5c
Reimplemented likeCount
2021-05-24 18:05:25 +02:00
litetex
01cfb55505
Update YoutubeCommentsInfoItemExtractor.java
2021-05-24 18:05:25 +02:00
litetex
bedcd87abb
Fixed occurrences where voteCount is not set (<=0 likes)
2021-05-24 18:05:25 +02:00
litetex
545c0a6f42
Fixed comment
2021-05-24 18:05:24 +02:00
Tobi
ff005122bf
Merge pull request #627 from TiA4f8R/use-snd-api-v2-everywhere
...
[SoundCloud] Use a lightweight request to check if the hardcoded client_id is valid, fix the extraction of mobile URLs and more
2021-05-23 22:52:40 +02:00
TiA4f8R
a00fdcbd3d
Format the indentation of the definition of the apiUrl string in SoundcloudParsingHelper.resolveFor method
...
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2021-05-23 20:27:45 +02:00
TiA4f8R
6b607eb38d
Update the hardcoded client_id
2021-05-23 18:08:44 +02:00
TiA4f8R
4552ea9c9f
Use the SOUNDCLOUD_API_V2_URL constant in all the SoundCloud package
2021-05-23 18:08:44 +02:00
TiA4f8R
86308d0603
Use a lightweight request to check if the client_id is valid
...
Request the api-v2 host with the client_id instead of checking if the streams of a SoundCloud track are not empty: if it is valid, the API returns 404, otherwise it should return 401.
2021-05-23 18:08:43 +02:00
TiA4f8R
c5c190500c
Format the SoundCloud package of the extractor and fix some warnings
...
Use final where possible in the package and format code to be in the 100 caracters per line limit.
Fix some warnings generated by Android Studio and do some code improvements
2021-05-23 18:08:42 +02:00
litetex
89ecc4e2e8
Update docs.yml
2021-05-23 14:34:29 +02:00
litetex
7791907480
Update ci.yml
2021-05-23 14:34:16 +02:00
Tobi
636e27333b
Merge pull request #603 from TeamNewPipe/peertube_hls
...
Add PeerTube HLS streams
2021-05-23 11:49:40 +02:00
Tobi
fadf677c42
Merge pull request #629 from TechComet/patch-1
...
[YouTube] Fix the fetch of the video info page
2021-05-23 09:41:24 +02:00
XiangRongLin
172394cd52
Update mocks
2021-05-22 20:33:20 +02:00
XiangRongLin
18c8cac40c
Fix MetaInfoTest.expectedMetaInfo
...
Expected meta changed
2021-05-21 21:00:08 +02:00
Hassan Igbaria
e8b5364745
Update YoutubeStreamExtractor.java
2021-05-21 16:22:24 +03:00
litetex
c3b837fe3b
Deduped code; Using default values...
2021-05-20 20:22:40 +02:00
litetex
b934c7ccbb
Removed unused import
2021-05-20 20:11:35 +02:00
litetex
10cf081145
Removed likeCount and added textualVoteCount
2021-05-20 20:10:14 +02:00
TobiGr
1acc53b8db
[PeerTube] Add support for HLS streams
2021-05-15 12:39:23 +02:00
Tobi
3a3ade20f4
Merge pull request #620 from TeamNewPipe/bandcamp-comments
...
Add Bandcamp comments (first page)
2021-05-02 13:43:53 +02:00
Fynn Godau
85a4270bd9
[Bandcamp] Fix comments test
2021-05-02 13:07:24 +02:00
TobiGr
ec50570053
Add mocks
2021-05-01 17:50:19 +02:00
TobiGr
f4404b5cc8
Add tests for more reasons
2021-05-01 17:50:19 +02:00
TobiGr
fc998589dc
Detect channels which have been terminated due to copyright infringement
2021-05-01 17:50:19 +02:00
TobiGr
bb3861ddce
Add AccountTerminatedException for better error handling of terminated channels
2021-05-01 17:50:19 +02:00
Tobi
284362fc15
Merge pull request #608 from TeamNewPipe/youtube-fix-segment-test
...
[YouTube] fix tests
2021-05-01 17:24:41 +02:00
TobiGr
dc678a9f6d
Release version 0.21.2
2021-04-26 18:06:30 +02:00
Tobi
7cf8edd5ac
Merge pull request #618 from TeamNewPipe/soundcloud_id
...
Fix Soundcloud extraction
2021-04-26 18:03:26 +02:00
TobiGr
9c12dc5609
[SoundCloud] Fix SoundCloud ID extraction
...
resolveIdWithEmbedPlayer() does not work anymore because the JSON data has been extracted to an API call. For this reason, replace resolveIdWithEmbedPlayer() with resolveIdWithWidgetApi)( which performs the API call.
2021-04-26 17:58:34 +02:00
Fynn Godau
9dc7a1d5ee
[Bandcamp] Add comments (first page)
2021-04-26 17:25:04 +02:00
TobiGr
8f023c1ec7
[SoundCloud] Update client id
2021-04-26 15:14:11 +02:00
Tobi
7f202db8b1
Fix Google URL detection ( #617 )
2021-04-26 11:49:47 +02:00
fynngodau
6db4bea8ca
Rework Bandcamp comments ( #613 )
2021-04-13 21:10:59 +02:00
Tobi
c14f6db14a
Merge pull request #610 from TeamNewPipe/bandcamp_related_items
...
Extract related bandcamp items
2021-04-13 21:05:27 +02:00
Fynn Godau
3671876721
Fix bandcamp radio stream extractor
2021-04-13 19:30:11 +02:00
Fynn Godau
90b5c00599
Extract related items #593
2021-04-13 17:56:19 +02:00
TobiGr
1fe645704f
[YouTube] FIx testMoreRelatedItems in RandomQueryNoMorePages
2021-04-11 23:21:35 +02:00
TobiGr
9a2b814d8f
Update mocks
2021-04-11 22:59:01 +02:00
TobiGr
50005ce937
[YouTube] Replace unavailable YouTube Stream Segments test
...
The stream is unavailable in many countries.
2021-04-11 22:58:48 +02:00
Tobi
82d11386df
Merge pull request #607 from TeamNewPipe/bandcamp_radio_segments
...
[Bandcamp] Add Radio stream segments
2021-04-11 22:30:03 +02:00
TobiGr
318cec7625
[Bandcamp] Add Radio stream segments
2021-04-11 21:19:44 +02:00
TobiGr
55e0f8e725
Merge branch 'master' into dev
2021-04-10 11:33:34 +02:00
Tobi
e747a89be2
Merge pull request #600 from TeamNewPipe/youtube-eu-cosent
...
[YouTube] Set EU consent cookie
2021-04-09 12:02:52 +02:00
TobiGr
33173eb3e6
Make CONSENT COOKIE constants private
2021-04-09 11:59:17 +02:00
TobiGr
b94b316558
Update mocks
2021-04-09 11:59:17 +02:00
TobiGr
4b8d4a84b6
Set the NumberGenerator for th remaining tests
2021-04-09 11:59:17 +02:00
XiangRongLin
db7ad9c27d
Update Mocks
2021-04-08 16:37:29 +02:00
XiangRongLin
4833df60e5
Add method to inject Random into YoutubeParsingHelper and use in tests
2021-04-08 16:36:55 +02:00
TobiGr
682ec27737
Update mocks
2021-04-08 14:22:31 +02:00
XiangRongLin
eafad3ad6b
Merge pull request #602 from XiangRongLin/soundcloud_go
...
Ignore unreliable SoundcloudGoPlusTrack.testRelatedItems
2021-04-07 19:41:44 +02:00
TobiGr
080d1e318d
Code improvement
2021-04-07 14:11:47 +02:00
TobiGr
4ee270fe01
Fix LiveStream Test
2021-04-07 14:11:41 +02:00
TobiGr
883f16e0ad
[YouTube] Set CONSENT cookie
2021-04-07 14:11:37 +02:00
XiangRongLin
aee3838ed5
Ignore unreliable SoundcloudGoPlusTrack.testRelatedItems
2021-04-06 18:26:45 +02:00
XiangRongLin
90afd6162a
Merge pull request #599 from XiangRongLin/post_body
...
Add body field to Page and use it
2021-04-06 18:23:26 +02:00
XiangRongLin
7b06c696e2
Use Page.body for YoutubeChannelExtractor
2021-04-06 18:14:38 +02:00
XiangRongLin
eda1b6e199
Use Page.body for YoutubePlaylistExtractor
2021-04-06 18:14:38 +02:00
XiangRongLin
3af26a2821
Add body field to Page
2021-04-04 11:39:58 +02:00
Tobi
4e0be60ddc
Merge pull request #601 from XiangRongLin/fix_tests
...
Fix SoundcloudGoPlusTrack Test
2021-04-04 09:55:54 +02:00
XiangRongLin
7b36469807
Fix SoundcloudGoPlusTrack
...
https://soundcloud.com/martinsolveig/places#t=0 has related streams
2021-04-04 09:39:26 +02:00
Tobi
b481a228cd
Merge pull request #598 from XiangRongLin/fix_tests
...
Fix a test and ignore rest
2021-04-03 23:44:23 +02:00
XiangRongLin
3d70b6d46a
Fix a test and ignore rest
2021-04-03 11:03:10 +02:00
XiangRongLin
1925dcf4dc
Merge pull request #597 from fynngodau/related-items
...
Rename getRelatedStreams to getRelatedItems and change return type
2021-04-02 19:54:22 +02:00
Fynn Godau
c877712647
Add deprecated old method calls to avoid breaking API
2021-04-01 22:32:36 +02:00
Fynn Godau
14f6f1b7c3
Generify related streams calls and rename method
2021-03-31 20:24:40 +02:00
Stypox
9dbc152d44
Merge pull request #594 from TeamNewPipe/conersions
...
Reduce number of type conversions
2021-03-31 14:34:46 +02:00
TobiGr
1498e1905e
Reduce number of type conversions in YouTubeSearchExtractor.collectStreamsFrom by introducing new variable
2021-03-31 13:30:38 +02:00
Tobi
7ea2cd73d3
Merge pull request #589 from fynngodau/bandcamp-more-recent-pages
...
Load more featured pages and fix featured cover arts not loading
2021-03-31 12:15:12 +02:00
Fynn Godau
705f6c6e33
Apply review
2021-03-31 11:02:30 +02:00
XiangRongLin
564d74c250
Merge pull request #578 from TeamNewPipe/code_improvements
...
Code improvements
2021-03-30 18:12:22 +02:00
TobiGr
657b00ca11
Smaller code improvements
2021-03-30 10:10:53 +02:00
TobiGr
8fbd1a2bcf
FIx potential NullPointerException
2021-03-30 10:07:20 +02:00
TobiGr
f2e7c7ebd6
Replace Vector with ArrayList for performance improvements
...
I could not figure out, why these two methods are necessary.
2021-03-30 10:07:20 +02:00
TobiGr
48fd8ca7b3
Remove redundant type declaration
2021-03-30 10:07:20 +02:00
TobiGr
22bd16fca3
Make vars private, final and static
2021-03-30 10:07:20 +02:00
TobiGr
6a0f6e846a
Fix possible bug when language.id field is not present and this the languageCode an empty String.
2021-03-30 10:07:20 +02:00
TobiGr
b9282bbe94
Remove checks which are always true
2021-03-30 10:07:20 +02:00
TobiGr
46eab1ec17
Remove declarations of exceptions which are not thrown from method signatures
2021-03-30 10:07:20 +02:00
TobiGr
c3e23559d7
Remove unused imports
2021-03-30 10:07:20 +02:00
TobiGr
e4c40ae6d1
Use List.isEmpty()
2021-03-30 10:07:20 +02:00
XiangRongLin
af2183855a
Merge pull request #591 from TotalCaesar659/update-url-to-https
...
Update URL to HTTPS in README
2021-03-29 09:33:59 +02:00
TotalCaesar659
cf3f54f438
Update URL to HTTPS
2021-03-28 22:06:56 +03:00
Fynn Godau
d663be5a78
[Bandcamp] Fix featured albums not showing thumbnails
2021-03-28 13:12:22 +02:00
Fynn Godau
dbcf61c6f7
[Bandcamp] Load more featured pages
2021-03-28 12:53:05 +02:00
Stypox
b4dee6d08f
Merge pull request #306 from B0pol/metadata
...
Extract metadata for youtube, soundcloud & mediaccc
2021-03-27 08:45:47 +01:00
TobiGr
e61ceef005
Release 0.21.0
2021-03-25 21:59:15 +01:00
Tobi
e2500fb7cc
Merge pull request #583 from TeamNewPipe/peertube_playlist
...
Fix some tests and bugs
2021-03-25 21:53:41 +01:00
TobiGr
070a40e181
[YouTube] Mix Playlist - Fix getting name
2021-03-25 21:47:16 +01:00
TobiGr
b029779217
[media.cc.de] Disable unreliable tests for now
2021-03-25 19:44:44 +01:00
TobiGr
bbee15474d
[PeerTube] Fix playlist extraction
2021-03-25 19:19:19 +01:00
Tobi
d4f83a1782
Merge pull request #575 from TiA4f8R/fix-peertube-test
...
Fix Peertube account subscribers extraction
2021-03-25 19:14:51 +01:00
TiA4f8R
ae283314da
Add a comment explaining subscribers count calculation
2021-03-25 17:54:46 +01:00
TiA4f8R
abb790f465
Remove println in getSubscriberCount from PeertubeAccountExtractor class
2021-03-25 17:49:22 +01:00
TiA4f8R
772de53a66
Increase the number of subscribers in PeertubeAccountExtractorTest
2021-03-25 17:49:20 +01:00
TiA4f8R
efe2b964f0
Extract real subscribers of a Peertube account
...
Apply fix provided in the PR and move all accounts/ strings to a constant, ACCOUNTS.
2021-03-25 17:47:44 +01:00
Tobi
12835bfae1
Merge pull request #551 from FireMasterK/ff-privacy
...
Change UA to privacy.resistFingerprinting.
2021-03-24 17:58:34 +01:00
Tobi
cae6cea551
Merge pull request #582 from Stypox/fix-trending
...
[YouTube] Fix trending getName()
2021-03-24 10:49:46 +01:00
Stypox
ce7cbbc9a0
[YouTube] Fix trending getName()
2021-03-24 09:14:09 +01:00
XiangRongLin
9009f1e277
Merge pull request #517 from XiangRongLin/flaky_test
...
Add MockOnlyRule to allow skipping specific tests based on downloader
2021-03-20 18:04:14 +01:00
bopol
152221c7fb
ignore a test with encoding problems
2021-03-19 10:14:43 +01:00
bopol
8806fb4e6b
address reviews
2021-03-19 10:14:43 +01:00
bopol
c47cc54908
Extract metadata for YouTube, SoundCloud & MediaCCC
2021-03-19 10:14:43 +01:00
Stypox
f71cfd489c
Merge pull request #526 from TiA4f8R/snd-hls-workaround
...
Support SoundCloud HLS-only tracks by using a workaround
2021-03-19 10:09:54 +01:00
TiA4f8R
379d7312fa
Don't use a regular expression to find the last segment URL and do code improvements
...
Apply suggestions provided in the PR and remove a redundant import.
2021-03-14 17:54:22 +01:00
TiA4f8R
0e3e420a25
Fix tests
2021-03-14 17:54:21 +01:00
TiA4f8R
d61d9d116d
Refactor getAudioStreams method of SoundcloudStreamExtractor
...
Split the method into private methods, in order to have a better reading.
2021-03-14 17:54:20 +01:00
TiA4f8R
a7b15b51e6
Change t to transcodingJsonObject in SoundcloudStreamExtractor
2021-03-14 17:54:19 +01:00
TiA4f8R
0438828e36
Add a test for the number of audioStreams in CreativeCommonsPlaysWellWithOthers test
...
It should be only two audio streams for track "Plays Well with Others, Ep 2: What Do an Army of Ants and an Online Encyclopedia Have in Common?" by Creative Commons (https://soundcloud.com/wearecc/plays-well-with-others-ep-2-what-do-an-army-of-ants-and-an-online-encyclopedia-have-in-common ):
- one which is a progressive stream, in MP3 format with a bitrate of 128 kbps
- one which is an HLS stream, in OPUS format with a bitrate of 64 kbps.
2021-03-14 17:54:18 +01:00
TiA4f8R
3bd08a2880
Adress requested changes and use final where possible in SoundcloudStreamExtractor
...
This commit moved the HLS parsing task to a separate method, did little performance improvements and used final where possible in the SoundcloudStreamExtractor file.
2021-03-14 17:54:17 +01:00
TiA4f8R
cbacd3c0a5
Add a check to don't show MP3 128kbps stream twice and catch IOException when fetching the HLS Manifest
...
If a progressive stream is present in the transcodings, it's unnecessary to show twice an MP3 128kbps stream so if this is the case, the MP3 HLS stream will be not added to the audioStreams, else it will.
This commit also catch fetching errors in HLS manifests parsing and don't add the corresponding stream if an error occurs.
2021-03-14 17:54:16 +01:00
TiA4f8R
26f1b4e7dc
Support SoundCloud HLS by using a workaround
...
This commit tries to support SoundCloud HLS streams by parsing M3U manifests, get the last segment URL (in order to get track length) and request a segment URL equals to track's duration so it's a single URL.
2021-03-14 17:54:14 +01:00
Tobi
def745b801
Merge pull request #579 from fynngodau/bandcamp-fix-linkhandler
...
[Bandcamp] Fix link handler acceptance behaviour
2021-03-14 12:56:10 +01:00
Fynn Godau
2e57a8f24f
[Bandcamp] Fix link handler acceptance behaviour
...
* Test for bandcamp footer instead of meta tag (which is not present on
all pages)
* Accept links to /music, not just /releases
* Correctly handle uppercase URLs
2021-03-14 09:51:30 +01:00
TobiGr
70d9e389b9
[Bandcamp] Add tests for external bandcamp artists
...
Necessary, because the external pages tested before were converted to bandcamp.com pages. See f9d06252f2
2021-03-13 20:51:59 +01:00
TobiGr
f9d06252f2
[Bandcamp] Fix tests
2021-03-13 17:42:39 +01:00
Tobi
7e6f464407
Merge pull request #509 from TiA4f8R/soundcloud-improvements
...
Add new exceptions to be able to display different error messages in apps
2021-03-07 13:51:15 +01:00
Tobi
ac51134aed
Merge pull request #546 from XiangRongLin/unignore2
...
Unignore more tests
2021-03-07 13:15:24 +01:00
Tobi
021da75f24
Merge pull request #232 from fynngodau/dev
...
Bandcamp support
2021-03-05 21:37:57 +01:00
TobiGr
22fa131922
Merge branch 'dev' into bandcamp
2021-03-05 19:45:37 +01:00
TiA4f8R
e55284bb8f
Code style improvements
...
Use final where possible in YoutubeStreamExtractor and do some other code style improvements
2021-03-05 16:38:53 +01:00
TobiGr
d0a1041afd
Prevent NullPointerException and use equalsIgnorecase
2021-03-05 16:38:52 +01:00
TiA4f8R
890cbba625
Update PaidContent exception
2021-03-05 16:38:51 +01:00
TobiGr
da3cfa967d
Handle age-restricted videos
2021-03-05 16:38:50 +01:00
TiA4f8R
448b68700c
Use final in exceptions and add AgeRestrictedException
...
AgeRestrictedException will be thrown only if the reason message equals to "Sign in to confirm your age" and if the age limit is 18.
2021-03-05 16:38:49 +01:00
TobiGr
22e6f33f0a
Ignore SoundCloudGeoRestrictedTest until HLS stream extraction is added
...
The ContentNotSupportedException is thrown because no supported audio streams where extracted. However, SoundCLoud does not check, whether there are any streams available.
This commit should be reverted in #526
2021-03-05 16:38:47 +01:00
TobiGr
21158744b1
Add expectedDescriptionIsEmpty() to DefaultStreamExtractorTest
...
Also check if related streams are empty if they are expected to be empty.
2021-03-05 16:38:46 +01:00
TiA4f8R
f15d7837a1
Fix SoundCloud tests
2021-03-05 16:38:45 +01:00
TiA4f8R
4e45aef2b3
Fix SoundCloud CreativeCommonsPlaysWellWithOthers test
2021-03-05 16:38:43 +01:00
TiA4f8R
59d6d3f04e
Fix YouTube tests
2021-03-05 16:38:42 +01:00
TiA4f8R
3925204658
Add tests and rename SoundcloudGoPlusException to SoundcloudGoPlusContentException
2021-03-05 16:38:40 +01:00
TiA4f8R
771bb1a2cb
Add three new exceptions for unavailable YouTube contents
...
These exceptions are thrown on a test with the error messages text, because YouTube returns only "UNPLAYABLE" status in most error cases.
Tests are based with English strings, so changing the lang used by
extractor will throw the generic exception (ContentNotAvailableException).
2021-03-05 16:38:39 +01:00
TiA4f8R
35325d980d
Add GeographicRestrictionException and SoundCloudGoPlusException in NewPipe Extractor to be able to display different error messages
...
This commit adds two new exceptions in NewPipe Extractor: GeographicRestrictionException and SoundCloudGoPlusException (which extend to ContentNotAvailableException). These exceptions allow showing different error messages to user when a content isn't available in his/her/its country (only used for now by SoundCloudStreamExtractor) or when the content is a SoundCloud Go+ track.
2021-03-05 16:38:37 +01:00
TobiGr
cb07ffa1eb
Release 0.20.11
2021-03-05 15:35:11 +01:00
Tobi
bc0cda68d1
Merge pull request #570 from TiA4f8R/fix-yt-channels-playlists
...
Fix YouTube channels contination
2021-03-05 15:26:49 +01:00
Tobi
a3c6fceef5
Merge pull request #573 from B0pol/comments-performance
...
[youtube] improve comments extraction performance
2021-03-05 14:46:27 +01:00
bopol
ff5273b882
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeCommentsExtractor.java
...
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2021-03-05 14:39:01 +01:00
Tobi
ec1127dd6a
Merge pull request #571 from TeamNewPipe/soundcloud_client_id
...
[SoundCloud] Update client_id and fix extraction
2021-03-05 14:21:14 +01:00
bopol
ed850d0688
[youtube] improve comments extraction performance
...
- do not parse responseBody twice for continuation
instead try to get commentsTokenInside with the new pattern ("sectionListRenderer")
and try again with the old pattern ("commentSectionRenderer") on failure
- do not unescape responseBody multiple times
-> parse responseBody less times
2021-03-05 13:37:08 +01:00
TobiGr
440a808b8a
Update mocks for channels
2021-03-05 13:13:36 +01:00
Tobi
b608587e4d
Merge pull request #572 from golfinq/fixed-comments
...
Fixes missing comments under videos
2021-03-05 11:42:40 +01:00
golfinq
05213175cd
minor edit, made variable final
2021-03-05 02:49:15 -05:00
golfinq
8bceb57a6c
formatting fix
2021-03-05 02:46:06 -05:00
golfinq
ae48bdea4c
Updated to reflect variable name change
2021-03-05 02:40:36 -05:00
TiA4f8R
b27efdc9a4
Remove clickTrackingParams in getNextPage method of YoutubeChannelExtractor
...
Because it's unused with the new request.
Co-authored-by: bopol <bopol@e.email>
2021-03-04 22:50:20 +01:00
TobiGr
2b671b15ce
[YouTube] Use new continuation API for channels
...
Co-authored-by: TiA4f8R <74829229+tia4f8r@users.noreply.github.com>
2021-03-04 21:47:16 +01:00
TobiGr
33594d4aff
[SoundCloud] Fix client id extraction
...
SoundCloud changed their Javascript structure. That caused the client_id to be in the middle of the file (at byte ~43000 ). To have a little buffer, we now fetch the first 50000 bytes.
2021-03-04 21:32:34 +01:00
TobiGr
fb2f37d223
[SoundCloud] Update hardcoded client id
2021-03-04 21:25:41 +01:00
XiangRongLin
9256b3b848
Merge pull request #567 from XiangRongLin/playlist_continuations
...
Playlist continuations
2021-03-04 19:05:32 +01:00
XiangRongLin
506cc5fbe9
regenerate mocks
2021-03-04 19:01:58 +01:00
XiangRongLin
03b00ff1d6
Move toJsonArray and toJsonObject to JsonUtils
2021-03-04 18:58:51 +01:00
FireMasterK
668b080b3a
Use dynamic key.
2021-03-04 08:45:14 +01:00
XiangRongLin
43b46bd408
Use new youtube continuations api for playlists
...
Requires sending a POST request instead of GET.
clientName and clientVersion, which were required as headers previously now need to be part of the request payload.
continuation id also needs to be part of request body.
quick and dirty solution.
2021-03-03 19:49:26 +01:00
XiangRongLin
beb05bd05c
Replace test url, because playlist was deleted
2021-03-03 19:46:14 +01:00
XiangRongLin
e13e237392
Fix typo and reword some explanations
2021-02-26 17:48:03 +01:00
Tobi
5d594cfded
Merge pull request #549 from FireMasterK/gradle-update
...
Update gradle wrapper.
2021-02-24 15:59:28 +01:00
Tobi
965a66bc87
Merge pull request #557 from TiA4f8R/fix-encoding-errors-windows
...
Set extractor encoding to UTF-8
2021-02-24 15:49:58 +01:00
TiA4f8R
b2cf41496d
Set extractor encoding to UTF-8
...
When compiling the extractor on Windows, you get more than 100 errors with the message error: unmappable character for encoding Cp1252, especially in timeago-parser tasks. This commit fixes it by setting the encoding for all compileJava and compileTestJava tasks to UTF-8.
2021-02-23 14:38:20 +01:00
TobiGr
a1688fe953
Move BandcampExtractorHelper.getJsonData(String, String) to JsonUtils
2021-02-21 13:51:12 +01:00
TobiGr
70814dcfef
Fix Utils.nonEmptyAndNullJoin
...
When using the index here, it the index needs to be decremented once an element is removed. To cirecumvent this, the native Collections.removeIf() method is used.
2021-02-21 12:51:47 +01:00
TobiGr
b9e8ee8450
Rename BandcampExtractorHelper.smartConcatenate(String[], String) to Utils.nonEmptyAndNullJoin(String, String[])
2021-02-21 12:35:14 +01:00
TobiGr
c07db80ef0
Add BASE_URL and BASE_API_URL to BandcampExtractorHelper
2021-02-21 12:15:45 +01:00
TobiGr
adde4332d1
Code improvements
2021-02-20 23:58:54 +01:00
FireMasterK
42bdd8adc5
Update to fix failing mock tests.
2021-02-20 17:14:41 +05:30
FireMasterK
e8bccfaf5d
Change UA to privacy.resistFingerprinting.
2021-02-20 17:14:41 +05:30
bopol
a9aa385bb3
Merge pull request #554 from B0pol/test-verified
...
use DownloaderFactory instead of DownloaderTestImpl for search Channe…
2021-02-20 12:12:43 +01:00
bopol
0c831afa7b
use DownloaderFactory instead of DownloaderTestImpl for search ChannelVerified
2021-02-20 11:41:30 +01:00
TobiGr
ea120a4637
remove print stacktrace
...
use interface
2021-02-20 00:23:45 +01:00
XiangRongLin
adf9d7d10f
Add reason field to MockOnly
...
This enforces developers to document why a test is skipped
2021-02-19 18:54:18 +01:00
XiangRongLin
ea52030613
Add MockOnlyRule to allow skipping specific tests based on downloader
2021-02-19 18:54:17 +01:00
TobiGr
02920fafa8
Add isUploaderVerified()
2021-02-19 15:51:23 +01:00
TobiGr
91e9309486
Merge remote-tracking branch 'origin/dev' into bandcamp
2021-02-19 13:37:45 +01:00
TobiGr
fa61b864f2
Code improvements
2021-02-19 13:37:24 +01:00
TobiGr
98268e351c
Move radio URL check into a function
2021-02-19 13:35:44 +01:00
FireMasterK
959d5d7db7
Add dependabot.
2021-02-19 11:52:56 +05:30
bopol
27a20e41cd
Merge pull request #550 from FireMasterK/add-more-data
...
Add more data.
2021-02-18 20:28:27 +01:00
FireMasterK
5333d8a98b
Add more data.
2021-02-19 00:55:03 +05:30
FireMasterK
88890d81a5
Update gradle wrapper.
2021-02-18 21:15:12 +05:30
bopol
9c1a7f7df6
Merge pull request #521 from B0pol/verified
...
Add uploader verified by service extraction
2021-02-18 16:39:53 +01:00
bopol
1a322ad8ed
Add uploader verified by service extraction
2021-02-18 16:36:40 +01:00
bopol
db253e202b
Merge pull request #535 from TeamPiped/adaptive-parsing
...
Adaptive Stream parsing
2021-02-18 15:55:27 +01:00
FireMasterK
c24afa2cbb
Permalink URL.
2021-02-18 20:16:53 +05:30
FireMasterK
8d54401233
Add fps field as well.
2021-02-18 20:16:53 +05:30
FireMasterK
4d096be14a
Revert "Update gradle wrapper."
...
This reverts commit 591856e3a11998c4b8ba87b5e361e2b9556d760f.
2021-02-18 20:16:53 +05:30
FireMasterK
5b70645d85
Add data to muxed streams aswell.
2021-02-18 20:16:53 +05:30
FireMasterK
e975d33fbe
Change jdoc / comments.
2021-02-18 20:16:53 +05:30
FireMasterK
f31b2a68fd
use getters/setters.
2021-02-18 20:16:53 +05:30
FireMasterK
d4945ac55c
Constructor cleanup.
2021-02-18 20:16:52 +05:30
FireMasterK
3f6a601be8
Fix jdoc / comments.
2021-02-18 20:16:52 +05:30
FireMasterK
a7c9905183
Fix requested changes.
2021-02-18 20:16:52 +05:30
FireMasterK
0c0f2d74bc
Make fields private
...
Co-authored-by: bopol <bopol@e.email>
2021-02-18 20:16:52 +05:30
FireMasterK
d1054338d3
Fix error with livestreams.
2021-02-18 20:16:52 +05:30
FireMasterK
11eb4932f4
Add data to respective classes.
2021-02-18 20:16:52 +05:30
FireMasterK
525e345ed8
remove duplicate field.
2021-02-18 20:16:52 +05:30
FireMasterK
e844d2aed3
Add dash fields to the ItagItem.
2021-02-18 20:16:52 +05:30
FireMasterK
b713a7af8c
Update gradle wrapper.
2021-02-18 20:16:52 +05:30
bopol
64f6b0478f
Merge pull request #545 from XiangRongLin/comment_reset
...
Add comment explaining YoutubeParsingHelper.resetClientVersionAndKey
2021-02-18 15:14:03 +01:00
XiangRongLin
a64579da2f
Unignore some tests that work on Linux
2021-02-17 20:31:26 +01:00
XiangRongLin
b99cb8624f
Unignore PublicBroadcasterTest
...
Tests only work on linux, mocks have to be generated on linux too
2021-02-17 20:31:26 +01:00
XiangRongLin
37366c65ca
Remove encoding of '&' to '&' in test and unignore it
2021-02-17 20:31:26 +01:00
XiangRongLin
c327922cc0
Add explanations to broken tests
2021-02-17 20:31:26 +01:00
XiangRongLin
26d50aca05
Add comment explaining YoutubeParsingHelper.resetClientVersionAndKey
2021-02-17 19:21:39 +01:00
TobiGr
54aa5b3042
Use propper structure in KioskExtractors
...
Made BandCampRadioExtractor a Kiosk which holds StreamInfoItems and not InfoItems.
2021-02-16 21:09:42 +01:00
TobiGr
54b8e54f80
Fix potential NPE
2021-02-16 20:59:13 +01:00
TobiGr
16973126a0
Use Collections.emptyList() instead of Collections.EMPTY_LIST or new ArrayList()
2021-02-16 20:25:48 +01:00
bopol
84380e4c3a
Merge pull request #536 from B0pol/watchendpoints
...
support /watch/, /v/ and /w/ subpaths
2021-02-16 20:23:27 +01:00
bopol
41f689b099
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/linkHandler/YoutubeStreamLinkHandlerFactory.java
...
Co-authored-by: Stypox <stypox@pm.me>
2021-02-16 19:17:10 +01:00
XiangRongLin
a9594f531f
Merge pull request #543 from XiangRongLin/unignore1
...
Unignore and fix some tests
2021-02-16 18:00:16 +01:00
XiangRongLin
a28e0226c5
Merge pull request #519 from XiangRongLin/better_logging_error_handling
...
Better logging, error handling, doc for mock tests
2021-02-16 15:06:06 +01:00
XiangRongLin
05cb22c654
Unignore Gronkh.testDescription
...
Remove special character ü from assertion
2021-02-16 08:59:35 +01:00
XiangRongLin
d9b4c2779f
Unignore YoutubeSearchQHTest.testRegularValues
2021-02-16 08:59:35 +01:00
XiangRongLin
137272b7a4
Unignore YoutubeCommentsExtractorTest.EmptyComment
...
Mock files were missing
2021-02-16 08:59:34 +01:00
TobiGr
5bf9fddba9
Code improvements
2021-02-15 22:45:17 +01:00
TobiGr
e062c8cb0d
Merge branch 'dev' into bandcamp
2021-02-15 22:09:41 +01:00
bopol
416cf17b7a
Merge pull request #532 from B0pol/utf8
...
Code improvement
2021-02-15 18:56:28 +01:00
bopol
7a3d9bdb7d
Description: add .equals() and .hashCode()
2021-02-15 18:51:47 +01:00
bopol
557934cb17
use EMPTY_STRING
2021-02-15 18:51:46 +01:00
bopol
9ca52ca68a
use only one constant UTF-8
2021-02-15 18:47:54 +01:00
XiangRongLin
50e5718bd1
Merge pull request #518 from XiangRongLin/generate_mocks
...
Use and generate mocks for youtube tests
2021-02-15 06:39:41 +01:00
Tobi
d7486f251a
Merge pull request #538 from B0pol/soundcloud-georestriction
...
Soundcloud: fix charts extraction when georestricted
2021-02-14 23:52:27 +01:00
TobiGr
432c68cdec
Update okhttp from 3.12.11 to 3.12.13
...
Changelog https://square.github.io/okhttp/changelog_3x/#version-31213
2021-02-14 23:44:19 +01:00
TobiGr
5eb75d72d3
Release 0.20.10
2021-02-14 22:09:17 +01:00
XiangRongLin
a88b2e6af4
Undo formatting of YoutubeStreamExtractorDefaultTest
2021-02-14 14:57:23 +01:00
bopol
8d89c82caa
Soundcloud: fix charts extraction when georestricted
2021-02-13 22:28:31 +01:00
bopol
ee2012c948
rename folder to subpath
2021-02-13 22:25:46 +01:00
bopol
8db40db4de
use substring instead of split
2021-02-13 22:20:05 +01:00
bopol
81317cf6bc
youtube: support /w/ folder
2021-02-13 22:20:05 +01:00
bopol
f15c0fcfed
support /watch/ and /v/ folders
2021-02-13 22:20:05 +01:00
XiangRongLin
971128c464
Ignore failing tests
2021-02-13 20:28:03 +01:00
XiangRongLin
92dea0806a
Regenerate json files for youtube tests
2021-02-13 20:28:03 +01:00
XiangRongLin
d6b9930c18
Use DownloaderFactory in YoutubeCommentsExtractorTest.Pinned
2021-02-13 20:12:39 +01:00
XiangRongLin
b555f38fa5
Use real downloader for YoutubeSubscriptionExtractorTest
...
It doesn't make network requests anyway
2021-02-13 20:12:39 +01:00
XiangRongLin
f352b0c7af
YoutubeMusicSearchExtractorTest can't be used with mocks
...
Requires further research
2021-02-13 20:00:51 +01:00
XiangRongLin
f45b5610bf
Unignore YoutubeChannelLocalizationTest
...
With mocks time is not a concern anymore
2021-02-13 20:00:51 +01:00
XiangRongLin
19737e06ee
Use DownloaderFactory in YoutubeChannelExtractorTest
...
KurzgesagtAdditional needed to be extracted because it ran a duplicated request.
2021-02-13 20:00:51 +01:00
XiangRongLin
84eb4b30c9
Use DownloaderFactory in YouTube tests
2021-02-13 20:00:51 +01:00
XiangRongLin
b43f3474f5
Make resetClientVersionAndKey public
2021-02-13 20:00:49 +01:00
Tobi
8d7b62914c
Merge pull request #540 from B0pol/fix-tests
...
fix tests, fix links in YouTube description
2021-02-13 13:20:29 +01:00
bopol
2859c7ba5c
add comment explaining why .substring(23)
2021-02-13 12:10:41 +01:00
bopol
73353996d7
fix more tests
2021-02-12 23:18:38 +01:00
bopol
a931e31239
fix tests
2021-02-12 22:36:18 +01:00
TobiGr
d116680ea3
Merge branch 'master' into dev
2021-02-12 22:26:35 +01:00
TobiGr
f8d769617a
Release 0.20.9
2021-02-12 22:14:23 +01:00
Tobi
c2525916ac
Merge pull request #539 from B0pol/yt-channel-continuation
...
[YouTube]: Fixed channel continuation
2021-02-12 22:09:31 +01:00
bopol
b0f356dd76
YT channels: fix continuation
2021-02-12 14:39:09 +01:00
bopol
fcdb50b825
make getInfo from Extractor public
...
In StreamInfo and CommentsInfo
2021-02-09 14:58:43 +01:00
bopol
44c54d403a
fix Invidious channels being accepted by YoutubeStreamLinkHandlerFactory ( #527 )
...
fixes https://github.com/TeamNewPipe/NewPipeExtractor/issues/524 , see it for more details
2021-02-05 18:57:58 +01:00
TiA4f8R
1414a6f178
Fix extraction of YouTube chapters ( #531 )
...
YouTube changed the name of the chapters in the JSON from engagement-panel-macro-markers to engagement-panel-macro-markers-description-chapters, so extracting chapters doesn't work.
2021-02-03 22:07:34 +01:00
Tobias Groza
bfa639950e
Merge pull request #528 from Stypox/ytmusic-0
...
[YouTube Music] Fix extracting search item view/subscriber count when = 0
2021-01-29 23:23:23 +01:00
TobiGr
d728c4fad0
Remove unused imports
...
I searched for "StandardCharsets" to prevent possible bugs with Android KitKat. The import is unused and is therefor removed.
There are other usages of the StandardCharsets class in the test implementation, but those are not relevant.
2021-01-29 23:16:40 +01:00
Stypox
a64dfd7343
[YouTube Music] Fix extracting search item view/subscriber count when = 0
2021-01-29 16:44:24 +01:00
Tobias Groza
c00ee75d74
Merge pull request #525 from B0pol/rename
...
rename getHeartedByCreator and getPinned to isPinned and isHeartedByC…
2021-01-23 13:45:24 +01:00
bopol
2662ceca22
rename getHeartedByCreator and getPinned to isPinned and isHeartedByCreator
2021-01-23 13:06:07 +01:00
Tobias Groza
10e8c543e8
Merge pull request #523 from B0pol/pinned
...
Extract whether the comment is pinned
2021-01-23 12:35:13 +01:00
bopol
66e4eb2f96
extract whether the comment is pinned
2021-01-22 23:59:22 +01:00
Tobias Groza
69f155d292
Merge pull request #522 from B0pol/invidious_instances
...
update invidious instances list
2021-01-22 21:04:10 +01:00
bopol
67ddfefdc6
keep invidious instances list in the same order
...
as suggested by @TobiGr
2021-01-22 20:58:14 +01:00
bopol
53d3f7989b
update invidious instances list
2021-01-22 19:30:25 +01:00
XiangRongLin
88e4c8667b
Add comments to RecordingDownloader about usage
2021-01-17 19:53:20 +01:00
XiangRongLin
eecfe09f2c
Check for non-existent folder in MockDownloader
2021-01-17 19:52:58 +01:00
XiangRongLin
a0288b8fd6
Echo which downloader is used during CI process
2021-01-17 19:52:28 +01:00
Tobias Groza
beb705013c
Merge pull request #516 from XiangRongLin/schedule_ci_with_real
...
Add if-else statement to run tests against real website on schedule
2021-01-16 12:59:05 +01:00
XiangRongLin
7a6d930e7b
Add if-else statement to run tests against real website on schedule
2021-01-16 10:03:47 +01:00
TobiGr
07570554b6
Fix build and optimize imports
2021-01-15 21:55:40 +01:00
TobiGr
78c2113094
Merge remote-tracking branch 'origin/dev' into bandcamp
2021-01-15 21:49:58 +01:00
Stypox
b9ba95614b
Merge pull request #514 from XiangRongLin/test_jvm_property
...
Pass on gradle JVM system property to test JVM
2021-01-15 20:44:23 +01:00
XiangRongLin
a6cfe11ae9
Pass on gradle JVM system property to test JVM
2021-01-15 20:37:14 +01:00
Stypox
48a9993edd
Merge pull request #513 from XiangRongLin/ci_use_mock
...
Adjust CI workflow to run tests with mocks
2021-01-15 20:20:23 +01:00
Stypox
588fbcc61f
Merge pull request #512 from XiangRongLin/ignore_failing
...
Ignore all failing tests
2021-01-15 20:18:03 +01:00
XiangRongLin
6888e7773f
Ignore all failing tests
2021-01-15 20:11:40 +01:00
XiangRongLin
e5dc62d9bd
Add argument -Ddownloader=MOCK to CI job
...
Now MockDownloader will be used in tests.
2021-01-15 18:04:33 +01:00
Stypox
7c774c84ce
Merge pull request #505 from TeamNewPipe/meida.ccc.de_search_fix
...
[media.ccc.de] Fix NPE in search results if they contain a future talk
2021-01-15 14:16:59 +01:00
Stypox
c2c4d97efd
Merge pull request #482 from XiangRongLin/mock_mix_pl_test
...
Mock mix pl tests
2021-01-15 12:54:48 +01:00
XiangRongLin
4f81d9d689
Regenerate mock files
2021-01-15 09:51:23 +01:00
Stypox
951159f75f
Merge pull request #504 from TeamNewPipe/media.ccc.de_recent_order
...
[media.ccc.de] Recent kiosk: order streams by upload date
2021-01-14 23:31:09 +01:00
Tobias Groza
cc51c5f6a3
Merge pull request #507 from vkay94/add-interval-to-stream-frames
...
Add durationPerFrame to Frameset
2021-01-14 20:27:40 +01:00
XiangRongLin
6ca7123d5d
Add comment for usage of YoutubeParsingHelper.resetClientVersionAndKey
2021-01-14 20:01:52 +01:00
vkay94
11dcfe638b
Add durationPerFrame to Frameset and getFrameBoundsAt method
2021-01-14 20:01:06 +01:00
Tobias Groza
c2ff6723d7
Merge pull request #511 from B0pol/errors
...
Fix PeerTube tests by changing instance
2021-01-14 19:15:22 +01:00
Tobias Groza
6773ec8699
Merge pull request #510 from B0pol/heartedbycreator
...
comments: add heartedByUploader and extract it for YouTube
2021-01-12 19:00:04 +01:00
TobiGr
0c4836c800
[media.ccc.de] Recent kiosk: Add comment clarifying "length" and "duration" fields have the same value
2021-01-12 18:48:42 +01:00
XiangRongLin
4af50c6870
Use @link for downloader
2021-01-12 09:41:57 +01:00
XiangRongLin
3455f0f23c
Add documentation to testing with mocks
2021-01-12 09:37:48 +01:00
bopol
5292cc1636
[peertube] test: remove usage of peertube.mastodon.host
...
It's down for months
2021-01-11 19:46:27 +01:00
bopol
f3c22da618
comments: add heartedByCreator and extract it for youtube
2021-01-11 18:48:08 +01:00
Tobias Groza
da968cf9c5
Merge pull request #508 from XiangRongLin/360_webm
...
[YouTube] Un-comment out 360p webm
2021-01-11 16:09:05 +01:00
XiangRongLin
63c237db41
Only create directories if they do not exist
2021-01-11 07:57:11 +01:00
XiangRongLin
e82cb1efea
Remove dependency to commons-io
2021-01-10 20:50:38 +01:00
XiangRongLin
1ea6c6ce54
Prefix mock file from RecordingDownloader with "generated_mock_"
...
Only read those files in MockDownloader
2021-01-10 20:46:42 +01:00
XiangRongLin
f91916c017
Remove unnecessary file object creation
2021-01-10 20:39:22 +01:00
XiangRongLin
255c726f20
Add resource path base to DownloaderFactory
2021-01-10 20:27:34 +01:00
XiangRongLin
35e299759e
Add method to allow resetting youtube client version and key
...
This is needed so that a request is made for each test class when running multiple at once. This way RecordingDownloader records all necessary requests.
This works as long as tests are run sequentially and not in parallel.
2021-01-10 20:24:50 +01:00
XiangRongLin
f447a7a450
fix import due to downloader package move
2021-01-10 19:49:14 +01:00
XiangRongLin
f8aa989c42
Add generated json and manually copy client_version.json
2021-01-10 19:38:47 +01:00
XiangRongLin
285c26eafa
Adjust YoutubeMixPlaylistExtractorTest to use DownloaderFactory
2021-01-10 19:38:41 +01:00
XiangRongLin
e6e8e39def
Add DownloaderFactory to return a specific downloader based on 2 variables.
...
If the system property 'downloader' is set that use that specific downloader. This is used from gradle by appending `-Ddownloader=ABCD to the command.
ABCD is one of DownloaderType.
The other variable is the static property `DEFAULT_DOWNLOADER` in DownloaderFactory, which can be easily changed as needed inside the IDE according to development needs`.
Normal workflow would be to first use the recording downloader and afterwards only use mocks, if the requests are always staying the same.
2021-01-10 19:36:24 +01:00
XiangRongLin
7c40fb8bf7
Add additional downloader implementations
...
RecordingDownloader relies on the real downloader and saves the request/response pair into a json file.
MockDownloader uses json files from above and mocks responses for specific requests.
2021-01-10 19:36:24 +01:00
XiangRongLin
1bcb9c76a7
Generate equals and hashCode for Request
2021-01-10 19:36:24 +01:00
XiangRongLin
4dad3d60d6
Move DownloaderTestImpl into downloader package
2021-01-10 19:36:21 +01:00
XiangRongLin
fe3902c669
Un-comment out 360p webm
2021-01-08 19:43:03 +01:00
TobiGr
2d93b23723
[media.ccc.de] Show conference title in search results instead of slug
2021-01-08 14:15:25 +01:00
TobiGr
52f3bd15f2
[media.ccc.de] Add conference title to "recent" kiosk
...
The "conference_title" field was added upstream recently (see https://github.com/voc/voctoweb/issues/495 )
2021-01-08 12:04:39 +01:00
Tobias Groza
ab62464e86
Merge pull request #495 from B0pol/sepiasearch
...
[PeerTube] Implement sepia search
2021-01-06 11:06:36 +01:00
bopol
0bcea9c2cd
add JDoc
2021-01-01 14:37:32 +01:00
bopol
0e45c25ea9
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/peertube/extractors/PeertubeStreamInfoItemExtractor.java
...
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2021-01-01 14:20:47 +01:00
TobiGr
fae67fbd45
Merge remote-tracking branch 'origin/master' into dev
2021-01-01 13:06:11 +01:00
Tobias Groza
c4b9183684
Merge pull request #506 from TeamNewPipe/gh-actions
...
Migrate to GitHub actions from Travis
2021-01-01 10:35:33 +01:00
TobiGr
4671e68304
Update links to Java SE components to Java SE 8
2020-12-31 18:05:49 +01:00
TobiGr
d63d20f943
Fix JDoc
2020-12-31 18:05:40 +01:00
FireMasterK
1ba689900e
Migrate to GitHub actions from Travis.
2020-12-31 18:05:35 +01:00
TobiGr
2cbc3ccb1a
[media.ccc.de] Fix live stream test if there are no current live streams
2020-12-31 17:03:23 +01:00
TobiGr
a896ec41b9
[media.ccc.de] Use Pattern for getting the ive stream id
2020-12-31 17:03:23 +01:00
TobiGr
5dc9a76e3c
[media.ccc.de] Recent kiosk: order streams by upload date
2020-12-30 18:38:09 +01:00
TobiGr
edf8dd0e92
[media.ccc.de] Fix NPE in search results if they contain a future talk
2020-12-30 18:34:28 +01:00
Tobias Groza
3c8c8e7307
Merge pull request #497 from TiA4f8R/fix-player-js-url-redirect
...
Fix YouTube player JavaScript URL redirection
2020-12-27 18:04:40 +01:00
TobiGr
288ddab65b
update version name
2020-12-27 18:02:16 +01:00
TiA4f8R
d30caeb091
Fix YouTube player JavaScript URL redirection
2020-12-27 15:55:03 +01:00
TobiGr
deb9af7bf5
[media.ccc.de] live kiosk - display info of the current talk
2020-12-27 15:52:09 +01:00
TobiGr
21c6a7884c
Release 0.20.8
2020-12-27 14:50:34 +01:00
Tobias Groza
1259521d53
Merge pull request #500 from TeamNewPipe/media.ccc.de_live
...
[media.ccc.de] live stream kiosk
2020-12-27 14:47:55 +01:00
TobiGr
b4e1913971
[media.ccc.de] Play live streams
2020-12-27 13:25:48 +01:00
TobiGr
80f4d42226
[Draft] Add live extractor
2020-12-27 11:01:22 +01:00
TobiGr
ed9402c002
[media.ccc.de] Escape "." in id pattern
2020-12-27 11:01:10 +01:00
TobiGr
674f2227f3
make url pattern static
2020-12-27 01:27:37 +01:00
TobiGr
090dade5b3
[media.ccc.de] Add short test for recent kiosk
2020-12-27 01:03:08 +01:00
TobiGr
0cfefe222a
Revert "[media.ccc.de] Get conference name from API URL"
...
This reverts commit d1a0686d2f
.
2020-12-26 16:55:44 +01:00
TobiGr
d1a0686d2f
[media.ccc.de] Get conference name from API URL
2020-12-26 13:15:03 +01:00
TobiGr
41699798fe
[media.ccc.de] Use web URL for uploaderUrl instead of API URL in recent
2020-12-26 12:47:32 +01:00
TobiGr
b2154c6e87
[media.ccc.de] Fix conferences kiosk test
2020-12-25 23:55:45 +01:00
TobiGr
ed6ae329b3
[media.ccc.de] Add "recent" kiosk
2020-12-25 22:26:58 +01:00
TobiGr
b2837698f5
Release 0.20.7
2020-12-25 18:55:06 +01:00
Tobias Groza
b5e50cc9fb
Merge pull request #484 from TeamNewPipe/yt_,music_search
...
Fix YouTube Music search
2020-12-25 18:30:25 +01:00
TobiGr
d9e2da53c3
Replace explicit string checks whether a playlist íd is a certain YouTube Mix type with calling the dedicated methods
2020-12-25 15:00:34 +01:00
TobiGr
abee0a8df1
Fix typos in YoutubePlaylistLinkHandlerFactory
2020-12-24 10:34:21 +01:00
TobiGr
3033c0b993
[YouTube] Music Mix: Fix playlist not being accepted
...
Regression introduced by YouTube Mix support (#280 )
2020-12-23 21:08:36 +01:00
TobiGr
b62144b49d
[YouTube] Music: Fix search extraction caused by changed JSON response
2020-12-23 21:08:36 +01:00
TobiGr
c9d9bd1e24
Fix typo in ListLinkHandlerFactory#getUrl
2020-12-23 21:05:58 +01:00
bopol
3ae924a7f1
Implement sepia search
2020-12-23 15:07:55 +01:00
TobiGr
942d840624
Fix typo in AffiliateService: rename "NO_AFILIATE" to "NO_AFFILIATE"
2020-12-23 12:50:45 +01:00
TobiGr
865c42e273
Fix line break
2020-12-23 12:49:50 +01:00
Tobias Groza
c682ea0d18
Merge pull request #451 from TeamNewPipe/meta_info
...
Extract stream and search meta info for YouTube
2020-12-22 23:05:33 +01:00
Fynn Godau
c9e9953bb0
[Bandcamp] Fix channel link handler factory
2020-12-22 20:02:53 +01:00
TobiGr
41a8ed625d
Make some vars final and add annotations to methods
2020-12-22 18:19:26 +01:00
TobiGr
bc6de14952
Extract stream and search meta info for YouTube
...
Add method to extract Google webcache URLs.
2020-12-22 18:19:26 +01:00
TobiGr
50903730b1
[Bandcamp] Fix accepting HTTP URLs
2020-12-22 08:53:10 +01:00
Tobias Groza
853a65a1a6
Merge pull request #491 from TeamNewPipe/dependecies
...
Update junit
2020-12-22 08:31:03 +01:00
TobiGr
838dc1ab54
Fix vulnerability in junit
2020-12-21 23:22:16 +01:00
Tobias Groza
7e481cf81a
Merge pull request #490 from B0pol/peertube_related
...
[peertube] fix related streams api url when no tags
2020-12-20 19:04:35 +01:00
bopol
8d3436565f
[peertube] fix related streams api url when no tags
...
fixes #489
2020-12-19 23:25:37 +01:00
Tobias Groza
22a415156f
Merge pull request #479 from vkay94/stream-segments
...
Extract stream segments for YouTube
2020-12-19 11:12:22 +01:00
TobiGr
74b46fed2d
[Bandcamp] Fix deprecation in parseDate helper function
2020-12-19 10:40:21 +01:00
TobiGr
116e921d6c
Merge remote-tracking branch 'origin/dev' into bandcamp
2020-12-19 00:26:11 +01:00
Tobias Groza
e8cc302d5f
Merge pull request #483 from XiangRongLin/cleanup-mix-pl
...
Cleanup mix pl
2020-12-19 00:13:17 +01:00
TobiGr
df16a8646f
[Bandcamp] Improve radio stream extractor test
2020-12-18 23:15:59 +01:00
XiangRongLin
cdcb66b93e
Clean up stuff from mix pl
...
Coming from unclean merge and missing hints from IDE
2020-12-16 08:35:28 +01:00
XiangRongLin
ba8782a9ed
Use lowercase string for extracting cookies
...
The map in a real response happens to ignore upper-/lowercase differences. Other maps used in unit test may not have that behaviour.
2020-12-16 08:32:04 +01:00
Stypox
85fa006214
Merge pull request #280 from XiangRongLin/mixPL
...
Extractor for youtube mix (auto-generated playlist)
2020-12-14 18:11:07 +01:00
XiangRongLin
f90f6fcf92
[YouTube] Don't escape & in getUrlFromNavigationEndpoint for playlists
2020-12-12 20:40:13 +01:00
Xiang Rong Lin
a338e4e08e
[Youtube] Apply review suggestions and avoid channel mix edge case
2020-12-12 20:32:43 +01:00
Xiang Rong Lin
22d2f7e400
[Youtube] Add cookies to youtube mix request
...
This way youtube wont return duplicates when getting more items of the mix (but youtube can also track us)
2020-12-12 20:32:40 +01:00
Stypox
421935401f
[Youtube] Add subchannel functions to mix and fix imports
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
3ff8619bcc
[Youtube] apply wb9688 suggestion (mix)
...
Channel mix adjusments and test
Don't accept youtube music mix urls as playlist
Don't override playlistData to keep getInitialPage()
Remove json constants
Indentation
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
822cf307f7
[Youtube] Add _ITEMS constants and improve code style
...
Move thumbnail id exctraction code to getThumbnailUrlFromId
Add test for "My mix" detection to service tests
Use ITEM_COUNT_UNKNOWN everywhere instead of -1 and add some tests
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
df38b1926c
[Youtube] Add tests and take thumbnail image always from first video of mix
...
Also fix getThumbnailUrl for "My Mix"
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
68a3948af6
[Youtube] Fix get banner url
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
ec6b99c082
[Youtube] Adjust mix extractor to new user agent
...
Also extract continuation now
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
0ff054acb4
[Youtube] Extract initial playlist info
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
a376792a5d
[Youtube] Handle case where url is in "youtube.com/playlist?list=listID" format.
...
This occurs when sharing a mix from the official youtube app.
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
327a5730a8
[Youtube] Add some comments to mix
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
d74265c846
[Youtube] Extract getThumbnailUrl into method and change getUploaderName
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
0efb854d27
[Youtube] Implement mix extractor for auto-generated playlists.
...
-New YoutubeMixPlaylistExtractor, that extracts from a mix (auto-generated playlist).
-The url has the format of "youtube.com/watch?v=videoID&playlistID",
where playlistID always starts with "RD" and usually followed by the videoID.
-Change YoutubePlaylistLinkHandlerFactory to create a linkhandler with the given url if it is a mix.
-Change YoutubeService to return YoutubeMixPlaylistExtractor if the url is a mix.
2020-12-12 20:30:57 +01:00
vkay94
2ba27b39af
Stream segments: Remove replaceAll from setPreviewUrl
2020-12-12 15:00:45 +01:00
Stypox
2b622fd2f1
Merge pull request #419 from Isira-Seneviratne/Use_Objects_methods
...
Use Objects methods.
2020-12-12 14:52:42 +01:00
vkay94
9a7a224a54
Add tests for stream segments
2020-12-12 10:24:55 +01:00
vkay94
d3eea4f4be
Add stream segments to StreamInfo
2020-12-12 10:24:29 +01:00
Isira Seneviratne
57be1f1123
Use Objects.requireNonNull().
2020-12-11 06:09:57 +05:30
Isira Seneviratne
4c19a88612
Use Objects.toString().
2020-12-11 06:09:50 +05:30
Isira Seneviratne
b8f64595a2
Use Objects' static equals() and hashCode() methods.
2020-12-11 06:09:44 +05:30
TobiGr
b3835bd616
Release 0.20.6
2020-12-09 23:30:23 +01:00
Tobias Groza
6fbcdd24ee
Merge pull request #477 from vkay94/fix-comments-parsing
...
Fix comments parsing
2020-12-09 23:25:51 +01:00
vkay94
9dbacbc618
Fix comments parsing
2020-12-09 14:57:38 +01:00
Fynn Godau
04dd3d4d32
Rework link handlers to correctly accept external websites
2020-12-05 15:08:26 +01:00
Tobias Groza
8ade913d9b
Merge pull request #472 from TeamNewPipe/fix-media-ccc-de
...
Fix name of media.ccc.de service
2020-12-02 22:16:52 +01:00
TheAssassin
ba3e2302bc
Fix name of media.ccc.de service
2020-12-01 23:54:27 +01:00
TobiGr
175df679e0
Release 0.20.5
2020-12-01 08:18:29 +01:00
Tobias Groza
b7a995187f
Merge pull request #463 from opusforlife2/update_invidious_URLs
...
Update Invidious URL List
2020-11-28 13:30:32 +01:00
Fynn Godau
be562b8436
Change tests
2020-11-27 16:48:40 +01:00
TobiGr
c91e21b37c
[Bandcamp] Tests - Add finals and improve code formatting
2020-11-24 14:13:31 +01:00
TobiGr
99e7ef013e
[Bandcamp] Apply small changes to code formatting and style
...
Make variables final when possible
Remove unused imports
Improve code formatting
2020-11-24 14:13:31 +01:00
fynngodau
8c369b0f79
Rephrase link in javadoc
...
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-11-24 14:01:31 +01:00
TobiGr
5a3f96b967
Release 0.20.4
2020-11-22 22:29:08 +01:00
Tobias Groza
650f0920fe
Merge pull request #465 from XiangRongLin/playlist_continuation
...
[YouTube] Fix playlist continuations extraction
2020-11-20 20:48:20 +01:00
TobiGr
334e1e9b53
Fix YouTube comments test
2020-11-20 19:53:53 +01:00
Fynn Godau
186936d041
Various changes regarding tests
2020-11-19 22:33:52 +01:00
Fynn Godau
6bc7e3420e
Merge TNP/dev into fynngodau/dev
2020-11-19 21:32:08 +01:00
TobiGr
3582a5189f
Release 0.20.3
2020-11-18 20:41:25 +01:00
XiangRongLin
5bceff0083
[YouTube] Fix extraction of next page url for the last page of playlist
2020-11-18 19:03:12 +01:00
XiangRongLin
8347e14952
[YouTube] Fix playlist continuations extraction
2020-11-17 21:04:53 +01:00
opusforlife2
2174055ef8
Update Invidious URL list in Parsing Helper
2020-11-11 15:12:31 +00:00
opusforlife2
c0afd5213a
Update Invidious URL list in Link Handler Factory
2020-11-11 14:54:16 +00:00
Tobias Groza
6701b0fe71
Merge pull request #447 from Stypox/better-deobfuscation
...
[YouTube] Cache deobfuscation code and more fixes
2020-11-08 00:32:36 +01:00
Stypox
89a77ae74a
[YouTube] Fix detection of ended livestreams and parse livestream upload date
2020-11-04 16:03:08 +01:00
Stypox
827f7bd137
[YouTube] Cache deobfuscation and improve requests made
...
Fix age restriction extraction
Automatically fixes more things
2020-11-04 16:02:12 +01:00
Stypox
b42a196f35
Merge pull request #454 from Isira-Seneviratne/Fix_date_time_issues
...
Fix some issues related to the date/time changes.
2020-11-04 14:48:20 +01:00
Isira Seneviratne
4fe28d7e3a
Fix YouTube parse error when only a date is present.
2020-11-04 05:49:42 +05:30
Isira Seneviratne
9cf9e7e980
Call existing constructor in DateWrapper.
2020-11-04 05:49:41 +05:30
Isira Seneviratne
fe31a90cb3
Remove DateTimeFormatter.ISO_OFFSET_DATE_TIME usage.
2020-11-04 05:49:41 +05:30
Tobias Groza
b13c7e1c1e
Merge pull request #452 from Stypox/yt-import
...
Implement YouTube subscription import from Google takeout
2020-11-03 20:32:17 +01:00
Tobias Groza
ac50068bbe
Merge pull request #456 from B0pol/youtube_comments_test
...
create YouTubeCommentsLinkHandlerFactoryTest
2020-11-03 20:14:14 +01:00
bopol
345e136f6c
create YouTubeCommentsLinkHandlerFactoryTest
...
and remove invidious test from YouTubeCommentsExtractorTest, because it was just testing if the URL is accepted, then the extractor does the same thing, we don't need to test the same thing twice
2020-11-03 19:10:10 +01:00
Stypox
501ec30152
Implement youtube subscription import from Google takeout
2020-11-02 11:04:48 +01:00
Tobias Groza
564a965810
Merge pull request #453 from bd0n4lds/fix-donation-typo
...
Fix typo in DonationLinkHelper and rewrote swtich statement
2020-11-02 00:15:04 +01:00
Bri@n
82746d172f
Fix typo in DonationLinkHelper and rewrote swtich statement
2020-11-01 17:34:34 -05:00
Stypox
2f02c0e6a4
Merge pull request #450 from B0pol/fileutils
...
create FileUtils
2020-11-01 18:13:57 +01:00
bopol
f69b3ef05d
create FileUtils
2020-11-01 17:15:34 +01:00
TobiGr
be9e160333
Fix build
2020-11-01 17:04:02 +01:00
Tobias Groza
8cbdec675b
Merge pull request #417 from Isira-Seneviratne/Use_Java_8_Date_Time_API
...
Use the Java 8 Date/Time API.
2020-11-01 16:49:06 +01:00
Isira Seneviratne
fcdfe7d939
Update README.md.
2020-11-01 06:28:54 +05:30
Isira Seneviratne
4f04cfccca
Switch from Calendar to OffsetDateTime in DateWrapper.
2020-11-01 06:28:54 +05:30
Isira Seneviratne
ee3af63c04
Switch to ChronoUnit.
2020-11-01 06:28:53 +05:30
Isira Seneviratne
0526a5148d
Set source and target versions to Java 8.
2020-11-01 06:28:52 +05:30
Isira Seneviratne
b2d0c098a3
Merge branch 'dev' of https://github.com/TeamNewPipe/NewPipeExtractor into dev
2020-11-01 06:27:59 +05:30
Stypox
6cc50b57e3
Merge pull request #410 from Scrxtchy/sets-track-title
...
Fix parsing Soundcloud tracks that contain the term 'sets'
2020-10-31 21:43:21 +01:00
Stypox
b242e1d113
Merge pull request #362 from Stypox/google-search-urls
...
Add support for Google search redirect url
2020-10-31 21:03:48 +01:00
Stypox
9e53cf0b56
Fix parameter reassignment and other style issues
...
Also remove left-behind debug statement
2020-10-31 20:58:28 +01:00
Stypox
3fe55b30ba
Add support for Google search redirect url
2020-10-31 20:58:28 +01:00
Stypox
c190a3029b
Consider protocol as base url when it is a custom one (e.g. vnd.youtube)
2020-10-31 20:58:28 +01:00
Isira Seneviratne
35c7b5640c
Merge branch 'dev' of https://github.com/TeamNewPipe/NewPipeExtractor into dev
2020-10-31 05:45:02 +05:30
Stypox
a9303b24a5
Merge pull request #445 from B0pol/peertube_fix
...
[PeerTube] fix account and channel extractors
2020-10-30 19:35:51 +01:00
bopol
b21e59925d
[PeerTube] fix account and channel extractors
2020-10-29 19:52:29 +01:00
Isira Seneviratne
61dcbbb800
Merge branch 'dev' of https://github.com/TeamNewPipe/NewPipeExtractor into dev
...
Conflicts:
extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java
2020-10-29 12:20:16 +05:30
Stypox
bcc01b99c7
Merge pull request #444 from Stypox/error-message
...
Remove any reference to decrypt and improve error message
2020-10-28 23:41:04 +01:00
Stypox
30ed4f2d63
Remove any reference to decrypt and improve error message
2020-10-28 20:26:00 +01:00
TobiGr
62912ee834
Merge branch 'master' into dev
2020-10-28 14:28:06 +01:00
TobiGr
94176c5d68
Merge branch 'master' into dev
2020-10-28 07:52:50 +01:00
Tobias Groza
19c0e8700d
0.20.2
2020-10-28 07:29:42 +01:00
Tobias Groza
354426fe5d
Merge pull request #441 from B0pol/fix-three-attempts
...
Support new YouTube JSON scheme
2020-10-28 07:24:31 +01:00
Scratch
947ce3ee10
Fix parsing Soundcloud tracks that contain the term 'sets'
2020-10-28 07:55:02 +11:00
bopol
6dc5ab4015
find playerUrl in another place when assetsPattern regex fails
2020-10-27 13:48:58 +01:00
bopol
db0ef83d6b
fix youtube decryption and three attemps bug
...
fixes teamnewpipe/newpipe#4572 fixes #439
2020-10-27 13:40:24 +01:00
Stypox
943334ba49
Merge pull request #440 from B0pol/fix-redirect-channels
...
Fix redirect channels
2020-10-26 20:00:06 +01:00
bopol
c1e9857960
fix subscriber count when subscribe is disabled
...
fixes #305
2020-10-26 19:06:01 +01:00
bopol
a39a2cca82
fix redirect channels
2020-10-26 19:05:19 +01:00
Tobias Groza
a8665fd97f
Merge pull request #442 from B0pol/polish
...
Polish
2020-10-26 18:51:47 +01:00
bopol
0a12300c5e
polish tests
2020-10-26 16:57:37 +01:00
bopol
01f49e8f66
polish strings
2020-10-26 16:32:39 +01:00
Tobias Groza
14c0c37d64
Merge pull request #309 from Stypox/improve-stream-tests
...
Improve stream extractor tests & various fixes
2020-10-25 18:49:53 +01:00
Stypox
df26f61cfc
Merge pull request #394 from kirisakow/dev
...
Fix TeamNewPipe/NewPipe#4213 : [YouTube] Ignore leading characters in video id
2020-10-25 14:39:03 +01:00
Kiril Isakov
366ed71523
[YouTube] Ignore leading characters in video id
2020-10-25 14:34:20 +01:00
Stypox
8026304a0a
Merge pull request #409 from Bartoshr/bugfix/short-channel-urls
...
Support short custom YouTube channel URLs
2020-10-25 11:43:50 +01:00
Stypox
57e7994c9e
Add some missing finals, nullables and comments
2020-10-25 08:12:41 +01:00
Stypox
f11fe87688
[YouTube] Replace outdated PewDiePie video test with another one
...
The old video was made private, and this video (wedding) is probably never going to be removed.
2020-10-24 18:41:41 +02:00
Stypox
3c55ea9321
[PeerTube] Change age restricted video in tests
...
The old one wasn't available anymore
2020-10-24 18:41:41 +02:00
Stypox
880ff04a5c
[Test] Add stream dash mpd url test
2020-10-24 18:41:41 +02:00
Stypox
d0b14644bb
[YouTube/MediaCCC] Consider dates as GMT and not as local
2020-10-24 18:41:41 +02:00
Stypox
8ce711f40f
[Test] Add sub channel name, url and thumbnail tests
2020-10-24 18:41:41 +02:00
Stypox
6127826571
[Test] Add stream metadata tests
2020-10-24 18:41:41 +02:00
Stypox
a087b092b4
[Test] Improve code style and add final
2020-10-24 18:39:55 +02:00
Stypox
a4097d8d01
[MediaCCC] Return empty list of video-only streams instead of null
2020-10-24 18:39:55 +02:00
Stypox
8dc3f28618
[PeerTube] Test one channel url with api and one without
2020-10-24 18:39:55 +02:00
Stypox
68d23defba
[YouTube] Do not catch every exception on getErrorMessage
...
@B0pol suggestion
2020-10-24 18:39:55 +02:00
Stypox
3191bd6c70
[YouTube] Use final when possible
2020-10-24 18:39:55 +02:00
Stypox
55bc01d1ce
[SoundCloud] Use final when possible, ide refactorings
2020-10-24 18:39:55 +02:00
Stypox
06430c4749
[PeerTube] Use final when possible, ide refactorings
2020-10-24 18:39:06 +02:00
Stypox
fcb9b6f855
[MediaCCC] Use final when possible, ide refactorings
...
Refactorings on `throws` clause
2020-10-24 18:36:22 +02:00
Stypox
af5b8b1915
[MediaCCC] Add tests for stream and conference link handlers
2020-10-24 18:36:21 +02:00
Stypox
07a90d116a
[MediaCCC] Use regex to parse stream and conference urls
2020-10-24 18:36:21 +02:00
Stypox
0c980b2d64
[PeerTube] Improve channel and stream link handler tests
2020-10-24 18:36:21 +02:00
Stypox
d130fd79c3
[PeerTube] Prepend "accounts/" to channel id for backward compatibility
2020-10-24 18:36:21 +02:00
Stypox
492db83ccf
[MediaCCC] Return null instead of empty items collector
...
As per the documentation in the base getRelatedStreams()
2020-10-24 18:34:57 +02:00
Stypox
3b2a1829d4
[MediaCCC] Extract tags
2020-10-24 18:34:57 +02:00
Stypox
b461da792f
[MediaCCC] Fix link handler inconsistency providing API links
2020-10-24 18:34:34 +02:00
Stypox
aeeae87641
[PeerTube] Parse timestamp from url (previously unimplemented)
2020-10-24 18:33:42 +02:00
Stypox
7ae3cb6d07
[PeerTube] Fix link handler inconsistency providing API links
2020-10-24 18:33:11 +02:00
Stypox
4349be13af
[PeerTube] Return empty audio stream list instead of null
2020-10-24 18:25:56 +02:00
Stypox
3b2cfb4ca2
[SoundCloud] Return empty video stream list instead of null
...
Also replace every instance of `return new ArrayList<>();` with `return Collections.emptyList();`
2020-10-24 18:23:46 +02:00
Stypox
072bae321f
[YouTube] Fix frame extraction for livestreams
...
Use saved playerResponse instead of parsing json every time
2020-10-24 18:23:46 +02:00
Stypox
7cd410f3fc
[YouTube] Return 0 when there is no timestamp, not -2, as per javadoc
2020-10-24 18:23:45 +02:00
Stypox
7fb867c166
[YouTube] Fix error message obtaining when there is none
2020-10-24 18:22:44 +02:00
Stypox
1d7a86e664
[Test] Add base classes for stream extractor tests
...
Refactor all stream extractor tests to use new base class.
Remove check if upload date is in the past: this does not have to hold true: youtube premieres turn up in search results even though they are in the future
2020-10-24 18:22:44 +02:00
Bartosz Rumiński
29695aed0a
Small field refactor
2020-10-23 16:42:30 +02:00
Bartosz Rumiński
0e67d820bc
Use static regex pattern for excluded path segments
2020-10-22 20:13:29 +02:00
Bartosz Rumiński
d3f80d1538
Exlude links which are not channels
2020-10-20 20:06:06 +02:00
Tobias Groza
3058af0e43
Merge pull request #418 from Stypox/yt-shorts
...
[YouTube] Handle URLs for Shorts
2020-10-18 13:06:10 +02:00
Stypox
6887d59570
[YouTube] Handle urls for Shorts
2020-10-18 12:03:01 +02:00
Scratch
e945f711c1
Fix SoundCloud test artist account name ( #416 )
2020-10-16 23:42:46 +02:00
TobiGr
350eed6214
Release v0.20.1
2020-10-16 22:03:57 +02:00
Tobias Groza
f82eda9f7d
Merge pull request #415 from wb9688/fix-ytinitialdata
...
Fix parsing new ytInitialData
2020-10-16 21:59:39 +02:00
wb9688
be9a6f931c
Fix parsing new ytInitialData
2020-10-16 20:27:40 +02:00
Tobias Groza
1a9bc8caf1
Merge pull request #414 from Scrxtchy/soundcloud-slash
...
Remove tailing slash from SoundCloud URLs
2020-10-16 09:48:44 +02:00
Scratch
6a70cb9d50
Remove tailing slash from SoundCloud URLs
...
Fixes #412
2020-10-16 17:17:40 +11:00
Tobias Groza
9a785f9a7e
Merge pull request #413 from Scrxtchy/soundcloud-test-update
...
Fix SoundCloud test artist account name
2020-10-16 08:05:44 +02:00
Scratch
538f5d3973
Fix SoundCloud test artist account name
2020-10-16 13:45:05 +11:00
Tobias Groza
527945eadb
Merge pull request #406 from Stypox/fix-decrypt
...
[YouTube] Fix some decryption exceptions by retrying
2020-10-15 22:03:25 +02:00
Stypox
19e862657a
[YouTube] Fix some decryption exceptions by retrying
2020-10-15 20:16:44 +02:00
Bartosz Rumiński
e3f996e014
Exlude links which are not channels
2020-10-12 20:59:56 +02:00
Bartosz Rumiński
5ab1b053d2
Update youtube channel link handler tests
...
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-10-12 20:11:28 +02:00
Bartosz Rumiński
7abb4b3713
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/linkHandler/YoutubeChannelLinkHandlerFactory.java
...
Fix typos
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-10-12 19:57:45 +02:00
Bartosz Rumiński
9d63211a66
Fix typos
...
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-10-12 19:56:53 +02:00
Bartosz Rumiński
8c38a5509e
Prevent attribution_link urls to be accepted by channel links handler
2020-10-09 19:07:38 +02:00
Bartosz Rumiński
4e04991762
Support short custom youtube channel urls
2020-10-09 17:06:59 +02:00
Fynn Godau
81b5e7cf3d
Fix extractor
2020-10-08 17:56:03 +02:00
TobiGr
acb04eb351
Update extractor version to 0.20.0
2020-10-04 13:37:19 +02:00
Stypox
2463884aa8
Merge pull request #400 from wb9688/skip-otf
...
Do not list YouTube's OTF streams
2020-09-29 20:16:34 +02:00
wb9688
ebbfe7f6d4
Skip YouTube's OTF streams
2020-09-29 10:48:02 +02:00
Tobias Groza
6633f26ec5
Merge pull request #386 from mhmdanas/remove-subtitles-stream-get-url
...
Remove SubtitlesStream#getURL()
2020-08-26 21:32:36 +02:00
Tobias Groza
070cd92857
Merge pull request #387 from wb9688/init-safe-standard-objects
...
Use initSafeStandardsObjects()
2020-08-26 21:25:46 +02:00
wb9688
7657c2ed1a
Use initSafeStandardsObjects()
2020-08-15 17:08:07 +02:00
mhmdanas
1a63dcb355
Remove overriden field `url`
2020-08-12 22:37:36 +03:00
Tobias Groza
de8edbe7a5
Merge pull request #382 from TobiGr/fix/empty_comment
...
[YouTube] Fix crash on empty comment
2020-08-12 14:18:16 +02:00
mhmdanas
f0f1c009b2
Remove SubtitlesStream#getURL()
2020-08-10 19:28:56 +03:00
Fynn Godau
cfe88a74c1
Throw ContentNotSupportedException when opening radio uploader channel
2020-08-09 10:07:49 +02:00
TobiGr
0fb73301e3
[YouTube] Fix crash on empty comment
...
Closes #380
2020-08-05 18:25:40 +02:00
Fynn Godau
932d094d6a
Upgrade featured playlist urls to https
2020-08-02 16:55:18 +02:00
Fynn Godau
c12ef3a02d
Merge TNP/dev into fynngodau/dev
2020-08-02 16:43:04 +02:00
TobiGr
5ac80624a4
Fix detection of YouTube's decryption function
...
Closes TeamNewPipe/NewPipe#3951
2020-07-28 01:01:10 +02:00
TobiGr
b4481dfec2
Update version to 0.19.7
2020-07-26 14:24:07 +02:00
Tobias Groza
32dff1541f
Merge pull request #370 from wb9688/fix-yt-age-restricted
...
Fix age restricted YouTube videos
2020-07-26 14:17:19 +02:00
TobiGr
ec778200d9
Add comments to the code
2020-07-26 14:15:13 +02:00
Tobias Groza
8627d01006
Merge pull request #373 from wb9688/fix-yt-continuations
...
Support YouTube's new continuations
2020-07-26 13:56:33 +02:00
wb9688
667dce034c
Only use fallback methods for clientVersion when clientVersion hasn't been successfully extracted yet
2020-07-26 13:14:25 +02:00
wb9688
f345f667e2
Extract YouTube's key
2020-07-26 12:01:59 +02:00
wb9688
4c987a5302
Support YouTube's new continuations for search
2020-07-26 10:01:03 +02:00
wb9688
81459e289f
Fix age restricted YouTube videos
2020-07-18 19:52:54 +02:00
Tobias Groza
df28a087de
Merge pull request #337 from wb9688/next-stream
...
Remove getNextStream() from StreamExtractor
2020-07-18 11:55:58 +02:00
wb9688
fc3a63fec5
Update Javadoc for getRelatedStreams()
2020-07-15 18:48:37 +02:00
wb9688
ff560e907a
Remove getNextStream() from StreamExtractor
2020-07-15 18:48:37 +02:00
TheAssassin
186193d366
Update version to 0.19.6
2020-07-14 17:04:03 +02:00
Tobias Groza
a70cb0283f
Merge pull request #314 from wb9688/remove-getnextpageurl
...
Next page stuff
2020-07-07 20:49:38 +02:00
wb9688
0a5a905bc7
Add final at more places
2020-07-07 20:45:47 +02:00
Tobias Groza
e5d23a89de
Merge pull request #366 from wb9688/yt-timestamp-start
...
Support start= timestamps for YouTube
2020-07-06 21:28:19 +02:00
wb9688
17ba8a57fa
Clean up the code
2020-07-06 20:19:31 +02:00
wb9688
9b6fe1dea6
Throw IllegalArgumentException when Page is invalid
2020-07-06 20:19:31 +02:00
wb9688
4cc312086a
Introduce Page class
2020-07-06 20:19:31 +02:00
wb9688
e3bfdba135
Remove getNextPageUrl() function from ListExtractor
2020-07-06 20:11:40 +02:00
wb9688
156a26b64b
Support start= timestamps for YouTube
2020-07-06 14:08:40 +02:00
Tobias Groza
54d9e5a2f8
Merge pull request #365 from B0pol/update_inv_instances
...
Update inv instances
2020-07-02 23:13:07 +02:00
bopol
261471e7f4
remove invidiou.sh from tests
...
dead instance
2020-07-02 21:33:45 +02:00
bopol
eb3901acee
update invidious instances
2020-07-02 21:31:05 +02:00
Tobias Groza
c0ceb5cb27
Merge pull request #340 from wb9688/peertube-deleted-comments
...
Handle isDeleted for PeerTube comments
2020-06-28 10:21:37 +02:00
Tobias Groza
f4ed7ce922
Merge pull request #361 from yausername/gradleScope
...
expose extractor via api scope
2020-06-27 21:45:04 +02:00
Tobias Groza
339f1d9c43
Merge pull request #349 from wb9688/okhttp
...
Use OkHttp for tests like in NewPipe
2020-06-27 20:08:02 +02:00
Tobias Groza
970bc7f69d
Merge pull request #359 from mauriciocolli/fix-client-id-bug
...
[SoundCloud] Fix concurrency issue when getting the client id
2020-06-27 17:36:24 +02:00
Tobias Groza
dbf6ab9e91
Merge pull request #355 from Redirion/patch-1
...
Added a comment
2020-06-27 11:47:50 +02:00
Robin
576754982e
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java
...
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-06-27 11:35:01 +02:00
yausername
89e0e3ee2d
expose extractor via api scope
2020-06-27 03:02:43 +05:30
Mauricio Colli
7505e828cc
[SoundCloud] Update client id
2020-06-24 01:50:09 -03:00
Mauricio Colli
d2f1c0f40d
[SoundCloud] Fix concurrency issue when getting the client id
2020-06-24 01:50:08 -03:00
Robin
b63ae93495
Added a comment
...
and fixed a "typo" in the method parseDurationString replacing Long with Integer.
These are not the droids you are looking for wb9688 😁
2020-06-15 11:27:44 +02:00
Tobias Groza
92f6754f0f
Merge pull request #338 from wb9688/parse-duration-string-test
...
Add tests for parseDurationString()
2020-06-15 11:01:21 +02:00
wb9688
e02d3c37b2
Merge pull request #354 from Stypox/peertube-age-restricted
...
[PeerTube] Change age restricted video in tests
2020-06-14 20:09:36 +02:00
Stypox
5808439c3f
[PeerTube] Change age restricted video in tests
...
The old one wasn't available anymore
2020-06-14 19:57:39 +02:00
wb9688
3239aa84f2
Handle isDeleted for PeerTube comments
2020-06-14 19:27:20 +02:00
wb9688
a65e46e4b1
Use OkHttp for tests like in NewPipe
2020-06-11 15:06:08 +02:00
Tobias Groza
5f523254cb
Merge pull request #344 from B0pol/peertube_embeds
...
For #342 : support PeerTube embeds
2020-06-06 18:53:07 +02:00
Fynn Godau
96de834b67
Bandcamp: parse date from radio info item
2020-06-04 19:36:58 +02:00
Fynn Godau
39b55b5b42
Bandcamp radio stream extractor: test uploader url
2020-06-04 19:32:19 +02:00
Fynn Godau
9a555d97e5
Remove useless bandcamp tests
2020-06-04 19:31:01 +02:00
Fynn Godau
e13f341a45
Bandcamp stream extractor: test service id
2020-06-04 19:29:37 +02:00
Fynn Godau
4dd9540782
Bandcamp license switch order
2020-06-04 19:27:39 +02:00
Fynn Godau
52103ac61f
Remove "useless" comments
2020-06-04 19:27:13 +02:00
Fynn Godau
8fa81537c0
Better bandcamp stream extractor description test
2020-06-04 19:17:51 +02:00
Fynn Godau
0950a95577
Fix for testing wrong object for null or empty
2020-06-04 19:16:08 +02:00
Fynn Godau
e6ecd91929
Bandcamp: Test full date
2020-06-04 19:12:10 +02:00
Fynn Godau
27e7aad159
Bandcamp: Improve code style
2020-06-04 19:04:25 +02:00
fynngodau
ea49202f64
Don't display internal license id
...
Co-authored-by: bopol <bopol@e.email>
2020-06-04 18:26:58 +02:00
bopol
063f4be766
For #342 : support PeerTube embeds
2020-06-04 14:26:42 +02:00
Fynn Godau
34b6928124
Don't print stacktrace before throwing new exception
2020-06-04 14:25:38 +02:00
Fynn Godau
1be20ceeec
Bandcamp: parse date
2020-06-04 14:09:21 +02:00
Fynn Godau
6822fe3dc8
Bandcamp: Get Stream upload date
2020-06-04 13:03:16 +02:00
Fynn Godau
e98bff8ae9
Improved Bandcamp radio stream extractor tests
2020-06-04 12:30:53 +02:00
Fynn Godau
8f6c00f8d8
Use default test in BandcampSearchExtractorTest
2020-06-04 12:12:20 +02:00
Fynn Godau
de776561b0
Bandcamp: fix loading uploader from streams in search
2020-06-04 12:02:18 +02:00
Fynn Godau
3940138fc5
Fix Bandcamp capitalization in one more spot
2020-06-03 22:41:27 +02:00
Fynn Godau
9fa9d920a9
Bandcamp: Implement new methods required due to interface changes
2020-06-03 21:35:11 +02:00
Fynn Godau
dd955c7f0c
Merge branch 'origin/dev' into dev
2020-06-03 21:13:40 +02:00
wb9688
33b1121fc7
Add tests for parseDurationString()
2020-05-30 17:20:54 +02:00
TobiGr
3aa6b93e91
Update version to 0.19.5
2020-05-30 10:45:39 +02:00
Tobias Groza
bda83fe6a5
Merge pull request #336 from wb9688/fix-time-parsing
...
Fix YouTube video duration parsing
2020-05-30 10:36:46 +02:00
wb9688
e39147202c
Fix YouTube video duration parsing
2020-05-30 09:31:08 +02:00
wb9688
98055a3c3c
Revert "Add more decryption function regexes"
...
This reverts commit 384d6acd5d
.
2020-05-29 18:41:40 +02:00
Tobias Groza
bdb0f2ae6b
Merge pull request #332 from wb9688/learning-playlist
...
Support YouTube's learning playlists
2020-05-29 10:11:08 +02:00
Tobias Groza
cf18cdb2f5
Merge pull request #329 from wb9688/decryption-regexes
...
Add more decryption function regexes
2020-05-29 10:10:50 +02:00
wb9688
ab77961b0a
Extract trailer thumbnail in YT learning playlists
2020-05-29 09:55:24 +02:00
wb9688
4d683e7655
Support YouTube's learning playlists
2020-05-29 09:17:03 +02:00
Tobias Groza
0b4977bb0c
Merge pull request #333 from wb9688/playerresponse-dashmanifesturl
...
Extract dashManifestUrl in playerResponse
2020-05-28 21:17:07 +02:00
Tobias Groza
0f8057ae56
Merge pull request #322 from wb9688/upgrade-jsoup
...
Upgrade dependencies
2020-05-28 21:16:33 +02:00
wb9688
384d6acd5d
Add more decryption function regexes
2020-05-28 11:40:38 +02:00
wb9688
8f71bde776
Fix bug for YT videos longer than 1000h
2020-05-28 11:34:42 +02:00
wb9688
aa277e1b6a
Upgrade other dependencies
2020-05-28 11:34:42 +02:00
wb9688
036cc7e26e
Upgrade jsoup
2020-05-28 11:34:41 +02:00
Fynn Godau
89b0639faa
New Bandcamp tests for channel extractor
2020-05-25 20:22:23 +02:00
Fynn Godau
e08256ebc6
Remove "url cleanup" in Bandcamp link handlers
2020-05-25 19:17:27 +02:00
Fynn Godau
692b2d06f4
Bandcamp: read info item data in place and not in advance
2020-05-25 18:53:15 +02:00
Fynn Godau
fff21caa87
Fix Bandcamp no avatar test
...
I own the bandcamp artist account called "NewPipeExtractorTest", so it
shouldn't break again.
2020-05-25 18:53:10 +02:00
wb9688
7ef3e80493
Extract dashManifestUrl in playerResponse
2020-05-15 12:07:26 +02:00
Stypox
60e6df7e6b
Merge pull request #330 from B0pol/gitignore
...
add vscode / eclipse files to .gitignore
2020-05-13 10:39:33 +02:00
wb9688
efafbbb88a
Merge pull request #331 from wb9688/playlist-subchannel
...
Add sub-channel stuff to PlaylistInfo
2020-05-12 10:49:13 +02:00
wb9688
baf5dd3e8f
Merge pull request #311 from B0pol/fixSearchSuggestion
...
Search: add isCorrectedSearch() and fix YoutubeSearchExtractor#getSea…
2020-05-12 09:29:15 +02:00
wb9688
102f04e297
Add sub-channel stuff to PlaylistInfo
2020-05-12 09:14:29 +02:00
bopol
cbf1163b80
ignore workspace and folder configs
2020-05-11 21:23:50 +02:00
Tobias Groza
b40ccb5075
Merge pull request #307 from B0pol/playlists_peertube
...
[PeerTube] playlist support
2020-05-11 20:53:03 +02:00
bopol
aeb2a6ff6f
add vscode / eclipse files to .gitignore
2020-05-11 13:06:02 +02:00
B0pol
3a858e6640
Merge pull request #315 from B0pol/isnullorempty
...
refactor: add Utils.isNullOrEmpty()
2020-05-11 11:04:56 +00:00
bopol
adaf196c99
remove unused import
2020-05-11 12:54:33 +02:00
bopol
de26e00079
changed all == null || isEmpty() to isNullOrEmpty()
2020-05-11 11:40:24 +02:00
bopol
202a73516c
refactor: add Utils.isNullOrEmpty()
2020-05-11 11:26:18 +02:00
Tobias Groza
3cae32b6db
Merge pull request #328 from wb9688/fix-327
...
Fix #327
2020-05-09 20:29:11 +02:00
wb9688
1de0876fb4
Fix #327
2020-05-09 20:06:50 +02:00
bopol
7cae95bf09
add tests for subchannel stuff in playlists
2020-05-09 10:15:35 +02:00
bopol
1a6e92ebf6
add getSubChannel Name, Url and AvatarUrl for playlists
2020-05-09 09:52:24 +02:00
bopol
cf99107745
address yausername change
2020-05-09 00:52:04 +02:00
bopol
5bab9d9fc0
[PeerTube] playlist support & refactoring
2020-05-09 00:52:04 +02:00
Tobias Groza
f3913e241e
Merge pull request #324 from Stypox/music-detector-url
...
[YouTube] Support stream urls in `vnd.youtube://ID` form
2020-05-08 17:43:04 +02:00
Tobias Groza
3ddd6e4d7b
Merge pull request #326 from wb9688/subchannel-parentchannel
...
Rename SubChannel in channels back to ParentChannel
2020-05-08 17:40:49 +02:00
wb9688
7320108c66
Rename SubChannel in channels back to ParentChannel
2020-05-08 16:02:59 +02:00
wb9688
44d382b4bd
Merge pull request #313 from Royosef/DisplayParentChannelDetails
...
Display parent channel details
2020-05-08 15:52:32 +02:00
Roy Yosef
a408661771
Fix PR review
...
Rename "ParentChannel" to "SubChannel"
Add Tests for sub-channel info fetching
2020-05-07 20:20:12 +02:00
Roy Yosef
1de1f97cf1
Add parent channel info to StreamExtractor: name, url & avatar url
2020-05-07 20:20:12 +02:00
Roy Yosef
4234740baa
Add parent channel info to ChannelExtractor: name, url & avatar url
2020-05-07 20:20:12 +02:00
bopol
6cff5de400
renaming methods for consistency
2020-05-07 15:40:41 +02:00
bopol
5760366d70
use JsonObject#isEmpty instead of JsonObject#equals(JsonUtils.EMPTY_OBJECT)
2020-05-07 15:40:41 +02:00
bopol
93e868634e
clarified javadoc for isCorrectedSearch and getSearchSuggestions in SearchExtractor
2020-05-07 15:40:41 +02:00
bopol
3757541a9c
adapt new nanojson version
2020-05-07 15:40:41 +02:00
bopol
79483a6dc0
«Use this in SearchInfo, for consistency's sake» @Stypox order xD
2020-05-07 15:40:41 +02:00
bopol
e8d58e09c9
getSearchSuggestion now returns the corrected query if isCorrectedSearch() is true
2020-05-07 15:40:41 +02:00
bopol
5d1c3b3fa2
Use setters in SearchInfo
2020-05-07 15:39:57 +02:00
bopol
29b639b454
Search: add isCorrectedSearch() and fix YoutubeSearchExtractor#getSearchSuggestion()
...
isCorrectedSearch: see the javadoc
getSearchSuggestion: in YoutubeSearchExtractor, it was giving the corrected search query. It now gives the suggested query, as it should
2020-05-07 15:39:57 +02:00
wb9688
665c69b530
Merge pull request #310 from B0pol/soundcloudComments
...
Add support for soundcloud comments
2020-05-07 15:35:57 +02:00
Stypox
a02bfbeae5
[YouTube] Support stream urls in `vnd.youtube://ID` form
2020-05-07 15:31:31 +02:00
bopol
bc13e0c616
Merge remote-tracking branch 'upstream/dev' into soundcloudComments
2020-05-05 14:49:21 +02:00
TobiGr
4086715a68
Merge branch 'dev'
2020-05-03 21:21:30 +02:00
TobiGr
8a79b14077
Update extractor version to 0.19.4
2020-05-03 15:11:20 +02:00
Tobias Groza
fc3a69ed54
Merge pull request #323 from wb9688/fix-yt-music-albums
...
Fix NPE in defaultAlertsCheck()
2020-05-03 14:08:47 +02:00
wb9688
6aae99f832
Fix NPE in defaultAlertsCheck()
2020-05-03 10:28:45 +02:00
Tobias Groza
2780e716b3
Merge pull request #321 from wb9688/fix-nanojson
...
Return null instead of "" in getTextFromObject()
2020-05-02 15:56:28 +02:00
wb9688
a1eabc7589
Return null instead of "" in getTextFromObject()
2020-05-02 08:02:25 +02:00
Fynn Godau
e7b046db9d
Merge remote-tracking branch 'origin/dev' into dev
2020-04-26 23:12:43 +02:00
Fynn Godau
a3e8e1c70e
Bandcamp: move own methods to helper class
2020-04-26 23:05:56 +02:00
Fynn Godau
9bc7a470ab
Replace fori with foreach loop
2020-04-26 23:02:14 +02:00
Tobias Groza
49157fcf0d
Merge pull request #317 from wb9688/tnp-nanojson
...
Use our fork of nanojson
2020-04-24 21:54:35 +02:00
bopol
2564bcf399
refactor comments
...
rename methods in CommentsInfoItemExtractor interface to match the other method names across NewPipeExtractor
remove getName in (Youtube|SoundCloud|Peertube)CommentsExtractor and move it up in CommentsExtractor, return "Comments" instead
2020-04-21 20:05:54 +02:00
Fynn Godau
f71bd29d58
Bandcamp playlist extractor: add more tests
2020-04-20 23:52:12 +02:00
Fynn Godau
5009d9f53b
Bandcamp tests: remove certain tests
2020-04-20 23:25:00 +02:00
Fynn Godau
b5e251c82f
Bandcamp channel extractor: package-private access for getImageUrl(…)
2020-04-20 23:12:43 +02:00
Fynn Godau
8c70dab8c5
Bandcamp search query handler factory: inline variable
2020-04-20 23:11:16 +02:00
Fynn Godau
10ae3db118
Bandcamp javadoc: Replace br tags with p tags
2020-04-20 23:03:12 +02:00
Fynn Godau
9201e0d3ea
Don't declare exceptions that can't actually be thrown
2020-04-20 22:46:54 +02:00
Fynn Godau
965bce00cf
Bandcamp stream extractor: return NO_AGE_LIMIT
2020-04-20 22:44:45 +02:00
Fynn Godau
82099592c7
Bandcamp: Move code from SearchExtractor to InfoItemExtractors
2020-04-20 22:44:35 +02:00
Fynn Godau
00c0333059
Bandcamp: Fetch channelInfo in onFetchPage
2020-04-20 22:41:44 +02:00
Fynn Godau
433d72c26a
Depend on nanojson fork in timeago-parser
2020-04-20 22:04:10 +02:00
Fynn Godau
67de0285e1
Bandcamp: Generate query JSON in a proper way
2020-04-20 22:03:29 +02:00
Fynn Godau
c133190c53
Bandcamp capitalization
2020-04-20 21:55:35 +02:00
Fynn Godau
b100b9873f
Bandcmap: Use nanojson fork instead of org.json
2020-04-20 21:53:04 +02:00
wb9688
b51699a20e
Implement @TobiGr's suggestions
2020-04-20 14:27:33 +02:00
wb9688
74f0ab1792
Merge pull request #301 from wb9688/media-ccc-non-api-urls
...
Use media.ccc.de instead of api.media.ccc.de
2020-04-19 17:06:57 +02:00
Fynn Godau
9c239371f4
Change nanojson dependency to @wb9688's fork
2020-04-19 14:29:39 +02:00
Fynn Godau
5c0a03328a
Merge 'origin/dev' into dev
2020-04-19 14:26:51 +02:00
bopol
0cbbc2a1f9
remove unused import
2020-04-19 14:16:38 +02:00
wb9688
979c5a7502
Use our fork of nanojson
2020-04-17 19:36:42 +02:00
bopol
d4352f9b84
support comments for SoundCloud
2020-04-10 20:41:41 +02:00
wb9688
1b522304c1
Merge pull request #308 from Royosef/PeerTubeChannelExtractor
...
[PeerTube] channels support
2020-04-10 19:07:51 +02:00
Roy Yosef
4afe657f6f
Add tests for getUrlFromId and test for video-channels in acceptUrlTest
2020-04-10 19:05:38 +03:00
Roy Yosef
2c9f1260eb
Fix wb9688 review comments
...
* Rename PeertubeUserExtractor to PeertubeAccountExtractor
* Add test for video-channels in PeertubeChannelLinkHandlerFactoryTest
* Compatibility support for older versions (use "accounts/" as default)
2020-04-10 16:40:53 +03:00
bopol
636c430743
refactor: create extractors and linkHandler packages for SoundCloud, move YoutubeParsingHelper to youtube package (not linkhandler anymore)
2020-04-10 10:51:05 +02:00
Roy Yosef
b6e6f403a8
add support for PeerTube channels extraction
2020-04-09 20:37:49 +03:00
wb9688
a5155fb562
Merge pull request #300 from B0pol/hls_opus_not_supported
...
throw ContentNotSupportedException when content is known to be unsuppo…
2020-04-08 19:54:16 +02:00
bopol
a20d53dd98
remove the mention of Opus on wb9688 advise when
...
ContentNotSupportedException is thrown
2020-04-08 15:31:11 +02:00
bopol
9a7c6b7ab0
use ContentNotSupportedException for channels without tabs
2020-04-07 14:33:43 +02:00
bopol
a22104cbda
add tests for ContentNotSupportedException
2020-04-07 14:33:43 +02:00
bopol
093762e793
throw ContentNotSupportedException when content is know to be unsupported
2020-04-07 14:33:43 +02:00
Tobias Groza
094b87c537
Merge pull request #299 from B0pol/sc_countries
...
add supported countries for soundcloud
2020-04-03 19:12:33 +02:00
Tobias Groza
a7d3d14910
Add link to JDoc to readme
2020-04-03 18:14:14 +02:00
Tobias Groza
e3263666bd
Merge pull request #303 from opusforlife2/correct-template
...
Corrected the pull request template
2020-04-03 17:56:02 +02:00
Tobias Groza
b81e22ddc9
Merge pull request #291 from wb9688/yt-music-search
...
Add support for YouTube Music search
2020-04-03 17:55:34 +02:00
wb9688
bce27a0e22
Rename getValidResponseBody() to getValidJsonResponseBody()
2020-04-03 17:23:18 +02:00
opusforlife2
4a1d8c804c
Corrected the pull request template
...
It was a bit weird before.
2020-04-02 10:42:54 +00:00
bopol
1eb3deb7fd
add supported countries for soundcloud
2020-04-02 11:26:19 +02:00
Tobias Groza
cef7c96d7d
Merge pull request #302 from wb9688/readme-local-build
...
Add another way to test changes locally to README.md
2020-04-01 19:23:04 +02:00
wb9688
8a9e137385
Extract some code to getValidResponseBody()
2020-04-01 16:01:21 +02:00
wb9688
2d0c301440
Add another way to test changes locally to README.md
2020-04-01 15:43:11 +02:00
wb9688
bf24caa5f2
Use media.ccc.de instead of api.media.ccc.de
2020-04-01 13:25:10 +02:00
wb9688
2af610e3e7
Add tests for other YT Music search types
2020-04-01 10:30:51 +02:00
wb9688
c7f7bd2442
Fix error when YT Music videos has no uploader URL
2020-04-01 10:30:51 +02:00
wb9688
ac15df4548
Move YouTube Music search to its own class
2020-04-01 10:30:51 +02:00
wb9688
cf0f2aff3e
Extract uploader url from certain YouTube Music videos
2020-04-01 10:30:51 +02:00
wb9688
aa8cea47f3
Refactor YouTube Music search tests
2020-04-01 10:30:51 +02:00
wb9688
dd434cca01
Fix issue when there is no didYouMeanRenderer in itemSectionRenderer
2020-04-01 10:30:51 +02:00
wb9688
5a775a4bbe
Use new way of specifying stream count
2020-04-01 10:30:51 +02:00
wb9688
d58c0f230d
Improve code for YouTube Music search
2020-04-01 10:30:51 +02:00
wb9688
dc29d87962
Extract YouTube search suggestions
2020-04-01 10:30:51 +02:00
wb9688
c852b13d5a
Add Referer header so that it also works with HttpsUrlConnection
2020-04-01 10:30:51 +02:00
wb9688
eb48524411
Add tests for YouTube Music search
2020-04-01 10:30:51 +02:00
wb9688
2b9b2a78e8
Handle 100+ items in playlist
2020-04-01 10:30:51 +02:00
wb9688
fb9b9691b7
Improve getYoutubeMusicKeys()
2020-04-01 10:30:51 +02:00
wb9688
4ddbdf0aee
Disable artist search for now
2020-04-01 10:30:51 +02:00
wb9688
1762a527c9
Add support for YouTube Music search
2020-04-01 10:30:51 +02:00
Tobias Groza
04af575764
Hey Travis, deploy the docs
2020-03-30 00:10:54 +02:00
Fynn Godau
a1523eb293
Update Bandcamp service to latest interface changes
2020-03-17 20:40:25 +01:00
fynngodau
b78f788017
Merge branch 'dev' into dev
2020-03-17 18:03:14 +01:00
Fynn Godau
36a316eea5
Merge branch 'dev' into dev
2020-02-08 09:43:13 +01:00
Fynn Godau
623e6a8c63
Merge branch 'dev' into dev
2020-01-17 22:37:42 +01:00
Fynn Godau
7b5702bdc2
Add bandcamp to list of supported services in README
2020-01-11 16:32:45 +01:00
Fynn Godau
ba967f1a15
Workaround enourmous load times for long bandcamp playlists
...
Additionally, get cover art from json instead of html
2020-01-03 13:17:04 +01:00
Fynn Godau
46e1f3922c
Refuse to load bandcamp playlists without content
2020-01-03 12:32:22 +01:00
Fynn Godau
f0d36dfa7e
Bandcamp radio: proper name, date shortened
2020-01-03 12:28:49 +01:00
Fynn Godau
808c8aa087
Add bandcamp weekly kiosk
...
Unfortunately a little messy
2020-01-03 12:28:49 +01:00
Fynn Godau
13ef11e9c8
Bandcamp featured kiosk
2020-01-03 12:28:49 +01:00
Fynn Godau
db02850d4f
Bandcamp playlist extractor: support playlists with just one stream
2020-01-03 12:28:49 +01:00
Fynn Godau
13e4908b83
Bandcamp search extractor: read track count from result page
2020-01-03 12:28:49 +01:00
Fynn Godau
9b16baffb7
Bandcamp search: multiple pages
2020-01-03 12:28:49 +01:00
Fynn Godau
ce2a88e56f
Add bandcamp search suggestion extractor
2020-01-03 12:28:49 +01:00
Fynn Godau
e12ddaef7f
Bandcamp playlist and search extractors: better and more tests
2020-01-03 12:28:49 +01:00
Fynn Godau
655df356a2
Various improvements concerning bandcamp playlists
2020-01-03 12:28:49 +01:00
Fynn Godau
ba700bfb3e
Add bandcamp playlists (albums)
2020-01-03 12:28:49 +01:00
Fynn Godau
d05b14ae48
Add channels (artists) to search results
2020-01-03 12:28:49 +01:00
Fynn Godau
794ca5eeae
Bandcamp channel extractor: handel nonexistent images better
2020-01-03 12:28:49 +01:00
Fynn Godau
91c0ec7cea
Bandcamp stream extractor: catch NullPointerException
2020-01-03 12:28:49 +01:00
Fynn Godau
d5cdc20be1
Bandcmap channel extractor: fix getting banner url
2020-01-03 12:28:49 +01:00
Fynn Godau
7730eb2ea1
Bandcamp channel extractor: enforce https when getting banner url
2020-01-03 12:28:49 +01:00
Fynn Godau
a42c77425d
Bandcamp channel extractor (ignoring everything but tracks)
2020-01-03 12:28:49 +01:00
Fynn Godau
5281456899
Bandcamp channel link handler factory
2020-01-03 12:28:49 +01:00
Fynn Godau
c3d127ccd9
Bandcamp audio stream test and fix
2020-01-03 12:28:49 +01:00
Fynn Godau
80d67e22b7
Return mp3-128 audio stream [experimental]
2020-01-03 12:28:49 +01:00
Fynn Godau
43dc3c3d4c
Return empty kiosk list instead of null
2020-01-03 12:28:49 +01:00
Fynn Godau
a579337c9a
Bandcamp service with support for streams and searches
2020-01-03 12:28:49 +01:00