TobiGr
fafd471606
[PeerTube] Fix test for like count
...
Number changed
2024-05-08 19:25:12 +02:00
TobiGr
4f477ad72b
[PeerTube] Fix testing comment content
...
The comment is not available anymore.
2024-05-08 19:25:12 +02:00
Tobi
964e429978
Merge pull request #1166 from TeamNewPipe/dependabot/github_actions/peaceiris/actions-gh-pages-4
...
Bump peaceiris/actions-gh-pages from 3 to 4
2024-05-08 19:24:40 +02:00
dependabot[bot]
10c6965a28
Bump peaceiris/actions-gh-pages from 3 to 4
...
Bumps [peaceiris/actions-gh-pages](https://github.com/peaceiris/actions-gh-pages ) from 3 to 4.
- [Release notes](https://github.com/peaceiris/actions-gh-pages/releases )
- [Changelog](https://github.com/peaceiris/actions-gh-pages/blob/main/CHANGELOG.md )
- [Commits](https://github.com/peaceiris/actions-gh-pages/compare/v3...v4 )
---
updated-dependencies:
- dependency-name: peaceiris/actions-gh-pages
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-05-08 17:16:08 +00:00
TobiGr
e54f38f5e7
[PeerTube] Fix test
...
UploaderName was changed by user
2024-05-08 19:13:49 +02:00
Tobi
5dd5c7a65b
Merge pull request #1173 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.8.5
...
Bump com.github.spotbugs:spotbugs-annotations from 4.8.3 to 4.8.5
2024-05-08 19:12:40 +02:00
Tobi
37438ff82f
Merge pull request #1174 from TeamNewPipe/dependabot/gradle/org.mozilla-rhino-1.7.15
...
Bump org.mozilla:rhino from 1.7.13 to 1.7.15
2024-05-08 19:02:57 +02:00
TobiGr
bba3b6c69b
version 0.24.0
2024-05-08 13:01:06 +02:00
TobiGr
b40a5784ed
Fix JDoc
...
YoutubePostLiveStreamDvrDashManifestCreator.java:90: error: unexpected end tag: </p>
2024-05-08 12:41:36 +02:00
dependabot[bot]
c6da4004e2
Bump org.mozilla:rhino from 1.7.13 to 1.7.15
...
Bumps [org.mozilla:rhino](https://github.com/mozilla/rhino ) from 1.7.13 to 1.7.15.
- [Release notes](https://github.com/mozilla/rhino/releases )
- [Changelog](https://github.com/mozilla/rhino/blob/master/RELEASE-NOTES.md )
- [Commits](https://github.com/mozilla/rhino/commits )
---
updated-dependencies:
- dependency-name: org.mozilla:rhino
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-05-06 09:43:05 +00:00
dependabot[bot]
f26e84d39f
Bump com.github.spotbugs:spotbugs-annotations from 4.8.3 to 4.8.5
...
Bumps [com.github.spotbugs:spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.8.3 to 4.8.5.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.8.3...4.8.5 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-05-06 09:43:03 +00:00
Tobi
ec3e8378c6
Merge pull request #1171 from TeamNewPipe/fix/jdoc
...
Fix JavaDocs and add Checkstyle exception for line length in JavaDocs' links
2024-04-23 21:22:12 +02:00
TobiGr
8d2a7a5281
[PeerTube] Fix test
2024-04-23 20:02:29 +02:00
TobiGr
7c29dbc965
Fix JDoc
...
Add Checkstyle exception for LineLength in JDoc links.
See https://github.com/TeamNewPipe/NewPipeExtractor/actions/runs/8804403691/job/24164703883
2024-04-23 19:55:51 +02:00
Stypox
fbe9e6223a
Merge pull request #1168 from AudricV/yt_upd-cver-rm-keys-and-do-fixes
...
[YouTube] Update clients versions, restore access to some streams and more
2024-04-20 11:54:24 +02:00
Stypox
4e9e7cb29c
Improve assertTabsContain() to also check size
2024-04-20 11:48:36 +02:00
Stypox
9d0dd36034
[YouTube] Create constants for client names/versions
2024-04-20 11:43:54 +02:00
Stypox
d4e6d22e64
[YouTube] Improve meta info code for review
2024-04-20 11:43:08 +02:00
AudricV
74bf000473
[YouTube] Update mocks
2024-04-11 18:59:11 +02:00
AudricV
f9792cf3a9
[YouTube] Fix InteractiveTabbedHeader.testTabs test
...
YouTube now returns a Shorts tab for InteractiveTabbedHeader gaming channels,
which contains Shorts about the game of the topic channel but are not uploaded
on the game topic channel.
As this tab is already supported by the extractor, fetching a gaming topic
channel now returns a tab instead of none.
Channel name and channel URL of these Shorts needs to be set to null in a
separate commit, as Shorts on this tab do not have the topic channel as their
uploader.
2024-04-11 18:59:10 +02:00
AudricV
f40fc0aa4f
[YouTube] Add support for new crisis meta info action data
...
The new action data can return multiple contact actions instead of only one,
which will be concatenated by a new line return.
This commit fixes tests of the CrisisResources test class of
YoutubeSearchExtractorTest.
2024-04-11 18:59:09 +02:00
AudricV
2a3c6f80d2
[YouTube] Fix YoutubeStreamExtractorRelatedMixTest
2024-04-10 21:19:03 +02:00
AudricV
657b4377aa
[YouTube] Fix YoutubeStreamExtractorDefaultTest tests
2024-04-10 21:19:03 +02:00
AudricV
7bf50bf1cb
[YouTube] Update Android client player parameters
...
YouTube disabled the effectiveness of the parameters which were used (the
player response we get redirects to another video), but new parameters which
work around Android's client integrity checks have been found.
2024-04-10 21:19:03 +02:00
AudricV
27dc1b1f50
[YouTube] Remove usage of API keys for InnerTube requests, bump versions
...
The API keys are not used anymore by official clients in almost all cases
(still used by the Android app until it gets a configuration) for all requests
we made.
Clients and device OS versions have been bumped to their latest stable version
known.
Methods and fields related to API keys have been renamed or deleted if they're
no longer relevant.
2024-04-10 21:19:02 +02:00
AudricV
e380bb4bc3
[YouTube] Add missing prettyPrint query parameter to mixes continuations
2024-04-10 19:06:36 +02:00
Audric V
6c3c2e25d7
Merge pull request #1163 from AudricV/yt-fix_comments_extraction
...
[YouTube] Support new comments data
2024-04-10 18:19:59 +02:00
Stypox
02274d5395
[YouTube] Avoid XSS attacks in description or comments
2024-04-08 11:21:31 +02:00
Stypox
3f7b2653e3
[YouTube] Add YoutubeDescriptionHelperTest
2024-04-08 11:21:31 +02:00
Stypox
a90237816a
[YouTube] Cleanup description helper
...
Remove unneeded isClose field, and make constants private
2024-04-08 11:21:31 +02:00
Stypox
b80c3f5d51
[YouTube] Replace link text with accessibility label
2024-04-08 00:14:28 +02:00
Stypox
09732d6785
[YouTube] Add support for styles in attributed descriptions
...
Also refactor descriptions parsing.
2024-04-04 21:14:27 +02:00
AudricV
293c3e9e47
[YouTube] Support new A/B tested comments data
...
Also improve current comments code by removing outdated comment
renderer data.
2024-04-04 21:14:26 +02:00
Stypox
e5b30ae8c3
Merge pull request #1151 from Profpatsch/localization-return-optional
...
LocaleCompat.forLanguageTag: return Optional if parsing fails
2024-03-29 13:50:48 +01:00
Stypox
23fc7aa209
Throw ParsingException instead of IllegalArg
2024-03-29 13:44:42 +01:00
Stypox
fb468a23f4
Merge pull request #1142 from TeamNewPipe/peertube-v6
...
[PeerTube] Add support for PeerTube v6 features
2024-03-29 12:25:38 +01:00
Stypox
6589e2c15d
Merge pull request #1148 from Stypox/mediaccc-channel-tab-handler
...
[MediaCCC] Allow obtaining channel tab link handler
2024-03-28 13:45:05 +01:00
Tobi
ad71864b23
Merge pull request #1162 from Stypox/fix-comment-description-npe
...
Make getCommentText @Nonnull
2024-03-27 20:04:17 +01:00
Stypox
c57016b79b
Make getCommentText @Nonnull
2024-03-27 15:26:06 +01:00
Tobi
adcc1f17ee
Merge pull request #1160 from TeamNewPipe/fix/tests
...
Fix some failing unit tests and detect new account termination messages for YouTube
2024-03-20 15:17:55 +01:00
TobiGr
51ddacc81d
[SoundCloud] Fix SoundcloudSearchExtractorTest.NoNextPage
...
Search did not return no item at all, causing a NothingFoundException. New search query yields three items on first page
2024-03-20 15:10:39 +01:00
TobiGr
8392d50ba6
Update mocks for YoutubeChannelExtractorTest.NotAvailable
2024-03-20 14:59:44 +01:00
TobiGr
aaccfecda8
[YouTube] Detect new account termination messages
2024-03-20 14:57:41 +01:00
TobiGr
73f0c63a9d
[PeerTube] Fix tests for "What is PeerTube?" video
2024-03-20 14:44:06 +01:00
Tobi
896a55e319
Merge pull request #1139 from TeamNewPipe/dependabot/github_actions/actions/upload-artifact-4
...
Bump actions/upload-artifact from 3 to 4
2024-03-18 08:59:35 +01:00
dependabot[bot]
e58fc652e0
Bump actions/upload-artifact from 3 to 4
...
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact ) from 3 to 4.
- [Release notes](https://github.com/actions/upload-artifact/releases )
- [Commits](https://github.com/actions/upload-artifact/compare/v3...v4 )
---
updated-dependencies:
- dependency-name: actions/upload-artifact
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-03-18 07:53:24 +00:00
Tobi
e3f2c9aec7
Merge pull request #1153 from TeamNewPipe/dependabot/github_actions/actions/cache-4
...
Bump actions/cache from 3 to 4
2024-03-18 08:51:24 +01:00
Tobi
6b0fc14c04
Merge pull request #1156 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.10.2
...
Bump org.junit:junit-bom from 5.10.0 to 5.10.2
2024-03-18 08:46:13 +01:00
dependabot[bot]
d579b608e5
Bump org.junit:junit-bom from 5.10.0 to 5.10.2
...
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5 ) from 5.10.0 to 5.10.2.
- [Release notes](https://github.com/junit-team/junit5/releases )
- [Commits](https://github.com/junit-team/junit5/compare/r5.10.0...r5.10.2 )
---
updated-dependencies:
- dependency-name: org.junit:junit-bom
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-02-05 09:15:26 +00:00
TobiGr
fe47a4311f
[PeerTube] Add test for segments and framesets
2024-01-29 10:22:06 +01:00
TobiGr
15e0e74b48
[PeerTube] Add support for stream frames/storyboards extraction
...
Implement PeerTubeStreamExtractor.getFrames()
2024-01-29 10:22:06 +01:00
dependabot[bot]
da04eded5d
Bump actions/cache from 3 to 4
...
Bumps [actions/cache](https://github.com/actions/cache ) from 3 to 4.
- [Release notes](https://github.com/actions/cache/releases )
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md )
- [Commits](https://github.com/actions/cache/compare/v3...v4 )
---
updated-dependencies:
- dependency-name: actions/cache
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-01-18 09:32:42 +00:00
Profpatsch
7408173246
LocaleCompat.forLanguageTag: return Optional if parsing fails
...
It’s not obvious that the function will fail in some cases and throw
an `IllegalArgumentException`.
So instead of just failing if parsing fails, return an Optional that
all callers have to decide what to do (e.g. the YoutubeExtractor can
just ignore the locale in that case, like it does with most other
fields in the json if they are unexpected).
2024-01-07 14:31:34 +01:00
Stypox
aaf3231fc7
[MediaCCC] Fix lambda link handler keeping reference to extractor
...
This caused problems in NewPipe, because extractors are not serializable, and well, keeping references to them is a bad idea anyway.
2023-12-30 23:23:19 +01:00
Stypox
137e924035
[MediaCCC] Add ChannelTabExtractorTest
2023-12-30 22:53:51 +01:00
Stypox
cc9ade962e
[MediaCCC] Allow obtaining channel tab extractor from scratch
...
i.e. without needing to pass through the conference/channel extractor
This was needed because clients (like NewPipe) might rely on link handlers to hold as little data as possible, since they might be kept around for long or passed around in system transactions, so this commit allows obtaining a standalone link handler that does not hold a JsonObject within itself.
2023-12-30 22:53:27 +01:00
Stypox
3402cdb666
Merge pull request #1147 from petlyh/youtube-albums
...
[YouTube] Add Albums channel tab
2023-12-30 21:58:19 +01:00
petlyh
6dc25f7b97
[YouTube] Add Albums channel tab mocks
2023-12-30 14:46:39 +01:00
petlyh
4408e2d0ac
[YouTube] Add Albums channel tab
2023-12-30 14:01:30 +01:00
TobiGr
9ab932e394
Rename testDoNotAcceptNonURLs() -> assertDoNotAcceptNonURLs()
2023-12-29 16:38:11 +01:00
Stypox
9d66debf3c
Merge pull request #1132 from TeamNewPipe/dependabot/github_actions/actions/setup-java-4
...
Bump actions/setup-java from 3 to 4
2023-12-29 16:20:46 +01:00
Tobi
038ebdedc4
Merge pull request #1143 from petlyh/peertube-non-urls
...
Avoid PeerTube accepting non-URLs
2023-12-29 12:47:30 +01:00
TobiGr
61d237de02
[PeerTube] Test onAccept(String URL) in LinkHandlerFactories for non-URLs
2023-12-29 12:45:02 +01:00
petlyh
2b2c1546d1
Avoid PeerTube accepting non-URLs
2023-12-29 12:27:39 +01:00
Tobi
1e93b1dc20
Merge pull request #1135 from Stypox/yt-emergency-info
...
[YouTube] Implement emergency meta info
2023-12-29 12:01:40 +01:00
dependabot[bot]
3400af99b3
Bump actions/setup-java from 3 to 4
...
Bumps [actions/setup-java](https://github.com/actions/setup-java ) from 3 to 4.
- [Release notes](https://github.com/actions/setup-java/releases )
- [Commits](https://github.com/actions/setup-java/compare/v3...v4 )
---
updated-dependencies:
- dependency-name: actions/setup-java
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-12-29 10:57:47 +00:00
Tobi
1f8a044462
Merge pull request #1138 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.8.3
...
Bump com.github.spotbugs:spotbugs-annotations from 4.8.0 to 4.8.3
2023-12-29 11:56:18 +01:00
dependabot[bot]
1470aa7303
Bump com.github.spotbugs:spotbugs-annotations from 4.8.0 to 4.8.3
...
Bumps [com.github.spotbugs:spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.8.0 to 4.8.3.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.8.0...4.8.3 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-12-29 10:53:28 +00:00
TobiGr
8f9ebdcb77
[PeerTube] Fix failing PeertubeTrendingLinkHandlerFactoryTest
...
The factory was updated in #1144
2023-12-29 11:52:19 +01:00
Stypox
1553931027
Merge pull request #1145 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.17.2
...
Bump org.jsoup:jsoup from 1.16.2 to 1.17.2
2023-12-29 11:27:01 +01:00
Stypox
b2ec1b15fb
Merge pull request #1144 from dragfyre/patch-1
...
Update PeertubeTrendingLinkHandlerFactory.java
2023-12-29 11:21:08 +01:00
dependabot[bot]
151ee99da3
Bump org.jsoup:jsoup from 1.16.2 to 1.17.2
...
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup ) from 1.16.2 to 1.17.2.
- [Release notes](https://github.com/jhy/jsoup/releases )
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES.md )
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.16.2...jsoup-1.17.2 )
---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-12-29 09:47:00 +00:00
dragfyre
65e7bc5b95
Update PeertubeTrendingLinkHandlerFactory.java
...
correcting Peertube local trending api URL (per #10685 in main NewPipe repo); see https://docs.joinpeertube.org/api-rest-reference.html#tag/Video/operation/getVideos
2023-12-28 14:50:31 +07:00
Stypox
f276caf54a
Release v0.23.1
2023-12-21 21:59:21 +01:00
Stypox
fc54fb2fdb
Merge pull request #1140 from Stypox/yt-shorts-no-duration
...
[YouTube] Always return -1 as duration of Shorts returned inside reel items
2023-12-21 21:52:40 +01:00
Stypox
0518487d26
Fix SearchInfo's non-null MetaInfo being null when initialized or when an extraction error occurs ( #1141 )
...
The meta info might have been null either when SearchInfo is first initialized, or when extractor.getMetaInfo() throws an exception in getInfo().
This caused NewPipe to crash instead of showing a nice error in https://www.reddit.com/r/youtube/comments/184ttmw/what_exactly_about_blue_whales_has_youtube_so/ .
2023-12-21 21:39:25 +01:00
Stypox
5b59a1a8c5
[YouTube] Move meta info extraction to separate file
...
YoutubeParsingHelper was longer than 2000 lines which caused checkstyle issues
2023-12-21 21:19:08 +01:00
Stypox
b8e12dd76c
[YouTube] Implement emergency meta info
...
YouTube provides that meta info panel when users search for really sensitive content like suicide (e.g. "blue whale").
It contains:
- an encouragement as title (e.g. "We are with you")
- a phone number as action
- details about how to call the phone number (e.g. availability)
- an url pointing to the website of an association
Also add a test that just checks if a meta info is properly extracted
2023-12-21 21:19:08 +01:00
Stypox
83c1737f70
[YouTube] Update shorts channel tab mocks
2023-12-21 20:54:16 +01:00
Stypox
2938067c2c
[YouTube] Shorts don't provide a duration anymore
2023-12-21 20:41:01 +01:00
Stypox
91419ec6e8
Release v0.23.0
2023-12-10 15:59:32 +01:00
Audric V
678c98f24c
Merge pull request #1127 from AudricV/yt_improvements-and-fixes
...
[YouTube] Make some improvements and fixes
2023-12-09 14:07:31 +01:00
AudricV
ec0194cfbf
[YouTube] Update mocks
2023-12-08 21:46:52 +01:00
AudricV
00a0f1a103
[YouTube] Add a blocking crisis resources bypass in search results test
...
This test only tests that search results are returned, when no content filters
are provided and crisis resources blocking search results should be returned.
Searches with blocking crisis resources and content filters should work too, as
the bypass has been implemented for them.
2023-12-08 21:46:49 +01:00
AudricV
06838d7245
[YouTube] Fix YoutubeSearchQHTest tests
...
As search parameters to bypass crisis resources blocking search results have
been implemented, they need to be added to search tests, in order to pass
them.
2023-12-08 21:46:49 +01:00
AudricV
642bb01388
[YouTube] Fix testSubscriberCount method of YoutubeChannelExtractorTest.CarouselHeader
...
The subscriber count is now lower than the expected count as some people
unsubscribed to the Sports system channel. The expected count has been so
lowered.
2023-12-08 21:46:49 +01:00
AudricV
bedc9e5bc0
[YouTube] Remove Channels channel tab in tests
2023-12-08 21:46:48 +01:00
AudricV
5fa22ae25b
[YouTube] Enable and change testRelatedItems method of YoutubePlaylistExtractorTest.LearningPlaylist
...
This change makes the method test whether there is no more items instead of
being disabled.
2023-12-08 21:46:48 +01:00
AudricV
29dc7625f2
[YouTube] Fix YoutubeSearchExtractorTest.Suggestion
...
YouTube doesn't return anymore a suggestion for the query "algorythm", but does
for the query "on board ing" ("on boarding"). This search query is now used and
had to be URL-encoded.
URL encoding in the complete YoutubeSearchExtractorTest test class uses now
extractor's Util class instead of Java's URLDecoder class directly.
2023-12-08 21:46:48 +01:00
AudricV
56ab35423e
[YouTube] Fix potential NullPointerException in YoutubeSearchExtractor.getSearchSuggestion
2023-12-08 21:46:48 +01:00
AudricV
9dc1eab28c
[YouTube] Fix expected upload dates of StreamExtractor tests
...
YouTube is rolling out or A/B testing a new date format returned inside player
responses, which are precise to the second instead of the day.
This commit makes the StreamExtractor tests use these more precise dates.
2023-12-08 21:46:48 +01:00
AudricV
ad3d187ac7
[YouTube] Fix testRelatedItems method of YoutubeStreamExtractorRelatedMixTest
...
This commit fixes the testRelatedItems test method by:
- accepting consent in the test class, in order to extract mixes in
recommendations;
- removing assertion of a music mix inside the recommendations, as YouTube
doesn't seem to return such mixes anymore, at least for the video used in the
test class.
2023-12-08 21:46:47 +01:00
AudricV
e111814401
[YouTube] Fix YoutubeStreamExtractorDefaultTest.RatingsDisabledTest
...
Replace the video used in this test class with another one publicly available
and update the corresponding expected test values.
The test class's mocks will be updated in a different commit.
2023-12-08 21:46:47 +01:00
AudricV
fc45941ead
[YouTube] Fix YoutubeChannelExtractorTest tests
...
- Change CarouselHeader test channel to Sports system one, as the Coachella one
doesn't return this channel header anymore;
- Fix InteractiveTabbedHeader test by checking whether the test's channel
description is not empty instead of containing some words, as it is changing
frequently.
2023-12-08 21:46:47 +01:00
AudricV
0bcb241c38
[YouTube] Fix expected data in YoutubeStreamExtractorRelatedMixTest
...
Video's title and tags have been changed by its uploader, so they have to be
updated.
Also make some package-private constants private, as they are not used outside
of the class, and remove unneeded test overrides.
2023-12-08 21:46:47 +01:00
AudricV
6ba8251be1
[YouTube] Bypass crisis resources blocking search results
...
These crisis resources are preventing search results to be returned. See
https://support.google.com/youtube/answer/10726080?hl=en for more info on them.
This commit changes search parameters to include the property allowing to show
search results.
2023-12-08 21:46:47 +01:00
AudricV
7dea2d0d27
[YouTube] Remove Channels channel tab support
...
This tab has been removed by YouTube.
2023-12-08 21:46:47 +01:00
AudricV
3782d9a02a
[YouTube] Support new A/B tested like data and avoid like count conversion from integer to long
...
Also make minor improvements to current like data extraction and remove
previous like count data support, as it is not returned anymore.
2023-12-08 21:46:46 +01:00
AudricV
b71ce1123f
[YouTube] Extract only search results corresponding to a search type
...
YouTube returns sometimes videos inside channel search results. As we only want
results corresponding to the type we requested, this commits makes
YoutubeSearchExtractor ignoring non-requested search results we get, using the
extractor LinkHandler's first content filter value.
Also remove an unneeded exception throwing declaration in
YoutubeSearchExtractor.
2023-12-08 21:46:46 +01:00
AudricV
ff8ed7247f
[YouTube] Switch to new consent cookie
...
Also move the documentation of the consent in its setter method in order to be
accessible publicly and improve it.
2023-12-08 21:46:46 +01:00
AudricV
ec838d7421
[YouTube] Add missing prettyPrint query parameter to some test InnerTube requests
...
This query parameter for which its value is set to false was not added to two
requests made in test classes of YoutubeMixPlaylistExtractorTest.
Also remove an unneeded ParsingException exception throwing declaration in a
test method.
2023-12-08 21:46:46 +01:00
AudricV
2c941794c0
[YouTube] Add utcOffsetMinutes to all InnerTube payloads
...
This should make returned dates consistent between timezones and countries on
which the extractor is ran.
It was previously only set on YouTube Music search continuations.
2023-12-08 21:46:46 +01:00
AudricV
d97c9e0db1
[YouTube] Improve payloads and URLs of InnerTube requests
...
For every InnerTube request:
- Always add a `request` object with the following properties:
- "internalExperimentFlags" set to an empty array;
- "useSsl" set to "true";
- "lockedSafetyMode" set to "false".
- Use proper TODO comment to provide a way to enable restricted mode on every
request and add it on requests on which it wasn't present.
For YouTube Music:
- Remove alt query parameter, as it is not used anymore by the website;
- Add prettyPrint query parameter with false value on YouTube Music search
continuations.
2023-12-08 21:46:45 +01:00
AudricV
8a9ebcc373
[YouTube] Update InnerTube clients' version and devices' OS version and model
2023-12-08 21:46:45 +01:00
Tobi
eac850ca10
Merge pull request #1114 from FineFindus/feat/comment-author-is-owner
...
[YouTube] Add channel owner to comments
2023-10-25 09:51:12 +02:00
Tobi
5ab1f784e8
Merge pull request #1117 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.16.2
...
Bump org.jsoup:jsoup from 1.16.1 to 1.16.2
2023-10-21 19:26:36 +02:00
dependabot[bot]
9d7bcba050
Bump org.jsoup:jsoup from 1.16.1 to 1.16.2
...
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup ) from 1.16.1 to 1.16.2.
- [Release notes](https://github.com/jhy/jsoup/releases )
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES )
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.16.1...jsoup-1.16.2 )
---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-10-20 09:13:21 +00:00
dependabot[bot]
e26065148a
Bump com.github.spotbugs:spotbugs-annotations from 4.7.3 to 4.8.0
...
Bumps [com.github.spotbugs:spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.7.3 to 4.8.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.3...4.8.0 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-10-12 14:19:10 +02:00
FineFindus
34b05a0dda
feat(youtube/comments): support creator replies
2023-10-09 16:33:43 +02:00
TobiGr
0821f09114
Add missing mocks
2023-10-09 16:33:43 +02:00
FineFindus
c1784a4bdb
[YouTube] Add channel owner to comments
2023-10-09 16:33:43 +02:00
TobiGr
f9846352ea
Fix wrong `@Nullable` annotation
2023-10-09 16:02:57 +02:00
Tobi
d6f5cba6e2
Merge pull request #1111 from FineFindus/feat/creator-reply
...
Add `hasCreatorReply()` to CommentsInfoItem
2023-10-09 12:45:56 +02:00
TobiGr
9d63c75623
Add missing mocks
2023-10-09 11:24:39 +02:00
TobiGr
d49f8411d7
[PeerTube] Implement CommentsInfoItemExtractor.hasCreatorReply()
2023-10-09 02:47:12 +02:00
Stypox
bb132167d5
Merge pull request #1113 from AudricV/snd_fix-non-jpg-images
...
[SoundCloud] Fix extraction of non-JPG images
2023-10-02 19:40:57 +02:00
AudricV
c98695fcea
[SoundCloud] Fix extraction of non-JPG images
...
Default image qualities were removed in image URLs with the jpg extension,
causing the addition of the image suffix to full non-JPG images URLs and so to
invalid image URLs.
Only the image quality name with its leading "-" character and the "."
character after the name is now removed and replaced by a string format
replaced itself with the image quality name for each quality.
As the image suffixes do not contain the image extension, the name of image
qualities lists has been adapted with these changes and some related comments
have been also improved.
2023-10-01 20:33:25 +02:00
AudricV
ac00459c1a
Change requirement of image extensions in ImageSuffix class' Javadoc to a possibility
...
Some services may provide different image formats using the same suffix,
without we know what format the service provide. Enforcing an image extension
could so lead to provide invalid image URLs, like for SoundCloud PNG images
currently.
With this documentation change, it is now clear that users of this class decide
of whether they want to include image extensions in the suffix. The previous
behavior described in the Javadoc was not enforced.
2023-09-30 21:11:09 +02:00
FineFindus
dd7b2d9798
feat(youtube/comments): support creator replies
2023-09-25 10:40:45 +02:00
Youssif Shaaban Alsager
917554acc4
[YouTube] Add support for ultralow audio formats ( #1063 )
2023-09-24 19:04:34 +02:00
Tobi
8b0068f8f4
Merge pull request #1110 from christian-2hu/chore/update-copyright
...
chore: Update copyright notices
2023-09-23 00:29:19 +02:00
Christian
fc67d49f59
Update copyright notices
...
Update copyright notices to comply to GPLv3 and change NewPipe to NewPipe Extractor on some notices that were not updated.
2023-09-22 19:10:15 -03:00
Stypox
289db1178a
Merge pull request #1108 from AudricV/yt_refactor-js-usage
...
[YouTube] Refactor JavaScript usage and fix extraction of obfuscated signature deobfuscation function
2023-09-22 10:41:57 +02:00
AudricV
6ed22099a2
[YouTube] Update stream mocks
2023-09-21 21:59:34 +02:00
AudricV
714b141ecb
[YouTube] Catch any exception when extracting something from JavaScript's base player
2023-09-21 21:59:33 +02:00
AudricV
588c6a8422
[YouTube] Quote signature deobfuscation function name and add semicolon only where needed
2023-09-21 21:59:33 +02:00
AudricV
1fa85ec6ca
[YouTube] Add tests for signature timestamp extraction and signature deobfuscation function extraction and execution
2023-09-21 21:59:33 +02:00
AudricV
a04bc320de
[YouTube] Convert signature timestamp to integer
...
The signature timestamp is used as a number by HTML5 clients, so it should be
used in the same way by the extractor too instead of being a string.
As the timestamp doesn't seem to exceed 5 digits, an integer is used to store
its value.
2023-09-21 21:59:32 +02:00
AudricV
7de3753a81
[YouTube] Refactor JavaScript player management API
...
This commit is introducing breaking changes.
For clients, everything is managed in a new class called
YoutubeJavaScriptPlayerManager:
- caching JavaScript base player code and its extracted code (functions and
variables);
- getting player signature timestamp;
- getting deobfuscated signatures of streaming URLs;
- getting streaming URLs with a throttling parameter deobfuscated, if
applicable.
The class delegates the extraction parts to external package-private classes:
- YoutubeJavaScriptExtractor, to extract and download YouTube's JavaScript base
player code: it always already present before and has been edited to mainly
remove the previous caching system and made it package-private;
- YoutubeSignatureUtils, for player signature timestamp and signature
deobfuscation function of streaming URLs, added in a recent commit;
- YoutubeThrottlingParameterUtils, which was originally
YoutubeThrottlingDecrypter, for throttling parameter of streaming URLs
deobfuscation function and checking whether this parameter is in a streaming
URL.
YoutubeJavaScriptPlayerManager caches and then runs the extracted code if it
has been executed successfully. The cache system of throttling parameters
deobfuscated values has been kept, its size can be get using the
getThrottlingParametersCacheSize method and can be cleared independently using
the clearThrottlingParametersCache method.
If an exception occurs during the extraction or the parsing of a function
property which is not related to JavaScript base player code fetching, it is
stored until caches are cleared, making subsequent failing extraction calls of
the requested function or property faster and consuming less resources, as the
result should be the same until the base player code changes.
All caches can be reset using the clearAllCaches method of
YoutubeJavaScriptPlayerManager.
Classes using JavaScript base player code and utilities directly (in the code
and its tests) have been also updated in this commit.
2023-09-21 21:59:32 +02:00
AudricV
6884d191cd
[YouTube] Add utility class around signatures and fix signature deobfuscation function extraction
...
The goal of this class is to decouple the extraction of signature timestamp and
signature deobfuscation function from YoutubeStreamExtractor.
The extraction of the signature deobfuscation function has been also adapted to
support the latest YouTube player versions.
This new class, YoutubeSignatureUtils, doens't store anything temporary such as
a copy of the player code, which has to be passed where required. It is not
public, as it will be used by a JavaScript player manager class in the future,
in order to handle in a better way fetching, caching and resetting cache of the
player code.
2023-09-21 21:59:26 +02:00
Tobi
3be76a6406
Merge pull request #1107 from Isira-Seneviratne/Locale_forLanguageTag
...
Use Locale.forLanguageTag() in tests
2023-09-18 16:48:30 +02:00
TobiGr
17790328cd
Improve doc
2023-09-18 16:44:51 +02:00
Isira Seneviratne
4bc8ae7812
Use Locale.forLanguageTag() in tests
2023-09-18 08:59:13 +05:30
Tobi
90aed06a63
Merge pull request #1105 from TeamNewPipe/fix/bandcamp-streame-extractor-test
...
[Badcamp] Fix StreamExtractorTest
2023-09-18 01:49:04 +02:00
TobiGr
cf49f4a31c
[Badcamp] Fix StreamExtractorTest
...
The song was renamed and the URL changed
2023-09-17 23:58:07 +02:00
Tobi
7c7ceaceab
Merge pull request #1103 from TeamNewPipe/dependabot/github_actions/actions/checkout-4
...
Bump actions/checkout from 3 to 4
2023-09-17 22:42:01 +02:00
dependabot[bot]
72c475d944
Bump actions/checkout from 3 to 4
...
Bumps [actions/checkout](https://github.com/actions/checkout ) from 3 to 4.
- [Release notes](https://github.com/actions/checkout/releases )
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md )
- [Commits](https://github.com/actions/checkout/compare/v3...v4 )
---
updated-dependencies:
- dependency-name: actions/checkout
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-09-05 10:00:13 +00:00
Stypox
1f08d28ae5
Merge pull request #889 from AudricV/multiple-images-support
...
Multiple images support
2023-08-13 11:35:11 +02:00
AudricV
e8bfd20170
[MediaCCC] Apply changes in extractor tests
...
Also remove some public test methods modifiers.
2023-08-12 22:56:33 +02:00
AudricV
0292c4f3e8
[Bandcamp] Apply changes in extractor tests
...
Also remove some public test methods modifiers, add missing Test annotations on
old Junit 4 tests (and update them if needed), and use final in some places
where it was possible.
BandcampChannelExtractorTest.testLength has been removed as the test is always
true.
2023-08-12 22:56:32 +02:00
AudricV
2578f22054
[Bandcamp] Add utility test method to test images
...
This method, testImages(Collection<Image>), will use first the default image
collection test in DefaultTests and then will check that each image URL
contains f4.bcbits.com/img and ends with .jpg or .png.
To do so, a new non-instantiable final class has been added: BandcampTestUtils.
2023-08-12 22:56:32 +02:00
AudricV
ba5315c72d
[PeerTube] Apply changes in extractor tests
...
Also remove some public test methods modifiers, add missing Test annotations on
old Junit 4 tests (and update them if needed), and improve some code.
2023-08-12 22:56:32 +02:00
AudricV
1d72bac53d
[SoundCloud] Apply changes in extractor tests
2023-08-12 22:56:32 +02:00
AudricV
93a210394d
[YouTube] Apply changes in extractor tests
...
Also remove some public test methods modifiers, add missing Test annotations on
old Junit 4 tests (and update them if needed), and use final in some places
where it was possible.
2023-08-12 22:56:31 +02:00
AudricV
2c436d428c
[YouTube] Add utility test method to test images in YoutubeTestsUtils
...
This method, testImages(Collection<Image>), will use first the default image
collection test in DefaultTests and then will check that each image URL
contains the string yt.
The JavaDoc of the class has been also updated to reflect the changes made in
it (it is now more general).
2023-08-12 22:56:31 +02:00
AudricV
d381f3b70b
Update avatar, banners and thumbnail methods' name and apply changes in DefaultStreamExtractorTest
2023-08-12 22:56:31 +02:00
AudricV
434e885708
Add utility methods in ExtractorAsserts to check whether a collection is empty and to test image collections
...
Two new methods have been added in ExtractorAsserts to check if a collection is
empty:
- assertNotEmpty(String, Collection<?>), checking:
- the non nullity of the collection;
- its non emptiness (if that's not case, an exception will be thrown using
the provided message).
- assertNotEmpty(Collection<?>), calling assertNotEmpty(String, Collection<?>)
with null as the value of the string argument.
A new one has been added to this assertion class to check the contrary:
assertEmpty(Collection<?>), checking emptiness of the collection only if it is
not null.
Three new methods have been added in ExtractorAsserts as utility test methods
for image collections:
- assertContainsImageUrlInImageCollection(String, Collection<Image>), checking
that:
- the provided URL and image collection are not null;
- the image collection contains at least one image which has the provided
string value as its URL (which is a string) property.
- assertContainsOnlyEquivalentImages(Collection<Image>, Collection<Image>),
checking that:
- both collections are not null;
- they have the same size;
- each image of the first collection has its equivalent in the second one.
This means that the properties of an image in the first collection must be
equal in an image of the second one.
- assertNotOnlyContainsEquivalentImages(Collection<Image>, Collection<Image>),
checking that:
- both collections are not null;
- one of the following conditions is met:
- they have different sizes;
- an image of the first collection has not its equivalent in the second one.
This means that the properties of an image in the first collection must
be not equal in an image of the second one.
These methods will be used by services extractors tests (and default ones) to
test image collections.
2023-08-12 22:56:31 +02:00
AudricV
5158472852
Apply changes in DefaultTests and add utility method to test image lists
...
This new method, defaultTestImageList(List<Image), will check that the image
list is not null.
For each image, it will test that its URL is secure and its height and width
are more than or equal to their relevant unknown constants in the Image class
(HEIGHT_UNKNOWN and WIDTH_UNKNOWN).
2023-08-12 22:56:31 +02:00
AudricV
70fb3aa38e
Update BaseExtractorTests image methods' name
...
Also suppress unused warnings in BaseStreamExtractorTest, like it is done on
other BaseExtractorTests interfaces.
2023-08-12 22:56:30 +02:00
AudricV
e16d521b7b
[MediaCCC] Apply changes in Extractors
...
Also remove usage of the conference logo as the banner of a conference, as it
is a logo and not a banner.
2023-08-12 22:56:30 +02:00
AudricV
306068a63b
[MediaCCC] Apply changes in InfoItemExtractors
2023-08-12 22:56:30 +02:00
AudricV
2f40861428
[MediaCCC] Add utility methods to get image lists from conference logos and streams
...
These three new methods, added in MediaCCCParsingHelper,
getImageListFromImageUrl(String), getThumbnailsFromStreamItem(JsonObject) and
getThumbnailsFromLiveStreamItem(JsonObject) (the last two are based on a common
method, getThumbnailsFromObject(JsonObject, String, String)), return an empty
list if the case no image URL could be extracted.
Images returned have their height and width unknown and a resolution level
depending on the image key of the JSON API response.
2023-08-12 22:56:30 +02:00
AudricV
71cda03c4c
[Bandcamp] Apply changes in Extractors
2023-08-12 22:56:29 +02:00
AudricV
7e01eaac33
[Bandcamp] Apply changes in InfoItemExtractors
2023-08-12 22:56:29 +02:00
AudricV
4b80d737a4
[Bandcamp] Add utility methods to get multiple images
...
Bandcamp images work with image IDs, which provide different resolutions.
Images on Bandcamp are not always squares, and some IDs respect aspect ratios
where some others not.
The extractor will only use the ones which preserve aspect ratio and will not
provide original images, for performance and size purposes.
Because of this aspect ratio preservation constraint, only one dimension will
be known at a time.
The image IDs with their respective dimension used are:
- 10: 1200w;
- 101: 90h;
- 170: 422h;
- 171: 646h;
- 20: 1024w;
- 200: 420h;
- 201: 280h;
- 202: 140h;
- 204: 360h;
- 205: 240h;
- 206: 180h;
- 207: 120h;
- 43: 100h;
- 44: 200h.
(Where w represents the width of the image and h the height of the image)
Note that these dimensions are theoretical because if the image size is less
than the dimensions of the image ID, it will be not upscaled but kept to its
original size.
All these resolutions are stored in a private static list of ThumbnailSuffixes
in BandcampExtractorHelper, in which the methods to get mutliple images have
been added:
- getImagesFromImageUrl(String): public method to get images from an image URL;
- getImagesFromImageId(long, boolean): public method to get images from an
image ID;
- getImagesFromImageBaseUrl(String): private utility method to get images from
the static list of ThumbnailSuffixes from a given image base URL, containing
the path to the image, a "a" letter if it comes from an album, its ID and an
underscore.
Some existing methods have been also edited:
- the documentation of getImageUrl(long, boolean) has been changed to reflect
the Bandcamp images findings;
- getThumbnailUrlFromSearchResult has been renamed to
getImagesFromSearchResult, and a documentation has been added to this method.
The method replaceHttpWithHttps of the Utils class has been also used in
BandcampExtractorHelper instead of doing manually what the method does.
2023-08-12 22:56:29 +02:00
AudricV
4e6fb368bc
[PeerTube] Apply changes in Extractors and remove usages of default avatar picture
...
The default avatar picture was used when no profile picture was found, but it
was removed and split in multiple images.
Thumbnails' size is not known, as this data is not provided by the API.
2023-08-12 22:56:29 +02:00
AudricV
0a6011a50e
[PeerTube] Apply changes in InfoItemExtractors
...
Also lower the visibility of attributes of channels and playlists InfoItems to
private.
2023-08-12 22:56:29 +02:00
AudricV
6f8331524b
[PeerTube] Add utility method to get thumbnails of playlists and videos
...
This method, getThumbnailsFromPlaylistOrVideoItem, has been added in
PeertubeParsingHelper and returns the two image variants for playlists and
videos.
2023-08-12 22:56:28 +02:00
AudricV
81c0d80a54
[PeerTube] Add utility methods to get avatars and banners of accounts and channels
...
Four new static methods have been added in PeertubeParsingHelper to do so:
- two public methods to get the corresponding image type:
getAvatarsFromOwnerAccountOrVideoChannelObject(String, JsonObject) and
getBannersFromAccountOrVideoChannelObject(String, JsonObject);
- two private methods as helper methods: getImagesFromAvatarsOrBanners(String,
JsonObject, String, String) and getImagesFromAvatarOrBannerArray(String,
JsonArray).
2023-08-12 22:56:28 +02:00
AudricV
31da5beb51
[SoundCloud] Apply changes in Extractors
2023-08-12 22:56:28 +02:00
AudricV
a3a74cd566
[SoundCloud] Apply changes in InfoItemExtractors and return track user avatars as uploader avatars in SoundcloudStreamInfoItemExtractor
2023-08-12 22:56:28 +02:00
AudricV
7f818217d2
[SoundCloud] Add utility methods to get images from track JSON objects and image URLs
...
These new public and static methods, added in SoundcloudParsingHelper,
getAllImagesFromArtworkOrAvatarUrl(String) and
getAllImagesFromVisualUrl(String) (which call a common private method,
getAllImagesFromImageUrlReturned(String, List<ImageSuffix>, List<Image>)),
return an unmodifiable list of JPEG images containing almost every image
resolution provided by SoundCloud except the original size and the tiny
resolution (for artworks and avatars, as the image size is 20x20 for artworks
and 18x18 for avatars, so very close to or equal to the t20x20 resolution):
- for artworks and avatars:
- mini: 16x16;
- t20x20: 20x20;
- small: 32x32;
- badge: 47x47;
- t50x50: 50x50;
- t60x60: 60x60;
- t67x67: 67x67;
- large: 100x100;
- t120x120: 120x120;
- t200x200: 200x200;
- t240x240: 240x240;
- t250x250: 250x250;
- t300x300: 300x300;
- t500x500: 500x500.
- for visuals/user banners:
- t1240x260: 1240x260;
- t2480x520: 2480x520.
Duplicated code in two methods of SoundcloudParsingHelper
(getUsersFromApi(ChannelInfoItemsCollector, String) and
getStreamsFromApi(StreamInfoItemsCollector, String, boolean)) has been merged
into one common private method, getNextPageUrlFromResponseObject(JsonObject).
2023-08-12 22:56:28 +02:00
AudricV
266cd1f76b
[YouTube] Apply changes in YoutubeMusicSearchExtractor and split its InfoItemExtractors into separate classes
...
Splitting YoutubeMusicSearchExtractor's InfoItemExtractors into separate
classes (YoutubeMusicSongOrVideoInfoItemExtractor,
YoutubeMusicAlbumOrPlaylistInfoItemExtractor and
YoutubeMusicArtistInfoItemExtractor) allows to simplify
YoutubeMusicSearchExtractor,improves reading and applying changes to InfoItems
(no more losing at least quarter of a line due to indentations).
These InfoItems, in which the image changes have been applied, don't extend the
YouTube ones anymore, as most methods were overridden and the few ones that are
not don't apply in YouTube Music items responses, so it was useless to extend
them.
The code of YoutubeMusicSearchExtractor have been also improved a bit.
2023-08-12 22:56:27 +02:00
AudricV
c1981ed54f
[YouTube] Apply changes in Extractors except YoutubeMusicSearchExtractor
...
Also improve a bit some code related to the changes.
2023-08-12 22:56:27 +02:00
AudricV
4cc99f9ce1
[YouTube] Apply changes in InfoItemExtractors except YouTube Music ones
2023-08-12 22:56:27 +02:00
AudricV
adfad086ac
[YouTube] Add utility methods to get images from InfoItems and thumbnails arrays
...
Unmodifiable lists of Images are returned, parsed from a given YouTube
"thumbnails" JSON array.
These methods will be used in all YouTube extractors and InfoItems, as the
structures between content types (videos, channels, playlists, ...) are common.
2023-08-12 22:56:27 +02:00
AudricV
d56b880cae
Replace avatar and thumbnail URLs attributes and methods to List<Image> in Infos
2023-08-12 22:56:26 +02:00
AudricV
9d8098576e
Replace avatar and thumbnail URLs attributes and methods to List<Image> in Extractors
2023-08-12 22:56:26 +02:00
AudricV
0f4a5a8184
Replace avatar and thumbnail URLs attributes and methods to List<Image> in InfoItemsCollectors
2023-08-12 22:56:26 +02:00
AudricV
ca1d4a6fa4
Replace avatar and thumbnail URLs attributes and methods to List<Image> in InfoItemExtractors
2023-08-12 22:56:26 +02:00
AudricV
2f3ee8a3f2
Replace avatar and thumbnail URLs attributes and methods to List<Image> in InfoItems
2023-08-12 22:56:25 +02:00
AudricV
78ce65769f
Add an ImageSuffix class to the extractor
...
The goal of this utility class is to simply store suffixes which need to be
appended to image URLs, in order to get images at the suffix resolution.
This class contains four properties: the suffix (as a string), the height,
the width (as integers) and the estimated resolution level of the image
corresponding to the one represented by the suffix.
2023-08-12 22:56:25 +02:00
AudricV
d85454186a
Add an Image class to the extractor
...
Objects of this serializable class contains four properties: a URL (as a
string), a width, a height (represented as integers) and an estimated
resolution level, which can be constructed from a given height.
Possible resolution levels are:
- UNKNOWN: for unknown heights or heights <= 0;
- LOW: for heights > 0 & < 175;
- MEDIUM: for heights >= 175 & < 720;
- HIGH: for heights >= 720.
Getters of these properties are available and the constructor needs these four
properties.
2023-08-12 22:56:25 +02:00
Stypox
7294675aea
Merge pull request #1093 from AudricV/yt_support-shorts-ui-playlists
...
[YouTube] Support Shorts UI in playlists
2023-08-12 11:11:36 +02:00
Stypox
93a90b816d
Merge pull request #1094 from AudricV/yt_support-more-channel-headers
...
[YouTube] Support more channel headers
2023-08-12 11:08:30 +02:00
Stypox
44b664af15
[YouTube] Simplify Optional chains in channel
2023-08-12 11:02:51 +02:00
AudricV
2f7bfd3e7f
[YouTube] Add mocks of interactiveTabbedHeaderRenderer channel header test
2023-08-08 19:12:29 +02:00
AudricV
b147904571
[YouTube] Add test for interactiveTabbedHeaderRenderer channel header
...
This test uses the Minecraft game topic channel.
2023-08-08 19:12:28 +02:00
AudricV
1852031a0b
[YouTube] Support pageHeaderRenderer and interactiveTabbedHeaderRenderer channel headers
...
The addition of this support required to turn the isCarouselHeader boolean into
an enum containing all supported channel headers named HeaderType.
Also assert that the page has been fetched where needed to avoid
NullPointerExceptions when the channel page has been not fetched and remove the
getChannelHeaderJson method in YoutubeChannelExtractor, method for which its
code has been moved to its sole usage after the new headers support changes.
2023-08-08 19:12:27 +02:00
AudricV
698c710685
Do not require knowledge of uploader in default StreamInfoItems tests
...
This change is required as some services can return no uploader info, such as
YouTube for playlists with a Shorts UI.
2023-08-07 19:43:15 +02:00
AudricV
8237052ef5
Fix wrong assertion in assertNotEmpty
...
The non-null assertion was made on the exception message instead of the string
to check, causing a NullPointerException if the string to check was null.
2023-08-07 19:43:09 +02:00
AudricV
162c261577
[YouTube] Add mocks of the playlist with Shorts UI test
2023-08-07 19:07:53 +02:00
AudricV
e2f4ee47b9
[YouTube] Add a playlist with Shorts UI test
...
The system Shorts videos uploads playlist of the YouTube official channel has
been chosen for this test.
2023-08-07 19:06:09 +02:00
AudricV
e6f371fb94
[YouTube] Support Shorts UI in playlists
...
Also remove an outdated A/B test comment.
2023-08-07 19:01:08 +02:00
Stypox
7936987955
Merge pull request #1092 from Stypox/channel-tabs-improvements
...
Channel tabs code improvements
2023-08-06 21:44:10 +02:00
Stypox
6d2227111f
[YouTube] Assert that videos tab is ready after channel fetching
2023-08-06 21:14:57 +02:00
Stypox
ee625c325c
Inherit from DefaultListExtractorTest in channel tab tests
2023-08-06 21:14:56 +02:00
Stypox
276c293889
Rename assertTabsContain
2023-08-06 21:14:56 +02:00
Stypox
9d3761a371
[YouTube] Directly use playlist collector in channel tabs wrapper
...
Note that this introduces a "Raw use of parameterized class 'InfoItemsPage'" warning, but it can be ignored since the type missing would be <InfoItem>, and StreamInfoItem extends InfoItem
2023-08-06 21:13:25 +02:00
Stypox
95a3cc0a17
Merge pull request #1083 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.10.0
...
Bump org.junit:junit-bom from 5.9.3 to 5.10.0
2023-08-06 18:58:27 +02:00
Stypox
e34b4f1978
[YouTube] Avoid using Consumer
2023-08-06 13:02:31 +02:00
Stypox
ef67c7cd74
[YouTube] Simplify usage of channel header json
2023-08-06 13:02:31 +02:00
Stypox
a104cf3227
[YouTube] Fix docs in channel helper
2023-08-06 13:02:31 +02:00
Stypox
bb47f05f89
Merge pull request #1091 from Stypox/update-mocks
...
[YouTube] Update stream mocks after #1087
2023-08-06 12:58:24 +02:00
Stypox
468bcc045d
[YouTube] Update mocks after #1087
2023-08-06 12:33:04 +02:00
Stypox
35f3a4ad01
Merge pull request #1082 from AudricV/channel-tabs-and-tags-support
...
Add support for channel tabs and channel tags
2023-08-06 12:21:42 +02:00
AudricV
e7d64099a7
[YouTube] Update channel mocks and add channel tabs mocks
2023-08-06 12:15:06 +02:00
AudricV
684101c47d
[YouTube] Implement age-restricted channels support, link handlers and channels tabs and tags changes on tests
...
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV
eaf2600ce0
[SoundCloud] Implement link handlers and channels tabs and tags changes on tests
...
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV
0ee2072de5
[PeerTube] Implement link handlers and channels tabs and tags changes on tests
...
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV
d3801dd0e9
[MediaCCC] Implement link handlers and channels tabs and tags changes on tests
...
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV
8baec04611
[Bandcamp] Implement link handlers and channels tabs and tags changes on tests
...
Tests in BandcampChannelExtractorTest and BandcampChannelLinkHandlerFactoryTest
have been also fixed.
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV
e0ba29cd19
Add utility method to assert that given channel tabs are in the ones returned by a channel extractor
...
Only the first content filter of the ListLinkHandler instances provided is
used when collecting all channel tabs of the ListLinkHandler list, as channel
tabs implementations only use one content filter per ListLinkHandler instance.
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV
18846baba7
Add tabs and tags methods in tests interfaces and annotate all methods with the Test JUnit annotation
...
These changes should help to detect tests as tests, when running a subset of
tests or all tests.
They should be also implemented in these interfaces' implementations (new and
existing ones).
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
ThetaDev
c70a0e3543
Add a test for textual durations parsing using TimeAgoParser's patterns
2023-08-06 12:15:06 +02:00
AudricV
7366eab156
[YouTube] Add support for channel tabs and tags and age-restricted channels
...
Support of tags and videos, shorts, live, playlists and channels tabs has been
added for non-age restricted channels.
Age-restricted channels are now also supported and always returned the videos,
shorts and live tabs, accessible using system playlists. These tabs are the
only ones which can be accessed using YouTube's desktop website without being
logged-in.
The videos channel tab parameter has been updated to the one used by the
desktop website and when a channel extraction is fetched, this tab is returned
in the list of tabs as a cached one in the corresponding link handler.
Visitor data support per request has been added, as a valid visitor data is
required to fetch continuations with contents on the shorts tab. It is only
used in this case to enhance privacy.
A dedicated shorts UI elements (reelItemRenderers) extractor has been added,
YoutubeReelInfoItemExtractor. These elements do not provide the exact view
count, any uploader info (name, URL, avatar, verified status) and the upload
date.
All service's LinkHandlers are now using the singleton pattern and some code
has been also improved on the files changed.
Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:15:04 +02:00
AudricV
4586067934
Add utility method to parse textual durations using TimeAgoParser's patterns
...
This is required to parse duration of YouTube's reelItemRenderers, returned
only inside accessibility data.
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:13:33 +02:00
AudricV
d4bfe791ee
[SoundCloud] Add tabs support for users
...
Support of tracks, playlists and albums has been added for users.
Also add the declaration of the UnsupportedOperationException exception to the
service's LinkHandlers.
Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:13:32 +02:00
AudricV
6f7d1f079f
[Bandcamp] Add tabs support for artists
...
Support of tracks and albums has been added for artists.
Also use the singleton pattern and add the declaration of the
UnsupportedOperationException exception to the service's LinkHandlers and
improved some code in the files changed.
Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:12:19 +02:00
AudricV
1e8474b22d
[PeerTube] Add tabs support for accounts and video channels
...
Support of channels and videos has been added for accounts and support of
videos and playlists has been added for video channels.
The following changes have been also done:
- collectStreamsFrom method in PeertubeParsingHelper has been renamed to
collectItemsFrom;
- PeertubeChannelInfoItemExtractor.getStreamCount method has been fixed due to
ChannelExtractor's new inheritance;
- the declaration of the UnsupportedOperationException exception thrown has
been added to the service's LinkHandlers;
- a channel tab LinkHandlerFactory has been added,
PeertubeChannelTabLinkHandlerFactory;
- all service's LinkHandlers are now using properly the singleton pattern.
Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:12:15 +02:00
AudricV
652c2c8408
Add a ListLinkHandler which can be used to be returned from ChannelInfo.getTabs() when a specific tab's data has already been fetched
...
This new ListLinkHandler, ReadyChannelTabListLinkHandler, should help saving
clients data, energy and time by helping to reduce duplicate requests.
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:11:12 +02:00
AudricV
de823a6b68
Add an UnsupportedTabException exception class
...
This class makes easier for LinkHandlerFactory implementations to declare an
UnsupportedOperationException.
2023-08-06 12:11:12 +02:00
AudricV
76fb9dcdd7
Add UnsupportedOperationException to exceptions which can be thrown by getId and getUrl methods of LinkHandlerFactory and its base implementations
...
This change advertise to clients that channel tabs' link handler factories can
return an UnsupportedOperationException when a tab provided to them is
unsupported.
2023-08-06 12:11:12 +02:00
AudricV
946eb9bd91
Add structure of channel tags
...
Tags' getters and/or setters have been added in ChannelExtractor and
ChannelInfo to do so.
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:11:12 +02:00
AudricV
356a888d6c
Add structure of channel tabs
...
This commit introduces the following breaking changes:
- Three new classes have been added:
- ChannelTabExtractor, class extending ListExtractor<InfoItem>, which
extracts InfoItems from a channel tab;
- ChannelTabInfo extending ListInfo<InfoItem>, which extracts InfoItems from
a ChannelTabExtractor and returns them as a ChannelTabInfo;
- ChannelTabs, an immutable class containing all supported channel tabs.
- StreamingService implementations must implement new methods returning a
channel tab LinkHandlerFactory (getChannelTabsLHFactory) and a
ChannelTabExtractor (getChannelTabExtractor);
- ChannelExtractor inherits Extractor instead of ListExtractor<StreamInfoItem>
and ChannelInfo inherits Info instead of ListInfo<StreamInfoItem>;
- ChannelExtractor and ChannelInfo have now getters and/or setters of tabs.
Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:11:11 +02:00
Stypox
3faaf4301c
Merge pull request #1087 from AudricV/yt_js-extractor-improvements-and-fixes
...
[YouTube] Improve and fix YoutubeJavaScriptExtractor
2023-08-06 12:01:00 +02:00
Stypox
8fb6ba36fa
Merge pull request #1081 from TeamNewPipe/fix/sc/search-next-page
...
[SoundCloud] Detect whether there are any more search results
2023-08-06 11:49:35 +02:00
Stypox
2947257111
[SoundCloud] Properly calculate if results have finished
2023-08-06 11:38:22 +02:00
Stypox
485bfbca9d
[SoundCloud] Move try-catch inside getOffsetFromUrl
2023-08-06 11:35:37 +02:00
Stypox
7c70fef197
Merge pull request #1089 from TeamNewPipe/ccc
...
[media.ccc.de] Only extract kiosk live stream rooms if they are streaming
2023-08-06 10:12:04 +02:00
Stypox
de0a9bb797
Merge pull request #1088 from FireMasterK/pseudo-rng
...
Replace cryptographically secure random with regular random
2023-08-05 11:37:49 +02:00
TobiGr
340095515d
Make Kiosk IDs accessible if possible
2023-08-05 03:18:40 +02:00
TobiGr
fe27d6a0ec
[media.ccc.de] Only extract live streams if the conference is streaming
2023-08-05 01:53:43 +02:00
Kavin
25082d78b0
Replace SecureRandom with Random
2023-08-03 23:00:02 +01:00
TobiGr
aa6c17dc77
[SoundCloud] Deduplicate some code
2023-08-03 14:41:30 +02:00
TobiGr
2fb9922a15
[SoundCloud] Detect whether there are any more search results
...
Add test for this edge case.
2023-08-03 14:37:13 +02:00
AudricV
a3d160edab
[YouTube] Improve and fix YoutubeJavaScriptExtractor
...
- Enhance documentation;
- Fix the regular expression fallback on HTML embed watch page;
- Use HTML scripts tag search first instead of the regular expression approach,
now used as a last resort;
- Compile regular expressions only once, in order to improve the performance of
subsequent extraction calls when clearing the cache;
- Provide original exceptions when fetching or parsing pages on which the base
JavaScript's player could be found failed, allowing clients to detect network
errors when they are the cause of the failures for instance;
- Remove delegate method which was not taking a video ID and hardcoding one, as
we can provide the video ID in all cases or do not provide a video ID at worse;
- Rename and make extraction methods package-private, as they are not intended
to be used publicly.
These breaking internal changes have been applied where needed, in
YoutubeJavaScriptExtractorTest and YoutubeStreamExtractor (in which an unneeded
initStsFromPlayerJsIfNeeded call have been removed).
2023-08-02 23:05:08 +02:00
Stypox
5492343b8e
Pre-release v0.22.7
2023-08-02 20:02:29 +02:00
Tobi
9a59afbcf5
Merge pull request #1086 from AudricV/yt_no-exception-channels-without-banner
...
[YouTube] Don't throw an exception when there is no banner available on a channel
2023-08-02 18:56:26 +02:00
AudricV
bb1ab166bf
[YouTube] Test that no banner is returned for carouselHeaderRenders
2023-08-01 22:19:43 +02:00
AudricV
f1fa84b4e3
[YouTube] Don't throw an exception when there is no banner available on a channel
...
Channels may not have a banner, so no exception should be thrown if no banner
is found.
2023-08-01 12:40:20 +02:00
Tobi
39a911db9f
Merge pull request #1084 from AudricV/yt_android-403s-workaround-and-streams-tests-fixes
...
[YouTube] Workaround again 403 HTTP issues on the ANDROID InnerTube client and fix stream tests
2023-07-31 23:51:10 +02:00
dependabot[bot]
bda3a3fc5d
Bump org.junit:junit-bom from 5.9.3 to 5.10.0
...
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5 ) from 5.9.3 to 5.10.0.
- [Release notes](https://github.com/junit-team/junit5/releases )
- [Commits](https://github.com/junit-team/junit5/compare/r5.9.3...r5.10.0 )
---
updated-dependencies:
- dependency-name: org.junit:junit-bom
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-07-24 09:53:05 +00:00
AudricV
522c78160f
[YouTube] Update stream tests mocks
2023-07-23 19:36:28 +02:00
AudricV
7528eb2bd9
[YouTube] Fix stream tests failures
...
- Fix testCheckAudioStreams test of
YoutubeStreamExtractorDefaultTest.AudioTrackLanguage test class, by updating
the excepted audio track name test to use the updated English audio track name
(audio track type info has been added on the video tested);
- Fix YoutubeStreamExtractorDefaultTest.PublicBroadcasterTest test class by
using a different video from a French and German public broadcast channel, as
the channel Dinge Erklärt – Kurzgesagt is not affiliated with a public
broadcast channel anymore;
- Fix YoutubeStreamExtractorLivestreamTest test class, by updating the excepted
name of the livestream to the current one.
2023-07-23 19:19:02 +02:00
AudricV
164c8e3abb
[YouTube] Workaround again 403 HTTP issues on the Android client by using new player parameters
...
These parameters are the only ones currently known to bypass 403 HTTP issues
related to failure of passing Android client integrity checks, as the ones of
stories (and the base of the shorts ones) do not work anymore, which may be
related to end of this format on the service.
2023-07-22 20:22:16 +02:00
Tobi
8ebbe1b7e1
Merge pull request #706 from FireMasterK/av1-itags
...
Add support for AV1 itags.
2023-07-22 13:31:45 +02:00
FireMasterK
6db0d116fe
Add support for AV1 itags.
2023-07-22 13:23:44 +02:00
Tobi
986a3f7747
Merge pull request #1080 from TeamNewPipe/proguard-rules
...
Add proguard rules to README
2023-07-07 12:52:10 +02:00
Tobi
d6132a9ed9
Merge pull request #1076 from AudricV/yt-fix-throttling-parameter-regex
...
[YouTube] Support multiple declarations for throttling parameter function name array
2023-07-07 12:51:26 +02:00
Tobi
87eb29a25d
Add proguard rules to README
2023-07-05 08:25:23 +02:00
AudricV
4e22c5ee87
[YouTube] Support multiple declarations for throttling parameter function name array
...
Also moved the corresponding regex parts in static constants for easier future
modifications
2023-06-26 15:25:53 +02:00
Kavin
d961d349c3
[YouTube] Check whether player responses are valid for all InnerTube clients used ( #1070 )
...
Co-authored-by: Audric V <74829229+AudricV@users.noreply.github.com>
2023-06-18 21:54:52 +02:00
ThetaDev
ad97f08048
[YouTube] Fix parsing short relative date formats (English only) ( #1068 )
2023-06-18 21:41:29 +02:00
Tobi
d294ccb433
Merge pull request #1071 from TeamNewPipe/feat/ServiceList
...
Init services at the correct place
2023-06-17 20:50:24 +02:00
Tobi
5d112aa772
Merge pull request #1074 from TeamNewPipe/imp/media-formats
...
Add more audio media formats and MediaFormat.getAllForMimeType(mimeType)
2023-06-17 20:49:38 +02:00
TobiGr
0e01d90562
Do not init services while creating services array
2023-06-17 18:08:09 +02:00
TobiGr
5809904cf7
Add annotations to MediaFormat
2023-06-17 18:01:54 +02:00
TobiGr
53dfd871e2
Add more audio media formats and MediaFormat.getAllForMimeType(mimeType)
2023-06-17 18:01:29 +02:00
Stypox
5eb886243f
Merge pull request #1072 from TeamNewPipe/fix/bandcamp_test
...
Fix BandcampRadioStreamExtractorTest.testGetAudioStreams()
2023-06-09 17:35:35 +02:00
TobiGr
19ce06fe00
Fix BandcampRadioStreamExtractorTest.testGetAudioStreams()
2023-06-06 23:07:56 +02:00
Audric V
533121fb81
Merge pull request #1045 from Theta-Dev/fix/trending-video-tab
...
[YouTube] Extract trends from A/B tested "Videos" tab and fix extraction of trends name from A/B tested new title design
2023-05-19 11:22:49 +02:00
Audric V
92a0024424
Merge pull request #1052 from TeamNewPipe/peertube/fix/nested-comment-replies
...
[PeerTube] Fix multi level comment replies
2023-05-18 18:49:06 +02:00
Tobi
3c036a9c03
Merge pull request #1061 from ChunkyProgrammer/develop/extract-yt-playlist-descrption
...
Extract playlists description
2023-05-17 22:43:17 +02:00
TobiGr
c70bb83801
[Bandcamp] Implement PlaylistExtractor.getDescsription()
2023-05-15 15:23:03 +02:00
TobiGr
ca0ce00753
Add PlaylistInfoItem.getDescription() and PlaylistInfoItemExtractor.getDescription()
...
[PeerTube] Implement the corresponding extractor method.
TODO: add tests
2023-05-12 01:43:59 +02:00
TobiGr
b218bf69bd
Implement PlaylistInfo.getDescription()
...
Implement PlaylistExtractor.getDescription() for PeerTube and SoundCloud.
Anotate PlaylistExtractor.getDescription() as Nonnull
2023-05-12 00:44:10 +02:00
chunky programmer
81f29116ba
switch from string to Description object
2023-05-11 00:36:57 -04:00
chunky programmer
e147867d41
Add tests
2023-05-11 00:00:34 -04:00
chunky programmer
5ab6cd7420
Extract YouTube playlist description
2023-05-11 00:00:22 -04:00
TobiGr
d358ba1c41
Improve PeertubeCommentsInfoItemExtractor constructor
2023-05-07 22:55:26 +02:00
TobiGr
aff3e795f8
[PeerTube] Fix multi level comment replies
2023-05-07 22:49:14 +02:00
ThetaDev
3673d4ae01
fix: YouTube trending name extraction
2023-05-03 21:16:35 +02:00
ThetaDev
0addb98cd7
tests: update mocks
2023-05-03 21:16:35 +02:00
ThetaDev
24eba62305
fix: extract YouTube trends from new "Videos" tab
2023-05-03 21:16:23 +02:00
Kavin
a9ca5c49e4
Merge pull request #1056 from AudricV/yt-improve-search-suggestions-extraction
...
[YouTube] Switch to new search suggestion domain and improve error handling
2023-05-02 20:17:48 +01:00
Stypox
00408db959
Merge pull request #1058 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.16.1
...
Bump org.jsoup:jsoup from 1.15.4 to 1.16.1
2023-05-01 19:59:02 +02:00
dependabot[bot]
108f8a7a17
Bump org.jsoup:jsoup from 1.15.4 to 1.16.1
...
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup ) from 1.15.4 to 1.16.1.
- [Release notes](https://github.com/jhy/jsoup/releases )
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES )
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.4...jsoup-1.16.1 )
---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-05-01 09:58:32 +00:00
Tobi
11565db17f
Merge pull request #1055 from AudricV/peertube-no-description-channels
...
[PeerTube] Don't return "No description" when there is no description for a channel or an account
2023-04-30 23:38:10 +02:00
AudricV
cf6040ddb3
Update YoutubeSuggestionExtractorTest mocks
2023-04-30 19:53:44 +02:00
AudricV
593122342f
[YouTube] Improve YoutubeSuggestionExtractorTest
...
- Remove useless concatenation on the downloader path;
- Remove unneeded public test modifier;
- Update license header;
- Specify the service class tested instead of the generic class.
2023-04-30 19:53:43 +02:00
AudricV
e923fca440
[YouTube] Switch to new search suggestion domain and improve error handling
...
- Switch to the new domain used by YouTube for search suggestions,
suggestqueries-clients6.youtube.com, and add the xhr query parameter with the
t value, to allow getting responses without requiring trim;
- Use the Java 8 Stream API to collect search suggestions and improve invalid
response detection by checking whether the content type of the response
returned is JSON;
- Move the licence header at the top of the file.
2023-04-30 19:53:42 +02:00
AudricV
945165a3c0
[PeerTube] Don't return "No description" when there is no description for a channel or an account
...
When a description is missing, no description should be returned, even the ones
indicating there is no description. This behavior is represented by a null
return instead.
Also update PeertubeAccountExtractorTest to reflect these changes.
2023-04-30 18:41:38 +02:00
Stypox
d61dc27406
Merge pull request #1054 from TeamNewPipe/fix-ci-variable
...
Make sure CI is not vulnerable to bash word splitting
2023-04-27 18:52:07 +02:00
Stypox
51cba5a287
Make sure CI is not vulnerable to bash word splitting
...
See https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-an-intermediate-environment-variable
2023-04-26 18:36:25 +02:00
Stypox
f819c2fe34
Merge pull request #1053 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.3
...
Bump org.junit:junit-bom from 5.9.2 to 5.9.3
2023-04-26 15:42:19 +02:00
dependabot[bot]
4b92c5a34d
Bump org.junit:junit-bom from 5.9.2 to 5.9.3
...
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5 ) from 5.9.2 to 5.9.3.
- [Release notes](https://github.com/junit-team/junit5/releases )
- [Commits](https://github.com/junit-team/junit5/compare/r5.9.2...r5.9.3 )
---
updated-dependencies:
- dependency-name: org.junit:junit-bom
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-04-26 09:57:57 +00:00
Stypox
2deb023da4
Merge pull request #1050 from Theta-Dev/fix/channel-carousel-header
...
[YouTube] Add support for CarouselHeaderRenderer
2023-04-25 15:17:31 +02:00
ThetaDev
4aada7f91b
refactor: rename carousel header channel test
2023-04-21 22:48:37 +02:00
Tobi
b2b426048f
Merge pull request #1051 from Theta-Dev/fix/ytm-client-version-rg
...
[YouTube] fix: set musicClientVersion regex capture group
2023-04-16 23:48:19 +02:00
ThetaDev
47aa9fed40
fix: set musicClientVersion regex capture group
2023-04-16 19:25:05 +02:00
ThetaDev
20370395c5
fix: add support for CarouselHeaderRenderer
2023-04-16 17:40:13 +02:00
Stypox
7dba6e3891
Merge pull request #1033 from petlyh/bandcamp-paywalled-content
...
[Bandcamp] Handle paywalled tracks
2023-04-12 13:04:26 +02:00
petlyh
e6aad117e7
[Bandcamp] Throw PaidContentException on paywalled albums
2023-04-03 19:27:09 +02:00
TobiGr
8495ad619e
Bump version to 0.22.6
2023-04-02 22:35:02 +02:00
fynngodau
69705138e4
[Bandcamp] Fix extraction of related playlist items URL ( #1047 )
...
Small change in HTML structure
2023-04-02 22:24:29 +02:00
Björn Sigurbergsson
1b6fe5edd6
[YouTube] Fix ParsingException when comments are unavailable in a video ( #1040 )
...
Co-authored-by: bjs <bjs@elect-it.com>
Co-authored-by: Audric V. <74829229+AudricV@users.noreply.github.com>
Co-authored-by: Kavin <20838718+FireMasterK@users.noreply.github.com>
2023-03-30 19:58:06 +02:00
ThetaDev
8d1303e18f
Add track types to audio streams ( #1041 )
2023-03-28 00:02:20 +02:00
Tobi
44ae139d33
Merge pull request #1034 from AudricV/peertube-fix-commentsinfo-test
...
[PeerTube] Fix comments info test
2023-03-20 19:30:35 +01:00
Tobi
d5e9df5381
Merge pull request #1038 from TeamNewPipe/docs-jdk-11
...
Update JDK to 11 in JDoc workflow
2023-03-20 19:25:21 +01:00
Tobi
0486ad7a08
Update JDK to 11 in JDoc workflow
2023-03-06 18:45:11 +01:00
AudricV
80a6fc2c63
[PeerTube] Fix testGetCommentsFromCommentsInfo test of PeertubeCommentsExtractorTest.Default
...
The tested comment has been removed, so it couldn't be found in the comments
list.
This comment has been replaced by a new one from the current comments of the
video.
Also, in the parent class PeertubeCommentsExtractorTest, final has been used as
much as possible and for-each loops of lists have been replaced by their
forEach method or the Stream API, in order to simplify code.
2023-03-04 16:49:10 +01:00
petlyh
5a9b6ed2e3
[Bandcamp] Support loading additional comments ( #1030 )
2023-03-04 14:01:06 +01:00
Stypox
6bdd698c25
Merge pull request #1026 from AudricV/audio-streams-descriptive-and-locale-properties
...
Add descriptive and locale properties to audio streams
2023-03-01 11:15:46 +01:00
Stypox
19e4b216c9
Merge pull request #1032 from AudricV/yt_fix-comments-hashtags-links-extraction
...
[YouTube] Fix hashtags links extraction and escape HTML links
2023-03-01 10:47:37 +01:00
Stypox
b1298490c0
Merge pull request #1029 from AudricV/yt_fix-no-views-extraction-playlist-items
...
[YouTube] Fix partial non-extraction of "No views" string in stream items
2023-03-01 10:46:52 +01:00
petlyh
9dc1832733
[Bandcamp] Handle paywalled tracks
2023-02-28 17:51:30 +01:00
fynngodau
3fdb6ee476
Merge pull request #1031 from petlyh/bandcamp-fix-radio-comments
...
[Bandcamp] Show comments as disabled on radio streams
2023-02-27 20:50:12 +01:00
AudricV
bd79b921e8
[YouTube] Refactor the code to get stream items' view count
...
This refactoring avoids code duplication as much as possible.
2023-02-27 10:25:46 +01:00
AudricV
51f9b39953
[YouTube] Fix partial non-extraction of no views string in stream items
...
As the "No views" string is returned in the case there is no view on a video, a
number cannot be parsed in this case, so -1 was returned.
This string is now detected in all methods to get the view count of a stream.
2023-02-27 10:18:45 +01:00
AudricV
95b3f5e391
[MediaCCC] Test audio language property extraction
2023-02-26 19:06:18 +01:00
AudricV
30a0f8c510
[MediaCCC] Extract audio language property for single language audio tracks
2023-02-26 19:06:18 +01:00
AudricV
7f0269c4c7
[YouTube] Edit YoutubeStreamExtractorDefaultTest.AudioTrackLanguage to test audio locale property
...
The Hindi audio track language presence test has been changed from audio track
label to audio locale.
2023-02-26 19:06:18 +01:00
AudricV
034f82dae7
[YouTube] Test language and descriptive audio in YoutubeDashManifestCreatorsTest
2023-02-26 19:06:17 +01:00
AudricV
05e8cb39f7
[YouTube] Add language and descriptive audio properties to DASH manifests
2023-02-26 19:06:17 +01:00
AudricV
bf30d70152
[YouTube] Add descriptive audio test
...
This test uses video TjxC-evzxdk.
Also improve a bit YoutubeStreamExtractorDefaultTest.AudioTrackLanguage test.
2023-02-26 19:06:17 +01:00
AudricV
76b7c19c5d
[YouTube] Extract whether a track is a descriptive audio and audio locale when available
...
Also use audio track setters only for audio itags.
2023-02-26 19:06:17 +01:00
AudricV
3bb5eeef30
[YouTube] Add descriptive and locale audio support in ItagItem
2023-02-26 19:06:16 +01:00
AudricV
14bf3fb05b
Add ability to know the locale of an audio stream
...
Getting audio tracks locales by parsing their ID or their label, should not be
done by clients, but by the extractor.
This commit adds the ability to store the Locale of an AudioStream, which is
used to compare similar AudioStreams (in the equalStats method).
2023-02-26 19:06:16 +01:00
AudricV
f92426560c
Add descriptive audio properties
...
Also improve AudioStream's audio language documentation
2023-02-26 19:06:16 +01:00
AudricV
a63f289667
[YouTube] Update mocks of YoutubeCommentsExtractorTest.FormattingTest
2023-02-26 18:50:07 +01:00
AudricV
9483dcd9fa
[YouTube] Update mocks of YoutubeCommentsExtractorTest.RepliesTest
2023-02-26 18:43:36 +01:00
AudricV
1556adbb2d
[YouTube] Fix hashtags links extraction and escape text in attribute descriptions + HTML links
...
webCommandMetadata object is contained inside a commandMetadata one, so it is
not accessible from the root of the navigationEndpoint object.
The corresponding statement has been moved at the bottom of the specific
endpoints parsing, as the webCommandMetadata object is present almost
everywhere, otherwise URLs of some endpoints would have be changed, such as
uploader URLs (from channel IDs to handles).
As no ParsingException is now thrown by getUrlFromNavigationEndpoint, and so by
getTextFromObject, getUrlFromObject and getTextAtKey, the methods which were
catching ParsingExceptions thrown by these methods had to be updated.
URLs got in the HTML version of getTextFromObject are now escaped properly to
provide valid HTML to clients. This has been also done for attribute
descriptions, with the description text for this type of descriptions.
As YouTube descriptions are in HTML format (except for the fallback on the JSON
player response, which is plain text and only happens when there is no visual
metadata or a breaking change), all URLs returned are escaped, so tests which
are testing presence of URLs with escaped characters had to be updated (it was
only the case for YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing).
2023-02-26 18:43:36 +01:00
Stypox
99ab9777ad
Release 0.22.5
2023-02-26 16:33:25 +01:00
Stypox
39c2e1fda0
Merge pull request #1028 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.4
...
Bump org.jsoup:jsoup from 1.15.3 to 1.15.4
2023-02-26 16:27:31 +01:00
petlyh
f7a7a236fb
[Bandcamp] Show comments as disabled on radio streams
2023-02-23 18:42:43 +01:00
dependabot[bot]
f5599ff08d
Bump org.jsoup:jsoup from 1.15.3 to 1.15.4
...
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup ) from 1.15.3 to 1.15.4.
- [Release notes](https://github.com/jhy/jsoup/releases )
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES )
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.3...jsoup-1.15.4 )
---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-02-20 10:05:26 +00:00
Stypox
7e793c11ae
Empty commit to make JitPack happy
...
The build for e920ab3f5f
succeeded, but for some reason it is considered as failed...
2023-02-08 22:35:26 +01:00
Stypox
e920ab3f5f
Merge pull request #1021 from TeamNewPipe/fix/yt-comment-text-hashtag
...
[YouTube] Fix getting the comment text if the comment contains a hashtag
2023-01-29 21:47:59 +01:00
TobiGr
3f7df9536e
[YouTube] Fix getting the comment text if the comment contains a hashtag
2023-01-29 20:33:51 +01:00
Stypox
999fb7f812
Merge pull request #1024 from AudricV/snd_fix-tracks-like-count
...
[SoundCloud] Fix extraction of tracks like count
2023-01-29 10:52:54 +01:00
Stypox
3519d4c367
Merge pull request #1015 from AudricV/yt_fix-channel-id-rss-feeds
...
[YouTube] Fix channel ID extraction of YouTube channels RSS feeds
2023-01-29 10:41:38 +01:00
Stypox
9aca710e86
Merge pull request #1013 from Stypox/fix-music-mixes
...
[YouTube] Now music mixes can be treated as normal mixes
2023-01-29 09:48:51 +01:00
Stypox
76eeabac45
Merge pull request #1020 from TeamNewPipe/fix/yt-subscriber-count
...
[YouTube] Fix NPE in search when getting channel items without subscriber count
2023-01-29 09:44:22 +01:00
AudricV
676622f6df
[SoundCloud] Fix expectedLikeCountAtLeast tests of SoundcloudStreamExtractorTest test classes
...
As like count is now returned by the extractor, we need to assert a positive
minimum like count, which is close to the actual value, in order to avoid test
failures due to lower like counts than the ones excepted.
2023-01-29 01:08:02 +01:00
AudricV
2a24d407d5
[SoundCloud] Fix extraction of tracks like count
...
SoundCloud is using likes_count to return the like count of a track, like it
was the case before they switched to favoritings_count.
2023-01-29 01:00:49 +01:00
Tobi
0e4e6a9bac
Merge pull request #1022 from AudricV/yt_support-live-urls
...
[YouTube] Support live URLs
2023-01-28 21:52:26 +01:00
AudricV
ba24976e41
[YouTube] Add live URLs test and do minor improvements to YoutubeStreamLinkHandlerFactoryTest
...
- Remove unused imports;
- Replace wildcard imports by single class imports;
- Suppress "HTTP links are not secured" warnings from IDEA IDEs;
- Replace removed video jZViOEv90dI by an existing video, 9Dpqou5cI08 (the
corresponding test has been of course renamed).
2023-01-28 19:36:21 +01:00
AudricV
57f850bc2d
[YouTube] Support live URLs and do minor improvements to YoutubeStreamLinkHandlerFactory
...
- Move license header at the top;
- Use an unmodifiable set for the subpaths instead of a modifiable list;
- Add missing Nonnull and Nullable annotations;
- Improve exception messages.
2023-01-28 19:36:20 +01:00
AudricV
1f4ed9dce9
[YouTube] Fix channel ID extraction of YouTube channel RSS feeds
...
The yt:channelId element doesn't provide the channel ID anymore and is empty,
like the id element, so we need now to extract it from the channel URL provided
in two elements: author -> uri and feed -> link.
Also avoid a NullPointerException in getUrl and getName methods.
2023-01-28 11:53:33 +01:00
Tobi
c589a2c1a2
Merge pull request #1014 from TeamNewPipe/fix/yt-comments
...
[YouTube] Fix getting next comments pages
2023-01-27 11:14:55 +01:00
Tobi
c2b36fd405
Merge pull request #1016 from lonewolf2208/bug-fix
...
[YouTube] Remove topStandaloneBadge check for view count of stream items
2023-01-25 08:30:36 +01:00
TobiGr
72573932cf
[YouTube] Fix NPE in search when getting channel items without subscriber count
2023-01-24 23:03:45 +01:00
TobiGr
f50b7275af
[YouTube] Fix getting next comments pages
2023-01-24 22:39:08 +01:00
Kunal
9bdad40b06
Removed topStandaloneBadge
2023-01-20 02:41:21 +05:30
Stypox
5945057227
[YouTube] Add music mix test
2023-01-15 23:30:30 +01:00
Stypox
7293991832
[YouTube] Now music mixes can be treated as normal mixes
...
Using a playlist extractor on them would result in "Unviewable playlist" errors
2023-01-15 23:28:59 +01:00
Stypox
ff94e9f30b
Merge pull request #1009 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.10.1
...
Bump gson from 2.10 to 2.10.1
2023-01-11 15:35:36 +01:00
Stypox
c1040bccac
Merge pull request #794 from FireMasterK/comments-count
...
[YouTube] Add support to extract total comment count
2023-01-11 15:32:19 +01:00
Stypox
6ccc43e57e
Merge pull request #1010 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.2
...
Bump junit-bom from 5.9.1 to 5.9.2
2023-01-11 13:19:33 +01:00
dependabot[bot]
1bc44cb949
Bump junit-bom from 5.9.1 to 5.9.2
...
Bumps [junit-bom](https://github.com/junit-team/junit5 ) from 5.9.1 to 5.9.2.
- [Release notes](https://github.com/junit-team/junit5/releases )
- [Commits](https://github.com/junit-team/junit5/compare/r5.9.1...r5.9.2 )
---
updated-dependencies:
- dependency-name: org.junit:junit-bom
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-01-11 09:04:12 +00:00
dependabot[bot]
f43049985e
Bump gson from 2.10 to 2.10.1
...
Bumps [gson](https://github.com/google/gson ) from 2.10 to 2.10.1.
- [Release notes](https://github.com/google/gson/releases )
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md )
- [Commits](https://github.com/google/gson/compare/gson-parent-2.10...gson-parent-2.10.1 )
---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-01-09 09:05:55 +00:00
Stypox
33f9f2cf66
Merge pull request #1006 from TeamNewPipe/yt/fix-html-escape
...
[YouTube] Fix escaping links in YouTubeParsingHelper.getTextFromObject
2023-01-05 19:02:23 +01:00
TobiGr
56aab4d971
[YouTube] Fix escaping links in YouTubeParsingHelper.getTextFromObject
2023-01-05 00:28:12 +01:00
Kavin
22a47da8c7
Fix requested change and remove outdated comment.
2023-01-02 20:42:32 +00:00
Kavin
98a90fd9c8
Don't cache comments count and return early on page fetch if no token.
2023-01-02 20:40:48 +00:00
Kavin
2974dfaa48
Only store ajaxJson for initial page and eager fetch the initial continuation.
2023-01-02 20:40:48 +00:00
Kavin
64d24aa09e
Fix request changes.
2023-01-02 20:40:48 +00:00
Kavin
67ef4f4c30
Cleanup and remove optional.
2023-01-02 20:40:48 +00:00
FireMasterK
22f71b010c
Fix for requested changes.
2023-01-02 20:40:48 +00:00
FireMasterK
656b7c1cd9
Improve method documentation.
2023-01-02 20:40:48 +00:00
FireMasterK
981aee4092
Add support to extract total comment count.
2023-01-02 20:40:48 +00:00
Stypox
45636b0d00
Merge pull request #986 from Isira-Seneviratne/Static_maps
...
Use immutable Map factory methods.
2023-01-02 18:11:14 +01:00
Stypox
219c5c5be5
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java
2023-01-02 18:11:03 +01:00
Stypox
259de3cba6
Merge pull request #995 from TeamNewPipe/feat/soundcloud-playlistinfoitemextractor
...
[SoundCloud] Implement getUploaderUrl() and isUploaderVerified() for PlaylistInfoItemExtractor
2023-01-02 15:10:40 +01:00
Stypox
991394b53a
Merge pull request #1005 from FireMasterK/fix-escaping-xss
...
Fix for potential XSS attacks and formatting issues
2023-01-02 15:06:17 +01:00
Stypox
ce15f7cc50
Merge pull request #996 from TeamNewPipe/feat/peeertube-playlists
...
[PeerTube] Support searching for playlists and channels
2023-01-02 15:04:09 +01:00
Isira Seneviratne
d8ce08d969
Use immutable Map factory methods.
2023-01-02 07:50:31 +05:30
Kavin
01acf79436
Fix for potential XSS attacks.
2022-12-31 20:05:32 +00:00
TobiGr
292e0d8ce7
[SoundCloud] Implement getUploaderUrl() and isUploaderVerified() for PlaylistInfoItemExtractor
2022-12-31 18:46:39 +01:00
TobiGr
2a8729aeb2
Apply suggestions
...
Co-authored-by: Stypox <stypox@pm.me>
2022-12-31 18:24:33 +01:00
TobiGr
d75a997611
[PeerTube] Support searching for channels
2022-12-31 18:24:33 +01:00
TobiGr
dea6d8ce4c
[PeerTube] Support searching for playlists
2022-12-31 18:24:33 +01:00
Stypox
95cc6aefbb
Merge pull request #994 from TeamNewPipe/fix/peertube-subtitles-exception
...
[PeerTube] Report Exceptions thrown while getting a stream's subtitles
2022-12-31 15:01:39 +01:00
Stypox
7b54457789
Merge pull request #941 from TeamNewPipe/feat/peertube-comment-replies
...
[PeerTube] Support comment replies
2022-12-31 14:57:51 +01:00
AudricV
f45966d449
Merge pull request #910 from Isira-Seneviratne/Locale_forLanguageTag
...
Add compat Locale.forLanguageTag() implementation.
2022-12-24 23:53:30 +01:00
AudricV
d5437e0bc5
Merge pull request #863 from AudricV/add-content-type-and-content-length-headers-to-post-requests
...
Add Content-Type header to all POST requests without an empty body
2022-12-16 19:32:56 +01:00
Tobi
88e07e555d
Merge pull request #1000 from AudricV/yt-streaminfoitemextractor-improvements
...
[YouTube] Improve YoutubeStreamInfoItemExtractor
2022-12-11 17:02:29 +01:00
AudricV
0766b1d211
[YouTube] Improve YoutubeStreamInfoItemExtractor
...
- Return duration of video premieres;
- Add another non-localized method to determine whether a stream is a running
livestream;
- Return view count and upload date of videos in playlists;
- Store isPremiere result;
- Remove shorts workaround code, as it was only useful on channels and shorts
have been moved into a separated channel tab;
- Improve some other code.
2022-12-08 13:59:12 +01:00
Tobi
896d7e09eb
Merge pull request #978 from Theta-Dev/fix/search-channel-handles
...
[YouTube] Fix search subscriber count extraction with channel handles
2022-12-05 17:52:05 +01:00
TobiGr
cd3262745d
[PeerTube] Report Exceptions thrown while getting a stream's subtitles
2022-12-03 16:11:21 +01:00
TobiGr
4e66b2287e
[PeerTube] Add support for comment replies
2022-12-01 14:05:18 +01:00
Tobi
41c8dce452
Merge pull request #992 from Isira-Seneviratne/String_isBlank
...
Use String.isBlank().
2022-11-30 17:48:54 +01:00
Isira Seneviratne
2bca56f0df
Use String.isBlank().
2022-11-30 08:26:21 +05:30
Isira Seneviratne
3b80547976
Add code review suggestions.
2022-11-30 07:57:45 +05:30
Tobi
a1128ecbdc
Merge pull request #991 from FireMasterK/fix-compile
...
Fix complication error in comment test
2022-11-29 19:20:01 +01:00
ThetaDev
016623131e
docs: update comment in YoutubeChannelInfoItemExtractor
2022-11-29 19:06:03 +01:00
Kavin
2e08eaad96
Fix complication error in comment test.
2022-11-29 16:07:48 +00:00
Kavin
abf08e1496
Merge pull request #990 from FireMasterK/bold-italic-strikethrough
...
[YouTube] Implement bold/italic/strike-through support
2022-11-29 15:59:38 +00:00
Tobi
72d5ed3318
Merge pull request #987 from FireMasterK/comments-text-description
...
Use Description object for comments text.
2022-11-29 16:54:56 +01:00
Kavin
57e7a6fb7c
Add mocks test.
2022-11-28 20:27:55 +00:00
Kavin
1d3d7fa5c3
Add test for formatting.
2022-11-28 20:26:37 +00:00
Kavin
52fda37915
Implement bold/italic/strike-through support.
2022-11-28 19:06:18 +00:00
Kavin
b566084cac
Use Description object for comments text.
2022-11-28 17:02:19 +00:00
Tobi
40f1ec4a54
Merge pull request #989 from Stypox/forbid-wrong-nullable-notnull-imports
...
Block wrong nullable/nonnull imports in checkstyle
2022-11-28 14:57:46 +01:00
Stypox
5f6101698b
Block wrong nullable/nonnull imports in checkstyle
2022-11-28 14:42:18 +01:00
Tobi
f8162b049d
Merge pull request #984 from FireMasterK/unused-dep
...
Remove unused autolink dependency
2022-11-28 11:28:42 +01:00
Tobi
ad7c3c2374
Merge pull request #985 from FireMasterK/checkstyle-update
...
Update checkstyle dependency.
2022-11-28 11:22:40 +01:00
Tobi
1da0190056
Merge pull request #980 from TeamNewPipe/fix/yt/unavailable
...
[YouTube] Fix extracting the detailed error message for unavailable streams
2022-11-28 10:07:34 +01:00
Stypox
60fb30f835
Merge pull request #928 from FireMasterK/comment-urls
...
Parse YouTube comments as HTML
2022-11-27 19:16:34 +01:00
Kavin
5abea22225
Fix throwing correct reason.
2022-11-26 21:09:08 +00:00
Kavin
7874d3c8aa
Update checkstyle dependency.
...
Since we now use JDK 11
2022-11-26 20:22:25 +00:00
Kavin
faf28f5c11
Remove unused dependency.
2022-11-26 20:17:25 +00:00
Kavin
a3ed947b2d
Merge pull request #983 from FireMasterK/update-countries
...
Update supported countries list.
2022-11-26 20:06:00 +00:00
Kavin
c043597255
Update supported countries list.
2022-11-26 19:01:33 +00:00
TobiGr
4680df0bdf
Fix throwing correct reason
2022-11-23 17:03:22 +01:00
TobiGr
9de8405c9f
[YouTube] Fix extracting the detailed error message of streams which are unavailable
2022-11-23 08:33:06 +01:00
Stypox
34d79bd267
[YouTube] Update mocks
2022-11-22 17:10:04 +01:00
AudricV
2ec296e674
Fix YoutubeSearchExtractorTest.MetaInfoTest
...
Not all the "learn more" button is uppercase anymore, that's only the case for
the first letter.
2022-11-22 16:34:54 +01:00
AudricV
3891542ca1
Use Downloader's postWithContentType and postWithContentTypeJson methods in services and extractors
2022-11-22 11:37:18 +01:00
AudricV
b2862f3cd1
Add postWithContentType and postWithContentTypeJson utility methods in Downloader
...
Co-authored-by: Stypox <stypox@pm.me>
2022-11-22 11:37:17 +01:00
AudricV
e9a0d3bd95
[YouTube] Send Content-Type header in all POST requests
...
This header was not sent partially before and was added and guessed by OkHttp. This can create issues when using other HTTP clients than OkHttp, such as Cronet.
Some code in the modified classes has been improved and / or deduplicated, and usages of the UTF_8 constant of the Utils class has been replaced by StandardCharsets.UTF_8 where possible.
Note that this header has been not added in except in YoutubeDashManifestCreatorsUtils, as an empty body is sent in the POST requests made by this class.
2022-11-22 11:37:16 +01:00
AudricV
b9e463de49
[Bandcamp] Send Content-Type header in POST requests
...
This header was not sent before and was added and guessed by OkHttp. This can create issues when using other HTTP clients than OkHttp, such as Cronet.
Also make use of StandardCharsets.UTF_8 when getting bytes of bodies instead of the platform default's charset, to make sure to prevent some encoding issues on some JVMs.
2022-11-22 11:35:46 +01:00
AudricV
65d6321e3d
Fix typos in Downloader.post JavaDocs
...
Post methods in Downloader return the result of a POST request and not the one of a GET request.
2022-11-22 11:35:46 +01:00
ThetaDev
5daabd1793
fix : #976 search subscriber count extraction with channel handles
2022-11-22 02:17:10 +01:00
Kavin
c953e23414
Merge pull request #968 from AudricV/yt-support-no-video-info-renderers-for-streams
...
[YouTube] Support lack of video info renderers for streams
2022-11-16 20:20:01 +00:00
Tobi
2211a24b69
Merge pull request #971 from lrusso96/patch-1
...
[YouTube] Improve duration parsing
2022-11-16 16:14:54 +01:00
Tobi
f6eefdc76c
Merge pull request #975 from FireMasterK/jitpack
...
Use JDK 11 for Jitpack gradle builds
2022-11-15 20:23:21 +01:00
Kavin
a1f22be0dd
Use JDK 11 for Jitpack gradle builds.
2022-11-15 18:43:19 +00:00
Kavin
9bdad55508
Merge pull request #973 from FireMasterK/audio-track
...
Add support for extracting audio track information
2022-11-15 08:19:47 +00:00
Kavin
86f06b333a
Address review.
2022-11-14 00:05:31 +00:00
Kavin
b16e6082e1
Add test for audio stream languages.
2022-11-13 23:10:44 +00:00
Kavin
30909da1df
Fix audio track similar comparison for IDs.
2022-11-13 23:08:54 +00:00
Kavin
6d59cdbe3a
Add support for extracting audio tracks.
2022-11-13 21:39:29 +00:00
Tobi
43d1c1f8b1
Merge pull request #880 from Isira-Seneviratne/Use_StandardCharsets
...
Use StandardCharsets.UTF_8.
2022-11-12 03:23:44 +01:00
Isira Seneviratne
e4d982c7ea
Fix license.
2022-11-12 07:29:15 +05:30
Isira Seneviratne
416089146e
Fix failing tests.
2022-11-12 07:29:15 +05:30
Isira Seneviratne
ff5f223d3f
Update README.md
2022-11-12 07:29:15 +05:30
Isira Seneviratne
ddbce3b83d
Add Utils methods for URL encoding/decoding using UTF-8.
2022-11-12 07:29:15 +05:30
Isira Seneviratne
366f5c1632
Use StandardCharsets.UTF_8.
2022-11-12 07:29:15 +05:30
Tobi
0ea16c0b73
Merge pull request #954 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.10
...
Bump gson from 2.9.1 to 2.10
2022-11-09 17:28:31 +01:00
dependabot[bot]
0b82fade51
Bump gson from 2.9.1 to 2.10
...
Bumps [gson](https://github.com/google/gson ) from 2.9.1 to 2.10.
- [Release notes](https://github.com/google/gson/releases )
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md )
- [Commits](https://github.com/google/gson/compare/gson-parent-2.9.1...gson-parent-2.10 )
---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-11-09 16:21:16 +00:00
Luigi Russo
c9635218e2
[YouTube] Improve duration parsing
2022-11-09 09:41:29 +01:00
Isira Seneviratne
1be270768f
Merge pull request #893 from Isira-Seneviratne/Static_sets
...
Use immutable sets in YoutubeParsingHelper.
2022-11-09 09:07:01 +05:30
Isira Seneviratne
316d8573fa
Use immutable sets in YoutubeParsingHelper.
2022-11-07 07:50:26 +05:30
AudricV
6a2c680d8f
[YouTube] Add mocks for YoutubeStreamExtractorDefaultTest.NoVisualMetadataVideoTest
2022-11-04 19:43:06 +01:00
AudricV
e66fed41d6
[YouTube] Add a StreamExtractor test for a video without visual metadata
...
The video "Makani’s first commercial-scale energy kite" (video ID:
An8vtD1FDqs), which has this behavior, is used for the new test,
NoVisualMetadataVideoTest, added in YoutubeStreamExtractorDefaultTest.
Tests of elements who throw an exception in this case (subscriber count, like
count, uploader avatar URL) test if the ParsingException exception is thrown by
YoutubeStreamExtractor.
2022-11-04 19:42:12 +01:00
AudricV
aa9a8ca23c
[YouTube] Make non-extraction of videoPrimaryInfoRenderer and/or videoSecondaryInfoRenderer not fatal
...
Also de-duplicated common code related to the obtain of these video info renderers.
This change allows extraction of videos without visual metadata.
2022-11-04 18:35:53 +01:00
AudricV
eb07d70a2c
Merge pull request #964 from AudricV/yt-support-handles-and-all-channel-usernames
...
[YouTube] Support handles and all channel usernames
2022-11-04 12:09:15 +01:00
AudricV
20cd8e8a4a
[YouTube] Update mocks of YoutubeChannelExtractorTest.Gronkh
2022-11-03 19:46:42 +01:00
AudricV
a34f060ba0
[YouTube] Use a handle for YoutubeChannelExtractorTest.Gronkh
2022-11-03 19:46:42 +01:00
AudricV
724f669ff7
[YouTube] Add tests for handles and user IDs with non ASCII characters support
...
Unneeded public modifiers in test methods of
YoutubeChannelLinkHandlerFactoryTest have been also removed.
2022-11-03 19:46:42 +01:00
AudricV
61ce041bda
[YouTube] Support handles and all custom channel names
...
More non-channel paths have been also added to the excluded custom name paths,
documentation and exception messages have been improved and fixed in some
places, and the licence header of YoutubeChannelLinkHandlerFactory has been
moved to its beginning and updated.
2022-11-03 19:46:42 +01:00
AudricV
ffffb04439
Merge pull request #953 from Theta-Dev/attributed-text-desc
...
[YouTube] Add support for attributed text description
2022-11-03 18:34:30 +01:00
ThetaDev
592e1d6386
fix: parsing attributed description with no command runs
2022-11-03 12:10:52 +01:00
AudricV
31bf7046b7
Merge pull request #944 from Theta-Dev/fix-rich-grid-renderer
...
[YouTube] Support richGridRenderer on channel page to fix extraction of videos in channels
2022-11-03 00:10:54 +01:00
ThetaDev
099b53cc4f
[YouTube] Add parser for attributedDescription
...
Also update the mock of the next InnerTube endpoint response of the
YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing test class with an
attributedDescription instead of a regular description
2022-11-02 23:11:33 +01:00
AudricV
e4c24d4c36
[YouTube] Regenerate supported channels mocks
2022-11-02 19:13:59 +01:00
Theta-Dev
20e4a35814
[YouTube] Support richGridRenderer on channel pages
...
YouTube is deploying a new layout on their channel pages, which uses richGridRenderer JSON objects.
2022-11-02 19:01:29 +01:00
AudricV
4cae66f1f9
Merge pull request #946 from chowder/dev
...
Add ability to identify short-form `StreamInfoItem`s
2022-11-01 12:19:58 +01:00
Tobi
eb40bb8458
Merge pull request #959 from FireMasterK/playlist-info-item-uploader
...
Add uploaderUrl and uploaderVerified to PlaylistInfoItem.
2022-10-31 13:10:33 +01:00
chowder
b1a899fd47
Fix null pointer exception
2022-10-31 11:12:23 +00:00
Stypox
a4db106a66
Merge pull request #960 from AudricV/yt-workaround-403-errors-android-client
...
[YouTube] Workaround 403 HTTP errors of ANDROID client streams
2022-10-30 21:53:26 +01:00
Kavin
b441910257
Mark uploaderUrl as nullable.
2022-10-30 13:36:04 +00:00
chowder
3fdc0e72cc
Line breaks for long docstrings
2022-10-30 13:28:39 +00:00
Kavin
6a256d0631
Add uploader url and verified to PlaylistInfoItem.
2022-10-30 13:00:19 +00:00
Tobi
430504b4b5
Merge pull request #958 from AudricV/yt-playlists-support-new-metadata-format
...
[YouTube] Support new metadata format of playlists
2022-10-30 12:31:43 +01:00
Kavin
f9bd08c649
Address reviews.
2022-10-30 01:25:30 +00:00
Caleb
9282c3c13b
Fix exception message for YoutubeStreamInfoItemExtractor#isShortFormContent
...
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
2022-10-30 01:23:15 +00:00
Caleb
c5216f7c12
Update docstring for StreamExtractor#isShortFormContent
...
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
2022-10-30 01:23:15 +00:00
Caleb
04795fe5d2
Use Stream API for ShortFormContent#testShortFormContent
...
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
2022-10-30 01:23:15 +00:00
chowder
975b38a0b9
Regenerate test mocks
2022-10-30 01:23:15 +00:00
chowder
4cccd33f3d
Implement isShortFormContent for StreamExtractor and StreamInfo
2022-10-30 01:23:15 +00:00
chowder
09544ceb23
Fix tests
2022-10-30 01:23:15 +00:00
chowder
644cc6cd76
Rename test function name
2022-10-30 01:23:15 +00:00
chowder
daf5674951
Add ability to identify short-form StreamInfoItems
2022-10-30 01:23:12 +00:00
Tobi
3d314169b9
Merge pull request #943 from TeamNewPipe/fix/sc/comments
...
[SoundCloud] Fix getting more comments
2022-10-29 22:19:50 +02:00
AudricV
65125a3eb4
[YouTube] Fix YoutubePlaylistExtractorTest.LearningPlaylist
...
The video count is now returned for this playlist, so it isn't unknown.
The testStreamCount method of this test class asserts now that the stream count
is greater than 40.
2022-10-29 18:12:10 +02:00
AudricV
7258a53225
[YouTube] Support new playlist layout
...
This new layout doesn't provide author thumbnails and is completely different
for metadata, so the code to get them has been refactored.
The code of learning playlists video count check has been also removed, as it
seems to be not relevant anymore (the video count seems to be returned for
these playlists with both layouts).
Finally, unneeded overrides of subchannel methods, which don't apply to the
YouTube service, have been removed.
2022-10-29 18:12:10 +02:00
AudricV
e37e8b358e
[YouTube] Update mocks that need to be updated
2022-10-29 18:10:18 +02:00
AudricV
60e97cd274
[YouTube] Workaround getting streaming URLs returning 403 HTTP response codes
...
Using the player parameters used to get stories seems to fix the issue, which
affects currently only certain countries such as UK.
This is a workaround and should be fixed in a better way (by changing the
InnerTube additional client used for videos or finding what is now required in
Android player requests).
2022-10-29 17:58:33 +02:00
AudricV
4bc90cd9d8
Merge pull request #957 from AudricV/fix-checkstyle-error
...
Fix Checkstyle error in YoutubeCommentsInfoItemExtractor
2022-10-29 17:55:33 +02:00
AudricV
c230d84df1
Fix Checkstyle error in YoutubeCommentsInfoItemExtractor
2022-10-29 13:24:19 +02:00
Tobi
9ffdd0948b
Merge pull request #945 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.3
...
Bump spotbugs-annotations from 4.7.2 to 4.7.3
2022-10-18 22:55:24 +02:00
dependabot[bot]
24137bb15f
Bump spotbugs-annotations from 4.7.2 to 4.7.3
...
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.7.2 to 4.7.3.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.2...4.7.3 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-10-17 09:20:51 +00:00
xz-dev
0ffcb32d9c
[YouTube] Add comment reply count support ( #936 )
...
Add comment reply count support for YouTube and introduce `CommentsInfoItem.UNKNOWN_REPLY_COUNT` constant
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2022-10-15 12:40:06 +02:00
Isira Seneviratne
b90a566dd8
Add backport implementation of Locale.forLanguageTag().
2022-10-12 09:21:39 +05:30
Isira Seneviratne
b232c29d22
Use Locale.forLanguageTag().
2022-10-12 09:21:38 +05:30
TobiGr
4d136599bd
[SoundCloud] Fix getting more comments
2022-10-11 15:44:54 +02:00
Tobi
a822e91909
Merge pull request #939 from TurtleArmyMc/fix/SoundcloudPlaylistExtractor_track_order
...
Fix SoundcloudPlaylistExtractor: tracks are in correct order
2022-10-10 22:28:18 +02:00
TobiGr
02810a7db7
Add a comment
2022-10-10 22:22:12 +02:00
Tobi
2d50369c77
Merge pull request #888 from Isira-Seneviratne/EnumMap
...
Use EnumMap in PatternsHolder.
2022-10-10 12:00:06 +02:00
Tobi
54092fc3c7
Merge pull request #930 from Isira-Seneviratne/Avoid_Comparator_NPE
...
Avoid possible NullPointerException in MediaCCCRecentKiosk.
2022-10-10 11:43:34 +02:00
Tobi
d7c23ef118
Merge pull request #924 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.2
...
Bump spotbugs-annotations from 4.7.1 to 4.7.2
2022-10-10 11:35:40 +02:00
dependabot[bot]
08339d6b0c
Bump spotbugs-annotations from 4.7.1 to 4.7.2
...
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.7.1 to 4.7.2.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.1...4.7.2 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-10-10 09:32:17 +00:00
Tobi
5537e84100
Merge pull request #931 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.1
...
Bump junit-bom from 5.9.0 to 5.9.1
2022-10-10 11:30:53 +02:00
dependabot[bot]
719664593f
Bump junit-bom from 5.9.0 to 5.9.1
...
Bumps [junit-bom](https://github.com/junit-team/junit5 ) from 5.9.0 to 5.9.1.
- [Release notes](https://github.com/junit-team/junit5/releases )
- [Commits](https://github.com/junit-team/junit5/compare/r5.9.0...r5.9.1 )
---
updated-dependencies:
- dependency-name: org.junit:junit-bom
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-10-10 09:25:42 +00:00
TurtleArmyMc
bf70d32eb4
Fix SoundcloudPlaylistExtractor: tracks are in correct order
2022-09-30 15:26:25 -04:00
Stypox
5c710da160
Merge pull request #934 from AudricV/fix-yt-throttling-decryption-function-regex
...
[YouTube] Fix throttling decryption function regex
2022-09-25 13:26:08 +02:00
AudricV
8067c43837
[YouTube] Don't use a specific letter for the decryption function name pattern
...
Use the same possible characters for variables everywhere, in order to avoid
potential future throttling parameter decryption function name parsing issues
related to the usage of other letter(s) than b.
2022-09-24 21:49:22 +02:00
AudricV
abcee87167
[YouTube] Fix throttling parameter decryption function regex
...
- Quote the function name, as it may contain special regex symbols, such as
dollar;
- Support multiple lines;
- Use what looks like the end of the function for the end of the regex (this
part is inspired from yt-dlp throttling parameter decryption regex);
- Move the throttling function body regex into a private and static constant.
2022-09-24 21:28:09 +02:00
ThetaDev
7244be7627
[YouTube] Add JavaScript lexer to parse completely throttling decryption function ( #905 )
2022-09-24 19:55:46 +02:00
Isira Seneviratne
0e31c86aee
Avoid possible NullPointerException in MediaCCCRecentKiosk.
2022-09-21 05:40:07 +05:30
Stypox
a99af9bb6e
Merge pull request #927 from Stypox/fix-feed-thumbnail
...
[YouTube] Return mqdefault thumbnails in fast feed
2022-09-19 08:52:01 +02:00
Kavin
4e14644c41
Ensure comments are parsed with URLs and timestamps.
2022-09-17 16:03:39 +05:30
Stypox
14ef430546
Merge pull request #920 from AudricV/fix-yt-trending-extraction
...
[YouTube] Fix extraction of the Trending kiosk with the new data model returned
2022-09-14 11:14:13 +02:00
Stypox
93ef73e2f2
[YouTube] Return mqdefault thumbnails in feed
...
The hqdefault thumbnails, which are used by default, have black bars at the top and at the bottom, and are thus not optimal.
2022-09-13 16:46:54 +02:00
AudricV
d8ddeb549c
[YouTube] Support new trending structure and filter recently trending section
...
This new structure allow us to filter easily Trending shorts and Recently
trending sections.
On the previous one, this Recently trending section is now filtered, by
checking whether sections have a title, which isn't the case for normal trends
contrary to the other ones.
This makes that the extractor returns now only the real 50 "Now" YouTube
trends.
Elements inside arrays are now extracted dynamically instead of only the ones
of the first index, using Java 8's Stream API.
The getInitialPage() method of YoutubeTrendingExtractor can now throw a
ParsingException if no selected tab (corresponding to the one of the trends
type extracted) has been found.
Finally, the licence header has been moved to the top of the file and updated.
2022-09-11 18:52:12 +02:00
AudricV
884da40f65
Merge pull request #926 from AudricV/fix-yt-like-count-extraction
...
[YouTube] Fix extraction of like count with the new data model
2022-09-11 12:21:38 +02:00
AudricV
58b09cf78b
Merge pull request #925 from Theta-Dev/fix-td-regex
...
[YouTube] Fix throttling parameter decryption on Android
2022-09-10 23:30:50 +02:00
AudricV
119b9129e2
[YouTube] Fix extraction of like count with the new data model
...
In the new data model currently A/B tested or deployed, the like count button
data is the same, but its path has been changed.
The extraction of the like count has been also improved, by using the multiple
accessibility data available instead of the only one which was used before.
This accessibility data has been also deprioritized, because it is language
dependent when there is no view.
Like count is always known, even for age-restricted videos, so returning -1 if
the like count extraction fails on this type of videos is not needed and hides
an extraction error: that's the reason why this return has been removed and the
exception will always be thrown, even if a video is age-restricted.
2022-09-10 23:28:38 +02:00
ThetaDev
4905f74700
[YouTube] Fix throttling parameter decryption on Android
...
Escape the curly brace in the regular expression used to parse the throttling
parameter decryption function to allow its compatibility on Android.
2022-09-10 17:03:39 +02:00
Isira Seneviratne
e101d58ae7
Use EnumMap in PatternsHolder.
2022-08-26 08:15:21 +05:30
Stypox
6a858368c8
Merge pull request #912 from Stypox/downgrade-rhino
...
Downgrade Rhino since it uses a class not available on Android
2022-08-25 10:10:27 +02:00
Stypox
fa25a7280f
Downgrade Rhino since it uses a class not available on Android
...
Reverts #774 , see TeamNewPipe/NewPipe#8876
2022-08-25 09:49:13 +02:00
Stypox
25f0ce7305
Merge pull request #911 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.3
...
Bump jsoup from 1.15.2 to 1.15.3
2022-08-24 14:23:44 +02:00
Stypox
d8e10dd3ad
Merge pull request #907 from Isira-Seneviratne/Remove_EmptyString
...
Remove EMPTY_STRING.
2022-08-24 14:22:55 +02:00
dependabot[bot]
1d7a3a90a4
Bump jsoup from 1.15.2 to 1.15.3
...
Bumps [jsoup](https://github.com/jhy/jsoup ) from 1.15.2 to 1.15.3.
- [Release notes](https://github.com/jhy/jsoup/releases )
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES )
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.2...jsoup-1.15.3 )
---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-08-24 09:14:26 +00:00
Isira Seneviratne
41254ae12a
Remove annotation.
2022-08-24 06:59:17 +05:30
Isira Seneviratne
943b7c033b
Remove EMPTY_STRING.
2022-08-24 06:59:17 +05:30
litetex
20914a6035
Merge pull request #903 from TacoTheDank/updateGradle
...
Update gradle wrapper to 7.5.1
2022-08-23 16:21:56 +02:00
litetex
d6577e5e0b
Merge pull request #882 from litetex/fix-all-tests
...
Fix all tests
2022-08-23 16:19:27 +02:00
litetex
0a5ad90724
Merge pull request #901 from AudricV/remove-deprecated-code-yt-throttling-decrypter
...
[YouTube] Remove deprecated throttling decryption code
2022-08-22 17:13:00 +02:00
AudricV
03d9a4fe9d
Fix typos in YoutubeStreamExtractor.tryDecryptUrl
2022-08-21 21:17:03 +02:00
litetex
0beb55a232
Improve ``YoutubeThrottlingDecrypterTest``
2022-08-21 20:17:05 +02:00
litetex
9e93d6b193
YoutubeThrottlingDecrypter: Check if returned string is a valid JavaScript function
2022-08-21 20:16:45 +02:00
litetex
ed0a07af4e
YoutubeThrottlingDecrypter: Patch regex
2022-08-21 20:15:53 +02:00
litetex
641a447d9b
Updated consent-related mock-data
2022-08-21 18:43:48 +02:00
litetex
8ff7a90f52
Improved consent cookie related constants and documentation
2022-08-21 18:41:40 +02:00
TacoTheDank
d8339d4060
Update gradle wrapper to 7.5.1
2022-08-18 15:09:35 -04:00
AudricV
cb64a480cd
[YouTube] Remove deprecated code in YoutubeThrottlingDecrypter
...
YoutubeThrottlingDecrypter is now a non-instantiable final class and non-static
attributes have been removed.
The static attributes of this class have been renamed, in order to respect the
naming specification used.
2022-08-16 15:01:05 +02:00
litetex
5bbea3a8f2
Fix ``YoutubeMixPlaylistExtractorTest`` again
...
Doesn't work for kurgesagt videos anymore, used music video (copyright free) instead
2022-08-14 15:12:27 +02:00
litetex
da06166065
Updated mock data of ``YoutubeSearchExtractorTest$Suggestion``
2022-08-14 15:12:26 +02:00
litetex
2e36ab1578
Fixed ``YoutubeSearchExtractorTest$Suggestion``
2022-08-14 15:12:26 +02:00
litetex
2a8a623643
``YoutubePlaylistLinkHandlerFactoryTest``: Use parameterized tests
2022-08-14 14:48:31 +02:00
litetex
938e69a16a
Fixed ``YoutubePlaylistLinkHandlerFactoryTest``
2022-08-14 14:48:30 +02:00
litetex
844de3e378
Fix ``YoutubeStreamLinkHandlerFactoryTest `` and parameterized the tests
...
used code from #833
2022-08-14 14:48:30 +02:00
litetex
c1a72b8240
Use Android Studios code style
2022-08-14 14:48:29 +02:00
litetex
fc27b8a5b8
Use preexisting ``ContentNotAvailableException``
2022-08-14 14:48:29 +02:00
Stypox
17bad6cedf
Fix test about exception type: NPE, not IllegalArgment
...
The relevant change was made in #877
2022-08-14 14:48:28 +02:00
litetex
ecfc370685
Fixed all YTMixPlaylists
...
Added option to choose if you want to consent or not - currently this is done by a static variable in ``YoutubeParsingHelper`` - may not be the best long-term solution but for now the tests work again (in EU countries) 🥳
2022-08-14 14:48:27 +02:00
litetex
2b6fe294b2
Fixed ``YoutubeMusicSearchExtractorTest``
...
The renderers and queries got changed.
2022-08-14 14:48:27 +02:00
litetex
d6586da614
Failing test works locally without any problems. Reformatted it a bit.
2022-08-14 14:48:27 +02:00
litetex
bf3ae5e679
Fixed ``SoundcloudStreamLinkHandlerFactoryTest``
...
* Removed the unnecessary ``public``
* Use parameterized tests
* One song got removed (which caused the test failure), replaced it with another song
2022-08-14 14:48:27 +02:00
litetex
504f81036e
Fixed ``PeertubePlaylistExtractorTest``
...
Also removed the unnecessary ``public`` and reformatted the code a bit
2022-08-14 14:48:26 +02:00
litetex
e7c12258f4
Fixed ``BandcampSearchExtractorTest``
...
Also removed the unnecessary ``public``
2022-08-14 14:48:26 +02:00
litetex
9e8724df4d
Fixed ``YoutubeStreamExtractorLivestreamTest``
...
The "Lofi Girl"-stream got interrupted by a copyright strike and had to be restarted. Because of this a new id is now used.
2022-08-14 14:48:26 +02:00
Tobi
76aad92fa5
Merge pull request #890 from AudricV/yt-clients-update-improvements-and-extraction-fixes
...
[YouTube] Update clients version and improve extraction of API key and client version of WEB client
2022-08-12 23:32:41 +02:00
AudricV
472f5d9e9c
[YouTube] Update mocks
2022-08-12 19:20:32 +02:00
AudricV
7bdca33a87
[YouTube] Ensure that an additional player response is the correct one
...
If YouTube detect that requests come from a third party client, they may
replace the real player response by another one of a video saying that this
content is not available on this app and to watch it on the latest version of
YouTube. We can detect this by checking whether the video ID of the player
response returned is the same as the one requested by the extractor.
2022-08-12 19:20:31 +02:00
AudricV
c82317e318
[YouTube] Spoof more mobile clients
...
Additional parameters have been added to the player requests of ANDROID and IOS
clients:
- for both clients: osName and osVersion: their respective values are:
- for the ANDROID one: Android and 12;
- for the IOS one: iOS and 15.6.0.19G71.
- for the ANDROID client: androidTargetSdkVersion, with the Android SDK version
corresponding to the Android version used in the player requests of this
client. This parameter is now required with this client to be sure to get a
correct player response, otherwise, the one of a video saying that this
content is not available in this app and to watch it with the latest version
of YouTube can be returned instead;
- for the IOS client: deviceMake, with Apple as its value.
The iOS version sent in the IOS client player requests has been also updated to
the version 15.6 of the OS.
Finally, a comment about the requirement to use the signature timestamp from
the player JavaScript base file for HTML5 player requests on videos with
obfuscated URLs has been added and replaces a previous one which may be not
true.
2022-08-12 19:20:31 +02:00
AudricV
d0549a5a52
[YouTube] Update client versions and use a real version for the iOS client
...
The iOS version can be got easily in fact, by looking at the What's New section of the App Store' app page.
2022-08-12 19:20:31 +02:00
AudricV
d7e678aca2
[YouTube] Improve WEB client version and API key HTML extraction
...
Common code in WEB client version HTML extraction has been deduplicated, usage of the Java 8 Stream API has been made and initial data fallback has been used as a last resort.
This means that the client version extraction from regexes will be used before this fallback, as it doesn't contain the full client version.
This can be used as a way to fingerprint the extractor, even if it seems to be not the case.
2022-08-12 19:20:30 +02:00
AudricV
6a885ef5ab
Merge pull request #891 from Theta-Dev/fix-throttling-decrypter
...
[YouTube] Fix extraction of more complex nsig functions
2022-08-12 18:24:21 +02:00
AudricV
5b548340e8
[YouTube] Catch any exception in YoutubeThrottlingDecrypter.apply and improve docs
...
This will prevent any future extractor break due to decryption failure, like it was excepted to be the case before.
Some documentation about the throttling decryption has been also improved.
2022-08-12 17:49:36 +02:00
ThetaDev
52ded6e3d7
Handle curly braces inside strings in StringUtils.matchToClosingParenthesis
...
This is required to extract fully more complex YouTube nsig functions.
2022-08-12 16:32:00 +02:00
Stypox
d12003651b
Merge pull request #886 from Isira-Seneviratne/toArray_improvements
...
Make improvements to methods using toArray().
2022-08-06 22:34:23 +02:00
Isira Seneviratne
7daca10a06
Make improvements to methods using toArray().
2022-08-06 05:21:12 +05:30
Stypox
2906be22af
Merge pull request #881 from Isira-Seneviratne/String_join
...
Use String.join() and Collectors.joining().
2022-08-04 12:09:33 +02:00
Stypox
4ddb96a86f
Merge pull request #883 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.9.1
...
Bump gson from 2.9.0 to 2.9.1
2022-08-04 11:20:41 +02:00
Isira Seneviratne
64771c5712
Use String.join() and Collectors.joining().
2022-08-04 05:18:13 +05:30
Stypox
fc8b5ebbc6
Merge pull request #878 from Isira-Seneviratne/Use_Collections
...
Use Collections methods.
2022-08-03 22:50:41 +02:00
Stypox
4a4939d89c
Merge pull request #877 from Isira-Seneviratne/Use_Objects_requireNonNull
...
Use Objects.requireNonNull().
2022-08-03 22:45:35 +02:00
Stypox
c336bd58a5
Merge pull request #879 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.0
...
Bump junit-bom from 5.8.2 to 5.9.0
2022-08-03 21:06:20 +02:00
dependabot[bot]
325af31e5f
Bump gson from 2.9.0 to 2.9.1
...
Bumps [gson](https://github.com/google/gson ) from 2.9.0 to 2.9.1.
- [Release notes](https://github.com/google/gson/releases )
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md )
- [Commits](https://github.com/google/gson/compare/gson-parent-2.9.0...gson-parent-2.9.1 )
---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-08-01 09:26:14 +00:00
dependabot[bot]
d905636021
Bump junit-bom from 5.8.2 to 5.9.0
...
Bumps [junit-bom](https://github.com/junit-team/junit5 ) from 5.8.2 to 5.9.0.
- [Release notes](https://github.com/junit-team/junit5/releases )
- [Commits](https://github.com/junit-team/junit5/compare/r5.8.2...r5.9.0 )
---
updated-dependencies:
- dependency-name: org.junit:junit-bom
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-07-27 09:20:12 +00:00
Isira Seneviratne
1af6b8eedb
Use Collections.singletonList().
2022-07-27 07:35:57 +05:30
Isira Seneviratne
ff60e05c76
Use Collections.singletonMap().
2022-07-27 07:35:57 +05:30
Isira Seneviratne
682a4263e5
Use Objects.requireNonNull().
2022-07-27 06:55:26 +05:30
Stypox
8c5f014a6f
Merge pull request #869 from mhmdanas/add-workflow-permissions
...
Use minimum required permissions for GitHub workflows
2022-07-13 18:58:32 +02:00
Stypox
954a294e27
Merge pull request #870 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.2
...
Bump jsoup from 1.15.1 to 1.15.2
2022-07-13 17:48:06 +02:00
Mohammed Anas
f57049d2c0
Use temurin instead of adopt ( #868 )
...
See
https://blog.adoptopenjdk.net/2021/08/goodbye-adoptopenjdk-hello-adoptium/ .
2022-07-10 18:38:08 +03:00
Stypox
5bd7bf20cc
Merge pull request #867 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.1
...
Bump spotbugs-annotations from 4.7.0 to 4.7.1
2022-07-06 14:23:44 +02:00
Stypox
5ab74b3631
Merge pull request #857 from FireMasterK/video-title
...
Get original untranslated title for YouTube
2022-07-06 10:26:45 +02:00
dependabot[bot]
122365005a
Bump jsoup from 1.15.1 to 1.15.2
...
Bumps [jsoup](https://github.com/jhy/jsoup ) from 1.15.1 to 1.15.2.
- [Release notes](https://github.com/jhy/jsoup/releases )
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES )
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.1...jsoup-1.15.2 )
---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-07-04 09:38:16 +00:00
mhmdanas
414186cff5
Use minimum required permissions for GitHub workflows
...
This reduces the attack surface if the workflows are ever compromised.
2022-07-03 23:55:28 +03:00
dependabot[bot]
e9b4be3e3c
Bump spotbugs-annotations from 4.7.0 to 4.7.1
...
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.7.0 to 4.7.1.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.0...4.7.1 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-06-28 09:15:07 +00:00
Stypox
5219a705ba
Merge pull request #864 from AudricV/fetch-android-client-for-ended/post-livestreams
...
[YouTube] Fetch the ANDROID client for ended/post livestreams
2022-06-24 14:55:20 +02:00
AudricV
090debd83b
[YouTube] Fetch the ANDROID client for ended/post livestreams
...
The ANDROID client was only fetched for video contents, where it can be useful on ended/post livestreams, if the n parameter of the WEB client cannot be decrypted, to avoid throttling issues (because the WEB client was only used before for ended/post livestreams).
It also provides an exclusive 48kbps M4A audio format in the adaptiveFormats array of the JSON player response, like other mobile clients (which can be also extracted from the response of the DASH manifest URL returned into the WEB client player's response, but the DASH manifest is not used by the extractor).
A note about non-fatality of fetching or parsing issues of the ANDROID and IOS clients has been added.
2022-06-21 18:53:49 +02:00
litetex
a26bcc55c4
Merge pull request #845 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.0
...
Bump spotbugs-annotations from 4.6.0 to 4.7.0
2022-06-19 15:36:55 +02:00
dependabot[bot]
424eb1c559
Bump spotbugs-annotations from 4.6.0 to 4.7.0
...
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.6.0 to 4.7.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.6.0...4.7.0 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-06-19 13:29:18 +00:00
litetex
2712c3d788
Merge pull request #847 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.1
...
Bump jsoup from 1.14.3 to 1.15.1
2022-06-19 15:28:43 +02:00
dependabot[bot]
281d2b9f81
Bump jsoup from 1.14.3 to 1.15.1
...
Bumps [jsoup](https://github.com/jhy/jsoup ) from 1.14.3 to 1.15.1.
- [Release notes](https://github.com/jhy/jsoup/releases )
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES )
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.14.3...jsoup-1.15.1 )
---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-06-19 13:15:40 +00:00
litetex
f775155d25
Merge pull request #846 from litetex/remove-unused-methods
...
Remove unused methods
2022-06-19 15:12:15 +02:00
litetex
9d625dd75f
Merge pull request #860 from gliptak/patch-1
...
Correct PeertubePlaylistExtractorTest unit tests
2022-06-19 15:09:27 +02:00
Gábor Lipták
28c9340d69
Correct unit tests
2022-06-18 11:18:38 -04:00
Stypox
1b51eab664
Merge pull request #859 from AudricV/delivery-methods-fixes-and-improvements
...
Fix extraction of some properties in ItagItems for YouTube livestreams and post-live streams and remove completely SoundCloud HLS workaround
2022-06-17 15:46:13 +02:00
AudricV
301a795ed3
[SoundCloud] Remove completely workaround for HLS streams
...
SoundCloud is currently removing this workaround completely, so there is no need to keep it, because it impacts the loading time (a HLS playlist was downloaded and parsed).
2022-06-16 12:12:54 +02:00
AudricV
e960a417ec
[YouTube] Fix extraction of fps, audioSampleRate and audioChannels fields for ItagItems of live streams and post live streams
...
These values were only set before for video streams.
A fallback for the audio channels count has been added, in order to prevent exceptions when generating DASH manifests of audio streams: the fallback value is 2, because most audio streams on YouTube have 2 audio channels.
2022-06-16 12:12:54 +02:00
Stypox
c8a77da2ab
Merge pull request #810 from TiA4f8R/delivery-methods-v2
...
Support delivery methods other than progressive HTTP
2022-06-02 22:44:24 +02:00
Kavin
7635aeed2c
Get original untranslated title for YouTube.
2022-06-02 09:57:52 +01:00
TiA4f8R
287d1dfd63
[SoundCloud] Use the HLS delivery method for all streams and extract only a single stream URL from HLS manifest for MP3 streams
...
SoundCloud broke the workaround used to get a single file from HLS manifests for Opus manifests, but it still works for MP3 ones.
The code has been adapted to prevent an unneeded request (the one to the Opus HLS manifest) and the HLS delivery method is now used for SoundCloud MP3 and Opus streams, plus the progressive one (for tracks which have a progressive stream (MP3) and for the ones which doesn't have one, it is still used by trying to get a progressive stream, using the workaround).
Streams extraction has been also moved to Java 8 Stream's API and the relevant test has been also updated.
2022-05-29 19:08:18 +02:00
Stypox
b3c620f0d8
Apply code review and Streams rework
2022-05-28 12:00:58 +02:00
Stypox
d652e05874
[MediaCCC] Fix comments about containsSimilarStream
2022-05-28 12:00:58 +02:00
Stypox
044639c32b
Solve some review comments
2022-05-28 12:00:57 +02:00
litetex
c33d392958
Fixed typo XEE → XXE (Xml eXternal Entity attack)
...
See also:
https://en.wikipedia.org/wiki/XML_external_entity_attack
https://owasp.org/www-community/vulnerabilities/XML_External_Entity_(XXE)_Processing
2022-05-28 12:00:57 +02:00
TiA4f8R
fffbbee7f3
[YouTube] Add missing Nonnull annotations in getCache method of YouTube DASH manifest creators
2022-05-28 12:00:56 +02:00
TiA4f8R
f7b1515290
[YouTube] Refactor DASH manifests creation
...
Move DASH manifests creation into a new subpackage of the YouTube package, dashmanifestcreators.
This subpackage contains:
- CreationException, exception extending Java's RuntimeException, thrown by manifest creators when something goes wrong;
- YoutubeDashManifestCreatorsUtils, class which contains all common methods and constants of all or a part of the manifest creators;
- a manifest creator has been added per delivery type of YouTube streams:
- YoutubeProgressiveDashManifestCreator, for progressive streams;
- YoutubeOtfDashManifestCreator, for OTF streams;
- YoutubePostLiveStreamDvrDashManifestCreator, for post-live DVR streams (which use the live delivery method).
Every DASH manifest creator has a getCache() static method, which returns the ManifestCreatorCache instance used to cache results.
DeliveryType has been also extracted from the YouTube DASH manifest creators part of the extractor and moved to the YouTube package.
YoutubeDashManifestCreatorTest has been updated and renamed to YoutubeDashManifestCreatorsTest, and YoutubeDashManifestCreator has been removed.
Finally, several documentation and exception messages fixes and improvements have been made.
2022-05-28 12:00:56 +02:00
TiA4f8R
f17f7b9842
Apply requested changes in YoutubeParsingHelper
2022-05-28 12:00:55 +02:00
TiA4f8R
301b9fa024
Remove hashCode and equals methods overrides of Stream classes
2022-05-28 12:00:55 +02:00
TiA4f8R
2f3920c648
[YouTube] Return approxDurationMs value from YouTube's player response in ItagItems generated
...
This change allows to build DASH manifests using YoutubeDashManifestCreator with the real duration of streams and prevent potential cuts of the end of progressive streams, because the duration in YouTube's player response is in seconds and not milliseconds.
2022-05-28 12:00:54 +02:00
TiA4f8R
4158fc46a0
[Bandcamp] Fix regression of Opus radio streams extraction
...
When moving opus-lo into a constant, opus-lo was renamed to opus_lo and was only used if no MP3 stream was available (which was not the case before the changes in BandcampRadioStreamExtractor related to the addition of the support of all delivery methods), so these changes removed the ability to get Opus streams of Bandcamp radios.
This commit reverts this unwanted change.
2022-05-28 12:00:54 +02:00
TiA4f8R
54d323c2ae
Fix Checkstyle issue in YoutubeDashManifestCreator
2022-05-28 12:00:53 +02:00
Stypox
2321822844
Rename Stream's baseUrl to manifestUrl
2022-05-28 12:00:53 +02:00
Stypox
cfc13f4a6f
[YouTube] Reduce exception generation code and move several attributes of MPD documents into constants
...
Rename YoutubeDashManifestCreationException to CreationException.
Also use these constants in YoutubeDashManifestCreatorTest.
2022-05-28 12:00:53 +02:00
Stypox
00bbe5eb4d
[YouTube] Make exception messages smaller
2022-05-28 12:00:52 +02:00
Stypox
4da05afe11
[YouTube] Inline collectSegmentsData in YoutubeDashManifestCreator
2022-05-28 12:00:52 +02:00
Stypox
3708ab9ed5
[YouTube] Refactor YoutubeDashManifestCreator
...
- Remove all of the methods used to access caches and replace them with three caches getters
- Rename caches to shorter and more meaningful names
- Remove redundant @throws tags that just say "if this method fails to do what it should do", which is obvious
2022-05-28 12:00:51 +02:00
Stypox
5c83409039
[YouTube] Rewrite manifest test and rename long methods
2022-05-28 12:00:51 +02:00
Stypox
8226fd044f
[YouTube] Suppress Sonar security warning for XEE
2022-05-28 12:00:50 +02:00
Stypox
ba68b8c014
[YouTube] Secure DashManifestCreator against XEE attack
...
Also remove duplicate lines from javadoc comments, otherwise checkstyle complaints.
XEE prevention is inspired from https://github.com/ChuckerTeam/chucker/pull/201/files
For an explanation of XEE see https://rules.sonarsource.com/java/RSPEC-2755 , though the solution reported there causes crashes on Android
2022-05-28 12:00:50 +02:00
Stypox
159d05c91b
Remove unused DashMpdParser (but kept in git history)
2022-05-28 12:00:49 +02:00
Stypox
50272db946
Apply reviews: improve comments, remove FILE, remove Stream#equals(Stream)
2022-05-28 12:00:49 +02:00
TiA4f8R
07b045f20d
[YouTube] Support the iOS client in YoutubeDashManifestCreator and decrypt again the n parameter, if present, for all clients
...
This commits reverts a new behavior introduced in this branch, which only applied the decryption if needed on streams from the WEB client.
Also fix rebase issues and documentations style in YoutubeDashManifestCreator.
2022-05-28 12:00:48 +02:00
TiA4f8R
436ddde29f
Use assertThrows in YoutubeDashManifestCreatorTest
2022-05-28 12:00:48 +02:00
TiA4f8R
d64d7bbd01
Move ManifestCreatorCache tests to a separate class and remove override of equals and hashCode methods in ManifestCreatorCache
...
These methods don't need to be overriden, as they are not excepted to be used in collections.
Also improve the toString method of this class, which contains also now clearFactor and maximumSize attributes and for each operations.
2022-05-28 12:00:48 +02:00
TiA4f8R
2fb1a412a6
Fix Checkstyle issues, revert resolution string changes for YouTube video streams and don't return the rn parameter in DASH manifests
...
This parameter is still used to get the initialization sequence of OTF and POST-live streams, but is not returned anymore in the manifests.
It has been removed in order to avoid fingerprinting based on the number sent (e.g. when starting to play a stream close to the end and using 123 as the request number where it should be 1) and should be added dynamically by clients in their requests.
The relevant test has been also updated.
Checkstyle issues in YoutubeDashManifestCreator have been fixed, and the changes in the resolution string returned for video streams in YoutubeStreamExtractor have been reverted, as they create issues on NewPipe right now.
2022-05-28 12:00:47 +02:00
TiA4f8R
f61e2092a1
[YouTube] Return a copy of the hardcoded ItagItem instead of returning the reference to the hardcoded one in ItagItem.getItag
...
To do so, a copy constructor has been added in the class.
This fixes, for instance, an issue in NewPipe, in which the ItagItem values where not the ones corresponsing to a stream but to another, when generating DASH manifests.
2022-05-28 12:00:47 +02:00
TiA4f8R
aa4c10e751
Improve documentation and adress most of the requested changes
...
Also fix some issues in several places, in the code and the documentation.
2022-05-28 12:00:46 +02:00
TiA4f8R
6985167e63
Add tests for YoutubeDashManifestCreator and ManifestCreatorCache
...
The test added in YoutubeDashManifestCreator uses a video of the Creative Commons channel, licenced under the Creative Commons Attribution licence (reuse allowed).
Also remove public keywords of tests in UtilsTest, as suggested by SonarLint, because they are not needed with Junit 5.
2022-05-28 12:00:46 +02:00
TiA4f8R
f6ec7f9a61
Update DefaultStreamExtractorTest and SoundcloudStreamExtractorTest to support changes made in Stream classes
2022-05-28 12:00:45 +02:00
TiA4f8R
7477ed0f3d
[YouTube] Add ability to generate manifests of progressive, OTF and post live streams
...
A new class has been added to do so: YoutubeDashManifestCreator.
It relies on a new class: ManifestCreatorCache, to cache the content, which relies on a new pair class named Pair.
Results are cached and there is a cache per delivery type, on which cache limit, clear factor, clearing and resetting can be applied to each cache and to all caches.
Look at code changes for more details.
2022-05-28 12:00:45 +02:00
TiA4f8R
a857684442
Apply changes in YoutubeStreamExtractor
...
Extract post live DVR streams as post live streams instead of live streams.
A new class has been in order to improve code: ItagInfo, which stores an itag, the content (URL) extracted and if its an URL or not.
A functional interface has been added in order to abstract the stream building: StreamBuilderHelper.
Also add the cver parameter added by the desktop web client on the corresponding streams (a new method has been added in YoutubeParsingHelper to check this and another for Android streams).
Some code in these classes has been also refactored/improved/optimized.
2022-05-28 12:00:44 +02:00
TiA4f8R
4330b5f7be
Add POST_LIVE_STREAM and POST_LIVE_AUDIO_STREAM stream types
...
This allows the extractor to determine if a content is an ended audio or video livestream.
2022-05-28 12:00:43 +02:00
TiA4f8R
881969f1da
Apply changes in all StreamExtractors except YouTube's one and fix extraction of PeerTube audio streams as video streams
...
Some code in these classes has been also refactored/improved/optimized.
Also fix the extraction of PeerTube audio streams as video streams, which are now returned as audio streams.
2022-05-28 12:00:43 +02:00
TiA4f8R
d5f3637fc3
[YouTube] Return more values returned inside the ItagItems of the player response and deprecate use of public audio and video fields
...
These fields can be now replaced by a getter and a setter.
New fields have been added and will allow the creation of DASH manifests for OTF and ended livestreams. There are:
- contentLength;
- approxDurationMs;
- targetDurationSec;
- sampleRate;
- audioChannels.
2022-05-28 12:00:42 +02:00
TiA4f8R
7c67d46e09
Move DashMpdParser to the YouTube package and fix extraction of streams
...
DashMpdParser is only working with YouTube streams, as it uses the ItagItem class.
Also update creation of AudioStreams and VideoStreams objects.
2022-05-28 12:00:41 +02:00
TiA4f8R
ad993b920f
Remove fetching of the DASH manifest extracted when getting information of a content with StreamInfo
...
DashMpdParser is only working with YouTube streams, as it uses the ItagItem class.
Also improve code and comments of StreamInfo (especially final use where possible).
2022-05-28 12:00:41 +02:00
TiA4f8R
2f061b8dbd
Add support of other delivery methods than progressive HTTP in Stream classes
...
Stream constructors are now private and streams can be constructed with new Builder classes per stream class. This change has been made to prevent creating and using several constructors in stream classes.
Some default cases have been also added in these Builder classes, so not everything has to be set, depending of the service and the content.
2022-05-28 12:00:27 +02:00
Stypox
1dc80957d8
Merge pull request #848 from TiA4f8R/fix_yt_music_website_extraction_eu
...
[YouTube] Fix extracting YouTube Music client version and key in the EU
2022-05-22 22:05:05 +02:00
TiA4f8R
c34b5e3a8b
[YouTube] Fix extraction of YouTube Music client version and API key when using YouTube Music's website in EU
...
Google returns now the consent page of YouTube for YouTube Music in EU, which can be also avoided by adding the ucbcb parameter to the URL with the value 1 ("?ucbcb=1").
2022-05-15 11:20:06 +02:00
litetex
2015eb374a
Removed more unused methods
2022-05-09 21:05:03 +02:00
litetex
f69b0ff77b
Remove unused methods
2022-05-09 20:59:25 +02:00
Stypox
23fa31a1ec
Merge pull request #843 from TiA4f8R/fix-checkstyle-issues
...
Remove Checkstyle suppressions file and fix Checkstyle issues
2022-05-02 22:07:27 +02:00
TiA4f8R
3c3cd78676
Remove Checkstyle suppressions file and fix Checkstyle issues introduced in 24e8399
and 8c1041d
...
The Checkstyle suppressions file is now replaced by // CHECKSTYLE:OFF and // CHECKSTYLE:ON comments.
2022-05-02 21:51:25 +02:00
Stypox
2e1c5c119d
Merge pull request #822 from Stypox/more-refactors
...
More refactors
2022-05-02 19:03:54 +02:00
Stypox
598ebb92ea
Merge pull request #839 from TeamNewPipe/bandcamp/extract-length
...
Bandcamp: extract stream length
2022-05-02 15:49:41 +02:00
litetex
5db4d1faf3
Merge pull request #782 from litetex/cleanup-yt-stream-extractor
...
Cleanup of ``YoutubeStreamExtractor`` and some related classes
2022-05-01 16:44:11 +02:00
litetex
fe30eb43a9
Cleanup ``YoutubeStreamExtractor`` and some related classes
...
* Fixed obvious sonar(lint) warnings
* Abstracted some code (get*Streams)
* Used some new lines to make code better readable
* Chopped down brace-jungle in some methods
* Use StandardCharset (Java 8 4tw)
2022-05-01 16:39:07 +02:00
Stypox
c2b5370517
Apply suggestions: improve switch and use EMPTY_STRING
2022-04-30 16:39:51 +02:00
Stypox
7c78c39230
Merge pull request #821 from litetex/cleanup-TimeAgoParser-java
...
Cleanup ``TimeAgoParser``
2022-04-30 16:20:09 +02:00
Stypox
ac1c22d81c
Release NewPipe Extractor 0.22.1
2022-04-28 11:05:52 +02:00
Stypox
3f5c8962ac
Merge pull request #841 from TiA4f8R/yt-respect-order-of-clients-for-streams
...
[YouTube] Use correct order of clients to get better streams first
2022-04-28 10:58:49 +02:00
TiA4f8R
9f9af35adb
[YouTube] Fix regression introduced in the order of streams used when adding more parameters to InnerTube requests, using the iOS client for livestreams and more
2022-04-25 20:23:04 +02:00
Fynn Godau
c38c016de5
Bandcamp: extract stream length
2022-04-24 21:24:19 +02:00
litetex
5f8f3929bd
Merge pull request #838 from Nickoriginal/update_user_agent
...
Update User-Agent in NPE
2022-04-24 14:52:18 +02:00
Nickoriginal
b5f3f9eb90
Update USER_AGENT to match the latest Firefox
2022-04-24 00:27:23 +03:00
litetex
9f665db4e1
Merge pull request #828 from TeamNewPipe/dependabot/github_actions/actions/setup-java-3
...
Bump actions/setup-java from 2 to 3
2022-04-23 00:09:44 +02:00
Stypox
52fa2d939a
Fix javadoc formatting error causing deployment to fail
2022-04-16 17:07:07 +02:00
Stypox
95a4b05acc
Release NewPipe Extractor 0.22.0
2022-04-16 16:57:58 +02:00
Tobi
b77c72fb88
Merge pull request #832 from Stypox/hotfix-decrypter
...
Fix YouTube throttling decrypter function parsing
2022-04-15 13:35:10 +02:00
Stypox
dcb7483dcf
Fix YouTube throttling decrypter function parsing
2022-04-15 13:10:19 +02:00
Robin
bebdc55ad4
Merge pull request #829 from TiA4f8R/yt-subtitles-age-restricted-videos-fix
...
[YouTube] Extract subtitles for age-restricted videos
2022-04-14 10:31:16 +02:00
TiA4f8R
ef49cd0007
[YouTube] Extract subtitles for age-restricted videos
...
Subtitles of age-restricted videos can be extracted since the InnerTube API migration, so there is no reason to not extract them anymore.
2022-04-11 22:09:56 +02:00
dependabot[bot]
00b51a2bf8
Bump actions/setup-java from 2 to 3
...
Bumps [actions/setup-java](https://github.com/actions/setup-java ) from 2 to 3.
- [Release notes](https://github.com/actions/setup-java/releases )
- [Commits](https://github.com/actions/setup-java/compare/v2...v3 )
---
updated-dependencies:
- dependency-name: actions/setup-java
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-04-11 09:16:36 +00:00
TiA4f8R
2e92d718a2
Merge pull request #780 from TiA4f8R/yt-more-params-innertube-requests
...
[YouTube] Add more parameters to InnerTube requests, use the iOS client for livestreams and fix extraction of embeddable age-restricted videos and contents with a warning before playback
2022-04-09 18:15:18 +02:00
TiA4f8R
b30e341559
Add link to learn more about the issue which makes YoutubeStreamExtractorRelatedMixTest.testRelatedItems test disabled
2022-04-04 20:36:59 +02:00
TiA4f8R
70812fa611
Update YouTube stream mocks and disable YoutubeStreamExtractorRelatedMixTest
...
Mixes seems to be not given by YouTube anymore if you use a PENDING consent cookie value.
As mocks needs to updated, the test is always failing because of this change.
2022-04-02 19:28:49 +02:00
TiA4f8R
67288a0191
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
...
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 19:06:36 +02:00
TiA4f8R
11b5a222c4
Deduplicate code of getStringResultFromRegexArray methods in Utils
...
Also revert indentation in Utils.mixedNumberWordToLong.
2022-04-02 18:40:00 +02:00
TiA4f8R
9ca647a750
Update mocks
2022-03-27 22:10:59 +02:00
TiA4f8R
dfa4239661
Fix missing imports and Checkstyle issues
2022-03-27 22:10:57 +02:00
TiA4f8R
6d27996ac4
Improve code of getStringResultFromRegexArray methods in Utils
2022-03-27 22:10:57 +02:00
TiA4f8R
2e3da445e6
[YouTube] Add documentation about parameters added and clients versions and key
...
Also move the iPhone device machine id to a constant, explain how it is used and move the licence in the header of the file, and fix missing imports in YoutubeStreamExtractor (due to a rebase issue).
2022-03-27 22:10:57 +02:00
TiA4f8R
1dad3bfe8b
[YouTube] Update again hardcoded client versions and update mobile user agents
...
Also provide ability to get mobile user-agents used for mobile InnerTube requests and deduplicate related code.
2022-03-27 20:52:40 +02:00
TiA4f8R
3d38459cf3
[YouTube] Reduce InnerTube response sizes by adding the prettyPrint parameter with the false value
...
InnerTube responses return pretty printed responses, which increase responses' size for nothing.
By using the prettyPrint parameter on requests and setting its value to false, responses are not pretty printed anymore, which reduces responses size, and so data transfer and processing times.
This usage has been recently deployed by YouTube on their websites.
2022-03-27 20:52:40 +02:00
litetex
349ba8db7f
Improve tests and randomness
...
- Use the existing RNG inside YoutubeParsingHelper
- Deduplicated test-setup for YouTube tests
- Minor improvements
2022-03-27 20:52:38 +02:00
TiA4f8R
52376949e5
Add setSeedForVideoTests method in YoutubeStreamExtractor tests
...
In order to use still use mocks with the generation of random strings in player requests, we need to use YoutubeParsingHelper.setSeedForVideoTests() method in every stream test.
2022-03-27 20:51:39 +02:00
TiA4f8R
d0d91e6690
Adress requested changes
2022-03-27 20:51:39 +02:00
TiA4f8R
b6bc521f0d
[YouTube] Update client versions again
2022-03-27 20:51:38 +02:00
TiA4f8R
26f93f5bb0
[YouTube] Extract streams of livestreams from the iOS client and disabled the Android client for livestreams
...
The iOS client is only enabled for livestreams and the Android client is now only enabled for videos, both by default.
A way to force, or not, the fetch of both clients have been added with two new static methods in YoutubeStreamExtractor.
2022-03-27 20:51:38 +02:00
TiA4f8R
7d07924de8
[YouTube] Try to use lighter requests when extracting client version and key from YouTube and YouTube Music
...
This is done by fetching https://www.youtube.com/sw.js for YouTube and https://music.youtube.com/sw.js for YouTube Music.
Two new methods in Utils class have been added which allow to try to get a match of regular expressions in a string array, or a Pattern array, on a content, on a specific index or 0.
Also some code refactoring has been made in this class.
2022-03-27 20:51:38 +02:00
TiA4f8R
05b7fee23b
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
...
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
2022-03-27 20:51:38 +02:00
TiA4f8R
83f374bff1
[YouTube] Update client versions and fix a bug when using resetClientVersionAndKey method
...
The boolean keyAndVersionExtracted in YoutubeParsingHelper was not set to false when resetting the client version and the key, which makes the extractor uses null on the next getting of the client version or the key if the clientVersion and the key were extracted before.
Also update client versions.
2022-03-27 20:51:38 +02:00
Stypox
8c1041def6
Add @ null annotations where Android Studio suggested it
...
That is, basically where the overriding function was missing an annotation from the base method.
Also apply renaming of emptyDescription to EMPTY_DESCRIPTION
2022-03-26 22:07:14 +01:00
Stypox
adbbdc7a5b
[YouTube] Fix regex warning: use ' {2}' instead of ' '
2022-03-26 22:07:14 +01:00
Stypox
24e83997b4
[Bandcamp] Add Java 8 streams
2022-03-26 22:07:12 +01:00
Stypox
349990fd48
Fix redundant escape \\ in regex in Utils
2022-03-26 22:01:30 +01:00
litetex
3bf7aa3762
Cleanup ``TimeAgoParser``
2022-03-26 21:09:31 +01:00
litetex
5a18730845
Merge pull request #813 from litetex/fix-test-2022-03
...
Fixed tests (+ Youtube shorts in channels)
2022-03-26 21:09:10 +01:00
litetex
af82edf9dc
Fix checkstyle problems
2022-03-26 20:54:20 +01:00
litetex
29408f9356
``YoutubeMusicSearchExtractorTest$Suggestion`` has no mock data
2022-03-26 20:52:28 +01:00
litetex
fcee247f9f
Switched to mockonly tests
...
* Made the reason why the tests are mockonly mandatory
2022-03-26 20:52:28 +01:00
litetex
0fceb4686b
Added missing mock data
2022-03-26 20:52:27 +01:00
litetex
70b20f0d6b
Updated mock data after conflicts
2022-03-26 20:52:27 +01:00
litetex
6a6c9359af
Fixed kurzgesagt test - They changed their description...
2022-03-26 20:52:27 +01:00
litetex
26596215fa
Refactored ``MediaCCCRecentListExtractorTest``
2022-03-26 20:52:26 +01:00
litetex
53962bfd7d
Disabled suggestion test for now
...
Has to be done once YT's backed works again...
2022-03-26 20:52:26 +01:00
litetex
66dc5e8bb8
API hardening against changes
2022-03-26 20:52:26 +01:00
litetex
5d58156cde
Update mock data
2022-03-26 20:52:25 +01:00
litetex
7598b40957
Workaround for incorrect duration for "YT shorts" videos in channels
...
As a workaround 0 is returned as duration for such videos.
See also https://github.com/TeamNewPipe/NewPipe/issues/8034
2022-03-26 20:52:24 +01:00
litetex
164e21b5af
Fixed ``MediaCCCRecentKiosk``
...
Ignore faulty data/items (with duration <= 0)
2022-03-26 20:52:23 +01:00
litetex
49dfdae5d2
Removed useless throw declaration
2022-03-26 20:51:38 +01:00
litetex
69a58fd3d1
Don't use sysout
2022-03-26 20:51:37 +01:00
litetex
639be7adda
Minimized some code
2022-03-26 20:51:37 +01:00
litetex
2bd4299563
Fixed test: Peertube - Use new comments and videos
...
Old comments/videos got deleted
2022-03-26 20:49:39 +01:00
litetex
606a386a98
Fix tests: What's peertube has less comments now
2022-03-26 20:49:39 +01:00
litetex
4d1a1c8fb8
Better test for ``MediaCCCRecentListExtractorTest``
...
* Use assertAll
* Show which item is affected
2022-03-26 20:49:38 +01:00
litetex
ba43dbaa28
Fix Tests: The channel lost abos
2022-03-26 20:49:38 +01:00
litetex
ff436e5740
Fixed test: YT mix playlist
2022-03-26 20:49:38 +01:00
litetex
bb49f7d857
Merge pull request #817 from Stypox/checkstyle
...
Add checkstyle
2022-03-26 20:46:49 +01:00
litetex
9c07e8a664
Fix useage of wrong object
2022-03-26 20:17:50 +01:00
litetex
93d6e5c4d5
CI: Upload test reports when failure occurs
2022-03-26 19:57:17 +01:00
litetex
804e57004f
Fixed new checkstyle problems from dev
2022-03-26 19:46:10 +01:00
litetex
33347ac18b
Removed unused methods
...
``contentFilters`` and ``sortfilter`` are get inside the ``ListLinkHandler`` and not the ``ListLinkHandlerFactory``
``ListLinkHandlerFactory`` only passes these values through when ``fromQuery`` is called
2022-03-26 19:43:11 +01:00
litetex
ec5b54c38b
Removed unused class
2022-03-26 19:43:10 +01:00
litetex
8771af7ba5
Restored original naming
2022-03-26 19:43:09 +01:00
litetex
01cfde0285
Fixed typo
2022-03-26 19:43:09 +01:00
Stypox
bdadcfa1f7
Legitimately suppress remaining checkstyle warnings
2022-03-26 19:43:08 +01:00
Stypox
740a37a2de
[YouTube] Fix checkstyle issues
2022-03-26 19:42:40 +01:00
Stypox
9dc17cd1ca
[Soundcloud] Fix checkstyle issues
2022-03-26 19:40:20 +01:00
Stypox
9ab32cb2e7
[Peertube] Fix checkstyle issues
2022-03-26 19:40:19 +01:00
Stypox
9f7e06c817
[MediaCCC] Fix checkstyle issues
2022-03-26 19:40:18 +01:00
Stypox
3a94839359
[Bandcamp] Fix checkstyle issues
2022-03-26 19:40:17 +01:00
Stypox
08dff33002
Use Java 8 streams in NewPipe class
2022-03-26 19:40:15 +01:00
Stypox
c2446ecff0
Use Java 8 streams and deduplicate code in MediaFormat class
2022-03-26 19:40:15 +01:00
Stypox
d79e20340c
Fix checkstyle issues in root package extractor/
...
Note: not all issues were fixed because MediaFormat and ServiceList use a specific formatting that makes sense for them
2022-03-26 19:40:14 +01:00
Stypox
ca7c63f273
Fix remaining checkstyle issues in utils/ subpackage
2022-03-26 19:40:13 +01:00
Stypox
1d5f22e41f
Fix checkstyle issues & more in JsonUtils
...
Also use Java 8 streams and extract duplicate code to getInstanceOf function
2022-03-26 19:40:13 +01:00
Stypox
87d2834986
Fix checkstyle issues & more in DonationLinkHelper
...
Also add comment about the class being unused and replace the fixLink function with Utils.stringToUrl()
2022-03-26 19:40:12 +01:00
Stypox
bd7b362040
Fix checkstyle issues & more in DashMpdParser
...
Also remove useless null check on ItagItem.getItag() as that function already throws an exception if there is no itag
2022-03-26 19:40:11 +01:00
Stypox
8aba2b47b0
Fix checkstyle issues in subpackages with abstract classes
2022-03-26 19:40:10 +01:00
Stypox
e4951a0623
Refactor code handling http headers in downloader.Request
2022-03-26 19:37:47 +01:00
Stypox
37690058d2
Add checkstyle to extractor gradle project
...
With respect to NewPipe's checkstyle.xml, checkstyle is disabled for javadoc comments. There is no need for strict rules over comments here in the extractor, as sometimes javadocs are just needed to clarify a small thing and having empty/meaningless @param or @throws is useless.
2022-03-26 19:37:46 +01:00
litetex
9284569c84
Merge pull request #774 from TeamNewPipe/dependabot/gradle/org.mozilla-rhino-1.7.14
...
Bump rhino from 1.7.13 to 1.7.14
2022-03-26 17:25:55 +01:00
litetex
358e619a27
Merge pull request #819 from TeamNewPipe/dependabot/github_actions/actions/cache-3
...
Bump actions/cache from 2 to 3
2022-03-23 21:02:18 +01:00
dependabot[bot]
a2c9ad104c
Bump actions/cache from 2 to 3
...
Bumps [actions/cache](https://github.com/actions/cache ) from 2 to 3.
- [Release notes](https://github.com/actions/cache/releases )
- [Commits](https://github.com/actions/cache/compare/v2...v3 )
---
updated-dependencies:
- dependency-name: actions/cache
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-03-21 09:12:36 +00:00
XiangRongLin
aa6b7272a4
Merge pull request #804 from Stypox/fix-yt-music-mix
...
[YouTube] Fix music mixes in some countries
2022-03-20 08:35:56 +01:00
Stypox
843945f2a6
Merge pull request #818 from litetex/update-license
...
Update the license to the latest version
2022-03-19 23:13:55 +01:00
litetex
fa76098750
Delete copyright
...
was replace by LICENSE-file
2022-03-19 21:44:38 +01:00
litetex
faaf4cbec3
Update the license to the latest version of https://www.gnu.org/licenses/gpl-3.0.txt
2022-03-19 17:59:27 +01:00
XiangRongLin
7f2ea133f0
Merge pull request #788 from Stypox/mix
...
Add MixInfoItem and extract YouTube mixes in related items
2022-03-19 16:46:01 +01:00
Stypox
09ddb6adbb
[YouTube] Add MockOnly to method testing mixes in related items
2022-03-19 10:54:38 +01:00
Stypox
8201b3b90e
[YouTube] Parse any playlist (including music mixes) in related items
2022-03-19 10:48:13 +01:00
Stypox
13f7900816
[YouTube] Add test for genre mix
2022-03-19 10:48:13 +01:00
Stypox
279f3a20fe
[YouTube] Fix mix tests with invalid video ids
...
Replaces mix tests based on a strange mix type RDQM{videoId} (only reference I could find is https://github.com/ytdl-org/youtube-dl/issues/26228 ) and with an invalid video id of 13 characters (the first two characters were QM, but even after removing QM there still wasn't a video available at that id).
Also updates mocks.
2022-03-19 10:48:13 +01:00
Stypox
d660c04838
[YouTube] Also test playlist type in playlist tests
2022-03-19 10:48:13 +01:00
Stypox
401082abe4
[YouTube] Extract playlist type in playlist extractor
2022-03-19 10:48:12 +01:00
Stypox
63ed06a710
[YouTube] Differentiate genre mixes from normal mixes
...
Note: genre mixes already worked, now they are just considered as such in various video id extraction and in related items
Note 2: now extracting a mix id from a *normal* youtube mix id will fail if the video id wouldn't be exactly 11 characters long
2022-03-19 10:46:31 +01:00
Stypox
f19660e7d9
[YouTube] Deduplicate code extracting video id from mix id
2022-03-19 10:46:30 +01:00
Stypox
8f9d5b858e
[YouTube] Remove useless comments about mixes
2022-03-19 10:44:06 +01:00
Stypox
34a4484c72
[YouTube] Add test for a video with a mix in related items
2022-03-19 10:44:06 +01:00
Stypox
50db871d89
[YouTube] Extract mixes from streams related items
2022-03-19 10:44:06 +01:00
Stypox
638da1756c
[Mix] Create MultiInfoItemsCollector
...
It is a collector that can handle many extractor types, to be used when a list contains items of different types (e.g. search). It was renamed from InfoItemsSearchCollector so that it can now be used not just for search but for any extractor needing it. It supports, streams, channels, playlists and *mixes*.
2022-03-19 10:44:06 +01:00
Stypox
53673d64c6
[Mix] Add type to playlists & playlist items, to distinguish mixes
2022-03-19 10:44:06 +01:00
Stypox
d8f2031619
Merge pull request #816 from Stypox/mock-only-extension
...
Add `@MockOnly` Junit 5 extension
2022-03-19 10:40:38 +01:00
litetex
cc2e4d7104
Merge pull request #815 from litetex/fix-soundcloud-id-once-and-for-all
...
Removed hardcoded soundcloud HARDCODED_CLIENT_ID
2022-03-17 13:54:08 +01:00
litetex
cd8088b217
Merge pull request #811 from TiA4f8R/playlists-improvements-and-yt-playlists-fixes
...
[YouTube] Fix the extraction of series playlists and don't return the view count as the stream count for learning playlists
2022-03-17 13:52:24 +01:00
TiA4f8R
c7757c0994
Apply requested changes
2022-03-16 20:14:08 +01:00
TiA4f8R
35e082248e
Fix YouTube and SoundCloud playlists tests
2022-03-16 19:40:30 +01:00
TiA4f8R
8b3f90eb7e
[YouTube] Fix extraction of series playlists and don't return the view count as the stream count for learning playlists
...
ITEM_COUNT_UNKNOWN is returned when the JSON array which contains usally the number of videos is less than 3 items.
Also apply the same type of optimizations done in other PlaylistExtractors in YoutubePlaylistExtractor.
2022-03-16 19:18:58 +01:00
TiA4f8R
58a247907e
Apply changes in all playlist extractors except YoutubePlaylistExtractor
...
Also fix some issues in the extractors, remove uneeded overrides, use the Java 8 Stream API where possible and replace usages of Utils.UTF_8 with StandardCharsets.UTF_8 in these classes.
2022-03-16 19:18:57 +01:00
TiA4f8R
fc6b45ee36
Implement some methods in PlaylistExtractor
...
This will prevent their override in each child class where the values corresponding to the methods could not be extracted.
2022-03-16 19:18:36 +01:00
Stypox
73d1fd472f
Add MockOnly junit 5 test extension
2022-03-16 19:03:08 +01:00
Stypox
ef71a5fa0f
static final instead of final static
2022-03-16 17:24:33 +01:00
Stypox
0c37c75981
Make getDownloader static & extract getDownloaderType
2022-03-16 17:22:42 +01:00
Stypox
40aa5104b1
Merge pull request #786 from XiangRongLin/throttling_resilience
...
[Youtube] Make throttling decryption more resilient to api change
2022-03-16 11:03:16 +01:00
litetex
ba56be8ef1
Removed hardcoded soundcloud id
...
It never works (long enough) so let's simply remove it...
2022-03-15 21:19:19 +01:00
XiangRongLin
e726437da3
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java
...
Co-authored-by: Stypox <stypox@pm.me>
2022-03-15 17:10:05 +01:00
litetex
e7aee0ca57
Merge pull request #807 from FireMasterK/no-commentsinfo-instance
...
Remove the need for a CommentsInfo instance in CommentsInfo.getMoreItems and fix PeertubeCommentsExtractorTest.Default test
2022-03-15 15:06:56 +01:00
litetex
d806984aa8
Merge pull request #800 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.9.0
...
Bump gson from 2.8.9 to 2.9.0
2022-03-14 19:47:54 +01:00
litetex
2aa5f98c26
Merge pull request #796 from FireMasterK/streaminfo-subscriber-count
...
Add support for extracting channel subscriber count in StreamInfo
2022-03-14 19:40:57 +01:00
Tobi
a267093e76
Merge pull request #809 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.6.0
...
Bump spotbugs-annotations from 4.5.3 to 4.6.0
2022-03-12 00:02:51 +01:00
dependabot[bot]
fdb603e9ac
Bump spotbugs-annotations from 4.5.3 to 4.6.0
...
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.5.3 to 4.6.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.5.3...4.6.0 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-03-08 09:16:28 +00:00
FireMasterK
60cc71e944
Remove the need for a CommentsInfo instance.
2022-03-03 11:48:41 +00:00
Stypox
3a6269bca6
Merge pull request #805 from TeamNewPipe/dependabot/github_actions/actions/checkout-3
...
Bump actions/checkout from 2 to 3
2022-03-02 11:09:28 +01:00
dependabot[bot]
e953794703
Bump actions/checkout from 2 to 3
...
Bumps [actions/checkout](https://github.com/actions/checkout ) from 2 to 3.
- [Release notes](https://github.com/actions/checkout/releases )
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md )
- [Commits](https://github.com/actions/checkout/compare/v2...v3 )
---
updated-dependencies:
- dependency-name: actions/checkout
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-03-02 09:25:59 +00:00
Stypox
dd8687f9fe
[YouTube] Fix music mixes in some countries
2022-03-01 23:02:56 +01:00
FireMasterK
e6d334765d
Apply requested codestyle improvements.
2022-02-26 17:12:51 +00:00
FireMasterK
6950b362f2
Add test values for failing tests.
2022-02-24 12:55:13 +00:00
FireMasterK
5b0ec694a6
Add requested changes.
2022-02-24 12:50:51 +00:00
FireMasterK
d290d2e393
Move variable to super Extractor classes.
2022-02-24 12:50:51 +00:00
FireMasterK
ab49cb6e18
Add requested changes.
2022-02-24 12:50:51 +00:00
Kavin
4bd59f65f4
Update exception message.
...
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2022-02-24 12:50:51 +00:00
FireMasterK
6f3f608ab6
Add support for extracting channel subscriber count in StreamInfo.
2022-02-24 12:50:51 +00:00
Stypox
69e18c80cb
Merge pull request #799 from litetex/imporve-yt-comments-extractor
...
Refactored YoutubeCommentsExtractor
2022-02-23 17:58:00 +01:00
litetex
1a67ea100a
Refactored code
...
according to review
2022-02-21 19:56:12 +01:00
litetex
f79ce1f52a
Refactored YoutubeCommentsExtractor
...
* Use Java Streaming API
* Use StandardCharsets
* Prevented several NPEs/ArrayIndexOutOfBound
* Reformatted some code so that it's easier readable
2022-02-21 19:56:09 +01:00
TobiGr
dfe8716f5f
Merge branch 'dev'
2022-02-18 20:30:38 +01:00
TobiGr
d07e16aadd
Release NewPipe Extractor 0.21.14
2022-02-18 20:29:57 +01:00
TobiGr
d337e537dc
[YouTube] Add a comment clarifying why the panelIdentfier can be null
2022-02-18 20:26:19 +01:00
Tobi
5c8fbeb166
Fix NullPointerException in YouTubeStreamExtractor.getStreamSegments() ( #797 )
...
Fix NullPointerException in YouTubeStreamExtractor.getStreamSegments()
Co-authored-by: litetex <40789489+litetex@users.noreply.github.com>
2022-02-16 23:39:26 +01:00
dependabot[bot]
89900431ab
Bump gson from 2.8.9 to 2.9.0
...
Bumps [gson](https://github.com/google/gson ) from 2.8.9 to 2.9.0.
- [Release notes](https://github.com/google/gson/releases )
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md )
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.9...gson-parent-2.9.0 )
---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-02-14 09:15:09 +00:00
Tobi
0d6aebce25
[SoundCloud] Update clientId ( #798 )
2022-02-11 17:16:32 +01:00
dependabot[bot]
a2a41e3bf2
Bump rhino from 1.7.13 to 1.7.14
...
Bumps [rhino](https://github.com/mozilla/rhino ) from 1.7.13 to 1.7.14.
- [Release notes](https://github.com/mozilla/rhino/releases )
- [Changelog](https://github.com/mozilla/rhino/blob/master/RELEASE-NOTES.md )
- [Commits](https://github.com/mozilla/rhino/commits )
---
updated-dependencies:
- dependency-name: org.mozilla:rhino
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-02-07 20:29:51 +00:00
XiangRongLin
545522b80f
[Youtube] Make throttling decryption more resilient to api change
2022-02-01 19:57:34 +01:00
TobiGr
ebc129c392
NewPipe Extractor 0.21.13
2022-02-01 13:20:34 +01:00
TobiGr
5306f7a988
Improve code style
2022-02-01 13:18:18 +01:00
Abdu Ameen
38e9da8d44
Adding removal of the braces
2022-02-01 13:18:10 +01:00
Abdu Ameen
36ebb633a4
Changed the regex to account for
...
nonword characters
2022-02-01 13:18:05 +01:00
Tobi
65129e65a8
Merge pull request #785 from AbduAmeen/regex_error
...
Regex error when parsing the Youtube JavaScript code
2022-02-01 13:17:19 +01:00
TobiGr
047d75a4c9
Improve code style
2022-02-01 12:56:59 +01:00
Abdu Ameen
50a4b026f8
Adding removal of the braces
2022-01-31 23:52:31 -08:00
Abdu Ameen
a9174f6b9f
Changed the regex to account for
...
nonword characters
2022-01-31 22:29:00 -08:00
dependabot[bot]
a4399fd4ad
Bump spotbugs-annotations from 4.5.2 to 4.5.3
...
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.5.2 to 4.5.3.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.5.2...4.5.3 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-01-24 18:45:21 +01:00
Stypox
59e0aeba88
Merge pull request #768 from litetex/junit-5
...
JUnit 5
2022-01-24 16:56:16 +01:00
Mohammed Anas
65df39b3e3
[YouTube] parse timestamps with >3 digits correctly ( #775 )
...
* [YouTube] parse timestamps with >3 digits correctly
Fixes https://github.com/TeamNewPipe/NewPipe/issues/7530 ; check the
issue for details.
* Remove outdated comment
2022-01-12 10:45:06 +01:00
litetex
a579ef2651
Fixed Soundcloud tests
2022-01-07 14:27:24 +01:00
litetex
a32edce750
Removed deprecation because AndroidSDK/API < 19
2022-01-07 14:15:55 +01:00
litetex
602068937d
Code cleanup
...
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2022-01-06 21:04:16 +01:00
litetex
dd3f3da828
Kurzgesagt mock data is outdated
2022-01-04 17:44:12 +01:00
litetex
a8ebd2cead
Update mock resources for YT Mix playlist
2022-01-04 17:39:14 +01:00
litetex
066afd8629
Restored test
2022-01-04 17:33:08 +01:00
litetex
3712a669b1
Code cleanup / improvements
...
* Deprecated Utils#UTF-8; see StandardCharsets
* Added more helpful methods to ``ExtractorAsserts``
* Use parameterized (cool new) tests
* Restore functionality of some tests + updated mockdata
* Other code cleanups + Sonarlint improvements
2022-01-04 17:28:31 +01:00
litetex
4291a90251
Fixed SoundCloud's search(for tests)
2021-12-27 21:42:43 +01:00
litetex
4995709871
Fixed SoundCloud's search(for tests)
...
Getting the initial page was not returning initial page
2021-12-27 21:18:39 +01:00
litetex
a2cbdf0991
Updated to JUnit 5
2021-12-27 21:08:08 +01:00
litetex
10f6cc7194
Merge pull request #755 from litetex/clean-up-yt-dis-likes
...
Cleanup: Remove old ways of getting YT dis/likes
2021-12-27 18:32:02 +01:00
litetex
148ba6641d
Updated mock data
2021-12-27 17:08:14 +01:00
litetex
4d0a689cb6
Disabled dislike count in tests
2021-12-27 17:04:46 +01:00
litetex
a6961c5efa
Fixed NumberFormatException when processing likes
...
See also #7525
2021-12-27 16:54:47 +01:00
litetex
65687f3b9b
Removed YT dislike count as it no longer works
2021-12-27 16:54:46 +01:00
litetex
1e2e0029fc
[StreamExtractor] Deduplicated a ton of code by using default methods
2021-12-27 16:54:45 +01:00
litetex
15b98ffdb4
Remove old ways of getting YT dis/likes
...
* Added additional check for averageRating (in dislikes)
2021-12-27 16:54:44 +01:00
dependabot[bot]
fe432425df
Bump spotbugs-annotations from 4.5.1 to 4.5.2
...
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.5.1 to 4.5.2.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.5.1...4.5.2 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-12-14 10:35:32 +01:00
dependabot[bot]
36663dcb37
Bump spotbugs-annotations from 4.5.0 to 4.5.1
...
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.5.0 to 4.5.1.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.5.0...4.5.1 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-12-12 13:28:30 +01:00
litetex
12a6a48b15
Merge pull request #761 from litetex/fix-soundcloud-georestricted
...
Disabled ``SoundcloudStreamExtractorTest#SoundcloudGeoRestrictedTrack#testRelatedItems`` as it's unreliable
2021-12-11 18:06:17 +01:00
litetex
7484b6a098
Merge pull request #758 from litetex/release/v0.21.12
...
Release 0.21.12
2021-12-11 16:14:11 +01:00
litetex
795e6df596
Update README.md
...
Made usage section more abstract
2021-12-11 16:06:29 +01:00
litetex
4dd737c9a0
Updated version to v0.21.12
2021-12-11 15:34:46 +01:00
litetex
7489502cc8
Disabled ``testRelatedItems`` as it's unreliable
2021-12-11 15:23:11 +01:00
Tobi
1e0c409bec
Merge pull request #760 from litetex/fix-test
...
Fix test
2021-12-11 12:44:45 +01:00
litetex
d9b2a2042a
Updated mock data for YoutubeCommentsExtractorTest
2021-12-08 22:11:45 +01:00
litetex
a3a0c1ee85
Fixed YoutubeCommentsExtractorTest
...
Old video was deleted / set to private
2021-12-08 22:11:34 +01:00
litetex
5c288eb8d7
Updated mock data for YoutubeChannelExtractorTest
2021-12-08 21:51:38 +01:00
litetex
75bb83a1b8
Fixed YoutubeChannelExtractorTest
...
Channel was restored, picked a new one
2021-12-08 21:51:15 +01:00
litetex
f276bacce5
Fixed SoundcloudStreamExtractorTest
2021-12-08 21:41:27 +01:00
litetex
7f99c1c193
Fixed YoutubeStreamExtractorLivestreamTest
2021-12-08 21:16:39 +01:00
litetex
652fdf2c36
Updated mock-data for YoutubeStreamExtractorAgeRestrictedTest
2021-12-08 21:08:25 +01:00
litetex
3c669d3cb5
Fixed YoutubeStreamExtractorAgeRestrictedTest
...
Looks like a lot of stuff (description, tags, name) got changed on the existing video.
Use a better suited video.
2021-12-08 21:08:03 +01:00
litetex
f735788a39
Fixed Soundcloud hardcoded client id
2021-12-08 21:00:53 +01:00
litetex
f6f2724634
Merge pull request #753 from B0pol/dis-likes
...
Fix YouTube likes + dislikes
2021-11-29 20:42:17 +01:00
bopol
56c8af710e
Code refactoring
2021-11-21 18:16:31 +01:00
bopol
c4eca91be9
Fix YouTube likes + dislikes
2021-11-19 21:36:03 +01:00
TobiGr
5028396405
Improve names and formatting
2021-11-14 13:44:13 +01:00
dependabot[bot]
1e0c802d8d
Bump spotbugs-annotations from 4.4.2 to 4.5.0
...
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.4.2 to 4.5.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.4.2...4.5.0 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-14 13:43:44 +01:00
Tobi
68d7e4e20a
Merge pull request #742 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.8.9
...
Bump gson from 2.8.8 to 2.8.9
2021-11-08 19:37:09 +01:00
bopol
8e4f9995bf
Merge pull request #745 from TeamNewPipe/update_SC_hardcoded_client_id
...
Update hardcoded SoundCloud client_id
2021-11-04 16:33:08 +01:00
opusforlife2
cb80a646d9
Update hardcoded SoundCloud client_id
...
Hah! I got here first, @TobiGr!
2021-11-04 15:28:26 +00:00
litetex
7e7b78f1b3
Merge pull request #743 from B0pol/peertube-shortlinks
...
PeerTube short links support
2021-11-04 16:10:23 +01:00
bopol
396aecef19
Make sure playlists aren't accepted by PeertubeStreamLinkHandlerFactory
2021-11-03 14:41:39 +01:00
bopol
38ad1eaac5
[PeerTube] Support /w/ short video links part 2
...
We also need to support it for comments
2021-11-03 13:54:37 +01:00
bopol
026751624f
[PeerTube] Support /w/p/ short playlist links
2021-11-03 13:35:30 +01:00
bopol
e0b8e142fc
[PeerTube] Support /a/ and /c/ short links
2021-11-03 13:26:27 +01:00
bopol
599a91c88c
[PeerTube] Support /w/ short video links
2021-11-03 11:51:58 +01:00
dependabot[bot]
198d37090b
Bump gson from 2.8.8 to 2.8.9
...
Bumps [gson](https://github.com/google/gson ) from 2.8.8 to 2.8.9.
- [Release notes](https://github.com/google/gson/releases )
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md )
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.8...gson-parent-2.8.9 )
---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 09:12:53 +00:00
litetex
4f60225ddc
Merge pull request #741 from mhmdanas/add-y2ube-link-support
...
Add support for y2u.be links
2021-10-26 18:43:36 +02:00
mhmdanas
3e8e2a1532
Add support for y2u.be links
2021-10-22 22:48:18 +03:00
Tobi
b425394e7a
Merge pull request #731 from FireMasterK/short-description
...
Extract Video Short Description in YouTube.
2021-10-15 23:10:00 +02:00
Tobi
4e9d9bffd0
Merge pull request #738 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.4.2
...
Bump spotbugs-annotations from 4.4.1 to 4.4.2
2021-10-12 12:09:36 +02:00
dependabot[bot]
1867a381a8
Bump spotbugs-annotations from 4.4.1 to 4.4.2
...
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.4.1 to 4.4.2.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.4.1...4.4.2 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-12 09:11:11 +00:00
FireMaskterK
62313962a0
Add default implementation for getShortDescription.
2021-10-04 01:42:29 +01:00
Kavin
d1add05bc1
Add requested changes.
...
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
Add final.
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-10-04 01:36:51 +01:00
Tobi
0b0c39aec2
Merge pull request #734 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.14.3
...
Bump jsoup from 1.14.2 to 1.14.3
2021-10-02 16:49:34 +02:00
TobiGr
8dcb74724a
Release NewPipe Extractor 0.21.11
2021-10-02 16:25:35 +02:00
dependabot[bot]
3ca9f51b30
Bump jsoup from 1.14.2 to 1.14.3
...
Bumps [jsoup](https://github.com/jhy/jsoup ) from 1.14.2 to 1.14.3.
- [Release notes](https://github.com/jhy/jsoup/releases )
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES )
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.14.2...jsoup-1.14.3 )
---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-02 08:20:38 +00:00
TobiGr
d8ec3ce370
[SoundCloud] Update client id
2021-10-01 02:51:38 +02:00
FireMasterK
94efe86c71
Extract Video Short Description in YouTube.
...
In Trending, and Search results.
2021-09-22 16:26:16 +01:00
Tobi
a9d214478d
Merge pull request #703 from FireMasterK/comment-replies
...
Add support for extracting comment replies continuation
2021-09-14 23:58:14 +02:00
TobiGr
ce8cabb9f8
Release NewPipe Exttractor 0.21.10
2021-09-13 13:59:35 +02:00
FireMasterK
6aabdc6d16
Fix for requested changes.
2021-09-12 01:15:19 +05:30
Kavin
0aad09fa22
Update JavaDocs for comment replies continuation.
...
Co-authored-by: Stypox <stypox@pm.me>
2021-09-12 01:05:20 +05:30
Tobi
a031289a63
Merge pull request #727 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.4.1
...
Bump spotbugs-annotations from 4.4.0 to 4.4.1
2021-09-10 18:25:27 +02:00
dependabot[bot]
82ae322cb8
Bump spotbugs-annotations from 4.4.0 to 4.4.1
...
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.4.0 to 4.4.1.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.4.0...4.4.1 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-10 09:11:27 +00:00
Tobi
62b87552f5
Merge pull request #725 from FireMasterK/related-uploader-avatar
...
Extract Channel Avatar in related videos.
2021-09-03 22:14:28 +02:00
FireMasterK
feb09e9997
Extract Channel Avatar in related videos.
2021-09-04 01:09:40 +05:30
Tobi
f0aa46b008
Merge pull request #724 from FireMasterK/streams-uploader-verified
...
Extract uploaderVerified to StreamInfo.
2021-09-03 18:18:04 +02:00
Tobi
c8037f5e00
Merge pull request #723 from FireMasterK/uploader-avatar
...
Extract Uploader's Avatar on YouTube and PeerTube
2021-09-03 18:13:59 +02:00
FireMasterK
857b2f39e8
Extract uploaderVerified to StreamInfo.
2021-09-03 21:27:58 +05:30
FireMasterK
1db463b55f
Add a default test for the Uploader's Avatar.
2021-09-03 21:05:36 +05:30
Kavin
db6b3b2c29
Extract uploader's avatar in peertube.
...
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2021-09-02 16:14:45 +05:30
FireMasterK
a0c1dcc8d8
Remove throws parsing exception.
2021-09-02 16:14:45 +05:30
FireMasterK
0ba03c552c
Improve based on review/feedback.
2021-09-02 00:35:56 +05:30
FireMasterK
b9fad4fcc8
Extract Uploader's Avatar on YouTube.
2021-09-02 00:19:00 +05:30
Stypox
68f1fa994a
Merge pull request #721 from Stypox/yt-csv-import
...
[YouTube] Small improvements to subscription import
2021-08-31 12:02:59 +02:00
TobiGr
b644f1d67b
[SoundCloud] Update client id
2021-08-31 07:42:21 +02:00
Stypox
a5e9eeb790
[YouTube] Small improvements to subscription import
2021-08-30 15:45:45 +02:00
Tobi
9570882c73
Merge pull request #712 from magicbrothers/add-eduvid-tubus
...
Add invidious instance EduVid Tubus
2021-08-24 16:07:07 +02:00
Stypox
db6c729678
Merge pull request #709 from talanc/dev
...
[YouTube] Support csv and zip subscription import (Google Takeout)
2021-08-24 15:46:03 +02:00
talanc
94a29fd63f
[YouTube] csv and zip subscription import (Google Takeout)
...
csv:
Improved error messages
Exits early if it hasnt found any items in the first few lines
zip:
Now checks all CSV files instead of hard-coded paths
final qualifiers for immutable locals and parameters
Co-authored-by: litetex <40789489+litetex@users.noreply.github.com>
2021-08-24 15:42:49 +02:00
dependabot[bot]
816bc7137f
Bump gson from 2.8.7 to 2.8.8
...
Bumps [gson](https://github.com/google/gson ) from 2.8.7 to 2.8.8.
- [Release notes](https://github.com/google/gson/releases )
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md )
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.7...gson-parent-2.8.8 )
---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-23 13:04:10 +02:00
TobiGr
6d487e0dad
Bump version to 0.21.9
2021-08-22 19:29:22 +02:00
Tobi
7823dcd99d
Merge pull request #715 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.4.0
...
Bump spotbugs-annotations from 4.2.3 to 4.4.0
2021-08-22 16:41:20 +02:00
dependabot[bot]
d75d54e1c6
Bump spotbugs-annotations from 4.2.3 to 4.4.0
...
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.2.3 to 4.4.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.2.3...4.4.0 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-22 07:12:54 +00:00
dependabot[bot]
72393995bb
Bump jsoup from 1.13.1 to 1.14.2 ( #716 )
...
Bumps [jsoup](https://github.com/jhy/jsoup ) from 1.13.1 to 1.14.2.
- [Release notes](https://github.com/jhy/jsoup/releases )
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES )
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.13.1...jsoup-1.14.2 )
---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-22 09:12:14 +02:00
Stypox
d8179dc35a
Merge pull request #713 from TiA4f8R/regexes-yt-signatureciphers
...
[YouTube] Consolidate the regular expressions used to find the cipher decryption function
2021-08-17 18:58:27 +02:00
Tobi
89bab241b9
Merge pull request #717 from XiangRongLin/fix_tests
...
Fix failing PeertubeStreamExtractorTest
2021-08-17 17:48:28 +02:00
XiangRongLin
61acdae4b8
Fix failing PeertubeStreamExtractorTest
2021-08-17 17:06:36 +02:00
FireMasterK
4f86b8ce43
Add verified info to ChannelInfo. ( #714 )
2021-08-17 13:08:49 +02:00
Stypox
52910ce970
Keep original exception when there is an error message ( #710 )
2021-08-13 16:13:58 +02:00
TiA4f8R
f8309d546a
[YouTube] Consolidate current regexes of the cipher decryption function
...
Also add more regexes
2021-08-12 18:26:43 +02:00
magicfelix
0e16091ce0
Add invidious instance EduVid Tubus
2021-08-12 10:06:41 +02:00
Tobi
b6f951edb4
Merge pull request #711 from gardenappl/dev
...
Fix minor typos in Javadoc
2021-08-11 18:16:34 +02:00
gardenapple
50bba32239
Fix typos in Javadoc
2021-08-11 12:01:25 +03:00
Tobi
78f2192cbb
Merge pull request #687 from TacoTheDank/gradle7
...
Upgrade to Gradle 7.1.1
2021-08-08 12:38:29 +02:00
TacoTheDank
4a7c1723a6
Use new 'maven-publish' plugin
2021-08-07 17:51:32 -04:00
TacoTheDank
8919a53d95
Update gradle wrapper to 7.1.1
2021-08-07 17:51:18 -04:00
TacoTheDank
cf9e678223
Declare common dependency versions in root gradle
2021-08-07 17:51:09 -04:00
FireMasterK
71b9fd0076
Faster iframe api based player extraction. ( #694 )
...
* Faster iframe api based player extraction.
Uses the IFrame API to reduce the required download to less than 1/50 of the size.
* Remove debug code.
* Extract to two methods.
* Add tests for player URL extraction.
* Add assertThat for tests.
2021-08-07 12:35:48 +02:00
Tobi
4b147863ec
Merge pull request #705 from FireMasterK/suppored-countries
...
Update supported countries list for YouTube.
2021-08-06 10:14:27 +02:00
FireMasterK
ee32317315
Update supported countries list for YouTube.
2021-08-05 23:48:38 +05:30
FireMasterK
e8e9e6cd00
Fix comment reply continuations.
2021-08-04 22:14:04 +05:30
FireMasterK
666b45958e
Fix test and update mocks.
2021-08-04 21:10:02 +05:30
FireMasterK
f6d054e5da
Add test for comments.
2021-08-04 21:03:00 +05:30
FireMasterK
ed84658055
Get it working.
2021-08-04 18:35:14 +05:30
FireMasterK
ea7e6526fd
Collect replies/page to CommentsInfoItem.
2021-08-04 18:09:42 +05:30
FireMasterK
4385cc1b7c
Add comment replies.
2021-08-04 17:56:04 +05:30
Stypox
8ca5e5e364
Merge pull request #702 from TeamNewPipe/master
...
This merges master into dev
2021-08-03 20:52:04 +02:00
Stypox
bb3815d19b
Merge pull request #701 from Stypox/v0.21.8
...
V0.21.8
2021-08-03 20:43:15 +02:00
Stypox
5b38b3ae97
Merge branch 'master' into v0.21.8
2021-08-03 20:38:07 +02:00
Stypox
4b5907e18c
Release version 0.21.8
2021-08-03 20:33:24 +02:00
Stypox
5a88263785
Merge pull request #604 from TiA4f8R/youtubei-api
...
[YouTube] Use the new internal API in NewPipe Extractor
2021-08-03 19:01:14 +02:00
TiA4f8R
79cc5c8d12
Fix tests of YoutubePlaylistExtractorTest and YoutubeMixPlaylistExtractorTest
...
Without removing RunWith and SuiteClasses annotations (and the corresponding imports) in YoutubePlaylistExtractorTest and YoutubeMixPlaylistExtractorTest, some mocks cannot be generated, so the CI fails because of the missing mocks. Mocks of workings tests have been also updated.
2021-08-01 17:28:01 +02:00
TiA4f8R
b74a39c176
Reformat some code and don't use the clickTrackingParams in continuations of YouTube Music search results
...
The clickTrackingParams of YouTube Music search results are not needed to get continuations. This commit removes their use, which may improve privacy.
2021-08-01 17:25:59 +02:00
TiA4f8R
f8197da09e
Update YouTube comments and YouTube playlist continuations mocks
2021-08-01 12:39:05 +02:00
FireMasterK
f3e4c9d689
Use the youtubei API for YouTube comments
...
Migrate YouTube comments to the desktop version by using the `next` endpoint of the InnerTube internal API.
With the desktop version, we are able to get the exact like count of YouTube comments (by parsing the accessibility data) (the current extraction is used as a fallback). We are also now able to get if the uploader of the comment is verified or not.
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-08-01 12:39:05 +02:00
TiA4f8R
286d839a3b
Update mocks (except mocks of YoutubeCommentsExtractorTest)
2021-08-01 12:39:04 +02:00
TiA4f8R
693f654f02
Ignore a broken test of YoutubeMixPlaylistExtractorTest
2021-08-01 12:39:03 +02:00
TiA4f8R
d55109258d
Fix YoutubeStreamExtractorAgeRestrictedTest
2021-08-01 12:39:03 +02:00
FireMasterK
2eeb0a3403
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
...
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:
- the desktop API is fetched.
If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status
If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.
If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.
We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.
Other code changes:
- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-08-01 12:39:03 +02:00
TiA4f8R
7753556e66
Adress the last requested changes + update YoutubeCommentsExtractor mocks
2021-08-01 12:39:03 +02:00
TiA4f8R
2320aecb7c
Update mocks
2021-08-01 12:39:02 +02:00
TiA4f8R
629dcd63b2
Ignore some broken tests
2021-08-01 12:39:01 +02:00
TiA4f8R
3adac6a150
Workaround for rate limits: always use the Android mobile API
2021-08-01 12:39:01 +02:00
TiA4f8R
8aa60d7e8f
Update clients version
2021-08-01 12:39:01 +02:00
TiA4f8R
27bd797f1e
Update mocks
2021-08-01 12:39:01 +02:00
TiA4f8R
609919db59
Adress again reviews, fix some rebase issues
2021-08-01 12:39:00 +02:00
TiA4f8R
4299d806a2
Adress changes
2021-08-01 12:38:59 +02:00
TiA4f8R
1a6b8da438
Annotate YoutubeParsingHelper methods with Nonnull when needed
2021-08-01 12:38:59 +02:00
TiA4f8R
d8177b57f6
Loop in all formats to check if the stream has URLs protected by signatureCiphers
2021-08-01 12:38:59 +02:00
TiA4f8R
a6a2c6eb80
Revert the use of Collections.singletonList instead of Arrays.asList in addCookieHeader of YoutubeParsingHelper
2021-08-01 12:38:59 +02:00
TiA4f8R
81013e5a8e
Add a static method to reset the YouTube deobfuscation code and use it in tests
...
This method is needed for YouTube stream tests, because when all YouTube tests are ran, the signatureTimestamp is known (the sts string) so a different body than the body present in the mocks is send by the extractor instance.
As a result, running all YouTube stream tests with the MockDownloader (like the CI does) will fail if this method is not called before fetching the page of a test.
2021-08-01 12:38:58 +02:00
TiA4f8R
cc798523cd
Adress requested changes in other classes
2021-08-01 12:38:58 +02:00
TiA4f8R
accd5ddef3
Adress requested changes in YoutubeStreamExtractor
2021-08-01 12:38:58 +02:00
TiA4f8R
632772d17f
Adress requested changes in YoutubeParsingHelper
2021-08-01 12:38:58 +02:00
TiA4f8R
657f165771
Update client version and mocks
2021-08-01 12:38:44 +02:00
TiA4f8R
8c1c7281b0
Real fix of the fetch of the JS player at each the signatureTimestamp is required
...
The strings playerJsUrl, sts and playerCode are now static in order to don't fetch again the JavaScript player at each time the signatureTimestamp is needed.
2021-08-01 12:38:43 +02:00
TiA4f8R
e97a685989
Format the YoutubeMusicSearchExtractor class to be in the 100 characters per line limit
2021-08-01 12:38:43 +02:00
TiA4f8R
34a9ccb0fd
Adress requested changes
2021-08-01 12:38:42 +02:00
TiA4f8R
54d4551ca6
Adress requested changes in YoutubeParsingHelper and update mobile client version
2021-08-01 12:38:42 +02:00
TiA4f8R
a59c2a3577
Catch every exception instead of only IOException and ExtractionException and add a Javadoc
...
Catch every exception instead of only IOException and ExtractionException.
Add JavaDoc for fetchAndroidMobileJsonPlayer method of YoutubeStreamExtractor
2021-08-01 12:38:42 +02:00
TiA4f8R
6921e80ded
Try again to don't fetch at each time the JavaScript player
2021-08-01 12:38:42 +02:00
TiA4f8R
70927ddade
Update client version and mocks
2021-08-01 12:38:40 +02:00
TiA4f8R
318bc46a8c
Readd the deleted code of views because watching count of livestreams was broken
...
The number shown was the total number of views that a livestream has. In order to fix this bug, the previous code is readded.
2021-08-01 12:38:39 +02:00
TiA4f8R
947baec805
Fetch again the desktop player JSON only if the content is protected by signatureCiphers
2021-08-01 12:38:39 +02:00
TiA4f8R
7474049fd1
Update mocks
2021-08-01 12:38:39 +02:00
TiA4f8R
c32bc6e534
Try to don't fetch again the first page when requesting a channel continuation
...
Try to don't fetch again the first page of a YouTube channel when requesting a continuation of it by trying to store the channel name and the channel id into the next page using the ids field of the Page class.
2021-08-01 12:38:38 +02:00
TiA4f8R
b52732a1c3
Fix the fetch of the playerCode at each time getStsFromPlayerJs is called
...
storePlayerJs was called even if sts was not empty in the getStsFromPlayerJs method. This commit fixes it.
2021-08-01 12:38:37 +02:00
TiA4f8R
ae5abc0c5d
Fix a typo in YoutubeChannelExtractor
2021-08-01 12:38:37 +02:00
TiA4f8R
0f9e9b8b4b
Use the youtubei API for YouTube mixes + update the corresponding test + do some improvements
...
Use the youtubei API for YouTube mixes. The corresponding has been updated because the new API breaks the tests of YoutubeMixPlaylistExtractorTest.
Remove some deprecated code (the old search code with the pbj JSON) and do some other improvements.
2021-08-01 12:38:37 +02:00
TiA4f8R
14569c4aa9
Readd the return of itags 134 and 136
2021-08-01 12:38:37 +02:00
TiA4f8R
3017dde67e
Fix some typos
2021-08-01 12:38:36 +02:00
TiA4f8R
013b902535
Use the Android mobile API when there are OTF streams or the content is protected by signatureCiphers
...
Use the Android mobile API to get the itag 22 (720p with audio), removed when the content is protected by signatureCiphers.
Also use this API when they are OTF streams, to get the itag 17 and 36, low 3GPP quality streams but also the itag 139.
Update the web client version.
2021-08-01 12:38:36 +02:00
TiA4f8R
e7d589edbf
Use the youtubei API for YouTube videos + update client version
...
Update the hardcoded client version to 2.20210520.09.00
Use the player and next endpoints of the Innertube API for YouTube videos
2021-08-01 12:38:36 +02:00
TiA4f8R
f73c923f60
Don't use the youtubei.googleapis.com but the websites domains + update client version of the desktop internal API
...
Use again www.youtube.com and music.youtube.com domains instead of youtubei.googleapis.com domain because it spoofs more a web client of YouTube or YouTube Music and may reduce Google's detection of NewPipe Extractor users.
2021-08-01 12:38:34 +02:00
TiA4f8R
effcdaa4f2
Update mocks, reenable a test and fix a test
...
This commit updates mocks and reenables the test invalidId of the NotAvailable class of the YoutubePlaylistExtractorTest class beacuse with the youtubei/innertube API, it returns "Not found" and doesn't redirect to the YouTube homepage.
The expectedMetaInfo test of the MetaInfoTest class of the YoutubeSearchExtractorTest class was broken because YouTube removes the vaccine progress link of the WHO from the meta info so this commit removes it and the test is now passing.
2021-08-01 12:38:04 +02:00
TiA4f8R
4d682834c3
Fix localization and update client version
2021-08-01 12:38:03 +02:00
TiA4f8R
f46cfb0f26
Adress reviews and do some improvements
...
Adress changes requested in reviews.
Do some improvements, remove unused imports and format some code to be in the 100 characters line limit.
2021-08-01 12:38:03 +02:00
TiA4f8R
e075dd5a63
Update client version, fix some tests, update mocks and do some improvements
...
Add the origin and the referer headers with the https://www.youtube.com value for YouTube JSON POST requests.
Don't add the consent cookie header for the requests which use the youtubei/innertube API because it's uneeded.
Fix some tests and update YouTube mocks
2021-08-01 12:38:02 +02:00
TiA4f8R
b49ae547a3
Do some improvements to YoutubeStreamExtractor
...
Get the real name of the uploader (for autogenerated channels and music artist channels), like before the migration to the JSON pbj.
Do some other improvements, especially reformatting some code to be in the 100 characters line limit and use final where possible.
2021-08-01 12:38:01 +02:00
TiA4f8R
58ce9b04a1
Fix channel extraction when channel URL is youtube.com/c/username
2021-08-01 12:38:01 +02:00
TiA4f8R
991b2c7d73
Use lightweight requests when getting and checking YouTube API key and client version
2021-08-01 12:38:01 +02:00
TiA4f8R
9ab9c66ddf
Use the youtubei.googleapis.com domain for YouTube Music searches + change a check + update client version and mocks
...
Change the domain from music.youtube.com to youtubei.googleapis.com.
Use a lightweight request to check if the hardcoded YouTubeMusic keys are valid. Increase the length of the response to 500 because if the key is invalid, the length of the response returned is higher than 250 and the response when the key is valid is higher than 1500.
Format the YoutubeMusicSearchExtractor file.
Update YouTube web client version and mocks
2021-08-01 12:37:56 +02:00
TiA4f8R
77c031a88a
Use the youtubei API for YouTube trends
2021-08-01 12:36:28 +02:00
TiA4f8R
f461224b2b
Use the youtubei API for YouTube searches + update mocks
...
Add getSearchParameter, a new method in YoutubeSearchQueryHandlerFactory class which returns the params field for a search, or an empty string if there is no one.
Update mocks of YoutubeSearchExtractorTest.
2021-08-01 12:36:28 +02:00
TiA4f8R
a12c69da7d
Use the youtubei API for YouTube channels
2021-08-01 12:36:27 +02:00
TiA4f8R
5794eb2350
Use the youtubei API for YouTube playlists
2021-08-01 12:36:12 +02:00
Stypox
c97a19d719
Merge pull request #696 from XiangRongLin/decrypt_pattern
...
Expand regex to match n param decrypt function
2021-08-01 10:24:22 +02:00
Stypox
06a5219c9c
Merge pull request #699 from FireMasterK/yt-music-search
...
Fix YouTube music search.
2021-08-01 10:17:15 +02:00
XiangRongLin
37df225556
Remove length check from StringUtils.matchToClosingParenthesis
2021-07-31 16:05:24 +02:00
FireMasterK
f4aad8b014
Fix tests again.
2021-07-31 14:40:51 +05:30
FireMasterK
88c11db4c5
Fix isCorrectedSearch test.
2021-07-31 02:18:18 +05:30
FireMasterK
fd19c53f65
Fix search suggestions test.
2021-07-31 02:08:50 +05:30
FireMasterK
a685941bba
Fix YouTube music search.
2021-07-31 01:33:38 +05:30
XiangRongLin
852a65ff18
Add tests for StringUtils
2021-07-30 19:56:57 +02:00
XiangRongLin
48d897e6ad
Add final and adjust utils class name
2021-07-29 21:33:45 +02:00
XiangRongLin
60794aea31
[YouTube] Add parenthesis matching as way to parse decrypt function
...
From @Stypox
2021-07-29 21:23:00 +02:00
XiangRongLin
2967d1ae6a
[YouTube] Compile YoutubeThrottlingDecrypter pattern statically
2021-07-29 20:27:29 +02:00
XiangRongLin
1c78976900
[YouTube] Expand regex to match n param decrypt function
...
Temporary solution
2021-07-29 20:26:36 +02:00
TobiGr
b62fe7141e
Fix JDoc 2
2021-07-29 12:34:04 +02:00
TobiGr
dbc4e01c17
Fix JDoc
2021-07-29 12:31:33 +02:00
TobiGr
6335823843
Merge branch 'master' into dev
2021-07-28 20:41:23 +02:00
TobiGr
7c6ff0a38f
Release version 0.21.7
2021-07-28 20:38:45 +02:00
TobiGr
8bfcb0ad59
[SoundCloud] Fix hardcoded client id
2021-07-28 20:33:21 +02:00
Tobi
027dc65434
pull request #683 from XiangRongLin/yt_throttling
...
[YouTube] Fix buffering by decoding n parameter of stream urls
2021-07-28 20:32:38 +02:00
Tobi
14c179f343
Merge pull request #692 from TeamNewPipe/fix/sc/id
...
[SoundCloud] Fix hardcoded client id
2021-07-28 18:28:07 +02:00
Tobi
394c02ad06
Merge pull request #683 from XiangRongLin/yt_throttling
...
[YouTube] Fix buffering by decoding n parameter of stream urls
2021-07-28 18:01:57 +02:00
TobiGr
785ff4aa32
[SoundCloud] Fix hardcoded client id
2021-07-28 17:02:56 +02:00
TobiGr
79f2d74b04
Fix test
...
The decryption code changes over time. Only check whether the n parameter was changed in the URL
2021-07-28 16:45:31 +02:00
TobiGr
d70adfdb8f
Add methods for cache control to YoutubeThrottlingDecrypter.
2021-07-27 20:32:22 +02:00
TiA4f8R
d13f531b6f
Use YoutubeThrottlingDecrypter also in getAudioStreams and getVideoOnlyStreams methods of YoutubeStreamExtractor
...
Without this commit, the n param is only decrypted for streams extracted in getVideoStreams (so only for streams in the formats object of the player response).
2021-07-27 17:28:32 +02:00
TiA4f8R
32055147e0
Do some code improvements
...
Use final where possible, annotate some methods and parameters as Nonnull and format new code to be in the 100 characters limit per line.
2021-07-27 17:24:27 +02:00
TobiGr
1c30a2725e
Cache nParams to prevent executing the JavaScript function for the same nParam multiple times.
...
Closes #689
2021-07-22 19:40:12 +02:00
XiangRongLin
3a3d1d7f2b
Make YoutubeJavaScriptExtractor and JavaScript methods static
...
Also address review and rewrite some comments
2021-07-20 20:48:11 +02:00
XiangRongLin
a683c8d278
Delete duplicated code to load youtube javascript file
2021-07-17 19:14:57 +02:00
XiangRongLin
a02ee2e952
Rewrite youtube throttling solution and add tests
2021-07-17 19:10:09 +02:00
Tobi
ada67d136a
Merge pull request #647 from litetex/playerSeekbarPreview
...
Code changes to enable player thumbnail seekbar preview in NewPipe
2021-07-17 17:49:54 +02:00
litetex
0c12b396e5
Review changes
2021-07-17 16:41:12 +02:00
litetex
ecf4232ce3
Using ``Collections.emptyList()``
2021-07-17 16:41:12 +02:00
litetex
0f9ed020a4
Frameset has to implement Serializable or NewPipe fails
...
java.lang.RuntimeException: Parcelable encountered IOException writing serializable object (name = org.schabi.newpipe.extractor.stream.StreamInfo)
...
Caused by: java.io.NotSerializableException: org.schabi.newpipe.extractor.stream.Frameset
...
2021-07-17 16:41:11 +02:00
litetex
17ccaf4b87
Update YoutubeStreamExtractor.java
...
To fix ``java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.String[] java.lang.String.split(java.lang.String)' on a null object reference``
2021-07-17 16:41:10 +02:00
litetex
4e41e172ea
Update Frameset.java
...
Reformatted code
2021-07-17 16:41:10 +02:00
litetex
09b670d745
Renamed frames to previewFrames (so it's better understandable)
2021-07-17 16:41:09 +02:00
litetex
b086655d54
Reverted useless change
2021-07-17 16:41:08 +02:00
litetex
93b0c5b15f
Added seekbar data
2021-07-17 16:41:08 +02:00
XiangRongLin
6956b72af7
Fix YoutubeThrottlingDecoder.replaceNParam
...
Previously it replaced the parameter itself not the value of the parameter.
2021-07-17 10:05:14 +02:00
XiangRongLin
a189f685dc
cache player js code
2021-07-16 19:20:40 +02:00
XiangRongLin
fcdb9bdbeb
add url to sout
2021-07-15 20:05:26 +02:00
XiangRongLin
80cf8b3acd
Extract separate YoutubeThrottlingDecoder
2021-07-15 20:05:25 +02:00
XiangRongLin
a86a30103f
[Youtube] bare bones version to solve throttling
...
Done by transforming the parameter "n" from videoplayback urls
https://github.com/ytdl-org/youtube-dl/issues/29326#issuecomment-865985377
2021-07-15 20:05:25 +02:00
bopol
c38a06e8dc
Merge pull request #661 from B0pol/peertube-livestreams
...
[PeerTube] Support livestreams
2021-07-13 15:48:13 +02:00
Tobi
b45bb411e8
Merge pull request #652 from litetex/fixYTCommentsAndAddDisabledComments
...
Fix yt comments and add disabled comments functionallity
2021-07-12 16:31:50 +02:00
TobiGr
6fd93cdb31
Release 0.21.6
2021-07-10 17:07:23 +02:00
litetex
fdebf3c6cd
Added javadoc warnings
2021-07-07 20:41:59 +02:00
litetex
6860543b07
Added more doc
2021-07-06 21:16:31 +02:00
litetex
a59fead0d7
Fixed typos
2021-07-06 21:16:30 +02:00
litetex
688a1c316b
Fixed Exception when YT comments are disabled and added ``commentsDisabled`` field
...
* Fixed code: Added missing finals (according to NewPipes Checkstyle guide)
* Fixed ``findValue`` method in ``YoutubeCommentsExtractor``
2021-07-06 21:16:29 +02:00
litetex
ca33f4f60b
Improved code format (final) and added ``isCommentsDisabled`` method
2021-07-06 21:16:28 +02:00
Tobi
f324772254
Merge pull request #676 from TeamNewPipe/fix/consent-cookie
...
Fix invalid CONSENT cookie value
2021-07-04 17:41:47 +02:00
TobiGr
dbdc962a07
Update mocks
2021-07-03 13:30:26 +02:00
TobiGr
9ca6dc26ed
Fix invalid CONSENT cookie value
2021-07-03 13:27:40 +02:00
Tobi
a023f08774
Merge pull request #673 from XiangRongLin/recording_downloader
...
Remove IPv4 addresses from response during mock recording
2021-07-03 13:25:41 +02:00
XiangRongLin
b9549ace25
[Youtube] Fix lofi girl test and ignore mix with id test
2021-06-29 21:12:20 +02:00
XiangRongLin
5982431ffb
Remove IPv4 addresses from response during mock recording
2021-06-29 21:03:33 +02:00
bopol
7d7cc087e2
[PeerTube] Support livestreams
2021-06-24 11:02:30 +02:00
bopol
3966178979
Merge pull request #665 from B0pol/agerestricted
...
[youtube] Fix extraction of age restricted videos
2021-06-23 21:07:14 +02:00
bopol
b475f09ba7
don't do useless concatenation
2021-06-23 20:33:32 +02:00
TiA4f8R
d1c56be744
Merge pull request #668 from B0pol/invidious-instances
...
Update Invidious instances and support Piped links
2021-06-23 18:36:48 +02:00
bopol
b96b2a6eba
Update invidious instances
2021-06-23 14:39:10 +02:00
bopol
c5e8bd368d
Update Mocks
2021-06-23 12:21:57 +02:00
bopol
e9a992b0a9
[youtube] Fix extraction of age restricted videos
2021-06-23 12:06:02 +02:00
bopol
35c1bdd012
Merge pull request #655 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.2.3
...
Bump spotbugs-annotations from 4.0.2 to 4.2.3
2021-06-23 12:01:56 +02:00
dependabot[bot]
ad403ce73e
Bump spotbugs-annotations from 4.0.2 to 4.2.3
...
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.0.2 to 4.2.3.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.0.2...4.2.3 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-06-23 09:57:42 +00:00
bopol
65fd8740aa
Merge pull request #658 from TeamNewPipe/revert-651-fix/streamType
...
[YouTube] Fix getting stream type of live streams
2021-06-23 11:33:24 +02:00
bopol
361f142621
Use badge style instead of label
2021-06-23 11:25:59 +02:00
bopol
58f109ddec
adress stypox review
2021-06-23 11:15:40 +02:00
bopol
fe29c78fe1
Merge pull request #656 from TeamNewPipe/dependabot/gradle/org.mozilla-rhino-1.7.13
...
Bump rhino from 1.7.12 to 1.7.13
2021-06-20 00:46:42 +02:00
bopol
e9412fdadc
Merge pull request #654 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.8.7
...
Bump gson from 2.8.6 to 2.8.7
2021-06-20 00:35:24 +02:00
XiangRongLin
1510dd5187
Merge pull request #660 from TeamNewPipe/fix/sc/client-id
...
[SoundCloud] Update HARDCODED_CLIENT_ID
2021-06-18 15:48:27 +02:00
TobiGr
1f9a99c931
[SoundCloud] Update HARDCODED_CLIENT_ID
2021-06-18 15:30:54 +02:00
Tobi
c998012c28
Revert "[YouTube] Fix getting stream type of live streams"
2021-06-18 09:20:32 +02:00
dependabot[bot]
38d0339753
Bump rhino from 1.7.12 to 1.7.13
...
Bumps [rhino](https://github.com/mozilla/rhino ) from 1.7.12 to 1.7.13.
- [Release notes](https://github.com/mozilla/rhino/releases )
- [Changelog](https://github.com/mozilla/rhino/blob/master/RELEASE-NOTES.md )
- [Commits](https://github.com/mozilla/rhino/commits )
---
updated-dependencies:
- dependency-name: org.mozilla:rhino
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-06-17 21:49:07 +00:00
dependabot[bot]
5a46878b09
Bump gson from 2.8.6 to 2.8.7
...
Bumps [gson](https://github.com/google/gson ) from 2.8.6 to 2.8.7.
- [Release notes](https://github.com/google/gson/releases )
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md )
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.6...gson-parent-2.8.7 )
---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-06-17 21:49:01 +00:00
bopol
598de2dd5f
Merge pull request #552 from FireMasterK/dependabot
...
Add dependabot
2021-06-17 23:48:41 +02:00
Tobi
0cb48e36bd
Merge pull request #651 from TeamNewPipe/fix/streamType
...
[YouTube] Fix getting stream type of live streams
2021-06-12 17:57:18 +02:00
TobiGr
6762fe3357
[YouTube] Fix getting stream type of live streams
2021-06-10 16:51:00 +02:00
Tobi
a27575021d
Merge pull request #625 from TeamNewPipe/commentsStreamPosition
...
Add streamPosition for comments
2021-06-08 11:03:15 +02:00
TobiGr
f7f727d19c
Add constant NO_LIKE_COUNT to CommentsInfoItem
2021-06-08 10:28:44 +02:00
TobiGr
b70c0f93c7
Add streamPosition for comments
...
SoundCloud is the only service which supports adding comments at a specific timestamp in the stream.
2021-06-08 10:27:52 +02:00
Tobi
d4186d100b
Merge pull request #580 from TeamNewPipe/accountTerminated
...
Add AccountTerminatedException for terminated channels
2021-06-08 09:55:58 +02:00
TobiGr
89e0514d8b
Fix Javadoc generation fix
2021-06-08 09:30:05 +02:00
TobiGr
80d3052033
Fix Javadoc generation
2021-06-08 09:26:01 +02:00
TobiGr
98825a2f01
Set version to 0.21.4
2021-06-07 23:45:12 +02:00
TobiGr
fa444c8298
Remove assertations of exception messages
2021-06-06 11:38:43 +02:00
Tobi
8c42a48673
Merge pull request #646 from Stypox/get-service
...
Add utility method getService() to Info object
2021-06-05 14:29:09 +02:00
Stypox
2158ca4060
Remove two unneded warning suppressions
2021-06-05 14:15:48 +02:00
Stypox
13f192704d
Add utility method getService() to Info object
2021-06-05 14:14:19 +02:00
XiangRongLin
0ad51e76fb
Merge pull request #638 from TeamNewPipe/spelling
...
Improve exception message thrown when not accepting an URL in LinkHandlerFactory#fromUrl(String url, String baseUrl)
2021-06-04 18:23:31 +02:00
Tobi
143fd3c81c
Merge pull request #645 from XiangRongLin/fix_test
...
Fix PeertubeAccountExtractorTest
2021-06-03 20:49:59 +02:00
XiangRongLin
faa503c7dd
Fix PeertubeAccountExtractorTest
2021-06-03 19:54:42 +02:00
XiangRongLin
db81384ae0
Merge pull request #644 from TiA4f8R/utf-8-encoding-for-mocks-windows
...
Specify UTF-8 file encoding in RecordingDownloader and MockDownloader
2021-06-03 18:55:00 +02:00
TiA4f8R
ac31f3a883
Specify UTF-8 file encoding in RecordingDownloader and MockDownloader
...
On Windows, mocks are recorded and read with the Cp1252 encoding so it breaks the mocks on non ASCII characters for Linux OS (and so the CI).
The project is in Java 8, so we can't use FileReader(File, Charset) and FileReader(File, Charset) because these methods require Java 11. Instead of changing the Java version of the extractor, use OutputStreamWriter and FileOutputStream instead of FileWriter and InputStreamReader and FileInputStream instead of FileReader.
2021-06-03 18:38:25 +02:00
Tobi
7e4332e0cf
Update CI badge to representt the scheduled tests
2021-05-30 22:12:09 +02:00
bopol
ff11c2df2a
Merge pull request #628 from litetex/fix-broken-yt-liked-comments
...
Fix broken yt likes in comments
2021-05-29 11:04:11 +02:00
TobiGr
1ff56a85df
Update junit from 4.13.1 to 4.13.2
...
https://github.com/junit-team/junit4/blob/main/doc/ReleaseNotes4.13.2.md
2021-05-28 15:09:26 +02:00
TobiGr
858e007804
Improve exception message thrown when not accepting an URL in LinkHandlerFactory#fromUrl(String url, String baseUrl)
2021-05-28 14:59:44 +02:00
Tobi
519bba70b3
Merge pull request #630 from XiangRongLin/fix_tests
...
Fix MetaInfoTest.expectedMetaInfo
2021-05-28 14:16:53 +02:00
litetex
289b84e68b
Removed unused code
2021-05-27 21:27:44 +02:00
litetex
e81b0e2885
Tweaked code
2021-05-27 19:48:31 +02:00
TobiGr
f4cb32cef5
Release version 0.21.3
2021-05-26 22:40:56 +02:00
XiangRongLin
bf2af4f316
Merge pull request #635 from litetex/update-setup-java
...
Update setup java
2021-05-26 17:44:57 +02:00
litetex
8c96545e57
Merge branch 'TeamNewPipe:dev' into fix-broken-yt-liked-comments
2021-05-24 18:06:45 +02:00
litetex
12fb18c310
Updated mock data (only the affected)
2021-05-24 18:05:26 +02:00
litetex
8e08a2aac7
Improved docs
2021-05-24 18:05:26 +02:00
litetex
020acfed71
Fixed tests and added new ones
2021-05-24 18:05:26 +02:00
litetex
b310922fc0
Better exception message
2021-05-24 18:05:25 +02:00
litetex
86bb9efb5d
Added likeCount back to tests
2021-05-24 18:05:25 +02:00
litetex
46654f5fca
Reformatted code
2021-05-24 18:05:25 +02:00
litetex
2174685c5c
Reimplemented likeCount
2021-05-24 18:05:25 +02:00
litetex
01cfb55505
Update YoutubeCommentsInfoItemExtractor.java
2021-05-24 18:05:25 +02:00
litetex
bedcd87abb
Fixed occurrences where voteCount is not set (<=0 likes)
2021-05-24 18:05:25 +02:00
litetex
545c0a6f42
Fixed comment
2021-05-24 18:05:24 +02:00
Tobi
ff005122bf
Merge pull request #627 from TiA4f8R/use-snd-api-v2-everywhere
...
[SoundCloud] Use a lightweight request to check if the hardcoded client_id is valid, fix the extraction of mobile URLs and more
2021-05-23 22:52:40 +02:00
TiA4f8R
a00fdcbd3d
Format the indentation of the definition of the apiUrl string in SoundcloudParsingHelper.resolveFor method
...
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2021-05-23 20:27:45 +02:00
TiA4f8R
6b607eb38d
Update the hardcoded client_id
2021-05-23 18:08:44 +02:00
TiA4f8R
4552ea9c9f
Use the SOUNDCLOUD_API_V2_URL constant in all the SoundCloud package
2021-05-23 18:08:44 +02:00
TiA4f8R
86308d0603
Use a lightweight request to check if the client_id is valid
...
Request the api-v2 host with the client_id instead of checking if the streams of a SoundCloud track are not empty: if it is valid, the API returns 404, otherwise it should return 401.
2021-05-23 18:08:43 +02:00
TiA4f8R
c5c190500c
Format the SoundCloud package of the extractor and fix some warnings
...
Use final where possible in the package and format code to be in the 100 caracters per line limit.
Fix some warnings generated by Android Studio and do some code improvements
2021-05-23 18:08:42 +02:00
litetex
89ecc4e2e8
Update docs.yml
2021-05-23 14:34:29 +02:00
litetex
7791907480
Update ci.yml
2021-05-23 14:34:16 +02:00
Tobi
636e27333b
Merge pull request #603 from TeamNewPipe/peertube_hls
...
Add PeerTube HLS streams
2021-05-23 11:49:40 +02:00
Tobi
fadf677c42
Merge pull request #629 from TechComet/patch-1
...
[YouTube] Fix the fetch of the video info page
2021-05-23 09:41:24 +02:00
XiangRongLin
172394cd52
Update mocks
2021-05-22 20:33:20 +02:00
XiangRongLin
18c8cac40c
Fix MetaInfoTest.expectedMetaInfo
...
Expected meta changed
2021-05-21 21:00:08 +02:00
Hassan Igbaria
e8b5364745
Update YoutubeStreamExtractor.java
2021-05-21 16:22:24 +03:00
litetex
c3b837fe3b
Deduped code; Using default values...
2021-05-20 20:22:40 +02:00
litetex
b934c7ccbb
Removed unused import
2021-05-20 20:11:35 +02:00
litetex
10cf081145
Removed likeCount and added textualVoteCount
2021-05-20 20:10:14 +02:00
TobiGr
1acc53b8db
[PeerTube] Add support for HLS streams
2021-05-15 12:39:23 +02:00
Tobi
3a3ade20f4
Merge pull request #620 from TeamNewPipe/bandcamp-comments
...
Add Bandcamp comments (first page)
2021-05-02 13:43:53 +02:00
Fynn Godau
85a4270bd9
[Bandcamp] Fix comments test
2021-05-02 13:07:24 +02:00
TobiGr
ec50570053
Add mocks
2021-05-01 17:50:19 +02:00
TobiGr
f4404b5cc8
Add tests for more reasons
2021-05-01 17:50:19 +02:00
TobiGr
fc998589dc
Detect channels which have been terminated due to copyright infringement
2021-05-01 17:50:19 +02:00
TobiGr
bb3861ddce
Add AccountTerminatedException for better error handling of terminated channels
2021-05-01 17:50:19 +02:00
Tobi
284362fc15
Merge pull request #608 from TeamNewPipe/youtube-fix-segment-test
...
[YouTube] fix tests
2021-05-01 17:24:41 +02:00
TobiGr
dc678a9f6d
Release version 0.21.2
2021-04-26 18:06:30 +02:00
Tobi
7cf8edd5ac
Merge pull request #618 from TeamNewPipe/soundcloud_id
...
Fix Soundcloud extraction
2021-04-26 18:03:26 +02:00
TobiGr
9c12dc5609
[SoundCloud] Fix SoundCloud ID extraction
...
resolveIdWithEmbedPlayer() does not work anymore because the JSON data has been extracted to an API call. For this reason, replace resolveIdWithEmbedPlayer() with resolveIdWithWidgetApi)( which performs the API call.
2021-04-26 17:58:34 +02:00
Fynn Godau
9dc7a1d5ee
[Bandcamp] Add comments (first page)
2021-04-26 17:25:04 +02:00
TobiGr
8f023c1ec7
[SoundCloud] Update client id
2021-04-26 15:14:11 +02:00
Tobi
7f202db8b1
Fix Google URL detection ( #617 )
2021-04-26 11:49:47 +02:00
fynngodau
6db4bea8ca
Rework Bandcamp comments ( #613 )
2021-04-13 21:10:59 +02:00
Tobi
c14f6db14a
Merge pull request #610 from TeamNewPipe/bandcamp_related_items
...
Extract related bandcamp items
2021-04-13 21:05:27 +02:00
Fynn Godau
3671876721
Fix bandcamp radio stream extractor
2021-04-13 19:30:11 +02:00
Fynn Godau
90b5c00599
Extract related items #593
2021-04-13 17:56:19 +02:00
TobiGr
1fe645704f
[YouTube] FIx testMoreRelatedItems in RandomQueryNoMorePages
2021-04-11 23:21:35 +02:00
TobiGr
9a2b814d8f
Update mocks
2021-04-11 22:59:01 +02:00
TobiGr
50005ce937
[YouTube] Replace unavailable YouTube Stream Segments test
...
The stream is unavailable in many countries.
2021-04-11 22:58:48 +02:00
Tobi
82d11386df
Merge pull request #607 from TeamNewPipe/bandcamp_radio_segments
...
[Bandcamp] Add Radio stream segments
2021-04-11 22:30:03 +02:00
TobiGr
318cec7625
[Bandcamp] Add Radio stream segments
2021-04-11 21:19:44 +02:00
TobiGr
55e0f8e725
Merge branch 'master' into dev
2021-04-10 11:33:34 +02:00
Tobi
e747a89be2
Merge pull request #600 from TeamNewPipe/youtube-eu-cosent
...
[YouTube] Set EU consent cookie
2021-04-09 12:02:52 +02:00
TobiGr
33173eb3e6
Make CONSENT COOKIE constants private
2021-04-09 11:59:17 +02:00
TobiGr
b94b316558
Update mocks
2021-04-09 11:59:17 +02:00
TobiGr
4b8d4a84b6
Set the NumberGenerator for th remaining tests
2021-04-09 11:59:17 +02:00
XiangRongLin
db7ad9c27d
Update Mocks
2021-04-08 16:37:29 +02:00
XiangRongLin
4833df60e5
Add method to inject Random into YoutubeParsingHelper and use in tests
2021-04-08 16:36:55 +02:00
TobiGr
682ec27737
Update mocks
2021-04-08 14:22:31 +02:00
XiangRongLin
eafad3ad6b
Merge pull request #602 from XiangRongLin/soundcloud_go
...
Ignore unreliable SoundcloudGoPlusTrack.testRelatedItems
2021-04-07 19:41:44 +02:00
TobiGr
080d1e318d
Code improvement
2021-04-07 14:11:47 +02:00
TobiGr
4ee270fe01
Fix LiveStream Test
2021-04-07 14:11:41 +02:00
TobiGr
883f16e0ad
[YouTube] Set CONSENT cookie
2021-04-07 14:11:37 +02:00
XiangRongLin
aee3838ed5
Ignore unreliable SoundcloudGoPlusTrack.testRelatedItems
2021-04-06 18:26:45 +02:00
XiangRongLin
90afd6162a
Merge pull request #599 from XiangRongLin/post_body
...
Add body field to Page and use it
2021-04-06 18:23:26 +02:00
XiangRongLin
7b06c696e2
Use Page.body for YoutubeChannelExtractor
2021-04-06 18:14:38 +02:00
XiangRongLin
eda1b6e199
Use Page.body for YoutubePlaylistExtractor
2021-04-06 18:14:38 +02:00
XiangRongLin
3af26a2821
Add body field to Page
2021-04-04 11:39:58 +02:00
Tobi
4e0be60ddc
Merge pull request #601 from XiangRongLin/fix_tests
...
Fix SoundcloudGoPlusTrack Test
2021-04-04 09:55:54 +02:00
XiangRongLin
7b36469807
Fix SoundcloudGoPlusTrack
...
https://soundcloud.com/martinsolveig/places#t=0 has related streams
2021-04-04 09:39:26 +02:00
Tobi
b481a228cd
Merge pull request #598 from XiangRongLin/fix_tests
...
Fix a test and ignore rest
2021-04-03 23:44:23 +02:00
XiangRongLin
3d70b6d46a
Fix a test and ignore rest
2021-04-03 11:03:10 +02:00
XiangRongLin
1925dcf4dc
Merge pull request #597 from fynngodau/related-items
...
Rename getRelatedStreams to getRelatedItems and change return type
2021-04-02 19:54:22 +02:00
Fynn Godau
c877712647
Add deprecated old method calls to avoid breaking API
2021-04-01 22:32:36 +02:00
Fynn Godau
14f6f1b7c3
Generify related streams calls and rename method
2021-03-31 20:24:40 +02:00
Stypox
9dbc152d44
Merge pull request #594 from TeamNewPipe/conersions
...
Reduce number of type conversions
2021-03-31 14:34:46 +02:00
TobiGr
1498e1905e
Reduce number of type conversions in YouTubeSearchExtractor.collectStreamsFrom by introducing new variable
2021-03-31 13:30:38 +02:00
Tobi
7ea2cd73d3
Merge pull request #589 from fynngodau/bandcamp-more-recent-pages
...
Load more featured pages and fix featured cover arts not loading
2021-03-31 12:15:12 +02:00
Fynn Godau
705f6c6e33
Apply review
2021-03-31 11:02:30 +02:00
XiangRongLin
564d74c250
Merge pull request #578 from TeamNewPipe/code_improvements
...
Code improvements
2021-03-30 18:12:22 +02:00
TobiGr
657b00ca11
Smaller code improvements
2021-03-30 10:10:53 +02:00
TobiGr
8fbd1a2bcf
FIx potential NullPointerException
2021-03-30 10:07:20 +02:00
TobiGr
f2e7c7ebd6
Replace Vector with ArrayList for performance improvements
...
I could not figure out, why these two methods are necessary.
2021-03-30 10:07:20 +02:00
TobiGr
48fd8ca7b3
Remove redundant type declaration
2021-03-30 10:07:20 +02:00
TobiGr
22bd16fca3
Make vars private, final and static
2021-03-30 10:07:20 +02:00
TobiGr
6a0f6e846a
Fix possible bug when language.id field is not present and this the languageCode an empty String.
2021-03-30 10:07:20 +02:00
TobiGr
b9282bbe94
Remove checks which are always true
2021-03-30 10:07:20 +02:00
TobiGr
46eab1ec17
Remove declarations of exceptions which are not thrown from method signatures
2021-03-30 10:07:20 +02:00
TobiGr
c3e23559d7
Remove unused imports
2021-03-30 10:07:20 +02:00
TobiGr
e4c40ae6d1
Use List.isEmpty()
2021-03-30 10:07:20 +02:00
XiangRongLin
af2183855a
Merge pull request #591 from TotalCaesar659/update-url-to-https
...
Update URL to HTTPS in README
2021-03-29 09:33:59 +02:00
TotalCaesar659
cf3f54f438
Update URL to HTTPS
2021-03-28 22:06:56 +03:00
Fynn Godau
d663be5a78
[Bandcamp] Fix featured albums not showing thumbnails
2021-03-28 13:12:22 +02:00
Fynn Godau
dbcf61c6f7
[Bandcamp] Load more featured pages
2021-03-28 12:53:05 +02:00
Stypox
b4dee6d08f
Merge pull request #306 from B0pol/metadata
...
Extract metadata for youtube, soundcloud & mediaccc
2021-03-27 08:45:47 +01:00
TobiGr
e61ceef005
Release 0.21.0
2021-03-25 21:59:15 +01:00
Tobi
e2500fb7cc
Merge pull request #583 from TeamNewPipe/peertube_playlist
...
Fix some tests and bugs
2021-03-25 21:53:41 +01:00
TobiGr
070a40e181
[YouTube] Mix Playlist - Fix getting name
2021-03-25 21:47:16 +01:00
TobiGr
b029779217
[media.cc.de] Disable unreliable tests for now
2021-03-25 19:44:44 +01:00
TobiGr
bbee15474d
[PeerTube] Fix playlist extraction
2021-03-25 19:19:19 +01:00
Tobi
d4f83a1782
Merge pull request #575 from TiA4f8R/fix-peertube-test
...
Fix Peertube account subscribers extraction
2021-03-25 19:14:51 +01:00
TiA4f8R
ae283314da
Add a comment explaining subscribers count calculation
2021-03-25 17:54:46 +01:00
TiA4f8R
abb790f465
Remove println in getSubscriberCount from PeertubeAccountExtractor class
2021-03-25 17:49:22 +01:00
TiA4f8R
772de53a66
Increase the number of subscribers in PeertubeAccountExtractorTest
2021-03-25 17:49:20 +01:00
TiA4f8R
efe2b964f0
Extract real subscribers of a Peertube account
...
Apply fix provided in the PR and move all accounts/ strings to a constant, ACCOUNTS.
2021-03-25 17:47:44 +01:00
Tobi
12835bfae1
Merge pull request #551 from FireMasterK/ff-privacy
...
Change UA to privacy.resistFingerprinting.
2021-03-24 17:58:34 +01:00
Tobi
cae6cea551
Merge pull request #582 from Stypox/fix-trending
...
[YouTube] Fix trending getName()
2021-03-24 10:49:46 +01:00
Stypox
ce7cbbc9a0
[YouTube] Fix trending getName()
2021-03-24 09:14:09 +01:00
XiangRongLin
9009f1e277
Merge pull request #517 from XiangRongLin/flaky_test
...
Add MockOnlyRule to allow skipping specific tests based on downloader
2021-03-20 18:04:14 +01:00
bopol
152221c7fb
ignore a test with encoding problems
2021-03-19 10:14:43 +01:00
bopol
8806fb4e6b
address reviews
2021-03-19 10:14:43 +01:00
bopol
c47cc54908
Extract metadata for YouTube, SoundCloud & MediaCCC
2021-03-19 10:14:43 +01:00
Stypox
f71cfd489c
Merge pull request #526 from TiA4f8R/snd-hls-workaround
...
Support SoundCloud HLS-only tracks by using a workaround
2021-03-19 10:09:54 +01:00
TiA4f8R
379d7312fa
Don't use a regular expression to find the last segment URL and do code improvements
...
Apply suggestions provided in the PR and remove a redundant import.
2021-03-14 17:54:22 +01:00
TiA4f8R
0e3e420a25
Fix tests
2021-03-14 17:54:21 +01:00
TiA4f8R
d61d9d116d
Refactor getAudioStreams method of SoundcloudStreamExtractor
...
Split the method into private methods, in order to have a better reading.
2021-03-14 17:54:20 +01:00
TiA4f8R
a7b15b51e6
Change t to transcodingJsonObject in SoundcloudStreamExtractor
2021-03-14 17:54:19 +01:00
TiA4f8R
0438828e36
Add a test for the number of audioStreams in CreativeCommonsPlaysWellWithOthers test
...
It should be only two audio streams for track "Plays Well with Others, Ep 2: What Do an Army of Ants and an Online Encyclopedia Have in Common?" by Creative Commons (https://soundcloud.com/wearecc/plays-well-with-others-ep-2-what-do-an-army-of-ants-and-an-online-encyclopedia-have-in-common ):
- one which is a progressive stream, in MP3 format with a bitrate of 128 kbps
- one which is an HLS stream, in OPUS format with a bitrate of 64 kbps.
2021-03-14 17:54:18 +01:00
TiA4f8R
3bd08a2880
Adress requested changes and use final where possible in SoundcloudStreamExtractor
...
This commit moved the HLS parsing task to a separate method, did little performance improvements and used final where possible in the SoundcloudStreamExtractor file.
2021-03-14 17:54:17 +01:00
TiA4f8R
cbacd3c0a5
Add a check to don't show MP3 128kbps stream twice and catch IOException when fetching the HLS Manifest
...
If a progressive stream is present in the transcodings, it's unnecessary to show twice an MP3 128kbps stream so if this is the case, the MP3 HLS stream will be not added to the audioStreams, else it will.
This commit also catch fetching errors in HLS manifests parsing and don't add the corresponding stream if an error occurs.
2021-03-14 17:54:16 +01:00
TiA4f8R
26f1b4e7dc
Support SoundCloud HLS by using a workaround
...
This commit tries to support SoundCloud HLS streams by parsing M3U manifests, get the last segment URL (in order to get track length) and request a segment URL equals to track's duration so it's a single URL.
2021-03-14 17:54:14 +01:00
Tobi
def745b801
Merge pull request #579 from fynngodau/bandcamp-fix-linkhandler
...
[Bandcamp] Fix link handler acceptance behaviour
2021-03-14 12:56:10 +01:00
Fynn Godau
2e57a8f24f
[Bandcamp] Fix link handler acceptance behaviour
...
* Test for bandcamp footer instead of meta tag (which is not present on
all pages)
* Accept links to /music, not just /releases
* Correctly handle uppercase URLs
2021-03-14 09:51:30 +01:00
TobiGr
70d9e389b9
[Bandcamp] Add tests for external bandcamp artists
...
Necessary, because the external pages tested before were converted to bandcamp.com pages. See f9d06252f2
2021-03-13 20:51:59 +01:00
TobiGr
f9d06252f2
[Bandcamp] Fix tests
2021-03-13 17:42:39 +01:00
Tobi
7e6f464407
Merge pull request #509 from TiA4f8R/soundcloud-improvements
...
Add new exceptions to be able to display different error messages in apps
2021-03-07 13:51:15 +01:00
Tobi
ac51134aed
Merge pull request #546 from XiangRongLin/unignore2
...
Unignore more tests
2021-03-07 13:15:24 +01:00
Tobi
021da75f24
Merge pull request #232 from fynngodau/dev
...
Bandcamp support
2021-03-05 21:37:57 +01:00
TobiGr
22fa131922
Merge branch 'dev' into bandcamp
2021-03-05 19:45:37 +01:00
TiA4f8R
e55284bb8f
Code style improvements
...
Use final where possible in YoutubeStreamExtractor and do some other code style improvements
2021-03-05 16:38:53 +01:00
TobiGr
d0a1041afd
Prevent NullPointerException and use equalsIgnorecase
2021-03-05 16:38:52 +01:00
TiA4f8R
890cbba625
Update PaidContent exception
2021-03-05 16:38:51 +01:00
TobiGr
da3cfa967d
Handle age-restricted videos
2021-03-05 16:38:50 +01:00
TiA4f8R
448b68700c
Use final in exceptions and add AgeRestrictedException
...
AgeRestrictedException will be thrown only if the reason message equals to "Sign in to confirm your age" and if the age limit is 18.
2021-03-05 16:38:49 +01:00
TobiGr
22e6f33f0a
Ignore SoundCloudGeoRestrictedTest until HLS stream extraction is added
...
The ContentNotSupportedException is thrown because no supported audio streams where extracted. However, SoundCLoud does not check, whether there are any streams available.
This commit should be reverted in #526
2021-03-05 16:38:47 +01:00
TobiGr
21158744b1
Add expectedDescriptionIsEmpty() to DefaultStreamExtractorTest
...
Also check if related streams are empty if they are expected to be empty.
2021-03-05 16:38:46 +01:00
TiA4f8R
f15d7837a1
Fix SoundCloud tests
2021-03-05 16:38:45 +01:00
TiA4f8R
4e45aef2b3
Fix SoundCloud CreativeCommonsPlaysWellWithOthers test
2021-03-05 16:38:43 +01:00
TiA4f8R
59d6d3f04e
Fix YouTube tests
2021-03-05 16:38:42 +01:00
TiA4f8R
3925204658
Add tests and rename SoundcloudGoPlusException to SoundcloudGoPlusContentException
2021-03-05 16:38:40 +01:00
TiA4f8R
771bb1a2cb
Add three new exceptions for unavailable YouTube contents
...
These exceptions are thrown on a test with the error messages text, because YouTube returns only "UNPLAYABLE" status in most error cases.
Tests are based with English strings, so changing the lang used by
extractor will throw the generic exception (ContentNotAvailableException).
2021-03-05 16:38:39 +01:00
TiA4f8R
35325d980d
Add GeographicRestrictionException and SoundCloudGoPlusException in NewPipe Extractor to be able to display different error messages
...
This commit adds two new exceptions in NewPipe Extractor: GeographicRestrictionException and SoundCloudGoPlusException (which extend to ContentNotAvailableException). These exceptions allow showing different error messages to user when a content isn't available in his/her/its country (only used for now by SoundCloudStreamExtractor) or when the content is a SoundCloud Go+ track.
2021-03-05 16:38:37 +01:00
TobiGr
cb07ffa1eb
Release 0.20.11
2021-03-05 15:35:11 +01:00
Tobi
bc0cda68d1
Merge pull request #570 from TiA4f8R/fix-yt-channels-playlists
...
Fix YouTube channels contination
2021-03-05 15:26:49 +01:00
Tobi
a3c6fceef5
Merge pull request #573 from B0pol/comments-performance
...
[youtube] improve comments extraction performance
2021-03-05 14:46:27 +01:00
bopol
ff5273b882
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeCommentsExtractor.java
...
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2021-03-05 14:39:01 +01:00
Tobi
ec1127dd6a
Merge pull request #571 from TeamNewPipe/soundcloud_client_id
...
[SoundCloud] Update client_id and fix extraction
2021-03-05 14:21:14 +01:00
bopol
ed850d0688
[youtube] improve comments extraction performance
...
- do not parse responseBody twice for continuation
instead try to get commentsTokenInside with the new pattern ("sectionListRenderer")
and try again with the old pattern ("commentSectionRenderer") on failure
- do not unescape responseBody multiple times
-> parse responseBody less times
2021-03-05 13:37:08 +01:00
TobiGr
440a808b8a
Update mocks for channels
2021-03-05 13:13:36 +01:00
Tobi
b608587e4d
Merge pull request #572 from golfinq/fixed-comments
...
Fixes missing comments under videos
2021-03-05 11:42:40 +01:00
golfinq
05213175cd
minor edit, made variable final
2021-03-05 02:49:15 -05:00
golfinq
8bceb57a6c
formatting fix
2021-03-05 02:46:06 -05:00
golfinq
ae48bdea4c
Updated to reflect variable name change
2021-03-05 02:40:36 -05:00
TiA4f8R
b27efdc9a4
Remove clickTrackingParams in getNextPage method of YoutubeChannelExtractor
...
Because it's unused with the new request.
Co-authored-by: bopol <bopol@e.email>
2021-03-04 22:50:20 +01:00
TobiGr
2b671b15ce
[YouTube] Use new continuation API for channels
...
Co-authored-by: TiA4f8R <74829229+tia4f8r@users.noreply.github.com>
2021-03-04 21:47:16 +01:00
TobiGr
33594d4aff
[SoundCloud] Fix client id extraction
...
SoundCloud changed their Javascript structure. That caused the client_id to be in the middle of the file (at byte ~43000 ). To have a little buffer, we now fetch the first 50000 bytes.
2021-03-04 21:32:34 +01:00
TobiGr
fb2f37d223
[SoundCloud] Update hardcoded client id
2021-03-04 21:25:41 +01:00
XiangRongLin
9256b3b848
Merge pull request #567 from XiangRongLin/playlist_continuations
...
Playlist continuations
2021-03-04 19:05:32 +01:00
XiangRongLin
506cc5fbe9
regenerate mocks
2021-03-04 19:01:58 +01:00
XiangRongLin
03b00ff1d6
Move toJsonArray and toJsonObject to JsonUtils
2021-03-04 18:58:51 +01:00
FireMasterK
668b080b3a
Use dynamic key.
2021-03-04 08:45:14 +01:00
XiangRongLin
43b46bd408
Use new youtube continuations api for playlists
...
Requires sending a POST request instead of GET.
clientName and clientVersion, which were required as headers previously now need to be part of the request payload.
continuation id also needs to be part of request body.
quick and dirty solution.
2021-03-03 19:49:26 +01:00
XiangRongLin
beb05bd05c
Replace test url, because playlist was deleted
2021-03-03 19:46:14 +01:00
XiangRongLin
e13e237392
Fix typo and reword some explanations
2021-02-26 17:48:03 +01:00
Tobi
5d594cfded
Merge pull request #549 from FireMasterK/gradle-update
...
Update gradle wrapper.
2021-02-24 15:59:28 +01:00
Tobi
965a66bc87
Merge pull request #557 from TiA4f8R/fix-encoding-errors-windows
...
Set extractor encoding to UTF-8
2021-02-24 15:49:58 +01:00
TiA4f8R
b2cf41496d
Set extractor encoding to UTF-8
...
When compiling the extractor on Windows, you get more than 100 errors with the message error: unmappable character for encoding Cp1252, especially in timeago-parser tasks. This commit fixes it by setting the encoding for all compileJava and compileTestJava tasks to UTF-8.
2021-02-23 14:38:20 +01:00
TobiGr
a1688fe953
Move BandcampExtractorHelper.getJsonData(String, String) to JsonUtils
2021-02-21 13:51:12 +01:00
TobiGr
70814dcfef
Fix Utils.nonEmptyAndNullJoin
...
When using the index here, it the index needs to be decremented once an element is removed. To cirecumvent this, the native Collections.removeIf() method is used.
2021-02-21 12:51:47 +01:00
TobiGr
b9e8ee8450
Rename BandcampExtractorHelper.smartConcatenate(String[], String) to Utils.nonEmptyAndNullJoin(String, String[])
2021-02-21 12:35:14 +01:00
TobiGr
c07db80ef0
Add BASE_URL and BASE_API_URL to BandcampExtractorHelper
2021-02-21 12:15:45 +01:00
TobiGr
adde4332d1
Code improvements
2021-02-20 23:58:54 +01:00
FireMasterK
42bdd8adc5
Update to fix failing mock tests.
2021-02-20 17:14:41 +05:30
FireMasterK
e8bccfaf5d
Change UA to privacy.resistFingerprinting.
2021-02-20 17:14:41 +05:30
bopol
a9aa385bb3
Merge pull request #554 from B0pol/test-verified
...
use DownloaderFactory instead of DownloaderTestImpl for search Channe…
2021-02-20 12:12:43 +01:00
bopol
0c831afa7b
use DownloaderFactory instead of DownloaderTestImpl for search ChannelVerified
2021-02-20 11:41:30 +01:00
TobiGr
ea120a4637
remove print stacktrace
...
use interface
2021-02-20 00:23:45 +01:00
XiangRongLin
adf9d7d10f
Add reason field to MockOnly
...
This enforces developers to document why a test is skipped
2021-02-19 18:54:18 +01:00
XiangRongLin
ea52030613
Add MockOnlyRule to allow skipping specific tests based on downloader
2021-02-19 18:54:17 +01:00
TobiGr
02920fafa8
Add isUploaderVerified()
2021-02-19 15:51:23 +01:00
TobiGr
91e9309486
Merge remote-tracking branch 'origin/dev' into bandcamp
2021-02-19 13:37:45 +01:00
TobiGr
fa61b864f2
Code improvements
2021-02-19 13:37:24 +01:00
TobiGr
98268e351c
Move radio URL check into a function
2021-02-19 13:35:44 +01:00
FireMasterK
959d5d7db7
Add dependabot.
2021-02-19 11:52:56 +05:30
bopol
27a20e41cd
Merge pull request #550 from FireMasterK/add-more-data
...
Add more data.
2021-02-18 20:28:27 +01:00
FireMasterK
5333d8a98b
Add more data.
2021-02-19 00:55:03 +05:30
FireMasterK
88890d81a5
Update gradle wrapper.
2021-02-18 21:15:12 +05:30
bopol
9c1a7f7df6
Merge pull request #521 from B0pol/verified
...
Add uploader verified by service extraction
2021-02-18 16:39:53 +01:00
bopol
1a322ad8ed
Add uploader verified by service extraction
2021-02-18 16:36:40 +01:00
bopol
db253e202b
Merge pull request #535 from TeamPiped/adaptive-parsing
...
Adaptive Stream parsing
2021-02-18 15:55:27 +01:00
FireMasterK
c24afa2cbb
Permalink URL.
2021-02-18 20:16:53 +05:30
FireMasterK
8d54401233
Add fps field as well.
2021-02-18 20:16:53 +05:30
FireMasterK
4d096be14a
Revert "Update gradle wrapper."
...
This reverts commit 591856e3a11998c4b8ba87b5e361e2b9556d760f.
2021-02-18 20:16:53 +05:30
FireMasterK
5b70645d85
Add data to muxed streams aswell.
2021-02-18 20:16:53 +05:30
FireMasterK
e975d33fbe
Change jdoc / comments.
2021-02-18 20:16:53 +05:30
FireMasterK
f31b2a68fd
use getters/setters.
2021-02-18 20:16:53 +05:30
FireMasterK
d4945ac55c
Constructor cleanup.
2021-02-18 20:16:52 +05:30
FireMasterK
3f6a601be8
Fix jdoc / comments.
2021-02-18 20:16:52 +05:30
FireMasterK
a7c9905183
Fix requested changes.
2021-02-18 20:16:52 +05:30
FireMasterK
0c0f2d74bc
Make fields private
...
Co-authored-by: bopol <bopol@e.email>
2021-02-18 20:16:52 +05:30
FireMasterK
d1054338d3
Fix error with livestreams.
2021-02-18 20:16:52 +05:30
FireMasterK
11eb4932f4
Add data to respective classes.
2021-02-18 20:16:52 +05:30
FireMasterK
525e345ed8
remove duplicate field.
2021-02-18 20:16:52 +05:30
FireMasterK
e844d2aed3
Add dash fields to the ItagItem.
2021-02-18 20:16:52 +05:30
FireMasterK
b713a7af8c
Update gradle wrapper.
2021-02-18 20:16:52 +05:30
bopol
64f6b0478f
Merge pull request #545 from XiangRongLin/comment_reset
...
Add comment explaining YoutubeParsingHelper.resetClientVersionAndKey
2021-02-18 15:14:03 +01:00
XiangRongLin
a64579da2f
Unignore some tests that work on Linux
2021-02-17 20:31:26 +01:00
XiangRongLin
b99cb8624f
Unignore PublicBroadcasterTest
...
Tests only work on linux, mocks have to be generated on linux too
2021-02-17 20:31:26 +01:00
XiangRongLin
37366c65ca
Remove encoding of '&' to '&' in test and unignore it
2021-02-17 20:31:26 +01:00
XiangRongLin
c327922cc0
Add explanations to broken tests
2021-02-17 20:31:26 +01:00
XiangRongLin
26d50aca05
Add comment explaining YoutubeParsingHelper.resetClientVersionAndKey
2021-02-17 19:21:39 +01:00
TobiGr
54aa5b3042
Use propper structure in KioskExtractors
...
Made BandCampRadioExtractor a Kiosk which holds StreamInfoItems and not InfoItems.
2021-02-16 21:09:42 +01:00
TobiGr
54b8e54f80
Fix potential NPE
2021-02-16 20:59:13 +01:00
TobiGr
16973126a0
Use Collections.emptyList() instead of Collections.EMPTY_LIST or new ArrayList()
2021-02-16 20:25:48 +01:00
bopol
84380e4c3a
Merge pull request #536 from B0pol/watchendpoints
...
support /watch/, /v/ and /w/ subpaths
2021-02-16 20:23:27 +01:00
bopol
41f689b099
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/linkHandler/YoutubeStreamLinkHandlerFactory.java
...
Co-authored-by: Stypox <stypox@pm.me>
2021-02-16 19:17:10 +01:00
XiangRongLin
a9594f531f
Merge pull request #543 from XiangRongLin/unignore1
...
Unignore and fix some tests
2021-02-16 18:00:16 +01:00
XiangRongLin
a28e0226c5
Merge pull request #519 from XiangRongLin/better_logging_error_handling
...
Better logging, error handling, doc for mock tests
2021-02-16 15:06:06 +01:00
XiangRongLin
05cb22c654
Unignore Gronkh.testDescription
...
Remove special character ü from assertion
2021-02-16 08:59:35 +01:00
XiangRongLin
d9b4c2779f
Unignore YoutubeSearchQHTest.testRegularValues
2021-02-16 08:59:35 +01:00
XiangRongLin
137272b7a4
Unignore YoutubeCommentsExtractorTest.EmptyComment
...
Mock files were missing
2021-02-16 08:59:34 +01:00
TobiGr
5bf9fddba9
Code improvements
2021-02-15 22:45:17 +01:00
TobiGr
e062c8cb0d
Merge branch 'dev' into bandcamp
2021-02-15 22:09:41 +01:00
bopol
416cf17b7a
Merge pull request #532 from B0pol/utf8
...
Code improvement
2021-02-15 18:56:28 +01:00
bopol
7a3d9bdb7d
Description: add .equals() and .hashCode()
2021-02-15 18:51:47 +01:00
bopol
557934cb17
use EMPTY_STRING
2021-02-15 18:51:46 +01:00
bopol
9ca52ca68a
use only one constant UTF-8
2021-02-15 18:47:54 +01:00
XiangRongLin
50e5718bd1
Merge pull request #518 from XiangRongLin/generate_mocks
...
Use and generate mocks for youtube tests
2021-02-15 06:39:41 +01:00
Tobi
d7486f251a
Merge pull request #538 from B0pol/soundcloud-georestriction
...
Soundcloud: fix charts extraction when georestricted
2021-02-14 23:52:27 +01:00
TobiGr
432c68cdec
Update okhttp from 3.12.11 to 3.12.13
...
Changelog https://square.github.io/okhttp/changelog_3x/#version-31213
2021-02-14 23:44:19 +01:00
TobiGr
5eb75d72d3
Release 0.20.10
2021-02-14 22:09:17 +01:00
XiangRongLin
a88b2e6af4
Undo formatting of YoutubeStreamExtractorDefaultTest
2021-02-14 14:57:23 +01:00
bopol
8d89c82caa
Soundcloud: fix charts extraction when georestricted
2021-02-13 22:28:31 +01:00
bopol
ee2012c948
rename folder to subpath
2021-02-13 22:25:46 +01:00
bopol
8db40db4de
use substring instead of split
2021-02-13 22:20:05 +01:00
bopol
81317cf6bc
youtube: support /w/ folder
2021-02-13 22:20:05 +01:00
bopol
f15c0fcfed
support /watch/ and /v/ folders
2021-02-13 22:20:05 +01:00
XiangRongLin
971128c464
Ignore failing tests
2021-02-13 20:28:03 +01:00
XiangRongLin
92dea0806a
Regenerate json files for youtube tests
2021-02-13 20:28:03 +01:00
XiangRongLin
d6b9930c18
Use DownloaderFactory in YoutubeCommentsExtractorTest.Pinned
2021-02-13 20:12:39 +01:00
XiangRongLin
b555f38fa5
Use real downloader for YoutubeSubscriptionExtractorTest
...
It doesn't make network requests anyway
2021-02-13 20:12:39 +01:00
XiangRongLin
f352b0c7af
YoutubeMusicSearchExtractorTest can't be used with mocks
...
Requires further research
2021-02-13 20:00:51 +01:00
XiangRongLin
f45b5610bf
Unignore YoutubeChannelLocalizationTest
...
With mocks time is not a concern anymore
2021-02-13 20:00:51 +01:00
XiangRongLin
19737e06ee
Use DownloaderFactory in YoutubeChannelExtractorTest
...
KurzgesagtAdditional needed to be extracted because it ran a duplicated request.
2021-02-13 20:00:51 +01:00
XiangRongLin
84eb4b30c9
Use DownloaderFactory in YouTube tests
2021-02-13 20:00:51 +01:00
XiangRongLin
b43f3474f5
Make resetClientVersionAndKey public
2021-02-13 20:00:49 +01:00
Tobi
8d7b62914c
Merge pull request #540 from B0pol/fix-tests
...
fix tests, fix links in YouTube description
2021-02-13 13:20:29 +01:00
bopol
2859c7ba5c
add comment explaining why .substring(23)
2021-02-13 12:10:41 +01:00
bopol
73353996d7
fix more tests
2021-02-12 23:18:38 +01:00
bopol
a931e31239
fix tests
2021-02-12 22:36:18 +01:00
TobiGr
d116680ea3
Merge branch 'master' into dev
2021-02-12 22:26:35 +01:00
TobiGr
f8d769617a
Release 0.20.9
2021-02-12 22:14:23 +01:00
Tobi
c2525916ac
Merge pull request #539 from B0pol/yt-channel-continuation
...
[YouTube]: Fixed channel continuation
2021-02-12 22:09:31 +01:00
bopol
b0f356dd76
YT channels: fix continuation
2021-02-12 14:39:09 +01:00
bopol
fcdb50b825
make getInfo from Extractor public
...
In StreamInfo and CommentsInfo
2021-02-09 14:58:43 +01:00
bopol
44c54d403a
fix Invidious channels being accepted by YoutubeStreamLinkHandlerFactory ( #527 )
...
fixes https://github.com/TeamNewPipe/NewPipeExtractor/issues/524 , see it for more details
2021-02-05 18:57:58 +01:00
TiA4f8R
1414a6f178
Fix extraction of YouTube chapters ( #531 )
...
YouTube changed the name of the chapters in the JSON from engagement-panel-macro-markers to engagement-panel-macro-markers-description-chapters, so extracting chapters doesn't work.
2021-02-03 22:07:34 +01:00
Tobias Groza
bfa639950e
Merge pull request #528 from Stypox/ytmusic-0
...
[YouTube Music] Fix extracting search item view/subscriber count when = 0
2021-01-29 23:23:23 +01:00
TobiGr
d728c4fad0
Remove unused imports
...
I searched for "StandardCharsets" to prevent possible bugs with Android KitKat. The import is unused and is therefor removed.
There are other usages of the StandardCharsets class in the test implementation, but those are not relevant.
2021-01-29 23:16:40 +01:00
Stypox
a64dfd7343
[YouTube Music] Fix extracting search item view/subscriber count when = 0
2021-01-29 16:44:24 +01:00
Tobias Groza
c00ee75d74
Merge pull request #525 from B0pol/rename
...
rename getHeartedByCreator and getPinned to isPinned and isHeartedByC…
2021-01-23 13:45:24 +01:00
bopol
2662ceca22
rename getHeartedByCreator and getPinned to isPinned and isHeartedByCreator
2021-01-23 13:06:07 +01:00
Tobias Groza
10e8c543e8
Merge pull request #523 from B0pol/pinned
...
Extract whether the comment is pinned
2021-01-23 12:35:13 +01:00
bopol
66e4eb2f96
extract whether the comment is pinned
2021-01-22 23:59:22 +01:00
Tobias Groza
69f155d292
Merge pull request #522 from B0pol/invidious_instances
...
update invidious instances list
2021-01-22 21:04:10 +01:00
bopol
67ddfefdc6
keep invidious instances list in the same order
...
as suggested by @TobiGr
2021-01-22 20:58:14 +01:00
bopol
53d3f7989b
update invidious instances list
2021-01-22 19:30:25 +01:00
XiangRongLin
88e4c8667b
Add comments to RecordingDownloader about usage
2021-01-17 19:53:20 +01:00
XiangRongLin
eecfe09f2c
Check for non-existent folder in MockDownloader
2021-01-17 19:52:58 +01:00
XiangRongLin
a0288b8fd6
Echo which downloader is used during CI process
2021-01-17 19:52:28 +01:00
Tobias Groza
beb705013c
Merge pull request #516 from XiangRongLin/schedule_ci_with_real
...
Add if-else statement to run tests against real website on schedule
2021-01-16 12:59:05 +01:00
XiangRongLin
7a6d930e7b
Add if-else statement to run tests against real website on schedule
2021-01-16 10:03:47 +01:00
TobiGr
07570554b6
Fix build and optimize imports
2021-01-15 21:55:40 +01:00
TobiGr
78c2113094
Merge remote-tracking branch 'origin/dev' into bandcamp
2021-01-15 21:49:58 +01:00
Stypox
b9ba95614b
Merge pull request #514 from XiangRongLin/test_jvm_property
...
Pass on gradle JVM system property to test JVM
2021-01-15 20:44:23 +01:00
XiangRongLin
a6cfe11ae9
Pass on gradle JVM system property to test JVM
2021-01-15 20:37:14 +01:00
Stypox
48a9993edd
Merge pull request #513 from XiangRongLin/ci_use_mock
...
Adjust CI workflow to run tests with mocks
2021-01-15 20:20:23 +01:00
Stypox
588fbcc61f
Merge pull request #512 from XiangRongLin/ignore_failing
...
Ignore all failing tests
2021-01-15 20:18:03 +01:00
XiangRongLin
6888e7773f
Ignore all failing tests
2021-01-15 20:11:40 +01:00
XiangRongLin
e5dc62d9bd
Add argument -Ddownloader=MOCK to CI job
...
Now MockDownloader will be used in tests.
2021-01-15 18:04:33 +01:00
Stypox
7c774c84ce
Merge pull request #505 from TeamNewPipe/meida.ccc.de_search_fix
...
[media.ccc.de] Fix NPE in search results if they contain a future talk
2021-01-15 14:16:59 +01:00
Stypox
c2c4d97efd
Merge pull request #482 from XiangRongLin/mock_mix_pl_test
...
Mock mix pl tests
2021-01-15 12:54:48 +01:00
XiangRongLin
4f81d9d689
Regenerate mock files
2021-01-15 09:51:23 +01:00
Stypox
951159f75f
Merge pull request #504 from TeamNewPipe/media.ccc.de_recent_order
...
[media.ccc.de] Recent kiosk: order streams by upload date
2021-01-14 23:31:09 +01:00
Tobias Groza
cc51c5f6a3
Merge pull request #507 from vkay94/add-interval-to-stream-frames
...
Add durationPerFrame to Frameset
2021-01-14 20:27:40 +01:00
XiangRongLin
6ca7123d5d
Add comment for usage of YoutubeParsingHelper.resetClientVersionAndKey
2021-01-14 20:01:52 +01:00
vkay94
11dcfe638b
Add durationPerFrame to Frameset and getFrameBoundsAt method
2021-01-14 20:01:06 +01:00
Tobias Groza
c2ff6723d7
Merge pull request #511 from B0pol/errors
...
Fix PeerTube tests by changing instance
2021-01-14 19:15:22 +01:00
Tobias Groza
6773ec8699
Merge pull request #510 from B0pol/heartedbycreator
...
comments: add heartedByUploader and extract it for YouTube
2021-01-12 19:00:04 +01:00
TobiGr
0c4836c800
[media.ccc.de] Recent kiosk: Add comment clarifying "length" and "duration" fields have the same value
2021-01-12 18:48:42 +01:00
XiangRongLin
4af50c6870
Use @link for downloader
2021-01-12 09:41:57 +01:00
XiangRongLin
3455f0f23c
Add documentation to testing with mocks
2021-01-12 09:37:48 +01:00
bopol
5292cc1636
[peertube] test: remove usage of peertube.mastodon.host
...
It's down for months
2021-01-11 19:46:27 +01:00
bopol
f3c22da618
comments: add heartedByCreator and extract it for youtube
2021-01-11 18:48:08 +01:00
Tobias Groza
da968cf9c5
Merge pull request #508 from XiangRongLin/360_webm
...
[YouTube] Un-comment out 360p webm
2021-01-11 16:09:05 +01:00
XiangRongLin
63c237db41
Only create directories if they do not exist
2021-01-11 07:57:11 +01:00
XiangRongLin
e82cb1efea
Remove dependency to commons-io
2021-01-10 20:50:38 +01:00
XiangRongLin
1ea6c6ce54
Prefix mock file from RecordingDownloader with "generated_mock_"
...
Only read those files in MockDownloader
2021-01-10 20:46:42 +01:00
XiangRongLin
f91916c017
Remove unnecessary file object creation
2021-01-10 20:39:22 +01:00
XiangRongLin
255c726f20
Add resource path base to DownloaderFactory
2021-01-10 20:27:34 +01:00
XiangRongLin
35e299759e
Add method to allow resetting youtube client version and key
...
This is needed so that a request is made for each test class when running multiple at once. This way RecordingDownloader records all necessary requests.
This works as long as tests are run sequentially and not in parallel.
2021-01-10 20:24:50 +01:00
XiangRongLin
f447a7a450
fix import due to downloader package move
2021-01-10 19:49:14 +01:00
XiangRongLin
f8aa989c42
Add generated json and manually copy client_version.json
2021-01-10 19:38:47 +01:00
XiangRongLin
285c26eafa
Adjust YoutubeMixPlaylistExtractorTest to use DownloaderFactory
2021-01-10 19:38:41 +01:00
XiangRongLin
e6e8e39def
Add DownloaderFactory to return a specific downloader based on 2 variables.
...
If the system property 'downloader' is set that use that specific downloader. This is used from gradle by appending `-Ddownloader=ABCD to the command.
ABCD is one of DownloaderType.
The other variable is the static property `DEFAULT_DOWNLOADER` in DownloaderFactory, which can be easily changed as needed inside the IDE according to development needs`.
Normal workflow would be to first use the recording downloader and afterwards only use mocks, if the requests are always staying the same.
2021-01-10 19:36:24 +01:00
XiangRongLin
7c40fb8bf7
Add additional downloader implementations
...
RecordingDownloader relies on the real downloader and saves the request/response pair into a json file.
MockDownloader uses json files from above and mocks responses for specific requests.
2021-01-10 19:36:24 +01:00
XiangRongLin
1bcb9c76a7
Generate equals and hashCode for Request
2021-01-10 19:36:24 +01:00
XiangRongLin
4dad3d60d6
Move DownloaderTestImpl into downloader package
2021-01-10 19:36:21 +01:00
XiangRongLin
fe3902c669
Un-comment out 360p webm
2021-01-08 19:43:03 +01:00
TobiGr
2d93b23723
[media.ccc.de] Show conference title in search results instead of slug
2021-01-08 14:15:25 +01:00
TobiGr
52f3bd15f2
[media.ccc.de] Add conference title to "recent" kiosk
...
The "conference_title" field was added upstream recently (see https://github.com/voc/voctoweb/issues/495 )
2021-01-08 12:04:39 +01:00
Tobias Groza
ab62464e86
Merge pull request #495 from B0pol/sepiasearch
...
[PeerTube] Implement sepia search
2021-01-06 11:06:36 +01:00
bopol
0bcea9c2cd
add JDoc
2021-01-01 14:37:32 +01:00
bopol
0e45c25ea9
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/peertube/extractors/PeertubeStreamInfoItemExtractor.java
...
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2021-01-01 14:20:47 +01:00
TobiGr
fae67fbd45
Merge remote-tracking branch 'origin/master' into dev
2021-01-01 13:06:11 +01:00
Tobias Groza
c4b9183684
Merge pull request #506 from TeamNewPipe/gh-actions
...
Migrate to GitHub actions from Travis
2021-01-01 10:35:33 +01:00
TobiGr
4671e68304
Update links to Java SE components to Java SE 8
2020-12-31 18:05:49 +01:00
TobiGr
d63d20f943
Fix JDoc
2020-12-31 18:05:40 +01:00
FireMasterK
1ba689900e
Migrate to GitHub actions from Travis.
2020-12-31 18:05:35 +01:00
TobiGr
2cbc3ccb1a
[media.ccc.de] Fix live stream test if there are no current live streams
2020-12-31 17:03:23 +01:00
TobiGr
a896ec41b9
[media.ccc.de] Use Pattern for getting the ive stream id
2020-12-31 17:03:23 +01:00
TobiGr
5dc9a76e3c
[media.ccc.de] Recent kiosk: order streams by upload date
2020-12-30 18:38:09 +01:00
TobiGr
edf8dd0e92
[media.ccc.de] Fix NPE in search results if they contain a future talk
2020-12-30 18:34:28 +01:00
Tobias Groza
3c8c8e7307
Merge pull request #497 from TiA4f8R/fix-player-js-url-redirect
...
Fix YouTube player JavaScript URL redirection
2020-12-27 18:04:40 +01:00
TobiGr
288ddab65b
update version name
2020-12-27 18:02:16 +01:00
TiA4f8R
d30caeb091
Fix YouTube player JavaScript URL redirection
2020-12-27 15:55:03 +01:00
TobiGr
deb9af7bf5
[media.ccc.de] live kiosk - display info of the current talk
2020-12-27 15:52:09 +01:00
TobiGr
21c6a7884c
Release 0.20.8
2020-12-27 14:50:34 +01:00
Tobias Groza
1259521d53
Merge pull request #500 from TeamNewPipe/media.ccc.de_live
...
[media.ccc.de] live stream kiosk
2020-12-27 14:47:55 +01:00
TobiGr
b4e1913971
[media.ccc.de] Play live streams
2020-12-27 13:25:48 +01:00
TobiGr
80f4d42226
[Draft] Add live extractor
2020-12-27 11:01:22 +01:00
TobiGr
ed9402c002
[media.ccc.de] Escape "." in id pattern
2020-12-27 11:01:10 +01:00
TobiGr
674f2227f3
make url pattern static
2020-12-27 01:27:37 +01:00
TobiGr
090dade5b3
[media.ccc.de] Add short test for recent kiosk
2020-12-27 01:03:08 +01:00
TobiGr
0cfefe222a
Revert "[media.ccc.de] Get conference name from API URL"
...
This reverts commit d1a0686d2f
.
2020-12-26 16:55:44 +01:00
TobiGr
d1a0686d2f
[media.ccc.de] Get conference name from API URL
2020-12-26 13:15:03 +01:00
TobiGr
41699798fe
[media.ccc.de] Use web URL for uploaderUrl instead of API URL in recent
2020-12-26 12:47:32 +01:00
TobiGr
b2154c6e87
[media.ccc.de] Fix conferences kiosk test
2020-12-25 23:55:45 +01:00
TobiGr
ed6ae329b3
[media.ccc.de] Add "recent" kiosk
2020-12-25 22:26:58 +01:00
TobiGr
b2837698f5
Release 0.20.7
2020-12-25 18:55:06 +01:00
Tobias Groza
b5e50cc9fb
Merge pull request #484 from TeamNewPipe/yt_,music_search
...
Fix YouTube Music search
2020-12-25 18:30:25 +01:00
TobiGr
d9e2da53c3
Replace explicit string checks whether a playlist íd is a certain YouTube Mix type with calling the dedicated methods
2020-12-25 15:00:34 +01:00
TobiGr
abee0a8df1
Fix typos in YoutubePlaylistLinkHandlerFactory
2020-12-24 10:34:21 +01:00
TobiGr
3033c0b993
[YouTube] Music Mix: Fix playlist not being accepted
...
Regression introduced by YouTube Mix support (#280 )
2020-12-23 21:08:36 +01:00
TobiGr
b62144b49d
[YouTube] Music: Fix search extraction caused by changed JSON response
2020-12-23 21:08:36 +01:00
TobiGr
c9d9bd1e24
Fix typo in ListLinkHandlerFactory#getUrl
2020-12-23 21:05:58 +01:00
bopol
3ae924a7f1
Implement sepia search
2020-12-23 15:07:55 +01:00
TobiGr
942d840624
Fix typo in AffiliateService: rename "NO_AFILIATE" to "NO_AFFILIATE"
2020-12-23 12:50:45 +01:00
TobiGr
865c42e273
Fix line break
2020-12-23 12:49:50 +01:00
Tobias Groza
c682ea0d18
Merge pull request #451 from TeamNewPipe/meta_info
...
Extract stream and search meta info for YouTube
2020-12-22 23:05:33 +01:00
Fynn Godau
c9e9953bb0
[Bandcamp] Fix channel link handler factory
2020-12-22 20:02:53 +01:00
TobiGr
41a8ed625d
Make some vars final and add annotations to methods
2020-12-22 18:19:26 +01:00
TobiGr
bc6de14952
Extract stream and search meta info for YouTube
...
Add method to extract Google webcache URLs.
2020-12-22 18:19:26 +01:00
TobiGr
50903730b1
[Bandcamp] Fix accepting HTTP URLs
2020-12-22 08:53:10 +01:00
Tobias Groza
853a65a1a6
Merge pull request #491 from TeamNewPipe/dependecies
...
Update junit
2020-12-22 08:31:03 +01:00
TobiGr
838dc1ab54
Fix vulnerability in junit
2020-12-21 23:22:16 +01:00
Tobias Groza
7e481cf81a
Merge pull request #490 from B0pol/peertube_related
...
[peertube] fix related streams api url when no tags
2020-12-20 19:04:35 +01:00
bopol
8d3436565f
[peertube] fix related streams api url when no tags
...
fixes #489
2020-12-19 23:25:37 +01:00
Tobias Groza
22a415156f
Merge pull request #479 from vkay94/stream-segments
...
Extract stream segments for YouTube
2020-12-19 11:12:22 +01:00
TobiGr
74b46fed2d
[Bandcamp] Fix deprecation in parseDate helper function
2020-12-19 10:40:21 +01:00
TobiGr
116e921d6c
Merge remote-tracking branch 'origin/dev' into bandcamp
2020-12-19 00:26:11 +01:00
Tobias Groza
e8cc302d5f
Merge pull request #483 from XiangRongLin/cleanup-mix-pl
...
Cleanup mix pl
2020-12-19 00:13:17 +01:00
TobiGr
df16a8646f
[Bandcamp] Improve radio stream extractor test
2020-12-18 23:15:59 +01:00
XiangRongLin
cdcb66b93e
Clean up stuff from mix pl
...
Coming from unclean merge and missing hints from IDE
2020-12-16 08:35:28 +01:00
XiangRongLin
ba8782a9ed
Use lowercase string for extracting cookies
...
The map in a real response happens to ignore upper-/lowercase differences. Other maps used in unit test may not have that behaviour.
2020-12-16 08:32:04 +01:00
Stypox
85fa006214
Merge pull request #280 from XiangRongLin/mixPL
...
Extractor for youtube mix (auto-generated playlist)
2020-12-14 18:11:07 +01:00
XiangRongLin
f90f6fcf92
[YouTube] Don't escape & in getUrlFromNavigationEndpoint for playlists
2020-12-12 20:40:13 +01:00
Xiang Rong Lin
a338e4e08e
[Youtube] Apply review suggestions and avoid channel mix edge case
2020-12-12 20:32:43 +01:00
Xiang Rong Lin
22d2f7e400
[Youtube] Add cookies to youtube mix request
...
This way youtube wont return duplicates when getting more items of the mix (but youtube can also track us)
2020-12-12 20:32:40 +01:00
Stypox
421935401f
[Youtube] Add subchannel functions to mix and fix imports
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
3ff8619bcc
[Youtube] apply wb9688 suggestion (mix)
...
Channel mix adjusments and test
Don't accept youtube music mix urls as playlist
Don't override playlistData to keep getInitialPage()
Remove json constants
Indentation
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
822cf307f7
[Youtube] Add _ITEMS constants and improve code style
...
Move thumbnail id exctraction code to getThumbnailUrlFromId
Add test for "My mix" detection to service tests
Use ITEM_COUNT_UNKNOWN everywhere instead of -1 and add some tests
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
df38b1926c
[Youtube] Add tests and take thumbnail image always from first video of mix
...
Also fix getThumbnailUrl for "My Mix"
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
68a3948af6
[Youtube] Fix get banner url
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
ec6b99c082
[Youtube] Adjust mix extractor to new user agent
...
Also extract continuation now
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
0ff054acb4
[Youtube] Extract initial playlist info
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
a376792a5d
[Youtube] Handle case where url is in "youtube.com/playlist?list=listID" format.
...
This occurs when sharing a mix from the official youtube app.
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
327a5730a8
[Youtube] Add some comments to mix
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
d74265c846
[Youtube] Extract getThumbnailUrl into method and change getUploaderName
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
0efb854d27
[Youtube] Implement mix extractor for auto-generated playlists.
...
-New YoutubeMixPlaylistExtractor, that extracts from a mix (auto-generated playlist).
-The url has the format of "youtube.com/watch?v=videoID&playlistID",
where playlistID always starts with "RD" and usually followed by the videoID.
-Change YoutubePlaylistLinkHandlerFactory to create a linkhandler with the given url if it is a mix.
-Change YoutubeService to return YoutubeMixPlaylistExtractor if the url is a mix.
2020-12-12 20:30:57 +01:00
vkay94
2ba27b39af
Stream segments: Remove replaceAll from setPreviewUrl
2020-12-12 15:00:45 +01:00
Stypox
2b622fd2f1
Merge pull request #419 from Isira-Seneviratne/Use_Objects_methods
...
Use Objects methods.
2020-12-12 14:52:42 +01:00
vkay94
9a7a224a54
Add tests for stream segments
2020-12-12 10:24:55 +01:00
vkay94
d3eea4f4be
Add stream segments to StreamInfo
2020-12-12 10:24:29 +01:00
Isira Seneviratne
57be1f1123
Use Objects.requireNonNull().
2020-12-11 06:09:57 +05:30
Isira Seneviratne
4c19a88612
Use Objects.toString().
2020-12-11 06:09:50 +05:30
Isira Seneviratne
b8f64595a2
Use Objects' static equals() and hashCode() methods.
2020-12-11 06:09:44 +05:30
TobiGr
b3835bd616
Release 0.20.6
2020-12-09 23:30:23 +01:00
Tobias Groza
6fbcdd24ee
Merge pull request #477 from vkay94/fix-comments-parsing
...
Fix comments parsing
2020-12-09 23:25:51 +01:00
vkay94
9dbacbc618
Fix comments parsing
2020-12-09 14:57:38 +01:00
Fynn Godau
04dd3d4d32
Rework link handlers to correctly accept external websites
2020-12-05 15:08:26 +01:00
Tobias Groza
8ade913d9b
Merge pull request #472 from TeamNewPipe/fix-media-ccc-de
...
Fix name of media.ccc.de service
2020-12-02 22:16:52 +01:00
TheAssassin
ba3e2302bc
Fix name of media.ccc.de service
2020-12-01 23:54:27 +01:00
TobiGr
175df679e0
Release 0.20.5
2020-12-01 08:18:29 +01:00
Tobias Groza
b7a995187f
Merge pull request #463 from opusforlife2/update_invidious_URLs
...
Update Invidious URL List
2020-11-28 13:30:32 +01:00
Fynn Godau
be562b8436
Change tests
2020-11-27 16:48:40 +01:00
TobiGr
c91e21b37c
[Bandcamp] Tests - Add finals and improve code formatting
2020-11-24 14:13:31 +01:00
TobiGr
99e7ef013e
[Bandcamp] Apply small changes to code formatting and style
...
Make variables final when possible
Remove unused imports
Improve code formatting
2020-11-24 14:13:31 +01:00
fynngodau
8c369b0f79
Rephrase link in javadoc
...
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-11-24 14:01:31 +01:00
TobiGr
5a3f96b967
Release 0.20.4
2020-11-22 22:29:08 +01:00
Tobias Groza
650f0920fe
Merge pull request #465 from XiangRongLin/playlist_continuation
...
[YouTube] Fix playlist continuations extraction
2020-11-20 20:48:20 +01:00
TobiGr
334e1e9b53
Fix YouTube comments test
2020-11-20 19:53:53 +01:00
Fynn Godau
186936d041
Various changes regarding tests
2020-11-19 22:33:52 +01:00
Fynn Godau
6bc7e3420e
Merge TNP/dev into fynngodau/dev
2020-11-19 21:32:08 +01:00
TobiGr
3582a5189f
Release 0.20.3
2020-11-18 20:41:25 +01:00
XiangRongLin
5bceff0083
[YouTube] Fix extraction of next page url for the last page of playlist
2020-11-18 19:03:12 +01:00
XiangRongLin
8347e14952
[YouTube] Fix playlist continuations extraction
2020-11-17 21:04:53 +01:00
opusforlife2
2174055ef8
Update Invidious URL list in Parsing Helper
2020-11-11 15:12:31 +00:00
opusforlife2
c0afd5213a
Update Invidious URL list in Link Handler Factory
2020-11-11 14:54:16 +00:00
Tobias Groza
6701b0fe71
Merge pull request #447 from Stypox/better-deobfuscation
...
[YouTube] Cache deobfuscation code and more fixes
2020-11-08 00:32:36 +01:00
Stypox
89a77ae74a
[YouTube] Fix detection of ended livestreams and parse livestream upload date
2020-11-04 16:03:08 +01:00
Stypox
827f7bd137
[YouTube] Cache deobfuscation and improve requests made
...
Fix age restriction extraction
Automatically fixes more things
2020-11-04 16:02:12 +01:00
Stypox
b42a196f35
Merge pull request #454 from Isira-Seneviratne/Fix_date_time_issues
...
Fix some issues related to the date/time changes.
2020-11-04 14:48:20 +01:00
Isira Seneviratne
4fe28d7e3a
Fix YouTube parse error when only a date is present.
2020-11-04 05:49:42 +05:30
Isira Seneviratne
9cf9e7e980
Call existing constructor in DateWrapper.
2020-11-04 05:49:41 +05:30
Isira Seneviratne
fe31a90cb3
Remove DateTimeFormatter.ISO_OFFSET_DATE_TIME usage.
2020-11-04 05:49:41 +05:30
Tobias Groza
b13c7e1c1e
Merge pull request #452 from Stypox/yt-import
...
Implement YouTube subscription import from Google takeout
2020-11-03 20:32:17 +01:00
Tobias Groza
ac50068bbe
Merge pull request #456 from B0pol/youtube_comments_test
...
create YouTubeCommentsLinkHandlerFactoryTest
2020-11-03 20:14:14 +01:00
bopol
345e136f6c
create YouTubeCommentsLinkHandlerFactoryTest
...
and remove invidious test from YouTubeCommentsExtractorTest, because it was just testing if the URL is accepted, then the extractor does the same thing, we don't need to test the same thing twice
2020-11-03 19:10:10 +01:00
Stypox
501ec30152
Implement youtube subscription import from Google takeout
2020-11-02 11:04:48 +01:00
Tobias Groza
564a965810
Merge pull request #453 from bd0n4lds/fix-donation-typo
...
Fix typo in DonationLinkHelper and rewrote swtich statement
2020-11-02 00:15:04 +01:00
Bri@n
82746d172f
Fix typo in DonationLinkHelper and rewrote swtich statement
2020-11-01 17:34:34 -05:00
Stypox
2f02c0e6a4
Merge pull request #450 from B0pol/fileutils
...
create FileUtils
2020-11-01 18:13:57 +01:00
bopol
f69b3ef05d
create FileUtils
2020-11-01 17:15:34 +01:00
TobiGr
be9e160333
Fix build
2020-11-01 17:04:02 +01:00
Tobias Groza
8cbdec675b
Merge pull request #417 from Isira-Seneviratne/Use_Java_8_Date_Time_API
...
Use the Java 8 Date/Time API.
2020-11-01 16:49:06 +01:00
Isira Seneviratne
fcdfe7d939
Update README.md.
2020-11-01 06:28:54 +05:30
Isira Seneviratne
4f04cfccca
Switch from Calendar to OffsetDateTime in DateWrapper.
2020-11-01 06:28:54 +05:30
Isira Seneviratne
ee3af63c04
Switch to ChronoUnit.
2020-11-01 06:28:53 +05:30
Isira Seneviratne
0526a5148d
Set source and target versions to Java 8.
2020-11-01 06:28:52 +05:30
Isira Seneviratne
b2d0c098a3
Merge branch 'dev' of https://github.com/TeamNewPipe/NewPipeExtractor into dev
2020-11-01 06:27:59 +05:30
Stypox
6cc50b57e3
Merge pull request #410 from Scrxtchy/sets-track-title
...
Fix parsing Soundcloud tracks that contain the term 'sets'
2020-10-31 21:43:21 +01:00
Stypox
b242e1d113
Merge pull request #362 from Stypox/google-search-urls
...
Add support for Google search redirect url
2020-10-31 21:03:48 +01:00
Stypox
9e53cf0b56
Fix parameter reassignment and other style issues
...
Also remove left-behind debug statement
2020-10-31 20:58:28 +01:00
Stypox
3fe55b30ba
Add support for Google search redirect url
2020-10-31 20:58:28 +01:00
Stypox
c190a3029b
Consider protocol as base url when it is a custom one (e.g. vnd.youtube)
2020-10-31 20:58:28 +01:00
Isira Seneviratne
35c7b5640c
Merge branch 'dev' of https://github.com/TeamNewPipe/NewPipeExtractor into dev
2020-10-31 05:45:02 +05:30
Stypox
a9303b24a5
Merge pull request #445 from B0pol/peertube_fix
...
[PeerTube] fix account and channel extractors
2020-10-30 19:35:51 +01:00
bopol
b21e59925d
[PeerTube] fix account and channel extractors
2020-10-29 19:52:29 +01:00
Isira Seneviratne
61dcbbb800
Merge branch 'dev' of https://github.com/TeamNewPipe/NewPipeExtractor into dev
...
Conflicts:
extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java
2020-10-29 12:20:16 +05:30
Stypox
bcc01b99c7
Merge pull request #444 from Stypox/error-message
...
Remove any reference to decrypt and improve error message
2020-10-28 23:41:04 +01:00
Stypox
30ed4f2d63
Remove any reference to decrypt and improve error message
2020-10-28 20:26:00 +01:00
TobiGr
62912ee834
Merge branch 'master' into dev
2020-10-28 14:28:06 +01:00
TobiGr
94176c5d68
Merge branch 'master' into dev
2020-10-28 07:52:50 +01:00
Tobias Groza
19c0e8700d
0.20.2
2020-10-28 07:29:42 +01:00
Tobias Groza
354426fe5d
Merge pull request #441 from B0pol/fix-three-attempts
...
Support new YouTube JSON scheme
2020-10-28 07:24:31 +01:00
Scratch
947ce3ee10
Fix parsing Soundcloud tracks that contain the term 'sets'
2020-10-28 07:55:02 +11:00
bopol
6dc5ab4015
find playerUrl in another place when assetsPattern regex fails
2020-10-27 13:48:58 +01:00
bopol
db0ef83d6b
fix youtube decryption and three attemps bug
...
fixes teamnewpipe/newpipe#4572 fixes #439
2020-10-27 13:40:24 +01:00
Stypox
943334ba49
Merge pull request #440 from B0pol/fix-redirect-channels
...
Fix redirect channels
2020-10-26 20:00:06 +01:00
bopol
c1e9857960
fix subscriber count when subscribe is disabled
...
fixes #305
2020-10-26 19:06:01 +01:00
bopol
a39a2cca82
fix redirect channels
2020-10-26 19:05:19 +01:00
Tobias Groza
a8665fd97f
Merge pull request #442 from B0pol/polish
...
Polish
2020-10-26 18:51:47 +01:00
bopol
0a12300c5e
polish tests
2020-10-26 16:57:37 +01:00
bopol
01f49e8f66
polish strings
2020-10-26 16:32:39 +01:00
Tobias Groza
14c0c37d64
Merge pull request #309 from Stypox/improve-stream-tests
...
Improve stream extractor tests & various fixes
2020-10-25 18:49:53 +01:00
Stypox
df26f61cfc
Merge pull request #394 from kirisakow/dev
...
Fix TeamNewPipe/NewPipe#4213 : [YouTube] Ignore leading characters in video id
2020-10-25 14:39:03 +01:00
Kiril Isakov
366ed71523
[YouTube] Ignore leading characters in video id
2020-10-25 14:34:20 +01:00
Stypox
8026304a0a
Merge pull request #409 from Bartoshr/bugfix/short-channel-urls
...
Support short custom YouTube channel URLs
2020-10-25 11:43:50 +01:00
Stypox
57e7994c9e
Add some missing finals, nullables and comments
2020-10-25 08:12:41 +01:00
Stypox
f11fe87688
[YouTube] Replace outdated PewDiePie video test with another one
...
The old video was made private, and this video (wedding) is probably never going to be removed.
2020-10-24 18:41:41 +02:00
Stypox
3c55ea9321
[PeerTube] Change age restricted video in tests
...
The old one wasn't available anymore
2020-10-24 18:41:41 +02:00
Stypox
880ff04a5c
[Test] Add stream dash mpd url test
2020-10-24 18:41:41 +02:00
Stypox
d0b14644bb
[YouTube/MediaCCC] Consider dates as GMT and not as local
2020-10-24 18:41:41 +02:00
Stypox
8ce711f40f
[Test] Add sub channel name, url and thumbnail tests
2020-10-24 18:41:41 +02:00
Stypox
6127826571
[Test] Add stream metadata tests
2020-10-24 18:41:41 +02:00
Stypox
a087b092b4
[Test] Improve code style and add final
2020-10-24 18:39:55 +02:00
Stypox
a4097d8d01
[MediaCCC] Return empty list of video-only streams instead of null
2020-10-24 18:39:55 +02:00
Stypox
8dc3f28618
[PeerTube] Test one channel url with api and one without
2020-10-24 18:39:55 +02:00
Stypox
68d23defba
[YouTube] Do not catch every exception on getErrorMessage
...
@B0pol suggestion
2020-10-24 18:39:55 +02:00
Stypox
3191bd6c70
[YouTube] Use final when possible
2020-10-24 18:39:55 +02:00
Stypox
55bc01d1ce
[SoundCloud] Use final when possible, ide refactorings
2020-10-24 18:39:55 +02:00
Stypox
06430c4749
[PeerTube] Use final when possible, ide refactorings
2020-10-24 18:39:06 +02:00
Stypox
fcb9b6f855
[MediaCCC] Use final when possible, ide refactorings
...
Refactorings on `throws` clause
2020-10-24 18:36:22 +02:00
Stypox
af5b8b1915
[MediaCCC] Add tests for stream and conference link handlers
2020-10-24 18:36:21 +02:00
Stypox
07a90d116a
[MediaCCC] Use regex to parse stream and conference urls
2020-10-24 18:36:21 +02:00
Stypox
0c980b2d64
[PeerTube] Improve channel and stream link handler tests
2020-10-24 18:36:21 +02:00
Stypox
d130fd79c3
[PeerTube] Prepend "accounts/" to channel id for backward compatibility
2020-10-24 18:36:21 +02:00
Stypox
492db83ccf
[MediaCCC] Return null instead of empty items collector
...
As per the documentation in the base getRelatedStreams()
2020-10-24 18:34:57 +02:00
Stypox
3b2a1829d4
[MediaCCC] Extract tags
2020-10-24 18:34:57 +02:00
Stypox
b461da792f
[MediaCCC] Fix link handler inconsistency providing API links
2020-10-24 18:34:34 +02:00
Stypox
aeeae87641
[PeerTube] Parse timestamp from url (previously unimplemented)
2020-10-24 18:33:42 +02:00
Stypox
7ae3cb6d07
[PeerTube] Fix link handler inconsistency providing API links
2020-10-24 18:33:11 +02:00
Stypox
4349be13af
[PeerTube] Return empty audio stream list instead of null
2020-10-24 18:25:56 +02:00
Stypox
3b2cfb4ca2
[SoundCloud] Return empty video stream list instead of null
...
Also replace every instance of `return new ArrayList<>();` with `return Collections.emptyList();`
2020-10-24 18:23:46 +02:00
Stypox
072bae321f
[YouTube] Fix frame extraction for livestreams
...
Use saved playerResponse instead of parsing json every time
2020-10-24 18:23:46 +02:00
Stypox
7cd410f3fc
[YouTube] Return 0 when there is no timestamp, not -2, as per javadoc
2020-10-24 18:23:45 +02:00
Stypox
7fb867c166
[YouTube] Fix error message obtaining when there is none
2020-10-24 18:22:44 +02:00
Stypox
1d7a86e664
[Test] Add base classes for stream extractor tests
...
Refactor all stream extractor tests to use new base class.
Remove check if upload date is in the past: this does not have to hold true: youtube premieres turn up in search results even though they are in the future
2020-10-24 18:22:44 +02:00
Bartosz Rumiński
29695aed0a
Small field refactor
2020-10-23 16:42:30 +02:00
Bartosz Rumiński
0e67d820bc
Use static regex pattern for excluded path segments
2020-10-22 20:13:29 +02:00
Bartosz Rumiński
d3f80d1538
Exlude links which are not channels
2020-10-20 20:06:06 +02:00
Tobias Groza
3058af0e43
Merge pull request #418 from Stypox/yt-shorts
...
[YouTube] Handle URLs for Shorts
2020-10-18 13:06:10 +02:00
Stypox
6887d59570
[YouTube] Handle urls for Shorts
2020-10-18 12:03:01 +02:00
Scratch
e945f711c1
Fix SoundCloud test artist account name ( #416 )
2020-10-16 23:42:46 +02:00
TobiGr
350eed6214
Release v0.20.1
2020-10-16 22:03:57 +02:00
Tobias Groza
f82eda9f7d
Merge pull request #415 from wb9688/fix-ytinitialdata
...
Fix parsing new ytInitialData
2020-10-16 21:59:39 +02:00
wb9688
be9a6f931c
Fix parsing new ytInitialData
2020-10-16 20:27:40 +02:00
Tobias Groza
1a9bc8caf1
Merge pull request #414 from Scrxtchy/soundcloud-slash
...
Remove tailing slash from SoundCloud URLs
2020-10-16 09:48:44 +02:00
Scratch
6a70cb9d50
Remove tailing slash from SoundCloud URLs
...
Fixes #412
2020-10-16 17:17:40 +11:00
Tobias Groza
9a785f9a7e
Merge pull request #413 from Scrxtchy/soundcloud-test-update
...
Fix SoundCloud test artist account name
2020-10-16 08:05:44 +02:00
Scratch
538f5d3973
Fix SoundCloud test artist account name
2020-10-16 13:45:05 +11:00
Tobias Groza
527945eadb
Merge pull request #406 from Stypox/fix-decrypt
...
[YouTube] Fix some decryption exceptions by retrying
2020-10-15 22:03:25 +02:00
Stypox
19e862657a
[YouTube] Fix some decryption exceptions by retrying
2020-10-15 20:16:44 +02:00
Bartosz Rumiński
e3f996e014
Exlude links which are not channels
2020-10-12 20:59:56 +02:00
Bartosz Rumiński
5ab1b053d2
Update youtube channel link handler tests
...
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-10-12 20:11:28 +02:00
Bartosz Rumiński
7abb4b3713
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/linkHandler/YoutubeChannelLinkHandlerFactory.java
...
Fix typos
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-10-12 19:57:45 +02:00
Bartosz Rumiński
9d63211a66
Fix typos
...
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-10-12 19:56:53 +02:00
Bartosz Rumiński
8c38a5509e
Prevent attribution_link urls to be accepted by channel links handler
2020-10-09 19:07:38 +02:00
Bartosz Rumiński
4e04991762
Support short custom youtube channel urls
2020-10-09 17:06:59 +02:00
Fynn Godau
81b5e7cf3d
Fix extractor
2020-10-08 17:56:03 +02:00
TobiGr
acb04eb351
Update extractor version to 0.20.0
2020-10-04 13:37:19 +02:00
Stypox
2463884aa8
Merge pull request #400 from wb9688/skip-otf
...
Do not list YouTube's OTF streams
2020-09-29 20:16:34 +02:00
wb9688
ebbfe7f6d4
Skip YouTube's OTF streams
2020-09-29 10:48:02 +02:00
Tobias Groza
6633f26ec5
Merge pull request #386 from mhmdanas/remove-subtitles-stream-get-url
...
Remove SubtitlesStream#getURL()
2020-08-26 21:32:36 +02:00
Tobias Groza
070cd92857
Merge pull request #387 from wb9688/init-safe-standard-objects
...
Use initSafeStandardsObjects()
2020-08-26 21:25:46 +02:00
wb9688
7657c2ed1a
Use initSafeStandardsObjects()
2020-08-15 17:08:07 +02:00
mhmdanas
1a63dcb355
Remove overriden field `url`
2020-08-12 22:37:36 +03:00
Tobias Groza
de8edbe7a5
Merge pull request #382 from TobiGr/fix/empty_comment
...
[YouTube] Fix crash on empty comment
2020-08-12 14:18:16 +02:00
mhmdanas
f0f1c009b2
Remove SubtitlesStream#getURL()
2020-08-10 19:28:56 +03:00
Fynn Godau
cfe88a74c1
Throw ContentNotSupportedException when opening radio uploader channel
2020-08-09 10:07:49 +02:00
TobiGr
0fb73301e3
[YouTube] Fix crash on empty comment
...
Closes #380
2020-08-05 18:25:40 +02:00
Fynn Godau
932d094d6a
Upgrade featured playlist urls to https
2020-08-02 16:55:18 +02:00
Fynn Godau
c12ef3a02d
Merge TNP/dev into fynngodau/dev
2020-08-02 16:43:04 +02:00
TobiGr
5ac80624a4
Fix detection of YouTube's decryption function
...
Closes TeamNewPipe/NewPipe#3951
2020-07-28 01:01:10 +02:00
TobiGr
b4481dfec2
Update version to 0.19.7
2020-07-26 14:24:07 +02:00
Tobias Groza
32dff1541f
Merge pull request #370 from wb9688/fix-yt-age-restricted
...
Fix age restricted YouTube videos
2020-07-26 14:17:19 +02:00
TobiGr
ec778200d9
Add comments to the code
2020-07-26 14:15:13 +02:00
Tobias Groza
8627d01006
Merge pull request #373 from wb9688/fix-yt-continuations
...
Support YouTube's new continuations
2020-07-26 13:56:33 +02:00
wb9688
667dce034c
Only use fallback methods for clientVersion when clientVersion hasn't been successfully extracted yet
2020-07-26 13:14:25 +02:00
wb9688
f345f667e2
Extract YouTube's key
2020-07-26 12:01:59 +02:00
wb9688
4c987a5302
Support YouTube's new continuations for search
2020-07-26 10:01:03 +02:00
wb9688
81459e289f
Fix age restricted YouTube videos
2020-07-18 19:52:54 +02:00
Tobias Groza
df28a087de
Merge pull request #337 from wb9688/next-stream
...
Remove getNextStream() from StreamExtractor
2020-07-18 11:55:58 +02:00
wb9688
fc3a63fec5
Update Javadoc for getRelatedStreams()
2020-07-15 18:48:37 +02:00
wb9688
ff560e907a
Remove getNextStream() from StreamExtractor
2020-07-15 18:48:37 +02:00
TheAssassin
186193d366
Update version to 0.19.6
2020-07-14 17:04:03 +02:00
Tobias Groza
a70cb0283f
Merge pull request #314 from wb9688/remove-getnextpageurl
...
Next page stuff
2020-07-07 20:49:38 +02:00
wb9688
0a5a905bc7
Add final at more places
2020-07-07 20:45:47 +02:00
Tobias Groza
e5d23a89de
Merge pull request #366 from wb9688/yt-timestamp-start
...
Support start= timestamps for YouTube
2020-07-06 21:28:19 +02:00
wb9688
17ba8a57fa
Clean up the code
2020-07-06 20:19:31 +02:00
wb9688
9b6fe1dea6
Throw IllegalArgumentException when Page is invalid
2020-07-06 20:19:31 +02:00
wb9688
4cc312086a
Introduce Page class
2020-07-06 20:19:31 +02:00
wb9688
e3bfdba135
Remove getNextPageUrl() function from ListExtractor
2020-07-06 20:11:40 +02:00
wb9688
156a26b64b
Support start= timestamps for YouTube
2020-07-06 14:08:40 +02:00
Tobias Groza
54d9e5a2f8
Merge pull request #365 from B0pol/update_inv_instances
...
Update inv instances
2020-07-02 23:13:07 +02:00
bopol
261471e7f4
remove invidiou.sh from tests
...
dead instance
2020-07-02 21:33:45 +02:00
bopol
eb3901acee
update invidious instances
2020-07-02 21:31:05 +02:00
Tobias Groza
c0ceb5cb27
Merge pull request #340 from wb9688/peertube-deleted-comments
...
Handle isDeleted for PeerTube comments
2020-06-28 10:21:37 +02:00
Tobias Groza
f4ed7ce922
Merge pull request #361 from yausername/gradleScope
...
expose extractor via api scope
2020-06-27 21:45:04 +02:00
Tobias Groza
339f1d9c43
Merge pull request #349 from wb9688/okhttp
...
Use OkHttp for tests like in NewPipe
2020-06-27 20:08:02 +02:00
Tobias Groza
970bc7f69d
Merge pull request #359 from mauriciocolli/fix-client-id-bug
...
[SoundCloud] Fix concurrency issue when getting the client id
2020-06-27 17:36:24 +02:00
Tobias Groza
dbf6ab9e91
Merge pull request #355 from Redirion/patch-1
...
Added a comment
2020-06-27 11:47:50 +02:00
Robin
576754982e
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java
...
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-06-27 11:35:01 +02:00
yausername
89e0e3ee2d
expose extractor via api scope
2020-06-27 03:02:43 +05:30
Mauricio Colli
7505e828cc
[SoundCloud] Update client id
2020-06-24 01:50:09 -03:00
Mauricio Colli
d2f1c0f40d
[SoundCloud] Fix concurrency issue when getting the client id
2020-06-24 01:50:08 -03:00
Robin
b63ae93495
Added a comment
...
and fixed a "typo" in the method parseDurationString replacing Long with Integer.
These are not the droids you are looking for wb9688 😁
2020-06-15 11:27:44 +02:00
Tobias Groza
92f6754f0f
Merge pull request #338 from wb9688/parse-duration-string-test
...
Add tests for parseDurationString()
2020-06-15 11:01:21 +02:00
wb9688
e02d3c37b2
Merge pull request #354 from Stypox/peertube-age-restricted
...
[PeerTube] Change age restricted video in tests
2020-06-14 20:09:36 +02:00
Stypox
5808439c3f
[PeerTube] Change age restricted video in tests
...
The old one wasn't available anymore
2020-06-14 19:57:39 +02:00
wb9688
3239aa84f2
Handle isDeleted for PeerTube comments
2020-06-14 19:27:20 +02:00
wb9688
a65e46e4b1
Use OkHttp for tests like in NewPipe
2020-06-11 15:06:08 +02:00
Tobias Groza
5f523254cb
Merge pull request #344 from B0pol/peertube_embeds
...
For #342 : support PeerTube embeds
2020-06-06 18:53:07 +02:00
Fynn Godau
96de834b67
Bandcamp: parse date from radio info item
2020-06-04 19:36:58 +02:00
Fynn Godau
39b55b5b42
Bandcamp radio stream extractor: test uploader url
2020-06-04 19:32:19 +02:00
Fynn Godau
9a555d97e5
Remove useless bandcamp tests
2020-06-04 19:31:01 +02:00
Fynn Godau
e13f341a45
Bandcamp stream extractor: test service id
2020-06-04 19:29:37 +02:00
Fynn Godau
4dd9540782
Bandcamp license switch order
2020-06-04 19:27:39 +02:00
Fynn Godau
52103ac61f
Remove "useless" comments
2020-06-04 19:27:13 +02:00
Fynn Godau
8fa81537c0
Better bandcamp stream extractor description test
2020-06-04 19:17:51 +02:00
Fynn Godau
0950a95577
Fix for testing wrong object for null or empty
2020-06-04 19:16:08 +02:00
Fynn Godau
e6ecd91929
Bandcamp: Test full date
2020-06-04 19:12:10 +02:00
Fynn Godau
27e7aad159
Bandcamp: Improve code style
2020-06-04 19:04:25 +02:00
fynngodau
ea49202f64
Don't display internal license id
...
Co-authored-by: bopol <bopol@e.email>
2020-06-04 18:26:58 +02:00
bopol
063f4be766
For #342 : support PeerTube embeds
2020-06-04 14:26:42 +02:00
Fynn Godau
34b6928124
Don't print stacktrace before throwing new exception
2020-06-04 14:25:38 +02:00
Fynn Godau
1be20ceeec
Bandcamp: parse date
2020-06-04 14:09:21 +02:00
Fynn Godau
6822fe3dc8
Bandcamp: Get Stream upload date
2020-06-04 13:03:16 +02:00
Fynn Godau
e98bff8ae9
Improved Bandcamp radio stream extractor tests
2020-06-04 12:30:53 +02:00
Fynn Godau
8f6c00f8d8
Use default test in BandcampSearchExtractorTest
2020-06-04 12:12:20 +02:00
Fynn Godau
de776561b0
Bandcamp: fix loading uploader from streams in search
2020-06-04 12:02:18 +02:00
Fynn Godau
3940138fc5
Fix Bandcamp capitalization in one more spot
2020-06-03 22:41:27 +02:00
Fynn Godau
9fa9d920a9
Bandcamp: Implement new methods required due to interface changes
2020-06-03 21:35:11 +02:00
Fynn Godau
dd955c7f0c
Merge branch 'origin/dev' into dev
2020-06-03 21:13:40 +02:00
wb9688
33b1121fc7
Add tests for parseDurationString()
2020-05-30 17:20:54 +02:00
TobiGr
3aa6b93e91
Update version to 0.19.5
2020-05-30 10:45:39 +02:00
Tobias Groza
bda83fe6a5
Merge pull request #336 from wb9688/fix-time-parsing
...
Fix YouTube video duration parsing
2020-05-30 10:36:46 +02:00
wb9688
e39147202c
Fix YouTube video duration parsing
2020-05-30 09:31:08 +02:00
wb9688
98055a3c3c
Revert "Add more decryption function regexes"
...
This reverts commit 384d6acd5d
.
2020-05-29 18:41:40 +02:00
Tobias Groza
bdb0f2ae6b
Merge pull request #332 from wb9688/learning-playlist
...
Support YouTube's learning playlists
2020-05-29 10:11:08 +02:00
Tobias Groza
cf18cdb2f5
Merge pull request #329 from wb9688/decryption-regexes
...
Add more decryption function regexes
2020-05-29 10:10:50 +02:00
wb9688
ab77961b0a
Extract trailer thumbnail in YT learning playlists
2020-05-29 09:55:24 +02:00
wb9688
4d683e7655
Support YouTube's learning playlists
2020-05-29 09:17:03 +02:00
Tobias Groza
0b4977bb0c
Merge pull request #333 from wb9688/playerresponse-dashmanifesturl
...
Extract dashManifestUrl in playerResponse
2020-05-28 21:17:07 +02:00
Tobias Groza
0f8057ae56
Merge pull request #322 from wb9688/upgrade-jsoup
...
Upgrade dependencies
2020-05-28 21:16:33 +02:00
wb9688
384d6acd5d
Add more decryption function regexes
2020-05-28 11:40:38 +02:00
wb9688
8f71bde776
Fix bug for YT videos longer than 1000h
2020-05-28 11:34:42 +02:00
wb9688
aa277e1b6a
Upgrade other dependencies
2020-05-28 11:34:42 +02:00
wb9688
036cc7e26e
Upgrade jsoup
2020-05-28 11:34:41 +02:00
Fynn Godau
89b0639faa
New Bandcamp tests for channel extractor
2020-05-25 20:22:23 +02:00
Fynn Godau
e08256ebc6
Remove "url cleanup" in Bandcamp link handlers
2020-05-25 19:17:27 +02:00
Fynn Godau
692b2d06f4
Bandcamp: read info item data in place and not in advance
2020-05-25 18:53:15 +02:00
Fynn Godau
fff21caa87
Fix Bandcamp no avatar test
...
I own the bandcamp artist account called "NewPipeExtractorTest", so it
shouldn't break again.
2020-05-25 18:53:10 +02:00
wb9688
7ef3e80493
Extract dashManifestUrl in playerResponse
2020-05-15 12:07:26 +02:00
Stypox
60e6df7e6b
Merge pull request #330 from B0pol/gitignore
...
add vscode / eclipse files to .gitignore
2020-05-13 10:39:33 +02:00
wb9688
efafbbb88a
Merge pull request #331 from wb9688/playlist-subchannel
...
Add sub-channel stuff to PlaylistInfo
2020-05-12 10:49:13 +02:00
wb9688
baf5dd3e8f
Merge pull request #311 from B0pol/fixSearchSuggestion
...
Search: add isCorrectedSearch() and fix YoutubeSearchExtractor#getSea…
2020-05-12 09:29:15 +02:00
wb9688
102f04e297
Add sub-channel stuff to PlaylistInfo
2020-05-12 09:14:29 +02:00
bopol
cbf1163b80
ignore workspace and folder configs
2020-05-11 21:23:50 +02:00
Tobias Groza
b40ccb5075
Merge pull request #307 from B0pol/playlists_peertube
...
[PeerTube] playlist support
2020-05-11 20:53:03 +02:00
bopol
aeb2a6ff6f
add vscode / eclipse files to .gitignore
2020-05-11 13:06:02 +02:00
B0pol
3a858e6640
Merge pull request #315 from B0pol/isnullorempty
...
refactor: add Utils.isNullOrEmpty()
2020-05-11 11:04:56 +00:00
bopol
adaf196c99
remove unused import
2020-05-11 12:54:33 +02:00
bopol
de26e00079
changed all == null || isEmpty() to isNullOrEmpty()
2020-05-11 11:40:24 +02:00
bopol
202a73516c
refactor: add Utils.isNullOrEmpty()
2020-05-11 11:26:18 +02:00
Tobias Groza
3cae32b6db
Merge pull request #328 from wb9688/fix-327
...
Fix #327
2020-05-09 20:29:11 +02:00
wb9688
1de0876fb4
Fix #327
2020-05-09 20:06:50 +02:00
bopol
7cae95bf09
add tests for subchannel stuff in playlists
2020-05-09 10:15:35 +02:00
bopol
1a6e92ebf6
add getSubChannel Name, Url and AvatarUrl for playlists
2020-05-09 09:52:24 +02:00
bopol
cf99107745
address yausername change
2020-05-09 00:52:04 +02:00
bopol
5bab9d9fc0
[PeerTube] playlist support & refactoring
2020-05-09 00:52:04 +02:00
Tobias Groza
f3913e241e
Merge pull request #324 from Stypox/music-detector-url
...
[YouTube] Support stream urls in `vnd.youtube://ID` form
2020-05-08 17:43:04 +02:00
Tobias Groza
3ddd6e4d7b
Merge pull request #326 from wb9688/subchannel-parentchannel
...
Rename SubChannel in channels back to ParentChannel
2020-05-08 17:40:49 +02:00
wb9688
7320108c66
Rename SubChannel in channels back to ParentChannel
2020-05-08 16:02:59 +02:00
wb9688
44d382b4bd
Merge pull request #313 from Royosef/DisplayParentChannelDetails
...
Display parent channel details
2020-05-08 15:52:32 +02:00
Roy Yosef
a408661771
Fix PR review
...
Rename "ParentChannel" to "SubChannel"
Add Tests for sub-channel info fetching
2020-05-07 20:20:12 +02:00
Roy Yosef
1de1f97cf1
Add parent channel info to StreamExtractor: name, url & avatar url
2020-05-07 20:20:12 +02:00
Roy Yosef
4234740baa
Add parent channel info to ChannelExtractor: name, url & avatar url
2020-05-07 20:20:12 +02:00
bopol
6cff5de400
renaming methods for consistency
2020-05-07 15:40:41 +02:00
bopol
5760366d70
use JsonObject#isEmpty instead of JsonObject#equals(JsonUtils.EMPTY_OBJECT)
2020-05-07 15:40:41 +02:00
bopol
93e868634e
clarified javadoc for isCorrectedSearch and getSearchSuggestions in SearchExtractor
2020-05-07 15:40:41 +02:00
bopol
3757541a9c
adapt new nanojson version
2020-05-07 15:40:41 +02:00
bopol
79483a6dc0
«Use this in SearchInfo, for consistency's sake» @Stypox order xD
2020-05-07 15:40:41 +02:00
bopol
e8d58e09c9
getSearchSuggestion now returns the corrected query if isCorrectedSearch() is true
2020-05-07 15:40:41 +02:00
bopol
5d1c3b3fa2
Use setters in SearchInfo
2020-05-07 15:39:57 +02:00
bopol
29b639b454
Search: add isCorrectedSearch() and fix YoutubeSearchExtractor#getSearchSuggestion()
...
isCorrectedSearch: see the javadoc
getSearchSuggestion: in YoutubeSearchExtractor, it was giving the corrected search query. It now gives the suggested query, as it should
2020-05-07 15:39:57 +02:00
wb9688
665c69b530
Merge pull request #310 from B0pol/soundcloudComments
...
Add support for soundcloud comments
2020-05-07 15:35:57 +02:00
Stypox
a02bfbeae5
[YouTube] Support stream urls in `vnd.youtube://ID` form
2020-05-07 15:31:31 +02:00
bopol
bc13e0c616
Merge remote-tracking branch 'upstream/dev' into soundcloudComments
2020-05-05 14:49:21 +02:00
TobiGr
4086715a68
Merge branch 'dev'
2020-05-03 21:21:30 +02:00
TobiGr
8a79b14077
Update extractor version to 0.19.4
2020-05-03 15:11:20 +02:00
Tobias Groza
fc3a69ed54
Merge pull request #323 from wb9688/fix-yt-music-albums
...
Fix NPE in defaultAlertsCheck()
2020-05-03 14:08:47 +02:00
wb9688
6aae99f832
Fix NPE in defaultAlertsCheck()
2020-05-03 10:28:45 +02:00
Tobias Groza
2780e716b3
Merge pull request #321 from wb9688/fix-nanojson
...
Return null instead of "" in getTextFromObject()
2020-05-02 15:56:28 +02:00
wb9688
a1eabc7589
Return null instead of "" in getTextFromObject()
2020-05-02 08:02:25 +02:00
Fynn Godau
e7b046db9d
Merge remote-tracking branch 'origin/dev' into dev
2020-04-26 23:12:43 +02:00
Fynn Godau
a3e8e1c70e
Bandcamp: move own methods to helper class
2020-04-26 23:05:56 +02:00
Fynn Godau
9bc7a470ab
Replace fori with foreach loop
2020-04-26 23:02:14 +02:00
Tobias Groza
49157fcf0d
Merge pull request #317 from wb9688/tnp-nanojson
...
Use our fork of nanojson
2020-04-24 21:54:35 +02:00
bopol
2564bcf399
refactor comments
...
rename methods in CommentsInfoItemExtractor interface to match the other method names across NewPipeExtractor
remove getName in (Youtube|SoundCloud|Peertube)CommentsExtractor and move it up in CommentsExtractor, return "Comments" instead
2020-04-21 20:05:54 +02:00
Fynn Godau
f71bd29d58
Bandcamp playlist extractor: add more tests
2020-04-20 23:52:12 +02:00
Fynn Godau
5009d9f53b
Bandcamp tests: remove certain tests
2020-04-20 23:25:00 +02:00
Fynn Godau
b5e251c82f
Bandcamp channel extractor: package-private access for getImageUrl(…)
2020-04-20 23:12:43 +02:00
Fynn Godau
8c70dab8c5
Bandcamp search query handler factory: inline variable
2020-04-20 23:11:16 +02:00
Fynn Godau
10ae3db118
Bandcamp javadoc: Replace br tags with p tags
2020-04-20 23:03:12 +02:00
Fynn Godau
9201e0d3ea
Don't declare exceptions that can't actually be thrown
2020-04-20 22:46:54 +02:00
Fynn Godau
965bce00cf
Bandcamp stream extractor: return NO_AGE_LIMIT
2020-04-20 22:44:45 +02:00
Fynn Godau
82099592c7
Bandcamp: Move code from SearchExtractor to InfoItemExtractors
2020-04-20 22:44:35 +02:00
Fynn Godau
00c0333059
Bandcamp: Fetch channelInfo in onFetchPage
2020-04-20 22:41:44 +02:00
Fynn Godau
433d72c26a
Depend on nanojson fork in timeago-parser
2020-04-20 22:04:10 +02:00
Fynn Godau
67de0285e1
Bandcamp: Generate query JSON in a proper way
2020-04-20 22:03:29 +02:00
Fynn Godau
c133190c53
Bandcamp capitalization
2020-04-20 21:55:35 +02:00
Fynn Godau
b100b9873f
Bandcmap: Use nanojson fork instead of org.json
2020-04-20 21:53:04 +02:00
wb9688
b51699a20e
Implement @TobiGr's suggestions
2020-04-20 14:27:33 +02:00
wb9688
74f0ab1792
Merge pull request #301 from wb9688/media-ccc-non-api-urls
...
Use media.ccc.de instead of api.media.ccc.de
2020-04-19 17:06:57 +02:00
Fynn Godau
9c239371f4
Change nanojson dependency to @wb9688's fork
2020-04-19 14:29:39 +02:00
Fynn Godau
5c0a03328a
Merge 'origin/dev' into dev
2020-04-19 14:26:51 +02:00
bopol
0cbbc2a1f9
remove unused import
2020-04-19 14:16:38 +02:00
wb9688
979c5a7502
Use our fork of nanojson
2020-04-17 19:36:42 +02:00
bopol
d4352f9b84
support comments for SoundCloud
2020-04-10 20:41:41 +02:00
wb9688
1b522304c1
Merge pull request #308 from Royosef/PeerTubeChannelExtractor
...
[PeerTube] channels support
2020-04-10 19:07:51 +02:00
Roy Yosef
4afe657f6f
Add tests for getUrlFromId and test for video-channels in acceptUrlTest
2020-04-10 19:05:38 +03:00
Roy Yosef
2c9f1260eb
Fix wb9688 review comments
...
* Rename PeertubeUserExtractor to PeertubeAccountExtractor
* Add test for video-channels in PeertubeChannelLinkHandlerFactoryTest
* Compatibility support for older versions (use "accounts/" as default)
2020-04-10 16:40:53 +03:00
bopol
636c430743
refactor: create extractors and linkHandler packages for SoundCloud, move YoutubeParsingHelper to youtube package (not linkhandler anymore)
2020-04-10 10:51:05 +02:00
Roy Yosef
b6e6f403a8
add support for PeerTube channels extraction
2020-04-09 20:37:49 +03:00
wb9688
a5155fb562
Merge pull request #300 from B0pol/hls_opus_not_supported
...
throw ContentNotSupportedException when content is known to be unsuppo…
2020-04-08 19:54:16 +02:00
bopol
a20d53dd98
remove the mention of Opus on wb9688 advise when
...
ContentNotSupportedException is thrown
2020-04-08 15:31:11 +02:00
bopol
9a7c6b7ab0
use ContentNotSupportedException for channels without tabs
2020-04-07 14:33:43 +02:00
bopol
a22104cbda
add tests for ContentNotSupportedException
2020-04-07 14:33:43 +02:00
bopol
093762e793
throw ContentNotSupportedException when content is know to be unsupported
2020-04-07 14:33:43 +02:00
Tobias Groza
094b87c537
Merge pull request #299 from B0pol/sc_countries
...
add supported countries for soundcloud
2020-04-03 19:12:33 +02:00
Tobias Groza
a7d3d14910
Add link to JDoc to readme
2020-04-03 18:14:14 +02:00
Tobias Groza
e3263666bd
Merge pull request #303 from opusforlife2/correct-template
...
Corrected the pull request template
2020-04-03 17:56:02 +02:00
Tobias Groza
b81e22ddc9
Merge pull request #291 from wb9688/yt-music-search
...
Add support for YouTube Music search
2020-04-03 17:55:34 +02:00
wb9688
bce27a0e22
Rename getValidResponseBody() to getValidJsonResponseBody()
2020-04-03 17:23:18 +02:00
opusforlife2
4a1d8c804c
Corrected the pull request template
...
It was a bit weird before.
2020-04-02 10:42:54 +00:00
bopol
1eb3deb7fd
add supported countries for soundcloud
2020-04-02 11:26:19 +02:00
Tobias Groza
cef7c96d7d
Merge pull request #302 from wb9688/readme-local-build
...
Add another way to test changes locally to README.md
2020-04-01 19:23:04 +02:00
wb9688
8a9e137385
Extract some code to getValidResponseBody()
2020-04-01 16:01:21 +02:00
wb9688
2d0c301440
Add another way to test changes locally to README.md
2020-04-01 15:43:11 +02:00
wb9688
bf24caa5f2
Use media.ccc.de instead of api.media.ccc.de
2020-04-01 13:25:10 +02:00
wb9688
2af610e3e7
Add tests for other YT Music search types
2020-04-01 10:30:51 +02:00
wb9688
c7f7bd2442
Fix error when YT Music videos has no uploader URL
2020-04-01 10:30:51 +02:00
wb9688
ac15df4548
Move YouTube Music search to its own class
2020-04-01 10:30:51 +02:00
wb9688
cf0f2aff3e
Extract uploader url from certain YouTube Music videos
2020-04-01 10:30:51 +02:00
wb9688
aa8cea47f3
Refactor YouTube Music search tests
2020-04-01 10:30:51 +02:00
wb9688
dd434cca01
Fix issue when there is no didYouMeanRenderer in itemSectionRenderer
2020-04-01 10:30:51 +02:00
wb9688
5a775a4bbe
Use new way of specifying stream count
2020-04-01 10:30:51 +02:00
wb9688
d58c0f230d
Improve code for YouTube Music search
2020-04-01 10:30:51 +02:00
wb9688
dc29d87962
Extract YouTube search suggestions
2020-04-01 10:30:51 +02:00
wb9688
c852b13d5a
Add Referer header so that it also works with HttpsUrlConnection
2020-04-01 10:30:51 +02:00
wb9688
eb48524411
Add tests for YouTube Music search
2020-04-01 10:30:51 +02:00
wb9688
2b9b2a78e8
Handle 100+ items in playlist
2020-04-01 10:30:51 +02:00
wb9688
fb9b9691b7
Improve getYoutubeMusicKeys()
2020-04-01 10:30:51 +02:00
wb9688
4ddbdf0aee
Disable artist search for now
2020-04-01 10:30:51 +02:00
wb9688
1762a527c9
Add support for YouTube Music search
2020-04-01 10:30:51 +02:00
Tobias Groza
04af575764
Hey Travis, deploy the docs
2020-03-30 00:10:54 +02:00
TobiGr
c8abc1abd5
v0.19.0
2020-03-29 23:47:26 +02:00
Tobias Groza
69e0624e35
Merge pull request #298 from B0pol/feed_peertube
...
[peertube] add feed url in channels
2020-03-28 12:53:47 +01:00
bopol
0ae53c1e50
[peertube] adapt test for channel feed url
2020-03-28 11:48:24 +01:00
bopol
bc43f8977d
[peertube] add feed url in channels
2020-03-28 10:22:47 +01:00
TobiGr
647e7cd450
Accept YouTube Music playlists but not YouTube Mixes
2020-03-25 22:24:18 +01:00
Tobias Groza
bff595c6c6
Merge pull request #276 from wb9688/no-yt-mix
...
Don't accept YouTube Mix playlists
2020-03-25 11:18:07 +01:00
wb9688
bcd2a1092b
Don't accept YouTube Mix playlists
2020-03-25 11:07:28 +01:00
Tobias Groza
e4874d554e
Merge pull request #296 from mauriciocolli/fix-search-errors-detection
...
Fix search errors detection and refactor search tests
2020-03-24 21:53:32 +01:00
Tobias Groza
8a9ae32e6b
Merge pull request #292 from wb9688/fix-yt-no-views
...
Fix YouTube videos with no views
2020-03-24 20:27:08 +01:00
Mauricio Colli
0a20c53f1a
Use easier-to-read asserts statements in tests
2020-03-23 18:08:59 -03:00
Mauricio Colli
c921e5d6a5
Remove unnecessary collector creation function from search extractor
2020-03-23 18:08:51 -03:00
Tobias Groza
6fd9b38ad9
Merge pull request #287 from mauriciocolli/fix-channel-redirect
...
[YouTube] Fix channel with redirects directly in the response
2020-03-23 18:41:56 +01:00
Mauricio Colli
04bfa0ec6a
[PeerTube] Update video used in age limit test (was 404 not found)
2020-03-21 15:55:40 -03:00
Mauricio Colli
921bf30bb7
[YouTube] Add check for channel items with no video count in search
2020-03-21 15:55:39 -03:00
Mauricio Colli
b7f8001a49
[YouTube] Add check for channel items without description in search
2020-03-21 15:55:38 -03:00
Mauricio Colli
9b7999fe54
[YouTube] Check if channel item has subscription count in search
2020-03-21 15:55:37 -03:00
Mauricio Colli
9704fc9952
Improve search extractor tests for services
2020-03-21 15:55:36 -03:00
Mauricio Colli
d72130edae
Always return a new item collector in the search extractor
2020-03-21 15:55:34 -03:00
Mauricio Colli
0c27198ba1
Include errors from child collectors in the search collector
2020-03-21 15:55:29 -03:00
TobiGr
2cf8cbf0df
Merge branch 'master' into dev
2020-03-18 22:29:10 +01:00
Tobias Groza
6f03c6e877
Merge pull request #293 from Stypox/fix-soundcloud
...
[SoundCloud] Fix by migrating to `api-v2` and other improvements
2020-03-18 17:49:54 +01:00
wb9688
222d659d9e
[SoundCloud] Don't make separate request for getAudioStreams() in StreamExtractor
...
Signed-off-by: Stypox <stypox@pm.me>
2020-03-18 11:30:15 +01:00
wb9688
904c9d681f
Fix YouTube videos with no views
2020-03-18 10:07:49 +01:00
Stypox
265cfb61f3
[SoundCloud] Add test for small playlist, use defaultTestGetPageInNewExtractor
2020-03-17 21:03:50 +01:00
Stypox
45bb646480
[SoundCloud] Do not overwrite nextPageUrl in PlaylistExtractor
...
Consistent with YouTube and the documentation
2020-03-17 20:56:47 +01:00
Fynn Godau
a1523eb293
Update Bandcamp service to latest interface changes
2020-03-17 20:40:25 +01:00
Stypox
c505d4e2b7
[SoundCloud] Remove trailing , in playlist page urls
2020-03-17 20:36:59 +01:00
Stypox
ae47c9587c
[SoundCloud] Optimize imports in edited files
2020-03-17 20:32:04 +01:00
Stypox
4389fd3b7b
[SoundCloud] Migrate StreamExtractor to api-v2
2020-03-17 18:06:13 +01:00
Stypox
5e4ddb368f
[SoundCloud] Fix extractors built from next playlist pages
...
They didn't have the information to calculate another next page url. So now `nextPageUrl` contains a full link with all video ids, and `getPage` takes the first part of the url (containing 15 streams) and produces another `nextPageUrl` with the remaining streams.
Also add a test for this.
2020-03-17 18:04:40 +01:00
fynngodau
b78f788017
Merge branch 'dev' into dev
2020-03-17 18:03:14 +01:00
Stypox
0e1b4bbf17
[SoundCloud] Test playlists banner: it should not exist
2020-03-17 15:53:25 +01:00
Stypox
65bdb3bc9d
[SoundCloud] Unignore ignored playlist tests
2020-03-17 15:49:58 +01:00
Stypox
1558da6f6b
[SoundCloud] Fix playlist next page generation
2020-03-17 15:46:35 +01:00
Stypox
ca8bf53b61
[SoundCloud] Fix playlist test: number of streams changed
2020-03-17 15:35:33 +01:00
Stypox
d4aa4a0763
[SoundCloud] Fix typo in ChartsExtractor
2020-03-17 15:25:11 +01:00
Stypox
4b1121aac7
[SoundCloud] Add tests for api-v2 channel urls
2020-03-17 15:23:13 +01:00
Stypox
c3d811fde5
[SoundCloud] Use api-v2 in SubscriptionExtractor
...
Also added
2020-03-17 15:22:25 +01:00
Stypox
d0e66cc600
[SoundCloud] Improve thumbnail url extraction in playlists
...
Prevent NullPointerExceptions and remove duplicate code
2020-03-17 15:13:28 +01:00
Stypox
f3095713f9
[SoundCloud] Use api-v2 in PlaylistExtractor
...
Rewrote methods to calculate next page url and to get items from it. `api-v2` is different from `api` since the initial playlist page contains (usually) the full info of the first 3 streams and only the id of the other. Then the single tracks can be requested in batch using `/tracks?ids=id1,id2,...`.
2020-03-17 15:12:13 +01:00
TobiGr
9eca7df947
Forget to fetch the page
2020-03-16 20:50:08 +01:00
TobiGr
edb57840d7
[SoundCloud] Update hard-coded client_id
2020-03-16 19:43:36 +01:00
TobiGr
efad0be99f
Test if the extractor can get AudiStreams with the hard-coded cliend_id
...
SoundCloud client_id are changed on a regular basis now. We might want to remove the hardcoded id completely on a later point of time when client_id is changed to often
2020-03-16 19:42:35 +01:00
TobiGr
dc20f2aa0c
version 0.18.7
2020-03-16 17:27:26 +01:00
TobiGr
5f39bc4e99
[SoundCloud] Update hardcoded client id
2020-03-16 17:26:30 +01:00
Tobias Groza
65a7eda446
Merge pull request #256 from B0pol/mediaccc
...
Support for non-api mediaccc links, invidious shortened links, refactor mediaccc tests
2020-03-15 17:04:09 +01:00
TobiGr
40b3bc138f
Remove hack to fix YoutubeStreamLinkHandlerFactoryTest of vnd.youtube videos
2020-03-14 21:54:54 +01:00
Tobias Groza
1c954fbded
Merge pull request #277 from XiangRongLin/vnd
...
Handle youtube stream urls in "vnd.youtube:videoId" format.
2020-03-14 21:52:49 +01:00
Tobias Groza
9edd0f2675
Merge pull request #288 from Stypox/fix-unlisted-video
...
[YouTube] Fix videos without next stream throwing error and add test for unlisted videos
2020-03-14 21:36:13 +01:00
Stypox
957b78ed9b
[YouTube] Add test for an unlisted video, which has no next stream
2020-03-14 14:04:01 +01:00
Stypox
325a51f73a
[YouTube] Do not crash if there is no next stream, but return null
...
Like every other extractor does if there is no next stream
2020-03-14 14:02:48 +01:00
TobiGr
aea0962cfe
[SoundCloud] Update hardcoded client id
2020-03-14 12:41:44 +01:00
Mauricio Colli
b086e9db3f
[YouTube] Fix id extraction for some channels
...
Some channels had no reliable way to get the redirected id in the response,
so saving it for later was a valid alternative.
2020-03-14 02:39:49 -03:00
Mauricio Colli
00d1ed439b
[YouTube] Fix channel extraction when redirects are in the response
...
Some redirects were embed directly into the response as instructions
for the page, instead of the usual http redirects.
2020-03-14 02:39:48 -03:00
bopol
66518ec444
check wether mediaccc streams url are secure
2020-03-13 20:07:36 +01:00
bopol
0cd5e05b7b
MediaCCCLH: use substring instead of replace; improve a bit tests, return Collections.emptyList(); instead of null where it's annotated @NonNull
2020-03-13 20:00:49 +01:00
bopol
f742a6bd3e
code optimization
2020-03-13 20:00:49 +01:00
bopol
9701c7d800
invidious shortened links
2020-03-13 20:00:49 +01:00
bopol
e8e535b815
mediaccc: update linkhandlers & refactor tests
2020-03-13 20:00:49 +01:00
Tobias Groza
e7be952fbf
Merge pull request #281 from mauriciocolli/add-check-live
...
[YouTube] Add additional check for live videos
2020-03-07 22:07:34 +01:00
Mauricio Colli
70abd57852
[YouTube] Cache commonly used stream type result
2020-03-07 16:48:32 -03:00
Mauricio Colli
1ef706f567
[YouTube] Detect LIVE videos in the trending page
2020-03-07 16:48:31 -03:00
Xiang Rong Lin
cc8fb486ee
Handle youtube stream urls in "vnd.youtube:videoId" format.
...
The official YouTube app accept intents in this format, which causes other developers to use it.
2020-03-02 20:40:33 +01:00
Tobias Groza
6446abc6d1
Merge pull request #271 from mauriciocolli/improve-status-checking
...
[YouTube] Improve response status checks
2020-03-01 15:30:53 +01:00
Mauricio Colli
6aa4d59b91
[PeerTube] Update video used in age limit test (the previous was 404)
2020-03-01 10:29:30 -03:00
Mauricio Colli
cfc278317d
[YouTube] Fail-fast if status exist and is anything other than "OK"
2020-03-01 10:29:29 -03:00
Tobias Groza
65f0ec6057
Merge pull request #265 from mauriciocolli/improve-tests
...
Test if services recognizes their own items urls
2020-03-01 12:59:12 +01:00
Mauricio Colli
90ae5fbea2
Improve kiosk tests across services
2020-03-01 12:57:18 +01:00
Mauricio Colli
e590417cc4
Test if services recognizes their own items urls
2020-03-01 12:57:18 +01:00
TobiGr
35252235b0
Go through all badges when checking if a StreamInfoItem is a live stream
2020-03-01 12:52:37 +01:00
Tobias Groza
830b7d3dbd
Merge pull request #267 from mauriciocolli/hotfixes
...
Hotfixes for YouTube and improve unavailable cases
2020-03-01 12:50:10 +01:00
Mauricio Colli
5686a6f562
[YouTube] Detect when a stream is deleted or doesn't exist
...
Added a test case as well.
2020-03-01 12:49:13 +01:00
Mauricio Colli
e65333c3ce
[YouTube] Detect deleted/nonexistent/invalid channels and playlists
...
- Added tests for these cases.
2020-03-01 12:49:13 +01:00
Mauricio Colli
98e359438a
[YouTube] Detect simple 404s in the standard fetch method
2020-03-01 12:49:13 +01:00
Mauricio Colli
408f042127
[YouTube] Fix bug when url isn't present in the browseEndpoint object
2020-03-01 12:49:13 +01:00
Mauricio Colli
342bdbb852
[YouTube] Avoid crashing by letting exceptions bubble up
2020-03-01 12:49:13 +01:00
Mauricio Colli
e9644e6216
[YouTube] Handle video premiere's date and duration
2020-03-01 12:49:13 +01:00
Mauricio Colli
2a470ac4f7
[YouTube] Handle videos with no views or with "Recommended to you" text
2020-03-01 12:49:13 +01:00
Mauricio Colli
da2c562eb0
[YouTube] Take into account videos that have their views hidden
...
Also remove catch because it may hide bugs when the page changes.
2020-03-01 12:49:13 +01:00
Mauricio Colli
f66c20de54
Ignore null-keyed entries when iterating through the response headers
2020-03-01 12:49:13 +01:00
Mauricio Colli
5edd774fc4
Add latest url to the response to make detection of a redirect possible
...
Will be latest one in this commit because there's need to check the
history of redirects as of now.
2020-03-01 12:49:13 +01:00
Mauricio Colli
3441946bea
Make test downloader return a response instead of throwing an exception
...
The test implementation was throwing an exception instead of just
returning the response and letting the caller handle it.
2020-03-01 12:49:13 +01:00
Tobias Groza
fcbc96a86c
Merge pull request #269 from wb9688/charset
...
Fix for Android versions earlier than KitKat
2020-03-01 11:31:26 +01:00
wb9688
df219a9453
Don't use StandardCharsets class
2020-03-01 09:41:06 +01:00
TobiGr
705896aca0
Version 0.18.6
2020-02-29 21:39:45 +01:00
Tobias Groza
5a101fd17f
Merge pull request #262 from wb9688/pbj
...
Improve yt_new
2020-02-29 21:39:02 +01:00
wb9688
2807079531
Catch NumberFormatException instead of checking whether viewCount equals "Recommended for you"
2020-02-29 21:28:38 +01:00
wb9688
3e4d2fd64e
Improve testChannelUrl()
2020-02-29 18:20:35 +01:00
wb9688
bfe3eb1409
Use suggested try-if code style
2020-02-29 17:18:50 +01:00
wb9688
a6a63e9570
Let isHardcodedClientVersionValid() throw IOException
2020-02-29 17:01:36 +01:00
wb9688
45df8248bf
Use Localization in getJsonResponse()
2020-02-29 16:55:07 +01:00
wb9688
157055fbfd
Create getJsonResponse() function
2020-02-29 16:42:04 +01:00
wb9688
4f406512c2
Use YoutubeChannelLinkHandlerFactory
2020-02-29 16:26:28 +01:00
wb9688
4ff655352d
Reimplement getAgeLimit()
2020-02-28 17:14:26 +01:00
wb9688
6ba03e542b
Cache getVideoTab()
2020-02-28 17:05:31 +01:00
wb9688
8bfbae904d
Cache getVideo(Primary|Secondary)InfoRenderer()
2020-02-28 17:03:21 +01:00
wb9688
0b940f5cf4
Use "getAgeLimit() != NO_AGE_LIMIT" instead of "getAgeLimit() == 18"
2020-02-28 16:40:50 +01:00
wb9688
beb07a4852
Add test to check whether hardcoded client version is still valid
2020-02-28 16:36:14 +01:00
wb9688
b89b6b1db2
Check if hard-coded client version is valid in separate function
2020-02-28 16:35:24 +01:00
wb9688
7898ba230d
Use pbj in YoutubeStreamExtractor
2020-02-28 15:17:47 +01:00
wb9688
612ec06839
Actually fix thumbnail URLs starting with //
2020-02-28 09:36:33 +01:00
wb9688
f946310af5
Fix search queries with a promotedSparklesTextSearchRenderer returning 0 results
2020-02-28 08:57:44 +01:00
wb9688
94a2d5be46
Fix thumbnail URLs starting with //
2020-02-27 19:08:46 +01:00
wb9688
838d834ce2
Get Uploads view of Videos tab
2020-02-27 18:54:27 +01:00
wb9688
365b0329f3
Implement getTextFromObject() function
2020-02-27 17:39:23 +01:00
wb9688
0798bdd5cd
Fix getSubscriberCount() for 0 subscribers
2020-02-27 16:26:22 +01:00
wb9688
8ebd971648
Fix channel descriptions consisting of multiple parts
2020-02-27 10:06:35 +01:00
wb9688
880b951088
Fix getUploaderAvatarUrl() in YoutubeStreamExtractor
2020-02-27 09:53:59 +01:00
wb9688
8aa560bef4
Fix video titles consisting of multiple parts due to @
2020-02-27 09:42:22 +01:00
wb9688
951ed9f0d8
Don't call getUploaderInfo() in onFetchPage() for playlists
...
This way YouTube Music playlists will keep working in NewPipe
2020-02-27 09:27:56 +01:00
wb9688
2399d46ab0
Fix YoutubeChannelExtractor for channels without videos
2020-02-27 09:21:25 +01:00
wb9688
cdbc751b87
Fix getUploaderUrl() in YoutubeStreamInfoItemExtractor
2020-02-27 09:05:41 +01:00
wb9688
db305408e9
Get all expandedShelfContentsRenderers in YoutubeTrendingExtractor
2020-02-26 20:37:20 +01:00
wb9688
a2d8489e16
Parse timestamps in description
2020-02-26 20:27:30 +01:00
wb9688
8d503407ef
Parse browseEndpoint and watchEndpoint in descriptions
2020-02-26 19:37:58 +01:00
wb9688
8dc39d517a
Fix tests
...
Except testGetFullLinksInDescription() in DescriptionTestUnboxing, since that requires other changes first.
2020-02-26 17:31:01 +01:00
wb9688
88282761f4
Fix YoutubeStreamExtractor not being able to getAgeLimit() in onFetchPage()
2020-02-26 16:19:42 +01:00
wb9688
bb3a3d70bf
Use pbj in YoutubeTrendingExtractor
2020-02-26 16:10:24 +01:00
wb9688
1c3a4e710b
Check if there are no related streams in getAgeLimit()
2020-02-26 16:03:06 +01:00
wb9688
985c3ec877
Improve getClientVersion()
2020-02-26 15:27:50 +01:00
wb9688
15afbea3e1
Use pbj in YoutubePlaylistExtractor
2020-02-26 15:27:50 +01:00
wb9688
89a41a7f69
Use pbj in YoutubeChannelExtractor
2020-02-26 15:27:50 +01:00
wb9688
0973263aab
Use pbj in YoutubeSearchExtractor
2020-02-26 15:27:46 +01:00
TobiGr
fc465c8bf7
Fix getUploadDate() for premiered videos with a given date
...
Detect publishDate and uploadDate
2020-02-25 22:54:37 +01:00
TobiGr
d1b7461d0e
Version 0.18.5
2020-02-25 22:00:47 +01:00
Tobias Groza
8838e2d136
Merge pull request #261 from TeamNewPipe/yt_new
...
Update YouTube to material version
2020-02-25 21:57:18 +01:00
TobiGr
96285e0815
Fix getUploadDate() for premiered videos with a given date
2020-02-25 21:51:30 +01:00
wb9688
5a35300a5e
Fix parsing upload date for premiered streams
2020-02-25 21:51:30 +01:00
wb9688
729fd2eaee
Fix parsing description for navigationEndpoint without urlEndpoint
2020-02-25 21:51:30 +01:00
wb9688
26fb44595f
Fix parsing description
2020-02-25 21:51:30 +01:00
TobiGr
0fff03038e
Get description with correct links
2020-02-25 21:51:30 +01:00
wb9688
5d08c34322
Fix parsing upload date in YoutubeStreamExtractor
2020-02-25 21:51:30 +01:00
wb9688
bbe1a3cd62
Use getVideo(Primary|Secondary)InfoRenderer()
2020-02-25 21:51:30 +01:00
wb9688
8e6be880d7
Simplify getId() in YoutubeChannelExtractor
2020-02-25 21:51:30 +01:00
TobiGr
ecfc27a92c
Implement fallback methods for getDescription() and getTextualUploadDate() in YouTubeStreamExtractor
2020-02-25 21:51:30 +01:00
wb9688
eed29ea734
Return null for related streams at age restricted videos
2020-02-25 21:51:30 +01:00
TobiGr
9efcc61ca6
Adapt some tests to match the new URLs
2020-02-25 21:51:30 +01:00
TobiGr
f39603f6ef
Implement getUploadDate() in YouTubeStreamInfoItemExtractor
2020-02-25 21:51:30 +01:00
wb9688
26ea3dceb6
Fix getViewCount() in YoutubeStreamExtractor
2020-02-25 21:51:30 +01:00
wb9688
76d54abdbf
Reimplement more methods in YoutubeStreamExtractor
2020-02-25 21:51:30 +01:00
wb9688
f13c0288cc
Reimplement some methods in YoutubeStreamExtractor
2020-02-25 21:51:30 +01:00
wb9688
02b59903fa
Remove useless code
2020-02-25 21:51:30 +01:00
TobiGr
dd4dd849dc
Get client version dynamically in YouTubeSearchExtractor
2020-02-25 21:51:30 +01:00
TobiGr
583e9c1571
Fix getPage()
...
Someone should check if there are unstaged changes before committing...
2020-02-25 21:51:30 +01:00
TobiGr
5842b9ad37
Add getClientVersion() and HARDCODED_CLIENT_VERSION to YouTubeParsingHelper
...
Prefer hardcoded client version above the current one when making requests to retrieve the same JSON structure for each request.
2020-02-25 21:51:30 +01:00
wb9688
5d883d100c
Implement pagination in YoutubeSearchExtractor
2020-02-25 21:51:30 +01:00
wb9688
c0a8e01889
Implement pagination in YoutubePlaylistExtractor
2020-02-25 21:51:30 +01:00
wb9688
4039409820
Fix getLikeCount() for certain streams
2020-02-25 21:51:30 +01:00
wb9688
627ab2459f
Uncomment fetchPage()
2020-02-25 21:51:30 +01:00
wb9688
2dfa2187ff
Implement pagination in YoutubeChannelExtractor
2020-02-25 21:51:30 +01:00
wb9688
a38ab9b791
Return null if channel has no banner
2020-02-25 21:51:30 +01:00
wb9688
1eca63089b
Count private and deleted videos as ad
2020-02-25 21:51:30 +01:00
wb9688
c7360e4a46
Fix getThumbnailUrl() in YoutubeChannelInfoItemExtractor
2020-02-25 21:51:30 +01:00
wb9688
51fb26625a
Implement getSubscriberCount() in YoutubeChannelInfoItemExtractor
2020-02-25 21:51:30 +01:00
TobiGr
6d20b2b1d0
Fix NPE on collecting page items
2020-02-25 21:51:30 +01:00
wb9688
4462cbe3f1
Reimplement YoutubePlaylistInfoItemExtractor
2020-02-25 21:51:30 +01:00
wb9688
8aea4d445b
Reimplement YoutubeChannelInfoItemExtractor
2020-02-25 21:51:30 +01:00
wb9688
3187116a63
Handle premium videos
2020-02-25 21:51:30 +01:00
wb9688
7442f91aa6
Don't assume which channel tab is videos
2020-02-25 21:51:30 +01:00
wb9688
316fe0109d
Fix first playlist page
2020-02-25 21:51:30 +01:00
wb9688
4e57e589ce
Fix first channel page
2020-02-25 21:51:30 +01:00
TobiGr
5816202cc7
Move getInitialData() method to YouTubeParsingHelper
...
Rename ytInitialData to initialData
2020-02-25 21:51:30 +01:00
wb9688
38aabc6aca
Use plain YoutubeStreamInfoItemExtractor in YoutubeTrendingExtractor
2020-02-25 21:51:30 +01:00
wb9688
957db062ef
Fix getting uploader for videoRenderer
2020-02-25 21:51:30 +01:00
wb9688
b88188d419
Move stuff from extractVideoPreviewInfo() into YoutubeStreamInfoItemExtractor and partially fix search
2020-02-25 21:51:30 +01:00
wb9688
af49b3c487
Fix live streams in extractVideoPreviewInfo()
2020-02-25 21:51:30 +01:00
TobiGr
f33f9466ce
Fix testDescription() in ChannelExtractor
2020-02-25 21:51:30 +01:00
TobiGr
10e9c16d8c
Fix LiveStream view count
2020-02-25 21:51:30 +01:00
wb9688
127c4d5893
Fix getNextStream() in YoutubeStreamExtractor
2020-02-25 21:51:30 +01:00
wb9688
7bfc0e62c6
Fix getRelatedStreams() in YoutubeStreamExtractor
2020-02-25 21:51:30 +01:00
TobiGr
12166ca60c
[StreamExtractor] Add fallback method for uploader avatar url
2020-02-25 21:51:30 +01:00
wb9688
3ec422a542
Fix like/dislike parsing
2020-02-25 21:51:30 +01:00
TobiGr
a34e5c18e1
Trending StreamInfoItemCollecetor
2020-02-25 21:51:30 +01:00
TobiGr
b705515da0
Adapt most playlist fields
2020-02-25 21:51:30 +01:00
TobiGr
21253abd9b
Use Firefox ESR User-Agent: 68
2020-02-25 21:51:30 +01:00
TobiGr
655c999795
Detect disabled subriber count correctly
...
Fix parsing of kiosk name
2020-02-25 21:51:30 +01:00
TobiGr
7dcc9f159b
Stream: fix uploader avatar url
2020-02-25 21:51:30 +01:00
TobiGr
be3e20e263
Adapt most channel changes
2020-02-25 21:51:30 +01:00
TobiGr
39bf1ff1eb
Add ytInitialData regex
2020-02-25 21:51:30 +01:00
TobiGr
a63371a2c5
Version 0.18.4
2020-02-21 22:29:20 +01:00
TobiGr
468092894f
Merge branch 'dev'
2020-02-21 22:27:54 +01:00
Tobias Groza
4bc5b8d33f
Merge pull request #254 from TeamNewPipe/formatting
...
Improve code formatting and optimize imports
2020-02-17 08:38:01 +01:00
Tobias Groza
a129c65c63
more improvements of comments
...
Co-Authored-By: Stypox <stypox@pm.me>
2020-02-16 23:59:31 +01:00
TobiGr
0a4115286b
Apply one suggestion
2020-02-15 13:36:54 +01:00
Tobias Groza
6822574588
Fix typos
...
Co-Authored-By: B0pol <bopol@e.email>
2020-02-15 13:36:54 +01:00
TobiGr
030465b5d4
Improve code formatting and optimise imports
2020-02-15 13:36:54 +01:00
TobiGr
fc9f03178f
Merge branch 'master' into dev
2020-02-15 00:35:45 +01:00
TobiGr
33961b2092
Version 0.18.3
2020-02-10 21:12:39 +01:00
TobiGr
aae3476ee4
[SoundCloud] Update client_id
2020-02-10 21:11:09 +01:00
TobiGr
9112a106ea
[SoundCloud] Update client_id
2020-02-08 23:09:03 +01:00
TobiGr
008b6f505c
[PeerTube] Fix subscriber count in ChannelExtractorTest
2020-02-08 23:08:37 +01:00
Tobias Groza
3add704f5d
Merge pull request #239 from B0pol/peertube
...
[PeerTube] added metadata, fix descriptions, fix thumbnail, fix upload date, fix age limit, update tests.
2020-02-08 22:31:02 +01:00
TobiGr
0e33249c85
Fix SoundCloud description test
2020-02-08 22:20:20 +01:00
Fynn Godau
36a316eea5
Merge branch 'dev' into dev
2020-02-08 09:43:13 +01:00
bopol
11bcc78d9c
Description implements Serializable. fix NotSerializableException
2020-02-07 14:10:54 +01:00
bopol
0f8a7f9df2
fix testGetUploadDate for PeerTubeStreamExtractor
2020-02-07 13:33:50 +01:00
bopol
70a40e7388
Description: rm constructor by serviceId
2020-02-07 13:28:27 +01:00
bopol
26c65b2948
Create class Description
2020-02-06 23:35:46 +01:00
bopol
5756df8dc7
Use GMT as base time (actually fix upload date)
2020-02-06 22:33:48 +01:00
B0pol
b671a4ba18
Merge branch 'dev' into peertube
2020-02-01 18:18:23 +00:00
Tobias Groza
62b81c3607
Merge pull request #222 from mauriciocolli/feed-extractor
...
Introduce FeedExtractor to fetch from dedicated feeds when available
2020-02-01 18:41:26 +01:00
Mauricio Colli
26234a1c0f
Introduce FeedExtractor making fetching from dedicated feeds possible
...
YouTube, for example, has a dedicated feed which was built to be used
like this.
2020-01-28 02:44:18 -03:00
Mauricio Colli
be81f2945c
[YouTube] Use correct upload date flag in StreamExtractor
2020-01-28 02:44:17 -03:00
Mauricio Colli
5c0c35064c
Use clean url and id when creating a channel info
2020-01-28 02:44:16 -03:00
TobiGr
033a9fb8e3
Revert change which I accidentally committed while fixing the decryption
...
function in 7b72fd2a7d
2020-01-25 21:08:17 +01:00
B0pol
e392b6c68f
getLanguageInfo returns Locale instead of String
...
so that java can automatically translate with Locale.getDisplayLanguage(), instead of always having English name of the language
2020-01-25 14:12:38 +01:00
B0pol
341372c0d0
reindenting (ctrl alt l) on JsonUtils and PeertubeStreamExtractor
2020-01-24 20:16:24 +01:00
B0pol
b816e48216
Merge branch 'dev' into peertube
2020-01-24 20:10:47 +01:00
TobiGr
f8d117f6cd
Merge remote-tracking branch 'origin/master' into dev
2020-01-24 19:22:40 +01:00
Tobias Groza
ff61e28424
Merge pull request #243 from TeamNewPipe/fix/yt_decryption
...
Fix/yt decryption
2020-01-24 02:18:40 +01:00
TobiGr
7b72fd2a7d
[YouTube] Add new decryption function
2020-01-24 02:10:56 +01:00
TobiGr
bce87f3d01
Improve getDescriptionFuncName by removing catches and adding a loop
2020-01-24 02:10:38 +01:00
TobiGr
9b45c61103
Typos, comments and formatting
2020-01-24 02:10:18 +01:00
TobiGr
29a4cc78bf
Fix typo in decryption constant
2020-01-24 02:10:08 +01:00
B0pol
812c8e0ad2
authorName in comments now follow PeerTube website
...
https://peertube.cpy.re/videos/watch/d2a5ec78-5f85-4090-8ec5-dc1102e022ea
anonyme zirbeldrüse is his displayName, what was displayed in NewPipe.
but on the website, it's shown 777@mastodon.xyz
2020-01-24 00:22:05 +01:00
bopol
c790261eed
update test
2020-01-23 19:08:41 +01:00
bopol
a691d6d0fc
fix upload date: there was a one hour delay
...
on peertube.co.uk: https://i.imgur.com/8Pai1rb.png
on newpipe, before this commit: https://i.imgur.com/NIRbs4Z.png
on newpipe, with this commit: https://i.ibb.co/mhKKtRD/Screenshot-20200123-185422-New-Pipe-Debug.png
2020-01-23 19:02:40 +01:00
bopol
20da4750f8
empty support returns "", same for empty description
2020-01-23 14:37:14 +01:00
bopol
74439f692a
add extraction for support info and rename getLanguageInfo function
2020-01-23 14:19:22 +01:00
B0pol
1a15c0e750
agelimit now returns 18 if the video is marked as nsfw, 0 otherwise
...
+ created getBoolean method in JsonUtils.java
2020-01-23 04:42:54 +01:00
B0pol
bcfe7be4e6
Merge branch 'dev' into peertube
2020-01-22 19:19:41 +01:00
Tobias Groza
b9afc9807c
Merge pull request #233 from Stypox/yt-stream-ext
...
Youtube stream extractor improvements by using playerResponse json
2020-01-20 23:17:36 +01:00
Tobias Groza
82eff772fa
Merge pull request #234 from TeamNewPipe/fix/YT_search_test
...
Fix and improve tests
2020-01-20 23:16:27 +01:00
Stypox
1cfdc4547a
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java
...
Co-Authored-By: Tobias Groza <TobiGr@users.noreply.github.com>
2020-01-20 22:52:48 +01:00
Stypox
1e1100ef76
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java
...
Co-Authored-By: Tobias Groza <TobiGr@users.noreply.github.com>
2020-01-20 22:52:36 +01:00
TobiGr
2308b074f0
[MediaCCC] Fix testGetTextualUploadDate()
2020-01-20 22:12:39 +01:00
TobiGr
089dbe8afb
Fix StreamLinkHandlerFactoryTest by passing baseUrl to vdn.youtube urls
2020-01-20 22:10:18 +01:00
B0pol
f403490bd9
Refactoring
...
Co-Authored-By: Tobias Groza <TobiGr@users.noreply.github.com>
2020-01-20 21:28:26 +01:00
TobiGr
69ccb7e69d
Update subtitle tests comments
2020-01-20 21:25:47 +01:00
TobiGr
7943130307
Throw IllegalArgumentException when url is null in LinkHandlerFactory
2020-01-20 21:25:16 +01:00
TobiGr
39de55dcd3
[YouTube] Fix and move streamCountTest
2020-01-20 21:24:51 +01:00
TobiGr
51fd9ba491
Merge branch 'master' into dev
2020-01-20 20:31:32 +01:00
bopol
ad7f97ae83
fix PeerTube description and add more description tests
...
actually, the max description length is 250 after request with our extractor.
during my tests, I made API requests with Firefox, copy/pasted into echo "insert description" | wc, and it was giving a wrong length, maybe due to the escapers, I have no idea
anyway, it's now fixed
2020-01-20 15:14:23 +01:00
bopol
b382416372
changed the way to extract peertube description
...
if the description length is above 254, and ends with ..., it means the description we got from the first request is shortened.
why above 254: because in fact, shortened descriptions length are between 255 : https://framatube.org/videos/watch/24efbe1b-73c0-4d72-a3ff-77c8b32d3fcd
https://framatube.org/videos/watch/1ca99f40-fb5b-4fa4-abe5-4d54325df7fc
and 269: https://framatube.org/videos/watch/4d661d5f-a884-4510-bca8-15cb19aa3fe5
also fixed a typo in StreamExtractor.java
2020-01-20 14:36:12 +01:00
bopol
a281519e77
added metadata, fix descriptions, fix thumbnail, update tests
...
thumbnail: quality before: https://peertube.cpy.re/static/thumbnails/d2a5ec78-5f85-4090-8ec5-dc1102e022ea.jpg
quality after: https://peertube.cpy.re/static/previews/d2a5ec78-5f85-4090-8ec5-dc1102e022ea.jpg
description: we were getting about the first 260 characters, we now get full description (with fallback to first 260 chars if the get request for full description fails)
test: updated tests to match description, also changed some test: it was assertEquals(extracted, expected), but the proper way to do it is assertEquals(expected, extracted)
metadata: got host, privacy (public, private, unlisted), licence, language, tags
2020-01-20 12:22:33 +01:00
TobiGr
c1e1ac1f57
Merge branch 'dev'
2020-01-19 10:13:14 +01:00
Tobias Groza
2ee558fbe7
Merge pull request #237 from B0pol/fix_invidious_comments
...
Fix invidious comments
2020-01-17 22:55:51 +01:00
Tobias Groza
5a259fca94
Merge pull request #238 from B0pol/c_links_support
...
add support to /c/shortened_url channel links
2020-01-17 22:53:46 +01:00
Tobias Groza
57cf47b9f3
Merge pull request #236 from B0pol/fix_typos
...
fix: typos in Test
2020-01-17 22:52:29 +01:00
Fynn Godau
623e6a8c63
Merge branch 'dev' into dev
2020-01-17 22:37:42 +01:00
bopol
221e8dd171
changed assertTrue(string.equals(string)) to assertEquals(string, string)
2020-01-17 21:24:52 +01:00
bopol
5e81ed1dff
add support to /c/shortened_url channel links
2020-01-17 19:27:54 +01:00
B0pol
7e8d00981f
changed getId() to remove duplicate code
2020-01-17 13:44:55 +01:00
B0pol
02930d08f1
added comments test for invidious
2020-01-17 12:58:50 +01:00
B0pol
a12c0e2a37
fix:invidious: getID and onAccpetURl for comments
2020-01-17 12:09:59 +01:00
B0pol
3743c60695
fix: typos in Test
2020-01-15 14:27:52 +01:00
TobiGr
4aaf123f24
Add PeerTube to supported services in README
2020-01-14 15:36:25 +01:00
Fynn Godau
7b5702bdc2
Add bandcamp to list of supported services in README
2020-01-11 16:32:45 +01:00
Stypox
0567c20549
Merge branch 'dev' into yt-stream-ext
2020-01-07 00:10:06 +01:00
Stypox
7ba04836eb
Add tests for ratings in yt stream extractor tests
2020-01-06 23:57:08 +01:00
Stypox
1bb6cdee22
Enable commented-out test for yt stream extractor
2020-01-06 21:43:24 +01:00
Stypox
d5ca02f3f6
Fix test failing because music channel could be Vevo
...
In yt stream extractor test.
The Vevo and the normal music channel are equivalent, so Youtube picks one of them at random, and in playerResponse the channel id is Vevo's one.
2020-01-06 21:41:15 +01:00
Stypox
1689037b01
Fix getLength() tests now that the value is rounded (not floor-ed)
...
In yt stream extractor
2020-01-06 20:49:08 +01:00
Stypox
1ed89aad3e
Use more often playerResponse in yt stream extractor
...
This enhances performance and should make the extractor more reliable since it get info from a stable json structure that shouldn't be subject to many changes. Fallback html methods have been kept. In case of error the thrown exception contains the data about the playerResponse failure, that should be clearer than a NPE caused by not-found html tags.
2020-01-06 20:45:57 +01:00
Tobias Groza
bdbfa26835
Merge pull request #231 from pihug12/dev
...
[Soundcloud] Update client_id
2020-01-06 11:21:27 +01:00
Fynn Godau
ba967f1a15
Workaround enourmous load times for long bandcamp playlists
...
Additionally, get cover art from json instead of html
2020-01-03 13:17:04 +01:00
Fynn Godau
46e1f3922c
Refuse to load bandcamp playlists without content
2020-01-03 12:32:22 +01:00
Fynn Godau
f0d36dfa7e
Bandcamp radio: proper name, date shortened
2020-01-03 12:28:49 +01:00
Fynn Godau
808c8aa087
Add bandcamp weekly kiosk
...
Unfortunately a little messy
2020-01-03 12:28:49 +01:00
Fynn Godau
13ef11e9c8
Bandcamp featured kiosk
2020-01-03 12:28:49 +01:00
Fynn Godau
db02850d4f
Bandcamp playlist extractor: support playlists with just one stream
2020-01-03 12:28:49 +01:00
Fynn Godau
13e4908b83
Bandcamp search extractor: read track count from result page
2020-01-03 12:28:49 +01:00
Fynn Godau
9b16baffb7
Bandcamp search: multiple pages
2020-01-03 12:28:49 +01:00
Fynn Godau
ce2a88e56f
Add bandcamp search suggestion extractor
2020-01-03 12:28:49 +01:00
Fynn Godau
e12ddaef7f
Bandcamp playlist and search extractors: better and more tests
2020-01-03 12:28:49 +01:00
Fynn Godau
655df356a2
Various improvements concerning bandcamp playlists
2020-01-03 12:28:49 +01:00
Fynn Godau
ba700bfb3e
Add bandcamp playlists (albums)
2020-01-03 12:28:49 +01:00
Fynn Godau
d05b14ae48
Add channels (artists) to search results
2020-01-03 12:28:49 +01:00
Fynn Godau
794ca5eeae
Bandcamp channel extractor: handel nonexistent images better
2020-01-03 12:28:49 +01:00
Fynn Godau
91c0ec7cea
Bandcamp stream extractor: catch NullPointerException
2020-01-03 12:28:49 +01:00
Fynn Godau
d5cdc20be1
Bandcmap channel extractor: fix getting banner url
2020-01-03 12:28:49 +01:00
Fynn Godau
7730eb2ea1
Bandcamp channel extractor: enforce https when getting banner url
2020-01-03 12:28:49 +01:00
Fynn Godau
a42c77425d
Bandcamp channel extractor (ignoring everything but tracks)
2020-01-03 12:28:49 +01:00
Fynn Godau
5281456899
Bandcamp channel link handler factory
2020-01-03 12:28:49 +01:00
Fynn Godau
c3d127ccd9
Bandcamp audio stream test and fix
2020-01-03 12:28:49 +01:00
Fynn Godau
80d67e22b7
Return mp3-128 audio stream [experimental]
2020-01-03 12:28:49 +01:00
Fynn Godau
43dc3c3d4c
Return empty kiosk list instead of null
2020-01-03 12:28:49 +01:00
Fynn Godau
a579337c9a
Bandcamp service with support for streams and searches
2020-01-03 12:28:49 +01:00
pihug12
d4e0930f44
[Soundcloud] Update client_id
2019-12-31 23:36:23 +01:00
Tobias Groza
d83787a5ca
Merge pull request #226 from 7FM/dev
...
fix switching on null causing NPE
2019-12-31 00:26:27 +01:00
Tobias Groza
1c0a5db6da
Merge pull request #215 from gzsombor/fix-search-paging
...
Fix search result paging
2019-12-31 00:25:23 +01:00
TobiGr
0b6e37e71f
Improve YouTubeSearchPagingtest
2019-12-31 00:18:53 +01:00
Zsombor Gegesy
64729e5357
Improve the tests
2019-12-29 23:14:55 +01:00
Zsombor Gegesy
68b0fd9650
Add test for search paging
2019-12-29 23:14:55 +01:00
Zsombor Gegesy
2123b3abd3
Fix search result paging - due to the way as the InfoItemsSearchCollector are re-used,
...
the returned item list just grows, which cause that same videos are returned.
2019-12-29 23:14:55 +01:00
TobiGr
305ff21770
use openjdk8 for travis build
2019-12-26 23:38:40 +01:00
TobiGr
f826c24749
Use UTF-8 encoding for docs and bump version to v0.18.0
2019-12-26 22:46:19 +01:00
TobiGr
01eb7a9658
Fix syntax
2019-12-26 22:22:46 +01:00
TobiGr
147322b526
Deploy JDoc even when tests on master fail
2019-12-26 22:11:12 +01:00
Zsombor Gegesy
0a3422a591
Fix invalid jsoup query
2019-12-26 14:03:40 +01:00
7FM
fd5ed07d33
Merge branch 'dev' into dev
2019-12-22 12:33:26 +01:00
TobiGr
8e53fdad16
[SoundCloud] Fix getAudioStreams 401 response
...
Fix TeamNewPipe/NewPipe#2823
2019-12-22 00:42:07 +01:00
7FM
f57d9d43ce
fix switching on null causing NPE
2019-12-21 14:15:42 +01:00
Tobias Groza
8cb32501e7
Merge pull request #217 from TeamNewPipe/fix/SC_client_id
...
Fix SoundCloud client_id extraction and update SoundCloud client_id
2019-12-12 18:11:42 +01:00
TobiGr
54d1a1a831
Fix SoundCloud client_id extraction and update SoundCloud client_id
...
Don't throw exception when hard coded client_id is wrong.
Addresses TeamNewPipe/NewPipe#2823
2019-12-12 18:01:48 +01:00
Tobias Groza
be79f20217
Merge pull request #219 from Redirion/patch-1
...
Allow PlayLists with missing uploader
2019-12-12 17:58:57 +01:00
Redirion
035994b3cf
Update PlaylistInfo.java
...
added missing imports
2019-12-09 16:55:28 +01:00
Redirion
e98112ea21
Update PlaylistInfo.java
...
Fixed typo
2019-12-07 14:34:15 +01:00
Redirion
cdf41b868c
Allow PlayLists with missing uploader
...
This supports PR 2724 in NewPipe
2019-12-07 12:49:42 +01:00
Tobias Groza
43b54ccc3c
Merge pull request #145 from yausername/peertube
...
Basic PeerTube support (without P2P)
2019-12-02 23:08:52 +01:00
yausername
00c23683b1
Merge remote-tracking branch 'upstream/dev' into peertube
2019-12-03 00:45:03 +05:30
yausername
5115541268
fix peertube trending, added most-liked kiosk
2019-12-03 00:42:06 +05:30
Tobias Groza
b6d3252290
Merge pull request #199 from kapodamy/yt-webm-opus
...
use webm as format for opus audio streams
2019-11-24 21:39:59 +01:00
kapodamy
c1d39f692d
Merge branch 'dev' into yt-webm-opus
2019-11-24 11:56:16 -03:00
yausername
6a7680ce17
added instance getter
2019-11-24 03:38:27 +05:30
yausername
281ccea39f
[peertube] constant name for service
2019-11-24 02:35:41 +05:30
yausername
bc75c6603b
added getBaseUrl method to linkhandler
2019-11-23 00:05:49 +05:30
yausername
279f175693
validate peertube instance
2019-11-22 22:59:14 +05:30
yausername
4e0adbefbc
[peertube] use baseUrl instead of global instance url
2019-11-21 05:35:22 +05:30
yausername
8a7aa6b9a9
added baseUrl param in linkhandlers. Required for multi instance services
2019-11-21 05:00:09 +05:30
yausername
f3a59a6cdc
merged upstream/dev
2019-11-20 03:08:17 +05:30
Mauricio Colli
0c6e2c8f23
Revert "Temp Commit: Ignore Test (Remove this before merge)"
...
This reverts commit 6ca4c8986a
.
2019-11-18 10:09:18 +01:00
Tobias Groza
5c420340ce
Merge pull request #158 from mauriciocolli/timeago-localization
...
Implement time ago parser and improve localization handling
2019-11-16 00:31:35 +01:00
yausername
318f600527
add subtitles support for peertube
2019-11-16 04:30:13 +05:30
yausername
bb5ad49fac
fix html comments
2019-11-16 03:20:35 +05:30
yausername
193442d01c
add original url for share
2019-11-16 02:42:39 +05:30
yausername
d7582734e7
merged upstream/dev
2019-11-16 00:07:45 +05:30
Mauricio Colli
3d21ef5dba
Introduce class that indicates when the time ago is an approximation
2019-11-03 15:46:28 -03:00
Mauricio Colli
6ca4c8986a
Temp Commit: Ignore Test (Remove this before merge)
2019-11-03 15:46:27 -03:00
Mauricio Colli
41b51adceb
[YouTube] Force english as the only supported language ( #196 )
2019-11-03 15:46:25 -03:00
Mauricio Colli
b42bebdee8
Add instructions to quickly test changes in the extractor
...
- Make use of the local Maven repository
2019-11-03 15:46:24 -03:00
Mauricio Colli
cb1e327a6e
[YouTube] Fix parsing of video reminders
2019-11-03 15:46:23 -03:00
Mauricio Colli
d8280ce0da
[YouTube] Parse watching count in live streams items
2019-11-03 15:46:22 -03:00
Mauricio Colli
3638f0e0ea
Implement time ago parser and improve localization handling
...
- Handle special cases for languages where the number is not shown
- Rework the Downloader base implementation, allowing for more
advanced things to be done
- Separate the localization from the content country (just like
YouTube let's the user choose both).
2019-11-03 15:46:21 -03:00
wojcik-online
180836c180
Base Implementation: Parse the upload date of StreamInfoItems
...
In the format '2 days ago' (in English) on a YouTube channel page.
(Parser extensible to other pages.)
2019-11-03 15:46:20 -03:00
Tobias Groza
514ed7bdc1
Merge pull request #211 from mauriciocolli/improve-recaptcha-detection
...
[YouTube] Improve reCAPTCHA detection
2019-11-01 12:05:49 +01:00
Mauricio Colli
c4fe2183ce
[YouTube] Improve detection of reCAPTCHA pages
2019-10-29 02:00:41 -03:00
Mauricio Colli
9a325b280d
[YouTube] Make detection of age restricted pages more reliable
2019-10-29 02:00:40 -03:00
Mauricio Colli
91c360df5e
Remove section of dead code
2019-10-29 02:00:39 -03:00
Mauricio Colli
ddd563fe78
[SoundCloud] Add test for hardcoded client id
2019-10-23 16:54:20 +02:00
Mauricio Colli
4fc18a6994
[SoundCloud] Fix extraction of client id
...
- Hardcoded id and check at the first usage.
- As a fallback, and considering that the scripts containing the client
id were all split up, try searching it in each of them.
2019-10-23 16:54:20 +02:00
Mauricio Colli
250c0bb1e8
Add head request to the current downloader implementation
2019-10-23 16:54:20 +02:00
kapodamy
3c6e93c96a
Merge branch 'dev' into yt-webm-opus
2019-10-12 14:01:50 -03:00
Tobias Groza
06f2144e4d
[YouTube] Remove GEMA exception handling ( #202 )
2019-10-05 14:59:05 +02:00
kapodamy
95f93bc71c
Merge branch 'dev' into yt-webm-opus
2019-10-01 13:11:43 -03:00
Tobias Groza
b9c0d71210
Merge pull request #190 from nv95/feature/frames
...
Fetching frames preview from youtube
2019-09-30 19:48:27 +02:00
Stypox
fbb9a86c9f
Merge branch 'dev' into yt-webm-opus
2019-09-29 18:16:14 +02:00
Vasiliy
d8279f91f6
Merge branch 'dev' of https://github.com/TeamNewPipe/NewPipeExtractor into feature/frames
2019-09-28 14:46:14 +03:00
TobiGr
8ab48c62b9
[YouTube] Fix NPE in ChennelExtractor.getSubsciberCount()
2019-09-25 08:57:08 +02:00
TobiGr
0710f31a39
Fix TeamNewPipe/NewPipeExtractor#197
2019-09-25 08:37:26 +02:00
Stypox
d1cd341592
Change comment
2019-09-25 08:32:00 +02:00
Stypox
f6088c4fc1
Add test for Eminem channel (it has no "Subscribe" button)
2019-09-25 08:32:00 +02:00
Stypox
35921345d9
Use Exception instead of Throwable (more consistent)
2019-09-25 08:32:00 +02:00
Stypox
db3596c818
Fix "Could not get id" for channels w/o "Subscribe" button
2019-09-25 08:32:00 +02:00
Stypox
94e7f0d3ab
Fix fallback method is not tried on exception
...
in YoutubeChannelInfoItem.getUrl()
2019-09-25 08:32:00 +02:00
kapodamy
12008fc6bb
Update MediaFormat.java
...
declare WEBMA_OPUS
2019-09-23 21:25:19 -03:00
kapodamy
5f3c2f5622
Update ItagItem.java
2019-09-23 21:25:19 -03:00
TobiGr
6d504e0883
Add test for mixedNumberWordToLong method
...
Add Billion to mixedNumberWordToLong
2019-09-17 09:15:05 +02:00
TobiGr
06016d1ae3
Fix YouTube subscriber count
...
Modify test to fail on too small subscriber count
2019-09-16 23:16:05 +02:00
Tobias Groza
dbdd9ed083
Merge pull request #195 from Stypox/livestream-fix
...
Fix livestreams
2019-09-12 22:40:44 +02:00
Tobias Groza
acf2b4c5fa
Merge pull request #194 from toehead2001/youtube-music
...
Add support for music.youtube.com to link handler
2019-09-12 22:36:04 +02:00
toehead2001
b709529cb6
Add link handler tests for music.youtube.com
2019-09-12 12:16:30 -06:00
toehead2001
075e6d51d6
Add music.youtube.com to link handler
2019-09-12 12:16:30 -06:00
Stypox
1a1672248a
Eliminate Android Studio warnings in livestream test
2019-09-12 15:11:23 +02:00
Stypox
3f1ba93be5
Fix NPE when extracting itags with non-existing streamingData key
2019-09-12 15:08:17 +02:00
Stypox
0d8fb65003
Fix NPE on determining whether stream is live on Youtube
2019-09-12 15:07:07 +02:00
Stypox
4453a63447
Add test for YouTube livestreams
...
The current livestream is https://www.youtube.com/watch?v=EcEMX-63PKY
2019-09-12 15:05:22 +02:00
Stypox
5f8e76eb87
Move stream-related youtube tests to subfolder
2019-09-12 14:36:42 +02:00
Tobias Groza
bfa6b4d14b
Merge pull request #182 from 2secslater/patch-1
...
Add Invidious instances
2019-09-12 09:19:11 +02:00
Tobias Groza
ec3554a2ea
Merge pull request #193 from Stypox/unavailable-video-fix
...
Fix TeamNewPipe/NewPipe#2615
2019-09-11 21:44:05 +02:00
Stypox
24a37b88a9
Use pre-generated playerResponse field in yt's getHlsUrl()
...
Also refactored code to always throw exception when the url can't be found
2019-09-11 20:12:30 +02:00
Stypox
9c423a0a40
Use FORMATS to get video+audio streams on yt
...
Not ADAPTIVE_FORMATS
2019-09-11 20:04:28 +02:00
Stypox
e5e8c66686
Readd signature decryption in YtStreamExtractor
2019-09-11 19:56:16 +02:00
Stypox
d9570d8634
Use pre-generated playerResponse field everywhere in YtStreamExtractor
2019-09-11 19:35:08 +02:00
Stypox
63a37c48e3
Remove println left behind
2019-09-11 19:31:39 +02:00
Stypox
bf017bf5b9
Fix TeamNewPipe/NewPipe#2615
2019-09-11 19:05:41 +02:00
Vasiliy
ecb8ad85a1
Update comments
2019-09-11 19:03:53 +03:00
Andrew
d0f1c31b34
Add Invidious instances to parsing helper for YouTube
2019-09-10 17:54:32 +01:00
Vasiliy
b55c05d04f
Update frameset extractor test
2019-09-10 19:42:55 +03:00
Vasiliy
ab32b0ed84
Merge remote-tracking branch 'upstream/dev' into feature/frames
2019-09-10 19:39:58 +03:00
Vasiliy
f084cfec24
Refactor frames extraction
2019-09-10 19:38:51 +03:00
Tobias Groza
c7e9ad57c4
Merge branch 'dev' into patch-1
2019-09-09 19:05:00 +02:00
Tobias Groza
7169bcfb75
Merge pull request #192 from gzsombor/fix-illegal-expression
...
Fix 'java.lang.IllegalArgumentException
2019-09-03 09:12:21 +02:00
Zsombor Gegesy
93aeb19bbc
Fix 'java.lang.IllegalArgumentException: Did not find balanced marker at 'class*="yt-lockup-video"'
...
at org.jsoup.helper.Validate.fail(Validate.java:110)'
2019-09-02 23:45:37 +02:00
Vasiliy
a6c94c7a9d
Grub frames preview from youtube
2019-08-26 19:14:09 +03:00
Christian Schabesberger
cfc72d8dc9
Merge pull request #188 from jimbo1qaz/fix-description-timestamps
...
Fix timestamp links in Youtube video descriptions
2019-08-22 11:25:16 +02:00
jimbo1qaz
e38d906ff9
Fix timestamp links in Youtube video descriptions
...
For some reason, in NewPipeExtractor,
comments were loaded from JSON by YoutubeCommentsInfoItemExtractor as text,
sent via CommentsInfoItem#getCommentText to NewPipe,
where timestamps are converted to hyperlinks using Linkify:
https://github.com/TeamNewPipe/NewPipe/pull/2168
On the other hand, video descriptions are handled in NewPipeExtractor
by scraping the watch-page HTML.
There, timestamp links were previously mangled (and now properly parsed),
before being sent as HTML via YoutubeStreamExtractor#getDescription
to NewPipe (where HTML gets converted to Spanned).
The logic introduced in this commit is different from the above PR,
since it operates in the extractor, and mutates the HTML DOM
rather than identifying via regex.
2019-08-18 01:05:41 -07:00
Christian Schabesberger
430da57350
Merge pull request #184 from Stypox/duplicate-subscription-fix
...
Fix inconsistency in youtube channel urls
2019-08-18 00:42:31 +02:00
Christian Schabesberger
58e3996edc
Merge branch 'dev' into duplicate-subscription-fix
2019-08-18 00:42:24 +02:00
Christian Schabesberger
f9da8bc0c8
Merge pull request #186 from Stypox/recaptcha-with-url
...
Add url to ReCaptchaException
2019-08-18 00:42:15 +02:00
Christian Schabesberger
3e2df12f12
Merge branch 'dev' into recaptcha-with-url
2019-08-18 00:41:27 +02:00
Christian Schabesberger
92846076ce
Merge branch 'dev' into duplicate-subscription-fix
2019-08-18 00:41:16 +02:00
Christian Schabesberger
a13304248f
Merge pull request #185 from Stypox/search-error-fix
...
Fix casual search error when playlists are in results
2019-08-18 00:37:59 +02:00
Stypox
06689a2f27
Add url to ReCaptchaException
...
Sometimes YouTube introduces recaptchas only on some pages. By adding an url to the ReCaptchaException the NewPipe app is able to use that url to load the page that originally caused the problem.
Also removed every instance of exception caught and rethrown with a different description: it makes no sense and it removes part of the useful stacktrace.
2019-08-17 09:09:07 +02:00
Stypox
d4e975e4fa
Fix search error with some playlists
...
Somtimes there were two divs with class "yt-lockup-meta", so the extractor couldn't get the correct one.
2019-08-16 22:47:02 +02:00
Stypox
216a4eb1f5
Complete fix inconsistency in youtube channel urls
...
It is not always possible to get the url in the form "https://www.youtube.com/channel/ ...", so a not has been added whenever that happens to be the case (i.e. only in InfoStreamItems).
2019-08-16 21:17:03 +02:00
Stypox
315c5c262f
Typo
2019-08-16 21:14:52 +02:00
Stypox
d14c45c948
Fix SoundCloud tests
2019-08-12 17:15:21 +02:00
Stypox
b09e402d4f
Fix wrong regex when channel id is at the end of the url
...
It had no "&" at the end.
2019-08-12 16:55:39 +02:00
Stypox
31e74253f8
Fix tests
2019-08-12 16:38:56 +02:00
Stypox
09c6822b1d
Change youtube channel url test
2019-08-12 13:13:41 +02:00
Stypox
b8bc57c53f
Add tests for youtube channel urls
...
They have to be in the form "https://www.youtube.com/channel/ID "
2019-08-12 11:58:50 +02:00
Stypox
6aa69a2df8
Fix inconsistency in youtube channel urls
...
Urls from the youtube search extractor were "https://www.youtube.com/user/NAME " instead of "https://www.youtube.com/channel/ID ". This fixes TeamNewPipe/NewPipe#2167
2019-08-12 11:57:29 +02:00
Andrew
7fb17684f5
Fixed missing colon causing builds to fail
2019-08-08 00:25:42 +00:00
Andrew
5a2dcdce16
Add Invidious instances
...
Added all publicly listed Invidious instances from omarroth/invidious wiki page to the link handler factory for YouTube.
2019-08-08 00:19:02 +00:00
Christian Schabesberger
5f65788a2f
Merge pull request #177 from TeamNewPipe/duration_fix
...
fix duration can not be paresd
2019-07-31 18:03:41 +02:00
Christian Schabesberger
5798c8fdf5
fix duration can not be paresd
...
update gradle to version 5.1
fix sts issue for agegated videos
GOD DAMN FUCKING BULLSHIT
add duratin for controversal/age gated videos
bring back sts
remove ignores
fix ogg test
2019-07-31 16:11:42 +02:00
Christian Schabesberger
4488c21765
Merge pull request #159 from ritiek/better-soundcloud-thumbnail
...
Improve thumbnail quality for Soundcloud service
2019-06-27 14:27:23 +02:00
Tobias Groza
cfdbc4e1d1
Merge pull request #170 from TobiGr/dev
...
Fix SoundCloud playlists parsing exception
2019-06-26 01:02:34 +02:00
TobiGr
0d09a9fd61
Fix SoundCloud playlists parsing exception
...
Closes TeamNewPipe/NewPipe#2344
2019-06-26 00:56:03 +02:00
Tobias Groza
c64c90a567
Merge pull request #168 from masozzi/dev
...
soundcloud parsing helper: fixed id parser regex
2019-06-02 21:06:57 +02:00
Tobias Groza
796c1b7bc0
Merge pull request #164 from TeamNewPipe/fix/test_comments
...
Fix failing YouTube comments tests
2019-06-02 11:36:16 +02:00
Matteo Sozzi
93d4299f07
soundcloud parsing helper: fixed id parser regex
2019-05-31 20:15:36 +02:00
Tobias Groza
2ac713e70f
Merge pull request #160 from Stypox/invalid-youtube-subscription-fix
...
Fixed youtube subscription import: ignore ones with invalid url and keep ones with empty title.
2019-05-14 22:26:25 +02:00
Tobias Groza
867ca1cabf
Fix failing YouTube comments tests
...
The comment function has been disabled for the video on which we ran the test. We are testing the comments of a different video now.
2019-05-14 22:11:40 +02:00
Tobias Groza
acebbaf073
Merge pull request #163 from Stypox/youtube-url-signature-fix
...
Fix invalid yt url: signature tag name is not always "signature"
2019-05-14 21:29:31 +02:00
Stypox
c70d28597b
Add fallback for urls not conaining the "sp" tag
...
If ever YouTube changes thing again (or uses old urls for some unknown reason), this prevents the extractor from crashing.
As suggested here: 133cc032d9 (r283529811)
2019-05-14 13:57:45 +02:00
Stypox
133cc032d9
Fix invalid yt url: signature tag name is not always "signature"
...
Thanks to @omarroth for the suggestion: see TeamNewPipe/NewPipeExtractor#155
2019-05-13 21:25:35 +02:00
Stypox
0eaca52c15
Add test for subscription with invalid url.
...
Also modified the test for empty title, since now subscriptions with empty title are not ignored anymore.
2019-04-28 14:19:33 +02:00
Stypox
171f2c49fe
Ignore subscriptions with invalid url and keep ones with empty title.
...
if a channel if deleted (thus it has an empty title), it is imported in NewPipe anyway, so that if it becomes undeleted in the future, it will be shown in the app.
2019-04-28 14:17:52 +02:00
Stypox
d5043cdf49
Add test for subscriptions with empty title.
...
(youtube subscription extractor)
2019-04-26 19:59:23 +02:00
Stypox
03893abd91
Fixed TeamNewPipe/NewPipe#2226 .
...
(in the youtube subscription extractor)
Ignore subscriptions that have an empty title instead of throwing an error: the youtube subscription_manager XML file can sometimes contain those (i.e. deleted channels).
2019-04-26 18:54:30 +02:00
Ritiek Malhotra
c7974b2aed
Fetch better quality thumbnails
...
and fallback to avatar thumbnail if track thumbnail isn't found
2019-04-26 02:59:15 +05:30
yausername
f60c973928
more javadoc
2019-03-23 19:32:06 +05:30
yausername
e4e6d44770
reordered services
2019-03-23 19:12:26 +05:30
yausername
aabc65b57d
added java doc for stream
2019-03-23 19:11:51 +05:30
yausername
07a8129a73
fixed peertube search test
2019-03-23 19:11:19 +05:30
yausername
bd22b55885
fixed youtube channel name test
2019-03-23 19:10:34 +05:30
yausername
2193119fab
youtube's comment censorship breaks test
2019-03-23 19:09:40 +05:30
Christian Schabesberger
d22786b817
Merge branch 'master' into dev
2019-03-23 10:07:28 +01:00
Christian Schabesberger
4c633c2166
Merge pull request #137 from connectety/invidious-support
...
Invidious support
2019-03-23 10:01:51 +01:00
Christian Schabesberger
5b93db04e6
Merge pull request #152 from yausername/fixMissingAuthorName
...
fix empty author name
2019-03-23 09:57:44 +01:00
yausername
4effd0b36d
fix empty author name
2019-03-23 00:13:18 +05:30
Christian Schabesberger
560c648e92
fix decrypt regex for akamai 2 times in file
2019-03-14 16:49:30 +01:00
Connectety
ed78f7dadc
Merge branch 'master' into invidious-support
2019-03-14 16:07:04 +01:00
Christian Schabesberger
dd61d66cf5
speed up finding decrypt function
2019-03-14 09:07:19 +01:00
Christian Schabesberger
5eac266a16
Merge branch 'dev' of github.com:TeamNewPipe/NewPipeExtractor into dev
2019-03-14 08:50:57 +01:00
Ritvik Saraf
0a7d42f58d
set soundcloud default kiosk
2019-03-14 08:49:37 +01:00
Ritvik Saraf
53058802e2
fix comment url
2019-03-14 08:49:37 +01:00
Christian Schabesberger
e072bf6461
fix dercrypt error due to wrong dollar sign detection
2019-03-14 08:49:11 +01:00
Christian Schabesberger
5f44031ce7
Merge pull request #146 from yausername/soundcloudDefaultKiosk
...
set soundcloud default kiosk
2019-03-10 22:28:29 +01:00
Ritvik Saraf
cb30b336ab
set soundcloud default kiosk
2019-03-11 02:14:58 +05:30
Ritvik Saraf
c220700c94
fixed tests
2019-03-10 00:37:01 +05:30
Ritvik Saraf
36139558c6
handle stream 404 error, fixed tests
2019-03-09 23:33:51 +05:30
Ritvik Saraf
e66d188381
throw content not available
2019-03-08 04:54:39 +05:30
Ritvik Saraf
22dac63698
merged upstream/dev
2019-03-08 03:09:05 +05:30
Christian Schabesberger
aa4f03a385
Merge pull request #144 from yausername/fixCommentUrl
...
fix comment url
2019-03-02 23:38:28 +01:00
Connectety
c255fdcdf7
Merge branch 'master' into invidious-support
2019-03-02 13:03:31 +01:00
Ritvik Saraf
a2735c4f97
fix comment url
2019-03-02 02:48:05 +05:30
Christian Schabesberger
536365c99e
Merge pull request #142 from yausername/serviceByUrlNPEfix
...
fix npe in fetching service by url
2019-03-01 09:57:52 +01:00
Christian Schabesberger
16d693ea7a
Merge branch 'master' into serviceByUrlNPEfix
2019-03-01 09:54:41 +01:00
Christian Schabesberger
8425936b67
Merge pull request #143 from yausername/sanitizeComment
...
Sanitize youtube comment
2019-03-01 09:52:22 +01:00
Ritvik Saraf
72262707bf
fixed youtube comment utf8 bom
2019-03-01 04:21:45 +05:30
Ritvik Saraf
9eff18252b
fix npe in fetching service by url
2019-02-28 22:55:18 +05:30
Christian Schabesberger
cffbd468cc
Merge branch 'master' into invidious-support
2019-02-24 21:58:59 +01:00
Christian Schabesberger
e7e411dc29
Merge pull request #141 from yausername/invalidChannelUrlFix
...
Invalid channel url fix
2019-02-24 21:58:26 +01:00
Ritvik Saraf
948a7826ab
fetch channel id for channel url
2019-02-23 19:03:52 +05:30
Connectety
28c5b86f6d
Merge branch 'master' into invidious-support
2019-02-20 17:58:21 +01:00
Christian Schabesberger
8de53111d9
Merge pull request #101 from yausername/master
...
added comments extractor
2019-02-19 17:31:25 +01:00
Christian Schabesberger
021e4e06b6
Merge branch 'master' into master
2019-02-19 17:28:38 +01:00
Christian Schabesberger
04460bba2c
add mediaCCC to supported sites in description.
2019-02-19 16:10:46 +01:00
Ritvik Saraf
3dadf63028
refactored comments capability
2019-02-16 00:57:00 +05:30
Ritvik Saraf
a97b6dfa7a
mediaccc merge
2019-02-16 00:41:23 +05:30
Ritvik Saraf
f8fb7b6500
Merge remote-tracking branch 'upstream/master'
2019-02-16 00:18:39 +05:30
Connectety
5079d5efe1
Merge branch 'master' into invidious-support
2019-02-07 23:06:19 +01:00
Christian Schabesberger
d15f32925b
Merge pull request #139 from TeamNewPipe/ccc
...
Ccc
2019-02-05 18:27:47 +01:00
Ritvik Saraf
f97d5ab4cd
Merge remote-tracking branch 'upstream/master'
2019-01-29 22:10:07 +05:30
Christian Schabesberger
79b0a19d1a
make search filter empty be equal to all
2019-01-29 16:49:37 +01:00
Christian Schabesberger
d0efe8bd47
fix ogg
2019-01-29 16:04:19 +01:00
Christian Schabesberger
644da4fc3c
add ogg as audioformat
2019-01-28 15:43:21 +01:00
Christian Schabesberger
14e1ccdb86
add conferences search
2019-01-28 15:30:17 +01:00
Christian Schabesberger
1503459292
add kiosk for conferences
2019-01-28 12:18:16 +01:00
Connectety-W
6390eb268b
fixed YoutubePlaylistLinkHandlerFactory accepting invalid links
2019-01-27 12:05:36 +01:00
Connectety-W
1ab7a1f930
added tests for YoutubePlaylistLinkHandlerFactory
2019-01-27 12:00:23 +01:00
Connectety-L
ec4aa9e0cd
added Invidio subdomain test and fixed error in test caused by Invidio playlist support
2019-01-27 02:12:12 +01:00
Connectety-L
10939efcce
added support playlists on Invidio
2019-01-27 01:44:46 +01:00
Connectety-L
cce5e4ad33
added support and tests for Invidio Trending-URLs
2019-01-27 01:29:23 +01:00
Connectety-L
7493ed903b
split isYoutubeALikeURL into multiple methods
2019-01-27 01:28:51 +01:00
Christian Schabesberger
cb51254aa3
return one null less
2019-01-26 22:55:34 +01:00
Christian Schabesberger
73232a7bad
stream names may not contain dots
2019-01-26 17:47:58 +01:00
Christian Schabesberger
50a17afc05
fix issue with originalUrl
2019-01-26 17:47:58 +01:00
Christian Schabesberger
cf3ea22753
make stream size have a p in the end
2019-01-26 17:47:58 +01:00
Christian Schabesberger
7fcb3f5790
fix crash when return null as a related streams collector
2019-01-26 17:47:58 +01:00
Christian Schabesberger
2148edc7d7
add channel and strem extractor for mediaccc
2019-01-26 17:47:58 +01:00
Christian Schabesberger
ef576341c8
implement searchextractor for mediaccc
2019-01-26 15:22:16 +01:00
Connectety-L
2ede47d36c
added hooktube and invidio http test
2019-01-24 14:19:44 +01:00
Connectety-L
2ae23a6f79
added support for videos on invidio.us
2019-01-24 11:13:01 +01:00
Connectety-L
399b4f2eef
added support for "vnd.youtube.launch" URI-scheme
2019-01-24 10:53:03 +01:00
Connectety-L
ae23059d66
added support for channels on invidio.us
2019-01-24 10:48:29 +01:00
Connectety-L
79ba6aba95
added tests from https://github.com/TeamNewPipe/NewPipeExtractor/pull/128
2019-01-24 10:27:01 +01:00
Christian Schabesberger
8114ce6ae4
Merge pull request #132 from connectety/master
...
refactored YouTube-linkHandler
2019-01-20 14:46:06 +01:00
Connectety-W
19288c1456
added forgotten break statements
2019-01-20 14:39:06 +01:00
Christian Schabesberger
76a3742084
Merge branch 'master' into master
2019-01-20 14:34:16 +01:00
Christian Schabesberger
f7c7b9df1a
Merge pull request #135 from yausername/ytlivestream-fix
...
[Youtube Live Streams] fixed extracting live HLS URL
2019-01-20 13:55:32 +01:00
Connectety-W
a6c972eff8
fixed tests by prepending HTTP to URLs without protocol and adding a check for null.
2019-01-20 01:31:30 +01:00
Ritvik Saraf
c60d6ec0ab
removed unused import
2019-01-19 18:20:02 +05:30
Ritvik Saraf
50c4783f71
fixed extracting hls url
2019-01-19 18:07:35 +05:30
Ritvik Saraf
c4e502b9d2
Merge branch 'master' into peertube
2019-01-19 14:33:01 +05:30
Ritvik Saraf
25ed9deaf3
Merge remote-tracking branch 'upstream/master'
2019-01-19 14:28:12 +05:30
Christian Schabesberger
97d72590fc
Merge branch 'master' into master
2019-01-18 22:31:37 +01:00
Christian Schabesberger
99915e4527
fix failing tests
2019-01-18 12:44:34 +01:00
Christian Schabesberger
6776abdd55
Merge https://github.com/The-Twisted-Samurai/NewPipeExtractor
2019-01-18 12:25:20 +01:00
Christian Schabesberger
09bafe0660
fix failing viewcount test
2019-01-18 12:24:30 +01:00
Kunal Sareen
b6790bbe6b
Updating regex
2019-01-18 21:47:34 +11:00
Connectety
43ffeac023
Merge branch 'master' into master
2019-01-13 13:38:35 +01:00
Connectety-W
98f49852d7
refactored YouTube-linkHandler to use less regex and more URL-methods
2019-01-13 12:52:07 +01:00
Ritvik Saraf
fade7bd749
getter for torrent url
2018-12-30 15:23:24 +05:30
Ritvik Saraf
ef0ffb2229
added torrent url in streams
2018-12-30 15:00:57 +05:30
Ritvik Saraf
b1a77fa484
update peertube instance
2018-12-29 22:23:35 +05:30
Ritvik Saraf
a83342de0f
update peertube instance
2018-12-29 21:58:43 +05:30
Ritvik Saraf
b05fa45080
resolved review comments
2018-12-28 13:02:00 +05:30
Ritvik Saraf
b2c6928459
added logic for fetching related streams
2018-12-26 14:29:23 +05:30
Ritvik Saraf
bc82a53f2c
replace base url
2018-12-26 13:07:06 +05:30
Ritvik Saraf
203c884d51
strip base url
2018-12-26 13:02:04 +05:30
Ritvik Saraf
14cd1dbcbb
fixed kiosk by url
2018-12-26 12:50:43 +05:30
Ritvik Saraf
8755c25349
format date of comment published. changed default instance
2018-12-26 11:08:39 +05:30
Ritvik Saraf
df0db8468d
merged master
2018-12-25 19:12:22 +05:30
Ritvik Saraf
c77050dc70
handle empty comments
2018-12-23 08:24:36 +05:30
Ritvik Saraf
9ad102df3f
fixed fetching youtube client name
2018-12-23 08:24:36 +05:30
Ritvik Saraf
4c49a347f6
merged upstream/master
2018-12-04 21:20:26 +05:30
Christian Schabesberger
91b1efc97e
add documentation of StreamingService
2018-11-10 10:50:13 +01:00
Christian Schabesberger
a78ad16235
rebase correctly
2018-11-09 19:55:40 +01:00
kapodamy
b40a1f3657
Delete Subtitles.java
...
change class name
2018-11-08 17:06:41 -03:00
kapodamy
388f4fd38b
Delete SubtitlesFormat.java
...
merge into MediaFormat
2018-11-08 16:29:06 -03:00
kapodamy
e4afb21862
make Subtitle object Stream compliant
...
* merge SubtitlesFormat into MediaFormat
* implement Stream interface on Subtitle class
* misc fixes: dont show a snackbar error on age-restricted videos, lint fix.
2018-11-08 16:29:05 -03:00
Christian Schabesberger
7129d6db55
add inline documentation for StreamExtractor
2018-11-07 18:28:44 +01:00
Christian Schabesberger
4de99ae28f
add support for segmented streams
2018-11-06 14:24:47 +01:00
Christian Schabesberger
32d316330c
fix broken search results
2018-10-25 15:46:47 +02:00
Ritvik Saraf
f58c914e73
merge fix
2018-10-19 20:17:26 +05:30
Ritvik Saraf
05ef80c2eb
Merge remote-tracking branch 'upstream/master'
2018-10-19 20:09:51 +05:30
Ritvik Saraf
99a0134b1c
removed unchecked cast
2018-10-19 19:33:36 +05:30
Ritvik Saraf
4883b6fb48
return suggestion extractor
2018-10-12 13:04:51 +05:30
Ritvik Saraf
706d4271a6
empty suggestions for search
2018-10-12 12:45:09 +05:30
Ritvik Saraf
c9fba9cb1e
fallback avatar for channel
2018-10-12 02:27:32 +05:30
Ritvik Saraf
26da3088ba
fallback avatar and description
2018-10-12 01:59:13 +05:30
Ritvik Saraf
9dee8d5fcd
updated kiosk accept link
2018-10-12 01:37:47 +05:30
Ritvik Saraf
20f280cb57
added peertube extractor
2018-10-12 00:40:22 +05:30
Christian Schabesberger
24cb684e23
fix failing test
2018-10-06 17:11:12 +02:00
Christian Schabesberger
0920680708
fix search country for next page
2018-10-06 17:03:55 +02:00
Christian Schabesberger
0c3dc86b07
add support for yt content country
2018-10-06 12:22:37 +02:00
Christian Schabesberger
3a865749e8
fix issues Kiosk issue about localization, nad rename to preferred localization
2018-10-05 16:01:10 +02:00
Ritvik Saraf
e85958b180
merged upstream/master
2018-09-29 13:19:00 +05:30
Ritvik Saraf
d1ff1c7589
using mobile website since it is faster
2018-09-28 04:54:57 +05:30
Ritvik Saraf
c2ed99b1b7
no comments
2018-09-27 23:01:27 +05:30
Ritvik Saraf
ad8066830d
getting client version and name from initial page
2018-09-27 22:32:13 +05:30
Ritvik Saraf
6b620914b6
moved cookie logic outside
2018-09-27 02:04:12 +05:30
Ritvik Saraf
fb1419608a
using getDataString
2018-09-27 00:44:55 +05:30
Ritvik Saraf
ce76885553
removed generics
2018-09-26 04:21:58 +05:30
Ritvik Saraf
8e27801183
removed jackson and java 8
2018-09-26 03:20:29 +05:30
Christian Schabesberger
5223aece7b
fix failing unit test
2018-09-24 15:17:57 +02:00
Christian Schabesberger
f334dd07de
fix youtub.be links in description
2018-09-24 15:06:00 +02:00
Christian Schabesberger
ad4b15ac66
Merge pull request #114 from TeamNewPipe/localization
...
Localization
2018-09-24 14:00:20 +02:00
Ritvik Saraf
c1199c8fcf
added isCommentsSupported method
2018-09-23 01:12:11 +05:30
Ritvik Saraf
0e86475891
removed commentsInfo from streamInfo
2018-09-22 23:36:41 +05:30
Ritvik Saraf
7ed0da0493
more testing
2018-09-22 19:09:31 +05:30
Ritvik Saraf
6a0341d59c
testing
2018-09-22 14:55:59 +05:30
Christian Schabesberger
49c2eb5185
fix failing watch links
2018-09-20 13:51:06 +02:00
Ritvik Saraf
e04787f340
fixed NPE for services where comments is not available
2018-09-19 05:32:14 +05:30
Ritvik Saraf
4794e16dcb
removed comment details form streamInfo. added commentsInfo instead
2018-09-19 04:58:40 +05:30
Ritvik Saraf
ee239985ae
added CommentsInfo
2018-09-19 04:58:40 +05:30
Ritvik Saraf
4ca23ab5c3
revering serialzable change
2018-09-19 04:58:40 +05:30
Ritvik Saraf
bf2d3deb03
make extractor serializable
2018-09-19 04:58:40 +05:30
Ritvik Saraf
823551170d
added loadable comments in stream info
2018-09-19 04:58:40 +05:30
Ritvik Saraf
9fb0622a24
better quality thumbnails
2018-09-19 04:58:40 +05:30
Ritvik Saraf
f62f147ea0
added comments to stream info
2018-09-19 04:58:40 +05:30
Ritvik Saraf
06898f47b2
close resource
2018-09-19 04:58:40 +05:30
Ritvik Saraf
95575756ee
added http post method in downloader, formatting
2018-09-19 04:58:40 +05:30
Ritvik Saraf
8b8779b176
removing catch exception
2018-09-19 04:58:40 +05:30
Ritvik Saraf
53d3997904
added comments extractor
2018-09-19 04:58:40 +05:30
Christian Schabesberger
ce2bbee1e9
add propper localization
2018-09-15 22:29:28 +02:00
Christian Schabesberger
dc0d0bda24
replace uiHandler name with linkhandler
2018-09-15 22:29:28 +02:00
Christian Schabesberger
1e7bcfbd66
Merge pull request #97 from MadderRagax/master
...
Fixed #1548 and #1546 - Playlist links not getting parsed properly
2018-09-15 22:29:08 +02:00
Christian Schabesberger
ae8107b182
Merge branch 'master' into master
2018-09-15 22:18:54 +02:00
Christian Schabesberger
96b4c97cee
Merge pull request #109 from karyogamy/sig-fix-update
...
Decryption Signature Regex Update
2018-09-15 22:10:31 +02:00
Christian Schabesberger
217d13b102
fix wrong subscription count
2018-09-11 15:14:22 +02:00
Christian Schabesberger
8a4afe2548
refactor split time parsing
2018-09-09 14:01:39 +02:00
Christian Schabesberger
49b72cbe5d
Merge pull request #112 from skil3z/master
...
Fix time formatting for different countries
2018-09-09 13:58:34 +02:00
skil3z
a83d020944
Accommodate time formatting for different countries
...
If there's a . in the time format, this detects it and uses is instead of :
This removes errors and lag related to "Could not get duration" while using NewPipe in countries with official time formatting with . (dot) instead of : (punctuation colon)
Tested to compile and work on real device
2018-09-09 12:53:10 +03:00
John Zhen Mo
ed73ae55f1
-Added more decrypt function name matching regex.
...
-Cleaned up decryption code generation method.
2018-09-07 22:25:07 -07:00
Christian Schabesberger
66c3c3f452
fix channel links in description part 2
2018-09-07 22:18:22 +02:00
Christian Schabesberger
4469d11307
fix channel links in description
2018-09-07 21:40:36 +02:00
John Zhen Mo
119843bfac
-Fix signature extraction parse exception.
2018-09-07 11:34:14 -07:00
Mauricio Colli
850670917f
Quick fix for the kiosks in SoundCloud
2018-09-05 07:25:24 -03:00
Mauricio Colli
834382111b
Fix NPE when playlist is empty
2018-09-03 22:37:31 -03:00
Mauricio Colli
0400ae026f
Fix channels with subscription count disabled
...
Related: TeamNewPipe/NewPipe#1649
2018-09-03 22:37:31 -03:00
Mauricio Colli
f6be85e2f5
Merge branch 'dev' into 'master'
2018-09-03 22:37:31 -03:00
Christian Schabesberger
f498dd7875
fix failing unit tests
...
fix yt share url error
2018-08-28 17:19:22 +02:00
Christian Schabesberger
fef71aeccc
roleback using href for description links
2018-08-28 11:55:12 +02:00
Christian Schabesberger
3aa7ff6d0e
make ytdescription alter href instead of the text of links
2018-08-27 20:45:43 +02:00
Christian Schabesberger
e662c97433
make dash parser ignore segmented streams
2018-08-21 17:23:56 +02:00
Christian Schabesberger
c4e16c7337
fix hashtag parsing
2018-08-17 16:22:50 +02:00
Christian Schabesberger
981cb333b5
fix link parsing for yt internal links
...
blub
2018-08-16 17:44:09 +02:00
Christian Schabesberger
389a87fc89
add link parsing for youtube description
2018-08-16 17:11:18 +02:00
Christian Schabesberger
aeb813840d
fix search querry tests
2018-08-05 14:14:36 +02:00
Christian Schabesberger
701666f498
block yt premium videos
...
fix logig error for block yt premium videos
2018-08-05 13:04:49 +02:00
Christian Schabesberger
af1dddb79e
Merge pull request #96 from Bleuzen/enable_opus
...
Enable opus format 251
2018-08-01 18:21:50 +02:00
oscar
a077cb9777
Small YoutubeStreamLinkHandlerFactory refactoring
2018-07-25 17:21:42 +02:00
oscar
a9ce0b29d0
Fixed improper handling of playlist deep links
2018-07-25 17:15:56 +02:00
Christian Schabesberger
dcca11fbeb
Merge branch 'dev'
2018-07-25 14:13:16 +02:00
Bleuzen
9af07a2562
re-enable opus
2018-07-24 17:25:42 +02:00
Christian Schabesberger
1eff8c5708
rename filter playlist to playlists
2018-07-19 16:16:37 +02:00
Christian Schabesberger
558a973e43
rename search filters
2018-07-19 16:06:44 +02:00
Christian Schabesberger
8ee068fd98
fix names in kiosklist
...
blub
2018-07-13 21:50:49 +02:00
Christian Schabesberger
28788a05db
rename uih and remove afiliate link foo
2018-07-13 18:02:40 +02:00
Christian Schabesberger
a1aaca1bea
get info from initial page
2018-07-08 20:49:13 +02:00
Christian Schabesberger
53f0bc9d8d
add getInfo without need for search extractor to searchinfo
2018-07-08 20:15:14 +02:00
Christian Schabesberger
35b46900c1
add outcomming items of searchextractor to searchinfo
2018-07-08 19:53:25 +02:00
Christian Schabesberger
4746a1c48a
upgrade kisok uriIdhandler to ListUIHandler
2018-07-08 15:27:35 +02:00
Christian Schabesberger
31b0480e22
fix failing tests
2018-07-08 14:43:49 +02:00
Christian Schabesberger
2f351be772
add getMoreInfo to SearchInfo
2018-07-08 14:32:31 +02:00
Christian Schabesberger
39332a283b
remove searchengine
2018-07-02 13:49:14 +02:00
Christian Schabesberger
812a785811
remove searchengine
2018-07-02 13:47:39 +02:00
Christian Schabesberger
291de18418
getUIHandler() to getUIHFactory()
...
handler to handlerFactory in kiosk
2018-07-01 19:01:39 +02:00
Christian Schabesberger
cf542db853
getUIHandler() to getUIHFactory()
2018-07-01 18:50:21 +02:00
Christian Schabesberger
bd5423fe2a
make less tests fail
2018-07-01 16:55:48 +02:00
Christian Schabesberger
17f46b8b12
rename UrlIdHandler to UIHFactory
2018-07-01 16:55:48 +02:00
Christian Schabesberger
9bda761103
add test for soundcloud search extractor
2018-07-01 16:55:48 +02:00
Christian Schabesberger
ef2ce6854c
add soundcloud SearchExtractor
2018-07-01 16:55:48 +02:00
Christian Schabesberger
5718d5b8b4
add tests for searchextractor
2018-07-01 16:55:48 +02:00
Christian Schabesberger
06c67763d2
remove soundcloud and make first search test work
2018-07-01 16:55:48 +02:00
Christian Schabesberger
b4544a67e8
add SearchExtractor
2018-07-01 16:55:48 +02:00
Christian Schabesberger
91aa52f65c
Merge branch 'dev'
2018-05-15 22:38:54 +02:00
Christian Schabesberger
0501a2f543
fix donation link parsing
2018-05-08 21:58:35 +02:00
Christian Schabesberger
94d1eb6f32
restructure youtube service
2018-05-08 21:19:03 +02:00
Christian Schabesberger
e5125c15f1
Merge pull request #78 from TeamNewPipe/donation
...
enable Donations for streams
2018-05-07 10:08:36 +02:00
Christian Schabesberger
08f0247d73
make UrlIdHandler actual wrapper for the url/id
...
fix tests
fix tests
2018-05-06 19:51:24 +02:00
Christian Schabesberger
b7c1f251d8
add filter to stream lists
2018-05-05 20:07:47 +02:00
Christian Schabesberger
bf1c771662
fix could not parese metadata despide available error
2018-05-04 16:11:25 +02:00
Christian Schabesberger
bad67f0bc3
Merge branch 'dev' of github.com:TeamNewPipe/NewPipeExtractor into dev
2018-04-29 21:31:45 +02:00
Christian Schabesberger
466865385e
add support for affiliate
2018-04-29 21:31:24 +02:00
Christian Schabesberger
016c2fc549
add channel donations for youtube
2018-04-08 15:58:42 +02:00
Mauricio Colli
1c8dafe4c6
Lightweight method to get the client id on SoundCloud
...
- Instead of downloading the whole ~1MB file, download just the first 16KB. With fallback to the old way in case of fail
- I've come across some technical difficulties as to how to limit the download (current implementation is too basic/simple for that), but in the end, a simple http header (Range) was enough
2018-04-07 22:12:27 -03:00
Christian Schabesberger
17e7e5a359
enable donations for streams
2018-04-07 21:36:20 +02:00
Christian Schabesberger
1d94e7d477
add ability to get donation links from channel
2018-04-07 16:45:07 +02:00
Mauricio Colli
77a74b8d20
Add "originalUrl" field to Info's object
2018-04-06 03:06:46 -03:00
Mauricio Colli
3fe9922d49
Fix failing test due to an invalid path
2018-04-06 03:06:33 -03:00
Christian Schabesberger
a6b6235644
return propper name for soundcloud kisok extractor
2018-04-01 20:27:44 +02:00
Christian Schabesberger
22c59a4895
changed implement me to SoundCloud Charts
2018-03-31 18:38:22 +02:00
Christian Schabesberger
9664658914
Merge branch 'master' of github.com:TeamNewPipe/NewPipeExtractor
2018-03-25 16:08:35 +02:00
Christian Schabesberger
f2067338db
move on to version 0.13.0, and made ListCollector only accept InfoItemExtracto
2018-03-25 16:08:14 +02:00
Mauricio Colli
27f61e049d
Add time units strings and utilities
2018-03-22 20:31:02 -03:00
Mauricio Colli
fa3974bf3a
Fix javadoc generation and some errors
...
- Generate javadoc containing all the classes from all submodules
2018-03-18 16:18:11 -03:00
Mauricio Colli
f787b375e5
Created gradle module and moved existing code to new one
2018-03-14 00:44:02 -03:00
Mauricio Colli
94e24a6e1c
Remove RunWith from tests
...
- It was causing tests to repeat
2018-03-13 23:50:57 -03:00
Mauricio Colli
d71a42d6cd
Ignore some tests due to some recent changes
2018-03-13 23:43:13 -03:00
Mauricio Colli
37f2e5cfda
Make some methods return the specific InfoItem type
...
- Some methods were returning a broader range of InfoItem types than it should be. For example, a ChannelInfo should return a List containing only StreamInfoItem, instead of the more general InfoItem.
- Renamed and changed return type of ListExtractor.getInfoItems to getInitialPage returning a InfoItemsPage, to be consistent with getPage(url)
2018-03-13 23:32:31 -03:00
Mauricio Colli
5dd2daad37
Finish transition to use getters on Info classes
...
- Renamed Collector method
2018-03-13 23:28:58 -03:00
Mauricio Colli
6c07b7851b
Fix Java 7 incompatibility
2018-03-13 23:25:23 -03:00
Mauricio Colli
29afd0c29c
Fix some bugs discovered by the new tests
2018-03-13 23:25:23 -03:00
Mauricio Colli
d9cf1fd631
Improve Channel and Playlist tests
2018-03-13 23:25:23 -03:00
Mauricio Colli
11216f361f
Fix bug and some re-structure
2018-03-13 23:24:51 -03:00
Christian Schabesberger
5e34556ac3
removed more nextStreams keywords
2018-03-13 23:24:51 -03:00
Christian Schabesberger
dc0ef3b89a
remove unused ExtractionExceptions
2018-03-13 23:24:51 -03:00
Christian Schabesberger
d812a114c5
made getPage() function be stand alone
2018-03-13 23:24:51 -03:00
Mauricio Colli
4366b7321f
Fail-fast in case of unsuccessful tests
2018-03-13 23:16:55 -03:00
Christian Schabesberger
fce324d1bc
fix playlist banner foo
2018-03-12 16:14:33 +01:00
Christian Schabesberger
b1130629bb
Merge pull request #73 from karyogamy/live
...
HLS Livestream Extraction
2018-03-06 19:39:18 +01:00
John Zhen Mo
cc3f3b8292
-Fixed Youtube page extraction on flagged / offensive content urls.
...
-Added test for flagged content url extraction.
2018-03-01 16:44:15 -08:00
John Zhen Mo
b4206479cb
-Fixed Youtube page extraction on flagged / offensive content urls.
2018-03-01 16:31:36 -08:00
John Zhen Mo
4cf4ee394f
-Fixed youtube stream type to no longer return none.
...
-Fixed search result errors on empty.
2018-02-27 18:16:29 -08:00
John Zhen Mo
837dbd6b86
-Fixed live stream info using video stream type.
2018-02-25 14:31:42 -08:00
John Zhen Mo
690f241357
-Added hls manifest url extraction.
...
-Changed manifest url fields to nonnull.
2018-02-25 13:03:51 -08:00
John Zhen Mo
e851c12230
-Removed live stream exceptions
2018-02-25 12:34:09 -08:00
Christian Schabesberger
86db415b18
architecture refacturing
2018-02-24 22:20:50 +01:00
Mauricio Colli
9dfcb3be06
Implement SubscriptionExtractor for services
...
- YouTube supports extracting from an export (from subscriptions manager)
- SoundCloud supports extracting the "followings" from an user
2018-02-22 11:52:38 -03:00
Christian Schabesberger
a99f466c28
Merge pull request #72 from wb9688/extractor-to-parsing
...
Move SoundcloudExtractorHelper methods to SoundcloudParsingHelper
2018-02-21 10:45:21 +01:00
wb9688
c08cef9331
Move SoundcloudExtractorHelper methods to SoundcloudParsingHelper
2018-02-21 09:23:57 +01:00
Christian Schabesberger
7716b14378
Merge pull request #70 from wb9688/fix-okhttp
...
Fix OkHttp
2018-02-20 13:46:00 +01:00
wb9688
4ac70e6966
Fix OkHttp
2018-02-15 14:23:45 +01:00
Christian Schabesberger
232b8699d6
add contribution notes to extractor
2018-02-12 20:34:24 +01:00
Christian Schabesberger
e51bc58a85
fix compatibility issue
2018-02-12 20:23:43 +01:00
Christian Schabesberger
09aca7cf90
Merge pull request #68 from TeamNewPipe/hook
...
add support for hooktube links
2018-02-12 20:06:06 +01:00
Christian Schabesberger
055cc41834
add support for hooktube links
...
fix some links
2018-02-12 20:04:55 +01:00
Mauricio Colli
f3865445f5
Improve ServiceList class
...
- [Minor] Fix wrong arguments order in KioskInfo
2018-02-10 00:46:05 -02:00
Mauricio Colli
42411099db
Revert and fix Idea's IML file
2018-02-10 00:37:40 -02:00
Mauricio Colli
4fb49d54b5
Merge pull request #66 from karyogamy/subtitles_fix
...
Subtitles Extraction NPE Fix
2018-02-09 00:42:25 -02:00
John Zhen Mo
ccc8cde80e
-Fix subtitles extraction NPE on certain video that contains no caption info.
...
-Moved subtitles to optional data extraction.
2018-02-06 10:45:58 -08:00
John Zhen Mo
7fd21ec085
-Modified subtitles to use locale rather than language code.
...
-Fixed locale parsing for youtube.
2018-02-03 13:53:03 -08:00
John Zhen Mo
02dd281395
-Added display name to subtitles that resolves language code into locale.
2018-02-03 13:28:28 -08:00
John Zhen Mo
502fd53cc4
-Refactored subtitle storing to reduce redundancy.
2018-02-01 23:24:22 -08:00
John Zhen Mo
c1cfb2b223
-Fixed url format property.
...
-Removed unused code.
2018-02-01 23:03:06 -08:00
John Zhen Mo
5b79ef3557
-Fixed extraction to only output manual and autogenerated scripts, omitting autotranslated scripts.
2018-02-01 22:51:53 -08:00
John Zhen Mo
aafe543334
-Removed new method in favor of current method.
...
-Fixed current method to output all available captions.
-Fixed tests.
2018-02-01 21:48:34 -08:00
John Zhen Mo
b3699cdcfc
-Fixed tests
...
-Simplified subtitle extraction url
2018-02-01 19:07:57 -08:00
John Zhen Mo
1f1bbaad57
-Fixed subtitles extraction to use method from youtube-dl.
...
-Expose subtitles during extraction.
-Make subtitle lists return nonnull empty collections instead of null.
2018-02-01 13:27:22 -08:00
Mauricio Colli
0061131d39
Merge pull request #60 from wb9688/fix-m
...
Fix mobile SoundCloud links
2018-01-21 05:11:30 -02:00
wb9688
6453805288
Fix mobile SoundCloud links
2018-01-20 18:48:15 +01:00