Commit Graph

575 Commits

Author SHA1 Message Date
TiA4f8R 287d1dfd63
[SoundCloud] Use the HLS delivery method for all streams and extract only a single stream URL from HLS manifest for MP3 streams
SoundCloud broke the workaround used to get a single file from HLS manifests for Opus manifests, but it still works for MP3 ones.

The code has been adapted to prevent an unneeded request (the one to the Opus HLS manifest) and the HLS delivery method is now used for SoundCloud MP3 and Opus streams, plus the progressive one (for tracks which have a progressive stream (MP3) and for the ones which doesn't have one, it is still used by trying to get a progressive stream, using the workaround).

Streams extraction has been also moved to Java 8 Stream's API and the relevant test has been also updated.
2022-05-29 19:08:18 +02:00
TiA4f8R f7b1515290
[YouTube] Refactor DASH manifests creation
Move DASH manifests creation into a new subpackage of the YouTube package, dashmanifestcreators.
This subpackage contains:

- CreationException, exception extending Java's RuntimeException, thrown by manifest creators when something goes wrong;
- YoutubeDashManifestCreatorsUtils, class which contains all common methods and constants of all or a part of the manifest creators;
- a manifest creator has been added per delivery type of YouTube streams:
  - YoutubeProgressiveDashManifestCreator, for progressive streams;
  - YoutubeOtfDashManifestCreator, for OTF streams;
  - YoutubePostLiveStreamDvrDashManifestCreator, for post-live DVR streams (which use the live delivery method).

Every DASH manifest creator has a getCache() static method, which returns the ManifestCreatorCache instance used to cache results.

DeliveryType has been also extracted from the YouTube DASH manifest creators part of the extractor and moved to the YouTube package.

YoutubeDashManifestCreatorTest has been updated and renamed to YoutubeDashManifestCreatorsTest, and YoutubeDashManifestCreator has been removed.

Finally, several documentation and exception messages fixes and improvements have been made.
2022-05-28 12:00:56 +02:00
Stypox cfc13f4a6f
[YouTube] Reduce exception generation code and move several attributes of MPD documents into constants
Rename YoutubeDashManifestCreationException to CreationException.
Also use these constants in YoutubeDashManifestCreatorTest.
2022-05-28 12:00:53 +02:00
Stypox 5c83409039
[YouTube] Rewrite manifest test and rename long methods 2022-05-28 12:00:51 +02:00
TiA4f8R 436ddde29f
Use assertThrows in YoutubeDashManifestCreatorTest 2022-05-28 12:00:48 +02:00
TiA4f8R 2fb1a412a6
Fix Checkstyle issues, revert resolution string changes for YouTube video streams and don't return the rn parameter in DASH manifests
This parameter is still used to get the initialization sequence of OTF and POST-live streams, but is not returned anymore in the manifests.
It has been removed in order to avoid fingerprinting based on the number sent (e.g. when starting to play a stream close to the end and using 123 as the request number where it should be 1) and should be added dynamically by clients in their requests.
The relevant test has been also updated.

Checkstyle issues in YoutubeDashManifestCreator have been fixed, and the changes in the resolution string returned for video streams in YoutubeStreamExtractor have been reverted, as they create issues on NewPipe right now.
2022-05-28 12:00:47 +02:00
TiA4f8R aa4c10e751
Improve documentation and adress most of the requested changes
Also fix some issues in several places, in the code and the documentation.
2022-05-28 12:00:46 +02:00
TiA4f8R 6985167e63
Add tests for YoutubeDashManifestCreator and ManifestCreatorCache
The test added in YoutubeDashManifestCreator uses a video of the Creative Commons channel, licenced under the Creative Commons Attribution licence (reuse allowed).
Also remove public keywords of tests in UtilsTest, as suggested by SonarLint, because they are not needed with Junit 5.
2022-05-28 12:00:46 +02:00
TiA4f8R f6ec7f9a61
Update DefaultStreamExtractorTest and SoundcloudStreamExtractorTest to support changes made in Stream classes 2022-05-28 12:00:45 +02:00
Fynn Godau c38c016de5 Bandcamp: extract stream length 2022-04-24 21:24:19 +02:00
TiA4f8R b30e341559
Add link to learn more about the issue which makes YoutubeStreamExtractorRelatedMixTest.testRelatedItems test disabled 2022-04-04 20:36:59 +02:00
TiA4f8R 70812fa611
Update YouTube stream mocks and disable YoutubeStreamExtractorRelatedMixTest
Mixes seems to be not given by YouTube anymore if you use a PENDING consent cookie value.
As mocks needs to updated, the test is always failing because of this change.
2022-04-02 19:28:49 +02:00
TiA4f8R 3d38459cf3
[YouTube] Reduce InnerTube response sizes by adding the prettyPrint parameter with the false value
InnerTube responses return pretty printed responses, which increase responses' size for nothing.

By using the prettyPrint parameter on requests and setting its value to false, responses are not pretty printed anymore, which reduces responses size, and so data transfer and processing times.
This usage has been recently deployed by YouTube on their websites.
2022-03-27 20:52:40 +02:00
litetex 349ba8db7f
Improve tests and randomness
- Use the existing RNG inside YoutubeParsingHelper
- Deduplicated test-setup for YouTube tests
- Minor improvements
2022-03-27 20:52:38 +02:00
TiA4f8R 52376949e5
Add setSeedForVideoTests method in YoutubeStreamExtractor tests
In order to use still use mocks with the generation of random strings in player requests, we need to use YoutubeParsingHelper.setSeedForVideoTests() method in every stream test.
2022-03-27 20:51:39 +02:00
litetex 29408f9356 ``YoutubeMusicSearchExtractorTest$Suggestion`` has no mock data 2022-03-26 20:52:28 +01:00
litetex fcee247f9f Switched to mockonly tests
* Made the reason why the tests are mockonly mandatory
2022-03-26 20:52:28 +01:00
litetex 6a6c9359af Fixed kurzgesagt test - They changed their description... 2022-03-26 20:52:27 +01:00
litetex 26596215fa Refactored ``MediaCCCRecentListExtractorTest`` 2022-03-26 20:52:26 +01:00
litetex 53962bfd7d Disabled suggestion test for now
Has to be done once YT's backed works again...
2022-03-26 20:52:26 +01:00
litetex 164e21b5af Fixed ``MediaCCCRecentKiosk``
Ignore faulty data/items (with duration <= 0)
2022-03-26 20:52:23 +01:00
litetex 49dfdae5d2 Removed useless throw declaration 2022-03-26 20:51:38 +01:00
litetex 69a58fd3d1 Don't use sysout 2022-03-26 20:51:37 +01:00
litetex 2bd4299563 Fixed test: Peertube - Use new comments and videos
Old comments/videos got deleted
2022-03-26 20:49:39 +01:00
litetex 606a386a98 Fix tests: What's peertube has less comments now 2022-03-26 20:49:39 +01:00
litetex 4d1a1c8fb8 Better test for ``MediaCCCRecentListExtractorTest``
* Use assertAll
* Show which item is affected
2022-03-26 20:49:38 +01:00
litetex ba43dbaa28 Fix Tests: The channel lost abos 2022-03-26 20:49:38 +01:00
litetex ff436e5740 Fixed test: YT mix playlist 2022-03-26 20:49:38 +01:00
Stypox 09ddb6adbb
[YouTube] Add MockOnly to method testing mixes in related items 2022-03-19 10:54:38 +01:00
Stypox 8201b3b90e
[YouTube] Parse any playlist (including music mixes) in related items 2022-03-19 10:48:13 +01:00
Stypox 13f7900816
[YouTube] Add test for genre mix 2022-03-19 10:48:13 +01:00
Stypox 279f3a20fe
[YouTube] Fix mix tests with invalid video ids
Replaces mix tests based on a strange mix type RDQM{videoId} (only reference I could find is https://github.com/ytdl-org/youtube-dl/issues/26228) and with an invalid video id of 13 characters (the first two characters were QM, but even after removing QM there still wasn't a video available at that id).
Also updates mocks.
2022-03-19 10:48:13 +01:00
Stypox d660c04838
[YouTube] Also test playlist type in playlist tests 2022-03-19 10:48:13 +01:00
Stypox 34a4484c72
[YouTube] Add test for a video with a mix in related items 2022-03-19 10:44:06 +01:00
Stypox d8f2031619
Merge pull request #816 from Stypox/mock-only-extension
Add `@MockOnly` Junit 5 extension
2022-03-19 10:40:38 +01:00
litetex cc2e4d7104
Merge pull request #815 from litetex/fix-soundcloud-id-once-and-for-all
Removed hardcoded soundcloud HARDCODED_CLIENT_ID
2022-03-17 13:54:08 +01:00
TiA4f8R 35e082248e
Fix YouTube and SoundCloud playlists tests 2022-03-16 19:40:30 +01:00
Stypox 0c37c75981
Make getDownloader static & extract getDownloaderType 2022-03-16 17:22:42 +01:00
Stypox 40aa5104b1
Merge pull request #786 from XiangRongLin/throttling_resilience
[Youtube] Make throttling decryption more resilient to api change
2022-03-16 11:03:16 +01:00
litetex ba56be8ef1 Removed hardcoded soundcloud id
It never works (long enough) so let's simply remove it...
2022-03-15 21:19:19 +01:00
litetex e7aee0ca57
Merge pull request #807 from FireMasterK/no-commentsinfo-instance
Remove the need for a CommentsInfo instance in CommentsInfo.getMoreItems and fix PeertubeCommentsExtractorTest.Default test
2022-03-15 15:06:56 +01:00
FireMasterK 60cc71e944
Remove the need for a CommentsInfo instance. 2022-03-03 11:48:41 +00:00
FireMasterK 6950b362f2
Add test values for failing tests. 2022-02-24 12:55:13 +00:00
FireMasterK 5b0ec694a6
Add requested changes. 2022-02-24 12:50:51 +00:00
FireMasterK 6f3f608ab6
Add support for extracting channel subscriber count in StreamInfo. 2022-02-24 12:50:51 +00:00
XiangRongLin 545522b80f [Youtube] Make throttling decryption more resilient to api change 2022-02-01 19:57:34 +01:00
Stypox 59e0aeba88
Merge pull request #768 from litetex/junit-5
JUnit 5
2022-01-24 16:56:16 +01:00
Mohammed Anas 65df39b3e3
[YouTube] parse timestamps with >3 digits correctly (#775)
* [YouTube] parse timestamps with >3 digits correctly

Fixes https://github.com/TeamNewPipe/NewPipe/issues/7530; check the
issue for details.

* Remove outdated comment
2022-01-12 10:45:06 +01:00
litetex a8ebd2cead Update mock resources for YT Mix playlist 2022-01-04 17:39:14 +01:00
litetex 066afd8629 Restored test 2022-01-04 17:33:08 +01:00