Commit Graph

2267 Commits

Author SHA1 Message Date
Stypox 8c5f014a6f
Merge pull request #869 from mhmdanas/add-workflow-permissions
Use minimum required permissions for GitHub workflows
2022-07-13 18:58:32 +02:00
Stypox 954a294e27
Merge pull request #870 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.2
Bump jsoup from 1.15.1 to 1.15.2
2022-07-13 17:48:06 +02:00
Mohammed Anas f57049d2c0
Use temurin instead of adopt (#868)
See
https://blog.adoptopenjdk.net/2021/08/goodbye-adoptopenjdk-hello-adoptium/.
2022-07-10 18:38:08 +03:00
Stypox 5bd7bf20cc
Merge pull request #867 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.1
Bump spotbugs-annotations from 4.7.0 to 4.7.1
2022-07-06 14:23:44 +02:00
Stypox 5ab74b3631
Merge pull request #857 from FireMasterK/video-title
Get original untranslated title for YouTube
2022-07-06 10:26:45 +02:00
dependabot[bot] 122365005a
Bump jsoup from 1.15.1 to 1.15.2
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.15.1 to 1.15.2.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.1...jsoup-1.15.2)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-07-04 09:38:16 +00:00
mhmdanas 414186cff5 Use minimum required permissions for GitHub workflows
This reduces the attack surface if the workflows are ever compromised.
2022-07-03 23:55:28 +03:00
dependabot[bot] e9b4be3e3c
Bump spotbugs-annotations from 4.7.0 to 4.7.1
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.7.0 to 4.7.1.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.0...4.7.1)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-06-28 09:15:07 +00:00
Stypox 5219a705ba
Merge pull request #864 from AudricV/fetch-android-client-for-ended/post-livestreams
[YouTube] Fetch the ANDROID client for ended/post livestreams
2022-06-24 14:55:20 +02:00
AudricV 090debd83b
[YouTube] Fetch the ANDROID client for ended/post livestreams
The ANDROID client was only fetched for video contents, where it can be useful on ended/post livestreams, if the n parameter of the WEB client cannot be decrypted, to avoid throttling issues (because the WEB client was only used before for ended/post livestreams).

It also provides an exclusive 48kbps M4A audio format in the adaptiveFormats array of the JSON player response, like other mobile clients (which can be also extracted from the response of the DASH manifest URL returned into the WEB client player's response, but the DASH manifest is not used by the extractor).

A note about non-fatality of fetching or parsing issues of the ANDROID and IOS clients has been added.
2022-06-21 18:53:49 +02:00
litetex a26bcc55c4
Merge pull request #845 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.0
Bump spotbugs-annotations from 4.6.0 to 4.7.0
2022-06-19 15:36:55 +02:00
dependabot[bot] 424eb1c559
Bump spotbugs-annotations from 4.6.0 to 4.7.0
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.6.0 to 4.7.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.6.0...4.7.0)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-06-19 13:29:18 +00:00
litetex 2712c3d788
Merge pull request #847 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.1
Bump jsoup from 1.14.3 to 1.15.1
2022-06-19 15:28:43 +02:00
dependabot[bot] 281d2b9f81
Bump jsoup from 1.14.3 to 1.15.1
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.14.3 to 1.15.1.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.14.3...jsoup-1.15.1)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-06-19 13:15:40 +00:00
litetex f775155d25
Merge pull request #846 from litetex/remove-unused-methods
Remove unused methods
2022-06-19 15:12:15 +02:00
litetex 9d625dd75f
Merge pull request #860 from gliptak/patch-1
Correct PeertubePlaylistExtractorTest unit tests
2022-06-19 15:09:27 +02:00
Gábor Lipták 28c9340d69
Correct unit tests 2022-06-18 11:18:38 -04:00
Stypox 1b51eab664
Merge pull request #859 from AudricV/delivery-methods-fixes-and-improvements
Fix extraction of some properties in ItagItems for YouTube livestreams and post-live streams and remove completely SoundCloud HLS workaround
2022-06-17 15:46:13 +02:00
AudricV 301a795ed3
[SoundCloud] Remove completely workaround for HLS streams
SoundCloud is currently removing this workaround completely, so there is no need to keep it, because it impacts the loading time (a HLS playlist was downloaded and parsed).
2022-06-16 12:12:54 +02:00
AudricV e960a417ec
[YouTube] Fix extraction of fps, audioSampleRate and audioChannels fields for ItagItems of live streams and post live streams
These values were only set before for video streams.

A fallback for the audio channels count has been added, in order to prevent exceptions when generating DASH manifests of audio streams: the fallback value is 2, because most audio streams on YouTube have 2 audio channels.
2022-06-16 12:12:54 +02:00
Stypox c8a77da2ab
Merge pull request #810 from TiA4f8R/delivery-methods-v2
Support delivery methods other than progressive HTTP
2022-06-02 22:44:24 +02:00
Kavin 7635aeed2c
Get original untranslated title for YouTube. 2022-06-02 09:57:52 +01:00
TiA4f8R 287d1dfd63
[SoundCloud] Use the HLS delivery method for all streams and extract only a single stream URL from HLS manifest for MP3 streams
SoundCloud broke the workaround used to get a single file from HLS manifests for Opus manifests, but it still works for MP3 ones.

The code has been adapted to prevent an unneeded request (the one to the Opus HLS manifest) and the HLS delivery method is now used for SoundCloud MP3 and Opus streams, plus the progressive one (for tracks which have a progressive stream (MP3) and for the ones which doesn't have one, it is still used by trying to get a progressive stream, using the workaround).

Streams extraction has been also moved to Java 8 Stream's API and the relevant test has been also updated.
2022-05-29 19:08:18 +02:00
Stypox b3c620f0d8
Apply code review and Streams rework 2022-05-28 12:00:58 +02:00
Stypox d652e05874
[MediaCCC] Fix comments about containsSimilarStream 2022-05-28 12:00:58 +02:00
Stypox 044639c32b
Solve some review comments 2022-05-28 12:00:57 +02:00
litetex c33d392958
Fixed typo XEE → XXE (Xml eXternal Entity attack)
See also:
https://en.wikipedia.org/wiki/XML_external_entity_attack
https://owasp.org/www-community/vulnerabilities/XML_External_Entity_(XXE)_Processing
2022-05-28 12:00:57 +02:00
TiA4f8R fffbbee7f3
[YouTube] Add missing Nonnull annotations in getCache method of YouTube DASH manifest creators 2022-05-28 12:00:56 +02:00
TiA4f8R f7b1515290
[YouTube] Refactor DASH manifests creation
Move DASH manifests creation into a new subpackage of the YouTube package, dashmanifestcreators.
This subpackage contains:

- CreationException, exception extending Java's RuntimeException, thrown by manifest creators when something goes wrong;
- YoutubeDashManifestCreatorsUtils, class which contains all common methods and constants of all or a part of the manifest creators;
- a manifest creator has been added per delivery type of YouTube streams:
  - YoutubeProgressiveDashManifestCreator, for progressive streams;
  - YoutubeOtfDashManifestCreator, for OTF streams;
  - YoutubePostLiveStreamDvrDashManifestCreator, for post-live DVR streams (which use the live delivery method).

Every DASH manifest creator has a getCache() static method, which returns the ManifestCreatorCache instance used to cache results.

DeliveryType has been also extracted from the YouTube DASH manifest creators part of the extractor and moved to the YouTube package.

YoutubeDashManifestCreatorTest has been updated and renamed to YoutubeDashManifestCreatorsTest, and YoutubeDashManifestCreator has been removed.

Finally, several documentation and exception messages fixes and improvements have been made.
2022-05-28 12:00:56 +02:00
TiA4f8R f17f7b9842
Apply requested changes in YoutubeParsingHelper 2022-05-28 12:00:55 +02:00
TiA4f8R 301b9fa024
Remove hashCode and equals methods overrides of Stream classes 2022-05-28 12:00:55 +02:00
TiA4f8R 2f3920c648
[YouTube] Return approxDurationMs value from YouTube's player response in ItagItems generated
This change allows to build DASH manifests using YoutubeDashManifestCreator with the real duration of streams and prevent potential cuts of the end of progressive streams, because the duration in YouTube's player response is in seconds and not milliseconds.
2022-05-28 12:00:54 +02:00
TiA4f8R 4158fc46a0
[Bandcamp] Fix regression of Opus radio streams extraction
When moving opus-lo into a constant, opus-lo was renamed to opus_lo and was only used if no MP3 stream was available (which was not the case before the changes in BandcampRadioStreamExtractor related to the addition of the support of all delivery methods), so these changes removed the ability to get Opus streams of Bandcamp radios.

This commit reverts this unwanted change.
2022-05-28 12:00:54 +02:00
TiA4f8R 54d323c2ae
Fix Checkstyle issue in YoutubeDashManifestCreator 2022-05-28 12:00:53 +02:00
Stypox 2321822844
Rename Stream's baseUrl to manifestUrl 2022-05-28 12:00:53 +02:00
Stypox cfc13f4a6f
[YouTube] Reduce exception generation code and move several attributes of MPD documents into constants
Rename YoutubeDashManifestCreationException to CreationException.
Also use these constants in YoutubeDashManifestCreatorTest.
2022-05-28 12:00:53 +02:00
Stypox 00bbe5eb4d
[YouTube] Make exception messages smaller 2022-05-28 12:00:52 +02:00
Stypox 4da05afe11
[YouTube] Inline collectSegmentsData in YoutubeDashManifestCreator 2022-05-28 12:00:52 +02:00
Stypox 3708ab9ed5
[YouTube] Refactor YoutubeDashManifestCreator
- Remove all of the methods used to access caches and replace them with three caches getters
- Rename caches to shorter and more meaningful names
- Remove redundant @throws tags that just say "if this method fails to do what it should do", which is obvious
2022-05-28 12:00:51 +02:00
Stypox 5c83409039
[YouTube] Rewrite manifest test and rename long methods 2022-05-28 12:00:51 +02:00
Stypox 8226fd044f
[YouTube] Suppress Sonar security warning for XEE 2022-05-28 12:00:50 +02:00
Stypox ba68b8c014
[YouTube] Secure DashManifestCreator against XEE attack
Also remove duplicate lines from javadoc comments, otherwise checkstyle complaints.
XEE prevention is inspired from https://github.com/ChuckerTeam/chucker/pull/201/files
For an explanation of XEE see https://rules.sonarsource.com/java/RSPEC-2755, though the solution reported there causes crashes on Android
2022-05-28 12:00:50 +02:00
Stypox 159d05c91b
Remove unused DashMpdParser (but kept in git history) 2022-05-28 12:00:49 +02:00
Stypox 50272db946
Apply reviews: improve comments, remove FILE, remove Stream#equals(Stream) 2022-05-28 12:00:49 +02:00
TiA4f8R 07b045f20d
[YouTube] Support the iOS client in YoutubeDashManifestCreator and decrypt again the n parameter, if present, for all clients
This commits reverts a new behavior introduced in this branch, which only applied the decryption if needed on streams from the WEB client.
Also fix rebase issues and documentations style in YoutubeDashManifestCreator.
2022-05-28 12:00:48 +02:00
TiA4f8R 436ddde29f
Use assertThrows in YoutubeDashManifestCreatorTest 2022-05-28 12:00:48 +02:00
TiA4f8R d64d7bbd01
Move ManifestCreatorCache tests to a separate class and remove override of equals and hashCode methods in ManifestCreatorCache
These methods don't need to be overriden, as they are not excepted to be used in collections.
Also improve the toString method of this class, which contains also now clearFactor and maximumSize attributes and for each operations.
2022-05-28 12:00:48 +02:00
TiA4f8R 2fb1a412a6
Fix Checkstyle issues, revert resolution string changes for YouTube video streams and don't return the rn parameter in DASH manifests
This parameter is still used to get the initialization sequence of OTF and POST-live streams, but is not returned anymore in the manifests.
It has been removed in order to avoid fingerprinting based on the number sent (e.g. when starting to play a stream close to the end and using 123 as the request number where it should be 1) and should be added dynamically by clients in their requests.
The relevant test has been also updated.

Checkstyle issues in YoutubeDashManifestCreator have been fixed, and the changes in the resolution string returned for video streams in YoutubeStreamExtractor have been reverted, as they create issues on NewPipe right now.
2022-05-28 12:00:47 +02:00
TiA4f8R f61e2092a1
[YouTube] Return a copy of the hardcoded ItagItem instead of returning the reference to the hardcoded one in ItagItem.getItag
To do so, a copy constructor has been added in the class.

This fixes, for instance, an issue in NewPipe, in which the ItagItem values where not the ones corresponsing to a stream but to another, when generating DASH manifests.
2022-05-28 12:00:47 +02:00
TiA4f8R aa4c10e751
Improve documentation and adress most of the requested changes
Also fix some issues in several places, in the code and the documentation.
2022-05-28 12:00:46 +02:00