The ANDROID client was only fetched for video contents, where it can be useful on ended/post livestreams, if the n parameter of the WEB client cannot be decrypted, to avoid throttling issues (because the WEB client was only used before for ended/post livestreams).
It also provides an exclusive 48kbps M4A audio format in the adaptiveFormats array of the JSON player response, like other mobile clients (which can be also extracted from the response of the DASH manifest URL returned into the WEB client player's response, but the DASH manifest is not used by the extractor).
A note about non-fatality of fetching or parsing issues of the ANDROID and IOS clients has been added.
These values were only set before for video streams.
A fallback for the audio channels count has been added, in order to prevent exceptions when generating DASH manifests of audio streams: the fallback value is 2, because most audio streams on YouTube have 2 audio channels.
This change allows to build DASH manifests using YoutubeDashManifestCreator with the real duration of streams and prevent potential cuts of the end of progressive streams, because the duration in YouTube's player response is in seconds and not milliseconds.
This commits reverts a new behavior introduced in this branch, which only applied the decryption if needed on streams from the WEB client.
Also fix rebase issues and documentations style in YoutubeDashManifestCreator.
This parameter is still used to get the initialization sequence of OTF and POST-live streams, but is not returned anymore in the manifests.
It has been removed in order to avoid fingerprinting based on the number sent (e.g. when starting to play a stream close to the end and using 123 as the request number where it should be 1) and should be added dynamically by clients in their requests.
The relevant test has been also updated.
Checkstyle issues in YoutubeDashManifestCreator have been fixed, and the changes in the resolution string returned for video streams in YoutubeStreamExtractor have been reverted, as they create issues on NewPipe right now.
Extract post live DVR streams as post live streams instead of live streams.
A new class has been in order to improve code: ItagInfo, which stores an itag, the content (URL) extracted and if its an URL or not.
A functional interface has been added in order to abstract the stream building: StreamBuilderHelper.
Also add the cver parameter added by the desktop web client on the corresponding streams (a new method has been added in YoutubeParsingHelper to check this and another for Android streams).
Some code in these classes has been also refactored/improved/optimized.
* Fixed obvious sonar(lint) warnings
* Abstracted some code (get*Streams)
* Used some new lines to make code better readable
* Chopped down brace-jungle in some methods
* Use StandardCharset (Java 8 4tw)
Use the TV embedded client technique to get streams of embeddable age-restricted videos.
This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.
The outdated code for these contents has been removed.
Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.
Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
Also move the iPhone device machine id to a constant, explain how it is used and move the licence in the header of the file, and fix missing imports in YoutubeStreamExtractor (due to a rebase issue).
InnerTube responses return pretty printed responses, which increase responses' size for nothing.
By using the prettyPrint parameter on requests and setting its value to false, responses are not pretty printed anymore, which reduces responses size, and so data transfer and processing times.
This usage has been recently deployed by YouTube on their websites.
The iOS client is only enabled for livestreams and the Android client is now only enabled for videos, both by default.
A way to force, or not, the fetch of both clients have been added with two new static methods in YoutubeStreamExtractor.
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.
For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.
Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).
For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.
This commit also fixes a small bug with the Android User-Agent string.
Some code improvements have been also made.
That is, basically where the overriding function was missing an annotation from the base method.
Also apply renaming of emptyDescription to EMPTY_DESCRIPTION
Note: genre mixes already worked, now they are just considered as such in various video id extraction and in related items
Note 2: now extracting a mix id from a *normal* youtube mix id will fail if the video id wouldn't be exactly 11 characters long
It is a collector that can handle many extractor types, to be used when a list contains items of different types (e.g. search). It was renamed from InfoItemsSearchCollector so that it can now be used not just for search but for any extractor needing it. It supports, streams, channels, playlists and *mixes*.
ITEM_COUNT_UNKNOWN is returned when the JSON array which contains usally the number of videos is less than 3 items.
Also apply the same type of optimizations done in other PlaylistExtractors in YoutubePlaylistExtractor.
Also fix some issues in the extractors, remove uneeded overrides, use the Java 8 Stream API where possible and replace usages of Utils.UTF_8 with StandardCharsets.UTF_8 in these classes.
AudricV
Stypox
TiA4f8R
litetex
XiangRongLin
FireMasterK
TobiGr
Mohammed Anas