Commit Graph

25 Commits

Author SHA1 Message Date
Stypox 401082abe4
[YouTube] Extract playlist type in playlist extractor 2022-03-19 10:48:12 +01:00
Stypox f19660e7d9
[YouTube] Deduplicate code extracting video id from mix id 2022-03-19 10:46:30 +01:00
Stypox 50db871d89
[YouTube] Extract mixes from streams related items 2022-03-19 10:44:06 +01:00
TiA4f8R c7757c0994
Apply requested changes 2022-03-16 20:14:08 +01:00
TiA4f8R 58a247907e
Apply changes in all playlist extractors except YoutubePlaylistExtractor
Also fix some issues in the extractors, remove uneeded overrides, use the Java 8 Stream API where possible and replace usages of Utils.UTF_8 with StandardCharsets.UTF_8 in these classes.
2022-03-16 19:18:57 +01:00
FireMasterK 2eeb0a3403
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:

- the desktop API is fetched.

If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status

If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.

If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.

We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.

Other code changes:

- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed

Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-08-01 12:39:03 +02:00
TiA4f8R 7753556e66
Adress the last requested changes + update YoutubeCommentsExtractor mocks 2021-08-01 12:39:03 +02:00
TiA4f8R 4299d806a2
Adress changes 2021-08-01 12:38:59 +02:00
TiA4f8R 0f9e9b8b4b
Use the youtubei API for YouTube mixes + update the corresponding test + do some improvements
Use the youtubei API for YouTube mixes. The corresponding has been updated because the new API breaks the tests of YoutubeMixPlaylistExtractorTest.
Remove some deprecated code (the old search code with the pbj JSON) and do some other improvements.
2021-08-01 12:38:37 +02:00
TobiGr 883f16e0ad [YouTube] Set CONSENT cookie 2021-04-07 14:11:37 +02:00
TobiGr 070a40e181 [YouTube] Mix Playlist - Fix getting name 2021-03-25 21:47:16 +01:00
XiangRongLin 03b00ff1d6 Move toJsonArray and toJsonObject to JsonUtils 2021-03-04 18:58:51 +01:00
bopol 1a322ad8ed Add uploader verified by service extraction 2021-02-18 16:36:40 +01:00
Xiang Rong Lin a338e4e08e [Youtube] Apply review suggestions and avoid channel mix edge case 2020-12-12 20:32:43 +01:00
Xiang Rong Lin 22d2f7e400 [Youtube] Add cookies to youtube mix request
This way youtube wont return duplicates when getting more items of the mix (but youtube can also track us)
2020-12-12 20:32:40 +01:00
Stypox 421935401f [Youtube] Add subchannel functions to mix and fix imports 2020-12-12 20:30:57 +01:00
Xiang Rong Lin 3ff8619bcc [Youtube] apply wb9688 suggestion (mix)
Channel mix adjusments and test
Don't accept youtube music mix urls as playlist
Don't override playlistData to keep getInitialPage()
Remove json constants
Indentation
2020-12-12 20:30:57 +01:00
Xiang Rong Lin 822cf307f7 [Youtube] Add _ITEMS constants and improve code style
Move thumbnail id exctraction code to getThumbnailUrlFromId
Add test for "My mix" detection to service tests
Use ITEM_COUNT_UNKNOWN everywhere instead of -1 and add some tests
2020-12-12 20:30:57 +01:00
Xiang Rong Lin df38b1926c [Youtube] Add tests and take thumbnail image always from first video of mix
Also fix getThumbnailUrl for "My Mix"
2020-12-12 20:30:57 +01:00
Xiang Rong Lin 68a3948af6 [Youtube] Fix get banner url 2020-12-12 20:30:57 +01:00
Xiang Rong Lin ec6b99c082 [Youtube] Adjust mix extractor to new user agent
Also extract continuation now
2020-12-12 20:30:57 +01:00
Xiang Rong Lin 0ff054acb4 [Youtube] Extract initial playlist info 2020-12-12 20:30:57 +01:00
Xiang Rong Lin 327a5730a8 [Youtube] Add some comments to mix 2020-12-12 20:30:57 +01:00
Xiang Rong Lin d74265c846 [Youtube] Extract getThumbnailUrl into method and change getUploaderName 2020-12-12 20:30:57 +01:00
Xiang Rong Lin 0efb854d27 [Youtube] Implement mix extractor for auto-generated playlists.
-New YoutubeMixPlaylistExtractor, that extracts from a mix (auto-generated playlist).
-The url has the format of "youtube.com/watch?v=videoID&playlistID",
where playlistID always starts with "RD" and usually followed by the videoID.
-Change YoutubePlaylistLinkHandlerFactory to create a linkhandler with the given url if it is a mix.
-Change YoutubeService to return YoutubeMixPlaylistExtractor if the url is a mix.
2020-12-12 20:30:57 +01:00