Commit Graph

72 Commits

Author SHA1 Message Date
Stypox 8c1041def6
Add @ null annotations where Android Studio suggested it
That is, basically where the overriding function was missing an annotation from the base method.

Also apply renaming of emptyDescription to EMPTY_DESCRIPTION
2022-03-26 22:07:14 +01:00
Stypox 740a37a2de [YouTube] Fix checkstyle issues 2022-03-26 19:42:40 +01:00
Stypox 638da1756c
[Mix] Create MultiInfoItemsCollector
It is a collector that can handle many extractor types, to be used when a list contains items of different types (e.g. search). It was renamed from InfoItemsSearchCollector so that it can now be used not just for search but for any extractor needing it. It supports, streams, channels, playlists and *mixes*.
2022-03-19 10:44:06 +01:00
FireMasterK 2eeb0a3403
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:

- the desktop API is fetched.

If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status

If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.

If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.

We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.

Other code changes:

- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed

Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-08-01 12:39:03 +02:00
TiA4f8R 4299d806a2
Adress changes 2021-08-01 12:38:59 +02:00
TiA4f8R 0f9e9b8b4b
Use the youtubei API for YouTube mixes + update the corresponding test + do some improvements
Use the youtubei API for YouTube mixes. The corresponding has been updated because the new API breaks the tests of YoutubeMixPlaylistExtractorTest.
Remove some deprecated code (the old search code with the pbj JSON) and do some other improvements.
2021-08-01 12:38:37 +02:00
TiA4f8R 3017dde67e
Fix some typos 2021-08-01 12:38:36 +02:00
TiA4f8R f73c923f60
Don't use the youtubei.googleapis.com but the websites domains + update client version of the desktop internal API
Use again www.youtube.com and music.youtube.com domains instead of youtubei.googleapis.com domain because it spoofs more a web client of YouTube or YouTube Music and may reduce Google's detection of NewPipe Extractor users.
2021-08-01 12:38:34 +02:00
TiA4f8R 4d682834c3
Fix localization and update client version 2021-08-01 12:38:03 +02:00
TiA4f8R f46cfb0f26
Adress reviews and do some improvements
Adress changes requested in reviews.
Do some improvements, remove unused imports and format some code to be in the 100 characters line limit.
2021-08-01 12:38:03 +02:00
TiA4f8R b49ae547a3
Do some improvements to YoutubeStreamExtractor
Get the real name of the uploader (for autogenerated channels and music artist channels), like before the migration to the JSON pbj.
Do some other improvements, especially reformatting some code to be in the 100 characters line limit and use final where possible.
2021-08-01 12:38:01 +02:00
TiA4f8R f461224b2b
Use the youtubei API for YouTube searches + update mocks
Add getSearchParameter, a new method in YoutubeSearchQueryHandlerFactory class which returns the params field for a search, or an empty string if there is no one.
Update mocks of YoutubeSearchExtractorTest.
2021-08-01 12:36:28 +02:00
TiA4f8R a12c69da7d
Use the youtubei API for YouTube channels 2021-08-01 12:36:27 +02:00
TobiGr 1498e1905e Reduce number of type conversions in YouTubeSearchExtractor.collectStreamsFrom by introducing new variable 2021-03-31 13:30:38 +02:00
bopol 9ca52ca68a use only one constant UTF-8 2021-02-15 18:47:54 +01:00
TobiGr bc6de14952 Extract stream and search meta info for YouTube
Add method to extract Google webcache URLs.
2020-12-22 18:19:26 +01:00
wb9688 f345f667e2 Extract YouTube's key 2020-07-26 12:01:59 +02:00
wb9688 4c987a5302 Support YouTube's new continuations for search 2020-07-26 10:01:03 +02:00
wb9688 9b6fe1dea6 Throw IllegalArgumentException when Page is invalid 2020-07-06 20:19:31 +02:00
wb9688 4cc312086a Introduce Page class 2020-07-06 20:19:31 +02:00
wb9688 e3bfdba135 Remove getNextPageUrl() function from ListExtractor 2020-07-06 20:11:40 +02:00
wb9688 baf5dd3e8f
Merge pull request #311 from B0pol/fixSearchSuggestion
Search: add isCorrectedSearch() and fix YoutubeSearchExtractor#getSea…
2020-05-12 09:29:15 +02:00
bopol de26e00079 changed all == null || isEmpty() to isNullOrEmpty() 2020-05-11 11:40:24 +02:00
bopol 5760366d70 use JsonObject#isEmpty instead of JsonObject#equals(JsonUtils.EMPTY_OBJECT) 2020-05-07 15:40:41 +02:00
bopol 3757541a9c adapt new nanojson version 2020-05-07 15:40:41 +02:00
bopol e8d58e09c9 getSearchSuggestion now returns the corrected query if isCorrectedSearch() is true 2020-05-07 15:40:41 +02:00
bopol 29b639b454 Search: add isCorrectedSearch() and fix YoutubeSearchExtractor#getSearchSuggestion()
isCorrectedSearch: see the javadoc
getSearchSuggestion: in YoutubeSearchExtractor, it was giving the corrected search query. It now gives the suggested query, as it should
2020-05-07 15:39:57 +02:00
bopol bc13e0c616 Merge remote-tracking branch 'upstream/dev' into soundcloudComments 2020-05-05 14:49:21 +02:00
wb9688 979c5a7502 Use our fork of nanojson 2020-04-17 19:36:42 +02:00
bopol 636c430743 refactor: create extractors and linkHandler packages for SoundCloud, move YoutubeParsingHelper to youtube package (not linkhandler anymore) 2020-04-10 10:51:05 +02:00
wb9688 c7f7bd2442 Fix error when YT Music videos has no uploader URL 2020-04-01 10:30:51 +02:00
wb9688 ac15df4548 Move YouTube Music search to its own class 2020-04-01 10:30:51 +02:00
wb9688 cf0f2aff3e Extract uploader url from certain YouTube Music videos 2020-04-01 10:30:51 +02:00
wb9688 aa8cea47f3 Refactor YouTube Music search tests 2020-04-01 10:30:51 +02:00
wb9688 dd434cca01 Fix issue when there is no didYouMeanRenderer in itemSectionRenderer 2020-04-01 10:30:51 +02:00
wb9688 5a775a4bbe Use new way of specifying stream count 2020-04-01 10:30:51 +02:00
wb9688 d58c0f230d Improve code for YouTube Music search 2020-04-01 10:30:51 +02:00
wb9688 dc29d87962 Extract YouTube search suggestions 2020-04-01 10:30:51 +02:00
wb9688 c852b13d5a Add Referer header so that it also works with HttpsUrlConnection 2020-04-01 10:30:51 +02:00
wb9688 2b9b2a78e8 Handle 100+ items in playlist 2020-04-01 10:30:51 +02:00
wb9688 1762a527c9 Add support for YouTube Music search 2020-04-01 10:30:51 +02:00
Mauricio Colli c921e5d6a5
Remove unnecessary collector creation function from search extractor 2020-03-23 18:08:51 -03:00
Mauricio Colli 408f042127 [YouTube] Fix bug when url isn't present in the browseEndpoint object 2020-03-01 12:49:13 +01:00
wb9688 45df8248bf Use Localization in getJsonResponse() 2020-02-29 16:55:07 +01:00
wb9688 157055fbfd Create getJsonResponse() function 2020-02-29 16:42:04 +01:00
wb9688 f946310af5 Fix search queries with a promotedSparklesTextSearchRenderer returning 0 results 2020-02-28 08:57:44 +01:00
wb9688 365b0329f3 Implement getTextFromObject() function 2020-02-27 17:39:23 +01:00
wb9688 985c3ec877 Improve getClientVersion() 2020-02-26 15:27:50 +01:00
wb9688 0973263aab Use pbj in YoutubeSearchExtractor 2020-02-26 15:27:46 +01:00
wb9688 02b59903fa Remove useless code 2020-02-25 21:51:30 +01:00