lasers
/
NewPipeExtractor
mirror of https://github.com/TeamNewPipe/NewPipeExtractor
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
2,058 Commits 11 Branches 72 Tags 112 MiB
Commit Graph

548 Commits

Author SHA1 Message Date
TiA4f8R d30caeb091
Fix YouTube player JavaScript URL redirection 2020-12-27 15:55:03 +01:00
TobiGr b62144b49d [YouTube] Music: Fix search extraction caused by changed JSON response 2020-12-23 21:08:36 +01:00
TobiGr bc6de14952 Extract stream and search meta info for YouTube 
Add method to extract Google webcache URLs.
 2020-12-22 18:19:26 +01:00
Tobias Groza 22a415156f
Merge pull request #479 from vkay94/stream-segments 
Extract stream segments for YouTube
 2020-12-19 11:12:22 +01:00
Stypox 85fa006214
Merge pull request #280 from XiangRongLin/mixPL 
Extractor for youtube mix (auto-generated playlist)
 2020-12-14 18:11:07 +01:00
Xiang Rong Lin a338e4e08e [Youtube] Apply review suggestions and avoid channel mix edge case 2020-12-12 20:32:43 +01:00
Xiang Rong Lin 22d2f7e400 [Youtube] Add cookies to youtube mix request 
This way youtube wont return duplicates when getting more items of the mix (but youtube can also track us)
 2020-12-12 20:32:40 +01:00
Stypox 421935401f [Youtube] Add subchannel functions to mix and fix imports 2020-12-12 20:30:57 +01:00
Xiang Rong Lin 3ff8619bcc [Youtube] apply wb9688 suggestion (mix) 
Channel mix adjusments and test
Don't accept youtube music mix urls as playlist
Don't override playlistData to keep getInitialPage()
Remove json constants
Indentation
 2020-12-12 20:30:57 +01:00
Xiang Rong Lin 822cf307f7 [Youtube] Add _ITEMS constants and improve code style 
Move thumbnail id exctraction code to getThumbnailUrlFromId
Add test for "My mix" detection to service tests
Use ITEM_COUNT_UNKNOWN everywhere instead of -1 and add some tests
 2020-12-12 20:30:57 +01:00
Xiang Rong Lin df38b1926c [Youtube] Add tests and take thumbnail image always from first video of mix 
Also fix getThumbnailUrl for "My Mix"
 2020-12-12 20:30:57 +01:00
Xiang Rong Lin 68a3948af6 [Youtube] Fix get banner url 2020-12-12 20:30:57 +01:00
Xiang Rong Lin ec6b99c082 [Youtube] Adjust mix extractor to new user agent 
Also extract continuation now
 2020-12-12 20:30:57 +01:00
Xiang Rong Lin 0ff054acb4 [Youtube] Extract initial playlist info 2020-12-12 20:30:57 +01:00
Xiang Rong Lin 327a5730a8 [Youtube] Add some comments to mix 2020-12-12 20:30:57 +01:00
Xiang Rong Lin d74265c846 [Youtube] Extract getThumbnailUrl into method and change getUploaderName 2020-12-12 20:30:57 +01:00
Xiang Rong Lin 0efb854d27 [Youtube] Implement mix extractor for auto-generated playlists. 
-New YoutubeMixPlaylistExtractor, that extracts from a mix (auto-generated playlist).
-The url has the format of "youtube.com/watch?v=videoID&playlistID",
where playlistID always starts with "RD" and usually followed by the videoID.
-Change YoutubePlaylistLinkHandlerFactory to create a linkhandler with the given url if it is a mix.
-Change YoutubeService to return YoutubeMixPlaylistExtractor if the url is a mix.
 2020-12-12 20:30:57 +01:00
vkay94 2ba27b39af Stream segments: Remove replaceAll from setPreviewUrl 2020-12-12 15:00:45 +01:00
vkay94 d3eea4f4be Add stream segments to StreamInfo 2020-12-12 10:24:29 +01:00
Isira Seneviratne 4c19a88612 Use Objects.toString(). 2020-12-11 06:09:50 +05:30
vkay94 9dbacbc618 Fix comments parsing 2020-12-09 14:57:38 +01:00
XiangRongLin 5bceff0083 [YouTube] Fix extraction of next page url for the last page of playlist 2020-11-18 19:03:12 +01:00
XiangRongLin 8347e14952 [YouTube] Fix playlist continuations extraction 2020-11-17 21:04:53 +01:00
Stypox 89a77ae74a
[YouTube] Fix detection of ended livestreams and parse livestream upload date 2020-11-04 16:03:08 +01:00
Stypox 827f7bd137
[YouTube] Cache deobfuscation and improve requests made 
Fix age restriction extraction
Automatically fixes more things
 2020-11-04 16:02:12 +01:00
Isira Seneviratne fe31a90cb3 Remove DateTimeFormatter.ISO_OFFSET_DATE_TIME usage. 2020-11-04 05:49:41 +05:30
Stypox 501ec30152
Implement youtube subscription import from Google takeout 2020-11-02 11:04:48 +01:00
Isira Seneviratne 4f04cfccca Switch from Calendar to OffsetDateTime in DateWrapper. 2020-11-01 06:28:54 +05:30
Stypox 30ed4f2d63
Remove any reference to decrypt and improve error message 2020-10-28 20:26:00 +01:00
TobiGr 62912ee834 Merge branch 'master' into dev 2020-10-28 14:28:06 +01:00
bopol 6dc5ab4015 find playerUrl in another place when assetsPattern regex fails 2020-10-27 13:48:58 +01:00
bopol db0ef83d6b fix youtube decryption and three attemps bug 
fixes teamnewpipe/newpipe#4572 fixes #439
 2020-10-27 13:40:24 +01:00
bopol c1e9857960 fix subscriber count when subscribe is disabled 
fixes #305
 2020-10-26 19:06:01 +01:00
bopol a39a2cca82 fix redirect channels 2020-10-26 19:05:19 +01:00
bopol 01f49e8f66 polish strings 2020-10-26 16:32:39 +01:00
Stypox 57e7994c9e
Add some missing finals, nullables and comments 2020-10-25 08:12:41 +01:00
Stypox 68d23defba
[YouTube] Do not catch every exception on getErrorMessage 
@B0pol suggestion
 2020-10-24 18:39:55 +02:00
Stypox 3191bd6c70
[YouTube] Use final when possible 2020-10-24 18:39:55 +02:00
Stypox 3b2cfb4ca2
[SoundCloud] Return empty video stream list instead of null 
Also replace every instance of `return new ArrayList<>();` with `return Collections.emptyList();`
 2020-10-24 18:23:46 +02:00
Stypox 072bae321f
[YouTube] Fix frame extraction for livestreams 
Use saved playerResponse instead of parsing json every time
 2020-10-24 18:23:46 +02:00
Stypox 7cd410f3fc
[YouTube] Return 0 when there is no timestamp, not -2, as per javadoc 2020-10-24 18:23:45 +02:00
Stypox 7fb867c166
[YouTube] Fix error message obtaining when there is none 2020-10-24 18:22:44 +02:00
Stypox 19e862657a
[YouTube] Fix some decryption exceptions by retrying 2020-10-15 20:16:44 +02:00
wb9688 ebbfe7f6d4 Skip YouTube's OTF streams 2020-09-29 10:48:02 +02:00
wb9688 7657c2ed1a Use initSafeStandardsObjects() 2020-08-15 17:08:07 +02:00
TobiGr 0fb73301e3 [YouTube] Fix crash on empty comment 
Closes #380
 2020-08-05 18:25:40 +02:00
TobiGr 5ac80624a4 Fix detection of YouTube's decryption function 
Closes TeamNewPipe/NewPipe#3951
 2020-07-28 01:01:10 +02:00
Tobias Groza 32dff1541f
Merge pull request #370 from wb9688/fix-yt-age-restricted 
Fix age restricted YouTube videos
 2020-07-26 14:17:19 +02:00
TobiGr ec778200d9 Add comments to the code 2020-07-26 14:15:13 +02:00
wb9688 f345f667e2 Extract YouTube's key 2020-07-26 12:01:59 +02:00
wb9688 4c987a5302 Support YouTube's new continuations for search 2020-07-26 10:01:03 +02:00
wb9688 81459e289f Fix age restricted YouTube videos 2020-07-18 19:52:54 +02:00
wb9688 ff560e907a Remove getNextStream() from StreamExtractor 2020-07-15 18:48:37 +02:00
Tobias Groza a70cb0283f
Merge pull request #314 from wb9688/remove-getnextpageurl 
Next page stuff
 2020-07-07 20:49:38 +02:00
wb9688 0a5a905bc7 Add final at more places 2020-07-07 20:45:47 +02:00
wb9688 17ba8a57fa Clean up the code 2020-07-06 20:19:31 +02:00
wb9688 9b6fe1dea6 Throw IllegalArgumentException when Page is invalid 2020-07-06 20:19:31 +02:00
wb9688 4cc312086a Introduce Page class 2020-07-06 20:19:31 +02:00
wb9688 e3bfdba135 Remove getNextPageUrl() function from ListExtractor 2020-07-06 20:11:40 +02:00
wb9688 156a26b64b Support start= timestamps for YouTube 2020-07-06 14:08:40 +02:00
wb9688 98055a3c3c Revert "Add more decryption function regexes" 
This reverts commit 384d6acd5d.
 2020-05-29 18:41:40 +02:00
Tobias Groza bdb0f2ae6b
Merge pull request #332 from wb9688/learning-playlist 
Support YouTube's learning playlists
 2020-05-29 10:11:08 +02:00
Tobias Groza cf18cdb2f5
Merge pull request #329 from wb9688/decryption-regexes 
Add more decryption function regexes
 2020-05-29 10:10:50 +02:00
wb9688 ab77961b0a Extract trailer thumbnail in YT learning playlists 2020-05-29 09:55:24 +02:00
wb9688 4d683e7655 Support YouTube's learning playlists 2020-05-29 09:17:03 +02:00
wb9688 384d6acd5d Add more decryption function regexes 2020-05-28 11:40:38 +02:00
wb9688 7ef3e80493 Extract dashManifestUrl in playerResponse 2020-05-15 12:07:26 +02:00
wb9688 baf5dd3e8f
Merge pull request #311 from B0pol/fixSearchSuggestion 
Search: add isCorrectedSearch() and fix YoutubeSearchExtractor#getSea…
 2020-05-12 09:29:15 +02:00
Tobias Groza b40ccb5075
Merge pull request #307 from B0pol/playlists_peertube 
[PeerTube] playlist support
 2020-05-11 20:53:03 +02:00
bopol de26e00079 changed all == null || isEmpty() to isNullOrEmpty() 2020-05-11 11:40:24 +02:00
bopol 202a73516c refactor: add Utils.isNullOrEmpty() 2020-05-11 11:26:18 +02:00
wb9688 1de0876fb4 Fix #327 2020-05-09 20:06:50 +02:00
bopol 1a6e92ebf6 add getSubChannel Name, Url and AvatarUrl for playlists 2020-05-09 09:52:24 +02:00
wb9688 7320108c66 Rename SubChannel in channels back to ParentChannel 2020-05-08 16:02:59 +02:00
Roy Yosef a408661771 Fix PR review 
Rename "ParentChannel" to "SubChannel"
Add Tests for sub-channel info fetching
 2020-05-07 20:20:12 +02:00
Roy Yosef 1de1f97cf1 Add parent channel info to StreamExtractor: name, url & avatar url 2020-05-07 20:20:12 +02:00
Roy Yosef 4234740baa Add parent channel info to ChannelExtractor: name, url & avatar url 2020-05-07 20:20:12 +02:00
bopol 5760366d70 use JsonObject#isEmpty instead of JsonObject#equals(JsonUtils.EMPTY_OBJECT) 2020-05-07 15:40:41 +02:00
bopol 3757541a9c adapt new nanojson version 2020-05-07 15:40:41 +02:00
bopol e8d58e09c9 getSearchSuggestion now returns the corrected query if isCorrectedSearch() is true 2020-05-07 15:40:41 +02:00
bopol 5d1c3b3fa2 Use setters in SearchInfo 2020-05-07 15:39:57 +02:00
bopol 29b639b454 Search: add isCorrectedSearch() and fix YoutubeSearchExtractor#getSearchSuggestion() 
isCorrectedSearch: see the javadoc
getSearchSuggestion: in YoutubeSearchExtractor, it was giving the corrected search query. It now gives the suggested query, as it should
 2020-05-07 15:39:57 +02:00
bopol bc13e0c616 Merge remote-tracking branch 'upstream/dev' into soundcloudComments 2020-05-05 14:49:21 +02:00
wb9688 a1eabc7589 Return null instead of "" in getTextFromObject() 2020-05-02 08:02:25 +02:00
bopol 2564bcf399 refactor comments 
rename methods in CommentsInfoItemExtractor interface to match the other method names across NewPipeExtractor
remove getName in (Youtube|SoundCloud|Peertube)CommentsExtractor and move it up in CommentsExtractor, return "Comments" instead
 2020-04-21 20:05:54 +02:00
wb9688 b51699a20e Implement @TobiGr's suggestions 2020-04-20 14:27:33 +02:00
wb9688 979c5a7502 Use our fork of nanojson 2020-04-17 19:36:42 +02:00
bopol 636c430743 refactor: create extractors and linkHandler packages for SoundCloud, move YoutubeParsingHelper to youtube package (not linkhandler anymore) 2020-04-10 10:51:05 +02:00
bopol 9a7c6b7ab0 use ContentNotSupportedException for channels without tabs 2020-04-07 14:33:43 +02:00
wb9688 bce27a0e22 Rename getValidResponseBody() to getValidJsonResponseBody() 2020-04-03 17:23:18 +02:00
wb9688 8a9e137385 Extract some code to getValidResponseBody() 2020-04-01 16:01:21 +02:00
wb9688 c7f7bd2442 Fix error when YT Music videos has no uploader URL 2020-04-01 10:30:51 +02:00
wb9688 ac15df4548 Move YouTube Music search to its own class 2020-04-01 10:30:51 +02:00
wb9688 cf0f2aff3e Extract uploader url from certain YouTube Music videos 2020-04-01 10:30:51 +02:00
wb9688 aa8cea47f3 Refactor YouTube Music search tests 2020-04-01 10:30:51 +02:00
wb9688 dd434cca01 Fix issue when there is no didYouMeanRenderer in itemSectionRenderer 2020-04-01 10:30:51 +02:00
wb9688 5a775a4bbe Use new way of specifying stream count 2020-04-01 10:30:51 +02:00
wb9688 d58c0f230d Improve code for YouTube Music search 2020-04-01 10:30:51 +02:00
wb9688 dc29d87962 Extract YouTube search suggestions 2020-04-01 10:30:51 +02:00
wb9688 c852b13d5a Add Referer header so that it also works with HttpsUrlConnection 2020-04-01 10:30:51 +02:00
wb9688 2b9b2a78e8 Handle 100+ items in playlist 2020-04-01 10:30:51 +02:00
wb9688 1762a527c9 Add support for YouTube Music search 2020-04-01 10:30:51 +02:00
Tobias Groza e4874d554e
Merge pull request #296 from mauriciocolli/fix-search-errors-detection 
Fix search errors detection and refactor search tests
 2020-03-24 21:53:32 +01:00
Tobias Groza 8a9ae32e6b
Merge pull request #292 from wb9688/fix-yt-no-views 
Fix YouTube videos with no views
 2020-03-24 20:27:08 +01:00
Mauricio Colli c921e5d6a5
Remove unnecessary collector creation function from search extractor 2020-03-23 18:08:51 -03:00
Tobias Groza 6fd9b38ad9
Merge pull request #287 from mauriciocolli/fix-channel-redirect 
[YouTube] Fix channel with redirects directly in the response
 2020-03-23 18:41:56 +01:00
Mauricio Colli 921bf30bb7
[YouTube] Add check for channel items with no video count in search 2020-03-21 15:55:39 -03:00
Mauricio Colli b7f8001a49
[YouTube] Add check for channel items without description in search 2020-03-21 15:55:38 -03:00
Mauricio Colli 9b7999fe54
[YouTube] Check if channel item has subscription count in search 2020-03-21 15:55:37 -03:00
wb9688 904c9d681f Fix YouTube videos with no views 2020-03-18 10:07:49 +01:00
Tobias Groza 9edd0f2675
Merge pull request #288 from Stypox/fix-unlisted-video 
[YouTube] Fix videos without next stream throwing error and add test for unlisted videos
 2020-03-14 21:36:13 +01:00
Stypox 325a51f73a
[YouTube] Do not crash if there is no next stream, but return null 
Like every other extractor does if there is no next stream
 2020-03-14 14:02:48 +01:00
Mauricio Colli b086e9db3f
[YouTube] Fix id extraction for some channels 
Some channels had no reliable way to get the redirected id in the response,
so saving it for later was a valid alternative.
 2020-03-14 02:39:49 -03:00
Mauricio Colli 00d1ed439b
[YouTube] Fix channel extraction when redirects are in the response 
Some redirects were embed directly into the response as instructions
for the page, instead of the usual http redirects.
 2020-03-14 02:39:48 -03:00
Mauricio Colli 70abd57852
[YouTube] Cache commonly used stream type result 2020-03-07 16:48:32 -03:00
Mauricio Colli 1ef706f567
[YouTube] Detect LIVE videos in the trending page 2020-03-07 16:48:31 -03:00
Mauricio Colli cfc278317d
[YouTube] Fail-fast if status exist and is anything other than "OK" 2020-03-01 10:29:29 -03:00
TobiGr 35252235b0 Go through all badges when checking if a StreamInfoItem is a live stream 2020-03-01 12:52:37 +01:00
Mauricio Colli 5686a6f562 [YouTube] Detect when a stream is deleted or doesn't exist 
Added a test case as well.
 2020-03-01 12:49:13 +01:00
Mauricio Colli e65333c3ce [YouTube] Detect deleted/nonexistent/invalid channels and playlists 
- Added tests for these cases.
 2020-03-01 12:49:13 +01:00
Mauricio Colli 408f042127 [YouTube] Fix bug when url isn't present in the browseEndpoint object 2020-03-01 12:49:13 +01:00
Mauricio Colli e9644e6216 [YouTube] Handle video premiere's date and duration 2020-03-01 12:49:13 +01:00
Mauricio Colli 2a470ac4f7 [YouTube] Handle videos with no views or with "Recommended to you" text 2020-03-01 12:49:13 +01:00
Mauricio Colli da2c562eb0 [YouTube] Take into account videos that have their views hidden 
Also remove catch because it may hide bugs when the page changes.
 2020-03-01 12:49:13 +01:00
wb9688 2807079531 Catch NumberFormatException instead of checking whether viewCount equals "Recommended for you" 2020-02-29 21:28:38 +01:00
wb9688 bfe3eb1409 Use suggested try-if code style 2020-02-29 17:18:50 +01:00
wb9688 45df8248bf Use Localization in getJsonResponse() 2020-02-29 16:55:07 +01:00
wb9688 157055fbfd Create getJsonResponse() function 2020-02-29 16:42:04 +01:00
wb9688 4f406512c2 Use YoutubeChannelLinkHandlerFactory 2020-02-29 16:26:28 +01:00
wb9688 4ff655352d Reimplement getAgeLimit() 2020-02-28 17:14:26 +01:00
wb9688 6ba03e542b Cache getVideoTab() 2020-02-28 17:05:31 +01:00
wb9688 8bfbae904d Cache getVideo(Primary|Secondary)InfoRenderer() 2020-02-28 17:03:21 +01:00
wb9688 0b940f5cf4 Use "getAgeLimit() != NO_AGE_LIMIT" instead of "getAgeLimit() == 18" 2020-02-28 16:40:50 +01:00
wb9688 7898ba230d Use pbj in YoutubeStreamExtractor 2020-02-28 15:17:47 +01:00
wb9688 612ec06839 Actually fix thumbnail URLs starting with // 2020-02-28 09:36:33 +01:00
wb9688 f946310af5 Fix search queries with a promotedSparklesTextSearchRenderer returning 0 results 2020-02-28 08:57:44 +01:00
wb9688 94a2d5be46 Fix thumbnail URLs starting with // 2020-02-27 19:08:46 +01:00
wb9688 838d834ce2 Get Uploads view of Videos tab 2020-02-27 18:54:27 +01:00
wb9688 365b0329f3 Implement getTextFromObject() function 2020-02-27 17:39:23 +01:00
wb9688 0798bdd5cd Fix getSubscriberCount() for 0 subscribers 2020-02-27 16:26:22 +01:00
wb9688 8ebd971648 Fix channel descriptions consisting of multiple parts 2020-02-27 10:06:35 +01:00
wb9688 880b951088 Fix getUploaderAvatarUrl() in YoutubeStreamExtractor 2020-02-27 09:53:59 +01:00
wb9688 8aa560bef4 Fix video titles consisting of multiple parts due to @ 2020-02-27 09:42:22 +01:00
wb9688 951ed9f0d8 Don't call getUploaderInfo() in onFetchPage() for playlists 
This way YouTube Music playlists will keep working in NewPipe
 2020-02-27 09:27:56 +01:00
wb9688 2399d46ab0 Fix YoutubeChannelExtractor for channels without videos 2020-02-27 09:21:25 +01:00
wb9688 cdbc751b87 Fix getUploaderUrl() in YoutubeStreamInfoItemExtractor 2020-02-27 09:05:41 +01:00
wb9688 db305408e9 Get all expandedShelfContentsRenderers in YoutubeTrendingExtractor 2020-02-26 20:37:20 +01:00
wb9688 a2d8489e16 Parse timestamps in description 2020-02-26 20:27:30 +01:00
wb9688 8d503407ef Parse browseEndpoint and watchEndpoint in descriptions 2020-02-26 19:37:58 +01:00
wb9688 88282761f4 Fix YoutubeStreamExtractor not being able to getAgeLimit() in onFetchPage() 2020-02-26 16:19:42 +01:00
wb9688 bb3a3d70bf Use pbj in YoutubeTrendingExtractor 2020-02-26 16:10:24 +01:00
wb9688 1c3a4e710b Check if there are no related streams in getAgeLimit() 2020-02-26 16:03:06 +01:00
wb9688 985c3ec877 Improve getClientVersion() 2020-02-26 15:27:50 +01:00
wb9688 15afbea3e1 Use pbj in YoutubePlaylistExtractor 2020-02-26 15:27:50 +01:00
wb9688 89a41a7f69 Use pbj in YoutubeChannelExtractor 2020-02-26 15:27:50 +01:00
wb9688 0973263aab Use pbj in YoutubeSearchExtractor 2020-02-26 15:27:46 +01:00
TobiGr fc465c8bf7 Fix getUploadDate() for premiered videos with a given date 
Detect publishDate and uploadDate
 2020-02-25 22:54:37 +01:00
TobiGr 96285e0815 Fix getUploadDate() for premiered videos with a given date 2020-02-25 21:51:30 +01:00
wb9688 5a35300a5e Fix parsing upload date for premiered streams 2020-02-25 21:51:30 +01:00
wb9688 729fd2eaee Fix parsing description for navigationEndpoint without urlEndpoint 2020-02-25 21:51:30 +01:00
wb9688 26fb44595f Fix parsing description 2020-02-25 21:51:30 +01:00
TobiGr 0fff03038e Get description with correct links 2020-02-25 21:51:30 +01:00
wb9688 5d08c34322 Fix parsing upload date in YoutubeStreamExtractor 2020-02-25 21:51:30 +01:00
wb9688 bbe1a3cd62 Use getVideo(Primary|Secondary)InfoRenderer() 2020-02-25 21:51:30 +01:00
wb9688 8e6be880d7 Simplify getId() in YoutubeChannelExtractor 2020-02-25 21:51:30 +01:00
TobiGr ecfc27a92c Implement fallback methods for getDescription() and getTextualUploadDate() in YouTubeStreamExtractor 2020-02-25 21:51:30 +01:00
wb9688 eed29ea734 Return null for related streams at age restricted videos 2020-02-25 21:51:30 +01:00
TobiGr f39603f6ef Implement getUploadDate() in YouTubeStreamInfoItemExtractor 2020-02-25 21:51:30 +01:00
wb9688 26ea3dceb6 Fix getViewCount() in YoutubeStreamExtractor 2020-02-25 21:51:30 +01:00
wb9688 76d54abdbf Reimplement more methods in YoutubeStreamExtractor 2020-02-25 21:51:30 +01:00
wb9688 f13c0288cc Reimplement some methods in YoutubeStreamExtractor 2020-02-25 21:51:30 +01:00
wb9688 02b59903fa Remove useless code 2020-02-25 21:51:30 +01:00
TobiGr dd4dd849dc Get client version dynamically in YouTubeSearchExtractor 2020-02-25 21:51:30 +01:00
TobiGr 583e9c1571 Fix getPage() 
Someone should check if there are unstaged changes before committing...
 2020-02-25 21:51:30 +01:00
TobiGr 5842b9ad37 Add getClientVersion() and HARDCODED_CLIENT_VERSION to YouTubeParsingHelper 
Prefer hardcoded client version above the current one when making requests to retrieve the same JSON structure for each request.
 2020-02-25 21:51:30 +01:00
wb9688 5d883d100c Implement pagination in YoutubeSearchExtractor 2020-02-25 21:51:30 +01:00
wb9688 c0a8e01889 Implement pagination in YoutubePlaylistExtractor 2020-02-25 21:51:30 +01:00
wb9688 4039409820 Fix getLikeCount() for certain streams 2020-02-25 21:51:30 +01:00
wb9688 627ab2459f Uncomment fetchPage() 2020-02-25 21:51:30 +01:00
wb9688 2dfa2187ff Implement pagination in YoutubeChannelExtractor 2020-02-25 21:51:30 +01:00
wb9688 a38ab9b791 Return null if channel has no banner 2020-02-25 21:51:30 +01:00
wb9688 1eca63089b Count private and deleted videos as ad 2020-02-25 21:51:30 +01:00
wb9688 c7360e4a46 Fix getThumbnailUrl() in YoutubeChannelInfoItemExtractor 2020-02-25 21:51:30 +01:00
wb9688 51fb26625a Implement getSubscriberCount() in YoutubeChannelInfoItemExtractor 2020-02-25 21:51:30 +01:00
TobiGr 6d20b2b1d0 Fix NPE on collecting page items 2020-02-25 21:51:30 +01:00
wb9688 4462cbe3f1 Reimplement YoutubePlaylistInfoItemExtractor 2020-02-25 21:51:30 +01:00
wb9688 8aea4d445b Reimplement YoutubeChannelInfoItemExtractor 2020-02-25 21:51:30 +01:00
wb9688 3187116a63 Handle premium videos 2020-02-25 21:51:30 +01:00
wb9688 7442f91aa6 Don't assume which channel tab is videos 2020-02-25 21:51:30 +01:00
wb9688 316fe0109d Fix first playlist page 2020-02-25 21:51:30 +01:00
wb9688 4e57e589ce Fix first channel page 2020-02-25 21:51:30 +01:00
TobiGr 5816202cc7 Move getInitialData() method to YouTubeParsingHelper 
Rename ytInitialData to initialData
 2020-02-25 21:51:30 +01:00
wb9688 38aabc6aca Use plain YoutubeStreamInfoItemExtractor in YoutubeTrendingExtractor 2020-02-25 21:51:30 +01:00
wb9688 957db062ef Fix getting uploader for videoRenderer 2020-02-25 21:51:30 +01:00
wb9688 b88188d419 Move stuff from extractVideoPreviewInfo() into YoutubeStreamInfoItemExtractor and partially fix search 2020-02-25 21:51:30 +01:00
wb9688 af49b3c487 Fix live streams in extractVideoPreviewInfo() 2020-02-25 21:51:30 +01:00
TobiGr 10e9c16d8c Fix LiveStream view count 2020-02-25 21:51:30 +01:00
wb9688 127c4d5893 Fix getNextStream() in YoutubeStreamExtractor 2020-02-25 21:51:30 +01:00
wb9688 7bfc0e62c6 Fix getRelatedStreams() in YoutubeStreamExtractor 2020-02-25 21:51:30 +01:00
TobiGr 12166ca60c [StreamExtractor] Add fallback method for uploader avatar url 2020-02-25 21:51:30 +01:00
wb9688 3ec422a542 Fix like/dislike parsing 2020-02-25 21:51:30 +01:00
TobiGr a34e5c18e1 Trending StreamInfoItemCollecetor 2020-02-25 21:51:30 +01:00
TobiGr b705515da0 Adapt most playlist fields 2020-02-25 21:51:30 +01:00
TobiGr 655c999795 Detect disabled subriber count correctly 
Fix parsing of kiosk name
 2020-02-25 21:51:30 +01:00
TobiGr 7dcc9f159b Stream: fix uploader avatar url 2020-02-25 21:51:30 +01:00
TobiGr be3e20e263 Adapt most channel changes 2020-02-25 21:51:30 +01:00
TobiGr 39bf1ff1eb Add ytInitialData regex 2020-02-25 21:51:30 +01:00
Tobias Groza a129c65c63
more improvements of comments 
Co-Authored-By: Stypox <stypox@pm.me>
 2020-02-16 23:59:31 +01:00
Tobias Groza 6822574588 Fix typos 
Co-Authored-By: B0pol <bopol@e.email>
 2020-02-15 13:36:54 +01:00
TobiGr 030465b5d4 Improve code formatting and optimise imports 2020-02-15 13:36:54 +01:00
bopol 70a40e7388 Description: rm constructor by serviceId 2020-02-07 13:28:27 +01:00
bopol 26c65b2948 Create class Description 2020-02-06 23:35:46 +01:00
B0pol b671a4ba18
Merge branch 'dev' into peertube 2020-02-01 18:18:23 +00:00
Mauricio Colli 26234a1c0f
Introduce FeedExtractor making fetching from dedicated feeds possible 
YouTube, for example, has a dedicated feed which was built to be used
like this.
 2020-01-28 02:44:18 -03:00
Mauricio Colli be81f2945c
[YouTube] Use correct upload date flag in StreamExtractor 2020-01-28 02:44:17 -03:00
TobiGr 033a9fb8e3 Revert change which I accidentally committed while fixing the decryption 
function in 7b72fd2a7d
 2020-01-25 21:08:17 +01:00
B0pol e392b6c68f getLanguageInfo returns Locale instead of String 
so that java can automatically translate with Locale.getDisplayLanguage(), instead of always having English name of the language
 2020-01-25 14:12:38 +01:00
B0pol b816e48216
Merge branch 'dev' into peertube 2020-01-24 20:10:47 +01:00
TobiGr f8d117f6cd Merge remote-tracking branch 'origin/master' into dev 2020-01-24 19:22:40 +01:00
TobiGr 7b72fd2a7d [YouTube] Add new decryption function 2020-01-24 02:10:56 +01:00
TobiGr bce87f3d01 Improve getDescriptionFuncName by removing catches and adding a loop 2020-01-24 02:10:38 +01:00
TobiGr 9b45c61103 Typos, comments and formatting 2020-01-24 02:10:18 +01:00
TobiGr 29a4cc78bf Fix typo in decryption constant 2020-01-24 02:10:08 +01:00
bopol 74439f692a add extraction for support info and rename getLanguageInfo function 2020-01-23 14:19:22 +01:00
B0pol bcfe7be4e6
Merge branch 'dev' into peertube 2020-01-22 19:19:41 +01:00
Stypox 1cfdc4547a
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java 
Co-Authored-By: Tobias Groza <TobiGr@users.noreply.github.com>
 2020-01-20 22:52:48 +01:00
Stypox 1e1100ef76
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java 
Co-Authored-By: Tobias Groza <TobiGr@users.noreply.github.com>
 2020-01-20 22:52:36 +01:00
bopol a281519e77 added metadata, fix descriptions, fix thumbnail, update tests 
thumbnail: quality before: https://peertube.cpy.re/static/thumbnails/d2a5ec78-5f85-4090-8ec5-dc1102e022ea.jpg
quality after: https://peertube.cpy.re/static/previews/d2a5ec78-5f85-4090-8ec5-dc1102e022ea.jpg

description: we were getting about the first 260 characters, we now get full description (with fallback to first 260 chars if the get request for full description fails)
test: updated tests to match description, also changed some test: it was assertEquals(extracted, expected), but the proper way to do it is assertEquals(expected, extracted)

metadata: got host, privacy (public, private, unlisted), licence, language, tags
 2020-01-20 12:22:33 +01:00
Stypox 1ed89aad3e
Use more often playerResponse in yt stream extractor 
This enhances performance and should make the extractor more reliable since it get info from a stable json structure that shouldn't be subject to many changes. Fallback html methods have been kept. In case of error the thrown exception contains the data about the playerResponse failure, that should be clearer than a NPE caused by not-found html tags.
 2020-01-06 20:45:57 +01:00
Tobias Groza d83787a5ca
Merge pull request #226 from 7FM/dev 
fix switching on null causing NPE
 2019-12-31 00:26:27 +01:00
Zsombor Gegesy 2123b3abd3 Fix search result paging - due to the way as the InfoItemsSearchCollector are re-used, 
the returned item list just grows, which cause that same videos are returned.
 2019-12-29 23:14:55 +01:00
Zsombor Gegesy 0a3422a591 Fix invalid jsoup query 2019-12-26 14:03:40 +01:00
7FM f57d9d43ce fix switching on null causing NPE 2019-12-21 14:15:42 +01:00
Mauricio Colli 3d21ef5dba
Introduce class that indicates when the time ago is an approximation 2019-11-03 15:46:28 -03:00
Mauricio Colli 41b51adceb
[YouTube] Force english as the only supported language (#196) 2019-11-03 15:46:25 -03:00
Mauricio Colli cb1e327a6e
[YouTube] Fix parsing of video reminders 2019-11-03 15:46:23 -03:00
Mauricio Colli d8280ce0da
[YouTube] Parse watching count in live streams items 2019-11-03 15:46:22 -03:00
Mauricio Colli 3638f0e0ea
Implement time ago parser and improve localization handling 
- Handle special cases for languages where the number is not shown
- Rework the Downloader base implementation, allowing for more
advanced things to be done
- Separate the localization from the content country (just like
YouTube let's the user choose both).
 2019-11-03 15:46:21 -03:00
wojcik-online 180836c180
Base Implementation: Parse the upload date of StreamInfoItems 
In the format '2 days ago' (in English) on a YouTube channel page.
(Parser extensible to other pages.)
 2019-11-03 15:46:20 -03:00
Mauricio Colli c4fe2183ce
[YouTube] Improve detection of reCAPTCHA pages 2019-10-29 02:00:41 -03:00
Mauricio Colli 9a325b280d
[YouTube] Make detection of age restricted pages more reliable 2019-10-29 02:00:40 -03:00
Mauricio Colli 91c360df5e
Remove section of dead code 2019-10-29 02:00:39 -03:00
Tobias Groza 06f2144e4d
[YouTube] Remove GEMA exception handling (#202) 2019-10-05 14:59:05 +02:00
Vasiliy d8279f91f6
Merge branch 'dev' of https://github.com/TeamNewPipe/NewPipeExtractor into feature/frames 2019-09-28 14:46:14 +03:00
TobiGr 8ab48c62b9 [YouTube] Fix NPE in ChennelExtractor.getSubsciberCount() 2019-09-25 08:57:08 +02:00
TobiGr 0710f31a39 Fix TeamNewPipe/NewPipeExtractor#197 2019-09-25 08:37:26 +02:00
Stypox d1cd341592 Change comment 2019-09-25 08:32:00 +02:00
Stypox 35921345d9 Use Exception instead of Throwable (more consistent) 2019-09-25 08:32:00 +02:00
Stypox db3596c818 Fix "Could not get id" for channels w/o "Subscribe" button 2019-09-25 08:32:00 +02:00
Stypox 94e7f0d3ab Fix fallback method is not tried on exception 
in YoutubeChannelInfoItem.getUrl()
 2019-09-25 08:32:00 +02:00