lasers
/
NewPipeExtractor
mirror of https://github.com/TeamNewPipe/NewPipeExtractor
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
2,362 Commits 11 Branches 72 Tags 113 MiB
Commit Graph

1415 Commits

Author SHA1 Message Date
Stypox 7b54457789
Merge pull request #941 from TeamNewPipe/feat/peertube-comment-replies 
[PeerTube]  Support comment replies
 2022-12-31 14:57:51 +01:00
AudricV f45966d449
Merge pull request #910 from Isira-Seneviratne/Locale_forLanguageTag 
Add compat Locale.forLanguageTag() implementation.
 2022-12-24 23:53:30 +01:00
AudricV d5437e0bc5
Merge pull request #863 from AudricV/add-content-type-and-content-length-headers-to-post-requests 
Add Content-Type header to all POST requests without an empty body
 2022-12-16 19:32:56 +01:00
AudricV 0766b1d211
[YouTube] Improve YoutubeStreamInfoItemExtractor 
- Return duration of video premieres;
- Add another non-localized method to determine whether a stream is a running
livestream;
- Return view count and upload date of videos in playlists;
- Store isPremiere result;
- Remove shorts workaround code, as it was only useful on channels and shorts
have been moved into a separated channel tab;
- Improve some other code.
 2022-12-08 13:59:12 +01:00
Tobi 896d7e09eb
Merge pull request #978 from Theta-Dev/fix/search-channel-handles 
[YouTube] Fix search subscriber count extraction with channel handles
 2022-12-05 17:52:05 +01:00
TobiGr cd3262745d [PeerTube] Report Exceptions thrown while getting a stream's subtitles 2022-12-03 16:11:21 +01:00
TobiGr 4e66b2287e [PeerTube] Add support for comment replies 2022-12-01 14:05:18 +01:00
Tobi 41c8dce452
Merge pull request #992 from Isira-Seneviratne/String_isBlank 
Use String.isBlank().
 2022-11-30 17:48:54 +01:00
Isira Seneviratne 2bca56f0df Use String.isBlank(). 2022-11-30 08:26:21 +05:30
Isira Seneviratne 3b80547976 Add code review suggestions. 2022-11-30 07:57:45 +05:30
ThetaDev 016623131e docs: update comment in YoutubeChannelInfoItemExtractor 2022-11-29 19:06:03 +01:00
ThetaDev c156c404cb Merge branch 'dev' of github.com:TeamNewPipe/NewPipeExtractor into channel-tabs 2022-11-29 17:50:32 +01:00
ThetaDev ffd02a4bc8 fix: shorts continuation 2022-11-29 17:50:14 +01:00
Kavin abf08e1496
Merge pull request #990 from FireMasterK/bold-italic-strikethrough 
[YouTube] Implement bold/italic/strike-through support
 2022-11-29 15:59:38 +00:00
Kavin 52fda37915
Implement bold/italic/strike-through support. 2022-11-28 19:06:18 +00:00
Kavin b566084cac
Use Description object for comments text. 2022-11-28 17:02:19 +00:00
Tobi 1da0190056
Merge pull request #980 from TeamNewPipe/fix/yt/unavailable 
[YouTube] Fix extracting the detailed error message for unavailable streams
 2022-11-28 10:07:34 +01:00
Stypox 60fb30f835
Merge pull request #928 from FireMasterK/comment-urls 
Parse YouTube comments as HTML
 2022-11-27 19:16:34 +01:00
Kavin 5abea22225
Fix throwing correct reason. 2022-11-26 21:09:08 +00:00
Kavin c043597255
Update supported countries list. 2022-11-26 19:01:33 +00:00
TobiGr 4680df0bdf Fix throwing correct reason 2022-11-23 17:03:22 +01:00
TobiGr 9de8405c9f [YouTube] Fix extracting the detailed error message of streams which are unavailable 2022-11-23 08:33:06 +01:00
AudricV 3891542ca1
Use Downloader's postWithContentType and postWithContentTypeJson methods in services and extractors 2022-11-22 11:37:18 +01:00
AudricV b2862f3cd1
Add postWithContentType and postWithContentTypeJson utility methods in Downloader 
Co-authored-by: Stypox <stypox@pm.me>
 2022-11-22 11:37:17 +01:00
AudricV e9a0d3bd95
[YouTube] Send Content-Type header in all POST requests 
This header was not sent partially before and was added and guessed by OkHttp. This can create issues when using other HTTP clients than OkHttp, such as Cronet.

Some code in the modified classes has been improved and / or deduplicated, and usages of the UTF_8 constant of the Utils class has been replaced by StandardCharsets.UTF_8 where possible.

Note that this header has been not added in except in YoutubeDashManifestCreatorsUtils, as an empty body is sent in the POST requests made by this class.
 2022-11-22 11:37:16 +01:00
AudricV b9e463de49
[Bandcamp] Send Content-Type header in POST requests 
This header was not sent before and was added and guessed by OkHttp. This can create issues when using other HTTP clients than OkHttp, such as Cronet.

Also make use of StandardCharsets.UTF_8 when getting bytes of bodies instead of the platform default's charset, to make sure to prevent some encoding issues on some JVMs.
 2022-11-22 11:35:46 +01:00
AudricV 65d6321e3d
Fix typos in Downloader.post JavaDocs 
Post methods in Downloader return the result of a POST request and not the one of a GET request.
 2022-11-22 11:35:46 +01:00
ThetaDev f7e3b713b5 Merge branch 'dev' into channel-tabs 2022-11-22 02:38:03 +01:00
ThetaDev 5daabd1793 fix: #976 search subscriber count extraction with channel handles 2022-11-22 02:17:10 +01:00
ThetaDev 8d3bc2bc4b fix: YoutubeParsingHelper formatting 2022-11-22 01:59:51 +01:00
Kavin c953e23414
Merge pull request #968 from AudricV/yt-support-no-video-info-renderers-for-streams 
[YouTube] Support lack of video info renderers for streams
 2022-11-16 20:20:01 +00:00
ThetaDev 2245de1197
fix: make getTab nonnull 
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
 2022-11-16 16:50:25 +01:00
Tobi 2211a24b69
Merge pull request #971 from lrusso96/patch-1 
[YouTube] Improve duration parsing
 2022-11-16 16:14:54 +01:00
Kavin 86f06b333a
Address review. 2022-11-14 00:05:31 +00:00
Kavin 30909da1df
Fix audio track similar comparison for IDs. 2022-11-13 23:08:54 +00:00
Kavin 6d59cdbe3a
Add support for extracting audio tracks. 2022-11-13 21:39:29 +00:00
Isira Seneviratne e4d982c7ea Fix license. 2022-11-12 07:29:15 +05:30
Isira Seneviratne ddbce3b83d Add Utils methods for URL encoding/decoding using UTF-8. 2022-11-12 07:29:15 +05:30
Isira Seneviratne 366f5c1632 Use StandardCharsets.UTF_8. 2022-11-12 07:29:15 +05:30
Luigi Russo c9635218e2
[YouTube] Improve duration parsing 2022-11-09 09:41:29 +01:00
Isira Seneviratne 316d8573fa Use immutable sets in YoutubeParsingHelper. 2022-11-07 07:50:26 +05:30
ThetaDev 7dba12be76 fix: link handler urls for tabs 2022-11-05 00:21:43 +01:00
ThetaDev 8a3545c8b2 fix: rename channel tab LIVE to LIVESTREAMS 2022-11-05 00:08:33 +01:00
ThetaDev abf04733a3 fix: support new PlaylistInfoItem interface 2022-11-05 00:02:08 +01:00
ThetaDev 73c182f817 Merge branch 'dev' of github.com:TeamNewPipe/NewPipeExtractor into channel-tabs 2022-11-04 23:50:04 +01:00
ThetaDev f71fdac166 refactor: API changes 2022-11-04 23:47:44 +01:00
AudricV aa9a8ca23c
[YouTube] Make non-extraction of videoPrimaryInfoRenderer and/or videoSecondaryInfoRenderer not fatal 
Also de-duplicated common code related to the obtain of these video info renderers.

This change allows extraction of videos without visual metadata.
 2022-11-04 18:35:53 +01:00
AudricV 61ce041bda
[YouTube] Support handles and all custom channel names 
More non-channel paths have been also added to the excluded custom name paths,
documentation and exception messages have been improved and fixed in some
places, and the licence header of YoutubeChannelLinkHandlerFactory has been
moved to its beginning and updated.
 2022-11-03 19:46:42 +01:00
AudricV ffffb04439
Merge pull request #953 from Theta-Dev/attributed-text-desc 
[YouTube] Add support for attributed text description
 2022-11-03 18:34:30 +01:00
ThetaDev 592e1d6386 fix: parsing attributed description with no command runs 2022-11-03 12:10:52 +01:00
ThetaDev 7ec6a44926 fix: channel short upload date parsing 2022-11-03 09:38:38 +01:00
ThetaDev 099b53cc4f
[YouTube] Add parser for attributedDescription 
Also update the mock of the next InnerTube endpoint response of the
YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing test class with an
attributedDescription instead of a regular description
 2022-11-02 23:11:33 +01:00
ThetaDev 856584f820 fix: channel shorts duration parsing 
1 minute was incorrectly parsed as 1s
 2022-11-02 19:42:25 +01:00
ThetaDev 0a458d8948 fix: NPE when extracting YT stream items without duration 2022-11-02 19:17:23 +01:00
ThetaDev f3b064a3b3 fix: change playlist tab parameter to include YTM albums 2022-11-02 19:07:07 +01:00
Theta-Dev 20e4a35814
[YouTube] Support richGridRenderer on channel pages 
YouTube is deploying a new layout on their channel pages, which uses richGridRenderer JSON objects.
 2022-11-02 19:01:29 +01:00
AudricV 4cae66f1f9
Merge pull request #946 from chowder/dev 
Add ability to identify short-form `StreamInfoItem`s
 2022-11-01 12:19:58 +01:00
Tobi eb40bb8458
Merge pull request #959 from FireMasterK/playlist-info-item-uploader 
Add uploaderUrl and uploaderVerified to PlaylistInfoItem.
 2022-10-31 13:10:33 +01:00
chowder b1a899fd47 Fix null pointer exception 2022-10-31 11:12:23 +00:00
Stypox a4db106a66
Merge pull request #960 from AudricV/yt-workaround-403-errors-android-client 
[YouTube] Workaround 403 HTTP errors of ANDROID client streams
 2022-10-30 21:53:26 +01:00
Kavin b441910257
Mark uploaderUrl as nullable. 2022-10-30 13:36:04 +00:00
chowder 3fdc0e72cc Line breaks for long docstrings 2022-10-30 13:28:39 +00:00
Kavin 6a256d0631
Add uploader url and verified to PlaylistInfoItem. 2022-10-30 13:00:19 +00:00
Tobi 430504b4b5
Merge pull request #958 from AudricV/yt-playlists-support-new-metadata-format 
[YouTube] Support new metadata format of playlists
 2022-10-30 12:31:43 +01:00
Kavin f9bd08c649 Address reviews. 2022-10-30 01:25:30 +00:00
Caleb 9282c3c13b Fix exception message for YoutubeStreamInfoItemExtractor#isShortFormContent 
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
 2022-10-30 01:23:15 +00:00
Caleb c5216f7c12 Update docstring for StreamExtractor#isShortFormContent 
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
 2022-10-30 01:23:15 +00:00
chowder 4cccd33f3d Implement isShortFormContent for StreamExtractor and StreamInfo 2022-10-30 01:23:15 +00:00
chowder daf5674951 Add ability to identify short-form StreamInfoItems 2022-10-30 01:23:12 +00:00
Tobi 3d314169b9
Merge pull request #943 from TeamNewPipe/fix/sc/comments 
[SoundCloud] Fix getting more comments
 2022-10-29 22:19:50 +02:00
AudricV 7258a53225
[YouTube] Support new playlist layout 
This new layout doesn't provide author thumbnails and is completely different
for metadata, so the code to get them has been refactored.

The code of learning playlists video count check has been also removed, as it
seems to be not relevant anymore (the video count seems to be returned for
these playlists with both layouts).

Finally, unneeded overrides of subchannel methods, which don't apply to the
YouTube service, have been removed.
 2022-10-29 18:12:10 +02:00
AudricV 60e97cd274
[YouTube] Workaround getting streaming URLs returning 403 HTTP response codes 
Using the player parameters used to get stories seems to fix the issue, which
affects currently only certain countries such as UK.

This is a workaround and should be fixed in a better way (by changing the
InnerTube additional client used for videos or finding what is now required in
Android player requests).
 2022-10-29 17:58:33 +02:00
AudricV c230d84df1
Fix Checkstyle error in YoutubeCommentsInfoItemExtractor 2022-10-29 13:24:19 +02:00
ThetaDev a592c96ed9 test: add channel tab extractor tests 2022-10-25 21:13:16 +02:00
ThetaDev 94523adeea feat: add Bandcamp album tab 2022-10-25 10:38:23 +02:00
ThetaDev 12537733c1 fix: store YouTube visitor data for channel tabs 2022-10-25 09:20:18 +02:00
ThetaDev edaaaac85f fix: checkstyle errors 2022-10-24 10:29:00 +02:00
ThetaDev 04c7e46d86 Merge branch 'dev' of github.com:TeamNewPipe/NewPipeExtractor into channel-tabs 2022-10-24 08:36:46 +02:00
ThetaDev e6907ca1bd fix: Peertube playlist urls 2022-10-24 00:37:12 +02:00
ThetaDev 53e772c3d9 feat: add tab support for Soundcloud 
- fix checkstyle errors
 2022-10-24 00:15:31 +02:00
ThetaDev aed685e58b feat: add tab support for Peertube 2022-10-23 23:09:40 +02:00
ThetaDev 57865e2195 feat: add visitor data config option 2022-10-23 21:57:15 +02:00
ThetaDev 667ab2a78c feat: add album tab 2022-10-23 21:36:55 +02:00
ThetaDev 9a9fae9a33 feat: prettier channel info page 2022-10-23 17:01:39 +02:00
ThetaDev 78bbbd405d fix: handle unsupported content, hide tab bar with < 2 tabs 2022-10-23 15:37:40 +02:00
ThetaDev 18e3758c92 feat: add channel tabs 2022-10-23 10:27:35 +02:00
ThetaDev 8b4b4310ea feat: add tab support to channel extractor 
- extract YouTube channel tabs: playlists, channels, shorts, live
 2022-10-22 15:29:35 +02:00
xz-dev 0ffcb32d9c
[YouTube] Add comment reply count support (#936) 
Add comment reply count support for YouTube and introduce `CommentsInfoItem.UNKNOWN_REPLY_COUNT` constant

Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
 2022-10-15 12:40:06 +02:00
Theta-Dev ed4559d4de fix: support richGridRenderer on channel page 
YouTube is currently A/B testing a new layout on their channel pages, which uses a RichGridRenderer.
 2022-10-12 15:29:36 +02:00
Isira Seneviratne b90a566dd8 Add backport implementation of Locale.forLanguageTag(). 2022-10-12 09:21:39 +05:30
Isira Seneviratne b232c29d22 Use Locale.forLanguageTag(). 2022-10-12 09:21:38 +05:30
TobiGr 4d136599bd [SoundCloud] Fix getting more comments 2022-10-11 15:44:54 +02:00
Tobi a822e91909
Merge pull request #939 from TurtleArmyMc/fix/SoundcloudPlaylistExtractor_track_order 
Fix SoundcloudPlaylistExtractor: tracks are in correct order
 2022-10-10 22:28:18 +02:00
TobiGr 02810a7db7 Add a comment 2022-10-10 22:22:12 +02:00
Tobi 54092fc3c7
Merge pull request #930 from Isira-Seneviratne/Avoid_Comparator_NPE 
Avoid possible NullPointerException in MediaCCCRecentKiosk.
 2022-10-10 11:43:34 +02:00
TurtleArmyMc bf70d32eb4 Fix SoundcloudPlaylistExtractor: tracks are in correct order 2022-09-30 15:26:25 -04:00
AudricV 8067c43837
[YouTube] Don't use a specific letter for the decryption function name pattern 
Use the same possible characters for variables everywhere, in order to avoid
potential future throttling parameter decryption function name parsing issues
related to the usage of other letter(s) than b.
 2022-09-24 21:49:22 +02:00
AudricV abcee87167
[YouTube] Fix throttling parameter decryption function regex 
- Quote the function name, as it may contain special regex symbols, such as
  dollar;
- Support multiple lines;
- Use what looks like the end of the function for the end of the regex (this
  part is inspired from yt-dlp throttling parameter decryption regex);
- Move the throttling function body regex into a private and static constant.
 2022-09-24 21:28:09 +02:00
ThetaDev 7244be7627
[YouTube] Add JavaScript lexer to parse completely throttling decryption function (#905) 2022-09-24 19:55:46 +02:00
Isira Seneviratne 0e31c86aee Avoid possible NullPointerException in MediaCCCRecentKiosk. 2022-09-21 05:40:07 +05:30