Commit Graph

1409 Commits

Author SHA1 Message Date
Kavin e3480ba2ae
Update the JavaDoc for the rate-limiting issue. 2023-06-02 22:54:26 +01:00
Kavin f063dd15ec
Fail-fast extraction if the initial player response is invalid. 2023-06-02 22:47:07 +01:00
Audric V 533121fb81
Merge pull request #1045 from Theta-Dev/fix/trending-video-tab
[YouTube] Extract trends from A/B tested "Videos" tab and fix extraction of trends name from A/B tested new title design
2023-05-19 11:22:49 +02:00
Audric V 92a0024424
Merge pull request #1052 from TeamNewPipe/peertube/fix/nested-comment-replies
[PeerTube] Fix multi level comment replies
2023-05-18 18:49:06 +02:00
TobiGr c70bb83801 [Bandcamp] Implement PlaylistExtractor.getDescsription() 2023-05-15 15:23:03 +02:00
TobiGr ca0ce00753 Add PlaylistInfoItem.getDescription() and PlaylistInfoItemExtractor.getDescription()
[PeerTube] Implement the corresponding extractor method.
TODO: add tests
2023-05-12 01:43:59 +02:00
TobiGr b218bf69bd Implement PlaylistInfo.getDescription()
Implement PlaylistExtractor.getDescription() for PeerTube and SoundCloud.
Anotate   PlaylistExtractor.getDescription() as Nonnull
2023-05-12 00:44:10 +02:00
chunky programmer 81f29116ba switch from string to Description object 2023-05-11 00:36:57 -04:00
chunky programmer 5ab6cd7420 Extract YouTube playlist description 2023-05-11 00:00:22 -04:00
TobiGr d358ba1c41 Improve PeertubeCommentsInfoItemExtractor constructor 2023-05-07 22:55:26 +02:00
TobiGr aff3e795f8 [PeerTube] Fix multi level comment replies 2023-05-07 22:49:14 +02:00
ThetaDev 3673d4ae01
fix: YouTube trending name extraction 2023-05-03 21:16:35 +02:00
ThetaDev 24eba62305
fix: extract YouTube trends from new "Videos" tab 2023-05-03 21:16:23 +02:00
Kavin a9ca5c49e4
Merge pull request #1056 from AudricV/yt-improve-search-suggestions-extraction
[YouTube] Switch to new search suggestion domain and improve error handling
2023-05-02 20:17:48 +01:00
AudricV e923fca440
[YouTube] Switch to new search suggestion domain and improve error handling
- Switch to the new domain used by YouTube for search suggestions,
suggestqueries-clients6.youtube.com, and add the xhr query parameter with the
t value, to allow getting responses without requiring trim;
- Use the Java 8 Stream API to collect search suggestions and improve invalid
response detection by checking whether the content type of the response
returned is JSON;
- Move the licence header at the top of the file.
2023-04-30 19:53:42 +02:00
AudricV 945165a3c0
[PeerTube] Don't return "No description" when there is no description for a channel or an account
When a description is missing, no description should be returned, even the ones
indicating there is no description. This behavior is represented by a null
return instead.

Also update PeertubeAccountExtractorTest to reflect these changes.
2023-04-30 18:41:38 +02:00
Stypox 2deb023da4
Merge pull request #1050 from Theta-Dev/fix/channel-carousel-header
[YouTube] Add support for CarouselHeaderRenderer
2023-04-25 15:17:31 +02:00
ThetaDev 47aa9fed40 fix: set musicClientVersion regex capture group 2023-04-16 19:25:05 +02:00
ThetaDev 20370395c5 fix: add support for CarouselHeaderRenderer 2023-04-16 17:40:13 +02:00
Stypox 7dba6e3891
Merge pull request #1033 from petlyh/bandcamp-paywalled-content
[Bandcamp] Handle paywalled tracks
2023-04-12 13:04:26 +02:00
petlyh e6aad117e7
[Bandcamp] Throw PaidContentException on paywalled albums 2023-04-03 19:27:09 +02:00
fynngodau 69705138e4
[Bandcamp] Fix extraction of related playlist items URL (#1047)
Small change in HTML structure
2023-04-02 22:24:29 +02:00
Björn Sigurbergsson 1b6fe5edd6
[YouTube] Fix ParsingException when comments are unavailable in a video (#1040)
Co-authored-by: bjs <bjs@elect-it.com>
Co-authored-by: Audric V. <74829229+AudricV@users.noreply.github.com>
Co-authored-by: Kavin <20838718+FireMasterK@users.noreply.github.com>
2023-03-30 19:58:06 +02:00
ThetaDev 8d1303e18f
Add track types to audio streams (#1041) 2023-03-28 00:02:20 +02:00
petlyh 5a9b6ed2e3
[Bandcamp] Support loading additional comments (#1030) 2023-03-04 14:01:06 +01:00
Stypox 6bdd698c25
Merge pull request #1026 from AudricV/audio-streams-descriptive-and-locale-properties
Add descriptive and locale properties to audio streams
2023-03-01 11:15:46 +01:00
Stypox 19e4b216c9
Merge pull request #1032 from AudricV/yt_fix-comments-hashtags-links-extraction
[YouTube] Fix hashtags links extraction and escape HTML links
2023-03-01 10:47:37 +01:00
Stypox b1298490c0
Merge pull request #1029 from AudricV/yt_fix-no-views-extraction-playlist-items
[YouTube] Fix partial non-extraction of "No views" string in stream items
2023-03-01 10:46:52 +01:00
petlyh 9dc1832733
[Bandcamp] Handle paywalled tracks 2023-02-28 17:51:30 +01:00
AudricV bd79b921e8
[YouTube] Refactor the code to get stream items' view count
This refactoring avoids code duplication as much as possible.
2023-02-27 10:25:46 +01:00
AudricV 51f9b39953
[YouTube] Fix partial non-extraction of no views string in stream items
As the "No views" string is returned in the case there is no view on a video, a
number cannot be parsed in this case, so -1 was returned.

This string is now detected in all methods to get the view count of a stream.
2023-02-27 10:18:45 +01:00
AudricV 30a0f8c510
[MediaCCC] Extract audio language property for single language audio tracks 2023-02-26 19:06:18 +01:00
AudricV 05e8cb39f7
[YouTube] Add language and descriptive audio properties to DASH manifests 2023-02-26 19:06:17 +01:00
AudricV 76b7c19c5d
[YouTube] Extract whether a track is a descriptive audio and audio locale when available
Also use audio track setters only for audio itags.
2023-02-26 19:06:17 +01:00
AudricV 3bb5eeef30
[YouTube] Add descriptive and locale audio support in ItagItem 2023-02-26 19:06:16 +01:00
AudricV 14bf3fb05b
Add ability to know the locale of an audio stream
Getting audio tracks locales by parsing their ID or their label, should not be
done by clients, but by the extractor.

This commit adds the ability to store the Locale of an AudioStream, which is
used to compare similar AudioStreams (in the equalStats method).
2023-02-26 19:06:16 +01:00
AudricV f92426560c
Add descriptive audio properties
Also improve AudioStream's audio language documentation
2023-02-26 19:06:16 +01:00
AudricV 1556adbb2d
[YouTube] Fix hashtags links extraction and escape text in attribute descriptions + HTML links
webCommandMetadata object is contained inside a commandMetadata one, so it is
not accessible from the root of the navigationEndpoint object.

The corresponding statement has been moved at the bottom of the specific
endpoints parsing, as the webCommandMetadata object is present almost
everywhere, otherwise URLs of some endpoints would have be changed, such as
uploader URLs (from channel IDs to handles).

As no ParsingException is now thrown by getUrlFromNavigationEndpoint, and so by
getTextFromObject, getUrlFromObject and getTextAtKey, the methods which were
catching ParsingExceptions thrown by these methods had to be updated.

URLs got in the HTML version of getTextFromObject are now escaped properly to
provide valid HTML to clients. This has been also done for attribute
descriptions, with the description text for this type of descriptions.

As YouTube descriptions are in HTML format (except for the fallback on the JSON
player response, which is plain text and only happens when there is no visual
metadata or a breaking change), all URLs returned are escaped, so tests which
are testing presence of URLs with escaped characters had to be updated (it was
only the case for YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing).
2023-02-26 18:43:36 +01:00
petlyh f7a7a236fb
[Bandcamp] Show comments as disabled on radio streams 2023-02-23 18:42:43 +01:00
TobiGr 3f7df9536e [YouTube] Fix getting the comment text if the comment contains a hashtag 2023-01-29 20:33:51 +01:00
Stypox 999fb7f812
Merge pull request #1024 from AudricV/snd_fix-tracks-like-count
[SoundCloud] Fix extraction of tracks like count
2023-01-29 10:52:54 +01:00
Stypox 3519d4c367
Merge pull request #1015 from AudricV/yt_fix-channel-id-rss-feeds
[YouTube] Fix channel ID extraction of YouTube channels RSS feeds
2023-01-29 10:41:38 +01:00
Stypox 9aca710e86
Merge pull request #1013 from Stypox/fix-music-mixes
[YouTube] Now music mixes can be treated as normal mixes
2023-01-29 09:48:51 +01:00
Stypox 76eeabac45
Merge pull request #1020 from TeamNewPipe/fix/yt-subscriber-count
[YouTube] Fix NPE in search when getting channel items without subscriber count
2023-01-29 09:44:22 +01:00
AudricV 2a24d407d5
[SoundCloud] Fix extraction of tracks like count
SoundCloud is using likes_count to return the like count of a track, like it
was the case before they switched to favoritings_count.
2023-01-29 01:00:49 +01:00
AudricV 57f850bc2d
[YouTube] Support live URLs and do minor improvements to YoutubeStreamLinkHandlerFactory
- Move license header at the top;
- Use an unmodifiable set for the subpaths instead of a modifiable list;
- Add missing Nonnull and Nullable annotations;
- Improve exception messages.
2023-01-28 19:36:20 +01:00
AudricV 1f4ed9dce9
[YouTube] Fix channel ID extraction of YouTube channel RSS feeds
The yt:channelId element doesn't provide the channel ID anymore and is empty,
like the id element, so we need now to extract it from the channel URL provided
in two elements: author -> uri and feed -> link.

Also avoid a NullPointerException in getUrl and getName methods.
2023-01-28 11:53:33 +01:00
Tobi c589a2c1a2
Merge pull request #1014 from TeamNewPipe/fix/yt-comments
[YouTube] Fix getting next comments pages
2023-01-27 11:14:55 +01:00
TobiGr 72573932cf [YouTube] Fix NPE in search when getting channel items without subscriber count 2023-01-24 23:03:45 +01:00
TobiGr f50b7275af [YouTube] Fix getting next comments pages 2023-01-24 22:39:08 +01:00