9483dcd9fa
[YouTube] Update mocks of YoutubeCommentsExtractorTest.RepliesTest
2023-02-26 18:43:36 +01:00
1556adbb2d
[YouTube] Fix hashtags links extraction and escape text in attribute descriptions + HTML links
...
webCommandMetadata object is contained inside a commandMetadata one, so it is
not accessible from the root of the navigationEndpoint object.
The corresponding statement has been moved at the bottom of the specific
endpoints parsing, as the webCommandMetadata object is present almost
everywhere, otherwise URLs of some endpoints would have be changed, such as
uploader URLs (from channel IDs to handles).
As no ParsingException is now thrown by getUrlFromNavigationEndpoint, and so by
getTextFromObject, getUrlFromObject and getTextAtKey, the methods which were
catching ParsingExceptions thrown by these methods had to be updated.
URLs got in the HTML version of getTextFromObject are now escaped properly to
provide valid HTML to clients. This has been also done for attribute
descriptions, with the description text for this type of descriptions.
As YouTube descriptions are in HTML format (except for the fallback on the JSON
player response, which is plain text and only happens when there is no visual
metadata or a breaking change), all URLs returned are escaped, so tests which
are testing presence of URLs with escaped characters had to be updated (it was
only the case for YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing).
2023-02-26 18:43:36 +01:00
f7a7a236fb
[Bandcamp] Show comments as disabled on radio streams
2023-02-23 18:42:43 +01:00
f5599ff08d
Bump org.jsoup:jsoup from 1.15.3 to 1.15.4
...
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup ) from 1.15.3 to 1.15.4.
- [Release notes](https://github.com/jhy/jsoup/releases )
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES )
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.3...jsoup-1.15.4 )
---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-02-20 10:05:26 +00:00
3f7df9536e
[YouTube] Fix getting the comment text if the comment contains a hashtag
2023-01-29 20:33:51 +01:00
999fb7f812
Merge pull request #1024 from AudricV/snd_fix-tracks-like-count
...
[SoundCloud] Fix extraction of tracks like count
2023-01-29 10:52:54 +01:00
3519d4c367
Merge pull request #1015 from AudricV/yt_fix-channel-id-rss-feeds
...
[YouTube] Fix channel ID extraction of YouTube channels RSS feeds
2023-01-29 10:41:38 +01:00
9aca710e86
Merge pull request #1013 from Stypox/fix-music-mixes
...
[YouTube] Now music mixes can be treated as normal mixes
2023-01-29 09:48:51 +01:00
76eeabac45
Merge pull request #1020 from TeamNewPipe/fix/yt-subscriber-count
...
[YouTube] Fix NPE in search when getting channel items without subscriber count
2023-01-29 09:44:22 +01:00
676622f6df
[SoundCloud] Fix expectedLikeCountAtLeast tests of SoundcloudStreamExtractorTest test classes
...
As like count is now returned by the extractor, we need to assert a positive
minimum like count, which is close to the actual value, in order to avoid test
failures due to lower like counts than the ones excepted.
2023-01-29 01:08:02 +01:00
2a24d407d5
[SoundCloud] Fix extraction of tracks like count
...
SoundCloud is using likes_count to return the like count of a track, like it
was the case before they switched to favoritings_count.
2023-01-29 01:00:49 +01:00
ba24976e41
[YouTube] Add live URLs test and do minor improvements to YoutubeStreamLinkHandlerFactoryTest
...
- Remove unused imports;
- Replace wildcard imports by single class imports;
- Suppress "HTTP links are not secured" warnings from IDEA IDEs;
- Replace removed video jZViOEv90dI by an existing video, 9Dpqou5cI08 (the
corresponding test has been of course renamed).
2023-01-28 19:36:21 +01:00
57f850bc2d
[YouTube] Support live URLs and do minor improvements to YoutubeStreamLinkHandlerFactory
...
- Move license header at the top;
- Use an unmodifiable set for the subpaths instead of a modifiable list;
- Add missing Nonnull and Nullable annotations;
- Improve exception messages.
2023-01-28 19:36:20 +01:00
1f4ed9dce9
[YouTube] Fix channel ID extraction of YouTube channel RSS feeds
...
The yt:channelId element doesn't provide the channel ID anymore and is empty,
like the id element, so we need now to extract it from the channel URL provided
in two elements: author -> uri and feed -> link.
Also avoid a NullPointerException in getUrl and getName methods.
2023-01-28 11:53:33 +01:00
c589a2c1a2
Merge pull request #1014 from TeamNewPipe/fix/yt-comments
...
[YouTube] Fix getting next comments pages
2023-01-27 11:14:55 +01:00
72573932cf
[YouTube] Fix NPE in search when getting channel items without subscriber count
2023-01-24 23:03:45 +01:00
f50b7275af
[YouTube] Fix getting next comments pages
2023-01-24 22:39:08 +01:00
9bdad40b06
Removed topStandaloneBadge
2023-01-20 02:41:21 +05:30
5945057227
[YouTube] Add music mix test
2023-01-15 23:30:30 +01:00
7293991832
[YouTube] Now music mixes can be treated as normal mixes
...
Using a playlist extractor on them would result in "Unviewable playlist" errors
2023-01-15 23:28:59 +01:00
ff94e9f30b
Merge pull request #1009 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.10.1
...
Bump gson from 2.10 to 2.10.1
2023-01-11 15:35:36 +01:00
c1040bccac
Merge pull request #794 from FireMasterK/comments-count
...
[YouTube] Add support to extract total comment count
2023-01-11 15:32:19 +01:00
f43049985e
Bump gson from 2.10 to 2.10.1
...
Bumps [gson](https://github.com/google/gson ) from 2.10 to 2.10.1.
- [Release notes](https://github.com/google/gson/releases )
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md )
- [Commits](https://github.com/google/gson/compare/gson-parent-2.10...gson-parent-2.10.1 )
---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-01-09 09:05:55 +00:00
56aab4d971
[YouTube] Fix escaping links in YouTubeParsingHelper.getTextFromObject
2023-01-05 00:28:12 +01:00
22a47da8c7
Fix requested change and remove outdated comment.
2023-01-02 20:42:32 +00:00
98a90fd9c8
Don't cache comments count and return early on page fetch if no token.
2023-01-02 20:40:48 +00:00
2974dfaa48
Only store ajaxJson for initial page and eager fetch the initial continuation.
2023-01-02 20:40:48 +00:00
64d24aa09e
Fix request changes.
2023-01-02 20:40:48 +00:00
67ef4f4c30
Cleanup and remove optional.
2023-01-02 20:40:48 +00:00
22f71b010c
Fix for requested changes.
2023-01-02 20:40:48 +00:00
656b7c1cd9
Improve method documentation.
2023-01-02 20:40:48 +00:00
981aee4092
Add support to extract total comment count.
2023-01-02 20:40:48 +00:00
45636b0d00
Merge pull request #986 from Isira-Seneviratne/Static_maps
...
Use immutable Map factory methods.
2023-01-02 18:11:14 +01:00
219c5c5be5
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java
2023-01-02 18:11:03 +01:00
259de3cba6
Merge pull request #995 from TeamNewPipe/feat/soundcloud-playlistinfoitemextractor
...
[SoundCloud] Implement getUploaderUrl() and isUploaderVerified() for PlaylistInfoItemExtractor
2023-01-02 15:10:40 +01:00
991394b53a
Merge pull request #1005 from FireMasterK/fix-escaping-xss
...
Fix for potential XSS attacks and formatting issues
2023-01-02 15:06:17 +01:00
d8ce08d969
Use immutable Map factory methods.
2023-01-02 07:50:31 +05:30
01acf79436
Fix for potential XSS attacks.
2022-12-31 20:05:32 +00:00
292e0d8ce7
[SoundCloud] Implement getUploaderUrl() and isUploaderVerified() for PlaylistInfoItemExtractor
2022-12-31 18:46:39 +01:00
2a8729aeb2
Apply suggestions
...
Co-authored-by: Stypox <stypox@pm.me>
2022-12-31 18:24:33 +01:00
d75a997611
[PeerTube] Support searching for channels
2022-12-31 18:24:33 +01:00
dea6d8ce4c
[PeerTube] Support searching for playlists
2022-12-31 18:24:33 +01:00
95cc6aefbb
Merge pull request #994 from TeamNewPipe/fix/peertube-subtitles-exception
...
[PeerTube] Report Exceptions thrown while getting a stream's subtitles
2022-12-31 15:01:39 +01:00
7b54457789
Merge pull request #941 from TeamNewPipe/feat/peertube-comment-replies
...
[PeerTube] Support comment replies
2022-12-31 14:57:51 +01:00
f45966d449
Merge pull request #910 from Isira-Seneviratne/Locale_forLanguageTag
...
Add compat Locale.forLanguageTag() implementation.
2022-12-24 23:53:30 +01:00
d5437e0bc5
Merge pull request #863 from AudricV/add-content-type-and-content-length-headers-to-post-requests
...
Add Content-Type header to all POST requests without an empty body
2022-12-16 19:32:56 +01:00
0766b1d211
[YouTube] Improve YoutubeStreamInfoItemExtractor
...
- Return duration of video premieres;
- Add another non-localized method to determine whether a stream is a running
livestream;
- Return view count and upload date of videos in playlists;
- Store isPremiere result;
- Remove shorts workaround code, as it was only useful on channels and shorts
have been moved into a separated channel tab;
- Improve some other code.
2022-12-08 13:59:12 +01:00
896d7e09eb
Merge pull request #978 from Theta-Dev/fix/search-channel-handles
...
[YouTube] Fix search subscriber count extraction with channel handles
2022-12-05 17:52:05 +01:00
cd3262745d
[PeerTube] Report Exceptions thrown while getting a stream's subtitles
2022-12-03 16:11:21 +01:00
4e66b2287e
[PeerTube] Add support for comment replies
2022-12-01 14:05:18 +01:00
41c8dce452
Merge pull request #992 from Isira-Seneviratne/String_isBlank
...
Use String.isBlank().
2022-11-30 17:48:54 +01:00
2bca56f0df
Use String.isBlank().
2022-11-30 08:26:21 +05:30
3b80547976
Add code review suggestions.
2022-11-30 07:57:45 +05:30
016623131e
docs: update comment in YoutubeChannelInfoItemExtractor
2022-11-29 19:06:03 +01:00
2e08eaad96
Fix complication error in comment test.
2022-11-29 16:07:48 +00:00
abf08e1496
Merge pull request #990 from FireMasterK/bold-italic-strikethrough
...
[YouTube] Implement bold/italic/strike-through support
2022-11-29 15:59:38 +00:00
57e7a6fb7c
Add mocks test.
2022-11-28 20:27:55 +00:00
1d3d7fa5c3
Add test for formatting.
2022-11-28 20:26:37 +00:00
52fda37915
Implement bold/italic/strike-through support.
2022-11-28 19:06:18 +00:00
b566084cac
Use Description object for comments text.
2022-11-28 17:02:19 +00:00
f8162b049d
Merge pull request #984 from FireMasterK/unused-dep
...
Remove unused autolink dependency
2022-11-28 11:28:42 +01:00
1da0190056
Merge pull request #980 from TeamNewPipe/fix/yt/unavailable
...
[YouTube] Fix extracting the detailed error message for unavailable streams
2022-11-28 10:07:34 +01:00
60fb30f835
Merge pull request #928 from FireMasterK/comment-urls
...
Parse YouTube comments as HTML
2022-11-27 19:16:34 +01:00
5abea22225
Fix throwing correct reason.
2022-11-26 21:09:08 +00:00
faf28f5c11
Remove unused dependency.
2022-11-26 20:17:25 +00:00
c043597255
Update supported countries list.
2022-11-26 19:01:33 +00:00
4680df0bdf
Fix throwing correct reason
2022-11-23 17:03:22 +01:00
9de8405c9f
[YouTube] Fix extracting the detailed error message of streams which are unavailable
2022-11-23 08:33:06 +01:00
34d79bd267
[YouTube] Update mocks
2022-11-22 17:10:04 +01:00
2ec296e674
Fix YoutubeSearchExtractorTest.MetaInfoTest
...
Not all the "learn more" button is uppercase anymore, that's only the case for
the first letter.
2022-11-22 16:34:54 +01:00
3891542ca1
Use Downloader's postWithContentType and postWithContentTypeJson methods in services and extractors
2022-11-22 11:37:18 +01:00
b2862f3cd1
Add postWithContentType and postWithContentTypeJson utility methods in Downloader
...
Co-authored-by: Stypox <stypox@pm.me>
2022-11-22 11:37:17 +01:00
e9a0d3bd95
[YouTube] Send Content-Type header in all POST requests
...
This header was not sent partially before and was added and guessed by OkHttp. This can create issues when using other HTTP clients than OkHttp, such as Cronet.
Some code in the modified classes has been improved and / or deduplicated, and usages of the UTF_8 constant of the Utils class has been replaced by StandardCharsets.UTF_8 where possible.
Note that this header has been not added in except in YoutubeDashManifestCreatorsUtils, as an empty body is sent in the POST requests made by this class.
2022-11-22 11:37:16 +01:00
b9e463de49
[Bandcamp] Send Content-Type header in POST requests
...
This header was not sent before and was added and guessed by OkHttp. This can create issues when using other HTTP clients than OkHttp, such as Cronet.
Also make use of StandardCharsets.UTF_8 when getting bytes of bodies instead of the platform default's charset, to make sure to prevent some encoding issues on some JVMs.
2022-11-22 11:35:46 +01:00
65d6321e3d
Fix typos in Downloader.post JavaDocs
...
Post methods in Downloader return the result of a POST request and not the one of a GET request.
2022-11-22 11:35:46 +01:00
5daabd1793
fix : #976 search subscriber count extraction with channel handles
2022-11-22 02:17:10 +01:00
c953e23414
Merge pull request #968 from AudricV/yt-support-no-video-info-renderers-for-streams
...
[YouTube] Support lack of video info renderers for streams
2022-11-16 20:20:01 +00:00
2211a24b69
Merge pull request #971 from lrusso96/patch-1
...
[YouTube] Improve duration parsing
2022-11-16 16:14:54 +01:00
86f06b333a
Address review.
2022-11-14 00:05:31 +00:00
b16e6082e1
Add test for audio stream languages.
2022-11-13 23:10:44 +00:00
30909da1df
Fix audio track similar comparison for IDs.
2022-11-13 23:08:54 +00:00
6d59cdbe3a
Add support for extracting audio tracks.
2022-11-13 21:39:29 +00:00
e4d982c7ea
Fix license.
2022-11-12 07:29:15 +05:30
416089146e
Fix failing tests.
2022-11-12 07:29:15 +05:30
ddbce3b83d
Add Utils methods for URL encoding/decoding using UTF-8.
2022-11-12 07:29:15 +05:30
366f5c1632
Use StandardCharsets.UTF_8.
2022-11-12 07:29:15 +05:30
0b82fade51
Bump gson from 2.9.1 to 2.10
...
Bumps [gson](https://github.com/google/gson ) from 2.9.1 to 2.10.
- [Release notes](https://github.com/google/gson/releases )
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md )
- [Commits](https://github.com/google/gson/compare/gson-parent-2.9.1...gson-parent-2.10 )
---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-11-09 16:21:16 +00:00
c9635218e2
[YouTube] Improve duration parsing
2022-11-09 09:41:29 +01:00
316d8573fa
Use immutable sets in YoutubeParsingHelper.
2022-11-07 07:50:26 +05:30
6a2c680d8f
[YouTube] Add mocks for YoutubeStreamExtractorDefaultTest.NoVisualMetadataVideoTest
2022-11-04 19:43:06 +01:00
e66fed41d6
[YouTube] Add a StreamExtractor test for a video without visual metadata
...
The video "Makani’s first commercial-scale energy kite" (video ID:
An8vtD1FDqs), which has this behavior, is used for the new test,
NoVisualMetadataVideoTest, added in YoutubeStreamExtractorDefaultTest.
Tests of elements who throw an exception in this case (subscriber count, like
count, uploader avatar URL) test if the ParsingException exception is thrown by
YoutubeStreamExtractor.
2022-11-04 19:42:12 +01:00
aa9a8ca23c
[YouTube] Make non-extraction of videoPrimaryInfoRenderer and/or videoSecondaryInfoRenderer not fatal
...
Also de-duplicated common code related to the obtain of these video info renderers.
This change allows extraction of videos without visual metadata.
2022-11-04 18:35:53 +01:00
20cd8e8a4a
[YouTube] Update mocks of YoutubeChannelExtractorTest.Gronkh
2022-11-03 19:46:42 +01:00
a34f060ba0
[YouTube] Use a handle for YoutubeChannelExtractorTest.Gronkh
2022-11-03 19:46:42 +01:00
724f669ff7
[YouTube] Add tests for handles and user IDs with non ASCII characters support
...
Unneeded public modifiers in test methods of
YoutubeChannelLinkHandlerFactoryTest have been also removed.
2022-11-03 19:46:42 +01:00
61ce041bda
[YouTube] Support handles and all custom channel names
...
More non-channel paths have been also added to the excluded custom name paths,
documentation and exception messages have been improved and fixed in some
places, and the licence header of YoutubeChannelLinkHandlerFactory has been
moved to its beginning and updated.
2022-11-03 19:46:42 +01:00
ffffb04439
Merge pull request #953 from Theta-Dev/attributed-text-desc
...
[YouTube] Add support for attributed text description
2022-11-03 18:34:30 +01:00
592e1d6386
fix: parsing attributed description with no command runs
2022-11-03 12:10:52 +01:00
099b53cc4f
[YouTube] Add parser for attributedDescription
...
Also update the mock of the next InnerTube endpoint response of the
YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing test class with an
attributedDescription instead of a regular description
2022-11-02 23:11:33 +01:00
e4c24d4c36
[YouTube] Regenerate supported channels mocks
2022-11-02 19:13:59 +01:00
AudricV
petlyh
dependabot[bot]
TobiGr
Stypox
Kunal
Kavin
Isira Seneviratne
ThetaDev
Luigi Russo