lasers
/
NewPipeExtractor
mirror of https://github.com/TeamNewPipe/NewPipeExtractor
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
1,961 Commits 11 Branches 72 Tags 112 MiB
Commit Graph

1448 Commits

Author SHA1 Message Date
Stypox 279f3a20fe
[YouTube] Fix mix tests with invalid video ids 
Replaces mix tests based on a strange mix type RDQM{videoId} (only reference I could find is https://github.com/ytdl-org/youtube-dl/issues/26228) and with an invalid video id of 13 characters (the first two characters were QM, but even after removing QM there still wasn't a video available at that id).
Also updates mocks.
 2022-03-19 10:48:13 +01:00
Stypox d660c04838
[YouTube] Also test playlist type in playlist tests 2022-03-19 10:48:13 +01:00
Stypox 401082abe4
[YouTube] Extract playlist type in playlist extractor 2022-03-19 10:48:12 +01:00
Stypox 63ed06a710
[YouTube] Differentiate genre mixes from normal mixes 
Note: genre mixes already worked, now they are just considered as such in various video id extraction and in related items
Note 2: now extracting a mix id from a *normal* youtube mix id will fail if the video id wouldn't be exactly 11 characters long
 2022-03-19 10:46:31 +01:00
Stypox f19660e7d9
[YouTube] Deduplicate code extracting video id from mix id 2022-03-19 10:46:30 +01:00
Stypox 8f9d5b858e
[YouTube] Remove useless comments about mixes 2022-03-19 10:44:06 +01:00
Stypox 34a4484c72
[YouTube] Add test for a video with a mix in related items 2022-03-19 10:44:06 +01:00
Stypox 50db871d89
[YouTube] Extract mixes from streams related items 2022-03-19 10:44:06 +01:00
Stypox 638da1756c
[Mix] Create MultiInfoItemsCollector 
It is a collector that can handle many extractor types, to be used when a list contains items of different types (e.g. search). It was renamed from InfoItemsSearchCollector so that it can now be used not just for search but for any extractor needing it. It supports, streams, channels, playlists and *mixes*.
 2022-03-19 10:44:06 +01:00
Stypox 53673d64c6
[Mix] Add type to playlists & playlist items, to distinguish mixes 2022-03-19 10:44:06 +01:00
Stypox d8f2031619
Merge pull request #816 from Stypox/mock-only-extension 
Add `@MockOnly` Junit 5 extension
 2022-03-19 10:40:38 +01:00
litetex cc2e4d7104
Merge pull request #815 from litetex/fix-soundcloud-id-once-and-for-all 
Removed hardcoded soundcloud HARDCODED_CLIENT_ID
 2022-03-17 13:54:08 +01:00
TiA4f8R c7757c0994
Apply requested changes 2022-03-16 20:14:08 +01:00
TiA4f8R 35e082248e
Fix YouTube and SoundCloud playlists tests 2022-03-16 19:40:30 +01:00
TiA4f8R 8b3f90eb7e
[YouTube] Fix extraction of series playlists and don't return the view count as the stream count for learning playlists 
ITEM_COUNT_UNKNOWN is returned when the JSON array which contains usally the number of videos is less than 3 items.
Also apply the same type of optimizations done in other PlaylistExtractors in YoutubePlaylistExtractor.
 2022-03-16 19:18:58 +01:00
TiA4f8R 58a247907e
Apply changes in all playlist extractors except YoutubePlaylistExtractor 
Also fix some issues in the extractors, remove uneeded overrides, use the Java 8 Stream API where possible and replace usages of Utils.UTF_8 with StandardCharsets.UTF_8 in these classes.
 2022-03-16 19:18:57 +01:00
TiA4f8R fc6b45ee36
Implement some methods in PlaylistExtractor 
This will prevent their override in each child class where the values corresponding to the methods could not be extracted.
 2022-03-16 19:18:36 +01:00
Stypox 73d1fd472f
Add MockOnly junit 5 test extension 2022-03-16 19:03:08 +01:00
Stypox ef71a5fa0f
static final instead of final static 2022-03-16 17:24:33 +01:00
Stypox 0c37c75981
Make getDownloader static & extract getDownloaderType 2022-03-16 17:22:42 +01:00
Stypox 40aa5104b1
Merge pull request #786 from XiangRongLin/throttling_resilience 
[Youtube] Make throttling decryption more resilient to api change
 2022-03-16 11:03:16 +01:00
litetex ba56be8ef1 Removed hardcoded soundcloud id 
It never works (long enough) so let's simply remove it...
 2022-03-15 21:19:19 +01:00
XiangRongLin e726437da3
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java 
Co-authored-by: Stypox <stypox@pm.me>
 2022-03-15 17:10:05 +01:00
litetex e7aee0ca57
Merge pull request #807 from FireMasterK/no-commentsinfo-instance 
Remove the need for a CommentsInfo instance in CommentsInfo.getMoreItems and fix PeertubeCommentsExtractorTest.Default test
 2022-03-15 15:06:56 +01:00
litetex d806984aa8
Merge pull request #800 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.9.0 
Bump gson from 2.8.9 to 2.9.0
 2022-03-14 19:47:54 +01:00
FireMasterK 60cc71e944
Remove the need for a CommentsInfo instance. 2022-03-03 11:48:41 +00:00
Stypox dd8687f9fe
[YouTube] Fix music mixes in some countries 2022-03-01 23:02:56 +01:00
FireMasterK e6d334765d
Apply requested codestyle improvements. 2022-02-26 17:12:51 +00:00
FireMasterK 6950b362f2
Add test values for failing tests. 2022-02-24 12:55:13 +00:00
FireMasterK 5b0ec694a6
Add requested changes. 2022-02-24 12:50:51 +00:00
FireMasterK d290d2e393
Move variable to super Extractor classes. 2022-02-24 12:50:51 +00:00
FireMasterK ab49cb6e18
Add requested changes. 2022-02-24 12:50:51 +00:00
Kavin 4bd59f65f4
Update exception message. 
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
 2022-02-24 12:50:51 +00:00
FireMasterK 6f3f608ab6
Add support for extracting channel subscriber count in StreamInfo. 2022-02-24 12:50:51 +00:00
litetex 1a67ea100a Refactored code 
according to review
 2022-02-21 19:56:12 +01:00
litetex f79ce1f52a Refactored YoutubeCommentsExtractor 
* Use Java Streaming API
* Use StandardCharsets
* Prevented several NPEs/ArrayIndexOutOfBound
* Reformatted some code so that it's easier readable
 2022-02-21 19:56:09 +01:00
TobiGr d337e537dc [YouTube] Add a comment clarifying why the panelIdentfier can be null 2022-02-18 20:26:19 +01:00
Tobi 5c8fbeb166
Fix NullPointerException in YouTubeStreamExtractor.getStreamSegments() (#797) 
Fix NullPointerException in YouTubeStreamExtractor.getStreamSegments()

Co-authored-by: litetex <40789489+litetex@users.noreply.github.com>
 2022-02-16 23:39:26 +01:00
dependabot[bot] 89900431ab
Bump gson from 2.8.9 to 2.9.0 
Bumps [gson](https://github.com/google/gson) from 2.8.9 to 2.9.0.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.9...gson-parent-2.9.0)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-02-14 09:15:09 +00:00
Tobi 0d6aebce25
[SoundCloud] Update clientId (#798) 2022-02-11 17:16:32 +01:00
dependabot[bot] a2a41e3bf2
Bump rhino from 1.7.13 to 1.7.14 
Bumps [rhino](https://github.com/mozilla/rhino) from 1.7.13 to 1.7.14.
- [Release notes](https://github.com/mozilla/rhino/releases)
- [Changelog](https://github.com/mozilla/rhino/blob/master/RELEASE-NOTES.md)
- [Commits](https://github.com/mozilla/rhino/commits)

---
updated-dependencies:
- dependency-name: org.mozilla:rhino
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-02-07 20:29:51 +00:00
XiangRongLin 545522b80f [Youtube] Make throttling decryption more resilient to api change 2022-02-01 19:57:34 +01:00
TobiGr 047d75a4c9 Improve code style 2022-02-01 12:56:59 +01:00
Abdu Ameen 50a4b026f8 Adding removal of the braces 2022-01-31 23:52:31 -08:00
Abdu Ameen a9174f6b9f Changed the regex to account for 
nonword characters
 2022-01-31 22:29:00 -08:00
Stypox 59e0aeba88
Merge pull request #768 from litetex/junit-5 
JUnit 5
 2022-01-24 16:56:16 +01:00
Mohammed Anas 65df39b3e3
[YouTube] parse timestamps with >3 digits correctly (#775) 
* [YouTube] parse timestamps with >3 digits correctly

Fixes https://github.com/TeamNewPipe/NewPipe/issues/7530; check the
issue for details.

* Remove outdated comment
 2022-01-12 10:45:06 +01:00
litetex a579ef2651 Fixed Soundcloud tests 2022-01-07 14:27:24 +01:00
litetex a32edce750 Removed deprecation because AndroidSDK/API < 19 2022-01-07 14:15:55 +01:00
litetex 602068937d
Code cleanup 
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
 2022-01-06 21:04:16 +01:00
litetex dd3f3da828 Kurzgesagt mock data is outdated 2022-01-04 17:44:12 +01:00
litetex a8ebd2cead Update mock resources for YT Mix playlist 2022-01-04 17:39:14 +01:00
litetex 066afd8629 Restored test 2022-01-04 17:33:08 +01:00
litetex 3712a669b1 Code cleanup / improvements 
* Deprecated Utils#UTF-8; see StandardCharsets
* Added more helpful methods to ``ExtractorAsserts``
* Use parameterized (cool new) tests
* Restore functionality of some tests + updated mockdata
* Other code cleanups + Sonarlint improvements
 2022-01-04 17:28:31 +01:00
litetex 4291a90251 Fixed SoundCloud's search(for tests) 2021-12-27 21:42:43 +01:00
litetex 4995709871 Fixed SoundCloud's search(for tests) 
Getting the initial page was not returning initial page
 2021-12-27 21:18:39 +01:00
litetex a2cbdf0991 Updated to JUnit 5 2021-12-27 21:08:08 +01:00
litetex 148ba6641d Updated mock data 2021-12-27 17:08:14 +01:00
litetex 4d0a689cb6 Disabled dislike count in tests 2021-12-27 17:04:46 +01:00
litetex a6961c5efa Fixed NumberFormatException when processing likes 
See also #7525
 2021-12-27 16:54:47 +01:00
litetex 65687f3b9b Removed YT dislike count as it no longer works 2021-12-27 16:54:46 +01:00
litetex 1e2e0029fc [StreamExtractor] Deduplicated a ton of code by using default methods 2021-12-27 16:54:45 +01:00
litetex 15b98ffdb4 Remove old ways of getting YT dis/likes 
* Added additional check for averageRating (in dislikes)
 2021-12-27 16:54:44 +01:00
litetex 7489502cc8 Disabled ``testRelatedItems`` as it's unreliable 2021-12-11 15:23:11 +01:00
litetex d9b2a2042a Updated mock data for YoutubeCommentsExtractorTest 2021-12-08 22:11:45 +01:00
litetex a3a0c1ee85 Fixed YoutubeCommentsExtractorTest 
Old video was deleted / set to private
 2021-12-08 22:11:34 +01:00
litetex 5c288eb8d7 Updated mock data for YoutubeChannelExtractorTest 2021-12-08 21:51:38 +01:00
litetex 75bb83a1b8 Fixed YoutubeChannelExtractorTest 
Channel was restored, picked a new one
 2021-12-08 21:51:15 +01:00
litetex f276bacce5 Fixed SoundcloudStreamExtractorTest 2021-12-08 21:41:27 +01:00
litetex 7f99c1c193 Fixed YoutubeStreamExtractorLivestreamTest 2021-12-08 21:16:39 +01:00
litetex 652fdf2c36 Updated mock-data for YoutubeStreamExtractorAgeRestrictedTest 2021-12-08 21:08:25 +01:00
litetex 3c669d3cb5 Fixed YoutubeStreamExtractorAgeRestrictedTest 
Looks like a lot of stuff (description, tags, name) got changed on the existing video.
Use a better suited video.
 2021-12-08 21:08:03 +01:00
litetex f735788a39 Fixed Soundcloud hardcoded client id 2021-12-08 21:00:53 +01:00
bopol 56c8af710e Code refactoring 2021-11-21 18:16:31 +01:00
bopol c4eca91be9 Fix YouTube likes + dislikes 2021-11-19 21:36:03 +01:00
TobiGr 5028396405 Improve names and formatting 2021-11-14 13:44:13 +01:00
Tobi 68d7e4e20a
Merge pull request #742 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.8.9 
Bump gson from 2.8.8 to 2.8.9
 2021-11-08 19:37:09 +01:00
opusforlife2 cb80a646d9
Update hardcoded SoundCloud client_id 
Hah! I got here first, @TobiGr!
 2021-11-04 15:28:26 +00:00
bopol 396aecef19 Make sure playlists aren't accepted by PeertubeStreamLinkHandlerFactory 2021-11-03 14:41:39 +01:00
bopol 38ad1eaac5 [PeerTube] Support /w/ short video links part 2 
We also need to support it for comments
 2021-11-03 13:54:37 +01:00
bopol 026751624f [PeerTube] Support /w/p/ short playlist links 2021-11-03 13:35:30 +01:00
bopol e0b8e142fc [PeerTube] Support /a/ and /c/ short links 2021-11-03 13:26:27 +01:00
bopol 599a91c88c [PeerTube] Support /w/ short video links 2021-11-03 11:51:58 +01:00
dependabot[bot] 198d37090b
Bump gson from 2.8.8 to 2.8.9 
Bumps [gson](https://github.com/google/gson) from 2.8.8 to 2.8.9.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.8...gson-parent-2.8.9)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-01 09:12:53 +00:00
mhmdanas 3e8e2a1532 Add support for y2u.be links 2021-10-22 22:48:18 +03:00
Tobi b425394e7a
Merge pull request #731 from FireMasterK/short-description 
Extract Video Short Description in YouTube.
 2021-10-15 23:10:00 +02:00
FireMaskterK 62313962a0 Add default implementation for getShortDescription. 2021-10-04 01:42:29 +01:00
Kavin d1add05bc1 Add requested changes. 
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
Add final.

Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
 2021-10-04 01:36:51 +01:00
dependabot[bot] 3ca9f51b30
Bump jsoup from 1.14.2 to 1.14.3 
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.14.2 to 1.14.3.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.14.2...jsoup-1.14.3)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-02 08:20:38 +00:00
TobiGr d8ec3ce370 [SoundCloud] Update client id 2021-10-01 02:51:38 +02:00
FireMasterK 94efe86c71
Extract Video Short Description in YouTube. 
In Trending, and Search results.
 2021-09-22 16:26:16 +01:00
Tobi a9d214478d
Merge pull request #703 from FireMasterK/comment-replies 
Add support for extracting comment replies continuation
 2021-09-14 23:58:14 +02:00
FireMasterK 6aabdc6d16
Fix for requested changes. 2021-09-12 01:15:19 +05:30
Kavin 0aad09fa22
Update JavaDocs for comment replies continuation. 
Co-authored-by: Stypox <stypox@pm.me>
 2021-09-12 01:05:20 +05:30
FireMasterK feb09e9997
Extract Channel Avatar in related videos. 2021-09-04 01:09:40 +05:30
Tobi f0aa46b008
Merge pull request #724 from FireMasterK/streams-uploader-verified 
Extract uploaderVerified to StreamInfo.
 2021-09-03 18:18:04 +02:00
FireMasterK 857b2f39e8
Extract uploaderVerified to StreamInfo. 2021-09-03 21:27:58 +05:30
FireMasterK 1db463b55f
Add a default test for the Uploader's Avatar. 2021-09-03 21:05:36 +05:30
Kavin db6b3b2c29
Extract uploader's avatar in peertube. 
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
 2021-09-02 16:14:45 +05:30
FireMasterK a0c1dcc8d8
Remove throws parsing exception. 2021-09-02 16:14:45 +05:30