bopol
c47cc54908
Extract metadata for YouTube, SoundCloud & MediaCCC
2021-03-19 10:14:43 +01:00
TiA4f8R
0e3e420a25
Fix tests
2021-03-14 17:54:21 +01:00
TiA4f8R
0438828e36
Add a test for the number of audioStreams in CreativeCommonsPlaysWellWithOthers test
...
It should be only two audio streams for track "Plays Well with Others, Ep 2: What Do an Army of Ants and an Online Encyclopedia Have in Common?" by Creative Commons (https://soundcloud.com/wearecc/plays-well-with-others-ep-2-what-do-an-army-of-ants-and-an-online-encyclopedia-have-in-common ):
- one which is a progressive stream, in MP3 format with a bitrate of 128 kbps
- one which is an HLS stream, in OPUS format with a bitrate of 64 kbps.
2021-03-14 17:54:18 +01:00
Fynn Godau
2e57a8f24f
[Bandcamp] Fix link handler acceptance behaviour
...
* Test for bandcamp footer instead of meta tag (which is not present on
all pages)
* Accept links to /music, not just /releases
* Correctly handle uppercase URLs
2021-03-14 09:51:30 +01:00
TobiGr
70d9e389b9
[Bandcamp] Add tests for external bandcamp artists
...
Necessary, because the external pages tested before were converted to bandcamp.com pages. See f9d06252f2
2021-03-13 20:51:59 +01:00
TobiGr
f9d06252f2
[Bandcamp] Fix tests
2021-03-13 17:42:39 +01:00
Tobi
7e6f464407
Merge pull request #509 from TiA4f8R/soundcloud-improvements
...
Add new exceptions to be able to display different error messages in apps
2021-03-07 13:51:15 +01:00
Tobi
ac51134aed
Merge pull request #546 from XiangRongLin/unignore2
...
Unignore more tests
2021-03-07 13:15:24 +01:00
TobiGr
22fa131922
Merge branch 'dev' into bandcamp
2021-03-05 19:45:37 +01:00
TobiGr
22e6f33f0a
Ignore SoundCloudGeoRestrictedTest until HLS stream extraction is added
...
The ContentNotSupportedException is thrown because no supported audio streams where extracted. However, SoundCLoud does not check, whether there are any streams available.
This commit should be reverted in #526
2021-03-05 16:38:47 +01:00
TobiGr
21158744b1
Add expectedDescriptionIsEmpty() to DefaultStreamExtractorTest
...
Also check if related streams are empty if they are expected to be empty.
2021-03-05 16:38:46 +01:00
TiA4f8R
f15d7837a1
Fix SoundCloud tests
2021-03-05 16:38:45 +01:00
TiA4f8R
4e45aef2b3
Fix SoundCloud CreativeCommonsPlaysWellWithOthers test
2021-03-05 16:38:43 +01:00
TiA4f8R
3925204658
Add tests and rename SoundcloudGoPlusException to SoundcloudGoPlusContentException
2021-03-05 16:38:40 +01:00
XiangRongLin
beb05bd05c
Replace test url, because playlist was deleted
2021-03-03 19:46:14 +01:00
XiangRongLin
e13e237392
Fix typo and reword some explanations
2021-02-26 17:48:03 +01:00
TobiGr
70814dcfef
Fix Utils.nonEmptyAndNullJoin
...
When using the index here, it the index needs to be decremented once an element is removed. To cirecumvent this, the native Collections.removeIf() method is used.
2021-02-21 12:51:47 +01:00
FireMasterK
e8bccfaf5d
Change UA to privacy.resistFingerprinting.
2021-02-20 17:14:41 +05:30
bopol
0c831afa7b
use DownloaderFactory instead of DownloaderTestImpl for search ChannelVerified
2021-02-20 11:41:30 +01:00
XiangRongLin
adf9d7d10f
Add reason field to MockOnly
...
This enforces developers to document why a test is skipped
2021-02-19 18:54:18 +01:00
XiangRongLin
ea52030613
Add MockOnlyRule to allow skipping specific tests based on downloader
2021-02-19 18:54:17 +01:00
TobiGr
02920fafa8
Add isUploaderVerified()
2021-02-19 15:51:23 +01:00
TobiGr
91e9309486
Merge remote-tracking branch 'origin/dev' into bandcamp
2021-02-19 13:37:45 +01:00
TobiGr
fa61b864f2
Code improvements
2021-02-19 13:37:24 +01:00
bopol
1a322ad8ed
Add uploader verified by service extraction
2021-02-18 16:36:40 +01:00
XiangRongLin
a64579da2f
Unignore some tests that work on Linux
2021-02-17 20:31:26 +01:00
XiangRongLin
b99cb8624f
Unignore PublicBroadcasterTest
...
Tests only work on linux, mocks have to be generated on linux too
2021-02-17 20:31:26 +01:00
XiangRongLin
37366c65ca
Remove encoding of '&' to '&' in test and unignore it
2021-02-17 20:31:26 +01:00
XiangRongLin
c327922cc0
Add explanations to broken tests
2021-02-17 20:31:26 +01:00
TobiGr
54aa5b3042
Use propper structure in KioskExtractors
...
Made BandCampRadioExtractor a Kiosk which holds StreamInfoItems and not InfoItems.
2021-02-16 21:09:42 +01:00
bopol
84380e4c3a
Merge pull request #536 from B0pol/watchendpoints
...
support /watch/, /v/ and /w/ subpaths
2021-02-16 20:23:27 +01:00
XiangRongLin
a9594f531f
Merge pull request #543 from XiangRongLin/unignore1
...
Unignore and fix some tests
2021-02-16 18:00:16 +01:00
XiangRongLin
a28e0226c5
Merge pull request #519 from XiangRongLin/better_logging_error_handling
...
Better logging, error handling, doc for mock tests
2021-02-16 15:06:06 +01:00
XiangRongLin
05cb22c654
Unignore Gronkh.testDescription
...
Remove special character ü from assertion
2021-02-16 08:59:35 +01:00
XiangRongLin
d9b4c2779f
Unignore YoutubeSearchQHTest.testRegularValues
2021-02-16 08:59:35 +01:00
XiangRongLin
137272b7a4
Unignore YoutubeCommentsExtractorTest.EmptyComment
...
Mock files were missing
2021-02-16 08:59:34 +01:00
TobiGr
5bf9fddba9
Code improvements
2021-02-15 22:45:17 +01:00
TobiGr
e062c8cb0d
Merge branch 'dev' into bandcamp
2021-02-15 22:09:41 +01:00
bopol
557934cb17
use EMPTY_STRING
2021-02-15 18:51:46 +01:00
bopol
9ca52ca68a
use only one constant UTF-8
2021-02-15 18:47:54 +01:00
XiangRongLin
a88b2e6af4
Undo formatting of YoutubeStreamExtractorDefaultTest
2021-02-14 14:57:23 +01:00
bopol
81317cf6bc
youtube: support /w/ folder
2021-02-13 22:20:05 +01:00
bopol
f15c0fcfed
support /watch/ and /v/ folders
2021-02-13 22:20:05 +01:00
XiangRongLin
971128c464
Ignore failing tests
2021-02-13 20:28:03 +01:00
XiangRongLin
d6b9930c18
Use DownloaderFactory in YoutubeCommentsExtractorTest.Pinned
2021-02-13 20:12:39 +01:00
XiangRongLin
b555f38fa5
Use real downloader for YoutubeSubscriptionExtractorTest
...
It doesn't make network requests anyway
2021-02-13 20:12:39 +01:00
XiangRongLin
f352b0c7af
YoutubeMusicSearchExtractorTest can't be used with mocks
...
Requires further research
2021-02-13 20:00:51 +01:00
XiangRongLin
f45b5610bf
Unignore YoutubeChannelLocalizationTest
...
With mocks time is not a concern anymore
2021-02-13 20:00:51 +01:00
XiangRongLin
19737e06ee
Use DownloaderFactory in YoutubeChannelExtractorTest
...
KurzgesagtAdditional needed to be extracted because it ran a duplicated request.
2021-02-13 20:00:51 +01:00
XiangRongLin
84eb4b30c9
Use DownloaderFactory in YouTube tests
2021-02-13 20:00:51 +01:00
bopol
73353996d7
fix more tests
2021-02-12 23:18:38 +01:00
bopol
44c54d403a
fix Invidious channels being accepted by YoutubeStreamLinkHandlerFactory ( #527 )
...
fixes https://github.com/TeamNewPipe/NewPipeExtractor/issues/524 , see it for more details
2021-02-05 18:57:58 +01:00
TiA4f8R
1414a6f178
Fix extraction of YouTube chapters ( #531 )
...
YouTube changed the name of the chapters in the JSON from engagement-panel-macro-markers to engagement-panel-macro-markers-description-chapters, so extracting chapters doesn't work.
2021-02-03 22:07:34 +01:00
bopol
2662ceca22
rename getHeartedByCreator and getPinned to isPinned and isHeartedByCreator
2021-01-23 13:06:07 +01:00
bopol
66e4eb2f96
extract whether the comment is pinned
2021-01-22 23:59:22 +01:00
XiangRongLin
88e4c8667b
Add comments to RecordingDownloader about usage
2021-01-17 19:53:20 +01:00
XiangRongLin
eecfe09f2c
Check for non-existent folder in MockDownloader
2021-01-17 19:52:58 +01:00
TobiGr
07570554b6
Fix build and optimize imports
2021-01-15 21:55:40 +01:00
TobiGr
78c2113094
Merge remote-tracking branch 'origin/dev' into bandcamp
2021-01-15 21:49:58 +01:00
XiangRongLin
6888e7773f
Ignore all failing tests
2021-01-15 20:11:40 +01:00
Stypox
7c774c84ce
Merge pull request #505 from TeamNewPipe/meida.ccc.de_search_fix
...
[media.ccc.de] Fix NPE in search results if they contain a future talk
2021-01-15 14:16:59 +01:00
Stypox
c2c4d97efd
Merge pull request #482 from XiangRongLin/mock_mix_pl_test
...
Mock mix pl tests
2021-01-15 12:54:48 +01:00
Tobias Groza
cc51c5f6a3
Merge pull request #507 from vkay94/add-interval-to-stream-frames
...
Add durationPerFrame to Frameset
2021-01-14 20:27:40 +01:00
vkay94
11dcfe638b
Add durationPerFrame to Frameset and getFrameBoundsAt method
2021-01-14 20:01:06 +01:00
Tobias Groza
c2ff6723d7
Merge pull request #511 from B0pol/errors
...
Fix PeerTube tests by changing instance
2021-01-14 19:15:22 +01:00
XiangRongLin
4af50c6870
Use @link for downloader
2021-01-12 09:41:57 +01:00
XiangRongLin
3455f0f23c
Add documentation to testing with mocks
2021-01-12 09:37:48 +01:00
bopol
5292cc1636
[peertube] test: remove usage of peertube.mastodon.host
...
It's down for months
2021-01-11 19:46:27 +01:00
bopol
f3c22da618
comments: add heartedByCreator and extract it for youtube
2021-01-11 18:48:08 +01:00
XiangRongLin
63c237db41
Only create directories if they do not exist
2021-01-11 07:57:11 +01:00
XiangRongLin
e82cb1efea
Remove dependency to commons-io
2021-01-10 20:50:38 +01:00
XiangRongLin
1ea6c6ce54
Prefix mock file from RecordingDownloader with "generated_mock_"
...
Only read those files in MockDownloader
2021-01-10 20:46:42 +01:00
XiangRongLin
f91916c017
Remove unnecessary file object creation
2021-01-10 20:39:22 +01:00
XiangRongLin
255c726f20
Add resource path base to DownloaderFactory
2021-01-10 20:27:34 +01:00
XiangRongLin
35e299759e
Add method to allow resetting youtube client version and key
...
This is needed so that a request is made for each test class when running multiple at once. This way RecordingDownloader records all necessary requests.
This works as long as tests are run sequentially and not in parallel.
2021-01-10 20:24:50 +01:00
XiangRongLin
f447a7a450
fix import due to downloader package move
2021-01-10 19:49:14 +01:00
XiangRongLin
285c26eafa
Adjust YoutubeMixPlaylistExtractorTest to use DownloaderFactory
2021-01-10 19:38:41 +01:00
XiangRongLin
e6e8e39def
Add DownloaderFactory to return a specific downloader based on 2 variables.
...
If the system property 'downloader' is set that use that specific downloader. This is used from gradle by appending `-Ddownloader=ABCD to the command.
ABCD is one of DownloaderType.
The other variable is the static property `DEFAULT_DOWNLOADER` in DownloaderFactory, which can be easily changed as needed inside the IDE according to development needs`.
Normal workflow would be to first use the recording downloader and afterwards only use mocks, if the requests are always staying the same.
2021-01-10 19:36:24 +01:00
XiangRongLin
7c40fb8bf7
Add additional downloader implementations
...
RecordingDownloader relies on the real downloader and saves the request/response pair into a json file.
MockDownloader uses json files from above and mocks responses for specific requests.
2021-01-10 19:36:24 +01:00
XiangRongLin
4dad3d60d6
Move DownloaderTestImpl into downloader package
2021-01-10 19:36:21 +01:00
Tobias Groza
ab62464e86
Merge pull request #495 from B0pol/sepiasearch
...
[PeerTube] Implement sepia search
2021-01-06 11:06:36 +01:00
TobiGr
2cbc3ccb1a
[media.ccc.de] Fix live stream test if there are no current live streams
2020-12-31 17:03:23 +01:00
TobiGr
80f4d42226
[Draft] Add live extractor
2020-12-27 11:01:22 +01:00
TobiGr
090dade5b3
[media.ccc.de] Add short test for recent kiosk
2020-12-27 01:03:08 +01:00
TobiGr
b2154c6e87
[media.ccc.de] Fix conferences kiosk test
2020-12-25 23:55:45 +01:00
bopol
3ae924a7f1
Implement sepia search
2020-12-23 15:07:55 +01:00
Fynn Godau
c9e9953bb0
[Bandcamp] Fix channel link handler factory
2020-12-22 20:02:53 +01:00
TobiGr
41a8ed625d
Make some vars final and add annotations to methods
2020-12-22 18:19:26 +01:00
TobiGr
bc6de14952
Extract stream and search meta info for YouTube
...
Add method to extract Google webcache URLs.
2020-12-22 18:19:26 +01:00
TobiGr
50903730b1
[Bandcamp] Fix accepting HTTP URLs
2020-12-22 08:53:10 +01:00
Tobias Groza
22a415156f
Merge pull request #479 from vkay94/stream-segments
...
Extract stream segments for YouTube
2020-12-19 11:12:22 +01:00
TobiGr
74b46fed2d
[Bandcamp] Fix deprecation in parseDate helper function
2020-12-19 10:40:21 +01:00
TobiGr
116e921d6c
Merge remote-tracking branch 'origin/dev' into bandcamp
2020-12-19 00:26:11 +01:00
TobiGr
df16a8646f
[Bandcamp] Improve radio stream extractor test
2020-12-18 23:15:59 +01:00
XiangRongLin
cdcb66b93e
Clean up stuff from mix pl
...
Coming from unclean merge and missing hints from IDE
2020-12-16 08:35:28 +01:00
Xiang Rong Lin
a338e4e08e
[Youtube] Apply review suggestions and avoid channel mix edge case
2020-12-12 20:32:43 +01:00
Xiang Rong Lin
22d2f7e400
[Youtube] Add cookies to youtube mix request
...
This way youtube wont return duplicates when getting more items of the mix (but youtube can also track us)
2020-12-12 20:32:40 +01:00
Xiang Rong Lin
3ff8619bcc
[Youtube] apply wb9688 suggestion (mix)
...
Channel mix adjusments and test
Don't accept youtube music mix urls as playlist
Don't override playlistData to keep getInitialPage()
Remove json constants
Indentation
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
822cf307f7
[Youtube] Add _ITEMS constants and improve code style
...
Move thumbnail id exctraction code to getThumbnailUrlFromId
Add test for "My mix" detection to service tests
Use ITEM_COUNT_UNKNOWN everywhere instead of -1 and add some tests
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
df38b1926c
[Youtube] Add tests and take thumbnail image always from first video of mix
...
Also fix getThumbnailUrl for "My Mix"
2020-12-12 20:30:57 +01:00
vkay94
9a7a224a54
Add tests for stream segments
2020-12-12 10:24:55 +01:00
Fynn Godau
04dd3d4d32
Rework link handlers to correctly accept external websites
2020-12-05 15:08:26 +01:00
Fynn Godau
be562b8436
Change tests
2020-11-27 16:48:40 +01:00
TobiGr
c91e21b37c
[Bandcamp] Tests - Add finals and improve code formatting
2020-11-24 14:13:31 +01:00
Tobias Groza
650f0920fe
Merge pull request #465 from XiangRongLin/playlist_continuation
...
[YouTube] Fix playlist continuations extraction
2020-11-20 20:48:20 +01:00
TobiGr
334e1e9b53
Fix YouTube comments test
2020-11-20 19:53:53 +01:00
Fynn Godau
186936d041
Various changes regarding tests
2020-11-19 22:33:52 +01:00
Fynn Godau
6bc7e3420e
Merge TNP/dev into fynngodau/dev
2020-11-19 21:32:08 +01:00
XiangRongLin
5bceff0083
[YouTube] Fix extraction of next page url for the last page of playlist
2020-11-18 19:03:12 +01:00
XiangRongLin
8347e14952
[YouTube] Fix playlist continuations extraction
2020-11-17 21:04:53 +01:00
Stypox
89a77ae74a
[YouTube] Fix detection of ended livestreams and parse livestream upload date
2020-11-04 16:03:08 +01:00
Stypox
827f7bd137
[YouTube] Cache deobfuscation and improve requests made
...
Fix age restriction extraction
Automatically fixes more things
2020-11-04 16:02:12 +01:00
Tobias Groza
b13c7e1c1e
Merge pull request #452 from Stypox/yt-import
...
Implement YouTube subscription import from Google takeout
2020-11-03 20:32:17 +01:00
bopol
345e136f6c
create YouTubeCommentsLinkHandlerFactoryTest
...
and remove invidious test from YouTubeCommentsExtractorTest, because it was just testing if the URL is accepted, then the extractor does the same thing, we don't need to test the same thing twice
2020-11-03 19:10:10 +01:00
Stypox
501ec30152
Implement youtube subscription import from Google takeout
2020-11-02 11:04:48 +01:00
Stypox
2f02c0e6a4
Merge pull request #450 from B0pol/fileutils
...
create FileUtils
2020-11-01 18:13:57 +01:00
bopol
f69b3ef05d
create FileUtils
2020-11-01 17:15:34 +01:00
Isira Seneviratne
4f04cfccca
Switch from Calendar to OffsetDateTime in DateWrapper.
2020-11-01 06:28:54 +05:30
Stypox
6cc50b57e3
Merge pull request #410 from Scrxtchy/sets-track-title
...
Fix parsing Soundcloud tracks that contain the term 'sets'
2020-10-31 21:43:21 +01:00
Stypox
b242e1d113
Merge pull request #362 from Stypox/google-search-urls
...
Add support for Google search redirect url
2020-10-31 21:03:48 +01:00
Stypox
3fe55b30ba
Add support for Google search redirect url
2020-10-31 20:58:28 +01:00
Stypox
c190a3029b
Consider protocol as base url when it is a custom one (e.g. vnd.youtube)
2020-10-31 20:58:28 +01:00
bopol
b21e59925d
[PeerTube] fix account and channel extractors
2020-10-29 19:52:29 +01:00
Scratch
947ce3ee10
Fix parsing Soundcloud tracks that contain the term 'sets'
2020-10-28 07:55:02 +11:00
bopol
0a12300c5e
polish tests
2020-10-26 16:57:37 +01:00
Tobias Groza
14c0c37d64
Merge pull request #309 from Stypox/improve-stream-tests
...
Improve stream extractor tests & various fixes
2020-10-25 18:49:53 +01:00
Kiril Isakov
366ed71523
[YouTube] Ignore leading characters in video id
2020-10-25 14:34:20 +01:00
Stypox
8026304a0a
Merge pull request #409 from Bartoshr/bugfix/short-channel-urls
...
Support short custom YouTube channel URLs
2020-10-25 11:43:50 +01:00
Stypox
57e7994c9e
Add some missing finals, nullables and comments
2020-10-25 08:12:41 +01:00
Stypox
f11fe87688
[YouTube] Replace outdated PewDiePie video test with another one
...
The old video was made private, and this video (wedding) is probably never going to be removed.
2020-10-24 18:41:41 +02:00
Stypox
3c55ea9321
[PeerTube] Change age restricted video in tests
...
The old one wasn't available anymore
2020-10-24 18:41:41 +02:00
Stypox
880ff04a5c
[Test] Add stream dash mpd url test
2020-10-24 18:41:41 +02:00
Stypox
d0b14644bb
[YouTube/MediaCCC] Consider dates as GMT and not as local
2020-10-24 18:41:41 +02:00
Stypox
8ce711f40f
[Test] Add sub channel name, url and thumbnail tests
2020-10-24 18:41:41 +02:00
Stypox
6127826571
[Test] Add stream metadata tests
2020-10-24 18:41:41 +02:00
Stypox
a087b092b4
[Test] Improve code style and add final
2020-10-24 18:39:55 +02:00
Stypox
8dc3f28618
[PeerTube] Test one channel url with api and one without
2020-10-24 18:39:55 +02:00
Stypox
af5b8b1915
[MediaCCC] Add tests for stream and conference link handlers
2020-10-24 18:36:21 +02:00
Stypox
0c980b2d64
[PeerTube] Improve channel and stream link handler tests
2020-10-24 18:36:21 +02:00
Stypox
492db83ccf
[MediaCCC] Return null instead of empty items collector
...
As per the documentation in the base getRelatedStreams()
2020-10-24 18:34:57 +02:00
Stypox
b461da792f
[MediaCCC] Fix link handler inconsistency providing API links
2020-10-24 18:34:34 +02:00
Stypox
aeeae87641
[PeerTube] Parse timestamp from url (previously unimplemented)
2020-10-24 18:33:42 +02:00
Stypox
7ae3cb6d07
[PeerTube] Fix link handler inconsistency providing API links
2020-10-24 18:33:11 +02:00
Stypox
1d7a86e664
[Test] Add base classes for stream extractor tests
...
Refactor all stream extractor tests to use new base class.
Remove check if upload date is in the past: this does not have to hold true: youtube premieres turn up in search results even though they are in the future
2020-10-24 18:22:44 +02:00
Bartosz Rumiński
d3f80d1538
Exlude links which are not channels
2020-10-20 20:06:06 +02:00
Stypox
6887d59570
[YouTube] Handle urls for Shorts
2020-10-18 12:03:01 +02:00
Scratch
e945f711c1
Fix SoundCloud test artist account name ( #416 )
2020-10-16 23:42:46 +02:00
Scratch
538f5d3973
Fix SoundCloud test artist account name
2020-10-16 13:45:05 +11:00
Bartosz Rumiński
5ab1b053d2
Update youtube channel link handler tests
...
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-10-12 20:11:28 +02:00
Bartosz Rumiński
8c38a5509e
Prevent attribution_link urls to be accepted by channel links handler
2020-10-09 19:07:38 +02:00