sjw/yt-dlp
1
0
Fork 0
mirror of https://github.com/yt-dlp/yt-dlp.git synced 2024-11-09 20:19:28 +01:00
Code Issues Packages Projects Releases Wiki Activity

Better Format Sorting (Squashed)

Browse Source 
* Added --format-sort (-S height,filesize)
* Made fields reversible (-S +height)
* Added --format-sort-force, --no-format-sort-force
* Added limit (-S height:720)
* Added codec preference (-S vcodec,acodec)
* Correct handling of preference<-1000

* Rebased to yt-dlc
* Automatically determine missing bitrates
* aext, vext, protocol, acodec, vcodec can now takes priority as string, not number (-S vext:webm)
* Correct handling of None in codec, audio_codec (None means the codec is unknown while 'none' means it doesn't exist)
* Correctly parse filesize (-S filesize:200M)
* Generalized preference calculation

* Rewrote entire code into the class FormatSort
* Correctly handle user input errors
* Combined fields (-S +ext:webm:webm)
* Closest mode (-S filesize~50M)
* Aliases (framerate=fps, br=bitrate etc)

* Documentation
This commit is contained in:
pukkandan 2020-10-26 21:20:09 +05:30
parent 98e248faa4
commit eb8a44336c
6 changed files with 448 additions and 103 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

171
README.md
View File

@ -32,7 +32,10 @@ youtube-dlc is a fork of youtube-dl with the intention of getting features teste
- [Output template and Windows batch files](#output-template-and-windows-batch-files)
- [Output template examples](#output-template-examples)
- [FORMAT SELECTION](#format-selection)
- [Format selection examples](#format-selection-examples)
- [Filtering Formats](#filtering-formats)
- [Sorting Formats](#sorting-formats)
- [Default Format Selection](#default-format-selection)
- [Format Selection examples](#format-selection-examples)
- [VIDEO SELECTION](#video-selection-1)
# INSTALLATION
@ -385,8 +388,16 @@ I will add some memorable short links to the binaries so you can download them e
## Video Format Options:
-f, --format FORMAT Video format code, see the "FORMAT
SELECTION" for all the info
-f, --format FORMAT Video format code, see "FORMAT SELECTION"
for more details
-S, --format-sort SORTORDER Sort the formats by the fields given, see
"Sorting Formats" for more details
--S-force, --format-sort-force Force user specified sort order to have
precedence over all fields, see "Sorting
Formats" for more details
--no-format-sort-force Some fields have precedence over the user
specified sort order, see "Sorting Formats"
for more details (default)
--all-formats Download all available video formats
--prefer-free-formats Prefer free video formats unless a specific
one is requested
@ -425,8 +436,8 @@ I will add some memorable short links to the binaries so you can download them e
## Adobe Pass Options:
--ap-mso MSO Adobe Pass multiple-system operator (TV
provider) identifier, use --ap-list-mso for
a list of available MSOs
provider) identifier, use --ap-list-mso
for a list of available MSOs
--ap-username USERNAME Multiple-system operator account login
--ap-password PASSWORD Multiple-system operator account password.
If this option is left out, youtube-dlc
@ -707,12 +718,17 @@ You can also use special names to select particular edge case formats:
- `bestaudio`: Select the best quality audio only-format. May not be available.
- `worstaudio`: Select the worst quality audio only-format. May not be available.
For example, to download the worst quality video-only format you can use `-f worstvideo`.
For example, to download the worst quality video-only format you can use `-f worstvideo`. It is however recomended to never actually use `worst` and related options. See [sorting formats](#sorting-formats) for more details.
If you want to download multiple videos and they don't have the same formats available, you can specify the order of preference using slashes. Note that slash is left-associative, i.e. formats on the left hand side are preferred, for example `-f 22/17/18` will download format 22 if it's available, otherwise it will download format 17 if it's available, otherwise it will download format 18 if it's available, otherwise it will complain that no suitable formats are available for download.
If you want to download multiple videos and they don't have the same formats available, you can specify the order of preference using slashes. Note that formats on the left hand side are preferred, for example `-f 22/17/18` will download format 22 if it's available, otherwise it will download format 17 if it's available, otherwise it will download format 18 if it's available, otherwise it will complain that no suitable formats are available for download.
If you want to download several formats of the same video use a comma as a separator, e.g. `-f 22,17,18` will download all these three formats, of course if they are available. Or a more sophisticated example combined with the precedence feature: `-f 136/137/mp4/bestvideo,140/m4a/bestaudio`.
You can merge the video and audio of multiple formats into a single file using `-f <format-1>+<format-2>` (requires ffmpeg or avconv installed), for example `-f bestvideo+bestaudio` will download the best video-only format, the best audio-only format and mux them together with ffmpeg/avconv.
## Filtering Formats
You can also filter the video formats by putting a condition in brackets, as in `-f "best[height=720]"` (or `-f "[filesize>10M]"`).
The following numeric meta fields can be used with comparisons `<`, `<=`, `>`, `>=`, `=` (equals), `!=` (not equals):
@ -741,35 +757,148 @@ Note that none of the aforementioned meta fields are guaranteed to be present si
Formats for which the value is not known are excluded unless you put a question mark (`?`) after the operator. You can combine format filters, so `-f "[height <=? 720][tbr>500]"` selects up to 720p videos (or videos where the height is not known) with a bitrate of at least 500 KBit/s.
You can merge the video and audio of two formats into a single file using `-f <video-format>+<audio-format>` (requires ffmpeg or avconv installed), for example `-f bestvideo+bestaudio` will download the best video-only format, the best audio-only format and mux them together with ffmpeg/avconv.
Format selectors can also be grouped using parentheses, for example if you want to download the best mp4 and webm formats with a height lower than 480 you can use `-f '(mp4,webm)[height<480]'`.
Since the end of April 2015 and version 2015.04.26, youtube-dlc uses `-f bestvideo+bestaudio/best` as the default format selection (see [#5447](https://github.com/ytdl-org/youtube-dl/issues/5447), [#5456](https://github.com/ytdl-org/youtube-dl/issues/5456)). If ffmpeg or avconv are installed this results in downloading `bestvideo` and `bestaudio` separately and muxing them together into a single file giving the best overall quality available. Otherwise it falls back to `best` and results in downloading the best available quality served as a single file. `best` is also needed for videos that don't come from YouTube because they don't provide the audio and video in two different files. If you want to only download some DASH formats (for example if you are not interested in getting videos with a resolution higher than 1080p), you can add `-f bestvideo[height<=?1080]+bestaudio/best` to your configuration file. Note that if you use youtube-dlc to stream to `stdout` (and most likely to pipe it to your media player then), i.e. you explicitly specify output template as `-o -`, youtube-dlc still uses `-f best` format selection in order to start content delivery immediately to your player and not to wait until `bestvideo` and `bestaudio` are downloaded and muxed.
## Sorting Formats
You can change the criteria for being considered the `best` by using `-S` (`--format-sort`). The general format for this is `--format-sort field1,field2...`. The available fields are:
- `video`, `has_video`: Gives priority to formats that has a video stream
- `audio`, `has_audio`: Gives priority to formats that has a audio stream
- `extractor`, `preference`, `extractor_preference`: The format preference as given by the extractor
- `lang`, `language_preference`: Language preference as given by the extractor
- `quality`: The quality of the format. This is a metadata field available in some websites
- `source`, `source_preference`: Preference of the source as given by the extractor
- `proto`, `protocol`: Protocol used for download (`https`/`ftps` > `http`/`ftp` > `m3u8-native` > `m3u8` > `http-dash-segments` > other > `mms`/`rtsp` > unknown > `f4f`/`f4m`)
- `vcodec`, `video_codec`: Video Codec (`av01` > `vp9` > `h265` > `h264` > `vp8` > `h263` > `theora` > other > unknown)
- `acodec`, `audio_codec`: Audio Codec (`opus` > `vorbis` > `aac` > `mp4a` > `mp3` > `ac3` > `dts` > other > unknown)
- `codec`: Equivalent to `vcodec,acodec`
- `vext`, `video_ext`: Video Extension (`mp4` > `flv` > `webm` > other > unknown). If `--prefer-free-formats` is used, `webm` is prefered.
- `aext`, `audio_ext`: Audio Extension (`m4a` > `aac` > `mp3` > `ogg` > `opus` > `webm` > other > unknown). If `--prefer-free-formats` is used, the order changes to `opus` > `ogg` > `webm` > `m4a` > `mp3` > `aac`.
- `ext`, `extension`: Equivalent to `vext,aext`
- `filesize`: Exact filesize, if know in advance. This will be unavailable for mu38 and DASH formats.
- `filesize_approx`: Approximate filesize calculated the manifests
- `size`, `filesize_estimate`: Exact filesize if available, otherwise approximate filesize
- `height`: Height of video
- `width`: Width of video
- `res`, `dimension`: Video resolution, calculated as the smallest dimension.
- `fps`, `framerate`: Framerate of video
- `tbr`, `total_bitrate`: Total average bitrate in KBit/s
- `vbr`, `video_bitrate`: Average video bitrate in KBit/s
- `abr`, `audio_bitrate`: Average audio bitrate in KBit/s
- `br`, `bitrate`: Equivalent to using `tbr,vbr,abr`
- `samplerate`, `asr`: Audio sample rate in Hz
All fields, unless specified otherwise, are sorted in decending order. To reverse this, prefix the field with a `+`. Eg: `+res` prefers the smallest resolution format. Additionally, you can suffix a prefered value for the fields, seperated by a `:`. Eg: `res:720` prefers larger videos, but no larger than 720p and the smallest video if there are no videos less than 720p. For `codec` and `ext`, you can provide two prefered values, the first for video and the second for audio. Eg: `+codec:avc:m4a` (equivalent to `+vcodec:avc,+acodec:m4a`) sets the video codec preference to `h264` > `h265` > `vp9` > `av01` > `vp8` > `h263` > `theora` and audio codec preference to `mp4a` > `aac` > `vorbis` > `opus` > `mp3` > `ac3` > `dts`. You can also make the sorting prefer the nearest values to the provided by using `~` as the delimiter. Eg: `filesize~1G` prefers the format with filesize closest to 1 GiB.
The fields `has_video`, `has_audio`, `extractor_preference`, `language_preference`, `quality` are always given highest priority in sorting, irrespective of the user-defined order. This behaviour can be changed by using `--force-format-sort`. Apart from these, the default order used by youtube-dlc is: `tbr,filesize,vbr,height,width,protocol,vext,abr,aext,fps,filesize_approx,source_preference,format_id`. Note that the extractors may override this default order (currently no extractor does this), but not the user-provided order.
If your format selector is `worst`, the last item is selected after sorting. This means it will select the format that is worst in all repects. Most of the time, what you actually want is the video with the smallest filesize instead. So it is generally better to use `-f best -S +size,+br,+res,+fps`.
**Tip**: You can use the `-v -F` to see how the formats have been sorted (worst to best).
## Default Format Selection
Since the end of April 2015 and version 2015.04.26, youtube-dlc uses `-f bestvideo+bestaudio/best` as the default format selection (see [#5447](https://github.com/ytdl-org/youtube-dl/issues/5447), [#5456](https://github.com/ytdl-org/youtube-dl/issues/5456)). If ffmpeg or avconv are installed this results in downloading `bestvideo` and `bestaudio` separately and muxing them together into a single file giving the best overall quality available. Otherwise it falls back to `best` and results in downloading the best available quality served as a single file. `best` is also needed for videos that don't come from YouTube because they don't provide the audio and video in two different files. Note that if you use youtube-dlc to stream to `stdout` (and most likely to pipe it to your media player then), i.e. you explicitly specify output template as `-o -`, youtube-dlc still uses `-f best` format selection in order to start content delivery immediately to your player and not to wait until `bestvideo` and `bestaudio` are downloaded and muxed.
If you want to preserve the old format selection behavior (prior to youtube-dlc 2015.04.26), i.e. you want to download the best available quality media served as a single file, you should explicitly specify your choice with `-f best`. You may want to add it to the [configuration file](#configuration) in order not to type it every time you run youtube-dlc.
#### Format selection examples
## Format Selection examples
Note that on Windows you may need to use double quotes instead of single.
```bash
# Download best mp4 format available or any other best if no mp4 available
$ youtube-dlc -f 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best'
# Download the worst video available
$ youtube-dlc -f 'worstvideo+worstaudio/worst'
# Download best format available but no better than 480p
$ youtube-dlc -f 'bestvideo[height<=480]+bestaudio/best[height<=480]'
# Download the best video available but with the smallest resolution
$ youtube-dlc -S '+res'
# Download best video only format but no bigger than 50 MB
$ youtube-dlc -f 'best[filesize<50M]'
# Download the smallest video available
$ youtube-dlc -S '+size,+bitrate'
# Download best format available via direct link over HTTP/HTTPS protocol
$ youtube-dlc -f '(bestvideo+bestaudio/best)[protocol^=http]'
# Download the best video format and the best audio format without merging them
# Download the best mp4 video available, or the best video if no mp4 available
$ youtube-dlc -f 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/bestvideo+bestaudio / best'
# Download the best video with the best extension
# (For video, mp4 > webm > flv. For audio, m4a > aac > mp3 ...)
$ youtube-dlc -S 'ext'
# Download the best video available but no better than 480p,
# or the worst video if there is no video under 480p
$ youtube-dlc -f 'bestvideo[height<=480]+bestaudio/best[height<=480] / worstvideo+bestaudio/worst'
# Download the best video available with the largest height but no better than 480p,
# or the best video with the smallest resolution if there is no video under 480p
$ youtube-dlc -S 'height:480'
# Download the best video available with the largest resolution but no better than 480p,
# or the best video with the smallest resolution if there is no video under 480p
# Resolution is determined by using the smallest dimension.
# So this works correctly for vertical videos as well
$ youtube-dlc -S 'res:480'
# Download the best video (that also has audio) but no bigger than 50 MB,
# or the worst video (that also has audio) if there is no video under 50 MB
$ youtube-dlc -f 'best[filesize<50M] / worst'
# Download largest video (that also has audio) but no bigger than 50 MB,
# or the smallest video (that also has audio) if there is no video under 50 MB
$ youtube-dlc -f 'best' -S 'filesize:50M'
# Download best video (that also has audio) that is closest in size to 50 MB
$ youtube-dlc -f 'best' -S 'filesize~50M'
# Download best video available via direct link over HTTP/HTTPS protocol,
# or the best video available via any protocol if there is no such video
$ youtube-dlc -f '(bestvideo+bestaudio/best)[protocol^=http][protocol!*=dash] / bestvideo+bestaudio/best'
# Download best video available via the best protocol
# (https/ftps > http/ftp > m3u8_native > m3u8 > http_dash_segments ...)
$ youtube-dlc -S 'protocol'
# Download the best video-only format and the best audio-only format without merging them
# For this case, an output template should be used since
# by default, bestvideo and bestaudio will have the same file name.
$ youtube-dlc -f 'bestvideo,bestaudio' -o '%(title)s.f%(format_id)s.%(ext)s'
# Download the best video with h264 codec, or the best video if there is no such video
$ youtube-dlc -f '(bestvideo+bestaudio/best)[vcodec^=avc1] / bestvideo+bestaudio/best'
# Download the best video with best codec no better than h264,
# or the best video with worst codec if there is no such video
$ youtube-dlc -S 'codec:h264'
# Download the best video with worst codec no worse than h264,
# or the best video with best codec if there is no such video
$ youtube-dlc -S '+codec:h264'
# More complex examples
# Download the best video no better than 720p prefering framerate greater than 30,
# or the worst video (prefering framerate greater than 30) if there is no such video
$ youtube-dlc -f '((bestvideo[fps>30]/bestvideo)[height<=720]/(worstvideo[fps>30]/worstvideo)) + bestaudio / (best[fps>30]/best)[height<=720]/(worst[fps>30]/worst)'
# Download the video with the largest resolution no better than 720p,
# or the video with the smallest resolution available if there is no such video,
# prefering larger framerate for formats with the same resolution
$ youtube-dlc -S 'res:720,fps'
# Download the video with smallest resolution no worse than 480p,
# or the video with the largest resolution available if there is no such video,
# prefering better codec and then larger total bitrate for the same resolution
$ youtube-dlc -S '+res:480,codec,br'
```
Note that in the last example, an output template is recommended as bestvideo and bestaudio may have the same file name.
# VIDEO SELECTION

8
youtube_dlc/YoutubeDL.py
View File

@ -162,7 +162,9 @@ class YoutubeDL(object):
dump_single_json: Force printing the info_dict of the whole playlist
(or video) as a single JSON line.
simulate: Do not download the video files.
format: Video format code. See options.py for more information.
format: Video format code. see "FORMAT SELECTION" for more details.
format_sort: How to sort the video formats. see "Sorting Formats" for more details.
format_sort_force: Force the given format_sort. see "Sorting Formats" for more details.
outtmpl: Template for output names.
restrictfilenames: Do not allow "&" and spaces in file names.
trim_file_name: Limit length of filename (extension excluded).
@ -2305,8 +2307,8 @@ class YoutubeDL(object):
[f['format_id'], f['ext'], self.format_resolution(f), self._format_note(f)]
for f in formats
if f.get('preference') is None or f['preference'] >= -1000]
if len(formats) > 1:
table[-1][-1] += (' ' if table[-1][-1] else '') + '(best)'
# if len(formats) > 1:
# table[-1][-1] += (' ' if table[-1][-1] else '') + '(best)'
header_line = ['format code', 'extension', 'resolution', 'note']
self.to_screen(

7
youtube_dlc/__init__.py
View File

@ -8,6 +8,7 @@ __license__ = 'Public Domain'
import codecs
import io
import os
import re
import random
import sys
@ -41,6 +42,7 @@ from .downloader import (
FileDownloader,
)
from .extractor import gen_extractors, list_extractors
from .extractor.common import InfoExtractor
from .extractor.adobepass import MSO_INFO
from .YoutubeDL import YoutubeDL
@ -245,6 +247,9 @@ def _real_main(argv=None):
parser.error('Cannot download a video and extract audio into the same'
' file! Use "{0}.%(ext)s" instead of "{0}" as the output'
' template'.format(outtmpl))
for f in opts.format_sort:
if re.match(InfoExtractor.FormatSort.regex, f) is None:
parser.error('invalid format sort string "%s" specified' % f)
any_getting = opts.geturl or opts.gettitle or opts.getid or opts.getthumbnail or opts.getdescription or opts.getfilename or opts.getformat or opts.getduration or opts.dumpjson or opts.dump_single_json
any_printing = opts.print_json
@ -347,6 +352,8 @@ def _real_main(argv=None):
'simulate': opts.simulate or any_getting,
'skip_download': opts.skip_download,
'format': opts.format,
'format_sort': opts.format_sort,
'format_sort_force': opts.format_sort_force,
'listformats': opts.listformats,
'outtmpl': outtmpl,
'autonumber_size': opts.autonumber_size,

336
youtube_dlc/extractor/common.py
View File

@ -32,6 +32,7 @@ from ..compat import (
compat_urlparse,
compat_xml_parse_error,
)
from ..downloader import FileDownloader
from ..downloader.f4m import (
get_base_url,
remove_encrypted_media,
@ -1354,81 +1355,270 @@ class InfoExtractor(object):
html, '%s form' % form_id, group='form')
return self._hidden_inputs(form)
def _sort_formats(self, formats, field_preference=None):
class FormatSort:
regex = r' *((?P<reverse>\+)?(?P<field>[a-zA-Z0-9_]+)((?P<seperator>[~:])(?P<limit>.*?))?)? *$'
default = ('hidden', 'has_video', 'has_audio', 'extractor', 'lang', 'quality',
'tbr', 'filesize', 'vbr', 'height', 'width', 'protocol', 'vext',
'abr', 'aext', 'fps', 'filesize_approx', 'source_preference', 'format_id')
settings = {
'vcodec': {'type': 'ordered', 'regex': True,
'order': ['av01', 'vp9', '(h265|he?vc?)', '(h264|avc)', 'vp8', '(mp4v|h263)', 'theora', '', None, 'none']},
'acodec': {'type': 'ordered', 'regex': True,
'order': ['opus', 'vorbis', 'aac', 'mp?4a?', 'mp3', 'e?a?c-?3', 'dts', '', None, 'none']},
'protocol': {'type': 'ordered', 'regex': True,
'order': ['(ht|f)tps', '(ht|f)tp$', 'm3u8.+', 'm3u8', '.*dash', '', 'mms|rtsp', 'none', 'f4']},
'vext': {'type': 'ordered', 'field': 'video_ext',
'order': ('mp4', 'flv', 'webm', '', 'none'), # Why is flv prefered over webm???
'order_free': ('webm', 'mp4', 'flv', '', 'none')},
'aext': {'type': 'ordered', 'field': 'audio_ext',
'order': ('m4a', 'aac', 'mp3', 'ogg', 'opus', 'webm', '', 'none'),
'order_free': ('opus', 'ogg', 'webm', 'm4a', 'mp3', 'aac', '', 'none')},
'hidden': {'visible': False, 'forced': True, 'type': 'extractor', 'max': -1000},
'extractor_preference': {'priority': True, 'type': 'extractor'},
'has_video': {'priority': True, 'field': 'vcodec', 'type': 'boolean', 'not_in_list': ('none',)},
'has_audio': {'priority': True, 'field': 'acodec', 'type': 'boolean', 'not_in_list': ('none',)},
'language_preference': {'priority': True, 'convert': 'ignore'},
'quality': {'priority': True, 'convert': 'float_none'},
'filesize': {'convert': 'bytes'},
'filesize_approx': {'convert': 'bytes'},
'format_id': {'convert': 'string'},
'height': {'convert': 'float_none'},
'width': {'convert': 'float_none'},
'fps': {'convert': 'float_none'},
'tbr': {'convert': 'float_none'},
'vbr': {'convert': 'float_none'},
'abr': {'convert': 'float_none'},
'asr': {'convert': 'float_none'},
'source_preference': {'convert': 'ignore'},
'codec': {'type': 'combined', 'field': ('vcodec', 'acodec')},
'bitrate': {'type': 'combined', 'field': ('tbr', 'vbr', 'abr'), 'same_limit': True},
'filesize_estimate': {'type': 'combined', 'same_limit': True, 'field': ('filesize', 'filesize_approx')},
'extension': {'type': 'combined', 'field': ('vext', 'aext')},
'dimension': {'type': 'multiple', 'field': ('height', 'width'), 'function': min}, # not named as 'resolution' because such a field exists
'res': {'type': 'alias', 'field': 'dimension'},
'ext': {'type': 'alias', 'field': 'extension'},
'br': {'type': 'alias', 'field': 'bitrate'},
'total_bitrate': {'type': 'alias', 'field': 'tbr'},
'video_bitrate': {'type': 'alias', 'field': 'vbr'},
'audio_bitrate': {'type': 'alias', 'field': 'abr'},
'framerate': {'type': 'alias', 'field': 'fps'},
'lang': {'type': 'alias', 'field': 'language_preference'}, # not named as 'language' because such a field exists
'proto': {'type': 'alias', 'field': 'protocol'},
'source': {'type': 'alias', 'field': 'source_preference'},
'size': {'type': 'alias', 'field': 'filesize_estimate'},
'samplerate': {'type': 'alias', 'field': 'asr'},
'video_ext': {'type': 'alias', 'field': 'vext'},
'audio_ext': {'type': 'alias', 'field': 'aext'},
'video_codec': {'type': 'alias', 'field': 'vcodec'},
'audio_codec': {'type': 'alias', 'field': 'acodec'},
'video': {'type': 'alias', 'field': 'has_video'},
'audio': {'type': 'alias', 'field': 'has_audio'},
'extractor': {'type': 'alias', 'field': 'extractor_preference'},
'preference': {'type': 'alias', 'field': 'extractor_preference'}}
_order = []
def _get_field_setting(self, field, key):
if field not in self.settings:
self.settings[field] = {}
propObj = self.settings[field]
if key not in propObj:
type = propObj.get('type')
if key == 'field':
default = 'preference' if type == 'extractor' else (field,) if type in ('combined', 'multiple') else field
elif key == 'convert':
default = 'order' if type == 'ordered' else 'float_string' if field else 'ignore'
else:
default = {'type': 'field', 'visible': True, 'order': [], 'not_in_list': (None,), 'function': max}.get(key, None)
propObj[key] = default
return propObj[key]
def _resolve_field_value(self, field, value, convertNone=False):
if value is None:
if not convertNone:
return None
else:
value = value.lower()
conversion = self._get_field_setting(field, 'convert')
if conversion == 'ignore':
return None
if conversion == 'string':
return value
elif conversion == 'float_none':
return float_or_none(value)
elif conversion == 'bytes':
return FileDownloader.parse_bytes(value)
elif conversion == 'order':
order_free = self._get_field_setting(field, 'order_free')
order_list = order_free if order_free and self._use_free_order else self._get_field_setting(field, 'order')
use_regex = self._get_field_setting(field, 'regex')
list_length = len(order_list)
empty_pos = order_list.index('') if '' in order_list else list_length + 1
if use_regex and value is not None:
for (i, regex) in enumerate(order_list):
if regex and re.match(regex, value):
return list_length - i
return list_length - empty_pos # not in list
else: # not regex or value = None
return list_length - (order_list.index(value) if value in order_list else empty_pos)
else:
if value.isnumeric():
return float(value)
else:
self.settings[field]['convert'] = 'string'
return value
def evaluate_params(self, params, sort_extractor):
self._use_free_order = params.get('prefer_free_formats', False)
self._sort_user = params.get('format_sort', [])
self._sort_extractor = sort_extractor
def add_item(field, reverse, closest, limit_text):
field = field.lower()
if field in self._order:
return
self._order.append(field)
limit = self._resolve_field_value(field, limit_text)
data = {
'reverse': reverse,
'closest': False if limit is None else closest,
'limit_text': limit_text,
'limit': limit}
if field in self.settings:
self.settings[field].update(data)
else:
self.settings[field] = data
sort_list = (
tuple(field for field in self.default if self._get_field_setting(field, 'forced'))
+ (tuple() if params.get('format_sort_force', False)
else tuple(field for field in self.default if self._get_field_setting(field, 'priority')))
+ tuple(self._sort_user) + tuple(sort_extractor) + self.default)
for item in sort_list:
match = re.match(self.regex, item)
if match is None:
raise ExtractorError('Invalid format sort string "%s" given by extractor' % item)
field = match.group('field')
if field is None:
continue
if self._get_field_setting(field, 'type') == 'alias':
field = self._get_field_setting(field, 'field')
reverse = match.group('reverse') is not None
closest = match.group('seperator') == '~'
limit_text = match.group('limit')
has_limit = limit_text is not None
has_multiple_fields = self._get_field_setting(field, 'type') == 'combined'
has_multiple_limits = has_limit and has_multiple_fields and not self._get_field_setting(field, 'same_limit')
fields = self._get_field_setting(field, 'field') if has_multiple_fields else (field,)
limits = limit_text.split(":") if has_multiple_limits else (limit_text,) if has_limit else tuple()
limit_count = len(limits)
for (i, f) in enumerate(fields):
add_item(f, reverse, closest,
limits[i] if i < limit_count
else limits[0] if has_limit and not has_multiple_limits
else None)
def print_verbose_info(self, to_screen):
to_screen('[debug] Sort order given by user: %s' % ','.join(self._sort_user))
if self._sort_extractor:
to_screen('[debug] Sort order given by extractor: %s' % ','.join(self._sort_extractor))
to_screen('[debug] Formats sorted by: %s' % ', '.join(['%s%s%s' % (
'+' if self._get_field_setting(field, 'reverse') else '', field,
'%s%s(%s)' % ('~' if self._get_field_setting(field, 'closest') else ':',
self._get_field_setting(field, 'limit_text'),
self._get_field_setting(field, 'limit'))
if self._get_field_setting(field, 'limit_text') is not None else '')
for field in self._order if self._get_field_setting(field, 'visible')]))
def _calculate_field_preference_from_value(self, format, field, type, value):
reverse = self._get_field_setting(field, 'reverse')
closest = self._get_field_setting(field, 'closest')
limit = self._get_field_setting(field, 'limit')
if type == 'extractor':
maximum = self._get_field_setting(field, 'max')
if value is None or (maximum is not None and value >= maximum):
value = 0
elif type == 'boolean':
in_list = self._get_field_setting(field, 'in_list')
not_in_list = self._get_field_setting(field, 'not_in_list')
value = 0 if ((in_list is None or value in in_list) and (not_in_list is None or value not in not_in_list)) else -1
elif type == 'ordered':
value = self._resolve_field_value(field, value, True)
# try to convert to number
val_num = float_or_none(value)
is_num = self._get_field_setting(field, 'convert') != 'string' and val_num is not None
if is_num:
value = val_num
return ((-10, 0) if value is None
else (1, value, 0) if not is_num # if a field has mixed strings and numbers, strings are sorted higher
else (0, -abs(value - limit), value - limit if reverse else limit - value) if closest
else (0, value, 0) if not reverse and (limit is None or value <= limit)
else (0, -value, 0) if limit is None or (reverse and value == limit) or value > limit
else (-1, value, 0))
def _calculate_field_preference(self, format, field):
type = self._get_field_setting(field, 'type') # extractor, boolean, ordered, field, multiple
get_value = lambda f: format.get(self._get_field_setting(f, 'field'))
if type == 'multiple':
type = 'field' # Only 'field' is allowed in multiple for now
actual_fields = self._get_field_setting(field, 'field')
def wrapped_function(values):
values = tuple(filter(lambda x: x is not None, values))
return (self._get_field_setting(field, 'function')(*values) if len(values) > 1
else values[0] if values
else None)
value = wrapped_function((get_value(f) for f in actual_fields))
else:
value = get_value(field)
return self._calculate_field_preference_from_value(format, field, type, value)
def calculate_preference(self, format):
# Determine missing protocol
if not format.get('protocol'):
format['protocol'] = determine_protocol(format)
# Determine missing ext
if not format.get('ext') and 'url' in format:
format['ext'] = determine_ext(format['url'])
if format.get('vcodec') == 'none':
format['audio_ext'] = format['ext']
format['video_ext'] = 'none'
else:
format['video_ext'] = format['ext']
format['audio_ext'] = 'none'
# if format.get('preference') is None and format.get('ext') in ('f4f', 'f4m'): # Not supported?
# format['preference'] = -1000
# Determine missing bitrates
if format.get('tbr') is None:
if format.get('vbr') is not None and format.get('abr') is not None:
format['tbr'] = format.get('vbr', 0) + format.get('abr', 0)
else:
if format.get('vcodec') != "none" and format.get('vbr') is None:
format['vbr'] = format.get('tbr') - format.get('abr', 0)
if format.get('acodec') != "none" and format.get('abr') is None:
format['abr'] = format.get('tbr') - format.get('vbr', 0)
return tuple(self._calculate_field_preference(format, field) for field in self._order)
def _sort_formats(self, formats, field_preference=[]):
if not formats:
raise ExtractorError('No video formats found')
for f in formats:
# Automatically determine tbr when missing based on abr and vbr (improves
# formats sorting in some cases)
if 'tbr' not in f and f.get('abr') is not None and f.get('vbr') is not None:
f['tbr'] = f['abr'] + f['vbr']
def _formats_key(f):
# TODO remove the following workaround
from ..utils import determine_ext
if not f.get('ext') and 'url' in f:
f['ext'] = determine_ext(f['url'])
if isinstance(field_preference, (list, tuple)):
return tuple(
f.get(field)
if f.get(field) is not None
else ('' if field == 'format_id' else -1)
for field in field_preference)
preference = f.get('preference')
if preference is None:
preference = 0
if f.get('ext') in ['f4f', 'f4m']: # Not yet supported
preference -= 0.5
protocol = f.get('protocol') or determine_protocol(f)
proto_preference = 0 if protocol in ['http', 'https'] else (-0.5 if protocol == 'rtsp' else -0.1)
if f.get('vcodec') == 'none': # audio only
preference -= 50
if self._downloader.params.get('prefer_free_formats'):
ORDER = ['aac', 'mp3', 'm4a', 'webm', 'ogg', 'opus']
else:
ORDER = ['webm', 'opus', 'ogg', 'mp3', 'aac', 'm4a']
ext_preference = 0
try:
audio_ext_preference = ORDER.index(f['ext'])
except ValueError:
audio_ext_preference = -1
else:
if f.get('acodec') == 'none': # video only
preference -= 40
if self._downloader.params.get('prefer_free_formats'):
ORDER = ['flv', 'mp4', 'webm']
else:
ORDER = ['webm', 'flv', 'mp4']
try:
ext_preference = ORDER.index(f['ext'])
except ValueError:
ext_preference = -1
audio_ext_preference = 0
return (
preference,
f.get('language_preference') if f.get('language_preference') is not None else -1,
f.get('quality') if f.get('quality') is not None else -1,
f.get('tbr') if f.get('tbr') is not None else -1,
f.get('filesize') if f.get('filesize') is not None else -1,
f.get('vbr') if f.get('vbr') is not None else -1,
f.get('height') if f.get('height') is not None else -1,
f.get('width') if f.get('width') is not None else -1,
proto_preference,
ext_preference,
f.get('abr') if f.get('abr') is not None else -1,
audio_ext_preference,
f.get('fps') if f.get('fps') is not None else -1,
f.get('filesize_approx') if f.get('filesize_approx') is not None else -1,
f.get('source_preference') if f.get('source_preference') is not None else -1,
f.get('format_id') if f.get('format_id') is not None else '',
)
formats.sort(key=_formats_key)
format_sort = self.FormatSort() # params and to_screen are taken from the downloader
format_sort.evaluate_params(self._downloader.params, field_preference)
if self._downloader.params.get('verbose', False):
format_sort.print_verbose_info(self._downloader.to_screen)
formats.sort(key=lambda f: format_sort.calculate_preference(f))
def _check_formats(self, formats, video_id):
if formats:

10
youtube_dlc/extractor/vimeo.py
View File

@ -181,11 +181,11 @@ class VimeoBaseInfoExtractor(InfoExtractor):
'preference': 1,
})
for f in formats:
if f.get('vcodec') == 'none':
f['preference'] = -50
elif f.get('acodec') == 'none':
f['preference'] = -40
# for f in formats:
# if f.get('vcodec') == 'none':
# f['preference'] = -50
# elif f.get('acodec') == 'none':
# f['preference'] = -40
subtitles = {}
text_tracks = config['request'].get('text_tracks')

19
youtube_dlc/options.py
View File

@ -397,7 +397,24 @@ def parseOpts(overrideArguments=None):
video_format.add_option(
'-f', '--format',
action='store', dest='format', metavar='FORMAT', default=None,
help='Video format code, see the "FORMAT SELECTION" for all the info')
help='Video format code, see "FORMAT SELECTION" for more details')
video_format.add_option(
'-S', '--format-sort',
dest='format_sort', default=[],
action='callback', callback=_comma_separated_values_options_callback, type='str',
help='Sort the formats by the fields given, see "Sorting Formats" for more details')
video_format.add_option(
'--format-sort-force', '--S-force',
action='store_true', dest='format_sort_force', metavar='FORMAT', default=False,
help=(
'Force user specified sort order to have precedence over all fields, '
'see "Sorting Formats" for more details'))
video_format.add_option(
'--no-format-sort-force',
action='store_false', dest='format_sort_force', metavar='FORMAT', default=False,
help=(
'Some fields have precedence over the user specified sort order (default), '
'see "Sorting Formats" for more details'))
video_format.add_option(
'--all-formats',
action='store_const', dest='format', const='all',