rust/src/librustc/metadata/cstore.rs

251 lines
8.4 KiB
Rust
Raw Normal View History

// Copyright 2012-2014 The Rust Project Developers. See the COPYRIGHT
// file at the top-level directory of this distribution and at
// http://rust-lang.org/COPYRIGHT.
//
// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
// option. This file may not be copied, modified, or distributed
// except according to those terms.
#![allow(non_camel_case_types)]
2011-07-08 03:39:44 +02:00
// The crate store - a central repo for information collected about external
// crates and libraries
pub use self::MetadataBlob::*;
pub use self::LinkagePreference::*;
pub use self::NativeLibraryKind::*;
use back::svh::Svh;
use metadata::decoder;
rustc: Optimize reading metadata by 4x We were previously reading metadata via `ar p`, but as learned from rustdoc awhile back, spawning a process to do something is pretty slow. Turns out LLVM has an Archive class to read archives, but it cannot write archives. This commits adds bindings to the read-only version of the LLVM archive class (with a new type that only has a read() method), and then it uses this class when reading the metadata out of rlibs. When you put this in tandem of not compressing the metadata, reading the metadata is 4x faster than it used to be The timings I got for reading metadata from the respective libraries was: libstd-04ff901e-0.9-pre.dylib => 100ms libstd-04ff901e-0.9-pre.rlib => 23ms librustuv-7945354c-0.9-pre.dylib => 4ms librustuv-7945354c-0.9-pre.rlib => 1ms librustc-5b94a16f-0.9-pre.dylib => 87ms librustc-5b94a16f-0.9-pre.rlib => 35ms libextra-a6ebb16f-0.9-pre.dylib => 63ms libextra-a6ebb16f-0.9-pre.rlib => 15ms libsyntax-2e4c0458-0.9-pre.dylib => 86ms libsyntax-2e4c0458-0.9-pre.rlib => 22ms In order to always take advantage of these faster metadata read-times, I sort the files in filesearch based on whether they have an rlib extension or not (prefer all rlib files first). Overall, this halved the compile time for a `fn main() {}` crate from 0.185s to 0.095s on my system (when preferring dynamic linking). Reading metadata is still the slowest pass of the compiler at 0.035s, but it's getting pretty close to linking at 0.021s! The next best optimization is to just not copy the metadata from LLVM because that's the most expensive part of reading metadata right now.
2013-12-17 05:58:21 +01:00
use metadata::loader;
rustc: Fix a leak in dependency= paths With the addition of separate search paths to the compiler, it was intended that applications such as Cargo could require a `--extern` flag per `extern crate` directive in the source. The system can currently be subverted, however, due to the `existing_match()` logic in the crate loader. When loading crates we first attempt to match an `extern crate` directive against all previously loaded crates to avoid reading metadata twice. This "hit the cache if possible" step was erroneously leaking crates across the search path boundaries, however. For example: extern crate b; extern crate a; If `b` depends on `a`, then it will load crate `a` when the `extern crate b` directive is being processed. When the compiler reaches `extern crate a` it will use the previously loaded version no matter what. If the compiler was not invoked with `-L crate=path/to/a`, it will still succeed. This behavior is allowing `extern crate` declarations in Cargo without a corresponding declaration in the manifest of a dependency, which is considered a bug. This commit fixes this problem by keeping track of the origin search path for a crate. Crates loaded from the dependency search path are not candidates for crates which are loaded from the crate search path. As a result of this fix, this is a likely a breaking change for a number of Cargo packages. If the compiler starts informing that a crate can no longer be found, it likely means that the dependency was forgotten in your Cargo.toml. [breaking-change]
2015-01-06 17:46:07 +01:00
use session::search_paths::PathKind;
use util::nodemap::{FnvHashMap, NodeMap};
2013-12-21 04:43:27 +01:00
use std::cell::RefCell;
2014-03-27 18:28:38 +01:00
use std::rc::Rc;
use flate::Bytes;
2013-03-26 21:38:07 +01:00
use syntax::ast;
use syntax::codemap::Span;
use syntax::parse::token::IdentInterner;
// A map from external crate numbers (as decoded from some crate file) to
// local crate numbers (as generated during this session). Each external
// crate may refer to types in other external crates, and each has their
// own crate numbers.
pub type cnum_map = FnvHashMap<ast::CrateNum, ast::CrateNum>;
pub enum MetadataBlob {
MetadataVec(Bytes),
rustc: Optimize reading metadata by 4x We were previously reading metadata via `ar p`, but as learned from rustdoc awhile back, spawning a process to do something is pretty slow. Turns out LLVM has an Archive class to read archives, but it cannot write archives. This commits adds bindings to the read-only version of the LLVM archive class (with a new type that only has a read() method), and then it uses this class when reading the metadata out of rlibs. When you put this in tandem of not compressing the metadata, reading the metadata is 4x faster than it used to be The timings I got for reading metadata from the respective libraries was: libstd-04ff901e-0.9-pre.dylib => 100ms libstd-04ff901e-0.9-pre.rlib => 23ms librustuv-7945354c-0.9-pre.dylib => 4ms librustuv-7945354c-0.9-pre.rlib => 1ms librustc-5b94a16f-0.9-pre.dylib => 87ms librustc-5b94a16f-0.9-pre.rlib => 35ms libextra-a6ebb16f-0.9-pre.dylib => 63ms libextra-a6ebb16f-0.9-pre.rlib => 15ms libsyntax-2e4c0458-0.9-pre.dylib => 86ms libsyntax-2e4c0458-0.9-pre.rlib => 22ms In order to always take advantage of these faster metadata read-times, I sort the files in filesearch based on whether they have an rlib extension or not (prefer all rlib files first). Overall, this halved the compile time for a `fn main() {}` crate from 0.185s to 0.095s on my system (when preferring dynamic linking). Reading metadata is still the slowest pass of the compiler at 0.035s, but it's getting pretty close to linking at 0.021s! The next best optimization is to just not copy the metadata from LLVM because that's the most expensive part of reading metadata right now.
2013-12-17 05:58:21 +01:00
MetadataArchive(loader::ArchiveMetadata),
}
pub struct crate_metadata {
pub name: String,
pub data: MetadataBlob,
pub cnum_map: cnum_map,
pub cnum: ast::CrateNum,
pub span: Span,
}
2015-01-28 14:34:18 +01:00
#[derive(Copy, Debug, PartialEq, Clone)]
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 23:03:29 +01:00
pub enum LinkagePreference {
RequireDynamic,
RequireStatic,
}
#[derive(Copy, Clone, PartialEq, FromPrimitive)]
pub enum NativeLibraryKind {
NativeStatic, // native static library (.a archive)
NativeFramework, // OSX-specific
NativeUnknown, // default way to specify a dynamic library
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 23:03:29 +01:00
}
// Where a crate came from on the local filesystem. One of these two options
// must be non-None.
#[derive(PartialEq, Clone)]
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 23:03:29 +01:00
pub struct CrateSource {
rustc: Fix a leak in dependency= paths With the addition of separate search paths to the compiler, it was intended that applications such as Cargo could require a `--extern` flag per `extern crate` directive in the source. The system can currently be subverted, however, due to the `existing_match()` logic in the crate loader. When loading crates we first attempt to match an `extern crate` directive against all previously loaded crates to avoid reading metadata twice. This "hit the cache if possible" step was erroneously leaking crates across the search path boundaries, however. For example: extern crate b; extern crate a; If `b` depends on `a`, then it will load crate `a` when the `extern crate b` directive is being processed. When the compiler reaches `extern crate a` it will use the previously loaded version no matter what. If the compiler was not invoked with `-L crate=path/to/a`, it will still succeed. This behavior is allowing `extern crate` declarations in Cargo without a corresponding declaration in the manifest of a dependency, which is considered a bug. This commit fixes this problem by keeping track of the origin search path for a crate. Crates loaded from the dependency search path are not candidates for crates which are loaded from the crate search path. As a result of this fix, this is a likely a breaking change for a number of Cargo packages. If the compiler starts informing that a crate can no longer be found, it likely means that the dependency was forgotten in your Cargo.toml. [breaking-change]
2015-01-06 17:46:07 +01:00
pub dylib: Option<(Path, PathKind)>,
pub rlib: Option<(Path, PathKind)>,
pub cnum: ast::CrateNum,
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 23:03:29 +01:00
}
pub struct CStore {
metas: RefCell<FnvHashMap<ast::CrateNum, Rc<crate_metadata>>>,
/// Map from NodeId's of local extern crate statements to crate numbers
extern_mod_crate_map: RefCell<NodeMap<ast::CrateNum>>,
used_crate_sources: RefCell<Vec<CrateSource>>,
used_libraries: RefCell<Vec<(String, NativeLibraryKind)>>,
used_link_args: RefCell<Vec<String>>,
pub intr: Rc<IdentInterner>,
}
2013-12-25 21:08:04 +01:00
impl CStore {
2014-03-27 18:28:38 +01:00
pub fn new(intr: Rc<IdentInterner>) -> CStore {
2013-12-25 21:08:04 +01:00
CStore {
metas: RefCell::new(FnvHashMap()),
extern_mod_crate_map: RefCell::new(FnvHashMap()),
used_crate_sources: RefCell::new(Vec::new()),
used_libraries: RefCell::new(Vec::new()),
used_link_args: RefCell::new(Vec::new()),
2013-12-25 21:08:04 +01:00
intr: intr
}
}
pub fn next_crate_num(&self) -> ast::CrateNum {
self.metas.borrow().len() as ast::CrateNum + 1
}
2014-04-17 14:06:25 +02:00
pub fn get_crate_data(&self, cnum: ast::CrateNum) -> Rc<crate_metadata> {
(*self.metas.borrow())[cnum].clone()
2013-12-25 21:08:04 +01:00
}
pub fn get_crate_hash(&self, cnum: ast::CrateNum) -> Svh {
2013-12-25 21:08:04 +01:00
let cdata = self.get_crate_data(cnum);
decoder::get_crate_hash(cdata.data())
}
2014-04-17 14:06:25 +02:00
pub fn set_crate_data(&self, cnum: ast::CrateNum, data: Rc<crate_metadata>) {
2014-03-21 03:49:20 +01:00
self.metas.borrow_mut().insert(cnum, data);
2013-12-25 21:08:04 +01:00
}
2014-12-09 02:26:43 +01:00
pub fn iter_crate_data<I>(&self, mut i: I) where
I: FnMut(ast::CrateNum, &crate_metadata),
{
2015-01-31 18:20:46 +01:00
for (&k, v) in &*self.metas.borrow() {
2014-04-17 14:06:25 +02:00
i(k, &**v);
2013-12-25 21:08:04 +01:00
}
}
/// Like `iter_crate_data`, but passes source paths (if available) as well.
2014-12-09 02:26:43 +01:00
pub fn iter_crate_data_origins<I>(&self, mut i: I) where
I: FnMut(ast::CrateNum, &crate_metadata, Option<CrateSource>),
{
2015-01-31 18:20:46 +01:00
for (&k, v) in &*self.metas.borrow() {
let origin = self.get_used_crate_source(k);
origin.as_ref().map(|cs| { assert!(k == cs.cnum); });
i(k, &**v, origin);
}
}
2013-12-21 05:00:58 +01:00
pub fn add_used_crate_source(&self, src: CrateSource) {
let mut used_crate_sources = self.used_crate_sources.borrow_mut();
2014-03-21 03:49:20 +01:00
if !used_crate_sources.contains(&src) {
used_crate_sources.push(src);
2013-12-25 21:08:04 +01:00
}
}
2013-12-25 19:10:33 +01:00
pub fn get_used_crate_source(&self, cnum: ast::CrateNum)
-> Option<CrateSource> {
2014-03-21 03:49:20 +01:00
self.used_crate_sources.borrow_mut()
.iter().find(|source| source.cnum == cnum).cloned()
2013-12-25 19:10:33 +01:00
}
pub fn reset(&self) {
self.metas.borrow_mut().clear();
self.extern_mod_crate_map.borrow_mut().clear();
self.used_crate_sources.borrow_mut().clear();
self.used_libraries.borrow_mut().clear();
self.used_link_args.borrow_mut().clear();
2013-12-25 19:10:33 +01:00
}
// This method is used when generating the command line to pass through to
// system linker. The linker expects undefined symbols on the left of the
// command line to be defined in libraries on the right, not the other way
// around. For more info, see some comments in the add_used_library function
// below.
//
// In order to get this left-to-right dependency ordering, we perform a
// topological sort of all crates putting the leaves at the right-most
// positions.
2013-12-25 21:08:04 +01:00
pub fn get_used_crates(&self, prefer: LinkagePreference)
-> Vec<(ast::CrateNum, Option<Path>)> {
let mut ordering = Vec::new();
fn visit(cstore: &CStore, cnum: ast::CrateNum,
ordering: &mut Vec<ast::CrateNum>) {
if ordering.contains(&cnum) { return }
let meta = cstore.get_crate_data(cnum);
2015-01-31 18:20:46 +01:00
for (_, &dep) in &meta.cnum_map {
visit(cstore, dep, ordering);
}
ordering.push(cnum);
};
2015-01-31 18:20:46 +01:00
for (&num, _) in &*self.metas.borrow() {
visit(self, num, &mut ordering);
}
ordering.reverse();
2014-03-21 03:49:20 +01:00
let mut libs = self.used_crate_sources.borrow()
.iter()
2013-12-25 21:08:04 +01:00
.map(|src| (src.cnum, match prefer {
rustc: Fix a leak in dependency= paths With the addition of separate search paths to the compiler, it was intended that applications such as Cargo could require a `--extern` flag per `extern crate` directive in the source. The system can currently be subverted, however, due to the `existing_match()` logic in the crate loader. When loading crates we first attempt to match an `extern crate` directive against all previously loaded crates to avoid reading metadata twice. This "hit the cache if possible" step was erroneously leaking crates across the search path boundaries, however. For example: extern crate b; extern crate a; If `b` depends on `a`, then it will load crate `a` when the `extern crate b` directive is being processed. When the compiler reaches `extern crate a` it will use the previously loaded version no matter what. If the compiler was not invoked with `-L crate=path/to/a`, it will still succeed. This behavior is allowing `extern crate` declarations in Cargo without a corresponding declaration in the manifest of a dependency, which is considered a bug. This commit fixes this problem by keeping track of the origin search path for a crate. Crates loaded from the dependency search path are not candidates for crates which are loaded from the crate search path. As a result of this fix, this is a likely a breaking change for a number of Cargo packages. If the compiler starts informing that a crate can no longer be found, it likely means that the dependency was forgotten in your Cargo.toml. [breaking-change]
2015-01-06 17:46:07 +01:00
RequireDynamic => src.dylib.clone().map(|p| p.0),
RequireStatic => src.rlib.clone().map(|p| p.0),
2013-12-25 21:08:04 +01:00
}))
rustc: Fix a leak in dependency= paths With the addition of separate search paths to the compiler, it was intended that applications such as Cargo could require a `--extern` flag per `extern crate` directive in the source. The system can currently be subverted, however, due to the `existing_match()` logic in the crate loader. When loading crates we first attempt to match an `extern crate` directive against all previously loaded crates to avoid reading metadata twice. This "hit the cache if possible" step was erroneously leaking crates across the search path boundaries, however. For example: extern crate b; extern crate a; If `b` depends on `a`, then it will load crate `a` when the `extern crate b` directive is being processed. When the compiler reaches `extern crate a` it will use the previously loaded version no matter what. If the compiler was not invoked with `-L crate=path/to/a`, it will still succeed. This behavior is allowing `extern crate` declarations in Cargo without a corresponding declaration in the manifest of a dependency, which is considered a bug. This commit fixes this problem by keeping track of the origin search path for a crate. Crates loaded from the dependency search path are not candidates for crates which are loaded from the crate search path. As a result of this fix, this is a likely a breaking change for a number of Cargo packages. If the compiler starts informing that a crate can no longer be found, it likely means that the dependency was forgotten in your Cargo.toml. [breaking-change]
2015-01-06 17:46:07 +01:00
.collect::<Vec<_>>();
libs.sort_by(|&(a, _), &(b, _)| {
ordering.position_elem(&a).cmp(&ordering.position_elem(&b))
});
libs
2013-12-25 21:08:04 +01:00
}
pub fn add_used_library(&self, lib: String, kind: NativeLibraryKind) {
2013-12-25 21:08:04 +01:00
assert!(!lib.is_empty());
2014-03-21 03:49:20 +01:00
self.used_libraries.borrow_mut().push((lib, kind));
2013-12-25 21:08:04 +01:00
}
pub fn get_used_libraries<'a>(&'a self)
-> &'a RefCell<Vec<(String,
NativeLibraryKind)>> {
&self.used_libraries
2013-12-25 21:08:04 +01:00
}
2013-12-21 05:00:58 +01:00
pub fn add_used_link_args(&self, args: &str) {
for s in args.split(' ').filter(|s| !s.is_empty()) {
self.used_link_args.borrow_mut().push(s.to_string());
2013-12-25 21:08:04 +01:00
}
}
pub fn get_used_link_args<'a>(&'a self) -> &'a RefCell<Vec<String> > {
&self.used_link_args
2013-12-25 21:08:04 +01:00
}
2013-12-21 05:00:58 +01:00
pub fn add_extern_mod_stmt_cnum(&self,
2013-12-25 21:08:04 +01:00
emod_id: ast::NodeId,
cnum: ast::CrateNum) {
2014-03-21 03:49:20 +01:00
self.extern_mod_crate_map.borrow_mut().insert(emod_id, cnum);
}
2013-12-25 21:08:04 +01:00
pub fn find_extern_mod_stmt_cnum(&self, emod_id: ast::NodeId)
-> Option<ast::CrateNum> {
self.extern_mod_crate_map.borrow().get(&emod_id).cloned()
2013-12-25 21:08:04 +01:00
}
2013-07-02 21:47:32 +02:00
}
impl crate_metadata {
pub fn data<'a>(&'a self) -> &'a [u8] { self.data.as_slice() }
pub fn name(&self) -> String { decoder::get_crate_name(self.data()) }
pub fn hash(&self) -> Svh { decoder::get_crate_hash(self.data()) }
}
impl MetadataBlob {
pub fn as_slice<'a>(&'a self) -> &'a [u8] {
let slice = match *self {
MetadataVec(ref vec) => vec.as_slice(),
rustc: Optimize reading metadata by 4x We were previously reading metadata via `ar p`, but as learned from rustdoc awhile back, spawning a process to do something is pretty slow. Turns out LLVM has an Archive class to read archives, but it cannot write archives. This commits adds bindings to the read-only version of the LLVM archive class (with a new type that only has a read() method), and then it uses this class when reading the metadata out of rlibs. When you put this in tandem of not compressing the metadata, reading the metadata is 4x faster than it used to be The timings I got for reading metadata from the respective libraries was: libstd-04ff901e-0.9-pre.dylib => 100ms libstd-04ff901e-0.9-pre.rlib => 23ms librustuv-7945354c-0.9-pre.dylib => 4ms librustuv-7945354c-0.9-pre.rlib => 1ms librustc-5b94a16f-0.9-pre.dylib => 87ms librustc-5b94a16f-0.9-pre.rlib => 35ms libextra-a6ebb16f-0.9-pre.dylib => 63ms libextra-a6ebb16f-0.9-pre.rlib => 15ms libsyntax-2e4c0458-0.9-pre.dylib => 86ms libsyntax-2e4c0458-0.9-pre.rlib => 22ms In order to always take advantage of these faster metadata read-times, I sort the files in filesearch based on whether they have an rlib extension or not (prefer all rlib files first). Overall, this halved the compile time for a `fn main() {}` crate from 0.185s to 0.095s on my system (when preferring dynamic linking). Reading metadata is still the slowest pass of the compiler at 0.035s, but it's getting pretty close to linking at 0.021s! The next best optimization is to just not copy the metadata from LLVM because that's the most expensive part of reading metadata right now.
2013-12-17 05:58:21 +01:00
MetadataArchive(ref ar) => ar.as_slice(),
};
if slice.len() < 4 {
&[] // corrupt metadata
} else {
let len = (((slice[0] as u32) << 24) |
((slice[1] as u32) << 16) |
((slice[2] as u32) << 8) |
((slice[3] as u32) << 0)) as uint;
if len + 4 <= slice.len() {
2015-01-18 01:15:52 +01:00
&slice[4.. len + 4]
} else {
&[] // corrupt or old metadata
}
}
}
}