Home | History | Annotate | Download | only in common
      1 #
      2 # CDDL HEADER START
      3 #
      4 # The contents of this file are subject to the terms of the
      5 # Common Development and Distribution License (the "License").
      6 # You may not use this file except in compliance with the License.
      7 #
      8 # You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
      9 # or http://www.opensolaris.org/os/licensing.
     10 # See the License for the specific language governing permissions
     11 # and limitations under the License.
     12 #
     13 # When distributing Covered Code, include this CDDL HEADER in each
     14 # file and include the License file at usr/src/OPENSOLARIS.LICENSE.
     15 # If applicable, add the following below this CDDL HEADER, with the
     16 # fields enclosed by brackets "[]" replaced with your own identifying
     17 # information: Portions Copyright [yyyy] [name of copyright owner]
     18 #
     19 # CDDL HEADER END
     20 #
     21 
     22 #
     23 # Copyright 2008 Sun Microsystems, Inc.  All rights reserved.
     24 # Use is subject to license terms.
     25 #
     26 
     27 
     28 Why 32-bit libelf is not Large File Aware
     29 -----------------------------------------
     30 
     31 The ELF format uses unsigned 32-bit integers for offsets, so the
     32 theoretical limit on a 32-bit ELF object is 4GB. However, libelf
     33 imposes a 2GB limit on the objects it can create. The Solaris
     34 link-editor and related tools are all based on libelf, so the
     35 32-bit version of the link-editor also has a 2GB limit, despite
     36 the theoretical limit of 4GB.
     37 
     38 Large file support (LFS) is a half step between the 32 and 64-bit
     39 worlds, in which an otherwise 32-bit limited process is allowed to
     40 read and write data to a file that can be larger than 2GB (the extent
     41 of a signed 32-bit integer, as represented by the system type off_t).
     42 LFS is useful if the program only needs to access a small subset of
     43 the file data at any given time (e.g. /usr/bin/cat). It is less useful
     44 if the program needs to access a large amount of data at once --- having
     45 been freed from the file limit, the program will simply hit the virtual
     46 memory limit (4GB).
     47 
     48 In particular, the link-editor generally requires twice as much
     49 memory as the size of the output object, half to hold the input
     50 objects, and half to hold the result. This means that a 32-bit
     51 link-editor process will hit the 2GB file size limit and the 4GB
     52 address space limit at roughly the same time. As a result, a
     53 large file aware 32-bit version of libelf has no significant value.
     54 Despite this, the question of what it would take to make libelf
     55 large file aware comes up from time to time.
     56 
     57 The first step would be to provide alternative versions of
     58 all public data structures that involve the off_t data type.
     59 These structs, found in /usr/include/libelf.h, are:
     60 
     61 	/*
     62 	 * Archive member header
     63 	 */
     64 	typedef struct {
     65 		char		*ar_name;
     66 		time_t		ar_date;
     67 		uid_t		ar_uid;
     68 		gid_t 		ar_gid;
     69 		mode_t		ar_mode;
     70 		off_t		ar_size;
     71 		char 		*ar_rawname;
     72 	} Elf_Arhdr;
     73 
     74 
     75 	/*
     76 	 * Data descriptor
     77 	 */
     78 	typedef struct {
     79 		Elf_Void	*d_buf;
     80 		Elf_Type	d_type;
     81 		size_t		d_size;
     82 		off_t		d_off;		/* offset into section */
     83 		size_t		d_align;	/* alignment in section */
     84 		unsigned	d_version;	/* elf version */
     85 	} Elf_Data;
     86 
     87 As off_t is a signed type, these alternative versions would have to use
     88 an off64_t type instead.
     89 
     90 In addition to providing alternative large file aware Elf_Arhdr and
     91 Elf_Data types, it would be necessary to implement large file aware
     92 versions of the public functions that use them, also found in
     93 /usr/include/libelf.h:
     94 
     95 	/*
     96 	 * Function declarations
     97 	 */
     98 	unsigned  elf_flagdata(Elf_Data *, Elf_Cmd, unsigned);
     99 	Elf_Arhdr *elf_getarhdr(Elf *);
    100 	off_t	  elf_getbase(Elf *);
    101 	Elf_Data  *elf_getdata(Elf_Scn *, Elf_Data *);
    102 	Elf_Data  *elf_newdata(Elf_Scn *);
    103 	Elf_Data  *elf_rawdata(Elf_Scn *, Elf_Data *);
    104 	off_t	  elf_update(Elf *, Elf_Cmd);
    105 	Elf_Data  *elf32_xlatetof(Elf_Data *, const Elf_Data *, unsigned);
    106 	Elf_Data  *elf32_xlatetom(Elf_Data *, const Elf_Data *, unsigned);
    107 	Elf_Data  *elf64_xlatetof(Elf_Data *, const Elf_Data *, unsigned);
    108 	Elf_Data  *elf64_xlatetom(Elf_Data *, const Elf_Data *, unsigned);
    109 
    110 It is important to note that these new versions cannot replace the
    111 original definitions. Those must continue to be available to support
    112 non-large-file-aware programs. These new types and functions would be in
    113 addition to the pre-existing versions.
    114 
    115 When you make code like this large file aware, it is necessary to undertake
    116 a careful analysis of the code to ensure that all the surrounding code uses
    117 variable types large enough to handle the increased range. Hence, this work
    118 is more complicated than simply supplying variants that use a bigger
    119 off_t and rebuilding --- that is just the first step.
    120 
    121 There are two standard preprocessor definitions used to control
    122 large file support:
    123 
    124 	_LARGEFILE64_SOURCE
    125 	_FILE_OFFSET_BITS
    126 
    127 These preprocessor definitions would be used to determine whether 
    128 a given program linked against libelf would see the regular, or
    129 the large file aware versions of the above types and routines.
    130 This is the same approach used in other large file capable software,
    131 such as libc.
    132 
    133 Finally, all the applications that rely on libelf would need to be made
    134 large file aware. As with libelf itself, there is more to such an effort
    135 than recompiling with preprocessor macros set. The code in these
    136 applications would need to be examined carefully. Some of these programs
    137 are very old, and were not originally written with such type portability
    138 in mind. Such code can be difficult to transition.
    139 
    140 To work around the 2GB limit in 32-bit libelf:
    141 
    142     - The fundamental limits of a 32-bit address space mean
    143       that a program this large should be 64-bit. Only a 64-bit
    144       address space has enough room for that much code, plus the
    145       stack and heap needed to do useful work with it.
    146 
    147     - The 64-bit version of libelf is also able to process
    148       32-bit objects, and does not have a 2GB file size limit.
    149       Therefore, the 64-bit link-editor can be used to build a 32-bit
    150       executable which is >2GB. The resulting program will consume over
    151       half the available address space just to start running. However,
    152       there may be enough address space left for it to do useful work.
    153 
    154       Note that the 32-bit limit for sharable objects remains at
    155       2GB --- imposed by the runtime linker, which is also not large
    156       file aware.
    157