QAR format¶
Introduction¶
A simple file format which allow combining many files together into a single archive.
Utilities¶
Usage: qar list path.qar
Usage: qar build-idx path.qar
Usage: qar create path.qar path
Usage: qar extract path.qar path
Usage: qar cp path_src path_dst
Usage: qar cat path1 path2 ...
Usage: qar l path1.qar path2.qar ...
Usage: qar lr path1.qar path2.qar ...
Usage: qar b path1.qar path2.qar ...
Usage: qar br path1.qar path2.qar ...
Usage: qar c path1 path2 ...
Usage: qar x path1.qar path2.qar ...
Usage: qar cr path1 path2 ...
Remove folders after qar files created
Usage: qar xr path1.qar path2.qar ...
Remove qar files after folder extracted
Usage: qar-glimpse path1.qar path2.qar ... path1 path2 ...
Multiple volumes¶
Can store the single folder in multiple qar archives. For example:
folder.qar
folder.qar.v1
folder.qar.v2
folder.qar.v3
Each file is a valid qar file on its own, but only contains a portion of the content of the original folder.
Environment variable q_qar_multi_vol_max_size controls the size of the each qar volume file.
Note¶
Only store file names and content of files (allow record some info about the file in FILE-INFO).
‘/’ is allowed in
FILE-NAME, and represent directory structure after extraction.Does not store directories in QAR format. During extraction, directories will only be created to allow the files to be extracted to the specified path.
Empty directories will not be recorded and recovered after extraction.
QAR file format (.qar)¶
Source code located in qlat-utils/include/qlat-utils/qar.h
Example file is located in docs/contents/qar-sample.qar.
FILE-FORMAT-LINE
FILE-HEADER
[newline character]
FILE-NAME
[newline character]
FILE-INFO
[newline character]
FILE-DATA
[newline character]
[newline character]
FILE-HEADER
[newline character]
FILE-NAME
[newline character]
FILE-INFO
[newline character]
FILE-DATA
[newline character]
[newline character]
...
FILE-FORMAT-LINE¶
"#!/usr/bin/env qar-glimpse\n\n"
NOTE: There are two newline characters in the end of the FILE-FORMAT-LINE.
Quote symbols are not part of the FILE-FORMAT-LINE.
FILE-HEADER¶
"QAR-FILE"
[one or more space characters]
[FILE-NAME size in bytes stored in ASCII]
[one or more space characters]
[FILE-INFO size in bytes stored in ASCII]
[one or more space characters]
[FILE-DATA size in bytes stored in ASCII]
Quote symbols are not part of the FILE-HEADER.
FILE-INFO¶
Can store some metadata information about the file. The default is simply empty.
Segment size¶
[FILE-HEADER size] + 1 + [FILE-NAME size] + 1 + [FILE-INFO size] + 1 + [FILE-DATA size] + 2
Example¶
#!/usr/bin/env qar-glimpse
QAR-FILE 13 0 20
filename1.txt
Contents for file1.
QAR-FILE 13 0 20
filename2.txt
Contents for file2.
QAR-FILE 13 0 20
filename3.txt
Contents for file3.
QAR-FILE 18 0 21
folder1/file-a.txt
Contents for file-a.
QAR-FILE 18 0 21
folder2/file-b.txt
Contents for file-b.
QAR-FILE 18 0 21
folder2/file-c.txt
Contents for file-c.
QAR Index file format (.qar.idx)¶
A .qar.idx file is a companion index to a .qar archive (or multi-volume set). It is generated by qar build-idx and loaded to avoid scanning the archive sequentially.
Source code: show_qar_index and parse_qar_index in qlat-utils/lib/qar.cpp.
IDX-FORMAT-LINE
ENTRY-HEADER
[newline character]
ENTRY-FILENAME
[newline character]
ENTRY-SEGMENT-INFO
[newline character]
[newline character]
ENTRY-HEADER
[newline character]
ENTRY-FILENAME
[newline character]
ENTRY-SEGMENT-INFO
[newline character]
[newline character]
...
IDX-FORMAT-LINE¶
"#!/usr/bin/env qar-idx-glimpse\n\n"
NOTE: There are two newline characters in the end of the IDX-FORMAT-LINE.
ENTRY-HEADER¶
"QAR-FILE-IDX"
[one or more space characters]
[volume index, stored in ASCII]
[one or more space characters]
[entry index within volume (0-based), stored in ASCII]
[one or more space characters]
[filename length in bytes, stored in ASCII]
ENTRY-FILENAME¶
The full path of the file as stored in the archive (may contain / for directory structure).
ENTRY-SEGMENT-INFO¶
[offset] [offset_fn] [offset_info] [offset_data] [offset_end] [fn_len] [info_len] [data_len]
Each value is a Long integer stored in ASCII, separated by single space characters:
Field |
Description |
|---|---|
|
Absolute byte offset of the segment’s FILE-HEADER start |
|
Absolute byte offset of the segment’s FILE-NAME |
|
Absolute byte offset of the segment’s FILE-INFO |
|
Absolute byte offset of the segment’s FILE-DATA |
|
Absolute byte offset past the end of the segment (after |
|
Length of FILE-NAME in bytes |
|
Length of FILE-INFO in bytes |
|
Length of FILE-DATA in bytes |
Index entry size¶
len(ENTRY-HEADER) + 1 + len(ENTRY-FILENAME) + 1 + len(ENTRY-SEGMENT-INFO) + 1 + 1
Example¶
#!/usr/bin/env qar-idx-glimpse
QAR-FILE-IDX 0 0 13
filename1.txt
28 45 59 60 82 13 0 20
QAR-FILE-IDX 0 1 13
filename2.txt
82 99 113 114 136 13 0 20
QAR-FILE-IDX 0 2 13
filename3.txt
136 153 167 168 190 13 0 20
QAR-FILE-IDX 0 3 18
folder1/file-a.txt
190 207 226 227 250 18 0 21
QAR-FILE-IDX 0 4 18
folder2/file-b.txt
250 267 286 287 310 18 0 21
QAR-FILE-IDX 0 5 18
folder2/file-c.txt
310 327 346 347 370 18 0 21
This example corresponds to the .qar sample above. All entries belong to volume index 0. The segment info values reference byte offsets into qar-sample.qar.