94 lines
3.2 KiB
Markdown
94 lines
3.2 KiB
Markdown
# CBSL: Compressed Binary Serialization Library
|
|
|
|
## What's this?
|
|
|
|
The library provides the binary serialization with the compression by [Zstandard](https://facebook.github.io/zstd/).
|
|
A motivation of the library is to implement the data-compressed checkpoint/restart, which is well-known technique to recover computer failures in high-performance computing.
|
|
This library aims to simple and lightweight access for users.
|
|
|
|
The library support C99 or later, and Fortran 2008 or later: perhaps we use the features are all supported by major compilers.
|
|
|
|
## How to build and test
|
|
|
|
The library uses [CMake](https://cmake.org/) version 3.3.x or later.
|
|
|
|
$ cmake --version
|
|
cmake version 3.14.3
|
|
|
|
CMake suite maintained and supported by Kitware (kitware.com/cmake).
|
|
$ mkdir build && cd build
|
|
$ cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/install/path ..
|
|
$ make
|
|
$ make test
|
|
$ make install
|
|
|
|
## Do you need help to install `zstd` package?
|
|
|
|
We can build and install `zstd` automatically in the build process.
|
|
Please pass `-D INSTALL_ZSTD=on` to cmake, we will install `zstd` package where `CMAKE_INSTALL_PREFIX` directory before building the library.
|
|
|
|
$ cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/install/path -D INSTALL_ZSTD=on ..
|
|
$ make
|
|
$ make test
|
|
$ make install
|
|
|
|
## Test environments
|
|
|
|
1. CMake version 3.14.3
|
|
2. GCC version 4.8.5
|
|
3. Zstandard version 1.4.0
|
|
4. CentOS Linux release 7.5.1804 (Core)
|
|
|
|
## Benchmark
|
|
|
|
We provide the library performance benchmark with best and worst case.
|
|
Please be reminded that **the benchmark results not indicate the performance of Zstandard**, these measure the overhead (use cost) of this library.
|
|
|
|
1. Best case : all data is zero filled (A compression ratio achieves up to 99%)
|
|
2. Worst case : data is generated by rand() (A compression ratio is lower than 1%)
|
|
|
|
`benchmark` target executes the benchmarks.
|
|
|
|
...
|
|
$ make benchmark
|
|
min data size = 262144.00 [B]
|
|
max data size = 134217728.00 [B]
|
|
data is zero filled (maximum compression)
|
|
<write data [Byte]> <time [seconds]> <speed [MiB/sec]>
|
|
262144 0.002397 109.360615
|
|
524288 0.002887 181.572393
|
|
1048576 0.003440 304.827177
|
|
...
|
|
|
|
## Use `zstd` command
|
|
|
|
A compressed file by the library can be decompressed by `zstd` command.
|
|
|
|
## License
|
|
|
|
Copyright 2019 Yuta Hirokawa (University of Tsukuba, Japan)
|
|
|
|
Licensed under the Apache License, Version 2.0 (the "License");
|
|
you may not use this file except in compliance with the License.
|
|
You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software
|
|
distributed under the License is distributed on an "AS IS" BASIS,
|
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
See the License for the specific language governing permissions and
|
|
limitations under the License.
|
|
|
|
## Future work
|
|
|
|
1. API error code
|
|
2. Refectoring tests
|
|
|
|
## NOTE
|
|
|
|
### Unsupported binary compatibility
|
|
|
|
The library does not support the binary compatibility and endianness conversion, which is required on the communication across machine.
|
|
If you want it, please consider using other serialization libraries such as [MessagePack](https://msgpack.org/).
|