Pravega v0.7.0-rc1 Release Notes
Release Date: 2020-03-13 // about 4 years ago-
🚀 The major goal of this release was performance and stability improvements. Core to these changes is a new Cache used by the Segment Store, designed from the ground up to support the type of streaming workloads that Pravega uses. It also contains full support for DellEMC ECS as Tier 2 binding.
🚀 This release resolves 100 issues; see the attached log file.
🐎 Performance
🚀 In terms of Performmance, Pravega 0.7 contains significant improvements over the previous release. These are the main improvements included in 0.7:
- 👀 #4044: Streaming Cache. See next section for details.
- #4303: Avoid Netty Flush from the Client (#4294). This change minimizes the amount of Netty flush calls issued by the client and avoids blocking on the futures returned from write() unless it is needed.
- 🐎 #4558: Low single segment write performance (#4559). This PR changes the default digest for Bookkeeper ledgers to CRC32C, which is much faster compared to the previous one used.
- #4578: Early wake reader in response to server reply (#4579). This PR helps to unblock reader threads waiting on idle segments in the event that data arrives from the server on another segment.
- #4597: Optimization of String splits for Metrics (#4598). This PR minimizes the overhead related to String manipulation on the write path, significantly improving throughput for small events.
- #4274: Write event padding more efficiently.
- #4396, #4147: Improved handling of transactions.
Streaming Cache
In 0.7, we have replaced RocksDB with a home-grown, block-based memory cache, which improved stability in containerized environments and eliminated a major bottleneck on the ingestion and read paths. A few highlights about the Streaming Cache:
- 📇 Index-less, block-based storage for arbitrary-sized cache entries. Metadata overhead is a constant 0.2%.
- 👉 Uses direct memory to store all data.
- Can be provided with an explicit upper bound for memory usage and it will never exceed this.
- All cache entries are broken down into equal-sized 4KB blocks which are memory-aligned to the OS page file boundaries.
- 🆓 Enables copy-free appends to existing cache entries, which significantly minimizes the amount of time required to process small Events.
- 🆓 Enables copy-free cache entry slicing (reading a sub-range of a cache entry without copying the whole entry into a heap buffer).
👍 DellEMC ECS Support
🚀 We are happy to announce that Pravega now fully supports DellEMC ECS object storage (ECS for short) as Tier 2 using the open-source Extended S3 Client. In this release we focused on testing ECS support more thoroughly. We found and fixed a few issues. We also changed the way we configure ECS to make it more user friendly and extensible. Pravega now uses the term "prefix" instead of "root" to be more consistent with the Object Storage community and accepts REST style uri for ECS client side configurations.
You can find more details about using ECS as Tier 2 in our Pravega Operator Doc.
🐎 Performance improvements:
- #4609: For smaller source segments, concat should read and append instead of using multi-part upload
- #4505: Add missing metrics for ExtendedS3Storage.
🐛 Bug Fixes
- 📇 #4591: Fixed ExtendedS3 exists(). This change uses getObjectMetadata instead of listObjects to implement exists. This fixes the problem where listObject based check behaved incorrectly when prefix instead of full path is provided as parameter.
🔄 Changes in how we configure ECS
- 💅 #4561: Using S3 Rest Style Config URI for Extended S3 client construction.
- #4550: Renamed ExtendedS3 config item from root to prefix.
- #4412: Fix bug in ExtendedS3StorageConfig to correctly handle missing trailing '/' in prefix.
Other changes