Changelog History
Page 2
-
v0.7.1-rc0 Changes
May 06, 2020🚀 This release contains several bug fix and performance improvements over 0.7.0:
Segment Store
- Issue #3992: (Data Recovery) Reconcile BookKeeperLog with BookKeeper Ledgers.
- Issue #4273: (Bug Fix) Revised Throtting Metrics.
- Issue #4642: (Bug Fix) Load TLS certificates from /etc/secret-volume/ca-bundle.
- 🛠 Issue #4652: (Bug Fix) Fixed Reconciliation of partially-flushed Storage append.
- Issue #4654: (Bug Fix) DataLogWriterNotPrimaryException thrown indefinitely.
- 🛠 Issue #4702: (Bug Fix) Fixed a deadlock condition in the Read Index.
- Issue #4706: (Bug Fix) Fix SegmentStoreMetrics.
- 📇 Issue #4721: (Bug Fix/Data Recovery) Auto-reconciliation of BookKeeperLog Metadata.
- Issue #4743: (Bug Fix) Infinite loop in StorageWriter if deleting a segment while flushing.
Client
- 🔊 Issue #4633: (Bug Fix) Improve controller client logs.
- 👻 Issue #4638: (Bug Fix) Propagate retryable exception to the client instead of returning Status.Failure on all exceptions.
- Issue #4669: (Bug Fix) Fix serialization in PositionImpl.
- 🐎 Issue #4720: (Performance) Reduce cpu use on client.
- Issue #4766: (Bug Fix) Ensure BatchClient handles partial response from SSS.
Controller
- Issue #4631: (Bug Fix) Failed transactions metric incorrectly reported.
- ⚡️ Issue #4639: (Bug Fix) Table store helper to retry only on unknownhost and auth failure for update requests.
General
-
v0.7.0 Changes
March 17, 2020🚀 The major goal of this release was performance and stability improvements. Core to these changes is a new Cache used by the Segment Store, designed from the ground up to support the type of streaming workloads that Pravega uses. It also contains full support for DellEMC ECS as Tier 2 binding.
🚀 This release resolves 100 issues; see the attached log file.
🐎 Performance
🚀 In terms of Performance, Pravega 0.7 contains significant improvements over the previous release. These are the main improvements included in 0.7:
- 👀 #4044: Streaming Cache. See next section for details.
- #4303: Avoid Netty Flush from the Client (#4294). This change minimizes the amount of Netty flush calls issued by the client and avoids blocking on the futures returned from write() unless it is needed.
- 🐎 #4558: Low single segment write performance (#4559). This PR changes the default digest for Bookkeeper ledgers to CRC32C, which is much faster compared to the previous one used.
- #4578: Early wake reader in response to server reply (#4579). This PR helps to unblock reader threads waiting on idle segments in the event that data arrives from the server on another segment.
- #4597: Optimization of String splits for Metrics (#4598). This PR minimizes the overhead related to String manipulation on the write path, significantly improving throughput for small events.
- #4274: Write event padding more efficiently.
- #4396, #4147: Improved handling of transactions.
Streaming Cache
In 0.7, we have replaced RocksDB with a home-grown, block-based memory cache, which improved stability in containerized environments and eliminated a major bottleneck on the ingestion and read paths. A few highlights about the Streaming Cache:
- 📇 Index-less, block-based storage for arbitrary-sized cache entries. Metadata overhead is a constant 0.2%.
- 👉 Uses direct memory to store all data.
- Can be provided with an explicit upper bound for memory usage and it will never exceed this.
- All cache entries are broken down into equal-sized 4KB blocks which are memory-aligned to the OS page file boundaries.
- 🆓 Enables copy-free appends to existing cache entries, which significantly minimizes the amount of time required to process small Events.
- 🆓 Enables copy-free cache entry slicing (reading a sub-range of a cache entry without copying the whole entry into a heap buffer).
👍 DellEMC ECS Support
👍 We are happy to announce that Pravega now fully supports DellEMC ECS object storage (ECS for short) as Tier 2 using the open-source Extended S3 Client. This is in addition to other Tier 2 options Pravega supports today (HDFS and Filesystem (NFS)).
🚀 In this release we focused on testing ECS support more thoroughly. We found and fixed a few issues. We also changed the way we configure ECS to make it more user friendly and extensible. Pravega now uses the term "prefix" instead of "root" to be more consistent with the Object Storage community and accepts REST style URIs for ECS client side configurations.
You can find more details in links below.
- Using ECS as Tier-2 in Pravega wiki.
- Using ECS as Tier 2 in our Pravega Operator Doc.
- 📚 Dell EMC ECS Object Storage
- 🚀 Extended S3 API
🐎 Performance improvements:
- #4609: For smaller source segments, concat should read and append instead of using multi-part upload
- #4505: Add missing metrics for ExtendedS3Storage.
🐛 Bug Fixes
- 📇 #4591: Fixed ExtendedS3 exists(). This change uses getObjectMetadata instead of listObjects to implement exists. This fixes the problem where listObject based check behaved incorrectly when prefix instead of full path is provided as parameter.
🔄 Changes in how we configure ECS
- 💅 #4561: Using S3 Rest Style Config URI for Extended S3 client construction.
- #4550: Renamed ExtendedS3 config item from root to prefix.
- #4412: Fix bug in ExtendedS3StorageConfig to correctly handle missing trailing '/' in prefix.
Other changes
-
v0.7.0-rc1 Changes
March 13, 2020🚀 The major goal of this release was performance and stability improvements. Core to these changes is a new Cache used by the Segment Store, designed from the ground up to support the type of streaming workloads that Pravega uses. It also contains full support for DellEMC ECS as Tier 2 binding.
🚀 This release resolves 100 issues; see the attached log file.
🐎 Performance
🚀 In terms of Performmance, Pravega 0.7 contains significant improvements over the previous release. These are the main improvements included in 0.7:
- 👀 #4044: Streaming Cache. See next section for details.
- #4303: Avoid Netty Flush from the Client (#4294). This change minimizes the amount of Netty flush calls issued by the client and avoids blocking on the futures returned from write() unless it is needed.
- 🐎 #4558: Low single segment write performance (#4559). This PR changes the default digest for Bookkeeper ledgers to CRC32C, which is much faster compared to the previous one used.
- #4578: Early wake reader in response to server reply (#4579). This PR helps to unblock reader threads waiting on idle segments in the event that data arrives from the server on another segment.
- #4597: Optimization of String splits for Metrics (#4598). This PR minimizes the overhead related to String manipulation on the write path, significantly improving throughput for small events.
- #4274: Write event padding more efficiently.
- #4396, #4147: Improved handling of transactions.
Streaming Cache
In 0.7, we have replaced RocksDB with a home-grown, block-based memory cache, which improved stability in containerized environments and eliminated a major bottleneck on the ingestion and read paths. A few highlights about the Streaming Cache:
- 📇 Index-less, block-based storage for arbitrary-sized cache entries. Metadata overhead is a constant 0.2%.
- 👉 Uses direct memory to store all data.
- Can be provided with an explicit upper bound for memory usage and it will never exceed this.
- All cache entries are broken down into equal-sized 4KB blocks which are memory-aligned to the OS page file boundaries.
- 🆓 Enables copy-free appends to existing cache entries, which significantly minimizes the amount of time required to process small Events.
- 🆓 Enables copy-free cache entry slicing (reading a sub-range of a cache entry without copying the whole entry into a heap buffer).
👍 DellEMC ECS Support
🚀 We are happy to announce that Pravega now fully supports DellEMC ECS object storage (ECS for short) as Tier 2 using the open-source Extended S3 Client. In this release we focused on testing ECS support more thoroughly. We found and fixed a few issues. We also changed the way we configure ECS to make it more user friendly and extensible. Pravega now uses the term "prefix" instead of "root" to be more consistent with the Object Storage community and accepts REST style uri for ECS client side configurations.
You can find more details about using ECS as Tier 2 in our Pravega Operator Doc.
🐎 Performance improvements:
- #4609: For smaller source segments, concat should read and append instead of using multi-part upload
- #4505: Add missing metrics for ExtendedS3Storage.
🐛 Bug Fixes
- 📇 #4591: Fixed ExtendedS3 exists(). This change uses getObjectMetadata instead of listObjects to implement exists. This fixes the problem where listObject based check behaved incorrectly when prefix instead of full path is provided as parameter.
🔄 Changes in how we configure ECS
- 💅 #4561: Using S3 Rest Style Config URI for Extended S3 client construction.
- #4550: Renamed ExtendedS3 config item from root to prefix.
- #4412: Fix bug in ExtendedS3StorageConfig to correctly handle missing trailing '/' in prefix.
Other changes
-
v0.7.0-rc0
March 06, 2020 -
v0.6.2 Changes
January 31, 2020🚀 This release contains two additional fixes over 0.6.1:
- ⬆️ Issue3696: Upgrade the Bookeeper server from 4.7.3 to 4.8.2.
- Issue 4357: Call noteWriterMark once per writer after committing a batch of transactions
-
v0.6.2-rc0 Changes
January 29, 2020🚀 This release contains two additional fixes over 0.6.1:
- ⬆️ Issue3696: Upgrade the Bookeeper server from 4.7.3 to 4.8.2.
- Issue 4357: Call noteWriterMark once per writer after committing a batch of transactions
-
v0.6.1 Changes
January 06, 2020🔖 Version 0.6.1 fixes 11 issues:
- Issue 4352: Turn retrieve token operation of DelegationTokenProvider asynchronous
🛠 This fixes a deadlock that was recently introduced as part of automatic refreshing of delegation token.
🔀 The delegation token provider implementation used client's internal executor to make a synchronous and blocking request for new delegation token which needs to be executed on same executor. The fix has made the DelegationTokenProvider interface operation retrieveToken() asynchronous and make its invocations in client non-blocking, in order to avoid client-side deadlocks. - Issue 4349: Complete partially created scopes before throwing ScopeExists
📇 This fixes an issue that was introduced in 0.5.0 when we moved the metadata to table segments. Creating scopes in pravega tables is a two step operation, where if we failed after first step, subsequent attempts to create the scope should succeed in running both steps in idempotent fashion. - 👻 Issue 4334: Fix auth and related exception handling
🌲 This fix improves exception handling and logging in gRPC interface to distinguish clearly among conditions representing authentication, authorization and internal failures. - Issue 4372: Split controller thread pools to avoid potential deadlock
🛠 This fixes a similar problem to 4352 where writer clients on controller process make a loopback call into local controller service. - 🔀 Issue 4383: Fix exception handling during state synchronizer compaction.
📇 This fix ensures SegmentIsTruncated response from the Segmentstore is handled while attempting to truncate segments using SegmentMetadataClient. - Issue 4369: (SegmentStore) Fix memory leak with Future Reads (#4373) (#4389)
🛠 Fixed a memory leak that can be triggered by abandoning Future Reads without any segment write activity. - Issue 4390: Ensure retries by client when delegation token expires. This change ensures that the client retries the request to segment store if the segment store throws an error indicating token expiry. Doing so will ensure that a new delegation token will be obtained after reauthentication and reauthorization with the Controller and used in the retried request.
- Issue 4399: BookKeeper write throttling. This enable Tier 1 throttling to prevent saturating Bookkeeper client.
- 🚚 Issue 4396: For completed transactions the ping fails with DataNotFoundException. This issue meant client could endlessly attempt to ping a completed transaction once its active record was moved.
- Issue 4147: Adjust txn ping to perform multiple pings per transaction. This ensures that transaction pings are more aggressively retried upon failure.
- 👻 Issue 4409: Reset tokenRefreshFuture upon exception in obtaining a new delegation token
🛠 This fixes a bug in token refresh logic after a token expiration.
🌲 Commit log file:
commitlog.txt - Issue 4352: Turn retrieve token operation of DelegationTokenProvider asynchronous