feat(server): store offsets of groups managed outside Iggy

[krishvishal]

Which issue does this PR address?

Relates to #3542

Rationale

The Kafka gateway needs to keep a committed offset for every Kafka group, topic and partition. The accepted design in gateways/kafka/docs/OFFSET_STORAGE.md stores it as an Iggy consumer group offset under kafka.cg.<group>. That kind does not fit
the gateway:

• The server lets only a member that owns the partition write a group offset, and the gateway never joins the group.
• The server refuses an offset past the last message. Kafka commits the next offset to read, which is one past the last message, and it can commit 0 on an empty partition.
• A group offset holds back retention. A Kafka committed offset never does.
• An Iggy consumer that polls that key with Next skips one record per partition, and only a naming rule stops it.

The gateway handlers and the update to that design doc come in a separate PR.

What changed?

Group offsets had to come from a member of an Iggy consumer group, inside the partition's range, and they held back retention. A new consumer kind, ExternalGroup (wire value 3, Consumer::external_group in the Rust SDK), now stores offsets for a group that lives outside Iggy. The key is the numeric id of an existing Iggy consumer group, so the gateway still creates that group but never joins it. These offsets have their own key space, so they never share a key with the group's own offsets.

For this kind the server:

• accepts a store or a delete for any existing group, with no membership check
• stores the value as sent, with no range check and no clamp at boot
• refuses a poll with FeatureUnavailable
• leaves it out of the retention barrier
• deletes the offsets of a deleted group, through the same replicated delete as group offsets

The server replicates these offsets, recovers them at boot from offsets/external_groups/, carries them in partition state transfer, and clears them on a purge. Boot creates that directory for a partition that an older build made. The HTTP client
returns FeatureUnavailable for this kind, because the REST API cannot name a consumer kind. The CLI accepts --kind external-group.

Some changes elsewhere follow from the new kind:

• A caught-up primary now serves a state transfer of an empty partition with committed ops. A client can store and then delete an offset on a partition that never took a message. Before, a backup behind the repair floor had no way past those ops.
• A purge whose offset directory sync fails leaves its generation unrecorded, so the reconciler runs it again. Boot never clamps an external offset, so without this a crash can bring back a deleted one.
• A state transfer install syncs offsets/ after the kind directories. The commit walk syncs it once per walk, not once per kind.

[codecov]

Codecov Report

❌ Patch coverage is 92.92543% with 74 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.90%. Comparing base (2d7fddb) to head (a53cdc6).

Files with missing lines	Patch %	Lines
core/partitions/src/iggy_partition.rs	95.14%	23 Missing and 7 partials ⚠️
core/partitions/src/partition_storage.rs	78.44%	23 Missing and 2 partials ⚠️
core/partitions/src/state_transfer.rs	85.71%	10 Missing and 5 partials ⚠️
...nds/binary_consumer_offsets/get_consumer_offset.rs	0.00%	1 Missing ⚠️
core/sdk/src/http/consumer_offsets.rs	83.33%	1 Missing ⚠️
core/server/src/dispatch/partition.rs	96.66%	0 Missing and 1 partial ⚠️
core/server/src/namespace.rs	85.71%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #4392      +/-   ##
============================================
- Coverage     87.96%   87.90%   -0.06%     
  Complexity     1579     1579              
============================================
  Files          1290     1290              
  Lines        230145   230276     +131     
  Branches     193480   193612     +132     
============================================
- Hits         202440   202429      -11     
- Misses        22980    23092     +112     
- Partials       4725     4755      +30     

Components	Coverage Δ
Rust Core	89.00% <92.92%> (-0.07%)	⬇️
Java SDK	68.74% <ø> (ø)
C# SDK	77.64% <ø> (+0.02%)	⬆️
Python SDK	91.24% <ø> (ø)
PHP SDK	85.67% <ø> (ø)
Node SDK	96.57% <ø> (-0.03%)	⬇️
Go SDK	70.26% <ø> (+0.07%)	⬆️

Files with missing lines	Coverage Δ
core/binary_protocol/src/primitives/consumer.rs	100.00% <100.00%> (ø)
core/common/src/types/consumer/consumer_kind.rs	59.67% <100.00%> (+12.80%)	⬆️
core/partitions/src/consumer_offset_capacity.rs	99.00% <100.00%> (+1.59%)	⬆️
core/partitions/src/iggy_partitions.rs	87.82% <100.00%> (+0.25%)	⬆️
core/partitions/src/lib.rs	0.00% <ø> (ø)
core/partitions/src/offset_recovery.rs	73.85% <100.00%> (+0.74%)	⬆️
core/partitions/src/persistence.rs	92.05% <100.00%> (ø)
core/partitions/src/types.rs	83.09% <100.00%> (+1.14%)	⬆️
core/sdk/src/http/messages.rs	92.55% <ø> (ø)
core/server/src/consumer_group.rs	98.34% <100.00%> (+<0.01%)	⬆️
... and 16 more

... and 42 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.