aws-bench compatibility — tracking (gap matrix + dual-track plan)

[lex00]

aws-bench on Floci — gap matrix (derived 2026-07-26, corrected count v2)

Source: offline cdk synth of all 8 aws-bench-datasets scenarios (211 distinct resource types) diffed against Floci main's CFN provisioner (75 unique types incl. extracted provisioners). v2 fixes a double-count of the provision/delete switches in v1.

Per-scenario CFN coverage

• api-and-observability: 31 of 80 types missing
• compute-and-data: 15 of 40 types missing
• databases-and-storage: 25 of 69 types missing
• ec2-multiregion: 4 of 18 types missing
• reference-architectures: 70 of 129 types missing
• serverless-apps: 21 of 62 types missing
• streaming-and-iot: 22 of 47 types missing
• troubleshooting-multiservice: 50 of 93 types missing

Missing CFN resource types (140), tagged by scenario

• AWS::AmazonMQ::Broker refe
• AWS::ApiGateway::Account api-,data,refe,serv
• AWS::ApiGateway::ApiKey refe
• AWS::ApiGateway::UsagePlan refe
• AWS::ApiGateway::UsagePlanKey refe
• AWS::AppConfig::Application trou
• AWS::AppConfig::ConfigurationProfile trou
• AWS::AppConfig::Deployment trou
• AWS::AppConfig::DeploymentStrategy trou
• AWS::AppConfig::Environment trou
• AWS::AppConfig::HostedConfigurationVersion trou
• AWS::AppSync::DataSource refe
• AWS::AppSync::FunctionConfiguration refe
• AWS::AppSync::GraphQLApi refe
• AWS::AppSync::GraphQLSchema refe
• AWS::AppSync::Resolver refe
• AWS::Athena::WorkGroup data,serv
• AWS::AutoScaling::LifecycleHook serv,stre,trou
• AWS::AutoScaling::ScalingPolicy refe
• AWS::Backup::BackupPlan refe
• AWS::Backup::BackupSelection refe
• AWS::Backup::BackupVault refe
• AWS::Bedrock::KnowledgeBase api-
• AWS::CloudFront::CloudFrontOriginAccessIdentity comp,refe
• AWS::CloudFront::ConnectionGroup api-
• AWS::CloudFront::Distribution api-,comp,refe,trou
• AWS::CloudFront::Function refe
• AWS::CloudFront::OriginAccessControl api-
• AWS::CloudWatch::Dashboard api-,refe,serv
• AWS::CloudWatch::MetricStream serv
• AWS::CodeBuild::Project data,refe
• AWS::CodeCommit::Repository refe
• AWS::CodeDeploy::Application refe
• AWS::CodeDeploy::DeploymentGroup refe
• AWS::CodePipeline::Pipeline refe
• AWS::Cognito::IdentityPool api-
• AWS::Cognito::IdentityPoolRoleAttachment api-
• AWS::Cognito::UserPoolDomain api-
• AWS::CustomerProfiles::Domain stre
• AWS::DocDB::DBCluster serv
• AWS::DocDB::DBInstance serv
• AWS::DocDB::DBSubnetGroup serv
• AWS::EC2::FlowLog api-,data,trou
• AWS::EC2::InstanceConnectEndpoint refe
• AWS::EC2::IPAM comp
• AWS::EC2::IPAMPool comp
• AWS::EC2::IPAMScope comp
• AWS::EC2::LaunchTemplate comp,data,ec2-,refe,serv,stre,trou
• AWS::EC2::NetworkAcl data,trou
• AWS::EC2::NetworkAclEntry data,trou
• AWS::EC2::NetworkInterface data,trou
• AWS::EC2::NetworkInterfaceAttachment trou
• AWS::EC2::SecurityGroupEgress refe,serv,stre,trou
• AWS::EC2::SecurityGroupIngress api-,refe,serv,stre,trou
• AWS::EC2::SubnetNetworkAclAssociation data,trou
• AWS::EC2::TransitGateway trou
• AWS::EC2::TransitGatewayAttachment trou
• AWS::EC2::TransitGatewayRoute trou
• AWS::EC2::TransitGatewayRouteTable trou
• AWS::EC2::TransitGatewayRouteTableAssociation trou
• AWS::EC2::TransitGatewayRouteTablePropagation trou
• AWS::EC2::Volume comp
• AWS::EC2::VPCEndpoint api-,data,refe,stre,trou
• AWS::EC2::VPCGatewayAttachment api-,comp,data,ec2-,refe,serv,stre,trou
• AWS::ECS::CapacityProvider data,serv,stre,trou
• AWS::ECS::ClusterCapacityProviderAssociations api-,data,refe,serv,stre,trou
• AWS::EFS::FileSystem refe
• AWS::EFS::MountTarget refe
• AWS::EKS::Addon comp
• AWS::ElastiCache::CacheCluster trou
• AWS::ElastiCache::SubnetGroup trou
• AWS::EMR::Cluster trou
• AWS::Events::EventBus stre
• AWS::Glue::Database api-,data,trou
• AWS::Glue::Job trou
• AWS::Glue::Table api-,data,trou
• AWS::IAM::Group refe
• AWS::ImageBuilder::Component refe
• AWS::ImageBuilder::ContainerRecipe refe
• AWS::ImageBuilder::DistributionConfiguration refe
• AWS::ImageBuilder::ImagePipeline refe
• AWS::ImageBuilder::InfrastructureConfiguration refe
• AWS::IoT::Thing stre
• AWS::LakeFormation::DataLakeSettings trou
• AWS::Lambda::Alias serv,trou
• AWS::Lambda::Permission api-,data,refe,serv,stre,trou
• AWS::Lambda::Version serv,trou
• AWS::Lex::Bot refe
• AWS::Lex::BotAlias refe
• AWS::Lex::BotVersion refe
• AWS::Logs::LogStream serv
• AWS::Logs::MetricFilter refe
• AWS::MSK::Cluster stre
• AWS::MSK::Topic stre
• AWS::Neptune::DBCluster stre
• AWS::Neptune::DBInstance stre
• AWS::Neptune::DBSubnetGroup stre
• AWS::NetworkFirewall::Firewall trou
• AWS::NetworkFirewall::FirewallPolicy trou
• AWS::OpenSearchService::Domain api-
• AWS::Redshift::Cluster api-,trou
• AWS::Redshift::ClusterParameterGroup api-,trou
• AWS::Redshift::ClusterSubnetGroup api-,trou
• AWS::RedshiftServerless::Namespace comp
• AWS::RedshiftServerless::Workgroup comp
• AWS::Route53Resolver::FirewallDomainList refe
• AWS::Route53Resolver::FirewallRuleGroup refe
• AWS::Route53Resolver::FirewallRuleGroupAssociation refe
• AWS::Route53Resolver::ResolverEndpoint refe
• AWS::Route53Resolver::ResolverQueryLoggingConfig refe
• AWS::Route53Resolver::ResolverQueryLoggingConfigAssociation refe
• AWS::S3Tables::TableBucket data,stre
• AWS::S3Vectors::Index api-,data
• AWS::S3Vectors::VectorBucket api-,data
• AWS::SageMaker::Domain api-
• AWS::SecretsManager::SecretTargetAttachment data,refe,serv,stre,trou
• AWS::ServiceCatalog::CloudFormationProduct refe
• AWS::ServiceCatalog::LaunchNotificationConstraint refe
• AWS::ServiceCatalog::LaunchRoleConstraint refe
• AWS::ServiceCatalog::LaunchTemplateConstraint refe
• AWS::ServiceCatalog::Portfolio refe
• AWS::ServiceCatalog::PortfolioPrincipalAssociation refe
• AWS::ServiceCatalog::PortfolioProductAssociation refe
• AWS::ServiceCatalog::TagOption refe
• AWS::ServiceCatalog::TagOptionAssociation refe
• AWS::SNS::TopicPolicy refe
• AWS::SSM::Association refe
• AWS::SSM::Document refe,trou
• AWS::Transfer::Server refe
• AWS::Transfer::User refe
• AWS::WAFv2::WebACL data,refe,trou
• AWS::WAFv2::WebACLAssociation trou
• Custom::AWS api-,comp,data,ec2-,refe
• Custom::CDKBucketDeployment api-,comp,data,stre,trou
• Custom::CloudwatchLogResourcePolicy api-
• Custom::LogRetention api-,refe,stre,trou
• Custom::OpenSearchAccessPolicy api-
• Custom::S3AutoDeleteObjects api-,comp,data,refe,serv,stre,trou
• Custom::S3BucketNotifications refe
• Custom::VpcRestrictDefaultSG api-,comp,data,ec2-,refe,serv,trou

Verifier/solution-blocked tasks (8 of 134)

• compute-and-data/amplify-deploy-frontend-from-s3: amplify
• compute-and-data/appstream-stack-with-streaming-vpce: appstream
• compute-and-data/create-medialive-channel: medialive
• compute-and-data/create-s3-tables-with-compaction: s3tables
• compute-and-data/ec2-image-builder-pipeline: imagebuilder
• compute-and-data/kinesis-flink-studio-realtime: kinesisanalyticsv2
• streaming-and-iot/connect-create-customer-profiles: customer-profiles
• streaming-and-iot/iotsitewise-windfarm-rollup-models: iotsitewise

Upstream floci-io/floci overlaps (do not reimplement)

• PR feat(sagemaker): add SageMaker emulation with Docker-backed training floci-io/floci#1949 sagemaker (Docker-backed training) / issue [FEAT] Adding SageMaker Service floci-io/floci#1945
• PR feat(kinesisanalytics): add Managed Service for Apache Flink (Kinesis Analytics V2) floci-io/floci#1914 kinesisanalyticsv2 (Managed Flink) / issue [FEAT] Kinesis Analytics V2 (Amazon Managed Service for Apache Flink) support floci-io/floci#1915 — unblocks kinesis-flink-studio-realtime
• CFN PRs: feat(cloudformation): provision AWS::SecretsManager::SecretTargetAttachment floci-io/floci#1804 SecretTargetAttachment, fix(cloudformation): model AWS::IAM::Policy as an inline policy (Fixes #1531) floci-io/floci#1800/CloudFormation: IAM inline-policy physical name not stack-scoped → cross-stack collision ("Policy … already exists") floci-io/floci#1531 IAM inline policy, feat(cloudformation): provision AWS::Events::EventBus as a real custom bus floci-io/floci#1807/feat(cloudformation): support AWS::Events::EventBus and EventBusPolicy floci-io/floci#1794 Events::EventBus, fix(cloudformation): provision AWS::ApiGatewayV2::Authorizer and wire Route.AuthorizerId floci-io/floci#1760 ApiGatewayV2::Authorizer, feat(cloudformation): resolve dynamic references in RDS credentials floci-io/floci#1809 dynamic refs, [BUG] UpdateStack re-creates existing resources instead of reconciling — rolls back on "log group already exists" floci-io/floci#1965/[BUG] CloudFormation DeleteStack fails with DELETE_FAILED when a resource is already gone, instead of treating it as deleted floci-io/floci#1966 UpdateStack/DeleteStack semantics
• Issue CloudFormation: AWS::IAM::Role inline Policies silently dropped while stack reports CREATE_COMPLETE floci-io/floci#1952 IAM Role inline Policies — claimed, fix branch fix/1952-cfn-iam-inline-policies
• Issue feat: emulate Amazon Bedrock AgentCore (control plane + data-plane invoke) floci-io/floci#1892 (ours) Bedrock AgentCore — fork PR feat(bedrockagentcore): emulate Amazon Bedrock AgentCore (control plane + invoke) #15

Custom:: resources (verify, may already work via Lambda-backed CR flow)

• Custom::AWS
• Custom::CDKBucketDeployment
• Custom::CloudwatchLogResourcePolicy
• Custom::LogRetention
• Custom::OpenSearchAccessPolicy
• Custom::S3AutoDeleteObjects
• Custom::S3BucketNotifications
• Custom::VpcRestrictDefaultSG

Thin service modules needed (no upstream work open)

• s3tables, medialive, imagebuilder, customer-profiles, iotsitewise, amplify, appstream (verifier-blocking, 1 task each)
• redshift + redshift-serverless, efs, codecommit, lex, lakeformation, network-firewall, servicecatalog, route53resolver (scenario-deploy / introspection surface)

Harness

• aws-bench fork: --emulator floci mode (skip org provisioning, inject AWS_ENDPOINT_URL into deploy/agent/verifier)

[lex00]

Filed upstream today:

• fix(cloudformation): create AWS::IAM::Role inline Policies instead of silently dropping them floci-io/floci#1969 — fix(cloudformation): IAM Role inline Policies (fixes upstream CloudFormation: AWS::IAM::Role inline Policies silently dropped while stack reports CREATE_COMPLETE floci-io/floci#1952). Branch fix/1952-cfn-iam-inline-policies, 182 cfn-package tests green.
• [FEAT] CloudFormation: provision AWS::EC2::VPCGatewayAttachment floci-io/floci#1970 — [FEAT] CFN: AWS::EC2::VPCGatewayAttachment (ec2-multiregion batch)
• [FEAT] CloudFormation: provision AWS::EC2::LaunchTemplate floci-io/floci#1971 — [FEAT] CFN: AWS::EC2::LaunchTemplate (ec2-multiregion batch)

Matrix corrected to v2: Floci provision coverage is 75 unique types (v1 double-counted the provision/delete switches), 140 of 211 scenario types missing. IAM::ManagedPolicy and SQS::QueuePolicy were false gaps. ec2-multiregion's real gap is just the two types above + Custom::AWS / Custom::VpcRestrictDefaultSG verification.

[lex00]

Milestone: ec2-multiregion deploys green on the integration build.

All four stacks reach CREATE_COMPLETE via the scenario's real flow (cdk bootstrap x3 regions + cdk deploy --all) against the awsbench-integration image (Docker compose mode, TLS proxy + embedded DNS on). Sanity: 6 EC2 instances across 3 regions, 27 IAM roles, both Custom:: resources (VpcRestrictDefaultSG, CreateAmi AwsCustomResource) executed for real in Docker-backed Lambdas.

Chased down live and filed/fixed along the way (each upstream, each with tests):

• [FEAT] EC2: support DescribeVpnGateways (CDK Vpc.fromLookup fails without it) floci-io/floci#1974 / PR feat(ec2): return an empty set for DescribeVpnGateways floci-io/floci#1975 — EC2 DescribeVpnGateways empty set (CDK Vpc.fromLookup died on it)
• [BUG] CloudFormation: Fn::GetAtt [Vpc, DefaultSecurityGroup] passes through unresolved floci-io/floci#1976 / PR fix(cloudformation): resolve Fn::GetAtt [Vpc, DefaultSecurityGroup] floci-io/floci#1977 — Fn::GetAtt [Vpc, DefaultSecurityGroup] passed through unresolved
• [FEAT] EC2: support CreateImage (create AMI from instance) floci-io/floci#1978 / PR feat(ec2): support CreateImage (create AMI from instance) floci-io/floci#1979 — EC2 CreateImage (Custom::AWS CreateAmi flow)
• PR feat(cloudformation): provision AWS::EC2::LaunchTemplate floci-io/floci#1973 got a second commit: EC2 instances resolve ImageId/InstanceType/KeyName from their LaunchTemplate reference

Run-mode note for the harness: Floci must run as the Docker image (TLS + DNS active). Bare-jar mode starts 'TLS: disabled (HTTP only)' and custom-resource Lambdas fail their HTTPS callback.

Branch awsbench-integration = upstream main + PRs floci-io#1969/floci-io#1972/floci-io#1973/floci-io#1975/floci-io#1977/floci-io#1979, pushed to this fork. Next: merge to fork main to publish the GHCR image, then the aws-bench --emulator floci harness mode against this scenario's 9 quickstart tasks.

[lex00]

First validated baseline row (clean estate, serial, claude-code + claude-haiku-4-5, Claude-judged):

9 tasks, 5 pass, 0 exceptions, 11m33s, $0 AWS, $0 API (subscription OAuth).

Failure triage — every failure root-caused:

• list-ec-private-ips-all-regions → emulator fidelity [BUG] EC2: DescribeInstances reports the Docker bridge IP as PrivateIpAddress, disagreeing with CloudFormation's allocated IP floci-io/floci#1983 (DescribeInstances reports Docker bridge IP; CFN exports report subnet IP — can never agree)
• list-ec-instances-all-regions-1 (SSH reachability) → emulator fidelity [BUG] EC2: every instance reports PublicIpAddress 127.0.0.1, regardless of subnet floci-io/floci#1984 (every instance reports PublicIpAddress 127.0.0.1)
• list-ec-instances-by-vpc-across-regions → genuine model failure: Haiku's summary contradicts its own listing (counts a VPC twice, claims 5 VPCs vs expected 4)
• list-unused-security-groups-all-regions → genuine model failure: correct queries (13× describe-security-groups, 7× describe-network-interfaces; Floci's group-id filter verified honored), then an empty final table — aggregation/finishing slip

Also filed floci-io#1982 (EC2 host-port collision + silent CREATE_COMPLETE on failed instance launch) from the contaminated first run; environment hygiene now = full volume wipe + verify all instances running before benchmarking.

Effective baseline on the 7 Floci-valid tasks: 5/7 (0.714). Both genuine failures are bookkeeping/aggregation errors — precisely the class of failure a typed chant estate view is hypothesized to eliminate. Next: the chant-workspace arm on the same tasks.

[lex00]

Update as of 2026-08-18: 22 of the 140 missing types now covered (fork+upstream union), mostly by merged PRs since filing: EC2 network ACL family, security group ingress/egress, VPCEndpoint, VPCGatewayAttachment, LaunchTemplate, FlowLog, ApiGateway::Account, AutoScaling::LifecycleHook, ECS::CapacityProvider+association, Lambda::Alias/Version/Permission, Events::EventBus, IAM::Group, DocDB::DBCluster/DBInstance, SecretsManager::SecretTargetAttachment, CloudFront::Distribution. 118 still missing. Gap matrix not rerun cohort by cohort, just the type list diff.