VPC firewall: a firewall rule on a new VPC public IP before any PF/LB rule prevents the IP from being configured on the VR #14669

problem

With the Firewall service for VPC public IPs (#12706), a firewall rule can be created on a
public IP of a VPC before the IP has any port forwarding, load balancing or static NAT rule. In
that order the IP is never configured on the VPC virtual router:

1. IpAddressManagerImpl.applyRules resolves only the VPC for a VPC firewall rule
   (networkId is null), skips the IP association and logs
   Unable to resolve network for firewall rules (networkId=null, vpcId=...); skipping IP association.
   The firewall rules are sent to the VR anyway and the FIREWALL_<ip> chain is created.
2. The first LB or PF rule on the IP is applied afterwards, but no IpAssocVpcCommand is sent
   for it. The IP never appears in the ips data bag of the VR and is not added to the public
   interface. Clients on the same public subnet get No route to host, others time out.
3. When the LB/PF rules are applied, the VR also deletes the previously added firewall rules
   from FIREWALL_<ip> (Delete rule -D FIREWALL_<ip> ... in /var/log/cloud.log), so only the
   final DROP remains. Re-applying the firewall rules restores them, but the IP is still not on
   the VR. (On a correctly associated IP, re-applying PF rules during a CKS scale kept the
   firewall rules.)

With the opposite order (LB/PF first, firewall rules afterwards) everything works: the LB/PF
apply sends IpAssocVpcCommand, the VR runs ip addr add dev eth1 <ip>/<mask>, and later
firewall rules take effect immediately. The skip in step 1 is harmless then, because the IP is
already associated.

Observed on 4.23.0.0 with a VPC offering that has the Firewall service (provider
VpcVirtualRouter), same VPC, same tier, same steps except for the order:

In the "before" case the management server also logged
Merovingian2: Was unable to find lock for the key user_ip_address<id> while applying the LB
rule on that IP.

A side observation: after an IP with firewall rules is released, its FIREWALL_<ip> chain stays
on the VR (with RELATED/ESTABLISHED and DROP), and the PREROUTING -d <ip> -j FIREWALL_<ip>
jump stays as well. When the IP is acquired again in the same VPC the chain is reused.

versions

• Apache CloudStack 4.23.0.0; IpAddressManagerImpl.applyRules is unchanged on main
  (a7be64a)
• KVM, advanced zone, VPC offering with the Firewall service (provider VpcVirtualRouter),
  SystemVM template for 4.23 on the VR

The steps to reproduce the bug

1. In a VPC with the Firewall service, get a new public IP that is associated with the VPC and
   with a tier but has no rules yet. We saw it with the IP that CKS acquires for a cluster
   (associateIPToVpc + associateIPToGuestNetwork); a plain associateIpAddress vpcid=...
   was not tested.
2. Create a firewall rule on that IP, for example TCP 22 from your address.
3. Create a port forwarding (or load balancing) rule on the same IP to a VM in the tier.
4. The port is not reachable. ip addr on the VR (or getDiagnosticsData files=ipaddr) does not
   show the IP, and the job that applied the port forwarding rule sent no IpAssocVpcCommand.

Doing step 3 before step 2 works.

What to do about it?

When firewall rules are applied for a VPC public IP that is not yet associated with the VR,
associate it first, as applyRules does for network-scoped rules (resolve the network from
the IP's associated network or use the VPC-level IP association). The VR should not drop
firewall rules of an IP when PF/LB rules for that IP are applied. Released IPs should have their
firewall chain removed.

Found while fixing CKS for VPCs with the Firewall service (#14668), whose fix creates the
firewall rules after the LB/PF rules to avoid this.

Related, but a different problem in the same feature: #14068 (NPE when destroying a VPC tier
with the Firewall service, PR #14133).