This is one of those that feels like it should be easier than it ends up being, but here is where I landed at to ensure a modal image preserves its 3/2 aspect ratio no mater the viewport width/height.

.modal {
  aspect-ratio: 3 / 2;
  height: min(calc(100vh - 2rem), calc((100vw - 2rem) * 2 / 3));
  margin: auto;
  position: relative;
}

See it in action on I Brew My Own Coffe.