Maybe you would be interested in a bacteria, rather than a virus? My suggestion would be Brucella abortus, a bacteria which causes Brucellosis.
First, a quick introduction. Brucellosis occurs in two main stages. The first stage is flu-like: fever, sweating, soreness, and GI symptoms are common at this stage. If the brucellosis remains untreated or is treated ineffectually, it can progress to become a chronic infection which can linger for life, as the bacteria infects the intracellular space in the body and is hard to target with antibiotics. The particular strain I am recommending is B. abortus, which occurs mainly in cattle but can infect humans and other animals. This strain also has a low mortality rate, typically around 2%, so you won't be outright murdering (too many) people.
Now, let's go over your wishlist, and I'll discuss what B. abortus gets you and what you would need to modify:
1. make women infertile (ideally permanently)
Notice the name of the bacteria, B. abortus. The main noticeable effect of the disease in cattle is infertility and miscarriage. I am uncertain to what degree this also occurs in humans, but one listed symptom of brucellosis is miscarriage so it's definitely possible. Perhaps apply a little bit of genetic tweaking or selective breeding, and you could easily have a pathogen which makes having children very difficult.
2. spread very rapidly, ideally airborne
Back in the 50s when the US had a bioweapons program, it was found that brucellosis-causing bacteria survive drying very well, and can be effectively spread as an aerosol. The US military even developed a cluster bomb meant to disperse the bacteria as an aerosol, although apparently they weren't happy with how effectively the bomb spread the bacteria.
In addition, the bacteria are easily spread via ingestion. Everyone needs to drink, so infecting the Olympic water supply would likely be equally as effective and possibly more subtle.
The main drawback here is that person-to-person spread is rare, and typically occurs via sexual transmission. You would need a modification here to allow the attack to affect more than just the initially infected population. Given how well the bacteria survive as an aerosol, breeding a strain that can live in the lungs and be spread via coughing/sneezing should do.
3. have a long incubation time, so that it spreads without being found
Typical incubation times are two weeks to a year. B. abortus has got this, no modification needed. As a bonus, the test to determine B. abortus infection takes weeks to culture and poses a significant risk of infection to laboratory workers, so you'll gain both some time and some victims.
4. be contagious as fast as possible
Again, it is typically not contagious human-to-human without sexual contact. But since you'll need to do something about this anyway if you intend to use B. abortus for your story, you may as well say that the aerosolized bacteria is present in the breath of infected persons before symptoms appear, or that the earliest symptoms of the modified form are sneezing and coughing. The same modification that gets you human-to-human transmission in the first place could get you this as well.
5. be hard to cure
I expect this is why you specified a virus in the first place, since we have antibiotics. Hear me out, though: antibiotic resistance. B. abortus is difficult to treat with antibiotics in the first place, requiring multi-week long courses of multiple antibiotics at once. This is because it hides so effectively in the intracellular spaces of the body. Now imagine that your mad scientist had been running a simple selective breeding program for a few years: culture some B. abortus, expose the colonies to an antibiotic at levels too low to kill the entire population, and re-culture the survivors. Repeat with all effective antibiotics (streptomycin, doxycycline, gentamicin, co-trimoxazole, rifampin, etc), and you have a bacterial infection which is untreatable.
It's even better than that, though. The bacteria is not only untreatable at this stage, but hospitals would likely not immediately notice that treatments are not working. Since a typical course of treatment takes between two weeks and a month and a half, it would likely be about that long before doctors noted that the infections were not getting better as expected. This would further delay effective response to the outbreak, and would help ensure widespread chronic infection leading to severe population decline.