The Boy Charlton Pool, or ABC pool as I shall call it to ease my typing, is a Summer only pool. However there are 4 other pools in Sydney run by the city that you can use. 2 more outdoor and 2 indoor.
Stolen directly from City of Sydney Pools detail as it is a Better summary than I could provide:
Aquatic centres in central Sydney
Sydneysiders love the water and the City of Sydney proudly provides the community with 5 outstanding aquatic centres. While each centre is different, they all boast 50-metre pools for lap swimming as well as a smaller pool for learners and leisure seekers. The stunning Andrew (Boy) Charlton Pool is located on the edge of the harbour and is famous for hosting its namesake's world-record races in the 1920s. During the 2012 off-season it was closed for a major renovation and it now looks better than ever. Cook + Phillip Park is partially underground and was designed to make the most of the natural light that streams in from outside. Its pools are indoors and open all year round. The new Prince Alfred Park Pool was opened on 13 May 2013. The pool is the first fully accessible, heated outdoor pool in the local area. Year round opening hours are being trialed until 2015.
The Ian Thorpe Aquatic Centre, named after the Olympic gold medallist, is a standout example of contemporary architecture. Designed by Harry Seidler and Associates, the building is shaped like a wave and cost $40 million to construct. It opened in 2007. The Victoria Park Pool was the first in-ground public pool to be built in Sydney when demand for such facilities boomed in the 1950s and 1960s. Opened in 1953, the pool has been refurbished many times but the original building has been preserved. All that and more Each aquatic centre offers a range of other facilities and services, including full-size gyms, indoor courts, fitness classes, massage/physiotherapy treatments and cafes.
Cook & Phillip is a good winter option as it is indoors, close to CBD or city hotels for any visitors. Can be a positive or a negative depending on your situation, but there are a lot of little kids at the Cook & Phillip pool on the weekends. It has a massive kid pool area. So if you have children, great. But if not, be prepared for a lot of children about. Not to be a Grinch about it, just pointing it out.
Both Victoria Park (outdoor) and the Ian Thorpe Aquatic Centre (indoor and has a spa/sauna) are other options, but I have not been to either so I cannot opine on their greatness or improvements.
Have a great Swim!