Former city council adviser Max Caller said the Games shifted focus away from issues amid the council’s legacy of financial problems