The Carlton club house was built in two stages, 1846-48 and 1854-56, by Sydney Smirke in a style similar to the Library of St. Mark's in Venice. Smirke originally won the commission in partnership with George Basevi, with whom he had previously completed the Conservative Club in St. James's Street. However, Basevi died in an accident in 1845 and Smirke was left to complete the job alone. The previous building had been built by his brother, Robert Smirke, and had been the club's first purpose-built clubhouse. The facade is described by the London Survey as a Grecian composition, and was demolished prior to the second building stage, the first being built on neighbouring plots to the west. The club was founded in 1832 as a conservative club for Tory politicians and the first premises were located in Carlton terrace, which is also the origin of the club's name. The building designed by Robert Smirke was completed in 1835, on the corner of Pall Mall and Carlton Gardens, next to the Reform Club.
The building was completely refaced in 1923-24, due to the crumbling Caen stone used on the facade. The facade also had polished red columns and pilasters in Peterhead granite. The ground storey was filled with a rusticated arcade, as opposed to the library of St. Mark's where there is an open loggia, and the doorway was placed in a projecting porch framed with coupled columns. Other deviations from the St. Mark's model were comparably minor. Extensive repairs had been carried out in 1896, but the final solution was to replace the exterior entirely. The new facade was by Reginald Bloomfield and was dressed in Portland stone. The building was struck by bombing on 14 October 1940. The current building was constructed in 1958-59 by architects Donald H. McMorran and Armstrong Smith, and the current premises of the Carlton Club is located at 69 St. James's Street, previously the premises of Arthur's Club (disbanded in 1940).