Neighbors originally called the structure "Ward's Folly" thinking it would collapse but later called it "Ward's Castle" after witnessing its durability.

The 17-room mansion was constructed entirely of concrete, utilizing 4,000 barrels of Portland cement, 8,000 barrels of sand, 12,000 barrels of machine-crushed North River bluestone, and 12,000 barrels of white beach pebbles. The concrete was reinforced with small, lightweight iron I-beams ranging from five to eight inches (13-20 cm) and 3/8-inch (1 cm) round iron rods. In preparing the concrete, William E. Ward followed the technique of French industrialist François Coignet, making the mixture very dry and tamping it into place.

Architect Robert Mook's design accentuated the use of concrete as a building material. No attempt was made to disguise the reinforced concrete, but rather to stress its character as a building material. The structure is dominated by a four-story, castle-like octagonal tower at one corner and the outside walls employ different colors of concrete decorated with graffito work. Only the doors, window frames, and paneling were built with wood. The mansard roofs are made out of solid concrete and hollow spaces in the floors and walls were connected to the furnace to provide a central heating system. The large imposing structure is locally called Ward's Castle, although when it was first built neighbors called it "Ward's Folly" because of their lack of confidence in the new building material.

Cartoonist Mort Walker, Ed Cohen, Mike Pender, and Winifred Ward (granddaughter of the original owner) at the Ward House dedication ceremony.

In building the house, William E. Ward wanted to test the idea of building a concrete house through an actual experiment without contracting out the work (he supervised all of the construction and only used his own workers). After conceiving the idea to build a concrete house in 1870, Ward conducted preliminary deflection tests with reinforced concrete in 1871 and 1872. These experiments determined the optimal size of aggregate and showed that iron reinforcements should be placed at the bottom of the beam to resist strain below the neutral axis. After constructing the house, Ward tested the strength of the floor by placing a 26 ton (24 t) weight between the two central floor beams and recorded a deflection of just one hundredth of an inch (0.3 mm).

A few years after completion of his house, William E. Ward authored a most informative paper on its construction which appeared in Transactions of the American Society of Mechanical Engineers, 1882-1883, with the title, "Béton in Combination with Iron as a Building Material." This paper is recommended reading for those interested in the early development of reinforced concrete construction. The Ward House was a milestone in structural engineering. A new type of engineered construction had made its debut which today is universally employed. The structure demonstrated the practicability of reinforced concrete as a construction material for fireproof permanent structures, stimulating an entire new field for civil engineers. Another inspiration for Ward to create a fireproof residence was that his mother suffered from pyrophobia.

The building served as the Ward's family home for four generations and was listed on the National Register of Historic Places in 1976. The eight acre (3.2 ha) parcel of land was subdivided for residential development and Ward's Castle was sold to world-renowned comic artist Mort Walker, creator of Beetle Bailey and Hi and Lois who converted it, without alteration to the Museum of Cartoon Art in 1977. The museum had a collection of more than 25,000 original pieces of artwork, including an original painting of The Yellow Kid, the first comic strip character in the United States.

A bronze plaque designating the Ward House as a National Historic Civil and Concrete Engineering Landmark was unveiled on May 20, 1978.

The Ward House was jointly designated as a National Historic Civil and Concrete Engineering Landmark by ASCE and the American Concrete Institute (ACI) in 1977. A bronze plaque was unveiled by officers of the ASCE and ACI in a ceremony on May 20, 1978 on the grounds of the Ward House. Ray C. Heun of the New York Concrete Construction Institute and David Caplan, Chairman of the ASCE Met Section's History & Heritage Committee, arranged and conducted the ceremony. Michael R. Pender, President of the Met Section represented ASCE and Edward Cohen (also the ASCE Met Section Vice President) represented the American Concrete Institute. Miss Winifred Ward, granddaughter of the builder addressed the attendees relating some historical highlights of the building and its residents, the prominent Ward family. Mort Walker spoke about the renovation of the building and the museum's valuable collection.

The Museum of Cartoon Art was relocated to Boca Raton, Florida in 1992 after which Ward's Castle again became a private residence.