The Lost Henry M. Flagler Mansion -- 685 Fifth Avenue

In the first years of the 20th century, the Flagler mansion (far left) sat next door to an elegant clubhouse -- NYPL Collection

By 1882, Cleveland-based oil tycoons Henry M. Flagler and John D. Rockefeller were spending more and more time in New York City.  The Standard Oil Corporation was the leader in the American oil refining field.  Within three years the firm’s headquarters would move to New York and in preparation Henry Flagler scouted out a home for his family.

The Fifth Avenue neighborhood just north of St. Patrick’s Cathedral already boasted brownstone mansions and in 1882 Flagler bought the house at 685 Fifth Avenue.  Like Rockefeller, he preferred a comparatively unassuming brownstone to the more opulent mansions making appearances along the avenue.  Located at the southeast corner of 54^th Street it was four stories tall, including the handsome mansard roof, and stretched a luxurious 30 feet wide.  Sitting squarely in what would become Rockefeller and Flagler family territory, the mansion announced Flagler’s financial and social position.

Flagler’s move to New York coincided with a turbulent time in his life.  A year earlier his wife, Mary Harkness Flagler, had died.  As her health had failed, Flagler began a romantic interest in Mary’s caregiver, Ida Alice Shourds, and in 1883 the couple married.   It began the end of his interest in New York and the Fifth Avenue mansion.

Following the wedding Henry and Alice traveled to Saint Augustine, Florida.  While the city and the area were delightful, the accommodations were not.  Flagler offered to buy millionaire Franklin W. Smith’s newly-completed Villa Zorayda—a Moorish-inspired mansion—for his honeymoon.  Smith refused and Flagler walked away with the realization that there was untapped financial potential in Florida real estate.

Although he held his seat on the Board of Directors of Standard Oil, Flagler set off for Saint Augustine in 1885 to begin construction of the 540-room Ponce de Leon Hotel.  And in order to ensure guests could reach his new venture, he bought up railroads—the beginning of the Florida East Coast Railway.

Henry M. Flagler -- NYPL Collection

Because Henry and his wife spent the bulk of their time in Florida, the mansion on Fifth Avenue required little full-time household staff.  Flagler hired an outside cleaning firm to handle tasks normally done by uniformed maids.  In 1890 James E. Garner, the proprietor of Manhattan House Cleaning, noted in his advertisement in Lain’s Business Directory that “the exquisitely furnished house of Mrs. Flagler, 685 Fifth Avenue" was among "the many buildings and dwellings I have cleaned."

While his father was building hotels and railroads, Harry Harkness Flagler remained in New York.  More interested in a gentleman’s lifestyle than one of a businessman, Henry had left Columbia University in 1893 after three years.  Shortly afterward his engagement to Annie Louise Lamont, who lived nearby at 555 Fifth Avenue, was announced.  Annie was one of two daughters of Charles A. Lamont “who left a fortune of several million dollars to his wife and two daughters,” according to The New York Times.

The pair was married in the Madison Avenue Baptist Church on April 25,. 1894.  Published accounts of the wedding do not mention that Harry’s father or stepmother attended the ceremony.  Relations between the father and his only son had already been strained.  Marrying a beautiful young woman with a fortune was, perhaps, not a bad idea for the younger Harry.

The Flaglers had not yet abandoned Fifth Avenue and New York altogether.  On October 6, 1894, The New York Times noted that “Mr. and Mrs. Henry M. Flagler are at their home, 685 Fifth Avenue, having closed their cottage at Mamaroneck-on-the-Sound.”

The tensions between father and son boiled over in 1897.  Flagler had his wife committed as mentally incapacitated that year.  Soon after he used his influence to have insanity added to state law as grounds for divorce in Florida.  A divorce was granted in August 1901 and ten days later Flagler married Mary Lily Kenan, reportedly the niece of his doctor.  The bride was about 40 years younger than the 67 year old groom.  The New York Times said, quite frankly, “When the statute under which the divorce was granted was, as a bill, introduced in the Florida Legislature, it was charged that it was drawn solely in the interest of Henry M. Flagler so as to permit him to cast off the insane wife and wed a younger woman.”

It was the last straw in a series of events that infuriated the tycoon’s son.  Harry Harkness Flagler vowed never to speak to his father thereafter.

Even while the domestic drama was brewing, it seemed in March 1900 that perhaps Henry M. Flagler was renewing his interest in New York.  The New York Times noted that he bought the country estate of J. T. Tower at Millbrook, New York “consisting of a mansion and a tract of about eighty acres.”   Two weeks later that illusion was dashed.

On April 14 the newspaper reported that “Henry M. Flagler, according to a report current yesterday, has sold his house 685 Fifth Avenue, southeast corner of Fifty-fourth Street.  Mr. Flagler, it is said, will in the future spend very little time in this city—only a few weeks out of the year—and has on this account deemed it advisable to dispose of his home.”

The comparatively staid Flagler house on the corner stood in start contrast to the opulent Criterion Club next door.  photograph "Both Sides of Fifth Avenue" J. F. L. Collins, 1911 (copyright expired)

The mogul’s separation from the city was underscored on October 9 when The New York Times reported that “Henry M. Flagler, the New York millionaire who has opened up the east coast of Florida by dotting it with hotels and by building a railroad system connecting all of the principal towns and orange shipping points, has announced his citizenship in Florida…He arrived from the East yesterday, and will in the future spend the greater part of his time in the State.”

The Fifth Avenue house was purchased by a Flagler relative, Charles W. Harkness, for $300,000 – about $7 million today.  By now Manhattan was dotted with mansions built by Standard Oil money—two Rockefeller mansions on West 54^th Street and the William Rockefeller mansion at 689 Fifth Avenue; the residences of William and Edward Harkness; and Henry H. Flager's home on Park Avenue.

A near tragedy occurred on January 24, 1909 after Charles Harkness was dropped off at home of his brother, Edward, at Fifth Avenue and 75^th Street.   His chauffeur, Barth Anderson, apparently took the opportunity to go to his home at 257 West 129^th Street.    At the corner of Seventh Avenue and 129^th Street, the limousine struck 8-year old Theresa Jaeger.  Although, after examination by a doctor, the little girl was deemed “not mortally injured” and was taken home, The New York Times took advantage of the opportunity to run the headline “Machine Owned by Standard Oil Man Runs Down Eight-Year-Old Girl.”

Charles Harkness died on May 1, 1916, and a year later the house was acquired by millionaire lawyer Samuel Untermyer.   The New York Times noted that “Mr. Untermyer owns 'Graystone,’ the palatial estate in Yonkers, which was formerly the home of Samuel J. Tilden.”

Although the Fifth Avenue neighborhood was becoming increasingly commercial, Untermyer lived on in the outdated brownstone mansion for years--despite repeated rumors that it would be demolished.  On October 1, 1924, The New York Times reported that “the city home of Samuel Untermyer” had been leased to the No. 685 Fifth Avenue Corporation “as a site for a business building.” 

Samuel Untermyer was quick to refute the article.  In a letter to the editor published two days later, the attorney said called the details of the article “new to me,” and protested “I herewith request that you give equal prominence to this denial of that alleged transaction.”

Finally, in 1926 Untermyer gave in.  He leased the land to the Midi Realty Corporation for 21 years with three renewals.   Before the end of the year a renovation by the architectural firm of Cohen & Siegel resulted in the parlor floor becoming a “restaurant with dancing.”  Kitchens were in the basement and all floors above were deemed by the Department of Buildings “not to be used.”

A 17-story building (left) replaced the Flagler mansion -- NYPL Collection

Empty floors and a restaurant in a building sitting on valuable Midtown real estate made no financial sense.  Within two years the Flagler mansion was gone, replaced by a 17-story office building that included a magnificent two-story penthouse.

The building survives today with a substantially-altered base.  photo - cpcexecutive.com