Minutes from the Fed's January meeting seemed to catch investors by surprise when they were released at 2 p.m. EST. Several Fed policymakers worried that the bank's program of buying $85 billion of bonds each month could eventually unsettle financial markets or cause the bank to take losses. Even so, most of the Fed officials thought the economy faced fewer risks than in December.
Judging by the market's reaction, the Fed appears to be closer to ending its support for the U.S. economy than traders had expected, said Dan Greenhaus, chief global strategist at the brokerage BTIG. "We're at a point now where we're discussing how we're going to end this, not whether it's going to end," he said.
The S&P 500 index sank 18.99 points to 1,511.95, a loss off 1.2 percent. That's the biggest one-day drop since Nov. 14, 2012.
By buying bonds, the Fed drives up their prices and lowers interest rates, which have stayed at record lows. That keeps costs low for mortgages and other types of loans.
The major indexes drifted sideways in morning trading then turned lower in the early afternoon after Caterpillar reported weaker sales of its heavy trucks and mining equipment. Stocks fell further after traders had time to digest the Fed minutes. The S&P 500 lost 11 points in the last hour and a half of trading.
The Dow Jones industrial average fell 108 points, or less than 1 percent, to close at 13,927. Merck helped curb the Dow's fall, rising 1 percent, on news that it teamed up with a Korean drugmaker to create drugs.
The Nasdaq composite fell 49 points, or 1.5 percent, to 3,164.
News that Apple's major supplier, Foxconn, stopped hiring at its largest plant in China helped push down Apple's stock. Foxconn reportedly said the hiring freeze was not caused by slumping orders for iPhones. Apple fell $11.14 to $448.85.
The stock market surged at the start of the year then drifted slightly higher in recent weeks with few major events to drive trading one way or another. That could change as soon as Congress returns from vacation next Monday. Deep federal spending cuts are scheduled to start March 1 unless Congress and the White House find a way to avoid them.
Both the Dow and the S&P 500 have gained 6 percent for the year. The Nasdaq is up 5 percent.
Before the Fed minutes came out, Phil Orlando, the chief market strategist at Federated Investors, said he believed the stock market had climbed too quickly and was prone to a big drop. He expected the S&P 500 to get knocked down by 3 percent or more in the coming weeks. Another budget battle in Washington could be the trigger.
"There are a lot of us who say, 'We're a little bit ahead of ourselves here,'" Orlando said. "I still expect an all-time high for the S&P 500 this year, but it's going to get there in fits and starts."
Even though housing construction slowed down in January, the Department of Commerce reported Wednesday that new housing starts remained strong. Builders started construction at an annual rate of 890,000 last month, down 8.5 percent from December. Applications for building permits increased.
The Dow closed at its highest level of the year Tuesday, bringing it within one percent of 14,164, the record high reached more than five years ago.
In the U.S. government bond market, the yield on the 10-year Treasury note slipped to 2.01 percent from 2.03 percent late Tuesday. The yield has climbed steadily higher since the start of the year, when it traded around 1.70 percent.
Among companies making moves:
— GPS device maker Garmin slumped 9 percent, the biggest drop in the S&P 500 index, after the company's results missed analysts' forecasts. Demand has waned for handheld navigation devices as more customers use maps on their smartphones. Garmin lost $3.70 to $35.54.
— Food giant ConAgra gained 20 cents to $33.65 after it raised its profit forecast for the year. The company, whose brands include Chef Boyardee, said its acquisition of Ralcorp will add a nickel per share to adjusted earnings this year.