Shares of brokerage and wealth management companies dropped Friday amid investor fears about the economic impact of tariffs.