The executive committee reversed its support for the Swiss football administrator in a meeting on September 25 after endorsing his candidacy in June.
"The decision follows a comprehensive reassessment of developments over the past few months," the SFV said in a statement.
"From the SFV's perspective, various incidents raise fundamental questions regarding leadership, governance, transparency and decision-making processes within FIFA," it added.
Infantino, who is seeking re-election in March, has faced growing opposition since his bid in July to sell a stake in FIFA tournaments, including the World Cup, which he was forced to abandon.
The withdrawal is the latest blow to Infantino, with football associations in England, Wales, France, Serbia and other European countries having already pulled their endorsements ahead of the election. UEFA has pledged to present a rival candidate before the November deadline.
The 56-year-old Infantino, who has been FIFA president since 2016, has faced calls for his resignation amid widespread pleas for reform in global football's governing body.
Infantino wrote this month in a letter to member associations that he was open to talks over reform, but the SFV said key questions regarding FIFA's responsibilities and decision-making process had been left unanswered.
"As these have not been convincingly clarified, we cannot justify continuing to support Gianni Infantino’s candidacy,” the SFV added.