Here's one way to change the favicon for your 2013 public site by editing a master page. The same technique can also be used to edit other elements.
First upload a favicon image to an anonymously accessible library (the 2010 image lib or 2013 site assets lib should work).
A favicon can be any image but should be square: 16 x 16 or 32 x 32 would be ideal. It should also be saved as favicon.ico – rename it if you use a jpg or png.
Open the public site's settings: domain.com/_layouts/15/settings.aspx
Download the desired master page from web designer galleries»master page and page layouts
RENAME the file to say "oslo1.master" (important!)
Open it in notepad or other text editor and change the image url in this line to your favicon's url – it's somewhere around line 21 in the Oslo master page:
<SharePoint:SPShortcutIcon runat="server" IconUrl="domain.com/siteimages/favicon.ico" />
Upload it back to the gallery and publish it
In settings go to Look and feel»master page
Select your new master from the 2 drop downs.
An example of the effect can be seen here: http://365new.webbrewers.com/