RSS
 

Posts Tagged ‘requests’

Restoring Facebook’s Birthday Calendar Export Feature (fb2cal)

31 Jul 2019

Context

Around 20 June 2019, Facebook removed their Facebook Birthday ics export option.
This change was unannounced and no reason was ever released.

As a heavy user of this feature I was very upset. I use the birthday export feature to be reminded of upcoming birthdays so I can congratulations friends and family. After it became clear that is was not a mistake I decided to write my own scraping tool to restore this functionality for personal use.

This posts includes some of my findings on how I did this.

Where can I find the tool?

If you are simply after the tool, I’ve opensourced it on Github:
https://github.com/mobeigi/fb2cal

Initial Research

I was sure a scraping solution would work as Facebook still displayed all your friends Birthdays at the /events/birthdays page which is located here
https://facebook.com/events/birthdays/ .

Facebook Birthday Events Page

The friend ‘bubbles’ are grouped by the birthday month. Upon hovering over a user a tooltip with the format Friend Name (DD/MM) is shown which reveals the friends birthday day and month which is all the data we need for our calendar.

Facebook Birthday Hover Tooltip

Scrolling down on the page would dynamically load in more birthday month groups which means AJAX endpoints were being called. Using Chrome Developer Tools I can easily monitor outgoing XHR network requests as I scroll down and trigger the AJAX calls I’d like to replicate.

Chrome Dev Tools XHR Monitor

 

Querying the Birthday Monthly Card AJAX Endpoint

The Birthday Monthly Card AJAX endpoint we found is responsible for returning the HTML that powers the monthly grouped bubbles pictured above.
We end up with this GET AJAX query (some query parameters have been redacted/snipped):

 

After some trial and error we notice that the endpoint still returns a valid response as long as we include the following three query parameters: date , fb_dtsg_ag and __a.
Required parameter explanations:

  • The  date parameter is an epoch timestamp. The month that the epoch lands in is the month that will be used for the response. So we can pass in any epoch within a particular month to get a response for that month.
  • The  fb_dtsg_ag parameter is an async token (CSRF protection token). This token seems to have a lifetime of 24 hours and can be reused between subsequent AJAX requests. It is stored in the source code of the same  /events/birthdays page so we can scrape it from there and pass it alongside our AJAX requests.
    "async_get_token":"AQxSD4ZC6HFv74axgbaCIcvRTKp29fSPxI3puZLEFiGfAQ:AQx1pSd8-cFSM6eRRV-VOQ4z_Bc9Hjp_dMYAuRIbhz9sgg"},3515]
  • The __a parameter seems to be a generic action parameter and must be set to 1.

 

The response from the endpoint looks a little like this:

Facebook likes to prefix all AJAX responses with for(;;);  as a security measure to prevent JSON hijacking. We can strip this away from the response. The rest of the response is a valid JSON object which we can parse.
We have a lot of useful information in this payload including:

  • Friends Full Name (in the  alt and  data-tooltip-content  fields)
  • Friends Birthday month/day (in the data-tooltip-content  field)
  • Link to friends Facebook profile page revealing their vanity_name or profile_id (in the href  field)
  • Link to Facebook profile display picture (in the img  src  field)

 

We will need all of this information to generate our calendar .ics file except for the Facebook profile display picture.

 

Parsing the data-tooltip-content

The data-tooltip-content is in the following format (for myself using Facebook locale en_UK ):

There are various problems here! The format is based on the current Facebook users locale.
In other words, the format will change based on the Facebook users selected language.
The date format as well as the ordering of Name/Date can change.

The solution here was to query another AJAX endpoint ( https://www.facebook.com/ajax/settings/language/account.php ) to retrieve the users locale. Each locale was then mapped to a date format. Finally, we strip away the users name, brackets, right-to-left mark, left-to-right mark and various other unicode characters leaving only the birthday day and birthday month with some separator character in between. It then becomes easy to parse the date using the locale to date format mapping.

Another issue was that Facebook would replace the date with a day name if the Birthday for the friend occurred in the next 7 days relative to the current date (and not the passed in epoch timestamp). So for example if today is the 01/01 and a friends birthday was the next day and that day was a Tuesday, the tooltip content would show  John Smith (Tuesday) . This logic was easy enough to add once the issue was discovered.

Getting the Facebook entity id

Our calendar .ics file will need a UID (unique identifier) for each friends Birthday event. Otherwise, every time the .ics file is imported into a calendar, duplicate events will be created which is not what we want as we would like to automate the updating process. The obvious candidate for a unique identifier is a Facebook users entity id which is unique per Facebook user, page etc. Vanity names are also unique on Facebook but we decided to not use them as they can change unlike entity ids. Unfortunately, our payload from the Birthday Monthly Card AJAX endpoint does not contain the entity id for every friend. Instead we get a URL to their profile page.

If a friend does not have a vanity name (custom profile page url) setup then we can simply extract the entity id from the id field:

Otherwise, the problem becomes much more difficult as I did not easily find an endpoint which takes in a vanity name and returns a unique identifier.
The best solution that was discovered was querying the COMPOSER QUERY endpoint ( https://www.facebook.com/ajax/mercury/composer_query.php and passing in the following query parameters: value , fb_dtsg_ag and __a. The  value parameters is your search query which in our case is the vanity name.

This endpoint is naturally used on Facebook when you are searching for a person to message.

Facebook Composer Endpoint Example Query

This is a typical response for the search term Vanity:

Note that the search results can return multiple matching entity results including users, pages, apps etc. However, we are guaranteed that our Facebook friend/user with the matching vanity name will appear in the list somewhere. All we have to do here is compare our vanity_name from before with the alias field in the composer query response payload. The matching entry is thus our Facebook friend and we can take the corresponding UID directly off the JSON object.

A consequence of this approach is that we now must perform 1 lookup per Facebook friend to get their entity id. This slows down the script significantly. However, no better solution was found. Third party websites such as findmyfbid.com scrape the users profile page directly to retrieve the entity id from the source code but this approach was profiled as being slower (including via mobile version of Facebook).

Rate limiting note: If the composer query endpoint is hit enough in a short period of times, it seems to somehow restrict the number of entries returned. It seems to limit the query results to only returning results matching friends names, page names exactly. This limitation then disappears over time. So one should be careful how often or quickly they hit this endpoint!

Generating our Calendar ICS File

At this stage, we simply query the Birthday Monthly Card AJAX endpoint passing in epoch timestamps belonging to the first day of every month for a full year (12 months total) and storing the results. We should have all the required fields needed for each friend (Birthday day, Birthday month, Name and UID) so we can generate our Calendar ICS file. This file can then be automatically pushed to the cloud or stored on the local file system for importing into third party applications like Google Calendar.

Final Remarks

This was a fun little project to undertake and a minimum viable product was produced in only a few hours which was nice. It is important to note that these methods do bypass the Facebook API entirely and as a result are against the Facebook TOS. However, as somebody who really wants to tell their friends HAPPY BIRTHDAY! and desperately needs reminders, it was a sin that had to be committed.

Happy Birthday with Cupcakes

 
8 Comments

Posted in Programming